服务端 scrapyd
1 环境安装
sudo apt update -y
sudo apt install -y build-essential libssl-dev libffi-dev libxml2 libxml2-dev libxslt1-dev zlib1g-dev python-dev
sudo pip install -y scrapy redis python_redis pymongo scrapyd
2 配置
find / -name scrapyd
bind 0.0.0.0
3 运行
scrapyd
客户端 scrapyd-client
1 web端查看
http://localhost:6800
2 打包上传
scrapyd-deploy
3 调度
curl http://localhost:6800/schedule.json -d project=zhihuuser -d spider=zhihu
4 取消
curl http:/localhost:6800/cancel.json -d project=zhihuuser -d job=51e8b2c0602011e9b97c000c2989fdcb
