安装
- 安装库
pip install scrapy-redis
- 安装redis
配置
在setting.py文件中配置
SCHEDULER = "scrapy_redis.scheduler.Scheduler"
SCHEDULER_PERSIST = True
SCHEDULER_QUEUE_CLASS = 'scrapy_redis.queue.SpiderQueue'
DUPEFILTER_CLASS = "scrapy_redis.dupefilter.RFPDupeFilter"
REDIS_HOST = 'localhost'
REDIS_PORT = 6379
编写spider
class MySpider(RedisSpider):
name = "myspider"
start_urls = ["http://weather.sina.com.cn/", ]
redis_key = "myspider:start_urls"
# 注意一定要重写start_requests,否则出不来数据
def start_requests(self):
for url in self.start_urls:
yield Request(url=url, callback=self.parse)
def parse(self, response):
.....
上述中一定要注意重写start_requests，不然一直没数据出来，这是本人走过的坑，希望能帮到您，如果你觉得这篇文章对你有帮助，麻烦点个赞哦！
