settings.py配置redis
SCHEDULER = "scrapy_redis.scheduler.Scheduler"
SCHEDULER_PERSIST = True
SCHEDULER_QUEUE_CLASS = 'scrapy_redis.queue.SpiderPriorityQueue'
REDIS_URL = None # 一般情况可以省去
REDIS_HOST = '127.0.0.1' # 也可以根据情况改成 localhost
REDIS_PORT = 6379
在scrapy中使用scrapy-redis
spider 继承RedisSpider
class tempSpider(RedisSpider)
name = "temp"
redis_key = ''temp:start_url"
