一、Scrapy-Redis框架
github 传送门: https://github.com/rmax/scrapy-redis
git clone https://github.com/rmax/scrapy-redis.git
1、安装环境
Python 2.7, 3.4 or 3.5
Redis >= 2.8
Scrapy >= 1.1
redis-py >= 2.10
2、Settings文件配置
Enables scheduling storing requests queue in redis.
SCHEDULER = "scrapy_redis.scheduler.Scheduler"
Ensure all spiders share same duplicates filter through redis.
DUPEFILTER_CLASS = "scrapy_redis.dupefilter.RFPDupeFilter"
Store scraped item in redis for post-processing.(非必须)
ITEM_PIPELINES = {
'scrapy_redis.pipelines.RedisPipeline': 300
}
3、举个栗子
1)spider genspider myspider www.abc.com
from scrapy_redis.spiders import RedisSpider
class MySpider(RedisSpider):
name = 'myspider'
def parse(self, response):
# do stuff
pass
2)spider genspider -t crawl myspider www.abc.com
class MySpider(RedisCrawlSpider):
rules = (
Rule(LinkExtractor(), callback='parse_item'),
)
def parse_item(self, response):
# do stuff
pass
