前言
scrapy爬取网站数据的时候，一般第一次爬取为全量爬取，以后需要的都是增量爬取，或者爬取中断之后需要继续爬取，那么这都需要爬取剩余未爬取的，而已经爬取过的则不需要爬取。为了提高爬取效率，已经爬取过的地址最好通过判断是否爬取，如果爬取过则丢弃，否则交给调度器，由调度器安排爬取。
image.png
根据爬虫框架的结构图可知，scrapy中有两个重要的中间件，一个是Downloader Middlewares一个是Spider Middlewares 在spiders中yield scrapy.Request()的请求都会经过spiderMiddlewares。查看官方文档关于scrapy.contrib.spidermiddleware.SpiderMiddleware的process_spider_output(response, result, spider)方法的介绍：
当Spider处理response返回result时，该方法被调用
可知在spider的parse方法中yied的item对象和request对象都会调用该方法。那么增量爬虫的判断是否爬取过，如果爬取过则丢弃，否则交给调度器，这一功能可在此实现。本篇在不改变原来spider的基础上，通过中间件实现增量爬虫。
具体实现
步骤一、
新建数据库操作文件db.py实现的功能：
- mysql数据库的配置信息
- 根据origin_url字段判断url在数据库中是否已经存在
- 管道中使用的插入数据方法
- 为避免sql查询时的数据库连接的反复建立，使用单例模式
#!/usr/bin/env python
# -*- encoding: utf-8 -*-
import pymysql
import logging
class DB_MySQL():
'''数据库操作类'''
HOST = 'localhost'
DBNAME = 'hebei'
USER = 'root'
PASSWD = '123456'
PORT = '3306'
CHARSET = 'utf8'
def __init__(self):
self.conn = pymysql.connect(host=self.HOST, port=int(self.PORT), user=self.USER, passwd=self.PASSWD,
db=self.DBNAME, charset=self.CHARSET)
self.cur = self.conn.cursor()
# 插入数据
def insert(self, item):
try:
fields = item.keys()
sql = 'insert into news(%s) value(%s)' % (','.join(fields), ','.join(['%s']*len(fields)))
self.cur.execute(sql,[item[x] for x in fields])
self.conn.commit()
except Exception as e:
logging.error('mysql插入数据执行异常: %s' % str(e))
# 判断url是否已经存在
def url_is_exist(self, url):
try:
if self.cur.execute('select 1 from news where origin_url = %s limit 1', (url,)):
return True
else:
return False
except Exception as e:
logging.error('mysql查询origin_url是否存在执行异常: ' + str(e))
def close(self):
self.cur.close()
self.conn.close()
db_mysql = DB_MySQL()
中间件实现
process_spider_output()方法中需要先判断对象是否为Request对象，然后获取该对象的url属性，并判断该url是否已经存在，如果存在则yield None。
class HbPolicyNewsSpiderMiddleware(object):
# Not all methods need to be defined. If a method is not defined,
# scrapy acts as if the spider middleware does not modify the
# passed objects.
@classmethod
def from_crawler(cls, crawler):
# This method is used by Scrapy to create your spiders.
s = cls()
crawler.signals.connect(s.spider_opened, signal=signals.spider_opened)
return s
def process_spider_input(self, response, spider):
# Called for each response that goes through the spider
# middleware and into the spider.
# Should return None or raise an exception.
return None
def process_spider_output(self, response, result, spider):
# Called with the results returned from the Spider, after
# it has processed the response.
# Must return an iterable of Request, dict or Item objects.
for i in result:
if isinstance(i, Request):
referer = i.headers[b'Referer'] if b'Referer' in i.headers.keys() else ''
if db_mysql.url_is_exist(i.url):
spider.logger.debug('url已存在丢弃请求：%s ,referer信息： %s' % (i.url, referer))
yield None
else:
spider.logger.debug('新url请求：%s ,referer信息： %s' % (i.url, referer))
yield i
else:
yield i
def process_spider_exception(self, response, exception, spider):
# Called when a spider or process_spider_input() method
# (from other spider middleware) raises an exception.
# Should return either None or an iterable of Response, dict
# or Item objects.
pass
def process_start_requests(self, start_requests, spider):
# Called with the start requests of the spider, and works
# similarly to the process_spider_output() method, except
# that it doesn’t have a response associated.
# Must return only requests (not items).
for r in start_requests:
yield r
def spider_opened(self, spider):
spider.logger.info('Spider opened: %s' % spider.name)
settings.py中启用中间件
SPIDER_MIDDLEWARES = {
'hb_policy_news.middlewares.HbPolicyNewsSpiderMiddleware': 543,
}
