还在用BeautifulSoup写爬虫?out了! 用lxml&xpath!
从上一篇python网络爬虫-爬取网页的三种方式(1) 我们知道爬取网页内容的方式有三种分别是:正则表达式、BeautifulSoup以及lxml的CSS selector 和xpath。
在上篇的代码测试中对网页内容的爬取,四种方式几乎同时显示爬取结果。由于爬取数据量有限,因此无法对比它们的性能。本篇将对比它们爬取大量数据时的性能。
目标:爬取网页所有显示内容,重复1000次,比较不同爬虫的爬取时间。
准备:从Python网络爬虫-你的第一个爬虫(requests库)中导入网页爬取函数download。
从python网络爬虫-爬取网页的三种方式(1)导入所有爬虫。
开始coding:
import time
import re
from chp2_all_scrapers import re_scraper, bs_scraper, lxml_scraper, lxml_xpath_scraper
from chp1_download import download
num_iterations = 1000 # 每个爬虫测试1000次
html = download('http://example.webscraping.com/places/default/view/Australia-14')
scrapers = [('re', re_scraper), ('bs', bs_scraper), ('lxml', lxml_scraper),
('lxml_xpath', lxml_xpath_scraper)]
for name, scraper in scrapers:
start_time = time.time()
for i in range(num_iterations):
if scraper == re_scraper:
re.purge() #清除缓存,保证公平性
result = scraper(html)
# 检测结果是否为想要的
assert result['area'] == '7,686,850 square kilometres'
end_time = time.time()
print("=================================================================")
print('%s: %.2f seconds' % (name, end_time - start_time))
请注意:re.purge() 是为了清除缓存。因为正则表达式默认情况下会缓存搜索,需要对其清除以保证公平性。如果不清除缓存,后面的999次都是在作弊,结果惊人!
运行结果:
Downloading: http://example.webscraping.com/places/default/view/Australia-14
=================================================================
re: 2.40 seconds
=================================================================
bs: 18.49 seconds
=================================================================
lxml: 3.82 seconds
=================================================================
lxml_xpath: 1.53 seconds
Process finished with exit code 0
#作弊结果
Downloading: http://example.webscraping.com/places/default/view/Australia-14
=================================================================
re: 0.19 seconds
=================================================================
bs: 18.12 seconds
=================================================================
lxml: 3.43 seconds
=================================================================
lxml_xpath: 1.50 seconds
Process finished with exit code 0
从结果可以明显看出正则表达式和lxml的性能差不多,采用xpath性能最佳,BeautifulSoup性能最差! 因为BeautifulSoup是纯python写的而lxml是c写的,性能可见一斑。
绝大部分网上教程都在使用BeautifulSoup写爬虫,这是因为BeautifulSoup可读性更高,更加人性,而css selector 和xpath需要学习他们特有的表达方式,但是绝对值得学习的。性能提高不止十倍(相对)!
在爬取少量数据时,BeautifulSoup还可以胜任,而且易读性高,但是大量数据采集时就建议使用lxml。