再次顺手写了一个爬虫,把数据存在数据库。然后利用time date模块分析了每天的发帖数,利用charts做成表格。
要点:
1.爬虫,requests,beautifulSoup的css选择器
2.py连接数据库操作insert_one,find({},{}),update()等
3.charts表格使用,折现type:line,很方便。
4.time date模块,理解了datetime是一个时间的类,timetelta是时间间隔,strptime,strftime,now等函数。详见<a href='http://www.liaoxuefeng.com/wiki/0014316089557264a6b348958f449949df42a6d3a2e542c000/001431937554888869fb52b812243dda6103214cd61d0c2000#0'>廖雪峰datetime部分</a>
import requests,datetime,pymongo,charts
from bs4 import BeautifulSoup
client = pymongo.MongoClient('localhost',27017)
tieba = client['tieba']
time_list = tieba['time_list']
time2_list = tieba['time2_list']
#把日期存入数据库
'''
start_url = 'https://tieba.baidu.com/p/5062169335?pn='
headers = {
'User-Agent':'Mozilla/5.0 (Windows NT 10.0; WOW64; rv:53.0) Gecko/20100101 Firefox/53.0'
}
for i in range(1,14):
try:
url = start_url + str(i)
r = requests.get(url,timeout = 20,headers = headers)
r.raise_for_status()
except:
print('erroe')
soup = BeautifulSoup(r.text,'lxml')
date = soup.select('div.post-tail-wrap > span:nth-of-type(4)')
for each in date:
time_list.insert_one({'date':each.text[:10]})
'''
#把数据从数据库取出处理
index_time = []
count_time = []
start_day = datetime.date(2017,4,8)
end_day = datetime.date(2017,5,6)
days = datetime.timedelta(days=1)
while start_day <= end_day:
index_time.append(start_day.strftime('%Y-%m-%d'))
start_day +=days
#print(index_time)
for date in index_time:
a = list(time_list.find({'date':date}))
count_time.append(len(a))
#print(count_time)
options = {
'chart' : {'zoomType':'xy'},
'title' : {'text': '发帖量统计'},
'subtitle': {'text': '可视化统计图表'},
'xAxis' : {'categories': index_time},
'yAxis' : {'title': {'text': '数量'}}
}
series = [{'data':count_time,'name':'在所有物是人非的','type':'line'}] #只统计了一个贴吧,可以统计多个贴吧,画多条曲线
charts.plot(series,show = 'inline',options=options)
结果,由于charts是由js生成的,可以放大缩小观看曲线,但不能保存,只能够截个图来看看了