最终成果
[{'image': 'img/pic_0000_073a9256d9624c92a05dc680fc28865f.jpg',
'price': '$24.99',
'review': '65 reviews',
'star': '★★★★★',
'titile': 'EarPod'},
{'image': 'img/pic_0005_828148335519990171_c234285520ff.jpg',
'price': '$64.99',
'review': '12 reviews',
'star': '★★★★☆',
'titile': 'New Pocket'},
{'image': 'img/pic_0006_949802399717918904_339a16e02268.jpg',
'price': '$74.99',
'review': '31 reviews',
'star': '★★★★☆',
'titile': 'New sunglasses'},
{'image': 'img/pic_0008_975641865984412951_ade7a767cfc8.jpg',
'price': '$84.99',
'review': '6 reviews',
'star': '★★★☆☆',
'titile': 'Art Cup'},
{'image': 'img/pic_0001_160243060888837960_1c3bcd26f5fe.jpg',
'price': '$94.99',
'review': '18 reviews',
'star': '★★★★☆',
'titile': 'iphone gamepad'},
{'image': 'img/pic_0002_556261037783915561_bf22b24b9e4e.jpg',
'price': '$214.5',
'review': '18 reviews',
'star': '★★★★☆',
'titile': 'Best Bed'},
{'image': 'img/pic_0011_1032030741401174813_4e43d182fce7.jpg',
'price': '$500',
'review': '35 reviews',
'star': '★★★★☆',
'titile': 'iWatch'},
{'image': 'img/pic_0010_1027323963916688311_09cc2d7648d9.jpg',
'price': '$15.5',
'review': '8 reviews',
'star': '★★★★☆',
'titile': 'Park tickets'}]
我的代码
from bs4 import BeautifulSoup
path='./1_2answer_of_homework/index.html'
with open(path,'r') as wb_data:
soup = BeautifulSoup(wb_data,'lxml')
reviews = soup.select(" body > div > div > div.col-md-9 > div > div > div > div.ratings > p.pull-right")
grades = soup.select(" body > div > div > div.col-md-9 > div > div > div > div.ratings > p > span") # 删除所有冒号后的信息
prices = soup.select(" body > div > div > div.col-md-9 > div > div > div > div.caption > h4.pull-right")
titles = soup.select(" body > div > div > div.col-md-9 > div > div > div > div.caption > h4 > a")
images = soup.select(" body > div > div > div.col-md-9 > div > div > div > img")
# print(reviews,stars,prices,titles,images,sep='\n---------------\n')
grades = str(grades)
grades = grades.replace('<span class="glyphicon glyphicon-star"></span>', '★')
grades = grades.replace('<span class="glyphicon glyphicon-star-empty"></span>', '☆') # 替换grades为星符
grades = grades.replace(', ','')
grades = grades[1:(len(grades)-1)] #选取所有星符
num = range(int(len(grades)/5))
stars = []
for s in num:
print(5*s,5*(s+1))
stars.append(''.join(grades[5*s:5*(s+1)])) # 每5个星符组成一个商品评价
print(stars)
info = []
# 采用字典进行集体查询
for review,star,price,title,image in zip(reviews,stars,prices,titles,images):
data = {
'review':review.get_text(),
'star': star,
'price': price.get_text(),
'titile': title.get_text(),
'image': image.get('src') # 图片形式
}
info.append(data)
list(info)
总结
- 使用相对路径
- 采用BeautifulSoup( )爬取网页信息
- 检查网页，拷贝元素的CSS Selector路径而非XPath路径
- 用 ______ .select 获取网页中某类元素的所有路径时，应把冒号后的部分去掉，如：body > div:nth-child(2) > div > div.col-md-9 > div:nth-child(2) > div:nth-child(3) > div > img 应改为body > div > div > div.col-md-9 > div > div > div > img （注意： ______ 表示变量名）
- 可以采用字典进行集体查询
- 遇到多对一的情况时，如：多个标签对应同一主题，______.select 应去除Selector路径的最后一项，并且在集体查询时用列表展示 list( ____.stripped_strings)
- 图片采用 ______.get('src')，文字采用______.get_text()
