1. 练习准备：

MongoDB的准备：

• MongoDB学习—（1）安装时出现The default storage engine 'wiredTiger' is not available问题解决 http://blog.csdn.net/u013457382/article/details/50775268

• windows下mongodb安装与使用整理:
  http://www.cnblogs.com/lecaf/archive/2013/08/23/mongodb.html

• MongoDB数据导入导出：

  
  
  image.png
  
  
  image.png
  
  
  image.png
  
  
  image.png

• 新建 collection：db.creatCollection('filename')

• 简单的MongoDB使用方法：

  
  
  image.png
  
  
  image.png
  
  
  image.png

2. CODE:

import requests
from bs4 import BeautifulSoup
import pymongo

client = pymongo.MongoClient('localhost',27017)
xiaozhu = client['xiaozhu']
sheet_tab = xiaozhu['sheet_tab']

headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36'
}

def main():
    urls = ["http://bj.xiaozhu.com/search-duanzufang-p{}-0/".format(number) for number in range(1, 2)]
    for url in urls:
        wb_data = requests.get(url)

    # 开始解析网页数据
        soup = BeautifulSoup(wb_data.text, 'lxml')

        # 鼠标放到图片上，右键，审查元素，找到链接的css selector
        links = soup.select("#page_list > ul > li > a")

        #  由于链接有好多个，soup.select返回的是列表，需要用for一个个取出来
        for link in links:
            # 由于链接地址在标签的href属性里面，所以要用get获取
            href = link.get("href")

            # 把得到的详情页链接，传给函数，这个函数可以得到详细数据
            get_detail_info(href)

def get_lorder_sex(class_name):
    if class_name == ['member_boy_ico']:
        return '男'
    elif class_name == ['member_girl_ico']:
        return '女'

def get_detail_info(url):
    wb_data = requests.get(url)

# 开始解析详情页数据
    soup = BeautifulSoup(wb_data.text, 'lxml')

# 获取名称
    titles = soup.select("body > div.wrap.clearfix.con_bg > div.con_l > div.pho_info > h4 > em")

# 获取地址
    addresss = soup.select("body > div.wrap.clearfix.con_bg > div.con_l > div.pho_info > p > span.pr5")

# 获取价格
    prices = soup.select("#pricePart > div.day_l > span")

# 获取图片
    images = soup.select("#curBigImage")

# 获取房东头像
    avartars = soup.select("#floatRightBox > div.js_box.clearfix > div.member_pic > a > img")

# 获取房东姓名
    names = soup.select("#floatRightBox > div.js_box.clearfix > div.w_240 > h6 > a")

# 获取房东性别
    sexs = soup.select("#floatRightBox > div.js_box.clearfix > div.w_240 > h6 > span")

    for title, address, price, image, avartar, name, sex in zip(titles, addresss, prices, images, avartars, names, sexs):
    # 从标签里面提取内容
        data = {
            "title": title.get_text(),
            "address": address.get_text(),
            "price": int(price.get_text()),
            "image": image.get("src"),
            "avartar": avartar.get("src"),
            "name": name.get_text(),
            "sex": get_lorder_sex(sex.get("class"))
        }
  
        sheet_tab.insert_one(data)
        for item in sheet_tab.find({'price':{'$gte':500}}):
            print('title: '+ item['title'],'价格： '+ str(item['price']) + '元')



if __name__ == '__main__':
    main()

3. 运行结果：

……
title: 劲松双井10号线地铁华腾园大三室整租家庭聚会 价格： 698元
title: 北京居住核心地段，国贸CBD双井，地铁站旁 价格： 548元
title: 简约大气，浪漫舒适，紧邻9号线科怡路 价格： 769元
Process finished with exit code 0