Scrapy 的环境搭建
找到python3
> which python3
/Users/macroot/virtualenvs/article_spider/bin/python3
生成虚拟环境
virtualenv --python=/Users/macroot/virtualenvs/article_spider/bin/python3 article_spider
进入文件夹
/Users/macroot [macroot@macroots-MacBook-Pro] [0:02]
> cd imooc
生成爬虫项目
/Users/macroot/imooc [macroot@macroots-MacBook-Pro] [0:02]
> scrapy startproject ArticleSpider
在项目外面创建spider是错误的。删掉
/Users/macroot/imooc [macroot@macroots-MacBook-Pro] [0:12]
> scrapy genspider jobbole blog.jobbole.com
Created spider 'jobbole' using template 'basic'
(article_spider)
/Users/macroot/imooc [macroot@macroots-MacBook-Pro] [0:13]
> ls
ArticleSpider BingSearch html_start jobbole.py py3rex
(article_spider)
/Users/macroot/imooc [macroot@macroots-MacBook-Pro] [0:13]
> rm -rf jobbole.py
进入目录去创建spider，scrapy会自己放到spider目录下。
/Users/macroot/imooc [macroot@macroots-MacBook-Pro] [0:14]
> ls
ArticleSpider BingSearch html_start py3rex
(article_spider)
/Users/macroot/imooc [macroot@macroots-MacBook-Pro] [0:14]
> cd ArticleSpider
(article_spider)
/Users/macroot/imooc/ArticleSpider [macroot@macroots-MacBook-Pro] [0:14]
> scrapy genspider jobbole blog.jobbole.com
Created spider 'jobbole' using template 'basic' in module:
ArticleSpider.spiders.jobbole
(article_spider)
