READ THIS
Paste_Image.png
Item.py
for making scrapy crawled data more ordered and serializable
how to use
- import botnameItem class from the projectname.items file
declare it () and yield it
Pipeline
receive and process item
how to use
- uncomment it inside settings
- use it
Settings
DOWNLOAD_DELAY = 3 be more friendly to scrapped site
USER_AGENT be more of a browser than a robot
ROBOTSTXT_OBEY = True
robotstxt in setting should be true if there is a robots.txt for the site, to be a good web citizen
r
Paste_Image.png
Paste_Image.png
