scrapy crawl test -s JOBDIR=data -0 items.csv
watch -n 1 -d 'wc -l items.csv;du -h items.csv'
附数据 items.csv
-
- GitHub
- GitLab
- Bitbucket
-
By logging in you accept
our terms of service
and privacy policy
用于爬取JD和github的代码，数据存储于mongodb。.
pip install scrapy-jingdong==1.2.1
scrapy crawl test -s JOBDIR=data -0 items.csv
watch -n 1 -d 'wc -l items.csv;du -h items.csv'
附数据 items.csv
The Tidelift Subscription provides access to a continuously curated stream of human-researched and maintainer-verified data on open source packages and their licenses, releases, vulnerabilities, and development practices.
Learn more →Login to resync this project