Scrapy MongoDB
MongoDB-based components for scrapy that allows distributed crawling
Available Scrapy components:
- Scheduler
- Duplication Filter
Installation
From pypi
$ pip install scrapy-mongodbFrom github
$ git clone https://github.com/jbinfo/scrapy-mongodb.git
$ cd scrapy-mongodb
$ python setup.py installUsage
Enable the components in your settings.py:
# Enables scheduling storing requests queue in redis.
SCHEDULER = "scrapy_mongodb.scheduler.Scheduler"
# Don't cleanup mongodb queues, allows to pause/resume crawls.
MONGODB_QUEUE_PERSIST = True
# Specify the host and port to use when connecting to Redis (optional).
MONGODB_SERVER = 'localhost'
MONGODB_PORT = 27017
MONGODB_DB = "my_db"
# MongoDB collection name
MONGODB_QUEUE_NAME = "my_queue"
