scrapy-dotpersistence

Scrapy extension to sync .scrapy folder to an S3 bucket.

Installation

You can install scrapy-dotpersistence using pip:

pip install scrapy-dotpersistence

You can then enable the extension in your settings.py:

EXTENSIONS = {
    ...
    'scrapy_dotpersistence.DotScrapyPersistence': 0
}

How to use it

Enable extension through settings.py:

DOTSCRAPY_ENABLED = True

Configure the exension through settings.py:

ADDONS_AWS_ACCESS_KEY_ID = "ABC"
ADDONS_AWS_SECRET_ACCESS_KEY = "DEF"
ADDONS_AWS_USERNAME = "username"
ADDONS_S3_BUCKET = "test-bucket-name"

You can change a dotpersistence folder path with environ:

export DOTSCRAPY_DIR='/tmp/.scrapy'