scrapy-job-parameters

Scrapy downloader middleware to enable persistent storage.


Licenses
CERN-OHL-P-2.0/OFL-1.1-RFN/CDDL-1.1
Install
pip install scrapy-job-parameters==0.1.10

Documentation

Scrapy Metas Extension

CircleCI

Scrapy extension to make env meta information available as spider fields.

Current implementation exposes to the spider a meta object with the following attributes:

  • project_id - as defined by Scrapinghub
  • spider_id -
  • job_id -
  • job_name - raw job id from Scrapinghub or uuid v4
  • job_time - excecution time