Scrapy Algolia Exporter

Directly populate a given Algolia index from a scrapy spider.

Usage

⚠️ Your crawled items must contains a unique ObjectID key. ⚠️

Add the mandatory scrapy settings to configure the Algolia API:

ALGOLIA_API_ID='my_algolia_api_id'
ALGOLIA_API_KEY='my_algolia_api_key'
ALGOLIA_INDEX_NAME='my_algolia_index_name'

Add the AlgoliaItemPipeline in the ITEM_PIPELINES scrapy setting:

ITEM_PIPELINES = {
   'scrapy_algolia_exporter.pipelines.AlgoliaItemPipeline': 10
}

The ALGOLIA_ITEM_BULK_NBR setting control how many items will be send to Algolia at the same time. If not provided, the items will be send by group of 100.

Install

pip install scrapy_algolia_exporter