Scrapy "CloudFlare" middleware
A Scrapy middleware to bypass the CloudFlare's anti-bot protection, based on cloudflare-scrape.
Installation
pip install scrapy_cloudflare_middleware
Usage
Add the middleware to your DOWNLOADER_MIDDLEWARES
settings
DOWNLOADER_MIDDLEWARES = {
# The priority of 560 is important, because we want this middleware to kick in just before the scrapy built-in `RetryMiddleware`.
'scraping_hub.middlewares.CloudFlareMiddleware': 560
}
Done. Happy scraping !