scrapy-rotating-free-proxies
Avoid being banned by webpages when you crawl them. This is an extension to the amazing scrapy-rotating-proxies library this with very minor changes. The main target of this library is to get proxies dynamically when the spider is running. This library automatically fetches freely available lists of proxies from free-proxy-list.net.
Installation
pip install scrapy-rotating-free-proxies
Usage
After installing you need to do just add following two variables in settings.py of your Scrapy project
ROTATING_PROXY_LIST_PATH = '/my/path/proxies.txt' DOWNLOADER_MIDDLEWARES = { 'rotating_proxies.middlewares.RotatingProxyMiddleware': 610, 'rotating_proxies.middlewares.BanDetectionMiddleware': 620, }
For further details on using this library, refer to the original readme.
Thank you!