scrapy-random-useragent-pro
Random UserAgent middleware for Scrapy.
You can chose your preferred ua in request.meta[desktop,mobile,bot].
Install
pip install scrapy-random-useragent-proUsage
settings.py
RANDOM_UA_ENABLED = True
RANDOM_UA_DEFAULT_TYPE = 'mobile'
# always change user-agent
RANDOM_UA_OVERWRITE = False
DOWNLOADER_MIDDLEWARES = {
'scrapy_random_useragent_pro.middleware.RandomUserAgentMiddleware': 100,
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,# important
}
In your code
yield Request('http://scrapytest.org/', meta={'ua': 'mobile'})
