requestspool
Multiprocessing pool providing a unique request session per process, thus avoiding the common problems like SSL errors when trying to use a global session for multiple processes.
Every process started by the map() or starmap() methods will have a
session assigned.
The session will be passed as additional argument to the
map()/starmap() target function.
requests.Session() is used by default to provide a session object, but
another function can be used by passing special_func, special_args and/or
special_kwargs parameters to RequestsPool.
Examples
Get multiple pages in parallel, here two at a time:
>>> from requestspool import RequestsPool
>>> URLS = ("https://pypi.org/", "https://git.io",
... "https://gentoo.org")
>>> def get_url(url, session):
... return session.get(url)
...
>>> with RequestsPool(2) as rp:
... print(rp.map(get_url, URLS))
...
[[<Response [200]>, <Response [200]>], [<Response [200]>]]Get three pages in parallel and pass a same timeout parameter to all target function calls:
>>> from requestspool import RequestsPool
>>> from itertools import product
>>> URLS = ("https://pypi.org/", "https://git.io",
... "https://gentoo.org")
>>> def get_url_timeout(url, timeout, session):
... return session.get(url, timeout=timeout)
...
>>> with RequestsPool(3) as rp:
... print(rp.starmap(get_url_timeout, product(URLS, (6,))))
...
[[<Response [200]>], [<Response [200]>], [<Response [200]>]]Installation
Requires Python 3 (currently only tested on 3.6.5+ with GNU/Linux).
From pip:
sudo pip3 install requestspoolManually:
git clone https://github.com/ccc032/requestspool
cd requestspool
sudo python3 setup.py install 
