Requests package with QOL improvements and anti-bot detection measures
- 1. About the Project
- 2. Getting Started
- 3. Usage
- 4. Roadmap
- 5. License
- 6. Contact
- 7. Acknowledgements
ak_requests
is a Python package that provides an interface for automating requests tasks using requests
package. It comes with quality of life improvements like retires, custom redirects, spacing out requests and shuffling requests to not trigger anti-bot measures
- Bulk requests handling
- Built-in retries and timeouts
- Can log processes to file
- Handles downloads of files/videos
- Implemented default rate-limiting checks and process
- Session objects are serialized to be able to save/load sessions from file
- Can choose to handle exceptions or skip it completely with
RAISE_EXCEPTIONS
attribute - Can support both basic
.basic_auth()
and OAuth.oauth2_auth()
authentications.
Download the repo and install with flit
pip install flit
flit install --deps production
Alternatively, you can use pip
pip install ak_requests
Install with flit
flit install --pth-file
from ak_requests import RequestsSession
# Initialize session
session = RequestsSession(log=False, retries=5, log_level='error', timeout=10)
## Can update session level variables
session.MIN_REQUEST_GAP = 1.5 # seconds, Change min time bet. requests
session.RAISE_ERRORS = False # raises RequestErrors, else returns None; defaults to True
# Update custom header
session.update_header({'Connection': 'keep-alive'})
# set cookies
session.update_cookies([{'name':'has_recent_activity', 'value':'1'}])
# Get requests
res = session.get('https://reqres.in/api/users?page=2', data={}, proxies = {} ) # Can accept any requests parameters
# Make bulk requests
urls = ['https://reqres.in/api/users?page=2', 'https://reqres.in/api/unknown']
responses = session.bulk_get(urls)
# use beautifulsoup
from ak_requests import soupify
res = session.get('https://reqres.in/api/users?page=2')
soup = soupify(res)
## or
soup, res = session.soup('https://reqres.in/api/users?page=2')
## Also works for bulk requests
soups, ress = session.bulk_soup(urls)
# Download files
## Check if file is downloadble
session.downloadble('https://www.youtube.com/watch?v=9bZkp7q19f0') #False
session.downloadble('http://google.com/favicon.ico') #True
session.download(
url = 'http://google.com/favicon.ico', #URL to download
fifopath='C:\\', #Can be folderpath, filename or filepath. If existing folder specified - will extract filename from url contents
confirm_downloadble = False #Will return `None`, if url not downloadble
)
# Download videos
from pathlib import Path
video_info = session.video(url='https://www.youtube.com/watch?v=BaW_jenozKc',
folderpath=Path('.'),
audio_only=False) #Downloads the video to specified path and returns dict of video info
# Save/Restore session to/from file
## Save the session state to a file
session.save_session('session_state.pkl')
## Later, you can load the session state back
restored_session = RequestsSession.load_session('session_state.pkl')
# Authentication
session.basic_auth(username="johndoe", password="12345678") ## basic auth
session.oauth2_auth(token='x0-xxxxxxxxxxxxxxxxxxxxxxxx') ## OAuth authentication
- Proxy
- Asynchronous Requests
- Response Caching
- Request Preprocessing and Postprocessing
- File Upload Support
See LICENSE for more information.
Arun Kishore - @rpakishore
Project Link: https://github.com/rpakishore/ak_requests