astel

Python package for asyncronous web crawling.


Keywords
async, async-task, asyncio, httpx, python, robots-txt, web-crawler, web-crawling
License
MIT
Install
pip install astel==0.1.1

Documentation

Astel logo

Astel

pypi python Build Status codecov Ruff

A simple, fast and reliable asyncronous web crawler for Python.

Features

The main goal of astel is to offer a simpler, efficient and performant solution to programmatically look for links in webpages: no need to extend any class (composition over inheritance), no configuration and as few dependencies as possible.

This package relies on HTTPX to send all requests in asynchronous operations, thus maximizing the number of pages processed during each execution.

Credits

This package was created with Cookiecutter and the waynerv/cookiecutter-pypackage project template.