A Python framework to build web scraping applications.
- I want to
- Scrape contents (images, texts, etc) from places on the web (RSS, Twitter, parsing HTML, etc)
- Moderate and modify them internally on a web UI
- Expose them in a public API
- Without having to re-implement, for different scraping applications
- Reliable "cron" jobs that scrape the content
- A pipeline that processes the contents stage by stage
- Internal web UIs that allow human to moderate the contents as a pipeline stage
This is a Python framework that extracts out common components in the scraping, processing and moderating of web contents, and provides interfaces for programmers to implement "business logic" with, so that they can build reliable and easy-to-use web scraping applications in less time.