-
- GitHub
- GitLab
- Bitbucket
-
By logging in you accept
our terms of service
and privacy policy
A simple web spider with pluggable recursion strategies
Homepage Repository PyPI Python
pip install spydey==0.5
Spydey ======= A simple web spider with several recursion strategies. Home page is at http://github.com/slinkp/spydey. It doesn't do much except follow links and report status. I mostly use it for quick and dirty smoke testing and link checking. The only unusual feature is the ``--traversal=pattern`` option, which does recursive traversal in an unusual order: It tries to recognize patterns in URLs, and will follow URLs of novel patterns before those with patterns it has seen before. When there are no novel patterns to follow, it follows random links to URLs of known patterns. If you use this for smoke-testing a typical modern web app that maps URL patterns to views/controllers, this will very quickly hit all your views/controllers at least once... usually. But it's not very interesting when pointed at a website that has arbitrarily deep trees (static files, VCS repositories, and the like). Also, it's designed so that adding a new recursion strategy is trivial. Spydey was originally written for the purpose of experimenting with different recursive crawling strategies. Read the source. Oh, and if you install Fabulous, console output is in color. For lazy, zero-configuration smoke testing, I typically run it like:: spydey -r --stop-on-error --max-requests=200 --traversal=pattern --profile --log-referrer URL There are a number of other command-line options, many stolen from wget. Use ``--help`` to see what they are.
The Tidelift Subscription provides access to a continuously curated stream of human-researched and maintainer-verified data on open source packages and their licenses, releases, vulnerabilities, and development practices.
Learn more →
Something wrong with this page? Make a suggestion
Export .ABOUT file for this package
Last synced: 2021-12-15 13:49:49 UTC
Login to resync this project