spiderfetch

Installation

$ pip install spiderfetch
$ spiderfetch

Usage

Fetching

Fetch all urls matching 2008.*.ogg from a page:

$ spiderfetch http://www.fosdem.org/2008/media/video 2008.*ogg

To dump the urls to a file instead of fetching:

$ spiderfetch http://www.fosdem.org/2008/media/video 2008.*ogg --dump > urls

Spidering

Spider a site to depth 3 while pausing 2 seconds between fetches. The urls that will be considered when spidering must match .*:

$ spiderfetch --host http://en.wikipedia.org --depth 3 --pause 2 '.*'