Bamboo Crawler

Deprecation Notice

This project is deprecated. No longer maintained. I don't use this project anymore. If you want to use this project, please fork it.

A Hobby Crawler. It is almost under construction.

Usage

Installation

$ pip install bamboo-crawler

Run

$ bamboo --recipe recipe.yml

Recipe

mytask:
  input:
    type: ConstantInputter
    options:
      value: http://httpbin.org/robots.txt
  process:
    type: HTTPCrawler
  output:
    type: StdoutOutputter

License

BSD-3-Clause License