DeliciousSoda
DeliciousSoda
is a webscraper for robot.txt
files. Basic usage:
>>> from soda import DeliciousSoda
>>> s = DeliciousSoda("https://www.google.com")
>>> s.get_allowed()
['Allow: /search/about', 'Allow: /search/static', ...]
License: Apache 2.0, see LICENSE for more details.