robots
A library that parses and validates rules from robots.txt.
Installation
This library is available on hex.pm.
To install this library, simply add the following lines to your
rebar.config:
{robots, "1.1.1"}Usage
Content = <<"User-Agent: bot\nAllow: /fish">>,
%% This will return an opaque type that contains all the rules and their agents
{ok, RulesIndex} = robots:parse(Content, 200),
true = robots:is_allowed(<<"bot/1.0.0">>, <<"/fish/salmon.html">>, RulesIndex),
true = robots:is_allowed(<<"bot/1.0.0">>, <<"/Fish.asp">>, RulesIndex),Development
Running all the tests and linters
You can run all the tests and linters with the rebar3 alias:
rebar3 check
