tarampampam/mikrotik-hosts-parser

Application for generating ADBlock script for MikroTik routers, based on Lumen framework


Keywords
mikrotik, lumen, golang, parser
License
MIT

Documentation

Logo

MikroTik hosts parser

Release version Build Status Coverage Image size License

This application provides HTTP server, that can generate script for RouterOS-based routers for blocking "AD" hosts using 3rd party host-lists (available by HTTP):

## Limit: 5000
## Cache lifetime: 30m0s
## Format: routeros
## Redirect to: 127.0.0.1
## Sources list:
##  - <https://adaway.org/hosts.txt>
##  - <https://www.malwaredomainlist.com/hostslist/hosts.txt>
## Excluded hosts:
##  - broadcasthost
##  - ip6-allhosts
##  - ip6-allnodes
##  - ip6-allrouters
## Cache HIT for <https://adaway.org/hosts.txt> (expires after 25m55s)
## Cache miss for <https://www.malwaredomainlist.com/hostslist/hosts.txt>

/ip dns static
add address=127.0.0.1 comment="ADBlock" disabled=no name="1-1ads.com"
add address=127.0.0.1 comment="ADBlock" disabled=no name="101com.com"
add address=127.0.0.1 comment="ADBlock" disabled=no name="101order.com"
add address=127.0.0.1 comment="ADBlock" disabled=no name="123freeavatars.com"

# ...

Hosts file format (example):

# Any comments
127.0.0.1   1-1ads.com
127.0.0.1   101com.com 101order.com
0.0.0.0     123freeavatars.com

All what you need is:

  • Start current application HTTP server
  • Make an HTTP request to the script generator endpoint /script/source?sources_urls=... with all required parameters (like records limit, hosts file URLs, exclusion list and others)
  • Generated script source execute on your RouterOS-based hardware

More information can be found here.

Previous version (PHP) can be found in php-version branch.

Installing

Download latest binary file for your os/arch from releases page or use our docker image (ghcr.io). Also you may need in configuration file ./configs/config.yml and ./web directory content for web UI access.

Usage

This application supports next sub-commands:

Sub-command Description
serve Start HTTP server
healthcheck Health checker for the HTTP server (use case - docker healthcheck) (hidden in CLI help)
version Display application version

And global flags:

Flag Description
--verbose, -v Verbose output
--debug Debug output
--log-json Logs in JSON format

HTTP server starting

serve sub-command allows to use next flags:

Flag Description Default value Environment variable
--listen, -l IP address to listen on 0.0.0.0 (all interfaces) LISTEN_ADDR
--port, -p TCP port number 8080 LISTEN_PORT
--resources-dir, -r Path to the directory with public assets ./web RESOURCES_DIR
--config, -c Config file path ./configs/config.yml CONFIG_PATH
--caching-engine Caching engine (memory or redis) memory CACHING_ENGINE
--cache-ttl Cached entries lifetime (examples: 50s, 1h30m) 30m CACHE_TTL
--redis-dsn Redis server DSN, required only if redis caching engine is enabled redis://127.0.0.1:6379/0 REDIS_DSN

Environment variables have higher priority then flag values.

Server starting command example:

$ ./mikrotik-hosts-parser serve \
    --config ./configs/config.yml \
    --listen 0.0.0.0 \
    --port 8080 \
    --resources-dir ./web

This command will start HTTP server using configuration from ./configs/config.yml on TCP port 8080 and use directory ./web for serving static files. Configuration file well-documented, so, feel free to change any settings on your choice!

Configuration file allows you to use environment variables with default values (used library)!

After that you can navigate your browser to http://127.0.0.1:8080/ and you will see something like that:

screenshot

Special endpoint /script/source?sources_urls=... generates RouterOS-based script using passed http-get parameters (watch examples on index page).

Using docker

image stats

All supported image tags can be found here and here.

Just execute in your terminal:

$ docker run --rm -p 8080:8080/tcp tarampampam/mikrotik-hosts-parser:X.X.X

Where X.X.X is image tag (application version).

Demo

I can't guarantee that this links will available forever, but you can use this application by the following links:

Testing

For application testing and building we use built-in golang testing feature and docker-ce + docker-compose as develop environment. So, just write into your terminal after repository cloning:

$ make test

Or build the binary file:

$ make build

Releasing

New versions publishing is very simple - just make required changes in this repository, update changelog file and "publish" new release using repo releases page.

Binary files and docker images will be build and published automatically.

New release will overwrite the latest docker image tag in both registers.

Changes log

Release date Commits since latest release

Changes log can be found here.

Support

Issues Issues

If you will find any package errors, please, make an issue in current repository.

License

This is open-sourced software licensed under the MIT License.