Automatic Network Documentation plugin for NetBox


License
GPL-3.0
Install
pip install netdoc==0.10.16

Documentation

NetDoc

NetDoc is an automatic network documentation plugin for NetBox. NetDoc aims to discover a partially known network populating netbox and drawing L2 and L3 diagrams.

NetDoc:

  • Discovers, via nornir+netmiko, network devices fetching information (routing, adjacencies, configuration...).
  • Populate netbox (devices, cables, IPAM).

Network diagrams are currently provided by netbox-topology-views plugin. See my blog post for more information.

Assumptions

Correlate data from different sources requires some assumptions:

  • Interface labels are used to correlate interfaces from different sources/output (e.g. show interfaces, show cdp neighbors...). Labels are built from interface name or shortname using the short_interface_name function.
  • Interface level VRFs have a local scope. Prefix and IP address level VRFs are used to globally track VRF. Because discovery scripts are not aware about global VRF, they set interface level VRF only. Discovery scripts do not modify IP address level VRF if IP address is already set. Users must manually update IP address and prefix level VRFs.
  • Netbox don't have network objects. To draw L3 diagrams, NetDoc uses prefixes and IP addresses. Because there is no hard association between prefixes and IP addresses, mind that IP addresses are part of a specific prefix if they both have the same VRF.
  • IP addresses of discoverable devices must be unique. They are used to correlate outputs and discoverables.
  • A method to get the hostname of discoverable devices exists. This hostname is used to create Netbox devices associated to discoverables.
  • Templates are used to parse hostnames and to select the right ingestor. If Netmiko can understand that "show ip arp" and "show ip arp vrf red" need the same template, ingestors must be aware of the specific VRF that can be found in the command line.
  • L2 neighbors (LLDP/CDP) are used to guess L1 cabling. Interfaces with multiple netibhors are silently skipped.

Installing netbox

You should follow the offical documentation, but just in case here is how I install netbox:

sudo apt install -y apache2 python3 python3-pip python3-venv python3-dev build-essential libxml2-dev libxslt1-dev libffi-dev libpq-dev libssl-dev zlib1g-dev postgresql redis
sudo useradd -M -U -d /opt/netbox netbox
sudo git clone -b v3.4.4 https://github.com/netbox-community/netbox /opt/netbox
sudo chown netbox:netbox /opt/netbox/ -R

Installing NetDoc prerequisites

sudo git clone --depth=1 https://github.com/networktocode/ntc-templates /opt/ntc-templates
sudo chown netbox:netbox /opt/ntc-templates -R

NetDoc must be included in netbox plugins and configured in the main netbox configuration file (see below).

Creating the netbox database

sudo -u postgres psql
create database netbox;
create user netbox with password '0123456789abcdef';
grant all privileges on database netbox to netbox;

Configuring netbox

You should follow offical documentation, but just in case here is how I configure netbox:

sudo -u netbox cp -a /opt/netbox/netbox/netbox/configuration_example.py /opt/netbox/netbox/netbox/configuration.py
sudo -u netbox cp /opt/netbox/contrib/gunicorn.py /opt/netbox/gunicorn.py
sudo -u netbox chmod 600 /opt/netbox/netbox/netbox/configuration.py
sudo openssl req -x509 -newkey rsa:4096 -keyout /etc/ssl/private/netbox.key -nodes -out /etc/ssl/certs/netbox.crt -sha256 -days 3650
sudo cp /opt/netbox/contrib/apache.conf /etc/apache2/sites-available/001-netbox.conf
sudo a2enmod proxy ssl headers proxy_http
sudo a2ensite 001-netbox
sudo find /opt/netbox/netbox/static/ -type f -exec chmod a+r {} \;
sudo find /opt/netbox/ -type d -exec chmod a+xr {} \;

Edit the configuration file (configuration.py) as following:

ALLOWED_HOSTS = ['*']
DEVELOPER = True
DATABASE = {
    'NAME': 'netbox',
    'USER': 'netbox',
    'PASSWORD': '0123456789abcdef',
    'HOST': 'localhost',
    'PORT': '',
    'CONN_MAX_AGE': 300,
}
REDIS = {
    'tasks': {
        'HOST': 'localhost',
        'PORT': 6379,
        'PASSWORD': '',
        'DATABASE': 0,
        'SSL': False,
    },
    'caching': {
        'HOST': 'localhost',
        'PORT': 6379,
        'PASSWORD': '',
        'DATABASE': 1,
        'SSL': False,
    }
}
PLUGINS = ['netdoc', 'netbox_topology_views']
PLUGINS_CONFIG = {
    'netdoc': {
        'NTC_TEMPLATES_DIR': '/opt/ntc-templates/ntc_templates/templates',
        'NORNIR_LOG': '/tmp/nornir.log',
        'NORNIR_TIMEOUT': 300,
    },
}
RQ_DEFAULT_TIMEOUT = 600
SECRET_KEY = '01234567890123456789012345678901234567890123456789'

Upgrade and install dependencies for netbox:

sudo -u netbox echo netdoc >> /opt/netbox/local_requirements.txt
sudo -u netbox /opt/netbox/upgrade.sh

Create first administrative user:

sudo -u netbox /opt/netbox/venv/bin/python3 /opt/netbox/netbox/manage.py createsuperuser

Starting netbox

Under /opt/netbox/contrib/ you can find startup scripts for both netbox and scheduler (netbox-rq).

sudo cp -v /opt/netbox/contrib/*.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable netbox netbox-rq apache2
sudo systemctl start netbox netbox-rq apache2

Netbox is listening by default on localhost:8001 (see /opt/netbox/contrib/gunicorn.py). Apache is serving as a reverse proxy.

Usint NetDoc on an existing netbox installation

NetDoc uses interface labels as a primary key for interfaces. Labels are populated with a interface shortname. The reason is simple: based on different outputs (especially on Cisco devices), interface name can be shortened. To correlate the interface name between many outputs, a shotname is created.

If you are using NetDoc on an existing netbox installation, you need to generate interface labels (shortnames) for each interface in the database. You can do that with the provided scripts:

./manage.py shell < set_interface_shortname.django

Developing NetDoc

Install NetDoc as a development module:

mkdir ~/src
git clone https://github.com/dainok/netdoc ~/src/netdoc

Starting netbox:

cd ~/src/netdoc
/opt/netbox/venv/bin/python3 setup.py develop
/opt/netbox/venv/bin/python3 manage.py runserver 0.0.0.0:8000 --insecure
/opt/netbox/venv/bin/python3 manage.py rqworker high default low

Debugging

Discover script:

from netdoc import tasks

tasks.discovery(["172.25.82.34","172.25.82.39","172.25.82.40"])

NTC template:

import textfsm
import pprint

template_file = 'ntc_templates/templates/cisco_xr_show_ipv4_interface.textfsm'
raw_output_file = 'tests/cisco_xr/show_ipv4_interface/cisco_xr_show_ipv4_interface.raw'

with open(template_file) as fd_t, open(raw_output_file) as fd_o:
    re_table = textfsm.TextFSM(fd_t)
    parsed_header = re_table.header
    parsed_output = re_table.ParseText(fd_o.read())

pprint.pprint(parsed_header)
pprint.pprint(parsed_output)

Parsers:

from netdoc import models
import importlib
from netdoc import functions
import logging
import pprint

request = "show vrf"
mode = "netmiko_cisco_nxos"
request = "show ip interface"
mode = None

logs = models.DiscoveryLog.objects.all()

request = None # or "show vrf"
mode = None    # or "netmiko_cisco_nxos"

if mode:
    logs = logs.filter(discoverable__mode=mode)
if request:
    logs = logs.filter(request=request)
logs = logs.filter(success=True)

for log in logs:
    try:
        functions.log_parse(log)
    except:
        pass
	print('Command: ', log.command)
	print('ID: ', log.id)
    print('Address: ', log.discoverable.address)
	print('Device: ', log.discoverable.device)
    print('Parsed: ', log.parsed)
    print('Items: ', len(log.parsed_output))
    pprint.pprint(log.parsed_output)
    print('-' * 70)

Ingest scripts:

from netdoc import models
import importlib
from netdoc.ingestors import functions
import logging

request = None # or "show vrf"
mode = None    # or "netmiko_cisco_nxos"

logs = models.DiscoveryLog.objects.all()
if mode:
    logs = logs.filter(discoverable__mode=mode)
if request:
    logs = logs.filter(request=request)
logs = logs.filter(parsed=True)

for log in logs:
        try:
            functions.log_ingest(log)
        except functions.NoIngestor:
            pass
        except functions.Postponed as err:
            print(err)

ToDo list

  • Implement diff and approve model for discovered objects (use the JSON validators).
  • Encrypt passwords and use select credentials in discovery script.
  • Implement a deny list to skip commands.

Known issues

  • Renaming credentials blanks passwords.
  • Cabling fails if neighbors are sending LLDP packets on subinterfaces (Check if the interface is physical).

References