snmp-fetch
An opinionated python3.7 SNMPv2 package designed for rapid database ingestion. This package is a source distribution that includes a C module wrapping net-snmp. No MIB processing is done as part of this package. The C module copies raw results from net-snmp into numpy arrays for fast post-processing with either numpy or pandas. Other libraries that wrap net-snmp will typically return control to python between every PDU request-response. Snmp-fetch is designed to be thread-safe and efficient by walking multiple targets within the C module with the GIL released. Helper modules are provided to aid in the post-processing with MIB-like definitions for converting the raw data into usable DataFrames.
Prerequisites
Snmp-fetch requires python 3.7, a c++17 compiler (currently only supports gcc-8), and cmake 3.12.4+. No other user installed dependencies should be required for building this package.
Attention!
Installation can take awhile as the install script will download boost and download and build a light-weight version of net-snmp 5.8 within the package.
The boost download can take awhile as it clones each submodule as oppose to downloading the compressed distribution. There is an issue with downloading the compressed distribution via cmake with 302 redirects to a file failing in cURL.
The cmake script will attempt to detect the number of cores on the host machine to speedup download and build times. Expect installation times to range from 5 minutes (4 cores with hyperthreading) to 30+ minutes (1 core).
Installation
# poetry
poetry add snmp-fetch --no-dev
# pip
pip install snmp-fetchExamples
The examples use jupyter and the dependencies can be installed using the following:
git clone --recurse-submodules -j8 https://github.com/higherorderfunctor/snmp-fetch.git
cd snmp_fetch
virtualenv -p python3.7 ENV
source ENV/bin/activate
poetry install -E notebooks
jupyter labDevelopment
Poetry is required for the development of snmp-fetch.
# clone the respository
git clone --recurse-submodules -j8 https://github.com/higherorderfunctor/snmp-fetch.git
cd snmp-fetch
# if working off an existing clone, update the current branch
git pull # pull the latest code
git submodule update --init --recursive --depth=1 # pull the latest submodule version
# setup the virtual environment - mypy uses symbolic links in the 'stubs' directory to
# expose packages that play nicely with the static type checker
virtualenv -p python3.7 ENV
source ENV/bin/activate
poetry install# C++ headers are in the following folders for linters
export CPLUS_INCLUDE_PATH="build/temp.linux-x86_64-3.7/include:lib/pybind11/include:lib/Catch2/single_include/catch2:lib/boost"
# python linting
poetry run isort -rc --atomic .
poetry run pylint snmp_fetch tests
poetry run flake8 snmp_fetch tests
poetry run mypy -p snmp_fetch -p tests
poetry run bandit -r snmp_fetch
# C++ linting
# TODO
# python testing
poetry run pytest -v --hypothesis-show-statistics tests
# fail fast testing
poetry run pytest -x --ff tests
# C++ testing
pushd build/temp.linux-x86_64-3.7/
cmake -DBUILD_TESTING=ON ../.. && make test_api test
popdUpgrading Dependencies
# boost
rm -rf lib/boost
mkdir lib/boost
wget https://dl.bintray.com/boostorg/release/X.Y.Z/source/boost_X_Y_Z.tar.gz
tar -xvf boost_X_Y_Z.tar.gz
cd boost_X_Y_Zi
./bootstrap.sh
cd tools/bcp
../../b2
cd ../../
chmod +x bin.v2/tools/bcp
bin.v2/tools/bcp/gcc-8/release/link-static/bcp LICENSE_1_0.txt boost/format.hpp boost/range/combine.hpp ../lib/boost
cd ..
rm -rf boost_X_Y_Z*Known Limitations
- Changes between v0.1.x versions may introduce breaking changes.
- The library only supports SNMPv2 at this time.
- BULKGET_REQUEST and NEXT_REQUEST will always perform a walk.
- Walks will always end if the root of the OID runs past the requested OID.
- Duplicate objects on the same host/request will be silently discarded.
- This includes the initial request; walks must be performed on an OID prior to the first desired.
- NO_SUCH_INSTANCE, NO_SUCH_OBJECT, and END_OF_MIB_VIEW response variable bindings are exposed as errors for handling by the client.
