An opinionated python3.7 SNMPv2 package designed for rapid database ingestion. This package is a source distribution that includes a C module wrapping net-snmp. No MIB processing is done as part of this package. The C module copies raw results from net-snmp into numpy arrays for fast post-processing with either numpy or pandas. Other libraries that wrap net-snmp will typically return control to python between every PDU request-response. Snmp-fetch is designed to be thread-safe and efficient by walking multiple targets within the C module with the GIL released. Helper modules are provided to aid in the post-processing with MIB-like definitions for converting the raw data into usable DataFrames.
Snmp-fetch requires python 3.7, a c++17 compiler (currently only supports gcc-8), and cmake 3.12.4+. No other user installed dependencies should be required for building this package.
Installation can take awhile as the install script will download boost and download and build a light-weight version of net-snmp 5.8 within the package.
The boost download can take awhile as it clones each submodule as oppose to downloading the compressed distribution. There is an issue with downloading the compressed distribution via cmake with 302 redirects to a file failing in cURL.
The cmake script will attempt to detect the number of cores on the host machine to speedup download and build times. Expect installation times to range from 5 minutes (4 cores with hyperthreading) to 30+ minutes (1 core).
# poetry poetry add snmp-fetch --no-dev # pip pip install snmp-fetch
The examples use jupyter and the dependencies can be installed using the following:
git clone --recurse-submodules -j8 https://github.com/higherorderfunctor/snmp-fetch.git cd snmp_fetch virtualenv -p python3.7 ENV source ENV/bin/activate poetry install -E notebooks jupyter lab
Poetry is required for the development of snmp-fetch.
# clone the respository git clone --recurse-submodules -j8 https://github.com/higherorderfunctor/snmp-fetch.git cd snmp-fetch # if working off an existing clone, update the current branch git pull # pull the latest code git submodule update --init --recursive --depth=1 # pull the latest submodule version # setup the virtual environment - mypy uses symbolic links in the 'stubs' directory to # expose packages that play nicely with the static type checker virtualenv -p python3.7 ENV source ENV/bin/activate poetry install
# C++ headers are in the following folders for linters export CPLUS_INCLUDE_PATH="build/temp.linux-x86_64-3.7/include:lib/pybind11/include:lib/Catch2/single_include/catch2:lib/boost" # python linting poetry run isort -rc --atomic . poetry run pylint snmp_fetch tests poetry run flake8 snmp_fetch tests poetry run mypy -p snmp_fetch -p tests poetry run bandit -r snmp_fetch # C++ linting # TODO # python testing poetry run pytest -v --hypothesis-show-statistics tests # fail fast testing poetry run pytest -x --ff tests # C++ testing pushd build/temp.linux-x86_64-3.7/ cmake -DBUILD_TESTING=ON ../.. && make test_api test popd
# boost rm -rf lib/boost mkdir lib/boost wget https://dl.bintray.com/boostorg/release/X.Y.Z/source/boost_X_Y_Z.tar.gz tar -xvf boost_X_Y_Z.tar.gz cd boost_X_Y_Zi ./bootstrap.sh cd tools/bcp ../../b2 cd ../../ chmod +x bin.v2/tools/bcp bin.v2/tools/bcp/gcc-8/release/link-static/bcp LICENSE_1_0.txt boost/format.hpp boost/range/combine.hpp ../lib/boost cd .. rm -rf boost_X_Y_Z*
- Changes between v0.1.x versions may introduce breaking changes.
- The library only supports SNMPv2 at this time.
- BULKGET_REQUEST and NEXT_REQUEST will always perform a walk.
- Walks will always end if the root of the OID runs past the requested OID.
- Duplicate objects on the same host/request will be silently discarded.
- This includes the initial request; walks must be performed on an OID prior to the first desired.
- NO_SUCH_INSTANCE, NO_SUCH_OBJECT, and END_OF_MIB_VIEW response variable bindings are exposed as errors for handling by the client.