fnv-c

Python 3.7+ FNV (fnv0, fnv1, fnv1a) non-cryptographic hash library implemented in C through libffi


Keywords
fnv-algorithms, fnv0, fnv1, fnv1a, fnv32, fnv64, libffi, python3
License
MIT
Install
pip install fnv-c==0.2.0

Documentation

fnv-c

GitHub Workflow Status Codecov pypi badge

What is it?

fnv-c is a Python 3.7+ FNV (fnv0, fnv1, fnv1a) non-cryptographic hash library implemented in C through libffi.

FNV ("Fowler–Noll–Vo") is is a non-cryptographic hash function created by Glenn Fowler, Landon Curt Noll, and Kiem-Phong. FNV is probably no the "best" non-cryptographic hash function but:

  • it has a reasonably good distribution
  • it's very fast
  • it's very easy to implement (even in some exotic stored procedures for example) so you can use it everywhere

More details on this Wikepedia article.

Features

  • speed:
    • up to 800 MB/s hashing speed (on Macbook Pro M1 (2020) with fnv0_64)
    • 6 800% faster than basic Python implementation, 70% faster than pyhash (when hashing 100 bytes with fnv0_64 on a cloud VM)
  • portability:
    • tested with recent Python versions (3.7+)
    • compatible with ARM64
    • compatible (and tested) with PyPy

Non features

  • other hash algorithms (this library is only about FNV algorithm)
  • too agressive CPU optimizations (we prefer maximizing binary portability)

Benchmark

You have a benchmark script here to bench fnv-c by yourself and to compare it with:

  • fnvhash (pure python implementation)
  • pyhash (more general hashing library with C++ extension)

Comparisons with other libraries (fnv0_64 on a cloud VM)

Differences with fnvhash are huge (from 35% for one byte hashing to 19 000% for 1 000 bytes hashing with fnv0_64)

Differences with pyhash (on fnv0_64) are shown with the following diagram:

Influence of string size on fnv-c hashing speed (on a Macbook Pro M1 (2020) with fnv0_64)

How to install/use it?

pip install fnv-c
import fnv_c

print(fnv_c.fnv0_32(b"foo bar"))
print(fnv_c.fnv0_64(b"foo bar"))
print(fnv_c.fnv1_32(b"foo bar"))
print(fnv_c.fnv1_64(b"foo bar"))
print(fnv_c.fnv1a_32(b"foo bar"))
print(fnv_c.fnv1a_64(b"foo bar"))

Function signatures / API

Full API doc is available at: https://botify-labs.github.io/fnv-c/fnv_c/

Dev

See this specific document