Sthir

Search using spectral bloom filters in static sites

Sthir can create memory efficient search feature for your static website. Sthir is equipped with an user friendly command-line interface. In two steps you can build a working search page for your website!

Description

Sthir is a library to create search functionality for your static websites. It scans your html pages for words and indexes these words in an efficient data structure called Spectral Bloom Filters. Spectral Bloom Filteres differs from regular ones as they can store counts for each hash (it can estimate, at minimum, how many times a hash was indexed). We are using an efficient base 15 decoding to compress and transfer the bloom filters at client side. Our goal can be described with a simple equation:

Less Memory Footprint + Term Frequency Knowledge = Perfect Search Functionality for Static Sites!

Installation

sthir runs on Python 3.5 or above.

Installation with pip via PyPI for OS X , Linux and Windows:

pip install sthir

To check installation run the following command:

sthir -h

If you see the help messages without any error then the installation was successful.

Documentation

Here is a working demo -

Our entire documentation is available in:

• PDF
• HTML

License

The library is licensed under MIT License. This project is developed by Parth Parikh, Mrunank Mistry, and Dhruvam Kothari.

Credits

• Spectral bloom filters by Saar Cohen and Yossi Matias. DOI: https://doi.org/10.1145/872757.872787
• Karan Lyons and Sascha Droste for MurmurHash3js.
• Zhuhongk and Nikolas for their code on Is there a pure python implementation of MurmurHash?.
• Data files are derived from the Google Web Trillion Word Corpus, as described by Thorsten Brants and Alex Franz, and distributed by the Linguistic Data Consortium. Subsets of this corpus distributed by Peter Novig. Corpus editing and cleanup by Josh Kaufman.

Contributing

1. Fork it (https://github.com/pncnmnp/Spectral-Bloom-Search/fork)
2. Create your feature branch (git checkout -b feature/fooBar)
3. Commit your changes (git commit -am 'Add some fooBar')
4. Push to the branch (git push origin feature/fooBar)
5. Create a new Pull Request