lazy-database

A simple lazy loaded key:value database


Keywords
lazy_db
License
MIT
Install
pip install lazy-database==0.1.0

Documentation

lazy_db

badge badge

A lazily loaded key:value db intended for use with large datasets that are too big to be loaded into memory. The database supports integers, strings, lists of integers, bytes, and dictionaries. This database is meant to strike a good balance of retrieval/insertion speed and memory usage. This database best fits a scenario where each key has a lot of data stored under it. Scenarios where values are under 100 bytes in size this database is not very well suited for.

Install with pip install lazy-database

Example usage:

from lazy_db import LazyDb

# Simple example usage
db = LazyDb("test.lazy")
db.write("test_value", "value")
print(db.read("test_value"))  # prints "value"
db.close()

# Or use a with statement to insure the database file is closed cleanly and avoid having to call db.close() on your own
with LazyDb("test2.lazy") as db:
    db.write("test_value", "value")
    print(db.read("test_value"))

How it works

File layout

All text in database files are encoded in utf-8 format. Each database has a json string at the start of the file that denotes the database's settings, who's end is marked with a NUL byte (00 in hex)

Each database entry is appended at the end of the file and is laid out in this format:

Name Size (bytes) Purpose
NUL 1 Marks the start of the entry. When the initial headers index, the starting byte of each entry is checked for this NUL byte to be sure the database hasn't been corrupted. This is the beginning to what is considered the "header" for the entry (NUL bytes carry a hex value of 0x00)
Key type 1 Marks if the key is an integer or a string.
Key any The key for the database entry.
NUL 1 Marks the end of the key. This is necessary since string keys don't have a set size.
Content length content_int_size An integer (little endian) depicting the length of the content (including the content type). Defaults to 4 bytes long. This is the end to what is considered the "header" for the entry
Content type 1 Marks if the content is a string, int, int list, dict, or bytes.
Content Content length Stores the content

Content type labels

Name Hex type value Type description
String 0x01 A utf-8 string
Int 0x02 An integer
Dict 0x03 A dictionary (internally stored as a utf-8 json string)
Int list 0x04 A list of integers. Max integer size is defined by int_list_size (default: 4 bytes)
Bytes 0x05 A bytes object

The algorithm

When loading a database, all entry headers are scanned for their key value and lengths. This allows for values to be retrieved very quickly without having to load the content of every entry, at the cost of having to store the key and content length in memory though. This approach makes the database best for cases where your database will be storing a lot of data in each key that you can't afford to store in memory, however you can afford to store the name values and lengths of each element in memory.