Disk Queue Processing (dqp)
This library makes it easy to serialize Python to disk using msgpack.
The library contains two modules:
-
dqp.disk_cache: to easily read/write data to disk and support caching iterators to disk. -
dqp.disk_queue: an approach to communicating lists of objects between runs with index/offset metadata.
Example of disk_cache
from typing import Iterator
from dqp.disk_cache import cached_iter
import time
@cached_iter()
def expensive_iter() -> Iterator[int]:
for idx in range(5):
time.sleep(1)
yield idx
def main():
#For replay, drop the cache
expensive_iter.cache_clear()
# First time it takes seconds
for i in expensive_iter():
print(time.asctime(), i)
# Other times the cache is already there
print(time.asctime(), list(expensive_iter()))
print(time.asctime(), list(expensive_iter()))
print(time.asctime(), list(expensive_iter()))
if __name__ == "__main__":
main()
