pyfiles

A Big file collection manager.

CLI

Create a settings.py files where you want to execute the cli with this configuration for file storage:

And for S3:

Then to store a file:

$ pyfiles store <file path> <file.namespace> <file.name> <version>

version should respect the format: YYYY.MM.DD-Rev or any semver like X.Y.Z

To list all version of a file:

$ pyfiles versions <file.namespace> <file.name>

To search for a file:

$ pyfiles search <file.namespace> <file.name> [<version-prefix>]

version-prefix can be YYYY or X or YYYY.MM or X.Y or YYYY.MM.DD or X.Y.Z or Latest. Latest by default if missing.

Finnaly to delete a file:

$ pyfiles delete <file.namespace> <file.name> <version>

To start the web api server:

$ pyfiles serve

Web API

GET on /search/<namespace>/<filename>[?version=<version>]

To get file version download link. Namespace is a namespace to organise data and filename is the file name. You can optionnaly add a version like latest or <year> or <year.month> or <major> or <major>.<minor>, ... You get the latest for the specified version.

GET on /versions/<namespace>/<filename>

To show all avaible file versions.

Python API

See pyfiles.storage classes for more informations.

You can use pyfiles.storage.get_storage(<backend path>, <options>) to initialize your storage.

Features

• An API to download files with rich version selection
• List all version of a file
• Can be used for CSV or Geojson files
• File can have version like 2018.01.10-01
• Find file by a part of the version. 2018 or 2018.01

Roadmap

• Allow authentification with private data
• Handle file diff between versions
• Get the update date of a file to ease caching
• Add a client library and CLI

License

• Free software: MIT license
• Documentation: https://pyfiles.readthedocs.io.

Credits

This package was created with Cookiecutter and the audreyr/cookiecutter-pypackage project template.