Hideout

Data processing programs such as machine learning takes sorts of input data. After input data are loaded, program creates intermediate objects using input data. Getting or converting data takes large amount of the time. This problem prevents programmers to refactor data processing programs.

Install

We can install hideout with pip. Run the following command.

$ pip install hideout

Basic Usage

Hideout save and load object with hideout.resume. If the cache file for the object exist, hideout loads it otherwise call specified function to generate expected object.

large_object = hideout.resume_or_generate(
    label="large_object",
    func=generate_large_object,
    func_args={"source": "s3-northeast-8.amazonaws.com/large-dic.txt"}
)

hideout.resume have func_args option which contains the parameters of specified function to generate the expected object. We can specify the prefix of cache file with label option. When we do not specify the label option, resume_or_generate method automatically name the cache file from function name and the arguments.

Usage

Enable / Disable Cache

In default, Hideout is not activated and therefore does not save and load cache files. To enable cache we set the provided environment variable HIDOUT_ENABLE_CACHE to True.

$ HIDEOUT_ENBALE_CACHE=True your_data_engineering_program.py

Disable Cache for Specified Stages

Hideout provide stage for skipping caches for specified points. Users can add the stage names to the object generation by hideout.resume_or_generate with stage parameter.

large_object = hideout.resume_or_generate(
    label="large_object",
    func=generate_large_object,
    func_args={"source": "s3-northeast-8.amazonaws.com/large-dic.txt"}
)

Specifing stage names with HIDEOUT_SKIP_STAGES, hideout skip the caching. For example, the following command skip caching named preliminaries and integrate.

$ HIDEOUT_SKIP_STAGES=preliminaries,integrate your_data_engineering_program.py

Specify directory to Store Cache Files

In default, Hideout saves the cache files in caches under the top project directory. If we specify the directory, we specify it with environment variable HIDEOUT_CACHE_DIR.

Inject logger

When you want to apply the logger which you use throughout an application, you can inject the logger with hideout.set_logger() function.

For Developers

We can install the hideout package and upload it to pypi repository.

Install to local environment

$ python setup.py install

Upload to pypi

$ python setup.py sdist upload

License

MIT

Contribution

See CONTRIBUTING.md.