cfg_load
Loading configuration files is a common task in many projects. This package does the job.
Installation
The recommended way to install cfg_load is:
$ pip install cfg_load[all] --userNote: You might have to escape [ and ] in some shells like ZSH.
If you want the latest version:
$ git clone https://github.com/MartinThoma/cfg_load.git; cd cfg_load
$ pip instell -e .[all] --userUsage
cfg_load is intended to be used as a library. In your code, it will mostly
be used like this:
import cfg_load
config = cfg_load.load("some/path.yaml")In order to check if it is doing what you expect, you can use it as a command line tool:
$ cfg_load tests/examples/cifar10_baseline.yaml
{ 'dataset': { 'script_path': '/home/moose/GitHub/cfg_loader/datasets/cifar10_keras.py'},
'evaluate': { 'augmentation_factor': 32,
'batch_size': 1000,
'data_augmentation': { 'channel_shift_range': 0,
'featurewise_center': False,
'height_shift_range': 0.15,
'horizontal_flip': True,
'rotation_range': 0,
'samplewise_center': False,
'samplewise_std_normalization': False,
'shear_range': 0,
'vertical_flip': False,
'width_shift_range': 0.15,
'zca_whitening': False,
'zoom_range': 0}},
'model': { 'script_path': '/home/moose/GitHub/cfg_loader/models/baseline.py'},
'optimizer': { 'initial_lr': 0.0001,
'script_path': '/home/moose/GitHub/cfg_loader/optimizers/adam_keras.py'},
'train': { 'artifacts_path': '/home/moose/GitHub/cfg_loader/artifacts/cifar10_baseline',
'batch_size': 64,
'data_augmentation': { 'channel_shift_range': 0,
'featurewise_center': False,
'height_shift_range': 0.1,
'horizontal_flip': True,
'rotation_range': 0,
'samplewise_center': False,
'samplewise_std_normalization': False,
'shear_range': 0,
'vertical_flip': False,
'width_shift_range': 0.1,
'zca_whitening': False,
'zoom_range': 0},
'epochs': 1000,
'script_path': '/home/moose/GitHub/cfg_loader/train/train_keras.py'}}You can see that it automatically detected that the file is a YAML file and
when you compare it to cfg_load examples/cifar10_baseline.yaml --raw you can
also see that it made the paths absolute.
Good Application Practice
import cfg_load
# Load defaults
base_cfg = cfg_load.load("some/path.yaml")
# Overwrite defaults if user defined it
user_cfg = cfg_load.load("other/path.yaml")
user_cfg = base_cfg.update(user_cfg)
# Overwrite user default with environment variables
env_mapping = cfg_load.load("other/env_mapping.yaml")
cfg = user_cfg.apply_env(env_mapping)Features
- You load your config like this:
cfg = cfg_load.load('examples/test.json') - No key that starts with
_will ever be touched. - Keys ending in
_pathwill be made absolute. - Don't worry about Unicode.
- Every key
[something]_module_pathtriggerscfg_loadto load the file found at[something]_module_pathas a Python module tocfg.modules['something']. - If an environment variable with the same name as a config key exists, the
take the value of the environment variable. Please note: If the type of
the overwritten key is not str, then
cfg_loadappliesjson.loadsto the environment variable. - Every key ending with
_load_urlhas to havesource_urlandsink_path. Files fromsource_urlwill be loaded automatically and stored in thesink_path. Apolicyparameter can specify if it should beload_alwaysorload_if_missing.
Not there, but planned fo the future:
- Every key
[something]_cfg_pathwill triggercfg_loadto search for another config file and append it at[something]. By this way you can define configuration files recursively.
Development
Check tests with tox.
