eneel is a cli utility for fast extracting and loading data to and from relational databases. The E and L in ELT!
The goal is to be the fastest tool in the market. Both in terms of development speed and data movements speed.
- Fastest way to extract and load large amounts of data between different databases
- Automatic table creation and conversion of datatypes
- Load a query from one database to another
- Loads are executed in parallel
- Incremental loads
- Pretty runtime logging of load progress
- Database logging
- Limit rows load in development
- Configuration with yaml files. Just list the tables you want to replicate
- Configuration files approach makes versioning with git etc easier
Make sure you have the cli-tools for your required databases then:
pip install eneel
Or clone the repository, go to the directory and:
python setup.py install
After installation an example connections.yml file will be in your home directory (~/.eneel). That's where you configure your connection info to your sources and targets.
Next, create a project configuration file in a directory for you EL projects.
Go to the directory with the project configuration file and run eneel with the project file name excluding yml. I.e for a project configuration file as load_from_postgres_prod_to_dw.yml run:
--connections: add a path to the connections.yml you would like to use
--target: specify which target in connections.yml you would like to use. This will be applied on both sources and targets
- Support for Snowflake
- Support for BigQuery
- Incremental loads with updates
- Incremental loads with deletes
Reporting bugs and contributing code
- Go ahead and report an issue
- Want to help? Pull requests are most welcome!