eneel

A package for fast loading av relational data


License
Apache-2.0
Install
pip install eneel==0.1.1

Documentation

eneel

eneel is a cli utility for fast extracting and loading data to and from relational databases. The E and L in ELT!

The goal is to be the fastest tool in the market. Both in terms of development speed and data movements speed.

Features:

  • Fastest way to extract and load large amounts of data between different databases
  • Automatic table creation and conversion of datatypes
  • Load a query from one database to another
  • Loads are executed in parallel
  • Incremental loads
  • Pretty runtime logging of load progress
  • Database logging
  • Limit rows load in development
  • Configuration with yaml files. Just list the tables you want to replicate
  • Configuration files approach makes versioning with git etc easier

Installation

Make sure you have the cli-tools for your required databases then:

  • With pip:

    pip install eneel

  • Or clone the repository, go to the directory and:

    python setup.py install

Configuration

After installation an example connections.yml file will be in your home directory (~/.eneel). That's where you configure your connection info to your sources and targets.

Next, create a project configuration file in a directory for you EL projects.

Running eneel

Go to the directory with the project configuration file and run eneel with the project file name excluding yml. I.e for a project configuration file as load_from_postgres_prod_to_dw.yml run:

eneel load_from_postgres_prod_to_dw

The output will then be something like below for at successfull run: alt text

Optional parameters:

  • --connections: add a path to the connections.yml you would like to use
  • --target: specify which target in connections.yml you would like to use. This will be applied on both sources and targets

Feature matrix

Database Source Target
Postgres YES YES
Sql Server YES YES
Oracle YES NO

Roadmap

  • Support for Snowflake
  • Support for BigQuery
  • Incremental loads with updates
  • Incremental loads with deletes

Reporting bugs and contributing code

  • Go ahead and report an issue
  • Want to help? Pull requests are most welcome!