MetalPipe: Modules for ETL Pipelines
MetalPipe is a lightweight, multithreaded framework for building ETL pipelines. It utilizes a design pattern similar to stream-processing frameworks such as Spark or Storm. But unlike those heavyweight systems, MetalPipe is designed for ETL, not data analytics.
The goals of MetalPipe are:
- To speed up ETL pipeline development by replacing as much code as possible with simple configurations.
- To make ETL pipelines faster by eliminating IO bottlenecks.
- To enable robust monitoring and error-handling into all ETL pipelines by default.
- To eliminate the need for specialized, heavyweight infrastructure for ETL jobs.
Documentation lives here:
https://metalpipe.readthedocs.io/en/latest/
MetalPipe is a work in progress. Although we use it in production, it should not generally be considered stable.
