blowpipe

blowpipe is a docker based data pipeline / workflow engine.


Keywords
blowpipe, data, pipeline, docker, workflow, container
License
Apache-2.0
Install
pip install blowpipe==0.0.8

Documentation

blowpipe

Introduction

Blowpipe is a tool to create and manage data pipelines, called Workflows. Each Step in a Workflow is implemented as a Container.

It began as an exercise in gaining a better understanding of data processing with forced isolation between Steps and has grown a bit in scope since then.

The arbitrary restriction of isolation means we get to investigate the impact in terms of design and implementation of data processing pipelines when your unit of code becomes a container.

It's a work in progress.

Installation

pip install blowpipe

Documentation

Learn how to use blowpipe starting with the first USER_GUIDE.md then move on to the full set of TUTORIALS.md.

Development

This is an open-source piece of software - I encourage you to take part! - please read the

to get started, then fork and PR away.

Current Plan

Please read DOING.md for what's in progress right now and the CHANGELOG.md for what has already been done.