Aggregate project for the kpipe data pipeline


License
MIT
Install
npm install kpipe@0.14.0

Documentation

kpipe

This kpipe module combines the current versions of the main core modules of the kpipe data processing system. Including this module in a project will enable the project to use the base kpipe functionality without managing dependencies of the individual modules.

The modules included are:

Module Description
kpipe-core Storage backends and basic readable and writable stream implementations.
kpipe-streams A set of low-level node transform streams to perform basic data conversion tasks such as compression, JSON translation, delineation, etc.
kpipe-sequence Provides an enhanced node pipeline running as a promise as well as tools to sequence a list of promisified actions.
kpipe-url Defines and supports a URL syntax for the kpipe stream backends. It automatically pipes together appropriate streams to deliver data in a particular format to/from a kpipe storage backend.

kpipe-core

Support for reading and writing to storage backends using standard node streams.

Supported backends:

Backend Description
stdio
fs
s3
kafka
buffer
random
Function Description
Reader
Writer
KafkaProducer
KafkaAdmin

kpipe-streams

Supported transforms:

Transform Description
Lineate Add newlines to incoming strings and pack into a buffer
Delineate Split buffer into individual strings delimited by newlines
JSONParse Parse an incoming JSON string into a JavaScript object
JSONStringify Generate a JSON string from incoming JavaScripts objects
Gzip Compress a buffer stream using the zlib library
Gunzip Decompress a buffer stream using the zlib library
SnappyCompress Compress a buffer with snappy
SnappyDecompress Decompress a buffer with snappy

kpipe-sequence

PipelinePromise

PromiseChain


kpipe-url

Translate URL-like paths into Kpipe backend stream generators.

Pattern Backend Description
stdio://
-
stdio
fs://[path]/[file].[ext]
file://[path]/[file].[ext]
fs
s3://[bucket](/[prefix])/[file].[ext] s3
kafka://[topic](/[partition](/[offset])) kafka
Function Description
readerUrl Return a Reader stream generator given a kpipe url
readStreamUrl Return a readable stream given a kpipe url
writerUrl Return a Writer stream generator given a kpipe url
writeStreamUrl Return a writable stream given a kpipe url
urlInStreams Given a kpipe url, return a readable stream which transforms backend data into the supplied content format (buffer, strings, or json)
urlOutStreams Given a kpipe url, return a writable stream which transforms data from the supplied content format (buffer, strings, or json) to a kpipe backend
urlOutMultiplex Special case of urlOutStreams which splits the incoming stream into one or more destination streams
urlPipeline Given an input URL, some transforms, and an output URL, return a promisified pipeline which performs a data conversion task
urlPipelineSorted Given an input URL, some transforms, and an output URL, return a promisified pipeline which first sorts the entire incoming dataset and then performs a data conversion task
parseUrl Helper function returns the parsed components of a kpipe url
contentModifiers Helper function returns an array of transforms to convert streams from one of the three content types (buffer, strings, or json) into another content type.
compressExt Helper function returns a transform stream (or passthrough) which compresses content according to a supplied filename extension
decompressExt Helper function returns a transform stream (or passthrough) which decompresses content according to a supplied filename extension