kpipe
This kpipe
module combines the current versions of the main core modules of the kpipe data processing system. Including this module in a project will enable the project to use the base kpipe functionality without managing dependencies of the individual modules.
The modules included are:
Module | Description |
---|---|
kpipe-core |
Storage backends and basic readable and writable stream implementations. |
kpipe-streams |
A set of low-level node transform streams to perform basic data conversion tasks such as compression, JSON translation, delineation, etc. |
kpipe-sequence |
Provides an enhanced node pipeline running as a promise as well as tools to sequence a list of promisified actions. |
kpipe-url |
Defines and supports a URL syntax for the kpipe stream backends. It automatically pipes together appropriate streams to deliver data in a particular format to/from a kpipe storage backend. |
kpipe-core
Support for reading and writing to storage backends using standard node streams.
Supported backends:
Backend | Description |
---|---|
stdio |
|
fs |
|
s3 |
|
kafka |
|
buffer |
|
random |
Function | Description |
---|---|
Reader |
|
Writer |
|
KafkaProducer |
|
KafkaAdmin |
kpipe-streams
Supported transforms:
Transform | Description |
---|---|
Lineate |
Add newlines to incoming strings and pack into a buffer |
Delineate |
Split buffer into individual strings delimited by newlines |
JSONParse |
Parse an incoming JSON string into a JavaScript object |
JSONStringify |
Generate a JSON string from incoming JavaScripts objects |
Gzip |
Compress a buffer stream using the zlib library |
Gunzip |
Decompress a buffer stream using the zlib library |
SnappyCompress |
Compress a buffer with snappy |
SnappyDecompress |
Decompress a buffer with snappy |
kpipe-sequence
PipelinePromise
PromiseChain
kpipe-url
Translate URL-like paths into Kpipe backend stream generators.
Pattern | Backend | Description |
---|---|---|
stdio:// - |
stdio |
|
fs://[path]/[file].[ext] file://[path]/[file].[ext] |
fs |
|
s3://[bucket](/[prefix])/[file].[ext] | s3 |
|
kafka://[topic](/[partition](/[offset])) | kafka |
Function | Description |
---|---|
readerUrl |
Return a Reader stream generator given a kpipe url |
readStreamUrl |
Return a readable stream given a kpipe url |
writerUrl |
Return a Writer stream generator given a kpipe url |
writeStreamUrl |
Return a writable stream given a kpipe url |
urlInStreams |
Given a kpipe url, return a readable stream which transforms backend data into the supplied content format (buffer , strings , or json ) |
urlOutStreams |
Given a kpipe url, return a writable stream which transforms data from the supplied content format (buffer , strings , or json ) to a kpipe backend |
urlOutMultiplex |
Special case of urlOutStreams which splits the incoming stream into one or more destination streams |
urlPipeline |
Given an input URL, some transforms, and an output URL, return a promisified pipeline which performs a data conversion task |
urlPipelineSorted |
Given an input URL, some transforms, and an output URL, return a promisified pipeline which first sorts the entire incoming dataset and then performs a data conversion task |
parseUrl |
Helper function returns the parsed components of a kpipe url |
contentModifiers |
Helper function returns an array of transforms to convert streams from one of the three content types (buffer , strings , or json ) into another content type. |
compressExt |
Helper function returns a transform stream (or passthrough) which compresses content according to a supplied filename extension |
decompressExt |
Helper function returns a transform stream (or passthrough) which decompresses content according to a supplied filename extension |