Functional programming style APIs in Go - with concurrency support
Concepts
-
Pipeline
- allows for the definition of a data pipeline with a sequence of PreFilters, Transformers and PostFilters -
PipelineComponents
- utility structure to define the pipeline components. Including concurrency. -
FilterOperation
- allows determination of whether an element should stay or be filtered out during pipeline processing -
TransformOperation
- transforms element. additionally supports filtering. additionally supports returning errors -
FoldOperation
- folds / reduces elements down
Pipeline
The Pipeline is the overarching concept that provides the executable behviours of:
-
Apply
- takes a slice and applies the data pipeline to it -
ApplyAndFold
- applies data pipelien to slice and then folds it down to a single element -
Stream
- continuously streams data through the pipeline until closure
Concurrency
- Set the Concurrency in
PipelineComponents
to greater than 1 to get transparent concurrency when processing the data pipeline. - Data is smartly distributed between independent jobs that execute the data pipeline across the filters, transforms and even the folding operation.
- During streaming this may naturally result in out of sequence data handling, so beware!
- During slice processing (
Apply
) the result is sorted by original sequence allowing sequence to be maintained. - Goes without saying that when using concurrency all functions must be thread-safe.