VMware-Go-KCL

Overview

Amazon Kinesis enables real-time processing of streaming data at massive scale. Kinesis Streams is useful for rapidly moving data off data producers and then continuously processing the data, be it to transform the data before emitting to a data store, run real-time metrics and analytics, or derive more complex data streams for further processing.

The VMware Kinesis Client Library for GO (VMware-Go-KCL) enables Go developers to easily consume and process data from [Amazon Kinesis][kinesis].

VMware-Go-KCL brings Go/Kubernetes community with Go language native implementation of KCL matching exactly the same API and functional spec of original Java KCL v2.0 without the resource overhead of installing Java based MultiLangDaemon.

Besides, vmware-go-kcl-v2 is the v2 version of VMWare KCL for the Go programming language by utilizing AWS Go SDK V2.

Try it out

Prerequisites

Install Go
Install docker
Install HyperMake
Config AWS CLI

Make sure hmake is version 1.3.1 or above and go is version 1.11 or above

hmake --version
1.3.1

Make sure to launch Docker daemon with specified DNS server --dns DNS-SERVER-IP

On Ubuntu, update the file /etc/default/docker to put --dns DNS-SERVER-IP in DOCKER_OPTS.

On Mac, set DNS in Docker Preferences – Daemon – Insecure registries

Build & Run

hmake

# security scan
hmake scanast

# run test
hmake check

# run integration test
# update the worker_test.go to let it point to your Kinesis stream
hmake test

Documentation

VMware-Go-KCL matches exactly the same interface and programming model from original Amazon KCL, the best place for getting reference, tutorial is from Amazon itself:

Contributing

The vmware-go-kcl project team welcomes contributions from the community. Before you start working with vmware-go-kcl, please read our Developer Certificate of Origin. All contributions to this repository must be signed as described on that page. Your signature certifies that you wrote the patch or have the right to pass it on as an open-source patch. For more detailed information, refer to CONTRIBUTING.md.