Kafka serialize python library

Kser is a bundle of python library whose purpose is to serialize tasks to be executed on Kafka consumers. It supports farious extensions:

Transport

librdkafka

You can choose to use the C bindings using confluent-kafka:

$ pip install kser[confluent]

kafka-python

You can choose the pure python library kafka-python:

$ pip install kser[pykafka]

http

There is also a light version using HTTP ideal to produce messages (not recommended for consumption)

$ pip install kser[http]

Other

encrypt data

It is possible to encrypt messages in the Kafka queue using libsodium.

$ pip install kser[crypto]

prometheus exporter

You can export metrics using the prometheus format:

$ pip install kser[prometheus]

A few set of environment variables allow to manage the exporter:

The exporter has only 2 metrics defined by default, it's just a sample. A good way to implement your own is to override the triggers methods (prefixed with '_') like the following example:

from kser import KSER_METRICS_ENABLED
from prometheus_client import Counter
from kser.entry import Entrypoint

MY_METRIC = Counter('kser_my_metric', 'a usefull metric')


class MyEntrypoint(Entrypoint):
    def _run(self):
        if KSER_METRICS_ENABLED == "yes":
            MY_METRIC.inc()

        return self.run()

See also:

prometheus_client documentation

Documentation of the python Prometheus client

opentracing support

You can use opentracing with Jaeger to follow operation and task:

$ pip install kser[opentracing]

Configuration can be set using environment variable:

See also:

Opentracing support with Jaeger

Kser documentation

Add-ons

• Flask extension: Flask integration.

Requirements

• Python 3.x

Documentations

• Project: http://kser.readthedocs.io/
• Libsodium: https://download.libsodium.org/doc/
• confluent-kafka-python: http://docs.confluent.io/current/clients/confluent-kafka-python
• kafka-python: http://kafka-python.readthedocs.io/en/master/

Other links

• PyPI: https://pypi.python.org/pypi/kser
• Project issues: https://github.com/cdumay/kser/issues

License

Licensed under MIT license (LICENSE or http://opensource.org/licenses/MIT)