QuintoAndar Kafka Python Library
QuintoAndar's kafka-python lib wrapper with additional functionalities.
KafkaIdempotentConsumer
A simple wrapper for kafka-python lib that uses redis to check duplicate events.
Configuration
| Name | Description |
|---|---|
| group_id | The consumer group id |
| bootstrap_servers | The bootstrap servers |
| redis_host | The topic to consume from |
| redis_port | The function that processes the event |
| idempotent_key | Function which extract an unique identifier from the event |
See examples
Development
Environment
At the bare minimum you'll need the following for your development environment:
It is strongly recommended to also install and use pyenv:
This tool eases the burden of dealing with virtualenvs and having to activate and
deactivate'em by hand. Once you run pyenv local my-project-venv the directory you're
in will be bound to the my-project-venv virtual environment and then you will have
never to bother again activating the correct venv.
Getting Started
1. Clone the project:
git clone git@github.com:quintoandar/kafka-python.git
cd kafka-python2. Setup the python environment for the project:
make environment3. Install dependencies
You can just the make recipe:
make install
