kinesis-stream-consumer
Kinesis stream consumer channelize through redis along with aws autorefreshable session
Usage
Requirements
- python >= 3.0
- boto3 >= 1.13.5
- kinesis-python >= 0.2.1
- redis >= 3.5.0
Installation
Install with:
pip install kinesis-stream-consumerOr, if you're using a development version cloned from this repository:
git clone https://github.com/harshittrivedi78/kinesis-stream-consumer.git
python kinesis-stream-consumer/setup.py installThis will install boto3 >= 1.13.5 and kinesis-python >= 0.2.1 and redis >= 3.5.0
How to use it?
There is two consumer which has to be run parallelly one is kinesis consumer and second is records queue consumer (redis). I have added a example.py file in this code base which can be used to check and test the code.
import threading
from kinesis_stream.consumer import KinesisConsumer
from kinesis_stream.record_queue import RecordQueueConsumer
from kinesis_stream.redis_wrapper import get_redis_conn
redis_conn = get_redis_conn(host="localhost", port=6379, db="0")
stream_name = "test-kinesis-stream"
region = "eu-west-1"
kinesis_consumer = KinesisConsumer(stream_name, region, redis_conn)
record_queue_consumer = RecordQueueConsumer(stream_name, redis_conn)
kinesis_consumer_thread = threading.Thread(name='kinesis_consumer', target=kinesis_consumer.start)
kinesis_consumer_thread.start()
record_queue_consumer_thread = threading.Thread(name='record_queue_consumer', target=record_queue_consumer.start)
record_queue_consumer_thread.start()Override handle_message func to do some stuff with the kinesis messages.
from kinesis_stream.record_queue import RecordQueueConsumer as BaseRecordQueueConsumer
class RecordQueueConsumer(BaseRecordQueueConsumer):
def handle_message(self, message):
# your code
print(message)
