Sensorizer
A python application built in a Docker container to simulate sensor data flow to disk or event hubs, is meant to be the starting point of a data pipeline.
The main characteristic is that it tries to simulate traffic with similar timings, that is, it will release up to 400K readings per second, one by one. Then we can go for a streaming option (Azure Event Hub implemented) or a disk option (Avro implemented), batch the data points or not, etc...
Getting Started
Clone the project from github and enjoy.
Prerequisites
This software has been tested in Linux, it might work in other OSs but it is definitely not warrantied.
- Ubuntu latest stable / Debian Stretch / Fedora +25
- Python 3.7 (Dataclasses and typing in the code)
- Docker (if you want a container deployment)
Installing
Python requirements:
pip install -r requirements.txt
Running the tests
As simple as:
pytest sensorizer/tests/
Deployment
The deployment is container based, you just need to build the docker container:
docker build .
The docker container defined in the Dockerfile will start up a process (bin/emulate_full_plants.py) that will push the data into the configured queue.
docker run -ti <image> /bin/bash
You can change the command that will run in the container by editing the file: deploy/sensor_emulator_task.conf
Built With
- Python 3.7
- Docker
Contributing
Simply put, per branch features, merge to master, so:
- Fork the repo.
- Make a feature branch and develop.
- Test :)
- Create a pull request for your new feature.