This is an event pipeline application where several producers can post their events to be validated and persisted so that another application can send the event to the end client.
Event contract validations will be described in a configuration file.
---
title: Integration Architecture
---
flowchart LR
EP1(Event Producer) ---> B
EP2(Event Producer) ---> B
EP3(Event Producer) ---> B
EP4(Event Producer) ---> B
B[(MessageBroker)] --> R
subgraph EPs[Event Processor]
direction LR
R[Receiver] --> V
V[Validator] --> P
P[Persist]
end
P --> DB
DB[(DataBase)] --> S
S(Sender) --> C1
S(Sender) --> C2
S(Sender) --> C3
C1([Client])
C2([Client])
C3([Client])
---
title: Event Processor activity diagram
---
flowchart TD
EV[(events)] -.-> R
R[Receiver] --save--> US
US[EventService] --validate--> Vt
Vt[Validator] --get--> C[/Schema/]
US --> V
V{Event is Valid} --false--> RE[Reject Event]
V --true--> P[Save on Repository]
RE[Return Error] -.-> DLQ[(events.dlq)]
- Cloning
.env.example
:
cd deployments
cp .env.example .env
You can adjust the .env
file to suit your infrastructure needs. Set LOG_LEVEL
to trace
to increase verbosity and gain detailed insights into the application's internal processes.
2. Starting dependencies:
docker compose up -d rabbitmq mongodb
- In deployments path, run terraform with:
export $(cat .env | xargs -I% echo TF_VAR_%)
cd terraform
terraform init
terraform apply
- Up event-processor service:
cd ../
docker compose up event-processor
After build image you will see:
starting event processor...
[*] Waiting for messages
- starting producers on another terminal:
cd deployments
docker compose up event-producer
Then you can see the events flowing from the producer to the processor
and being saved in mongoDb in the events
collection.
You can use this website to create a JSON Schema:
https://www.jsonschemavalidator.net/
using this file as a template: event-base.schema.json and following specification draft/2019-09
After building and validating schema, save on path configs/events-schemas/
with name <eventType>.schema.json
,
replacing <eventType>
with the eventType value expected to use this schema to validate them.
You can put this schema file on configs/events-schemas/transaction-allowed.schema.json
:
{
"$id": "file://configs/events-schemas/transaction-allowed.schema.json",
"$schema": "https://json-schema.org/draft/2019-09/schema",
"title": "transaction allowed event",
"description": "schema for all transaction allowed event",
"required": [
"eventType",
"tenantId"
],
"type": "object",
"properties": {
"eventType": {
"type": "string",
"minLength": 3,
"maxLength": 255
},
"tenantId": {
"type": "string",
"format": "uuid"
},
"transactionId": {
"type": "string",
"format": "uuid"
},
"userId": {
"type": "string",
"format": "uuid"
},
"value": {
"type": "number"
},
"latitude": {
"type": "number",
"minimum": -90,
"maximum": 90
},
"longitude": {
"type": "number",
"minimum": -180,
"maximum": 180
}
},
"additionalProperties": false
}
Then the event-processor
will be ready to validate this new event. Like, for example:
{
"eventType": "transaction-created",
"tenantId": "f25d45d5-213b-4de1-a35f-210f07fc55c4",
"transactionId": "f25d45d5-213b-4de1-a35f-210f07fc55c4",
"userId": "f25d45d5-213b-4de1-a35f-210f07fc55c4",
"value": 2345.23
}
Or
{
"eventType": "transaction-created",
"tenantId": "f25d45d5-213b-4de1-a35f-210f07fc55c4",
"transactionId": "f25d45d5-213b-4de1-a35f-210f07fc55c4",
"userId": "f25d45d5-213b-4de1-a35f-210f07fc55c4",
"value": 2345.23,
"latitude": 31,
"longitude": 48
}
For testing, you can put this json events on testdata
as a json file.
And run event-producer
to publish these events on queue.
- asyncapi.yaml can be viewed on the website: https://studio.asyncapi.com/
- JSON Schema
- JSON Schema specification: draft/2019-09
Before any commit you need golangci-lint
and mockery
to generate mocks,
you can add manually or running:
sh ./scripts/install.sh
Before running unit tests, ensure that all mocks have been generated using:
mockery --all
- Create Rabbitmq instance and provision it via terraform
- Create a consumer for the events
-
Create a configuration contract reader - Create a validator that uses the contract to reject invalid events
- Use a database to persist the events and be read in the future by the sender
- Create a docker container for the app
- Increase coverage test
- Improve logs
- Create acceptance tests
- Create load tests
- Create user on rabbitmq for application only
- Move to LocalStack
- Use terraform to declare resources
- Create a parameterizable producer to put events in the pipeline
- Change Dockerfile to be multi-stage build
- Optimize flow