github.com/periclescesar/event-processor


Install
go get github.com/periclescesar/event-processor

Documentation

Event Processor

This is an event pipeline application where several producers can post their events to be validated and persisted so that another application can send the event to the end client.

Event contract validations will be described in a configuration file.

Architecture

---
title: Integration Architecture
---
flowchart LR
    EP1(Event Producer) ---> B
    EP2(Event Producer) ---> B
    EP3(Event Producer) ---> B
    EP4(Event Producer) ---> B
    B[(MessageBroker)]   --> R

    subgraph EPs[Event Processor]
        direction LR
        R[Receiver] --> V
        V[Validator] --> P
        P[Persist]
    end

    P --> DB
    DB[(DataBase)] --> S
    S(Sender) --> C1
    S(Sender) --> C2
    S(Sender) --> C3
    C1([Client])
    C2([Client])
    C3([Client])
Loading
---
title: Event Processor activity diagram
---
flowchart TD
    EV[(events)] -.-> R
    R[Receiver]    --save--> US
    US[EventService] --validate--> Vt
    Vt[Validator] --get--> C[/Schema/]
    US --> V
    V{Event is Valid} --false--> RE[Reject Event]
    V --true--> P[Save on Repository]
    RE[Return Error] -.-> DLQ[(events.dlq)]
Loading

Running the project

  1. Cloning .env.example:
cd deployments
cp .env.example .env

You can adjust the .env file to suit your infrastructure needs. Set LOG_LEVEL to trace to increase verbosity and gain detailed insights into the application's internal processes. 2. Starting dependencies:

docker compose up -d rabbitmq mongodb
  1. In deployments path, run terraform with:
export $(cat .env | xargs -I% echo TF_VAR_%)
cd terraform
terraform init
terraform apply
  1. Up event-processor service:
cd ../
docker compose up event-processor

After build image you will see:

starting event processor...
 [*] Waiting for messages
  1. starting producers on another terminal:
cd deployments
docker compose up event-producer

Then you can see the events flowing from the producer to the processor and being saved in mongoDb in the events collection.

Creating a new event schema

You can use this website to create a JSON Schema:

https://www.jsonschemavalidator.net/

using this file as a template: event-base.schema.json and following specification draft/2019-09

After building and validating schema, save on path configs/events-schemas/ with name <eventType>.schema.json, replacing <eventType> with the eventType value expected to use this schema to validate them.

example:

You can put this schema file on configs/events-schemas/transaction-allowed.schema.json:

{
	"$id": "file://configs/events-schemas/transaction-allowed.schema.json",
	"$schema": "https://json-schema.org/draft/2019-09/schema",
	"title": "transaction allowed event",
	"description": "schema for all transaction allowed event",
	"required": [
		"eventType",
		"tenantId"
	],
	"type": "object",
	"properties": {
		"eventType": {
			"type": "string",
			"minLength": 3,
			"maxLength": 255
		},
		"tenantId": {
			"type": "string",
			"format": "uuid"
		},
		"transactionId": {
			"type": "string",
			"format": "uuid"
		},
		"userId": {
			"type": "string",
			"format": "uuid"
		},
		"value": {
			"type": "number"
		},
		"latitude": {
			"type": "number",
			"minimum": -90,
			"maximum": 90
		},
		"longitude": {
			"type": "number",
			"minimum": -180,
			"maximum": 180
		}
	},
	"additionalProperties": false
}

Then the event-processor will be ready to validate this new event. Like, for example:

{
	"eventType": "transaction-created",
	"tenantId": "f25d45d5-213b-4de1-a35f-210f07fc55c4",
	"transactionId": "f25d45d5-213b-4de1-a35f-210f07fc55c4",
	"userId": "f25d45d5-213b-4de1-a35f-210f07fc55c4",
	"value": 2345.23
}

Or

{
	"eventType": "transaction-created",
	"tenantId": "f25d45d5-213b-4de1-a35f-210f07fc55c4",
	"transactionId": "f25d45d5-213b-4de1-a35f-210f07fc55c4",
	"userId": "f25d45d5-213b-4de1-a35f-210f07fc55c4",
	"value": 2345.23,
	"latitude": 31,
	"longitude": 48
}

For testing, you can put this json events on testdata as a json file. And run event-producer to publish these events on queue.

Docs

ADRs

For Devs:

Before any commit you need golangci-lint and mockery to generate mocks, you can add manually or running:

sh ./scripts/install.sh

Test

Before running unit tests, ensure that all mocks have been generated using:

mockery --all

@TODO

  • Create Rabbitmq instance and provision it via terraform
  • Create a consumer for the events
  • Create a configuration contract reader
  • Create a validator that uses the contract to reject invalid events
  • Use a database to persist the events and be read in the future by the sender
  • Create a docker container for the app
  • Increase coverage test
  • Improve logs
  • Create acceptance tests
  • Create load tests
  • Create user on rabbitmq for application only
  • Move to LocalStack
  • Use terraform to declare resources
  • Create a parameterizable producer to put events in the pipeline
  • Change Dockerfile to be multi-stage build
  • Optimize flow