Hsdq
###High Speed Distributed Queue: #####A messaging layer allowing distributed applications/scripts to communicate easily and exchange data at high speed. Also allow to offload deferred tasks to outside applications/scripts.
Features
- Connect with ease applications and/or scripts together.
- Easy to scale horizontally by adding or removing new application on line.
- Easy to balance the load, no round-robbing, the first listening application listening will get the message, then the next one will get the next message.
- hot swap, hot deployment. If the app message API is compatible with the previous API version, just deploy the new version and stop the old one.
- No need to have the target application on line to send a message, they will be stored in the bus and the target app will begin processing as soon as back online. (HTTP API need to be online to get the request).
Dependencies
- HSDQ rely on Redis for the transport layer
Recommended
- Ruby 2.3.1, Minimum ruby 2
- Jruby 9.1.2 Minimum 9.0.3
How to use HSDQ
####Step 0: Getting a bit accustomed
Play with some quick and dirty scripts to get familiar, check the examples folder.
####Step 1: create the connected class
- Create a class in your application or script (for a Rails app in lib or accustomed in models)
- Prefix this class name with
Hsdq, ie:HsdqMyClass - Extend Hsdq into your class
The listening channel will then be created based on your class name ie: my_class
####Step 2: to get the events
Into your class override the 5 following methods:
| Method | Comment |
|---|---|
hsdq_request |
Called when a request is received |
hsdq_ack |
Called when an acknowledgment your request has been received by the receiving app |
hsdq_callback |
Called when the other other app respond to your request |
hsdq_feedback |
Called when the other app is sending intermediate feedback |
hsdq_error |
Called when an error occur |
You must implement these class methods into your class. They will receive the event. All methods are receiving 2 parameters: A Hash for the message and a Hash or nil for the context.
def self.hsdq_request(message, context)
self.new.hsdq_request message, context
end
def self.hsdq_request(message, context)
# Start your processing here
end Important: Unless options[:threaded] is set to false, each event received are totally decoupled and independent. So you need to keep track of the context if needed.
####Step 3: sending messages
Four methods are used to send and respond to messages.
They are implemented internally in Hsdq and need a Hash as parameter.
| Method | Comment |
|---|---|
hsdq_send_request(message) |
To send a request |
hsdq_send_calback(message) |
To send the final response to a request |
hsdq_send_feedback(message) |
To send intermediate data |
hsdq_send_error(message) |
To send an error message in case of error |
####Step 4:
Set the authorized topics and tasks in your class:
hsdq_authorized_topics :beer, :wine, :cheese, :dishes
hsdq_authorized_tasks :drink, :eat, :clean
Note you can in simple cases like in scripts only use topic or task but it is recommended to use both.
####Step 5: Setup the hsdq config file:
In the config folder create a file name hsdq_my_class.yml There ia a sample file in hsdq config folder.
- Each class you connect to HSDQ must have it's specific yml file. This allow you to shard the bus for heavy traffic application.
- Each subsection can point to a different Redis instance/host/db.
For heavy traffic sites, you can shard messages and have for example a unique host for admin. - Session can be split to different host if different applications do not share the sessions data
development:
redis:
message:
host: 127.0.0.1
port: 6379
db: 0
admin:
host: 127.0.0.1
port: 6379
db: 0
session:
host: 127.0.0.1
port: 6379
db: 0
Message specifications
#####Five type of message are running into the bus:
| Type | Description | |
|---|---|---|
| 1 | Request | Initial message sent |
| 2 | Ack | Acknowledgment of reception |
| 3 | Feedback | Intermediate response (progress) |
| 4 | Callback | Final response with the data |
| 5 | Error | Final response when an error occur |
#####A message event is composed of 2 parts:
1 - The spark:
- An ephemeral tiny part that include the minimum but sufficient information pointing to the 2nd part, the message itself.
- The spark is what the listener will be getting from the list queue and this will ignite the process.
- The spark is pushed to a redis list and popped by the listener. Once popped it will not be available in the redis layer anymore.
2 - The Burst:
- An event in the life of the message, the burst is stored as one value in a Redis hash and can be retrieved using the spark data.
#####The complete message
- It will contain all the operations executed during the life cycle of the message.
- It is stored into a hash and each element of this hash is one message event.
- The message element is stored in a Redis Hash with a default expiration of 72 hours (can be adjusted)
A message is composed of multiple events. At minimum there are 3 events:
- Request
- Ack
- Callback or Error
Multiple feedback can be sent before the callback/error and multiple events can be present in the case of chained events.
#####Structure of a message: The burst:
| Key | Type | Description | |
|---|---|---|---|
| sent_to | M | String | Name channel you publish |
| topic | O | Symbol | Topic to be processed (ex: :beer, :wine, :cheese etc...). |
| task | O | Symbol | Task to be processed (ex: :drink, eat, clean ,etc...). |
| params | M | Hash | parameters matching the receiver message API (can be empty hash) |
| data | C | Hash | Mandatory in the responses as this is the response |
| type | I | Symbol | Type of the message ie: :request, :callback, etc |
| sender | I | String | the listening channel of the sender. Used to reply |
| uid | I | UUID | Unique identifier for the message container (redis hash) |
| spark_uid | I | UUID | Unique identifier for the 'spark' and the 'burst' |
| tstamp | I | UTC | Timestamp for the event UTC |
| context | I | Hash | Data from the previous request. keys: :reply_to, spark_uid |
| previous_sender | I | String | The previous sender of a request. used in chained queries |
| hsdq_session | O | String | Key to load session data related to the context |
M Mandatory, O Optional (but recommended for pre-filtering), I Hsdq internal
- Topic and task values must be present respectively in the topics or task white lists when used.
- Redis return symbols as strings in the responses
The spark:
The spark has the same structure and values as the burst except it do not carry the payload. The params and data as well as eventual custom keys keys which can contain heavy payloads are not included.
