Messager

Reliable message passing in distributed systems.

Architecture

A Messager system consists of servers and clients. Once a client has successfully connected to a server, messages defined in their protocol specification can be sent between them.

Below is a sequence diagram showing typical communication between a server and a client. The messages FooReq, FooRsp and BarInd are defined in the protocol my_protocol, which is defined later in this document.

+--------+                               +--------+
| client |                               | server |
+--------+                               +--------+
    |             1. TCP connect              |
    |========================================>|
    |                2. ping                  |
    |---------------------------------------->|
    |                   pong                  |
    |<----------------------------------------|
    |               3. FooReq                 |
    |========================================>|
    |                  FooRsp                 |
    |<========================================|
    |               4. BarInd                 |
    |========================================>|
    |                  BarInd                 |
    |========================================>|
    .                                         .
    .                                         .
    .                                         .
    |                5. ping                  |
    |---------------------------------------->|
    |                   pong                  |
    |<----------------------------------------|
    .                                         .
    .                                         .
    .                                         .

---: Background communication. No user interaction needed.

===: User initiated communication.

1. The client connects to the server.
2. The client sends a ping message to the server, which responds with a pong message. This is done in the background. The user interaction needed.
3. The client sends FooReq to the server, which responds with FooRsp.
4. The client sends BarInd twice to the server. No response is defined.
5. Another pair of ping-pong messages.

Messager protocol specification

All messages sent on the wire consists of a type, a size and optional payload. This enables both streaming and packet based transport protocols. The transport protocol must be reliable (guaranteed in order delivery).

+---------+---------+-----------------+
| 4b type | 4b size | <size>b payload |
+---------+---------+-----------------+

TYPE  SIZE  DESCRIPTION
------------------------------------------------
   1     n  User message.

            Encoded "message ClientToServer" for client to server
            messages and "message ServerToClient" for server to
            client messages.

   2     0  Ping message.
   3     0  Pong message.

User messages

User messages are defined using Googles Protocol Buffers language.

Here is an example defining a protocol called my_protocol.

syntax = "proto3";

// The protocol name.
package my_protocol;

// Messages sent from client to server.
message ClientToServer {
    oneof messages {
        FooReq foo_req = 1;
        BarInd bar_ind = 2;
        FieRsp fie_rsp = 3;
    }
}

// Messages sent from server to client.
message ServerToClient {
    oneof messages {
        FooRsp foo_rsp = 1;
        FieReq fie_req = 2;
    }
}

// Message definitions.
message FooReq {
}

message FooRsp {
}

message BarInd {
}

message FieReq {
}

message FieRsp {
}

Ping and pong messages

Clients pings the server periodically. A client will close the connection and report an error if the server does not answer with pong within given time. Likewise, the server will close the connection and report an error if it does not receive ping within given time.

The ping-pong mechanism is only used if the transport layer does not provide equivalent functionality.

C source code

Generate server and client side C source code.

$ messager generate_c_source examples/hello_world/hello_world.proto

Client side

Per client.

void PROTO_client_init();       // Initialize given client.
void PROTO_client_connect();    // Connect to the server.
void PROTO_client_disconnect(); // Disconnect from the server.
void PROTO_client_send();       // Send prepared message to server.

Per message.

void PROTO_client_init_MESSAGE(); // Initialize given message.

Server side

Per server.

void PROTO_server_init();          // Initialize given server.
void PROTO_server_serve_forever(); // Serve clients forever.
void PROTO_server_broadcast();     // Send prepared message to all clients.
void PROTO_server_send();          // Send prepared message to current client.
void PROTO_server_reply();         // Send prepared message to current client.
void PROTO_server_disconnect();    // Disconnect given client.

Per message.

void PROTO_server_init_MESSAGE(); // Initialize given message.