hayrack

A tool for capturing messages from STDIN, such as when run as a program destination by Syslog-NG, and then load balance these messages over ZeroMQ to downstream worker nodes for processing.


License
Other
Install
pip install hayrack==0.0.6

Documentation

Hayrack

A tool for capturing messages from STDIN, such as when run as a program destination by Syslog-NG, and then load balance these messages over ZeroMQ to downstream worker nodes for processing.

Example Hayrack Server:

from hayrack import server

#start the IO Loop to read from STDIN
server.start_io('127.0.0.1', '5000')

The IO Loop will block the current process. The server binds to a ZeroMQ PUSH socket and forwards all messages received via STDIN. Messages sent over a PUSH socket are load balanced acorss all connected ZeroMQ PULL sockets. The server has a signal handler that will properly close sockets of a sigterm is recevied.

Example Client Side:

from hayrack import transport

#create a receiver
rcvr = transport.ZeroMQReceiver(connect_host_tuples=[('127.0.0.1', '5000')])
rcvr.connect()

#receive a single message
message = rcvr.get()

rcvr.close()

The client side receiver creates a ZeroMQ PULL socket that can connect to 1 or more ZeroMQ sockets, and pull messages across the socket.

Installing and running Hayrack

Hayrack can be used as a library as shown above or run as an application. The application is available for installation on PyPi.

To build a debian package for Hayrack you can run make_deb.sh file included in the repo. This will also create a configuration file at /etc/hayrack/hayrack.conf that will allow you to configure the Hayrack server. When installed the server can be run using /usr/share/hayrack/bin/run.py.

The initial intended use for this application is to serve as a transport mechanism for logging-as-a-service. Hayrack is used to receive messages from a local Syslog-NG server on STDIN and then load balance these messages over ZeroMQ to downstream worker nodes for processing.

The Chef cookbook syslogng-transport can be used for deploying Hayrack with Syslog-NG.