github.com/eliquious/hraftd

A reference use of Hashicorp's Raft implementation


License
MIT
Install
go get github.com/eliquious/hraftd

Documentation

For background on this project check out this blog post.

hraftd Circle CI GoDoc

hraftd is a reference example use of the Hashicorp Raft implementation, inspired by raftd. Raft is a distributed consensus protocol, meaning its purpose is to ensure that a set of nodes -- a cluster -- agree on the state of some arbitrary state machine, even when nodes are vulnerable to failure and network partitions. Distributed consensus is a fundamental concept when it comes to building fault-tolerant systems.

A simple example system like hraftd makes it easy to study the Raft consensus protocol in general, and Hashicorp's Raft implementation in particular.

Reading and Writing Keys

Like raftd, the implementation is a very simple key-value store. You can set a key like so:

curl -XPOST localhost:11000/key -d '{"foo": "bar"}'

You can read the value for a key like so:

curl -XGET localhost:11000/key/foo

Running hraftd

Starting and running a hraftd cluster is easy. Download hraftd like so:

mkdir hraftd
cd hraftd/
export GOPATH=$PWD
go get github.com/otoolep/hraftd

Run your first hraftd node like so:

$GOPATH/bin/hraftd ~/node0

You can now set a key and read its value back:

curl -XPOST localhost:11000/key -d '{"user1": "batman"}'
curl -XGET localhost:11000/key/user1

Bring up a cluster

Let's bring up 2 more nodes, so we have a 3-node cluster. That way we can tolerate the failure of 1 node:

$GOPATH/bin/hraftd -haddr :11001 -raddr :12001 -join :11000 ~/node1
$GOPATH/bin/hraftd -haddr :11002 -raddr :12002 -join :11000 ~/node2

This tells each new node to join the existing node. Once joined, each node now knows about the key:

curl -XGET localhost:11000/key/user1
curl -XGET localhost:11001/key/user1
curl -XGET localhost:11002/key/user1

Furthermore you can add a second key:

curl -XPOST localhost:11000/key -d '{"user2": "robin"}'

Confirm that the new key has been set like so:

curl -XGET localhost:11000/key/user2
curl -XGET localhost:11001/key/user2
curl -XGET localhost:11002/key/user2

Stale reads

Because any node will answer a GET request, and nodes may "fall behind" updates, stale reads are possible. Again, hraftd is a simple program, for the purpose of demonstrating a distributed key-value store. These shortcomings can be addressed by enhancements to the existing source.

Tolerating failure

Kill the leader process and watch one of the other nodes be elected leader. The keys are still available for query on the other nodes, and you can set keys on the new leader. Furthermore when the first node is restarted, it will rejoin the cluster and learn about any updates that occurred while it was down.

A 3-node cluster can tolerate the failure of a single node, but a 5-node cluster can tolerate the failure of two nodes. But 5-node clusters require that the leader contact a larger number of nodes before any change e.g. setting a key's value, can be considered committed.

Leader-forwarding

Automatically forwarding requests to set keys to the current leader is not implemented. The client must always send requests to change a key to the leader or an error will be returned.

Credits

Thanks to the authors of raftd for providing the inspiration for this system. The current use of Raft by InfluxDB was also helpful.