elm-broker

Data stream buffer for Elm application, inspired by Apache Kafka.

What is this?

• Broker is essentially a circular buffer, internally using Array
• Read pointers (Offset) are exposed to clients, allowing "pull"-style data consumption, just as in Kafka
• Insert(append), read, and update all take O(1)
• A buffer is made of multiple Segments. Buffer size (= number of Segments and size of each Segment) can be configured
• When the whole buffer is filled up, a new "cycle" begins and old Segments are evicted one by one

Expected Usage

• A Broker accepts incoming data stream. †
• Several consumers reads ("pulls") data from the Broker individually, while maintaining each Offset as their internal states.
• Consumers perform arbitrary operations against acquired data, then read Broker again after previous Offset. Rinse and repeat.

† It is possible to have multiple Brokers in your application for different purposes, however you must be careful so that you do not mix up Offsets produced from one Broker to ones from others. Since Offsets are only valid for their generating Broker. Wrapping Offsets in phantom types is a possible technique to enforce this restriction.

Remarks

• Technically, it can also perform O(1) delete, but it is still unclear whether we want delete API
  • Original Kafka now supports this as an admin command
• There are several major features I am interested in:
  • More sophisticated/efficient dump and reload. Current implementation is plain old encode and decoder pair, which is potentially inefficient for big-capacity Brokers.
  • Bulk append and bulk read
  • Callback mechanism around Segment eviction

Development

Install Elm Platform.

$ elm make
$ elm-test                    # full test
$ elm-test tests/MainTest.elm # only light-weight tests

License

Copyright © 2018, Yu Matsuzawa

BSD-3-Clause