dlog monitor

go get


dlog monitor

Author: Gao Peng <>
Description: kaixin dlog monitor framework, simulation of hadoop map reduce framework
Revision: $Id$

1   Introduction

This is a single-node map reduce framework implementation. It is widely used in kaixin's dlog analysis and reporting system.

2   Package with potential usage

2.1   consisent hash


2.2   msgpack


3   Todo

  • why first strings.Contains(100.123) will be slower to 280000 per sec

    enhance performance of strings.Contains

  • signal skipped not work

  • chain of workers and reducers

  • Job abstraction

    it has member Option

  • MapReduce

    Input -> Map -> Shuffle -> Reduce -> Output

  • select count(*) as count, service from dlog.20121212 where time>2000 group by service order by count desc

  • if group, each group is a dedicated reducer and goroutine

  • func return chan

    func f() chan int {
        ch := make(chan int)
        go func() {
            for i:=0; i<10; i++ {
                ch <- i
        return ch
    x := f()