Deprecated, see https://github.com/eleme/node-bell

Bell

Realtime anomalies detection based on statsd, for periodic time series.

Latest version: v0.5.6 Pre-Alpha

Requirements

• python 2.7+ (currently exludes python3 due to beanstalkc)
• ssdb (datastore)
• beanstalkd (job queue)
• statsd (metrics source)

Installation

Install backend statsd-bell for statsd:

$ npm install statsd-bell

and then add statsd-bell to statsd's backends in statsd's config.js:

{
, backends: ["statsd-bell"]
}

Next install bell:

$ mkdir workspace-bell && cd workspace-bell
$ virtualenv venv
$ . venv/bin/activate
$ pip install bell

Quick Start

1. Start statsd & ssdb & beanstalkd.

2. Generate sample configuration and edit it, default: bell/res/configs.toml:

   $ bell sample configs   # generate configs.toml
   $ vi configs.toml

3. Start listener & analyzers (optional: webapp).

   $ bell analyzer configs.toml
   $ bell listener configs.toml
   $ bell webapp configs.toml

   You can view site on 0.0.0.0:8989.

Services

• listener: receives incomming metrics from carbon-relay over tcp, then put them to job queue.
• analyzer(s): get jobs from job queue, and then analyze if current metric an anomaly or not.
• webapp: visualizes analyzation result on web.

Cli Usage

Usage:
  bell sample configs
  bell <service> [<configs>] [--log-level=<l>]
  bell [-h|-v]

examples:

$ bell sample configs   # generate a copy of default configs here
$ bell analyzer ./configs.toml  # boot an analyzer instance using ./config.toml

API Reference

Python API

from bell.api import around, latest, now

Protos:

def around(series, timestamp=None, offset=10):
    """Return datapoints around a timestamp within offsets"""

def latest(series):
    """Return the latest datapoints in a timeseries, `None` if not found."""

def now(series):
    """Return current datapoint in a timeseries, `None` if not found."""

return value format:

(timestamp, value, is_anomaly)

examples:

>>> around('stats.zzz', 1406629167)
[(1406629157, 5.6, 1), (1406629167, 10.4, 1), (1406629177, 2.4, 0)]
>>> latest('stats.zzz')
(1406631147, 2.9, 0)

Analyzer Events&Hooks

When the specified events happen, analyzers will trigger the functions you provide. Bell comes with a built-in hook module: hooks/hipchat.py.

Use Hooks

To enable a hook function, we take hipchat as an example:

1. Add hook module to hooks.modules:

   [hooks]
   enable = true
   modules = ["bell.hooks.hipchat"]

2. Configure hooks.hipchat:

   [hooks.hipchat]
   roomId = 12345
   token = "your-hipchat-api-token"
   weburl = "http://bell.example.com"

3. Restart all analyzers.

Write Hooks

Sample code can be found at bell/hooks/hipchat.py.

All events and their parameters format:

@on_datapoint_reserved
def hook(datapoint):
    series_name, (timestamp, value) = datapoint
    pass

@on_anomaly_detected
def hook(datapoint):
    series_name, (timestamp, value) = datapoint
    pass

@on_analyzation_done
def hook(datapoint):
    series_name, (timestamp, value, is_anomaly) = datapoint
    pass

Note: analyzers call hook functions in a thread independent of main thread, so hooks like sendmail won't block analyzation.

Look Inside

Algorithm

3-sigma or called 68-95-99.7 rule.

Storage

Analyzers store metrics in ssdb, using zset, here is storage format for a single time series:

key       |  score
--------------------------------------
timestamp | value:is_anomaly:timestamp

Data Flow

 [statsd]
    |
    v        send to queue
[listener] -----------------> [beanstalkd]
                                  |
                                  | reserve
            history metrics       v       record anomalies
            ---------------> [analyzers] ----------------
            |                     |                     |
            |                     | put to ssdb         |
            |                     v                     |
            ------------------- [ssdb] <-----------------
                                  |
                                  |
                                  v
                               [webapp]

License

Copyright (c) 2014 Eleme, Inc. Detail see LICENSE-MIT