AlephBet

AlephBet is a pure-javascript A/B (multivariate) testing framework for developers.

Key Features:

• NEW: run your own tracking backend on Rails with the alephbet rubygem (still experimental)
• NEW: run your own tracking backend on AWS Lambda with Lamed (recommended)
• Weighted variants. See #20
• user-based / cross-device experiments. See #16
• run your own tracking backend on AWS Lambda with Gimel (recommended)
• Pluggable backends: event tracking (defaults to Google Universal Analytics), and storage (defaults to localStorage)
• Supports multiple variants and goals
• Tracks unique visitors and goal completions
• Flexible triggers
• Ideal for use with page and fragment caching
• Developer-friendly for both usage and contirbution (using npm / webpack)

What does AlephBet mean?

Aleph (אלף) Bet (בית) are the first two letters in the Hebrew alphabet. Similar to A and B.

Inspiration

AlephBet was heavily inspired by Optimizely (sans WYSIWYG and reporting) and Cohorts.js. The code structure and some code elements were taken from cohorts.js, with some notable changes to terminology and built-in support for unique goals and visitor tracking.

For more detailed info about the background and rationale for creating AlephBet, please check out this blog post

Screencast

code snippet used on the screencast

for more screencasts, tips and info, please check the wiki

Quick Start

• Make sure your Google Universal analytics is set up.
• Download and include alephbet.min.js in the head section of your HTML.
• Or import it in your javascript code

import AlephBet from "alephbet" # ES6 module syntax
const AlephBet = require("alephbet") # commonJS syntax

• Create an experiment:

import AlephBet from "alephbet";

const button_color_experiment = new AlephBet.Experiment({
  name: 'button color',  // the name of this experiment; required.
  variants: {  // variants for this experiment; required.
    blue: {
      activate: function() {  // activate function to execute if variant is selected
        $('#my-btn').attr('style', 'color: blue;');
      }
    },
    red: {
      activate: function() {
        $('#my-btn').attr('style', 'color: red;');
      }
    }
  },
});

• Track goals for your experiment:

// creating a goal
const button_clicked_goal = new AlephBet.Goal('button clicked');
$('#my-btn').on('click', function() {
  // The chosen variant will be tied to the goal automatically
  button_clicked_goal.complete();
});

// adding experiment to the goal
button_clicked_goal.add_experiment(button_color_experiment);

// alternatively - add the goal to the experiment
button_color_experiment.add_goal(button_clicked_goal);

// tracking non-unique goals, e.g. page views
const page_views = new AlephBet.Goal('page view', {unique: false});

• view results on your Google Analytics Event Tracking Section. The experiment name + variation will be assigned to actions, and Visitors or Goals to label. e.g.

  • action: button color | red, label: Visitors : unique count of visitors assigned to the red variant.
  • button color | blue, button clicked : unique visitors clicking on the button assigned to the blue variant.
  • button color | red, viewed page : count of pages viewed by all visitors (not-unique) after the experiment started.

• important note: whilst Google Analytics is the easiest way to start playing with Alephbet, it's definitely not the best way to use it. GA starts sampling events after you reach a certain volume, and the built-in GA adapter does not support more advanced features like cross-device tracking. If you're serious about running A/B tests, I would urge you to consider using Lamed, Gimel or another backend instead.

Advanced Usage

Recommended Usage Pattern

AlephBet was meant to be used across different pages, tracking multiple goals over simultaneous experiments. It is therefore recommended to keep all experiments in one javascript file, shared across all pages. This allows sharing goals across different experiments. Experiments can be triggered based on a set of conditions, allowing to fine-tune the audience for the experiments (e.g. mobile users, logged-in etc).

Triggers

Experiments automatically start by default. However, a trigger function can be provided, to limit the audience or the page(s) where the experiment "kicks-off".

import AlephBet from "alephbet";

const button_color_experiment = new Alephbet.Experiment({
  name: 'button color',
  trigger: () => {
    return window.location.href.match(/pricing/);
  },
  variants: { // ...
  },
});

// triggers can be assigned to a variable and shared / re-used
const logged_in_user = function() { return document.cookie.match(/__session/); };
const mobile_browser = function() { // test if mobile browser };

const big_header_experiment = new Alephbet.Experiment({
  name: 'big header',
  trigger: () => { return logged_in_user() && mobile_browser(); },
  // ...
});

NOTE: once a user participates in an experiment, the trigger is no longer checked. See #9

Sample size

You can specify a sample float (between 0.0 and 1.0) to limit the number of visitors participating in an experiment.

Weights

Whilst sample will limit the entire experiment to a subset of potential participants, weight allows you to apply a weighted-random selection of variants. This can be considered a first step (manual) way to implement Multi Armed Bandit testing.

NOTE: Weights can be any integer value. Do not use floats. You can use any number, but it's probably easiest to treat it as a percentage, e.g. use weights of 80, 20 to allocate ~80% to one variant vs. ~20% to the other.

import {Experiment} from "alephbet"

const button_color_experiment = new Experiment({
  name: 'button color',  // the name of this experiment; required.
  variants: {  // variants for this experiment; required.
    blue: {
      activate: function() {  // activate function to execute if variant is selected
        $('#my-btn').attr('style', 'color: blue;');
      },
      weight: 50 // optional, can be any integer value
    },
    red: {
      activate: function() {
        $('#my-btn').attr('style', 'color: red;');
      },
      weight: 50
    }
  },
});

Visitors

Visitors will be tracked once they participate in an experiment (and only once). Once a visitor participates in an experiment, the same variant will always be shown to them. If visitors are excluded from the sample, they will be permanently excluded from seeing the experiment. Triggers however will be checked more than once, to allow launching experiments under specific conditions for the same visitor.

User-based / Cross-device tracking

You can now pass a user_id to the experiment as an optional parameter. This allows experiment to work across devices on a per-user basis.

import AlephBet from "alephbet";

const button_color_experiment = new Alephbet.Experiment({
  name: 'button color',
  user_id: get_user_id(),  // pass over the unique user id bound to this experiment
  trigger: () => {
    // do not trigger this expeirment without a user_id
    return get_user_id() && other_condition();
  },
  variants: {  // variants for this experiment; required.
    blue: {
      activate: function() {  // activate function to execute if variant is selected
        $('#my-btn').attr('style', 'color: blue;');
      }
    },
    red: {
      activate: function() {
        $('#my-btn').attr('style', 'color: red;');
      }
    }
  },
});

// do not assign goals without a user_id
if (get_user_id()) {
  button_color_experiment.add_goal(my_goal);
}

Notes:

• For user-based tracking, make sure you always have a user_id. Do not mix visitors (without an id) and users (with an id) in the same experiment.
• Cross-device tracking only works with the Gimel, Lamed or keen.io tracking backends. It does not work with Google Analytics.

See this Wiki page for more information

Goals

Goals are uniquely tracked by default. i.e. if a goal is set to measure how many visitors clicked on a button, multiple clicks won't generate another goal completion. Only one per visitor. Non-unique goals can be set by passing unique: false to the goal when creating it.

Goals will only be tracked if the experiment was launched and a variant selected before. Tracking goals is therefore safe and idempotent (unless unique is false).

Here's a short sample of tracking multiple goals over multiple experiments:

import AlephBet from "alephbet";

// main goal - button click
const button_click_goal = new AlephBet.Goal('button click');
$('#my-btn').on('click', function() {
  button_clicked_goal.complete();
});

// engagement - any click on the page
const engagement = new AlephBet.Goal('engagement');
$('html').on('click', function() {
  engagement.complete();
});

const all_goals = [button_click_goal, engagement];

// experiments
const button_color_experiment = new AlephBet.Experiment({ /* ... */ });
const buy_button_cta_experiment = new AlephBet.Experiment({ /* ... */ });

// adding all goals to experiments
_(all_goals).each(function (goal) {
  button_color_experiment.add_goal(goal);
  buy_button_cta_experiment.add_goal(goal);
});

// alternatively, you can use the add_goals method and pass it an array of goals
button_color_experiment.add_goals(all_goals);
buy_button_cta_experiment.add_goals(all_goals);

Custom Tracking Adapter

AlephBet comes with a built-in Google Analytics adapter and three, currently experimental, adapters with potentially better accuracy:

Persistent Queue GA Adapter

Persistent Queue Keen Adapter

Gimel adapter

Lamed adapter

Creating custom adapters is however very easy.

Here's an example for integrating an adapter for keen.io

(For a more complete implementation, you should use the built-in Alephbet.PersistentQueueKeenAdapter)

<script src="https://d26b395fwzu5fz.cloudfront.net/3.2.4/keen.min.js" type="text/javascript"></script>
<script src="alephbet.min.js"></script>
<script type="text/javascript">
    window.keen_client = new Keen({
        projectId: "ENTER YOUR PROJECT ID",
        writeKey: "ENTER YOUR WRITE KEY"
    });
    const tracking_adapter = {
        experiment_start: function(experiment, variant) {
            keen_client.addEvent(experiment.name, {variant: variant, event: 'participate'});
        },
        goal_complete: function(experiment, variant, event_name, _props) {
            keen_client.addEvent(experiment.name, {variant: variant, event: event_name});
        }
    };
    const my_experiment = new AlephBet.Experiment({
        name: 'my experiment',
        variants: { // ...
        },
        tracking_adapter: tracking_adapter,
        // ...
    });
</script>

Custom Storage Adapter

Similar to the tracking adapter, you can customize the storage adapter. AlephBet uses localStorage by default, but if you want to use cookies or customize how data is persisted on the client, creating an adapter is very easy.

Here's a simple example of a cookie storage adapter with expiry of 30 days, using js-cookie:

<script src="/path/to/js.cookie.js"></script>
<script type="text/javascript">
    // NOTE: using JSON stringify / parse to allow storing more complex values
    const storage_adapter = {
        set: function(key, value) {
            Cookies.set(key, JSON.stringify(value), {expires: 30});
        },
        get: function(key) {
            try { return JSON.parse(Cookies.get(key)); } catch(e) { return Cookies.get(key); }
        }
    };
    const my_experiment = new AlephBet.Experiment({
        name: 'my experiment',
        variants: { // ...
        },
        storage_adapter: storage_adapter,
        // ...
    });
</script> 

Debug mode

To set more verbose logging to the browser console, use

import Alephbet from "alephbet"

AlephBet.options.debug = true`

Other install options

• download alephbet.min.js from the dist folder on github
• npm install alephbet
• bower install alephbet

Analyzing results

AlephBet is built for developers, so there's no fancy interface or WYSIWYG editors. The best way to analyze the results and determine the best variant from an experiment is to look at the raw data and calculate the statistical significance. A couple of recommended resources:

• Thumbtack's ABBA A/B Test Calculator
• Evan Miller's A/B Testing tools

Development

Commands

• yarn run build - to build distribution files
• yarn run watch - will watch files and re-build using jest

License

AlephBet is distributed under the MIT license. All 3rd party libraries and components are distributed under their respective license terms.

Copyright (C) 2015 Yoav Aner

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated
documentation files (the "Software"), to deal in the Software without restriction, including without limitation the
rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit
persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the
Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE
WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.