Dockbot

A continuous integration system which uses Docker and Buildbot. Dockbot is a Python program which starts a master Docker instance and a number of build slaves. The master coordinates builds, offers a Web interface for monitoring and controlling builds and triggers builds based on changes to source repositories. The slaves perform the builds under different configurations.

Installation

You can install dockbot with pip as follows:

pip install dockbot

Or from GitHub:

git clone https://github.com/CauldronDevelopmentLLC/dockbot
cd dockbot
sudo python setup.py install

Configuring a Dockbot Project

To use dockbot you need to create a dockbot.json configuration file in a directory of it's own and a number of slaves in a directory called slaves. In the slave directories you also have dockbot.json files which configure which projects the slave will run. The directory structure should look like this:

dockbot.json
lib/
slaves/
  slave1/
    dockbot.json
    projectA.docker
    projectB.docker
    ...
  slave2/
    ...
  slaveN/
    ...

The optional lib directory can contain Docker file fragments which may be used by multiple slaves.

Basic configuration

The top level dockbot.json file needs a some basic configuration information to setup the dockbot install:

"project": "Dockbot",
"url": "https://github.com/CauldronDevelopmentLLC/dockbot",
"namespace": "dockbot",
"admin": "Joseph Coffland <jcoffland@cauldrondevelopment.com>",
"ip": "127.0.0.1",
"http-port": 8049,
"buildbot-port": 8050,

These configuration values have the following meanings:

• project - The dockbot project name.
• url - The project URL, cosmetic only.
• namespace - A unique name prefix for docker images in this project.
• admin - A email address, cosmetic only.
• ip - The IP address to which ports should be bound.
• http-port - The HTTP port for the build master's Web interface.
• buildbot-port - Opening this port allows other build slaves to connect.

Build modes

Often software can be built in more than one way. Build modes make it possible to configure different builds configurations for the same software. Build mode configuration looks like this:

"modes": {
  "debug": {"scons": {"debug": 1}},
  "release": {}
},

Above we configure two build modes debug and release. The names are arbitrary and can be anything you like. The dictionaries under these names override options specific to those moves.

Projects

"projects": {
  "_default_": {
    "compile": ["scons", "-k"],
  },

  "cbang": {
    "test": true,
    "repo": {
      "type": "git",
      "url": "https://github.com/CauldronDevelopmentLLC/cbang",
      "branch": "master"
    }
  }
}

The projects section is a dictionary of project configurations. The special _default_ section sets the default configuration options for all projects. Project names are arbitrary and may contain the following configuration options:

• compile - The compile command.
• test - If true testing should be performed after the build is complete
• repo - The source repository configuration
  • type - One of git, github or svn
  • url - The repository url.
  • org - An organization, used only with github repos with out a url.
  • user - The repo user name.
  • pass - The repo password, if needed.
  • branch - The repository branch.
• deps - Projects which must be built first.
• packages - A list of package types, appended to the compile command.
• build - If false the build step is omitted.

Slaves

A slave represents a particular build configuration. Each slave has its own directory under slaves. The dockbot.json file in this directory defines one or more images that will be built and adds any extra configuration options particular to the slave.

Slave Images

A slave image is a Docker image which will be run in one or more modes. Images are defined in the slave's dockbot.json as follows:

"images": {
  "cbang": {"projects": ["cbang"]}
}

In the above example one image cbang is defined with one project cbang. The project must be defined in the top level dockbot.json. To build the image dockbot will look for a file cbang.docker in the slave's directory. It will then process this file with m4 to produce a final Dockerfile used to build the image. The m4 preprocessor will combine Dockerfile fragments to create the complete Dockerfile. These fragments may exist either in the slave's own directory, in a top level lib directory or in the dockbot default libs.

An example project .docker file make look like this:

include(base.m4)
include(slave.m4)

base.m4 is defined as:

FROM debian:testing

include(debian.m4)
include(boost-1.59.0.m4)
include(buildbot.m4)
include(gcc-4.8.m4)

Each of these include() lines references a docker file fragment located in one of the lib directories.

Building Dockbot Images

To build a dockbot image run the following command:

dockbot build <image>

Where <image> is the name of the image. e.g. dockbot-debian-testing-64bit-cbang. If the image is successfully build then it can be started.

To rebuild a dockbot image run the following:

dockbot rebuild <image>

This command first deletes the old build than builds it again applying any configuration changes. In order to do a rebuild all of the images's containers must first be stopped.

Starting & Running Master

The build master is a special docker container which coordinates the builds. Like the slaves it too must be built and run. This is accomplished as follows:

dockbot build master
dockbot start master

The build master must be restarted whenever the dockbot configuration is changed. To stop the master run the following:

dockbot stop master

Dockbot Status

To view the status of the dockbot containers run the following:

dockbot status

or simply:

dockbot

This will list the status of all configured dockbot containers. The full name of container in the current dockbot project will be displayed on a line with it's current status. Possible status are:

• [Not Found] - The container's image has not been built yet.
• [Built] - The image has been built but the container does not yet exist.
• [Running] - The container is currently running.
• [Offline] - The container exists but is not currently running.
• [Building] - The image is currently being built.
• [Starting] - The container is starting.
• [Stopping] - The container is stopping.
• [Deleting] - The image is currently being deleted.

To check the status of a particular dockbot container run:

dockbot status <container>

You can also view the complete configuration of a dockbot container like this:

dockbot config <container>

Shell Access

Some times it is desirable to login and inspect the contents of a container. This is a accomplished with the dockbot shell command:

dockbot shell <container>

If the image was previously built this will open a shell in the container. If the container was already running it will attach a new shell to the running container.

The Web Interface

The build master's Web interface makes it possible to monitor and control builds. To view the Web interface navigate a brower to the IP and port specified in the top level dockbot.json file. At least the build master container must be running. Some slave containers should also be running to make it useful.

Triggering Builds

Builds may be manually triggered using dockbot's trigger command. The syntax is:

dockbot trigger <container> [project]

If [project] is omitted all projects on the slave will be triggered.

Triggering Builds from GitHub

TBD

Publishing Builds

The last 5 completed builds are placed in run/buildmaster/builds/. Once all the builds for a particular project are correct they can be published using the dockbot-publish tool.

TBD

Publishing Builds to GitHub

TBD