The Plugin Oriented Programming System

coordination deployment services zookeeper
pip install pop==12


Let pop run your services!

Skip to acknowledgements and credits.


Software builds, deployment and service management, are tasks that become exceedingly difficult to carry out without the assistance of a computer.

Typically, this assistance is a collection of scripts that require some amount of handholding. When multiple machines are involved, this handholding becomes a problem of distributed coordination.

We need a tool!

This is not meant to replace a configuration tool such as Chef or Puppet. You'll probably still need those.

To add to the confusion, it's also not meant to replace Supervisor, an excellent process control system written in Python. In fact, we recommend that you run your machine agents using this software.

What pop is meant to replace is you!

Now, we've already got a language – Python. All we need is to model the actions that you would carry out if you were the best possible system administrator.


With these principles we try to cover the basic design and philosophy.

  1. Adaptive. The system is dynamic and the equilibrium is the target state. There's always a state and if it's not the target state, then we must take the necessary steps to reach that state.
  2. Distributed. The system is comprised of services running on multiple machines. We use Apache's ZooKeeper for "highly reliable distributed coordination" (via txzookeeper, a Twisted library written for the Ubuntu Juju cloud deployment tool).
  3. Integrated. This is a Python-based tool for running Python-based services. The system is extensible through plugins.
  4. Open. The system should not be tied to a particular platform.


These steps will be necessary for each machine in the system – except for the final step which applies only to the machine that should run ZooKeeper.

We assume that Python 2.7 is installed and available as python2.

  1. To get started, log in to a machine, and install pop using setuptools and virtualenv:

    $ wget
    $ sudo python2
    $ sudo easy_install-2.7 virtualenv

    That's it for sudo. We can continue with an unprivileged user.

    Let's install pop into an isolated, virtual environment under the home directory and enter the environment:

    $ virtualenv --python=python2 ~/pop
    $ source ~/pop/bin/activate
    $ easy_install pop

    This installs the pop command-line utility as well as supporting libraries.

  2. Build and install ZooKeeper on the system (see instructions for more details).

    Make sure your system has Apache Ant installed (a build tool for Java) before proceeding:

    $ cd ~/pop
    $ git clone git://
    $ cd zookeeper
    $ ant

    First we need to build the C-bindings:

    $ cd src/c
    $ autoreconf -if
    $ ./configure --prefix=~/pop/zookeeper
    $ make && make install

    Then the Python-bindings:

    $ cd ../contrib/zkpython
    $ ant install

    That's it for the installation.

  3. Configure and start ZooKeeper:

    $ cd ~/pop/zookeeper/conf
    $ cp zoo_sample.cfg zoo.cfg

    You should edit zoo.cfg before you start a real deployment because the default storage setting is /tmp/zookeeper.

    There's a script included to run the service:

    $ ../bin/ start

    If you want to keep track of what's going on, use start-foreground. Note that the pop utility expects ZooKeeper to run on localhost unless the --host argument is provided.

    If everything's gone to plan, we're ready to use the system.


As a first look, in this tutorial we'll learn how to use pop to run a vanilla installation of Plone.

  1. Initialize pop namespace:

    $ pop init

    This adds a number of nodes to the hierarchy that pop requires for its operation. Note that the command effectively resets the configuration although this requires the --force option if an existing configuration is in place.

  2. Add the plone4 service which runs the latest version of Plone:

    $ pop add plone4

    The service will be set up with the default configuration with the name "plone4" (unless the --name argument was provided). We can confirm this using the status command:

    $ pop status plone4

    In the default configuration, Plone is set to listen on port 8080. To change this, use the --port parameter when you first add the service.

  3. To make the local machine available as a system that we can run services on, we need to start the machine agent:

    $ pop start

    This process can also run in the foreground using pop fg.

  4. Finally, to deploy the Plone service on the local machine:

    $ pop deploy plone4

    This works because the utility assumes that we want to deploy the service on the local machine.

The order of the last two steps is not important. We could easily have deployed the service first, then made the machine available.


Pop keeps all state in ZooKeeper (ZK).

The table below lists the various paths involved.

Path Data Format Type
/machines/<machine-id>/<service-name> PID [1] Integer Ephemeral
/services/<service-name>/machines Machines [2] JSON  
/services/<service-name>/settings Settings [3] JSON  
/services/<service-name>/state/<machine-id> State [4] JSON Ephemeral


[1] The process identifier.
[2] This is a list of machines on which the service should run.
[3] These are the settings used to bring up the service (regardless of the machine). If changed, the service will be restarted.
[4] This is set by the service when it's up and running; the contents is specific to the implementation, but must be in JSON-format.

Note that an ephemeral node is one that's created using the zookeeper.EPHEMERAL flag and immediately removed when the creator disconnects.


Machine identification

The machine id is a hardware UUIDs. On Linux this is the value returned by HAL for the "system.hardware.uuid" key.

Service identification

Each service is required to have a unique service name (a string).


To carry out tasks such as upgrades, coordinated script execution is needed.

Plugins can define tasks on a service level and make them available on the command-line:

$ pop run <service> <command> [args]


The PythonEnvironment base class comes with a set of tasks that help to set up the interpreter environment. These are available to all services that derive from this base class.

  1. Install packages. To make a Python package available in the instance environment:

    $ pop run plone install lxml==2.3.5

    Multiple packages can be listed, separate with space.

  2. List packages. Return a list of installed Python libraries. For each package, print time of installation, source and version:

    $ pop run plone packages


This section documents the development of pop (not with).


To run the automated test suite, you need a running ZooKeeper service and the nose test runner.

In your virtualenv environment:

$ python nosetests

It's often useful to get a more verbose output:

$ python nosetests --verbosity=3

Note that a higher verbosity setting will result in a great amount of noise from the nose test runner itself.

Acknowledgements and Credits

The architecture and technical implementation of this software was inspired by Canonical's Juju cloud deployment tool, originally designed by Kapil Thangavelu. We deliberately use the same terms and conventions when possible (for example machines and services).

The author of this software:

Malthe Borch –


Pop is available under the GPL.