Let pop run your services!

Skip to acknowledgements and credits.

Motivation

Software builds, deployment and service management, are tasks that become exceedingly difficult to carry out without the assistance of a computer.

Typically, this assistance is a collection of scripts that require some amount of handholding. When multiple machines are involved, this handholding becomes a problem of distributed coordination.

We need a tool!

This is not meant to replace a configuration tool such as Chef or Puppet. You'll probably still need those.

To add to the confusion, it's also not meant to replace Supervisor, an excellent process control system written in Python. In fact, we recommend that you run your machine agents using this software.

What pop is meant to replace is you!

Now, we've already got a language – Python. All we need is to model the actions that you would carry out if you were the best possible system administrator.

Principles

With these principles we try to cover the basic design and philosophy.

1. Adaptive. The system is dynamic and the equilibrium is the target state. There's always a state and if it's not the target state, then we must take the necessary steps to reach that state.
2. Distributed. The system is comprised of services running on multiple machines. We use Apache's ZooKeeper for "highly reliable distributed coordination" (via txzookeeper, a Twisted library written for the Ubuntu Juju cloud deployment tool).
3. Integrated. This is a Python-based tool for running Python-based services. The system is extensible through plugins.
4. Open. The system should not be tied to a particular platform.

Installation

These steps will be necessary for each machine in the system – except for the final step which applies only to the machine that should run ZooKeeper.

We assume that Python 2.7 is installed and available as python2.

1. To get started, log in to a machine, and install pop using setuptools and virtualenv:

   $ wget http://peak.telecommunity.com/dist/ez_setup.py
   $ sudo python2 ez_setup.py
   $ sudo easy_install-2.7 virtualenv
   

   That's it for sudo. We can continue with an unprivileged user.

   Let's install pop into an isolated, virtual environment under the home directory and enter the environment:

   $ virtualenv --python=python2 ~/pop
   $ source ~/pop/bin/activate
   $ easy_install pop
   

   This installs the pop command-line utility as well as supporting libraries.

2. Build and install ZooKeeper on the system (see instructions for more details).

   Make sure your system has Apache Ant installed (a build tool for Java) before proceeding:

   $ cd ~/pop
   $ git clone git://git.apache.org/zookeeper.git
   $ cd zookeeper
   $ ant
   

   First we need to build the C-bindings:

   $ cd src/c
   $ autoreconf -if
   $ ./configure --prefix=~/pop/zookeeper
   $ make && make install
   

   Then the Python-bindings:

   $ cd ../contrib/zkpython
   $ ant install
   

   That's it for the installation.

3. Configure and start ZooKeeper:

   $ cd ~/pop/zookeeper/conf
   $ cp zoo_sample.cfg zoo.cfg
   

   You should edit zoo.cfg before you start a real deployment because the default storage setting is /tmp/zookeeper.

   There's a script included to run the service:

   $ ../bin/zkServer.sh start
   

   If you want to keep track of what's going on, use start-foreground. Note that the pop utility expects ZooKeeper to run on localhost unless the --host argument is provided.

   If everything's gone to plan, we're ready to use the system.

Tutorial

As a first look, in this tutorial we'll learn how to use pop to run a vanilla installation of Plone.

1. Initialize pop namespace:

   $ pop init
   

   This adds a number of nodes to the hierarchy that pop requires for its operation. Note that the command effectively resets the configuration although this requires the --force option if an existing configuration is in place.

2. Add the plone4 service which runs the latest version of Plone:

   $ pop add plone4
   

   The service will be set up with the default configuration with the name "plone4" (unless the --name argument was provided). We can confirm this using the status command:

   $ pop status plone4
   

   In the default configuration, Plone is set to listen on port 8080. To change this, use the --port parameter when you first add the service.

3. To make the local machine available as a system that we can run services on, we need to start the machine agent:

   $ pop start
   

   This process can also run in the foreground using pop fg.

4. Finally, to deploy the Plone service on the local machine:

   $ pop deploy plone4
   

   This works because the utility assumes that we want to deploy the service on the local machine.

The order of the last two steps is not important. We could easily have deployed the service first, then made the machine available.

State

Pop keeps all state in ZooKeeper (ZK).

The table below lists the various paths involved.

Notes:

Note that an ephemeral node is one that's created using the zookeeper.EPHEMERAL flag and immediately removed when the creator disconnects.

Terminology

Machine identification

The machine id is a hardware UUIDs. On Linux this is the value returned by HAL for the "system.hardware.uuid" key.

Service identification

Each service is required to have a unique service name (a string).

Scripts

To carry out tasks such as upgrades, coordinated script execution is needed.

Plugins can define tasks on a service level and make them available on the command-line:

$ pop run <service> <command> [args]

Environment

The PythonEnvironment base class comes with a set of tasks that help to set up the interpreter environment. These are available to all services that derive from this base class.

1. Install packages. To make a Python package available in the instance environment:

   $ pop run plone install lxml==2.3.5
   

   Multiple packages can be listed, separate with space.

2. List packages. Return a list of installed Python libraries. For each package, print time of installation, source and version:

   $ pop run plone packages
   

Development

This section documents the development of pop (not with).

Tests

To run the automated test suite, you need a running ZooKeeper service and the nose test runner.

In your virtualenv environment:

$ python setup.py nosetests

It's often useful to get a more verbose output:

$ python setup.py nosetests --verbosity=3

Note that a higher verbosity setting will result in a great amount of noise from the nose test runner itself.

Acknowledgements and Credits

The architecture and technical implementation of this software was inspired by Canonical's Juju cloud deployment tool, originally designed by Kapil Thangavelu. We deliberately use the same terms and conventions when possible (for example machines and services).

The author of this software:

Malthe Borch – mborch@gmail.com

License

Pop is available under the GPL.

■