worch - Let the orchestration waf through the suite.

This package provides a waf based Python module orch, short for “orchestrate”, which allows the creation of a meta-build system for a suite of related packages. The system is centered around a simple configuration language describing the installation and a number of interpreting methods to produce installation tasks. Finally, these tasks are handed to waf for sequenced execution.

Features

Some features of worch:

• a simple, concise and flexible text-based configuration language which can fully describe the installation of a complex software suite.
• the Worch machinery itself is distributed in standard Python packages available in PyPI to allow for a flexible and automated installation. Add-on tools or user configuration may be distributed likewise.
• automated installation includes downloading of source archives in common formats, their unpacking, source preparation using popular configuration methods, building and finally installing and any additional steps the user defines
• the user defines the conventions that set file/directory installation patterns.
• batteries included for driving some popular native build mechanisms. Novel build systems can accommodated by providing custom feature methods which may be dynamically incorporated based on the configuration.
• in order to assure proper build order while allowing parallel builds, dependencies can be defined either implicitly by linking installation steps via the files they require or produce or by explicitly declaring dependencies between any two steps by name.
• idempotent build steps, repeating a build does not repeat successful steps.
• no hidden failures, errors abort the installation.
• packages can be grouped to assure parts of the suite are entirely built before others.
• while heavy use of environment variables is discouraged one can define build environment variables in the configuration. These can be applied on a per-package basis or can be defined on a package or group basis and applied to those packages which require them.

Limitations

While waf supports all the way back to Python 2.3, worch requires at least Python 2.6 (eg, Sci. Linux 6). A script is provided to help build a modern version of python from source. See this doc for more info.

Concepts

Some of the terms and concepts used by Worch.

suite

the collection of all the software installed together with Worch

package

software from a single source to be installed as an atomic unit. A package has a name and a version

group

a collection of related packages that must be fully installed independent from others in the suite

step

the installation of a package is broken down into a linear, series of steps. The steps include “download”, “unpack”, “patch”, “prepare”, “build” and “install”. Fully qualified steps names include the name of the package on which they operate, for example: <package>_<step>

feature

one or more steps collected together in a named and reusable manner (“feature” is in the waf sense of the word)

tool

a mechanism to load external Python code largely to provide new features

dependencies

before a step can run it may require another step to run either in an absolute way or by depending on the production of some file by that other step.

Getting started with Worch

The use of Worch consists largely of editing a configuration file set and running waf with the orch Python module to interpret them.

Worch comes with example configuration files that build a few simple Free Software programs. To exercise them:

Install waf

Worch includes a copy of waf program in the top level directory but see the waf book for if you wish to explicitly install.

Install Worch

Worch installs in the usual manner for Python packages with the exception that the code relies on the waflib module which is made available by importing the orch module via the waf command.

Recommended installation method

It is recommended to install Worch in a Virtual Environment. To create one do:

$ virtualenv /path/to/venv
$ source /path/to/venv/bin/activate

If you do not have the virtualenv script, see this link for downloading it.

Whether installing into a Virtual Environment or not, Worch may be installed like:

$ pip install worch

Development installation

To develop Worch itself, first set up a Virtual Environment as above and then do:

$ git clone git://github.com/brettviren/worch.git
$ cd worch
$ python setup.py sdist
$ pip install dist/worch-X.Y.Z.tar.gz
(hack away)
$ pip uninstall -y worch
$ python setup.py sdist
$ pip install dist/worch-X.Y.Z.tar.gz

First run of waf

Regardless of the installation method (and of Worch itself) the first time waf is run by you it will unpack itself. This means it needs to reside in a location that is writable by the user. To trigger an unpacking one may run:

$ waf --version

Run waf on the worch configuration files

Worch installs some example configuration files which you can browse in the examples source directory. A configuration is given to Worch by specifying the main file. If specified as a relative path the file will be searched for in the current working directory or, if not found, in the installed location.

Configure

Worch needs a waf wscript file to drive the build. A generally suitable one is provided with the Worch distribution

$ cd /path/to/work
$ cp /path/to/venv/share/worch/wscripts/worch/wscript .
$ waf --prefix=/path/to/install \
      --orch-config=examples/simple/*.cfg \
      configure
Setting top to  : /path/to/work
Setting out to  : /path/to/work/tmp
'configure' finished successfully (0.065s)

$ ls ./tmp
c4che  config.log

Notes:

• if multiple configuration files are given; they are effectively concatenated and the wildcard character (*) will be interpreted by Worch. (tcsh users: beware that your shell sucks)
• often just a main file needs to be specified and any other ones are implicitly loaded.
• the ./tmp directory is created as directed by the out variable in the main wscript file and holds all intermediate build files
• if the configure step is repeated it requires repetition of the options as well.
• --zones=orch can be passed (to configure and build) to get more verbose output from waf.

Configuration file location

Worch will try to find configuration files specified by --orch-config following these rules in order:

1. If absolute path, find the file absolutely, otherwise
2. first check assuming relative to current working directory
3. relative to any paths listed as elements of the environment variable WORCH_CONFIG_PATH
4. relative to sys.prefix
5. relative to =sys.prefix + ‘/share/worch’

Build

The build command is default and need not be explicitly stated.

$ waf [-j4] [-vvv] > log
Waf: Entering directory `/path/to/work/tmp'
[ 1/18] cmake_seturl:  -> tmp/cmake-2.8.8.url
[ 2/18] cmake_download: tmp/cmake-2.8.8.url -> tmp/downloads/cmake-2.8.8.tar.gz
[ 3/18] cmake_unpack: tmp/downloads/cmake-2.8.8.tar.gz -> tmp/sources/cmake-2.8.8/bootstrap
[ 4/18] cmake_prepare: tmp/sources/cmake-2.8.8/bootstrap -> tmp/builds/cmake-2.8.8-debug/cmake_install.cmake
[ 5/18] cmake_build: tmp/builds/cmake-2.8.8-debug/cmake_install.cmake -> tmp/builds/cmake-2.8.8-debug/bin/cmake
[ 6/18] cmake_install: tmp/builds/cmake-2.8.8-debug/bin/cmake -> ../../../../../../tmp/worch-simple-example/cmake/2.8.8/debug/bin/cmake
[ 7/18] hello_seturl:  -> tmp/hello-2.8.url
[ 8/18] bc_seturl:  -> tmp/bc-1.06.url
[ 9/18] bc_download: tmp/bc-1.06.url -> tmp/downloads/bc-1.06.tar.gz
[10/18] hello_download: tmp/hello-2.8.url -> tmp/downloads/hello-2.8.tar.gz
[11/18] bc_unpack: tmp/downloads/bc-1.06.tar.gz -> tmp/sources/bc-1.06/configure
[12/18] hello_unpack: tmp/downloads/hello-2.8.tar.gz -> tmp/sources/hello-2.8/configure
[13/18] bc_prepare: tmp/sources/bc-1.06/configure -> tmp/builds/bc-1.06-debug/config.status
[14/18] hello_prepare: tmp/sources/hello-2.8/configure -> tmp/builds/hello-2.8-debug/config.status
[15/18] bc_build: tmp/builds/bc-1.06-debug/config.status -> tmp/builds/bc-1.06-debug/bc/bc
[16/18] bc_install: tmp/builds/bc-1.06-debug/bc/bc -> ../../../../../../tmp/worch-simple-example/bc/1.06/debug/bin/bc
[17/18] hello_build: tmp/builds/hello-2.8-debug/config.status -> tmp/builds/hello-2.8-debug/src/hello
[18/18] hello_install: tmp/builds/hello-2.8-debug/src/hello -> ../../../../../../tmp/worch-simple-example/hello/2.8/debug/bin/hello
Waf: Leaving directory `/path/to/work/tmp'
'build' finished successfully (8m3.605s)

$ waf
Waf: Entering directory `/path/to/work/tmp'
Waf: Leaving directory `/path/to/work/tmp'
'build' finished successfully (0.028s)

$ ls ./tmp
bc-1.06.url  builds  c4che  cmake-2.8.8.url  config.log  downloads hello-2.8.url  sources

$ ls /tmp/worch-simple-example/*/*/*
/tmp/worch-simple-example/bc/1.06/debug:
bin  info  man

/tmp/worch-simple-example/cmake/2.8.8/debug:
bin  doc  man  share

/tmp/worch-simple-example/hello/2.8/debug:
bin  share

Notes:

• parallelism can be used with the -j option, verbosity increased with -v
• logging from each step is kept atomic and is not printed until that step finishes
• ordering of steps is determined by dependencies
• rerunning waf does not repeat the successful steps
• waf users may expect an explicit “waf install” but it is not used by Worch
• all installation files are placed under the directory set by the --prefix option in the configure step
• this example installs each package into a specific <name>/<version>/<qualifier> directory, but other patterns are possible

Configuration File Syntax and Interpretation

The main user interaction, besides running waf as above, is in writing configuration files to describe the installation.

The Worch configuration files are in the standard syntax expected by the Python ConfigParser module (aka “INI” format). They consist of a number of named sections followed by key/value pair settings. They section title is surrounded by square brackets ”[]” and the key/value pairs are separated by either “===” or ”:”.

# this is a comment
[section]
key = value
key: value

Worch adds to this simple syntax some these features:

• string value interpolation
• hierarchical structure

Interpolation

Most values are interpreted having a scalar string type. These values may contain the names of other keys surrounded by curly braces ”{}”. These will have their value replaced by Worch.

[section]
key1 = World
key2 = Hello {key1}

The result is that the value of key2 will be ”Hello World”. Keys must be used in the same hierarchical scope as they are defined. The hierarchy is described in the next section. In addition to interpolation being run on the items in the configuration, Worch provides a few additional key/value pairs:

uname

output of uname stored as kernelname, hostname, kernelversion, vendorstring, machine

platform

a name formed from the kernelname and machine

gcc_dumpversion

the native GCC version

gcc_dumpmachine

the native GCC notion of the hosting machine architecture

gcc_multiarch

the native multiarch string (Debian extension)

libc_version

the libc version

ups_flavor

the UPS flavor string

Additional keys may be provided based on the existence of keys in the configuration.

version_2digit

at most the first two digits of the “.”-separated version string

version_underscore

version string with “.” replaced with “_”

version_nodots

version string with “.” removed

tagsdashed

all tags concatenated with dashes

tagsunderscore

all tags concatenated with underscores

Hierarchical configuration

Worch partitions the configuration logically into packages and groups of packages. This partitioning is done by interpreting certain keys as holding a list of sections names of a certain type. The mapping of key to type is held in the special keytype section. The keytype section used by Worch is:

[keytype]
packages = package
groups = group

This means that if the keys packages or groups are encountered, their values are interpreted as a list of section names of the “type” ”package” or ”group”. The interpretation begins at with one section, ”start” by default and follows down any keytype keys.

[start]
groups = group1, group2
key = value_from_start

[group group1]
packages = package1, package2
key = value_from_group1

[package package1]
key = value_from_package1

[package package2]
some_other_key = {key}

The hierarchy built in this way causes all simple, scalar values to be copied down to the leafs, which are packages in this case. This means that each package gets a copy, possibly customized, of all scalar key/value pairs. The interpolation occurs late so resolution is performed with this final, leaf set. Using the example above:

package1

has key set to value_from_package1

package2

has key and some_other_key both set to value_from_group1

Specifying inter-package dependencies

The configuration file can expresses dependencies between steps of different packages in two ways.

• implicitly through required/produced files
• explicitly by naming a package+step on which the current a particular package step depends

Implicit file dependencies

Explicit package step dependencies

To express an explicit dependency a package configuration section specifies a depends key with a comma-separated list of <step>:<package>_<step> elements. For example:

depends = prepare:gmp_install

Steps

Building a package is split into a number of steps. A step is identified by a simple name. There is no limit to step names but a limited set are identified as covering most meta-build operations. They are:

seturl

write the URL of the source archive file (or repository) into a file to start the package dependencies

download

produce the source archive file (or repository clone) based on the URL

unpack

produce a directory of pristine source code

patch

modify the source code, in place, typically by applying a patch

prepare

prepare the source for building, for example running cmake or autoconf’s configure script

build

produce binaries from the source

install

place build results to a final installation location

A step may have a default, associated directory in which it is run. The directories are specified by the following configuration variables. These locations and their associated steps are:

download_dir

download

source_dir

unpack, patch

build_dir

prepare, build, install

Features

The common steps are then grouped and implemented by “features” which can then be applied to different packages. Features use the steps as “touch stones” so that different features can be swapped while others can be shared. An example is the tarball and vcs features both provide through to the “unpack” step. The “cmake” and “autoconf” features provide the “prepare” step.

Here is a list of “features” that worch provides and the steps they implement:

tarball

download and unpack a tar/zip file (seturl, download, unpack)

vcs

clone or checkout source from a version control system (git, hg, cvs, svn), (seturl, download, unpack)

patch

apply a patch to the source (patch)

prepare

a generic source preparation feature (prepare)

autoconf

prepare source using autoconf configure script (prepare)

cmake

prepare source by calling cmake script (prepare)

makemake

run make/make install (build, install)

pypackage

install a Python package via setup.py (prepare, build, install)

pythiainst

special purpose feature for installing Pythia6 (prepare, build, install, and feature-specific steps)

The rest of this section gives some examples

Download and unpack

Almost all packages start by a download of a source archive (tar or zip file or git repository). Worch will handle these steps using the tarball feature. The example below shows how the GNU hello package makes use of this feature. A full, working example is in ./examples/simple.

[group gnuprograms]
features = tarball autoconf
srcpkg_ext = tar.gz
source_unpacked = {package}-{version}
source_package = {source_unpacked}.{srcpkg_ext}
download_dir = downloads
source_dir = sources
source_url = http://ftp.gnu.org/gnu/{package}/{source_package}

[package hello]
version: 2.8

Notes:

• The tarball feature is added to a special features key which is interpreted as a space separated list (fixme: should allow for comma-separated - space separation exposes a waf detail)
• The package section is brief as it inherits from the group and only provides the information unique to the pacakge
• The tarball feature needs to know where the download and source directories are, how the source package, URL and eventual unpacked directory are named
• The extension is pulled out to its own variable to accommodate multiple packages that are similar but may be archived/compressed differently (eg, another GNU package that happens to be compressed with BZ2)

Autoconf

The vast majority of packages are built with the configure/make/make install pattern provided by GNU autoconf. The autoconf feature can invoke this pattern. It follows on from the tarball feature and thus requires some of the same keys to be defined. One does not typically need to redefine these but rather they are used in the same context. Here is a follow-on to the hello example above but just showing the parts relevant to the autoconf feature. Again, see the simple example for a fully working instance.

[group gnuprograms]
tags = debug
features = tarball autoconf
source_unpacked = {package}-{version}
source_package = {source_unpacked}.{srcpkg_ext}
build_dir = builds/{package}-{version}-{tagsdashed}
install_dir = {PREFIX}/{package}/{version}/{tagsdashed}

[package hello]
version: 2.8
depends = prepare:bc_install
build_target = src/hello
install_target = bin/hello

Notes:

• Here a tags key is introduced. Tags are used to indicate variants in the build. In this example a debug version of hello should be built (fixme: tags are not yet supported).
• The build and install directories are specified while some source-related keys are reused from the tarball feature
• A build and install target must be specified in order to satisfy waf requirements
• A depends key is used to place an artificial, contrived dependency on another package step.

Mimicking autoconf

Many native build systems can use the autoconf feature by explicitly defining some variables that it uses. For example, building CMake does not use autoconf but it is close. Its package section can be defined like:

[package cmake]
features = tarball autoconf
unpacked_target = bootstrap
prepare_script = bootstrap

This causes the tarball and autoconf features to look for a bootstrap instead of a configure script.

Writing your own tool providing features

See ./doc/tools.org.

waf/worch tricks

Rerunning a step

Waf honors expressed dependencies and will rerun a step when a dependency changes. However, not all dependencies that could be expressed are. In particular, if a step completes successfully and then one changes either its source code (which is not in a file explicitly depended on) or the worch configuration files then waf may have no way to notice a change.

However, waf provides a ”step” command which will rerun an isolated step or steps without regards to dependencies. To indicate the step on uses the --files options. Waf finds the step that has been declared to produce the given file(s) and reruns it.

In general, one must have detailed understanding of the implementation of a feature and its steps in order to know what to give to the --files option. However, worch consistently creates a special “control” file after the successful completion of each step. This control file is consistently named like:

{control_dir}/{package}_{step}

The control_dir may be defined in the configuration but defaults to simply ”controls/” and is found in the “out” directory.

Example

As an example, in the ORKA build it was found that the Geant4VMC package requires Geant4 to include the G3toG4.hh header (despite that we try telling the package NO_G3TOG4). To reconfigure Geant4 to include this header in the install requires adding -DGEANT4_USE_G3TOG4=ON to the CMake command line. In order to avoid rebuilding the entire suite and just rerunning prepare, build and install steps for Geant4 one can do:

# rerun the configure step to pick up 
# the changes to the configuration
$ waf [...] configure  

# manually have waf (re)run each Geant4 step
$ waf step --files=tmp/controls/geant_prepare
$ waf step --files=tmp/controls/geant_build
$ waf step --files=tmp/controls/geant_install

# ditto for g4vmc
$ waf step --files=tmp/controls/geant4_prepare
$ waf step --files=tmp/controls/geant4_build
$ waf step --files=tmp/controls/geant4_install

# sop up any collateral changes, or continue with steps not yet run
$ waf

Debugging info

Worch can produce a lot of debugging information. It has the concept of “zones” of logging. To add some verbosity just for worch logs one can do:

$ waf --zones=orch [...]

Log files

Every step which involves running an executable produces a log file in:

{out}/logs/worch_{package}_{step}.log.txt

The log file is composed of sections beginning with the following:

WORCH CMD

the command line run

WORCH CWD

the current working directory in which the command ran

WORCH TSK

the step’s dependencies (input and output files) followed by a detailed dump of internal waf information

WORCH ENV

a dump of the shell environment variables

WORCH command output

the last part of the log file shows any output from the command itself

The log file for long-running steps may be found and monitored “live” by doing something similar to the following commands:

# Find the updating log
$ ls -ltr tmp/logs/ | tail

$ tail -f tmp/logs/worch_ilcroot_build.log.txt

Here ”tmp/” is the directory specified by the ”waf --out=tmp” flag.

Reproducing failures

If a command that is run by a step fails a shell script will be produced with everything that should reproduce the failure in-place (it is very much not portable). The location of the shell script is the current working directory where the command ran and is reported by waf. It should be run from the directory that holds it in order to reproduce the failure.

Others in this space

A truly unique project is rare and worch is no exception. This section lists some projects that are similar to worch that I’ve come across.

guix

The GNU package manager (more details in ./doc/guix.org)