BenchExec

A Framework for Reliable Benchmarking and Resource Measurement

News and Updates:

BenchExec 3.18 brings support for systems with cgroups v2!
Linux kernel 5.11 finally makes it possible to use all BenchExec features on distributions other than Ubuntu!
We now provide an Ubuntu PPA that makes installing and upgrading BenchExec easier (docs).
An extended version of our paper on BenchExec and its background was published as open access in the journal STTT, you can read Reliable Benchmarking: Requirements and Solutions online. We also provide a set of overview slides.

BenchExec provides three major features:

execution of arbitrary commands with precise and reliable measurement and limitation of resource usage (e.g., CPU time and memory), and isolation against other running processes
an easy way to define benchmarks with specific tool configurations and resource limits, and automatically executing them on large sets of input files
generation of interactive tables and plots for the results

Unlike other benchmarking frameworks, BenchExec is able to reliably measure and limit resource usage of the benchmarked tool even if the latter spawns subprocesses. In order to achieve this, it uses the cgroups feature of the Linux kernel to correctly handle groups of processes. For proper isolation of the benchmarks, it uses (if available) Linux user namespaces and an overlay filesystem to create a container that restricts interference of the executed tool with the benchmarking host. BenchExec is intended for benchmarking non-interactive tools on Linux systems. It measures CPU time, wall time, and memory usage of a tool, and allows to specify limits for these resources. It also allows to limit the CPU cores and (on NUMA systems) memory regions, and the container mode allows to restrict filesystem and network access. In addition to measuring resource usage, BenchExec can verify that the result of the tool was as expected, and extract further statistical data from the output. Results from multiple runs can be combined into CSV and interactive HTML tables, of which the latter provide scatter and quantile plots (have a look at our demo table).

BenchExec works only on Linux and needs a one-time setup of cgroups by the machine's administrator. The actual benchmarking can be done by any user and does not need root access.

BenchExec was originally developed for use with the software verification framework CPAchecker and is now developed as an independent project at the Software Systems Lab of the Ludwig-Maximilians-Universität München (LMU Munich).

Literature

Reliable Benchmarking: Requirements and Solutions, by D. Beyer, S. Löwe, and P. Wendler. International Journal on Software Tools for Technology Transfer (STTT), 21(1):1-29, 2019. doi:10.1007/s10009-017-0469-y. Journal paper about BenchExec (open access, also with a supplementary webpage and overview slides)
CPU Energy Meter: A Tool for Energy-Aware Algorithms Engineering, by D. Beyer and P. Wendler. In Proc. TACAS 2020, part 2, LNCS 12079, pages 126-133, 2020. Springer. doi:10.1007/978-3-030-45237-7_8
Benchmarking and Resource Measurement, by D. Beyer, S. Löwe, and P. Wendler. In Proc. SPIN 2015, LNCS 9232, pages 160-178, 2015. Springer. doi:10.1007/978-3-319-23404-5_12

License and Copyright

BenchExec is licensed under the Apache 2.0 License, copyright Dirk Beyer. Exceptions are some tool-info modules and third-party code that is bundled in the HTML tables, which are available under several other free licenses (cf. folder LICENSES).

Authors

Maintainer: Philipp Wendler

Contributors:

Aditya Arora
Dirk Beyer
Laura Bschor
Thomas Bunk
Montgomery Carter
Andreas Donig
Karlheinz Friedberger
Robin Gloster
Peter Häring
Florian Heck
Hugo
George Karpenkov
Mike Kazantsev
Michael Lachner
Thomas Lemberger
Sebastian Ott
Stefan Löwe
Stephan Lukasczyk
Alexander von Rhein
Alexander Schremmer
Dennis Simon
Andreas Stahlbauer
Thomas Stieglmaier
Martin Yankov
Ilja Zakharov
and lots of more people who integrated tools into BenchExec

Users of BenchExec

BenchExec was successfully used for benchmarking in all instances of the international competitions on Software Verification and Software Testing with a wide variety of benchmarked tools and hundreds of thousands benchmark runs. It is integrated into the cluster-based logic-solving service StarExec (GitHub).

The developers of the following tools use BenchExec: