testdef

Testdef is an emerging test definition interchange intended for storing and describing tests in a public repository.

Scope

Testdef is intending to define the following areas of test definition management:

• Storing framework-independent information:
  • Name and version - required
  • Description and other supplementary human-readable fields
  • License and other legal requirements
  • Classifiers - framework independent meta-data
• Storing framework-specific information:
  • Intended software and hardware platforms
  • Software platform dependencies (such as platform version and any optional components or packages)
  • Hardware platform dependencies (such as specific hardware or add-on peripherials)
  • Level of automation (fully automated, manual, partially automated, manual verification, etc)
  • Description of test execution suitable for the framework
• File formats and languages:
  • Defining the basic format of a test definition
  • Defining the basic format of a test definition package (that may include other files, beyond the test definition itself, such as source code and data files)
• Tools, APIs and protocols:
  • Providing tools and APIs for registering and releasing test definitions.
  • Defining the web protocols used between the repository server and client side tools, regardless of the implementation details of the server.
  • Providing reference implementation of the test definition repository server that can be used with the client side tools.

Goals

• Create a set of tools that allow anyone to define, and publish test definitions to a supported repository.
• Create supplementary Python APIs for working with test definitions that may be of use to some test frameworks as either support or base libraries.
• Create a reference implementation of the test definition repository as well as the client-server APIs.

Non-goals

Testdef is not intended to be the one-size-fits-all test framework. It is natural and expected that domain-specific frameworks exist and will continue to exist. Testdef is not even bound to python (apart from the client tools defined here) and can equally well support Android tests written in Java and low-level numeric benchmarks written in Fortran and C.

Testdef is also not intended as a test result repository or test result processing system. We believe that test results are specific to given framework and the problem of coming up with the ultimate test result repository is well beyond the scope of test definitions.

Supported environment

The main environment for the tools created here is Linux but the format and tools must not limit support for other platforms, including Windows.

For the purpose of this repository the source code must support python 2.7 and python 3.2 (using automatic 2to3 conversion). The results must be installable and usable in both major python versions, at least during the django transition to python3.

Installation

Currently there's not much to install but all releases shall be made to pypi. You can obtain a source tarball from: http://pypi.python.org/pypi/testdef

As usual, you can install anything python with pip:

$ pip install testdef

Development

• Send pull requests or patches to https://github.com/zyga/testdef
• Discuss this on linaro-validation@lists.linaro.org