PyCTest

Travis	AppVeyor

PyCTest Documentation (readthedocs)

GitHub

git clone https://github.com/jrmadsen/pyctest.git

PyPi

pip install pyctest

Anaconda Cloud

conda install -c jrmadsen pyctest

Name	Version	Platforms	Downloads

conda-forge

conda install -c conda-forge pyctest

Name	Version	Platforms	Downloads

Benefits

Integration into continuous integration systems (e.g. Travis, AppVeyor, CircleCI, etc.) and pushing to CDash dashboard will combine all the results in one place
- The warnings and errors are enumerated in CDash (no more parsing stdout logs for errors)
Easily create platform-independent testing
No need to migrate build system to CMake -- just specify pyctest.BUILD_COMMAND

General Setup

Add PyCTest to conda environment: conda install pyctest
Write a driver Python script, e.g. pyctest-runner.py, in top of the source code tree

Example for Python project

The following is an example for a Python code with a compiled C extension that uses nosetests for unit-testing:

#!/usr/bin/env python

import os, sys, platform
import pyctest.pyctest as pyctest
import pyctest.helpers as helpers

parser = helpers.ArgumentParser("ProjectName", source_dir=os.getcwd(),binary_dir=os.getcwd(), vcs_type="git")
args = parser.parse_args()

pyctest.BUILD_NAME = "{}".format(args.build)
pyctest.BUILD_COMMAND = "python setup.py build_ext --inplace"

test = pyctest.test()
test.SetName("unittest")
# insert the command to run the tests for project
test.SetCommand(["nosetests"])

pyctest.run()

Example for autotools project

#!/usr/bin/env python

import os, sys, platform
import multiprocessing as mp
import pyctest.pyctest as pyctest
import pyctest.helpers as helpers

parser = helpers.ArgumentParser("ProjectName", source_dir=os.getcwd(), binary_dir=os.getcwd(),
                                vcs_type="git")
parser.add_argument("-n", "--build", type=str, required=True, help="Build name for identification")
args = parser.parse_args()

# CONFIGURE_COMMAND can only run one command so if autogen is required, just execute it here
cmd = pyctest.command(["./autogen.sh"])
cmd.SetWorkingDirectory(pyctest.SOURCE_DIRECTORY)
cmd.SetErrorQuiet(False)
cmd.Execute()

pyctest.BUILD_NAME = "{}".format(args.build)
pyctest.CONFIGURE_COMMAND = "./configure"
pyctest.BUILD_COMMAND = "make -j{}".format(mp.cpu_count())

test = pyctest.test()
test.SetName("unittest")
# insert the command to run the tests for project
test.SetCommand(["./run-testing.sh"])

pyctest.run()

Example for CMake project

#!/usr/bin/env python

import os
import sys
import platform
import multiprocessing as mp
import pyctest.pyctest as pyctest
import pyctest.helpers as helpers

binary_dir = os.path.join(os.getcwd(), "build-ProjectName")
parser = helpers.ArgumentParser("ProjectName", os.getcwd(), binary_dir)
parser.add_argument("-n", "--build", type=str, required=True, help="Build name for identification")
args = parser.parse_args()

pyctest.BUILD_NAME = "{}".format(args.build)
pyctest.UPDATE_COMMAND = "git"
pyctest.CONFIGURE_COMMAND = "cmake {}".format(pyctest.SOURCE_DIRECTORY)
pyctest.BUILD_COMMAND = "cmake --build {} --target all -- -j{}".format(pyctest.BINARY_DIRECTORY, mp.cpu_count())

test = pyctest.test()
test.SetName("unittest")
# insert the command to run the tests for project
test.SetCommand(["./run-testing.sh"])

pyctest.run()

Python Modules

import pyctest -- global package
import pyctest.pyctest -- CTest module
import pyctest.pycmake -- CMake module
import pyctest.helpers -- Helpers module
- includes command line arguments (argparse) for PyCTest

Direct Access to CMake/CTest/CPack Executables

python -m pyctest.cmake <ARGS> == cmake <ARGS>
python -m pyctest.ctest <ARGS> == ctest <ARGS>
python -m pyctest.cpack <ARGS> == cpack <ARGS>

Following Python code:

from pyctest.ctest import CTest
from pyctest.cmake import CMake
from pyctest.cpack import CPack

CMake({"CMAKE_BUILD_TYPE":"Release"}, os.getcwd(), "-G", "Ninja")
CTest("--build-and-test", os.getcwd(), "-VV")
CPack("-G", "TGZ")

is equivalent to the following shell commands:

$ cmake -DCMAKE_BUILD_TYPE=Release ${PWD} -G Ninja
$ ctest --build-and-test ${PWD} -VV
$ cpack -G TGZ

Standard Configuration Variables

pyctest.PROJECT_NAME
pyctest.SOURCE_DIRECTORY
pyctest.BINARY_DIRECTORY
pyctest.SITE
pyctest.BUILD_NAME
pyctest.TRIGGER
pyctest.CHECKOUT_COMMAND
pyctest.BUILD_COMMAND
pyctest.MODEL
pyctest.CUSTOM_COVERAGE_EXCLUDE
pyctest.CUSTOM_MAXIMUM_NUMBER_OF_ERRORS
pyctest.CUSTOM_MAXIMUM_NUMBER_OF_WARNINGS
pyctest.CUSTOM_MAXIMUM_PASSED_TEST_OUTPUT_SIZE

Setting Arbitrary Variables

    pyctest.set("CTEST_TOKEN_FILE", "${CMAKE_CURRENT_LIST_DIR}/.ctest-token")

Generating a Test

    test = pyctest.test()
    test.SetName("nosetests")
    test.SetCommand(["nosetests", "test", "--cover-xml", "--cover-xml-file=coverage.xml"])
    # set directory to run test
    test.SetProperty("WORKING_DIRECTORY", pyctest.BINARY_DIRECTORY)
    test.SetProperty("RUN_SERIAL", "ON")
    test.SetProperty("ENVIRONMENT", "OMP_NUM_THREADS=1")

Examples

Basic example
Advanced example
- includes submission to CDash dashboard

CDash Integration Example

Results from running the TomoPy example can be found at the TomoPy CDash Testing Dashboard @ NERSC

Python code with C extensions without CMake build system
The build logs from "python setup.py install" are registered in the "Build" section
The nosetests test command + other are wrapped into CTests

Testing Example

PyCTest can be used to simple execute tests and submit to a dashboard without any configuration, build, etc. steps

#!/usr/bin/env python

import os
import sys
import shutil
import argparse
import platform
import traceback

import pyctest.pyctest as pyctest
import pyctest.pycmake as pycmake
import pyctest.helpers as helpers

if __name__ == "__main__":

    directory = os.path.join(os.getcwd(), "pycm-test")

    # these are required
    pyctest.PROJECT_NAME = "PyCTest"
    pyctest.SOURCE_DIRECTORY = directory
    pyctest.BINARY_DIRECTORY = directory

    args = helpers.ArgumentParser(pyctest.PROJECT_NAME,
                                  pyctest.SOURCE_DIRECTORY,
                                  pyctest.BINARY_DIRECTORY).parse_args()

    # set explicitly
    pyctest.MODEL = "Continuous"
    pyctest.SITE = platform.node()

    # create a Test object
    test = pyctest.test()
    test.SetName("list_directory")
    test.SetCommand(["ls", directory])
    test.SetProperty("WORKING_DIRECTORY", os.getcwd())

    # create a second test
    # previous test is already stored by PyCTest
    test = pyctest.test()
    test.SetName("hostname")
    test.SetCommand(["hostname"])
    test.SetProperty("TIMEOUT", "10")

    # generate the CTestConfig.cmake and CTestCustom.cmake
    pyctest.generate_config(directory)

    # generate the CTestTestfile.cmake file
    pyctest.generate_test_file(directory)

    # run CTest -- e.g. ctest -VV ${PWD}/pycm-test
    pyctest.run(pyctest.ARGUMENTS, directory)

CTest arguments (default): '-V -DSTAGES=Start;Update;Configure;Build;Test;Coverage;MemCheck -S Stages.cmake -j1'
Writing CTest test file: "/Users/jrmadsen/devel/c++/pyctest-master/pycm-test/CTestTestfile.cmake"...
Generating test "list_directory"...
Generating test "hostname"...
-- STAGES = Start;Update;Configure;Build;Test;Coverage;MemCheck
-- [[Darwin macOS 10.13.6 x86_64] [Python 3.6.7]] Running CTEST_START stage...
Run dashboard with model Continuous
   Source directory: /Users/jrmadsen/devel/c++/pyctest-master/pycm-test
   Build directory: /Users/jrmadsen/devel/c++/pyctest-master/pycm-test
   Track: Continuous
   Reading ctest configuration file: /Users/jrmadsen/devel/c++/pyctest-master/pycm-test/CTestConfig.cmake
   Site: JRM-macOS-DOE.local
   Build name: [Darwin macOS 10.13.6 x86_64] [Python 3.6.7]
   Use Continuous tag: 20181129-2118
-- [[Darwin macOS 10.13.6 x86_64] [Python 3.6.7]] Skipping CTEST_UPDATE stage...
-- [[Darwin macOS 10.13.6 x86_64] [Python 3.6.7]] Skipping CTEST_CONFIGURE stage...
-- [[Darwin macOS 10.13.6 x86_64] [Python 3.6.7]] Skipping CTEST_BUILD stage...
-- [[Darwin macOS 10.13.6 x86_64] [Python 3.6.7]] Running CTEST_TEST stage...
Test project /Users/jrmadsen/devel/c++/pyctest-master/pycm-test
    Start 1: list_directory
1/2 Test #1: list_directory ...................   Passed    0.00 sec
    Start 2: hostname
2/2 Test #2: hostname .........................   Passed    0.00 sec

100% tests passed, 0 tests failed out of 2

Total Test time (real) =   0.01 sec
-- [[Darwin macOS 10.13.6 x86_64] [Python 3.6.7]] Skipping CTEST_COVERAGE stage...
-- [[Darwin macOS 10.13.6 x86_64] [Python 3.6.7]] Skipping CTEST_MEMCHECK stage...
-- [[Darwin macOS 10.13.6 x86_64] [Python 3.6.7]] Skipping CTEST_SUBMIT stage...
-- [[Darwin macOS 10.13.6 x86_64] [Python 3.6.7]] Finished Continuous Stages (Start;Update;Configure;Build;Test;Coverage;MemCheck)

Command Line Interface

PyCTest args: ['--help']
  CTest args: []
  CMake args: []
usage: pyctest-runner.py [-h] [-F] [-S] [-A] [-j <INT>] [-m <TYPE>] [-b <NAME>] [-i <PATH>] [-o <PATH>] [-M <TYPE>] [-H <SITE>] [-P <EXE>] [-N <NAME>]
                         [-C <PATH> [<PATH> ...]] [-L <LABEL> [<LABEL> ...]] [-T [<TYPE> [<TYPE> ...]]] [-fc <EXE>] [-cc <EXE>] [-cxx <EXE>]
                         [--pyctest-token <TOKEN>] [--pyctest-token-file <FILE>] [--pyctest-vcs-type <TYPE>] [--pyctest-build-type <TYPE>]
                         [--pyctest-trigger <TYPE>] [--pyctest-use-launchers <BOOL>] [--pyctest-ctest-args [<ARG> [<ARG> ...]]] [--pyctest-cdash-version <STR>]
                         [--pyctest-submit-retry-count <INT>] [--pyctest-submit-retry-delay <INT>] [--pyctest-curl-options <LIST>]
                         [--pyctest-drop-location <STR>] [--pyctest-drop-method <STR>] [--pyctest-drop-site <STR>] [--pyctest-drop-site-password <STR>]
                         [--pyctest-drop-site-user <STR>] [--pyctest-nightly-start-time <STR>] [--pyctest-update-version-only]

PyCTest argparse. Arguments after first '--' are passed directly to CTest, arguments after second '--' are passed directly to CMake

optional arguments:
  -h, --help                                                                   show this help message and exit
  -F, --pyctest-clean-first                                                    Remove standard PyCTest files and binary directory (if not source directory)
  -S, --pyctest-submit                                                         Enable submission to dashboard
  -A, --pyctest-append                                                         Append to last invocation of CTest
  -j <INT>, --pyctest-jobs <INT>                                               number of concurrent jobs
  -m <TYPE>, --pyctest-mode <TYPE>                                             Run specific stage. Choices:
                                                                               [Start,Update,Configure,Build,Test,Coverage,MemCheck,Submit,Stages]
  -b <NAME>, --pyctest-build-name <NAME>                                       Build name for identification
  -i <PATH>, --pyctest-source-dir <PATH>                                       Source directory
  -o <PATH>, --pyctest-binary-dir <PATH>                                       Binary/build/working directory
  -M <TYPE>, --pyctest-model <TYPE>                                            CTest submission model (track). Choices: [Nightly,Continuous,Experimental]
  -H <SITE>, --pyctest-site <SITE>                                             CTest submission site
  -P <EXE>, --pyctest-python-exe <EXE>                                         Python executable to use. This can be absolue, relative, or CMake path
  -N <NAME>, --pyctest-project-name <NAME>                                     Name of project using PyCTest
  -C <PATH> [<PATH> ...], --pyctest-cleanup <PATH> [<PATH> ...]                Remove standard PyCTest files and binary directory (if not source directory) and
                                                                               exit
  -L <LABEL> [<LABEL> ...], --pyctest-subproject-labels <LABEL> [<LABEL> ...]  Add labels for subproject
  -T [<TYPE> [<TYPE> ...]], --pyctest-stages [<TYPE> [<TYPE> ...]]             Run multiple stages. Choices:
                                                                               [Start,Update,Configure,Build,Test,Coverage,MemCheck,Submit]
  -fc <EXE>, --pyctest-fortran-compiler <EXE>                                  Specify Fortan compiler (if needed)
  -cc <EXE>, --pyctest-c-compiler <EXE>                                        Specify the C compiler (if needed)
  -cxx <EXE>, --pyctest-cxx-compiler <EXE>                                     Specify C++ compiler (if needed)
  --pyctest-token <TOKEN>                                                      CTest site token for submission
  --pyctest-token-file <FILE>                                                  Path to file for CTest site token for submission
  --pyctest-vcs-type <TYPE>                                                    Set to enable revision ID discovery during the Update stage. Choices:
                                                                               [bzr,cvs,git,hg,p4,svn]
  --pyctest-build-type <TYPE>                                                  Specify CMAKE_BUILD_TYPE (if using CMake). Choices:
                                                                               [Release,RelWithDebInfo,Debug,MinSizeRel]
  --pyctest-trigger <TYPE>                                                     DEPRECATED
  --pyctest-use-launchers <BOOL>                                               Enable launchers
  --pyctest-ctest-args [<ARG> [<ARG> ...]]                                     CTest arguments
  --pyctest-cdash-version <STR>                                                Set CTest variable: 'CTEST_CDASH_VERSION'
  --pyctest-submit-retry-count <INT>                                           Set CTest variable: 'CTEST_SUBMIT_RETRY_COUNT'
  --pyctest-submit-retry-delay <INT>                                           Set CTest variable: 'CTEST_SUBMIT_RETRY_DELAY'
  --pyctest-curl-options <LIST>                                                Set CTest variable: 'CTEST_CURL_OPTIONS'
  --pyctest-drop-location <STR>                                                Set CTest variable: 'CTEST_DROP_LOCATION'
  --pyctest-drop-method <STR>                                                  Set CTest variable: 'CTEST_DROP_METHOD'
  --pyctest-drop-site <STR>                                                    Set CTest variable: 'CTEST_DROP_SITE'
  --pyctest-drop-site-password <STR>                                           Set CTest variable: 'CTEST_DROP_SITE_PASSWORD'
  --pyctest-drop-site-user <STR>                                               Set CTest variable: 'CTEST_DROP_SITE_USER'
  --pyctest-nightly-start-time <STR>                                           Set CTest variable: 'CTEST_NIGHTLY_START_TIME'
  --pyctest-update-version-only                                                Specify that you want the version control update command to only discover the
                                                                               current version that is checked out, and not to update to a different version

pyctest
Release 0.0.12

Release 0.0.12

0.0.12

0.0.11

0.0.10

0.0.10rc0

0.0.9

0.0.9rc0

0.0.8rc0

0.0.7rc0

0.0.6

0.0.6.dev2

Documentation

PyCTest

PyCTest Documentation (readthedocs)

GitHub

PyPi

Anaconda Cloud

conda-forge

Benefits

General Setup

Example for Python project

Example for autotools project

Example for CMake project

Python Modules

Direct Access to CMake/CTest/CPack Executables

Standard Configuration Variables

Setting Arbitrary Variables

Generating a Test

Examples

CDash Integration Example

Testing Example

Command Line Interface

Stats

Development practices

Releases

Contributors

pyctest Release 0.0.12

Release 0.0.12 Toggle Dropdown 0.0.12 0.0.11 0.0.10 0.0.10rc0 0.0.9 0.0.9rc0 0.0.8rc0 0.0.7rc0 0.0.6 0.0.6.dev2

Documentation

PyCTest

PyCTest Documentation (readthedocs)

GitHub

PyPi

Anaconda Cloud

conda-forge

Benefits

General Setup

Example for Python project

Example for autotools project

Example for CMake project

Python Modules

Direct Access to CMake/CTest/CPack Executables

Standard Configuration Variables

Setting Arbitrary Variables

Generating a Test

Examples

CDash Integration Example

Testing Example

Command Line Interface

Stats

Development practices

Releases

Contributors

pyctest
Release 0.0.12

Release 0.0.12

0.0.12

0.0.11

0.0.10

0.0.10rc0

0.0.9

0.0.9rc0

0.0.8rc0

0.0.7rc0

0.0.6

0.0.6.dev2