OutputCheck

OutputCheck is a tool for checking the output of console programs that is inspired by the FileCheck tool used by LLVM. It has its own small language (Check Directives) for describing the expected output of a tool that is considerably more convenient and more powerful than GNU grep.

This tool was originally written for STP but it ended up being a very useful tool for other projects so it became its own project!

Obtaining OutputCheck

OutputCheck can be obtained in multple ways.

Cloning Git repository

$ git clone https://github.com/stp/OutputCheck.git

the tool can now be run as

$ cd OutputCheck/
$ bin/OutputCheck --help

Installing PyPi package

The tool is available in the Python package index. It can be installed using the pip tool.

$ pip install OutputCheck

the tool can now be run as

$ OutputCheck --help

Please note that this package may not be up to date.

It is recommended that you use virtualenv in conjunction with pip so that you do not need to install python packages as root.

Check Directives

Check Directives declare what output is expected from a tool. They are written as single line comments in a file (this file is usually used to by the tool being tested by OutputCheck).

The advantage of writing directives in this way is that the directives can be written next to the code that generates the output that the directive is checking for.

All directives use the regular expression syntax used by the re python module. It is also important to note that the any spaces after the : until the first non-whitespace character are not considered part of the regular expression.

The following directives are supported

CHECK: <regex>

This declares that that regular expression <regex> should match somewhere on a single line. This match must occur after previously declared Check directives.

Succesful example

HelloWorld.c

#include <stdio.h>

int main()
{
    // CHECK: Hello World
    printf("Hello World\n");

    // CHECK: Goodbye
    printf("Goodbye\n");
    
    return 0;
}

$ cc HelloWorld.c -o HelloWorld
$ ./HelloWorld | OutputCheck HelloWorld.c

This example shows a simple C program being compiled and its output being checked. There are two CHECK: declarations which effectively say...

1. At least one of the lines must match the regular expression Hello World.
2. At least one line after the previous match must match regular expression Goodbye.

It can be seen that the order the CHECK: directives are declared is important. If the directives were specified the other way round then the OutputCheck tool would report an error as shown below

Failing example

BrokenHelloWorld.c

#include <stdio.h>

int main()
{
    // CHECK: Goodbye
    printf("Hello World\n");

    // CHECK: Hello World
    printf("Goodbye\n");
    
    return 0;
}

$ cc BrokenHelloWorld.c -o BrokenHelloWorld
$ ./BrokenHelloWorld | OutputCheck BrokenHelloWorld.c
ERROR: Could not find a match for Check Directive (BrokenHelloWorld.c:8 Pattern: 'Hello World')

CHECK-L: <string>

This is the string literal version of the CHECK: directive. This is identical to the CHECK: directive except that <string> is a literal string rather than a regular expression. This is useful if using python's regular expression syntax is too cumbersome.

For example

// CHECK: foo\.c\(5\) '\*' is not a valid identifier\.
// CHECK-L: foo.c(5) '*' is not a valid identifier.

the above directives are equivalent but the CHECK-L: is much easier to write.

CHECK-NEXT: <regex>

This declares that the next line after the previous match must match the regular expression <regex>. If there was no previous directive then CHECK-NEXT: matches the first line of the tool's output.

Succesful example

HelloWorld2.c

#include <stdio.h>

int main()
{
    // CHECK: Hello World
    // CHECK-NEXT: Goodbye
    printf("Hello World\nGoodbye");

    return 0;
}

$ cc HelloWorld2.c -o HelloWorld2
$ ./HelloWorld2 | OutputCheck HelloWorld2.c

Failing example

BrokenHelloWorld2.c

#include <stdio.h>

int main()
{
    // CHECK: Hello World
    
    printf("Hello World\n");
    printf("Testing...\n");
    
    // CHECK-NEXT: Goodbye
    printf("Goodbye\n");

    return 0;
}

$ cc BrokenHelloWorld2.c -o BrokenHelloWorld2
$ ./BrokenHelloWorld2 | OutputCheck BrokenHelloWorld2.c
ERROR: Could not find a match for CheckNext Directive (BrokenHelloWorld2.c:10 Pattern: 'Goodbye') expected at <stdin>:2

CHECK-NEXT-L: <string>

This is the string literal version of the CHECK-NEXT: directive. This is identical to the CHECK-NEXT: directive except that <string> is a literal string rather than a regular expression. This is useful if using python's regular expression syntax is too cumbersome.

CHECK-NOT: <regex>

This declares that between the previous match (if there is none, search starts from the first line of tool's output) and the next match (if there is none the search will search to the end of the tool's output) that no line will match the regular expression <regex>.

Succesful example

HelloWorld3.c

#include <stdio.h>

int main()
{
    // CHECK: Hello World
    // CHECK-NOT: Testing
    // CHECK: Goodbye
    printf("Hello World\nGoodbye");

    return 0;
}

$ cc HelloWorld3.c -o HelloWorld3
$ ./HelloWorld3 | OutputCheck HelloWorld3.c

Failing example

BrokenHelloWorld3.c

#include <stdio.h>

int main()
{
    // CHECK: Hello World
    printf("Hello World\n");
    
    // CHECK-NOT: Testing
    // CHECK: Goodbye
    printf("Testing...\n");
    printf("Goodbye\n");

    return 0;
}

$ cc BrokenHelloWorld3.c -o BrokenHelloWorld3
$ ./BrokenHelloWorld3 | OutputCheck BrokenHelloWorld3.c
ERROR: Found a match for CheckNot Directive (BrokenHelloWorld3.c:8 : Pattern: 'Testing') in <stdin>:2

CHECK-NOT-L: <string>

This is the string literal version of the CHECK-NOT: directive. This is identical to the CHECK-NOT: directive except that <string> is a literal string rather than a regular expression. This is useful if using python's regular expression syntax is too cumbersome.

Substitutions

For convenience several substitutions are provided for use in OutputCheck. These substitutions can used in <regex> or <string> for all the check directives. All directives can be escaped by prepending \.

${LINE}

This is substituted by the line number that the check directive is on in the check file.

// CHECK-L : file.c(${LINE})

${LINE:+N}

This is substituted by the line number that the check directive is on in the check file plus an offset (N).

// CHECK-L : file.c(${LINE:+5})

${LINE:-N}

This is substituted by the line number that the check directive is on in the check file minus an offset (N).

// CHECK-L : file.c(${LINE:-5})

${CHECKFILE_NAME}

This is substituted by the name of the check file.

// CHECK-L : ${CHECKFILE_NAME}(5)

${CHECKFILE_ABS_PATH}

This is substituted by the absolute path to check file.

// CHECK-L : ${CHECKFILE_ABS_PATH}(5)

Tests

A small set of tests are present in the tests/ directory. These tests are designed to be driven using llvm-lit from LLVM >=3.4 . These tests are not cross platform and will only work on systems with the sed and grep programs. It should be noted that the OutputCheck tool is implemented purely in python so the tool should work on platforms that support Python.