FileCheck.py
Attempt to reimplement LLVM's FileCheck using Python.
Background
We know at least two software projects that would benefit from a suite of LLVM LIT integration tests:
The problem is that you need to have to build FileCheck
from LLVM sources
which is not a trivial task for 1) people who are not aware with LLVM
infrastructure and 2) Python-based projects that would prefer to not have
to build anything from LLVM sources in their CI process.
The option of having pre-compiled binaries is a workaround, but we don't like to keep third-party binary artifacts in source code, (see https://github.com/doorstop-dev/doorstop/pull/431#issuecomment-549237579).
Goals
The first goal is to make FileCheck.py pass on Mull's current integration tests suite which uses only a very limited subset of FileCheck's features which are as follows:
- Commands (both substring and regex matching):
CHECK
CHECK-NEXT
CHECK-NOT
CHECK-EMPTY
- Options:
--strict-whitespace
--match-full-lines
--check-prefix
When this is done, it feels like a good idea to implement full FileCheck's contract and be 100% compatible with the C++ version.
There is a file in this repository: FileCheck.pdf which has some rough implementation coverage: green color means something is implemented, yellow color means that the text is not relevant to implementation, no highlighting means "still to be implemented".
Notes on implementation
The implementation is made against the latest FileCheck - Flexible pattern matching file verifier document. The plan is to skip looking into the LLVM's implementation of FileCheck, unless some very advanced implementation details are encountered.
It a nice programming exercise of implementing something from a spec but we also want to make FileCheck.py to conform to the same contract this is why FileCheck.py is tested using LIT and the LLVM's FileCheck:
First, real FileCheck is run against a LIT/FileCheck test suite, then FileCheck.py is run on the same tests to make sure that it has the same behavior as its LLVM FileCheck parent.