Tasty SUGAR - Search Using Golden Answer References

Tasty SUGAR - Search Using Golden Answer References

The tasty-sugar package extends the tasty testing framework with the ability to generate tests based on golden answer files found for specific inputs. Multiple answers may be specified with different parameterization, and there can be associated files that are presented to the test as well.

The primary use of tasty-sugar is to generate test cases based on the contents of a directory, where the presence of various files determine which tests are generated.

Elements of tasty-sugar:

Tasty.Sugar.CUBE

Configuration Using Base Expectations

Describes the configuration for tasty-sugar tests, including where they are located and what syntax the files should have.

Tasty.Sugar.Sweets

Specifications With Existing Expected Testing Samples

The tasty-sugar library uses one or more CUBE’s to generate a set of test configurations based on the existing files found, outputting a list of Sweets representing the existing test data.

How to use tasty-sugar

Full information on tasty-sugar features and capabilities will be provided in a later Detailed Information section, but this is a quick introduction describing various testing use cases and how tasty-sugar can be used in those use cases.

For these motivational use cases, the example scenario basis is that the target to be tested is a tool to parse binary ELF files and generate various output information about those files (e.g. similar to objdump, but in a Haskell library form).

Single Expected Output per Test

Scenario

When running a test, it generates output that should be compared to the expected data maintained in a file. There is a simple, single expected output for each test, and no inputs other than the test name.

For the example scenario, several actual ELF binary files were collected and placed in the test/samples directory:

$ ls test/samples
empty.txt
fibonacci.c
fibonacci
foo.tar
hello.c
hello
ls.c
ls
tmux.c
tmux
$

Note that there are a couple of non-ELF files in there as well, to verify the errors generated by our library when given

The actual outputs aren’t known yet, but tasty-sugar can help with that. Setup the tasty-sugar CUBE configuration as follows:

cube = mkCUBE { inputDir = "test/samples"
              , rootName = "*.c"
              , expectedSuffix = "exp"
              , associatedNames = [ ("binary", "") ]
              }
main = do testSweets <- findSugar cube
          tests <- withSugarGroups testSweets testGroup $
            \sweets expIdx expectation -> return $
              testCase (rootMatchName sweets <> " #" <> show expIdx) $ do
                let Just binaryName = lookup "binary" $ associated expectation
                r <- runTestOn binaryName
                e <- readFile $ expectedFile expectation
                r @?= e
          defaultMain $ testGroup "elf" tests

runTestOn :: FilePath -> IO String
runTestOn f = ...

The tasty-sugar framework does not provide the actual testing: that is still provided by the developer. Instead, the tasty-sugar framework reads the contents of the test/samples directory and analyses the available files to create a list of tests that should be run. The tasty-sugar package also provides a function that can organize the tests and invoke the user’s test function once for each test configuration.

If the tests are run at this point, the tasty-sugar framework will do nothing.

Why?

The tasty-sugar framework will ignore any files in the target directory that do not have an associated expected file describing the expected output. This can be confirmed by running the tests with the --showsearch argument, which will use an alternate tasty ingredient that does not actually run the tests but write out the search process and search results.

To get actual tests to run, simply create an expected file for each of the input candidates. The contents of the file can be empty, or any random data.

Running the tests now will result in a test created for each input file that has a corresponding *.exp file. Note that tasty-sweet doesn’t actually read in any of the files, just invokes the test creation function with the Sweets and Expectation data structures that let the test do whatever is appropriate.

$ ls test/samples
empty.txt
empty.txt.exp
fibonacci
finonacci.c
fibonacci.exp
foo.tar
foo.tar.exp
hello
hello.c
hello.exp
ls
ls.c
ls.exp
tmux
tmux.c
$

Note that empty.txt and foo.tar will be ignored, even though there is an .exp file for them because they don’t match the source target of *.c. Similarly, tmux will be ignored because there is no .exp file for it.

At this point, any changes to the target library that cause output changes will be identified when running the tests.

Single Input and Output per Test

Scenario

Similar to the previous scenario, but there is a file containing the expected input needed by the test to generate the output.

To extend the previous example, let us assume that in addition to the pre-existing binaries that we will be generating a number of “interesting” binaries to run the target library on. These will be kept in a different directory where a different part of the build will compile the sources to generate the binaries for testing:

$ ls test/src_samples
foo.c
foo
foo.expct
simple.c
simple
simple.expct
recursive.rs
recursive
recursive.expct
functional.hs
functional
functional.expct
$

Note that there are several different source types (C, Rust, Haskell) involved, but each of them has an associated output binary that the target library should be tested on.

In an initial approach, the source files can be ignored by the testing code: simply create a FILE.expct file for each of the binaries and use the same Tasty.Sugar.CUBE configuration for this directory as for the previous directory.

However however an approach where the actual test written by the user needs access to the source file itself for some reason. This can be handled by specifying an “associated” file in the Tasty.Sugar.CUBE configuration:

cube = mkCUBE { inputDir = "test/samples"
              , rootName = "*.exe"
              , expectedSuffix = "expct"
              , associatedNames = [ ("c-source", ".c")
                                  , ("rust-source", ".rs")
                                  , ("haskell", ".hs)
                                  ]
              }

ingredients = includingOptions sugarOptions :
              sugarIngredients [cube] <> defaultIngredients

main = do testSweets <- findSugar cube
          tests <- withSugarGroups testSweets testGroup $
            \sweets expIdx expectation -> return $
              testCase (rootMatchName sweets <> " #" <> show expIdx) $ do
                e <- readFile $ expectedFile expectation
                let assoc = associated expectation
                    f = rootFile sweets
                r <- case lookup "c-source" assoc of
                       Just c -> runCTestOn f
                       Nothing ->
                         case lookup "rust-source" assoc of
                           Just r -> runRustTestOn f
                           Nothing ->
                             runHaskellTestOn f
                r @?= e
          defaultMainWithIngredients ingredients $
            testGroup "elf" tests

runCTestOn :: FilePath -> IO String
runCTestOn f = ...

runRustTestOn :: FilePath -> IO String
runRustTestOn f = ...

runHaskellTestOn :: FilePath -> IO String
runHaskellTestOn f = ...

Now when tasty-sugar generates the test configurations, each test will have a name, a source file, an expected file, and a single associated file. The test is free to use these files in any way it sees fit. For the configuration above, there would be 4 test configurations provided to the test:

Note that if both “simple.c” and “simple.hs” files existed, then the simple test configuration would get both as associated files.

Single Input with different parameters producing different outputs

Scenario

For each input file, multiple tests should be run, each with different parameters, and the expected output may or may not depend on the parameter.

Using the previous example scenario, let’s now assume that for each of the source sample files, two different executables were built: one with and one without optimization. Additionally, if they were a C source file, then there was a version built with GCC and a version built with Clang. The output executables are now named accordingly:

$ ls test/src_samples
foo.c
foo.noopt.clang.exe
foo.O0.gcc.exe
foo.opt.clang.exe
foo.O2.gcc.exe
foo.O3.gcc.exe
simple.c
simple.noopt.clang.exe
simple.noopt.gcc.exe
simple.opt-clang.exe
simple-opt.gcc-exe
recursive.rs
recursive.noopt.exe
recursive.opt.exe
functional.hs
functional.noopt.exe
functional.opt.exe
$

While the filenames are fairly regular, there are different numbers of executables and different naming conventions for different files.

The opt/noopt/O0/O2/O3 and gcc/clang information is known to tasty-sugar as a “parameter”. Parameters can appear in the filename in a specific order, and each parameter may have one of a set of valid values (e.g. gcc or clang) or it may have any (free-form) value (as with the optimization specification).

The Tasty.Sugar.CUBE confguration for is scenario is updated to:

cube = mkCUBE { inputDir = "test/samples"
              , rootName = "*"
              , separators = "-."
              , expectedSuffix = "expct"
              , associatedNames = [ ("c-source", ".c")
                                  , ("rust-source", ".rs")
                                  , ("haskell", ".hs")
                                  ]
              , validParams = [
                 ("optimization", Nothing)
                ,("c-compiler", Just ["gcc", "clang"])
                ]
              }

ingredients = includingOptions sugarOptions :
              sugarIngredients [cube] <> defaultIngredients

main = do testSweets <- findSugar cube
          tests <- withSugarGroups testSweets testGroup $
            \sweets expIdx expectation -> return $
              testCase (rootMatchName sweets <> " #" <> show expIdx) $ do
                e <- readFile $ expectedFile expectation
                let assoc = associated expectation
                    f = rootFile sweets
                r <- case lookup "c-source" assoc of
                       Just c -> runCTestOn f
                       Nothing ->
                         case lookup "rust-source" assoc of
                           Just r -> runRustTestOn f
                           Nothing -> runHaskellTestOn f
                r @?= e
          defaultMainWithIngredients ingredients $
          testGroup "elf" tests

runCTestOn :: FilePath -> String
runCTestOn f = ...

runRustTestOn :: FilePath -> String
runRustTestOn f = ...

runHaskellTestOn :: FilePath -> String
runHaskellTestOn f = ...

Parameters are separated by designated separator characters and must appear in the order declared. The default separators are “.” and “-” (e.g. both of the two optimized executable files for the simple.c source above are accepted).

Filenames may omit later parameter values: the file is assumed to apply to all unspecified parameter values if there is no more specific override. This can be very useful to avoid repetition and copying when specifying test files.

In the above example, a simple.expected file would be used for all four executables, but if there was also a simple.noopt-gcc.expected and a simple-opt.expected then the former would be used only for the simple.noopt.gcc.exe and the latter would be used for both the gcc and the clang executables, leaving the sample.expected to be used only for the simple.opt-clang.exe file.

Comparisons

tasty-KAT

• The tasty-KAT package reads both the inputs and the outputs from a single file, instaed of allowing the inputs to be a separate file that can be processed by the target under test.
  • The tasty-sugar package allows inputs and outputs to be in separate files, and additional “associated” files to be provided as inputs to the test.
• The tasty-KAT package inputs and outputs must be specifiable in a file with other KAT markup; this does not easily handle text markup conflicts and binary inputs/outputs.
  • The tasty-sugar package does not attempt to interpret the contents of the files, but simply passes them to the test itself.
• The tasty-KAT package does not allow auxiliary files, or different parameterized tests.
  • As mentioned above, tasty-sugar allows multiple auxiliary files per tests, and allows test inputs and expected outputs to be filename parameterized (with either constrained or free-form parameter values).

tasty-golden

• The tasty-golden package requires a 1:1 association between tests and corresponding golden expected output files; it does not support file-provided inputs, or associated files.
  • The tasty-sugar package allows multiple associated files in addition to the primary input file.
  • The tasty-sugar package supports parameterization of expected results (and associated files) as part of the filenames to allow multiple tests per input.
• The tasty-golden package will write the expected results if the expected file is missing.
  • The tasty-sugar package does not actually read or write any of the files identified, it simply provides the names of the files to the test generator function. The user’s test code is responsible for all file processing.

tasty-silver

Similar to tasty-golden in functionality.

Features unique to tasty-sugar

• Multiple potential outputs, parameterized by filename elements.
• Multiple associated input files.
• Search analysis mode showing how tests are generated based on the available files.
• Automatic grouping of generated tests by parameter values.

Limitations

• Huge directories
• Huge files
• Will throw any exception that the listDirectory function can throw.

Detailed Information

Requirements

• There must be a root (input) file to feed to the test
• There must be one or more “expected” results files for a root file
• There may be associated files for the root file required for the test
• All three groups of files may be parameterized by additional fields.
• All fields are represented by a common basename with optional parameters and required associated suffixes, separated by allowable separators.

All of the above may utilize globbing as provided by System.FilePath.Glob

Examples

Example:

For example, a test which would verify that the size of a compiled file meets the expectations would specify:

CUBE =
  { inputDir = "tests/samples"  -- relative to cabal file
  , separators = ".-"
  , rootName = "*.c"
  , associatedNames = [ ("exe", "exe")
                      , ("object", "o")
                      ]
  , expectedSuffix = "expected"
  , validParams = [ ("arch" : Just ["ppc", "x86_64"]) ]
  }

And given the following directory configuration:

tests/samples/
   foo.c
   bar.c
   bar.exe
   bar.ppc.exe
   bar.expected
   cow.c
   cow.ppc.exe
   cow.x86_64.exe
   cow.expected
   cow.ppc-expected
   cow.x86.expected
   moo.c
   moo.exe
   moo-expected
   dog.exe
   dog.expected

The result would be:

sweets =
  [ Sweets
    { rootMatchName = "bar"
    , rootBaseName = "bar"
    , rootFile = "tests/samples/bar.c"
    , expected =
        [ Expectation
          { expectedFile = "tests/samples/bar.expected"
          , associated = [ ("exe", "tests/samples/bar.exe") ]
          , expParamsMatch = []
          }
        , Expectation
          { expectedFile = "tests/samples/bar.expected"
          , associated = [ ("exe", "tests/samples/bar.ppc.exe") ]
          , expParamsMatch = [ ("arch", "ppc") ]
          }
        ]
    },
  , Sweets
    { rootMatchName = "cow"
    , rootBaseName = "cow"
    , rootFile = "tests/samples/cow.c"
    , expected =
        [ Expectation
          { expectedFile = "tests/samples/cow.ppc-expected"
          , associated = [ ("exe", "tests/samples/cow.ppc.exe") ]
          , expParamsMatch = [ { "arch", "ppc" } ]
          }
        , Expectedfile
          { expectedFile = "tests/samples/cow.expected"
          , associated = [ ("exe", "tests/samples/cow.x86_64.exe") ]
          , expParamsMatch = [ ("arch", "x86_64") ]
          }
        ]
    },
  , Sweets
    { rootMatchName = "moo"
    , rootBaseName = "moo"
    , rootFile = "tests/samples/moo.c"
    , expected =
        [ Expectation
          { expectedFile = "tests/samples/moo-expected"
          , associated = [ ("exe", "tests/samples/moo.exe") ]
          , expParamsMatch = []
          }
        ]
    },

FAQ

Why do the configurations need to be described by a Tasty.Sugar.CUBE data object? Why can’t they be passed in on the command-line?

Answer

They could be, but there are a couple of issues that would make that more awkward:

1. There would need to be a number of command-line arguments to describe all of the CUBE information.
2. The tasty framework provides command-line parsing and argument handling (and expects to do so). Handling some command-line arguments prior to tasty and some within tasty would be difficult and brittle (and also note that the set of all tests must be known prior to invoking the tasty main code; they cannot be added dynamically after that point).