Unit Tests for NodeJS and the Browser

Table of Contents generated with DocToc

• GuyTest
  • ██████████████
  • Suggested Organization of Tests
  • Public API
    • Test Configuration
    • Running Tests
  • Results and Stats
  • Proper Usage of Async Testing Methods
  • Notes on Private API
  • To Do
  • Is Done

• a 'test' is a single run of one or more 'tasks'

• a 'task' is a named function with any number of named 'checks'

• each 'check' consists of an 'assumption', a named 'probing' function, and a number of matchers (depending on what the assumption method allows)

  • example:

    my_task = ->
      @eq ( my_check_1 = -> FOO.multiply 2, 2 ), 4
      @eq ( my_check_2 = -> FOO.multiply 3, 9 ), 27
      ^^^            ^^^^^^^^^                   ^^

  assumption check matcher

  
  * a probing function used in an `Test::eq()`uality assumption (or its asynchronous version,
  `Test::async_eq()`) should return a value or throw an error; a `pass` is counted if the return value
  equals the matcher; if the return value doesn't equal the matcher or an error is thrown, the check is
  considered a failure
  
  * a probing function used in an `Test::throws()` assumption (or its asynchronous version,
  `Test::async_throws()`) should throw an error; if no matcher is given, any exception thrown is regarded
  a success; otherwise, a `pass` is counted if the error either equals the matcher (if it is a string) or
  satisfies the regular expression, a `fail` is recorded if the probing function didn't throw an error or
  threw one that doesn't satisfy the matcher
  
  

• Assumptions (assumption methods) are

  • @eq: ( check, matcher ) ->: (assumption of synchronous equality)
  • @throws: ( check, matcher ) ->: (assumption of synchronous failure)
  • @async_eq: ( check, matcher ) ->: (assumption of asynchronous equality)
  • @async_throws: ( check, matcher ) ->: (assumption of asynchronous failure)

• In addition, there's

  • @pass: ( ref, message ): to record a pass, and
  • @fail: ( ref, message ): to record a failure that may result from 'free-form' tests

• task and probing functions will called in the context of the Test instance to make the assumption methods available as properties of @/this

• use simple objects whose properties are tasks

• properties can also recursively be objects with tasks, this allows for hierarchical structure that will be reflected in the names of tasks and checks in the result display

• start test by passing in your task objects, preferrably by wrapping the (outermost) task objects in an ad-hoc objects so you get meaningful names:

  taskgroup_A =
    test_1: -> ...
    better_use_meaningful_names: ->
      @eq ( t__20 = -> my_math_lib.mul 3, 4 ), 12
      @eq ( t__21 = -> my_math_lib.add 3, 4 ), 7
    subgroup:
      foo: -> ...
      bar: -> ...
  
  taskgroup_B = ...
  
  ( new Test() ).test { taskgroup_A, taskgroup_B, }
  
  # or, as the case may be:
  
  await ( new Test() ).async_test { taskgroup_A, taskgroup_B, }

  The report will then show, among other things:

  —————————————————————————————————————————————————————————————————
                          🙤 GUY TEST 🙦
  —————————————————————————————————————————————————————————————————
  ...
  taskgroup_A.better_use_meaningful_names.t__20 { passes: 1, fails: 0 }
  taskgroup_A.better_use_meaningful_names.t__21 { passes: 1, fails: 0 }
  ...
  —————————————————————————————————————————————————————————————————
  *                    { passes: 298, fails: 2 }
  —————————————————————————————————————————————————————————————————
  

NOTE some of the below settings are not yet implemented

• new Test: ( cfg ) ->: Perform tests, return statistics including total counts and per-test

• auto_reset (false): whether stats should be reset to zero whenever the Test::test() and Test::async_test() methods is called

• show_report (true): whether to show totals when the Test::test() and Test::async_test() methods terminates

• show_results (true): whether the outcome of each check should be printed to the terminal as soon as it concludes

• show_fails (true): whether failed checks should be printed to the terminal

• show_passes (true): whether passed checks should be printed to the terminal

• throw_on_error (false): whether an exception that is thrown from a check should cause the Test::test() and Test::async_test() methods to throw that error; if set to false, such errors will be caught and reported as fails

• throw_on_fail (false): whether a failed assumption should cause the Test::test() and Test::async_test() methods to throw an exception

• message_width (300): length limit on reported messages

• prefix (''): string to prefix each reported line with

The settings below can also be used as third argument to the module-level equals() method:

• ordered_objects (false): whether to consider two objects as equal even when their epynomous properties are not in the same order
• ordered_sets (false): whether to consider two sets as equal even when their epynomous elements are not in the same order
• ordered_maps (false): whether to consider two maps as equal even when their epynomous key / value pairs are not in the same order
• signed_zero (false): whether +0 and -0 should be considered equal

• Test::test: ( tests... ) ->: Perform tests, return statistics including total counts and per-test counts. Only synchronous tasks and checks will be run; when asynchronous tasks or checks are encountered, they will cause a fail

• await Test::async_test: ( tests... ) ->: Perform asynchronous and synchronous tests

• for each check: count
  • how often it has been called (redundant, is sum of all passes and fails)
  • how many passes and fails occurred with this check
• for each task: count
  • how often it has been called (redundant, is sum of all passes and fails)
  • how many passes and fails occurred with this task
• for each test: count
  • how many passes and fails occurred

af1 = ->       after 0.1, ->       throw new Error 'oops' ### not OK ###
af2 = -> await after 0.1, ->       throw new Error 'oops' ### not OK ###
af3 = ->       after 0.1, -> await throw new Error 'oops' ### OK ###
af4 = -> await after 0.1, -> await throw new Error 'oops' ### OK ###
# debug 'Ω_101', validate.asyncfunction af
f1 = ->
  try
    result = await af2()
  catch error
    warn error.message
  help result
await f1()

• Test::_on_task_not_called: ( ............... ) ->: Called when a task could not be run (e.g. because it was an async function when sync Test::test() was used, or because it was not a function or an object with properties)
• Test::_on_eq_passed: ( ............... ) ->: Called when Test::eq() or Test::async_eq() found that result equals matcher
• Test::_on_eq_failed: ( ............... ) ->: Called when Test::eq() or Test::async_eq() found that result doesn't equal matcher
• Test::_on_throws_passed: ( ............... ) ->: Called when Test::throws() or Test::async_throws() found that calling the check method did result in an error
• Test::_on_throws_didnt_throw: ( ............... ) ->:
• Test::_on_throws_mismatch: ( ............... ) ->:

• [–] docs

• [–] consider to move equality testing so its use doesn't depend on Test instance

• [–] custom error classes

• [–] provide stats per module, per method and totals

• [–] use proper namespacing for types

• [–] make sure ref is available in warnings at least when iterating over object properties

• [–] confirm that running Test::test() and / or Test::async_test() repeatedly on same instance sums up all stats, introduce counter to count the times one of these methods is called; what about using only assumptions such as test.eq() on their own outside of a call to Test::test() and / or Test::async_test()?

• [–] rename Test class to something more meaningful(?)

• [–] rename parameter f in assumption methods to check

• [–] allow to pass in multiple matchers to Test::throws(), Test::async_throws() so we can check both class and error message

• [–] implement equality for Maps

• [–] methods Types::pass(), Types::fail() whould take three arguments ref, cat and message; there could be an additional method Types::fail_eq() to display two lines with first cat result or error, second cat with doesn't match

• [–] implement instance-level and check-level configuration:

  • auto_reset: false,
  • show_report: true,
  • show_results: true,
  • show_fails: true,
  • show_passes: true,
  • throw_errors: false,

• [–] check that three-argument style calling is used everywhere for Test::pass() and Test::fail(), including in tests, docs

• [–] use call to Tests::_warn() to also display warning when so configured

• [–] introduce methods to also display ongoing messages when so configured

• [–] use wrapping methods to set and reset task ref as state to consolidate internal formation of compound refs

• [–] standardize cats, choose better name

• [–] replace Tests::_test_ref as it won't work in async tests

• [–] modify behavior of assumptions (eq(), throws(), async_eq(), async_throws(), pass(), fail()):

• [–] rename either @_upref or upref

• [–] provide a simple, configurable way to run all tests in a given location, matching a glob &c so it becomes easy to provide the code for a test-all module

• [–] provide a directory/project with aliased versions of projects to allow for testing of past versions; provide API to do that; PNPM CLI for this is pnpm add ${my-name}@npm:${registered-name}@${version}, e.g. pnpm add foo-3.2.1@npm:foo@3.2.1

• [–] use dbay to store results, calculate totals

• [–] provide hierarchical totals for each level

• [–] before calling a check function, print the current ref to the terminal; erase that line only when terminated without error and replace with success message

  print("\u1b[1F")
  #Moves cursor to beginning of the line n (default 1) lines up
  print(str)   #prints the new line
  print("\u1b[0K")
  # clears  part of the line.
  #If n is 0 (or missing), clear from cursor to the end of the line.
  #If n is 1, clear from cursor to beginning of the line.
  #If n is 2, clear entire line.
  #Cursor position does not change.

  • see https://discourse.julialang.org/t/how-clear-the-printed-content-in-terminal-and-print-to-the-same-line/19549/3
  • see https://stackoverflow.com/questions/1508490/erase-the-current-printed-console-line
  • see https://www.reddit.com/r/C_Programming/comments/502xun/how_do_i_clear_a_line_on_console_in_c/
  • see https://superuser.com/questions/1230544/is-there-any-way-to-clear-specific-number-of-lines-in-a-terminal

• [–] implement the ability to 'couple' a Test instance to another one such that past and future test results are transferred upstream; this will allow to perform different tests with different configurations and see all results in one summary nonetheless

• [–] implement benchmarking for Hengist-NG

• [–] implement remaining settings for Test CFG

• [+] should an asyncfunction be required for check functions used with async_eq(), async_throws()? doesn't work, cf. tests where timeout (after()) is used
• [+] include message with each fail
• [+] call check methods with single argument @/this (conventional parameter t or me) so as to allow bound check methods; also convenient for JavaScript where there are only fat-arrow functions
• [+] standardize handling and display of compound refs (using dot notation or slashes)
• [+] implement a Test::clear() method to reset stats?
• [+] use _record_failure() &c instead of _increment_fails() to do all the associated tasks in one step
  • [+] get rid of level kludge in _increment(), preferrably by using more specialized methods
• [+] list all the steps to be taken by _record_failure() and related methods, arguments needed
• [+] remove ? {} in create.gt_report_cfg cfg ? {} in _report() after bug fixed in InterType
• [+] consolidate calls to Test::_increment_fails(), warn(), Test::_warn() into single call
• [+] avoid null as ref for test
• [+] use 'task' as a better synonym for the ubiquitous 'test'
• [+] restructure use of t2 in tests to run inside of named functions
• [+] can tasks be nested? Does it make sense to have one task call one or more other tasks?
• [+] implement configuration to enable distinguishing positive and negative zero, key ordering in objects, sets and maps (call it 'strict' mode when all of the above are enabled)