E.g. test

I personally use this to test Python examples and it works well. However if you need better support for other languages, open a new issue. Tests cover basic usage of egtest.

An incorrect example is worse than no example at all. E.g. test parses code blocks from documentation, runs them and reports possible errors. Your examples can be written in any language. There are similar projects but they didn't suit my needs. (Motivation)

Example

Python code block, which tries to demonstrate printing

# We have syntax error in the example
print('This is how you print a line)

Running egtest README.md looks like this:

E.g. test supports piping, reporting in JSON format and everything needed to integrate with other tools. Check out detailed usage.

Install

You'll need Python to run egtest. Python versions 2.6, 2.7 and 3.3 are supported and tested against.

Install latest release with pip:

pip install egtest

Install latest development version using pip:

pip install git+git://github.com/kimmobrunfeldt/egtest.git

Install latest development version using setup.py:

git clone git@github.com:kimmobrunfeldt/egtest.git
cd egtest
python setup.py install

Detailed usage

Code examples are written into temporary files. They are run with command parsed from code block, and if the temporary code exits with non-zero return value, egtest reports errors.

Output of egtest --help

E.g. test - Test example code blocks in documentation

Usage:
  egtest [<filename>] [--reporter=<reporter>] [--parser=<parser>] [--config=<config>]
  egtest -h | --help
  egtest --version

Examples:
  egtest readme.md
  egtest --reporter json readme.md
  cat readme.md | egtest
  egtest < readme.md

Options:
  -r --reporter=<reporter>  Sets reporter. Valid values: basic, json.
  -p --parser=<parser>      Sets parser. Valid values: markdown.
  -c --config=<config>      External configuration. File path to config JSON.
  -h --help                 Show this screen.
  -v --version              Show version.

You can use external JSON file along with command line parameters to control egtest. Check example-config.json.

Reporters

E.g. test supports two different reporter types: basic and json.

When JSON reporter is used, egtest outputs results in JSON format. For example egtest --reporter json README.md | python -m json.tool outputs:

{
    "executions": [
        {
            "code": "# We have syntax error in the example\nprint('This is how you print a line)",
            "command": "python",
            "output": {
                "returnValue": 1,
                "stderr": "  File \"/var/folders/2l/qmg1cgh90h1fdzjcdp9_ss580000gp/T/tmpOJfmPa\", line 6\n    print('This is how you print a line)\n                                       ^\nSyntaxError: EOL while scanning string literal\n",
                "stdout": ""
            }
        }
    ]
}

Parsers

Parser reads documentation in text format and extracts all code examples including what command should be used to run them.

Currently only one format is supported, which is GitHub's markdown. It parses only fenced code blocks to extract the language of code.

Injecting code

It is possible to define custom code which will be injected on each example before running. For Python code, egtest adds current working directory to sys.path to make imports possible.

Currently there's no sensible way to add custom injections without modifying egtest/injecthooks.py.

Motivation

I wrote this to test examples for Nap, which is a convenience wrapper for requests. Nap's examples are tested for each commit with Travis CI. Doctest has similar functionality but you need to start every line in example with >>>. It makes the examples harder to follow. Sphinx supports running examples like egtest does but Sphinx is a bit heavyweight and it supports only Python.

Contributing

Documentation for Egtest developers