pytest plugin for regression tests
pip install pytest-regtest==1.3.1
pytest-regtest is a pytest-plugin for implementing regression tests. Compared to functional testing a regression test does not test if software produces correct results, instead a regression test checks if software behaves the same way as it did before introduced changes.
More about regression testing at https://en.wikipedia.org/wiki/Regression_testing. Regression testing is a common technique to get started when refactoring legacy code lacking a test suite.
pytest-regtest allows capturing selected output which then can be compared to the captured output from former runs.
To install and activate this plugin execute:
$ pip install pytest-regtest
pytest-regtest plugin provides a fixture named regtest which can be used as a file handle for recording data:
from __future__ import print_function def test_squares_up_to_ten(regtest): result = [i*i for i in range(10)] # one way to record output: print(result, file=regtest) # alternative method to record output: regtest.write("done")
If you run this test script with pytest the first time there is no recorded output for this test function so far and thus the test will fail with a message including a diff:
$ py.test ... regression test output differences for test_demo.py::test_squares_up_to_ten: > --- current > +++ tobe > @@ -1,2 +1 @@ > -[0, 1, 4, 9, 16, 25, 36, 49, 64, 81] > -done > +
The output tells us what the current output is, and that the "tobe" output is still empty.
For accepting this output, we run pytest with the --reset-regtest flag:
$ py.test --regtest-reset
Now the next execution of py.test will succeed:
Now we break the test by modifying the code under test to compute the first eleven square numbers:
from __future__ import print_function def test_squares_up_to_ten(regtest): result = [i*i for i in range(11)] # changed ! # one way to record output: print(result, file=regtest) # alternative method to record output: regtest.write("done")
The next run of pytest delivers a nice diff of the current and expected output from this test function:
$ py.test ... > --- current > +++ tobe > @@ -1,2 +1,2 @@ > -[0, 1, 4, 9, 16, 25, 36, 49, 64, 81, 100] > +[0, 1, 4, 9, 16, 25, 36, 49, 64, 81] > done
The recorded output was written to files in the subfolder
_regtest_outputs next to your test script(s). You might keep this
folder under version control.
Another way to record output is the regtest_redirect fixture:
def test_squares_up_to_ten(regtest_redirect): result = [i*i for i in range(10)] with regtest_redirect(): print result
You can reset recorded output of files and functions individually as:
$ py.test --regtest-reset tests/test_00.py $ py.test --regtest-reset tests/test_00.py::test_squares_up_to_ten
To supress the diff and only see the stats use:
$ py.test --regtest-nodiff
To see recorded output during test execution run:
$ py.test --regtest-tee -s
If you develop on mixed platforms it might be usefull to ignore white spaces at the end of the lines when comparing output. This can be achieved by specifying:
$ py.test --regtest-ignore-line-endings
The recorded output can contain data which is changing from test run to test
run, e.g. pathes created with the
tmpdir fixture or hexadecimal object ids,
when objects are printed.
The plugin already replaces such changing data in the recorded output,
and one can register own converters in
conftest.py in the tests
folder. For example:
import pytest_regtest @pytest_regtest.register_converter_pre def fix_before(txt): """modify recorded output before the default fixes like temp folders or hex object ids are applied""" # remove lines with passwords: lines = txt.split('\n') lines = [l for l in lines if "password is" not in l] return '\n'.join(lines) @pytest_regtest.register_converter_post def after(txt): """modify recorded output after the default fixes like temp folders or hex object ids are applied""" # for demo only return txt.upper()
This can be used to fix substrings like "computation need 1.23 seconds" to "computation needed
One can register multiple such converters which will be applied in order of registration.