Parameterized testing in Python sucks.
parameterized
fixes that. For everything. Parameterized testing for nose, parameterized testing for py.test, parameterized testing for unittest.
# test_math.py
from nose.tools import assert_equal
from parameterized import parameterized, parameterized_class
import unittest
import math
@parameterized([
(2, 2, 4),
(2, 3, 8),
(1, 9, 1),
(0, 9, 0),
])
def test_pow(base, exponent, expected):
assert_equal(math.pow(base, exponent), expected)
class TestMathUnitTest(unittest.TestCase):
@parameterized.expand([
("negative", -1.5, -2.0),
("integer", 1, 1.0),
("large fraction", 1.6, 1),
])
def test_floor(self, name, input, expected):
assert_equal(math.floor(input), expected)
@parameterized_class(('a', 'b', 'expected_sum', 'expected_product'), [
(1, 2, 3, 2),
(5, 5, 10, 25),
])
class TestMathClass(unittest.TestCase):
def test_add(self):
assert_equal(self.a + self.b, self.expected_sum)
def test_multiply(self):
assert_equal(self.a * self.b, self.expected_product)
@parameterized_class([
{ "a": 3, "expected": 2 },
{ "b": 5, "expected": -4 },
])
class TestMathClassDict(unittest.TestCase):
a = 1
b = 1
def test_subtract(self):
assert_equal(self.a - self.b, self.expected)
With nose (and nose2):
$ nosetests -v test_math.py
test_floor_0_negative (test_math.TestMathUnitTest) ... ok
test_floor_1_integer (test_math.TestMathUnitTest) ... ok
test_floor_2_large_fraction (test_math.TestMathUnitTest) ... ok
test_math.test_pow(2, 2, 4, {}) ... ok
test_math.test_pow(2, 3, 8, {}) ... ok
test_math.test_pow(1, 9, 1, {}) ... ok
test_math.test_pow(0, 9, 0, {}) ... ok
test_add (test_math.TestMathClass_0) ... ok
test_multiply (test_math.TestMathClass_0) ... ok
test_add (test_math.TestMathClass_1) ... ok
test_multiply (test_math.TestMathClass_1) ... ok
test_subtract (test_math.TestMathClassDict_0) ... ok
----------------------------------------------------------------------
Ran 12 tests in 0.015s
OK
As the package name suggests, nose is best supported and will be used for all further examples.
With py.test (version 2.0 and above):
$ py.test -v test_math.py
============================= test session starts ==============================
platform darwin -- Python 3.6.1, pytest-3.1.3, py-1.4.34, pluggy-0.4.0
collecting ... collected 13 items
test_math.py::test_pow::[0] PASSED
test_math.py::test_pow::[1] PASSED
test_math.py::test_pow::[2] PASSED
test_math.py::test_pow::[3] PASSED
test_math.py::TestMathUnitTest::test_floor_0_negative PASSED
test_math.py::TestMathUnitTest::test_floor_1_integer PASSED
test_math.py::TestMathUnitTest::test_floor_2_large_fraction PASSED
test_math.py::TestMathClass_0::test_add PASSED
test_math.py::TestMathClass_0::test_multiply PASSED
test_math.py::TestMathClass_1::test_add PASSED
test_math.py::TestMathClass_1::test_multiply PASSED
test_math.py::TestMathClassDict_0::test_subtract PASSED
==================== 12 passed, 4 warnings in 0.16 seconds =====================
With unittest (and unittest2):
$ python -m unittest -v test_math
test_floor_0_negative (test_math.TestMathUnitTest) ... ok
test_floor_1_integer (test_math.TestMathUnitTest) ... ok
test_floor_2_large_fraction (test_math.TestMathUnitTest) ... ok
test_add (test_math.TestMathClass_0) ... ok
test_multiply (test_math.TestMathClass_0) ... ok
test_add (test_math.TestMathClass_1) ... ok
test_multiply (test_math.TestMathClass_1) ... ok
test_subtract (test_math.TestMathClassDict_0) ... ok
----------------------------------------------------------------------
Ran 8 tests in 0.001s
OK
(note: because unittest does not support test decorators, only tests created with @parameterized.expand
will be executed)
With green:
$ green test_math.py -vvv
test_math
TestMathClass_1
. test_method_a
. test_method_b
TestMathClass_2
. test_method_a
. test_method_b
TestMathClass_3
. test_method_a
. test_method_b
TestMathUnitTest
. test_floor_0_negative
. test_floor_1_integer
. test_floor_2_large_fraction
TestMathClass_0
. test_add
. test_multiply
TestMathClass_1
. test_add
. test_multiply
TestMathClassDict_0
. test_subtract
Ran 12 tests in 0.121s
OK (passes=9)
$ pip install parameterized
Yes (mostly).
Py3.7 | Py3.8 | Py3.9 | Py3.10 | Py3.11 | PyPy3 | @mock.patch |
|
---|---|---|---|---|---|---|---|
nose | yes | yes | yes | yes | no§ | no§ | yes |
nose2 | yes | yes | yes | yes | yes | yes | yes |
py.test 2 | no* | no* | no* | no* | no* | no* | no* |
py.test 3 | yes | yes | yes | yes | no* | no* | yes |
py.test 4 | no** | no** | no** | no** | no** | no** | no** |
py.test fixtures | no† | no† | no† | no† | no† | no† | no† |
unittest ( @parameterized.expand ) |
yes | yes | yes | yes | yes | yes | yes |
unittest2 ( @parameterized.expand ) |
yes | yes | yes | yes | no§ | no§ | yes |
§: nose and unittest2 - both of which were last updated in 2015 - sadly do not appear to support Python 3.10 or 3.11.
*: py.test 2 does not appear to work under Python 3 (#71), and py.test 3 does not appear to work under Python 3.10 or 3.11 (#154).
**: py.test 4 is not yet supported (but coming!) in issue #34
†: py.test fixture support is documented in issue #81
(this section left intentionally blank)
The @parameterized
and @parameterized.expand
decorators accept a list or iterable of tuples or param(...)
, or a callable which returns a list or iterable:
from parameterized import parameterized, param
# A list of tuples
@parameterized([
(2, 3, 5),
(3, 5, 8),
])
def test_add(a, b, expected):
assert_equal(a + b, expected)
# A list of params
@parameterized([
param("10", 10),
param("10", 16, base=16),
])
def test_int(str_val, expected, base=10):
assert_equal(int(str_val, base=base), expected)
# An iterable of params
@parameterized(
param.explicit(*json.loads(line))
for line in open("testcases.jsons")
)
def test_from_json_file(...):
...
# A callable which returns a list of tuples
def load_test_cases():
return [
("test1", ),
("test2", ),
]
@parameterized(load_test_cases)
def test_from_function(name):
...
Note that, when using an iterator or a generator, all the items will be loaded into memory before the start of the test run (we do this explicitly to ensure that generators are exhausted exactly once in multi-process or multi-threaded testing environments).
The @parameterized
decorator can be used test class methods, and standalone functions:
from parameterized import parameterized
class AddTest(object):
@parameterized([
(2, 3, 5),
])
def test_add(self, a, b, expected):
assert_equal(a + b, expected)
@parameterized([
(2, 3, 5),
])
def test_add(a, b, expected):
assert_equal(a + b, expected)
And @parameterized.expand
can be used to generate test methods in situations where test generators cannot be used (for example, when the test class is a subclass of unittest.TestCase
):
import unittest
from parameterized import parameterized
class AddTestCase(unittest.TestCase):
@parameterized.expand([
("2 and 3", 2, 3, 5),
("3 and 5", 3, 5, 8),
])
def test_add(self, _, a, b, expected):
assert_equal(a + b, expected)
Will create the test cases:
$ nosetests example.py
test_add_0_2_and_3 (example.AddTestCase) ... ok
test_add_1_3_and_5 (example.AddTestCase) ... ok
----------------------------------------------------------------------
Ran 2 tests in 0.001s
OK
Note that @parameterized.expand
works by creating new methods on the test class. If the first parameter is a string, that string will be added to the end of the method name. For example, the test case above will generate the methods test_add_0_2_and_3
and test_add_1_3_and_5
.
The names of the test cases generated by @parameterized.expand
can be customized using the name_func
keyword argument. The value should be a function which accepts three arguments: testcase_func
, param_num
, and params
, and it should return the name of the test case. testcase_func
will be the function to be tested, param_num
will be the index of the test case parameters in the list of parameters, and param
(an instance of param
) will be the parameters which will be used.
import unittest
from parameterized import parameterized
def custom_name_func(testcase_func, param_num, param):
return "%s_%s" %(
testcase_func.__name__,
parameterized.to_safe_name("_".join(str(x) for x in param.args)),
)
class AddTestCase(unittest.TestCase):
@parameterized.expand([
(2, 3, 5),
(2, 3, 5),
], name_func=custom_name_func)
def test_add(self, a, b, expected):
assert_equal(a + b, expected)
Will create the test cases:
$ nosetests example.py
test_add_1_2_3 (example.AddTestCase) ... ok
test_add_2_3_5 (example.AddTestCase) ... ok
----------------------------------------------------------------------
Ran 2 tests in 0.001s
OK
The param(...)
helper class stores the parameters for one specific test case. It can be used to pass keyword arguments to test cases:
from parameterized import parameterized, param
@parameterized([
param("10", 10),
param("10", 16, base=16),
])
def test_int(str_val, expected, base=10):
assert_equal(int(str_val, base=base), expected)
If test cases have a docstring, the parameters for that test case will be appended to the first line of the docstring. This behavior can be controlled with the doc_func
argument:
from parameterized import parameterized
@parameterized([
(1, 2, 3),
(4, 5, 9),
])
def test_add(a, b, expected):
""" Test addition. """
assert_equal(a + b, expected)
def my_doc_func(func, num, param):
return "%s: %s with %s" %(num, func.__name__, param)
@parameterized([
(5, 4, 1),
(9, 6, 3),
], doc_func=my_doc_func)
def test_subtraction(a, b, expected):
assert_equal(a - b, expected)
$ nosetests example.py
Test addition. [with a=1, b=2, expected=3] ... ok
Test addition. [with a=4, b=5, expected=9] ... ok
0: test_subtraction with param(*(5, 4, 1)) ... ok
1: test_subtraction with param(*(9, 6, 3)) ... ok
----------------------------------------------------------------------
Ran 4 tests in 0.001s
OK
Finally @parameterized_class
parameterizes an entire class, using either a list of attributes, or a list of dicts that will be applied to the class:
from yourapp.models import User
from parameterized import parameterized_class
@parameterized_class([
{ "username": "user_1", "access_level": 1 },
{ "username": "user_2", "access_level": 2, "expected_status_code": 404 },
])
class TestUserAccessLevel(TestCase):
expected_status_code = 200
def setUp(self):
self.client.force_login(User.objects.get(username=self.username)[0])
def test_url_a(self):
response = self.client.get('/url')
self.assertEqual(response.status_code, self.expected_status_code)
def tearDown(self):
self.client.logout()
@parameterized_class(("username", "access_level", "expected_status_code"), [
("user_1", 1, 200),
("user_2", 2, 404)
])
class TestUserAccessLevel(TestCase):
def setUp(self):
self.client.force_login(User.objects.get(username=self.username)[0])
def test_url_a(self):
response = self.client.get("/url")
self.assertEqual(response.status_code, self.expected_status_code)
def tearDown(self):
self.client.logout()
The @parameterized_class
decorator accepts a class_name_func
argument, which controls the name of the parameterized classes generated by @parameterized_class
:
from parameterized import parameterized, parameterized_class
def get_class_name(cls, num, params_dict):
# By default the generated class named includes either the "name"
# parameter (if present), or the first string value. This example shows
# multiple parameters being included in the generated class name:
return "%s_%s_%s%s" %(
cls.__name__,
num,
parameterized.to_safe_name(params_dict['a']),
parameterized.to_safe_name(params_dict['b']),
)
@parameterized_class([
{ "a": "hello", "b": " world!", "expected": "hello world!" },
{ "a": "say ", "b": " cheese :)", "expected": "say cheese :)" },
], class_name_func=get_class_name)
class TestConcatenation(TestCase):
def test_concat(self):
self.assertEqual(self.a + self.b, self.expected)
$ nosetests -v test_math.py
test_concat (test_concat.TestConcatenation_0_hello_world_) ... ok
test_concat (test_concat.TestConcatenation_0_say_cheese__) ... ok
If a test function only accepts one parameter and the value is not iterable, then it is possible to supply a list of values without wrapping each one in a tuple:
@parameterized([1, 2, 3])
def test_greater_than_zero(value):
assert value > 0
Note, however, that if the single parameter is iterable (such as a list or tuple), then it must be wrapped in a tuple, list, or the param(...)
helper:
@parameterized([
([1, 2, 3], ),
([3, 3], ),
([6], ),
])
def test_sums_to_6(numbers):
assert sum(numbers) == 6
(note, also, that Python requires single element tuples to be defined with a trailing comma: (foo, )
)
parameterized
can be used with mock.patch
, but the argument ordering can be confusing. The @mock.patch(...)
decorator must come below the @parameterized(...)
, and the mocked parameters must come last:
@mock.patch("os.getpid")
class TestOS(object):
@parameterized(...)
@mock.patch("os.fdopen")
@mock.patch("os.umask")
def test_method(self, param1, param2, ..., mock_umask, mock_fdopen, mock_getpid):
...
Note: the same holds true when using @parameterized.expand
.
To migrate a codebase from nose-parameterized
to parameterized
:
- Update your requirements file, replacing
nose-parameterized
withparameterized
. -
Replace all references to
nose_parameterized
withparameterized
:$ perl -pi -e 's/nose_parameterized/parameterized/g' your-codebase/
- You're done!
- What happened to Python 2.X, 3.5, and 3.6 support?
-
As of version 0.9.0,
parameterized
no longer supports Python 2.X, 3.5, or 3.6. Previous versions ofparameterized
- 0.8.1 being the latest -will continue to work, but will not receive any new features or bug fixes. - What do you mean when you say "nose is best supported"?
-
There are small caveates with
py.test
andunittest
:py.test
does not show the parameter values (ex, it will showtest_add[0]
instead oftest_add[1, 2, 3]
), andunittest
/unittest2
do not support test generators so@parameterized.expand
must be used. - Why not use
@pytest.mark.parametrize
? -
Because spelling is difficult. Also,
parameterized
doesn't require you to repeat argument names, and (usingparam
) it supports optional keyword arguments. - Why do I get an
AttributeError: 'function' object has no attribute 'expand'
with@parameterized.expand
? -
You've likely installed the
parametrized
(note the missing e) package. Useparameterized
(with the e) instead and you'll be all set. - What happened to
nose-parameterized
? -
Originally only nose was supported. But now everything is supported, and it only made sense to change the name!