A recent post
described the new ability to run a subset of
from the tdda library
by tagging tests or test classes
Initially, this ability was only available for
From version 1.0 of the tdda library,
we have extended this capability to work with
This post is very similar to the
unittest-based tests, but adapted for
A decorator called
tagcan be imported and used to decorate individual tests or whole test classes (by preceding the test function or class with
pytestis run using the
--taggedoption, only tagged tests and tests from tagged test classes will be run.
There is a second new option,
--istagged. When this is used, the software will report which test classes are tagged, or contain tests that are tagged, but will not actually run any tests. This is helpful if you have a lot of test classes, spread across different files, and want to change the set of tagged tests.
The situations where we find this particularly helpful are:
Fixing a broken test or working on a new feature or dataset. We often find ourselves with a small subset of tests failing (perhaps, a single test) either because we're adding a new feature, or because something has changed, or because we are working with data that has slightly different characteristics. If the tests of interest run in a few seconds, but the whole test suite takes minutes or hours to run, we can iterate dramatically faster if we have an easy way to run only the subset of tests currently failing.
Re-writing test output. The
tddalibrary provides the ability to re-write the expected ("reference") output from tests with the actual result from the code, using the
--write-allcommand-line flag. If it's only a subset of the tests that have failed, there is real benefit in re-writing only their output. This is particularly true if the reference outputs contain some differences each time (version numbers, dates etc.) that are being ignored using the
ignore-patternsoptions provided by the library. If we regenerate all the test outputs, and then look at which files have changed, we might see differences in many reference files. In contrast, if we only regenerate the tests that need to be updated, we avoid committing unnecessary changes and reduce the likelihood of overlooking changes that may actually be incorrect.
In order to use the reference test functionality with
you have always needed to add some boilerplate code to
conftest.py in the directory from which you are running
To use the tagging capability, you need to add one more function definition,
The recommended imports in
conftest.py are now:
from tdda.referencetest.pytestconfig import (pytest_addoption, pytest_collection_modifyitems, set_default_data_location, ref)
conftest.py is also a good place to set the reference file location
if you want to do so using
We'll illustrate this with a simple example. The code below implements four trivial tests, two in a class and two as plain functions.
Note the import of the
tag decorator function near the top,
test_a and the class
are decorated with the
### test_all.py from tdda.referencetest import tag @tag def test_a(ref): assert 'a' == 'a' def test_b(ref): assert 'b' == 'b' @tag class TestClassA: def test_x(self): assert 'x' * 2 == 'x' + 'x' def test_y(self): assert 'y' > 'Y'
If we run this as normal, all four tests run and pass:
$ pytest ============================= test session starts ============================== platform darwin -- Python 3.5.1, pytest-3.2.1, py-1.4.34, pluggy-0.4.0 rootdir: /Users/njr/tmp/referencetest_examples/pytest, inifile: plugins: hypothesis-3.4.2 collected 4 items test_all.py .... =========================== 4 passed in 0.02 seconds ===========================
But if we add the
–tagged flag, only three tests run:
$ pytest --tagged ============================= test session starts ============================== platform darwin -- Python 3.5.1, pytest-3.2.1, py-1.4.34, pluggy-0.4.0 rootdir: /Users/njr/tmp/referencetest_examples/pytest, inifile: plugins: hypothesis-3.4.2 collected 4 items test_all.py ... =========================== 3 passed in 0.02 seconds ===========================
–-verbose flag confirms that these three are the tagged
test and the tests in the tagged class, as expected:
$ pytest --tagged --verbose ============================= test session starts ============================== platform darwin -- Python 3.5.1, pytest-3.2.1, py-1.4.34, pluggy-0.4.0 -- /usr/local/Cellar/python/3.5.1/bin/python3.5 cachedir: .cache rootdir: /Users/njr/tmp/referencetest_examples/pytest, inifile: plugins: hypothesis-3.4.2 collected 4 items test_all.py::test_a PASSED test_all.py::TestClassA::test_x PASSED test_all.py::TestClassA::test_y PASSED =========================== 3 passed in 0.01 seconds ===========================
Finally, if we want to find out which classes include tagged tests,
we can use the
pytest --istagged ============================= test session starts ============================== platform darwin -- Python 3.5.1, pytest-3.2.1, py-1.4.34, pluggy-0.4.0 rootdir: /Users/njr/tmp/referencetest_examples/pytest, inifile: plugins: hypothesis-3.4.2 collected 4 items test_all.test_a test_all.TestClassA ========================= no tests ran in 0.01 seconds =========================
This is particularly helpful when our tests are spread across multiple files, as the filenames are then shown as well as the class names.
Information about installing the library is available in this post.
Other features of the
ReferenceTest capabilities of the
are described in this post.
Its capabilities in the area of constraint discovery and verification
in this post,
and this post.