Reference Testing Exercise 1 (pytest flavour)
Posted on Tue 29 October 2019 in TDDA
This exercise (video 8m 54s) shows how to migrate a test from using pytest
directly to the exploiting the referencetest
capabilities in
the TDDA library.
(If you do not currently use pytest
for writing
tests, you might prefer the
unittest-flavoured version
of this exercise, since unittest
is in Python's standard library.)
We will see how even simple use of referencetest
- makes it much easier to see how tests have failed when complex outputs are generated
- helps us to update reference outputs (the expected values) when we have verified that a new behaviour is correct
- allows us easily to write tests of code whose outputs are not identical from run to run. We do this by specifying exclusions from the comparisons used in assertions.
Prerequisites
★ You need to have the TDDA Python library installed (version 1.0.31 or newer) see installation. Use
tdda version
to check the version that you have.
Step 1: Copy the exercises
You need to change to some directory in which you're happy to create three
new directories with data. We are use ~/tmp
for this. Then copy the
example code.
$ cd ~/tmp
$ tdda examples # copy the example code
Step 2: Go the exercise files and examine them:
$ cd referencetest_examples/exercises-pytest/exercise1 # Go to exercise1
You should have at least the following four files:
$ ls
conftest.py expected.html generators.py test_all.py
-
generators.py
contains a function calledgenerate_string
that, when called, returns HTML text suitable for viewing as a web page. -
expected.html
is the result of calling that function, saved to file -
test_all.py
contains a singleunittest
-based test of that file. -
conftest.py
imports keyreferencetest
functionality from thetdda
library intopytest
.
It's probably useful to look at the web page expected.html
in a browser,
either by navigating to it in a file browser and double clicking it,
or by using
open expected.html
if your OS supports this. As you can see, it's just some text and an image. The image is an inline SVG vector image, generated along with the text.
Also have a look at the test code. The core part of it is very short:
from generators import generate_string
def testExampleStringGeneration():
actual = generate_string()
with open('expected.html') as f:
expected = f.read()
assert actual == expected
The code
- calls
generate_string()
to create the content - stores its output in the variable
actual
- reads the expected content into the variable
expected
- asserts that the two strings are the same.
Step 3. Run the test, which should fail
$ pytest # This will whether pytest uses Python2 or Python3
You should get a failure, and pytest tries quite hard to show what's causing the failure:
=================================== FAILURES ===================================
_________________________ testExampleStringGeneration __________________________
def testExampleStringGeneration():
actual = generate_string()
with open('expected.html') as f:
expected = f.read()
> assert actual == expected
E AssertionError: assert '<!DOCTYPE ht...y>\n</html>\n' == '<!DOCTYPE htm...y>\n</html>\n'
E Skipping 69 identical leading characters in diff, use -v to show
E - Solutions, 2016
E + Solutions Limited, 2016
E ? ++++++++
E - Version 1.0.0
E ? ^
E + Version 0.0.0...
E
E ...Full output truncated (31 lines hidden), use '-vv' to show
test_all.py:24: AssertionError
=========================== 1 failed in 0.11 seconds ===========================
You can certainly see that there's a different in the Version number in
the output and also a line including 2016
(a copyright notice, in fact).
But it also says:
...Full output truncated (31 lines hidden), use '-vv' to show
and if you do that, the output becomes a bit overwhelming.
We'll convert the test to use the TDDA libraries referencetest and see how that helps.
Step 4. Change the code to use referencetest.
The key change we need to make is the to the assertion, which will now be:
ref.assertStringCorrect(actual, 'expected.html')
ref
is object made available by conftest.py
, and is passed into our test
function by pytest
. We therefore need to change the function declaration to
take ref
as an argument:
def testExampleStringGeneration(ref):
Finally, because assertStringCorrect
compares a string in memory
to content from a file, we don't need the lines in the middle that
read the file:
* Delete the middle two lines of the test function.
The result is:
from generators import generate_string
def testExampleStringGeneration(ref):
actual = generate_string()
ref.assertStringCorrect(actual, 'expected.html')
Step 5. Run the modified test
$ pytest
You should see very different output, that includes, near the end, something like this:
E AssertionError: 2 lines are different, starting at line 5
E Expected file expected.html
E Compare raw with:
E diff /var/folders/zv/3xvhmvpj0216687_pk__2f5h0000gn/T/actual-raw-expected.html expected.html
E
E Compare post-processed with:
E diff /var/folders/zv/3xvhmvpj0216687_pk__2f5h0000gn/T/actual-expected.html /var/folders/zv/3xvhmvpj0216687_pk__2f5h0000gn/T/expected-expected.html
/Users/njr/python/tdda/tdda/referencetest/referencepytest.py:187: AssertionError
(You will probably need to scroll right to see all of the message on this page.)
Because the test failed, the TDDA library has written a copy of the
actual ouput to file to make it easy for us to examine it and to use diff
commands to see how it actually differs from what we expected. (In fact,
it's written out two copies, a "raw" and a "post-precocessed" one, but we
haven't used any processing, so they will be the same in our case. So
we ignore the second diff command suggested for now.)
It's also given us the precise diff
command we need to see the differences
between our actual and expected output.
Step 6. Copy the first diff
command and run it. You should see something similar to this:
$ diff /var/folders/zv/3xvhmvpj0216687_pk__2f5h0000gn/T/actual-raw-expected.html expected.html
5,6c5,6
< Copyright (c) Stochastic Solutions, 2016
< Version 1.0.0
—
> Copyright (c) Stochastic Solutions Limited, 2016
> Version 0.0.0
35c35
< </html>
\ No newline at end of file
—
> </html>
(If you have a visual diff tool, can also use that. For example,
on a Mac, if you have Xcode installed, you should have the
opendiff
command available.)
The diff makes it clear that there are three differences:
- The copyright notice has changed slightly
- The version number has changed
- The string doesn't have a newline at the end, whereas the file does.
The Copyright and version numbers lines are both in comments in the HTML,
so don't affect the rendering at all. You might want to confirm that if
you look at the actual file it saved (/var/folders/zv/3xvhmvpj0216687_pk__2f5h0000gn/T/actual-raw-expected.html
, the first file in the diff command),
you should see that it looks identical.
In this case, therefore, we might now feel that we should simply
update expected.html
with what generate_string()
is now
producing. It would be (by design) extremely easy to change the diff
in the command it gave is to cp
to achieve that.
However, there's better thing we can do in this case.
Step 7. Specify exclusions
Standing back, it seems obvious likely that periodically the version number and Copyright line written to comments in the HTML will change. If the only difference between out expected output and what we actually generate are those, we'd probably prefer the test didn't fail.
The ref.assertStringCorrect
function from referencetest
gives us
several mechanisms for specifying changes that can be ignored when
checking whether a string is correct. The simplest one, which will be
enough for our example, is just to specify strings which, if they
occur on a line in the output, case differences in those lines to be
ignored, so that the assertion doesn't fail.
** Step 7a. Add the ignore_substrings
parameter to assertStringCorrect
as follows:**
ref.assertStringCorrect(actual, 'expected.html',
ignore_substrings=['Copyright', 'Version'])
Step 7b. Run the test again. It should now pass:
$ pytest
============================= test session starts ==============================
test_all.py . [100%]
=========================== 1 passed in 0.04 seconds ===========================
Recap: What we have seen
We've seen
-
Converting standard
pytest
-based tests to usereferencetestcase
is straightfoward. -
When we do that, we gain access to powerful new kinds of assertion such as
assertStringCorrect
. Among the immediate benefits:- When there is failure, this saves the failing output to a temporary file
- It tells you the exact
diff
command you need to see be able to see differences - This also makes it very easy to copy the new "known good" answer into place if you've verified that the new answer is now correct. (In fact, the library also has a more powerful way to do this, as we'll see in a later exercise).
-
The
ref.assertStringCorrect
fucntion also has a number of mechanisms for allowing specific expected differences to occur without causing the test to fail. The simplest of these mechanisms is theignore_substrings
keyword argument we used here.