To see what a test error looks like, let's remove the failing doctest introduced in the previous section and introduce a different kind of mistake into our sample unit test. Let's assume that instead of wanting to test that 1 + 1 equals the literal 2, we want to test that it equals the result of a function, sum_args, that is supposed to return the sum of its arguments. But we're going to make a mistake and forget to import that function. So change self.failUnlessEqual to:
self.failUnlessEqual(1 + 1, sum_args(1, 1))
Now when the tests are run we see:
kmt@lbox:/dj_projects/marketr$ python manage.py test survey Creating test database... Creating table auth_permission Creating table auth_group Creating table auth_user Creating table auth_message Creating table django_content_type Creating table django_session Creating table django_site Creating table django_admin_log Installing index for auth.Permission model Installing index for auth.Message model Installing index for admin.LogEntry model E. ====================================================================== ERROR: test_basic_addition (survey.tests.SimpleTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/dj_projects/marketr/survey/tests.py", line 15, in test_basic_addition self.failUnlessEqual(1 + 1, sum_args(1, 1)) NameError: global name 'sum_args' is not defined ---------------------------------------------------------------------- Ran 2 tests in 0.041s FAILED (errors=1) Destroying test database...
The test runner encountered an exception before it even got to the point where it could compare 1 + 1 to the return value of sum_args, as sum_args was not imported. In this case, the error is in the test itself, but it would still have been reported as an error, not a failure, if the code in sum_args was what caused a problem. Failures mean actual results didn't match what was expected, whereas errors mean some other problem (exception) was encountered during the test run. Errors may imply a mistake in the test itself, but don't necessarily have to imply that.
Note that a similar error made in a doctest is reported as a failure, not an error. For example, we can change the doctest 1 + 1 line to:
>>> 1 + 1 == sum_args(1, 1)
If we then run the tests, the output will be:
kmt@lbox:/dj_projects/marketr$ python manage.py test survey Creating test database... Creating table auth_permission Creating table auth_group Creating table auth_user Creating table auth_message Creating table django_content_type Creating table django_session Creating table django_site Creating table django_admin_log Installing index for auth.Permission model Installing index for auth.Message model Installing index for admin.LogEntry model EF ====================================================================== ERROR: test_basic_addition (survey.tests.SimpleTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/dj_projects/marketr/survey/tests.py", line 15, in test_basic_addition self.failUnlessEqual(1 + 1, sum_args(1, 1)) NameError: global name 'sum_args' is not defined ====================================================================== FAIL: Doctest: survey.tests.__test__.doctest ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.5/site-packages/django/test/_doctest.py", line 2180, in runTest raise self.failureException(self.format_failure(new.getvalue())) AssertionError: Failed doctest test for survey.tests.__test__.doctest File "/dj_projects/marketr/survey/tests.py", line unknown line number, in doctest ---------------------------------------------------------------------- File "/dj_projects/marketr/survey/tests.py", line ?, in survey.tests.__test__.doctest Failed example: 1 + 1 == sum_args(1, 1) Exception raised: Traceback (most recent call last): File "/usr/lib/python2.5/site-packages/django/test/_doctest.py", line 1267, in __run compileflags, 1) in test.globs File "<doctest survey.tests.__test__.doctest[0]>", line 1, in <module> 1 + 1 == sum_args(1, 1) NameError: name 'sum_args' is not defined ---------------------------------------------------------------------- Ran 2 tests in 0.044s FAILED (failures=1, errors=1) Destroying test database...
Thus, the error versus failure distinction made for unit tests does not necessarily apply to doctests. So, if your tests include doctests, the summary of failure and error counts printed at the end doesn't necessarily reflect how many tests produced unexpected results (unit test failure count) or had some other error (unit test error count). However, in any case, neither failures nor errors are desired. The ultimate goal is to have zero for both, so if the difference between them is a bit fuzzy at times that's not such a big deal. It can be useful though, to understand under what circumstances one is reported instead of the other.
We have now seen how to run tests, and what the results look like for both overall success and a few failures and errors. Next we will examine the various command line options supported by the manage.py test command.