Automated testing is, in practice, the art of writing another piece of software to test an original piece of software.
As testing a whole piece of software has to take millions of variables and possible code paths into account, a single program trying to test another one would be very complex and hard to maintain. For this reason, it's usually convenient to split that program into smaller isolated programs, each being a test case.
Each test case contains all the instructions that are required to set up the target software in a state where the parts that are the test case areas of interest can be tested, the tests can be done, and all the conditions can be verified and reset back to the state of the target software so a subsequent test case can find a known state from which to start.
When using the unittest module that comes with the Python Standard Library, each test case is declared by subclassing from the unittest.TestCase class and adding a method whose name starts with test, which will contain the test itself:
import unittest
class MyTestCase(unittest.TestCase):
def test_one(self):
pass
Trying to run our previous test will do nothing by the way:
$ python 01_automatictests.py
$
We declared our test case, but we have nothing that runs it.
As for manually executed tests, the automatic tests need someone in charge of gathering all test cases and running them all. That's the role of a test runner.
Test runners usually involve a discovery phase (during which they detect all test cases) and a run phase (during which they run the discovered tests).
The unittest module provides all the components necessary to build a test runner that does both the discovery and execution of tests. For convenience, it even provides the unittest.main() method, which configures a test runner that, by default, will run the tests in the current module:
import unittest
class MyTestCase(unittest.TestCase):
def test_one(self):
pass
if __name__ == '__main__':
unittest.main()
By adding a call to unittest.main() at the end of our tests, Python will automatically execute our tests when the module is invoked:
$ python 01_automatictests.py
.
----------------------------------------------------------------------
Ran 1 test in 0.000s
OK
We can confirm that the test we cared about was executed by using the -v option to print a more verbose output:
$ python 01_automatictests.py -v
test_one (__main__.MyTestCase) ... ok
----------------------------------------------------------------------
Ran 1 test in 0.000s
OK
During the discovery phase, unittest.main will look for all classes that inherit from unittest.TestCase within the module that is recognized as the main Python module (sys.modules['__main__']), and all those subclasses will be registered as test cases for the runner.
Individual tests are then defined by having methods with names starting with test in the test case classes. This means that if we add more methods with names that don't start with test, they won't be treated as tests:
class MyTestCase(unittest.TestCase):
def test_one(self):
pass
def notatest(self):
pass
Trying to start the test runner again will continue to run only the test_one test:
$ python 01_automatictests.py -v
test_one (__main__.MyTestCase) ... ok
----------------------------------------------------------------------
Ran 1 test in 0.000s
OK
In the previous example, only the test_one method was executed as a test, while notatest was recognized as not being a test but instead as a method that we are going to use ourselves in tests.
Being able to distinguish between tests (methods whose names start with test_) and other methods allows us to create helpers and utility methods within our test cases that the individual tests can reuse.
Given that a test suite is a collection of multiple test cases, to grow our test suite, we need to be able to actually write more than one single TestCase subclass and run its tests.
Multiple test cases
We already know that unittest.main is the function in charge of executing our test suite, but how can we make it execute more than one TestCase?
The discovery phase of unittest.main (the phase during which unittest.main decides which tests to run) looks for all subclasses or unittest.TestCase.
The same way we had MyTestCase tests executed, adding more test cases is as simple as declaring more classes:
import unittest
class MyTestCase(unittest.TestCase):
def test_one(self):
pass
def notatest(self):
pass
class MySecondTestCase(unittest.TestCase):
def test_two(self):
pass
if __name__ == '__main__':
unittest.main()
Running the 01_automatictests.py module again will lead to both test cases being verified:
$ python 01_automatictests.py -v
test_two (__main__.MySecondTestCase) ... ok
test_one (__main__.MyTestCase) ... ok
----------------------------------------------------------------------
Ran 2 tests in 0.000s
OK
If a test case is particularly complex, it can even be divided into multiple individual tests, each checking a specific subpart of it:
class MySecondTestCase(unittest.TestCase):
def test_two(self):
pass
def test_two_part2(self):
pass
This allows us to divide the test cases into smaller pieces and eventually share setup and teardown code between the individual tests. The individual tests will be executed by the test runner in alphabetical order, so in this case, test_two will be executed before test_two_part2:
$ python 01_automatictests.py -v
test_two (__main__.MySecondTestCase) ... ok
test_two_part2 (__main__.MySecondTestCase) ... ok
test_one (__main__.MyTestCase) ... ok
In that run of the tests, we can see that MySecondTestCase was actually executed before MyTestCase because "MyS" is less than "MyT".
In any case, generally, it's a good idea to consider your tests as being executed in a random order and to not rely on any specific sequence of execution, because other developers might add more test cases, add more individual tests to a case, or rename classes, and you want to allow those changes with no additional issues. Especially since relying on a specific known execution order of your tests might limit your ability to parallelize your test suite and run test cases concurrently, which will be required as the size of your test suite grows.
Once more tests are added, adding them all into the same class or file quickly gets confusing, so it's usually a good idea to start organizing tests.
Organizing tests
If you have more than a few tests, it's generally a good idea to group your test cases into multiple modules and create a tests directory where you can gather the whole test plan:
├── 02_tests
│ ├── tests_div.py
│ └── tests_sum.py
Those tests can be executed through the unittest discover mode, which will look for all modules with names matching test*.py within a target directory and will run all the contained test cases:
$ python -m unittest discover 02_tests -v
test_div0 (tests_div.TestDiv) ... ok
test_div1 (tests_div.TestDiv) ... ok
test_sum0 (tests_sum.TestSum) ... ok
test_sum1 (tests_sum.TestSum) ... ok
----------------------------------------------------------------------
Ran 4 tests in 0.000s
OK
You can even pick which tests to run by filtering them with a substring with the -k parameter; for example, -k sum will only run tests that contain "sum" in their names:
$ python -m unittest discover 02_tests -k sum -v
test_sum0 (tests_sum.TestSum) ... ok
test_sum1 (tests_sum.TestSum) ... ok
----------------------------------------------------------------------
Ran 2 tests in 0.000s
OK
And yes, you can nest tests further as long as you use Python packages:
├── 02_tests
│ ├── tests_div
│ │ ├── __init__.py
│ │ └── tests_div.py
│ └── tests_sum.py
Running tests structured like the previous directory tree will properly navigate into the subfolders and spot the nested tests.
So running unittest in discovery mode over that direction will properly find the TestDiv and TestSum classes declared inside the files even when they are nested in subdirectories:
$ python -m unittest discover 02_tests -v
test_div0 (tests_div.tests_div.TestDiv) ... ok
test_div1 (tests_div.tests_div.TestDiv) ... ok
test_sum0 (tests_sum.TestSum) ... ok
test_sum1 (tests_sum.TestSum) ... ok
----------------------------------------------------------------------
Ran 4 tests in 0.000s
OK
Now that we know how to write tests, run them, and organize multiple tests in a test suite. We can start introducing the concept of TDD and how unit tests allow us to achieve it.