Understanding test metrics

Now that we understand how to deliver projects with tests first, it’s time to look at some metrics that can quantify how well-tested a project is. It’s important to deliver tests across the entire test pyramid, as it’s important to be able to ensure the application is working correctly end-to-end as well as working well with its external dependencies.

Important test metrics

There is a wide range of metrics that we can measure when quantifying the quality of software:

• Requirement coverage: This indicates the percentage of your project requirements that are covered by tests. A test could cover multiple requirements, but no customer requirement should be left untested.
• Defect count and distribution: This indicates how many defects or bugs are discovered in each module/part of the application. The distribution will also signal whether there are any particular problem areas in the system that could be refactored.
• Defect resolution time: This indicates how quickly the development team is able to fix bugs once they are detected. A long Mean Time To Resolution (MTTR) can indicate that the development team is short-staffed, while a long max resolution time in a particular area of the system can indicate that the code in that particular part is difficult to change.
• Code coverage: This indicates the percentage of your code base that is exercised by unit tests. Since tests should be written first, coverage also shows whether the development team is using TDD. Low test coverage can also indicate issues with the system design.
• Burndown rates and charts: These indicate the rate at which the team is able to deliver functionality. As development and testing are a unified task, a user story or requirement cannot be considered complete unless it is tested, so the burndown rate will include only stories that are ready for delivery. Burndown charts can indicate delays in project timelines.

Code coverage

Since the code coverage metric is such an important TDD indicator, let’s explore it further. In order to achieve a high coverage percentage, tests should cover the following:

• The functions you implemented
• The statements that your functions are composed of
• The different execution paths of your functions
• The different conditions of your Boolean variables
• The different parameter values that can be passed to your functions

The Go test runner provides the coverage percentage for Go applications. We will have a look at how to do this in Chapter 2, Unit Testing Essentials.

Figure 1.11 shows a flow chart of the implementation of the Divide(x,y) function from the simple terminal calculator:

Figure 1.11 – Execution flow of the Divide function in the simple calculator

Tests should be written to cover and verify the following:

• The execution path for y != 0
• The execution path for y == 0
• The error message of the DivideZero error
• The output from the result calculation statements
• The output from the print result statements

Code coverage percentage

In large projects, it will be unfeasible to reach 100% test coverage for the code base. There have been many discussions in the tech community about what a good test coverage percentage is. It is generally accepted that a good coverage amount is around the 80% mark. After that point, experience shows there can be diminishing returns.

The code coverage percentage will also depend on the kind of project you are running. A legacy code base with a low code coverage percentage will require considerable effort to bring up to the 80% mark. Similarly, a greenfield project will also be difficult to test if there are many unknowns.

Just like any code you write, test code needs to be maintained and refactored. Keeping a high coverage percentage requires maintaining and updating a lot of test code. This can increase the development cost of refactoring or other code changes, as it potentially requires updating many test cases. The business value of maintaining tests that do not cover requirements is very low, so you should ensure that your tests are providing value to your test suites.

Well-tested code is not necessarily bug-free code

Your tests should aim to provide verification for important code behavior, as opposed to simply writing code to get a certain code coverage percentage. The team should embrace a testing culture using TDD and a good coverage percentage will follow.