Testing is an important and intrinsic component of software. Tests are required to ensure quality and maintainability of code. Consider the code below and the tests associated with it.
The code is straightforward and the tests also cover 100% of the lines of the code. Now let us make a small code change which alters the functionality. If we run the tests they still pass and the code coverage is still 100%. Below is the changed code:
In the above code example, the small code change impacted the functionality of the code around a boundary condition. Does 100% code coverage equate to 100% functionality coverage? The answer is No.
The metric of code coverage is not enough, we need another metric to evaluate the comprehensiveness of tests. Essentially we need tests to test the tests. The most rudimentary way to determine if all functionality is covered is through code reviews. Code reviews should involve test reviews as well. However, code reviews are manual, and pose a scalability, reliability, and repeatability problem. So, we need to automate the process to evaluate the comprehensiveness of tests.
Let us consider the main purpose of tests, tests are used to check the functionality of code. If functionality of code changes then at least 1 test should fail. Therein lies the fundamental principle of Mutation Testing. Mutation Testing involves mutating the functionality of code and running all tests. If there is at least 1 failing test then we can conclude that the code is covered by mutation testing.
PIT Mutation Testing library is a testing framework which performs mutation testing and generates reports. For the code with original functionality, below is the mutation test coverage using PIT.
As seen above, lines 6 and 9 indicate missing mutation test coverage. The mutations are explained: when the conditional boundary was changed i.e. less than (<) was changed to less than or equal to (<=) and similarly greater than or equal to (>=) was changed to greater than (>) no tests failed.
Let us now add the missing test for original functionality and see the mutation test coverage.
As seen above, all the lines of code have 100% mutation test coverage.
There are many mutators available in PIT. PIT is configurable to either use specific mutators or mutator groups for evaluation. PIT also generates code coverage report. In addition to this PIT comes with support to run with commonly used build systems (Maven, Gradle, Ant, command line). This makes it possible to integrate it in a CI/CD pipeline. There is also a Mutation Testing Kata available which you can try on your own to learn more about the framework. There are additional resources available at the end of the blog for Mutation Testing.
Conclusion
This blog covered the importance of Mutation Testing. A simple code example with tests was considered to demonstrate 100% code coverage does not mean 100% comprehensive tests. There can be missing tests. Mutation testing frameworks like PIT can help automate the process of detecting missing tests. Do the Kata for self-study to learn the concept and framework.
Resources
steinhauer.software
Pedro Rijo December 22, 2019
I wrote a little bit more in detail regarding mutation test and PIT in specific, in case anyone is interested in reading any more resources on the subject: https://pedrorijo.com/blog/intro-mutation/