What is Mutation Testing?
Mutation testing is a software testing technique used to evaluate the effectiveness of a test suite. It involves introducing artificial defects, known as mutations, into a program’s source code and then running the existing test suite to determine if the defects are detected.
The primary goal of mutation testing is to assess the test suite's quality by measuring its ability to identify these mutations.
How is Mutation Testing different from Code Coverage Metrics?
Mutation testing provides a more rigorous assessment of the quality of a test suite compared to code coverage metrics like JaCoCo. While JaCoCo measures the extent to which the tests execute the code, mutation testing checks the ability of the tests to detect artificial defects.
By introducing mutations, it evaluates the effectiveness of the test suite in identifying these intentional code changes, providing a higher level of confidence in the test suite’s ability to catch actual defects.
How does it work?
What happens behind the scene is -
Mutants (defects) are introduced to our program (API in our case), and both the defect code and original code are run against the test cases.
The expectation is — If our test case has covered all the scenarios well, then a defective program (mutant) that’s run against those test cases should fail.
2. pitest introduces a set of default mutants to our program before comparing it against the unit tests.
A few examples are:
CONDITIONALS_BOUNDARY
EMPTY_RETURNS
FALSE_RETURNS
INCREMENTS
INVERT_NEGS
MATH
NEGATE_CONDITIONALS
NULL_RETURNS
PRIMITIVE_RETURNS
TRUE_RETURNS
VOID_METHOD_CALLS
Suppose we write a test case to test our API -
public void testGreetingsResponse(){
assertEquals("Hello", helloController.getGreetings().getBody());
}
A NULL_RETURNS mutant will change our API to this -
@GetMapping
public ResponseEntity<String> getGreetings(){
return null;
}
In this case, our test case “testGreetingsResponse()” will throw an error as getBody() will be invoked on a null object.
pitest will just check that the test case failed and that respective mutant will be killed.
An exception thrown from a test during pitest is considered as test failed.
Also note that a mutant needs to be killed in any one of the test cases, not in every single one.
Example
Check out this article for a practical example and complete code to a sample project implemented in spring boot and pitest.