Are your tests any good? Check them with mutation testing
Short answer: Code coverage tells you which lines your tests run, not whether the tests would notice if those lines were wrong. Mutation testing checks that. A tool such as Stryker (for JavaScript, TypeScript and .NET) or PIT (for Java) makes small deliberate changes to your code, such as flipping a > to >=, and reruns your tests. If the tests fail, the change was caught. If they still pass, you've found a gap. Run it on your most important code, not the whole codebase, and use the surviving changes as a to-do list for better tests.
Why coverage can fool you
Many teams track a coverage percentage and feel safe when it's high. But coverage only measures execution. The PIT project's site puts it plainly: traditional coverage such as line, statement or branch measures only which code is executed by your tests, and doesn't check that your tests are actually able to detect faults in that code.
A test that calls a function and checks nothing earns full coverage. So does a test that asserts the wrong thing, or only checks that no error was thrown. In business apps, the riskiest logic, such as discounts, delivery fees, stock reservations and permission checks, is exactly where weak tests hide behind good coverage numbers.
How mutation testing works
The idea is simple. Stryker's documentation describes it this way: bugs, called mutants, are automatically inserted into your production code; your tests are run for each mutant; if the tests fail, the mutant is killed, and if they pass, the mutant survived. The higher the share of mutants killed, the more effective your tests are.
Typical mutations include:
- Changing comparison operators (
<to<=,==to!=). - Swapping arithmetic (
+to-). - Replacing a condition with
trueorfalse. - Removing a function call or a line.
- Changing a returned value, such as returning an empty string or zero.
Each mutant is a plausible bug. If your tests can't tell the difference between your code and a broken version of it, they wouldn't catch that bug in real life either.
Reading the results
After a run, you get a report listing mutants by file and line, with their status:
- Killed: a test failed. Good.
- Survived: all tests passed despite the change. A test is missing or too weak.
- No coverage: no test runs that line at all.
- Timeout: the change caused an infinite loop or a very slow run; tools usually count these as caught.
The useful part is not the overall score but the survivors. Each one is a specific, concrete question: "why didn't any test notice when the free-delivery threshold changed from 50 to 49?"
Equivalent mutants
Some mutants don't actually change behaviour, for example a change to code whose result is never used, or two conditions that always produce the same outcome. These "equivalent mutants" can't be killed by any test. Don't chase a perfect score; when a survivor genuinely changes nothing, mark it as ignored or accept it and move on.
Where to start
Mutation testing is slow, because it runs your tests many times. Make it practical:
- Pick one high-value area. Pricing, order totals, booking rules or permission checks are good candidates.
- Run it locally on that folder and read the report.
- Fix the first few survivors by adding or tightening assertions. Check that the improved tests now kill those mutants.
- Add it to CI for that folder, perhaps nightly rather than on every commit.
- Use incremental mode if your tool supports it, so later runs only test mutants in changed code.
Fast unit tests make mutation testing far quicker. Slow tests that hit databases or browsers multiply the run time, so focus on code covered by unit tests.
Writing tests that kill mutants
Survivors usually point to one of a few patterns:
- Missing boundary tests. Test exactly at thresholds: an order of exactly the free-delivery amount, a booking at exactly closing time.
- Assertions that are too loose. "Result is truthy" passes for many wrong answers; check the exact value.
- Unchecked side effects. If a function should send an email or write a log entry, assert that it did.
- Only happy paths. Add cases for invalid input, empty lists and zero quantities.
Explaining it to the rest of the business
Mutation results are easier to share than coverage numbers, because each survivor is a concrete story. "If someone accidentally changed the free-delivery rule, no test would have noticed" is a sentence an owner or manager understands immediately. Use a few of these examples when you ask for time to improve tests, and report back when those specific gaps are closed. It turns testing from an abstract percentage into protection for named parts of the business.
Setting goals sensibly
Some tools let you fail a build below a mutation score threshold. If you use one, set it per folder for the code you've worked on, and raise it slowly. A sudden company-wide target tends to produce tests written to please the tool rather than to protect customers.
A worked example
Take a hypothetical Adelaide wine subscription business whose shipping and discount logic had high line coverage. A developer ran Stryker on the pricing folder and found many surviving mutants, including one where changing the free-shipping comparison from "greater than or equal to" to "greater than" didn't fail any test. No test used an order exactly at the threshold. She added boundary tests for each pricing rule and tightened several assertions that only checked a total was positive. The team then scheduled a nightly mutation run for that folder.
FAQ
Does mutation testing replace coverage?
No. Coverage shows untested code; mutation testing shows weakly tested code. Use both.
How long does a run take?
It depends on the number of mutants and how fast your tests are. Start with a small folder to get a feel for it.
Which languages are supported?
Stryker supports JavaScript, TypeScript, C# and Scala, PIT covers Java and other JVM languages, and other tools exist for languages like Python and PHP.
Should every survivor be fixed?
No. Prioritise survivors in important logic, and accept equivalent mutants that change nothing.
If you'd like to know how much your tests really protect your app's money and booking logic, tell us about your test setup.