Mutation Testing
Who tests the tests? Deliberately breaking the code to see whether anything fails, reading a surviving mutant, and the cost that keeps this off every pipeline.
Who tests the tests? Deliberately breaking the code to see whether anything fails, reading a surviving mutant, and the cost that keeps this off every pipeline.
Coverage tells you which lines ran. It cannot tell you whether anything checked what they did, which is why a suite can sit at 92% and be largely decorative.
Mutation testing answers the question coverage cannot: if the code were wrong, would any test notice? It does this by deliberately breaking the code and re-running the suite. By the end of this lesson you will know how to read a mutation report, why the score is expensive to compute, and how to get most of the value without running it on everything.
The mechanism is mechanical and slightly startling.
Change one line of your code
Flip > to >=, swap + for -, replace a return value
with None, invert a condition, delete a statement. That is
a mutant.
Run the tests
The whole suite, or the subset that covers the mutated line.
A test fails — the mutant is killed
Your suite noticed. That is the outcome you want, and it is proof that something in there is genuinely checking.
Every test passes — it survived
That change to your code is invisible to your entire suite. Somebody could make it by accident and nothing would report it.
Unlike coverage, this cannot be satisfied by executing code: a test with no assertion kills nothing.
def discount_percent(total, is_member):
if total >= 100:
return 20
if is_member and total >= 20:
return 10
return 0Mutants a tool would generate from those five lines:
total >= 100 -> total > 100 boundary
total >= 100 -> total <= 100 condition flip
return 20 -> return 21 value
return 20 -> return None value
and -> or logic
total >= 20 -> total > 20 boundary
return 10 -> return 0 value
return 0 -> return 10 valueA surviving total >= 100 → total > 100 means nothing tests a total of
exactly 100 — which is precisely the boundary the foundations course said
matters, discovered automatically rather than by discipline.
# Python
pip install mutmut
mutmut run --paths-to-mutate src/discounts.py
mutmut results
mutmut show 7 # the diff of one surviving mutant# JavaScript / TypeScript
npx stryker runSurvived mutants:
--- src/discounts.py
+++ src/discounts.py
@@ -2,5 +2,5 @@
def discount_percent(total, is_member):
- if total >= 100:
+ if total > 100:
return 20
--- src/discounts.py
+++ src/discounts.py
@@ -4,5 +4,5 @@
- if is_member and total >= 20:
+ if is_member or total >= 20:
return 10Two survivors, and each is a specific, actionable gap:
The first says no test uses a total of exactly 100. Add one.
The second is more interesting: and became or and nothing failed. That
means no test has a non-member with a total between 20 and 100 — so the
member condition is doing no work in any test. That is a whole business rule
with no coverage, and no coverage tool would have said so.
Each survivor is one of four things, and telling them apart is the skill.
A missing test. The most common and the point of the exercise. Add the case; the mutant dies.
An equivalent mutant. The change does not alter behaviour, so no test
can kill it. x * 1 versus x, or a mutation in unreachable code. False
positives, and unavoidable — good tools have fewer.
A weak assertion. The line is executed and the test asserts something too
loose to notice. assert result is not None where assert result == 10 was
meant.
Dead or pointless code. Nothing tests it because nothing needs it. The correct response is deletion, and mutation testing is unusually good at finding this.
survivor usually means
boundary >= became > no test on the exact boundary
condition inverted, tests pass that branch has no test at all
and became or, tests pass one side of the condition is never
exercised alone
return value changed, pass an assertion is too weak, or absent
statement deleted, tests pass the statement may be unnecessaryThat last row is worth acting on. A deleted line that no test misses is either untested or unneeded, and both are worth knowing.
Mutation testing runs your test suite once per mutant. A file with 200 mutants and a 30-second suite is 100 minutes — and that is one file.
This is why the technique is not on every pipeline, and there are four practical ways to make it affordable.
Run it on the top of the risk model only. The payment calculation, the authorisation check, the invoice arithmetic. Not the whole codebase.
Run it on the diff. Mutate only the lines a pull request changed. Cheap, targeted, and immediately relevant.
mutmut run --paths-to-mutate src/pricing.py
npx stryker run --since=mainRun it nightly or weekly, not per commit. A scheduled job whose report is read on a Monday.
Give it only the relevant tests. Some tools map mutants to the tests that cover that line and run only those, which is a large saving.
Bad — a test written to kill a mutant, coupled to how the code works:
def test_discount_internals(mocker):
spy = mocker.spy(discounts, "_tier_for")
discount_percent(100, False)
spy.assert_called_once_with(100)
assert discounts._LAST_TIER == "high"Good — a test written for the gap the mutant revealed:
def test_twenty_percent_applies_at_exactly_one_hundred():
assert discount_percent(100, False) == 20
def test_a_non_member_gets_no_discount_between_twenty_and_ninety_nine():
assert discount_percent(45, False) == 0Both kill mutants. The first does it by asserting on a private helper and a module-level variable, so it fails on any refactor and tells a reader nothing about the business rule. It has raised the score and made the suite worse — the exact outcome a score target produces.
The second pair states two real rules. Each name is a specification, each
survives a rewrite, and between them they kill the boundary mutant and the
and/or mutant that revealed the untested member condition.
The discipline is: read the survivor, work out what behaviour is untested, and test that behaviour. Do not aim at the mutant.
Four uses that justify the runtime.
Auditing a critical module once. Point it at the payment calculation, spend an afternoon on the survivors, and you will find real gaps. Then do not run it again for six months.
Validating a suite you inherited. A codebase with 85% coverage and no history of catching regressions — mutation testing tells you whether the suite is real. This is the single most informative use.
Checking a test suite written to a coverage target. If a team was measured on coverage, mutation testing reveals what that produced, usually starkly.
Teaching. Watching a mutant survive is the most convincing possible demonstration that a passing test can assert nothing, and it changes how people write tests afterwards more effectively than any explanation.
And one thing to be honest about: on a well-tested module with careful assertions, mutation testing often finds nothing but equivalent mutants. That is a good result, and it is also a reason to point it at areas you are unsure about rather than areas you trust.
# The mechanism
# 1. change one line of code (a MUTANT)
# 2. run the tests
# 3. a test fails -> KILLED. The suite noticed.
# 4. all pass -> SURVIVED. That break is invisible to your
# entire suite.
# mutation score = proportion killed
# unlike coverage, it CANNOT be satisfied by executing code
# Tools
mutmut run --paths-to-mutate src/pricing.py
mutmut results ; mutmut show 7
npx stryker run --since=main
# also: cosmic-ray, PIT (Java), cargo-mutants (Rust)
# Reading a survivor — four possibilities
a missing test add the case; the mutant dies
an equivalent mutant behaviour unchanged; unkillable. False positive.
a weak assertion "is not None" where "== 10" was meant
dead code nothing needs it. Delete it.
# What each survivor usually means
>= became > no test on the exact boundary
condition inverted, passes that branch has no test at all
and became or, passes one side is never exercised alone
return value changed, passes an assertion is absent or too loose
statement deleted, passes the statement may be unnecessary
# The cost: the whole suite runs once PER MUTANT
# 200 mutants x a 30-second suite = 100 minutes, for one file
affordable versions
the top of the risk model only — not the codebase
the DIFF only: --since=main
nightly or weekly, not per commit
only the tests that cover the mutated line
# Never set a mutation score target
# it is satisfiable by asserting on internals, which kills mutants
# and makes the suite brittle
# the survivor LIST is the artefact, not the percentage
# The discipline
# read the survivor -> what BEHAVIOUR is untested -> test that
# do not aim at the mutant
# Best uses
audit one critical module, once
validate a suite you inherited <- most informative
check what a coverage target produced
teaching: nothing else demonstrates an assertion-free test so wellProperty-based testing generates inputs against a rule; mutation testing generates faults against your tests. The next lesson generates inputs against robustness: fuzzing, which feeds a program input nobody would ever write and is the standard technique for finding crashes and security defects in parsers and input handlers.
Before that, run a mutation tool on one small, important module — a pricing function, a validator, a permission check. An afternoon on the survivor list of the module you care most about is the highest-value way to spend this lesson.