advanced30 minby DevFox

Mutation Testing

Who tests the tests? Deliberately breaking the code to see whether anything fails, reading a surviving mutant, and the cost that keeps this off every pipeline.

Testing

Coverage tells you which lines ran. It cannot tell you whether anything checked what they did, which is why a suite can sit at 92% and be largely decorative.

Mutation testing answers the question coverage cannot: if the code were wrong, would any test notice? It does this by deliberately breaking the code and re-running the suite. By the end of this lesson you will know how to read a mutation report, why the score is expensive to compute, and how to get most of the value without running it on everything.

Break the code, see who complains

The mechanism is mechanical and slightly startling.

Change one line of your code
Flip > to >=, swap + for -, replace a return value with None, invert a condition, delete a statement. That is a mutant.
Run the tests
The whole suite, or the subset that covers the mutated line.
A test fails — the mutant is killed
Your suite noticed. That is the outcome you want, and it is proof that something in there is genuinely checking.
Every test passes — it survived
That change to your code is invisible to your entire suite. Somebody could make it by accident and nothing would report it.

Repeat for hundreds of mutants. The proportion killed is the mutation score.

Unlike coverage, this cannot be satisfied by executing code: a test with no assertion kills nothing.

python

def discount_percent(total, is_member):
    if total >= 100:
        return 20
    if is_member and total >= 20:
        return 10
    return 0

Mutants a tool would generate from those five lines:

text

total >= 100   ->  total > 100      boundary
total >= 100   ->  total <= 100     condition flip
return 20      ->  return 21        value
return 20      ->  return None      value
and            ->  or               logic
total >= 20    ->  total > 20       boundary
return 10      ->  return 0         value
return 0       ->  return 10        value

A surviving total >= 100 → total > 100 means nothing tests a total of exactly 100 — which is precisely the boundary the foundations course said matters, discovered automatically rather than by discipline.

Running it

Two survivors, and each is a specific, actionable gap:

The first says no test uses a total of exactly 100. Add one.

The second is more interesting: and became or and nothing failed. That means no test has a non-member with a total between 20 and 100 — so the member condition is doing no work in any test. That is a whole business rule with no coverage, and no coverage tool would have said so.

Reading survivors

Each survivor is one of four things, and telling them apart is the skill.

A missing test. The most common and the point of the exercise. Add the case; the mutant dies.

An equivalent mutant. The change does not alter behaviour, so no test can kill it. x * 1 versus x, or a mutation in unreachable code. False positives, and unavoidable — good tools have fewer.

A weak assertion. The line is executed and the test asserts something too loose to notice. assert result is not None where assert result == 10 was meant.

Dead or pointless code. Nothing tests it because nothing needs it. The correct response is deletion, and mutation testing is unusually good at finding this.

That last row is worth acting on. A deleted line that no test misses is either untested or unneeded, and both are worth knowing.

The cost, and how to live with it

Mutation testing runs your test suite once per mutant. A file with 200 mutants and a 30-second suite is 100 minutes — and that is one file.

This is why the technique is not on every pipeline, and there are four practical ways to make it affordable.

Run it on the top of the risk model only. The payment calculation, the authorisation check, the invoice arithmetic. Not the whole codebase.

Run it on the diff. Mutate only the lines a pull request changed. Cheap, targeted, and immediately relevant.

Run it nightly or weekly, not per commit. A scheduled job whose report is read on a Monday.

Give it only the relevant tests. Some tools map mutants to the tests that cover that line and run only those, which is a large saving.

Chasing the number

Bad — a test written to kill a mutant, coupled to how the code works:

Good — a test written for the gap the mutant revealed:

Both kill mutants. The first does it by asserting on a private helper and a module-level variable, so it fails on any refactor and tells a reader nothing about the business rule. It has raised the score and made the suite worse — the exact outcome a score target produces.

The second pair states two real rules. Each name is a specification, each survives a rewrite, and between them they kill the boundary mutant and the and/or mutant that revealed the untested member condition.

The discipline is: read the survivor, work out what behaviour is untested, and test that behaviour. Do not aim at the mutant.

What it is genuinely good for

Four uses that justify the runtime.

Auditing a critical module once. Point it at the payment calculation, spend an afternoon on the survivors, and you will find real gaps. Then do not run it again for six months.

Validating a suite you inherited. A codebase with 85% coverage and no history of catching regressions — mutation testing tells you whether the suite is real. This is the single most informative use.

Checking a test suite written to a coverage target. If a team was measured on coverage, mutation testing reveals what that produced, usually starkly.

Teaching. Watching a mutant survive is the most convincing possible demonstration that a passing test can assert nothing, and it changes how people write tests afterwards more effectively than any explanation.

And one thing to be honest about: on a well-tested module with careful assertions, mutation testing often finds nothing but equivalent mutants. That is a good result, and it is also a reason to point it at areas you are unsure about rather than areas you trust.

Cheat sheet

Where to go next

Property-based testing generates inputs against a rule; mutation testing generates faults against your tests. The next lesson generates inputs against robustness: fuzzing, which feeds a program input nobody would ever write and is the standard technique for finding crashes and security defects in parsers and input handlers.

Before that, run a mutation tool on one small, important module — a pricing function, a validator, a permission check. An afternoon on the survivor list of the module you care most about is the highest-value way to spend this lesson.