intermediate25 minby DevFox

What Coverage Does Not Tell You

Line, branch and path coverage, what each measures, and why a 90% number can sit on top of tests that assert nothing. Using coverage as a question, not a target.

Testing

Coverage is the number everyone asks for and almost nobody interprets correctly. It is genuinely useful, and it measures something much narrower than what people take it to mean — which is why a codebase can sit at 92% and be full of untested behaviour.

By the end of this lesson you will know what each kind of coverage actually counts, how to use the report rather than the percentage, and what happens to a team that makes the percentage a target.

What it measures

Coverage tools record which parts of the code were executed while the tests ran. That is the whole mechanism.

Note the word. Not "tested" — executed. A line counts as covered if it ran, whether or not anything checked what it did.

There are several kinds, and the difference between them matters.

Line coverage — which lines ran. The most commonly quoted and the weakest.

Branch coverage — which decisions went both ways. An if with no else needs two tests to be branch-covered and one to be line-covered.

Condition coverage — within a compound condition, whether each part was evaluated both ways. if a and b has four combinations.

Path coverage — which routes through the function were taken. Complete path coverage is usually infeasible: a function with ten sequential two-way branches has 1,024 paths.

python

def discount_percent(total, is_member):
    if total >= 100:              # line 2
        return 20                 # line 3
    if is_member and total >= 20: # line 4
        return 10                 # line 5
    return 0                      # line 6

One test — discount_percent(150, False) — executes lines 2 and 3. That is 40% line coverage. Add discount_percent(0, False) and lines 4 and 6 run too: 100% line coverage, and line 5 never returned 10, and the is_member condition has only ever been False.

That is the shape of the problem. Full line coverage, and the member discount is completely untested.

Running it

bash

# Python
pytest --cov=src --cov-branch --cov-report=term-missing
pytest --cov=src --cov-branch --cov-report=html   # then open htmlcov/

# JavaScript / TypeScript
vitest run --coverage
npx jest --coverage

text

Name                     Stmts   Miss Branch BrPart  Cover   Missing
----------------------------------------------------------------------
src/discounts.py            12      0      6      2    89%   4->5
src/importer.py             84     31     22      6    58%   45-59, 71-88
src/notifications.py        26     26      4      0     0%   1-26
----------------------------------------------------------------------
TOTAL                      122     57     32      8    62%

The Missing column is the part to read. 45-59 in importer.py and 0% on notifications.py are facts you can act on; 62% is not.

The coverage that proves nothing

Bad — 100% coverage of a function, and it asserts nothing about it:

python

def test_import_runs():
    import_contacts(csv_text="name,email\nAda,ada@example.com\n")

Good — the same execution, with a claim attached:

python

def test_import_saves_a_valid_row(repository):
    result = import_contacts(
        csv_text="name,email\nAda,ada@example.com\n",
        repository=repository,
    )
    assert result.imported == 1
    assert repository.all()[0].name == "Ada"

The first test executes every line of import_contacts and reports it as covered. It passes whether the import saves the row, saves the wrong row, saves it twice, or saves nothing at all — the only thing it can catch is an unhandled exception.

Coverage tools cannot tell these two tests apart. Both light the lines green. This is why a high number and a weak suite coexist so easily, and why the number alone should never be treated as reassurance.

The technique that can tell them apart is mutation testing: change the code deliberately and see whether any test notices. It is a whole lesson in the advanced course, and it is the honest answer to "are these tests actually checking anything?"

Goodhart's law, applied to testing

When a measure becomes a target, it stops measuring what it did. Coverage is a textbook case, and the failure modes are specific.

A blanket threshold produces assertion-free tests
"Every pull request must be 80% covered" is satisfied fastest by calling functions without asserting on them — exactly the bad example above.
It rewards testing the easy things
Getters, constructors, thin glue and configuration are trivially coverable and almost never broken. The gnarly branch in the payment retry logic is hard to cover, and that is where the risk is.
It punishes deleting dead code
Removing an untested unused function lowers your percentage — a perverse incentive against a good change.
It says nothing about what is missing
Coverage can only report on code that exists. The requirement nobody implemented, the error case nobody handled, the empty state nobody built — all invisible, and all real defects.
Coverage measures your tests against your code, not against the requirements. A feature that was specified and never written scores perfectly.

Each of these makes the number go up and the suite get weaker

Using it well

Coverage is a good tool for asking questions and a bad one for answering them. Five ways to use it that hold up.

Read the report, not the number. Sort by lowest covered and look at what appears. A file at 0% is a fact worth knowing; the total is not.

Look for surprises. "The payment retry module is at 12%" is the information you came for. "Overall 74%" is not.

Check the diff, not the codebase. Coverage of the lines a pull request changed is far more actionable than a project-wide figure. Most tools support it, and platforms like Codecov report it directly on the pull request.

Use it to find untested error paths. These are where coverage earns its place: the except block nobody triggers, the early return nobody reaches, the fallback nobody exercises. They are exactly the paths that run during an incident.

Treat 100% as a smell in most codebases. Reaching it usually means tests were written for code that did not need them, or that # pragma: no cover is doing a lot of work. Some libraries genuinely aim for it and can defend the cost; an application usually cannot.

bash

# the two most useful invocations
pytest --cov=src --cov-branch --cov-report=term-missing | sort -k6
pytest --cov=src --cov-branch --cov-report=html   # then click around

The HTML report is underused. It shows each file with covered lines in green and missed ones in red, and five minutes of scrolling through the files you consider risky is worth more than any threshold.

What to measure instead

If coverage is a weak proxy for suite quality, what is a better one? Four candidates, all harder to game.

Escaped defects. How many bugs reached production, and how many of them were in areas the suite claimed to cover. This is the outcome measure, and it is the one that matters.

Mutation score. What proportion of deliberately introduced faults the suite catches. Directly measures whether tests assert anything. Expensive to run, so usually on a subset, nightly.

Time to detect. How long between a defect being introduced and being caught. A fast pipeline with fewer tests can beat a slow one with more.

Flaky test count. A suite's trustworthiness, which the last lesson argued is its most important property.

The metrics lesson at the end of this course goes further into which measures survive contact with incentives. The short version: measure outcomes, not activity.

Cheat sheet

text

# Coverage measures which code was EXECUTED. Not tested. Executed.

# Kinds, weakest to strongest
line       which lines ran
branch     which decisions went BOTH ways      <- measure this one
condition  each part of a compound condition
path       routes through a function — usually infeasible
           (10 sequential two-way branches = 1,024 paths)

# 100% line coverage can leave a whole rule untested
#   two tests can execute every line while one return value has
#   never been produced and one flag has only ever been False

# The report is useful; the number is not
pytest --cov=src --cov-branch --cov-report=term-missing
pytest --cov=src --cov-branch --cov-report=html   # open htmlcov/
vitest run --coverage
#   read the Missing column and the 0% files

# Why a threshold backfires (Goodhart's law)
assertion-free tests   a test with no assert covers every line
rewards easy code      getters and glue, not the payment retry branch
punishes deletion      removing dead code lowers the number
blind to what is       coverage measures tests against CODE, never
  missing              against requirements. Unwritten features
                       score perfectly.
#   a better rule: "this PR must not LOWER coverage"

# Five ways to use it that hold up
read the report, not the number
look for surprises — "payments is at 12%"
check the DIFF's coverage, not the project's
hunt untested ERROR paths — the ones that run during an incident
treat 100% as a smell unless you can defend the cost

# Better measures of suite quality
escaped defects    how many reached production, and where
mutation score     does the suite notice a deliberately broken line
time to detect     introduced -> caught
flaky test count   whether the suite is trusted at all

Where to go next

The next lesson covers a kind of check that assertions cannot express: visual and snapshot testing, where the expected result is a recorded artefact rather than a value — along with how to keep either from becoming a rubber stamp.

Before that, generate an HTML coverage report for something you work on and open the three files you consider riskiest. What is red in those files is a concrete list of work, and it is the version of this exercise that produces better tests rather than a better number.