Bug Triage and Quality Metrics
Running triage so the queue stays meaningful, and choosing measures that survive contact with incentives — escaped defects and time-to-detect over test counts and pass rates.
Running triage so the queue stays meaningful, and choosing measures that survive contact with incentives — escaped defects and time-to-detect over test counts and pass rates.
Two process questions close this course, and they are related. How does a defect queue stay meaningful rather than becoming a graveyard of nine hundred tickets nobody reads? And how does anyone tell whether the testing is working?
Both have the same trap: the obvious answer — hold more triage meetings, count more things — makes it worse. By the end of this lesson you will know how to run triage so the queue reflects reality, and which handful of measures survive being turned into targets.
A defect queue grows faster than it is emptied, which is not a failure — there will always be more known imperfections than capacity to fix them. The failure is when the queue stops being an accurate list of what is known.
The symptoms are recognisable:
tickets from two years ago, in features that no longer exist
duplicates nobody noticed, so one bug has four tickets
"cannot reproduce" left open indefinitely
everything marked High, because that was how to get attention
tickets with no reproduction, kept because deleting feels wrong
nobody reads the queue, so bugs get reported in chat insteadThe last one is the end state, and it is fatal: once the real defect conversation happens in a chat channel, the tracker is a formality and the knowledge is unsearchable.
Triage is a short recurring session — twenty minutes, twice a week works — where new defects get a decision. Not a discussion; a decision.
Four questions per ticket, fast:
1. Is it real and reproducible?
No reproduction -> back to the reporter with what is missing.
Not into the backlog. A ticket nobody can reproduce is not
actionable, and parking it pretends otherwise.
2. Is it a duplicate?
Link and close. Two minutes of searching saves a fix being done
twice and a report being triaged four times.
3. Is the severity right?
Sanity-check what the reporter set, against the workaround
question. Adjust with a reason recorded.
4. What is the priority?
Fix now / this sprint / backlog / will not fix.Three things make the session work.
Timebox it. Twenty minutes for the new tickets. Anything needing a real discussion is taken out of the meeting, not resolved in it while eight people watch.
Have the impact numbers ready. How many users, how often, how many support tickets. This is the tester's contribution and it is preparation, not improvisation — it is what turns priority from an argument into a decision.
Decide, including "no". A ticket that leaves triage without a decision will be looked at again next week and the week after, which is the mechanism by which queues grow.
The habit that keeps a queue readable is closing tickets on purpose, with a reason. Four legitimate outcomes besides "fixed":
Will not fix. A real defect that is not worth the cost — rare, low impact, in a feature being retired. Record why. This is a decision, not an admission.
Not reproducible, closed. After a genuine attempt and a request to the reporter. Reopenable if it recurs, with more detail.
By design. The behaviour is intended. Two obligations follow: say where that is documented, and if it was not documented, document it — because the next person will report it too.
Obsolete. The feature changed or was removed.
A useful and slightly uncomfortable practice: a stale-ticket sweep. Anything untouched for a year, at low priority, gets closed with a note that it can be reopened. If it mattered, somebody reports it again — and a queue of eighty live tickets is worth more than nine hundred where nobody can tell which are live.
Now the second half, and it needs Goodhart's law again: when a measure becomes a target, it stops measuring what it measured.
Bugs found per tester
Rewards volume, so you get many shallow reports — and a tester who improves acceptance criteria so the defects never exist scores zero.
Number of test cases
Rewards writing cases, not covering risk. A thousand near-duplicate cases beats two hundred good ones.
Test pass rate
The fastest way to 100% is deleting the failing test. The second fastest is writing tests that assert nothing.
Code coverage as a target
The whole subject of an earlier lesson: satisfied fastest by assertion-free tests.
Bugs closed per developer
Rewards closing, which is not the same as fixing, and encourages splitting one defect into five tickets.
Bad — a report that measures work done:
This sprint: 47 bugs found, 312 test cases executed, 94% pass
rate, coverage up to 81%.Good — a report that measures what happened to users:
This sprint:
- 3 defects reached production (down from 7). Two were in the
import flow, which has no integration tests — proposed work
attached.
- Median time from a defect being introduced to being caught:
40 minutes (was 3 days before the pipeline change).
- 2 flaky tests remain, both quarantined with owners.
- Import flow: no coverage of partial failures. This is the
largest known gap.The first report cannot distinguish a good sprint from a bad one. Every number in it can be improved without the product improving — more shallow bugs, more duplicate cases, deleted failing tests, assertion-free coverage. It also invites the reading that testing is a volume of activity.
The second says what changed for users, why, and what is still unsafe. It is harder to produce and much harder to game, and it is the version that leads to a decision — which is the whole purpose of reporting.
Five that survive being targets, because improving them requires actually improving.
Escaped defects. How many reached production. The outcome measure. It can only be improved by finding things earlier.
Escape route. For each escaped defect, which gate should have caught it. This is the metric that generates work: three escapes all missed by the same gap tells you what to build.
Time to detect. From a defect being introduced to being caught. Fast feedback is the thing most under your control, and it improves by tightening the pipeline rather than by writing more tests.
Time to recover. From a defect being reported to being fixed in production. Measures the whole chain, including whether you can deploy quickly.
Flaky test count. The suite's trustworthiness. A suite with flakes is a suite whose verdict is advisory.
Two more worth tracking without treating as targets: suite runtime, because it grows silently until people stop running it, and support ticket volume by area, which points at where the pain actually is.
More useful than any dashboard: for each defect that reached production, what would have caught it?
That question turns an incident into a change. The answers are usually structural, and they name work:
"nothing tested the empty case anywhere"
-> add empty-state cases to the factory defaults
"every test used an admin account"
-> the authorisation cases from the API lesson
"the tests mocked the thing that was broken"
-> an integration test at that seam
"it only happens across two requests"
-> a sequence test, not a unit test
"staging has 200 records and production has 2 million"
-> a large-data test, and realistic seed volumes
"we had a test and it was flaky, so it had been quarantined"
-> fix the flake; that is the whole findingEach of those is worth more than the fix, because each closes a class rather than an instance. It is the same argument the regression lesson made about individual bugs, applied to the testing system itself — and it is the most valuable habit in this course.
# Triage: 20 minutes, twice a week, four questions per ticket
1. real and reproducible? no repro -> back to the reporter, NOT
into the backlog
2. duplicate? link and close
3. severity right? check the workaround question; note why
4. priority? now / this sprint / backlog / will not fix
# timebox it; have the impact NUMBERS ready; always decide
# Close deliberately — four legitimate non-fix outcomes
will not fix a real defect not worth the cost. Record why.
not reproducible after a genuine attempt. Reopenable.
by design say where it is documented — and document it if
it was not, or it gets reported again
obsolete the feature changed or went
# plus a stale sweep: untouched for a year at low priority ->
# closed with a note. If it mattered, it comes back.
# Metrics that get gamed (all measure ACTIVITY)
bugs found per tester rewards volume, punishes prevention
number of test cases rewards duplicates
test pass rate fastest fix is deleting the failing test
coverage as a target fastest fix is assertion-free tests
bugs closed per developer closing is not fixing
# Measures that hold up (all measure OUTCOME)
escaped defects how many reached production
escape route WHICH gate should have caught each one
<- the one that generates work
time to detect introduced -> caught
time to recover reported -> fixed in production
flaky test count whether the suite is trusted
track but do not target: suite runtime, support tickets by area
# Never per-person. It converts a diagnostic into a performance
# review, and then people stop reporting near-misses.
# The question worth asking every time
# "for each defect that reached production, what would have
# caught it?"
# the answers are structural, and each closes a CLASS:
# nothing tested the empty case
# every test used an admin account
# the tests mocked the broken thing
# it only happens across two requests
# staging has 200 records, production has 2 million
# we had a test and it was quarantined for flakingThat closes Testing in Practice. You can structure a suite, place tests at the right level, fake what you do not own, keep state isolated, test an API and a browser, kill flakiness, run it all in CI, read coverage honestly, catch what assertions cannot, check the automatable part of accessibility, and report in a way that leads to decisions.
Advanced Testing is the next track, and it takes on the problems this one deliberately left: designing a strategy and spending by risk, property-based and mutation testing, fuzzing, contract testing between services, load models and resilience, testing in production, scaling a suite, quality gates, non-deterministic systems, and leading quality across a team.
Before that, do the escape-route exercise on the last three defects that reached production in something you work on. Three answers, three structural gaps, and a list of work that is better justified than anything a coverage report will give you.