intermediate30 minby DevFox

Bug Triage and Quality Metrics

Running triage so the queue stays meaningful, and choosing measures that survive contact with incentives — escaped defects and time-to-detect over test counts and pass rates.

Testing
QA

Two process questions close this course, and they are related. How does a defect queue stay meaningful rather than becoming a graveyard of nine hundred tickets nobody reads? And how does anyone tell whether the testing is working?

Both have the same trap: the obvious answer — hold more triage meetings, count more things — makes it worse. By the end of this lesson you will know how to run triage so the queue reflects reality, and which handful of measures survive being turned into targets.

Why queues rot

A defect queue grows faster than it is emptied, which is not a failure — there will always be more known imperfections than capacity to fix them. The failure is when the queue stops being an accurate list of what is known.

The symptoms are recognisable:

text

tickets from two years ago, in features that no longer exist
duplicates nobody noticed, so one bug has four tickets
"cannot reproduce" left open indefinitely
everything marked High, because that was how to get attention
tickets with no reproduction, kept because deleting feels wrong
nobody reads the queue, so bugs get reported in chat instead

The last one is the end state, and it is fatal: once the real defect conversation happens in a chat channel, the tracker is a formality and the knowledge is unsearchable.

Running triage

Triage is a short recurring session — twenty minutes, twice a week works — where new defects get a decision. Not a discussion; a decision.

Four questions per ticket, fast:

text

1. Is it real and reproducible?
   No reproduction -> back to the reporter with what is missing.
   Not into the backlog. A ticket nobody can reproduce is not
   actionable, and parking it pretends otherwise.

2. Is it a duplicate?
   Link and close. Two minutes of searching saves a fix being done
   twice and a report being triaged four times.

3. Is the severity right?
   Sanity-check what the reporter set, against the workaround
   question. Adjust with a reason recorded.

4. What is the priority?
   Fix now / this sprint / backlog / will not fix.

Three things make the session work.

Timebox it. Twenty minutes for the new tickets. Anything needing a real discussion is taken out of the meeting, not resolved in it while eight people watch.

Have the impact numbers ready. How many users, how often, how many support tickets. This is the tester's contribution and it is preparation, not improvisation — it is what turns priority from an argument into a decision.

Decide, including "no". A ticket that leaves triage without a decision will be looked at again next week and the week after, which is the mechanism by which queues grow.

Closing things deliberately

The habit that keeps a queue readable is closing tickets on purpose, with a reason. Four legitimate outcomes besides "fixed":

Will not fix. A real defect that is not worth the cost — rare, low impact, in a feature being retired. Record why. This is a decision, not an admission.

Not reproducible, closed. After a genuine attempt and a request to the reporter. Reopenable if it recurs, with more detail.

By design. The behaviour is intended. Two obligations follow: say where that is documented, and if it was not documented, document it — because the next person will report it too.

Obsolete. The feature changed or was removed.

A useful and slightly uncomfortable practice: a stale-ticket sweep. Anything untouched for a year, at low priority, gets closed with a note that it can be reopened. If it mattered, somebody reports it again — and a queue of eighty live tickets is worth more than nine hundred where nobody can tell which are live.

Metrics that get gamed

Now the second half, and it needs Goodhart's law again: when a measure becomes a target, it stops measuring what it measured.

Bugs found per tester
Rewards volume, so you get many shallow reports — and a tester who improves acceptance criteria so the defects never exist scores zero.
Number of test cases
Rewards writing cases, not covering risk. A thousand near-duplicate cases beats two hundred good ones.
Test pass rate
The fastest way to 100% is deleting the failing test. The second fastest is writing tests that assert nothing.
Code coverage as a target
The whole subject of an earlier lesson: satisfied fastest by assertion-free tests.
Bugs closed per developer
Rewards closing, which is not the same as fixing, and encourages splitting one defect into five tickets.

Every one of these measures activity rather than outcome, and names the fastest way to game itself

Bad — a report that measures work done:

Good — a report that measures what happened to users:

The first report cannot distinguish a good sprint from a bad one. Every number in it can be improved without the product improving — more shallow bugs, more duplicate cases, deleted failing tests, assertion-free coverage. It also invites the reading that testing is a volume of activity.

The second says what changed for users, why, and what is still unsafe. It is harder to produce and much harder to game, and it is the version that leads to a decision — which is the whole purpose of reporting.

Measures that hold up

Five that survive being targets, because improving them requires actually improving.

Escaped defects. How many reached production. The outcome measure. It can only be improved by finding things earlier.

Escape route. For each escaped defect, which gate should have caught it. This is the metric that generates work: three escapes all missed by the same gap tells you what to build.

Time to detect. From a defect being introduced to being caught. Fast feedback is the thing most under your control, and it improves by tightening the pipeline rather than by writing more tests.

Time to recover. From a defect being reported to being fixed in production. Measures the whole chain, including whether you can deploy quickly.

Flaky test count. The suite's trustworthiness. A suite with flakes is a suite whose verdict is advisory.

Two more worth tracking without treating as targets: suite runtime, because it grows silently until people stop running it, and support ticket volume by area, which points at where the pain actually is.

The one question worth asking every time

More useful than any dashboard: for each defect that reached production, what would have caught it?

That question turns an incident into a change. The answers are usually structural, and they name work:

Each of those is worth more than the fix, because each closes a class rather than an instance. It is the same argument the regression lesson made about individual bugs, applied to the testing system itself — and it is the most valuable habit in this course.

Cheat sheet

Where to go next

That closes Testing in Practice. You can structure a suite, place tests at the right level, fake what you do not own, keep state isolated, test an API and a browser, kill flakiness, run it all in CI, read coverage honestly, catch what assertions cannot, check the automatable part of accessibility, and report in a way that leads to decisions.

Advanced Testing is the next track, and it takes on the problems this one deliberately left: designing a strategy and spending by risk, property-based and mutation testing, fuzzing, contract testing between services, load models and resilience, testing in production, scaling a suite, quality gates, non-deterministic systems, and leading quality across a team.

Before that, do the escape-route exercise on the last three defects that reached production in something you work on. Three answers, three structural gaps, and a list of work that is better justified than anything a coverage report will give you.