Risk-Based Testing
Testing effort is a budget. Modelling likelihood against impact, spending on the parts that would actually hurt, and defending the choice when someone asks why an area is thin.
Testing effort is a budget. Modelling likelihood against impact, spending on the parts that would actually hurt, and defending the choice when someone asks why an area is thin.
Testing effort is a fixed budget being spent against an infinite surface. That means every hour spent somewhere is an hour not spent elsewhere, and the only question that matters is whether the allocation is defensible.
Risk-based testing is how you make that allocation explicit rather than accidental. By the end of this lesson you will be able to build a risk model for a real system, spend against it, and answer the question that arrives after every incident: why was that area thin?
Risk is likelihood multiplied by impact. Both halves are needed, and teams routinely use only one.
Using impact alone gives you a strategy where payments is tested exhaustively and the frequently-broken import flow is not, because nobody dies when an import fails. Using likelihood alone gives you enormous effort on the flaky area that nobody uses.
The two together produce an ordering you can defend.
It will happen, and nobody minds much. A unit test and move on.
The obvious tier, and the one everybody already funds.
Say so out loud, so it is a decision rather than an oversight.
Nothing has gone wrong yet, so the area feels safe. That feeling is the failure mode.
Likelihood is not a guess. There is evidence, and the factors are consistent enough to use as a checklist.
+ recently changed bugs cluster in new code
+ complex many branches, special cases, exceptions
+ has broken before the difficulty that caused it remains
+ written under pressure a rushed release, a hotfix
+ nobody understands it original author gone, no documentation
+ many dependencies each one is a seam
+ concurrent or async the hardest defects live here
+ thin tests no safety net, so mistakes survive
+ new technology unfamiliarity produces mistakes
- stable for years in production, unchanged, exercised daily
- simple one branch, obvious behaviour
- heavily tested mistakes get caughtTwo of those are worth trusting more than the rest, because they are measurable rather than felt. Recent change — you have the commit history. Has broken before — you have the defect tracker. Together they predict where the next defect is better than any amount of reading the code.
# the twenty most frequently changed files in the last six months
git log --since="6 months ago" --name-only --pretty=format: \
| sort | uniq -c | sort -rn | head -20Cross-reference that list with your defect history and the top few entries are, empirically, where to look. It is a five-minute exercise and it is more informative than intuition.
Impact is a question about consequence, and it needs several axes because one number hides too much.
users affected one, some, all
reversible? a rollback fixes it / data is corrupted / money
moved / it was published
detectability obvious immediately / found in a week / silent
forever
^ silent is worse, because the damage accumulates
financial refunds, chargebacks, lost sales, SLA penalties
regulatory data protection, financial reporting, medical
trust would a user tell their colleagues about this?Detectability is the axis most often forgotten and frequently the most important. A loud failure — an error page, a failed payment — is bad and bounded: you know, you fix it, you apologise. A silent one — a report that is wrong by 3%, an export missing every hundredth row — compounds for months and may have been acted on. Silent failures deserve disproportionate testing.
The artefact is a table, and half its value is in the argument that produces it. Involve a developer, a product owner and a support person: each knows a different half of the likelihood and impact.
area like. imp. risk where confidence comes from
-------------------------------------------------------------------
payment capture 2 5 10 property-based on amounts,
contract tests, manual pass,
prod reconciliation
authorisation checks 3 5 15 full authz matrix at API
level; annual pentest
CSV import 4 3 12 integration + large-data +
exploratory each release
invoice PDF totals 2 4 8 property-based; silent
failure, so also a monthly
sampled check
session handling 2 4 8 integration + one e2e
search ranking 4 1 4 a few API tests
admin reporting 4 2 8 API only
marketing pages 3 1 3 visual snapshotsRead the ordering rather than the numbers. Authorisation at 15 outranks payment capture at 10 — which is often correct and rarely how effort is actually distributed, because payments feel like the risky area while authorisation is a quiet cross-cutting concern that every endpoint has to get right.
The numbers themselves are not precise and do not need to be. A 1–5 scale argued over by three people produces a defensible ordering, which is all that is required. Resist the temptation to add decimal places; it creates an impression of rigour the inputs do not support.
The model changes four concrete things.
Depth by tier. High risk gets several mechanisms and human attention; low risk gets one cheap automated check.
Order of work. Test high-risk areas first, so that if the release is cut short you have covered the things that matter. This is why the plan lesson insisted on ordering exploratory charters by risk.
Where new mechanisms go. Property-based testing, mutation testing, a pentest — expensive tools, aimed at the top of the table only.
What is skipped, explicitly. The bottom of the table is the accepted-risk section of your strategy.
Bad — effort allocated by what is easy to test:
Sprint testing summary
- 60 new browser tests for the admin reporting redesign
- 12 API tests for search ranking
- Payments unchanged this sprint, so not retestedGood — effort allocated by risk, including what was skipped:
Sprint testing summary
- Authorisation: added the full matrix for the three new
endpoints (5 cases each: no token, expired, wrong role, other
account read, other account write). Two defects found — both
allowed cross-account reads. BUG-511, BUG-512.
- Payments: unchanged, but the provider updated its API version.
Ran the contract suite against their sandbox; one response
field renamed. Fake updated.
- Admin reporting redesign: API tests updated. Deliberately no
browser tests — tolerable consequence, redesigned twice this
year.
- Search ranking: not retested. Low risk, no changes.The bad summary reports a lot of activity aimed at the cheapest-consequence area of the system, and it contains the sentence that precedes most payment incidents: "unchanged this sprint, so not retested". Code does not have to change to break — a dependency, a provider, a data volume or a configuration can change underneath it.
The good summary spends where the risk is, finds two real cross-account defects because it went looking in the highest-risk row, and states the skips as decisions.
The model's second job is to answer the question that comes after an incident. Two versions of that conversation:
Without a model: "Why wasn't this tested?" — "We didn't get to it." That reads as negligence, and there is no argument available.
With one: "Search ranking is likelihood 4, impact 1. We put it in the bottom tier in March, and the reasoning is in the strategy. This incident is evidence that the impact score was wrong — a bad ranking sent traffic to a deprecated page and cost us a day of conversions. I am moving it to impact 3, which puts it above admin reporting."
The second is a professional conversation. It shows the decision was made deliberately, that it was wrong in a specific and identifiable way, and what changes as a result. That is what a model is for — not to be right first time, but to make being wrong correctable.
Update the model from incidents as a matter of course. An escaped defect is evidence about your likelihood and impact estimates, and it is the best evidence you will get.
# Risk = likelihood x impact. Both halves, or the allocation is wrong.
# LOW IMPACT HIGH IMPACT
# HIGH LIKELIHOOD test cheaply test hardest, first
# LOW LIKELIHOOD ignore test carefully; monitor
# the bottom-right quadrant is where catastrophes live, and where
# intuition is worst — nothing has gone wrong yet
# irreversible consequences: treat impact as DOMINANT, not a factor
# Likelihood — the factors, most trustworthy first
recently changed you have the git history
has broken before you have the defect tracker
complex branches, special cases
nobody understands it author gone, no docs
many dependencies each is a seam
concurrent or async the hardest defects
thin tests / new tech / written under pressure
lower: stable for years, simple, heavily tested
git log --since="6 months ago" --name-only --pretty=format: \
| sort | uniq -c | sort -rn | head -20
# Impact — several axes, because one number hides too much
users affected one / some / all
reversible? rollback fixes it / data corrupt / money moved
DETECTABILITY obvious / found in a week / SILENT FOREVER
^ most-forgotten axis; silent damage compounds
financial / regulatory / trust
# Build the model with a developer, a PO and a support person
# 1-5 scales, argued over. Read the ORDERING, not the numbers.
# no decimal places — they imply rigour the inputs lack
# authorisation usually outranks payments, and rarely gets the
# effort, because it is a quiet cross-cutting concern
# Spend against it
depth by tier top: several mechanisms + human attention
order of work high risk FIRST, so a cut release still covered
new mechanisms property-based, mutation, pentest -> top tier only
explicit skips the bottom of the table is your accepted risks
# Unchanged code is not unchanged risk
# ask: has a dependency, provider, data volume, config or
# certificate changed underneath it?
# Update the model from every incident
# "impact was scored 1; this cost a day of conversions, so it is
# now 3" is a professional conversation.
# "we didn't get to it" is not.The model says where to spend. The next three lessons are the expensive mechanisms it points you at: property-based testing, which replaces examples with invariants; mutation testing, which measures whether your tests assert anything; and fuzzing, which attacks robustness with input nobody would write.
Before that, build the table for a system you work on — eight to twelve rows, two numbers each, and the current testing effort noted beside them. The rows where effort and risk disagree are your next quarter's work, and the table is the argument for it.