advanced30 minby DevFox

Risk-Based Testing

Testing effort is a budget. Modelling likelihood against impact, spending on the parts that would actually hurt, and defending the choice when someone asks why an area is thin.

Testing
QA

Testing effort is a fixed budget being spent against an infinite surface. That means every hour spent somewhere is an hour not spent elsewhere, and the only question that matters is whether the allocation is defensible.

Risk-based testing is how you make that allocation explicit rather than accidental. By the end of this lesson you will be able to build a risk model for a real system, spend against it, and answer the question that arrives after every incident: why was that area thin?

Risk has two dimensions

Risk is likelihood multiplied by impact. Both halves are needed, and teams routinely use only one.

Using impact alone gives you a strategy where payments is tested exhaustively and the frequently-broken import flow is not, because nobody dies when an import fails. Using likelihood alone gives you enormous effort on the flaky area that nobody uses.

The two together produce an ordering you can defend.

How often it happensLow impactHigh impactHigh likelihood

Test, cheaply

It will happen, and nobody minds much. A unit test and move on.

Test hardest, first

The obvious tier, and the one everybody already funds.

Low likelihood

Ignore

Say so out loud, so it is a decision rather than an oversight.

Test carefully; monitor

Nothing has gone wrong yet, so the area feels safe. That feeling is the failure mode.

What it costs when it happens

The interesting cell is the bottom right — where catastrophes live and intuition is least reliable

Estimating likelihood

Likelihood is not a guess. There is evidence, and the factors are consistent enough to use as a checklist.

Two of those are worth trusting more than the rest, because they are measurable rather than felt. Recent change — you have the commit history. Has broken before — you have the defect tracker. Together they predict where the next defect is better than any amount of reading the code.

Cross-reference that list with your defect history and the top few entries are, empirically, where to look. It is a five-minute exercise and it is more informative than intuition.

Estimating impact

Impact is a question about consequence, and it needs several axes because one number hides too much.

Detectability is the axis most often forgotten and frequently the most important. A loud failure — an error page, a failed payment — is bad and bounded: you know, you fix it, you apologise. A silent one — a report that is wrong by 3%, an export missing every hundredth row — compounds for months and may have been acted on. Silent failures deserve disproportionate testing.

Building the model

The artefact is a table, and half its value is in the argument that produces it. Involve a developer, a product owner and a support person: each knows a different half of the likelihood and impact.

Read the ordering rather than the numbers. Authorisation at 15 outranks payment capture at 10 — which is often correct and rarely how effort is actually distributed, because payments feel like the risky area while authorisation is a quiet cross-cutting concern that every endpoint has to get right.

The numbers themselves are not precise and do not need to be. A 1–5 scale argued over by three people produces a defensible ordering, which is all that is required. Resist the temptation to add decimal places; it creates an impression of rigour the inputs do not support.

Spending against it

The model changes four concrete things.

Depth by tier. High risk gets several mechanisms and human attention; low risk gets one cheap automated check.

Order of work. Test high-risk areas first, so that if the release is cut short you have covered the things that matter. This is why the plan lesson insisted on ordering exploratory charters by risk.

Where new mechanisms go. Property-based testing, mutation testing, a pentest — expensive tools, aimed at the top of the table only.

What is skipped, explicitly. The bottom of the table is the accepted-risk section of your strategy.

Bad — effort allocated by what is easy to test:

Good — effort allocated by risk, including what was skipped:

The bad summary reports a lot of activity aimed at the cheapest-consequence area of the system, and it contains the sentence that precedes most payment incidents: "unchanged this sprint, so not retested". Code does not have to change to break — a dependency, a provider, a data volume or a configuration can change underneath it.

The good summary spends where the risk is, finds two real cross-account defects because it went looking in the highest-risk row, and states the skips as decisions.

Defending the allocation

The model's second job is to answer the question that comes after an incident. Two versions of that conversation:

Without a model: "Why wasn't this tested?" — "We didn't get to it." That reads as negligence, and there is no argument available.

With one: "Search ranking is likelihood 4, impact 1. We put it in the bottom tier in March, and the reasoning is in the strategy. This incident is evidence that the impact score was wrong — a bad ranking sent traffic to a deprecated page and cost us a day of conversions. I am moving it to impact 3, which puts it above admin reporting."

The second is a professional conversation. It shows the decision was made deliberately, that it was wrong in a specific and identifiable way, and what changes as a result. That is what a model is for — not to be right first time, but to make being wrong correctable.

Update the model from incidents as a matter of course. An escaped defect is evidence about your likelihood and impact estimates, and it is the best evidence you will get.

Cheat sheet

Where to go next

The model says where to spend. The next three lessons are the expensive mechanisms it points you at: property-based testing, which replaces examples with invariants; mutation testing, which measures whether your tests assert anything; and fuzzing, which attacks robustness with input nobody would write.

Before that, build the table for a system you work on — eight to twelve rows, two numbers each, and the current testing effort noted beside them. The rows where effort and risk disagree are your next quarter's work, and the table is the argument for it.