advanced35 minby DevFox

Designing a Test Strategy

A strategy is a set of decisions about where confidence comes from and what you are willing not to know. Writing one for a real system, and the trade-offs it has to name.

Testing
Architecture

A test plan covers a feature. A strategy covers a system, and it is a different kind of document: not a list of what will be checked, but a set of decisions about where confidence comes from and what you are deliberately choosing not to know.

That second half is what distinguishes a strategy from an aspiration. By the end of this lesson you will be able to write one for a real system, name the trade-offs it commits to, and defend the parts you left uncovered.

A strategy is a set of decisions

The failure mode is writing a document that says everything will be tested thoroughly at every level. That is not a strategy — it describes no choice, so it constrains nothing and cannot be wrong.

A strategy answers questions that have costs on both sides:

text

Where does confidence come from for each part of the system?
  unit tests? contract tests? production monitoring? a human?

What are we deliberately not testing, and why?

What must never break, and what is tolerable?

How fast must feedback be, and what are we paying for that?

What do we do when the answer only exists in production?

Who decides a release goes out, on what evidence?

Each of those has a defensible answer and several indefensible ones. Writing them down is what lets a team disagree productively — and what lets a new engineer understand why the suite looks the way it does.

Start from consequence, not from coverage

The first section of any strategy worth reading is a map of the system by what happens when each part fails. Not by size, not by complexity — by consequence.

text

CATASTROPHIC — irreversible, or existential
  customer data exposed or lost
  money moved incorrectly
  a regulatory breach
  destroyed trust

SEVERE — the product does not work
  cannot sign in
  cannot complete the core task
  data corrupted but recoverable

TOLERABLE — degraded and survivable
  a secondary feature is broken
  a report is slow
  a non-blocking flow fails and can be retried

COSMETIC — noticeable, not harmful

Then, for each area, the testing follows from where it sits:

text

area              consequence    confidence comes from
payments          catastrophic   unit + integration + contract +
                                 property-based on the money maths +
                                 a manual pass each release +
                                 reconciliation monitoring in prod
authentication    catastrophic   unit + integration + the full
                                 authorisation matrix + a pentest
                                 annually
core CRUD         severe         integration + API + one e2e journey
import/export     severe         integration + large-data tests
notifications     tolerable      unit + a fake at the boundary +
                                 delivery monitoring
admin reporting   tolerable      API tests; no e2e
marketing pages   cosmetic       visual snapshots; nothing else

That table is the strategy. Everything else in the document explains and qualifies it.

Two things to notice. Effort is wildly uneven, on purpose. And for payments, part of the confidence comes from production monitoring rather than from tests — which is a legitimate and often superior answer for a property that must hold continuously rather than at build time.

Deciding where confidence comes from

For each area, the question is which mechanism is cheapest for the risk it addresses. The mechanisms available, and what each is actually good for:

The advanced-course lessons after this one are each about one row. The strategic skill is the assignment: matching mechanism to risk, rather than applying every mechanism everywhere.

The strategy that commits to nothing

Bad — every sentence is unobjectionable and none constrains anything:

Good — decisions, with reasons and costs:

Unobjectionable

Worse than nothing.

Specifically because it reads like a strategy. Nobody can act on it, nobody can disagree with it, and it will not change a single decision.

When something escapes to production it offers no account of why that area was thin, because it never said any area was thin.

Arguable

Every line has a cost on both sides.

Somebody may think admin reporting deserves browser tests — and now there is a stated position to argue against, including the two prior attempts that failed.

That argument is the document doing its job.

A strategy you cannot disagree with is not doing any work

Naming what you will not know

Every strategy has holes. A good one lists them.

Three properties make that section useful: a stated consequence, a mitigation where one exists, and a trigger or owner for revisiting. "We do not test X" alone is an admission; with those three it is a decision.

Keeping it alive

A strategy written once and never revisited becomes fiction within a year, and a fictional strategy is worse than none because people cite it.

Three mechanisms keep it true:

Review it on a schedule. Quarterly, briefly. What changed about the system, the team, the traffic, the risks?

Update it from incidents. Every escaped defect is evidence about the strategy. If the escape route points at a mechanism you decided not to use, that decision now has new evidence against it — which is either a change or a reaffirmation, and both belong in the document.

Keep it short. Two to four pages. A thirty-page strategy is not read, and an unread strategy is not a strategy. If it is growing, the growth is usually plan-level detail that belongs elsewhere.

Cheat sheet

Where to go next

The strategy assigned effort by consequence. The next lesson makes that allocation systematic: risk-based testing — modelling likelihood against impact, spending the budget where it matters, and defending the choice when somebody asks why an area is thin.

Before that, write the consequence map for a system you work on. Four bands, every area placed in one, and the current testing effort noted beside it. The mismatches are usually obvious once the table exists, and they are the argument for everything else in this course.