Designing a Test Strategy
A strategy is a set of decisions about where confidence comes from and what you are willing not to know. Writing one for a real system, and the trade-offs it has to name.
A strategy is a set of decisions about where confidence comes from and what you are willing not to know. Writing one for a real system, and the trade-offs it has to name.
A test plan covers a feature. A strategy covers a system, and it is a different kind of document: not a list of what will be checked, but a set of decisions about where confidence comes from and what you are deliberately choosing not to know.
That second half is what distinguishes a strategy from an aspiration. By the end of this lesson you will be able to write one for a real system, name the trade-offs it commits to, and defend the parts you left uncovered.
The failure mode is writing a document that says everything will be tested thoroughly at every level. That is not a strategy — it describes no choice, so it constrains nothing and cannot be wrong.
A strategy answers questions that have costs on both sides:
Where does confidence come from for each part of the system?
unit tests? contract tests? production monitoring? a human?
What are we deliberately not testing, and why?
What must never break, and what is tolerable?
How fast must feedback be, and what are we paying for that?
What do we do when the answer only exists in production?
Who decides a release goes out, on what evidence?Each of those has a defensible answer and several indefensible ones. Writing them down is what lets a team disagree productively — and what lets a new engineer understand why the suite looks the way it does.
The first section of any strategy worth reading is a map of the system by what happens when each part fails. Not by size, not by complexity — by consequence.
CATASTROPHIC — irreversible, or existential
customer data exposed or lost
money moved incorrectly
a regulatory breach
destroyed trust
SEVERE — the product does not work
cannot sign in
cannot complete the core task
data corrupted but recoverable
TOLERABLE — degraded and survivable
a secondary feature is broken
a report is slow
a non-blocking flow fails and can be retried
COSMETIC — noticeable, not harmfulThen, for each area, the testing follows from where it sits:
area consequence confidence comes from
payments catastrophic unit + integration + contract +
property-based on the money maths +
a manual pass each release +
reconciliation monitoring in prod
authentication catastrophic unit + integration + the full
authorisation matrix + a pentest
annually
core CRUD severe integration + API + one e2e journey
import/export severe integration + large-data tests
notifications tolerable unit + a fake at the boundary +
delivery monitoring
admin reporting tolerable API tests; no e2e
marketing pages cosmetic visual snapshots; nothing elseThat table is the strategy. Everything else in the document explains and qualifies it.
Two things to notice. Effort is wildly uneven, on purpose. And for payments, part of the confidence comes from production monitoring rather than from tests — which is a legitimate and often superior answer for a property that must hold continuously rather than at build time.
For each area, the question is which mechanism is cheapest for the risk it addresses. The mechanisms available, and what each is actually good for:
unit tests logic with many cases; fast feedback
integration tests the seams: schema, transactions, serialisation
contract tests two services agreeing, without running both
API tests the service's whole contract, incl. authorisation
e2e journeys that the assembled product works at all
property-based invariants over generated input
mutation testing whether the tests assert anything
fuzzing robustness against input nobody would write
load tests behaviour under a modelled arrival rate
chaos experiments behaviour when a dependency fails
static analysis whole classes of mistake, on every keystroke
code review judgement, and everything a machine cannot see
exploratory testing the defects nobody predicted
canary release real traffic, small blast radius
monitoring + SLOs properties that must hold continuously
manual pass experience, and the irreducibly humanThe advanced-course lessons after this one are each about one row. The strategic skill is the assignment: matching mechanism to risk, rather than applying every mechanism everywhere.
Bad — every sentence is unobjectionable and none constrains anything:
TEST STRATEGY
We follow the testing pyramid and aim for comprehensive coverage
at all levels. Unit tests will cover business logic, integration
tests will cover component interactions, and end-to-end tests
will cover user journeys. We target 80% code coverage. Quality
is everyone's responsibility.Good — decisions, with reasons and costs:
TEST STRATEGY (extract)
Payments is the only catastrophic-consequence area, so it gets
the most expensive treatment: property-based tests on the amount
arithmetic, contract tests against the provider, a manual pass
each release, and a nightly reconciliation job in production
that alerts on any mismatch. We accept the cost of the manual
pass because a rounding error is not recoverable by a rollback.
Admin reporting is tolerable-consequence and changes weekly. It
gets API tests only. We are deliberately not writing browser
tests for it: the maintenance cost of e2e tests against a
frequently redesigned internal tool exceeded their value twice
before, and a broken report is a day of inconvenience for four
internal users.
Feedback: pull requests under 10 minutes, which caps what runs
there. The full e2e suite therefore runs nightly, and we accept
that an e2e regression can be up to a day old when we find it.
The mitigation is that e2e covers only assembly failures, which
the smoke test after each staging deploy also catches within
minutes.Worse than nothing.
Specifically because it reads like a strategy. Nobody can act on it, nobody can disagree with it, and it will not change a single decision.
When something escapes to production it offers no account of why that area was thin, because it never said any area was thin.
Every line has a cost on both sides.
Somebody may think admin reporting deserves browser tests — and now there is a stated position to argue against, including the two prior attempts that failed.
That argument is the document doing its job.
Every strategy has holes. A good one lists them.
ACCEPTED RISKS
Concurrency across services is not tested. We have no mechanism
for it and the cost of building one is not justified at current
volume. Consequence: a race between the import and billing jobs
would corrupt an invoice. Mitigation: both are single-worker
today. REVISIT if either is parallelised.
Safari is tested nightly, not per pull request, because the CI
cost triples. Consequence: a Safari-only regression can be a day
old. Mitigation: Safari is 12% of traffic and the nightly run
blocks the release.
Load beyond 500 concurrent users is untested. We have not
modelled peak arrival rates. Consequence: unknown behaviour
under a traffic spike. This is the largest unquantified risk in
this strategy. OWNER: platform team, Q3.Three properties make that section useful: a stated consequence, a mitigation where one exists, and a trigger or owner for revisiting. "We do not test X" alone is an admission; with those three it is a decision.
A strategy written once and never revisited becomes fiction within a year, and a fictional strategy is worse than none because people cite it.
Three mechanisms keep it true:
Review it on a schedule. Quarterly, briefly. What changed about the system, the team, the traffic, the risks?
Update it from incidents. Every escaped defect is evidence about the strategy. If the escape route points at a mechanism you decided not to use, that decision now has new evidence against it — which is either a change or a reaffirmation, and both belong in the document.
Keep it short. Two to four pages. A thirty-page strategy is not read, and an unread strategy is not a strategy. If it is growing, the growth is usually plan-level detail that belongs elsewhere.
# A strategy is a set of DECISIONS with costs on both sides.
# A document that says "test thoroughly at all levels" decides
# nothing and cannot be wrong.
# The questions it answers
where does confidence come from, for each part?
what are we deliberately NOT testing, and why?
what must never break, and what is tolerable?
how fast must feedback be, and what does that cap?
what can only be learned in production?
who decides a release ships, on what evidence?
# Map the system by CONSEQUENCE, not by size
catastrophic data exposed or lost, money wrong, regulatory
severe cannot sign in, cannot do the core task
tolerable a secondary feature broken, a slow report
cosmetic noticeable, not harmful
# Then assign mechanisms to risk — effort should be UNEVEN
payments unit + integration + contract + property-based +
a manual pass + production reconciliation
core CRUD integration + API + one e2e journey
notifications unit + a fake + delivery monitoring
admin reports API only — no e2e, deliberately
marketing visual snapshots
# The mechanisms, and what each is for
unit logic with many cases
integration the seams
contract two services agreeing without running both
API the whole contract, including authorisation
e2e that the assembled product works at all
property-based invariants over generated input
mutation whether the tests assert anything
fuzzing robustness against input nobody would write
load behaviour under a modelled arrival rate
chaos behaviour when a dependency fails
static whole classes of mistake, continuously
review judgement
exploratory the unpredicted
canary real traffic, small blast radius
SLOs properties that must hold continuously
manual the irreducibly human
# ACCEPTED RISKS — the section that makes it honest
# each entry needs three things:
consequence what happens if it bites
mitigation what reduces it, if anything
trigger/owner when to revisit, and who
# "we do not test X" alone is an admission
# with those three it is a decision
# Keep it alive
review quarterly / update from every escaped defect / 2-4 pagesThe strategy assigned effort by consequence. The next lesson makes that allocation systematic: risk-based testing — modelling likelihood against impact, spending the budget where it matters, and defending the choice when somebody asks why an area is thin.
Before that, write the consequence map for a system you work on. Four bands, every area placed in one, and the current testing effort noted beside it. The mismatches are usually obvious once the table exists, and they are the argument for everything else in this course.