Testing for Resilience and Failure
Injecting the failures production will deliver anyway: latency, partitions, dependency outages and clock skew. Running an experiment with a hypothesis and a blast radius.
Injecting the failures production will deliver anyway: latency, partitions, dependency outages and clock skew. Running an experiment with a hypothesis and a blast radius.
Your dependencies will fail. The database will be unreachable for forty seconds during a failover, a third-party API will time out, a network will partition, and a machine's clock will jump. None of that is hypothetical — it is the ordinary weather of distributed systems.
The question is not whether it happens but whether your system does something sensible when it does. That is testable, and the discipline for testing it is an experiment: a hypothesis, a controlled fault, a bounded blast radius, and a measured result. By the end of this lesson you will know how to run one and what to inject.
Every earlier level assumed the environment worked. Unit tests mock the dependency away, integration tests use a healthy one, load tests apply demand to a functioning system.
But the code that handles failure is the least-exercised code you have, and it runs at the worst possible time. A retry loop with no backoff, a timeout that is never set, a fallback that throws its own exception, a circuit breaker that never closes again — each is written once, never executed in anger, and discovered during an incident.
Resilience testing exercises it deliberately, while somebody is watching.
Five parts, and skipping any of them turns an experiment into an outage.
Steady state
A measurable property that is true when things are fine — "p95 checkout latency under 800ms and error rate under 0.5%".
Measurable, or you cannot tell whether you broke it.
Hypothesis
What you believe will happen, stated before running. "If the recommendations service becomes unavailable, checkout continues without recommendations and the error rate stays under 0.5%."
Blast radius
The smallest scope that can answer the question. One instance, one region, 1% of traffic, one dependency, staging first.
The fault
Injected in a way you can stop instantly.
Stop condition
The measurement at which you abort, decided in advance. "Abort if the error rate exceeds 2% or p95 exceeds 3s."
The hypothesis is the part that makes it valuable. An experiment that confirms your belief tells you the resilience works. One that disproves it has found a defect before a customer did — and the disproving is the point, so state a belief specific enough to be wrong.
Start with the failures your system will certainly meet, in rough order of likelihood.
Latency is the one to start with. Injecting slowness rather than failure finds more defects than any other single fault, because most code has an error path for "it failed" and no path at all for "it is taking forever". A missing timeout turns one slow dependency into an exhausted connection pool, and then into total unavailability of a service that was itself perfectly healthy.
You do not need a platform to start. Most of this is available in an ordinary integration test, using the faking techniques from the intermediate course.
That second test is the valuable one, and it is rarely written. It asserts that a timeout exists and is short enough — a resilience property rather than a functional one, and it fails loudly the day somebody removes the timeout in a refactor.
The same faults at the infrastructure level, for an integration or staging environment:
Purpose-built tools do this more safely and repeatably — Toxiproxy for network faults between services, Chaos Mesh and LitmusChaos for Kubernetes, AWS Fault Injection Simulator, Gremlin. Toxiproxy is the easiest to adopt because it sits in an integration test as a proxy you can misconfigure on demand.
Resilience experiments are testing whether specific patterns are present and correctly configured. Knowing the patterns tells you what to assert.
Two of those are the most commonly broken.
A thundering herd.
Missed almost universally. Every client that failed together retries together, and the service that was recovering goes back down.
A permanent outage from a 30-second one.
Which is why the experiment has to include restoring the dependency and checking that traffic resumes — not only that the breaker opened.
Bad — a fault injected into production with no stated belief and no stop condition:
Good — the same fault, as an experiment:
The bad version is not an experiment; it is an unplanned outage on a Friday evening. It produced one fact — the error rate went to 34% — and no diagnosis, because nothing was instrumented for the question and nobody had written down what was expected.
The good version found two defects in staging, at no cost to anybody, and established the resilience before going near production traffic. Note also that it ran on a Tuesday morning — in working hours, with people available. Running experiments when the team is present is not caution, it is the whole design.
The scaled-up version is a game day: a scheduled session where a team injects a realistic failure and practises responding to it, end to end — detection, diagnosis, mitigation, communication.
What it tests is broader than the software:
Most game days find that the software is more resilient than the response process. Runbooks are out of date, the dashboard needed does not exist, and the one person who knows how to fail over the database is on holiday. Those are findings you want on a Tuesday.
The next lesson covers the testable part of security — authorisation matrices, injection at every boundary, dependency and secret scanning — and the honest line between a scan and a penetration test.
Before that, write the timeout test from this lesson for one dependency you call. Assert that a slow response returns in under a second or two. It is five lines, and on first writing it people frequently discover there is no timeout at all.
LATENCY a dependency responds in 5s instead of 50ms
the most common real failure, and the most
revealing — it exposes missing timeouts
ERRORS a dependency returns 500s or 429s
UNAVAILABILITY connection refused, DNS failure
PARTIAL FAILURE one of three instances is broken
worse than total failure: retries may land on it
PACKET LOSS 1-5% loss, which degrades rather than breaks
DISK FULL logs stop, temporary files fail, uploads fail
MEMORY PRESSURE the process is killed, or thrashes
CPU SATURATION everything slows, health checks time out
CLOCK SKEW a token expires early; an ordering assumption
breaks
INSTANCE LOSS a machine disappears with no notice
DEPENDENCY OF A your cache's cache. Failures are transitive.
DEPENDENCYTIMEOUT every network call has one, and it is shorter
than the caller's own timeout
RETRY WITH retry only what is retryable, with exponential
BACKOFF + JITTER backoff and randomness. Without jitter, every
client retries in lockstep and hammers a
recovering service.
CIRCUIT BREAKER after n failures, stop calling for a while. Tests
should verify it OPENS and, crucially, CLOSES again.
BULKHEAD separate connection pools per dependency, so one
slow dependency cannot exhaust the pool everything
else needs
FALLBACK a degraded answer: cached data, empty list, a
default. And the fallback must not itself fail.
LOAD SHEDDING reject work you cannot do, fast, rather than
queueing it
IDEMPOTENCY a retried request must not charge twiceFriday 16:40. Terminated the recommendations service in production
to see what happens.
Checkout error rate went to 34%. Rolled back at 16:52.EXPERIMENT: recommendations unavailable during checkout
Steady state p95 checkout < 800ms, error rate < 0.5%
(measured over the last 24h: 610ms, 0.2%)
Hypothesis checkout completes without recommendations; error
rate stays under 0.5%; p95 rises by less than 200ms
Blast radius staging first. Then production, 1% of traffic, one
availability zone, 10 minutes, Tuesday 10:00.
Fault Toxiproxy: connection refused on the
recommendations upstream
Stop condition abort if error rate > 2% or p95 > 3s
RESULT (staging)
Hypothesis DISPROVED. Error rate 100%: the recommendations call
was not wrapped in a fallback, and the exception propagated to
the checkout handler. Also found: no timeout on the call, so the
"slow" variant would have exhausted the connection pool.
Fixed: fallback to an empty list; 500ms timeout; bulkhead pool.
Re-ran: hypothesis confirmed. Production run scheduled.[ ] did monitoring detect it, and how long did that take?
[ ] did the alert reach a human, and was it actionable?
[ ] could the on-call engineer find the cause from the dashboards
that exist?
[ ] was the runbook accurate, or had it drifted?
[ ] did the mitigation work — and had anybody run it before?
[ ] was the rollback rehearsed, or attempted for the first time?
[ ] did anyone know who to tell?# The experiment — five parts, none optional
1. steady state a MEASURABLE property true when things are fine
2. hypothesis what you believe will happen, stated BEFORE
3. blast radius the smallest scope that answers the question
4. the fault injected so you can stop it instantly
5. stop condition the measurement at which you abort, decided first
# an experiment that DISPROVES the hypothesis is the valuable one
# What to inject, in order of value
LATENCY <- start here. Most code has an error path and no
"taking forever" path. Missing timeouts ->
exhausted pools -> total unavailability.
errors (500, 429) / unavailability / PARTIAL failure (worse than
total: retries land on the broken one) / packet loss / disk full /
memory pressure / CPU saturation / clock skew / instance loss /
your dependency's dependency# The patterns you are verifying
timeout on every network call, shorter than the caller's
retry + backoff retry only the retryable — AND ADD JITTER, or
+ JITTER every client retries in lockstep
circuit breaker verify it opens AND that it CLOSES again
bulkhead a separate pool per dependency
fallback a degraded answer that cannot itself fail
load shedding reject fast rather than queueing
idempotency a retried request must not charge twice
# most commonly broken: missing jitter, and a breaker that never
# closes — which turns a 30-second outage into a permanent one
# Before any production experiment, all three:
# you can observe the steady state live
# you can stop the fault in seconds
# the people who would be paged know it is happening
# ...and run it on a Tuesday morning, not a Friday evening
# Game days test the RESPONSE, not just the software
detection time / did the alert reach a human / was it actionable /
could on-call diagnose from existing dashboards / was the runbook
accurate / had anyone run the mitigation before / rollback rehearsed /
did anyone know who to tell@respx.mock
def test_checkout_survives_slow_recommendations():
respx.get(RECOMMENDATIONS).mock(
side_effect=httpx.TimeoutException("timeout")
)
result = checkout(basket=basket_with_two_items())
assert result.status == "completed" # the order still went
assert result.recommendations == [] # degraded, not failed
@respx.mock
def test_checkout_does_not_wait_forever_for_recommendations():
respx.get(RECOMMENDATIONS).mock(side_effect=slow_response(delay=30))
start = time.monotonic()
checkout(basket=basket_with_two_items())
elapsed = time.monotonic() - start
assert elapsed < 2.0 # there IS a timeout, and it is short# 3 seconds of added latency on outbound traffic
tc qdisc add dev eth0 root netem delay 3000ms
tc qdisc del dev eth0 root netem # remove
# 5% packet loss
tc qdisc add dev eth0 root netem loss 5%
# in Docker Compose: kill a dependency mid-test
docker compose pause payments
docker compose unpause payments
# in Kubernetes: remove an instance
kubectl delete pod -l app=recommendations --field-selector=...# In an ordinary integration test
respx.get(URL).mock(side_effect=httpx.TimeoutException("timeout"))
assert result.status == "completed" # degraded, not failed
# The test people never write, and should
assert elapsed < 2.0 # there IS a timeout, and it is shorttc qdisc add dev eth0 root netem delay 3000ms # add latency
tc qdisc add dev eth0 root netem loss 5% # packet loss
docker compose pause payments # kill a dependency
# tools: Toxiproxy (easiest, sits in an integration test),
# Chaos Mesh, LitmusChaos, AWS FIS, Gremlin