Observability as a Testing Tool
The system tells you what your tests could not. Assertions on telemetry, service level objectives as continuous tests, and closing the loop from an incident back to a test case.
The system tells you what your tests could not. Assertions on telemetry, service level objectives as continuous tests, and closing the loop from an incident back to a test case.
A test suite checks the properties somebody thought of, at the moment the suite runs. Production has properties that must hold continuously, against inputs nobody imagined, and the only instrument that reaches them is the system's own telemetry.
This lesson is about treating that telemetry as a testing tool: asserting on it in tests, running service level objectives as continuous checks, and closing the loop from an incident back to something that would have caught it.
Is the output correct for this input?
At build time, on inputs you chose.
It can prove a discount calculation is right for every boundary you identified — and says nothing about the ones you did not.
Is it behaving correctly right now?
Continuously, on inputs nobody chose.
Only this can tell you that 4% of yesterday's orders took a code path nobody knew existed.
The three signals, and what each is for:
metrics aggregate numbers over time: request rate, error rate,
latency distribution, orders per minute. Cheap, and they
answer "is something wrong?"
traces the path of one request through every service, with
timings. Answers "where is the time going, and what
called what?"
logs individual events with context. Answers "what exactly
happened to this one request?"The pattern in practice: a metric alerts, a trace localises, a log explains.
Telemetry designed as an afterthought answers nothing. The discipline is to emit the dimensions you will want to slice by, at the point where the information exists.
from opentelemetry import trace
tracer = trace.get_tracer(__name__)
def place_order(basket, user):
with tracer.start_as_current_span("place_order") as span:
span.set_attribute("account.id", user.account_id)
span.set_attribute("account.plan", user.account.plan)
span.set_attribute("basket.item_count", len(basket.items))
span.set_attribute("basket.total_pence", basket.total_pence)
span.set_attribute("payment.provider", provider_for(user))
span.set_attribute("feature.new_checkout",
flags.enabled("new-checkout", user=user))
order = create_order(basket, user)
span.set_attribute("order.id", order.id)
return orderEvery attribute there is a dimension you can group by later. The flag attribute is what lets you compare the canary against the old path — which the previous lesson said was essential, and which is only possible if somebody emitted the flag state.
Three rules for what to emit:
High cardinality is a feature, not a cost problem. The account id, the order id, the request id. Aggregate metrics cannot carry them, and they are exactly what you need to find the one broken customer. This is what traces are for.
Emit the business event, not only the technical one. order_completed,
payment_declined, import_finished with a row count. A dashboard of HTTP
status codes cannot tell you that orders stopped.
Never emit secrets or personal data. Tokens, passwords, card numbers, and personal data beyond what you have a basis to process. Telemetry goes to a third party and is retained; the security lesson's rule about logs applies to every signal.
Instrumentation is code, and it breaks like code — silently, because nothing fails when a span attribute disappears. So test it.
def test_placing_an_order_records_the_provider_and_flag(
span_exporter, user_factory
):
user = user_factory(plan="pro")
place_order(basket_with_two_items(), user)
span = one_span_named(span_exporter, "place_order")
assert span.attributes["account.plan"] == "pro"
assert span.attributes["payment.provider"] == "stripe"
assert "feature.new_checkout" in span.attributes
assert "card_number" not in span.attributes # no secretsTwo things this catches that nothing else does. A refactor that drops an attribute — after which your canary comparison silently has no way to distinguish versions. And a secret leaking into telemetry, which is worth a dedicated assertion because the failure is invisible and serious.
The same idea for metrics and for logs:
def test_a_declined_payment_increments_the_counter(metrics_reader):
with declined_gateway():
place_order(basket_with_two_items(), a_user())
assert counter_value(metrics_reader, "payments.declined") == 1A service level objective is a target for a measurable property of the service, over a window. It is the closest thing production has to an assertion.
SLI the indicator: the proportion of checkout requests that
complete successfully in under 1 second
SLO the objective: 99.5% of them, over 28 days
error budget the 0.5% you are allowed to fail — about 3.4 hours
in 28 daysThe error budget is what makes an SLO useful rather than decorative, because it converts reliability into a quantity you can spend.
budget remaining, plenty ship faster, take more risk, run
experiments
budget nearly gone stop feature work, fix reliability
budget exhausted a freeze, by prior agreementThat is a quality gate driven by production behaviour rather than by a test result, and it is the most honest one there is — it measures what users actually experienced.
Two habits make SLOs work:
Pick indicators users would recognise. "Checkout completes in under a second" is one. "CPU under 80%" is not — a user has no opinion about CPU, and a system can be at 95% CPU and perfectly fine.
Alert on burn rate, not on the threshold. An alert saying "the error rate is above 0.5%" fires constantly during a blip. An alert saying "at the current rate, this month's budget will be exhausted in six hours" is actionable and does not cry wolf. Two burn-rate alerts — a fast one and a slow one — is the standard arrangement.
Bad — an incident that ends when service is restored:
INCIDENT-142
Checkout errors spiked to 40% for 22 minutes.
Cause: the payment provider began returning 429s and our retry
loop had no backoff, so we hammered them.
Resolution: added backoff. Deployed. Closed.Good — the incident produces tests and telemetry, not only a fix:
INCIDENT-142
Checkout errors spiked to 40% for 22 minutes.
Cause: the payment provider began returning 429s and our retry
loop had no backoff, so we hammered them.
Fix: exponential backoff with jitter, and a circuit breaker.
WHAT WOULD HAVE CAUGHT IT
- No test injected a 429. Added: a rate-limit fault test
asserting backoff intervals grow and total attempts are capped.
It fails against the old code.
- No test asserted a bounded total time. Added: the slow-dependency
test from the resilience lesson.
- Detection took 9 minutes because the alert was on total error
rate, not on the payment-provider error rate. Added an SLI for
provider call success, with a burn-rate alert.
- The trace had no attribute for the provider's response code, so
diagnosis took 11 of the 22 minutes. Added, plus a test that
asserts it is present.
CLASS
Every outbound dependency call. Audited the other four: two also
lack backoff. PROJ-880.The bad version fixes one instance. Nothing prevents the same class in the other four dependencies, nothing improves the nine-minute detection time, and the next occurrence will be diagnosed just as slowly because the missing trace attribute is still missing.
The good version treats the incident as evidence about three systems: the code, the test suite, and the telemetry. It produces a regression test that fails against the old behaviour, a better indicator, a better trace, and a list of the same defect elsewhere. That is the escape-route habit from the intermediate course, applied at full strength.
Four questions worth asking of telemetry deliberately, rather than only during incidents.
Which code paths are never taken? If a branch has had no traffic in three months, it is either dead or reachable only by a case you do not understand. Both are worth knowing, and it is a much better dead-code detector than reading.
What inputs are real? The distribution of basket sizes, file sizes, page depths, retry counts. Compare it with your test data — the gap is usually startling, and it is a direct instruction for what your fixtures should contain.
Where is the time actually going? A trace of a slow request beats any amount of reasoning about which query is expensive, and it frequently contradicts the reasoning.
What errors are being swallowed? Search for handled exceptions that never
surface. A try/except that logs and continues is invisible in every test and
may be firing thousands of times a day.
Each of those turns telemetry into an input for testing work, which is the loop this lesson is really about: production tells you what to test next.
# Two instruments, two questions
tests "is the output correct for inputs I chose?" at build time
telemetry "is it behaving correctly NOW, on real inputs?"
# The signals
metrics aggregates over time -> IS something wrong
traces one request across services, with timings -> WHERE
logs individual events with context -> WHAT exactly happened
# a metric alerts, a trace localises, a log explains
# monitoring = known failure modes
# observability = asking questions you did not plan for
# Instrument for the questions you will ask
span.set_attribute("account.plan", ...) dimensions to group by
span.set_attribute("feature.new_checkout", ...) <- makes canary
comparison possible
rules
HIGH CARDINALITY is a feature: account id, order id, request id
emit the BUSINESS event, not only the technical one
never emit secrets or personal data — telemetry is retained by
a third party
# Test the instrumentation — it breaks silently
assert span.attributes["payment.provider"] == "stripe"
assert "card_number" not in span.attributes
assert counter_value(reader, "payments.declined") == 1
# catches a refactor that drops an attribute (after which your
# canary cannot distinguish versions) and secrets in telemetry
# also: exercise the ALERT path, not just the metric
# SLOs as continuous tests
SLI the proportion of checkouts completing in under 1s
SLO 99.5% over 28 days
error budget the 0.5% you may fail — ~3.4 hours in 28 days
budget plenty -> ship faster, take more risk
budget nearly gone -> stop feature work, fix reliability
exhausted -> a freeze, by prior agreement
pick indicators a USER would recognise — not CPU under 80%
alert on BURN RATE, not the threshold: "budget gone in 6 hours"
(a fast and a slow burn alert is the standard pair)
# Close the loop on every incident
what would have caught it? -> a test that fails against the old code
why did detection take N min? -> a better indicator and alert
why was diagnosis slow? -> a missing trace attribute
where else does this exist? -> audit the class, file the work
# Ask telemetry these, deliberately
which code paths are never taken? dead, or misunderstood
what inputs are real? compare with your test data
where is the time actually going? traces beat reasoning
what errors are being swallowed? handled exceptions firing
thousands of times a dayThe next lesson is about the suite itself at scale: parallelism, sharding, test selection by impact, and the isolation guarantees you need before any of it is safe — because an hour-long suite gets skipped, and a skipped suite protects nothing.
Before that, pick the last incident you were involved in and write the "what would have caught it" section. Four lines, and each one is a piece of work better justified than anything on a backlog.