Testing in Production Safely
Some things are only true in production. Feature flags, canaries, shadow traffic, synthetic journeys and a rollback you have actually rehearsed.
Some things are only true in production. Feature flags, canaries, shadow traffic, synthetic journeys and a rollback you have actually rehearsed.
Some things are only true in production. The real data volume, the real traffic pattern, the real network, the real third parties, the real users doing things nobody modelled. No environment reproduces that, and the honest conclusion is not to pretend otherwise — it is to make production a place you can learn from safely.
"Testing in production" sounds reckless and describes a set of disciplined techniques: feature flags, canaries, shadow traffic, synthetic journeys and a rehearsed rollback. By the end of this lesson you will know each, and the prerequisites without which none of them is safe.
Everything before this lesson tried to learn about the system somewhere cheaper. That works until it does not, and the gaps are specific:
data two million rows behave differently from two hundred
traffic real arrival patterns, real concurrency, real
cache-hit ratios
scale twelve instances behind a load balancer, not one
integrations the real payment provider, with its real rate limits
and its real intermittent slowness
users doing things nobody wrote a requirement for
time state accumulated over years; accounts in states no
seed script creates
network real latency, real packet loss, real DNS
config production's own settings, secrets and feature flagsYou can narrow each of those and never close them. So the goal changes: rather than trying to know everything before release, make releasing safe enough that learning in production is cheap.
Four things must be true first. Without them, these techniques convert into incidents.
Observability, per version, in near real time
Error rates, latency percentiles and business metrics. If you cannot tell a canary is failing, a canary is just a slow deployment.
A kill switch
You can disable the change in seconds, without a deploy. A flag, not a revert-and-rebuild.
A small blast radius
You can expose the change to 1% rather than 100%.
A rehearsed rollback
You have actually rolled something back, recently, and know how long it takes.
This is the one teams believe and have not verified. An unexecuted rollback procedure is a hypothesis, and the failure modes are mundane and fatal: a migration that cannot be reversed, an image no longer in the registry, a configuration that changed shape.
A feature flag separates deploying code from releasing behaviour. The code ships dark; the flag decides who sees it.
if flags.enabled("new-checkout", user=current_user):
return new_checkout(basket)
return legacy_checkout(basket)That decoupling is what makes everything else in this lesson possible. Deployment becomes routine and low-risk, and release becomes a decision that can be made — and unmade — in seconds.
progressive rollout internal users -> 1% -> 5% -> 25% -> 100%
targeted release one account, one region, one plan tier
a kill switch off in seconds, no deploy
an experiment two variants, measuredThe cost is real and worth naming. Every flag doubles a code path, and the combinations multiply. Untended flags become permanent conditional branches that nobody dares remove, and a codebase with two hundred of them has an untestable number of possible states.
[ ] every flag has an owner and an expected removal date
[ ] the flag's state in production is visible somewhere
[ ] both paths are tested while the flag exists
[ ] removing the flag is a scheduled task, not an aspiration
[ ] permanent operational switches are labelled as such, and are
a different category from release flagsA canary sends a small slice of real traffic to the new version and compares it against the old on the metrics that matter.
1. deploy the new version alongside the old
2. route 1% of traffic to it
3. compare, for at least long enough to be meaningful:
error rate, latency percentiles, and a BUSINESS metric
4. if it is not worse, increase: 5%, 25%, 50%, 100%
5. if it is worse, route back to 0% — no deploy neededTwo details decide whether a canary works.
Compare against the current version, not against a threshold. "Error rate under 1%" fails to notice a canary at 0.9% when the old version is at 0.05%. Comparison is what makes small regressions visible.
Include a business metric. Error rate and latency can both look perfect while the checkout conversion rate halves — the bug is in the logic, not in the plumbing. Orders per minute, sign-ups per hour, searches with results: whatever the feature exists to produce.
old version canary (1%) verdict
error rate 0.04% 0.05% fine
p95 latency 210ms 230ms fine
orders / minute 12.1 6.8 STOPNothing in the first two rows would have caught that. The third row is the canary earning its existence.
Automated analysis of exactly this comparison is what tools like Argo Rollouts and Flagger provide, and it is worth adopting once canaries are manual and routine.
Shadow traffic — or mirroring — sends a copy of real requests to the new version while the old one continues to serve the real responses. The shadow's answers are discarded; only its behaviour is observed.
what it is good for
a rewrite of a read path, compared against the original
performance under genuinely real traffic before serving any
comparing outputs: does the new implementation agree with the old?
what it cannot do
writes. A shadowed write is a duplicate write.That last point is the whole risk. A mirrored POST /charges charges the
card twice. Shadowing is safe for read paths and requires deliberate work —
a separate database, or stubbed side effects — for anything that mutates.
Get this wrong once and it is a serious incident.
Synthetic monitoring runs your end-to-end journeys continuously against production, from outside, as a fake user.
// the same Playwright test, run every 5 minutes against production
test('a user can sign in and see their contacts', async ({ page }) => {
await page.goto(process.env.PRODUCTION_URL!);
await signIn(page, SYNTHETIC_ACCOUNT);
await expect(page.getByRole('heading', { name: 'Contacts' })).toBeVisible();
await expect(page.getByRole('row')).toHaveCount({ min: 1 });
});This is the only technique that tests the whole thing as a user meets it — including DNS, TLS, the CDN, third parties and the actual configuration.
Three practicalities. Use a dedicated synthetic account, marked as such so it is excluded from analytics and billing. Run from more than one region, because a regional network problem is invisible from one. And make the journey non-destructive, or clean up after it — a synthetic test creating a record every five minutes produces a hundred thousand rows a year.
Bad — a release that cannot be undone:
1. Deploy the migration that drops the old column
2. Deploy the code that uses the new column
3. Release to 100%Good — every step reversible, the schema change decoupled from the code change:
1. Migration: ADD the new column, nullable. Old code unaffected.
2. Deploy code that writes BOTH columns and reads the old one.
3. Backfill the new column for existing rows.
4. Deploy code that reads the new column, behind a flag. Roll out
1% -> 100%, comparing.
5. Wait. Days, not minutes.
6. Deploy code that stops writing the old column.
7. Migration: DROP the old column.The bad sequence is a one-way door disguised as a deployment. After step one the old code cannot run, so "roll back" means restoring a database — minutes to hours of downtime, and any data written since is at risk. The rollback everyone assumed existed does not.
The good sequence never has a moment where the previous version cannot run. Each step is individually reversible, the flag makes the read switch instant to undo, and step five exists precisely so that a problem is found while going back is still trivial. It is more steps and more days, and it is the difference between a routine change and an incident waiting for a bad Tuesday.
Testing in production leaves traces, and unmanaged traces become their own problem.
[ ] synthetic accounts marked and excluded from metrics, billing
and marketing emails
[ ] test data created in production is identifiable and cleaned up
[ ] canary versions removed after promotion, not left running
[ ] flags removed once fully rolled out
[ ] shadowed writes verified as impossible, not merely intended
[ ] experiments have an end date and a decision, not a permanent
50/50 splitThe last one is a common quiet failure: an experiment that was never concluded, so half of your users have been getting an unfinished variant for two years and nobody remembers why.
# Why pre-production has a ceiling
data volume / traffic patterns / scale / real integrations / real
users / accumulated state / real network / production config
# goal shifts: not "know everything first" but "make releasing
# safe enough that learning in production is cheap"
# Four prerequisites — without all of them these are incidents
1. observability error rate, latency percentiles and a BUSINESS
metric, per version, near real time
2. a kill switch off in seconds, no deploy
3. small blast radius 1%, not 100%
4. a REHEARSED you have actually rolled back, recently, and
rollback know how long it takes
# Feature flags: deploy != release
progressive rollout / targeted release / kill switch / experiment
costs: every flag doubles a path; combinations multiply
every flag: an owner and a removal date
both paths tested while it exists
removal is a scheduled task
permanent operational switches are a DIFFERENT category
# Canary
1% -> 5% -> 25% -> 100%, comparing at each step
compare against the CURRENT VERSION, not a threshold
("under 1%" misses a canary at 0.9% when old is at 0.05%)
include a BUSINESS metric — error rate and latency can be perfect
while conversion halves
tools: Argo Rollouts, Flagger
# Shadow traffic
good for: a read-path rewrite, real-traffic performance, comparing
outputs between implementations
CANNOT do writes — a mirrored POST /charges charges twice
# Synthetic monitoring
your journey tests, run continuously against production from
outside; the only check that covers DNS, TLS, CDN, third parties
and real config
a dedicated, excluded account / more than one region /
non-destructive or self-cleaning
# EXPAND AND CONTRACT — the pattern to internalise
1. add the new column, nullable
2. write both, read the old
3. backfill
4. read the new, behind a flag, 1% -> 100%
5. WAIT — days, not minutes
6. stop writing the old
7. drop the old column
# never a moment where the previous version cannot run
# a drop-then-deploy is a one-way door disguised as a deployment
# Clean up
synthetics excluded from metrics and billing / production test data
identifiable and removed / canaries removed after promotion / flags
removed / shadowed writes proven impossible / experiments concludedTesting in production depends entirely on being able to see what is happening. The next lesson is about that: observability as a testing tool — asserting on telemetry, service level objectives as continuous tests, and closing the loop from an incident back to a test case.
Before that, answer the fourth prerequisite honestly for a system you work on: when did you last roll something back, and how long did it take? If the answer is "we never have", that is the most valuable thing to change before adopting anything else in this lesson.