Quality Gates and Release Decisions
Turning test results into a decision: which gates block, which only inform, what happens on a red main, and why an unreliable gate is worse than no gate.
Turning test results into a decision: which gates block, which only inform, what happens on a red main, and why an unreliable gate is worse than no gate.
Every test in this course produces information. A gate is where that information becomes a decision: this change may merge, this build may deploy, this release may go out.
Gates are where testing meets delivery, and where most of the friction between the two lives. By the end of this lesson you will know which checks should block and which should only inform, why an unreliable gate is worse than no gate, and how to make a release decision that is a decision rather than a feeling.
The first distinction, and getting it wrong is the source of most gate problems.
Prevents progress.
Must be fast, reliable, and about something genuinely unacceptable.
If a production incident is waiting and this check is red, do you stop? If the honest answer is no, it must not block.
Reports, and stops nothing.
Can be slow, occasionally noisy, and about something worth knowing rather than something worth halting for.
This is also where a flaky check belongs until it is fixed.
BLOCK on
compilation and type errors
lint errors (not warnings)
unit and integration test failures
a required security scan at high severity
a contract test that a consumer would fail
INFORM on
coverage change
bundle size change
performance benchmark drift
new low-severity dependency advisories
complexity and code-smell metrics
visual differences awaiting reviewA gate that gets overridden trains everyone that gates are advisory, and that lesson is not one you can un-teach.
Four places, with different costs and therefore different contents.
1. PRE-COMMIT (seconds)
format, lint the changed files, secret scan
local, fast, easy to bypass — and that is acceptable here
2. PULL REQUEST (minutes) — the main gate
type-check, unit, integration, API tests, security scan,
contract verification
REQUIRED in branch protection. Under 10 minutes.
3. PRE-DEPLOY (minutes)
the artefact is built, migrations apply cleanly,
can-i-deploy against consumer contracts
4. POST-DEPLOY (minutes)
a smoke test against the deployed environment
a canary comparison before promoting
...and this one can trigger an automatic rollbackGate two is where most of the value is, and gate four is the one most often missing. A deployment that succeeded and a system that works are different claims, and only a post-deploy check distinguishes them.
A passing job that nobody enforces is not a gate. Three things have to be true:
[ ] the check is REQUIRED in branch protection, so red blocks merge
[ ] the branch must be up to date before merging, so a change is
tested against what it will actually merge into
[ ] nobody can bypass it silently — an override is possible,
logged, and requires a reasonThe second is easy to overlook and causes a specific failure: two changes each pass on their own and break when combined. This is semantic conflict — no textual conflict, no merge marker, and a broken main branch. Requiring the branch to be current before merge is what catches it, and merge queues are the scaled-up version: each change is tested against the actual merge result, serially.
The third matters because emergencies are real. An override that exists, requires a reason, and is recorded is honest. One that is impossible gets worked around in ways nobody sees.
The rule that makes everything else work: when the main branch is red, that is the team's most urgent problem.
The reason is compounding. Every subsequent pull request now inherits a failure that is not theirs, so nobody can tell whether their own change is safe, and within an hour three people are debugging the same unrelated failure. Two more changes merge on top and the original cause is buried.
[ ] a red main branch notifies a human immediately
[ ] the person whose change broke it fixes forward or reverts,
quickly — reverting is not a failure, it is the fast path
[ ] nothing else merges until it is green
[ ] if it cannot be fixed in minutes, revert and diagnose after"Revert first" is worth adopting explicitly, because the instinct is to fix. Fixing takes an unknown amount of time; reverting takes two minutes and returns everyone else to a working baseline.
For anything not continuously deployed, someone decides a release goes out. That decision should rest on evidence, and the plan lesson's exit criteria are where the evidence is defined.
Bad — a decision made on confidence:
Release 2.15.0 — go / no-go
QA: "We've tested it pretty thoroughly, I'm fairly happy."
Eng: "The pipeline's green apart from the flaky e2e ones."
Product: "We need it today."
Decision: ship.Good — a decision made on stated criteria:
Release 2.15.0 — go / no-go
Exit criteria (agreed 3 March):
[x] all pull-request gates green on the release commit
[x] full regression suite green (nightly, 04:12 today)
[x] both exploratory charters run and reported
[ ] no open Critical or High defects
-> BUG-604 (High): import duplicates rows when a file is
re-uploaded after a partial failure. Not fixed.
[x] p95 checkout latency within 10% of the previous release
[x] rollback rehearsed on staging (2 March, 4m 20s)
Residual risk:
Concurrent imports untested. Localised date formats untested;
EU customers will see incorrect dates in the import preview.
Recommendation: ship behind the flag, with import disabled for
the four accounts that use bulk import, and BUG-604 as the first
fix in 2.15.1. Import re-enabled when it lands.
Decision: ship on that basis — Product, 3 March.
Accepted risk owner: Product.The bad version has three problems beyond vagueness. "Pretty thoroughly" is not checkable, so nobody can disagree with it on evidence. "Apart from the flaky ones" means a real failure is indistinguishable from noise. And the deadline did the deciding, which means the same conversation will produce the same outcome next time regardless of what the tests found.
The good version states what was agreed weeks earlier, shows one criterion unmet with the specific defect, quantifies what remains unknown, and proposes a mitigation. Then the decision is a recorded choice by the person entitled to make it. Note that it still ships — the point of a gate is not to prevent shipping, it is to make what is being accepted visible.
The most advanced form is a gate on how the last release actually behaved, which the observability lesson set up.
An error budget policy turns reliability into a gate automatically:
budget healthy (>50% remaining) normal releases; run experiments
budget low (<25% remaining) releases require review; risky
changes wait
budget exhausted feature freeze; reliability work
only, until the window rolls overThat is a quality gate whose input is what users experienced rather than what a test asserted, and it has a property no test-based gate has: it cannot be gamed by writing better tests.
Two related production-driven gates worth knowing:
Automatic rollback on canary failure. If the canary's error rate or business metric is worse than the current version by a threshold, the rollout reverses without a human. Argo Rollouts and Flagger do this.
A deployment freeze window, for a genuinely risky period — a launch, a peak trading day, a holiday when nobody is on call. Worth having as a policy rather than as an argument each time.
Four patterns to recognise and avoid, because each converts a gate into a tax.
A gate on a metric rather than on a defect. "Coverage must be 80%" blocks a correct change for a number, and the coverage lesson explained what that produces. Gate on failures, inform on metrics.
A manual approval that is always given. A sign-off nobody has ever withheld is a delay pretending to be a control. Either it can say no, or it should be removed.
A gate slower than the work it protects. A four-hour suite on every pull request means people batch changes to avoid it, and large batches are riskier than small ones — so the gate has made delivery less safe.
A gate that reports what nobody reads. An informing check whose output nobody opens is pure cost. Either surface it where a decision is made, or turn it off.
# Blocking vs informing
BLOCK compilation, type errors, lint errors, unit and integration
failures, high-severity security findings, contract
verification
INFORM coverage change, bundle size, benchmark drift, low-severity
advisories, complexity, visual diffs awaiting review
# TEST: would you hold a critical fix for this? If no, it must
# not block — it will be overridden, and then nothing blocks
# An unreliable gate is worse than no gate
# re-run -> override -> ignore. Move a flaky check to INFORMING
# until it is fixed.
# Where the gates are
pre-commit seconds: format, lint changed files, secret scan
pull request minutes: the MAIN gate. Required. Under 10 minutes.
pre-deploy the artefact builds, migrations apply, can-i-deploy
post-deploy smoke test the deployed environment; canary comparison
<- most often missing. "Deployed" != "works".
# Making a gate real
required in branch protection
branch up to date before merge <- catches SEMANTIC conflicts:
two changes that each pass and
break together
(merge queues are the scaled version)
overrides possible, logged, and requiring a reason
# A red main branch is an emergency
notify immediately / fix forward or REVERT quickly / nothing else
merges until green / cannot fix in minutes -> revert and diagnose
# reverting is the fast path, not a failure
# The release decision — evidence, not confidence
state the exit criteria AGREED IN ADVANCE, with each one ticked or not
name the specific defect behind any unmet criterion
quantify the RESIDUAL RISK
recommend, with a mitigation
record who decided and who owns the accepted risk
# "pretty thoroughly" cannot be disagreed with on evidence
# "green apart from the flaky ones" hides real failures
# the point of a gate is not to prevent shipping — it is to make
# what is being accepted visible
# Gates driven by production
error budget policy healthy -> normal; low -> review; exhausted ->
freeze. Cannot be gamed by writing better tests.
automatic rollback canary worse than current by a threshold
freeze windows a launch, a peak day, nobody on call
# Gates that damage delivery
a gate on a METRIC rather than a defect (coverage thresholds)
an approval that is always given a delay, not a control
a gate slower than the work -> people batch changes
-> larger, riskier batches
an informing check nobody reads pure costThe next lesson takes on the case where the fundamental assumption of every gate breaks down: systems whose output is not deterministic, where the same input gives a different answer and equality assertions stop working.
Before that, list your own blocking checks and apply the critical-fix test to each. Anything you would wave through during an incident is not really a gate, and moving it to informing makes the remaining set mean something.