Killing Flaky Tests
A flaky test is a bug report about your suite. The four real causes — timing, shared state, ordering, real non-determinism — how to reproduce each, and why retries are not a fix.
A flaky test is a bug report about your suite. The four real causes — timing, shared state, ordering, real non-determinism — how to reproduce each, and why retries are not a fix.
A test fails. Somebody re-runs it and it passes. Everyone moves on.
That moment is the most expensive one in a test suite's life, because it establishes a habit: a red result no longer means anything. Once re-running is normal, a real failure and a flake are indistinguishable, and the suite has stopped providing information while continuing to cost time.
By the end of this lesson you will know the four real causes of flakiness, how to reproduce each deliberately, and why retries are a way to keep the pipeline usable rather than a way to fix anything.
A flaky test is one that passes and fails against the same code. It is non-deterministic, and non-determinism has a cause — there is no such thing as a test that fails randomly for no reason.
The cause is in one of three places, and it is worth being honest that the first is common.
The test is telling the truth, intermittently.
A race condition. An operation that usually finishes in time and sometimes does not.
Under load, a real user hits it — which is exactly what the test is doing.
It waits for the wrong thing.
Or depends on order, or assumes it has exclusive use of something.
Shared, slow, or ticking.
A shared resource, an overloaded CI machine, a clock, a network.
Deleting or retrying a flaky test without diagnosing it may be discarding a genuine production defect.
The largest category. The test proceeds before the thing it needs has happened.
symptoms fails more often in CI than locally
fails more often under parallel load
passes when you watch it with --headed
passes when you add a sleepThat third symptom is the giveaway: watching it slows everything down enough for the race to resolve.
The fix is to wait for a condition, never a duration, which the
Playwright lesson covered. Beyond expect retrying automatically:
// wait for the network, not for the clock
await page.waitForResponse(
(r) => r.url().includes('/api/contacts') && r.status() === 201,
);
// wait for a state the app actually reaches
await expect(page.getByRole('status')).toHaveText('Saved');Where the code under test is asynchronous, the same rule applies at other levels: poll for the outcome rather than sleeping.
def wait_until(predicate, timeout=5.0, interval=0.05):
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
if predicate():
return
time.sleep(interval)
raise AssertionError("condition not met within timeout")There is a specific case worth naming, because it is the most common
timing flake in web testing: a background job. The request returns
202, a worker processes it, and the test asserts on the result
immediately. The fix is either to poll for the outcome, or — better — to run
the worker synchronously in tests, so the behaviour is deterministic.
The subject of the isolation lesson, and the second-largest category.
symptoms passes alone, fails in the suite
fails only in parallel
fails only in a particular order
the failure moves to a different test when you add oneThat last symptom is diagnostic: if adding an unrelated test changes which test fails, the tests are sharing something.
pytest path::the_failing_test # alone: does it pass?
pytest --random-order # order dependence
pytest -n 4 # parallel safety
npx playwright test --workers=1 # does serialising fix it?If serialising fixes it, you have shared state; the remaining work is finding what. The usual culprits: a fixed email address or record id, a cache, a file path, a global counter, a session-scoped fixture something mutates.
The fix is always the same shape — create what you need, uniquely:
user = create_user(email=f"user-{uuid4()}@example.com")Related to shared state and worth separating, because the fix is different: here one test needs another to have run.
symptoms fails when run alone
passes only in the full suite
breaks when a test earlier in the file is deleted"Fails when run alone" is the inverse signature, and it means the test has no setup of its own — it is relying on data another test left behind.
The fix is to give it its own arrangement. If two tests genuinely must happen in sequence — create, then verify the created thing — they are one test that was split for tidiness, and joining them is correct.
Something in the test's inputs varies by nature.
time a test that fails at midnight, at month end, or in
another time zone
randomness an unseeded generator, an unordered set
concurrency a real race in the application <- a REAL BUG
external a third-party service that is slow or down
resources a port already in use, a disk filling upTime and randomness are controllable, and controlling them is the fix:
from freezegun import freeze_time
@freeze_time("2026-03-15 12:00:00")
def test_a_receipt_shows_todays_date():
assert receipt().date_line == "15 March 2026"random.seed(12345) # reproducible
sorted(result) # a set has no orderExternals are what the faking lesson was for — a test that depends on a third party being up is a test that fails when they deploy.
And the concurrency case is the one to take seriously. A test that occasionally sees a duplicate record, or a lost update, may have found a real race. The right response is to make it reproducible — hammer it in a loop, add contention — and then fix the application.
You cannot fix what you cannot reproduce, and the trick is to amplify whatever varies.
pytest --count 50 path::test_name # pytest-repeat
npx playwright test --repeat-each=50 -g "create a contact"
pytest -n 8 # add contention
pytest --random-order --random-order-seed=12345 # reproducible orderThen make the machine less forgiving, because CI is slower than your laptop and that is often the entire difference:
# throttle the browser's network to expose timing assumptions
npx playwright test --project=chromium --workers=8# add artificial latency to a fake, so races surface
respx.post(CHARGES).mock(side_effect=slow_response(delay=1.5))A flake that reproduces once in fifty runs is a flake you can now bisect. A flake you have never reproduced is one you are guessing at.
Bad — retries as the fix, applied to everything:
export default defineConfig({
retries: 3,
});Good — retries to keep the pipeline usable, plus a mechanism that surfaces what retried:
export default defineConfig({
retries: process.env.CI ? 2 : 0,
reporter: [['html'], ['github'], ['./flaky-reporter.ts']],
use: { trace: 'on-first-retry' },
});The first configuration hides the problem completely. A test that fails twice and passes on the third attempt reports as green, so nobody learns it is unreliable — and the underlying cause, which may be a real race, is now invisible. Meanwhile every genuine failure takes four times as long to report.
The second keeps the build usable while making flakiness visible: a reporter that records which tests needed a retry, a trace captured on the first retry so the evidence exists, and no retries locally so a developer sees the failure honestly.
Then treat the list as a defect queue:
[ ] every test that needed a retry is recorded, with its trace
[ ] the list is reviewed weekly, like a bug backlog
[ ] a test that flakes repeatedly is fixed or quarantined
[ ] quarantined tests have an owner and a date, not a permanent skipMost flakiness is designed in, and a handful of habits prevent the majority of it:
never sleep wait for conditions
never index .nth(2) depends on data and order
create your own data unique values, every test
one assertion's worth a test doing five things has five chances
control time freeze it; never assert on "now"
seed randomness or sort before comparing
fake externals a third party's deploy is not your failure
run workers synchronously in tests, so background jobs are
deterministic
prove it --random-order and -n 4 in CI, alwaysThat last line is the one that stops new flakiness arriving. Running the suite in a random order in CI turns a future intermittent failure into a deterministic one today.
# A flaky test is a bug report. The cause is in one of three places:
# the application — a real race. Do not discard this possibility.
# the test — waits wrongly, shares state, depends on order
# the environment — shared resources, slow CI, clock, network
# Diagnose by signature
fails in CI, passes locally timing
passes with --headed or a sleep timing
passes alone, fails in the suite shared state
fails only in parallel shared state
FAILS alone, passes in the suite order dependence
adding a test moves the failure shared state
fails at midnight / month end time
fails when a third party is slow external dependency
# Reproduce by amplifying
pytest --count 50 path::test_name
npx playwright test --repeat-each=50 -g "name"
pytest -n 8 add contention
pytest --random-order-seed=12345 reproducible order
# a flake you have never reproduced is one you are guessing at
# Fixes by cause
timing wait for a CONDITION, never a duration
waitForResponse, expect(...).toHaveText, poll
run background workers synchronously in tests
shared state create what you need, uniquely (uuid4)
order give the test its own setup; or join the two tests
time freeze the clock (freezegun, sinon)
randomness seed it, or sort before comparing
externals fake them
real race make it reproducible, then FIX THE APPLICATION
# Retries
retries: process.env.CI ? 2 : 0 keeps the build usable
trace: 'on-first-retry' captures the evidence
a reporter that lists what retried makes flakiness visible
# a green-after-retry result that nobody records is the problem,
# not the solution
# Treat the retry list as a defect queue
recorded with traces / reviewed weekly / fixed or quarantined
quarantine has an owner and a date — test.skip with no ticket is
lost coverage nobody remembers
# Prevent
no sleeps / no .nth() / own data / control time / seed randomness /
fake externals / run --random-order and -n 4 in CIA suite you can trust is worth putting somewhere it runs automatically, which is the next lesson: what belongs on a pull request against what belongs nightly, caching, artefacts on failure, and keeping the pipeline honest.
Before that, add a random-order run to your own CI configuration. It is a one-line change, and whatever it turns up is flakiness you were going to meet anyway — on your terms rather than on a Friday afternoon.