Scaling a Suite: Parallelism and Sharding
An hour-long suite gets skipped. Parallel workers, sharding, test selection by impact, and the isolation guarantees you need before any of it is safe.
An hour-long suite gets skipped. Parallel workers, sharding, test selection by impact, and the isolation guarantees you need before any of it is safe.
An hour-long suite gets skipped. That is the whole problem, and it is a behavioural fact rather than an engineering one: people do not wait an hour for feedback, so they push without running it, and a suite that runs after the fact protects much less than one that runs before.
Making a large suite fast is a solved problem, in a specific order. By the end of this lesson you will know that order, why the third step is dangerous without the first, and what to measure so the runtime stops creeping up again.
Suites get slow in unevenly distributed ways, and intuition is unreliable. Almost always a small number of tests account for most of the runtime.
pytest --durations=25 # the 25 slowest tests
pytest --durations=0 --durations-min=1.0 # everything over a secondnpx vitest run --reporter=verbose
npx playwright test --reporter=json | jq '.suites[].specs[] |
{title, duration: .tests[0].results[0].duration}' | sort=========================== slowest 10 durations ==========================
94.21s test_import_ten_thousand_rows
88.04s test_full_reindex_after_bulk_update
41.55s test_checkout_journey_chromium
39.87s test_checkout_journey_webkit
12.03s test_report_generation_for_a_year
8.44s test_migrations_from_zero
...
0.31s test_discount_boundariesTwo facts usually fall out. A handful of tests dominate — those two import tests at 94 and 88 seconds are three minutes of a five-minute suite. And the per-test setup, summed across hundreds of fast tests, is often larger than any individual slow test.
# is setup the real cost? compare with an empty test body
pytest --setup-plan | head -40Everything after this depends on tests being independent, and attempting parallelism without it produces intermittent failures that are attributed to the parallelism rather than to the coupling.
The requirements are the ones from the intermediate course, and they are worth restating as preconditions:
[ ] every test creates the data it needs, with unique values
[ ] no test depends on another having run
[ ] no test leaves state behind — transaction rollback, truncation,
or a schema per worker
[ ] no shared fixed identifiers: emails, ids, file paths, ports
[ ] no shared mutable session-scoped fixture
[ ] caches, queues and clocks reset between testsProve it before you scale it:
pytest --random-order # order dependence
pytest -n 4 # parallel safety
pytest --count 2 # repeat safetyIf any of those fails, that is the work. Parallelising a coupled suite converts a deterministic problem into a probabilistic one, which is strictly worse — the failure now appears one run in eight and gets re-run away.
Multiple workers on one machine, each running a subset. This is the cheapest large win available.
Several workers, one machine.
One flag, and you use cores you were already paying for.
The distribution strategy matters more than the worker count: grouping by module means expensive per-module setup happens once per worker rather than once per test.
Several machines, one suite.
Real money, and a fixed startup cost paid on every shard — checkout, install, build, container pull.
Worth it only once a single machine is genuinely saturated.
pytest -n auto # pytest-xdist, one per core
pytest -n 8 --dist loadscope # group by class or modulenpx vitest run --pool=threads # threads, for pure-logic tests
npx jest --maxWorkers=50%
npx playwright test --workers=4The distribution strategy matters more than the worker count. loadscope
groups tests by module or class so that per-module setup happens once per
worker rather than once per test — which, for a suite with expensive
class-level fixtures, can matter more than the parallelism itself.
Each worker needs its own isolation for anything shared:
@pytest.fixture(scope="session")
def database_url(worker_id): # xdist provides this
name = f"test_{worker_id}" # gw0, gw1, gw2...
create_database(name)
run_migrations(name)
yield url_for(name)
drop_database(name)That per-worker schema is the pattern from the isolation lesson, and it is what makes parallelism safe for anything touching a database.
Two costs to know. Worker start-up is not free — for a suite of fast
tests, eight workers each importing your application may be slower than two.
And more workers than cores contend, so -n auto is usually right and
-n 32 on an eight-core machine is not.
When one machine is not enough, split the suite across several CI jobs.
strategy:
fail-fast: false
matrix:
shard: [1, 2, 3, 4]
steps:
- run: npx playwright test --shard=${{ matrix.shard }}/4pytest --splits 4 --group ${{ matrix.shard }} # pytest-splitTwo things decide whether sharding helps.
Balance the shards by duration, not by count. Splitting alphabetically puts both ninety-second import tests in one shard, and that shard becomes the runtime. Tools do this with recorded timings:
pytest --store-durations # record, once
pytest --splits 4 --group 1 # then split by durationfail-fast: false, so one failing shard does not cancel the others. You
want the whole picture from one run, not to discover a second failure after
fixing the first.
The limits are worth stating: fixed per-job overhead — checkout, dependency install, container start — is paid per shard, so past a certain point adding shards adds overhead without reducing wall-clock time. Caching and a prebuilt image are what push that point further out.
The largest saving available is not running tests that cannot be affected by the change.
npx nx affected --target=test # monorepo dependency graph
npx turbo run test --filter=...[HEAD^]
pytest --testmon # per-test coverage mapping
npx jest --changedSince=mainTwo mechanisms, with different guarantees.
A dependency graph — Nx, Turborepo, Bazel — knows which packages depend on which, so a change to one package runs its tests and its dependents'. This is sound: the graph is derived from real imports.
Coverage-based selection — testmon, or Jest's --changedSince — records
which tests executed which lines, then runs only the tests touching changed
lines. Effective and not sound: it cannot see dependencies that do not appear
as executed lines. Configuration files, data files, templates, environment
variables, and anything loaded dynamically.
Bad — selection by impact as the only gate:
- run: pytest --testmon # only affected tests, on every push
# ...and nothing ever runs the full suite.Good — fast selection on push, the whole suite before merge:
# on every push to a branch
- run: pytest --testmon
# on the pull request, and before merge — required
- run: pytest -n auto --random-orderThe bad version is fast and eventually wrong. A change to a YAML configuration file, a template, or a dependency version affects behaviour without changing any line that coverage attributed to a test, so the affected tests are not selected and the change merges untested. It fails in the way that is hardest to attribute: weeks later, in an area nobody connected to that commit.
The good version uses selection for the fast inner loop, where being occasionally incomplete costs a few minutes, and runs everything at the gate that actually protects the main branch. Selection accelerates feedback; it does not replace the gate.
Once measured, parallelised and sharded, the remaining lever is when things run — which the CI lesson introduced and which becomes essential at scale.
on save the unit tests for the file you are editing
under 2 seconds, in the editor
on push affected unit + integration
under 3 minutes
on pull request all unit + integration + API, parallel,
random order — the required gate
under 10 minutes
nightly full e2e across browsers, large-data tests,
soak, mutation on the risk model's top rows,
long property-based runs
hours, and nobody is waiting
weekly the expensive analyses: full mutation testing,
long fuzzing runs, a capacity testTwo rules keep the tiers honest. A nightly failure is triaged the next morning, or the tier is decoration. And each tier's coverage is written down, so nobody believes the pull-request gate covers what only the nightly run does.
Runtime creeps back. A test added here, a fixture widened there, and in a year you are back to an hour.
[ ] the suite's runtime is a tracked metric with a graph
[ ] a budget: the pull-request gate must stay under 10 minutes
[ ] the slowest 20 tests are reviewed monthly
[ ] a new test slower than a threshold needs a justification
[ ] quarterly pruning: has this test ever failed? is it unique?
[ ] flaky tests fixed or quarantined, because retries multiply
runtimeThat last point compounds unpleasantly: with retries: 2, a suite where 2%
of tests flake spends a meaningful fraction of its runtime re-running them.
Fixing flakiness is a performance optimisation as well as a trust one.
# The order matters. Do them in this sequence.
# 0. MEASURE
pytest --durations=25
pytest --durations=0 --durations-min=1.0
pytest --setup-plan # is per-test setup the real cost?
# a handful of tests usually dominate. Fix those FIRST — deleting
# a needless 90-second test beats a week of infrastructure work.
# 1. INDEPENDENCE — non-negotiable, and first
unique data per test / no order dependence / no state left behind /
no shared fixed emails, ids, paths, ports / no mutable session
fixtures / caches, queues and clocks reset
prove it: pytest --random-order ; pytest -n 4 ; pytest --count 2
# parallelising a coupled suite turns a deterministic failure into
# a probabilistic one, which is strictly worse
# 2. PARALLELISM within a machine
pytest -n auto one worker per core
pytest -n 8 --dist loadscope group by module: setup once
npx playwright test --workers=4
npx vitest run --pool=threads
# per-worker isolation: a schema per worker_id
# worker start-up is not free; more workers than cores contend
# 3. SHARDING across machines
--shard=${{ matrix.shard }}/4 Playwright
pytest --store-durations record once...
pytest --splits 4 --group 1 ...then split BY DURATION
fail-fast: false one bad shard must not cancel
the rest
# per-job overhead is paid per shard — caching and prebuilt images
# push the useful limit further out
# 4. RUN LESS — the biggest saving, with a caveat
dependency graph nx affected, turbo --filter SOUND: real imports
coverage-based pytest --testmon, jest --changedSince
effective, NOT sound: blind to config files,
templates, data files, dynamic loading
# use selection for the fast inner loop
# run EVERYTHING at the required gate before merge
# and LOG what was skipped — "412 of 3,190 (affected)"
# 5. TIERS
on save the file's unit tests < 2s
on push affected unit + integration < 3 min
pull request all unit + integration + API < 10 min REQUIRED
nightly full e2e, large data, soak, mutation, long property
weekly full mutation, long fuzzing, capacity
# a nightly failure triaged next morning, or the tier is decoration
# write down what each tier covers
# Keep it fast
track runtime as a metric / a budget for the gate / review the
slowest 20 monthly / justify slow new tests / prune quarterly /
fix flakes — with retries: 2, a 2% flake rate costs real runtimeThe suite is fast enough to be a gate. The next lesson is about the gate itself: which checks block a merge and which only inform, what happens on a red main branch, and why an unreliable gate is worse than no gate.
Before that, run the durations command on your own suite and look at the top ten. In most codebases that list contains at least one test whose cost nobody has ever noticed, and removing or fixing it is the fastest win in this lesson.