intermediate30 minby DevFox

Running the Suite in CI

Getting from works-on-my-machine to a signal the team trusts: what runs on a pull request versus nightly, caching, artefacts on failure, and keeping the pipeline honest.

Testing
CI/CD

A test suite that runs when somebody remembers is not a safety net. It is a suite that goes red on a Tuesday, is discovered on Friday, and by then contains four failures nobody can attribute to a change.

Continuous integration turns it into a gate: every change is tested before it can merge, automatically, and a failure is attached to the commit that caused it. By the end of this lesson you will know what belongs at which gate, how to keep the pull-request run fast enough that people wait for it, and what to capture so a CI failure can be diagnosed without reproducing it.

The gates, and what each is for

Not everything runs every time. Each gate exists where its cost is affordable and its information still timely.

On every push — under 5 minutes
Lint, type-check, unit tests, integration tests. This is the gate people sit and watch.
On a pull request — under 10 minutes
The above, plus API tests and a small end-to-end set.
Past ten minutes people switch context and come back an hour later, and the feedback loop that justified the whole apparatus is gone.
On merge — plus a deploy and a smoke test
Build, deploy to staging, then smoke-test the deployed thing rather than the thing you built.
Nightly — the expensive suite
Full end-to-end across browsers, large-data tests, the slow ones, and a random-order run.
Before a release — everything, plus people
The full regression suite, and manual exploratory work no pipeline can do.

The number to defend is the second one — past about ten minutes, people stop waiting

Every decision below is in service of the second number.

A pipeline, concretely

Five things in there are worth naming.

concurrency with cancel-in-progress stops a second push from queueing behind the first. On a busy repository this is the difference between a five-minute wait and a forty-minute one.

services: postgres gives the job a real database of the same version as production — the integration lesson's requirement, satisfied by the CI platform.

A health check on the service. Without it the job starts before Postgres is accepting connections and fails on the first connection. This is the start-up race from the Docker course, in a pipeline.

-n auto --random-order runs in parallel and in a random order, which is the flakiness prevention from the last lesson applied where it does the most good.

if: always() on the artefact upload, so results are collected even when the tests failed — which is exactly when you want them.

Keeping it fast

Four levers, in order of return.

Cache dependencies. The single biggest win, and usually one line. A dependency install can be most of a short job's runtime.

Parallelise. Split by test level into separate jobs that run concurrently, and use multiple workers within each. Ten minutes of tests on four workers is under three.

Shard the slow suite. End-to-end tests split across machines:

Run only what is affected. For a monorepo, tools like Turborepo or Nx determine which packages a change touches and skip the rest — which is where the largest savings are once a repository is big.

And one thing not to do: reduce the number of tests to make the pipeline faster. That trades information for speed, and the other four levers have not run out.

Capturing evidence

A CI failure you cannot diagnose from the log is a failure you have to reproduce locally, which is where the real time goes. Four artefacts remove most of that.

Test results in a machine-readable format — JUnit XML — so the platform annotates the pull request with which tests failed rather than making somebody read a log.

Traces, screenshots and video for browser tests, which the Playwright lesson set up with trace: 'on-first-retry'.

Application logs from the run. A test failure and the server's own error at that moment, side by side, is usually the whole diagnosis.

The failing test's name in the job summary, so a reviewer sees it without opening anything.

The gate that does not block

Bad — tests run, and a failure is advisory:

Good — the tests decide whether the change can merge:

continue-on-error on a test job is the most self-defeating line in CI. The job goes green, the badge stays green, and the failures accumulate invisibly — so the suite costs its full runtime and returns nothing. It usually arrives as a temporary measure during a flaky spell and stays for a year.

The good version needs the platform's branch protection to make the check required; a passing job that nobody enforces is only marginally better than the bad version.

Two related rules worth setting up once:

No test.only or fdescribe reaches the main branch. Playwright's forbidOnly in CI, and a lint rule for the equivalents. A stray focus marker silently skips the rest of the suite and reports green.

A red main branch is an emergency. Whoever's change broke it fixes or reverts it before anything else merges, because every subsequent pull request now inherits a failure that is not theirs.

Reporting so people read it

A pipeline nobody looks at might as well be a pipeline nobody has.

That last one deserves a note: notify on a newly broken main branch, not on every pull-request failure. A channel that fires on every red run gets muted within a week, and then the notification for the one that mattered is muted too.

Making it reproducible locally

The most useful property of a pipeline is that a developer can run what it runs, with one command, before pushing.

Then CI calls the same targets. The pipeline and the local command cannot drift, because they are the same thing — and "it passes locally but fails in CI" becomes a much rarer sentence, usually about a genuine environment difference rather than about a step somebody forgot.

For the parts that genuinely need the CI environment, running the pipeline locally is possible — act for GitHub Actions, or the platform's own CLI — and worth reaching for when debugging a pipeline rather than a test.

Cheat sheet

Where to go next

The suite runs automatically and blocks bad changes. The next lesson looks at the number everyone asks about and few interpret well: coverage — what line, branch and path coverage actually measure, and why a 90% figure can sit on top of tests that assert nothing.

Before that, time your own pull-request pipeline. If it is over ten minutes, the four levers above are in priority order, and dependency caching is usually a one-line change with a surprising effect.