advanced30 minby DevFox

Ephemeral Environments and Test Infrastructure

Test infrastructure is a product with users. Environments created per branch and destroyed after, seeding them quickly, and why a shared staging box becomes a queue.

Testing
Infrastructure

One shared staging environment, six teams. Someone is always mid-test, someone else needs it for a demo, and a third person has just deployed a broken branch to it. The environment becomes a queue, and the queue is invisible in every metric anybody reports.

The alternative is environments created on demand and destroyed after. By the end of this lesson you will know what makes that feasible, why seeding is the part that decides whether it works, and how to think about test infrastructure as a product with users rather than as a chore.

Why a shared environment degrades

The failure is structural rather than a matter of discipline. A single long-lived environment has properties that get worse as more people use it:

Contention
One team at a time can trust its state, so everyone else is queueing without a queue to point at.
Drift
Hand-fixed over months, so it resembles nothing — neither production nor a clean install.
Unclear state
Is that data seeded, left over, or a bug? Nobody knows, and nobody dares delete it.
No isolation, and one broken deploy blocks all
Two teams' test data interact in undocumented ways, and a bad deployment stops everybody.
Irreproducible
"It works in staging" is unfalsifiable, because staging is unique. A bug that only appears there cannot be attributed — and a bug that does not appear there proves nothing.

Every one of these gets worse as more people use the environment, and none is fixable by discipline

Ephemeral environments

An ephemeral environment is created for one purpose — a pull request, a test run, a demo — and destroyed when that purpose ends.

What they buy is worth listing because each addresses one row of the table above: no contention, no drift, reproducibility by construction, no blocking, and full isolation.

The properties that make one work:

The word shape is deliberate. An ephemeral environment cannot have production's data volume or instance count, and that is fine for correctness testing. It must have the same topology: the same services, the same database engine and version, the same configuration mechanism. Otherwise it answers a different question from the one you asked.

Building one

The whole thing hangs off two ideas from earlier in this track: containerisation makes the stack reproducible, and seeding makes it useful.

Managed platforms do much of this: Vercel and Netlify give preview deployments for front ends automatically, Render and Railway do it for services, and Kubernetes plus Helm or Kustomize is the do-it-yourself version.

Seeding is the hard part

An empty environment is nearly useless: no accounts to sign in with, no data to list, nothing to exercise. Seeding is what makes it a place you can work, and it is the part that decides whether the whole idea succeeds.

Three approaches, and the trade-offs mirror the test-data lesson:

Two properties make any of them work.

Idempotent. Running it twice gives the same state — no duplicate keys, no accumulation. That is what lets a broken environment be reset rather than repaired.

Fast. Under a few minutes, or people stop creating environments. If seeding is slow, the answer is usually a pre-built snapshot restored rather than a script replayed — a database dump created once by the script.

Prefer seeding through the application's own interfaces — the API, or the service layer — rather than by direct database inserts. Then the seed cannot create states the application could not, it exercises validation, and it stays correct when the schema changes because it goes through the same code path production does.

Test infrastructure is a product

Bad — infrastructure as a chore, owned by nobody:

Good — infrastructure with an owner, a contract and a way to report problems:

The bad version is not unusual and it has a specific cost: every engineer pays a tax in waiting, in guessing, and in interrupting whoever set it up. Because the cost is spread across everyone it never becomes anyone's priority.

The good version treats the people using it as users. It states what they get, how long it takes, how to reset it, who owns it and where to complain — and having a target makes degradation visible rather than gradual. That framing is what gets the work funded.

What still needs a long-lived environment

Ephemeral environments do not remove the need for a stable one, and pretending otherwise causes its own problems.

So the arrangement that works is both: ephemeral environments per pull request for correctness and review, and one or two long-lived environments — production-shaped, rebuilt from code, with realistic data — for the questions that need scale and time.

The important difference from the shared-staging failure is that the long-lived environment is now rebuilt from the repository rather than maintained by hand. It can be destroyed and recreated, which is what stops it drifting into something unique.

Cheat sheet

Where to go next

The next lesson goes to the one environment that cannot be replicated: production. Feature flags, canaries, shadow traffic, synthetic journeys and a rollback you have actually rehearsed — the things that are only true where the real users are.

Before that, time how long it takes to get a working environment for the project you work on, from nothing. If the answer is more than an hour or involves asking somebody, that number is the argument for this lesson, and it is worth writing down.