Test Data and Environments
Most confusing test results are really data or environment problems. Building the state a test needs, keeping environments comparable, and never testing with real customer data.
Most confusing test results are really data or environment problems. Building the state a test needs, keeping environments comparable, and never testing with real customer data.
The case passed on Tuesday and failed on Wednesday. Nothing was deployed in between.
Somebody else had run a test that used up the one account with an unrefunded order on it. The account was still there; the order was refunded. Two hours went into investigating a defect that did not exist.
These are not bugs in the product, and they will take more of your week than bugs do if you let them. By the end of this lesson you will know how to get the data a test needs, how environments differ in the ways that matter, and why real customer data in a test system is a problem rather than a convenience.
A test case has preconditions — the earlier lesson insisted on stating them — and those preconditions are test data: the accounts, records and relationships that must exist before step one makes sense.
"Verify that an order can be refunded" needs a customer, a payment method, a completed order, a payment that succeeded, and a user with permission to refund. Five things, none of which the test creates, all of which must be true.
Which raises the question this lesson exists to answer: where do they come from?
Each has a real trade-off, and most teams end up using more than one.
The test makes what it needs, at the start.
Through the API, a fixture or a factory. Self-contained, repeatable, runs anywhere, safe in parallel.
Costs setup code, and gets slow when the object graph is deep.
A script builds a fixed cast that all tests share.
Fast, and it makes tests shorter.
Tests become coupled to the seed, and any test that modifies shared data breaks the others.
Real volume, real mess, generated names.
Finds bugs synthetic data never will, performance problems among them.
Significant work to build, and the anonymising is easy to get subtly wrong.
Nobody chooses this one.
Tests depend on records somebody made by hand months ago. Nobody knows which matter, and clearing the environment breaks everything.
| Repeatable | Fast | Realistic | Parallel-safe | |
|---|---|---|---|---|
| Create in test | yes | no | no | yes |
| Seeded set | yes | yes | no | no |
| Anonymised prod | no | yes | yes | no |
| Whatever is there | no | yes | — | no |
Whichever route you take, the data should include cases you would not choose. The input shapes lesson listed them; here they become records rather than form values.
Test data made only of Test User and test@example.com produces a
suite that passes and a product that breaks on the first customer whose
name has a hyphen. Building the awkward cases into the seed once means
every future test runs against them for free.
An environment is a complete running copy of the system. Most teams have several, and the names are broadly conventional.
Local
A developer's own machine, with whatever data they happened to make.
CI
Automated tests, on every change, against data created fresh for each run and thrown away after.
Development
Shared, integrated and deliberately unstable. Seeded.
Staging
Production-like on purpose, for release testing. Seeded or anonymised. This is the one that has to earn its name.
Production
Real users, real data, real money.
The entire point of staging is being like production — same versions, same shape of configuration, same integrations — so that a test there predicts what production will do. Every way it differs is a way that prediction can be wrong.
The differences that actually cause trouble:
Bad — two cases that pass alone and fail together:
Good — each case owns the account it uses:
The first pair works until they run in the other order, or in parallel, or twice — and then TC-044 fails because Bob is deactivated. The failure looks like a login defect, someone spends an hour on it, and the answer is that TC-018 ran first.
Worse, the failure is intermittent, which is how it becomes the flaky test everyone re-runs. Shared mutable state is the single largest source of unreliable suites, and creating what you need is the structural fix rather than a workaround.
Where sharing is unavoidable, at least separate by intent: read-only accounts that no test modifies, and per-test accounts for anything that changes state.
The tempting shortcut is to copy production into staging and test against that. Realistic volume, realistic mess, no fixtures to write.
It is also a serious problem, for four separate reasons.
Legally. Personal data may only be used for the purpose it was collected for. Testing is not that purpose. Under the GDPR and similar regimes this is a compliance matter with real penalties, not a formality.
Practically. Staging is less protected than production — more people have access, credentials are shared, backups are casual. A copy of production in a weakly protected environment is a breach waiting for a mistake.
Operationally. Test systems send email. A staging environment with real addresses has emailed real customers more than once, and the emails say things like "your subscription has been cancelled".
Sincerely. Somebody's medical appointment, salary, or private message is not test material, whatever the access controls say.
The workable answer is anonymisation: take the structure and volume, replace the content.
The free-text problem is why anonymisation is harder than it looks, and why the safe default for a comment field is to discard it rather than to attempt cleaning.
The property to aim for is that anyone can get a working environment from nothing, without asking anyone.
The two-runs requirement is the one worth insisting on. An idempotent setup means a broken environment is thrown away rather than repaired, and that changes how much time gets spent on environment archaeology — from hours a week to none.
Everything so far has been testing you do. The next lesson is about handing some of it to a machine: what automation is genuinely good at, what only a person can judge, and how to tell a test worth automating from one that will only ever break.
Before that, take a test case you rely on and trace its preconditions. Where does each piece of data come from, and what happens if somebody else's test modifies it? That question is usually where an intermittent failure has been hiding.
a name with an apostrophe O'Brien
a name with an emoji or accents José, a display name with 🙂
a very long name 300 characters
a user with no data at all new account, empty everything
a user with a great deal 10,000 records — the slow case
a deleted or deactivated user do their records still resolve?
an account with no permissions
amounts that do not divide £10.00 split three ways
a record from a different zone Auckland, so "today" differsdata volume staging has 200 records, production has 2 million.
The query that is instant on one takes 40 seconds
on the other.
integrations staging talks to a sandbox payment provider that
behaves slightly differently and never rate-limits.
configuration a feature flag on in one and off in the other; a
timeout of 30s against 5s.
scale one instance in staging, twelve behind a load
balancer in production — so session and caching
bugs only appear in production.
freshness staging is on last week's build.TC-018 Deactivate a user
Precondition: the account bob@example.com exists
Steps: deactivate bob@example.com
TC-044 A user can update their profile
Precondition: the account bob@example.com exists
Steps: sign in as bob@example.com, change the display nameTC-018 Deactivate a user
Precondition: create a new active account
Steps: deactivate that account
TC-044 A user can update their profile
Precondition: create a new active account
Steps: sign in as that account, change the display namenames replaced with generated names
emails replaced, and pointed at a sink domain
phone numbers replaced with reserved test ranges
addresses replaced
payment details removed entirely, never masked
free text this is the hard one — notes and comments contain
names, addresses and medical details, and no
pattern will find them all. Truncate or discard.
volumes, dates, KEPT — this is the realism you came for
relationships[ ] one command builds the database schema from zero
[ ] one command seeds the data the tests need
[ ] running it twice gives the same result — no duplicate keys,
no leftovers
[ ] configuration comes from the environment, not from a file
somebody edited by hand
[ ] the build or version is visible from the running system
[ ] no test depends on a record created manually
[ ] emails go to a sink or a local mail catcher, never out# Test data is the precondition, made concrete
# "verify a refund" needs: customer, payment method, completed
# order, successful payment, a user with permission
# Four ways to get it
create in test repeatable, parallel-safe, runs anywhere. Slower.
-> the default for anything automated
seeded set fast, shorter tests. Couples tests to the seed.
anonymised prod real volume and mess, finds performance bugs.
Significant work; easy to anonymise wrongly.
whatever is there nobody chooses this; it is where teams end up
# Build the awkward cases into the data once
O'Brien / José / 🙂 / a 300-character name
a user with nothing / a user with 10,000 records
a deleted user / an account with no permissions
£10.00 split three ways / a user in another time zone
# Environments, and how they differ from production
local / CI / development / staging / production
data volume 200 rows vs 2 million — the slow query only shows up
integrations a sandbox provider that never rate-limits
configuration a flag on here and off there; 30s vs 5s timeouts
scale one instance vs twelve — session and cache bugs
freshness staging is on last week's build
# FIRST question on any environment-specific failure: what build
# is actually deployed there?
# The largest source of flaky suites
# shared mutable data. Two cases using one account pass alone and
# fail together, in the other order, or in parallel.
# fix: each test creates what it needs.
# if sharing is unavoidable, separate read-only accounts from
# per-test ones.
# Real customer data in a test environment
# legally: not the purpose it was collected for (GDPR)
# practically: staging is less protected than production
# operationally: test systems send email to real people
# anonymise: names, emails (to a sink), phones, addresses;
# DELETE payment details; discard free text — no
# pattern finds every name inside a comment
# keep: volumes, dates, relationships. That is the realism.
# Reproducible environment checklist
one command builds the schema from zero
one command seeds what the tests need
running it twice gives the same state
configuration from the environment, not a hand-edited file
the build/version visible from the running system
no test depends on a manually created record
email goes to a sink, never out