Structuring a Test Suite
How a suite is laid out decides whether people run it. Naming, grouping, shared setup, the fast/slow split, and keeping the failure message enough to diagnose from.
How a suite is laid out decides whether people run it. Naming, grouping, shared setup, the fast/slow split, and keeping the failure message enough to diagnose from.
A suite's structure decides whether anybody runs it. That sounds like an
overstatement until you have worked in a repository where the tests take
twenty minutes, the fixtures are three files deep, and a failure message
says AssertionError: False is not true — at which point people stop
running them locally and start relying on CI, and the feedback loop that
was the whole point is gone.
This lesson is about the arrangement decisions: where tests live, how they are named, what setup they share, and how the fast ones stay separable from the slow ones. None of it is glamorous and all of it compounds.
The first decision is where tests live, and there are two conventions.
Beside the code, which is common in JavaScript and Go:
src/
├── discounts.ts
├── discounts.test.ts
├── orders/
│ ├── import.ts
│ └── import.test.tsIn a parallel tree, which is common in Python and Java:
src/
├── discounts.py
└── orders/
└── import.py
tests/
├── unit/
│ ├── test_discounts.py
│ └── orders/
│ └── test_import.py
├── integration/
│ └── test_import_writes.py
└── e2e/
└── test_import_journey.pyEither works. What matters is that the location is predictable — given a file, you know where its tests are without searching — and that the levels are separable, because you will want to run the fast ones alone dozens of times a day.
The parallel tree makes the level split structural, which is why it is worth preferring when a project has all three levels. Beside-the-code is pleasant for unit tests and tends to need a separate directory for the slow ones anyway.
You will read test names far more often than test bodies, usually in a list of failures with no context. The name has to carry the claim.
# useless in a failure list
def test_import_1(): ...
def test_import_2(): ...
def test_validation(): ...
# states what should be true
def test_rejects_a_row_with_an_invalid_date(): ...
def test_imports_valid_rows_when_a_later_row_fails(): ...
def test_a_second_import_of_the_same_file_creates_no_duplicates(): ...The pattern that scales is
test_<subject>_<behaviour>_<condition>, and long names are correct:
nobody types them, and the payoff is a failure list that reads like a
specification of what just broke.
Some ecosystems favour nested descriptions, which produce the same readable sentence:
describe('CSV import', () => {
describe('when a row has an invalid date', () => {
it('rejects that row and reports its number', () => {
// ...
});
});
});CSV import > when a row has an invalid date > rejects that row and reports its number. Either style is fine; the test is whether the name
alone tells a colleague what is broken.
Every test needs a starting state. How much of that state is shared is the decision that most affects both readability and reliability.
Maximum clarity, some repetition.
The premise is right there in the test body. Right when there is one of it, or when the values are the point.
When several tests need the same thing.
A verified admin user, a running database. Things a test needs and is not about.
When they need similar-but-different.
Defaults for everything irrelevant, and the test names the one thing that matters. This is where a growing fixture should go.
Inline, in the test. Maximum clarity, some repetition:
def test_a_forty_five_pound_basket_gets_ten_percent():
basket = Basket(items=[Item("book", 25.00), Item("pen", 20.00)])
assert basket.discount_percent() == 10A fixture, when several tests need the same thing:
@pytest.fixture
def admin_user(database):
return create_user(database, role="admin", verified=True)A factory, when tests need similar-but-different objects — which is the subject of its own lesson shortly:
def test_a_deactivated_admin_cannot_import(user_factory):
user = user_factory(role="admin", active=False)
assert not can_import(user)The rule for choosing is about what the test is about. Whatever the test is testing should be visible in the test; everything else can be hidden in setup.
Bad — the interesting value is in another file:
@pytest.fixture
def basket():
return Basket(items=[Item("book", 25.00), Item("pen", 20.00)])
def test_forty_five_pounds_gets_ten_percent(basket):
assert basket.discount_percent() == 10Good — the total the test is named for is in the test:
def test_forty_five_pounds_gets_ten_percent(basket_factory):
basket = basket_factory(total=45.00)
assert basket.discount_percent() == 10In the first version, a reader has to open the fixture to discover that 45.00 is where the number comes from — and if somebody adds a third item to that fixture for an unrelated test, this one fails for a reason its name does not mention. The relationship between the test and the value it depends on is invisible.
The second states its own premise. The fixture supplies defaults for everything irrelevant; the test names the one thing that matters.
Tests differ in cost by four orders of magnitude, so treat the fast ones as a separate suite you can run constantly.
The mechanism is markers or tags:
@pytest.mark.slow
def test_importing_ten_thousand_rows_completes_in_time():
...pytest tests/unit # milliseconds
pytest -m "not slow" # everything quick
pytest # all of it// vitest
test.skipIf(!process.env.RUN_SLOW)('imports 10,000 rows', () => {});A workable set of gates, which mirrors the regression lesson:
while working the unit tests for what you are touching, on save
before pushing all unit + integration, under 2 minutes
on a pull request the same, plus the API tests
nightly end-to-end across browsers, large data, long runsThe number to defend is the "before pushing" one. Under two minutes and people run it; past five and they stop.
Every test must pass alone, in any order, and in parallel. This is not tidiness — it is the difference between a suite that can be trusted and one that produces intermittent failures nobody can explain.
Three rules deliver it:
No test depends on another running first. If test B needs the record test A created, they are one test badly split.
No test leaves state behind. Whatever it creates, it removes — or the harness rolls it back. A transaction per test, wrapped and discarded, is the cleanest version for anything touching a database.
No test assumes exclusive use of anything. A hard-coded email address, a fixed record id, a shared account. This was the largest source of flakiness in the foundations course, and the fix is the same: create what you need, with a unique identity.
# fragile: two tests running at once collide
user = create_user(email="test@example.com")
# safe under parallelism
user = create_user(email=f"test-{uuid4()}@example.com")Then prove it rather than hoping:
pytest -p no:randomly -q # baseline
pytest --random-order # order-dependence
pytest -n 4 # parallel; needs pytest-xdistRunning the suite in a random order occasionally is the cheapest way to find hidden coupling before it finds you.
The last structural concern is the message. A test's value at the moment it fails is entirely in how fast someone understands why.
# tells you nothing
assert is_valid(row)
# tells you which row and what was wrong
assert is_valid(row), f"row {index} rejected: {row!r}"
# a specific assertion beats a boolean
assert errors == [] # shows the errors
assert response.status_code == 201 # shows what it wasThe general principle: assert on values, not on booleans. A framework
can show you 403 != 201; it cannot show you anything useful about
False.
Two habits that pay for themselves:
One logical assertion per test, so a failure names one thing. Several
assert statements checking facets of one outcome is fine; four
unrelated checks in one test is the bad case from the foundations lesson.
Add context to loops. A parametrised test shows its parameters; a test that loops over ten rows and fails on the seventh needs to say so.
# Layout — predictable, and separable by level
tests/
unit/ milliseconds; run constantly
integration/ seconds; run before pushing
e2e/ minutes; run nightly and on release
# "run only the fast tests" must be ONE command
# Naming — the failure list is what you read
test_<subject>_<behaviour>_<condition>
test_rejects_a_row_with_an_invalid_date
test_a_second_import_of_the_same_file_creates_no_duplicates
# long names are correct; nobody types them
# Setup — three levels of sharing
inline maximum clarity, some repetition
fixture several tests need the SAME thing
factory tests need SIMILAR-but-different things
# rule: what the test is ABOUT belongs in the test;
# everything else can be hidden in setup
# a fixture that grows to serve new tests wants to be a factory
# Fast/slow split
pytest tests/unit milliseconds
pytest -m "not slow" everything quick
pytest all of it
# keep "before pushing" under 2 minutes or people stop running it
# Independence — the anti-flake requirement
no test depends on another running first
no test leaves state behind (a transaction per test, rolled back)
no test assumes exclusive use of an account, email or id
create_user(email=f"test-{uuid4()}@example.com")
# Prove it
pytest --random-order finds order-dependence
pytest -n 4 finds parallel-safety problems
# Failures that diagnose themselves
assert errors == [] not assert not errors
assert response.status_code == 201 not assert response.ok
assert is_valid(row), f"row {index}: {row!r}"
# assert on VALUES, not booleans — a framework can show 403 != 201The suite has a shape. The next lesson is about what goes in the largest part of it: unit tests that are worth having, and the distinction between testing behaviour and testing implementation — which decides whether your tests help a refactor or block it.
Before that, run your own suite in a random order and in parallel. If either produces a failure, you have found a coupling worth fixing today, and it would otherwise have surfaced as a flake on a Friday afternoon.