Regression Testing and Why Suites Grow
Fixed things break again. Where a regression suite comes from, why every bug should leave a test behind, and how a suite turns into something nobody wants to run.
Fixed things break again. Where a regression suite comes from, why every bug should leave a test behind, and how a suite turns into something nobody wants to run.
The bug was fixed in March. It was signed off, closed, and released. In September a developer improved the user lookup, dropped a field the new code did not need, and the same behaviour came back — this time in the notification service, which had been quietly relying on that field for two years.
Nobody was careless. The connection between those two pieces of code was not written down anywhere, and nothing re-checked the March fix afterwards.
By the end of this lesson you will know where a regression suite comes from, why every bug should leave a test behind, and what to do when the suite has grown into a four-hour job that finds nothing.
A regression is something that used to work and now does not. Not a new feature failing — an old one, previously verified, broken by a change somewhere else.
They happen because software is connected in ways nobody has written down. A change to how dates are formatted for the invoice screen also changes the export, because both call the same helper. A performance improvement to the user lookup drops a field nothing in the new code needed and the notification service did. A dependency upgrade changes how a library handles empty strings.
That was the last of the five shapes from the earlier lesson: an assumption that used to be true. Regression is the form that shape takes over time.
Regression testing is therefore re-checking that what used to work still does. Not a technique — a purpose, in the vocabulary from the smoke-and-sanity lesson, and it can be pursued at any level.
A regression suite is not designed in one sitting. It accumulates, from three sources.
Written first. Change least.
The handful of things the product must be able to do at all: sign in, complete the main task, pay.
Where most of the volume comes from.
When a feature is accepted, the cases that verified it are kept and become regression cases.
One case per defect, and it must have failed once.
So it cannot come back silently. This is the source worth arguing for, and the rest of this section does.
A defect that reached working software is evidence about two things: the software, and the testing. The bug is the smaller finding.
If a bug got through, some check that would have caught it did not exist. Fixing the code fixes this instance. Writing a test that fails against the old behaviour is what prevents the class.
The order matters, and it is worth insisting on.
Write a test that reproduces the bug
Before touching the code that is wrong.
Run it. It must fail.
If it passes, you have not reproduced the bug. Whatever you wrote is testing something else, and you are about to lock that in forever.
Fix the code
Now, and not before.
Run it. It must pass.
Now the test has done both halves of its job, and it means what it says.
Step two is the one people skip, and skipping it is how you end up with a test that passes against the broken code and the fixed code — a test that looks like protection and is none. The only defence is watching it fail first.
Bad — written after the fix, against the fixed code:
Good — the same test, but confirmed to fail against the old behaviour first:
The code is identical; the difference is whether anyone watched it fail. Written afterwards and never run against the broken version, this test might be asserting on the wrong return value, using a fixture that invalidates the link for an unrelated reason, or exercising a code path the bug never touched. It passes, it goes green forever, and the regression it was written to prevent walks straight past it.
Watching it fail costs thirty seconds — check out the previous commit, or revert the fix temporarily — and it is the only thing that makes the test mean anything.
Every source of growth is legitimate. The sum is not. A suite that grows for three years and is never pruned develops the same four problems every time.
Too slow
Four hours means it does not run on every change, which means feedback arrives long after the change was made — exactly where the cost curve punishes you.
Unreliable
A suite that fails randomly twice a week teaches everyone to re-run it. Once "just run it again" is the habit, a real failure is indistinguishable from noise, and the suite costs time while giving no information at all.
Repetitive
Forty cases exercising one validation branch, because each was added by a different person for a different ticket.
Stale
Cases for features that changed, expected results nobody updated, cases that pass because they stopped asserting anything. Worse than absent ones: they produce confidence without coverage.
Five things, and none of them is one-off work.
Automate what repeats. A case run on every release is automation's best case: the cost is paid once and the benefit is every run. A case run twice a year usually is not worth automating.
Push cases down a level. The pyramid argument again. Forty invalid inputs checked through a browser is hours; the same forty as unit tests is milliseconds. Most bloated suites are bloated at the top.
Prune deliberately. Once a quarter, ask of each case: has this ever failed? Does it test something no other case does? Does the feature still exist? A case that has passed for two years and duplicates another is costing time and returning nothing.
Select by risk. A full suite before a release, and a targeted subset on each change: the area that changed, plus what depends on it, plus the core journeys. The next course covers doing this by impact analysis.
Fix flakiness as a defect. A test that fails intermittently is a bug report about your suite. Diagnose it or delete it — never re-run it and move on, because that is the habit that kills the suite.
Not everything, every time. A workable arrangement:
The pattern is that cost and coverage rise together, and each gate is placed where its cost is affordable and its information still timely.
Automated tests re-check what somebody predicted. They are excellent at it and they see nothing else — an automated suite will not notice that the page now looks wrong, that a step became confusing, or that a neighbouring feature was quietly affected.
So a release still deserves some human regression, and the efficient form is a short exploratory session around the change: not re-running scripts the machine already ran, but looking at the area with the heuristics from the exploratory lesson. Twenty minutes of that, on the area that changed, finds a category of regression no assertion covers.
The next lesson is about the thing every kind of testing quietly depends on and nobody plans: the data a test needs, and the environment it runs in. Most confusing test results turn out to be data or environment problems wearing a defect's clothing.
Before that, pick one bug that was fixed recently in something you work on and check whether a test now exists that would fail against the old behaviour. If not, that is a gap worth closing — and the reason it is missing is usually more interesting than the bug was.
on every commit unit and integration tests
under 10 minutes, or people stop waiting
on every deployment the smoke test
minutes; is this build usable at all
on every pull request the areas the change touches, plus the
core journeys
before a release the full regression suite
the only time the four-hour cost is justified
nightly the slow suite: all browsers, large data,
long-running scenarios
after a fix a sanity check on the fix and its neighbours# A regression: something that used to work and now does not
# caused by connections nobody wrote down — a shared helper, a
# dropped field, a dependency upgrade
# regression testing is a PURPOSE, available at any level
# Where a suite comes from
1. the core journeys written first, change least
2. each feature as it ships its acceptance cases become regression
3. every bug found a test that fails against the old code
# Every bug gets a test — in this order
1. write a test that reproduces it
2. RUN IT AND WATCH IT FAIL <- the step people skip
3. fix the code
4. run it and watch it pass
# a test never seen failing may prove nothing at all
# The better question
# not "is there a test for this bug?"
# but "why did the tests not catch it?"
# answers are structural: nothing tested the empty case; every
# test used an admin; the tests mocked the broken thing
# How suites go bad
too slow 4 hours means it does not run on every change
unreliable random failures train everyone to re-run it
repetitive 40 cases on one validation branch
stale passing cases that assert nothing — false confidence
# Keeping it healthy
automate what repeats pay once, benefit every run
push cases DOWN a level 40 browser checks -> 40 unit tests
prune quarterly has it ever failed? is it unique?
select by risk changed area + dependents + core
treat flakiness as a defect diagnose or delete; never just re-run
# What to run, when
every commit unit + integration under 10 minutes
every deployment smoke minutes
every PR changed areas + core journeys
before release the full suite the one justified 4 hours
nightly slow: all browsers, big data, long scenarios
after a fix sanity check: the fix AND its neighbours
# And keep some humans in it
# 20 minutes of exploratory testing around the change finds the
# regressions no assertion was written fordef test_reset_link_is_single_use():
link = request_reset("alice@example.com")
use_link(link, new_password="first1")
assert use_link(link, new_password="second1") is False# Run against the pre-fix build: FAILS (returns True).
# Run against the fix: PASSES.
def test_reset_link_is_single_use():
link = request_reset("alice@example.com")
use_link(link, new_password="first1")
assert use_link(link, new_password="second1") is False