beginner25 minby DevFox

When to Automate and When Not To

Automation is code you now maintain. What it is genuinely good at, what only a human can judge, and how to tell a test worth automating from one that will just break.

Testing
Automation

The team automated 340 browser tests in a quarter. Everyone was pleased. Then the checkout screen was redesigned, and one person spent nine days updating selectors in tests that had never once caught a bug.

"Automate the tests" sounds like an unambiguous improvement and is not. An automated test is code: it has to be written, reviewed, run, kept working when the product changes, and diagnosed when it fails. You are not removing work. You are exchanging repeated manual effort for permanent maintenance.

Sometimes that trade is overwhelmingly worth it. Sometimes it produces a suite costing more than the testing it replaced. By the end of this lesson you will be able to tell which is which before writing anything.

What you are actually trading

Automating a test turns a variable cost into a fixed cost plus a small recurring one — and a maintenance bill nobody budgets for.

Run it by hand

Twenty minutes, every single time.

Ten runs is three and a half hours. It also needs a person, so it can only happen when somebody is free.

Nothing to maintain. The cost is entirely in repetition.

Automate it

Four hours once, then seconds a run.

Pays for itself somewhere around the twelfth run.

Then: maintenance, every time the feature changes. That bill arrives forever and nobody put it in the estimate.

The maintenance line is the one that decides it, and the one people leave out

So automation wins on repetition and loses on churn. Fifty runs against a stable feature is an obvious yes. Two runs against a screen being redesigned next month is an obvious no. Most real cases sit in between, and the questions below are how to place them.

What machines are good at

Automation is not a faster human. It is a different capability, and knowing the shape of it prevents most bad decisions.

Repetition without fatigue. The four hundredth run is as careful as the first. A person checking the same twelve fields for the twentieth time is not.

Exactness. Comparing a response to an expected value, byte for byte, every field. People skim; machines do not.

Speed and volume. Ten thousand input combinations in a minute. This is where equivalence classes and boundary values become almost free — every value you identified, checked on every commit.

Doing the impossible. Fifty simultaneous users. A thousand records. The same request twice in the same millisecond. Nobody can do these by hand at all.

Being awake. Running on every commit, at 3am, without being asked — which is what turns a four-hour feedback loop into a four-minute one.

What only a person can do

The other half, and it is not a smaller half.

Judgement about whether something is right. An automated test asserts that the total is £43.00. A person notices that the total is £43.00 and that no reasonable customer would expect that, given the discount just applied. Machines check agreement with an expectation; people question the expectation.

Noticing the unasked. The layout is broken on a narrow window. The error message is technically accurate and incomprehensible. Two buttons say almost the same thing. Nobody wrote an assertion for any of it, because nobody thought of it — which was the whole argument for exploratory testing.

Following a hunch. "That was slower than last time." "Why did it flash before loading?" A person's next action changes based on what just happened; an automated test does exactly what it was written to do.

Judging experience. Confusing, tedious, frightening, condescending — all real defects, none machine-detectable.

Testing something that does not exist yet. A design, a requirement, a mock-up. The static-testing lesson was entirely about work a machine cannot do.

Five questions before automating a case

How often will it run?
Every commit is an easy yes. Once a release is usually yes. Twice a year is a no.
How stable is the thing it tests?
A business rule that has held for three years is stable. A screen being redesigned this quarter is not. Automate against settled behaviour, never against a moving interface.
How expensive is it by hand?
A five-second check is cheap to repeat forever. A forty-minute setup-heavy scenario earns automation even at modest frequency.
Could it only be done automatically?
Concurrency, volume, hundreds of input combinations. Here there is nothing to trade off — a person cannot do it at all.
What does it cost when it breaks?
A test that fails intermittently and takes an hour to diagnose has negative value unless it catches a great deal. Ask this one first.

Five answers, and the last one is the question nobody asks

Two rules of thumb worth carrying:

Automate the boring, keep the interesting. Repetitive, exact, high-volume, frequently repeated — automate. Judgement, novelty, experience — keep for people.

Automate at the lowest level that can catch it. The pyramid argument. Automating a validation rule through a browser is the most expensive possible way to check it.

The mistake: automating the manual script

Bad — a manual case translated faithfully into a browser test:

Good — the same coverage, placed where each part belongs:

The bad version is what happens when automation is treated as "do the manual tests, but with a robot". It takes three minutes instead of forty milliseconds, needs a browser and a running application, tests one pure function through six layers of software, and gives twenty-four red results for one broken selector — so the actual signal is buried.

The good version has the same coverage, runs 4,500 times faster, and fails in one place with a clear reason. Nothing was lost.

The general lesson: when you automate, redesign the test for automation. A manual case is written for a human reader who can improvise; an automated test needs to be aimed at the smallest thing that can be wrong.

What to automate first

If you are starting from nothing, this order gets the most value quickest:

The smoke test. Automated and running on every deployment. Highest value per hour of work in all of testing.
Unit tests on the rules with many cases. Every boundary and equivalence class you identified — nearly free once the harness exists.
The API-level tests of the core flows. Fast, stable, and they cover most of what a browser test would without the browser.
The regression test for each fixed bug. One per defect, and it fails against the old behaviour first.
A handful of end-to-end journeys. The two or three paths that must work. Kept genuinely reliable, and deliberately few.

Notice what is last and small. That ordering is the opposite of how most teams start, and it is why so many suites are slow and untrusted.

When not to automate at all

Five situations where the honest answer is no:

The feature is still changing weekly. Wait until it settles. You will write it twice otherwise.

It runs twice a year. Write a good manual case instead.

The check is a judgement. "The report looks right" cannot be automated, and forcing it produces a test that asserts something narrower than what you meant.

Nobody will maintain it. An unmaintained suite decays into noise, and noise is worse than nothing because it consumes attention while providing no signal.

The application is not testable yet. No stable identifiers on elements, no way to create data, no way to reach a known state. Fix that first — it is a legitimate finding, and automating around it produces exactly the fragile tests everyone complains about.

Cheat sheet

Where to go next

You know what to automate. The next lesson writes one: a real automated test, run and read, including what a good failure message looks like and why a test that has never failed proves nothing.

Before that, look at a manual suite you know and sort its cases by the five questions. The ones that are frequent, stable, expensive and exact are your automation backlog, in order — and the exercise usually also turns up cases nobody should be running at all.