Beyond Functional Testing
The requirements nobody writes down: usability, accessibility, compatibility, performance and security. What each one is, and the cheapest first check for each.
The requirements nobody writes down: usability, accessibility, compatibility, performance and security. What each one is, and the cheapest first check for each.
The export worked. It produced the right rows, in the right order, with the right totals, and every test agreed.
It also took nine minutes for the customer with 50,000 orders, during which the page had to stay open, and it could not be operated with a keyboard at all. None of that was a bug against the ticket. The ticket said "the user can export their orders as CSV", and the user could.
By the end of this lesson you will know the five requirements nobody writes down, and the cheapest first check for each — enough to find real problems in an afternoon, without specialist tools.
A functional requirement says what the software does. A non-functional one says how well.
The user can export their orders as CSV.
Testable, tested, and green.
Nobody wrote any of this, and all of it was true.
Within five seconds, for up to 50,000 orders. On a phone as well as a desktop. Usable with a keyboard alone. And only ever their own orders.
Asking for those during the requirements review — the fourth of the four questions from that lesson — is the cheap version of this whole lesson. What follows is what to do when nobody did.
Five categories, in the order they most often bite.
Software is rarely slow when it is built. It becomes slow when real volumes arrive, because it was built against a hundred test records and production has two million.
That was the fifth failure shape: an assumption that used to be true.
The cheapest first check needs no tools at all:
1. Create a lot of data — 10,000 records, not 20. A script or a
loop through the API will do it.
2. Use the feature. Time it with a stopwatch.
3. Watch the list, the search, the export and the report — these
are where per-record work hides.The bug you are looking for has a name worth knowing: the N+1 query. A page fetches a list of 50 orders in one query, then fetches the customer for each one separately — 51 queries instead of two. It is instant with 10 records, and it is a four-second page with 500. Your browser's network tab, or the application's own query log, shows it immediately.
Three numbers worth having a view on: how long the slowest page takes with realistic data, how many queries a page makes, and how large the largest response is.
It works on your machine, in your browser, at your window size. Users have none of those.
1. Resize the browser window to 375 pixels wide — a phone. Does
the layout survive? Can you reach every control?
2. Try a second browser engine. Chrome and Firefox share little
code with Safari, and Safari is where iPhone users are.
3. Zoom to 200%. Text should reflow, not disappear behind a
header or get clipped.
4. Try the actual devices your analytics say people use, rather
than the ones you own.The 375-pixel check is the highest-return of these, and it takes ten seconds — drag the window narrow. Overflowing tables, buttons pushed off screen and modals with an unreachable Close are all routine findings.
Accessibility means people with disabilities can use the software. It affects a large minority of users permanently and everyone temporarily — a broken wrist, bright sunlight, a noisy room — and in many jurisdictions it is a legal requirement rather than a nicety.
Four checks, none needing a specialist:
1. Keyboard only. Put the mouse aside. Tab through the page. Can
you reach and operate everything? Can you see where the focus
is? Can you escape a modal?
2. Automated scan. Run axe DevTools or Lighthouse in the browser.
Instant, and finds missing labels, poor contrast, images with
no alt text.
3. Colour is not the only signal. If an error is red text with
no icon or message, a colour-blind user gets nothing.
4. Screen reader. Turn on VoiceOver on macOS or NVDA on Windows
and try one flow. Uncomfortable at first, and the single most
informative half hour available on this subject.Usability is whether a person can accomplish their goal without frustration, and it cannot be automated at all — which is precisely why it is the part a human tester should be looking at.
The most effective technique is also the simplest: watch somebody use it. Give a person who has not seen the feature a realistic task, and say nothing while they do it. Where they hesitate is a defect. Where they click the wrong thing is a defect. What they say out loud is the report.
Five people is enough to find most problems, and the discipline is not helping.
Common findings, worth checking for directly:
error messages that state a problem and not a remedy
"Invalid input" versus "Dates must be in the past"
no feedback — a button that does nothing visible for three
seconds, so it gets pressed again
destructive actions with no confirmation, or no undo
inconsistency — Save here, Submit there, Apply somewhere else
required information that is not marked as required
losing the user's work on an error, so they retype everythingThat last one is the most infuriating and the most common: a validation error that clears the form.
Security testing in depth is a specialism. There is a set of checks any tester can run, and they find real problems.
1. Change an id in the URL. If /orders/1042 is yours, try 1041.
If you can see somebody else's order, that is a serious
defect — and it is the single most common web vulnerability
there is.
2. Do it without logging in. Copy an authenticated URL into a
private window.
3. Do it as the wrong role. A standard user reaching an admin
endpoint directly, without the button being visible.
4. Bypass the front end. Front-end validation is a convenience;
send the request with curl or the browser's own tools and see
whether the server checks too.
5. Read the error messages. A stack trace, a SQL fragment or an
internal hostname in a user-facing error tells an attacker
about your system.
6. Look for what should not be there. Credentials in the page
source, tokens in a URL, an API key in a JavaScript bundle.That first check has a name — insecure direct object reference — and it consistently appears near the top of the OWASP Top Ten, the standard list of common web vulnerabilities. It takes ten seconds to try.
Bad — a non-functional requirement written as an aspiration:
The export should be fast and work well on mobile.Good — the same intent, as something that can pass or fail:
The export completes within 5 seconds for 10,000 orders and
within 30 seconds for 100,000.
The order list is usable at 375px wide: no horizontal scrolling,
every control reachable.
The export button is operable by keyboard and has an accessible
name.The first version cannot be tested, so it will not be tested, so it will not be met — and because nobody can point at a failed requirement, the resulting slowness is nobody's defect. It reads like a requirement and functions as a hope.
The second version names numbers and observable conditions. Each line can fail, which means each line can be checked, which means somebody has to decide whether it is acceptable — and that decision is the entire value of writing it down.
Turning vague non-functional wishes into testable statements is one of the most useful things a tester does, and it happens in the requirements conversation rather than in a test run.
# Non-functional requirements: how WELL, not what.
# Almost never in the ticket. Ask for them in the review.
PERFORMANCE
create 10,000 records, then use the feature with a stopwatch
watch the list, the search, the export, the report
look for the N+1 query: 1 query for the list + 1 per row
ask for percentiles (p95, p99), never averages
COMPATIBILITY
resize the window to 375px — the ten-second check
a second browser engine — Safari shares little with Chrome
zoom to 200% — text should reflow, not clip
the devices your analytics show, not the ones you own
ACCESSIBILITY
keyboard only: tab everywhere, visible focus, escape a modal
automated scan: axe DevTools or Lighthouse
colour must not be the only signal
a screen reader for one flow: VoiceOver or NVDA
automated tooling finds roughly a third — a green score is a floor
USABILITY (cannot be automated — this is the human's part)
watch someone do a real task and say nothing
five people finds most problems
look for: errors with no remedy, no feedback on a slow action,
destructive actions with no undo, inconsistent labels,
unmarked required fields, and forms that lose the user's work
SECURITY (only against systems you are authorised to test)
change an id in the URL — /orders/1042 -> 1041
try an authenticated URL logged out, in a private window
try an admin endpoint as a standard user
send the request without the front end — is the server checking?
read error messages for stack traces, SQL, internal hostnames
look for credentials in page source, tokens in URLs, keys in JS
# Make the requirement testable or it will not be met
bad "should be fast and work well on mobile"
good "completes within 5s for 10,000 orders; usable at 375px
with no horizontal scrolling"The last lesson of this course brings everything together into a test plan: for one real feature, what you will test, what you deliberately will not, in what environment, with what data, and how you will know when you are done.
Before that, spend twenty minutes on something you use daily: narrow the window to 375 pixels, tab through it with no mouse, and change one id in the URL. Three checks, twenty minutes, and the hit rate is higher than anyone expects.