Property-Based Testing
Stop writing examples and state the invariant instead. Generators, shrinking, choosing properties that hold, and the class of bug that example-based tests structurally cannot find.
Stop writing examples and state the invariant instead. Generators, shrinking, choosing properties that hold, and the class of bug that example-based tests structurally cannot find.
An example-based test asserts that one input produces one output. You chose the input, which means the test can only find bugs in cases you thought of — and the boundary analysis from the foundations course is a discipline for thinking of more of them, not for thinking of all of them.
Property-based testing inverts the relationship. You state a rule that must hold for every input, and the framework generates hundreds of inputs trying to break it. By the end of this lesson you will know how to find properties worth stating, why shrinking is what makes the technique usable, and the class of defect this catches that examples structurally cannot.
# example-based: three inputs you chose
def test_reverse():
assert reverse([1, 2, 3]) == [3, 2, 1]
assert reverse([]) == []
assert reverse([1]) == [1]
# property-based: a rule, over every list the framework can invent
from hypothesis import given, strategies as st
@given(st.lists(st.integers()))
def test_reversing_twice_gives_the_original(items):
assert reverse(reverse(items)) == itemsThree inputs you chose.
Precise, readable, and it covers exactly the cases you already thought of — which are, by definition, not the ones with the bug in them.
A rule, over every input it can invent.
A hundred runs by default: empty, one element, a thousand elements, negative numbers, huge integers, duplicates.
reverse(reverse(x)) == x is true of reversal by definition,
whatever the input.
Finding statements like that is the whole skill, and it is harder than writing examples — which is why the technique belongs at the top of your risk model rather than everywhere.
A generated failure is usually enormous and unreadable. The framework's most valuable feature is that it then shrinks the input — repeatedly simplifying while the test still fails — down to a minimal counterexample.
Falsifying example: test_discount_never_exceeds_total(
total=0.01, discount_percent=1
)
Original failing input was:
total=8734.29, discount_percent=73That is the minimisation exercise from the bug-reporting lesson, done automatically. What you get is not "it failed with some big random input" but "it fails with total=0.01 and a 1% discount" — a one-line reproduction that names the cause, in this case a rounding step that produces a negative amount at the smallest possible value.
Hypothesis also keeps a database of failing examples and retries them first on subsequent runs, so a discovered failure becomes a deterministic regression test until it is fixed.
The hard part. Six patterns cover most of what is findable, and it is worth having them as prompts.
Round trip. Encode then decode, save then load, serialise then parse. The result must equal the original. This is the highest-value pattern and it applies almost everywhere.
@given(st.text())
def test_json_round_trip(value):
assert json.loads(json.dumps(value)) == value
@given(contact_strategy())
def test_a_saved_contact_reads_back_identically(repository, contact):
saved = repository.save(contact)
assert repository.get(saved.id) == contactInvariants — things always true of the output. A price is never negative. A sorted list has the same length as its input. A total equals the sum of its lines.
@given(st.decimals(min_value=0, max_value=100000, places=2),
st.integers(min_value=0, max_value=100))
def test_a_discount_never_makes_a_price_negative(total, percent):
assert final_price(total, percent) >= 0
@given(st.decimals(min_value=0, places=2), st.integers(0, 100))
def test_a_discount_never_increases_the_price(total, percent):
assert final_price(total, percent) <= totalTwo ways to the same answer. An optimised implementation against an obviously-correct slow one. This is exceptionally powerful when you have just rewritten something.
@given(st.lists(st.integers()))
def test_the_fast_sort_agrees_with_the_obvious_one(items):
assert fast_sort(items) == sorted(items)Idempotence. Doing it twice equals doing it once. Normalising, sorting, deduplicating, applying a migration.
@given(st.text())
def test_normalising_is_idempotent(value):
once = normalise_email(value)
assert normalise_email(once) == onceMetamorphic relations. A known relationship between two outputs, when the exact output is hard to state. Searching for a shorter prefix must return at least as many results; adding an item must not reduce the total.
@given(st.lists(item_strategy()), item_strategy())
def test_adding_an_item_never_reduces_the_total(items, extra):
assert basket_total(items + [extra]) >= basket_total(items)It does not crash. The weakest property and still worth having on a parser or an input handler: any input produces either a valid result or a declared exception, never an unhandled one.
@given(st.text())
def test_parsing_never_raises_an_unexpected_error(value):
try:
parse_date(value)
except InvalidDateError:
pass # declared and handledBad — the "property" restates the implementation, so it can only be wrong in the same way:
@given(st.decimals(min_value=0, places=2))
def test_discount_calculation(total):
expected = total - (total * Decimal("0.10"))
assert final_price(total, 10) == expectedGood — properties that constrain the result without recomputing it:
@given(st.decimals(min_value=0, max_value=100000, places=2),
st.integers(min_value=0, max_value=100))
def test_final_price_properties(total, percent):
result = final_price(total, percent)
assert result >= 0 # never negative
assert result <= total # never a markup
assert result.as_tuple().exponent >= -2 # never sub-penny
if percent == 0:
assert result == total # 0% is a no-op
if percent == 100:
assert result == 0 # 100% is freeThe bad version duplicates the arithmetic in the test. If the implementation rounds with banker's rounding and the test uses the same expression, both agree — including when both are wrong. And it fails whenever the calculation is legitimately restructured, which is the implementation-coupling problem from the intermediate course.
The good version says what must be true of the answer without computing the answer. It catches the negative result, the sub-penny artefact, the markup at some odd percentage, and the boundaries — none of which the bad version can see.
Default strategies generate anything, which is usually right and sometimes
produces noise — a test that fails on NaN when the domain excludes it.
Constrain deliberately, and prefer constraining the domain over filtering
after the fact.
# a domain-specific generator, composed
@st.composite
def order_strategy(draw):
return Order(
total=draw(st.decimals(min_value=0, max_value=10000, places=2)),
currency=draw(st.sampled_from(["GBP", "EUR", "USD"])),
items=draw(st.lists(item_strategy(), min_size=1, max_size=20)),
)
@given(order_strategy())
def test_an_orders_total_equals_the_sum_of_its_lines(order):
assert order.total == sum(line.amount for line in order.lines)# assume() discards a generated case — use sparingly
@given(st.integers())
def test_something_about_positive_numbers(value):
assume(value > 0) # half the cases are thrown away
...assume is convenient and wasteful: it discards generated inputs, and a
strict filter makes the framework give up looking. Generating from
st.integers(min_value=1) is better in every way.
Two settings worth knowing:
from hypothesis import settings
@settings(max_examples=1000, deadline=None)
@given(order_strategy())
def test_a_thorough_property(order):
...More examples nightly than on a pull request is the usual arrangement: a hundred per run in CI, a thousand or ten thousand in the nightly job, which finds the rarer counterexamples without slowing the gate.
Property-based testing takes real thought per test, so aim it at the top of the risk model. Five places it consistently pays:
money arithmetic rounding, currency, discounts, tax, splitting
parsers and encoders round trips; and never crashing
data transformations import, export, migration, normalisation
anything with an a sort, a scheduler, a deduplicator, a state
obvious invariant machine
a rewrite compare the new implementation against the oldThat last one is the easiest sell. When you replace a working function with a faster one, a property test comparing the two over generated input is worth more than any number of examples, and it can be deleted once the old implementation is gone.
Where it does not pay: thin glue, code with no statable invariant, and anything where the correct answer genuinely requires a table of known values — a tax-rate lookup has no property, only data.
from hypothesis import given, assume, settings, strategies as st
@given(st.lists(st.integers()))
def test_reversing_twice_gives_the_original(items):
assert reverse(reverse(items)) == items# Six patterns for finding properties
round trip encode/decode, save/load, serialise/parse == original
^ the highest-value pattern; applies almost everywhere
invariant always true of the output: never negative, same
length, total == sum of lines
two ways the fast implementation agrees with the obvious one
^ best when the reference is INDEPENDENTLY correct
idempotence doing it twice == doing it once
metamorphic a known RELATION between outputs when the exact
output is hard to state: adding an item never
reduces the total
no crash any input -> a valid result or a DECLARED exception
# The mistake: reimplementing the calculation in the test
# both versions agree, including when both are wrong
# instead: constrain the result without computing it
# >= 0 / <= total / no sub-penny / 0% is a no-op / 100% is free
# Shrinking is what makes it usable
# failing input total=8734.29 pct=73 shrinks to total=0.01 pct=1
# a one-line reproduction that names the cause
# Hypothesis stores failures and retries them first — a found
# counterexample becomes a deterministic regression test
# Generators: constrain the DOMAIN, do not filter after
st.integers(min_value=1) good
assume(value > 0) wasteful — discards half the cases
st.decimals(min_value=0, max_value=10000, places=2)
st.sampled_from(["GBP", "EUR", "USD"])
@st.composite compose domain objects
# Settings
@settings(max_examples=1000, deadline=None)
# 100 examples on a pull request, 1,000+ nightly
# Where it pays
money arithmetic / parsers and encoders / import, export, migration /
anything with an obvious invariant / verifying a REWRITE against the
old implementation
# Where it does not
thin glue / no statable invariant / lookup tables (data, not rules)Property-based testing generates inputs to break a stated rule. The next lesson turns the technique on the tests themselves: mutation testing breaks the code deliberately and asks whether any test notices — which is the honest answer to whether a suite asserts anything at all.
Before that, write one round-trip property for something you work on: a serialiser, an import/export pair, a normaliser. Round trips are the easiest properties to find and the most likely to fail on the first run.