advanced35 minby DevFox

The Data Model and Dunder Methods

The protocols behind Python's syntax: how len, in, iteration, comparison and arithmetic are all method calls, and how implementing them makes your types behave like built-in ones.

Python

You write a Money class. Adding two of them needs a.add(b). Comparing them needs a.equals(b). Putting one in a set fails. Printing one gives a memory address. Meanwhile int does all of it with +, == and print, and nothing about int is special — it is implemented in C, but the mechanism it uses is available to you.

Python's syntax is a facade over method calls. a + b is a method. len(x) is a method. for x in y is two methods. By the end of this lesson you will know which methods, how Python chooses between them, and where implementing them makes your types feel native rather than bolted on.

Syntax is dispatch

The data model is the set of hooks — named with double underscores, hence dunder — that Python's syntax compiles into.

python

a + b          ->  type(a).__add__(a, b)
len(x)         ->  type(x).__len__(x)
x[0]           ->  type(x).__getitem__(x, 0)
x in y         ->  type(y).__contains__(y, x)
for i in x     ->  type(x).__iter__(x), then __next__
with x:        ->  type(x).__enter__(x) / __exit__
f(1)           ->  type(f).__call__(f, 1)
print(x)       ->  type(x).__str__(x)

Two details in that arrow are worth stating, because they explain behaviour that otherwise looks arbitrary.

Lookup is on the type, not the instance
Setting obj.__len__ = something does not change what len(obj) returns, which is why you cannot patch a dunder method onto one object.
The protocol matters, not inheritance
Nothing has to subclass anything. Define __len__ and len() works — which is how a file, a generator and a list are all loopable without sharing an ancestor.

Both follow from `type(a)` being on the left of every arrow above

Repr and str

Two ways of turning an object into text, for two audiences:

python

from dataclasses import dataclass

class Money:
    def __init__(self, amount: Decimal, currency: str) -> None:
        self.amount = amount
        self.currency = currency

    def __repr__(self) -> str:
        return f"Money({self.amount!r}, {self.currency!r})"

    def __str__(self) -> str:
        return f"{self.amount:.2f} {self.currency}"

python

money = Money(Decimal("19.99"), "EUR")
print(money)          # 19.99 EUR              __str__, for a user
print(repr(money))    # Money(Decimal('19.99'), 'EUR')
print([money])        # [Money(Decimal('19.99'), 'EUR')]

__str__ is for humans reading output. __repr__ is for developers — at the prompt, in a debugger, in a log, and inside a container, which is why the list shows reprs.

Define __repr__ on everything. If you define only one, define that one: str() falls back to __repr__, so you get both. The convention is that it should look like the call that would recreate the object.

Equality and hashing travel together

python

    def __eq__(self, other: object) -> bool:
        if not isinstance(other, Money):
            return NotImplemented
        return (self.amount, self.currency) == (other.amount, other.currency)

    def __hash__(self) -> int:
        return hash((self.amount, self.currency))

Returning NotImplemented rather than False for an unknown type is the correct move: it tells Python to try the other operand's __eq__ before giving up. Returning False claims they are unequal, which forecloses a comparison that might have succeeded.

The rule that catches people is what happens next.

You define __eq__
Two objects with the same fields now compare equal.
Python sets __hash__ to None
Deliberately. Objects that compare equal must hash equal, and it cannot guess your definition — so it removes the inherited one rather than leave a wrong one in place.
You define __hash__ from the same fields
Or you accept unhashable, which for a mutable object is the correct answer.

Which is why a class that suddenly cannot go in a set is usually a class that just gained an __eq__

python

class Money:
    def __eq__(self, other): ...
    # no __hash__

{Money(...)}       # TypeError: unhashable type: 'Money'

A key that changes after being filed away cannot be found again, which is the same fact the dictionaries lesson established from the other side.

Ordering, for free

Comparison operators map to six methods, and you rarely need all six:

python

from functools import total_ordering

@total_ordering
class Version:
    def __init__(self, major: int, minor: int, patch: int) -> None:
        self.parts = (major, minor, patch)

    def __eq__(self, other: object) -> bool:
        if not isinstance(other, Version):
            return NotImplemented
        return self.parts == other.parts

    def __lt__(self, other: "Version") -> bool:
        if not isinstance(other, Version):
            return NotImplemented
        return self.parts < other.parts

@total_ordering derives <=, >, >= from __eq__ and __lt__. Now sorting works, max() works, and bisect works — all of them are defined in terms of <.

Note the tuple trick in both methods. Comparing tuples compares element by element, so (1, 2, 3) < (1, 3, 0) is exactly the version ordering you want, without writing it out.

Making a container

Three methods give you most of what a collection can do:

python

class Album:
    def __init__(self, photos: list[Photo]) -> None:
        self._photos = photos

    def __len__(self) -> int:
        return len(self._photos)

    def __getitem__(self, index):
        return self._photos[index]

    def __contains__(self, photo: Photo) -> bool:
        return photo in self._photos

python

len(album)            # __len__
album[0]              # __getitem__
album[1:3]            # __getitem__ with a slice object
photo in album        # __contains__
for photo in album:   # __getitem__ from 0 until IndexError!
list(album)

That last behaviour is a genuine piece of Python history: __getitem__ alone makes an object iterable, because the for loop falls back to calling it with 0, 1, 2… until IndexError. Prefer defining __iter__ explicitly — it is clearer and works for things that are not indexable.

Two more that matter:

__bool__ decides truthiness. Without it, Python falls back to __len__, so an empty album is falsy for free. Without either, every instance is truthy — which is why a custom object that "should" be empty still passes if obj.

__iter__ returning a generator is the easiest correct implementation:

python

    def __iter__(self):
        yield from self._photos

Arithmetic, and the reflected forms

python

    def __add__(self, other: "Money") -> "Money":
        if not isinstance(other, Money):
            return NotImplemented
        if other.currency != self.currency:
            raise ValueError(
                f"cannot add {self.currency} to {other.currency}"
            )
        return Money(self.amount + other.amount, self.currency)

    def __mul__(self, factor: int | Decimal) -> "Money":
        return Money(self.amount * factor, self.currency)

    def __rmul__(self, factor: int | Decimal) -> "Money":
        return self.__mul__(factor)          # 3 * money

__rmul__ is the reflected form. When Python evaluates 3 * money, it asks int.__mul__ first, which returns NotImplemented because it knows nothing about Money — then it tries money.__rmul__. Without it, money * 3 works and 3 * money raises, which is the kind of asymmetry that produces a bug report six months later.

NotImplemented is doing real work here. It is not an error; it is "I decline, ask the other operand".

Implement what you mean

Bad — operators borrowed for something they do not mean.

python

class Album:
    def __add__(self, photo: Photo) -> "Album":
        self._photos.append(photo)      # mutates, despite being `+`
        return self

    def __lt__(self, other: "Album") -> bool:
        return len(self) < len(other)   # albums have no natural order

Good — methods with names, and operators only where the meaning is obvious.

python

class Album:
    def with_photo(self, photo: Photo) -> "Album":
        return Album([*self._photos, photo])

    def __len__(self) -> int:
        return len(self._photos)

album + photo reads like addition and is a mutation returning self, so b = a + photo leaves a changed too — the aliasing surprise, imported into an operator where nobody will look for it. And __lt__ comparing lengths means sorted(albums) orders by size while album_a < album_b reads as something about albums, so any reader has to check.

The test: would a reader guess what this operator does without reading the class? Money + Money passes. Album + Photo does not. A named method costs six characters and removes the question.

The rest, by name

text

__call__          make an instance callable: obj(x)
__enter__/__exit__ a context manager
__getattr__       called only when normal lookup FAILS
__setattr__       called on EVERY attribute assignment
__slots__         fixed attributes: less memory, no __dict__
__format__        controls f"{obj:>10.2f}"
__index__         lets it be used as a list index
__copy__/__deepcopy__     control copying
__reduce__        control pickling

Two of those bite. __getattr__ runs only when the usual lookup fails, which makes it perfect for proxies and dangerous for typos — every misspelling becomes a call rather than an AttributeError. And __setattr__ intercepts every assignment, including the ones in __init__, so a naive implementation recurses infinitely; assign through object.__setattr__(self, name, value) inside it.

A cheat sheet to keep

text

THE PRINCIPLE
  syntax compiles to method calls, looked up on the TYPE
  protocols, not inheritance - define the method, it works
  setting obj.__len__ on an instance changes nothing

TEXT
  __repr__    for developers; define this on EVERYTHING
  __str__     for users; falls back to __repr__

EQUALITY
  __eq__ returning NotImplemented for unknown types
  defining __eq__ sets __hash__ to None
  define both, from the same fields - or stay unhashable
  mutable? unhashable is correct

ORDERING
  @total_ordering + __eq__ + __lt__  -> all six
  compare tuples of fields; they compare element by element

CONTAINERS
  __len__  __getitem__  __contains__  __iter__
  __bool__ falls back to __len__
  __getitem__ alone makes it iterable (0,1,2... until IndexError)
  prefer an explicit __iter__ yielding

ARITHMETIC
  __add__ __sub__ __mul__ ...
  __radd__ __rmul__ ...     for `3 * money`
  return NotImplemented to decline; Python tries the other side

  NotImplemented   a VALUE you return
  NotImplementedError  an EXCEPTION you raise
  confusing them makes == silently True

DESIGN
  would a reader guess what this operator does?
  Money + Money  yes.   Album + Photo  no - name the method
  never mutate in an operator that reads as producing a value

TRAPS
  __getattr__ runs only on FAILED lookup - typos become calls
  __setattr__ runs on every assignment - recurses without
    object.__setattr__(self, name, value)

Where to go next

You can now make your own types behave like the built-in ones, and — more useful — you know that there was never a difference in kind. Everything Python's syntax does is a method call on a type, which means the language has very few special cases and a lot of protocol.

Next is Descriptors and Properties, which explains the one piece of attribute access this lesson skipped. @property looks like syntax; it is an ordinary class implementing three dunder methods, and understanding it explains how methods themselves get bound to instances.

Before you move on, take a class you have written and add __repr__, __eq__ and __hash__. Then put two equal instances in a set and confirm you get one. If you get two, the hash and the equality disagree — which is exactly the bug those three methods exist to make visible.