advanced35 minby DevFox

Metaprogramming and Its Limits

inspect, ast and code generation; why eval and exec are almost never the answer; and recognising the point where clever machinery costs more than the duplication it removed.

Python

The new engineer has been stuck for two days. They are trying to find where Photo.from_api is defined, and it is not defined anywhere — a metaclass reads a schema file at import time and generates one classmethod per endpoint. Grep finds nothing. The editor's go-to-definition finds nothing. The code that built it is four hundred lines and removed about sixty lines of duplication.

Every technique in this course can be used to write code that writes code. This closing lesson is about the boundary: what metaprogramming genuinely buys, what it costs in ways that do not appear in a diff, and how to tell which side of the line you are on.

The tools, briefly

You have met most of these already.

python

getattr(obj, name)              # attribute access by name
setattr(obj, name, value)
hasattr(obj, name)
type(name, bases, namespace)    # build a class at runtime

__init_subclass__               # react to subclass creation
__set_name__                    # a descriptor learns its own name
__getattr__                     # handle a failed lookup

Two more from the standard library.

inspect reads the structure of live objects:

python

import inspect

inspect.signature(func)          # parameters, defaults, annotations
inspect.getsource(func)          # the source text, if available
inspect.stack()                  # the call stack

ast parses source into a tree without running it:

python

import ast

tree = ast.parse(Path("captions.py").read_text(encoding="utf-8"))
for node in ast.walk(tree):
    if isinstance(node, ast.FunctionDef):
        print(node.name, node.lineno)

ast is the right tool for anything that analyses or rewrites code — a linter, a codemod, a migration. It is static, so it cannot be surprised by what happens at runtime, and it cannot execute anything.

The line

Bad — generating an interface nobody can read.

python

class APIClient(metaclass=EndpointMeta):
    _schema = "endpoints.yaml"      # 40 methods appear from here

Good — writing the interface out.

python

class APIClient:
    def get_photo(self, photo_id: str) -> Photo:
        return self._request("GET", f"/photos/{photo_id}", Photo)

    def list_albums(self, *, limit: int = 100) -> list[Album]:
        return self._request("GET", "/albums", list[Album],
                             params={"limit": limit})
    # ...38 more, each two lines

The generated version removes eighty lines of near-identical code and costs: no autocomplete, no go-to-definition, no type checking, no signature in help(), a stack trace through generated frames, and a two-day onboarding delay per new engineer. The eighty lines it saved were boring, and boring is not the same as bad — they are greppable, individually type-annotated, and any one of them can diverge when an endpoint turns out to be different.

The generic _request helper is where the real duplication lives, and extracting it is an ordinary refactor. What remains is a readable list of what this client can do.

The test: does the abstraction remove duplication, or only repetition? Duplication is the same decision expressed several times, and changing it means changing all of them. Repetition is a similar shape with different content — forty endpoints that will each drift independently. Metaprogramming is good at collapsing the first and bad at the second.

What it costs

Worth stating plainly, because each one is invisible in the diff that introduces it.

Tools stop working
Type checkers, editors, documentation generators and refactoring tools all read source. What exists only at runtime is invisible to them, and dynamic attributes become Any — the contagion from the typing lesson.
Errors move
A typo in a generated name is not an AttributeError at the definition. It is a missing method, discovered later, somewhere else.
Stack traces get longer and less useful
Frames from generated or wrapped code sit between the caller and the failure.
The audience shrinks
Everyone can read a method. Fewer people can read a metaclass, and the ones who can are not always available when it breaks.
Import time grows
Anything computed at class creation runs on every import, for everyone.

Five costs, none of which shows up in the pull request that introduces them

None of these are reasons never to do it. They are the price, and the mistake is paying it without noticing.

eval and exec

The most direct form, and almost always the wrong one.

python

value = eval(user_input)          # arbitrary code execution
exec(generated_source)

eval on anything a user influenced is remote code execution. There is no safe subset achievable by filtering — the standard attacks reach the interpreter through attribute chains on ordinary objects, and every proposed sandbox in Python's history has been broken.

The legitimate uses are narrow and specific:

python

ast.literal_eval("{'a': 1}")     # literals ONLY - safe

literal_eval parses literals and refuses anything else. It is the right answer for "this string contains a Python literal".

For arithmetic from users, parse it yourself with ast and walk the tree, allowing only the node types you intend. For configuration, use a data format — JSON, TOML, YAML with safe_load — as the external-data lesson set out.

The one place exec is defensible is generating code you fully control for performance, which is what dataclasses does when it builds __init__. Note what makes that acceptable: the inputs are your own field definitions, the generated source is available for inspection, and it is a heavily tested standard library module. Very little application code meets that bar.

The order to try things

Almost everything people reach for metaprogramming to do has a simpler answer that keeps the tooling working.

A function
The answer more often than anyone expects.
A dataclass
Generated __init__, __repr__ and __eq__ that a checker can still see.
A decorator
Behaviour wrapped around a function.
A descriptor
Attribute behaviour that repeats across fields.
__init_subclass__
React to a subclass being created.
A class decorator
Transform a class that already exists.
A metaclass
The first thing on this list that sees the namespace before the class is created — which is the only reason to prefer it to the two above.
Code generation into a file
If you must generate, generate something you can commit.
exec / eval
Almost never.

Work down this list, and stop at the first thing that does the job. Most projects never get past four.

Item eight deserves the attention. If you genuinely have forty similar methods derived from a schema, generate them into a .py file, commit it, and regenerate when the schema changes:

bash

python -m tools.generate_client endpoints.yaml > src/client/_api.py

Now the code exists. Grep finds it, the editor navigates to it, the checker checks it, and the diff shows what changed when the schema did — you get the generation and keep every tool. It is how protobuf, OpenAPI clients and database model generators work, and it is almost always better than doing it at import time.

If you do it anyway

Sometimes it is right — a framework, a serialisation library, a genuine case where the alternative is worse. Then:

Keep it in one place. One module that does the clever thing, with a docstring explaining the mechanism and why.

Make it statically visible where possible. Ship a .pyi stub so checkers and editors see the generated surface even though it does not exist in source.

Fail at import, not at use. Validate what you generate as you generate it, so a mistake is a startup error rather than a missing attribute later.

Test the generated result, not the generator. Assert that Photo.from_api exists and behaves correctly — that is what users depend on.

Write down why the simpler options were rejected. In six months that note is what tells someone whether the constraint still holds.

A cheat sheet to keep

text

THE TOOLS
  getattr/setattr/hasattr        by name, at runtime
  type(name, bases, ns)          a class at runtime
  __init_subclass__, __set_name__, __getattr__
  inspect                        signatures, source, the stack
  ast                            parse WITHOUT running - codemods,
                                 linters, analysis

THE TEST
  does this remove DUPLICATION or only REPETITION?
    duplication  one decision expressed several times    -> abstract
    repetition   a similar shape, different content      -> write it

WHAT IT COSTS - none of it visible in the diff
  type checkers, editors, docs and refactoring tools go blind
  errors move from the definition to somewhere later
  stack traces gain frames nobody wrote
  fewer people can maintain it
  import time grows, for everyone

EVAL AND EXEC
  eval on user input IS remote code execution
  no filter makes it safe; every Python sandbox has been broken
  ast.literal_eval          literals only - the safe one
  a data format for configuration
  exec is defensible only for code you fully control, inspectable,
  and heavily tested - what dataclasses does

THE ORDER
  function > dataclass > decorator > descriptor >
  __init_subclass__ > class decorator > metaclass >
  GENERATE TO A COMMITTED FILE > exec

  generating to a file keeps grep, the editor, the checker and
  the diff - it is how protobuf and OpenAPI clients work

IF YOU DO IT
  one module, documented
  ship a .pyi so tools can see the surface
  validate at import, so failures are early
  test the RESULT, not the generator
  record why the simpler options were rejected

Where to go next

That closes Advanced Python. You started this course able to build and maintain real Python, and you now know what is underneath it: the data model that makes syntax into method calls, descriptors and the MRO that decide where an attribute comes from, the import system, how memory is reclaimed, what the GIL actually prevents, three models of concurrency, where performance really goes, and what it takes to publish something other people depend on.

The thread running through the whole course is the same question, asked in different places: what does this cost, and who pays it? A metaclass costs everyone who inherits. An unbounded cache costs the process at 3am. A bare Callable costs every caller their type checking. A native extension costs your build matrix. None of those are reasons not to — they are the second half of a decision that is usually made with only the first half in view.

Before you go, take the largest piece of machinery in your own codebase — the cleverest thing you or someone else wrote — and try to state what it costs and who pays. If the answer comes easily, it was probably a good decision. If it takes a while, that is worth knowing too.