advanced35 minby DevFox

The Import System

Finders, loaders and sys.path; packages and namespace packages; circular imports and the restructuring that fixes them; and deferring an expensive import without hiding it.

Python

Two modules import each other, and the error names a "partially initialized module". A test passes alone and fails in the suite. A module-level constant is stale because something imported it by value before it was set. A colleague's checkout imports a different file than yours from the same line of code.

All four are the import system, and all four are explicable once you know what happens between typing import x and having x. By the end of this lesson you will know the search, the cache, why circular imports fail exactly where they do, and how to defer an import without hiding it.

What import does, step by step

python

import photo_tools.captions

Check sys.modules
If the name is already there, bind it and stop. This is why importing something twenty times costs almost nothing.
Find it
Ask each finder in sys.meta_path in turn until one produces a spec — a description of where the module is and how to load it.
Create the module object
And, critically, put it in sys.modules before executing anything in it.
Execute the body
Into that module's namespace, top to bottom.
Bind the name
In the namespace that asked for it.

Step three explains circular imports. Step one explains almost everything else.

python

import sys

sys.modules["photo_tools.captions"]      # the module object
sys.modules.keys()                       # everything loaded

Where it looks

sys.path is a list of locations, searched in order:

python

import sys
for entry in sys.path:
    print(entry)

text

''                                   <- the script's directory
/usr/lib/python3.12
.../site-packages                    <- installed packages

The first entry is the directory of the script you ran — not your current working directory, which is a distinction that matters when a program is run from elsewhere.

Because it is searched first, a local file shadows anything installed:

text

project/
├── random.py          <- yours
└── main.py            import random  ->  finds YOURS

The failure is spectacular and the message never mentions your file: something deep inside a library calls random.choice, gets an AttributeError, and the traceback points at the library. Naming a file after a standard-library module is a mistake worth checking for whenever imports behave strangely.

The finders themselves are extensible — sys.meta_path is an ordinary list, which is how import hooks for zip files, lazy loading and test tooling work. You will rarely write one, and knowing they are objects in a list explains how such tools are possible.

Packages, and what init costs

A directory with __init__.py is a regular package. The file runs when the package is first imported — before any submodule.

python

# photo_tools/__init__.py
from photo_tools.captions import make_caption
from photo_tools.loading import load_all_photos

That gives a tidy public surface, and it has a cost: importing anything from the package now executes both submodules and everything they import. A CLI whose startup imports numpy because __init__.py re-exports one function from a module that uses it pays that on every invocation.

Keep __init__.py cheap. Re-exports are fine; work is not.

A directory without __init__.py is a namespace package, which can be split across several sys.path entries. That is occasionally what you want and is more often an accident — a missing __init__.py in a subpackage still imports, so the mistake surfaces later as a file missing from your built wheel, which is the packaging bug the src layout was designed to catch.

Circular imports, precisely

python

# loading.py
from photo_tools.output import save_photo

def load_photo(path): ...

python

# output.py
from photo_tools.loading import load_photo

def save_photo(photo): ...

text

ImportError: cannot import name 'load_photo' from partially
initialized module 'photo_tools.loading' (most likely due to a
circular import)

Trace it against the five steps. Python starts executing loading, puts the empty module in sys.modules, and reaches line one. That imports output, which starts executing and reaches its line one, which imports loading — found in sys.modules at step one, so no error, but it is the empty module from a moment ago. load_photo has not been defined yet.

Two things follow that are worth knowing.

import x survives where from x import y does not:

python

from photo_tools import loading        # binds the module object

def save_photo(photo):
    return loading.load_photo(...)     # looked up when CALLED

By the time anything calls save_photo, loading has finished executing. The name is resolved at call time rather than at import time. This works and it is a workaround, not a fix.

The real fix is structural. A cycle means the two modules are one thing, or a third thing is trying to get out. In order of preference: extract the shared piece into a module both can import; merge them if they were always one subject; or invert the dependency by passing what is needed as an argument rather than importing it.

Type-only imports do not have to exist at runtime

Annotations frequently create cycles that nothing else needs. TYPE_CHECKING is false at runtime and true for a type checker, so the import happens only where it is needed:

python

from __future__ import annotations
from typing import TYPE_CHECKING

if TYPE_CHECKING:
    from photo_tools.output import Writer

def save(writer: "Writer") -> None: ...

The future import makes all annotations strings, so the quotes become optional and no runtime lookup happens.

Import by reference, not by value

Bad — copying a value out of a module at import time.

python

from photo_tools.settings import BATCH_LIMIT

def process(photos):
    return photos[:BATCH_LIMIT]

Good — reading it through the module when it is used.

python

from photo_tools import settings

def process(photos):
    return photos[:settings.BATCH_LIMIT]

from x import y binds the current value of y into your namespace, once, at import time. If anything later reassigns settings.BATCH_LIMIT — configuration loaded at startup, a test monkeypatching it, a feature flag — your module keeps the old value and nothing indicates it.

This is also why monkeypatch.setattr("photo_tools.settings.BATCH_LIMIT", 5) appears to do nothing in a test: it changes the attribute on the settings module, and the module under test is holding its own copy. The mocking lesson's rule — patch where the name is looked up — is this same fact from the testing side.

For constants that genuinely never change, from x import Y is fine and clearer. For anything configurable, go through the module.

Deferring an import

An import inside a function runs on first call, not at module load:

python

def generate_chart(data):
    import matplotlib.pyplot as plt      # only if this is called
    ...

Three legitimate reasons: the dependency is heavy and rarely used, it is optional and may not be installed, or it breaks a cycle you cannot restructure today.

The cost is that it hides a dependency from anyone reading the imports at the top, and moves an ImportError from startup to whenever that function first runs — which is the same late-failure problem as the configuration lesson. Comment it:

python

def generate_chart(data):
    # Deferred: matplotlib adds ~400ms to startup and only the
    # `report --chart` path needs it.
    import matplotlib.pyplot as plt

For an optional dependency, fail with something useful:

python

try:
    import orjson as json_impl
except ImportError:
    import json as json_impl

Reloading, and why not to

python

import importlib
importlib.reload(module)

reload re-executes a module's body into the existing module object. Every name imported from it elsewhere still points at the old objects, and every instance of a class defined in it is an instance of the old class — so isinstance starts returning False for things that plainly are instances.

It is a tool for interactive experimentation. For a program, restart. The -X importtime flag is the related tool worth knowing, because it answers "why does startup take two seconds":

bash

python -X importtime -m photo_tools.main 2>&1 | sort -k2 -rn | head

That prints the cumulative time for every import, and a slow CLI is nearly always one heavy library pulled in by an __init__.py that did not need it.

A cheat sheet to keep

text

THE FIVE STEPS
  1 sys.modules hit?  bind and stop      <- the cache
  2 finders in sys.meta_path -> a spec
  3 create the module, ADD TO sys.modules
  4 execute the body                      <- step 3 precedes this
  5 bind the name locally

WHERE IT LOOKS
  sys.path, in order; sys.path[0] is the SCRIPT's directory
  a local random.py shadows the standard library, and the
  traceback will point at the library, not at your file

PACKAGES
  __init__.py runs before any submodule - keep it cheap
  re-exports fine, work not; it is paid on every startup
  no __init__.py = namespace package: usually an accident

CIRCULAR IMPORTS
  the module is in sys.modules but only PARTLY executed
  `from x import y` fails; `import x` + x.y at call time works
  that is a workaround - the fix is structural:
    extract a third module, merge them, or pass it as an argument
  TYPE_CHECKING + `from __future__ import annotations`
    for cycles that only annotations need

BY VALUE VS BY REFERENCE
  from settings import LIMIT     copies the value ONCE
  from x import settings         reads it when used     <- prefer
  this is why patching a module attribute "does nothing"

DEFERRING
  import inside a function: heavy, optional, or breaks a cycle
  always comment why - it hides a dependency and moves
  ImportError from startup to first call

TOOLS
  python -X importtime           what is slowing startup
  importlib.reload               interactive only; breaks isinstance

Where to go next

You can now explain every import failure you are likely to meet: the shadowed module, the partially initialised one, the stale constant, and the patch that does nothing. The distinction to carry is by-value versus by-reference — it decides both how circular imports break and why some monkeypatches have no effect.

Next is Memory, Reference Counting, and Garbage Collection, which asks what happens to all these objects when nothing needs them. It explains when memory is actually released, the cycles that reference counting cannot free on its own, and why measuring memory in Python is harder than it looks.

Before you move on, run python -X importtime on something you have written and read the slowest lines. Then look at whether your package's __init__.py is pulling in things most callers never use. Startup time is the cost of that file, and almost nobody has ever looked.