beginner25 minby DevFox

Organising Code into Modules

Splitting one long file into several that import each other, what the if __name__ line is really doing, and a project layout that will not fight you later.

Python

Your photo script is four hundred lines. To change how captions are generated you scroll. To find where files are written you search. You know there is a function called something like clean_name but not whether it is above or below the thing that calls it.

The file is not too long because you wrote too much. It is too long because everything in it is in the same place. By the end of this lesson you will split a program across several files, know exactly what import does when it runs, and understand the line at the bottom of every serious Python script that beginners copy without knowing why.

A module is a file

That is the whole definition. Any .py file is a module, and importing it makes what is inside available somewhere else.

Put this in captions.py:

python

def make_caption(filename):
    """Turn a filename into a human-readable caption."""
    words = filename.replace("_", " ").replace(".jpg", "")
    return words.title()

And use it from main.py, sitting beside it:

python

import captions

print(captions.make_caption("sunrise_over_lisbon.jpg"))

No configuration, no registration, no build step. The file is the module, and its name without .py is the module name.

Four ways to import

python

import captions
captions.make_caption(name)          # always says where it came from

from captions import make_caption
make_caption(name)                   # shorter at every use

from captions import make_caption, clean_name    # several

import captions as cap               # renamed
cap.make_caption(name)

The first two are both fine and the choice is about reading. import captions keeps the origin visible at every call, which is valuable when a reader is trying to work out where something lives. from captions import make_caption is shorter, which is valuable when you use it forty times.

The one to avoid entirely:

python

from captions import *        # don't

That pulls in every name from the module without listing them. Now a reader cannot tell which names came from where, two modules can silently overwrite each other's names, and adding a function to captions.py can break an unrelated file that happened to have a name of its own.

What import actually does

The behaviour that surprises people: importing a module runs it, top to bottom, the first time.

Put this in captions.py:

python

print("captions module is being loaded")

def make_caption(filename):
    ...

Then import captions prints that line. Not when you call the function — when you import.

That is why definitions are all a module should normally contain at the top level. def and class statements just create things; anything else happens, and it happens to everyone who imports you, whether they wanted it or not.

Running is once per program, not once per import. Python keeps a record of what it has loaded, so importing the same module from five files loads it once and hands the same module out five times.

The line at the bottom

You have seen this and probably copied it:

python

if __name__ == "__main__":
    main()

Here is what it means. Python sets a variable called __name__ in every module. When a file is run directly it is set to "__main__"; when the file is imported it is set to the module's name.

So that line reads: "only do this if I am the program being run, not if I am being imported."

python captions.py

__name__ is "__main__"

You are running the file directly, so the guard passes and main() runs. This is the script.

import captions

__name__ is "captions"

Somebody wants one function out of it. The guard fails, main() does not run, and they get exactly what they asked for. This is the library.

One file, two situations. The guard is what lets it behave correctly in both.

Without it, a module cannot be both.

Bad — work at the top level of an importable file.

python

# captions.py
def make_caption(filename):
    ...

photos = load_all_photos()          # runs on import
for photo in photos:
    print(make_caption(photo.name))

Good — work behind the guard.

python

# captions.py
def make_caption(filename):
    ...

def main():
    for photo in load_all_photos():
        print(make_caption(photo.name))

if __name__ == "__main__":
    main()

In the first version, any file that writes from captions import make_caption to reuse one function also loads every photo and prints several hundred lines — because importing runs the file. Your test suite does it too, on every run. The guard costs two lines and makes the module importable and runnable at the same time, which is what you wanted from the start.

Packages: a folder of modules

When one folder of files is not enough structure, a package is a directory of modules:

text

photo_tools/
├── __init__.py
├── captions.py
├── loading.py
└── output.py
main.py

python

from photo_tools import captions
from photo_tools.loading import load_all_photos

The __init__.py file marks the directory as a package. It can be completely empty, and often is. It runs when the package is first imported, so it is where you would put anything that should happen once — or, commonly, a few imports that give your package a tidy public surface:

python

# photo_tools/__init__.py
from photo_tools.captions import make_caption
from photo_tools.loading import load_all_photos

Now users of the package write from photo_tools import make_caption without needing to know which file it lives in, which means you can move it later without breaking them.

A layout that will not fight you

For a program of any size:

text

project/
├── photo_tools/
│   ├── __init__.py
│   ├── captions.py
│   ├── loading.py
│   └── output.py
├── tests/
│   └── test_captions.py
├── main.py
└── README.md

Three ideas are doing the work here.

Code in a package, not loose at the top. Everything importable lives in one named folder, so there is one obvious answer to "where does this go".

One entry point. main.py is the thing you run. It should be short — parse arguments, call into the package, handle the top-level errors.

Tests beside the code, in their own folder. They are not part of what ships, but they live in the same repository so they travel with it.

Split modules by subject, not by kind. captions.py, loading.py, output.py each own an area. A helpers.py or utils.py starts as a convenience and becomes the folder where everything nobody could categorise goes to be forgotten — when you find yourself adding a fourth unrelated function to one, that is the signal to split it by subject.

Circular imports

Two modules importing each other is the structural problem you will meet first:

python

# loading.py
from output import save_photo

# output.py
from loading import load_photo

text

ImportError: cannot import name 'save_photo' from partially
initialized module 'output' (most likely due to a circular import)

The message is accurate. Python was part-way through running output.py when it reached the import of loading, which imports output, which is not finished yet.

You can defer an import inside a function to break the cycle, and occasionally that is the pragmatic answer. But the cycle is almost always telling you something true about your design: these two modules are entangled. Three fixes cover nearly every case.

Extract a third module
Move whatever both of them need into a module they can each import. Neither has to know about the other any more.
Merge them
If every function in one calls something in the other, they were one subject split across two files.
Pass the value in
Instead of importing what you need, take it as an argument. The caller already has both modules loaded.

Try them in this order — the first is right most of the time

Import order, by convention

Group imports at the top of the file in three blocks, separated by a blank line: the standard library, then installed packages, then your own modules. Tools do this automatically, and the habit makes it obvious at a glance how many dependencies a file has actually taken on.

python

import json
from pathlib import Path

import requests

from photo_tools.captions import make_caption

A cheat sheet to keep

text

MODULES
  a module IS a .py file
  import captions              captions.make_caption(x)
  from captions import make_caption
  import captions as cap
  from captions import *       <- never

WHAT IMPORT DOES
  runs the file top to bottom, the first time
  once per program, however many times imported
  so top level should be definitions only

THE GUARD
  if __name__ == "__main__":   run only when executed directly
      main()                   not when imported

PACKAGES
  a folder with __init__.py
  from photo_tools import captions
  from photo_tools.loading import load_all_photos
  __init__.py can re-export, giving a tidy public surface

LAYOUT
  project/
    photo_tools/   __init__.py + one module per subject
    tests/
    main.py        short: arguments, call in, top-level errors

  split by subject, never a utils.py

IMPORT ORDER
  standard library / installed packages / your own

TRAPS
  never name a file json.py, random.py, csv.py - it shadows
  circular imports mean two modules are entangled;
    extract a third, merge them, or pass the value in

Where to go next

You can now split a program across files and folders, and you know that import runs code — which explains the __main__ guard, the shadowing trap, and why circular imports fail the way they do. Those four hundred lines can become six files of sixty, each with one subject.

Next is Objects and Classes, the last major piece of Python syntax in this course. Modules group related functions; classes group data with the behaviour that belongs to it. The lesson also covers the cases — more common than tutorials admit — where a function and a dictionary are the better answer.

Before you move on, take the longest program you have and split it into at least two modules with a main.py that imports them. Add the __main__ guard. Then import one of your modules from the interactive prompt and call a function from it directly. Doing that once makes the difference between a script and a program concrete rather than theoretical.