intermediate30 minby DevFox

Files, Paths, and the Filesystem

pathlib beyond the basics: globbing, temporary files, atomic writes that survive a crash mid-write, permissions, and why string paths keep breaking on someone else's machine.

Python

The report job writes a JSON file every night. Last Tuesday the machine ran out of disk halfway through the write, and the process died. This morning every downstream job is failing on json.decoder.JSONDecodeError, because the file on disk is a truncated fragment — valid enough to open, invalid enough to break everything that reads it.

The foundations course covered opening files. This lesson is about the parts that only matter once real data and real failures are involved: writing so a crash cannot leave a half-file, scratch space that cleans itself up, and the permission and path assumptions that hold on your machine and not on the server.

Writing so a crash cannot corrupt

Bad — writing in place.

python

def write_report(path: Path, report: dict) -> None:
    path.write_text(json.dumps(report, indent=2), encoding="utf-8")

Good — write beside it, then rename.

python

def write_report(path: Path, report: dict) -> None:
    temporary = path.with_suffix(path.suffix + ".tmp")
    try:
        temporary.write_text(
            json.dumps(report, indent=2), encoding="utf-8"
        )
        temporary.replace(path)          # atomic on the same volume
    except BaseException:
        temporary.unlink(missing_ok=True)
        raise

Written in place

A file that exists, opens, and is wrong.

Opening for writing truncates the destination immediately, before a single byte of the new content arrives.

A full disk, a killed process or a power loss between those two moments leaves half a file — and the old good version is already gone.

Written beside, then renamed

Either the whole old file or the whole new one.

The destination is untouched until a complete file exists elsewhere, and the rename swaps them in one indivisible step.

A crash before the rename leaves a stray .tmp and a perfectly good original.

The question both versions are answering: what is on disk if the process dies halfway through the write?

Path.replace is a rename, which the operating system performs atomically on the same filesystem. That caveat is why the temporary goes beside the target rather than in /tmp, which is frequently a different volume and turns the rename into a copy — no longer atomic.

except BaseException rather than Exception is deliberate here: it also catches KeyboardInterrupt, so a Ctrl-C during the write still removes the fragment. The bare raise means it cleans up and stays out of the way, per the errors lesson.

Scratch space that cleans itself

Building a temporary path by hand invites two problems: the cleanup that does not run, and the collision when two copies of the program run at once.

dir=target.parent puts it on the right filesystem for the rename. delete=False is needed because you want the file to outlive the block — you are renaming it, not discarding it.

Never build temporary names yourself from a timestamp or a counter. Two processes doing the same thing in the same second get the same name, and the resulting corruption is intermittent and miserable to reproduce.

Paths that are not what they look like

pathlib handles separators. It does not handle the assumptions underneath.

expanduser matters because ~ is shell syntax — Python does not expand it, so open("~/config.json") looks for a directory literally named ~.

resolve matters whenever you compare paths or use one to make a security decision. Two strings can name the same file, and == on unresolved paths compares text:

The security case is worth spelling out. If a user supplies a filename and you join it to a directory, ../../etc/passwd escapes:

Resolve first, then check containment. Checking before resolving checks the wrong string.

Asking about a file

st_mtime is a float, and turning it into a real time is where the previous lesson applies:

Without tz=, you get a naive local datetime — the exact thing that lesson said to keep out of your program.

And the check-then-act problem, which is the same one from the foundations course in a new setting:

Finding things

Both return generators, which is what makes them usable on directories with a million entries — the memory lesson from generators, arriving where it matters.

Two practical notes. Case sensitivity follows the filesystem, so *.jpg finds DAWN.JPG on macOS and Windows and not on Linux; if it matters, filter on suffix.lower(). And rglob follows into everything, including .git and node_modules, so a recursive walk over a project directory is often far more work than intended:

Permissions, and failing usefully

Three failures, three different exceptions:

Catching OSError covers all of them and more, which is right when the response is the same and wrong when it is not — a missing config file may be fine and an unreadable one almost never is.

When creating something sensitive, set the mode as you create it rather than afterwards:

Changing permissions after creation leaves a window where the file existed with the default mode, and on a shared machine that window is enough.

A cheat sheet to keep

Where to go next

Your programs can now write files that a crash cannot corrupt, use scratch space that cleans itself up, and treat a user-supplied path as the security question it is. The write-then-rename pattern is the one to carry: it costs four lines and eliminates a class of corruption that is otherwise impossible to test for.

Next is Subprocesses and the Shell, which is the same territory one step out — running other programs, capturing what they say, and the argument-quoting mistake that turns a filename into a command.

Before you move on, replace one in-place write in your code with the temporary-and-rename version. Then kill the process mid-write — a sleep and a Ctrl-C will do it — and confirm the original file is still intact. That demonstration takes two minutes and is more convincing than the paragraph explaining it.