intermediate30 minby DevFox

Files, Paths, and the Filesystem

pathlib beyond the basics: globbing, temporary files, atomic writes that survive a crash mid-write, permissions, and why string paths keep breaking on someone else's machine.

Python

The report job writes a JSON file every night. Last Tuesday the machine ran out of disk halfway through the write, and the process died. This morning every downstream job is failing on json.decoder.JSONDecodeError, because the file on disk is a truncated fragment — valid enough to open, invalid enough to break everything that reads it.

The foundations course covered opening files. This lesson is about the parts that only matter once real data and real failures are involved: writing so a crash cannot leave a half-file, scratch space that cleans itself up, and the permission and path assumptions that hold on your machine and not on the server.

Writing so a crash cannot corrupt

Bad — writing in place.

python

def write_report(path: Path, report: dict) -> None:
    path.write_text(json.dumps(report, indent=2), encoding="utf-8")

Good — write beside it, then rename.

python

def write_report(path: Path, report: dict) -> None:
    temporary = path.with_suffix(path.suffix + ".tmp")
    try:
        temporary.write_text(
            json.dumps(report, indent=2), encoding="utf-8"
        )
        temporary.replace(path)          # atomic on the same volume
    except BaseException:
        temporary.unlink(missing_ok=True)
        raise

Written in place

A file that exists, opens, and is wrong.

Opening for writing truncates the destination immediately, before a single byte of the new content arrives.

A full disk, a killed process or a power loss between those two moments leaves half a file — and the old good version is already gone.

Written beside, then renamed

Either the whole old file or the whole new one.

The destination is untouched until a complete file exists elsewhere, and the rename swaps them in one indivisible step.

A crash before the rename leaves a stray .tmp and a perfectly good original.

The question both versions are answering: what is on disk if the process dies halfway through the write?

Path.replace is a rename, which the operating system performs atomically on the same filesystem. That caveat is why the temporary goes beside the target rather than in /tmp, which is frequently a different volume and turns the rename into a copy — no longer atomic.

except BaseException rather than Exception is deliberate here: it also catches KeyboardInterrupt, so a Ctrl-C during the write still removes the fragment. The bare raise means it cleans up and stays out of the way, per the errors lesson.

When the data must survive a power cut

A rename is atomic with respect to other processes, but the bytes may still be in the operating system's cache. If the machine losing power must not lose the file, flush and os.fsync before renaming:

python

with open(temporary, "w", encoding="utf-8") as file:
    file.write(text)
    file.flush()
    os.fsync(file.fileno())

Most applications do not need this. Databases and anything claiming durability do.

Scratch space that cleans itself

Building a temporary path by hand invites two problems: the cleanup that does not run, and the collision when two copies of the program run at once.

python

import tempfile
from pathlib import Path

with tempfile.TemporaryDirectory() as name:
    workspace = Path(name)
    (workspace / "resized.jpg").write_bytes(data)
    ...
# the whole directory is removed here, exception or not

python

with tempfile.NamedTemporaryFile(
    mode="w", suffix=".json", dir=target.parent, delete=False
) as file:
    file.write(text)
    temporary = Path(file.name)

temporary.replace(target)

dir=target.parent puts it on the right filesystem for the rename. delete=False is needed because you want the file to outlive the block — you are renaming it, not discarding it.

Never build temporary names yourself from a timestamp or a counter. Two processes doing the same thing in the same second get the same name, and the resulting corruption is intermittent and miserable to reproduce.

Paths that are not what they look like

pathlib handles separators. It does not handle the assumptions underneath.

python

p = Path("~/photos/../photos/dawn.jpg")

print(p.expanduser())      # ~ becomes the home directory
print(p.resolve())         # absolute, symlinks followed, .. removed
print(p.absolute())        # absolute, but NOT normalised

expanduser matters because ~ is shell syntax — Python does not expand it, so open("~/config.json") looks for a directory literally named ~.

resolve matters whenever you compare paths or use one to make a security decision. Two strings can name the same file, and == on unresolved paths compares text:

python

Path("photos/dawn.jpg") == Path("./photos/dawn.jpg")     # False
Path("photos/dawn.jpg").resolve() == Path("./photos/dawn.jpg").resolve()

The security case is worth spelling out. If a user supplies a filename and you join it to a directory, ../../etc/passwd escapes:

python

root = Path("uploads").resolve()
candidate = (root / user_supplied).resolve()

if not candidate.is_relative_to(root):
    raise ValueError(f"path escapes the upload directory")

Resolve first, then check containment. Checking before resolving checks the wrong string.

Asking about a file

python

p.exists()          # is there anything there
p.is_file()         # ...and is it a regular file
p.is_dir()
p.is_symlink()

stat = p.stat()
stat.st_size        # bytes
stat.st_mtime       # last modified, as a Unix timestamp

st_mtime is a float, and turning it into a real time is where the previous lesson applies:

python

from datetime import datetime, timezone
modified = datetime.fromtimestamp(stat.st_mtime, tz=timezone.utc)

Without tz=, you get a naive local datetime — the exact thing that lesson said to keep out of your program.

And the check-then-act problem, which is the same one from the foundations course in a new setting:

python

if path.exists():        # between these two lines, anything may happen
    path.unlink()        # FileNotFoundError

python

path.unlink(missing_ok=True)      # one operation, no gap

Finding things

python

root.glob("*.jpg")            # this directory
root.rglob("*.jpg")           # and every subdirectory
root.glob("**/2026-*/*.jpg")  # a pattern with structure

Both return generators, which is what makes them usable on directories with a million entries — the memory lesson from generators, arriving where it matters.

Two practical notes. Case sensitivity follows the filesystem, so *.jpg finds DAWN.JPG on macOS and Windows and not on Linux; if it matters, filter on suffix.lower(). And rglob follows into everything, including .git and node_modules, so a recursive walk over a project directory is often far more work than intended:

python

for path in root.rglob("*.jpg"):
    if any(part.startswith(".") for part in path.parts):
        continue

Permissions, and failing usefully

Three failures, three different exceptions:

python

try:
    data = path.read_text(encoding="utf-8")
except FileNotFoundError:
    ...                 # nothing there
except PermissionError:
    ...                 # there, but not yours to read
except IsADirectoryError:
    ...                 # there, but not a file

Catching OSError covers all of them and more, which is right when the response is the same and wrong when it is not — a missing config file may be fine and an unreadable one almost never is.

When creating something sensitive, set the mode as you create it rather than afterwards:

python

path.touch(mode=0o600)                    # owner read/write only
path.parent.mkdir(mode=0o700, parents=True, exist_ok=True)

Changing permissions after creation leaves a window where the file existed with the default mode, and on a shared machine that window is enough.

A cheat sheet to keep

text

WRITING SAFELY
  write to path.tmp, then temporary.replace(path)
  atomic ONLY on the same filesystem - keep the temp beside it
  except BaseException: unlink(missing_ok=True); raise
  flush + os.fsync before rename if power loss must not lose it

TEMPORARY SPACE
  tempfile.TemporaryDirectory()          removed on exit, always
  NamedTemporaryFile(dir=..., delete=False)   when you will rename it
  never build temp names yourself - collisions between processes

PATHS
  p.expanduser()     ~ is shell syntax; Python will not expand it
  p.resolve()        absolute, normalised, symlinks followed
  compare RESOLVED paths, never raw strings

  candidate = (root / user_input).resolve()
  candidate.is_relative_to(root)         resolve THEN check

ASKING
  exists() is_file() is_dir() is_symlink()
  stat().st_size   stat().st_mtime
  fromtimestamp(mtime, tz=timezone.utc)  <- never without tz

  do not check-then-act
  unlink(missing_ok=True), mkdir(exist_ok=True)

FINDING
  glob("*.jpg")   rglob("*.jpg")   glob("**/2026-*/*.jpg")
  generators - fine on a million files
  case sensitivity follows the filesystem
  rglob descends into .git and node_modules

PERMISSIONS
  FileNotFoundError / PermissionError / IsADirectoryError
  OSError covers all - only if the response is the same
  touch(mode=0o600), mkdir(mode=0o700) at creation, not after

ALWAYS
  encoding="utf-8"      newline="" for csv

Where to go next

Your programs can now write files that a crash cannot corrupt, use scratch space that cleans itself up, and treat a user-supplied path as the security question it is. The write-then-rename pattern is the one to carry: it costs four lines and eliminates a class of corruption that is otherwise impossible to test for.

Next is Subprocesses and the Shell, which is the same territory one step out — running other programs, capturing what they say, and the argument-quoting mistake that turns a filename into a command.

Before you move on, replace one in-place write in your code with the temporary-and-rename version. Then kill the process mid-write — a sleep and a Ctrl-C will do it — and confirm the original file is still intact. That demonstration takes two minutes and is more convincing than the paragraph explaining it.