advanced35 minby DevFox

Distributing and Versioning

Wheels and source distributions, native dependencies and the platforms they force you to build for, publishing safely, and the supply-chain concerns that come with being a dependency.

Python
Security

The release works. Two hours later an issue arrives: install fails on Windows with a wall of compiler errors, because your package has a C extension and you published only a source distribution. Everyone on Linux is fine. Everyone else needs a toolchain they do not have.

Publishing is where a library meets every machine that is not yours. By the end of this lesson you will know what the two artefact formats actually are, what native code does to your build matrix, how to publish without long-lived credentials, and what being a dependency obliges you to do about supply-chain risk.

Wheels and source distributions

bash

python -m build

text

dist/
├── photo_tools-1.2.0-py3-none-any.whl
└── photo_tools-1.2.0.tar.gz

A wheel

Already built.

Installing it is unpacking files into place. No build step, no compiler, fast and predictable.

A source distribution

Built on the user's machine.

The fallback when no compatible wheel exists — so it needs whatever your build needs, on a machine you know nothing about.

It is also what distributions and auditors use to reproduce the build.

Publish both. One is what almost everyone gets; the other is what happens when they do not.

The filename is a specification, not decoration:

text

photo_tools - 1.2.0 - py3 - none - any .whl
   name       version   |      |     |
                     python   ABI  platform

py3-none-any is a pure Python wheel: any Python 3, any platform, one file. That is what you get when there is no compiled code, and it is the easy case.

What native code costs

With a C or Rust extension, one wheel is no longer enough:

text

photo_tools-1.2.0-cp311-cp311-manylinux_2_17_x86_64.whl
photo_tools-1.2.0-cp311-cp311-macosx_11_0_arm64.whl
photo_tools-1.2.0-cp312-cp312-win_amd64.whl
...

The matrix is Python versions times platforms times architectures — commonly twenty or more files, each built on (or for) the right system. cibuildwheel in CI is how this is done in practice; doing it by hand is not viable.

manylinux is the tag that makes Linux wheels portable: built in a controlled image against an old glibc, so they run on distributions newer than the build environment. auditwheel checks and repairs them.

The judgement, from the performance lesson: native code is a distribution decision as much as a speed one. A measured speedup that turns one file into a twenty-job build matrix, plus a category of install failure you cannot reproduce, may still be correct — but it should be a decision made with the cost in view.

One source of the version number

Bad — the version written in two places.

toml

# pyproject.toml
version = "1.2.0"

python

# photo_tools/__init__.py
__version__ = "1.1.0"        # somebody forgot

Good — one source, read by the other.

toml

[project]
dynamic = ["version"]

[tool.setuptools.dynamic]
version = { attr = "photo_tools.__version__" }

python

from importlib.metadata import version

__version__ = version("photo-tools")     # from the installed metadata

Two hand-maintained copies diverge — not usually, but on the release where someone was in a hurry. Then a bug report says 1.1.0, PyPI says 1.2.0, and nobody can tell which code is actually running. Either read the version from the package metadata at runtime, or derive it from a git tag with a plugin like setuptools-scm, so the tag is the single source.

Publishing safely

bash

python -m build
python -m twine check dist/*
python -m twine upload --repository testpypi dist/*
python -m twine upload dist/*

Rehearsing on TestPyPI is worth the two minutes, because a version on PyPI can never be reused. You can yank it — hiding it from new installs while leaving it available to anything that pinned it — but you cannot replace or re-upload it. A mistake is a permanent record plus a new version number.

For credentials, use trusted publishing rather than a token:

yaml

jobs:
  publish:
    environment: release
    permissions:
      id-token: write
    steps:
      - uses: pypa/gh-action-pypi-publish@release/v1

The CI job proves its identity to PyPI directly and receives a short-lived credential. There is no API token in your secrets to leak, rotate, or accidentally print into a log — which the configuration lesson identified as the failure mode that keeps happening.

If you must use a token, scope it to the single project.

Being a dependency

Once people install your package, you are part of their supply chain, and a few obligations follow.

Pin nothing, constrain sensibly. The packaging lesson's rule: >=10.0,<11. An exact pin in a library is an unsatisfiable conflict for anyone using you and anything else.

Keep your own build inputs pinned. Your CI and release pipeline should install from a lock file, so a compromised release of a build dependency does not silently end up inside your wheel.

Publish provenance. Trusted publishing produces attestations linking the artefact to the workflow and commit that built it — which is what lets a consumer verify that a file on PyPI came from your repository rather than from someone with your password.

Have a security contact. A SECURITY.md saying where to report an issue privately, because the alternative is a public issue describing a vulnerability in something people are running.

Respond to advisories. pip-audit in CI covers your own dependencies; a vulnerability in one of them is one in you, and your users find out from a scanner rather than from you.

And the two that catch new maintainers:

text

enable 2FA on your PyPI account       an account takeover is
                                      a supply-chain attack
name your package carefully           a typo-squatted neighbour
                                      installs from a mistyped name

Supporting versions

toml

requires-python = ">=3.11"

That line is a promise, and raising it is a breaking change from the previous lesson's table — someone on 3.10 gets an older release from a routine pip install -U, which is the mechanism working correctly and is still a surprise if you did not announce it.

A workable policy: support the versions that are not end-of-life, drop them when they are, and say so in the changelog a release in advance. Test on every one you claim:

yaml

strategy:
  matrix:
    python-version: ["3.11", "3.12", "3.13"]

Claiming support you do not test is how a release goes out that does not import on the oldest version in the range.

A cheat sheet to keep

text

ARTEFACTS
  python -m build   ->  a wheel and an sdist; publish BOTH
  wheel  built; installing is unpacking - no compiler needed
  sdist  source; built on the user's machine if no wheel fits

  name-version-pythontag-abi-platform.whl
  py3-none-any = pure Python, one file, the easy case

NATIVE CODE
  one wheel per python x platform x architecture - often 20+
  cibuildwheel in CI; auditwheel + manylinux for portability
  native code is a DISTRIBUTION decision, not only a speed one

VERSION NUMBER
  one source of truth
  dynamic version from the package, or setuptools-scm from a tag
  two hand-maintained copies diverge on the rushed release

PUBLISHING
  twine check dist/*
  unzip -l dist/*.whl        look inside BEFORE uploading
  TestPyPI first
  a version on PyPI can NEVER be reused - only yanked

  trusted publishing (OIDC) instead of a long-lived token
  no secret to leak, rotate, or print into a log
  otherwise: a token scoped to one project

BEING A DEPENDENCY
  constrain ranges, never pin exactly, in a library
  pin your OWN build inputs with a lock file
  publish attestations; keep a SECURITY.md
  pip-audit in CI - their vulnerability report is your problem
  2FA on the account; account takeover IS a supply-chain attack

VERSIONS
  requires-python = ">=3.11"   raising it is a BREAKING change
  test every version you claim, in a matrix
  support what is not end-of-life; announce drops a release ahead

Where to go next

You can now ship something installable on machines unlike yours, publish without a credential that can leak, and meet the obligations that come with being in someone else's dependency tree. The one to act on first is trusted publishing — it removes an entire category of incident and takes about ten minutes to set up.

Next is Metaprogramming and Its Limits, the last lesson of this course. It returns to the machinery from the first four lessons and asks the question this course keeps asking in different forms: when does clever code cost more than the duplication it removed?

Before you move on, build a wheel of something you have written and look inside it with unzip -l. Check that every module you expect is there. In a project with any subpackages at all, there is a real chance something is missing — and finding that before publishing is the whole reason to look.