intermediate30 minby DevFox

Configuration and Secrets

Layering defaults, files and environment variables; validating configuration at startup so it fails immediately; and keeping credentials out of your repository for good.

Python
Security

The deploy went out at five. At two in the morning the on-call engineer is looking at KeyError: 'PHOTO_API_KEY' from inside a worker, six hours after the process started, because that particular code path had not run until then. The variable was never set. Nothing checked at startup, so nothing said so while anyone was awake.

Configuration is the part of a program that differs between your laptop and production, and secrets are the part that must not be in your repository. By the end of this lesson you will have a place for both, validation that fails at startup rather than at 2am, and a clear rule about what never gets committed.

What counts as configuration

Anything that differs between environments, or that you would change without changing behaviour:

text

database URLs, API endpoints        differ per environment
credentials, tokens, signing keys   secret, differ per environment
timeouts, batch sizes, limits       tuning
feature flags                       operational switches
log level                           operational

What is not configuration: anything the program needs to be correct. A tax rate is data. A field name is code. Making them configurable adds a way to be wrong without adding anything.

The test that works: would you change this without changing any code, and would you expect a different value in production? If not, it is a constant, and a constant belongs in the source where it can be reviewed.

The environment, and files

Environment variables are the common denominator — every platform, container system and process manager sets them, and nothing has to parse a file format:

python

import os

api_key = os.environ["PHOTO_API_KEY"]           # required
folder = os.environ.get("OUTPUT_FOLDER", "out") # optional, defaulted

Note the two forms and the deliberate difference. os.environ[...] raises when something required is absent; .get with a default is for things that genuinely have one. That is the KeyError-versus-get distinction from the dictionaries lesson, and here it decides whether a misconfigured deploy fails loudly or runs wrongly.

Every value is a string, so anything else needs converting — and converting is where validation belongs:

python

limit = int(os.environ.get("BATCH_LIMIT", "100"))
debug = os.environ.get("DEBUG", "").lower() in {"1", "true", "yes"}

That boolean line matters more than it looks. bool("false") is True, because it is a non-empty string. Every codebase that gets this wrong has a flag that cannot be turned off.

For local development, a .env file keeps you from exporting variables by hand:

text

# .env  - never committed
PHOTO_API_KEY=dev-key-not-a-real-one
OUTPUT_FOLDER=./out
LOG_LEVEL=DEBUG

python

from dotenv import load_dotenv
load_dotenv()          # real environment variables still win

Commit a .env.example with every key and no real values, so a new contributor knows what to set.

Load it once, and validate it

Bad — reading the environment where it is used.

python

def upload(photo):
    key = os.environ["PHOTO_API_KEY"]
    limit = int(os.environ.get("BATCH_LIMIT", "100"))
    ...

Good — one settings object, built and validated at startup.

python

from pydantic import Field
from pydantic_settings import BaseSettings


class Settings(BaseSettings):
    api_key: str
    output_folder: Path = Path("out")
    batch_limit: int = Field(default=100, gt=0)
    log_level: str = "INFO"

    model_config = {"env_prefix": "PHOTO_", "env_file": ".env"}


def main() -> None:
    settings = Settings()          # raises here, at startup, if wrong
    ...

The first version is the 2am pager. A missing key is discovered whenever that function first runs, which may be hours in and is certainly not at deploy time. A misspelled BATCH_LIMIT silently becomes 100. Every call re-reads and re-parses. And nothing lists what the program actually needs — you find out by grepping for os.environ.

The second fails immediately, with a message naming every missing or invalid field at once. The class is the documentation of what this program requires, and the types are enforced, because this is a boundary and a type hint is not a runtime check — the rule from the JSON lesson, applied to the environment.

pydantic-settings is one option; a hand-written dataclass with a from_environment classmethod is entirely reasonable and the same shape.

Layers, and who wins

Configuration usually comes from several places at once. Fix the order and write it down:

Command-line arguments — highest
Environment variables
A config file
Defaults in code — lowest

More specific beats more general. A flag typed by a person outranks a file, because they are standing there fixing something.

python

parser.add_argument("--limit", type=int, default=None)
args = parser.parse_args()

limit = args.limit if args.limit is not None else settings.batch_limit

default=None rather than default=100 is what makes this work: it distinguishes "the user asked for 100" from "the user said nothing", which is the falsy-zero problem from the conditions lesson in its most practical form.

Secrets

The rules are short and absolute.

Never commit a secret. Not in code, not in a config file, not in a test fixture, not in a comment, not "temporarily". Git keeps history, so a secret committed once is in the repository until the history is rewritten — deleting it in the next commit achieves nothing.

If it was ever committed, it is compromised. Rotate it. The repository may have been cloned, mirrored, forked, or indexed by something scanning public code. Treat the exposure as real because you cannot prove it was not.

Keep them out of the process listing. ps shows command lines to other users on the machine, so a secret passed as --api-key=... is visible. Environment variables and files are better.

Where they should live:

Local
A gitignored .env, beside a committed .env.example that names every variable and sets none of them.
CI
The platform's encrypted secret store, injected into the job.
Production
A secret manager, or the platform's own mechanism, injected as environment variables at start.

Three environments, three mechanisms, one shape — the program always reads an environment variable

Add the protections that make a mistake unlikely:

text

.gitignore   .env  *.pem  secrets/  credentials.json

and a pre-commit secret scanner — gitleaks, detect-secrets — so the check happens before the commit rather than after the incident.

A secret in a log is a secret in an index

The logging lesson covered this and it belongs here too: logs are copied, retained and indexed. A settings object printed during startup diagnostics puts the API key into every system that touches your logs. Give secret fields a repr that hides them:

python

class Settings(BaseSettings):
    api_key: SecretStr        # prints as **********

pydantic's SecretStr requires an explicit .get_secret_value() to read, so printing it by accident is not possible.

Failing usefully at startup

The whole point of loading configuration first is the error message:

text

$ photo-tools photos/
Configuration error:
  PHOTO_API_KEY      field required
  PHOTO_BATCH_LIMIT  input should be greater than 0 (got -5)

See .env.example for the full list.

Two missing things reported together, named by their real environment variable, with a pointer to the documentation — and the process exits non-zero, so a deployment system notices.

python

def main() -> None:
    try:
        settings = Settings()
    except ValidationError as error:
        print("Configuration error:", file=sys.stderr)
        for problem in error.errors():
            field = problem["loc"][0]
            print(f"  PHOTO_{str(field).upper():<18} {problem['msg']}",
                  file=sys.stderr)
        raise SystemExit(2)

Compare that with KeyError: 'PHOTO_API_KEY' six hours in. Same underlying mistake; entirely different night.

A cheat sheet to keep

text

WHAT IS CONFIGURATION
  differs per environment, or changes without changing behaviour
  URLs, credentials, timeouts, limits, flags, log level
  NOT: anything the program needs to be correct - that is code

READING IT
  os.environ["KEY"]              required - raises if absent
  os.environ.get("KEY", "d")     genuinely optional
  everything is a string; convert AND validate

  bool("false") is True          <- the flag that cannot be off
  use: value.lower() in {"1", "true", "yes"}

STRUCTURE
  one Settings object, built ONCE in main()
  validated at startup, not at first use
  the class is the documentation of what the program needs

  never read os.environ deep inside the code

LAYERS - most specific wins
  defaults < config file < environment < command line
  argparse default=None, so "not given" differs from a real value

SECRETS
  never commit one - git history keeps it forever
  committed once = compromised = rotate it
  not on the command line - ps shows it to other users

  local .env (gitignored) + a committed .env.example
  CI: encrypted secret store
  prod: secret manager, injected as environment variables

  .gitignore: .env *.pem secrets/
  a pre-commit secret scanner: gitleaks, detect-secrets
  SecretStr so printing the settings cannot leak it

FAILING
  report every problem at once, named as the real variable
  exit non-zero so the deploy notices

Where to go next

Your program now knows what it needs, says so immediately when it does not have it, and keeps its credentials out of your repository. The habit that matters most is the startup validation: every configuration bug becomes a message at deploy time rather than an incident later.

Next is Code Quality Tooling, which automates the conventions this course has been describing. Formatters and linters catch a surprising number of the mistakes from earlier lessons — the mutable default, the bare except, the unused import — without anyone having to remember them in review.

Before you move on, take a program you have and list everything it reads from the environment. Then build one settings object that loads all of it at startup and fails with a readable message. Run it with a variable deliberately missing. The difference between that message and a KeyError six hours in is the entire lesson.