intermediate30 minby DevFox

Working with JSON and External Data

Parsing data you did not create: validating shape before trusting it, custom encoders, numbers and dates that do not survive a round trip, and failing loudly at the boundary.

Python

The API returns a photo record. You read data["size"], multiply it, and store the result. It has worked for a year. Today the provider shipped a change where size is sometimes null for photos still being processed, and your code raised a TypeError deep inside a formatting function, forty lines from the thing that was actually wrong.

Data from outside your program is a hope, not a fact. By the end of this lesson you will parse it safely, know what JSON silently does to your values, and be able to draw the boundary where untrusted shapes become types you can rely on.

Parsing, and the four functions

python

import json

config = json.loads(text)                  # from a string
report = json.dumps(data, indent=2)        # to a string

with open(path, encoding="utf-8") as file:
    config = json.load(file)               # from a file

with open(path, "w", encoding="utf-8") as file:
    json.dump(data, file, indent=2)        # to a file

The s means string. That is the only difference, and mixing them up gives an error that names neither.

Two arguments to dumps earn their place:

python

json.dumps(data, indent=2, ensure_ascii=False, sort_keys=True)

indent=2 makes it readable and diffable. ensure_ascii=False keeps "Praça" as "Praça" rather than "Praça" — correct whenever you are writing UTF-8, which you always are. sort_keys=True makes output deterministic, which matters if the file is committed or compared.

What JSON does to your types

JSON has six types. Python has many. Something is lost each way.

text

JSON            Python
object      ->  dict
array       ->  list          (a tuple becomes a list, and stays one)
string      ->  str
number      ->  int or float
true/false  ->  bool
null        ->  None

Three consequences that produce real bugs.

Tuples become lists
Round-trip a (1920, 1080) and you get [1920, 1080]. Anything comparing it to a tuple now fails.
Dictionary keys become strings
Always. A lookup by data[1] that worked before saving fails after loading, with a KeyError naming a number you can see in the file.
Dates and decimals have no representation
Both raise TypeError on dumps, so you convert on the way out and back on the way in.
Storing money as a JSON number would make it a float, which the numbers lesson explained is the one thing you must not do.

Every one of these survives a round trip looking almost right

Keys, concretely:

python

json.loads(json.dumps({1: "a"}))     # {'1': 'a'} - the key changed type

And the conversion for dates and decimals, on the way out:

python

json.dumps({"at": moment.isoformat(), "price": str(price)})

And convert back on the way in, because what you get is a string:

python

at = datetime.fromisoformat(data["at"])
price = Decimal(data["price"])

Custom encoders for your own types

For a type you control, default= handles the conversion in one place rather than at every call site:

python

def encode(obj):
    if isinstance(obj, datetime):
        return obj.isoformat()
    if isinstance(obj, Decimal):
        return str(obj)
    raise TypeError(f"cannot serialise {type(obj).__name__}")

json.dumps(data, default=encode)

Raising for the unknown case is deliberate — returning str(obj) instead would silently serialise anything at all, including objects whose repr contains a memory address.

Parse, do not cast

Now the central habit.

Bad — trusting the shape and reaching into it.

python

def load_photo(raw: str) -> Photo:
    data = json.loads(raw)
    return Photo(
        name=data["name"],
        size=data["size"],
        city=data.get("city", ""),
    )

Good — checking the shape at the point it arrives.

python

def load_photo(raw: str) -> Photo:
    try:
        data = json.loads(raw)
    except json.JSONDecodeError as error:
        raise InvalidPhotoError(f"not valid JSON: {error}") from error

    if not isinstance(data, dict):
        raise InvalidPhotoError(f"expected an object, got {type(data)}")

    try:
        name = data["name"]
        size = data["size"]
    except KeyError as error:
        raise InvalidPhotoError(f"missing field {error}") from error

    if not isinstance(size, int):
        raise InvalidPhotoError(f"size must be a whole number, got {size!r}")

    return Photo(name=name, size=size, city=data.get("city", ""))

The first version does not fail on bad input — it succeeds and builds a Photo whose size is None, or a string, or a dictionary. The type hint says int and nothing checks it, so the failure surfaces later, in arithmetic or formatting, with a message about the symptom rather than the cause. The second is longer and every one of its failures names the field, the expectation and the value.

That length is the honest argument for a validation library.

Doing it in three lines instead

python

from pydantic import BaseModel, ValidationError


class PhotoIn(BaseModel):
    name: str
    size: int
    city: str = ""


def load_photo(raw: str) -> PhotoIn:
    return PhotoIn.model_validate_json(raw)

text

ValidationError: 1 validation error for PhotoIn
size
  Input should be a valid integer [type=int_type, input_value=None]

Every check from the previous version, from a declaration rather than from twenty lines — and a better error, because it names the field and the received value. The annotations are the same type hints from earlier in this course, now enforced at runtime because this is a boundary and there is no checker.

The dividing line from the dataclasses lesson holds: dataclasses for data you created, validation models for data that arrived.

jsonschema is the alternative when the schema itself must be shared with other languages or published as a contract.

Nested, missing and optional

Real payloads are nested, and reaching through them is where None errors come from:

python

city = data["photographer"]["address"]["city"]

Any of three levels may be absent, and the traceback names only the last. Where a library is not warranted, be explicit:

python

photographer = data.get("photographer") or {}
address = photographer.get("address") or {}
city = address.get("city")

The or {} rather than a default argument matters: .get(k, {}) returns None if the key exists with a null value, which is exactly the case that breaks. or {} handles both missing and null.

Distinguish the three states that people collapse into one:

text

key absent          the provider did not send it
key present, null   the provider says there is no value
key present, ""     the provider says the value is empty

Treating all three as "no city" is often right and should be a decision you made, not one .get() made for you.

Other formats, briefly

CSV looks simple and is not. Quoted fields containing commas and newlines are why split(",") is wrong:

python

import csv

with open(path, encoding="utf-8", newline="") as file:
    for row in csv.DictReader(file):
        size = int(row["size"])      # everything arrives as text

Every CSV value is a string, including numbers. Converting is your job, and the conversion is where the validation goes.

YAML is common for configuration and has one rule:

python

import yaml
config = yaml.safe_load(text)        # safe_load, never load

Plain load can construct arbitrary Python objects, which makes loading an untrusted YAML file a code-execution vulnerability — the same hazard as pickle from the standard-library lesson.

TOML reads configuration and is in the standard library:

python

import tomllib
with open("pyproject.toml", "rb") as file:      # binary mode
    config = tomllib.load(file)

A cheat sheet to keep

text

THE FOUR FUNCTIONS
  json.loads(text)   json.dumps(obj)        strings
  json.load(file)    json.dump(obj, file)   files

  dumps(obj, indent=2, ensure_ascii=False, sort_keys=True)

WHAT SURVIVES A ROUND TRIP
  dict list str int float bool None      yes
  tuple  -> becomes a list, permanently
  dict keys -> become strings, always
  datetime, Decimal -> TypeError; convert yourself
    .isoformat() / fromisoformat()
    str(price) / Decimal(text)      never a JSON number for money

VALIDATION
  parse, do not cast
  check at the boundary, once, where the data arrives
  a type hint is NOT a runtime check

  by hand: isinstance checks, KeyError -> your own error
  or a library:
    class PhotoIn(BaseModel): name: str; size: int
    PhotoIn.model_validate_json(raw)

  created by you -> dataclass
  arrived from outside -> validate

NESTED
  data.get(k) or {}          not .get(k, {}) - null is not missing
  absent / null / empty are three different states

OTHER FORMATS
  csv.DictReader, newline=""     every value is a string
  yaml.safe_load                 never yaml.load
  tomllib.load(file)             open in "rb"

SAFETY
  cap the size before parsing; set a timeout
  never parse CSV with split(",")

Where to go next

Data crossing into your program is now checked where it arrives rather than trusted until it breaks something. The habit worth keeping is the boundary itself: one place where untrusted shapes become types the rest of your code can rely on, so nothing downstream needs defensive checks.

Next is Dates, Times, and Time Zones, which is the field you just serialised as a string and the one most likely to be subtly, silently wrong. It is a bigger subject than it looks, and the reason is that "now" is not a single thing.

Before you move on, take a JSON file your code reads and feed it something malformed — a missing field, a string where a number belongs, a null. Watch where the error surfaces. In most code it is nowhere near the parse, and that distance is what this lesson is for.