intermediate30 minby DevFox

Iterators, Generators, and Laziness

The iterator protocol, yield, and pipelines that process a file larger than memory. Where laziness wins, and the bugs that appear when a generator is consumed twice.

Python

The log file is nine gigabytes. Your function reads it into a list, filters it, transforms it, and returns the result. It works perfectly on the sample file and gets killed by the operating system on the real one, because building that list needed more memory than the machine has.

The fix is not a bigger machine. It is to stop building the list at all — to hand each line onward as it is read and never hold more than one at a time. By the end of this lesson you will write generators, build pipelines out of them, and recognise the one bug that laziness introduces, which catches people who otherwise understand it perfectly.

What a for loop actually does

Every for loop you have written rests on a protocol:

python

numbers = iter([1, 2, 3])     # get an iterator
print(next(numbers))          # 1
print(next(numbers))          # 2
print(next(numbers))          # 3
print(next(numbers))          # StopIteration

for x in thing calls iter(thing) to get an iterator, then calls next() on it repeatedly until StopIteration is raised, which ends the loop.

That is the whole mechanism, and the useful consequence is that anything implementing it can be looped over. The thing does not need to be a collection, or finite, or to exist yet.

A generator

Writing an iterator by hand means a class with __iter__ and __next__. A generator gets you the same thing from a function:

python

def read_captions(path):
    with open(path, encoding="utf-8") as file:
        for line in file:
            line = line.strip()
            if line:
                yield line

yield is what makes it a generator. Calling the function runs none of the body — it hands back a generator object. The body runs when something asks for a value, produces one at yield, and suspends there, keeping its local variables and its position, until the next value is asked for.

python

captions = read_captions("captions.txt")     # nothing has happened
print(next(captions))                        # now the file opens

for caption in read_captions("captions.txt"):
    print(caption)                           # one line in memory

That is the nine-gigabyte fix in six lines. Memory use is one line, whatever the file size, because there is never a list.

Pipelines

Generators compose, and that is where they stop being a memory trick and become a way of writing.

python

def read_lines(path):
    with open(path, encoding="utf-8") as file:
        for line in file:
            yield line.rstrip("\n")

def only_errors(lines):
    for line in lines:
        if " ERROR " in line:
            yield line

def parse(lines):
    for line in lines:
        timestamp, _, message = line.partition(" ERROR ")
        yield {"at": timestamp.strip(), "message": message.strip()}

python

errors = parse(only_errors(read_lines("app.log")))

for record in errors:
    store(record)

Each stage is a few lines, testable on its own, and does one thing. Reading that assembly line tells you what happens to the data without opening any of them.

And nothing is stored between stages. A line is read, tested, parsed and stored before the second line is read. The pipeline processes a file larger than memory with no stage aware of that fact.

Compare the eager version, which is the same logic and needs three full copies of a nine-gigabyte file in memory:

python

lines = read_all_lines(path)
errors = [line for line in lines if " ERROR " in line]
records = [parse_one(line) for line in errors]

The bug laziness introduces

Here is the one to know about.

Bad — using a generator twice.

python

def find_large(photos):
    return (p for p in photos if p.size > 1000)

large = find_large(all_photos)

print(f"Found {sum(1 for _ in large)} large photos")
for photo in large:                  # nothing. Silently.
    archive(photo)

Good — materialising it when you need it more than once.

python

large = list(find_large(all_photos))

print(f"Found {len(large)} large photos")
for photo in large:                  # works
    archive(photo)

A generator

Exhausted once consumed.

Counting it ran it to the end, so the loop that follows has nothing left.

It does not error. The count prints correctly and the archiving silently does not happen, which is the worst combination of symptoms available.

A list

Traversable as often as you like.

Call list() when you need to go over it twice, know how many there are, or index into it.

And accept that you are now holding it all in memory, which is the trade you were avoiding.

A generator is a stream, not a collection — and nothing in the language reminds you which you are holding

Generator expressions

The comprehension form, from the foundations course, is the same thing without a def:

python

sizes = (p.size for p in photos)          # a generator
total = sum(p.size for p in photos)       # brackets optional inside

Use the expression form for something short and immediate, and a def when the logic needs a name, a docstring or more than one line. A pipeline stage almost always deserves a name.

The laziness pays off with early exit:

python

first_large = next((p for p in photos if p.size > 1000), None)

That stops at the first match. The list comprehension version examines all four hundred thousand photos and then takes one. The second argument to next is the default when nothing matches, and omitting it means StopIteration instead — usually not what you want.

Tools that expect streams

itertools is built for this, and the functions worth knowing are the ones that would be awkward by hand:

python

from itertools import islice, chain, groupby, takewhile

islice(photos, 10)                 # first ten, lazily
islice(photos, 100, 200)           # a window, without a list
chain(folder_a, folder_b)          # several streams as one
takewhile(lambda p: p.size > 1000, photos)   # until the test fails

groupby groups consecutive items, which is a genuine trap:

python

for city, group in groupby(photos, key=lambda p: p.city):
    print(city, len(list(group)))

It only groups items that are adjacent, so the input must be sorted by the same key first. Unsorted input gives you the same city several times, quietly. It works that way precisely because it is lazy — grouping non-adjacent items would mean reading everything first.

Returning a value, and yield from

Two smaller features that come up.

A generator can return, which ends it and attaches a value to the StopIteration — most useful for reporting a total alongside the stream:

python

def read_valid(path):
    skipped = 0
    for line in read_lines(path):
        if not line.startswith("#"):
            yield line
        else:
            skipped += 1
    return skipped

And yield from delegates to another iterable, which flattens nested generators without a loop:

python

def all_photos(folders):
    for folder in folders:
        yield from read_photos(folder)

A cheat sheet to keep

text

THE PROTOCOL
  iter(x)  ->  an iterator
  next(it) ->  the next value, or raises StopIteration
  a for loop is exactly that, in a loop

GENERATORS
  def f():
      yield value            calling f() runs NOTHING
                             the body runs on demand and suspends
  (x for x in things)        the expression form

  no len(), no indexing, no second pass

PIPELINES
  parse(only_errors(read_lines(path)))
  each stage small, named, testable
  nothing stored between stages

THE BUG
  a generator is exhausted after ONE pass
  the second use does nothing, silently, with no error
  need it twice? list() it - and accept the memory

USEFUL SHAPES
  next((x for x in xs if test), None)     first match, stops early
  sum(p.size for p in photos)             no list built

ITERTOOLS
  islice(xs, 10)        first ten, lazily
  chain(a, b)           several streams as one
  takewhile(f, xs)      until the test fails
  groupby(xs, key=...)  ADJACENT only - sort by the key first

ALSO
  yield from other       delegate to another iterable
  return value           ends it; value rides on StopIteration

WHEN NOT TO
  small collections you reuse - a list is simpler
  anything needing len, indexing or two passes

Where to go next

You can now write functions that produce values on demand, chain them into pipelines that handle inputs larger than memory, and recognise the exhaustion bug that produces a silent no-op rather than an error. The idea to carry is that a generator is a stream: reusable exactly once, and cheap precisely because of that.

Next is Context Managers, which formalises the other half of what the pipeline examples relied on — with, and the guarantee that setup is always matched by cleanup. You have used it on files; that lesson shows how to write your own, and why the guarantee is worth more than the syntax.

Before you move on, take a function that builds and returns a list and convert it to a generator. Then deliberately consume it twice and watch the second pass do nothing. Causing that silently-empty result once, on purpose, is what makes you check for it later.