How Python Generators Work: Pause, Yield, Resume

by Souy Soeng /November 08, 2026

Diagram of a Python generator pausing at yield and resuming with next(), handing out values one at a time

Python generators are functions that hand out values one at a time, pausing after each one and resuming exactly where they left off when you ask for the next. Instead of building a full list in memory, a generator produces each value only at the moment you need it.

This article follows one small generator from the moment you call it to the moment it finishes. Along the way you'll see what yield actually does, how next() brings a paused function back to life, why StopIteration rarely shows up in your code, and when a generator is the right tool. It also covers the two mistakes that catch almost everyone.

In short

Any function that contains yield becomes a generator function.
Calling it runs no code; it only returns a generator object.
Each next() runs the body to the next yield, then pauses with all local variables intact.
for loops call next() for you and stop cleanly at StopIteration.
Generators run once and don't support len() or indexing.

Why do whole lists eat so much memory?

Imagine a script that reads a huge file and builds a list of every line before doing anything with them. On small test files it works fine. On real data it slows to a crawl or runs out of memory. The loop itself isn't doing anything expensive, so the slowdown can feel like a mystery.

The real cost is holding every result at the same time. A list stores all of its items before you touch the first one, and that up-front cost grows with the input. You pay it even if you stop after a few lines. Yet in most loops, each line is only ever used once, one at a time.

Generators take the opposite approach. They are lazy: each item is computed only when someone asks for it. That single idea, producing values on demand instead of all at once, is the reason generators exist, and it explains everything else about how they behave.

Whole list built: every item sits in memory at once.
Lazy instead: each item is computed only when requested.

What makes a function a generator?

A generator is a function that uses the yield keyword. You don't import anything or inherit from a special class. If Python finds yield anywhere in a function's body, the whole function becomes a generator function, even though it looks almost exactly like a normal one.

The example used throughout this article is count_up. It takes n and starts a counter i at one. While i is at most n, it yields i, handing that value out. Then it adds one and loops round for the next value. There is no list anywhere in the function.

The key difference is yield versus return. return ends the function and throws its state away. yield hands a value back to the caller but leaves the function alive, ready to continue from the same spot. Because count_up produces numbers on request, counting to a huge n uses no more memory than counting to three.

def count_up(n):
    i = 1
    while i <= n:
        yield i
        i += 1

What happens when you call a generator function?

This is the first surprise. Calling count_up(3) returns immediately, and not a single line of the body has run. That's the opposite of a normal function, which executes its body as soon as you call it. A helpful picture is being handed a paused video: everything is loaded, but nothing has played.

In the interpreter session, printing gen doesn't show a number. It shows a generator object, and type(gen) confirms its class is generator. Even the first line, i = 1, hasn't executed yet. If you put a print statement at the very top of the function, nothing would appear at this point.

The generator object wraps the function together with its future state, waiting for you to ask for a value. That's why creating one is cheap. The real work only begins when something calls next() on it.

python3 -i counter.py

How do next() and yield pass control back and forth?

Each call to next(gen) runs the function forward until it reaches a yield. The function hands that value back and stops right there. The following call picks up on the line just after that yield. Think of a bookmark: you read to the next chapter break, close the book, and later reopen it on the same page.

With count_up(3), the first next(gen) runs until yield 1. The second resumes, adds one, loops, and sends back 2. The third returns 3. Between those calls the generator isn't running at all and uses no CPU; it simply sits on the yield line until asked again.

Just as important, control returns to your code after every yield. You can process the value, decide whether you want another one, or stop early. The generator never computes the next value until you ask, so stopping early means the remaining work is never done.

Paused, not finished: between calls, the function waits.
Control goes back: your code runs between each value.

How does a generator remember where it stopped?

When a generator pauses, Python keeps its whole frame: the local variables and the exact line it stopped on. Nothing is recomputed and nothing is lost. This is what separates a generator from calling an ordinary function over and over, since a normal function starts fresh every time.

Stepping through count_up(3) makes this visible. After creation, no code has run. The first next(gen) gives 1 with i at 1, the second gives 2 with i at 2, and the third gives 3 with i at 3. On the fourth call the while loop ends, the body finishes, and Python raises StopIteration.

That ending is permanent. Once the body has finished, every later next() call raises StopIteration again. The saved state is gone and the generator cannot restart. This detail matters later, because it's the source of one of the most common generator bugs.

Why don't for loops raise StopIteration?

In everyday code you almost never call next() by hand or catch StopIteration yourself. A for loop does both. Under the hood it runs a small cycle: call next(), let the generator run to its yield, run the loop body with that value, and repeat. When StopIteration arrives, the loop stops cleanly.

The loop treats StopIteration as the normal end signal, not as an error. You never see it, and the code after the loop simply carries on. That's why generators feel just like lists when you iterate over them.

The same protocol is used well beyond for loops. sum(), list(), sorted() and str.join() all consume iterables this way, so any generator you write plugs straight into tools you already use.

No try/except needed.
Works with sum(), list(), sorted() and join().

Example: finding errors in a huge log file

Suppose you need every error line from a log file far too big to load at once. Generators let you build a pipeline that holds only one line in memory at a time. Each stage pulls a line from the stage before it, does its job, and passes the line on.

The pipeline starts with app.log on disk. The file object returned by open() is already lazy: looping over it yields one line at a time. Next, a generator expression, written in round brackets, keeps only the lines that contain ERROR. Finally, a for loop pulls the matches one by one and prints them.

Whether the file is small or enormous, memory use stays flat, because only the current line is ever held. Nothing is collected into a list along the way. The whole pattern fits in four lines of code, and it shows the core benefit of generators in a practical setting.

with open('app.log') as f:
    errors = (l for l in f if 'ERROR' in l)
    for line in errors:
        print(line, end='')

Where are Python generators used?

The log pipeline is one pattern, but the same shape appears anywhere values arrive over time or there are too many to hold at once. A useful test: if you need values one by one rather than all together, a generator is probably the natural way to write the code.

Common cases include reading large files line by line, handling streaming data such as network responses that arrive in pieces, producing endless sequences like a counter that never stops, and building pipelines out of generator expressions. A list could never hold an endless sequence, but a generator can describe one easily.

An infinite generator, written with while True and a yield inside, is perfectly safe because it only runs when asked. Pair it with itertools.islice or a break statement so you take only the values you actually need.

Large files
Streaming data
Endless values
Pipelines

Common mistakes: reusing generators and calling len()

A generator looks like a list inside a for loop, but it isn't one. Both common mistakes come from that same root: a generator only knows its next value, nothing before it and nothing after it. In the session shown, gen doubles each number in range(3). The first list(gen) gives 0, 2, 4. The second gives an empty list.

That second result is the dangerous one. A spent generator doesn't raise an error; it silently produces nothing, which makes the bug hard to spot. For a second pass, call the generator function again to get a fresh generator, or store the results in a list the first time.

The other mistake is treating a generator as a sequence. len(gen) raises a TypeError, and gen0 fails because generators aren't subscriptable. If you genuinely need the length or specific positions, convert to a list and accept the memory cost that comes with it.

python3

Key takeaways

yield turns a function into a generator.
Calling a generator function runs no code yet; you just get an object.
next() runs to the next yield, then pauses.
Local variables survive every pause.
for loops handle StopIteration for you.
Generators run once, with no len() or indexing.

Frequently asked questions

What is the difference between yield and return in Python?

return ends the function and discards its local state. yield hands a value back to the caller but keeps the function paused, so it can resume from the same line on the next request.

Why does my generator return nothing the second time I loop over it?

The generator is exhausted. Once its body finishes, it stays finished and silently yields nothing. Call the generator function again for a fresh one, or save the values in a list during the first pass.

Can I use len() or indexing on a generator?

No. len() raises a TypeError and indexing fails because generators aren't subscriptable. Convert the generator to a list if you need those features, knowing that this stores every value in memory.

Do I need to catch StopIteration when using a generator?

Usually not. for loops, along with functions like sum(), list(), sorted() and join(), treat StopIteration as the normal end signal. You only see it if you call next() manually after the generator is finished.

Is an infinite generator safe to write?

Yes. A generator with while True and a yield only runs when asked for a value, so it never loops on its own. Use itertools.islice or a break to take only as many values as you need.

When should I use a generator instead of a list?

Use a generator when you need values one by one rather than all at once, such as with large files, streaming data, endless sequences or processing pipelines. Use a list when you need to reuse the data, know its length or access items by position.

Watch the full video on YouTube →

Tag

Ad