Iterators and generators

Python’s iteration is built on a protocol — the __iter__ and __next__ methods — that admits any class to participate in the language’s standard iteration constructs (for, in, comprehensions, iter(), next()). Generators — functions containing yield — are the conventional Python way to write iterators; they admit lazy sequence production with stateful, function-shaped logic. Generator expressions admit one-line generators that look like comprehensions. The combination — protocol + generators + expressions — is the foundation of much of Python’s data-processing style and is used pervasively in idiomatic code.

This page covers the iterator protocol, generator functions, generator expressions, the itertools module, async iteration, and the conventional patterns. The choice between generators and other iteration mechanisms is in Loops.

The iterator protocol

A Python iterator is an object with two methods:

class CounterIter:
    def __init__(self, limit: int):
        self.n = 0
        self.limit = limit

    def __iter__(self):
        return self           # iterators return themselves

    def __next__(self):
        if self.n >= self.limit:
            raise StopIteration
        self.n += 1
        return self.n

it = CounterIter(5)
for n in it:
    print(n)
# 1, 2, 3, 4, 5

The protocol:

• __iter__() — returns an iterator (often self).
• __next__() — returns the next value, or raises StopIteration.

The for loop, iter(), and next() builtins use this protocol:

it = iter([1, 2, 3])
next(it)        # 1
next(it)        # 2
next(it)        # 3
next(it)        # StopIteration

# or with a default:
next(it, None)  # None instead of raising

Iterables vs iterators

Python distinguishes:

• Iterable — an object with __iter__; can be iterated multiple times. Examples: list, tuple, dict, set, range, str.
• Iterator — an object with __iter__ and __next__; the state of an in-progress iteration. Can be iterated once; after exhaustion, stays exhausted.

xs = [1, 2, 3]
# xs is iterable; it is not itself an iterator

it1 = iter(xs)            # produces a fresh iterator
it2 = iter(xs)            # another fresh iterator

next(it1)                 # 1
next(it1)                 # 2
next(it2)                 # 1 (independent of it1)

# After exhaustion, an iterator stays exhausted:
list(it1)                  # remaining: [3]
list(it1)                  # []; exhausted

The conventional uses:

• for x in iterable: calls iter() on the iterable each time.
• next(iterator) advances the iterator.
• A user-defined iterable typically returns a new iterator from __iter__; an iterator returns itself.

Generator functions

A function containing yield is a generator function; calling it produces a generator (an iterator):

def counter(limit: int):
    n = 0
    while n < limit:
        n += 1
        yield n

c = counter(5)
for n in c:
    print(n)
# 1, 2, 3, 4, 5

The semantics:

• Calling counter(5) does not run the body; it returns a generator.
• The body runs each time next() is called on the generator.
• Each yield expr returns expr and pauses the function; the next next() resumes after the yield.
• When the function returns (or falls off the end), StopIteration is raised.

The generator captures the function’s local state (variables, the program counter); the function’s body is effectively transformed into a state machine. The mechanism admits writing iterators with the conventional sequential-code style.

Lazy evaluation

Generators are lazy — values are produced on demand:

def fibonacci():
    a, b = 0, 1
    while True:
        yield a
        a, b = b, a + b

# An infinite generator; consumers take only what they need:
for n in fibonacci():
    if n > 1000:
        break
    print(n)

The infinite loop is fine because no value is computed until the consumer requests it. The pattern admits generators for arbitrarily-long sequences without exhausting memory.

yield returning the sent value

yield is also an expression. The send() method on a generator passes a value into the function:

def echoer():
    while True:
        received = yield
        print(f"got: {received}")

e = echoer()
next(e)              # advance to the first yield
e.send("hello")      # prints "got: hello"
e.send("world")      # prints "got: world"

The send() mechanism admits two-way communication. It is rare in routine code; the principal use is in coroutine-style architectures (often replaced by async/await in modern Python).

yield from

The yield from admits delegating iteration to another iterable:

def flatten(items):
    for item in items:
        if isinstance(item, list):
            yield from flatten(item)
        else:
            yield item

list(flatten([1, [2, [3, [4]], 5], 6]))
# [1, 2, 3, 4, 5, 6]

The yield from sub is roughly equivalent to for x in sub: yield x but also forwards send(), throw(), and return values. The mechanism is the conventional Python form for composing generators.

Generator expressions

A generator expression is a one-line generator with comprehension-like syntax:

gen = (x ** 2 for x in range(10))      # generator, not list
list(gen)                                 # [0, 1, 4, ..., 81]

The form: (expr for var in iterable [if cond]). The result is a generator that produces values lazily; the entire collection is not materialised.

When passed as a single argument to a function, the parentheses may be omitted:

sum(x ** 2 for x in range(1000))         # the parens are the function call's
max(x for x in items if x > 0)
",".join(str(x) for x in items)

The pattern is the conventional Python form for “produce a sequence and feed it to a consumer” without intermediate materialisation.

For materialised collections, the alternatives:

Form	Result
(x for x in xs)	generator (lazy)
[x for x in xs]	list
{x for x in xs}	set
{k: v for k, v in items}	dict

itertools

The itertools module provides a substantial collection of iterator primitives:

import itertools

# Infinite iterators:
itertools.count(start=0, step=1)          # 0, 1, 2, ...
itertools.cycle([1, 2, 3])                 # 1, 2, 3, 1, 2, 3, ...
itertools.repeat(42)                        # 42, 42, 42, ...
itertools.repeat(42, times=5)               # five 42s

# Combinatoric:
itertools.product([1, 2], [3, 4])           # (1,3), (1,4), (2,3), (2,4)
itertools.permutations([1, 2, 3])           # all permutations
itertools.combinations([1, 2, 3], 2)        # (1,2), (1,3), (2,3)
itertools.combinations_with_replacement([1, 2, 3], 2)  # (1,1), (1,2), ...

# Sequencing:
itertools.chain([1, 2], [3, 4])            # 1, 2, 3, 4
itertools.chain.from_iterable([[1, 2], [3, 4]])  # 1, 2, 3, 4
itertools.zip_longest([1, 2, 3], [4, 5], fillvalue=0)
                                            # (1,4), (2,5), (3,0)
itertools.tee(iterable, n)                 # n independent iterators

# Filtering and dropping:
itertools.takewhile(lambda x: x < 5, [1, 2, 3, 6, 7])  # 1, 2, 3
itertools.dropwhile(lambda x: x < 5, [1, 2, 3, 6, 7])  # 6, 7
itertools.filterfalse(predicate, iterable)              # opposite of filter
itertools.compress(data, selectors)                      # data where selectors true

# Slicing:
itertools.islice(iterable, stop)
itertools.islice(iterable, start, stop, step)

# Grouping:
itertools.groupby(sorted_iterable, key=key_func)

# Accumulating:
itertools.accumulate([1, 2, 3, 4])         # 1, 3, 6, 10 (running sum)
itertools.accumulate([1, 2, 3, 4], operator.mul)  # 1, 2, 6, 24

The conventional uses:

• chain for concatenating sequences without materialising.
• takewhile/dropwhile for prefix/suffix discrimination.
• groupby for grouping consecutive equal elements.
• product and combinations for combinatoric search.
• accumulate for running totals and similar.

The module is one of the standard library’s most useful; nearly every non-trivial iteration-heavy program uses some of it.

Async iteration

Python admits async iteration through the __aiter__ and __anext__ methods:

class AsyncCounter:
    def __init__(self, limit: int):
        self.n = 0
        self.limit = limit

    def __aiter__(self):
        return self

    async def __anext__(self):
        if self.n >= self.limit:
            raise StopAsyncIteration
        await asyncio.sleep(0.1)
        self.n += 1
        return self.n

async def main():
    async for n in AsyncCounter(5):
        print(n)

The async for consumes async iterators. The conventional uses are:

• Streaming I/O (network responses, paginated APIs).
• Async producer-consumer queues.
• Polling for events.

Async generator functions:

async def fetch_pages(url):
    while url:
        response = await http_client.get(url)
        yield response.body
        url = response.next_page_url

async for body in fetch_pages("https://api.example.com/items?page=1"):
    process(body)

The yield inside an async def produces an async generator — the conventional Python form for asynchronous streams.

The full treatment of async is in Async and concurrency.

Common patterns

Read-loop with sentinel

def read_lines(file):
    while True:
        line = file.readline()
        if not line:
            break
        yield line

# Or, more concisely:
def read_lines(file):
    yield from iter(file.readline, "")

The two-argument iter(callable, sentinel) admits “call until the sentinel is returned”.

Pairwise iteration

import itertools

def pairwise(iterable):
    """[a, b, c] -> [(a, b), (b, c)]"""
    a, b = itertools.tee(iterable)
    next(b, None)
    return zip(a, b)

list(pairwise([1, 2, 3, 4]))    # [(1, 2), (2, 3), (3, 4)]

In Python 3.10+, itertools.pairwise is built in.

Chunking

def chunked(iterable, size):
    iterator = iter(iterable)
    while True:
        chunk = list(itertools.islice(iterator, size))
        if not chunk:
            break
        yield chunk

list(chunked([1, 2, 3, 4, 5, 6, 7], 3))
# [[1, 2, 3], [4, 5, 6], [7]]

In Python 3.12+, itertools.batched is built in.

Generator pipeline

def numbers():
    for i in range(10):
        yield i

def squared(items):
    for x in items:
        yield x ** 2

def positive(items):
    for x in items:
        if x > 0:
            yield x

# Compose:
result = list(positive(squared(numbers())))

Each function is a generator; the pipeline is lazy until materialised. The pattern admits substantial composability without intermediate allocations.

Generator with state

def with_index(iterable, start=0):
    i = start
    for item in iterable:
        yield i, item
        i += 1

# Equivalent to enumerate:
for i, x in with_index(items):
    print(f"{i}: {x}")

Generator-based file reader

def parse_csv(filename):
    with open(filename) as f:
        for line in f:
            fields = line.strip().split(",")
            yield fields

for row in parse_csv("data.csv"):
    process(row)

The pattern admits processing arbitrarily-large files without loading them entirely. The with statement keeps the file open for the duration of the generator’s lifetime.

Custom iterator class

class Tree:
    def __init__(self, value, children=None):
        self.value = value
        self.children = children or []

    def __iter__(self):
        yield self.value
        for child in self.children:
            yield from child

The class is iterable; for v in tree yields the values in pre-order traversal. The yield from recursively delegates.

Generators vs lists

The conventional choice between a generator and a list:

Use a generator when	Use a list when
The sequence is lazy or infinite	The sequence is finite and small
Memory matters	Random access matters
The consumer takes a fraction	Multiple traversals are needed
The sequence is consumed once	The sequence is consumed multiple times

# Generator — lazy, single-use:
def squares(n):
    return (x ** 2 for x in range(n))

# List — eager, multi-use:
def squares(n):
    return [x ** 2 for x in range(n)]

For substantial sequences, the generator form is preferable; for small sequences, the difference is negligible.

A note on the for-loop boundary

Generators interact with for loops in a subtle way:

def evens():
    yield 2
    yield 4
    yield 6

# A fresh generator each call:
for n in evens():
    print(n)

# Same generator, exhausted after the first loop:
g = evens()
for n in g:
    print(n)             # 2, 4, 6
for n in g:
    print(n)             # nothing; exhausted

The conventional discipline is to call the generator function fresh for each iteration; reuse of a single generator is rare.

For multiple traversals, materialise into a list:

xs = list(evens())
for n in xs: ...
for n in xs: ...

A note on the conventional discipline

The contemporary Python iterator advice:

• Use generator expressions for one-line transformations of an iterable.
• Use generator functions for substantial logic with sequential state.
• Use itertools for the conventional combinator patterns.
• Use yield from for generator composition.
• Use list comprehensions when materialisation is needed and the input is small.
• Use the iterator protocol (__iter__, __next__) for custom iterators that can’t be expressed as generator functions.

The combination admits a substantial fraction of Python’s expressive power; nearly every non-trivial Python codebase uses generators heavily.