Iterators

Lua’s generic-for loop admits iterator-based iteration. An iterator is a function (or a triple of state and function) that produces values one at a time. The standard library provides pairs (all keys), ipairs (sequence), string.gmatch (pattern matches), io.lines (file lines). Custom iterators are conventional via closures (stateful) or via the generic-for protocol (stateless). Coroutines (treated separately) admit substantial generator-style iterators with substantial flexibility. The combination — generic-for protocol, the standard iterators (pairs/ipairs/gmatch/lines), custom-iterator patterns via closures and coroutines — is the substance of Lua’s iteration mechanism.

The generic-for protocol

The full form:

for var_1, ..., var_n in iter, state, ctrl do
    -- body
end

The semantics:

1. The expression iter, state, ctrl is evaluated once.
2. On each iteration, iter(state, ctrl) is called.
3. The first return value becomes the new ctrl; if ctrl is nil, the loop ends.
4. All return values are assigned to the variables.

Most iterators use only the function form (with closure-based state):

for v in custom_iter() do
    print(v)
end

Stateful (closure-based) iterators

The closure-based iterator stores state in upvalues:

local function range(start, stop, step)
    step = step or 1
    local current = start - step
    return function()
        current = current + step
        if (step > 0 and current <= stop) or (step < 0 and current >= stop) then
            return current
        end
    end
end

for i in range(1, 5) do
    print(i)                                       -- 1, 2, 3, 4, 5
end

for i in range(10, 1, -2) do
    print(i)                                       -- 10, 8, 6, 4, 2
end

The closure-based form admits substantial state encapsulation; the conventional pattern.

Stateless iterators

The stateless iterator passes state explicitly:

local function ipairs_stateless(t, i)
    i = i + 1
    if t[i] ~= nil then
        return i, t[i]
    end
end

local function my_ipairs(t)
    return ipairs_stateless, t, 0                  -- iter, state, ctrl
end

for i, v in my_ipairs({10, 20, 30}) do
    print(i, v)                                    -- 1 10, 2 20, 3 30
end

The stateless form admits substantial efficiency — no closure allocation per iteration.

pairs and ipairs

The standard library provides:

-- pairs: all keys (order undefined):
for k, v in pairs(t) do
    -- ...
end

-- ipairs: sequence (1, 2, ..., until nil):
for i, v in ipairs(t) do
    -- ...
end

The pairs and ipairs are stateless iterators internally (they admit substantial efficiency).

string.gmatch

Iterator over pattern matches:

for word in string.gmatch("the quick brown fox", "%w+") do
    print(word)                                    -- the, quick, brown, fox
end

for k, v in string.gmatch("a=1; b=2; c=3", "(%w+)=(%d+)") do
    print(k, v)                                    -- a 1, b 2, c 3
end

io.lines

Iterator over file lines:

for line in io.lines("file.txt") do
    print(line)
end

-- From a file handle:
local f = io.open("data.txt", "r")
for line in f:lines() do
    print(line)
end
f:close()

Custom iterators via coroutines

Coroutines admit substantial generator-style iterators:

local function fibonacci()
    return coroutine.wrap(function()
        local a, b = 0, 1
        while true do
            coroutine.yield(a)
            a, b = b, a + b
        end
    end)
end

for n in fibonacci() do
    if n > 100 then break end
    io.write(n, " ")
end
-- 0 1 1 2 3 5 8 13 21 34 55 89

The coroutine.wrap admits the coroutine being used as an iterator function. Treated in Coroutines.

Common patterns

Range

local function range(start, stop, step)
    step = step or 1
    local i = start - step
    return function()
        i = i + step
        if (step > 0 and i <= stop) or (step < 0 and i >= stop) then
            return i
        end
    end
end

for i in range(1, 10) do print(i) end

Reversed

local function reversed(t)
    local i = #t + 1
    return function()
        i = i - 1
        if i >= 1 then return i, t[i] end
    end
end

for i, v in reversed({10, 20, 30}) do
    print(i, v)                                    -- 3 30, 2 20, 1 10
end

Sorted pairs

local function sorted_pairs(t)
    local keys = {}
    for k in pairs(t) do keys[#keys + 1] = k end
    table.sort(keys)

    local i = 0
    return function()
        i = i + 1
        local k = keys[i]
        if k ~= nil then return k, t[k] end
    end
end

for k, v in sorted_pairs({c = 3, a = 1, b = 2}) do
    print(k, v)                                    -- a 1, b 2, c 3
end

The pattern admits deterministic iteration over a map.

Mapped iterator

local function imap(iter_fn, transform)
    return function()
        local v = iter_fn()
        if v ~= nil then return transform(v) end
    end
end

local doubled = imap(range(1, 5), function(x) return x * 2 end)
for v in doubled do print(v) end                   -- 2, 4, 6, 8, 10

Filtered iterator

local function ifilter(iter_fn, pred)
    return function()
        for v in iter_fn do
            if pred(v) then return v end
        end
    end
end

local evens = ifilter(range(1, 10), function(n) return n % 2 == 0 end)
for v in evens do print(v) end                     -- 2, 4, 6, 8, 10

Take N

local function take(iter_fn, n)
    local count = 0
    return function()
        if count >= n then return end
        count = count + 1
        return iter_fn()
    end
end

for v in take(naturals(), 5) do print(v) end

Skip N

local function skip(iter_fn, n)
    for _ = 1, n do iter_fn() end                  -- consume
    return iter_fn
end

for v in skip(range(1, 10), 5) do print(v) end     -- 6, 7, 8, 9, 10

Zipper (parallel iteration)

local function zip(iter_a, iter_b)
    return function()
        local a = iter_a()
        local b = iter_b()
        if a ~= nil and b ~= nil then
            return a, b
        end
    end
end

local names = {"Alice", "Bob", "Charlie"}
local ages = {30, 25, 35}

for n, a in zip(values(names), values(ages)) do
    print(n, a)
end

Where values is:

local function values(t)
    local i = 0
    return function()
        i = i + 1
        return t[i]
    end
end

Counter

local function counter(start)
    start = (start or 0) - 1
    return function()
        start = start + 1
        return start
    end
end

local c = counter()
print(c(), c(), c())                               -- 0, 1, 2

Lines from a string

local function lines(s)
    return s:gmatch("[^\n]+")
end

for line in lines("first\nsecond\nthird") do
    print(line)                                    -- first, second, third
end

Words from a string

local function words(s)
    return s:gmatch("%S+")
end

for w in words("hello world foo") do
    print(w)
end

Pairs from a table

local function table_pairs(t)
    return coroutine.wrap(function()
        for k, v in pairs(t) do
            coroutine.yield(k, v)
        end
    end)
end

Cycling iterator

local function cycle(t)
    return coroutine.wrap(function()
        while true do
            for _, v in ipairs(t) do
                coroutine.yield(v)
            end
        end
    end)
end

local c = cycle({"red", "green", "blue"})
for _ = 1, 7 do io.write(c(), " ") end             -- "red green blue red green blue red"

Concatenated iterators

local function chain(...)
    local iters = {...}
    return coroutine.wrap(function()
        for _, iter_fn in ipairs(iters) do
            for v in iter_fn do
                coroutine.yield(v)
            end
        end
    end)
end

for v in chain(range(1, 3), range(10, 12)) do
    print(v)                                       -- 1, 2, 3, 10, 11, 12
end

Index-and-value from sequence

local function enumerate(iter_fn)
    local i = 0
    return function()
        local v = iter_fn()
        if v ~= nil then
            i = i + 1
            return i, v
        end
    end
end

for i, v in enumerate(values({"a", "b", "c"})) do
    print(i, v)                                    -- 1 a, 2 b, 3 c
end

Iterator to table

local function collect(iter_fn)
    local result = {}
    for v in iter_fn do
        result[#result + 1] = v
    end
    return result
end

local arr = collect(range(1, 5))                   -- {1, 2, 3, 4, 5}

Iterator with multiple returns

local function with_index(iter_fn)
    local i = 0
    return function()
        local v = iter_fn()
        if v ~= nil then
            i = i + 1
            return i, v
        end
    end
end

for i, v in with_index(values({"a", "b", "c"})) do
    print(i, v)
end

File-based iterator

local function file_words(filename)
    local f = io.open(filename)
    if not f then return function() return nil end end

    local line_iter = f:lines()
    local current_words = function() return nil end

    return function()
        while true do
            local w = current_words()
            if w then return w end

            local line = line_iter()
            if line == nil then
                f:close()
                return nil
            end
            current_words = string.gmatch(line, "%S+")
        end
    end
end

A note on the iterator protocol details

The full generic-for unfolds to:

for var1, ..., varn in explist do
    body
end

-- Equivalent to:
do
    local iter, state, ctrl = explist
    while true do
        local var1, ..., varn = iter(state, ctrl)
        if var1 == nil then break end
        ctrl = var1
        body
    end
end

The stateless form passes state and ctrl to the iterator on each call; the stateful form has the iterator function close over its state.

A note on the conventional discipline

The contemporary Lua iterators advice:

• Use pairs for arbitrary key iteration.
• Use ipairs for sequence iteration.
• Use string.gmatch for pattern-based iteration.
• Use io.lines for file iteration.
• Use closures for stateful custom iterators.
• Use coroutines (coroutine.wrap) for substantial generator-style iterators.
• Use stateless iterators (returning iter, state, ctrl) for substantial efficiency.
• Compose iterators — take, filter, map, chain.
• Sort keys for deterministic map iteration.

The combination — the generic-for protocol with stateful and stateless variants, the standard iterators (pairs/ipairs/gmatch/lines), the closure-based and coroutine-based custom iterators, the composition patterns — is the substance of Lua’s iteration surface. The discipline admits substantial flexibility through the small protocol; substantial generator-style code is admitted via coroutines.