Mastering Python's itertools: Lazy, Memory-Efficient Iteration Done Right

If you have ever built up a giant list just to loop over it once, Python's itertools module can make your code shorter, faster, and dramatically more memory-friendly. itertools is a standard-library collection of building blocks for working with iterators — lazy sequences that produce values one at a time instead of all at once. Because nothing is materialized until you ask for it, you can express elegant data pipelines that behave the same whether they process ten items or ten million.

This guide walks through the most useful functions in the module, each with a runnable example you can paste straight into a REPL. By the end you will know how to generate, combine, filter, group, and chunk data without writing a single manual index or throwaway temporary list.

Why itertools? Laziness as a feature

An iterator computes values on demand and remembers where it left off. That single idea has two big consequences: you can work with sequences that are too large (or even infinite) to fit in memory, and you can chain operations together so data flows through the pipeline one element at a time. Compare an eager list with a lazy stream:

# Eager: materializes a million integers in memory first
squares = [n * n for n in range(1_000_000)]
total = sum(squares)

# Lazy: one value at a time, near-zero extra memory
total = sum(n * n for n in range(1_000_000))

Generator expressions already give you laziness. itertools goes further by providing ready-made, C-optimized iterators for the patterns you reach for again and again.

Infinite iterators: count, cycle, and repeat

Three functions produce streams that never end on their own — you decide when to stop. count is a numeric counter, cycle loops a sequence forever, and repeat yields the same value (optionally a fixed number of times).

from itertools import count, cycle, repeat

out = []
for i in count(10, 2):       # start at 10, step by 2
    if i > 18:
        break
    out.append(i)
print(out)                    # [10, 12, 14, 16, 18]

colors = cycle(["red", "green", "blue"])
print([next(colors) for _ in range(5)])  # ['red', 'green', 'blue', 'red', 'green']

print(list(repeat("x", 3)))   # ['x', 'x', 'x']

# repeat shines as a constant supply for map/zip
print(list(map(pow, range(5), repeat(2))))  # [0, 1, 4, 9, 16]

Because these never terminate by themselves, always pair them with something that limits output (a break, islice, or takewhile).

Combining and slicing: chain and islice

chain stitches multiple iterables into one continuous stream without building an intermediate list. chain.from_iterable does the same for an iterable of iterables — perfect for flattening.

from itertools import chain

a = [1, 2]
b = (3, 4)
c = range(5, 7)
print(list(chain(a, b, c)))               # [1, 2, 3, 4, 5, 6]

matrix = [[1, 2], [3, 4], [5, 6]]
print(list(chain.from_iterable(matrix)))  # [1, 2, 3, 4, 5, 6]

islice is "slicing for iterators." Unlike my_list[2:10:2], it works on any iterable, including infinite ones and file objects, without loading everything first.

from itertools import islice, count

print(list(islice(count(), 5)))         # [0, 1, 2, 3, 4]
print(list(islice(count(), 2, 10, 2)))  # [2, 4, 6, 8]

# Read just the first 10 lines of a huge file
# with open("big.log") as f:
#     for line in islice(f, 10):
#         print(line, end="")

Filtering and transforming streams

Several functions filter or reshape a stream. takewhile yields items until a predicate first fails; dropwhile skips the leading run that matches and then yields everything after; filterfalse is the inverse of the built-in filter; and compress selects items using a parallel mask.

from itertools import takewhile, dropwhile, filterfalse, compress

nums = [1, 3, 5, 8, 9, 2]
print(list(takewhile(lambda x: x % 2, nums)))  # [1, 3, 5]  stop at first even
print(list(dropwhile(lambda x: x % 2, nums)))  # [8, 9, 2]  drop leading odds

print(list(filterfalse(lambda x: x % 2, range(10))))  # [0, 2, 4, 6, 8]

data = ["a", "b", "c", "d"]
mask = [1, 0, 1, 0]
print(list(compress(data, mask)))  # ['a', 'c']

Two more transformers are worth knowing. starmap is like map but unpacks each tuple as arguments, and zip_longest zips uneven iterables, padding the short one instead of stopping early.

from itertools import starmap, zip_longest

points = [(1, 2), (3, 4), (5, 6)]
print(list(starmap(lambda x, y: x + y, points)))  # [3, 7, 11]

names = ["Ann", "Bob", "Cy"]
scores = [90, 85]
print(list(zip_longest(names, scores, fillvalue=0)))
# [('Ann', 90), ('Bob', 85), ('Cy', 0)]

Running totals with accumulate

accumulate produces a running reduction. By default it sums, but you can pass any two-argument function, and an initial value (added in Python 3.8).

from itertools import accumulate
import operator

print(list(accumulate([1, 2, 3, 4])))                 # [1, 3, 6, 10]   running sum
print(list(accumulate([1, 2, 3, 4], operator.mul)))   # [1, 2, 6, 24]   running product
print(list(accumulate([3, 1, 4, 1, 5], max)))         # [3, 3, 4, 4, 5]  running max
print(list(accumulate([1, 2, 3], initial=100)))       # [100, 101, 103, 106]

Sliding windows with pairwise

Added in Python 3.10, pairwise yields consecutive overlapping pairs — exactly what you need for diffs, deltas, and simple sliding-window logic.

from itertools import pairwise

prices = [10, 12, 9, 15]
print(list(pairwise(prices)))                  # [(10, 12), (12, 9), (9, 15)]
print([b - a for a, b in pairwise(prices)])    # [2, -3, 6]  day-over-day change

Combinatorics: product, permutations, and combinations

The combinatoric iterators replace deeply nested loops and hand-rolled recursion. product is the Cartesian product (a clean substitute for nested for loops), while permutations, combinations, and combinations_with_replacement cover ordered and unordered selections.

from itertools import product, permutations, combinations, combinations_with_replacement

print(list(product([0, 1], repeat=2)))
# [(0, 0), (0, 1), (1, 0), (1, 1)]

print(list(permutations("ABC", 2)))
# [('A','B'), ('A','C'), ('B','A'), ('B','C'), ('C','A'), ('C','B')]

print(list(combinations("ABC", 2)))
# [('A','B'), ('A','C'), ('B','C')]

print(list(combinations_with_replacement("ABC", 2)))
# [('A','A'), ('A','B'), ('A','C'), ('B','B'), ('B','C'), ('C','C')]

Using product instead of nesting flattens your indentation and makes intent obvious:

for x, y in product(range(2), range(2)):
    print(f"{x},{y}", end=" ")   # 0,0 0,1 1,0 1,1

Grouping consecutive items with groupby

groupby collapses consecutive elements that share a key into groups. The word "consecutive" is the single most important — and most commonly missed — detail: if your data is not already ordered by the key, you must sort it first, or you will get fragmented groups.

from itertools import groupby

records = [
    ("fruit", "apple"),
    ("fruit", "banana"),
    ("veg", "carrot"),
    ("veg", "pea"),
    ("fruit", "cherry"),
]

records.sort(key=lambda row: row[0])   # sort by the grouping key FIRST

grouped = {}
for key, group in groupby(records, key=lambda row: row[0]):
    grouped[key] = [item for _, item in group]

print(grouped)
# {'fruit': ['apple', 'banana', 'cherry'], 'veg': ['carrot', 'pea']}

Note also that each group is itself a lazy iterator that shares the underlying stream — consume it before advancing to the next group, or copy it into a list as shown above.

Chunking with batched (Python 3.12+)

Splitting a stream into fixed-size chunks used to require a helper. Since Python 3.12, itertools.batched does it directly — ideal for paginating database writes or batching API requests. The final batch may be shorter; in Python 3.13+ a strict=True argument raises if it is.

from itertools import batched   # Python 3.12+

ids = range(1, 10)
for batch in batched(ids, 3):
    print(batch)
# (1, 2, 3)
# (4, 5, 6)
# (7, 8, 9)

If you are on an older Python, the documented equivalent is a tiny recipe built from islice:

from itertools import islice

def batched(iterable, n):
    it = iter(iterable)
    while batch := tuple(islice(it, n)):
        yield batch

Composing a lazy pipeline

The real payoff comes from composition. Each function returns an iterator, so you can stack them and only the values you actually request get computed. Here is an infinite source filtered, transformed, and capped — all lazily:

from itertools import count, islice

evens = (n for n in count() if n % 2 == 0)   # infinite stream of even numbers
squares = map(lambda n: n * n, evens)         # square each, still lazy
print(list(islice(squares, 5)))               # [0, 4, 16, 36, 64]

Nothing past the fifth square is ever calculated. Swap islice(..., 5) for ..., 5000 and the memory footprint barely changes.

Common pitfalls

Iterators are single-use. Once exhausted, an iterator is empty — looping again yields nothing. If you need the data twice, store it in a list or use itertools.tee.

from itertools import islice, count

it = islice(count(), 3)
print(list(it))   # [0, 1, 2]
print(list(it))   # []  already exhausted!

Never call list() on an infinite iterator. list(count()) will hang and eventually exhaust memory. Always bound it with islice or takewhile.

groupby needs sorted input when you want global groups, as covered above. And tee can be memory-hungry: if the copied iterators advance at very different speeds, Python buffers everything in between, so it is not a free way to "rewind."

Wrap-up and next steps

The itertools module rewards a small upfront investment with code that is shorter, faster, and far easier to scale. Reach for chain and islice to combine and slice streams, takewhile/dropwhile/compress to filter them, accumulate and pairwise for running and windowed computations, the combinatoric functions to retire nested loops, and groupby and batched to organize data into groups and chunks.

From here, read the official itertools documentation, which includes a famous "recipes" section showing how to compose these primitives into higher-level tools like sliding_window, unique_everseen, and roundrobin. Many of those recipes are packaged, tested, and ready to install in the excellent third-party more-itertools library. Start by replacing one manual loop in your codebase with an itertools equivalent — you will quickly develop an eye for where they fit.