How-to › Move data in and out

How to paginate an API with Python generators#

Turn a cursor-paged endpoint into a sync and an async Python generator, take the first N items without fetching every page, and raise an error naming the page.

Audience
API consumer
Level
intermediate
Topic
Paginate collections
Languages
Python
Verified

Read first: Loop over every page of a REST collection

Your list call returns three books and a next cursor, and every caller writes its own loop to follow it. One loop stops after the first page, another collects all forty thousand rows into a list before showing the first, and a third swallows the 503 on page twelve. Each bug lives at a different call site.

What you get

You will end up with a sync generator over requests and an async generator over httpx, which callers read with a plain for or async for, fetching pages only as they are read. This is for you if you call a cursor-paged API from Python and want one loop instead of one per caller.

Short answer

Write a Python generator that requests one page, yields its items, and follows the next cursor until it is null, so callers iterate with a plain for loop. Write the same walk as an async generator over httpx.AsyncClient. Take the first N with itertools.islice, which fetches only the pages it needs, and raise an exception that carries the page number and cursor. Give each consumer its own generator, because one is used up after a single pass.

You will need

Python 3.11 or later, requests, httpx, and an API whose list endpoint returns a cursor. Verified 2026-11-21 against Python 3.13.18, requests 2.34.2, httpx 0.28.1, @voxgig/sdkgen 4.44.3 and openapi-python-client 0.30.0. The document, api.json, has one operation, GET /books, taking cursor and limit and answering { "items": [...], "next": "c3" }, with next null on the last page. server.py stands in for it on 127.0.0.1.

Approaches compared

ApproachWhen it fitsWhat it costs youWhen to pick something else
boto3 get_paginatorYou call AWS, which ships a paginator per operationA Paginator and PageIterator shape that covers AWS operations and nothing elseThe API is not an AWS service
Hand-written generator with requests or httpxAny cursor API, any client, and callers who should see one for loopYou own the loop guard, the error type and the async twinA generated SDK already pages correctly for your API
openapi-python-clientYou want typed models and will write the loop around themNo paging code at all, so the generator on this page is still yoursYou want paging generated with the client
sdkgen paging featureYou generate several languages and want one paging signal in eachAn opt-in feature whose control object you must pass and patch by handYour body names its cursor next, unless you add the wrapper here
Speakeasy x-speakeasy-paginationYou generate with Speakeasy and can add an extension to the documentAn account to generate, and a while loop over next() instead of a forYou cannot add vendor extensions to the published document

Voxgig maintains sdkgen. It is one of five options here, and the one with a gap this page works around.

A hand-written generator costs the most code and works against any client. boto3’s paginator is reusable where a generator is not, but it covers AWS alone. Stainless is not a row: on May 18, 2026 it announced it is winding down all hosted Stainless products, its SDK generator included, with no new signups.

Yield books from each page

Two generators split the job. pages() owns the HTTP and the cursor; books() flattens pages into items, so a caller never sees a page boundary.1

def pages(session, base, limit=3):
    """Yield one page of books at a time, with requests. Nothing is fetched until the caller asks."""
    cursor, page, used = None, 1, set()
    while True:
        params = {"limit": limit} if cursor is None else {"limit": limit, "cursor": cursor}
        r = session.get(f"{base}/books", params=params, timeout=10)
        if r.status_code != 200:
            raise PageError(page, cursor, r.status_code, r.text)
        body = r.json()
        yield body["items"]
        cursor = _next_cursor(page, cursor, body, used)
        if cursor is None:
            return
        used.add(cursor)
        page += 1


def books(session, base, limit=3):
    """Yield one book at a time, across every page."""
    for items in pages(session, base, limit):
        yield from items

The raise sits inside the generator, so it surfaces at the caller’s for line, on the item after the last good page. PageError carries the page number and the cursor that was sent for it. _next_cursor raises too when the server repeats a cursor, which would otherwise loop forever.

apages() is the same body with async def and await client.get(...) over an httpx.AsyncClient, and abooks() flattens it with async for.

Stop early and see which page failed

demo.py runs four walks against the stand-in and prints every query it received.

python3 demo.py
for loop over books()
  books: b1 b2 b3 b4 b5 b6 b7
  requests: 3, limit=3 | limit=3&cursor=c3 | limit=3&cursor=c6
islice(books(), 4)
  books: b1 b2 b3 b4
  requests: 2, limit=3 | limit=3&cursor=c3
async for over abooks(), break after 5
  books: b1 b2 b3 b4 b5
  requests: 2, limit=3 | limit=3&cursor=c3
page 2 answers 503
  books before the error: b1 b2 b3
  raised: page 2 (cursor 'c3') failed: 503 {"code": "unavailable", "message": "try again"}
  err.page=2 err.cursor='c3' err.status=503

islice stopped after four books, so the third page was never requested. The async walk has no islice, so it breaks out of the loop inside contextlib.aclosing, which closes the async generator at once instead of leaving it to the garbage collector. The failed walk delivered page 1 before raising, and the exception names page 2 and c3, which is the request to retry.

Follow the cursor with sdkgen’s paging feature

The sdkgen Python target scaffolded with -f paging reads the page signal and records it on the control dict, the second argument of list(). The feature sends a recorded cursor back as the cursor parameter, and records a body field named next as next, which it does not send. Its options cursorParam, pageParam and limitParam name query parameters only, so no setting points the feature at a different response field. sdks.py shows both, then the wrapper that copies one into the other.

python3 sdks.py
openapi-python-client list_books.sync()
  returns BookPage: 3 items, next_='c3'
  requests: limit=3
openapi-python-client, wrapped in opc_books()
  books: b1 b2 b3 b4 b5 b6 b7
  requests: limit=3 | cursor=c3&limit=3 | cursor=c6&limit=3
sdkgen paging feature, passing the same ctrl three times
  call 1: b1 b2 b3
  call 2: b1 b2 b3
  call 3: b1 b2 b3
  ctrl paging after: next='c3' cursor=None hasMore=True
  requests: limit=3&page=1 | limit=3&page=1 | limit=3&page=1
sdkgen, wrapped in sdkgen_books()
  books: b1 b2 b3 b4 b5 b6 b7
  requests: limit=3&page=1 | cursor=c3&limit=3 | cursor=c6&limit=3

With this document, the loop the features reference shows, while ctrl.paging.hasMore, never ends: hasMore stays true and each call asks for page=1. @voxgig/sdkgen 4.45.0, the newest release, generated the same Python tree and repeated the same page. A body that names its cursor nextCursor, cursor or next_cursor fills cursor instead, and check.py runs the documented loop to the last page on each.

    ctrl, page, used = {"paging": {}}, 1, set()
    while True:
        sent = ctrl["paging"].get("cursor")  # a failed call rewrites ctrl["paging"], so keep it first
        try:
            items = sdk.Book().list({"limit": limit}, ctrl)
        except Exception as err:
            raise PageError(page, sent, getattr(err, "status", None), err) from err
        for entity in items:
            yield entity.data_get()
        nxt = ctrl["paging"].get("next")
        if nxt is None:
            return
        if nxt in used:
            raise PageError(page + 1, nxt, "loop", "the server sent a cursor it already sent")
        used.add(nxt)
        ctrl["paging"]["cursor"] = nxt
        page += 1

The feature also adds page=1 to the first request, which this stand-in ignores as an undeclared parameter. A server that refuses unknown parameters answers 400 instead.

Page with Speakeasy’s extension or a wrapped Python client

openapi-python-client generates list_books.sync() and sync_detailed(), which return one BookPage per call. Following next is left to you. It renames the field next_, as the first block of sdks.py shows.2 opc_books() wraps sync_detailed() in the same generator shape and raises PageError on any response that did not parse as a BookPage, including a 503 the document does not list.

Speakeasy was not run, because generating needs an account. Its pagination page describes x-speakeasy-pagination with type: cursor, an input with name, in: parameters and type: cursor, and an output nextCursor written as a JSONPath, here $.next. The generated response gains next(), and the page’s loop, while response is not None: response = response.next(), stops when next() returns nothing. Wrap that loop in a generator to give callers a for.

Check it worked

check.py asserts every claim on this page and exits 1 on any failure.

python3 check.py
ok   books() walks every page in order
ok   a generator fetches only the pages its consumer reads
ok   the async generator walks and stops the same way
ok   a failed page raises PageError naming page 2 and its cursor
ok   the stand-in refuses what api.json does not allow
ok   a second consumer of one generator gets nothing
ok   a repeated cursor raises instead of looping
ok   openapi-python-client, wrapped, walks every page
ok   sdkgen paging needs next copied to cursor, and loses it on a failure
ok   sdkgen's documented loop advances on nextCursor, cursor and next_cursor
0 failed

When it goes wrong

The second consumer of a generator gets nothing. A generator runs once, so a count followed by a list comprehension over the same object reads it dry on the first pass.

python3 pitfall.py
one generator, two consumers
  count: 7
  titles: []
  requests: 3
a fresh generator per consumer
  count: 7
  titles: 7, first 'Emma', last 'Kindred'
  requests: 6

Call books() once per consumer, as the second block does, at the price of a second walk. Use itertools.tee only when both consumers advance together, since it buffers whatever one reads ahead of the other.

A retried sdkgen walk starts over. After a failed call, ctrl["paging"] holds cursor=None and hasMore=False, so a loop that catches the error and tests hasMore stops early. Keep the cursor before each call, as sdkgen_books() does.

The walk never ends. A server that repeats a cursor loops a naive generator forever, so keep the cursors already followed and raise on a repeat.

When not to do this

Do not hand-write the loop when the client you use already pages correctly. A boto3 paginator, or a Speakeasy SDK whose document carries x-speakeasy-pagination, is less code to keep. If you already hold a Speakeasy account and can add the extension, take Speakeasy: its generated next() follows the cursor with no wrapper.

Do not turn on sdkgen’s paging feature and trust its loop without running it against your own response shape. This page needed a wrapper because the cursor field is named next.

Choose openapi-python-client when typed models matter more than generated paging; the generator here is fifteen lines on top of it. Do not return a generator from a function whose callers expect a list they can read twice.

Last verified

Verified 2026-11-21 against Python 3.13.18, requests 2.34.2, httpx 0.28.1, httpx2 2.16.2, @voxgig/create-sdkgen 0.31.9, @voxgig/sdkgen 4.44.3, @voxgig/apidef 8.32.4 and openapi-python-client 0.30.0. Every output block is what the command preceding it printed, against a stand-in on 127.0.0.1, and Python 3.12.3 printed the same four blocks. The two generated clients were generated once and committed, and @voxgig/sdkgen 4.45.0 was run once on the same document to repeat the paging test. Speakeasy was not run, and x-speakeasy-pagination is described from its pagination page.

Footnotes

  1. Generators arrived in Python 2.2 through PEP 255, created in May 2001 by three authors. The asynchronous kind needed PEP 525, created in July 2016 for Python 3.6, so async for over a generator is fifteen years younger than for. boto3 describes its own Paginator as reusable, the one property neither PEP gives a generator. ↩︎ Back to text

  2. The generator appends an underscore to every name in Python’s builtins module, which its strings.py builds from dir(builtins). The built-in functions include next(), the call that advances an iterator, so the cursor becomes next_. The list exempts one builtin, id, which is why Book.id keeps its name. ↩︎ Back to text

Read this page as markdown · All how-to guides

Generate the client instead of writing it#

Retries, timeouts, pagination and auth are the same problems in every client. Voxgig generates them from your OpenAPI description, in 23 languages, from one model.

Listen to Fireside, the Voxgig podcast

Since 2018, Richard Rodger has talked with developer advocates, event organizers, and the people who build developer-facing companies about the specifics of the work.