Your sync job imports 100 customers and stops. The API holds 4,000. The loop broke on the first page that came back shorter than the page size, because that looked like the end, and the vendor returns short pages whenever a filter excludes rows. Nobody noticed for a week, because the job kept exiting zero.
What you get
You will end up with one generator that walks a collection to its real end. It also carries a guard that turns a server-side pagination bug into a raised error rather than an endless loop. This is for you if you are writing the import side of an integration.
Short answer
Loop until the server says stop, and read the signal the API documents rather than the length of
the page. Stripe sends has_more, Slack empties next_cursor, and GitHub drops rel="next" from
its Link header. Keep the cursors you have already requested in a set, and raise an error when one
repeats, so a buggy server cannot spin your worker forever.
You will need
Node 22 or later, and a list endpoint that returns more rows than one page holds. The sample serves all three signal shapes from one local server, so the loop can be exercised against each without three vendor accounts.
Voxgig maintains sdkgen. This page compares its paging feature with Octokit, Speakeasy, and a loop you write yourself.
Approaches compared
| Approach | When it fits | What it costs you | When to pick something else |
|---|---|---|---|
| A hand-written cursor loop | One or two endpoints, and you want the stop condition visible in the code | You repeat the loop per endpoint, and each copy can get the stop signal wrong | The API already has a generated client that iterates for you |
| Octokit paginate.iterator | You are calling GitHub, where Link headers and rate limits are already handled | A client tied to one API, so a second vendor means a second iteration model | You call several APIs and want one shape for all of them |
| sdkgen paging feature | You generate the client anyway and want paging on without writing it per operation | An opt-in feature to activate, and a generated client to regenerate when the spec moves | You hand-write the client, or the API paginates in a way no feature models |
| Speakeasy pagination | You already generate with Speakeasy and can annotate the spec | Pagination described in the OpenAPI document, which the API owner has to accept | You do not own the spec and cannot add vendor extensions to it |
The real division is who owns the stop condition. A hand-written loop puts it in front of the
reader, where a wrong guess is visible in review. Every generated iterator hides it, which is
correct until the server does something the generator did not model, and then the bug is in a
dependency rather than in your file. Generated iterators also fetch the whole collection by
default, so an early break matters more than it looks.
Read the signal, not the page length
Three signals cover most of what you will meet, and the sample serves each from its own route. Stripe sends a boolean.1 Slack empties its cursor.2 GitHub drops a Link header entry, in the form RFC 8288 defines.
export async function* pages(firstUrl, next, { maxPages = 1000 } = {}) {
let url = firstUrl
const seen = new Set()
for (let n = 0; url; n++) {
if (n >= maxPages) throw new Error(`pagination exceeded ${maxPages} pages`)
if (seen.has(url)) throw new Error(`pagination repeated a page: ${url}`)
seen.add(url)
const res = await fetch(url)
const body = await res.json()
yield body.data
url = next(body, res, url)
}
}The next callback is the only part that changes per API, and it gets the parsed body, the
response, and the URL it came from. Stripe needs the body, GitHub needs the headers,3 and a cursor
that has to be incremented needs the previous URL. Passing all three means one walker covers the
three shapes with no branching inside it.
Parse a Link header, do not pattern match it. RFC 8288 puts no order on a link-value’s parameters,
so rel can sit after type or title, it can be unquoted, and it can name several relation types
at once. A regular expression that expects rel straight after the URI returns null for a header
that is entirely valid. The walk then stops early, returning a partial result without an error. That is the
worst shape a pagination bug can take, because every row it did fetch looks right.
An empty page is not a stop signal. A server that filters after paginating returns zero rows on a page in the middle of the collection, and a loop that treats that as the end silently imports a prefix. The first test below pins that.
Refuse to walk forever
A server that echoes the cursor you sent back to you is a real failure mode, and it turns a sync job into a billing incident. The walker keeps the URLs it has requested and fails on the second sighting.
node demo.mjsstripe 3 requests, ids 1,2,3,4,5,6,7
slack 3 requests, ids 1,2,3,4,5,6,7
github 3 requests, ids 1,2,3,4,5,6,7
stuck 2 requests, stopped: pagination repeated a pageThree signals, one loop, the same seven rows. The fourth route repeats its cursor, and the walk stops on the second request rather than on the ten-thousandth. A page cap alone would also stop it, eventually, after 1,000 pointless requests against a rate-limited API.
Check it worked
The tests cover the two cases a hand-rolled loop usually gets wrong, plus the header parsing.
node --test paginate.test.mjs1..5
# tests 5
# suites 0
# pass 5
# fail 0
# cancelled 0
# skipped 0
# todo 0
# duration_ms 322.025089Test four asserts that breaking out of the for await makes no further request. Confirm that
against your own client before you rely on it: an iterator that fetches the following page ahead of time
has already spent a request by the time your break runs.
When it goes wrong
The import stops short. The loop used page length as its stop condition and the server returned a short page. Read the documented signal, and treat a short page as ordinary.
The job runs for hours and the row count climbs past the collection size. The cursor is being echoed, or the sort key is not stable and rows move between pages while you walk. Add the repeated cursor guard, and sort by an immutable key such as the id.
Rows go missing from the middle. Offset pagination shifts when a row is inserted or deleted during the walk, so page two skips whatever moved. Use the cursor form for anything that takes more than a moment.
The walk succeeds, but the totals still disagree with the vendor’s dashboard. Check whether the endpoint hides soft-deleted or archived rows behind a query parameter. That is a filter question rather than a pagination one, and it is the first thing to rule out before rewriting the loop.
When not to do this
Do not walk a collection you only need the head of. A search endpoint sorted by relevance answers the question in its first page, and pulling 40 more costs the API owner real money for rows you throw away. Ask for a filter instead.
Do not run the walk inside a request handler. A caller waiting on 4,000 rows gets a timeout, and a retry restarts the walk from page one. Push it to a job, and keep the last cursor so a restart resumes.
Do not turn on the sdkgen paging feature and assume the loop is solved. It walks the shapes its model describes, and an API that signals the end some other way needs the escape hatch and a loop you write. Check the generated client against a collection with a short page in the middle before you trust it in a sync.
Related how-tos
Last verified
Verified 2026-09-14 against Node 22.22.2. Both output blocks are what the preceding command printed.
Footnotes
-
The boolean has a companion number. Stripe’s pagination reference gives
limita default of 10 and a range of 1 to 100, so a collection of 4,000 rows is 400 requests at the default and 40 at the ceiling.has_moreis the stop signal, and it is a boolean. It says nothing about how many more, which is the boolean’s charm, since a count would have to be right. ↩︎ Back to text -
The cursor is less opaque than the word suggests. Slack’s pagination guide walks a workspace of three users with a limit of two. The
next_cursorit shows isdXNlcjpVMEc5V0ZYTlo=, which base64 decodes touser:U0G9WFXNZ, a user id with a label on it. The guide also warns against checking the size of a page against the limit to conclude the results are complete, and calls that a temptation, which is the right word for it. ↩︎ Back to text -
Absence is the signal, and it is also the default. GitHub’s pagination guide says the Link header is omitted when an endpoint does not paginate at all, and also when every result fits on one page. A client that walks until
rel="next"is missing handles a one-page collection without noticing. One that expects the header to be present, as proof that it is on a paginated endpoint, meets its first single page as a surprise. ↩︎ Back to text