# How to loop over every page of a REST collection

> Walk a paginated list endpoint to the end, read the three stop signals APIs actually send, and refuse to loop forever when a server repeats its cursor.

Source: https://voxgig.com/howto/loop-over-every-page-of-a-collection

- Audience: api-consumer
- Level: beginner
- Languages: typescript, javascript
- Verified: 2026-09-14
- Published: 2026-09-14
- Updated: 2026-09-24

## Short answer

Loop until the server says stop, and read the signal the API documents rather than the length of the page. Stripe sends `has_more`, Slack empties `next_cursor`, and GitHub drops `rel="next"` from its Link header. Keep the cursors you have already requested in a set, and raise an error when one repeats, so a buggy server cannot spin your worker forever.

---
## You will need

Node 22 or later, and a list endpoint that returns more rows than one page holds. The sample serves
all three signal shapes from one local server, so the loop can be exercised against each without
three vendor accounts.

Voxgig maintains sdkgen. This page compares its paging feature with Octokit, Speakeasy, and a loop
you write yourself.

## Approaches compared

| Approach | When it fits | What it costs you | When to pick something else |
| --- | --- | --- | --- |
| [A hand-written cursor loop](https://docs.stripe.com/api/pagination) | One or two endpoints, and you want the stop condition visible in the code | You repeat the loop per endpoint, and each copy can get the stop signal wrong | The API already has a generated client that iterates for you |
| [Octokit paginate.iterator](https://github.com/octokit/plugin-paginate-rest.js) | You are calling GitHub, where Link headers and rate limits are already handled | A client tied to one API, so a second vendor means a second iteration model | You call several APIs and want one shape for all of them |
| [sdkgen paging feature](https://github.com/voxgig/sdkgen) | You generate the client anyway and want paging on without writing it per operation | An opt-in feature to activate, and a generated client to regenerate when the spec moves | You hand-write the client, or the API paginates in a way no feature models |
| [Speakeasy pagination](https://www.speakeasy.com/docs/customize-sdks/pagination) | You already generate with Speakeasy and can annotate the spec | Pagination described in the OpenAPI document, which the API owner has to accept | You do not own the spec and cannot add vendor extensions to it |

The real division is who owns the stop condition. A hand-written loop puts it in front of the
reader, where a wrong guess is visible in review. Every generated iterator hides it, which is
correct until the server does something the generator did not model, and then the bug is in a
dependency rather than in your file. Generated iterators also fetch the whole collection by
default, so an early `break` matters more than it looks.

## Read the signal, not the page length

Three signals cover most of what you will meet, and the sample serves each from its own route.
[Stripe sends a boolean](https://docs.stripe.com/api/pagination).[^1]
[Slack empties its cursor](https://docs.slack.dev/apis/web-api/pagination).[^2]
[GitHub drops a Link header entry](https://docs.github.com/en/rest/using-the-rest-api/using-pagination-in-the-rest-api),
in the form [RFC 8288](https://www.rfc-editor.org/rfc/rfc8288.html) defines.

```js title="paginate.mjs"
export async function* pages(firstUrl, next, { maxPages = 1000 } = {}) {
  let url = firstUrl
  const seen = new Set()
  for (let n = 0; url; n++) {
    if (n >= maxPages) throw new Error(`pagination exceeded ${maxPages} pages`)
    if (seen.has(url)) throw new Error(`pagination repeated a page: ${url}`)
    seen.add(url)
    const res = await fetch(url)
    const body = await res.json()
    yield body.data
    url = next(body, res, url)
  }
}
```

The `next` callback is the only part that changes per API, and it gets the parsed body, the
response, and the URL it came from. Stripe needs the body, GitHub needs the headers,[^3] and a cursor
that has to be incremented needs the previous URL. Passing all three means one walker covers the
three shapes with no branching inside it.

Parse a Link header, do not pattern match it. RFC 8288 puts no order on a link-value's parameters,
so `rel` can sit after `type` or `title`, it can be unquoted, and it can name several relation types
at once. A regular expression that expects `rel` straight after the URI returns null for a header
that is entirely valid. The walk then stops early, returning a partial result without an error. That is the
worst shape a pagination bug can take, because every row it did fetch looks right.

An empty page is not a stop signal. A server that filters after paginating returns zero rows on a
page in the middle of the collection, and a loop that treats that as the end silently imports a
prefix. The first test below pins that.

## Refuse to walk forever

A server that echoes the cursor you sent back to you is a real failure mode, and it turns a sync
job into a billing incident. The walker keeps the URLs it has requested and fails on the second
sighting.

```bash
node demo.mjs
```

```text output
stripe     3 requests, ids 1,2,3,4,5,6,7
slack      3 requests, ids 1,2,3,4,5,6,7
github     3 requests, ids 1,2,3,4,5,6,7
stuck      2 requests, stopped: pagination repeated a page
```

Three signals, one loop, the same seven rows. The fourth route repeats its cursor, and the walk
stops on the second request rather than on the ten-thousandth. A page cap alone would also stop it,
eventually, after 1,000 pointless requests against a rate-limited API.

## Check it worked

The tests cover the two cases a hand-rolled loop usually gets wrong, plus the header parsing.

```bash
node --test paginate.test.mjs
```

```text output
1..5
# tests 5
# suites 0
# pass 5
# fail 0
# cancelled 0
# skipped 0
# todo 0
# duration_ms 322.025089
```

Test four asserts that breaking out of the `for await` makes no further request. Confirm that
against your own client before you rely on it: an iterator that fetches the following page ahead of time
has already spent a request by the time your `break` runs.

## When it goes wrong

The import stops short. The loop used page length as its stop condition and the server returned a
short page. Read the documented signal, and treat a short page as ordinary.

The job runs for hours and the row count climbs past the collection size. The cursor is being
echoed, or the sort key is not stable and rows move between pages while you walk. Add the repeated
cursor guard, and sort by an immutable key such as the id.

Rows go missing from the middle. Offset pagination shifts when a row is inserted or deleted during
the walk, so page two skips whatever moved. Use the cursor form for anything that takes more than a
moment.

The walk succeeds, but the totals still disagree with the vendor's dashboard. Check whether the
endpoint hides soft-deleted or archived rows behind a query parameter. That is a filter question
rather than a pagination one, and it is the first thing to rule out before rewriting the loop.

## When not to do this

Do not walk a collection you only need the head of. A search endpoint sorted by relevance answers
the question in its first page, and pulling 40 more costs the API owner real money for rows you
throw away. Ask for a filter instead.

Do not run the walk inside a request handler. A caller waiting on 4,000 rows gets a timeout, and a
retry restarts the walk from page one. Push it to a job, and keep the last cursor so a restart
resumes.

Do not turn on the sdkgen paging feature and assume the loop is solved. It walks the shapes its
model describes, and an API that signals the end some other way needs the escape hatch and a loop
you write. Check the generated client against a collection with a short page in the middle before
you trust it in a sync.

## Related how-tos

- [Choose a pagination style for a list endpoint](/howto/choose-a-pagination-style-for-a-list-endpoint)

- [Advertise next and previous pages with Link headers](/howto/link-headers-for-pagination-rfc-8288)

## Last verified

Verified 2026-09-14 against Node 22.22.2. Both output blocks are what the preceding command
printed.

[^1]: The boolean has a companion number. Stripe's
[pagination reference](https://docs.stripe.com/api/pagination) gives `limit` a default of 10 and a
range of 1 to 100, so a collection of 4,000 rows is 400 requests at the default and 40 at the
ceiling. `has_more` is the stop signal, and it is a boolean. It says nothing about how many more,
which is the boolean's charm, since a count would have to be right.

[^2]: The cursor is less opaque than the word suggests. Slack's
[pagination guide](https://docs.slack.dev/apis/web-api/pagination) walks a workspace of three
users with a limit of two. The `next_cursor` it shows is `dXNlcjpVMEc5V0ZYTlo=`, which base64
decodes to `user:U0G9WFXNZ`, a user id with a label on it. The guide also warns against checking
the size of a page against the limit to conclude the results are complete, and calls that a
temptation, which is the right word for it.

[^3]: Absence is the signal, and it is also the default. GitHub's
[pagination guide](https://docs.github.com/en/rest/using-the-rest-api/using-pagination-in-the-rest-api)
says the Link header is omitted when an endpoint does not paginate at all, and also when every
result fits on one page. A client that walks until `rel="next"` is missing handles a one-page
collection without noticing. One that expects the header to be present, as proof that it is on a
paginated endpoint, meets its first single page as a surprise.