# How to paginate an API with PHP generators

> Walk a paginated collection with a generator, so a caller writes one foreach, pages are fetched only as they are consumed, and an early break costs nothing.

Source: https://voxgig.com/howto/paginate-an-api-with-php-generators

- Audience: api-consumer
- Level: intermediate
- Languages: php
- Verified: 2026-09-14
- Published: 2026-09-14
- Updated: 2026-09-24

## Short answer

Write a generator that yields one page at a time and asks for the next only when the consumer pulls. Wrap it in a second generator that yields rows, so calling code is a single foreach with no page index. Nothing is requested until the first row is pulled, and breaking out of the loop stops the requests immediately.

---
## You will need

PHP 8.1 or later, and a list endpoint that returns a cursor. The behavior rests on
[generators](https://www.php.net/manual/en/language.generators.overview.php) being lazy, which is
the property that separates them from a function returning an array.[^1] A generator is also an
[Iterator](https://www.php.net/manual/en/class.iterator.php), so anything that accepts an iterable
accepts it without a wrapper.

## Approaches compared

| Approach | When it fits | What it costs you | When to pick something else |
| --- | --- | --- | --- |
| [A generator over pages](https://www.php.net/manual/en/language.generators.syntax.php) | Any walk whose length you do not control, and any caller that may stop early | You cannot count the rows before walking them, and rewinding means starting again | You need the whole set in memory anyway, to sort it |
| [A while loop collecting into an array](https://www.php.net/manual/en/control-structures.while.php) | Small collections with a known ceiling, where the array is what you want | Memory proportional to the collection, and the whole walk before the first row | The collection can grow without a bound you set |
| [An SDK paginator](https://docs.aws.amazon.com/sdk-for-php/v3/developer-guide/guide_paginators.html) | The vendor ships one and it covers the endpoints you call | An interface per vendor, so two APIs mean two shapes to learn | You call several APIs and want one walk for all of them |
| [Guzzle Pool with concurrent requests](https://docs.guzzlephp.org/en/stable/quickstart.html#concurrent-requests) | Pages you can address without walking, such as numbered offsets | Concurrency against an API that rate limits, and no cursor to follow | Pages are linked by a cursor you only learn by fetching |

The dividing line is whether the next request is knowable in advance. Cursor pagination hides it, so
the walk is inherently sequential and a generator fits exactly. Offset pagination exposes it, which
makes concurrency possible and makes the rows shift under you when somebody writes. Pick the
concurrency only when the endpoint offers numbered pages and you can accept the skew.

## Yield pages, then yield rows

Two small generators, each doing one thing.

```php title="Pager.php"
function rows(callable $fetch): Generator
{
    foreach (pages($fetch) as $page) {
        foreach ($page['data'] as $row) {
            yield $row;
        }
    }
}
```

The inner generator knows about cursors and the outer one knows about rows. A caller sees neither.
That separation is what keeps the page arithmetic in one place, and it means adding a second cursor
style later touches `pages` alone. When a vendor switches from a cursor to a Link header, only one
function changes, so no caller notices.

Take the fetcher as an argument. Passing a callable rather than a client makes the walk testable
without a network, and it is why the tests below run offline.

## Guard the cursor you have already used

A server that returns the cursor you sent is a real failure, and a generator will follow it forever.

```php title="Pager.php"
if ($cursor !== null && isset($seen[$cursor])) {
    throw new RuntimeException("pagination repeated the cursor {$cursor}");
}
```

Raise rather than return. A silent stop looks like the end of the collection, and a job that imports
a prefix and exits zero is worse than one that fails, because nobody investigates a success. Keep
the whole set of cursors you have used rather than only the previous one. Some servers alternate
between two values, and a check against the last cursor sees nothing wrong.

## Check it worked

Walk the whole collection, then stop after four rows, then meet the broken server.

```bash
php demo.php
```

```text output
every row: 1,2,3,4,5,6,7
requests:  3

stopping at four rows
rows:      1,2,3,4
requests:  GET /meters?cursor=0 | GET /meters?cursor=3

server repeats its cursor
stopped after 2 requests: pagination repeated the cursor 0
```

Read the middle block. Four rows came from two requests, and the third page was never asked for.
That is the property an array-collecting loop cannot have, and it is what makes a generator worth
the slightly odd control flow.

The third block is the guard earning its place. The broken server was stopped on the second request
rather than the millionth, and the job failed loudly. A cap on the page count would also have
stopped it, after a thousand pointless requests against an API that charges you for them.

```bash
php -d zend.assertions=1 test.php
```

```text output
ok a full walk visits every row once
ok breaking early stops the requests
ok nothing is fetched until the first row is pulled
ok an empty next_cursor ends the walk
ok a repeated cursor raises instead of looping
5 cases, 0 failures
```

## When it goes wrong

Nothing happens when you call the pager. A generator runs no code until it is iterated. Calling it
and discarding the result does nothing at all, which is correct and surprising.

`iterator_to_array` fills memory again. It collects everything the generator yields, which undoes
the reason for the generator. Use it in tests and avoid it in the job.

Keys collide in the collected array. `iterator_to_array` preserves keys by default, so rows from
later pages overwrite earlier ones when the keys repeat. Pass `false` as the second argument.[^2]

The walk cannot be resumed. A generator holds no state you can persist.[^3] Record the last cursor you
finished with, and pass it back as the starting point. The `pages` function already accepts one,
which is the smallest change that makes a long import restartable. Store that cursor beside the rows
you wrote, in the same transaction, or a restart repeats a batch.

## When not to do this

Do not use a generator where you need the count first. Deciding whether to proceed based on the
total means fetching the total, and a generator gives you rows rather than a number. Ask the API for
a count, or accept walking twice. Some endpoints return a total in the first page, which costs one
extra field and answers the question.

Do not open a database transaction around the whole walk. A generator can run for minutes, and a
transaction held open for the length of a remote API walk is a lock nobody predicted. Commit per
batch, and make the write idempotent so a restart can repeat one.

Do not yield rows straight into a write without batching. One insert per row across 90,000 rows is
slow in a way that has nothing to do with pagination. Chunk the rows, and keep the walk lazy. A
buffer of a few hundred rows gets you both properties, and it fits in the same foreach.

## Related how-tos

- [Loop over every page of a REST collection](/howto/loop-over-every-page-of-a-collection)

- [Choose a pagination style for a list endpoint](/howto/choose-a-pagination-style-for-a-list-endpoint)

## Last verified

Verified 2026-09-14 against PHP 8.4.19. Both output blocks are what the preceding command printed.

[^1]: Generators reached PHP through [an RFC](https://wiki.php.net/rfc/generators) dated June 2012,
by Nikita Popov. Its opening example is a function that reads every line of a file into an array
before returning any of them. The
[manual](https://www.php.net/manual/en/language.generators.overview.php) introduces the feature the
same way, as a means of feeding a foreach without building an array in memory ahead of time, which
may exceed a memory limit. The failure in the opening paragraph of this page is the one the feature
was built for, with a file in place of an API.

[^2]: The [signature](https://www.php.net/manual/en/function.iterator-to-array.php) is
`iterator_to_array(Traversable|array $iterator, bool $preserve_keys = true)`. The changelog records
one change, in 8.2.0: the first parameter, an iterator by name and by type, also accepts an array,
so the function can convert an array into an array. There are uses for this. The name does not
mention them.

[^3]: Nor can it go back. The manual likens the object to a forward-only iterator, and
[`Generator::rewind`](https://www.php.net/manual/en/generator.rewind.php) exists all the same. It is
the first method foreach calls. It runs the generator up to its first yield, does nothing if it is
already there, and throws if it has ever advanced beyond one. The message is that it cannot rewind a
generator that was already run. Of its three documented outcomes, none is a rewind.