How to paginate an API with PHP generators#

The import job collects every page into one array and the worker runs out of memory on the customer with 90,000 records. The code is correct and it loads the whole collection before it processes the first row. Splitting the loop in two, one for pages and one for rows, has spread page arithmetic through the job.

What you get

You will end up with a pager that yields rows one at a time, holding one page in memory, and a caller that reads as a single foreach. Stopping early stops the requests. This is for you if a PHP job imports collections whose size you do not control.

Short answer

Write a generator that yields one page at a time and asks for the next only when the consumer pulls. Wrap it in a second generator that yields rows, so calling code is a single foreach with no page index. Nothing is requested until the first row is pulled, and breaking out of the loop stops the requests immediately.

You will need

PHP 8.1 or later, and a list endpoint that returns a cursor. The behavior rests on generators being lazy, which is the property that separates them from a function returning an array.¹ A generator is also an Iterator, so anything that accepts an iterable accepts it without a wrapper.

Approaches compared

The dividing line is whether the next request is knowable in advance. Cursor pagination hides it, so the walk is inherently sequential and a generator fits exactly. Offset pagination exposes it, which makes concurrency possible and makes the rows shift under you when somebody writes. Pick the concurrency only when the endpoint offers numbered pages and you can accept the skew.

Yield pages, then yield rows

Two small generators, each doing one thing.

function rows(callable $fetch): Generator
{
    foreach (pages($fetch) as $page) {
        foreach ($page['data'] as $row) {
            yield $row;
        }
    }
}

The inner generator knows about cursors and the outer one knows about rows. A caller sees neither. That separation is what keeps the page arithmetic in one place, and it means adding a second cursor style later touches pages alone. When a vendor switches from a cursor to a Link header, only one function changes, so no caller notices.

Take the fetcher as an argument. Passing a callable rather than a client makes the walk testable without a network, and it is why the tests below run offline.

Guard the cursor you have already used

A server that returns the cursor you sent is a real failure, and a generator will follow it forever.

if ($cursor !== null && isset($seen[$cursor])) {
    throw new RuntimeException("pagination repeated the cursor {$cursor}");
}

Raise rather than return. A silent stop looks like the end of the collection, and a job that imports a prefix and exits zero is worse than one that fails, because nobody investigates a success. Keep the whole set of cursors you have used rather than only the previous one. Some servers alternate between two values, and a check against the last cursor sees nothing wrong.

Check it worked

Walk the whole collection, then stop after four rows, then meet the broken server.

php demo.php

every row: 1,2,3,4,5,6,7
requests:  3

stopping at four rows
rows:      1,2,3,4
requests:  GET /meters?cursor=0 | GET /meters?cursor=3

server repeats its cursor
stopped after 2 requests: pagination repeated the cursor 0

Read the middle block. Four rows came from two requests, and the third page was never asked for. That is the property an array-collecting loop cannot have, and it is what makes a generator worth the slightly odd control flow.

The third block is the guard earning its place. The broken server was stopped on the second request rather than the millionth, and the job failed loudly. A cap on the page count would also have stopped it, after a thousand pointless requests against an API that charges you for them.

php -d zend.assertions=1 test.php

ok a full walk visits every row once
ok breaking early stops the requests
ok nothing is fetched until the first row is pulled
ok an empty next_cursor ends the walk
ok a repeated cursor raises instead of looping
5 cases, 0 failures

When it goes wrong

Nothing happens when you call the pager. A generator runs no code until it is iterated. Calling it and discarding the result does nothing at all, which is correct and surprising.

iterator_to_array fills memory again. It collects everything the generator yields, which undoes the reason for the generator. Use it in tests and avoid it in the job.

Keys collide in the collected array. iterator_to_array preserves keys by default, so rows from later pages overwrite earlier ones when the keys repeat. Pass false as the second argument.²

The walk cannot be resumed. A generator holds no state you can persist.³ Record the last cursor you finished with, and pass it back as the starting point. The pages function already accepts one, which is the smallest change that makes a long import restartable. Store that cursor beside the rows you wrote, in the same transaction, or a restart repeats a batch.

When not to do this

Do not use a generator where you need the count first. Deciding whether to proceed based on the total means fetching the total, and a generator gives you rows rather than a number. Ask the API for a count, or accept walking twice. Some endpoints return a total in the first page, which costs one extra field and answers the question.

Do not open a database transaction around the whole walk. A generator can run for minutes, and a transaction held open for the length of a remote API walk is a lock nobody predicted. Commit per batch, and make the write idempotent so a restart can repeat one.

Do not yield rows straight into a write without batching. One insert per row across 90,000 rows is slow in a way that has nothing to do with pagination. Chunk the rows, and keep the walk lazy. A buffer of a few hundred rows gets you both properties, and it fits in the same foreach.

Related how-tos

• Loop over every page of a REST collection

• Choose a pagination style for a list endpoint

Last verified

Verified 2026-09-14 against PHP 8.4.19. Both output blocks are what the preceding command printed.