# How to paginate an API with a Ruby enumerator > Wrap a paged list in an Enumerator that fetches pages as rows are pulled, so first(50) stays cheap, lazy.select stops early, and a 503 surfaces where you iterate. Source: https://voxgig.com/howto/paginate-an-api-with-a-ruby-enumerator - Audience: api-consumer - Level: intermediate - Languages: ruby - Verified: 2026-09-24 - Published: 2026-09-24 ## Short answer Build the list with Enumerator.new and a yielder that requests the next page when the caller pulls past the current one. first(n) stops early, each walks to the end, and an HTTP failure raised inside the yielder surfaces at whichever call is iterating. Put .lazy in front of select and map, or they walk every page before returning. Prefer internal iteration: next runs the block on a Fiber, and an abandoned Fiber never runs its ensure. --- ## You will need Ruby 3.1 or later, and a list endpoint that returns a cursor. The sample takes the fetch as a callable, so the demos and tests run against an in-memory API with a request log. The same `Pager` runs unchanged over [Faraday](https://lostisland.github.io/faraday/) against a real endpoint. The class behind all of it is [`Enumerator`](https://docs.ruby-lang.org/en/3.3/Enumerator.html), and the property the page depends on is that its block runs only as far as the caller pulls. ## Approaches compared | Approach | When it fits | What it costs you | When to pick something else | | --- | --- | --- | --- | | [Enumerator.new with a yielder](https://docs.ruby-lang.org/en/3.3/Enumerator.html#method-c-new) | Any API, any HTTP client, and callers who want to chain Enumerable methods | The cursor loop, the repeat guard and the error translation are yours to write and test, and external iteration needs care | The vendor's gem already iterates and you call that one API | | [Octokit.rb `auto_paginate`](https://github.com/octokit/octokit.rb) | Small GitHub collections you want whole, from one call | Every page is fetched and concatenated into one Array before you see a row, and the walk stops without a word when the rate limit reaches zero | Collections longer than a few pages, or a caller that stops early | | [stripe-ruby `auto_paging_each`](https://docs.stripe.com/api/pagination/auto?lang=ruby) | Stripe lists, walked forward from a list call or backward from `ending_before` | A block-based walk tied to Stripe's list objects, with the direction decided by which filter you passed | Any API that is not Stripe, or code that wants an Enumerator to chain | The three differ in who decides when the next request happens. `auto_paginate` decides before you see a row, and fetches everything. `auto_paging_each` decides per row inside a block, and the block's control flow is what you get. The Enumerator hands the decision to whichever Enumerable method the caller chose, which is the property the rest of this page is about. ## Yield rows, fetch pages Two enumerators: one over pages, one over rows. The page one knows about cursors and the row one knows about neither. ```ruby title="pager.rb" def self.pages(fetch) Enumerator.new do |yielder| cursor = nil seen = {} loop do raise "pagination repeated the cursor #{cursor.inspect}" if seen.key?(cursor) seen[cursor] = true page = fetch.call(cursor) yielder << page["data"] cursor = page["next_cursor"] break if cursor.nil? || cursor.empty? end end end # One row at a time. Callers see neither pages nor cursors. def self.rows(fetch) Enumerator.new do |yielder| pages(fetch).each { |page| page.each { |row| yielder << row } } end end ``` `yielder << page["data"]` is where the block stops. Nothing after that line runs until the caller asks for another element, so `fetch.call` for page two happens when a caller reads past row three, and not before. The `seen` hash is the guard against a server that echoes the cursor it was sent, which turns a walk into a loop with no end. Raising is right: a silent stop looks like the end of the collection. `fetch` is a callable rather than a client, and that decision is what makes the rest of the page testable. The demos pass a method on an in-memory API. Production passes a lambda over Faraday. ## Put lazy in front of select [`first(n)`](https://docs.ruby-lang.org/en/3.3/Enumerable.html#method-i-first) stops iterating after `n` elements, so it requests only the pages those elements sit on. `select` does not stop, because it has to see every element to know which ones match, and that is true even when a `first(2)` follows it. The lazy form changes the order: the filter runs one row at a time and the `first(2)` at the end decides when to stop. ```bash ruby demo.rb ``` ```text output first(5) from seven rows in pages of three rows: 1,2,3,4,5 requests: 2 GET /items?cursor= | GET /items?cursor=c3 select, then first(2) rows: 2,4 requests: 3 GET /items?cursor= | GET /items?cursor=c3 | GET /items?cursor=c6 lazy.select, then first(2) rows: 2,4 requests: 2 GET /items?cursor= | GET /items?cursor=c3 the third page answers 503 rows delivered before it: 1,2,3,4,5,6 raised at the each call: Pager::FetchError: GET page "c6" answered 503 the server repeats its cursor stopped after 2 requests: pagination repeated the cursor "c3" ``` The second and third blocks return the same two rows. One of them fetched the whole collection to do it. Against seven rows the difference is a request; against nine thousand it is the difference between a script that finishes and one that gets rate limited. Every [`Enumerator::Lazy`](https://docs.ruby-lang.org/en/3.3/Enumerator/Lazy.html) method returns another lazy enumerator, so `map`, `reject` and `take_while` chain the same way, and the chain runs when something at the end of it asks.[^1] ## Raise from inside the yielder The fourth block is an HTTP failure on the third page. The error is raised inside `fetch.call`, which is inside the block that yields, which is running because a caller called `each`. So the exception arrives at that `each`, after the six rows before it were delivered. ```ruby title="pager.rb" class FetchError < StandardError attr_reader :status, :cursor def initialize(status, cursor) @status = status @cursor = cursor super("GET page #{cursor.inspect} answered #{status}") end end ``` The cursor in the message is the useful part. A caller that rescues `FetchError` can log which page failed and, if the rows already delivered were written somewhere, resume from that cursor rather than from the start. Faraday's [`raise_error` middleware](https://lostisland.github.io/faraday/#/middleware/included/raising-errors) turns a 4xx or 5xx into an exception of its own, and the fetcher translates it so the cursor rides along. ```ruby title="faraday_fetch.rb" def faraday_fetch(base_url, path: "/items", per_page: 100) conn = Faraday.new(url: base_url) do |f| f.response :raise_error f.response :json end lambda do |cursor| params = { limit: per_page } params[:cursor] = cursor if cursor conn.get(path, params).body rescue Faraday::Error => e raise Pager::FetchError.new(e.response ? e.response[:status] : "no response", cursor) end end ``` That lambda is the whole production adapter. Everything else on the page runs against the in-memory fetch and is unchanged when this one is passed instead. ## Check it worked The tests count requests rather than rows, because the number of requests is the property an Array cannot have. ```bash ruby test.rb ``` ```text output ok first(n) requests only the pages it needs ok nothing is fetched until the first row is pulled ok each walks every page and visits every row once ok select walks every page, lazy.select stops at the second match ok an HTTP failure surfaces at the call site with the cursor in the message ok a repeated cursor raises instead of looping ok breaking out of each runs the ensure block, abandoning next does not 7 cases, 0 failures ``` The second case is the one to keep when you cut the rest: calling `Pager.rows` makes no request, and the first `first` makes exactly one. ## See what next leaves behind [`next`](https://docs.ruby-lang.org/en/3.3/Enumerator.html#method-i-next) is external iteration. The enumerator's block runs on a Fiber that is suspended between calls, and a caller who stops calling `next` leaves it suspended. Anything the block was holding, a connection, a file, a transaction, stays held, and an `ensure` around the loop never runs. The script below opens a connection inside the block and closes it in `ensure`. ```bash ruby external.rb ``` ```text output first(2), internal iteration: connection open? false next twice, then abandoned: connection open? true after rewind: connection open? true after the enumerator is collected: connection open? true next until StopIteration: connection open? false ``` `first(2)` breaks out of the block, the `ensure` runs, and the connection is closed. Two calls to `next` followed by nothing leave it open. `rewind` drops the Fiber without running it to its end, so the connection stays open, and so does collecting the enumerator itself.[^2] The only way out through `next` is to call it until it raises `StopIteration`, which means walking every page. Use internal iteration for anything that holds a resource. When external iteration is the shape you need, keep the resource outside the block and close it yourself. ## When it goes wrong `select` fetches the whole collection before returning. The chain is eager because `.lazy` is missing or sits after the `select` rather than before it. Move `.lazy` to the front of the chain, and count requests in a test. The walk repeats rows or never ends. The server returned the cursor it was given, or rows moved between pages because the sort is not stable. The `seen` guard catches the first. For the second, sort on an immutable key such as the id. The exception has no page in it. The fetch raised a Faraday error that was passed through untouched, or a plain `RuntimeError`. Translate at the fetch, as `faraday_fetch` does, so the cursor is in the message. Callers get `StopIteration` from somewhere they did not expect. They called `next` past the end. `loop` rescues it, and `each` never raises it, which is one more reason to hand callers an Enumerator and let them use Enumerable. ## When not to do this Do not return an Enumerator when the caller needs the count first. An Enumerator has no size until it has walked, and `size` on this one answers `nil`. Ask the API for a total, or return the first page and its count as a plain object. Do not hold a database transaction open around a walk. An Enumerator can run for as long as the collection is long, and a lock held for the duration of a remote API walk is a lock nobody planned. Commit per batch. Do not reach for `auto_paginate` on a collection you cannot bound. Octokit fetches every page into one Array before returning, and its loop condition also checks the rate limit, so a long collection comes back truncated with no error.[^3] Do not use `auto_paging_each` as a general iterator. It is a method on Stripe's list objects, it walks backward when `ending_before` is the only filter, and it gives you a block rather than an Enumerable.[^4] Wrap it in an Enumerator if your callers want to chain. ## Related how-tos - [Loop over every page of a REST collection](/howto/loop-over-every-page-of-a-collection) - [Paginate an API with PHP generators](/howto/paginate-an-api-with-php-generators) ## Last verified Verified 2026-09-24 against Ruby 3.3.6 and Faraday 2.14.0. Every output block is what the preceding command printed, against the in-memory API in the sample directory. `faraday_fetch.rb` was run against a local HTTP server with the same page shape and is not exercised by the output blocks, because the runner has no gems. The Octokit and stripe-ruby behavior was read from the sources of Octokit 10.0.0 and stripe-ruby 19.6.2, linked in the notes. [^1]: `Enumerable#lazy` arrived with [Ruby 2.0.0 in February 2013](https://www.ruby-lang.org/en/news/2013/02/24/ruby-2-0-0-p0-is-released/), in a release whose notes describe it as being for possibly infinite lazy streams. A paged API is not infinite, only unbounded from where the caller stands, which is the same thing for the purpose. `select` has been eager for longer than that, and the five characters that change it have been available for over a decade. [^2]: The Ruby documentation lists [what external iteration changes](https://docs.ruby-lang.org/en/3.3/Enumerator.html#class-Enumerator-label-External+Iteration): the Fiber adds overhead, the stack trace stops at the Enumerator, and Fiber-local variables are not inherited. It does not mention that an `ensure` in a block that is never resumed will never run. That is the ordinary behavior of a Fiber, and so did not need saying. [^3]: Octokit's [`paginate`](https://github.com/octokit/octokit.rb/blob/main/lib/octokit/connection.rb) loops while the response has a `next` link and `rate_limit.remaining > 0`, and concatenates each page's data onto the first. When the second condition fails the loop ends and the Array is returned, shorter than the collection, and the caller has no way to tell that from the end. [^4]: The comment in stripe-ruby's [`auto_paging_each`](https://github.com/stripe/stripe-ruby/blob/master/lib/stripe/list_object.rb) explains that backward iteration activates with an `ending_before` constraint and only an `ending_before` constraint, and that if `starting_after` was also used it iterates forwards. The direction of a walk, decided by which of two filters was absent.