How to paginate an API with a Ruby enumerator#

Your Ruby client’s list_invoices returns an Array, so the caller who wants the first fifty waits for all nine thousand. The caller who wants to stop at the first unpaid invoice has no way to say so. The alternative you tried, handing callers a cursor and a loop to write, put page arithmetic into every script that calls you.

What you get

You will end up with a list method that returns an Enumerator. Callers chain first, take, lazy.select and each, pages are requested as rows are consumed, and an HTTP failure surfaces where the caller is iterating. This is for you if a Ruby client of yours hands back whole collections as Arrays.

Short answer

Build the list with Enumerator.new and a yielder that requests the next page when the caller pulls past the current one. first(n) stops early, each walks to the end, and an HTTP failure raised inside the yielder surfaces at whichever call is iterating. Put .lazy in front of select and map, or they walk every page before returning. Prefer internal iteration: next runs the block on a Fiber, and an abandoned Fiber never runs its ensure.

You will need

Ruby 3.1 or later, and a list endpoint that returns a cursor. The sample takes the fetch as a callable, so the demos and tests run against an in-memory API with a request log. The same Pager runs unchanged over Faraday against a real endpoint. The class behind all of it is Enumerator, and the property the page depends on is that its block runs only as far as the caller pulls.

Approaches compared

The three differ in who decides when the next request happens. auto_paginate decides before you see a row, and fetches everything. auto_paging_each decides per row inside a block, and the block’s control flow is what you get. The Enumerator hands the decision to whichever Enumerable method the caller chose, which is the property the rest of this page is about.

Yield rows, fetch pages

Two enumerators: one over pages, one over rows. The page one knows about cursors and the row one knows about neither.

  def self.pages(fetch)
    Enumerator.new do |yielder|
      cursor = nil
      seen = {}
      loop do
        raise "pagination repeated the cursor #{cursor.inspect}" if seen.key?(cursor)

        seen[cursor] = true
        page = fetch.call(cursor)
        yielder << page["data"]
        cursor = page["next_cursor"]
        break if cursor.nil? || cursor.empty?
      end
    end
  end

  # One row at a time. Callers see neither pages nor cursors.
  def self.rows(fetch)
    Enumerator.new do |yielder|
      pages(fetch).each { |page| page.each { |row| yielder << row } }
    end
  end

yielder << page["data"] is where the block stops. Nothing after that line runs until the caller asks for another element, so fetch.call for page two happens when a caller reads past row three, and not before. The seen hash is the guard against a server that echoes the cursor it was sent, which turns a walk into a loop with no end. Raising is right: a silent stop looks like the end of the collection.

fetch is a callable rather than a client, and that decision is what makes the rest of the page testable. The demos pass a method on an in-memory API. Production passes a lambda over Faraday.

Put lazy in front of select

first(n) stops iterating after n elements, so it requests only the pages those elements sit on. select does not stop, because it has to see every element to know which ones match, and that is true even when a first(2) follows it. The lazy form changes the order: the filter runs one row at a time and the first(2) at the end decides when to stop.

ruby demo.rb

first(5) from seven rows in pages of three
  rows:     1,2,3,4,5
  requests: 2  GET /items?cursor= | GET /items?cursor=c3
select, then first(2)
  rows:     2,4
  requests: 3  GET /items?cursor= | GET /items?cursor=c3 | GET /items?cursor=c6
lazy.select, then first(2)
  rows:     2,4
  requests: 2  GET /items?cursor= | GET /items?cursor=c3
the third page answers 503
  rows delivered before it: 1,2,3,4,5,6
  raised at the each call:  Pager::FetchError: GET page "c6" answered 503
the server repeats its cursor
  stopped after 2 requests: pagination repeated the cursor "c3"

The second and third blocks return the same two rows. One of them fetched the whole collection to do it. Against seven rows the difference is a request; against nine thousand it is the difference between a script that finishes and one that gets rate limited. Every Enumerator::Lazy method returns another lazy enumerator, so map, reject and take_while chain the same way, and the chain runs when something at the end of it asks.¹

Raise from inside the yielder

The fourth block is an HTTP failure on the third page. The error is raised inside fetch.call, which is inside the block that yields, which is running because a caller called each. So the exception arrives at that each, after the six rows before it were delivered.

  class FetchError < StandardError
    attr_reader :status, :cursor

    def initialize(status, cursor)
      @status = status
      @cursor = cursor
      super("GET page #{cursor.inspect} answered #{status}")
    end
  end

The cursor in the message is the useful part. A caller that rescues FetchError can log which page failed and, if the rows already delivered were written somewhere, resume from that cursor rather than from the start. Faraday’s raise_error middleware turns a 4xx or 5xx into an exception of its own, and the fetcher translates it so the cursor rides along.

def faraday_fetch(base_url, path: "/items", per_page: 100)
  conn = Faraday.new(url: base_url) do |f|
    f.response :raise_error
    f.response :json
  end
  lambda do |cursor|
    params = { limit: per_page }
    params[:cursor] = cursor if cursor
    conn.get(path, params).body
  rescue Faraday::Error => e
    raise Pager::FetchError.new(e.response ? e.response[:status] : "no response", cursor)
  end
end

That lambda is the whole production adapter. Everything else on the page runs against the in-memory fetch and is unchanged when this one is passed instead.

Check it worked

The tests count requests rather than rows, because the number of requests is the property an Array cannot have.

ruby test.rb

ok first(n) requests only the pages it needs
ok nothing is fetched until the first row is pulled
ok each walks every page and visits every row once
ok select walks every page, lazy.select stops at the second match
ok an HTTP failure surfaces at the call site with the cursor in the message
ok a repeated cursor raises instead of looping
ok breaking out of each runs the ensure block, abandoning next does not
7 cases, 0 failures

The second case is the one to keep when you cut the rest: calling Pager.rows makes no request, and the first first makes exactly one.

See what next leaves behind

next is external iteration. The enumerator’s block runs on a Fiber that is suspended between calls, and a caller who stops calling next leaves it suspended. Anything the block was holding, a connection, a file, a transaction, stays held, and an ensure around the loop never runs. The script below opens a connection inside the block and closes it in ensure.

ruby external.rb

first(2), internal iteration:      connection open? false
next twice, then abandoned:        connection open? true
after rewind:                      connection open? true
after the enumerator is collected: connection open? true
next until StopIteration:          connection open? false

first(2) breaks out of the block, the ensure runs, and the connection is closed. Two calls to next followed by nothing leave it open. rewind drops the Fiber without running it to its end, so the connection stays open, and so does collecting the enumerator itself.² The only way out through next is to call it until it raises StopIteration, which means walking every page.

Use internal iteration for anything that holds a resource. When external iteration is the shape you need, keep the resource outside the block and close it yourself.

When it goes wrong

select fetches the whole collection before returning. The chain is eager because .lazy is missing or sits after the select rather than before it. Move .lazy to the front of the chain, and count requests in a test.

The walk repeats rows or never ends. The server returned the cursor it was given, or rows moved between pages because the sort is not stable. The seen guard catches the first. For the second, sort on an immutable key such as the id.

The exception has no page in it. The fetch raised a Faraday error that was passed through untouched, or a plain RuntimeError. Translate at the fetch, as faraday_fetch does, so the cursor is in the message.

Callers get StopIteration from somewhere they did not expect. They called next past the end. loop rescues it, and each never raises it, which is one more reason to hand callers an Enumerator and let them use Enumerable.

When not to do this

Do not return an Enumerator when the caller needs the count first. An Enumerator has no size until it has walked, and size on this one answers nil. Ask the API for a total, or return the first page and its count as a plain object.

Do not hold a database transaction open around a walk. An Enumerator can run for as long as the collection is long, and a lock held for the duration of a remote API walk is a lock nobody planned. Commit per batch.

Do not reach for auto_paginate on a collection you cannot bound. Octokit fetches every page into one Array before returning, and its loop condition also checks the rate limit, so a long collection comes back truncated with no error.³

Do not use auto_paging_each as a general iterator. It is a method on Stripe’s list objects, it walks backward when ending_before is the only filter, and it gives you a block rather than an Enumerable.⁴ Wrap it in an Enumerator if your callers want to chain.

Related how-tos

• Loop over every page of a REST collection

• Paginate an API with PHP generators

Last verified

Verified 2026-09-24 against Ruby 3.3.6 and Faraday 2.14.0. Every output block is what the preceding command printed, against the in-memory API in the sample directory. faraday_fetch.rb was run against a local HTTP server with the same page shape and is not exercised by the output blocks, because the runner has no gems. The Octokit and stripe-ruby behavior was read from the sources of Octokit 10.0.0 and stripe-ruby 19.6.2, linked in the notes.