How-to › Make calls that survive failure

How to queue bursty agent traffic behind a rate limiter#

Pace a turn of 30 agent tool calls under a limit of five a second with one p-queue, send the urgent call first, and refuse a runaway loop with a readable error.

Audience
Agent builder
Level
advanced
Topic
Set and respect rate limits
Languages
TypeScript
Verified

The model asks for 30 parts in one turn, and the tool handler sends 30 requests at once. The parts API allows five requests a second and answers the other 25 with 429. The model gets 25 errors back in one turn, and each call it repeats competes with the rest of the burst.

What you get

You will end up with one queue that sends a turn of 30 calls with no 429, the urgent call first, and refuses a runaway loop with an error the model reads. This is for you if your agent’s tools call a rate-limited API.

Short answer

Put one p-queue between the agent’s tool handlers and the API, with intervalCap: 1, an interval a quarter longer than the limit’s spacing, and strict: true. Give each call a priority and a deadline passed as the queue’s signal, because p-queue’s timeout starts only when a task runs. Refuse calls past a bound with an error the model can read. The SDKs from sdkgen, openapi-fetch and OpenAPI Generator all take that signal.

You will need

Node 22 or later, p-queue 10, and an API with a documented per-second limit. The sdkgen packages ask for Node 24, so npm warns with EBADENGINE on Node 22, and every step here ran on Node 22 regardless. The document, api.json, describes one operation, GET /parts/{partId}. A stand-in on 127.0.0.1 allows five requests in any one-second period and answers the rest 429 with Retry-After, as RFC 6585 describes.1

Approaches compared

ApproachWhen it fitsWhat it costs youWhen to pick something else
bottleneck highWater and strategiesA Node limiter that sheds by priority, with LEAK dropping the calls that waited longesthighWater shed nothing here until maxConcurrent held jobs in the queueYou need a deadline on the wait, which expiration does not cover
OpenAPI Generator typescript-fetchA generated API class, with initOverrides per call and middleware per clientNo limiter, queue or retry in the output, so the queue sits around each callYou want the limit set by configuration in the SDK
openapi-typescript and openapi-fetchA thin typed fetch client, with signal, fetch and middleware options per requestTypes only from the generator, so pacing, deadlines and shedding are all your codeYou want a client that carries its own limiter
p-queue with priority and a deadlineOne process serving many conversations, where the model’s next step waits on one callA queue to size and a deadline to choose, and the model no longer sees the limitCalls come from several processes sharing one key
return the 429 to the agentFew calls per turn, and a model that reads Retry-After and waitsEvery refused call costs a model turn, and a retry can land in the same secondThe agent fans out more calls than the limit allows
sdkgen ratelimit featureSerial calls through one generated client, set by one constructor optionConcurrent calls on one client were not held apart, so a turn of 30 drew 25 refusalsThe calls in a turn run concurrently

Voxgig maintains sdkgen. It is one of six options here, not the recommendation.

The queue keeps the conversation moving, at the price of hiding the limit from the model and making a long wait look like a hung tool. bottleneck adds shedding strategies but needs maxConcurrent before they act. Both generators leave pacing to you, and both pass a signal through, so one queue serves either.

Fire one turn of 30 at a limit of five

generate.sh scaffolds the sdkgen project with the ratelimit feature and builds the TypeScript SDK that the lookup tool calls.

cd "$work"
npx --yes @voxgig/create-sdkgen@0.31.9 parts --def api.json --target ts --feature ratelimit
cd parts-sdk/.sdk
npm run generate

turn.mjs sends the same turn three ways: straight to the SDK, through the SDK with its ratelimit feature on, and through the queue.

node turn.mjs
30 lookup_part calls in one turn, API limit 5 per second
  scenario                                 ok  429  other errors
  no queue, all 30 at once                 5   25   -
  sdkgen ratelimit { rate: 4, burst: 1 }   5   25   -
  p-queue, one start per 250 ms            29  0    1 queue_deadline
  what the agent reads for a 429 with no queue: {"ok":false,"code":"request_status","message":"PartsSDK: load: request: 429: Too Many Requests","status":429,"retryAfter":"1"}
  first to return: p17, the call queued with priority 1
  p30, deadline 1000 ms: queue_deadline, sent: false
    "lookup_part p30 was not sent: it waited 1000 ms for the rate limit. Call again later."

The ratelimit feature waits for a token instead of failing, as the feature reference says, but 30 concurrent calls on one client mostly went out together, and 25 came back 429. In the generated RatelimitFeature.js, each call that finds the bucket empty computes the same wait, sleeps it, and empties the bucket, so the 29 behind the first left together. sdkgen 4.45.0 generated identical source for this document and gave the same 25.

The queue sent 29 calls with no 429. turn() pauses the queue while it adds a turn’s calls, so priority orders all of them before the first one starts, and p17 came back first. p30 had a one-second deadline and was still waiting when it passed. The stand-in counts on the real clock, so the spacing keeps a fifth of headroom. A test repeats the ratelimit result on a frozen clock, where machine speed cannot change it.

Give each call a deadline the queue enforces

p-queue’s README says its timeout starts when the task starts running, not when it is added, so it cannot bound a wait in the queue. A signal can: if it aborts while the task waits, p-queue removes the task and add() rejects.

  async function call(conversation, what, fn, { priority = 0, deadline = deadlineMs } = {}) {
    if (queue.size >= maxWaiting) throw new QueueFullError(what, 'queue', maxWaiting)
    if (waiting(conversation) >= maxWaitingPerConversation) {
      throw new QueueFullError(what, 'conversation', maxWaitingPerConversation)
    }

    // Fair share: each call a conversation already has open lowers the rank of
    // its next one, so one conversation's backlog cannot delay another's first call.
    const rank = priority * 1000 - (fairShare ? get(openBy, conversation) : 0)
    // A timer of its own rather than AbortSignal.timeout(), whose timer is unref'd
    // and so lets a process with nothing else to do exit with the call pending.
    const expiry = Number.isFinite(deadline) ? new AbortController() : undefined
    const timer = expiry && setTimeout(() => expiry.abort(new Error('deadline')), deadline)
    let started = false
    add(waitingBy, conversation, 1)
    add(openBy, conversation, 1)
    try {
      return await queue.add(
        ({ signal }) => {
          started = true
          add(waitingBy, conversation, -1)
          return fn({ signal })
        },
        { priority: rank, signal: expiry?.signal },
      )
    } catch (err) {
      if (expiry?.signal.aborted) throw new QueueDeadlineError(what, deadline, started)
      throw err
    } finally {
      clearTimeout(timer)
      if (!started) add(waitingBy, conversation, -1)
      add(openBy, conversation, -1)
    }
  }

sent in the error is the fact the model needs. A call refused before it started never reached the API, so calling again is safe. A call that timed out while running may have reached it.

The default of 40 waiting comes from the deadline: the last of 40 starts after about 10 seconds, inside 12. The rate comes from the limit: one start per 250 ms is four a second, a fifth under five, which leaves room for the network to bunch requests on the way. strict: true makes p-queue count starts in a sliding window; its fixed windows can let tasks burst at a boundary.

Shed a runaway loop before it fills memory

shed.mjs has one conversation queue 100 lookup calls at once and counts what each queue keeps.

node shed.mjs
100 lookups queued at once by a loop in one conversation
  queue                                                accepted  refused
  p-queue, no bounds                                   100       0
  p-queue, 30 per conversation, 40 in all              31        69, p32 to p100
  bottleneck highWater 40, minTime 250                 100       0
  bottleneck highWater 40, maxConcurrent 1, OVERFLOW   41        59, p42 to p100
  bottleneck highWater 40, maxConcurrent 1, LEAK       41        59, p2 to p60
what a refused call returns
  p-queue:    QueueFullError: lookup_part p32 was not sent: 30 calls from this conversation are already waiting for the rate limit. Ask for fewer items, or call again after those return.
  bottleneck: BottleneckError: This job has been dropped by Bottleneck

With minTime alone, bottleneck schedules every job at once and counts none as queued, so highWater never acts. With maxConcurrent: 1 it sheds 59: OVERFLOW refuses the newest and LEAK drops the ones that waited longest. Its error says that a job was dropped, so wrap it before a model reads it.

Put the same queue in front of openapi-fetch and OpenAPI Generator

Neither generator writes a limiter. openapi-typescript writes types and nothing else, and openapi-fetch takes any fetch option on each request, signal among them, plus per-request fetch and middleware options. OpenAPI Generator’s typescript-fetch output takes initOverrides, a RequestInit, on every method, and its options page lists no rate or retry option. Each tool call therefore hands its queue signal to the client.

// Each entry turns (base) into a function that loads one part, given a signal.
export const clients = {
  'sdkgen 4.44.3': (base) => {
    const sdk = new PartsSDK({ base, apikey: 'sk-test-1' })
    return async (id, signal) => (await sdk.Part().load({ id }, { signal })).data()
  },
  'openapi-fetch 0.17.0': (base) => {
    const client = createClient({ baseUrl: base, headers: { authorization: 'Bearer sk-test-1' } })
    return async (id, signal) => {
      const { data, error, response } = await client.GET('/parts/{partId}', { params: { path: { partId: id } }, signal })
      if (error !== undefined || !response.ok) throw Object.assign(new Error(`status ${response.status}`), { status: response.status })
      return data
    }
  },
  'OpenAPI Generator 7.27.0': (base) => {
    const api = new DefaultApi(new Configuration({ basePath: base, headers: { authorization: 'Bearer sk-test-1' } }))
    return (id, signal) => api.getPart({ partId: id }, { signal })
  },
node seams.mjs
six lookups and one slow one (p59, deadline 500 ms) through the queue
  sdkgen 4.44.3              ok 6  429s 0  p59: queue_deadline, sent true, stand-in saw the request closed: yes
  openapi-fetch 0.17.0       ok 6  429s 0  p59: queue_deadline, sent true, stand-in saw the request closed: yes
  OpenAPI Generator 7.27.0   ok 6  429s 0  p59: queue_deadline, sent true, stand-in saw the request closed: yes

In all three, the deadline reached the socket: the stand-in saw the slow request closed.

Check it worked

Run the suite. It sends the turn of 30, the runaway loop and the slow call through each client.

node --test queue.test.mjs
# tests 16
# suites 0
# pass 16
# fail 0

Two tests guard the checks themselves. One passes a client that drops the signal and expects the stand-in to see no close. The other shows that a deadline from AbortSignal.timeout() lets Node exit before the deadline fires.

When it goes wrong

Another conversation’s call waits behind a loop. Without bounds, one conversation’s 100 calls sit ahead of everyone else’s. Cap each conversation, and rank each call below the ones its own conversation already has open.

node pitfall.mjs
a loop in conversation b queues 100 lookups of p1, then conversation c asks for p42
  no bounds, no deadline, no fair share   b queued 100; c waits behind 99, 24.8 s at one start per 250 ms
  bounds, deadline and fair share         b queued 31, 69 refused; c returned ok, request 2 to reach the API

The agent runtime gives up before the queue does. The MCP lifecycle tells a sender to set a timeout on every request and cancel when it passes.2 Keep the queue’s deadline under that timeout, so the model reads queue_deadline and not a timeout of its own.

The process exits with calls still waiting. A deadline built on AbortSignal.timeout() keeps nothing alive, so a script with no server open ends early. Use your own timer, as call() does.

When not to do this

Do not hide the limit from the model when a turn makes two or three calls. Returning the 429 with its Retry-After is cheaper, and the model can decide to wait.

Do not rely on sdkgen’s ratelimit feature for a turn whose calls run concurrently. Here 30 concurrent calls on one client drew 25 refusals in 4.44.3 and 4.45.0. It suits serial calls, and it limits one client instance only.

Do not use a queue in one process when several processes share the key. Each keeps its own count, and the sum goes over.

Choose openapi-fetch when you want the smallest typed client and already own the pacing code, since it adds nothing you would need to switch off. OpenAPI Generator fits when the same document also needs clients in languages beyond TypeScript, where its generators share one command line.

Last verified

Verified 2026-11-21 against Node 22.23.5, @voxgig/create-sdkgen 0.31.9, @voxgig/sdkgen 4.44.3, @voxgig/apidef 8.32.4, p-queue 10.0.0, bottleneck 2.19.5, openapi-typescript 7.13.0, openapi-fetch 0.17.0, OpenAPI Generator 7.27.0 and TypeScript 5.9.3. Every output block is what the command preceding it printed. All three generators ran on api.json, and their output is committed; the OpenAPI Generator client runs as JavaScript compiled from its TypeScript. sdkgen 4.45.0 ran once, on the concurrent turn, and is not committed.

  1. Its example response carries a Retry-After of 3600 and a body, written in the first person, that allows 50 requests an hour to each logged-in user. The same section says the specification does not define how the origin server identifies the user, nor how it counts requests, which leaves the counting to each server.

Footnotes

  1. RFC 6585 gave 429 its name in April ↩︎ Back to text

  2. The MCP lifecycle says implementations SHOULD establish timeouts for all sent requests and SHOULD always enforce a maximum timeout, even while progress notifications arrive. It names no number for either, which leaves the figure to each client and makes the queue’s deadline a guess about someone else’s. ↩︎ Back to text

Read this page as markdown · All how-to guides

Generate the client instead of writing it#

Retries, timeouts, pagination and auth are the same problems in every client. Voxgig generates them from your OpenAPI description, in 23 languages, from one model.

Listen to Fireside, the Voxgig podcast

Since 2018, Richard Rodger has talked with developer advocates, event organizers, and the people who build developer-facing companies about the specifics of the work.