# How to queue bursty agent traffic behind a rate limiter

> Pace a turn of 30 agent tool calls under a limit of five a second with one p-queue, send the urgent call first, and refuse a runaway loop with a readable error.

Source: https://voxgig.com/howto/queue-bursty-agent-traffic-behind-a-limiter

- Audience: agent-builder
- Level: advanced
- Languages: typescript
- Verified: 2026-11-21
- Published: 2026-11-21

## Short answer

Put one p-queue between the agent's tool handlers and the API, with `intervalCap: 1`, an interval a quarter longer than the limit's spacing, and `strict: true`. Give each call a priority and a deadline passed as the queue's `signal`, because p-queue's `timeout` starts only when a task runs. Refuse calls past a bound with an error the model can read. The SDKs from sdkgen, openapi-fetch and OpenAPI Generator all take that signal.

---
## You will need

Node 22 or later, p-queue 10, and an API with a documented per-second limit. The sdkgen
packages ask for Node 24, so npm warns with `EBADENGINE` on Node 22, and every step here ran on
Node 22 regardless. The document, `api.json`,
describes one operation, `GET /parts/{partId}`. A stand-in on 127.0.0.1 allows five requests in
any one-second period and answers the rest 429 with `Retry-After`, as
[RFC 6585](https://www.rfc-editor.org/rfc/rfc6585.html#section-4) describes.[^1]

## Approaches compared

| Approach | When it fits | What it costs you | When to pick something else |
| --- | --- | --- | --- |
| [bottleneck highWater and strategies](https://github.com/SGrondin/bottleneck#strategies) | A Node limiter that sheds by priority, with `LEAK` dropping the calls that waited longest | `highWater` shed nothing here until `maxConcurrent` held jobs in the queue | You need a deadline on the wait, which `expiration` does not cover |
| [OpenAPI Generator `typescript-fetch`](https://openapi-generator.tech/docs/generators/typescript-fetch/) | A generated API class, with `initOverrides` per call and middleware per client | No limiter, queue or retry in the output, so the queue sits around each call | You want the limit set by configuration in the SDK |
| [openapi-typescript and openapi-fetch](https://openapi-ts.dev/openapi-fetch/api) | A thin typed `fetch` client, with `signal`, `fetch` and `middleware` options per request | Types only from the generator, so pacing, deadlines and shedding are all your code | You want a client that carries its own limiter |
| [p-queue with priority and a deadline](https://github.com/sindresorhus/p-queue) | One process serving many conversations, where the model's next step waits on one call | A queue to size and a deadline to choose, and the model no longer sees the limit | Calls come from several processes sharing one key |
| [return the 429 to the agent](https://www.rfc-editor.org/rfc/rfc9110.html#name-retry-after) | Few calls per turn, and a model that reads `Retry-After` and waits | Every refused call costs a model turn, and a retry can land in the same second | The agent fans out more calls than the limit allows |
| [sdkgen ratelimit feature](https://github.com/voxgig/sdkgen/blob/main/docs/reference/features.md) | Serial calls through one generated client, set by one constructor option | Concurrent calls on one client were not held apart, so a turn of 30 drew 25 refusals | The calls in a turn run concurrently |

Voxgig maintains sdkgen. It is one of six options here, not the recommendation.

The queue keeps the conversation moving, at the price of hiding the limit from the model and
making a long wait look like a hung tool. bottleneck adds shedding strategies but needs
`maxConcurrent` before they act. Both generators leave pacing to you, and both pass a signal
through, so one queue serves either.

## Fire one turn of 30 at a limit of five

`generate.sh` scaffolds the sdkgen project with the `ratelimit` feature and builds the TypeScript
SDK that the lookup tool calls.

```bash title="generate.sh"
cd "$work"
npx --yes @voxgig/create-sdkgen@0.31.9 parts --def api.json --target ts --feature ratelimit
cd parts-sdk/.sdk
npm run generate
```

`turn.mjs` sends the same turn three ways: straight to the SDK, through the SDK with its
`ratelimit` feature on, and through the queue.

```bash
node turn.mjs
```

```text output
30 lookup_part calls in one turn, API limit 5 per second
  scenario                                 ok  429  other errors
  no queue, all 30 at once                 5   25   -
  sdkgen ratelimit { rate: 4, burst: 1 }   5   25   -
  p-queue, one start per 250 ms            29  0    1 queue_deadline
  what the agent reads for a 429 with no queue: {"ok":false,"code":"request_status","message":"PartsSDK: load: request: 429: Too Many Requests","status":429,"retryAfter":"1"}
  first to return: p17, the call queued with priority 1
  p30, deadline 1000 ms: queue_deadline, sent: false
    "lookup_part p30 was not sent: it waited 1000 ms for the rate limit. Call again later."
```

The `ratelimit` feature waits for a token instead of failing, as the
[feature reference](https://github.com/voxgig/sdkgen/blob/main/docs/reference/features.md)
says, but 30 concurrent calls on one client mostly went out together, and 25 came back 429. In the
generated `RatelimitFeature.js`, each call that finds the bucket empty computes the same wait,
sleeps it, and empties the bucket, so the 29 behind the first left together. sdkgen 4.45.0
generated identical source for this document and gave the same 25.

The queue sent 29 calls with no 429. `turn()` pauses the queue while it adds a turn's calls, so
priority orders all of them before the first one starts, and `p17` came back first. `p30` had a
one-second deadline and was still waiting when it passed. The stand-in counts on the real clock,
so the spacing keeps a fifth of headroom. A test repeats the `ratelimit` result on a frozen clock,
where machine speed cannot change it.

## Give each call a deadline the queue enforces

p-queue's README says its `timeout` starts when the task starts running, not when it is added,
so it cannot bound a wait in the queue. A `signal` can: if it aborts while
the task waits, p-queue removes the task and `add()` rejects.

```js title="toolqueue.mjs"
  async function call(conversation, what, fn, { priority = 0, deadline = deadlineMs } = {}) {
    if (queue.size >= maxWaiting) throw new QueueFullError(what, 'queue', maxWaiting)
    if (waiting(conversation) >= maxWaitingPerConversation) {
      throw new QueueFullError(what, 'conversation', maxWaitingPerConversation)
    }

    // Fair share: each call a conversation already has open lowers the rank of
    // its next one, so one conversation's backlog cannot delay another's first call.
    const rank = priority * 1000 - (fairShare ? get(openBy, conversation) : 0)
    // A timer of its own rather than AbortSignal.timeout(), whose timer is unref'd
    // and so lets a process with nothing else to do exit with the call pending.
    const expiry = Number.isFinite(deadline) ? new AbortController() : undefined
    const timer = expiry && setTimeout(() => expiry.abort(new Error('deadline')), deadline)
    let started = false
    add(waitingBy, conversation, 1)
    add(openBy, conversation, 1)
    try {
      return await queue.add(
        ({ signal }) => {
          started = true
          add(waitingBy, conversation, -1)
          return fn({ signal })
        },
        { priority: rank, signal: expiry?.signal },
      )
    } catch (err) {
      if (expiry?.signal.aborted) throw new QueueDeadlineError(what, deadline, started)
      throw err
    } finally {
      clearTimeout(timer)
      if (!started) add(waitingBy, conversation, -1)
      add(openBy, conversation, -1)
    }
  }
```

`sent` in the error is the fact the model needs. A call refused before it started never reached
the API, so calling again is safe. A call that timed out while running may have reached it.

The default of 40 waiting comes from the deadline: the last of 40 starts after about 10 seconds,
inside 12. The rate comes from the limit: one start per 250 ms is four a second, a fifth under
five, which leaves room for the network to bunch requests on the way. `strict: true` makes p-queue
count starts in a sliding window; its fixed windows can let tasks burst at a boundary.

## Shed a runaway loop before it fills memory

`shed.mjs` has one conversation queue 100 lookup calls at once and counts what each queue keeps.

```bash
node shed.mjs
```

```text output
100 lookups queued at once by a loop in one conversation
  queue                                                accepted  refused
  p-queue, no bounds                                   100       0
  p-queue, 30 per conversation, 40 in all              31        69, p32 to p100
  bottleneck highWater 40, minTime 250                 100       0
  bottleneck highWater 40, maxConcurrent 1, OVERFLOW   41        59, p42 to p100
  bottleneck highWater 40, maxConcurrent 1, LEAK       41        59, p2 to p60
what a refused call returns
  p-queue:    QueueFullError: lookup_part p32 was not sent: 30 calls from this conversation are already waiting for the rate limit. Ask for fewer items, or call again after those return.
  bottleneck: BottleneckError: This job has been dropped by Bottleneck
```

With `minTime` alone, bottleneck schedules every job at once and counts none as queued, so
`highWater` never acts. With `maxConcurrent: 1` it sheds 59: `OVERFLOW` refuses the newest and
`LEAK` drops the ones that waited longest. Its error says that a job was dropped, so wrap it
before a model reads it.

## Put the same queue in front of openapi-fetch and OpenAPI Generator

Neither generator writes a limiter. [`openapi-typescript`](https://openapi-ts.dev/introduction)
writes types and nothing else, and `openapi-fetch` takes any `fetch` option on each request,
`signal` among them, plus per-request `fetch` and `middleware` options. OpenAPI Generator's
`typescript-fetch` output takes `initOverrides`, a `RequestInit`, on every method, and its options
page lists no rate or retry option. Each tool call therefore hands its queue signal to the client.

```js title="seams.mjs"
// Each entry turns (base) into a function that loads one part, given a signal.
export const clients = {
  'sdkgen 4.44.3': (base) => {
    const sdk = new PartsSDK({ base, apikey: 'sk-test-1' })
    return async (id, signal) => (await sdk.Part().load({ id }, { signal })).data()
  },
  'openapi-fetch 0.17.0': (base) => {
    const client = createClient({ baseUrl: base, headers: { authorization: 'Bearer sk-test-1' } })
    return async (id, signal) => {
      const { data, error, response } = await client.GET('/parts/{partId}', { params: { path: { partId: id } }, signal })
      if (error !== undefined || !response.ok) throw Object.assign(new Error(`status ${response.status}`), { status: response.status })
      return data
    }
  },
  'OpenAPI Generator 7.27.0': (base) => {
    const api = new DefaultApi(new Configuration({ basePath: base, headers: { authorization: 'Bearer sk-test-1' } }))
    return (id, signal) => api.getPart({ partId: id }, { signal })
  },
```

```bash
node seams.mjs
```

```text output
six lookups and one slow one (p59, deadline 500 ms) through the queue
  sdkgen 4.44.3              ok 6  429s 0  p59: queue_deadline, sent true, stand-in saw the request closed: yes
  openapi-fetch 0.17.0       ok 6  429s 0  p59: queue_deadline, sent true, stand-in saw the request closed: yes
  OpenAPI Generator 7.27.0   ok 6  429s 0  p59: queue_deadline, sent true, stand-in saw the request closed: yes
```

In all three, the deadline reached the socket: the stand-in saw the slow request closed.

## Check it worked

Run the suite. It sends the turn of 30, the runaway loop and the slow call through each client.

```bash
node --test queue.test.mjs
```

```text output
# tests 16
# suites 0
# pass 16
# fail 0
```

Two tests guard the checks themselves. One passes a client that drops the signal and expects the
stand-in to see no close. The other shows that a deadline from `AbortSignal.timeout()` lets Node
exit before the deadline fires.

## When it goes wrong

Another conversation's call waits behind a loop. Without bounds, one conversation's 100 calls sit
ahead of everyone else's. Cap each conversation, and rank each call below the ones its own
conversation already has open.

```bash
node pitfall.mjs
```

```text output
a loop in conversation b queues 100 lookups of p1, then conversation c asks for p42
  no bounds, no deadline, no fair share   b queued 100; c waits behind 99, 24.8 s at one start per 250 ms
  bounds, deadline and fair share         b queued 31, 69 refused; c returned ok, request 2 to reach the API
```

The agent runtime gives up before the queue does. The
[MCP lifecycle](https://modelcontextprotocol.io/specification/2025-06-18/basic/lifecycle#timeouts)
tells a sender to set a timeout on every request and cancel when it passes.[^2] Keep the queue's
deadline under that timeout, so the model reads `queue_deadline` and not a timeout of its own.

The process exits with calls still waiting. A deadline built on `AbortSignal.timeout()` keeps
nothing alive, so a script with no server open ends early. Use your own timer, as `call()` does.

## When not to do this

Do not hide the limit from the model when a turn makes two or three calls. Returning the 429 with
its `Retry-After` is cheaper, and the model can decide to wait.

Do not rely on sdkgen's `ratelimit` feature for a turn whose calls run concurrently. Here 30
concurrent calls on one client drew 25 refusals in 4.44.3 and 4.45.0. It suits serial calls, and
it limits one client instance only.

Do not use a queue in one process when several processes share the key. Each keeps its own count,
and the sum goes over.

Choose openapi-fetch when you want the smallest typed client and already own the pacing code,
since it adds nothing you would need to switch off. OpenAPI Generator fits when the same document
also needs clients in languages beyond TypeScript, where its generators share one command line.

## Related how-tos

- [Throttle outbound API calls with a token bucket](/howto/throttle-outbound-calls-with-a-token-bucket)

- [Read RateLimit headers and slow down before you get a 429](/howto/read-ratelimit-headers-and-pace-before-429)

- [Share one rate limiter across many workers with Redis](/howto/share-a-rate-limiter-across-workers-with-redis)

- [Design error responses an autonomous agent can act on](/howto/error-responses-an-agent-can-act-on)

- [Ship a client-side rate limiter inside an SDK you generate](/howto/ship-a-rate-limiter-in-a-generated-sdk)

## Last verified

Verified 2026-11-21 against Node 22.23.5, `@voxgig/create-sdkgen` 0.31.9, `@voxgig/sdkgen`
4.44.3, `@voxgig/apidef` 8.32.4, p-queue 10.0.0, bottleneck 2.19.5, `openapi-typescript`
7.13.0, `openapi-fetch` 0.17.0, OpenAPI Generator 7.27.0 and TypeScript 5.9.3. Every output block
is what the command preceding it printed. All three generators ran on `api.json`, and their output
is committed; the OpenAPI Generator client runs as JavaScript compiled from its TypeScript.
sdkgen 4.45.0 ran once, on the concurrent turn, and is not committed.

[^1]: [RFC 6585](https://www.rfc-editor.org/rfc/rfc6585.html#section-4) gave 429 its name in April
2012. Its example response carries a `Retry-After` of 3600 and a body, written in the first person,
that allows 50 requests an hour to each logged-in user. The same section says the specification
does not define how the origin server identifies the user, nor how it counts requests, which
leaves the counting to each server.

[^2]: The [MCP lifecycle](https://modelcontextprotocol.io/specification/2025-06-18/basic/lifecycle#timeouts)
says implementations SHOULD establish timeouts for all sent requests and SHOULD always enforce a
maximum timeout, even while progress notifications arrive. It names no number for either, which
leaves the figure to each client and makes the queue's deadline a guess about someone else's.