The model asks for 30 parts in one turn, and the tool handler sends 30 requests at once. The parts API allows five requests a second and answers the other 25 with 429. The model gets 25 errors back in one turn, and each call it repeats competes with the rest of the burst.
What you get
You will end up with one queue that sends a turn of 30 calls with no 429, the urgent call first, and refuses a runaway loop with an error the model reads. This is for you if your agent’s tools call a rate-limited API.
Short answer
Put one p-queue between the agent’s tool handlers and the API, with intervalCap: 1, an
interval a quarter longer than the limit’s spacing, and strict: true. Give each call a priority
and a deadline passed as the queue’s signal, because p-queue’s timeout starts only when a
task runs. Refuse calls past a bound with an error the model can read. The SDKs from sdkgen,
openapi-fetch and OpenAPI Generator all take that signal.
You will need
Node 22 or later, p-queue 10, and an API with a documented per-second limit. The sdkgen
packages ask for Node 24, so npm warns with EBADENGINE on Node 22, and every step here ran on
Node 22 regardless. The document, api.json,
describes one operation, GET /parts/{partId}. A stand-in on 127.0.0.1 allows five requests in
any one-second period and answers the rest 429 with Retry-After, as
RFC 6585 describes.1
Approaches compared
| Approach | When it fits | What it costs you | When to pick something else |
|---|---|---|---|
| bottleneck highWater and strategies | A Node limiter that sheds by priority, with LEAK dropping the calls that waited longest | highWater shed nothing here until maxConcurrent held jobs in the queue | You need a deadline on the wait, which expiration does not cover |
OpenAPI Generator typescript-fetch | A generated API class, with initOverrides per call and middleware per client | No limiter, queue or retry in the output, so the queue sits around each call | You want the limit set by configuration in the SDK |
| openapi-typescript and openapi-fetch | A thin typed fetch client, with signal, fetch and middleware options per request | Types only from the generator, so pacing, deadlines and shedding are all your code | You want a client that carries its own limiter |
| p-queue with priority and a deadline | One process serving many conversations, where the model’s next step waits on one call | A queue to size and a deadline to choose, and the model no longer sees the limit | Calls come from several processes sharing one key |
| return the 429 to the agent | Few calls per turn, and a model that reads Retry-After and waits | Every refused call costs a model turn, and a retry can land in the same second | The agent fans out more calls than the limit allows |
| sdkgen ratelimit feature | Serial calls through one generated client, set by one constructor option | Concurrent calls on one client were not held apart, so a turn of 30 drew 25 refusals | The calls in a turn run concurrently |
Voxgig maintains sdkgen. It is one of six options here, not the recommendation.
The queue keeps the conversation moving, at the price of hiding the limit from the model and
making a long wait look like a hung tool. bottleneck adds shedding strategies but needs
maxConcurrent before they act. Both generators leave pacing to you, and both pass a signal
through, so one queue serves either.
Fire one turn of 30 at a limit of five
generate.sh scaffolds the sdkgen project with the ratelimit feature and builds the TypeScript
SDK that the lookup tool calls.
cd "$work"
npx --yes @voxgig/create-sdkgen@0.31.9 parts --def api.json --target ts --feature ratelimit
cd parts-sdk/.sdk
npm run generate
turn.mjs sends the same turn three ways: straight to the SDK, through the SDK with its
ratelimit feature on, and through the queue.
node turn.mjs
30 lookup_part calls in one turn, API limit 5 per second
scenario ok 429 other errors
no queue, all 30 at once 5 25 -
sdkgen ratelimit { rate: 4, burst: 1 } 5 25 -
p-queue, one start per 250 ms 29 0 1 queue_deadline
what the agent reads for a 429 with no queue: {"ok":false,"code":"request_status","message":"PartsSDK: load: request: 429: Too Many Requests","status":429,"retryAfter":"1"}
first to return: p17, the call queued with priority 1
p30, deadline 1000 ms: queue_deadline, sent: false
"lookup_part p30 was not sent: it waited 1000 ms for the rate limit. Call again later."
The ratelimit feature waits for a token instead of failing, as the
feature reference
says, but 30 concurrent calls on one client mostly went out together, and 25 came back 429. In the
generated RatelimitFeature.js, each call that finds the bucket empty computes the same wait,
sleeps it, and empties the bucket, so the 29 behind the first left together. sdkgen 4.45.0
generated identical source for this document and gave the same 25.
The queue sent 29 calls with no 429. turn() pauses the queue while it adds a turn’s calls, so
priority orders all of them before the first one starts, and p17 came back first. p30 had a
one-second deadline and was still waiting when it passed. The stand-in counts on the real clock,
so the spacing keeps a fifth of headroom. A test repeats the ratelimit result on a frozen clock,
where machine speed cannot change it.
Give each call a deadline the queue enforces
p-queue’s README says its timeout starts when the task starts running, not when it is added,
so it cannot bound a wait in the queue. A signal can: if it aborts while
the task waits, p-queue removes the task and add() rejects.
async function call(conversation, what, fn, { priority = 0, deadline = deadlineMs } = {}) {
if (queue.size >= maxWaiting) throw new QueueFullError(what, 'queue', maxWaiting)
if (waiting(conversation) >= maxWaitingPerConversation) {
throw new QueueFullError(what, 'conversation', maxWaitingPerConversation)
}
// Fair share: each call a conversation already has open lowers the rank of
// its next one, so one conversation's backlog cannot delay another's first call.
const rank = priority * 1000 - (fairShare ? get(openBy, conversation) : 0)
// A timer of its own rather than AbortSignal.timeout(), whose timer is unref'd
// and so lets a process with nothing else to do exit with the call pending.
const expiry = Number.isFinite(deadline) ? new AbortController() : undefined
const timer = expiry && setTimeout(() => expiry.abort(new Error('deadline')), deadline)
let started = false
add(waitingBy, conversation, 1)
add(openBy, conversation, 1)
try {
return await queue.add(
({ signal }) => {
started = true
add(waitingBy, conversation, -1)
return fn({ signal })
},
{ priority: rank, signal: expiry?.signal },
)
} catch (err) {
if (expiry?.signal.aborted) throw new QueueDeadlineError(what, deadline, started)
throw err
} finally {
clearTimeout(timer)
if (!started) add(waitingBy, conversation, -1)
add(openBy, conversation, -1)
}
}
sent in the error is the fact the model needs. A call refused before it started never reached
the API, so calling again is safe. A call that timed out while running may have reached it.
The default of 40 waiting comes from the deadline: the last of 40 starts after about 10 seconds,
inside 12. The rate comes from the limit: one start per 250 ms is four a second, a fifth under
five, which leaves room for the network to bunch requests on the way. strict: true makes p-queue
count starts in a sliding window; its fixed windows can let tasks burst at a boundary.
Shed a runaway loop before it fills memory
shed.mjs has one conversation queue 100 lookup calls at once and counts what each queue keeps.
node shed.mjs
100 lookups queued at once by a loop in one conversation
queue accepted refused
p-queue, no bounds 100 0
p-queue, 30 per conversation, 40 in all 31 69, p32 to p100
bottleneck highWater 40, minTime 250 100 0
bottleneck highWater 40, maxConcurrent 1, OVERFLOW 41 59, p42 to p100
bottleneck highWater 40, maxConcurrent 1, LEAK 41 59, p2 to p60
what a refused call returns
p-queue: QueueFullError: lookup_part p32 was not sent: 30 calls from this conversation are already waiting for the rate limit. Ask for fewer items, or call again after those return.
bottleneck: BottleneckError: This job has been dropped by Bottleneck
With minTime alone, bottleneck schedules every job at once and counts none as queued, so
highWater never acts. With maxConcurrent: 1 it sheds 59: OVERFLOW refuses the newest and
LEAK drops the ones that waited longest. Its error says that a job was dropped, so wrap it
before a model reads it.
Put the same queue in front of openapi-fetch and OpenAPI Generator
Neither generator writes a limiter. openapi-typescript
writes types and nothing else, and openapi-fetch takes any fetch option on each request,
signal among them, plus per-request fetch and middleware options. OpenAPI Generator’s
typescript-fetch output takes initOverrides, a RequestInit, on every method, and its options
page lists no rate or retry option. Each tool call therefore hands its queue signal to the client.
// Each entry turns (base) into a function that loads one part, given a signal.
export const clients = {
'sdkgen 4.44.3': (base) => {
const sdk = new PartsSDK({ base, apikey: 'sk-test-1' })
return async (id, signal) => (await sdk.Part().load({ id }, { signal })).data()
},
'openapi-fetch 0.17.0': (base) => {
const client = createClient({ baseUrl: base, headers: { authorization: 'Bearer sk-test-1' } })
return async (id, signal) => {
const { data, error, response } = await client.GET('/parts/{partId}', { params: { path: { partId: id } }, signal })
if (error !== undefined || !response.ok) throw Object.assign(new Error(`status ${response.status}`), { status: response.status })
return data
}
},
'OpenAPI Generator 7.27.0': (base) => {
const api = new DefaultApi(new Configuration({ basePath: base, headers: { authorization: 'Bearer sk-test-1' } }))
return (id, signal) => api.getPart({ partId: id }, { signal })
},
node seams.mjs
six lookups and one slow one (p59, deadline 500 ms) through the queue
sdkgen 4.44.3 ok 6 429s 0 p59: queue_deadline, sent true, stand-in saw the request closed: yes
openapi-fetch 0.17.0 ok 6 429s 0 p59: queue_deadline, sent true, stand-in saw the request closed: yes
OpenAPI Generator 7.27.0 ok 6 429s 0 p59: queue_deadline, sent true, stand-in saw the request closed: yes
In all three, the deadline reached the socket: the stand-in saw the slow request closed.
Check it worked
Run the suite. It sends the turn of 30, the runaway loop and the slow call through each client.
node --test queue.test.mjs
# tests 16
# suites 0
# pass 16
# fail 0
Two tests guard the checks themselves. One passes a client that drops the signal and expects the
stand-in to see no close. The other shows that a deadline from AbortSignal.timeout() lets Node
exit before the deadline fires.
When it goes wrong
Another conversation’s call waits behind a loop. Without bounds, one conversation’s 100 calls sit ahead of everyone else’s. Cap each conversation, and rank each call below the ones its own conversation already has open.
node pitfall.mjs
a loop in conversation b queues 100 lookups of p1, then conversation c asks for p42
no bounds, no deadline, no fair share b queued 100; c waits behind 99, 24.8 s at one start per 250 ms
bounds, deadline and fair share b queued 31, 69 refused; c returned ok, request 2 to reach the API
The agent runtime gives up before the queue does. The
MCP lifecycle
tells a sender to set a timeout on every request and cancel when it passes.2 Keep the queue’s
deadline under that timeout, so the model reads queue_deadline and not a timeout of its own.
The process exits with calls still waiting. A deadline built on AbortSignal.timeout() keeps
nothing alive, so a script with no server open ends early. Use your own timer, as call() does.
When not to do this
Do not hide the limit from the model when a turn makes two or three calls. Returning the 429 with
its Retry-After is cheaper, and the model can decide to wait.
Do not rely on sdkgen’s ratelimit feature for a turn whose calls run concurrently. Here 30
concurrent calls on one client drew 25 refusals in 4.44.3 and 4.45.0. It suits serial calls, and
it limits one client instance only.
Do not use a queue in one process when several processes share the key. Each keeps its own count, and the sum goes over.
Choose openapi-fetch when you want the smallest typed client and already own the pacing code, since it adds nothing you would need to switch off. OpenAPI Generator fits when the same document also needs clients in languages beyond TypeScript, where its generators share one command line.
Related how-tos
Last verified
Verified 2026-11-21 against Node 22.23.5, @voxgig/create-sdkgen 0.31.9, @voxgig/sdkgen
4.44.3, @voxgig/apidef 8.32.4, p-queue 10.0.0, bottleneck 2.19.5, openapi-typescript
7.13.0, openapi-fetch 0.17.0, OpenAPI Generator 7.27.0 and TypeScript 5.9.3. Every output block
is what the command preceding it printed. All three generators ran on api.json, and their output
is committed; the OpenAPI Generator client runs as JavaScript compiled from its TypeScript.
sdkgen 4.45.0 ran once, on the concurrent turn, and is not committed.
- Its example response carries a
Retry-Afterof 3600 and a body, written in the first person, that allows 50 requests an hour to each logged-in user. The same section says the specification does not define how the origin server identifies the user, nor how it counts requests, which leaves the counting to each server.
Footnotes
-
RFC 6585 gave 429 its name in April ↩︎ Back to text
-
The MCP lifecycle says implementations SHOULD establish timeouts for all sent requests and SHOULD always enforce a maximum timeout, even while progress notifications arrive. It names no number for either, which leaves the figure to each client and makes the queue’s deadline a guess about someone else’s. ↩︎ Back to text