How-to › Make calls that survive failure

How to throttle outbound API calls with a token bucket#

Keep one process under a documented limit such as 10 requests a second with a client-side token bucket, and check the send log for a second that holds more.

Audience
API consumer
Level
beginner
Topic
Set and respect rate limits
Languages
TypeScript, Go and Python
Verified

A places API documents a limit of 10 requests per second per key. Your import loop sends 15 in its first second, and the API answers the last five with 429. The client’s retries then send those five again into the same second. The limiter you added lives inside a client you build per request, so it never holds anything back.

What you get

You will end up with one limiter per process in front of every call, and a send log that shows no second over the limit. This is for you if you call an API with a published per-second limit from one process.

Short answer

Put one token bucket in front of every call to the API, set just under the documented limit with a burst of one, and share it across the process. sdkgen’s ratelimit feature takes rate and burst in the constructor but lets concurrent calls through together, so wrap system.fetch with bottleneck for those. In Go, call Limiter.Wait from an oapi-codegen request editor. Then log send times and count the busiest second.

You will need

Node 22 or later, Go 1.24 or later for the Go section, and an API with a documented per-second limit. npm prints EBADENGINE warnings when it installs sdkgen on Node 22, because its packages declare Node 24, and the scaffold and the generation for this page still completed. The document is api.json, a places API, titled Geocode, with one operation, GET /places. A stand-in on 127.0.0.1 answers it and records every request without refusing any, so each row sends all 15 calls.

Approaches compared

ApproachWhen it fitsWhat it costs youWhen to pick something else
a fixed sleep between callsOne serial loop and a limit well beyond what you needThe sleep adds to every call’s latency, and concurrent callers ignore itCalls come from more than one place in the code
bottleneck minTime and maxConcurrentNode code with concurrent callers, wrapped around one function every call passes throughA dependency last released in 2019, and gaps that can run several milliseconds short of minTimeYou need the limit in more than one language
Fern custom fetcherA Fern TypeScript SDK, where allowCustomFetcher in sdk-config.yml adds a fetcher that wraps every requestNo limiter of its own, so you bring one, and generating needs a Fern login or DockerYou cannot change the SDK’s generator configuration
oapi-codegen request editor with x/time/rateA Go client, where an editor runs before each request and can blockYou wire the limiter into every client, and its long-run rate allows a short burstThe client is not Go
Python ratelimit and pyrate-limiterPython code, with a decorator or a limiter object around the callA decorator covers one function, so every other call path needs its ownThe calls go through a generated client you cannot decorate
sdkgen ratelimit featureSerial calls through one generated client, with the same option block in each targetConcurrent calls on one client pass together, and the default burst equals the rateCalls overlap, or several clients share one key

Voxgig maintains sdkgen. The table holds six options and recommends none of them.

The sdkgen feature costs one line in the constructor and holds serial calls, but only those. oapi-codegen generates no limiter, and the editor it does generate gives x/time/rate a place that every request passes. Fern documents no limiter either, so its fetcher is where bottleneck would go.

Turn on the ratelimit feature with a burst of one

generate.sh scaffolds the project with the feature and generates the TypeScript SDK.

cp api.json "$work/"
cd "$work"
npx --yes @voxgig/create-sdkgen@0.31.9 geocode --def api.json --target ts --feature ratelimit
cd geocode-sdk/.sdk
npm run generate

The feature is off until the constructor activates it. The feature reference gives rate in tokens per second and a burst that defaults to rate, and calls it a budget per client instance.

export const limitedClient = (base, fetch, ratelimit) => new GeocodeSDK({
  base,
  system: { fetch },
  feature: { ratelimit: { active: true, ...ratelimit } },
})

The fetch passed in system records the moment each request leaves, after the limiter has let it go. throttle.mjs makes 15 calls per row and counts the most sends in any one-second window.

node throttle.mjs
15 calls to GET /places, documented limit 10 per second
  scenario                                   sent  busiest second
  no limiter                                 15    15, over the limit
  sdkgen ratelimit { rate: 9 }               15    15, over the limit
  sdkgen ratelimit { rate: 9, burst: 1 }     15    10 or fewer
    the same, a client per call              15    15, over the limit
    the same, 15 calls at once               15    15, over the limit
  bottleneck { minTime: 110 }, 15 at once    15    10 or fewer
  a fixed 110 ms sleep between calls         15    10 or fewer

rate: 9 alone starts with nine tokens, sends them at once, then refills, so its first second held all 15. A limit stated per second is a window, and a bucket as deep as the rate can put up to twice the rate into one. The test replays rate: 10 on a clock it owns, through the feature’s now and sleep options, and counts 19 in one second.

Set burst: 1, and set the rate below the limit. At rate: 10 the bucket spaces serial calls exactly 100 ms apart, which leaves no room for timer jitter, and sends here came as close as 96 ms. Every paced row runs at nine a second, so each verdict has about 100 ms of headroom.

Hold concurrent calls with bottleneck in system.fetch

Fifteen calls started together on one limited client all went out in the same second. In 4.44.3 the feature checks the bucket, sees it empty, and every waiting call sleeps for the same one token, then proceeds. The Go and Python targets have the same acquire step and no lock. In a scratch run, 15 goroutines on one Go client went out within 102 ms, and go test -race reported a data race in the feature. Fifteen Python threads went out in clumps, and one run of three held 11 in a second. sdkgen 4.45.0 generated identical source for all three targets and sent the 15 TypeScript calls within 100 ms.

The SDK takes its transport in system.fetch, so a limiter that queues goes there.

  'bottleneck { minTime: 110 }, 15 at once': async (base) => {
    const limiter = new Bottleneck({ maxConcurrent: 1, minTime: 110 })
    const log = sendLog()
    const client = new GeocodeSDK({ base, system: { fetch: limiter.wrap(log.fetch) } })
    await Promise.all(Array.from({ length: CALLS }, () => client.Place().list()))
    return log.sends
  },

minTime: 110 is nine a second, not 10. Bottleneck’s own default is 0, so 110 is this page’s choice. With minTime: 100, the shortest gap in each of three scratch runs here was under 100 ms, 97.2 ms at the lowest, so 11 calls fitted into one second.

Wait on x/time/rate in an oapi-codegen request editor

oapi-codegen writes a Go client with no limiter. Each method takes reqEditors, and WithRequestEditorFn adds one for every call; an editor that returns after Limiter.Wait holds the request until a token is free.

// waitFor returns a request editor that blocks until the limiter allows a request.
func waitFor(l *rate.Limiter) geocode.RequestEditorFn {
	return func(ctx context.Context, req *http.Request) error {
		return l.Wait(ctx)
	}
}
cd go && go run .
15 calls to GET /places through the oapi-codegen client, limit 10 per second
  scenario                                     sent  busiest second
  rate.NewLimiter(10, 10), one client          15    15, over the limit
  rate.Every(110ms), burst 1, one client       15    10 or fewer
    the same, 15 goroutines at once            15    10 or fewer
    the same, a client and limiter per call    15    15, over the limit
rate.NewLimiter(10, 1) on a controlled clock, the second request let go 30 ms late
  sends at ms: 0 130 200 300 400 500 600 700 800 900 1000 1100
  busiest second: 11

The package documentation says a limiter is safe for simultaneous use by multiple goroutines, and 15 at once stayed under the limit. The last block replays NewLimiter(10, 1) on a clock the program sets. The second request went 30 ms late, and the third went 70 ms after it. The limiter booked the third token from the second one’s reserved time, so the one-second window from 130 ms held 11.1 The paced rows use rate.Every(110ms) for that reason.

Check it worked

The test runs each row against a fresh stand-in and asserts on the busiest second of the send log.

test('sdkgen ratelimit at rate 9, burst 1, holds serial calls under the limit', async () => {
  const r = await measure('sdkgen ratelimit { rate: 9, burst: 1 }')
  assert.equal(r.sent, CALLS)
  assert.equal(r.arrived, CALLS)
  assert.ok(r.most <= LIMIT, `busiest second held ${r.most}`)
})
node --test throttle.test.mjs
1..11
# tests 11
# suites 0
# pass 11
# fail 0
# cancelled 0
# skipped 0
# todo 0
# duration_ms 5236.241507

# pass 11 holds the verdicts in the demo output, including the concurrent row going over, and the two counts from the controlled clock. go test ./... in go/ asserts the Go rows and the 70 ms gap.

When it goes wrong

The busiest second holds 15 although each client has a limiter. A client built per request, or per worker, brings a full bucket with it, so the rate multiplies by the number of clients. Build one client or one limiter for the process, and pass it to every caller.

The API answers 429 although the send log shows no second over 10. The server counts arrivals, and network delay can close a gap between two sends. Leave headroom below the documented limit, as the paced rows do, and read the API’s own headers where it sends them.

A Fern TypeScript client goes over the limit after a 429. Fern’s retries page says its SDKs retry 429 up to two times by default, and a retry is a request. Fern’s documentation does not say whether those retries pass back through a custom fetcher, so set maxRetries: 0 and retry inside the fetcher, behind the limiter. Fern’s pricing page puts TypeScript and Python SDKs on the Free plan.

When not to do this

Do not rely on sdkgen’s ratelimit feature for calls that overlap. It holds serial calls on one client at burst: 1, and it let 15 concurrent calls through within about 100 ms. Put bottleneck in system.fetch instead, or keep the calls serial.

Do not add a client-side bucket when several processes share the key. Each process gets its own bucket, and the sum goes over. A shared limiter in Redis is the answer there.

Pick oapi-codegen with x/time/rate when the caller is Go and calls run in goroutines, because that limiter is safe for concurrent use and one editor covers every method. Pick Fern when a hosted pipeline owns your SDKs and the team already has a limiter to put in its fetcher.

Last verified

Verified 2026-11-21 against Node 22.23.5, Go 1.24.7, @voxgig/create-sdkgen 0.31.9, @voxgig/sdkgen 4.44.3, @voxgig/apidef 8.32.4, bottleneck 2.19.5, oapi-codegen v2.8.2 and golang.org/x/time v0.14.0. Every output block is what the command preceding it printed. sdkgen and oapi-codegen ran on api.json, and their output is committed. The Go and Python targets of the same project, and sdkgen 4.45.0, ran once on the concurrent case and are not committed. The gap figures in prose come from scratch runs on this machine. Fern was not run, and its fetcher option and the plan each language needs are read from its documentation and its pricing page.

Footnotes

  1. The package documentation for x/time/rate states its guarantee in one qualified sentence: informally, in any large enough time interval, the limiter limits the rate to r tokens per second. The qualifier matters, because a late wake is repaid out of the next token’s wait, and the repayment can fall inside the same second as the late request. ↩︎ Back to text

Read this page as markdown · All how-to guides

Generate the client instead of writing it#

Retries, timeouts, pagination and auth are the same problems in every client. Voxgig generates them from your OpenAPI description, in 23 languages, from one model.

Listen to Fireside, the Voxgig podcast

Since 2018, Richard Rodger has talked with developer advocates, event organizers, and the people who build developer-facing companies about the specifics of the work.