How-to › Move data in and out

How to backfill a year of API data under a rate limit#

Load a year of history in daily windows, checkpoint each so a rerun resumes, and pace calls to the published limit with a token bucket or the RateLimit field.

Audience
API consumer
Level
intermediate
Topic
Build scheduled pipelines from APIs
Languages
Python
Verified

Your backfill asks for twelve months of orders as fast as the loop runs. The API allows 60 calls a minute per key, so the job spends most of its time collecting 429 responses. When it dies at month seven, the rerun starts again from the first day.

What you get

You will end up with a backfill that walks a year in daily windows, resumes from a checkpoint, and never sends more than the limit in a minute. This is for you if you load history from an API that publishes a quota.

Short answer

Split the year into daily windows, land each one, and write a checkpoint after it so a rerun starts at the next window. Pace calls to the published limit with one token bucket per API key, or read the RateLimit field and wait out its reset, and honor Retry-After on a 429. The sdkgen ratelimit feature paces a single client, so split the budget when several workers share a key.

You will need

Python 3.10 or later, the API’s documented rate limit, and the headers it sends. The sample’s npm ci installs requests, which the generated SDK uses, into pydeps/. The stand-in API allows 60 calls per fixed minute per key, and sends the RateLimit and RateLimit-Policy fields of the IETF draft.1 Every run uses a fake clock, so a year of pacing takes seconds and nothing sleeps.

Approaches compared

ApproachWhen it fitsWhat it costs youWhen to pick something else
Hand-rolled token bucketOne process, any client, and a limit you can read off the docsTwenty lines you own, and a bucket each process keeps to itselfSeveral processes share the key
Kiota retry middlewareYou call the API through a Kiota clientRetries after a 429 but paces nothing, so the budget is learned by hitting itYou want to stay under the limit, not recover
pyrate-limiterWorkers on one host or across hosts need one shared budgetA dependency, and a SQLite or Redis bucket to run for the shared caseA single process, where a small bucket does
sdkgen ratelimit featureYou call the API through an sdkgen SDKA budget per client instance, so two workers each spend the whole quotaWorkers share a key and nothing splits it
Swagger Codegen Python clientYou already ship its generated clientNo pacing in the generated code, and urllib3 under it sleeps out Retry-After inside the callYou want the limiter inside the client

Voxgig maintains sdkgen. It is one of five options here, not the recommendation.

A limiter inside the client keeps pacing out of your loop, and it knows only its own calls. A bucket you own sees what you give it, which can be every worker. pyrate-limiter adds shared backends for the case the other four do not cover.

Split the year into windows and checkpoint each one

backfill() asks for one window at a time, retries it after a 429, lands its rows, and then records the last day done:

def backfill(fetch, first, last, checkpoint, land, pacer=None, stop_after=None):
    """Calls fetch(start, end) -> (status, headers, rows) once per window, retrying 429s."""
    calls = 0
    for n, (start, end) in enumerate(windows(checkpoint.resume_from(first), last), 1):
        while True:
            if pacer:
                pacer.before()
            status, headers, rows = fetch(start, end)
            calls += 1
            if pacer:
                pacer.after(status, headers)
            if status != 429:
                break
        if status != 200:
            raise RuntimeError(f"{start}..{end} answered {status}")
        land(start, rows)
        checkpoint.save(end)
        if stop_after is not None and n >= stop_after:
            raise KeyboardInterrupt(f"stopped after {n} windows")
    return calls

The checkpoint is written with os.replace after the window lands, so a crash between the two repeats that window rather than skipping it. Landing overwrites by window, which makes the repeat harmless. resume.py stops a run after 100 windows and starts it again:

python3 resume.py
first run stopped:   calls 100  checkpoint {'done_through': '2026-02-08'}
second run finished: calls 265  checkpoint {'done_through': '2026-10-31'}
windows landed 365  orders 546

Pace with the SDK, a bucket, or the headers

Generate the Python SDK with the feature, as generate.sh does:

npx --yes @voxgig/create-sdkgen@0.31.9 ledger --def api.json -t py -f ratelimit

The feature takes a rate in tokens per second, a burst, and an injectable clock and wait in milliseconds. The sample’s client factory:

def sdk_for(ledger, clock, key="sk-test-1", rate=None):
    """A generated client. With `rate`, its ratelimit feature paces calls per second."""
    feature = {}
    if rate is not None:
        feature["ratelimit"] = {"active": True, "rate": rate, "burst": 1,
                                "now": clock.now_ms, "sleep": clock.sleep_ms}
    return LedgerSDK({"base": ledger.base, "apikey": key, "feature": feature})

The feature reads no response headers. The header pacer does, and slows down before the 429 arrives:

class HeaderPacer:
    """Slows down before a 429: once `r` reaches `reserve`, waits `t` seconds."""

    def __init__(self, sleep, reserve=0):
        self.sleep, self.reserve = sleep, reserve

    def before(self):
        pass

    def after(self, status, headers):
        # The draft says Retry-After takes precedence over RateLimit.
        if retry_after(status, headers, self.sleep):
            return
        field = ratelimit_field(headers)
        if "r" in field and "t" in field and field["r"] <= self.reserve:
            self.sleep(field["t"])

demo.py runs the year four ways. The peak is the most calls the server admitted in one minute:

python3 demo.py
365 daily windows, 2025-11-01 to 2026-10-31, limit 60 calls a minute per key
  pacing                      calls  429s  peak/min  simulated  orders
  Retry-After only              371     6        60     6m 00s     546
  sdkgen ratelimit, rate=1      365     0        60     6m 04s     546
  token bucket, 60 a minute     365     0        60     6m 04s     546
  RateLimit header pacer        365     0        60     6m 00s     546

Waiting only on Retry-After finishes the year, at the cost of one refused call each minute. Both buckets space calls a second apart. The header pacer runs at full speed until the field says r=0, then waits out t.

Pace the Swagger Codegen client from its headers

Swagger Codegen’s Python client generates no pacing and no retry for this document. The urllib3 PoolManager it builds keeps urllib3’s default Retry(3), though, and that waits out Retry-After on a 429 inside the call. swagger_retry.py makes three calls on a key with two left in the minute:

python3 swagger_retry.py
default Retry: Retry(total=3, connect=None, read=None, redirect=None, status=None) statuses that wait out Retry-After: [413, 429, 503]
  call 1: 200, RateLimit "minute";r=1;t=60, waited 0 s inside the call
  call 2: 200, RateLimit "minute";r=0;t=60, waited 0 s inside the call
  call 3: 200, RateLimit "minute";r=1;t=60, waited 60 s inside the call
  the server refused 1 call; the caller saw no 429

On a real clock that third call blocks for a minute inside the client, where a loop that counts its own 429 responses cannot see it. The _with_http_info twin of each method returns the status and headers beside the data, which is what a header pacer needs:

        if _return_http_data_only:
            return (return_data)
        else:
            return (return_data, response_data.status,
                    response_data.getheaders())

Call api.list_orders_with_http_info(_from, to), note the _from the generator chose because from is a Python keyword, and pass the headers to HeaderPacer.after(). A token bucket goes around the call. Swagger Codegen is the better fit when the client is one of many it generates for your team, because the wrapper is the same in each.

Pace the Kiota client before each request

Kiota’s generated orders.get() holds no HTTP code: it calls send_collection_async on the request adapter. Kiota’s middleware page says the default retry handler retries on a 429 or 503, which recovers from the limit but does not keep you under it. Call TokenBucket.before() ahead of each await, or write a middleware handler that does the same for every request. Kiota suits a backfill that already uses its typed models for the rest of the integration.

Check it worked

check.py counts admitted calls per minute of the fake clock on the server side, and asserts that the peak stays at or under 60:

python3 check.py
ok  365 windows cover every day once
ok  weekly windows end on the last day
ok  sdkgen ratelimit: 365 calls, no 429, peak 60 of 60, every order landed
ok  token bucket: 365 calls, no 429, peak 60 of 60, every order landed
ok  RateLimit header pacer: 365 calls, no 429, peak 60 of 60, every order landed
ok  with no pacing the server refuses calls, and Retry-After still finishes the year
ok  Retry-After takes precedence over the RateLimit field
ok  the RateLimit field parses to r and t
ok  a rerun resumes after the checkpoint: 100 then 265 calls, 546 orders
ok  two clients at rate=1 on one key: 429s
ok  two clients at rate=0.5 on one key: no 429s
ok  the Swagger Codegen and Kiota files name retries or rate limits only in the API description
ok  Swagger Codegen's client: urllib3 waits out Retry-After on a 429, and the caller sees 200

When it goes wrong

Two workers split the windows and share one key. Each has its own sdkgen client at the full rate, so each bucket believes it owns the quota:

python3 parallel.py
each client at rate=1, the full budget
  requests 545, peak received in a minute 90, 429s 180, list() failures 180
each client at rate=0.5, half each
  requests 365, peak received in a minute 60, 429s 0, list() failures 0

The features reference says the same: the budget is per client instance, not distributed. Give each of N workers rate / N, or share one bucket. pyrate-limiter’s documentation lists a SQLiteBucket with use_file_lock=True for many processes on one host and Redis buckets across hosts.

A rerun starts at the first day. The checkpoint was written before the rows landed, or not at all. Write it after land(), with a rename, so the file is either the old value or the new one.

A pacer set to the published limit still draws 429 responses. The server’s minute is fixed and the client’s is not, or the clocks disagree. Leave a reserve, such as HeaderPacer(sleep, reserve=2).

When not to do this

Do not run windows in parallel processes to go faster. The quota is per key, so parallelism buys nothing past the limit and turns pacing into a shared-state problem.

Do not rely on the sdkgen ratelimit feature when more than one process holds the key. It is the simplest option for one client, and it cannot see the others.

Do not backfill through an API that offers a bulk export. One export job beats 365 paced calls.

Pick Swagger Codegen when its clients are already your team’s standard and a wrapper is cheap, and Kiota when its retry handler and typed models already carry the rest of the integration.

Last verified

Verified 2026-11-20 against Python 3.13.18, Node 22.23.5, @voxgig/create-sdkgen 0.31.9, @voxgig/sdkgen 4.44.3, requests 2.34.2, urllib3 2.8.0, Swagger Codegen 3.0.85, and Kiota 1.38.0. Every output block is what the command preceding it printed. The generators ran once, through generate.sh, and their output is committed. The Swagger Codegen client runs in swagger_retry.py and check.py. The Kiota client and pyrate-limiter were not run; what the page says of them comes from their documentation.

Footnotes

  1. The draft is draft-ietf-httpapi-ratelimit-headers-11, dated May 23, 2026, and its first page gives an expiry of November 24, 2026, four days after this page was verified. It says that when a response carries both RateLimit and Retry-After, the latter must take precedence, so the older header outranks the newer one by the newer one’s rule. ↩︎ Back to text

Read this page as markdown · All how-to guides

Generate the client instead of writing it#

Retries, timeouts, pagination and auth are the same problems in every client. Voxgig generates them from your OpenAPI description, in 23 languages, from one model.

Listen to Fireside, the Voxgig podcast

Since 2018, Richard Rodger has talked with developer advocates, event organizers, and the people who build developer-facing companies about the specifics of the work.