# How to backfill a year of API data under a rate limit

> Load a year of history in daily windows, checkpoint each so a rerun resumes, and pace calls to the published limit with a token bucket or the RateLimit field.

Source: https://voxgig.com/howto/backfill-a-year-under-a-rate-limit

- Audience: api-consumer
- Level: intermediate
- Languages: python
- Verified: 2026-11-20
- Published: 2026-11-20

## Short answer

Split the year into daily windows, land each one, and write a checkpoint after it so a rerun starts at the next window. Pace calls to the published limit with one token bucket per API key, or read the RateLimit field and wait out its reset, and honor Retry-After on a 429. The sdkgen ratelimit feature paces a single client, so split the budget when several workers share a key.

---
## You will need

Python 3.10 or later, the API's documented rate limit, and the headers it sends. The sample's
`npm ci` installs `requests`, which the generated SDK uses, into `pydeps/`. The stand-in API
allows 60 calls per fixed minute per key, and sends the `RateLimit` and `RateLimit-Policy` fields
of the [IETF draft](https://datatracker.ietf.org/doc/draft-ietf-httpapi-ratelimit-headers/).[^1]
Every run uses a fake clock, so a year of pacing takes seconds and nothing sleeps.

## Approaches compared

| Approach | When it fits | What it costs you | When to pick something else |
| --- | --- | --- | --- |
| [Hand-rolled token bucket](https://docs.python.org/3/library/time.html#time.sleep) | One process, any client, and a limit you can read off the docs | Twenty lines you own, and a bucket each process keeps to itself | Several processes share the key |
| [Kiota retry middleware](https://learn.microsoft.com/en-us/openapi/kiota/middleware) | You call the API through a Kiota client | Retries after a 429 but paces nothing, so the budget is learned by hitting it | You want to stay under the limit, not recover |
| [pyrate-limiter](https://pyratelimiter.readthedocs.io/en/latest/) | Workers on one host or across hosts need one shared budget | A dependency, and a SQLite or Redis bucket to run for the shared case | A single process, where a small bucket does |
| [sdkgen ratelimit feature](https://github.com/voxgig/sdkgen/blob/main/docs/reference/features.md) | You call the API through an sdkgen SDK | A budget per client instance, so two workers each spend the whole quota | Workers share a key and nothing splits it |
| [Swagger Codegen Python client](https://github.com/swagger-api/swagger-codegen) | You already ship its generated client | No pacing in the generated code, and urllib3 under it sleeps out `Retry-After` inside the call | You want the limiter inside the client |

Voxgig maintains sdkgen. It is one of five options here, not the recommendation.

A limiter inside the client keeps pacing out of your loop, and it knows only its own calls. A
bucket you own sees what you give it, which can be every worker. pyrate-limiter adds shared
backends for the case the other four do not cover.

## Split the year into windows and checkpoint each one

`backfill()` asks for one window at a time, retries it after a 429, lands its rows, and then
records the last day done:

```python title="backfill.py"
def backfill(fetch, first, last, checkpoint, land, pacer=None, stop_after=None):
    """Calls fetch(start, end) -> (status, headers, rows) once per window, retrying 429s."""
    calls = 0
    for n, (start, end) in enumerate(windows(checkpoint.resume_from(first), last), 1):
        while True:
            if pacer:
                pacer.before()
            status, headers, rows = fetch(start, end)
            calls += 1
            if pacer:
                pacer.after(status, headers)
            if status != 429:
                break
        if status != 200:
            raise RuntimeError(f"{start}..{end} answered {status}")
        land(start, rows)
        checkpoint.save(end)
        if stop_after is not None and n >= stop_after:
            raise KeyboardInterrupt(f"stopped after {n} windows")
    return calls
```

The checkpoint is written with [`os.replace`](https://docs.python.org/3/library/os.html#os.replace)
after the window lands, so a crash between the two repeats that window rather than skipping it.
Landing overwrites by window, which makes the repeat harmless. `resume.py` stops a run after 100
windows and starts it again:

```bash
python3 resume.py
```

```text output
first run stopped:   calls 100  checkpoint {'done_through': '2026-02-08'}
second run finished: calls 265  checkpoint {'done_through': '2026-10-31'}
windows landed 365  orders 546
```

## Pace with the SDK, a bucket, or the headers

Generate the Python SDK with the feature, as `generate.sh` does:

```bash title="generate.sh"
npx --yes @voxgig/create-sdkgen@0.31.9 ledger --def api.json -t py -f ratelimit
```

The feature takes a rate in tokens per second, a burst, and an injectable clock and wait in
milliseconds. The sample's client factory:

```python title="runs.py"
def sdk_for(ledger, clock, key="sk-test-1", rate=None):
    """A generated client. With `rate`, its ratelimit feature paces calls per second."""
    feature = {}
    if rate is not None:
        feature["ratelimit"] = {"active": True, "rate": rate, "burst": 1,
                                "now": clock.now_ms, "sleep": clock.sleep_ms}
    return LedgerSDK({"base": ledger.base, "apikey": key, "feature": feature})
```

The feature reads no response headers. The header pacer does, and slows down before the 429
arrives:

```python title="backfill.py"
class HeaderPacer:
    """Slows down before a 429: once `r` reaches `reserve`, waits `t` seconds."""

    def __init__(self, sleep, reserve=0):
        self.sleep, self.reserve = sleep, reserve

    def before(self):
        pass

    def after(self, status, headers):
        # The draft says Retry-After takes precedence over RateLimit.
        if retry_after(status, headers, self.sleep):
            return
        field = ratelimit_field(headers)
        if "r" in field and "t" in field and field["r"] <= self.reserve:
            self.sleep(field["t"])
```

`demo.py` runs the year four ways. The peak is the most calls the server admitted in one minute:

```bash
python3 demo.py
```

```text output
365 daily windows, 2025-11-01 to 2026-10-31, limit 60 calls a minute per key
  pacing                      calls  429s  peak/min  simulated  orders
  Retry-After only              371     6        60     6m 00s     546
  sdkgen ratelimit, rate=1      365     0        60     6m 04s     546
  token bucket, 60 a minute     365     0        60     6m 04s     546
  RateLimit header pacer        365     0        60     6m 00s     546
```

Waiting only on [`Retry-After`](https://www.rfc-editor.org/rfc/rfc9110.html#section-10.2.3)
finishes the year, at the cost of one refused call each minute. Both buckets space calls a
second apart. The header pacer runs at full speed until the field says `r=0`, then waits out `t`.

## Pace the Swagger Codegen client from its headers

Swagger Codegen's Python client generates no pacing and no retry for this document. The urllib3
`PoolManager` it builds keeps urllib3's default `Retry(3)`, though, and that waits out
`Retry-After` on a 429 inside the call. `swagger_retry.py` makes three calls on a key with two
left in the minute:

```bash
python3 swagger_retry.py
```

```text output
default Retry: Retry(total=3, connect=None, read=None, redirect=None, status=None) statuses that wait out Retry-After: [413, 429, 503]
  call 1: 200, RateLimit "minute";r=1;t=60, waited 0 s inside the call
  call 2: 200, RateLimit "minute";r=0;t=60, waited 0 s inside the call
  call 3: 200, RateLimit "minute";r=1;t=60, waited 60 s inside the call
  the server refused 1 call; the caller saw no 429
```

On a real clock that third call blocks for a minute inside the client, where a loop that counts
its own 429 responses cannot see it. The `_with_http_info` twin of each method returns the status and headers beside the
data, which is what a header pacer needs:

```python title="other/swagger_client/api_client.py"
        if _return_http_data_only:
            return (return_data)
        else:
            return (return_data, response_data.status,
                    response_data.getheaders())
```

Call `api.list_orders_with_http_info(_from, to)`, note the `_from` the generator chose because
`from` is a Python keyword, and pass the headers to `HeaderPacer.after()`. A token bucket goes
around the call. Swagger Codegen is the better fit when the client is one of many it generates
for your team, because the wrapper is the same in each.

## Pace the Kiota client before each request

Kiota's generated `orders.get()` holds no HTTP code: it calls `send_collection_async` on the
request adapter. Kiota's [middleware page](https://learn.microsoft.com/en-us/openapi/kiota/middleware)
says the default retry handler retries on a 429 or 503, which recovers from the limit but does not
keep you under it. Call `TokenBucket.before()` ahead of each `await`, or write a middleware handler
that does the same for every request. Kiota suits a backfill that already uses its typed models
for the rest of the integration.

## Check it worked

`check.py` counts admitted calls per minute of the fake clock on the server side, and asserts
that the peak stays at or under 60:

```bash
python3 check.py
```

```text output
ok  365 windows cover every day once
ok  weekly windows end on the last day
ok  sdkgen ratelimit: 365 calls, no 429, peak 60 of 60, every order landed
ok  token bucket: 365 calls, no 429, peak 60 of 60, every order landed
ok  RateLimit header pacer: 365 calls, no 429, peak 60 of 60, every order landed
ok  with no pacing the server refuses calls, and Retry-After still finishes the year
ok  Retry-After takes precedence over the RateLimit field
ok  the RateLimit field parses to r and t
ok  a rerun resumes after the checkpoint: 100 then 265 calls, 546 orders
ok  two clients at rate=1 on one key: 429s
ok  two clients at rate=0.5 on one key: no 429s
ok  the Swagger Codegen and Kiota files name retries or rate limits only in the API description
ok  Swagger Codegen's client: urllib3 waits out Retry-After on a 429, and the caller sees 200
```

## When it goes wrong

Two workers split the windows and share one key. Each has its own sdkgen client at the full rate,
so each bucket believes it owns the quota:

```bash
python3 parallel.py
```

```text output
each client at rate=1, the full budget
  requests 545, peak received in a minute 90, 429s 180, list() failures 180
each client at rate=0.5, half each
  requests 365, peak received in a minute 60, 429s 0, list() failures 0
```

The features reference says the same: the budget is per client instance, not distributed. Give
each of N workers `rate / N`, or share one bucket. pyrate-limiter's documentation lists a
`SQLiteBucket` with `use_file_lock=True` for many processes on one host and Redis buckets across
hosts.

A rerun starts at the first day. The checkpoint was written before the rows landed, or not at all.
Write it after `land()`, with a rename, so the file is either the old value or the new one.

A pacer set to the published limit still draws 429 responses. The server's minute is fixed and the
client's is not, or the clocks disagree. Leave a reserve, such as `HeaderPacer(sleep, reserve=2)`.

## When not to do this

Do not run windows in parallel processes to go faster. The quota is per key, so parallelism
buys nothing past the limit and turns pacing into a shared-state problem.

Do not rely on the sdkgen `ratelimit` feature when more than one process holds the key. It is
the simplest option for one client, and it cannot see the others.

Do not backfill through an API that offers a bulk export. One export job beats 365 paced calls.

Pick Swagger Codegen when its clients are already your team's standard and a wrapper is cheap,
and Kiota when its retry handler and typed models already carry the rest of the integration.

## Related how-tos

- [Read RateLimit headers and slow down before you get a 429](/howto/read-ratelimit-headers-and-pace-before-429)

- [Choose between 429 and 503 when you throttle a request](/howto/choose-429-or-503-when-throttling)

- [Rate limit by API key and tenant, not client IP](/howto/key-rate-limits-by-api-key-and-tenant)

- [Share one rate limiter across many workers with Redis](/howto/share-a-rate-limiter-across-workers-with-redis)

## Last verified

Verified 2026-11-20 against Python 3.13.18, Node 22.23.5, @voxgig/create-sdkgen 0.31.9,
@voxgig/sdkgen 4.44.3, requests 2.34.2, urllib3 2.8.0, Swagger Codegen 3.0.85, and Kiota 1.38.0. Every output
block is what the command preceding it printed. The generators ran once, through `generate.sh`,
and their output is committed. The Swagger Codegen client runs in `swagger_retry.py` and
`check.py`. The Kiota client and pyrate-limiter were not run; what the page says of them comes from
their documentation.

[^1]: The draft is
[draft-ietf-httpapi-ratelimit-headers-11](https://www.ietf.org/archive/id/draft-ietf-httpapi-ratelimit-headers-11.html),
dated May 23, 2026, and its first page gives an expiry of November 24, 2026, four days after this
page was verified. It says that when a response carries both `RateLimit` and `Retry-After`, the
latter must take precedence, so the older header outranks the newer one by the newer one's rule.