How to poll a job status endpoint with backoff#

Your export script asks the vendor for a report, then checks the job every second until it says succeeded. The job fails at 02:00. The script keeps checking a failed job until the scheduler kills it at 06:00, and the vendor’s rate limiter has been counting every check. Nobody sees the failure until the report is missed.

What you get

You will end up with a poll loop that honors Retry-After and spreads its checks out when the server gives no hint. It stops on every terminal state, or at a deadline you set. This is for you if a script of yours waits on somebody else’s job.

Short answer

Loop on GET /jobs/{id} and stop on every terminal state the API documents, and not on success alone. Wait the Retry-After the server sends. When it sends none, wait one to two seconds, then double up to a cap with jitter. Track the total elapsed time and raise with the job id when it passes a deadline, so a caller can come back for the job instead of waiting forever.

You will need

Python 3.11 or later, an API that exposes a job status endpoint, and an HTTP client that lets you read response headers. The sample uses urllib and a fake job server on 127.0.0.1 built with http.server, so it runs without an account at any vendor. Retry-After is defined in RFC 9110 and takes either a number of seconds or an HTTP date, so a parser has to handle both.

Approaches compared

The loop is the same against all three shapes, and only the terminal set and the schedule differ. Azure sends Retry-After on every status answer, so the schedule is the server’s. Stripe and Salesforce send none, so the schedule is yours, and Stripe’s webhook is the alternative for anyone who can host an endpoint. Polling works from any script and costs a request per check. A webhook costs nothing per check and needs a URL the vendor can reach.

Enumerate every state that means stop

The loop stops when the state is in a set, and the set is the whole design. A loop that stops on succeeded alone has no exit on failed, so write down every terminal state the API documents before writing the loop.

SHAPES = {
    "stripe report run": ("status", frozenset({"succeeded", "failed"})),
    "salesforce bulk 2.0 job": ("state", frozenset({"JobComplete", "Failed", "Aborted"})),
    "azure operation": ("status", frozenset({"Succeeded", "Failed", "Canceled"})),
}

Three vendors, three field names, three spellings of done. Stripe’s report run object moves from pending to succeeded or failed. A Salesforce ingest job leaves UploadComplete for JobComplete or Failed, and becomes Aborted if you abort it. The Azure guidelines name NotStarted, Running, Succeeded, Failed and Canceled. Casing differs, and a comparison that ignores case would be a guess about a contract you do not own.

Read the state from the body and treat the status code as a hint. Azure’s monitor answers 200 while the operation is still running. The fake server here answers 202 until the job is done, which is the code RFC 9110 reserves for accepted and not finished.¹ Either way the body carries the state, and the loop reads that.

Take the server’s schedule when it sends one

Retry-After is the one number in the exchange both sides agree on. When it is present, use it and skip your own arithmetic.

def retry_after_seconds(value: str | None, now: float) -> float | None:
    """Retry-After is a count of seconds or an HTTP date. Either becomes seconds."""
    if value is None:
        return None
    value = value.strip()
    if value.isdigit():
        return float(value)
    try:
        when = parsedate_to_datetime(value)
    except (TypeError, ValueError):
        return None
    if when.tzinfo is None:
        when = when.replace(tzinfo=timezone.utc)
    return max(0.0, when.timestamp() - now)

Both forms are legal, and a parser that calls int() on the header raises on the date form.² parsedate_to_datetime handles the date, and max stops a date already in the past from producing a negative sleep. A value the parser cannot read returns None, which hands the decision back to the backoff schedule rather than guessing.

The client is one GET that reads the body and the header together, because the state comes from one and the schedule from the other.

def check_job(base: str, job_id: str) -> Status:
    """One GET. The state comes from the body and Retry-After from the headers."""
    with urllib.request.urlopen(f"{base}/jobs/{job_id}") as res:
        body = json.load(res)
        return Status(state=body["state"], retry_after=res.headers.get("Retry-After"), body=body)

Spread the checks out when it does not

Without a hint from the server, the wait starts between one and two seconds and doubles until it reaches a cap. The jitter is what stops fifty scripts that all started at 02:00 from checking in lockstep.

def backoff(attempt: int, *, first: float = 1.0, cap: float = 30.0, rng=random.random) -> float:
    """Equal jitter. Attempt 0 waits between first and 2 * first, doubling up to cap."""
    raw = min(cap, first * 2 ** (attempt + 1))
    return raw / 2 + rng() * raw / 2

This is the equal jitter schedule from the AWS analysis of backoff, which keeps a floor under each wait. Full jitter spreads retries better under contention, and it can also produce a wait of a few milliseconds, which is a bad first move against a job that takes minutes. The floor is the point here. rng is a parameter so the tests and the demo can fix it and print an exact schedule.

Stop at a deadline with the id in hand

The loop itself is short, and the deadline check is the line that matters.

def poll(
    check: Callable[[], Status],
    job_id: str,
    *,
    terminal: frozenset[str],
    deadline: float = 600.0,
    first: float = 1.0,
    cap: float = 30.0,
    sleep=time.sleep,
    clock=time.monotonic,
    rng=random.random,
) -> Status:
    """Call check() until its state is in terminal, or raise JobTimeout at the deadline."""
    started = clock()
    for attempt in itertools.count():
        status = check()
        if status.state in terminal:
            return status
        delay = retry_after_seconds(status.retry_after, time.time())
        if delay is None:
            delay = backoff(attempt, first=first, cap=cap, rng=rng)
        else:
            # Retry-After: 0, or a date already past, is a hot loop. The floor is yours.
            delay = max(delay, first)
        elapsed = clock() - started
        if elapsed + delay > deadline:
            raise JobTimeout(job_id, status.state, elapsed)
        sleep(delay)

The deadline is checked before the sleep, not after, so the loop never sleeps past it. A Retry-After of zero, or a date already past, is floored at first, because a server that says wait nothing has asked for a hot loop. The exception carries the job id and the last state, because a job that is still running at the deadline is not lost. It is a job somebody can come back for, and an error message that names it is the difference between a retry and a support ticket. Elapsed time comes from time.monotonic, which a clock adjustment cannot move backwards.

Four jobs against the fake server, with the sleeps recorded rather than slept and the random source fixed at one half:

python3 demo.py

202 three times, then 200 (job_1)
  GET /jobs/job_1 -> 202 running
  GET /jobs/job_1 -> 202 running
  GET /jobs/job_1 -> 202 running
  GET /jobs/job_1 -> 200 succeeded
  waits: 1.5s, 3.0s, 6.0s
  returned state=succeeded
server sends Retry-After: 5 (job_2)
  GET /jobs/job_2 -> 202 running
  GET /jobs/job_2 -> 202 running
  GET /jobs/job_2 -> 200 succeeded
  waits: 5.0s, 5.0s
  returned state=succeeded
the job fails (job_3)
  GET /jobs/job_3 -> 202 running
  GET /jobs/job_3 -> 200 failed
  waits: 1.5s
  returned state=failed
the job never finishes, deadline 60s (job_4)
  GET /jobs/job_4 -> 202 running
  GET /jobs/job_4 -> 202 running
  GET /jobs/job_4 -> 202 running
  GET /jobs/job_4 -> 202 running
  GET /jobs/job_4 -> 202 running
  GET /jobs/job_4 -> 202 running
  waits: 1.5s, 3.0s, 6.0s, 12.0s, 22.5s
  raised JobTimeout: job job_4 still running after 45s

Read the waits. The first job gets the schedule, doubling from 1.5 seconds. The second job gets exactly the five seconds the server asked for, twice. The third returns on failed after one wait, which is the whole point of the terminal set. The fourth never finishes, and the loop refuses the sixth wait because 45 seconds plus 22.5 more would pass the 60 second deadline.

Check it worked

The tests pin the schedule to exact numbers, which is what fixing rng buys, and the last one drives the loop through the real server over HTTP.

@case
def the_fake_server_answers_202_three_times_then_200():
    server = JobServer().start()
    try:
        job_id = create_job(server.url, {"pending": 3})
        got = poll(lambda: check_job(server.url, job_id), job_id,
                   terminal=frozenset({"succeeded", "failed"}), **NO_SLEEP)
        assert got.state == "succeeded", got
        gets = [r for r in server.requests if r.startswith("GET")]
        assert [r.split(" -> ")[1] for r in gets] == ["202 running"] * 3 + ["200 succeeded"], gets
    finally:
        server.stop()

python3 test_poll.py

ok stops_on_every_terminal_state
ok retry_after_in_seconds_beats_the_schedule
ok retry_after_as_an_http_date_becomes_seconds
ok backoff_doubles_from_one_second_to_the_cap
ok the_deadline_raises_with_the_job_id
ok the_fake_server_answers_202_three_times_then_200
6 cases, 0 failures

The first case runs succeeded, failed and canceled through the loop in turn and asserts on calls["n"], so a terminal state that the loop ignored would show up as a fourth check.

When it goes wrong

The loop runs until the deadline on a job that failed hours ago. The terminal set names success alone, so failed reads as keep going. Here is the same loop with terminal set to one state, against a job that fails on its third check:

python3 pitfall.py

terminal = {succeeded} against a job that fails
  checks made: 30
  state has been failed since check 3
  gave up: job job_1 still failed after 585s

Twenty-seven checks of a job that was never going to change, each one counted against a rate limit. Then a ten minute wait for an answer that arrived in the first few seconds. Put every terminal state in the set, and treat a state you have never seen as an error, not as running.

The checks arrive faster than the server asked. The header was read as an integer and the date form raised, or the header was never read at all. Parse both forms, and test the date form with a fixed clock, as retry_after_as_an_http_date_becomes_seconds does.

The loop spins with no wait at all. A server that answers Retry-After: 0 on every check has handed you a hot loop, and a loop that takes the header at face value runs it. Floor the wait at your first interval, as poll does, and take the disagreement up with the vendor.

When not to do this

Do not poll when the vendor offers a webhook and you can host an endpoint. Stripe sends reporting.report_run.succeeded the moment a run finishes, and a poll loop learns the same fact seconds to minutes later at a request per check. Polling earns its place in a script, a CI job or a laptop, where no endpoint exists.

Do not poll from inside a request handler. A caller waiting on your handler is waiting on the vendor’s job, and the handler’s own timeout arrives before the job does. Return the job id, and poll from a worker.

Do not retry a status check that fails with a 5xx as if it were a running job. The status endpoint can be down while the job is fine, and the two need different handling. Give the check its own retry with its own budget, and keep the job deadline separate.

Do not let Retry-After override your deadline. A server that asks for a wait longer than the time you have left is telling you the answer will not arrive in time. Raise with the job id and let the caller decide.³

Related how-tos

• Retry fetch calls with exponential backoff in Node.js

• Enforce one total deadline across all retry attempts

Last verified

Verified 2026-09-24 against Python 3.11.15. Every output block is what the preceding command printed, against the fake job server in the sample directory and not against Stripe, Salesforce or Azure. The state names for those three come from the documentation linked beside each.