How to measure the token cost of an agent task against your API#

An agent that records one meter reading through your API costs far more than the task deserves, and the bill arrives as one total. You cannot tell whether the tokens went on your tool definitions, on one tool’s result, or on the model’s reasoning, so you cannot tell which endpoint to fix.

What you get

You will end up with a cost per completed task, split between tool definitions, tool results, and reasoning, that names the single most expensive tool result. This is for you if other people’s agents call an API you publish.

Short answer

Run the task with a scripted agent, save every provider response, and read only the usage fields. A tool result costs the growth in input between two turns, less the output the model sent back, and it is paid again on every later turn. Count the first request with and without tools for the definitions’ share. Cap the largest result, run the task again, and compare the cost per task.

You will need

Python 3.11 or later, a task an agent can complete against your API, and provider responses saved from scripted runs. Verified 2026-10-15 against Python 3.13 with the standard library only. Scripting repeatable audit runs is a separate task; this page starts from the saved responses. No model is called here. record.py drives a scripted agent through the task and saves responses from stand-ins that answer in the documented shapes of the Messages API and the Responses API. The stand-ins count tokens as compact JSON length over four, so every absolute number below is illustrative, priced at Claude Sonnet 5.5’s list prices but never counted by that model. The differences between runs come from the requests.

Approaches compared

The usage fields are what the provider bills, and getting them means running the task yourself. Claude Code reports a session total with no effort and no breakdown. Estimating from your own logs works on traffic you did not script. It falls short by whatever the agent sent that your server never saw, a share that grows once the tool results shrink.

Read the usage fields of a recorded run

The task is to record a reading of 42.5 kWh for the Cork meter, and the API’s list_meters returns all 500 meters. The scripted agent lists them, finds Cork, and writes the reading, three turns in all.

python3 measure.py fixtures/uncapped

fixtures/uncapped: 3 turns, claude-sonnet-5-5, $2.00 in and $10.00 out per million tokens

turn    input  output  reasoning  then called     result tokens
   1      445      17          0  list_meters            24,149
   2   24,611      26          0  create_reading             45
   3   24,682      17          0  -                           -

input tokens over the task: 49,738
  tool results        48,343   97.2%  each paid again on every later turn
  tool definitions     1,215    2.4%  405 a turn
  everything else        180    0.4%
output tokens: 60, of which reasoning 0

cost per task: $0.1001
most expensive tool result: list_meters on turn 1, 24,149 tokens from 82,474 bytes,
  paid on 2 later turns, $0.0966 of the task

One result is 97 percent of the stand-in’s input count. The list arrived on turn 1, and every later request carried it again, because each request in a conversation carries everything before it. Prices are Claude Sonnet 5.5’s list prices from the pricing page, held in a dictionary you replace.

The attribution needs no token counter of its own. A tool result’s size is the growth in input from one turn to the next, less the turn’s own output that goes back as input. That is the visible output, plus the reasoning when the model keeps it.

    # What a tool result added to the next request: the growth in context, less the turn's own
    # output that is sent back as input, which includes its reasoning when the model keeps it.
    # A turn that called two tools in parallel gets one delta for both, and the log's next two lines.
    results, logged = [], iter(log)
    for k in range(n - 1):
        added = steps[k + 1]["context"] - steps[k]["context"] - steps[k]["resent"]
        calls = [next(logged) for _ in steps[k]["called"]]
        results.append({"turn": k + 1, "tool": "+".join(c["tool"] for c in calls), "bytes": sum(c["bytes"] for c in calls),
                        "tokens": added, "paid_on": n - 1 - k})

The definitions’ share comes from the token counting endpoint, which is free and answers with one number: the first request counted with tools, minus the same request counted without. On Claude that difference includes a tool-use system prompt the API adds whenever tools are present, 286 tokens for Claude Sonnet 5.5 with tool_choice set to auto or none, according to the pricing page. The stand-in adds it too, which is why 405 tokens a turn come from 119 tokens of definitions. OpenAI has an equivalent endpoint at /v1/responses/input_tokens, described in its token counting guide.

Read the cache fields the way each API defines them

The two APIs report cached tokens differently, and a parser that treats them alike miscounts both.

def usage_of(response, reasoning_mode=None):
    """One shape for both APIs. The cache fields are typed as nullable, so None counts as 0."""
    u = response["usage"]
    if response.get("object") == "response":
        # Responses API: input_tokens already includes the cached tokens.
        details = u.get("input_tokens_details") or {}
        cached, written, written_1h = details.get("cached_tokens") or 0, details.get("cache_write_tokens") or 0, 0
        uncached = u["input_tokens"] - cached - written
        reasoning = (u.get("output_tokens_details") or {}).get("reasoning_tokens") or 0
        called = [i["name"] for i in response["output"] if i["type"] == "function_call"]
        # The response names the mode it used; all_turns carries this turn's reasoning forward.
        kept = (response.get("reasoning") or {}).get("context") == "all_turns"
    else:
        # Messages API: input_tokens excludes cache reads and writes, which are reported beside it.
        cached, written = u.get("cache_read_input_tokens") or 0, u.get("cache_creation_input_tokens") or 0
        written_1h = (u.get("cache_creation") or {}).get("ephemeral_1h_input_tokens") or 0
        uncached = u["input_tokens"]
        reasoning = (u.get("output_tokens_details") or {}).get("thinking_tokens") or 0
        called = [b["name"] for b in response["content"] if b["type"] == "tool_use"]
        # Claude Opus 4.5, Sonnet 4.6, and later models keep earlier thinking in context by default.
        kept = True
    if reasoning_mode is not None:
        kept = reasoning_mode == "resent"
    output = u["output_tokens"]
    return {
        "context": uncached + cached + written, "uncached": uncached, "cache_read": cached,
        "cache_write": written - written_1h, "cache_write_1h": written_1h, "output": output,
        "reasoning": reasoning, "resent": output if kept else output - reasoning, "called": called,
    }

The Messages API reference says the total input is the sum of input_tokens and the two cache fields. The Responses API lists cached tokens as a breakdown of input_tokens. Reasoning sits in output_tokens_details in both, as thinking_tokens and reasoning_tokens, and is billed as output.¹ Claude prices a one-hour cache write higher than a five-minute one, so the function reads the cache_creation breakdown too, and the cost line prices each tier at its own rate.

Cap the result and run the task again

Give list_meters a page size of 20 and a cursor, record the same task, and measure it:

python3 record.py fixtures/capped --page-size 20 && python3 measure.py fixtures/capped

recorded 3 turns to fixtures/capped
fixtures/capped: 3 turns, claude-sonnet-5-5, $2.00 in and $10.00 out per million tokens

turn    input  output  reasoning  then called     result tokens
   1      445      17          0  list_meters               997
   2    1,459      26          0  create_reading             45
   3    1,530      17          0  -                           -

input tokens over the task: 3,434
  tool results         2,039   59.4%  each paid again on every later turn
  tool definitions     1,215   35.4%  405 a turn
  everything else        180    5.2%
output tokens: 60, of which reasoning 0

cost per task: $0.0075
most expensive tool result: list_meters on turn 1, 997 tokens from 3,302 bytes,
  paid on 2 later turns, $0.0040 of the task

In the stand-in’s counts, the cost per task fell from $0.1001 to $0.0075. The Cork meter is on the first page, so the turn count did not change. A meter on page 20 would cost 19 more turns, each carrying the pages before it, so put a filter beside the cap.

Compare the two cheaper sources

Claude Code reports cost per session when run with --output-format json, as its headless guide documents. The fixture below is not a Claude Code run: record.py writes it in that documented shape from the stand-in’s numbers.

python3 claude_code.py fixtures/uncapped/claude-code-result.json

3 turns, $0.1001 for the session (a client-side estimate)
  claude-sonnet-5-5  in 49,738  out 60  cache read 0  $0.1001
  no field names a tool, so the session total cannot be split by tool result

The cost tracking guide calls total_cost_usd a client-side estimate computed from a bundled price table. It answers what a session cost, and it cannot say which result made it cost that.

Your server’s log holds the results it returned, so it can estimate without the agent:

python3 estimate.py fixtures/uncapped && python3 estimate.py fixtures/capped

fixtures/uncapped, counted with stand-in
  estimated from the server log   48,616 input tokens over 3 turns
  usage fields from the run       49,738
  the log cannot see               1,122  (2.3% of the task)
  largest single result           24,120
fixtures/capped, counted with stand-in
  estimated from the server log    2,312 input tokens over 3 turns
  usage fields from the run        3,434
  the log cannot see               1,122  (32.7% of the task)
  largest single result              968

The same 1,122 tokens are missing from both: the system prompt, the task, the agent’s turns, and the tool-use prompt on every turn. The estimate is close while one result dominates, and a third short once the result is fixed. Run in this container, not in CI, estimate.py --tokenizer tiktoken counted the 500-row result as 30,538 o200k_base tokens against the stand-in’s 24,120. That is about 2.7 characters per token for this JSON, so a four-characters rule counts it low.

Check it worked

Ten tests hold the numbers on this page, including one that records both runs again and compares the fixtures byte for byte.

python3 test_measure.py -v 2>&1

test_both_apis_agree_on_the_expensive_result (__main__.Measure.test_both_apis_agree_on_the_expensive_result) ... ok
test_cache_fields_are_read_the_way_each_api_defines_them (__main__.Measure.test_cache_fields_are_read_the_way_each_api_defines_them) ... ok
test_capping_the_list_cuts_the_cost_per_task (__main__.Measure.test_capping_the_list_cuts_the_cost_per_task) ... ok
test_each_result_is_attributed_its_own_size (__main__.Measure.test_each_result_is_attributed_its_own_size) ... ok
test_one_hour_cache_writes_are_priced_at_their_own_rate (__main__.Measure.test_one_hour_cache_writes_are_priced_at_their_own_rate) ... ok
test_reasoning_the_model_keeps_is_not_billed_to_the_tool_result (__main__.Measure.test_reasoning_the_model_keeps_is_not_billed_to_the_tool_result) ... ok
test_recording_again_reproduces_the_fixtures_byte_for_byte (__main__.Measure.test_recording_again_reproduces_the_fixtures_byte_for_byte) ... ok
test_the_claude_code_report_has_a_total_and_no_tool_split (__main__.Measure.test_the_claude_code_report_has_a_total_and_no_tool_split) ... ok
test_the_server_log_misses_what_only_the_agent_sends (__main__.Measure.test_the_server_log_misses_what_only_the_agent_sends) ... ok
test_two_calls_in_one_turn_share_that_turn_s_delta (__main__.Measure.test_two_calls_in_one_turn_share_that_turn_s_delta) ... ok

The capping test asserts the uncapped run costs more than ten times the capped one. The attribution test checks each delta against the logged result’s own size, within the few tokens of framing a message adds. Two more pin the cases a stand-in run never reaches: reasoning the model keeps in context, and a one-hour cache write.

When it goes wrong

The result’s share includes reasoning. Some models keep earlier turns’ reasoning as input. Claude’s context windows guide says Claude Sonnet 4.6 and later keep previous thinking blocks by default, and OpenAI’s reasoning guide says GPT-5.6 models render earlier reasoning by default. measure.py treats Claude’s thinking as kept and reads reasoning.context from each Responses API response. Pass --reasoning dropped for a model that discards it.

The comparison goes negative. The estimate and the measurement counted tokens in different ways. Compare counts made the same way, or compare each run against itself.

The task fails instead of costing more. The list no longer fits the context window. The Responses API’s truncation: "auto", which its reference now marks deprecated, drops items from the beginning of the conversation, which is where the task is. Its default refuses the request with a 400.

A cached run looks dearer than it is. The “paid on later turns” figure prices a result at the full input rate. The cost per task uses each field’s own price, so trust that line.

When not to do this

Do not measure one run and call it the cost. A model chooses different calls on different runs, so record several and report the spread, with the scripted agent kept for the regression check.

Do not cap a list without a filter. The cap moves the cost into extra turns when the record the agent needs is not on the first page.

Do not quote the stand-in’s numbers as a price. They show where the tokens go and how a change moves them; the provider’s usage fields are the only figures to bill against.

Do not use this audit as accounting. It answers what your API costs an agent once. Per-task, per-tenant cost tracking in an agent you run is a running system with its own storage.

Related how-tos

• Cut agent cost with provider prompt caching

• Choose a pagination style for a list endpoint

• Sanitize tool results before they re-enter the prompt

• Cache tool results across the steps of one agent run

• Map one tool definition to MCP, OpenAI, Claude, and Gemini

Last verified

Verified 2026-10-15 against Python 3.13, standard library only. Every output block is what the command preceding it printed. No model was called: the responses come from the stand-ins in stand_in.py, in the documented shapes, with tokens counted as JSON length over four. The tiktoken 0.14.0 count was run once in the writing container and is not re-run by the gate.