# How to measure the token cost of an agent task against your API > Read the usage fields from a scripted run, split each task's tokens between tool definitions, tool results, and reasoning, and find the costliest result. Source: https://voxgig.com/howto/measure-token-cost-per-agent-task - Audience: platform-team - Level: advanced - Languages: python - Verified: 2026-10-15 - Published: 2026-10-15 ## Short answer Run the task with a scripted agent, save every provider response, and read only the usage fields. A tool result costs the growth in input between two turns, less the output the model sent back, and it is paid again on every later turn. Count the first request with and without tools for the definitions' share. Cap the largest result, run the task again, and compare the cost per task. --- ## You will need Python 3.11 or later, a task an agent can complete against your API, and provider responses saved from scripted runs. Verified 2026-10-15 against Python 3.13 with the standard library only. Scripting repeatable audit runs is a separate task; this page starts from the saved responses. No model is called here. `record.py` drives a scripted agent through the task and saves responses from stand-ins that answer in the documented shapes of the [Messages API](https://platform.claude.com/docs/en/api/messages) and the [Responses API](https://developers.openai.com/api/reference/resources/responses/methods/create). The stand-ins count tokens as compact JSON length over four, so every absolute number below is illustrative, priced at Claude Sonnet 5.5's list prices but never counted by that model. The differences between runs come from the requests. ## Approaches compared | Approach | When it fits | What it costs you | When to pick something else | | --- | --- | --- | --- | | [API usage fields from scripted runs](https://platform.claude.com/docs/en/api/messages) | You can run the task yourself, against each model you care about | A scripted run per model, and a split you derive from deltas rather than read | You cannot drive the agent and see only its traffic | | [Claude Code JSON output cost report](https://code.claude.com/docs/en/headless) | Your users drive Claude Code, and one number per session answers the question | A client-side estimate per session, with no field that names a tool | You need the cost of one tool's result | | [`tiktoken` or `count_tokens` estimates from logs](https://pypi.org/project/tiktoken/) | Any client's traffic, counted from logs your server already keeps | Misses the system prompt, the agent's own turns, and tokens the provider adds | You need each result's exact share of the task | The usage fields are what the provider bills, and getting them means running the task yourself. Claude Code reports a session total with no effort and no breakdown. Estimating from your own logs works on traffic you did not script. It falls short by whatever the agent sent that your server never saw, a share that grows once the tool results shrink. ## Read the usage fields of a recorded run The task is to record a reading of 42.5 kWh for the Cork meter, and the API's `list_meters` returns all 500 meters. The scripted agent lists them, finds Cork, and writes the reading, three turns in all. ```bash python3 measure.py fixtures/uncapped ``` ```text output fixtures/uncapped: 3 turns, claude-sonnet-5-5, $2.00 in and $10.00 out per million tokens turn input output reasoning then called result tokens 1 445 17 0 list_meters 24,149 2 24,611 26 0 create_reading 45 3 24,682 17 0 - - input tokens over the task: 49,738 tool results 48,343 97.2% each paid again on every later turn tool definitions 1,215 2.4% 405 a turn everything else 180 0.4% output tokens: 60, of which reasoning 0 cost per task: $0.1001 most expensive tool result: list_meters on turn 1, 24,149 tokens from 82,474 bytes, paid on 2 later turns, $0.0966 of the task ``` One result is 97 percent of the stand-in's input count. The list arrived on turn 1, and every later request carried it again, because each request in a conversation carries everything before it. Prices are Claude Sonnet 5.5's list prices from the [pricing page](https://platform.claude.com/docs/en/about-claude/pricing), held in a dictionary you replace. The attribution needs no token counter of its own. A tool result's size is the growth in input from one turn to the next, less the turn's own output that goes back as input. That is the visible output, plus the reasoning when the model keeps it. ```python title="measure.py" # What a tool result added to the next request: the growth in context, less the turn's own # output that is sent back as input, which includes its reasoning when the model keeps it. # A turn that called two tools in parallel gets one delta for both, and the log's next two lines. results, logged = [], iter(log) for k in range(n - 1): added = steps[k + 1]["context"] - steps[k]["context"] - steps[k]["resent"] calls = [next(logged) for _ in steps[k]["called"]] results.append({"turn": k + 1, "tool": "+".join(c["tool"] for c in calls), "bytes": sum(c["bytes"] for c in calls), "tokens": added, "paid_on": n - 1 - k}) ``` The definitions' share comes from the [token counting endpoint](https://platform.claude.com/docs/en/build-with-claude/token-counting), which is free and answers with one number: the first request counted with tools, minus the same request counted without. On Claude that difference includes a tool-use system prompt the API adds whenever tools are present, 286 tokens for Claude Sonnet 5.5 with `tool_choice` set to `auto` or `none`, according to the pricing page. The stand-in adds it too, which is why 405 tokens a turn come from 119 tokens of definitions. OpenAI has an equivalent endpoint at `/v1/responses/input_tokens`, described in its [token counting guide](https://developers.openai.com/api/docs/guides/token-counting). ## Read the cache fields the way each API defines them The two APIs report cached tokens differently, and a parser that treats them alike miscounts both. ```python title="measure.py" def usage_of(response, reasoning_mode=None): """One shape for both APIs. The cache fields are typed as nullable, so None counts as 0.""" u = response["usage"] if response.get("object") == "response": # Responses API: input_tokens already includes the cached tokens. details = u.get("input_tokens_details") or {} cached, written, written_1h = details.get("cached_tokens") or 0, details.get("cache_write_tokens") or 0, 0 uncached = u["input_tokens"] - cached - written reasoning = (u.get("output_tokens_details") or {}).get("reasoning_tokens") or 0 called = [i["name"] for i in response["output"] if i["type"] == "function_call"] # The response names the mode it used; all_turns carries this turn's reasoning forward. kept = (response.get("reasoning") or {}).get("context") == "all_turns" else: # Messages API: input_tokens excludes cache reads and writes, which are reported beside it. cached, written = u.get("cache_read_input_tokens") or 0, u.get("cache_creation_input_tokens") or 0 written_1h = (u.get("cache_creation") or {}).get("ephemeral_1h_input_tokens") or 0 uncached = u["input_tokens"] reasoning = (u.get("output_tokens_details") or {}).get("thinking_tokens") or 0 called = [b["name"] for b in response["content"] if b["type"] == "tool_use"] # Claude Opus 4.5, Sonnet 4.6, and later models keep earlier thinking in context by default. kept = True if reasoning_mode is not None: kept = reasoning_mode == "resent" output = u["output_tokens"] return { "context": uncached + cached + written, "uncached": uncached, "cache_read": cached, "cache_write": written - written_1h, "cache_write_1h": written_1h, "output": output, "reasoning": reasoning, "resent": output if kept else output - reasoning, "called": called, } ``` The Messages API reference says the total input is the sum of `input_tokens` and the two cache fields. The Responses API lists cached tokens as a breakdown of `input_tokens`. Reasoning sits in `output_tokens_details` in both, as `thinking_tokens` and `reasoning_tokens`, and is billed as output.[^1] Claude prices a one-hour cache write higher than a five-minute one, so the function reads the `cache_creation` breakdown too, and the cost line prices each tier at its own rate. ## Cap the result and run the task again Give `list_meters` a page size of 20 and a cursor, record the same task, and measure it: ```bash python3 record.py fixtures/capped --page-size 20 && python3 measure.py fixtures/capped ``` ```text output recorded 3 turns to fixtures/capped fixtures/capped: 3 turns, claude-sonnet-5-5, $2.00 in and $10.00 out per million tokens turn input output reasoning then called result tokens 1 445 17 0 list_meters 997 2 1,459 26 0 create_reading 45 3 1,530 17 0 - - input tokens over the task: 3,434 tool results 2,039 59.4% each paid again on every later turn tool definitions 1,215 35.4% 405 a turn everything else 180 5.2% output tokens: 60, of which reasoning 0 cost per task: $0.0075 most expensive tool result: list_meters on turn 1, 997 tokens from 3,302 bytes, paid on 2 later turns, $0.0040 of the task ``` In the stand-in's counts, the cost per task fell from $0.1001 to $0.0075. The Cork meter is on the first page, so the turn count did not change. A meter on page 20 would cost 19 more turns, each carrying the pages before it, so put a filter beside the cap. ## Compare the two cheaper sources Claude Code reports cost per session when run with `--output-format json`, as its [headless guide](https://code.claude.com/docs/en/headless) documents. The fixture below is not a Claude Code run: `record.py` writes it in that documented shape from the stand-in's numbers. ```bash python3 claude_code.py fixtures/uncapped/claude-code-result.json ``` ```text output 3 turns, $0.1001 for the session (a client-side estimate) claude-sonnet-5-5 in 49,738 out 60 cache read 0 $0.1001 no field names a tool, so the session total cannot be split by tool result ``` The [cost tracking guide](https://code.claude.com/docs/en/agent-sdk/cost-tracking) calls `total_cost_usd` a client-side estimate computed from a bundled price table. It answers what a session cost, and it cannot say which result made it cost that. Your server's log holds the results it returned, so it can estimate without the agent: ```bash python3 estimate.py fixtures/uncapped && python3 estimate.py fixtures/capped ``` ```text output fixtures/uncapped, counted with stand-in estimated from the server log 48,616 input tokens over 3 turns usage fields from the run 49,738 the log cannot see 1,122 (2.3% of the task) largest single result 24,120 fixtures/capped, counted with stand-in estimated from the server log 2,312 input tokens over 3 turns usage fields from the run 3,434 the log cannot see 1,122 (32.7% of the task) largest single result 968 ``` The same 1,122 tokens are missing from both: the system prompt, the task, the agent's turns, and the tool-use prompt on every turn. The estimate is close while one result dominates, and a third short once the result is fixed. Run in this container, not in CI, `estimate.py --tokenizer tiktoken` counted the 500-row result as 30,538 `o200k_base` tokens against the stand-in's 24,120. That is about 2.7 characters per token for this JSON, so a four-characters rule counts it low. ## Check it worked Ten tests hold the numbers on this page, including one that records both runs again and compares the fixtures byte for byte. ```bash python3 test_measure.py -v 2>&1 ``` ```text output test_both_apis_agree_on_the_expensive_result (__main__.Measure.test_both_apis_agree_on_the_expensive_result) ... ok test_cache_fields_are_read_the_way_each_api_defines_them (__main__.Measure.test_cache_fields_are_read_the_way_each_api_defines_them) ... ok test_capping_the_list_cuts_the_cost_per_task (__main__.Measure.test_capping_the_list_cuts_the_cost_per_task) ... ok test_each_result_is_attributed_its_own_size (__main__.Measure.test_each_result_is_attributed_its_own_size) ... ok test_one_hour_cache_writes_are_priced_at_their_own_rate (__main__.Measure.test_one_hour_cache_writes_are_priced_at_their_own_rate) ... ok test_reasoning_the_model_keeps_is_not_billed_to_the_tool_result (__main__.Measure.test_reasoning_the_model_keeps_is_not_billed_to_the_tool_result) ... ok test_recording_again_reproduces_the_fixtures_byte_for_byte (__main__.Measure.test_recording_again_reproduces_the_fixtures_byte_for_byte) ... ok test_the_claude_code_report_has_a_total_and_no_tool_split (__main__.Measure.test_the_claude_code_report_has_a_total_and_no_tool_split) ... ok test_the_server_log_misses_what_only_the_agent_sends (__main__.Measure.test_the_server_log_misses_what_only_the_agent_sends) ... ok test_two_calls_in_one_turn_share_that_turn_s_delta (__main__.Measure.test_two_calls_in_one_turn_share_that_turn_s_delta) ... ok ``` The capping test asserts the uncapped run costs more than ten times the capped one. The attribution test checks each delta against the logged result's own size, within the few tokens of framing a message adds. Two more pin the cases a stand-in run never reaches: reasoning the model keeps in context, and a one-hour cache write. ## When it goes wrong The result's share includes reasoning. Some models keep earlier turns' reasoning as input. Claude's [context windows guide](https://platform.claude.com/docs/en/build-with-claude/context-windows) says Claude Sonnet 4.6 and later keep previous thinking blocks by default, and OpenAI's [reasoning guide](https://developers.openai.com/api/docs/guides/reasoning) says GPT-5.6 models render earlier reasoning by default. `measure.py` treats Claude's thinking as kept and reads `reasoning.context` from each Responses API response. Pass `--reasoning dropped` for a model that discards it. The comparison goes negative. The estimate and the measurement counted tokens in different ways. Compare counts made the same way, or compare each run against itself. The task fails instead of costing more. The list no longer fits the context window. The Responses API's `truncation: "auto"`, which its reference now marks deprecated, drops items from the beginning of the conversation, which is where the task is. Its default refuses the request with a 400. A cached run looks dearer than it is. The "paid on later turns" figure prices a result at the full input rate. The cost per task uses each field's own price, so trust that line. ## When not to do this Do not measure one run and call it the cost. A model chooses different calls on different runs, so record several and report the spread, with the scripted agent kept for the regression check. Do not cap a list without a filter. The cap moves the cost into extra turns when the record the agent needs is not on the first page. Do not quote the stand-in's numbers as a price. They show where the tokens go and how a change moves them; the provider's usage fields are the only figures to bill against. Do not use this audit as accounting. It answers what your API costs an agent once. Per-task, per-tenant cost tracking in an agent you run is a running system with its own storage. ## Related how-tos - [Cut agent cost with provider prompt caching](/howto/cut-agent-cost-with-prompt-caching) - [Choose a pagination style for a list endpoint](/howto/choose-a-pagination-style-for-a-list-endpoint) - [Sanitize tool results before they re-enter the prompt](/howto/sanitise-tool-results-before-reprompting) - [Cache tool results across the steps of one agent run](/howto/cache-tool-results-across-agent-steps) - [Map one tool definition to MCP, OpenAI, Claude, and Gemini](/howto/convert-tool-definitions-between-mcp-openai-claude-gemini) ## Last verified Verified 2026-10-15 against Python 3.13, standard library only. Every output block is what the command preceding it printed. No model was called: the responses come from the stand-ins in `stand_in.py`, in the documented shapes, with tokens counted as JSON length over four. The `tiktoken` 0.14.0 count was run once in the writing container and is not re-run by the gate. [^1]: The Messages API reference adds that the counts in `usage` will not match the visible content of a request or response one to one. Its example is `output_tokens`, which is non-zero even when Claude's response is an empty string. The [reference](https://platform.claude.com/docs/en/api/messages) attributes this to a parsing stage between the model and the API response. An empty answer still arrives on a bill, itemized.