How to compare agent task success rates across model providers#

Your evaluation shows one model at 85% and another at 90% on twenty tasks, and the team switches providers on the strength of it. On two hundred tasks the order reverses. The other failure is quieter: a gateway drops a request parameter one provider does not support, and its model is scored down for the route’s behavior.

What you get

You will end up with a runner that varies only the model, a grader that sorts failures into classes, and a table of success with intervals, tokens, and wall time. This is for you if you choose a model for a tool-using agent.

Short answer

Fix the tools, the system prompt, the cases, and every request parameter, and vary only the model, through one OpenAI-compatible gateway such as LiteLLM with drop_params off. Grade each run into a failure class, so a protocol failure caused by the route is not counted as the model’s mistake. Report each model’s rate with a bootstrap interval drawn over cases, and compare two models paired by case before you rank them.

You will need

Node 22 or later, Python 3.12 or later, and a tool-using suite whose cases name the tool calls a correct run makes. Verified 2026-10-15 against Node 22.22.0 and Python 3.13.16, with no packages beyond the standard libraries. The three models on this page are scripted stubs behind a local gateway, so the numbers show what the method finds rather than how any provider performs. Choosing an evaluation framework is a separate task, and so is asserting a single tool call.

Approaches compared

A gateway buys one code path, and the code path is where fairness lives. LiteLLM raises an error by default when a provider cannot take a parameter, and silently drops it with drop_params on. Braintrust aligns cases across experiments against a baseline, which is the pairing, and promptfoo reruns each test with --repeat. Check whether a report gives an interval before you rank on it. Native SDKs are the only route that sends each provider exactly what it supports, at the cost of writing the same runner three times.

Hold everything fixed except the model

The runner reads the tools, the system prompt, and 200 cases from suite/, and sends the same request body for every model. Only model changes:

  for (let turn = 0; turn < Number(opt['max-turns']); turn++) {
    const res = await fetch(`${base}/v1/chat/completions`, {
      method: 'POST',
      headers,
      // seed only goes to the stub, which needs it to vary between trials. A gateway that
      // refuses parameters a provider lacks, as LiteLLM does by default, would reject it.
      body: JSON.stringify({ model, messages, tools, tool_choice: 'auto', parallel_tool_calls: PARALLEL_TOOL_CALLS, ...(stub ? { seed: trial } : {}) }),
    })

The tools declare strict: true, and the request sets parallel_tool_calls: false. Both are OpenAI function-calling parameters, so a gateway has to translate them for every other provider. Each case carries the calls a correct run makes, and grade.ts checks a run in a fixed order: the protocol, then the schema, the tool, the arguments, and last the answer. A write tool called on a read task fails as the wrong tool, whatever the answer says.

For real models, run the LiteLLM proxy with one model_name per provider and point the runner at it:

model_list:
  - model_name: claude
    litellm_params:
      model: anthropic/CLAUDE_MODEL_ID
      api_key: os.environ/ANTHROPIC_API_KEY
  - model_name: gpt
    litellm_params:
      model: openai/OPENAI_MODEL_ID
      api_key: os.environ/OPENAI_API_KEY
  - model_name: gemini
    litellm_params:
      model: gemini/GEMINI_MODEL_ID
      api_key: os.environ/GEMINI_API_KEY

litellm_settings:
  # The default. A parameter a provider does not support is an error, not a silent drop.
  drop_params: false

litellm --config litellm.yaml
node --experimental-strip-types run.ts --gateway http://localhost:4000 --models claude,gpt,gemini

Those two commands call real models and were not run here. Without --gateway, the runner starts stub-gateway.ts in a process of its own and uses three scripted models.

Read the failure classes before the rates

Three trials of 200 cases against each stub:

node --experimental-strip-types run.ts --trials 3 --out results/latest.jsonl

200 cases x 3 trials x 3 models = 1800 runs, parallel_tool_calls: false
stub-a  515/600 succeeded
    33  wrong tool
    32  no tool call
    20  wrong arguments
stub-b  494/600 succeeded
    42  wrong arguments
    33  no tool call
    31  wrong tool
stub-c  388/600 succeeded
   141  parallel calls despite parallel_tool_calls: false
    30  no tool call
    25  wrong arguments
    16  wrong tool

stub-c sits behind a route that drops parallel_tool_calls. On every comparison task it asks for both meters’ readings in one turn, which the request forbade. Allow parallel calls for everyone and run the two again:

node --experimental-strip-types run.ts --models stub-a,stub-c --parallel --out results/parallel.jsonl

200 cases x 3 trials x 2 models = 1200 runs, parallel_tool_calls: true
stub-a  515/600 succeeded
    33  wrong tool
    32  no tool call
    20  wrong arguments
stub-c  514/600 succeeded
    32  wrong arguments
    30  no tool call
    24  wrong tool

The 21-point gap was the route. The providers place the switch differently, which is why a gateway has to translate it: OpenAI takes parallel_tool_calls at the top of the request, and Anthropic takes disable_parallel_tool_use inside tool_choice.¹ Decide which behavior you are measuring, then send it to every provider or none.

Report an interval drawn over cases

stats.py reads the recorded results, the file one run of run.ts wrote, and reports each model’s rate with a 95% interval:

python3 stats.py results/recorded.jsonl

model    cases  runs  success   95% CI, by case  tokens/run  wall ms p50
stub-a     200   600    85.8%  [ 83.2%,  88.5%]         984           34
stub-b     200   600    82.3%  [ 79.3%,  85.5%]        1174           29
stub-c     200   600    64.7%  [ 59.2%,  70.2%]         831           20

The interval is a bootstrap over cases, not runs.² Three trials of a case a model cannot do are three failures of one case, and drawing runs instead would count them as three independent pieces of evidence. One test in test_stats.py builds that data and shows the run-level interval coming out narrower than the case-level one.

def bootstrap(values, rounds, seed):
    """95% percentile interval of the mean, resampling the given per-case values."""
    # random() is the one method whose sequence Python promises to keep across versions for a
    # given seed; choice() is not, so the index is drawn from random() directly.
    rng = random.Random(seed)
    n = len(values)
    means = sorted(sum(values[int(rng.random() * n)] for _ in range(n)) / n for _ in range(rounds))
    return means[int(0.025 * rounds)], means[int(0.975 * rounds) - 1]

Read the other two columns with care. Each provider counts tokens its own way, so tokens/run compares what you are billed for, not how much text moved. The wall times here are a stub sleeping a set time per model, measured at the runner; they show the column works, not how fast anything is.

Do not rank on twenty cases

The first twenty cases rank the first two models the other way round:

python3 stats.py results/recorded.jsonl --cases 20 --compare stub-a stub-b
python3 stats.py results/recorded.jsonl --compare stub-a stub-b --detect 0.05

stub-a minus stub-b over 20 cases: -5.0%, 95% CI [-16.7%, +5.0%], inside the noise
stub-a minus stub-b over 200 cases: +3.5%, 95% CI [-0.5%, +7.5%], inside the noise
cases needed to detect a 5% difference at this spread: 286

The stubs were scripted six points apart. Twenty cases put the wrong one ahead, and two hundred still could not separate them at 95%. The comparison is paired by case, which removes the case-to-case spread both models share, and --detect turns the remaining spread into the number of cases a real difference would need.

Check it worked

The grader tests cover each failure class, a number found inside a longer number, and malformed arguments. run.test.ts reruns the suite and checks results/recorded.jsonl is what it writes, apart from wall time:

node --experimental-strip-types --test grade.test.ts run.test.ts

# tests 11
# suites 0
# pass 11
# fail 0

The statistics tests run under Python’s own test runner:

python3 test_stats.py 2>&1

test_a_failed_request_is_not_a_failed_task (__main__.Stats.test_a_failed_request_is_not_a_failed_task) ... ok
test_a_model_missing_a_case_is_refused (__main__.Stats.test_a_model_missing_a_case_is_refused) ... ok
test_halving_the_difference_needs_four_times_the_cases (__main__.Stats.test_halving_the_difference_needs_four_times_the_cases) ... ok
test_resampling_runs_reports_a_narrower_interval_than_the_data_supports (__main__.Stats.test_resampling_runs_reports_a_narrower_interval_than_the_data_supports) ... ok
test_the_interval_is_the_same_for_the_same_seed (__main__.Stats.test_the_interval_is_the_same_for_the_same_seed) ... ok
test_the_reported_rate_is_the_one_the_interval_is_drawn_around (__main__.Stats.test_the_reported_rate_is_the_one_the_interval_is_drawn_around) ... ok

When it goes wrong

One provider’s rate drops sharply in the middle of a run. The provider is rate limiting you, and a runner that counts a 429 as a failed task scores the provider’s quota rather than its model. run.ts records a failed request with success: null, and stats.py leaves it out of the rate. The rate is the mean of the case rates, so a case that lost two of its three requests still counts once. stats.py refuses a comparison when a model has no graded run for a case, because the pairing is then broken.

The gateway rejects the whole run with an unsupported-parameter error. That is LiteLLM doing its job. Remove the parameter for every provider, or drop that provider from the comparison. Turning on drop_params would silence the error by sending each provider a different request.

A model’s argument JSON does not parse. The runner keeps the call with null arguments and the grader fails it as arguments are not JSON, rather than crashing the run.

When not to do this

Do not pick a model on a rate difference whose interval includes zero. Run more cases, or treat the models as tied and choose on price or latency.

Do not route the comparison through a gateway when the behavior you care about is specific to one provider, for example parallel calls. Use each provider’s own SDK for that one measurement.

Do not grade on the final answer alone. A model that reads the right number from the wrong call passes an answer check, and so does one that registers a meter while answering a question. The grader checks the calls before the answer to catch both.

Related how-tos

• Validate JSON model output before you use it

• Name tools so an agent picks the right one from fifty

• Cut agent cost with provider prompt caching

• Measure the token cost of an agent task against your API

• Map one tool definition to MCP, OpenAI, Claude, and Gemini

Last verified

Verified 2026-10-15 against Node 22.22.0 and Python 3.13.16. Every output block is what the command preceding it printed. The models are scripted stubs behind a local gateway, so no command on this page with an output block called a model. The behavior of LiteLLM, promptfoo, and Braintrust described here is from their documentation, not from a run.