How to run agent evals in CI without flaky failures#

Your eval suite runs on every pull request and fails about one run in three. The cases that fail change each time while the model does not, so the team has learned to click rerun. The day a change breaks refunds for half the customers, its red check looks like every other red check, so it merges.

What you get

You will end up with a gate that reruns each case, judges the aggregate against a band from history, and names a drop as regression, noise, or outage. This is for you if a red eval check gets rerun before it gets read.

Short answer

Pin the model snapshot and run every case five times. Gate the pull request on the aggregate pass rate against a tolerance band drawn from the nightly history of the same cases, not on every case passing. Treat provider errors as missing evidence that makes the run inconclusive. Run a twelve-case smoke suite per pull request and the full suite nightly, and record each night so a drop can be read against its own history.

You will need

Node 22 or later, an eval suite whose cases name the tool call a correct run makes, and a CI system with a schedule trigger. Verified 2026-10-21 against Node 22.22.0, with no packages beyond the standard library. The model on this page is a scripted stub with two dated snapshot ids, so the numbers show what the gate does and not how any provider behaves.

Approaches compared

A mocked run passes or fails for a reason you can read in the diff, because the model’s answers are fixed, which is also why it cannot see the model regress. A live run sees the model and pays for it. promptfoo’s --repeat buys the trials under one threshold for the run. The split keeps the live bill to twelve cases per push and finds a regression in the other twenty-eight after the merge rather than before it.

Pin what the provider lets you pin, then repeat

The configuration names a dated snapshot, a temperature, and a trial count, and the runner reads nothing else about the model. A floating alias resolves to a different snapshot on the day the provider decides, so a drop under an alias can be their change or yours.

{
  "model": "stub-2026-06-01",
  "temperature": 0,
  "trials": 5,
  "cases": "suite/cases.jsonl",
  "floor": 0.75,
  "sigma": 2,
  "maxErrorShare": 0.1
}

Temperature and seed are weaker than they look. OpenAI’s reference documents seed as a best effort at determinism and marks it deprecated, along with the system_fingerprint field it points to. Anthropic’s reference says temperature 0.0 was never fully deterministic, and that models released after Claude Opus 4.6 reject any value other than 1.0.¹ The lever that works at every provider is repetition. Five trials per case, under a seed of your own that names the run, make each rate a measurement with noise you can estimate.

// A pass is the one expected call with exactly the expected arguments. Anything else is a reason
// the report can count, because "no tool call" and "wrong arguments" are different regressions.
// Arguments compare as values, not as JSON text: a model may emit the keys in any order.
export function grade(c: Case, call: Call | null): 'pass' | string {
  if (!call) return 'no tool call'
  if (call.tool !== c.expect.tool) return `called ${call.tool}`
  if (!isDeepStrictEqual(call.args, c.expect.args)) return 'wrong arguments'
  return 'pass'
}

export function runSuite(o: { config: Config; cases: Case[]; seed: string; filter: string; model?: string; outage?: number }): Run {
  const model = o.model ?? o.config.model
  const results: CaseResult[] = []
  for (const c of o.cases) {
    const r: CaseResult = { id: c.id, kind: c.kind, suite: c.suite, passes: 0, fails: 0, errors: 0, reasons: [] }
    for (let trial = 1; trial <= o.config.trials; trial++) {
      const reply = complete({ model, c, trial, seed: o.seed, temperature: o.config.temperature, outage: o.outage })
      if (reply.status === 'error') { r.errors++; continue }
      const verdict = grade(c, reply.call)
      if (verdict === 'pass') r.passes++
      else { r.fails++; r.reasons.push(verdict) }
    }
    results.push(r)
  }
  return { run: o.seed, model, temperature: o.config.temperature, trials: o.config.trials, filter: o.filter, cases: results }
}

The grader is strict and records why a trial failed, because a run that stopped calling tools and a run that acts on the wrong order id are different regressions.

node --experimental-strip-types run.ts --seed pr-412 --out results/pr.jsonl

run pr-412: stub-2026-06-01, temperature 0, 40 cases x 5 trials = 200 calls
pass 175  fail 25  error 0
failures: called get_order 10, wrong arguments 10, no tool call 5
aggregate pass rate 0.8750
wrote results/pr.jsonl

Twenty-five failed trials out of two hundred, from a snapshot that has not changed. A gate that wanted every case green would have failed this run on seventeen cases, which is the flakiness in the title. The noise is in the model, so a rerun samples it again.

Build the baseline from the nights

The baseline is the suite’s own history on main. make-history.ts plays ten nightly runs of the pinned snapshot, decides each against the nights before it, and appends the verdict and every case’s rate to history/main.jsonl. In CI the nightly job does the same with gate.ts --record.

node --experimental-strip-types make-history.ts

main-2026-09-01: pass, aggregate 0.8800, lower limit none yet
main-2026-09-02: pass, aggregate 0.8850, lower limit none yet
main-2026-09-03: pass, aggregate 0.9050, lower limit none yet
main-2026-09-04: pass, aggregate 0.9100, lower limit 0.8389
main-2026-09-05: pass, aggregate 0.9250, lower limit 0.8465
main-2026-09-06: pass, aggregate 0.8750, lower limit 0.8547
main-2026-09-07: FAIL, aggregate 0.8500, lower limit 0.8502
main-2026-09-08: pass, aggregate 0.8950, lower limit 0.8362
main-2026-09-09: pass, aggregate 0.9000, lower limit 0.8411
main-2026-09-10: pass, aggregate 0.8850, lower limit 0.8451
wrote 10 runs to history/main.jsonl

Night seven failed its own gate by two ten-thousandths, on a snapshot that had not changed. A band two units of noise wide fails about one unchanged run in forty by construction, and that night was the one.² The record keeps it, because a baseline built only from the nights that passed drifts upward while its spread shrinks.

Gate on the aggregate against the band

// The baseline is every completed run in history, each re-averaged over the cases this run has, so
// a smoke run is compared with the smoke cases of every nightly run and not with the whole suite.
// Failed nights stay in: a baseline built from the nights that passed drifts upward and its spread
// shrinks, and the band then fails more runs for nothing. A night that failed is main's problem,
// and the record says so.
export function baseline(history: HistoryRun[], ids: string[]) {
  const rates: number[] = []
  for (const h of history) {
    const xs = ids.map((id) => h.cases[id]).filter((x): x is number => x !== undefined)
    if (xs.length) rates.push(mean(xs))
  }
  return rates.length < 3 ? null : { n: rates.length, mean: mean(rates), sd: sd(rates) }
}

// The spread the band is drawn from. Ten nights estimate the between-run spread badly, and the
// run being judged has sampling noise of its own that no history can remove, so the band is never
// narrower than the binomial noise of this many calls, and it widens for the error in the mean.
export function noise(b: { n: number; mean: number; sd: number }, calls: number): number {
  const binomial = Math.sqrt((b.mean * (1 - b.mean)) / calls)
  return Math.max(b.sd, binomial) * Math.sqrt(1 + 1 / b.n)
}

The baseline mean and standard deviation come from the ten nights, re-averaged over the cases this run has. The noise is the larger of that standard deviation and the binomial noise of this many calls, widened for the error in the mean, because ten nights estimate a spread badly.

node --experimental-strip-types gate.ts results/pr.jsonl

pr-412: stub-2026-06-01, 40 cases x 5 trials, aggregate pass rate 0.8750
floor 0.7500: ok
baseline over 10 runs on these cases: mean 0.8910, sd 0.0209, noise used 0.0231
band 2 x noise below the baseline, lower limit 0.8448: ok
largest drops below each case's own history (mean, sd in history -> this run):
  refund-02      0.90 (0.14) -> 0.40
  escalate-03    0.82 (0.20) -> 0.40
  refund-04      0.90 (0.14) -> 0.60
  escalate-07    0.88 (0.14) -> 0.60
  reschedule-03  0.88 (0.14) -> 0.60
flaky in history (sd over 0.20), not ranked: escalate-05, reschedule-08
verdict: pass

The aggregate sits a point and a half under the baseline and inside the band. The ranked list is for the reviewer: one case from each kind near the top is an ordinary Tuesday. Two cases that move by more than 0.20 between nights are left out, because one run of theirs says nothing.

The second snapshot handles refunds worse and everything else the same:

node --experimental-strip-types run.ts --seed pr-412 --model stub-2026-09-15 --out results/upgrade.jsonl
node --experimental-strip-types gate.ts results/upgrade.jsonl

pr-412: stub-2026-09-15, 40 cases x 5 trials, aggregate pass rate 0.8300
floor 0.7500: ok
baseline over 10 runs on these cases: mean 0.8910, sd 0.0209, noise used 0.0231
band 2 x noise below the baseline, lower limit 0.8448: FAIL
largest drops below each case's own history (mean, sd in history -> this run):
  refund-05      0.92 (0.10) -> 0.40
  refund-06      0.92 (0.19) -> 0.40
  refund-08      0.86 (0.17) -> 0.40
  refund-07      0.94 (0.10) -> 0.60
  refund-02      0.90 (0.14) -> 0.60
flaky in history (sd over 0.20), not ranked: escalate-05, reschedule-08
verdict: FAIL

The floor passed it, the band failed it, and the list is five refund cases, which is the shape a regression has. A case that passed every night and never passed in this run fails the gate on its own, because sampling does not produce that.

Split the suite between the pull request and the night

Twelve cases run live on every pull request and forty run every night, under the same gate. A smoke run’s baseline is the smoke cases of every night, so the two are comparable:

node --experimental-strip-types run.ts --filter smoke --seed pr-412 --out results/smoke.jsonl
node --experimental-strip-types gate.ts results/smoke.jsonl

pr-412: stub-2026-06-01, 12 cases x 5 trials, aggregate pass rate 0.8333
floor 0.7500: ok
baseline over 10 runs on these cases: mean 0.9000, sd 0.0248, noise used 0.0406
band 2 x noise below the baseline, lower limit 0.8188: ok

Sixty calls carry more noise than two hundred, so the band widens to 0.0406 from 0.0231 and the smoke gate catches only a larger drop.

jobs:
  smoke:
    if: github.event_name == 'pull_request'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
      # Twelve cases, five trials each, against the live provider. The seed is the PR number
      # plus the attempt, so a rerun of the same commit draws different samples.
      - run: node --experimental-strip-types run.ts --filter smoke --seed pr-${{ github.event.number }}-${{ github.run_attempt }} --out results/pr.jsonl
        env:
          PROVIDER_API_KEY: ${{ secrets.PROVIDER_API_KEY }}
      # Compared with the smoke cases of the committed nightly history. Exit 1 fails the check;
      # exit 2 (inconclusive) fails it too, and the log says to rerun rather than to debug.
      - run: node --experimental-strip-types gate.ts results/pr.jsonl

The nightly job in the same file runs the full suite, gates it with --record so the night joins the history, and commits history/main.jsonl back. Its schedule is 17 3 * * *, because GitHub delays schedules that land at the top of the hour. Braintrust’s bt eval --first 20 is the same split, a non-final smoke run on the pull request and the full run on merge. promptfoo’s --repeat runs each test several times under PROMPTFOO_PASS_RATE_THRESHOLD, one percentage for the run, and in Inspect AI --epochs repeats each sample and reduces the scores with mean or pass_at_{k}.

Check it worked

Sixteen tests pin the gate: the unchanged snapshot passes under a fresh seed, the regressed one fails the band while clearing the floor, and an outage is inconclusive with exit code 2.

node --experimental-strip-types --test gate.test.ts

1..16
# tests 16
# suites 0
# pass 16
# fail 0
# cancelled 0
# skipped 0
# todo 0
# duration_ms 148.739107

When it goes wrong

A regression merges under a green check. The regressed snapshot dropped six points, which a floor set fourteen points below the baseline cannot see. Three gates over the same two runs:

node --experimental-strip-types pitfall.ts

unchanged snapshot stub-2026-06-01, seed pr-412: aggregate 0.8750
  every case must pass every trial:  FAIL, 17 of 40 cases had a failing trial
  floor 0.75 alone:                 pass
  floor 0.75 plus the band:         pass
regressed snapshot stub-2026-09-15, seed pr-412: aggregate 0.8300
  every case must pass every trial:  FAIL, 23 of 40 cases had a failing trial
  floor 0.75 alone:                 pass
  floor 0.75 plus the band:         FAIL, lower limit 0.8448

Every-case gating fails the unchanged snapshot on seventeen cases, so it cannot tell the two apart. The floor alone passes both. Only the band separates them, at a limit derived rather than typed.

A red check on a day the provider’s status page is also red: sixty-one calls answered 529, and a gate that counts those as failures scores the outage.

node --experimental-strip-types run.ts --seed pr-412 --outage 0.3 --out results/outage.jsonl
node --experimental-strip-types gate.ts results/outage.jsonl

pr-412: stub-2026-06-01, 40 cases x 5 trials, aggregate pass rate 0.8512
61 of 200 calls did not complete, over the 0.1 share a rate can carry
verdict: inconclusive, rerun before reading the rate

Exit code 2 fails the check as a 1 would, and the log says rerun rather than debug. Below the share, errors are left out of the rate: a case with three passes and two errors is 1.0, not 0.6.

The band compares against a baseline weeks old. GitHub runs a schedule only on the default branch, may delay or drop it under load, and in a public repository turns it off after sixty days without activity. Keep workflow_dispatch on the workflow so a person can run the night by hand. Add a check that the newest history record is less than a week old; the gate on this page has none.

When not to do this

Do not draw a band on a suite of ten cases. At ten cases and five trials the binomial noise is about 0.045, so a band of two units spans nine points and a smaller regression is invisible. Add cases or trials instead.

Do not let the band replace the floor. A suite that loses a point a week moves its baseline a point a week, and the band follows it down. The floor is the number below which the product is broken, set from what the product needs.

Do not use this gate to decide a model upgrade. A change of snapshot is a change of baseline, so compare the two snapshots side by side on the same cases instead.

Do not treat the per-case list as a gate. One case dipping at five trials is a sampling event in most runs. Several cases of one kind near the top of the list is a regression, and a person reads that difference faster than a rule does.

Related how-tos

• Compare agent task success rates across model providers

• Run a coding agent headless in CI to keep a client current

• Cut agent cost with provider prompt caching

• Measure the token cost of an agent task against your API

• Replay recorded API traffic in an agent regression suite

Last verified

Verified 2026-10-21 against Node 22.22.0. Every output block is what the command preceding it printed. The model is a scripted stub with two snapshot ids, so no command on this page called a provider, and the workflow file was not run. The behavior of promptfoo, Braintrust, Inspect AI, and the GitHub schedule event is taken from their documentation.