# How to run agent evals in CI without flaky failures > Rerun each eval case, gate the pull request on the aggregate against a band from nightly history, and keep outages and ordinary variance from reading as regressions. Source: https://voxgig.com/howto/run-agent-evals-in-ci-without-flakes - Audience: platform-team - Level: intermediate - Languages: typescript - Verified: 2026-10-21 - Published: 2026-10-19 ## Short answer Pin the model snapshot and run every case five times. Gate the pull request on the aggregate pass rate against a tolerance band drawn from the nightly history of the same cases, not on every case passing. Treat provider errors as missing evidence that makes the run inconclusive. Run a twelve-case smoke suite per pull request and the full suite nightly, and record each night so a drop can be read against its own history. --- ## You will need Node 22 or later, an eval suite whose cases name the tool call a correct run makes, and a CI system with a schedule trigger. Verified 2026-10-21 against Node 22.22.0, with no packages beyond the standard library. The model on this page is a scripted stub with two dated snapshot ids, so the numbers show what the gate does and not how any provider behaves. ## Approaches compared | Approach | When it fits | What it costs you | When to pick something else | | --- | --- | --- | --- | | [Live provider run](https://developers.openai.com/api/reference/resources/chat/subresources/completions/methods/create) | The change touches a prompt, a tool description, or the loop, and only the model can say what it did to them | Money and minutes on every push, sampling noise on every rate, and a red check whenever the provider has a bad hour | The change cannot alter model behavior, such as a refactor of the harness | | [Mocked run with scripted responses](https://nodejs.org/api/test.html#mocking) | Changes to the harness, the grader, or the gate, where the question is whether the code reads a fixed answer right | Deterministic by construction, so it cannot see a model regression or a prompt the model reads differently | The change is to a prompt or a tool description | | [Nightly full suite plus a per-PR smoke suite](https://docs.github.com/en/actions/reference/workflows-and-actions/events-that-trigger-workflows#schedule) | A suite too large or too expensive to run live on every push | A regression in a case outside the smoke set is found the next morning, after the merge, by a schedule that can itself be delayed or turned off | The whole suite runs live in a few minutes for a few cents | | [promptfoo in GitHub Actions](https://www.promptfoo.dev/docs/integrations/github-action/) | Prompt-centric suites in YAML, with a before and after comparison posted to the pull request | The pass-rate threshold is one percentage for the whole run, set in an environment variable, with no band from history behind it | You want a verdict that reads this run against the last ten | A mocked run passes or fails for a reason you can read in the diff, because the model's answers are fixed, which is also why it cannot see the model regress. A live run sees the model and pays for it. promptfoo's `--repeat` buys the trials under one threshold for the run. The split keeps the live bill to twelve cases per push and finds a regression in the other twenty-eight after the merge rather than before it. ## Pin what the provider lets you pin, then repeat The configuration names a dated snapshot, a temperature, and a trial count, and the runner reads nothing else about the model. A floating alias resolves to a different snapshot on the day the provider decides, so a drop under an alias can be their change or yours. ```json title="eval.config.json" { "model": "stub-2026-06-01", "temperature": 0, "trials": 5, "cases": "suite/cases.jsonl", "floor": 0.75, "sigma": 2, "maxErrorShare": 0.1 } ``` Temperature and seed are weaker than they look. OpenAI's reference documents `seed` as a best effort at determinism and marks it deprecated, along with the `system_fingerprint` field it points to. Anthropic's reference says [`temperature`](https://platform.claude.com/docs/en/api/messages) `0.0` was never fully deterministic, and that models released after Claude Opus 4.6 reject any value other than `1.0`.[^1] The lever that works at every provider is repetition. Five trials per case, under a seed of your own that names the run, make each rate a measurement with noise you can estimate. ```ts title="run.ts" // A pass is the one expected call with exactly the expected arguments. Anything else is a reason // the report can count, because "no tool call" and "wrong arguments" are different regressions. // Arguments compare as values, not as JSON text: a model may emit the keys in any order. export function grade(c: Case, call: Call | null): 'pass' | string { if (!call) return 'no tool call' if (call.tool !== c.expect.tool) return `called ${call.tool}` if (!isDeepStrictEqual(call.args, c.expect.args)) return 'wrong arguments' return 'pass' } export function runSuite(o: { config: Config; cases: Case[]; seed: string; filter: string; model?: string; outage?: number }): Run { const model = o.model ?? o.config.model const results: CaseResult[] = [] for (const c of o.cases) { const r: CaseResult = { id: c.id, kind: c.kind, suite: c.suite, passes: 0, fails: 0, errors: 0, reasons: [] } for (let trial = 1; trial <= o.config.trials; trial++) { const reply = complete({ model, c, trial, seed: o.seed, temperature: o.config.temperature, outage: o.outage }) if (reply.status === 'error') { r.errors++; continue } const verdict = grade(c, reply.call) if (verdict === 'pass') r.passes++ else { r.fails++; r.reasons.push(verdict) } } results.push(r) } return { run: o.seed, model, temperature: o.config.temperature, trials: o.config.trials, filter: o.filter, cases: results } } ``` The grader is strict and records why a trial failed, because a run that stopped calling tools and a run that acts on the wrong order id are different regressions. ```bash node --experimental-strip-types run.ts --seed pr-412 --out results/pr.jsonl ``` ```text output run pr-412: stub-2026-06-01, temperature 0, 40 cases x 5 trials = 200 calls pass 175 fail 25 error 0 failures: called get_order 10, wrong arguments 10, no tool call 5 aggregate pass rate 0.8750 wrote results/pr.jsonl ``` Twenty-five failed trials out of two hundred, from a snapshot that has not changed. A gate that wanted every case green would have failed this run on seventeen cases, which is the flakiness in the title. The noise is in the model, so a rerun samples it again. ## Build the baseline from the nights The baseline is the suite's own history on main. `make-history.ts` plays ten nightly runs of the pinned snapshot, decides each against the nights before it, and appends the verdict and every case's rate to `history/main.jsonl`. In CI the nightly job does the same with `gate.ts --record`. ```bash node --experimental-strip-types make-history.ts ``` ```text output main-2026-09-01: pass, aggregate 0.8800, lower limit none yet main-2026-09-02: pass, aggregate 0.8850, lower limit none yet main-2026-09-03: pass, aggregate 0.9050, lower limit none yet main-2026-09-04: pass, aggregate 0.9100, lower limit 0.8389 main-2026-09-05: pass, aggregate 0.9250, lower limit 0.8465 main-2026-09-06: pass, aggregate 0.8750, lower limit 0.8547 main-2026-09-07: FAIL, aggregate 0.8500, lower limit 0.8502 main-2026-09-08: pass, aggregate 0.8950, lower limit 0.8362 main-2026-09-09: pass, aggregate 0.9000, lower limit 0.8411 main-2026-09-10: pass, aggregate 0.8850, lower limit 0.8451 wrote 10 runs to history/main.jsonl ``` Night seven failed its own gate by two ten-thousandths, on a snapshot that had not changed. A band two units of noise wide fails about one unchanged run in forty by construction, and that night was the one.[^2] The record keeps it, because a baseline built only from the nights that passed drifts upward while its spread shrinks. ## Gate on the aggregate against the band ```ts title="gate.ts" // The baseline is every completed run in history, each re-averaged over the cases this run has, so // a smoke run is compared with the smoke cases of every nightly run and not with the whole suite. // Failed nights stay in: a baseline built from the nights that passed drifts upward and its spread // shrinks, and the band then fails more runs for nothing. A night that failed is main's problem, // and the record says so. export function baseline(history: HistoryRun[], ids: string[]) { const rates: number[] = [] for (const h of history) { const xs = ids.map((id) => h.cases[id]).filter((x): x is number => x !== undefined) if (xs.length) rates.push(mean(xs)) } return rates.length < 3 ? null : { n: rates.length, mean: mean(rates), sd: sd(rates) } } // The spread the band is drawn from. Ten nights estimate the between-run spread badly, and the // run being judged has sampling noise of its own that no history can remove, so the band is never // narrower than the binomial noise of this many calls, and it widens for the error in the mean. export function noise(b: { n: number; mean: number; sd: number }, calls: number): number { const binomial = Math.sqrt((b.mean * (1 - b.mean)) / calls) return Math.max(b.sd, binomial) * Math.sqrt(1 + 1 / b.n) } ``` The baseline mean and standard deviation come from the ten nights, re-averaged over the cases this run has. The noise is the larger of that standard deviation and the binomial noise of this many calls, widened for the error in the mean, because ten nights estimate a spread badly. ```bash node --experimental-strip-types gate.ts results/pr.jsonl ``` ```text output pr-412: stub-2026-06-01, 40 cases x 5 trials, aggregate pass rate 0.8750 floor 0.7500: ok baseline over 10 runs on these cases: mean 0.8910, sd 0.0209, noise used 0.0231 band 2 x noise below the baseline, lower limit 0.8448: ok largest drops below each case's own history (mean, sd in history -> this run): refund-02 0.90 (0.14) -> 0.40 escalate-03 0.82 (0.20) -> 0.40 refund-04 0.90 (0.14) -> 0.60 escalate-07 0.88 (0.14) -> 0.60 reschedule-03 0.88 (0.14) -> 0.60 flaky in history (sd over 0.20), not ranked: escalate-05, reschedule-08 verdict: pass ``` The aggregate sits a point and a half under the baseline and inside the band. The ranked list is for the reviewer: one case from each kind near the top is an ordinary Tuesday. Two cases that move by more than `0.20` between nights are left out, because one run of theirs says nothing. The second snapshot handles refunds worse and everything else the same: ```bash node --experimental-strip-types run.ts --seed pr-412 --model stub-2026-09-15 --out results/upgrade.jsonl node --experimental-strip-types gate.ts results/upgrade.jsonl ``` ```text output pr-412: stub-2026-09-15, 40 cases x 5 trials, aggregate pass rate 0.8300 floor 0.7500: ok baseline over 10 runs on these cases: mean 0.8910, sd 0.0209, noise used 0.0231 band 2 x noise below the baseline, lower limit 0.8448: FAIL largest drops below each case's own history (mean, sd in history -> this run): refund-05 0.92 (0.10) -> 0.40 refund-06 0.92 (0.19) -> 0.40 refund-08 0.86 (0.17) -> 0.40 refund-07 0.94 (0.10) -> 0.60 refund-02 0.90 (0.14) -> 0.60 flaky in history (sd over 0.20), not ranked: escalate-05, reschedule-08 verdict: FAIL ``` The floor passed it, the band failed it, and the list is five refund cases, which is the shape a regression has. A case that passed every night and never passed in this run fails the gate on its own, because sampling does not produce that. ## Split the suite between the pull request and the night Twelve cases run live on every pull request and forty run every night, under the same gate. A smoke run's baseline is the smoke cases of every night, so the two are comparable: ```bash node --experimental-strip-types run.ts --filter smoke --seed pr-412 --out results/smoke.jsonl node --experimental-strip-types gate.ts results/smoke.jsonl ``` ```text output pr-412: stub-2026-06-01, 12 cases x 5 trials, aggregate pass rate 0.8333 floor 0.7500: ok baseline over 10 runs on these cases: mean 0.9000, sd 0.0248, noise used 0.0406 band 2 x noise below the baseline, lower limit 0.8188: ok ``` Sixty calls carry more noise than two hundred, so the band widens to `0.0406` from `0.0231` and the smoke gate catches only a larger drop. ```yaml title="eval.yml" jobs: smoke: if: github.event_name == 'pull_request' runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-node@v4 with: node-version: 22 # Twelve cases, five trials each, against the live provider. The seed is the PR number # plus the attempt, so a rerun of the same commit draws different samples. - run: node --experimental-strip-types run.ts --filter smoke --seed pr-${{ github.event.number }}-${{ github.run_attempt }} --out results/pr.jsonl env: PROVIDER_API_KEY: ${{ secrets.PROVIDER_API_KEY }} # Compared with the smoke cases of the committed nightly history. Exit 1 fails the check; # exit 2 (inconclusive) fails it too, and the log says to rerun rather than to debug. - run: node --experimental-strip-types gate.ts results/pr.jsonl ``` The nightly job in the same file runs the full suite, gates it with `--record` so the night joins the history, and commits `history/main.jsonl` back. Its schedule is `17 3 * * *`, because GitHub [delays schedules](https://docs.github.com/en/actions/reference/workflows-and-actions/events-that-trigger-workflows#schedule) that land at the top of the hour. Braintrust's [`bt eval --first 20`](https://www.braintrust.dev/docs/evaluate/run-in-ci) is the same split, a non-final smoke run on the pull request and the full run on merge. promptfoo's [`--repeat`](https://www.promptfoo.dev/docs/usage/command-line/) runs each test several times under `PROMPTFOO_PASS_RATE_THRESHOLD`, one percentage for the run, and in Inspect AI [`--epochs`](https://inspect.aisi.org.uk/options.html) repeats each sample and reduces the scores with `mean` or `pass_at_{k}`. ## Check it worked Sixteen tests pin the gate: the unchanged snapshot passes under a fresh seed, the regressed one fails the band while clearing the floor, and an outage is inconclusive with exit code `2`. ```bash node --experimental-strip-types --test gate.test.ts ``` ```text output 1..16 # tests 16 # suites 0 # pass 16 # fail 0 # cancelled 0 # skipped 0 # todo 0 # duration_ms 148.739107 ``` ## When it goes wrong A regression merges under a green check. The regressed snapshot dropped six points, which a floor set fourteen points below the baseline cannot see. Three gates over the same two runs: ```bash node --experimental-strip-types pitfall.ts ``` ```text output unchanged snapshot stub-2026-06-01, seed pr-412: aggregate 0.8750 every case must pass every trial: FAIL, 17 of 40 cases had a failing trial floor 0.75 alone: pass floor 0.75 plus the band: pass regressed snapshot stub-2026-09-15, seed pr-412: aggregate 0.8300 every case must pass every trial: FAIL, 23 of 40 cases had a failing trial floor 0.75 alone: pass floor 0.75 plus the band: FAIL, lower limit 0.8448 ``` Every-case gating fails the unchanged snapshot on seventeen cases, so it cannot tell the two apart. The floor alone passes both. Only the band separates them, at a limit derived rather than typed. A red check on a day the provider's status page is also red: sixty-one calls answered `529`, and a gate that counts those as failures scores the outage. ```bash node --experimental-strip-types run.ts --seed pr-412 --outage 0.3 --out results/outage.jsonl node --experimental-strip-types gate.ts results/outage.jsonl ``` ```text output pr-412: stub-2026-06-01, 40 cases x 5 trials, aggregate pass rate 0.8512 61 of 200 calls did not complete, over the 0.1 share a rate can carry verdict: inconclusive, rerun before reading the rate ``` Exit code `2` fails the check as a `1` would, and the log says rerun rather than debug. Below the share, errors are left out of the rate: a case with three passes and two errors is `1.0`, not `0.6`. The band compares against a baseline weeks old. GitHub [runs a schedule](https://docs.github.com/en/actions/reference/workflows-and-actions/events-that-trigger-workflows#schedule) only on the default branch, may delay or drop it under load, and in a public repository turns it off after sixty days without activity. Keep `workflow_dispatch` on the workflow so a person can run the night by hand. Add a check that the newest history record is less than a week old; the gate on this page has none. ## When not to do this Do not draw a band on a suite of ten cases. At ten cases and five trials the binomial noise is about `0.045`, so a band of two units spans nine points and a smaller regression is invisible. Add cases or trials instead. Do not let the band replace the floor. A suite that loses a point a week moves its baseline a point a week, and the band follows it down. The floor is the number below which the product is broken, set from what the product needs. Do not use this gate to decide a model upgrade. A change of snapshot is a change of baseline, so compare the two snapshots side by side on the same cases instead. Do not treat the per-case list as a gate. One case dipping at five trials is a sampling event in most runs. Several cases of one kind near the top of the list is a regression, and a person reads that difference faster than a rule does. ## Related how-tos - [Compare agent task success rates across model providers](/howto/measure-task-success-across-model-providers) - [Run a coding agent headless in CI to keep a client current](/howto/run-a-coding-agent-headless-in-ci) - [Cut agent cost with provider prompt caching](/howto/cut-agent-cost-with-prompt-caching) - [Measure the token cost of an agent task against your API](/howto/measure-token-cost-per-agent-task) - [Replay recorded API traffic in an agent regression suite](/howto/replay-recorded-api-traffic-in-agent-evals) ## Last verified Verified 2026-10-21 against Node 22.22.0. Every output block is what the command preceding it printed. The model is a scripted stub with two snapshot ids, so no command on this page called a provider, and the workflow file was not run. The behavior of promptfoo, Braintrust, Inspect AI, and the GitHub schedule event is taken from their documentation. [^1]: The two references retire the same knob by different routes. OpenAI's lists [`seed`](https://developers.openai.com/api/reference/resources/chat/subresources/completions/methods/create) as in Beta and deprecated in the same breath, promises a best effort at determinism, and refers the reader to `system_fingerprint`, which the page also marks deprecated. Anthropic's marks [`temperature`](https://platform.claude.com/docs/en/api/messages) deprecated for models released after Claude Opus 4.6 and rejects every value but `1.0` with a `400`. Under that notice the page still says that even `0.0` never made results fully deterministic, so the parameter leaves with its caveat intact. [^2]: The band is the lower control limit of a control chart. The [NIST/SEMATECH handbook](https://www.itl.nist.gov/div898/handbook/pmc/section3/pmc31.htm) describes limits chosen so that almost all points fall inside them while the process is in control. It calls the customary two-in-a-thousand risk "a purely arbitrary number" and leaves the choice to how much risk the management of the quality program will take. Two units of noise accept a higher false-alarm rate than the customary three, on purpose: a regression that slips through costs more than a rerun, and the rerun is one click.