# How to replay recorded API traffic in an agent regression suite

> Record an agent's tool calls once, commit the cassette without the key, and replay it in CI so an uncovered request fails instead of reaching the network.

Source: https://voxgig.com/howto/replay-recorded-api-traffic-in-agent-evals

- Audience: agent-builder
- Level: advanced
- Languages: typescript, javascript, python
- Verified: 2026-10-21
- Published: 2026-10-19

## Short answer

Run the task once against the real API with nock's recorder on, keep only the headers that decide a match, pin the scope to one host, and commit the cassette. In the suite, call `nock.disableNetConnect()`, load the cassette with `nock.define`, and listen for nock's `no match` event, because a `fetch` no interaction covers gets a 501 rather than an exception. Schedule one live run so the cassette is compared with the API it stands in for.

---
## You will need

Node 22 or later, Python 3.11 or later for the vcrpy comparison, an agent whose tools wrap one
HTTP API, and a key for that API on the day you record. Verified 2026-10-21 against Node 22.22.0,
nock 15.0.2, Python 3.13.16, and vcrpy 8.3.0. The sample's agent is scripted: its policy stands in
for the model, so the suite replays the API traffic and nothing else. A stand-in server in
`api.mjs` plays the real API during recording, so the recording itself reproduces offline; against
your API, the base URL and the key are the real ones.

## Approaches compared

| Approach | When it fits | What it costs you | When to pick something else |
| --- | --- | --- | --- |
| [nock recorder](https://github.com/nock/nock) | Node, with tools on `http` or `fetch`, and cassettes you post-process in code | A scrub step you write, and an uncovered `fetch` answered with a 501 unless the suite listens for the miss | The tools are Python or Go, or you want a HAR file other tools can read |
| [Polly.js](https://netflix.github.io/pollyjs/) | Browser or Node code on `fetch` or XHR, and recordings in HAR for other tools to read | Strict matching on headers and order by default, three packages to install, and `recordIfMissing` true until you turn it off | You need the suite to refuse an uncovered request by default |
| [The sdkgen test and netsim features](https://voxgig.com/sdk/features) | The agent's tools call an SDK that sdkgen generated, and the test needs no transport | A seed shaped as the mock expects and no recording of the real API, so a shape change upstream is invisible | The tools call the API through anything other than that SDK |
| [vcrpy](https://vcrpy.readthedocs.io/en/latest/configuration.html) | Python tools, with `filter_headers` stripping the key as the cassette is written | The body left out of the default match, so a changed payload replays the old answer until you add `body` | The suite is Node, or you want a request body mismatch caught by default |

Voxgig maintains sdkgen. It is one of four options here, not the recommendation.

The three recorders differ most where a request does not match. nock answers a `fetch` with a 501
and emits an event. vcrpy raises `CannotOverwriteExistingCassetteException` in `none` mode, and in
its default `once` mode it records when the cassette file is missing. Polly.js records whatever is
missing until `recordIfMissing` is false. The sdkgen test feature records nothing: it is a mock
seeded by you, with netsim adding failures to it, so it proves your client's handling and never the
API's shape.

## Record once, against the API you will not call again

The agent's tools are two calls over `fetch`, with the key in a header the recording must not keep.
The order id comes from the task, so `segment` escapes it into one path segment of the URL.

```js title="agent.mjs"
// An id from the task or the model becomes exactly one path segment: escaped, and never empty or
// a dot segment, which fetch would resolve to another endpoint, called with the same key.
function segment(id) {
  if (typeof id !== 'string' || id === '' || id === '.' || id === '..') throw new TypeError(`not a usable id: ${JSON.stringify(id)}`)
  return encodeURIComponent(id)
}

export function tools({ baseUrl, apiKey }) {
  const headers = { authorization: `Bearer ${apiKey}`, 'content-type': 'application/json' }
  async function call(method, path, body) {
    const res = await fetch(baseUrl + path, { method, headers, body: body ? JSON.stringify(body) : undefined })
    const json = await res.json()
    if (!res.ok) throw new ApiError(res.status, json)
    return json
  }
  return {
    get_order: async (id) => call('GET', `/v1/orders/${segment(id)}`),
    create_refund: (order_id, amount) => call('POST', '/v1/refunds', { order_id, amount }),
  }
}
```

nock's recorder leaves request headers out unless you ask, and when you ask it records
`authorization` with the rest.[^1] `record.mjs` asks, because `content-type` is worth matching on,
and then scrubs. It keeps that one header, drops the response headers that vary between runs, and
pins every interaction's scope to the host the suite will call.

```js title="record.mjs"
export function scrub(defs) {
  return defs.map((def) => {
    const reqheaders = {}
    let secrets = 0
    for (const [name, value] of Object.entries(def.reqheaders ?? {})) {
      if (name === 'authorization') secrets++
      if (KEEP_REQUEST_HEADERS.includes(name)) reqheaders[name] = value
    }
    const rawHeaders = Object.fromEntries(
      Object.entries(def.rawHeaders ?? {}).filter(([name]) => !DROP_RESPONSE_HEADERS.includes(name)),
    )
    return { ...def, scope: HOST, reqheaders, rawHeaders, secrets }
  })
}
```

```bash
node record.mjs
```

```text output
task outcome: refunded ref_7f3k2m
recorded 2 interactions against https://api.example.com
  GET /v1/orders/ord_1001 -> 200
  POST /v1/refunds {"order_id":"ord_1001","amount":1200} -> 201
authorization headers removed: 2
request headers kept: content-type
wrote cassettes/refund-task.json
```

The cassette is plain JSON, one object per interaction, and it is what `nock.define` reads back.
The `body` of the POST is recorded as an object, so the replay matches the refund amount and not
only the path.

```json title="cassettes/refund-task.json"
  {
    "scope": "https://api.example.com",
    "method": "POST",
    "path": "/v1/refunds",
    "body": {
      "order_id": "ord_1001",
      "amount": 1200
    },
    "status": 201,
    "response": {
      "id": "ref_7f3k2m",
      "order_id": "ord_1001",
      "amount": 1200,
      "status": "pending"
    },
```

Review the file before you commit it, as you would any fixture: a cassette is a copy of what the
API said, customer records included.

## Replay with the network off, and listen for the miss

`nock.disableNetConnect()` turns an unmocked host into a thrown `NetConnectNotAllowedError`. A
mocked host is different. When a `fetch` reaches `api.example.com` and no interaction matches, nock
replies with a 501 whose body carries `ERR_NOCK_NO_MATCH`, and the agent's tool sees an API error,
not a test failure.[^2] The suite listens for the `no match` event and asserts on what it collected.

```js title="replay.test.mjs"
const misses = []
nock.emitter.on('no match', (req) => misses.push(`${req.method} ${req.path ?? req.url}`))

function load() {
  nock.cleanAll()
  misses.length = 0
  if (LIVE) {
    nock.enableNetConnect(new URL(LIVE).host)
    return []
  }
  nock.disableNetConnect()
  return nock.define(JSON.parse(readFileSync(CASSETTE, 'utf8')))
}
```

The second test asks the agent about an order the cassette never saw, and pins both halves of the
behavior: the tool's 501, and the single miss the event reported.

```js title="replay.test.mjs"
test('a request the cassette does not cover fails the task instead of leaving the machine', { skip: Boolean(LIVE) }, async () => {
  load()
  await assert.rejects(
    runRefundTask({ baseUrl: base, apiKey: key, orderId: 'ord_2002' }),
    (e) => e instanceof ApiError && e.status === 501 && e.body.code === 'ERR_NOCK_NO_MATCH',
  )
  assert.deepEqual(misses, ['GET https://api.example.com/v1/orders/ord_2002'])
})
```

The first test asserts `misses` is empty and every scope `isDone()`, so a cassette with an
interaction the agent stopped making fails too. A recording the code no longer exercises is a
second kind of drift.

## Run the same task through vcrpy

vcrpy patches `http.client`, so the standard library's `urllib` is enough to compare it on the
same three questions.[^3] `vcr_modes.py` records against the stand-in with `filter_headers`, then
replays in `none` mode with another key, with another refund amount, and with a request the
cassette does not hold.

```python title="vcr_modes.py"
cassette = os.path.join(tempfile.mkdtemp(), "refund-task.yaml")
recorder = vcr.VCR(record_mode="once", filter_headers=["authorization"])
with recorder.use_cassette(cassette) as tape:
    order = call("GET", "/v1/orders/ord_1001")
    refund = call("POST", "/v1/refunds", {"order_id": order["id"], "amount": order["amount"]})
    print(f"recorded {len(tape)} interactions, refund {refund['id']}")
```

```bash
PYTHONPATH=vendor python3 vcr_modes.py
```

```text output
recorded 2 interactions, refund ref_7f3k2m
request headers written: connection, content-type, host, user-agent
replay with another key: {'id': 'ord_1001', 'amount': 1200, 'currency': 'EUR'}
POST with another amount: {'id': 'ref_7f3k2m', 'order_id': 'ord_1001', 'amount': 1200}
uncovered request: CannotOverwriteExistingCassetteException: Can't overwrite existing cassette
POST with another amount, body matched: CannotOverwriteExistingCassetteException: Can't overwrite existing cassette
```

Three answers to read. `filter_headers` removed the key on the way to disk and left the other
request headers in the file. The fourth line is the one to act on. The
[default `match_on`](https://vcrpy.readthedocs.io/en/latest/configuration.html) compares method,
scheme, host, port, path, and query, and the body is not in the list, so a refund for 9999 replayed
the recording for 1200. Adding `body` to `match_on` turns that into the same exception an uncovered
request raises.

Go has the same shape in [go-vcr](https://pkg.go.dev/gopkg.in/dnaeon/go-vcr.v4/pkg/recorder).
`ModeReplayOnly` returns `ErrInteractionNotFound` for an interaction the cassette lacks, the
default matcher takes an option to ignore the `Authorization` header, and a `BeforeSaveHook`
deletes it before the file is written. Polly.js matches on method, headers, body, and order by
default, and excludes headers by name. Its `recordIfMissing` default is `true`, so an uncovered
request in `replay` mode goes to the server and is recorded until you set it to `false`.

## Check it worked

Run the suite with the network off. Six tests cover the replay, the uncovered request, the body
mismatch, the unmocked host, an order id that tries to leave its segment, and the live run itself.
The last starts the stand-in and runs the suite live against both versions of it.

```bash
node --test replay.test.mjs
```

```text output
1..6
# tests 6
# suites 0
# pass 6
# fail 0
# cancelled 0
# skipped 0
# todo 0
# duration_ms 871.670602
```

`# pass 6` with `LIVE_API_BASE` unset means the cassette alone carried the task. Set
`LIVE_API_BASE` and `LIVE_API_KEY`, and the first test runs against the real API while the other
five skip. Live, it checks the outcome and the steps, not the recorded refund id or nock's misses:
a live API mints a new id, and nock reports every live request as unmatched.

## When it goes wrong

The suite stays green after the API changes shape. `pitfall.mjs` replays the task from the
cassette, then runs it live against version 2 of the stand-in, where the order amount moved under
a `total` object as a decimal string.

```bash
node pitfall.mjs
```

```text output
replay from the cassette: refunded ref_7f3k2m for 1200
live against the moved API: escalate (order amount missing), steps get_order
```

The cassette froze the API as it was on the day of recording, and the agent that passes against it
escalates every refund in production. The fix is a scheduled live run of the same suite, which is
what `LIVE_API_BASE` is for. `live-check.yml` runs it weekly under
[a `schedule` trigger](https://docs.github.com/en/actions/reference/workflows-and-actions/events-that-trigger-workflows)
with the key from a repository secret; copy it under `.github/workflows/`.

```yaml title="live-check.yml"
on:
  schedule:
    - cron: '17 6 * * 1'
  workflow_dispatch:
```

The tool retries the 501. A client with retry on 5xx treats nock's answer as a transient server
error, and each retry is another miss and another 501. The task ends with its retry budget spent
and a confusing error. Treat `ERR_NOCK_NO_MATCH` as fatal in the test client, or let the `no match`
listener fail the test on the first miss.

The cassette carries the key. nock's recorder wrote it because header recording was on, and a
scrub that runs after the fact is the only thing between the recording and the commit. Keep the
scrub in `record.mjs`, and add a CI step that fails when the string `authorization` appears in a
cassette.

## When not to do this

Do not record the model's own replies into the same cassette. The API exchanges change when the API
changes; a model's replies change when the provider changes a snapshot, and a cassette of both goes
stale for two unrelated reasons. Script the policy in the regression suite, as here, and evaluate
the model's choices with a separate suite.

Do not let the cassette replace the live run for good. A recording proves the agent still handles
what the API said once. The weekly run is what proves the API still says it.

Do not use the sdkgen test feature as a recorder. It is a mock seeded from data you write, and
netsim injects failures into it, so together they prove how your client behaves under a 429 or an
outage. They know nothing about the real API's current shape, and a seed is as frozen as a
cassette. Pick them when the tools call a generated SDK and the question is your code's handling,
not the wire.

Do not commit a cassette you have not read. It is a copy of what the API returned, customer names
and all, and a scrub that removes the key leaves the body as it was.

## Related how-tos

- [Stub outbound HTTP calls in Node tests with nock](/howto/stub-outbound-node-http-calls-with-nock)

- [Run an in-memory mock of your API inside SDK tests](/howto/run-an-in-memory-mock-inside-sdk-tests)

- [Mock a generated SDK in your application tests](/howto/mock-a-generated-sdk-in-application-tests)

- [Compare agent task success rates across model providers](/howto/measure-task-success-across-model-providers)

- [Simulate slow and broken networks with Toxiproxy](/howto/simulate-slow-networks-with-toxiproxy)

- [Run agent evals in CI without flaky failures](/howto/run-agent-evals-in-ci-without-flakes)

## Last verified

Verified 2026-10-21 against Node 22.22.0, nock 15.0.2, Python 3.13.16, and vcrpy 8.3.0 with
PyYAML 6.0.3 and `wrapt` 2.5.1 under `vendor/`. Every output block is what the command preceding it
printed. No request left the machine: the stand-in in `api.mjs` and the one in `vcr_modes.py`
played the API, and the model was a script. The Polly.js and go-vcr behavior is read from their
documentation and not run.

[^1]: The nock [README](https://github.com/nock/nock#readme) explains the default. Request
headers depend on timestamps and other values that change after the tests are recorded, so
recording them is deemed more trouble than it is worth. It adds that `user-agent` is never
recorded even when the option is on, because the value changes with the Node version and the
operating system. All it can indicate, the README says, is that the user agent is not the one
that recorded the tests. The authorization header gets no such exemption, which is the reason the
scrub exists.

[^2]: 501 is Not Implemented, which [RFC 9110](https://www.rfc-editor.org/rfc/rfc9110.html#name-501-not-implemented)
defines as a server that does not support what the request requires of it. A library
that has no interceptor for the request is, on a strict reading, a server in exactly that
position. The body it attaches names the condition in a way the status alone cannot.

[^3]: The vocabulary is Ruby's. The vcrpy [usage page](https://vcrpy.readthedocs.io/en/latest/usage.html)
describes its record modes as having the same behavior as Ruby's VCR, and go-vcr's
[README](https://github.com/dnaeon/go-vcr) says it was inspired by the same library. Three
languages record HTTP onto cassettes and replay them, keeping the tape metaphor well past the point
where the people reading the documentation have handled one.