Your agent’s test sends its prompt to the real model and its three tool calls to the live help desk API. Every CI run closes a real ticket and spends tokens. The assertion on which operations ran, and in what order, fails whenever the model answers the same prompt another way. A red run cannot tell a broken loop from a model that chose a different path.
What you get
You will end up with a test that runs an agent loop end to end against a scripted model and a mock API, then asserts each operation in order. This is for you if your agent calls an API through a generated SDK.
Short answer
Keep the loop and replace both ends. Script the model’s turns as data behind the Anthropic SDK’s
fetch option, then build the tools on sdkgen’s HelpdeskSDK.test(seed) for an in-memory run, or
give the client a transport-level mock. Record each tool call and assert the list. A guard on
globalThis.fetch turns a forgotten injection into a refusal rather than a call to a live service.
You will need
Node 22 or later, an OpenAPI 3 document for the API the agent calls, and an SDK generated from
it. The sample’s api.json describes a help desk API with three operations and a bearer scheme.
The loop follows the hand-written one on
the tool-calling loop page, with plain JSON
Schema in place of Zod. This page tests the loop and the tools, with the model replaced: no captured
command calls a model, and nothing leaves the process.
Approaches compared
| Approach | When it fits | What it costs you | When to pick something else |
|---|---|---|---|
| Hey API MSW plugin | You generate with Hey API and want typed handlers per operation | A plugin its documentation marks In Progress, and msw as a dependency the output imports | You want writes to show in later reads without writing the resolver |
| MSW | Any client that calls fetch, with handlers shared between Node and a browser | Handlers written per route by hand, and a patched global fetch every client shares | The SDK has its own mode that skips the transport |
| openapi-typescript and openapi-fetch | Types-only generation, and a test that can hand the client a fake fetch | No mock is generated, so the stand-in and its routes are yours to keep in step | You want handlers derived from the document |
| Prism | A mock process that validates each request against the document, for any language | Answers built from examples or schemas, so a write never shows in the next read | The assertion is about state the agent changed |
| sdkgen test feature | The SDK comes from sdkgen and the test is about the loop, not the wire | No request is built, so a missing token or a wrong header passes unseen | You need to assert on headers, status codes, or the body |
Voxgig maintains sdkgen, the worked example here and one of five rows, none of them the recommendation.
The split is fidelity against setup. A transport-level mock, whether MSW, a fake fetch, or
WireMock for a JVM shop, sees the request your client built. It
catches a header or serialization bug, and costs you a stand-in to write. Prism checks requests
against the document with no stand-in to write, and gives back placeholders.1 An in-memory
mode costs the least and sees the least.
Script the model’s turns as data
The model is the part that makes the assertion flaky. The Anthropic SDK takes a fetch option,
so model.mjs answers each POST /v1/messages from a list of turns. A turn can say what it
expects in the tool result it was sent. If the loop or the tools change, the script refuses with
a 400 rather than feeding its next answer to the wrong question.
export const closeOldestBillingTicket = [
{ say: { tool: 'list_tickets', input: {} } },
{ expect: { tool: 'list_tickets', contains: '"id":"tk-102"' }, say: { tool: 'get_ticket', input: { id: 'tk-102' } } },
{
expect: { tool: 'get_ticket', contains: '"status":"open"' },
say: { tool: 'update_ticket', input: { id: 'tk-102', status: 'closed', note: 'Refund issued.' } },
},
{ expect: { tool: 'update_ticket', contains: '"status":"closed"' }, say: { text: 'Closed tk-102, the oldest open billing ticket, with a note that the refund was issued.' } },
]
A script records one plausible path, not what the model will do.2 Write it from a transcript of a real run, then keep it fixed, so the test checks the loop and the tools against that path.
Serve the tools from the generated test mode
Scaffold with the test feature, then build the client with the static test() the generated SDK
exposes. The features reference
says load, list, create, update, patch and remove are served from the seed map, and
that the live transport refuses to send anything while the feature is active.
npx --yes @voxgig/create-sdkgen@0.31.9 helpdesk --def api.json --target ts --feature test
// sdkgen test mode: the generated test feature answers from the seed, and no request is built.
export function sdkgenTestMode({ token = TOKEN } = {}) {
const seed = { entity: { ticket: Object.fromEntries(seedTickets().map((t) => [t.id, t])) } }
const sdk = HelpdeskSDK.test(seed, { apikey: token })
return {
label: 'sdkgen, HelpdeskSDK.test(seed)',
client: sdkgenClient(sdk),
requests: [],
status: async () => (await sdk.Ticket().load({ id: 'tk-102' })).data().status,
close: async () => {},
}
}
The tools in tools.mjs call sdk.Ticket().list(), load() and update(), and push each call
onto a list before it runs. The second setup builds the same SDK in live mode with the stand-in in
helpdesk.mjs as its system.fetch, so every request is built and sent.
node demo.mjs sdkgenTestMode sdkgenLive
sdkgen, HelpdeskSDK.test(seed)
tools list_tickets {}
get_ticket {"id":"tk-102"}
update_ticket {"id":"tk-102","status":"closed","note":"Refund issued."}
requests none built
tk-102 closed
model 4 requests, stop_reason end_turn
sdkgen, new HelpdeskSDK({ system: { fetch } })
tools list_tickets {}
get_ticket {"id":"tk-102"}
update_ticket {"id":"tk-102","status":"closed","note":"Refund issued."}
requests GET /tickets, GET /tickets/tk-102, PATCH /tickets/tk-102
tk-102 closed
model 4 requests, stop_reason end_turn
requests refused by the network guard: 0
Both runs make the same three calls and leave tk-102 closed. Only the live run built requests,
and only the stand-in checked the Authorization: Bearer header on each.
Mock the same tools in Hey API and openapi-fetch
Hey API’s answer is its msw plugin,
added to plugins in the configuration beside the fetch client and the SDK. It writes one
handler factory per operation into msw.gen.ts, typed by the document.3 A generated handler
answers 501 until you give it a body or a resolver, which a test checks, so the sample hands each
one the same stand-in.
// Hey API: the handlers its msw plugin generated, each one answered by the stand-in.
export function heyApiMsw({ token = TOKEN } = {}) {
const helpdesk = createHelpdesk()
const answer = ({ request }) => helpdesk.handle(request)
const handlers = createMswHandlers({ baseUrl: BASE }).all({
pick: { listTickets: answer, getTicket: answer, updateTicket: answer },
})
const server = setupServer(...handlers)
server.listen({ onUnhandledRequest: 'error' })
const client = createHeyApiClient(createConfig({ baseUrl: BASE, auth: token ? () => token : undefined }))
openapi-typescript generates types and no functions, so there is no mock to generate. Its
testing page passes a spy as the fetch option
of createClient, and recommends MSW for responses. The sample passes the stand-in’s fetch.
node demo.mjs heyApiMsw openapiFetch
Hey API, generated MSW handlers
tools list_tickets {}
get_ticket {"id":"tk-102"}
update_ticket {"id":"tk-102","status":"closed","note":"Refund issued."}
requests GET /tickets, GET /tickets/tk-102, PATCH /tickets/tk-102
tk-102 closed
model 4 requests, stop_reason end_turn
openapi-fetch, createClient({ fetch })
tools list_tickets {}
get_ticket {"id":"tk-102"}
update_ticket {"id":"tk-102","status":"closed","note":"Refund issued."}
requests GET /tickets, GET /tickets/tk-102, PATCH /tickets/tk-102
tk-102 closed
model 4 requests, stop_reason end_turn
requests refused by the network guard: 0
Hey API gets credentials through the auth callback its
Fetch client page documents, and
openapi-fetch through default headers. Both runs match the live sdkgen run request for request.
Check it worked
Four of the fourteen tests run the loop over one setup each and assert the calls, the requests, and the closed ticket. The other ten cover the claims on this page.
node --test loop.test.mjs
1..14
# tests 14
# suites 0
# pass 14
# fail 0
# cancelled 0
# skipped 0
# todo 0
# duration_ms 476.846982
One test gives sdkgen’s test mode a live fetch and asserts it is never called. Another gets a
model id of .. or tk-101/../../admin and asserts the stand-in saw no request, because the tool
checks the id is one path segment first. A third gives openapi-fetch a bodySerializer that
form-encodes the PATCH, and the stand-in answers 400.
When it goes wrong
The suite is green and production answers 401. A test-mode client was built without its token, which went unnoticed because that mode builds no request. Keep one transport-level run of the same script:
node pitfall.mjs
the client is built with no token
sdkgen, HelpdeskSDK.test(seed) passed, tk-102 closed
sdkgen, new HelpdeskSDK({ system: { fetch } }) failed: list_tickets: HelpdeskSDK: list: request: 401:
Hey API, generated MSW handlers failed: list_tickets: listTickets failed with 401: unauthorized
openapi-fetch, createClient({ fetch }) failed: list_tickets: GET /tickets failed with 401: unauthorized
the loop is given a client with no scripted fetch
APIConnectionError: Connection error.
refused by the guard: 3 x POST https://api.anthropic.com/v1/messages
The test calls the real model. One construction of new Anthropic() without the scripted fetch
reaches the model on any machine with a key in its environment. Under the guard it fails instead,
after the SDK’s default two retries, so install the guard in every agent test.
The script refuses turn 3. A tool changed what it returns, and the turn’s expect no longer
matches. Update the script from a fresh transcript; do not loosen the expect.
When not to do this
Do not treat sdkgen’s test mode as the whole suite. It is the cheapest of the five because it skips the transport, which is also why the missing token passed. Pair it with a transport-level run, or pick Hey API’s generated handlers when that client is already Hey API and you want the mock typed by the same document.
Skip Prism when the agent’s next step depends on its last write. Its answers come from the
document, so a write does not show in the next read. The same loop stopped at turn 2 there,
because Prism’s list held no tk-102, as prism-run.txt records. Choose it when request
validation against the document matters more than state.
Do not record real model turns on every run. A script is a fixed point; a recording refreshed
each time is the flaky test again. openapi-fetch suits the reader who generates types only and
already owns a fetch stand-in.
Related how-tos
Last verified
Verified 2026-11-21 against Node 22.23.5, @voxgig/create-sdkgen 0.31.9, @voxgig/sdkgen 4.44.3,
@voxgig/apidef 8.32.4, @hey-api/openapi-ts 0.99.0, openapi-typescript 7.13.0,
openapi-fetch 0.17.0, msw 2.15.0, and @anthropic-ai/sdk 1.12.0. Every output block is what
the command preceding it printed. All three generators ran once, in generate.sh, and their
output is committed under generated/; the captured commands read it. Prism 5.16.0 ran once, in
record-prism.sh: five requests, then one run of the loop. No request reached the Claude API or any host outside the process.
Footnotes
-
With no examples in the document, Prism answered every field of a ticket with the word
string, the name of the field’s type. That is the static mode its mocking guide describes as the default. A PATCH that setstatustoclosedcame back as"status":"string"too, and so did the read after it. ↩︎ Back to text -
Anthropic’s glossary, under Temperature, says that even with temperature set to 0 the results will not be fully deterministic, and that identical inputs might produce different outputs across API calls. It adds that this applies to Anthropic’s own inference service and to third-party cloud providers alike. ↩︎ Back to text
-
The plugin page names each handler creator
<operationId>Mock, the collectiongetAllMocks, and the field of a static responseresult. From this page’s document the release on npm, 0.99.0, wrotehandleListTickets,createMswHandlers, andbody. The page says In Progress at the top, in a badge, before any of the three names. ↩︎ Back to text