# How to compare an AI-written API client with generated clients

> Score an agent-written client and three generated ones on the same calls, type probes, and mock, then measure what one schema change costs each.

Source: https://voxgig.com/howto/compare-an-ai-written-client-with-generator-output

- Audience: api-producer
- Level: advanced
- Languages: typescript, python
- Verified: 2026-11-21
- Published: 2026-11-21

## Short answer

Generate clients from the same OpenAPI document with Swagger Codegen, openapi-typescript, and sdkgen. Run each, beside the agent-written client, through one script: every operation against a stand-in, type probes compiled with tsc, a Prism mock, a 503, and a .. id. Then change one schema and measure what each takes to agree with it again, because only the generated clients have a step that does it.

---
## You will need

Node 22 or later, an OpenAPI 3 document, and the client an agent wrote from it. Swagger Codegen
needs Java 11 or later. The document here, `api.v1.json`, is a Bookshelf API with five operations
and cursor pages. This page measures TypeScript clients only, so it says nothing about Python output.
An AI coding agent, in a [Claude Code](https://code.claude.com/docs/en/overview) session, wrote
`agent/client.ts` in one pass with `api.v1.json` as its only input, before any generator ran. It is
one specimen, not edited after the measurements, and `agent/PROMPT.md` holds the prompt.

## Approaches compared

| Approach | When it fits | What it costs you | When to pick something else |
| --- | --- | --- | --- |
| [Agent-written client](https://code.claude.com/docs/en/overview) | Your codebase has conventions a generator ignores, or the document describes an endpoint badly | Every line is yours to keep in step with the spec, and nothing reports when it falls behind | The API changes often, or several languages need clients |
| [`openapi-typescript` with `openapi-fetch`](https://openapi-ts.dev/openapi-fetch/) | A TypeScript codebase that wants exact types over a thin fetch wrapper | No retry, no paging, and path ids sent as given, so each is code you add | You want the client to carry behavior, not only types |
| [sdkgen](https://github.com/voxgig/sdkgen) | Retry, paging, and one behavior across several languages matter more than size | About 7,300 lines of compiled JavaScript, the slowest regeneration here, and no page or error type | One TypeScript client where types are all you need |
| [Swagger Codegen `typescript-fetch`](https://swagger.io/docs/open-source-tools/swagger-codegen/codegen-v3/about/) | Several languages from one tool, with Java already in your build | A Java runtime, types that drop `nullable`, and a regeneration that rewrites the spec version in four file headers | You want retry or paging without writing a fetch wrapper |

Voxgig maintains sdkgen, one of the four clients measured here, and the page does not recommend
it.

The agent-written client is the shortest and the only one with nothing to regenerate, which is
also why it has no way back to the spec. `openapi-fetch` costs a single types file and regenerates in
under a second, and leaves retry and paging to you. Swagger Codegen brings Java and one types
defect. sdkgen carries the most behavior and the most code.

## Score the four clients on the same axes

`score.mjs` runs each client through the same script against `stand-in.mjs`, a local stand-in held
to the document: unknown paths get 404, other methods 405, and bodies are checked against the
schemas. It compiles six type probes per client with TypeScript, such as a book whose `published`
is a string, which the document forbids. It also reads what [Prism](https://stoplight.io/open-source/prism)
returned when `generate.sh` ran every client against it.

```bash
node score.mjs
```

```text output
agent-written
  operations that work, of 5              5
  getBook('..')                           sent /v1/
  one 503, then a 200                     failed
  books reached by its own paging, of 3   3
  type probes that agree with v1          6 of 6
  operations ok against Prism, v1         5 of 5
  lines written by hand                   94
  compiled JavaScript lines               70
Swagger Codegen
  operations that work, of 5              5
  getBook('..')                           sent /v1/
  one 503, then a 200                     failed
  books reached by its own paging, of 3   2, one page
  type probes that agree with v1          5 of 6
  operations ok against Prism, v1         5 of 5
  lines written by hand                   0
  compiled JavaScript lines               526
openapi-fetch
  operations that work, of 5              5
  getBook('..')                           sent /v1/
  one 503, then a 200                     failed
  books reached by its own paging, of 3   2, one page
  type probes that agree with v1          6 of 6
  operations ok against Prism, v1         5 of 5
  lines written by hand                   0
  compiled JavaScript lines               0, types only
sdkgen
  operations that work, of 5              5
  getBook('..')                           refused, nothing sent
  one 503, then a 200                     failed; ok with retry on
  books reached by its own paging, of 3   3, paging on
  type probes that agree with v1          4 of 4, 2 with no type
  operations ok against Prism, v1         5 of 5
  lines written by hand                   0
  compiled JavaScript lines               7294
```

Coverage does not separate them: every client covers all five operations and passes Prism. Two of
the three generated clients are thin endpoint wrappers, because Swagger Codegen and `openapi-fetch`
neither retry nor page. Both send a `..` id through, which the URL resolves
to `/v1/`.[^1] sdkgen does not retry by default either, but its `retry` and `paging` features,
added at scaffold time and switched on per client, do both, and it refused the `..` id.

The agent-written client pages through `listAllBooks`, which the prompt did not ask for, and it
sent `..` through too. Swagger Codegen's one failed probe is a last page whose
`nextCursor` is `null`: the document marks the field [`nullable`](https://spec.openapis.org/oas/v3.0.3),
and the generated type is a plain `string`. sdkgen has no type for `BookPage` or `Error`, so two
probes have nothing to compile against.

## Change one schema and measure the way back

`api.v2.json` changes `published` from an integer to a string with `format: date` in three
schemas. `generate.sh` timed each generator on it and recorded what changed. For the
agent-written client, the change is `agent/client.v2.ts`, an edit made by hand. `regen.mjs` compiles
the v2 probes against each client before and after.

```bash
node regen.mjs
```

```text output
api.v2.json: published is a date string, not an integer year
agent-written
  back in line by        edit client.ts by hand, not timed
  files changed          1, 6 lines
  v2 probes, before      4 of 6
  v2 probes, after       6 of 6
Swagger Codegen
  back in line by        generate again, 1.6 s
  files changed          4, 20 lines
  v2 probes, before      3 of 6
  v2 probes, after       5 of 6
openapi-fetch
  back in line by        openapi-typescript again, 0.7 s
  files changed          1, 13 lines
  v2 probes, before      4 of 6
  v2 probes, after       6 of 6
sdkgen
  back in line by        npm run generate, 3.6 s
  files changed          2, 9 lines
  v2 probes, before      2 of 4, 2 with no type
  v2 probes, after       4 of 4, 2 with no type
openapi-typescript --check, v1 output against api.v2.json: exit 1
```

Every client was stale before the change was applied, the generated ones included. The difference
is the step back. Each generator took under four seconds, and `openapi-typescript`'s
[`--check`](https://openapi-ts.dev/cli) exits 1 when its output no longer matches the document, so
CI can see the drift. The hand edit is six lines, made only because someone read the change. Eight of Swagger
Codegen's 20 lines are the spec version in four file headers, and sdkgen's count covers `ts/src`.

## Add retry to the two thinner generated clients

Neither generator offers retry or paging as an option. Swagger Codegen's
`config-help -l typescript-fetch` lists ten options, none of them for either, and the
[API page](https://openapi-ts.dev/openapi-fetch/api), middleware page, and examples page of
`openapi-fetch` describe neither. Both clients take a fetch function, so the substitute goes there. Swagger Codegen's
generated base class:

```ts title="generated/swagger/v1/api.ts"
export class BaseAPI {
    protected configuration: Configuration;

    constructor(configuration?: Configuration, protected basePath: string = BASE_PATH, protected fetch: FetchAPI = isomorphicFetch) {
        if (configuration) {
            this.configuration = configuration;
            this.basePath = configuration.basePath || this.basePath;
        }
    }
}
```

`openapi-fetch` takes the same function as the `fetch` option of `createClient`. `retry-fetch.mjs`
retries a 429 or 503 twice and sends a request with a body once, so a retried create cannot write
twice.

```bash
node substitute.mjs
```

```text output
Swagger Codegen
  its default fetch:   failed, status 503
  retryingFetch():     3 books, 2 requests
openapi-fetch
  its default fetch:   status 503
  retryingFetch():     3 books, 2 requests
```

Paging takes the same kind of loop the agent-written client already has, written once per client.

## Check it worked

The tests hold the stand-in to the document, then assert each scorecard row, the probes that
disagree, the ids sdkgen refuses, the stale caller, the recorded Prism runs, and the retrying fetch.

```bash
node --test compare.test.mjs
```

```text output
# tests 18
# suites 0
# pass 18
# fail 0
# cancelled 0
# skipped 0
# todo 0
```

## When it goes wrong

The day-one score is clean and the second version breaks a caller. The agent-written client has
no path back to the spec, so after the change its types still say `number`. Everything around it
stays green, and the arithmetic returns `NaN`.

```bash
node pitfall.mjs
```

```text output
the agent-written client, not edited, against api.v2.json
  tsc of a caller, with the client's own types: 0 errors
  tsc of the same caller, with openapi-typescript's v2 types: 1 error, TS2363
  getBook('b1').published at run time: "1965-01-01", a string
  2026 - published in the caller: NaN
  Prism mock of v2, the five operations: 5 of 5 ok
  Prism mock of v2, a create body with published: 1965: failed: 400
```

The Prism run passes because Prism validates the request that arrives, not the types that produced
it. Only a caller still sending an integer gets the 400. Score a second version of the document,
and compile your callers against types regenerated from it.

The `..` id reaches another endpoint. Three of the four clients sent `getBook('..')` to `/v1/`,
because escaping leaves the dots in place. Refuse an empty id, `.`, and `..` before the call.
sdkgen refused `.` and `..`, but sent an empty id as `/v1/books/`.

## When not to do this

Do not score one agent-written client and call the result a verdict on agents. This page measured
one specimen from one prompt, and a second run from the same prompt is a different client.

Do not pick sdkgen for one TypeScript client whose callers need only types. Its 7,294 compiled
lines, its features, and a 3.6 second regeneration buy retry, paging, and the same behavior in other
languages, which that client does not use.

Swagger Codegen is the stronger option when a team already runs Java and wants Go, Python, and
TypeScript clients from one tool. `openapi-fetch` suits a TypeScript
codebase that wants types to track the document exactly and keeps its own fetch wrapper.

Do not keep an agent-written client for an API that changes, unless a check like `--check` or a
regenerated type file tells you when it has fallen behind.

## Related how-tos

- [Measure drift between two AI-written clients](/howto/measure-drift-between-two-ai-written-clients)

- [Generate your first SDK with sdkgen](/howto/first-sdk-with-sdkgen)

- [Rename a generated SDK method without editing the output](/howto/rename-generated-methods-without-editing-output)

- [Let a coding agent customize a generated SDK safely](/howto/customise-a-generated-sdk-with-an-agent-safely)

- [Score SDK generators on your API with a repeatable rubric](/howto/score-sdk-generators-with-a-repeatable-rubric)

## Last verified

Verified 2026-11-21 against Node 22.23.5, `@voxgig/create-sdkgen` 0.31.9, `@voxgig/sdkgen`
4.44.3, Swagger Codegen 3.0.85 on OpenJDK 21.0.13, `openapi-typescript` 7.13.0, `openapi-fetch`
0.17.0, Prism 5.14.2, and TypeScript 5.9.3. Generation, its timings, and both Prism runs happened
once in `generate.sh`, with network access, and their output is committed under `generated/`. The
gate re-runs the commands on this page against the committed clients and a local stand-in.

[^1]: [RFC 3986](https://www.rfc-editor.org/rfc/rfc3986.html#section-5.2.4) gives the algorithm,
`remove_dot_segments`, that turns `/v1/books/..` into `/v1/`. [`encodeURIComponent`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/encodeURIComponent)
leaves a dot alone, because the dot is one of the characters it never escapes, so the escaped id is still a
dot segment.