# How to contract test a generated SDK against its spec

> Run one operation list through each generated client against a document mock and a sandbox, then rename a field in the spec and see which runs fail.

Source: https://voxgig.com/howto/contract-test-a-generated-sdk

- Audience: library-maintainer
- Level: advanced
- Languages: typescript, go, python
- Verified: 2026-11-22
- Published: 2026-11-22

## Short answer

Run the same operation list through each generated client twice: against a mock that knows only the OpenAPI document, then against a sandbox, where a wrong document shows. In an sdkgen project the shared cases are .sdk/test/entity/<entity>/<Entity>TestData.json, read by every target's entity test, and BOOKSHELF_TEST_LIVE=TRUE sends them to a server. A renamed required field fails only live. A renamed optional field fails only with the validate feature checking responses.

---
## You will need

Node 22 or later, Go 1.25 or later, Python 3.12 or later with pytest, a generated SDK in at least
two languages, and an OpenAPI 3 document. The sample's `api.json` has five Bookshelf operations
and a `Book` that requires `id`, `title` and `author`. There are no real credentials here, so the
sandbox is `sandbox.mjs`, a local stand-in that implements `api.json` and seeds one book. It plays
the deployed API, so it disagrees with a renamed document in that one place.

## Approaches compared

| Approach | When it fits | What it costs you | When to pick something else |
| --- | --- | --- | --- |
| [Hey API with Zod validators](https://heyapi.dev/docs/openapi/typescript/plugins/sdk) | A TypeScript client whose calls should refuse a response the document does not describe | Validation is off by default, and every response pays for a parse | You ship more than TypeScript from one test set |
| [`ogen`](https://ogen.dev/docs/intro/) | A Go client that refuses a response missing a required field | Go only, and no generated test runs the operations against a server | You need the same cases in Python and TypeScript |
| [Pact consumer tests](https://docs.pact.io/) | A provider team that replays what your SDK sends in its own build | Covers only the calls consumers record, and the pact files must reach the provider's build | The API is public and its team runs nothing of yours |
| [Per-language test files](https://docs.python.org/3/library/unittest.html) | One language, or a small API that rarely changes | Each language's assertions drift once one copy is edited by hand | You ship the same operations in three languages |
| [sdkgen generated suites](https://github.com/voxgig/sdkgen/blob/main/docs/how-to/run-a-live-suite.md) | One set of entity cases, run by the TypeScript, Go and Python suites alike | Offline runs follow the document, and an optional rename passes live unless `validate` is added | Your clients are not generated by sdkgen |

Voxgig maintains sdkgen, and this page treats it as one of the five ways in.

A shared case file reaches every language at the price of a runner in each, which sdkgen
generates and a hand-rolled JSON runner would have to write. Pact sees only the calls a consumer
makes, and its pact files have to be
[shared with the provider](https://docs.pact.io/getting_started/sharing_pacts). Per-language files
drift. `ogen` and Hey API check each response's shape at run time,
in one language each.

## Find where the shared cases live

`generate.sh` scaffolded a project with `-t ts -t go -t py` and recorded where each target's
suite reads its cases from.

```bash
node report.mjs corpus
```

```text output
.sdk/test/feature/:
  cost.aontu
  feature-test-index.aontu
.sdk/test/test.json, feature sections: cost
.sdk/test/entity/:
  book/BookTestData.json
test files that name BookTestData.json:
  go/test/book_entity_test.go
  py/test/test_book_entity.py
  ts/test/entity/book/BookEntity.test.ts
test files that name the path .sdk/test/test.json:
  go/test/omniresolver_test.go
  go/test/struct_utility_test.go
  go/test/testsupport_test.go
  py/test/omni.py
  py/test/test_feature_corpus.py
  py/test/test_primary_utility.py
  py/test/test_struct_utility.py
  ts/test/utility/index.ts
definition suites (read the document, not the model):
  ts/test/definition-runner.ts
  ts/test/definition.test.ts
```

The feature corpus holds `cost` alone, and its cases test the SDK's own features, not the API.
The cases about the API are in `BookTestData.json`: the records and the request each operation
expects, written by `npm run generate` and read by all three entity tests. The definition suite,
which reads the document rather than the model, exists for TypeScript only, as the
[definition how-to](https://github.com/voxgig/sdkgen/blob/main/docs/how-to/check-against-the-definition.md)
says of its `ts` and `js` targets. That page also calls them the only two with a live suite, but
the Go and Python suites below sent their flows to the sandbox too.

## Run each suite offline, then live

Offline, an entity test answers every call from those fixtures. Live, the same test sends the
create, list, update, load and remove to the server the document names. `generate.sh` runs it
both ways, starting a fresh sandbox each time:

```bash title="generate.sh"
run_suite() { # <project> <lang> <mode> <sandbox host>
  local p=$1 l=$2 m=$3 h=$4
  local env=(BOOKSHELF_TEST_LIVE=FALSE)
  [ "$m" = live ] && env=(BOOKSHELF_TEST_LIVE=TRUE "BOOKSHELF_SERVER_SANDBOX=$h" BOOKSHELF_APIKEY=sandbox-key)
  case $l in
    ts) (cd "$p/ts" && env "${env[@]}" node --test --test-reporter=tap dist-test/entity/book/BookEntity.test.js) ;;
    go) (cd "$p/go" && env "${env[@]}" go test -count=1 -run TestBookEntity ./test/) ;;
    py) (cd "$p/py" && env "${env[@]}" python3 -m pytest -q test/test_book_entity.py) ;;
  esac
```

`BOOKSHELF_SERVER_SANDBOX` fills the `{sandbox}` variable in the document's server URL, which
is how a live run reaches a sandbox without editing the SDK.

## Rename one field and read which runs fail

`rename-field.mjs` renames a property in every component schema, in `properties` and in
`required`. `generate.sh` made two documents with it, one per kind of field, and generated all
three targets from each.

```bash
node rename-field.mjs api.json author writer > api.renamed.json
```

```bash
node report.mjs
```

```text output
api.json, as written
  offline, generated mock ts pass  go pass  py pass
  live, sandbox           ts pass  go pass  py pass
  sandbox requests        ts 6  go 6  py 6
  ts definition suite     pass, tests 5
api.renamed.json, author renamed to writer (required)
  offline, generated mock ts pass  go pass  py pass
  live, sandbox           ts fail  go fail  py fail
  sandbox requests        ts 4  go 1  py 1
  ts definition suite     pass, tests 5
    ts: book.create.0 failed: POST 400 (request_status)
    go: BookshelfSDK: create: request: 400: Bad Request
    py: BookshelfSDK: create: request: 400: Bad Request
api.year.json, published renamed to year (optional)
  offline, generated mock ts pass  go pass  py pass
  live, sandbox           ts pass  go pass  py pass
  sandbox requests        ts 6  go 6  py 6
  ts definition suite     pass, tests 5
```

The required rename failed live in every language, at the create. The sandbox answered 400,
`author is required`, to a body that carried `writer`. Offline, every suite passed, because the
fixtures were regenerated from the renamed document, and the definition suite read that document
too. The request counts show each live run reached the sandbox, which no offline run did.

The optional rename passed everywhere, live included, after the whole flow of six requests. The
sandbox stores no `year` and sends `published`, and no step of the flow reads either field.

## Catch an optional rename with the validate feature

The scaffold carries the `test` feature alone. `generate.sh` added
[`validate`](https://github.com/voxgig/sdkgen/blob/main/docs/reference/features.md#validate) to a
copy of each project with `npm run add-feature validate`, regenerated, and set `active`,
`response` and `strict` under `test.client.options` in each language's
`test/sdk-test-control.json`, which its live suite merges into the client's options.

```bash
node report.mjs validate
```

```text output
validate added, with active, response and strict in each test/sdk-test-control.json
  api.json, live, sandbox       ts pass  go pass  py pass
  api.year.json, live, sandbox  ts fail  go fail  py fail
    ts: book.list.1 failed: GET 200 (validate_failed); book.list.5 failed: GET 200 (validate_failed)
    go: BookshelfSDK: list: Invalid response for entity "book": Unexpected keys at field <root>: published
    py: BookshelfSDK: list: Invalid response for entity "book": Unexpected keys at field <root>: published
response false strict false load bk0: ok, keys author id published title (1 request)
response false strict true  load bk0: ok, keys author id published title (1 request)
response true  strict false load bk0: ok, keys author id published title (1 request)
response true  strict true  load bk0: validate_failed: BookshelfSDK: load: Invalid response for entity "book": Unexpected keys at field <root>: published (1 request)
```

All three languages failed the optional rename at the list, and passed the document as written.
The last four lines load the seeded book with each pair of options: only `response` and `strict`
together refuse it, because `year` is optional and `published` is a key the model lacks. A
record the flow created comes back with neither name, so the check fires on the seeded book alone.
Seed the sandbox with a record written the way the deployed API writes it.

## Replay the list through two other generators' clients

In these runs neither `ogen` nor Hey API wrote a test that sends operations to a server. Both check a
response's shape at run time, which turns a hand-written operation list into a contract test.
`ogen` does this by default, and its decoders refuse a body missing a required field. Hey API
does it once its SDK plugin's `validator` option names a validator:

```js title="openapi-ts.config.mjs"
export default {
  input: './api.renamed.json',
  output: './client',
  plugins: ['@hey-api/client-fetch', '@hey-api/typescript', 'zod', { name: '@hey-api/sdk', validator: 'zod' }],
}
```

`run-clients.mjs` sends one list through three clients generated from `api.renamed.json`: the
sdkgen TypeScript suite, a Go driver over the `ogen` client, and a driver over the Hey API client.
Each runs against `mock.mjs`, which answers from the document's examples, and against the sandbox.

```bash
node run-clients.mjs
```

```text output
api.renamed.json: author renamed to writer; the sandbox still sends and requires author
sdkgen TypeScript suite, generated/sdkgen-ts
  offline, its own mock          pass, 5 tests
  live, mock from the document   fail: book.update.2 PATCH 200; book.load.3 GET 200; book.list.5 GET 200 (3 of 6 steps passed)
  live, sandbox                  fail: book.create.0 POST 400 (3 of 6 steps passed, 2 blocked)
ogen client, ogen/
  mock from the document: pass
    create  ok, writer "Frank Herbert"
    list    ok, length 1
    load    ok, writer "Frank Herbert"
    update  ok, title "Dune"
    remove  ok, 204
  sandbox: fail
    create  error: decode response: unexpected status code: 400
    list    error: decode response: decode application/json: callback: invalid: writer (field required)
    load    error: decode response: decode application/json: invalid: writer (field required)
    update  error: decode response: decode application/json: invalid: writer (field required)
    remove  ok, 204
Hey API client with zod, generated/heyapi
  mock from the document: pass
    create  ok, writer "Frank Herbert"
    list    ok, length 1
    load    ok, writer "Frank Herbert"
    update  ok, title "Dune"
    remove  ok, 204
  sandbox: fail
    create  error: 400 {"error":"author is required"}
    list    error: 200 ZodError: 0.writer Invalid input: expected string, received undefined
    load    error: 200 ZodError: writer Invalid input: expected string, received undefined
    update  error: 200 ZodError: writer Invalid input: expected string, received undefined
    remove  ok, 204
```

Both drivers failed the reads as well as the create, because each client checked the seeded book
and found no `writer`. sdkgen's flow failed the create and blocked the two steps that needed its
record. Its list and load passed, because without `validate` the client does not check a response
against the model's fields. It also
failed against the document mock, because the flow checks that an update sticks, which a mock
that answers from examples and keeps no state cannot do.

`ogen`'s opt-in `debug/example_tests` feature also writes tests, which encode and decode fake
data for each schema.[^1] Hey API's [MSW plugin](https://heyapi.dev/docs/openapi/typescript/plugins/msw)
is marked In Progress.

## Check it worked

```bash
node --test contract.test.mjs
```

```text output
1..9
# tests 9
# suites 0
# pass 9
# fail 0
# cancelled 0
# skipped 0
# todo 0
# duration_ms 13602.440902
```

One test fails unless every offline run of the required rename passes and every live run fails
with a 400. It also refuses a live run that reached no sandbox. Another fails unless
`validate` fails the optional rename in all three languages and passes `api.json`. Others hold the
sandbox to `api.json`, and the last runs all three clients against both servers.

## When it goes wrong

The suite stays green after a rename. It ran offline, against fixtures regenerated from the same
document. Run it live against a sandbox before the release, with the variable your SDK reads.

Callers lose a field that the live run never checked. The renamed field was optional, so the
server ignored the name it did not know, and no step read the field. Add `validate` with
`response` and `strict`, and seed the sandbox with a record that carries the deployed name.

The live run fails against a mock at the update, the load, and the last list. The mock answers
from the document's examples and keeps no state, while sdkgen's flow expects an update to stick. Use the
offline mode for the first run and a stateful server for the second.

## When not to do this

Do not count sdkgen's offline suite as a contract test. It proves the SDK matches its document in
every language, but it never sends a request, so it cannot show that the server agrees. Its live
suite checks no response against the model's field types until `validate` is added, and `strict`
then refuses every field a server adds, so keep that setting in tests.

Do not run a live suite against a production account. The flow creates and deletes records, so
give it a sandbox reset to a known state, as `generate.sh` does by starting `sandbox.mjs` afresh,
with its one seeded book, for each run.

Choose `ogen` over sdkgen when you ship a Go client only and want every read checked against the
document's required fields, which its decoders did here with no extra code. Choose Hey API with
Zod when the SDK is TypeScript and callers should see a response that breaks the document as an
error. Pact earns its broker once a provider team will replay what your SDK sends.

## Related how-tos

- [Choose between a shared test corpus and Pact contracts](/howto/choose-between-a-shared-corpus-and-pact-contracts)

- [Choose consumer-driven or spec-driven contract tests](/howto/choose-consumer-driven-or-spec-driven-contract-tests)

- [Write an SDK feature test corpus in aontu instead of JSON](/howto/write-a-feature-corpus-in-aontu)

- [Run a scheduled live suite that gates an SDK release](/howto/scheduled-live-suite-before-an-sdk-release)

- [Contract test an API you do not own](/howto/contract-test-an-api-you-do-not-own)

## Last verified

Verified 2026-11-22 against Node 22.23.5, Python 3.13.18 with pytest 9.1.1,
`@voxgig/create-sdkgen` 0.31.9, `@voxgig/sdkgen` 4.44.3 with `@voxgig/apidef` 8.32.4, Go 1.25.0,
`ogen` v1.24.0, and Hey API 0.99.0 with `zod` 4.6.7. Every output block is what the command
preceding it printed. The Go and Python suites ran in `generate.sh`, and their results are in
`runs/`. The sandbox is a local stand-in, since no sandbox with real credentials was available.
Pact was not run; its rows come from its documentation, read on 2026-11-22.

[^1]: The feature's description in [`features.go`](https://github.com/ogen-go/ogen/blob/v1.24.0/gen/features.go)
is `Enables example tests generation`. On this document it wrote three, `TestBook_EncodeDecode`
and one each for `BookPatch` and `NewBook`, as `runs/ogen-example-tests.txt` lists. Each fills a
value with fake data, encodes it, and decodes it again, so the generated encoder is tested against
the generated decoder rather than against a server.