How-to › Test and mock integrations

How to contract test a generated SDK against its spec#

Run one operation list through each generated client against a document mock and a sandbox, then rename a field in the spec and see which runs fail.

Audience
Library maintainer
Level
advanced
Topic
Run contract and spec tests
Languages
TypeScript, Go and Python
Verified

Your TypeScript, Go and Python SDKs pass their generated tests after a regeneration, but every caller’s create call fails in production. The OpenAPI document renamed author to writer while the deployed API kept author. Each suite answered itself from fixtures made from that same document, so the rename looked correct in all three languages.

What you get

You will end up with one operation list run through each generated client against a document mock and against a sandbox, and a record of which runs notice a renamed field. This is for you if you ship one API’s SDK in several languages.

Short answer

Run the same operation list through each generated client twice: against a mock that knows only the OpenAPI document, then against a sandbox, where a wrong document shows. In an sdkgen project the shared cases are .sdk/test/entity/<entity>/<Entity>TestData.json, read by every target’s entity test, and BOOKSHELF_TEST_LIVE=TRUE sends them to a server. A renamed required field fails only live. A renamed optional field fails only with the validate feature checking responses.

You will need

Node 22 or later, Go 1.25 or later, Python 3.12 or later with pytest, a generated SDK in at least two languages, and an OpenAPI 3 document. The sample’s api.json has five Bookshelf operations and a Book that requires id, title and author. There are no real credentials here, so the sandbox is sandbox.mjs, a local stand-in that implements api.json and seeds one book. It plays the deployed API, so it disagrees with a renamed document in that one place.

Approaches compared

ApproachWhen it fitsWhat it costs youWhen to pick something else
Hey API with Zod validatorsA TypeScript client whose calls should refuse a response the document does not describeValidation is off by default, and every response pays for a parseYou ship more than TypeScript from one test set
ogenA Go client that refuses a response missing a required fieldGo only, and no generated test runs the operations against a serverYou need the same cases in Python and TypeScript
Pact consumer testsA provider team that replays what your SDK sends in its own buildCovers only the calls consumers record, and the pact files must reach the provider’s buildThe API is public and its team runs nothing of yours
Per-language test filesOne language, or a small API that rarely changesEach language’s assertions drift once one copy is edited by handYou ship the same operations in three languages
sdkgen generated suitesOne set of entity cases, run by the TypeScript, Go and Python suites alikeOffline runs follow the document, and an optional rename passes live unless validate is addedYour clients are not generated by sdkgen

Voxgig maintains sdkgen, and this page treats it as one of the five ways in.

A shared case file reaches every language at the price of a runner in each, which sdkgen generates and a hand-rolled JSON runner would have to write. Pact sees only the calls a consumer makes, and its pact files have to be shared with the provider. Per-language files drift. ogen and Hey API check each response’s shape at run time, in one language each.

Find where the shared cases live

generate.sh scaffolded a project with -t ts -t go -t py and recorded where each target’s suite reads its cases from.

node report.mjs corpus
.sdk/test/feature/:
  cost.aontu
  feature-test-index.aontu
.sdk/test/test.json, feature sections: cost
.sdk/test/entity/:
  book/BookTestData.json
test files that name BookTestData.json:
  go/test/book_entity_test.go
  py/test/test_book_entity.py
  ts/test/entity/book/BookEntity.test.ts
test files that name the path .sdk/test/test.json:
  go/test/omniresolver_test.go
  go/test/struct_utility_test.go
  go/test/testsupport_test.go
  py/test/omni.py
  py/test/test_feature_corpus.py
  py/test/test_primary_utility.py
  py/test/test_struct_utility.py
  ts/test/utility/index.ts
definition suites (read the document, not the model):
  ts/test/definition-runner.ts
  ts/test/definition.test.ts

The feature corpus holds cost alone, and its cases test the SDK’s own features, not the API. The cases about the API are in BookTestData.json: the records and the request each operation expects, written by npm run generate and read by all three entity tests. The definition suite, which reads the document rather than the model, exists for TypeScript only, as the definition how-to says of its ts and js targets. That page also calls them the only two with a live suite, but the Go and Python suites below sent their flows to the sandbox too.

Run each suite offline, then live

Offline, an entity test answers every call from those fixtures. Live, the same test sends the create, list, update, load and remove to the server the document names. generate.sh runs it both ways, starting a fresh sandbox each time:

run_suite() { # <project> <lang> <mode> <sandbox host>
  local p=$1 l=$2 m=$3 h=$4
  local env=(BOOKSHELF_TEST_LIVE=FALSE)
  [ "$m" = live ] && env=(BOOKSHELF_TEST_LIVE=TRUE "BOOKSHELF_SERVER_SANDBOX=$h" BOOKSHELF_APIKEY=sandbox-key)
  case $l in
    ts) (cd "$p/ts" && env "${env[@]}" node --test --test-reporter=tap dist-test/entity/book/BookEntity.test.js) ;;
    go) (cd "$p/go" && env "${env[@]}" go test -count=1 -run TestBookEntity ./test/) ;;
    py) (cd "$p/py" && env "${env[@]}" python3 -m pytest -q test/test_book_entity.py) ;;
  esac

BOOKSHELF_SERVER_SANDBOX fills the {sandbox} variable in the document’s server URL, which is how a live run reaches a sandbox without editing the SDK.

Rename one field and read which runs fail

rename-field.mjs renames a property in every component schema, in properties and in required. generate.sh made two documents with it, one per kind of field, and generated all three targets from each.

node rename-field.mjs api.json author writer > api.renamed.json
node report.mjs
api.json, as written
  offline, generated mock ts pass  go pass  py pass
  live, sandbox           ts pass  go pass  py pass
  sandbox requests        ts 6  go 6  py 6
  ts definition suite     pass, tests 5
api.renamed.json, author renamed to writer (required)
  offline, generated mock ts pass  go pass  py pass
  live, sandbox           ts fail  go fail  py fail
  sandbox requests        ts 4  go 1  py 1
  ts definition suite     pass, tests 5
    ts: book.create.0 failed: POST 400 (request_status)
    go: BookshelfSDK: create: request: 400: Bad Request
    py: BookshelfSDK: create: request: 400: Bad Request
api.year.json, published renamed to year (optional)
  offline, generated mock ts pass  go pass  py pass
  live, sandbox           ts pass  go pass  py pass
  sandbox requests        ts 6  go 6  py 6
  ts definition suite     pass, tests 5

The required rename failed live in every language, at the create. The sandbox answered 400, author is required, to a body that carried writer. Offline, every suite passed, because the fixtures were regenerated from the renamed document, and the definition suite read that document too. The request counts show each live run reached the sandbox, which no offline run did.

The optional rename passed everywhere, live included, after the whole flow of six requests. The sandbox stores no year and sends published, and no step of the flow reads either field.

Catch an optional rename with the validate feature

The scaffold carries the test feature alone. generate.sh added validate to a copy of each project with npm run add-feature validate, regenerated, and set active, response and strict under test.client.options in each language’s test/sdk-test-control.json, which its live suite merges into the client’s options.

node report.mjs validate
validate added, with active, response and strict in each test/sdk-test-control.json
  api.json, live, sandbox       ts pass  go pass  py pass
  api.year.json, live, sandbox  ts fail  go fail  py fail
    ts: book.list.1 failed: GET 200 (validate_failed); book.list.5 failed: GET 200 (validate_failed)
    go: BookshelfSDK: list: Invalid response for entity "book": Unexpected keys at field <root>: published
    py: BookshelfSDK: list: Invalid response for entity "book": Unexpected keys at field <root>: published
response false strict false load bk0: ok, keys author id published title (1 request)
response false strict true  load bk0: ok, keys author id published title (1 request)
response true  strict false load bk0: ok, keys author id published title (1 request)
response true  strict true  load bk0: validate_failed: BookshelfSDK: load: Invalid response for entity "book": Unexpected keys at field <root>: published (1 request)

All three languages failed the optional rename at the list, and passed the document as written. The last four lines load the seeded book with each pair of options: only response and strict together refuse it, because year is optional and published is a key the model lacks. A record the flow created comes back with neither name, so the check fires on the seeded book alone. Seed the sandbox with a record written the way the deployed API writes it.

Replay the list through two other generators’ clients

In these runs neither ogen nor Hey API wrote a test that sends operations to a server. Both check a response’s shape at run time, which turns a hand-written operation list into a contract test. ogen does this by default, and its decoders refuse a body missing a required field. Hey API does it once its SDK plugin’s validator option names a validator:

export default {
  input: './api.renamed.json',
  output: './client',
  plugins: ['@hey-api/client-fetch', '@hey-api/typescript', 'zod', { name: '@hey-api/sdk', validator: 'zod' }],
}

run-clients.mjs sends one list through three clients generated from api.renamed.json: the sdkgen TypeScript suite, a Go driver over the ogen client, and a driver over the Hey API client. Each runs against mock.mjs, which answers from the document’s examples, and against the sandbox.

node run-clients.mjs
api.renamed.json: author renamed to writer; the sandbox still sends and requires author
sdkgen TypeScript suite, generated/sdkgen-ts
  offline, its own mock          pass, 5 tests
  live, mock from the document   fail: book.update.2 PATCH 200; book.load.3 GET 200; book.list.5 GET 200 (3 of 6 steps passed)
  live, sandbox                  fail: book.create.0 POST 400 (3 of 6 steps passed, 2 blocked)
ogen client, ogen/
  mock from the document: pass
    create  ok, writer "Frank Herbert"
    list    ok, length 1
    load    ok, writer "Frank Herbert"
    update  ok, title "Dune"
    remove  ok, 204
  sandbox: fail
    create  error: decode response: unexpected status code: 400
    list    error: decode response: decode application/json: callback: invalid: writer (field required)
    load    error: decode response: decode application/json: invalid: writer (field required)
    update  error: decode response: decode application/json: invalid: writer (field required)
    remove  ok, 204
Hey API client with zod, generated/heyapi
  mock from the document: pass
    create  ok, writer "Frank Herbert"
    list    ok, length 1
    load    ok, writer "Frank Herbert"
    update  ok, title "Dune"
    remove  ok, 204
  sandbox: fail
    create  error: 400 {"error":"author is required"}
    list    error: 200 ZodError: 0.writer Invalid input: expected string, received undefined
    load    error: 200 ZodError: writer Invalid input: expected string, received undefined
    update  error: 200 ZodError: writer Invalid input: expected string, received undefined
    remove  ok, 204

Both drivers failed the reads as well as the create, because each client checked the seeded book and found no writer. sdkgen’s flow failed the create and blocked the two steps that needed its record. Its list and load passed, because without validate the client does not check a response against the model’s fields. It also failed against the document mock, because the flow checks that an update sticks, which a mock that answers from examples and keeps no state cannot do.

ogen’s opt-in debug/example_tests feature also writes tests, which encode and decode fake data for each schema.1 Hey API’s MSW plugin is marked In Progress.

Check it worked

node --test contract.test.mjs
1..9
# tests 9
# suites 0
# pass 9
# fail 0
# cancelled 0
# skipped 0
# todo 0
# duration_ms 13602.440902

One test fails unless every offline run of the required rename passes and every live run fails with a 400. It also refuses a live run that reached no sandbox. Another fails unless validate fails the optional rename in all three languages and passes api.json. Others hold the sandbox to api.json, and the last runs all three clients against both servers.

When it goes wrong

The suite stays green after a rename. It ran offline, against fixtures regenerated from the same document. Run it live against a sandbox before the release, with the variable your SDK reads.

Callers lose a field that the live run never checked. The renamed field was optional, so the server ignored the name it did not know, and no step read the field. Add validate with response and strict, and seed the sandbox with a record that carries the deployed name.

The live run fails against a mock at the update, the load, and the last list. The mock answers from the document’s examples and keeps no state, while sdkgen’s flow expects an update to stick. Use the offline mode for the first run and a stateful server for the second.

When not to do this

Do not count sdkgen’s offline suite as a contract test. It proves the SDK matches its document in every language, but it never sends a request, so it cannot show that the server agrees. Its live suite checks no response against the model’s field types until validate is added, and strict then refuses every field a server adds, so keep that setting in tests.

Do not run a live suite against a production account. The flow creates and deletes records, so give it a sandbox reset to a known state, as generate.sh does by starting sandbox.mjs afresh, with its one seeded book, for each run.

Choose ogen over sdkgen when you ship a Go client only and want every read checked against the document’s required fields, which its decoders did here with no extra code. Choose Hey API with Zod when the SDK is TypeScript and callers should see a response that breaks the document as an error. Pact earns its broker once a provider team will replay what your SDK sends.

Last verified

Verified 2026-11-22 against Node 22.23.5, Python 3.13.18 with pytest 9.1.1, @voxgig/create-sdkgen 0.31.9, @voxgig/sdkgen 4.44.3 with @voxgig/apidef 8.32.4, Go 1.25.0, ogen v1.24.0, and Hey API 0.99.0 with zod 4.6.7. Every output block is what the command preceding it printed. The Go and Python suites ran in generate.sh, and their results are in runs/. The sandbox is a local stand-in, since no sandbox with real credentials was available. Pact was not run; its rows come from its documentation, read on 2026-11-22.

Footnotes

  1. The feature’s description in features.go is Enables example tests generation. On this document it wrote three, TestBook_EncodeDecode and one each for BookPatch and NewBook, as runs/ogen-example-tests.txt lists. Each fills a value with fake data, encodes it, and decodes it again, so the generated encoder is tested against the generated decoder rather than against a server. ↩︎ Back to text

Read this page as markdown · All how-to guides

Generate the client instead of writing it#

Retries, timeouts, pagination and auth are the same problems in every client. Voxgig generates them from your OpenAPI description, in 23 languages, from one model.

Listen to Fireside, the Voxgig podcast

Since 2018, Richard Rodger has talked with developer advocates, event organizers, and the people who build developer-facing companies about the specifics of the work.