Your TypeScript, Go and Python SDKs pass their generated tests after a regeneration, but every
caller’s create call fails in production. The OpenAPI document renamed author to writer
while the deployed API kept author. Each suite answered itself from fixtures made from that
same document, so the rename looked correct in all three languages.
What you get
You will end up with one operation list run through each generated client against a document mock and against a sandbox, and a record of which runs notice a renamed field. This is for you if you ship one API’s SDK in several languages.
Short answer
Run the same operation list through each generated client twice: against a mock that knows only
the OpenAPI document, then against a sandbox, where a wrong document shows. In an sdkgen project
the shared cases are .sdk/test/entity/<entity>/<Entity>TestData.json, read by every target’s
entity test, and BOOKSHELF_TEST_LIVE=TRUE sends them to a server. A renamed required field fails
only live. A renamed optional field fails only with the validate feature checking responses.
You will need
Node 22 or later, Go 1.25 or later, Python 3.12 or later with pytest, a generated SDK in at least
two languages, and an OpenAPI 3 document. The sample’s api.json has five Bookshelf operations
and a Book that requires id, title and author. There are no real credentials here, so the
sandbox is sandbox.mjs, a local stand-in that implements api.json and seeds one book. It plays
the deployed API, so it disagrees with a renamed document in that one place.
Approaches compared
| Approach | When it fits | What it costs you | When to pick something else |
|---|---|---|---|
| Hey API with Zod validators | A TypeScript client whose calls should refuse a response the document does not describe | Validation is off by default, and every response pays for a parse | You ship more than TypeScript from one test set |
ogen | A Go client that refuses a response missing a required field | Go only, and no generated test runs the operations against a server | You need the same cases in Python and TypeScript |
| Pact consumer tests | A provider team that replays what your SDK sends in its own build | Covers only the calls consumers record, and the pact files must reach the provider’s build | The API is public and its team runs nothing of yours |
| Per-language test files | One language, or a small API that rarely changes | Each language’s assertions drift once one copy is edited by hand | You ship the same operations in three languages |
| sdkgen generated suites | One set of entity cases, run by the TypeScript, Go and Python suites alike | Offline runs follow the document, and an optional rename passes live unless validate is added | Your clients are not generated by sdkgen |
Voxgig maintains sdkgen, and this page treats it as one of the five ways in.
A shared case file reaches every language at the price of a runner in each, which sdkgen
generates and a hand-rolled JSON runner would have to write. Pact sees only the calls a consumer
makes, and its pact files have to be
shared with the provider. Per-language files
drift. ogen and Hey API check each response’s shape at run time,
in one language each.
Find where the shared cases live
generate.sh scaffolded a project with -t ts -t go -t py and recorded where each target’s
suite reads its cases from.
node report.mjs corpus
.sdk/test/feature/:
cost.aontu
feature-test-index.aontu
.sdk/test/test.json, feature sections: cost
.sdk/test/entity/:
book/BookTestData.json
test files that name BookTestData.json:
go/test/book_entity_test.go
py/test/test_book_entity.py
ts/test/entity/book/BookEntity.test.ts
test files that name the path .sdk/test/test.json:
go/test/omniresolver_test.go
go/test/struct_utility_test.go
go/test/testsupport_test.go
py/test/omni.py
py/test/test_feature_corpus.py
py/test/test_primary_utility.py
py/test/test_struct_utility.py
ts/test/utility/index.ts
definition suites (read the document, not the model):
ts/test/definition-runner.ts
ts/test/definition.test.ts
The feature corpus holds cost alone, and its cases test the SDK’s own features, not the API.
The cases about the API are in BookTestData.json: the records and the request each operation
expects, written by npm run generate and read by all three entity tests. The definition suite,
which reads the document rather than the model, exists for TypeScript only, as the
definition how-to
says of its ts and js targets. That page also calls them the only two with a live suite, but
the Go and Python suites below sent their flows to the sandbox too.
Run each suite offline, then live
Offline, an entity test answers every call from those fixtures. Live, the same test sends the
create, list, update, load and remove to the server the document names. generate.sh runs it
both ways, starting a fresh sandbox each time:
run_suite() { # <project> <lang> <mode> <sandbox host>
local p=$1 l=$2 m=$3 h=$4
local env=(BOOKSHELF_TEST_LIVE=FALSE)
[ "$m" = live ] && env=(BOOKSHELF_TEST_LIVE=TRUE "BOOKSHELF_SERVER_SANDBOX=$h" BOOKSHELF_APIKEY=sandbox-key)
case $l in
ts) (cd "$p/ts" && env "${env[@]}" node --test --test-reporter=tap dist-test/entity/book/BookEntity.test.js) ;;
go) (cd "$p/go" && env "${env[@]}" go test -count=1 -run TestBookEntity ./test/) ;;
py) (cd "$p/py" && env "${env[@]}" python3 -m pytest -q test/test_book_entity.py) ;;
esac
BOOKSHELF_SERVER_SANDBOX fills the {sandbox} variable in the document’s server URL, which
is how a live run reaches a sandbox without editing the SDK.
Rename one field and read which runs fail
rename-field.mjs renames a property in every component schema, in properties and in
required. generate.sh made two documents with it, one per kind of field, and generated all
three targets from each.
node rename-field.mjs api.json author writer > api.renamed.json
node report.mjs
api.json, as written
offline, generated mock ts pass go pass py pass
live, sandbox ts pass go pass py pass
sandbox requests ts 6 go 6 py 6
ts definition suite pass, tests 5
api.renamed.json, author renamed to writer (required)
offline, generated mock ts pass go pass py pass
live, sandbox ts fail go fail py fail
sandbox requests ts 4 go 1 py 1
ts definition suite pass, tests 5
ts: book.create.0 failed: POST 400 (request_status)
go: BookshelfSDK: create: request: 400: Bad Request
py: BookshelfSDK: create: request: 400: Bad Request
api.year.json, published renamed to year (optional)
offline, generated mock ts pass go pass py pass
live, sandbox ts pass go pass py pass
sandbox requests ts 6 go 6 py 6
ts definition suite pass, tests 5
The required rename failed live in every language, at the create. The sandbox answered 400,
author is required, to a body that carried writer. Offline, every suite passed, because the
fixtures were regenerated from the renamed document, and the definition suite read that document
too. The request counts show each live run reached the sandbox, which no offline run did.
The optional rename passed everywhere, live included, after the whole flow of six requests. The
sandbox stores no year and sends published, and no step of the flow reads either field.
Catch an optional rename with the validate feature
The scaffold carries the test feature alone. generate.sh added
validate to a
copy of each project with npm run add-feature validate, regenerated, and set active,
response and strict under test.client.options in each language’s
test/sdk-test-control.json, which its live suite merges into the client’s options.
node report.mjs validate
validate added, with active, response and strict in each test/sdk-test-control.json
api.json, live, sandbox ts pass go pass py pass
api.year.json, live, sandbox ts fail go fail py fail
ts: book.list.1 failed: GET 200 (validate_failed); book.list.5 failed: GET 200 (validate_failed)
go: BookshelfSDK: list: Invalid response for entity "book": Unexpected keys at field <root>: published
py: BookshelfSDK: list: Invalid response for entity "book": Unexpected keys at field <root>: published
response false strict false load bk0: ok, keys author id published title (1 request)
response false strict true load bk0: ok, keys author id published title (1 request)
response true strict false load bk0: ok, keys author id published title (1 request)
response true strict true load bk0: validate_failed: BookshelfSDK: load: Invalid response for entity "book": Unexpected keys at field <root>: published (1 request)
All three languages failed the optional rename at the list, and passed the document as written.
The last four lines load the seeded book with each pair of options: only response and strict
together refuse it, because year is optional and published is a key the model lacks. A
record the flow created comes back with neither name, so the check fires on the seeded book alone.
Seed the sandbox with a record written the way the deployed API writes it.
Replay the list through two other generators’ clients
In these runs neither ogen nor Hey API wrote a test that sends operations to a server. Both check a
response’s shape at run time, which turns a hand-written operation list into a contract test.
ogen does this by default, and its decoders refuse a body missing a required field. Hey API
does it once its SDK plugin’s validator option names a validator:
export default {
input: './api.renamed.json',
output: './client',
plugins: ['@hey-api/client-fetch', '@hey-api/typescript', 'zod', { name: '@hey-api/sdk', validator: 'zod' }],
}
run-clients.mjs sends one list through three clients generated from api.renamed.json: the
sdkgen TypeScript suite, a Go driver over the ogen client, and a driver over the Hey API client.
Each runs against mock.mjs, which answers from the document’s examples, and against the sandbox.
node run-clients.mjs
api.renamed.json: author renamed to writer; the sandbox still sends and requires author
sdkgen TypeScript suite, generated/sdkgen-ts
offline, its own mock pass, 5 tests
live, mock from the document fail: book.update.2 PATCH 200; book.load.3 GET 200; book.list.5 GET 200 (3 of 6 steps passed)
live, sandbox fail: book.create.0 POST 400 (3 of 6 steps passed, 2 blocked)
ogen client, ogen/
mock from the document: pass
create ok, writer "Frank Herbert"
list ok, length 1
load ok, writer "Frank Herbert"
update ok, title "Dune"
remove ok, 204
sandbox: fail
create error: decode response: unexpected status code: 400
list error: decode response: decode application/json: callback: invalid: writer (field required)
load error: decode response: decode application/json: invalid: writer (field required)
update error: decode response: decode application/json: invalid: writer (field required)
remove ok, 204
Hey API client with zod, generated/heyapi
mock from the document: pass
create ok, writer "Frank Herbert"
list ok, length 1
load ok, writer "Frank Herbert"
update ok, title "Dune"
remove ok, 204
sandbox: fail
create error: 400 {"error":"author is required"}
list error: 200 ZodError: 0.writer Invalid input: expected string, received undefined
load error: 200 ZodError: writer Invalid input: expected string, received undefined
update error: 200 ZodError: writer Invalid input: expected string, received undefined
remove ok, 204
Both drivers failed the reads as well as the create, because each client checked the seeded book
and found no writer. sdkgen’s flow failed the create and blocked the two steps that needed its
record. Its list and load passed, because without validate the client does not check a response
against the model’s fields. It also
failed against the document mock, because the flow checks that an update sticks, which a mock
that answers from examples and keeps no state cannot do.
ogen’s opt-in debug/example_tests feature also writes tests, which encode and decode fake
data for each schema.1 Hey API’s MSW plugin
is marked In Progress.
Check it worked
node --test contract.test.mjs
1..9
# tests 9
# suites 0
# pass 9
# fail 0
# cancelled 0
# skipped 0
# todo 0
# duration_ms 13602.440902
One test fails unless every offline run of the required rename passes and every live run fails
with a 400. It also refuses a live run that reached no sandbox. Another fails unless
validate fails the optional rename in all three languages and passes api.json. Others hold the
sandbox to api.json, and the last runs all three clients against both servers.
When it goes wrong
The suite stays green after a rename. It ran offline, against fixtures regenerated from the same document. Run it live against a sandbox before the release, with the variable your SDK reads.
Callers lose a field that the live run never checked. The renamed field was optional, so the
server ignored the name it did not know, and no step read the field. Add validate with
response and strict, and seed the sandbox with a record that carries the deployed name.
The live run fails against a mock at the update, the load, and the last list. The mock answers from the document’s examples and keeps no state, while sdkgen’s flow expects an update to stick. Use the offline mode for the first run and a stateful server for the second.
When not to do this
Do not count sdkgen’s offline suite as a contract test. It proves the SDK matches its document in
every language, but it never sends a request, so it cannot show that the server agrees. Its live
suite checks no response against the model’s field types until validate is added, and strict
then refuses every field a server adds, so keep that setting in tests.
Do not run a live suite against a production account. The flow creates and deletes records, so
give it a sandbox reset to a known state, as generate.sh does by starting sandbox.mjs afresh,
with its one seeded book, for each run.
Choose ogen over sdkgen when you ship a Go client only and want every read checked against the
document’s required fields, which its decoders did here with no extra code. Choose Hey API with
Zod when the SDK is TypeScript and callers should see a response that breaks the document as an
error. Pact earns its broker once a provider team will replay what your SDK sends.
Related how-tos
Last verified
Verified 2026-11-22 against Node 22.23.5, Python 3.13.18 with pytest 9.1.1,
@voxgig/create-sdkgen 0.31.9, @voxgig/sdkgen 4.44.3 with @voxgig/apidef 8.32.4, Go 1.25.0,
ogen v1.24.0, and Hey API 0.99.0 with zod 4.6.7. Every output block is what the command
preceding it printed. The Go and Python suites ran in generate.sh, and their results are in
runs/. The sandbox is a local stand-in, since no sandbox with real credentials was available.
Pact was not run; its rows come from its documentation, read on 2026-11-22.
Footnotes
-
The feature’s description in
features.goisEnables example tests generation. On this document it wrote three,TestBook_EncodeDecodeand one each forBookPatchandNewBook, asruns/ogen-example-tests.txtlists. Each fills a value with fake data, encodes it, and decodes it again, so the generated encoder is tested against the generated decoder rather than against a server. ↩︎ Back to text