How-to › Describe your API

How to test an API release against the spec it replaces#

Replay what the last release promised against a preview of the next one, and catch the behavior changes that a schema diff and a schema check both pass.

Audience
Platform team
Level
advanced
Topic
Version and retire APIs
Languages
Python and TypeScript
Verified

Read first: Detect breaking changes between two OpenAPI versions in CI

The breaking-change check on your pull request is green, because the field you renamed was optional in the old document. You deploy. Every client built from the last release reads author as undefined, and a second change, a list that sorts newest first, breaks the pagination code your largest customer wrote.

What you get

You will end up with a CI job that replays the previous release’s promises against a preview of the next one, and a matrix showing which check catches which change. This is for you if you deploy an API that generated clients already depend on.

Short answer

Check out the previous release tag in CI and point three checks at a preview deployment. Run Schemathesis with the previous spec, the SDK test corpus generated from that spec, and a hand-kept fixture set. A schema check misses a renamed optional field and a changed default sort order. The sdkgen corpus catches the rename; only a fixture that asserts the order catches the sort.

You will need

A tagged previous release of the OpenAPI document, a preview deployment per pull request, and Node 22 or later for the sample. Comparing the two documents is a separate check; this one compares the old document with the new server. The sample’s previous.json is the Bookshelf document as release 1.4.0 shipped it, with a templated server, http://{preview}/v1.

Approaches compared

ApproachWhen it fitsWhat it costs youWhen to pick something else
Fern handwritten integration testsFern generates your SDKs and you keep integration tests beside themIts generated wire tests run against a mock, so the live check is tests you writeYou want a corpus generated for you
Hey API client with kept fixturesA TypeScript or Python client and a fixture set you curate per releaseEvery expectation is written and updated by hand, release after releaseYou need coverage of every operation
Pact provider verificationConsumers you know record what they actually callEach consumer has to adopt Pact, and nothing covers unrecorded callsYour consumers are unknown
Schemathesis with the previous release’s specBroad generated coverage of every operation in the old documentIt checks responses against schemas, so a valid response with changed behavior passesYou need to assert behavior
sdkgen live suite from the previous releaseAn sdkgen SDK, whose generated flow round-trips every entity operationIts flow checks fields it wrote, not order, counts, or defaultsYour API is mostly reads with no writes allowed

Voxgig maintains sdkgen. It is one of five options here, not the recommendation.

The trade-off is breadth against precision. Schemathesis and a generated corpus cover every operation without a person writing a case, and both stop at what a schema or a generic flow can state. Pact contracts and kept fixtures assert what real callers depend on, at the price of someone writing them.

Replay the previous corpus against each candidate

The SDK and its test suite were generated from previous.json. The suite has a live mode, documented in the run-a-live-suite how-to: BOOKSHELF_TEST_LIVE=TRUE, the key in BOOKSHELF_APIKEY, and the host in BOOKSHELF_SERVER_PREVIEW. corpus.mjs sets them and runs the suite as a child process:

  const env = { ...process.env }
  delete env.NODE_TEST_CONTEXT // inherited, it makes the inner node --test exit 0 on failure
  Object.assign(env, { BOOKSHELF_TEST_LIVE: 'TRUE', BOOKSHELF_APIKEY: key, BOOKSHELF_SERVER_PREVIEW: host })
  const child = spawn(process.execPath, ['--test', SUITE], { cwd: fileURLToPath(new URL('.', import.meta.url)), env })

preview.mjs stands in for five candidate releases. One is 1.4.0 unchanged, one adds an isbn field, one renames author to writer, one lists newest first, and one returns published as a string. replay.mjs runs three checks against each:

node replay.mjs
previous
  1.4.0 response schema   pass
  1.4.0 sdkgen corpus     pass
  1.4.0 kept fixtures     pass
  records left            0
additive
  1.4.0 response schema   pass
  1.4.0 sdkgen corpus     pass
  1.4.0 kept fixtures     pass
  records left            0
renamed
  1.4.0 response schema   pass
  1.4.0 sdkgen corpus     FAIL book.update.2 failed; book.load.3 failed
  1.4.0 kept fixtures     FAIL bk2 lacks author; bk3 lacks author; bk4 lacks author
  records left            0
sorted
  1.4.0 response schema   pass
  1.4.0 sdkgen corpus     pass
  1.4.0 kept fixtures     FAIL list order fixture-3,fixture-2,fixture-1
  records left            0
retyped
  1.4.0 response schema   FAIL bk2 published is not integer; bk3 published is not integer; bk4 published is not integer
  1.4.0 sdkgen corpus     pass
  1.4.0 kept fixtures     pass
  records left            0

The schema row passes the rename. In 1.4.0 every Book property is optional, and JSON Schema allows properties the schema does not list, so { id, title, writer } is a valid 1.4.0 book. schema.mjs applies exactly those rules.1 The corpus fails it, because its update step writes a mark into author and its load step reads the mark back.

The sort change passes the corpus too. The generated flow checks that its record is in the list, not where.

Only the schema row fails the type change. The generated flow asserts the id at each step and the mark it wrote into author, and the fixtures assert order, presence, and author. Neither looks at the type of published.

Keep fixtures for behavior, with Hey API

Hey API generated a client from previous.json with one function per operation. It wires the X-API-Key scheme to the auth option and leaves the base URL to you, since the document’s server is templated. The fixtures are data, kept with the release, so a Python client from @hey-api/openapi-python, Hey API’s HTTPX generator, could read the same file:

  "expect": {
    "listOrder": ["fixture-1 Emma", "fixture-2 Ulysses", "fixture-3 Beloved"],
    "loadedFields": ["author", "id", "published", "title"]
  }

fixtures.mjs creates the three books through the client, compares the default list order and each loaded record with the expectations, and deletes what it made in a finally. The order assertion is the one no generated check on this page makes. Write one for every default a caller can observe: sort order, page size, which fields appear when none are requested.

Run the same checks with Fern

Fern was not run here. Its testing page describes generated unit tests that make no network calls. Its mock server tests, switched on by generation.wireTests.enabled in sdk-config.yml, Fern’s current configuration format, run the SDK against a mock generated from your definition. Neither reaches your preview. The page names handwritten integration tests as the ones that run against your real server, and it marks the mock server tests and integration tests as Enterprise plan features. Keep those tests at the release tag and run them as the previous/ step in the next section runs the sdkgen corpus.

Check out the previous tag in CI

The job adds the previous release as a git worktree, writes its document to previous.json, and runs each check against the preview for the pull request:

      - name: Check out the previous release beside this one
        run: |
          TAG=$(git describe --tags --abbrev=0 --match 'v*')
          git worktree add previous "$TAG"
          git show "$TAG:openapi.json" > previous.json

fetch-depth: 0 on the checkout step is what lets git describe find the tag. The runner image has no uv, so the job installs it with astral-sh/setup-uv. The Schemathesis step then runs uvx schemathesis run previous.json --url with the key in --header, the forms its quick start gives. A failing run looks like the renamed block in the matrix: named steps, with the field each one lacked.

Check it worked

The tests pin the matrix. An additive release passes, and a rename fails the corpus and the fixtures but not the schema. A sort change fails only the fixtures, and a type change fails only the schema check. They also check cleanup, a refused key, and the workflow’s shape.

node --test previous.test.mjs | tail -8
# tests 8
# suites 0
# pass 8
# fail 0
# cancelled 0
# skipped 0
# todo 0
# duration_ms 44003.412364

When it goes wrong

A customer reports a changed order after a deploy that passed every check, because no check asserted the default. Add the assertion to fixtures.json at the release that introduced the default, so the next release is checked against it.

The corpus passes against a preview that refuses the key. A nested node --test that inherits NODE_TEST_CONTEXT reports to its parent and exits 0. Remove the variable before spawning, as corpus.mjs does; one test runs the corpus with a stale key and expects a failure.

The schema row passes a response with a wrong type. A validator that takes the schema as a second parameter was handed straight to flatMap, which passes the index there, so every record was checked against nothing. Wrap it in a one-argument function, as fixtures.mjs does; one test changes a type and expects the row to fail.

The preview fills with test books. A check stopped between its create and its delete. Delete in a finally, as fixtures.mjs does, and give the preview a database that is thrown away with it.

When not to do this

Do not run this against production. The corpus and the fixtures both write, and a replay that dies midway leaves records behind.

Do not treat a green matrix as a compatibility promise. Its coverage stops at the operations the corpus generates and the defaults someone wrote down.

Do not adopt sdkgen to get the corpus. If your SDKs come from Fern, its integration tests kept at the release tag do the same job with the generator you already run. With one TypeScript client from Hey API, a fixture file like this one is less machinery than a second generator. It is also the only check on this page that sees order.

Last verified

Verified 2026-11-20 against Node 22.23.5, @voxgig/create-sdkgen 0.31.9, @voxgig/sdkgen 4.44.3, @hey-api/openapi-ts 0.99.0 and TypeScript 5.9.3. Every output block is what the command preceding it printed. Hey API was run once and its compiled output committed. Fern was not run. Its options come from its documentation. Schemathesis and Pact were not run, and the workflow was not run on GitHub.

Footnotes

  1. The object reference says it in two sentences: properties defined by properties are not required by default, and by default any additional properties are allowed. A schema written without required describes every object that has the right types where it has anything, which includes an empty one. ↩︎ Back to text

Read this page as markdown · All how-to guides

Generate the client instead of writing it#

Retries, timeouts, pagination and auth are the same problems in every client. Voxgig generates them from your OpenAPI description, in 23 languages, from one model.

Listen to Fireside, the Voxgig podcast

Since 2018, Richard Rodger has talked with developer advocates, event organizers, and the people who build developer-facing companies about the specifics of the work.