How-to › Describe your API

How to regression test model extraction in CI#

Commit the extracted model as sorted JSON, fail CI with a readable list when an entity or operation moves, and refuse the empty model a broken document gives.

Audience
Platform team
Level
intermediate
Topic
Extract a semantic model from a spec
Languages
TypeScript
Verified

A lockfile update moves @voxgig/apidef from 8.32.4 to 8.33.1 and leaves your OpenAPI document alone. The entity tide_forecast becomes forecast, and the next SDK generated from the model renames the class it built for it. The spec diff in your pipeline reports no changes, because the document has none, so the rename reaches a release before anyone reviews it.

What you get

You will end up with the extracted model committed as a snapshot and a CI step that fails with one readable line per change. A documented command accepts a change you meant. This is for you if you generate SDKs, CLIs or MCP servers from a model of your API.

Short answer

Extract the model with @voxgig/apidef pinned in a lockfile, write res.apimodel.main.kit as sorted JSON, and commit it as a snapshot. In CI, extract again and compare, print each entity, operation or field that moved, and fail on any difference. Refuse an empty entity map, because a truncated document builds without error. Keep a spec diff such as oasdiff beside it for the changes on the wire.

You will need

Node 22 or later, an OpenAPI 3 document, and a CI pipeline you can add a step to. The sample pins apidef 8.32.4, the release a project scaffolded with @voxgig/create-sdkgen installs, and installs 8.33.1 beside it under an alias to stand for the bump. The document, tides.json, has three operations: a tide forecast, a list of stations and one station.

Approaches compared

ApproachWhen it fitsWhat it costs youWhen to pick something else
apidef model snapshot with node —testYou generate from apidef’s model and want the extractor’s own changes caughtA snapshot file to commit and an update step, which every intended change also has to passNothing you ship is generated from that model
Fern checkFern generates your SDKs, and CI should refuse an invalid definitionIt validates the definition and configuration only, without a list of what the SDK gained or lostYou need to see a renamed method before it ships
Golden files checked with git diffAny extractor that writes files, checked with no test runnergit diff --exit-code fails with a raw diff, and a file git does not track passes itYou want a failure that names the entity that moved
Hey API generated output in gitHey API generates a TypeScript client from operation idsNo model between the document and the code, so the snapshot is the generated codeYour SDK comes from an entity model rather than operations
oasdiffYou want every change on the wire, with a severity, before generationIt reads the document only, so an extractor upgrade never showsYour risk is the extractor, not the document
Vitest snapshot testsThe repository already runs VitestA test dependency, and a raw diff of the JSON as the failure messageNode’s own runner is enough, as it is here

Voxgig maintains apidef. The snapshot on this page tests apidef’s output, and the other five rows test other things or other tools.

The rows split on what they read. oasdiff and fern check read the document, so an extractor bump passes them unseen. The snapshot reads what the extractor made, so it sees both kinds of change and needs an update step for every change you meant. Keep one of each.

Snapshot the model as sorted JSON

The model build runs as a library with its generate step off, as apidef’s pipeline how-to describes, and snapshot writes res.apimodel.main.kit with every key sorted. guard stands in front of it.

// apidef reports success for a document it read no paths from. A snapshot of that is empty.
export function guard(kit) {
  const names = Object.keys(kit?.entity ?? {})
  if (names.length === 0) throw new Error('the model has no entities: the document lost its paths, or apidef could not read them')
  return kit
}

function sortKeys(v) {
  if (Array.isArray(v)) return v.map(sortKeys)
  if (v && typeof v === 'object') return Object.fromEntries(Object.keys(v).sort().map((k) => [k, sortKeys(v[k])]))
  return v
}

node check.mjs --update wrote models/tides.model.json, which holds the model under model and the apidef release that made it under apidef. That command is the documented update step, as vitest -u is the update step for Vitest’s toMatchFileSnapshot.1 The snapshot is a committed file, so an update shows in the pull request as a diff of it. The check compares a fresh extraction with that file.

node check.mjs
tides.json on apidef 8.32.4: model matches models/tides.model.json

Make the failure say what moved

Delete GET /v1/stations/{stationId} from the document and run the check on the result, tides.trimmed.json.

node check.mjs tides.trimmed.json; echo "exit $?"
tides.trimmed.json on apidef 8.32.4: model differs from models/tides.model.json
  entity renamed: tide_forecast -> forecast
  operation removed: station.load GET /v1/stations/{stationId}
if the change is intended, run node check.mjs --update and commit the snapshot
exit 1

The second line is the one you expected. The first is why the snapshot reads the model and not the document: on 8.32.4, removing a station operation also renamed the forecast entity, although the deleted lines never mention the forecast. On 8.33.1 the same deletion reports the removal alone.

8.32.4 names an entity after its response schema only while that schema is rare. The test on line 2365 of heuristic01.ts asks for references from under 21% of the methods or 41% of the paths, the constants on lines 100 and 101. TideForecastResponse sits on one path of three, so it named tide_forecast. One path of two is 50%, so the path named forecast, and a test that adds a third path brings tide_forecast back.

changes describes entities, operations by route, and fields. When nothing of those differs, the first other difference is reported by its JSON path, so a change with no description still fails.

Catch an extractor bump the spec diff cannot see

Run the check on the unchanged document with the newer release.

node check.mjs --release 8.33.1; echo "exit $?"
tides.json on apidef 8.33.1: model differs from models/tides.model.json
  apidef release changed: 8.32.4 -> 8.33.1
  entity renamed: tide_forecast -> forecast
if the change is intended, run node check.mjs --update and commit the snapshot
exit 1

The release line fails a bump even when the model comes out the same, so the snapshot always names the extractor that made it. 8.33.1 names the entity after the path, /v1/forecast, because it lets the path win when the schema’s name wraps the path’s own word, the path-over-wrapper branch on line 3584 of its heuristic01.ts. record.sh ran oasdiff breaking v1.39.0 on the same cases.

cat records/oasdiff.txt
oasdiff version 1.39.0
a deleted operation, with and without --fail-on
$ oasdiff breaking tides.json tides.trimmed.json --fail-on ERR
1 changes: 1 error, 0 warning, 0 info
error	[api-path-removed-without-deprecation] at tides.json
	in API GET /v1/stations/{stationId}
		API path removed without deprecation


exit 1
$ oasdiff breaking tides.json tides.trimmed.json
1 changes: 1 error, 0 warning, 0 info
error	[api-path-removed-without-deprecation] at tides.json
	in API GET /v1/stations/{stationId}
		API path removed without deprecation


exit 0
the same document, which is what an apidef bump from 8.32.4 to 8.33.1 sees
$ oasdiff breaking tides.json tides.json --fail-on ERR
No changes detected
exit 0

oasdiff names the removed path and nothing about the forecast. Its getting-started page says breaking exits 1 when it finds a breaking change, but v1.39.0 exited 0 until --fail-on ERR was added, as its breaking-changes guide says.2 For the bump it has nothing to compare.

Refuse an empty model

A document cut short after "paths": still builds, and the build reports success with no entities. A snapshot taken from it is a valid file that describes nothing.

node check.mjs tides.truncated.json; echo "exit $?"
tides.truncated.json on apidef 8.32.4: the model has no entities: the document lost its paths, or apidef could not read them
exit 1

A test runs the truncated document on both releases and finds the entity map empty on each.

Keep clock times and paths out of the snapshot

times.mjs builds the model twice with the library and runs the voxgig-apidef CLI twice, each time in a different folder, and compares what each wrote.

node times.mjs
output                            same twice  clock time  absolute path
library model, sorted JSON        yes         no          no
CLI model files, *.aontu          yes         no          no
CLI .jostraca/jostraca.meta.log   no          yes         no
CLI stdout                        no          yes         yes

The model holds neither, in JSON or in the CLI’s model files. The CLI’s log lines on stdout and its .jostraca/jostraca.meta.log change on every run. Snapshot the model, never the CLI’s output, and leave .jostraca/ out of any golden directory.

Snapshot the generated client in Hey API and Fern

Hey API builds its client from the document’s operations. Every file record.sh listed is generated code rather than a model of the API, so the regression test is that code itself. Commit it, regenerate in CI, and fail on a diff. record.sh ran @hey-api/openapi-ts 0.99.0 on both documents.

cat records/hey-api.txt
@hey-api/openapi-ts 0.99.0, exported functions in sdk.gen.ts
  tides.json:          getStation getTideForecast listStations
  tides.trimmed.json:  getTideForecast listStations
  tides.json twice:    identical trees
  files: client.gen.ts client/client.gen.ts client/index.ts client/types.gen.ts client/utils.gen.ts core/auth.gen.ts core/bodySerializer.gen.ts core/params.gen.ts core/pathSerializer.gen.ts core/queryKeySerializer.gen.ts core/serverSentEvents.gen.ts core/types.gen.ts core/utils.gen.ts index.ts sdk.gen.ts types.gen.ts

Two runs on one release gave identical trees, so a diff in CI means an input changed. The exported names follow the operation ids, which is why the deletion removes getStation and nothing else.

Fern’s documentation calls generators.yml legacy and reads sdk-config.yml, which is the format described here. Fern turns the definition into code inside its generator containers, as its how it works page describes, and its general and SDK command references list no command that writes that model out. fern check validates the definition in CI. For a regression test, run fern generate --preview, which its SDK command reference says writes to a local .preview/ folder, and diff that folder against a committed copy. The same page says remote generation needs a login, so the preview runs with one or with --local on Docker.

Check it worked

Thirteen tests run, covering the snapshot, the guard on both paths, the deletion, the bump, a moved operation, the 41% rule, a malformed snapshot, and both records.

node --test model.test.mjs
1..13
# tests 13
# suites 0
# pass 13
# fail 0
# cancelled 0
# skipped 0
# todo 0

Add npm ci and node --test model.test.mjs to the CI job. The first test fails with the same lines check.mjs prints, and its message names the update command.

When it goes wrong

The check fails on every run. Something other than the model reached the snapshot: the CLI’s stdout, or .jostraca/jostraca.meta.log. Build the snapshot from res.apimodel.main.kit alone.

The check passes on a broken document. The snapshot was regenerated from an empty model, with the guard left out. Keep guard on the update path as well as the check.

A pull request updates the snapshot without a word. The update step is one command, so review the snapshot diff like code, and require the changes output in the pull request description.

When not to do this

Do not snapshot apidef’s model if nothing you ship is generated from it. A spec diff such as oasdiff covers the wire, and the model adds a file to review for no consumer.

Do not install apidef through a version range. The 8.32.4 to 8.33.1 step renamed an entity with no change to the document, so pin an exact version, and treat its bump as a change to review.

Hey API is the better fit when the client is TypeScript and its names follow operation ids. The generated code is then the only artifact, and a git diff of it is the whole test. Fern suits a team that wants a hosted pipeline, where fern check guards the definition and the preview folder is the thing to diff.

Last verified

Verified 2026-11-22 against Node 22.23.5 and @voxgig/apidef 8.32.4 and 8.33.1, which the captured commands run. record.sh ran oasdiff v1.39.0 on the Go 1.26 toolchain it fetched, and @hey-api/openapi-ts 0.99.0 with TypeScript 5.9.3. Fern was not run, because generation needs a login, a guest session or Docker; its commands come from its documentation. Vitest was not run.

Footnotes

  1. Vitest’s snapshot guide says that by default it does not write snapshots when process.env.CI is set to a true value. There, a mismatch, a missing snapshot or an obsolete one fails the run. The same vitest command therefore writes a missing snapshot on a laptop and fails on it in CI, with one environment variable deciding which. ↩︎ Back to text

  2. oasdiff’s breaking changes guide for v1.39.0 lists four uses of --fail-on. The last exits 1 only when a breaking change comes without a major version increase in info.version, so a team can let a removal through by editing one field of the document. The extractor’s version is in no field of the document, so no flag of that kind can see an apidef bump. ↩︎ Back to text

Read this page as markdown · All how-to guides

Generate the client instead of writing it#

Retries, timeouts, pagination and auth are the same problems in every client. Voxgig generates them from your OpenAPI description, in 23 languages, from one model.

Listen to Fireside, the Voxgig podcast

Since 2018, Richard Rodger has talked with developer advocates, event organizers, and the people who build developer-facing companies about the specifics of the work.