# How to regression test model extraction in CI

> Commit the extracted model as sorted JSON, fail CI with a readable list when an entity or operation moves, and refuse the empty model a broken document gives.

Source: https://voxgig.com/howto/regression-test-model-extraction-in-ci

- Audience: platform-team
- Level: intermediate
- Languages: typescript
- Verified: 2026-11-22
- Published: 2026-11-22

## Short answer

Extract the model with @voxgig/apidef pinned in a lockfile, write res.apimodel.main.kit as sorted JSON, and commit it as a snapshot. In CI, extract again and compare, print each entity, operation or field that moved, and fail on any difference. Refuse an empty entity map, because a truncated document builds without error. Keep a spec diff such as oasdiff beside it for the changes on the wire.

---
## You will need

Node 22 or later, an OpenAPI 3 document, and a CI pipeline you can add a step to. The sample
pins apidef 8.32.4, the release a project scaffolded with `@voxgig/create-sdkgen` installs, and
installs 8.33.1 beside it under an alias to stand for the bump. The document, `tides.json`, has
three operations: a tide forecast, a list of stations and one station.

## Approaches compared

| Approach | When it fits | What it costs you | When to pick something else |
| --- | --- | --- | --- |
| [apidef model snapshot with node --test](https://github.com/voxgig/apidef/blob/main/docs/how-to/control-the-pipeline.md) | You generate from apidef's model and want the extractor's own changes caught | A snapshot file to commit and an update step, which every intended change also has to pass | Nothing you ship is generated from that model |
| [Fern check](https://buildwithfern.com/learn/cli-api-reference/cli-reference/general-commands#fern-check) | Fern generates your SDKs, and CI should refuse an invalid definition | It validates the definition and configuration only, without a list of what the SDK gained or lost | You need to see a renamed method before it ships |
| [Golden files checked with git diff](https://git-scm.com/docs/git-diff) | Any extractor that writes files, checked with no test runner | `git diff --exit-code` fails with a raw diff, and a file git does not track passes it | You want a failure that names the entity that moved |
| [Hey API generated output in git](https://heyapi.dev/docs/openapi/typescript/output) | Hey API generates a TypeScript client from operation ids | No model between the document and the code, so the snapshot is the generated code | Your SDK comes from an entity model rather than operations |
| [oasdiff](https://www.oasdiff.com/docs/getting-started) | You want every change on the wire, with a severity, before generation | It reads the document only, so an extractor upgrade never shows | Your risk is the extractor, not the document |
| [Vitest snapshot tests](https://vitest.dev/guide/snapshot) | The repository already runs Vitest | A test dependency, and a raw diff of the JSON as the failure message | Node's own runner is enough, as it is here |

Voxgig maintains apidef. The snapshot on this page tests apidef's output, and the other five
rows test other things or other tools.

The rows split on what they read. oasdiff and `fern check` read the document, so an extractor
bump passes them unseen. The snapshot reads what the extractor made, so it sees both kinds of
change and needs an update step for every change you meant. Keep one of each.

## Snapshot the model as sorted JSON

The model build runs as a library with its generate step off, as apidef's
[pipeline how-to](https://github.com/voxgig/apidef/blob/main/docs/how-to/control-the-pipeline.md)
describes, and `snapshot` writes `res.apimodel.main.kit` with every key sorted. `guard` stands in
front of it.

```js title="snapshot.mjs"
// apidef reports success for a document it read no paths from. A snapshot of that is empty.
export function guard(kit) {
  const names = Object.keys(kit?.entity ?? {})
  if (names.length === 0) throw new Error('the model has no entities: the document lost its paths, or apidef could not read them')
  return kit
}

function sortKeys(v) {
  if (Array.isArray(v)) return v.map(sortKeys)
  if (v && typeof v === 'object') return Object.fromEntries(Object.keys(v).sort().map((k) => [k, sortKeys(v[k])]))
  return v
}
```

`node check.mjs --update` wrote `models/tides.model.json`, which holds the model under `model`
and the apidef release that made it under `apidef`. That command is the documented update step,
as `vitest -u` is the update step for Vitest's `toMatchFileSnapshot`.[^1] The snapshot is a
committed file, so an update shows in the pull request as a diff of it. The check compares a
fresh extraction with that file.

```bash
node check.mjs
```

```text output
tides.json on apidef 8.32.4: model matches models/tides.model.json
```

## Make the failure say what moved

Delete `GET /v1/stations/{stationId}` from the document and run the check on the result,
`tides.trimmed.json`.

```bash
node check.mjs tides.trimmed.json; echo "exit $?"
```

```text output
tides.trimmed.json on apidef 8.32.4: model differs from models/tides.model.json
  entity renamed: tide_forecast -> forecast
  operation removed: station.load GET /v1/stations/{stationId}
if the change is intended, run node check.mjs --update and commit the snapshot
exit 1
```

The second line is the one you expected. The first is why the snapshot reads the model and not
the document: on 8.32.4, removing a station operation also renamed the forecast entity,
although the deleted lines never mention the forecast. On 8.33.1 the same deletion reports the
removal alone.

8.32.4 names an entity after its response schema only while that schema is rare. The test on
line 2365 of
[`heuristic01.ts`](https://github.com/voxgig/apidef/blob/v8.32.4/ts/src/guide/heuristic01.ts)
asks for references from under 21% of the methods or 41% of the paths, the constants on lines
100 and 101. `TideForecastResponse` sits on one path of three, so it named
`tide_forecast`. One path of two is 50%, so the path named `forecast`, and a test that adds a
third path brings `tide_forecast` back.

`changes` describes entities, operations by route, and fields. When nothing of those differs,
the first other difference is reported by its JSON path, so a change with no description still
fails.

## Catch an extractor bump the spec diff cannot see

Run the check on the unchanged document with the newer release.

```bash
node check.mjs --release 8.33.1; echo "exit $?"
```

```text output
tides.json on apidef 8.33.1: model differs from models/tides.model.json
  apidef release changed: 8.32.4 -> 8.33.1
  entity renamed: tide_forecast -> forecast
if the change is intended, run node check.mjs --update and commit the snapshot
exit 1
```

The release line fails a bump even when the model comes out the same, so the snapshot always
names the extractor that made it. 8.33.1 names the entity after the path, `/v1/forecast`,
because it lets the path win when the schema's name wraps the path's own word, the
`path-over-wrapper` branch on line 3584 of its
[`heuristic01.ts`](https://github.com/voxgig/apidef/blob/v8.33.1/ts/src/guide/heuristic01.ts).
`record.sh` ran
[`oasdiff breaking`](https://github.com/oasdiff/oasdiff/blob/v1.39.0/docs/BREAKING-CHANGES.md)
v1.39.0 on the same cases.

```bash
cat records/oasdiff.txt
```

```text output
oasdiff version 1.39.0
a deleted operation, with and without --fail-on
$ oasdiff breaking tides.json tides.trimmed.json --fail-on ERR
1 changes: 1 error, 0 warning, 0 info
error	[api-path-removed-without-deprecation] at tides.json
	in API GET /v1/stations/{stationId}
		API path removed without deprecation


exit 1
$ oasdiff breaking tides.json tides.trimmed.json
1 changes: 1 error, 0 warning, 0 info
error	[api-path-removed-without-deprecation] at tides.json
	in API GET /v1/stations/{stationId}
		API path removed without deprecation


exit 0
the same document, which is what an apidef bump from 8.32.4 to 8.33.1 sees
$ oasdiff breaking tides.json tides.json --fail-on ERR
No changes detected
exit 0
```

oasdiff names the removed path and nothing about the forecast. Its getting-started page says
`breaking` exits 1 when it finds a breaking change, but v1.39.0 exited 0 until `--fail-on ERR`
was added, as its breaking-changes guide says.[^2] For the bump it has nothing to compare.

## Refuse an empty model

A document cut short after `"paths": ` still builds, and the build reports success with no
entities. A snapshot taken from it is a valid file
that describes nothing.

```bash
node check.mjs tides.truncated.json; echo "exit $?"
```

```text output
tides.truncated.json on apidef 8.32.4: the model has no entities: the document lost its paths, or apidef could not read them
exit 1
```

A test runs the truncated document on both releases and finds the entity map empty on each.

## Keep clock times and paths out of the snapshot

`times.mjs` builds the model twice with the library and runs the `voxgig-apidef` CLI twice, each
time in a different folder, and compares what each wrote.

```bash
node times.mjs
```

```text output
output                            same twice  clock time  absolute path
library model, sorted JSON        yes         no          no
CLI model files, *.aontu          yes         no          no
CLI .jostraca/jostraca.meta.log   no          yes         no
CLI stdout                        no          yes         yes
```

The model holds neither, in JSON or in the CLI's model files. The CLI's log lines on stdout and
its `.jostraca/jostraca.meta.log` change on every run. Snapshot the model, never the CLI's
output, and leave `.jostraca/` out of any golden directory.

## Snapshot the generated client in Hey API and Fern

Hey API builds its client from the document's operations. Every file `record.sh` listed is
generated code rather than a model of the API, so the regression test is that code itself. Commit it, regenerate in CI, and
fail on a diff. `record.sh` ran `@hey-api/openapi-ts` 0.99.0 on both documents.

```bash
cat records/hey-api.txt
```

```text output
@hey-api/openapi-ts 0.99.0, exported functions in sdk.gen.ts
  tides.json:          getStation getTideForecast listStations
  tides.trimmed.json:  getTideForecast listStations
  tides.json twice:    identical trees
  files: client.gen.ts client/client.gen.ts client/index.ts client/types.gen.ts client/utils.gen.ts core/auth.gen.ts core/bodySerializer.gen.ts core/params.gen.ts core/pathSerializer.gen.ts core/queryKeySerializer.gen.ts core/serverSentEvents.gen.ts core/types.gen.ts core/utils.gen.ts index.ts sdk.gen.ts types.gen.ts
```

Two runs on one release gave identical trees, so a diff in CI means an input changed. The
exported names follow the operation ids, which is why the deletion removes `getStation` and
nothing else.

Fern's documentation calls `generators.yml` legacy and reads `sdk-config.yml`, which is the
format described here. Fern turns the definition into code inside its generator containers, as its
[how it works page](https://buildwithfern.com/learn/sdks/current/overview/how-it-works)
describes, and its general and SDK command references list no command that writes that model
out. `fern check`
validates the definition in CI. For a regression test, run `fern generate --preview`, which its
[SDK command reference](https://buildwithfern.com/learn/cli-api-reference/cli-reference/sdk-commands)
says writes to a local `.preview/` folder, and diff that folder against a committed copy. The
same page says remote generation needs a login, so the preview runs with one or with `--local`
on Docker.

## Check it worked

Thirteen tests run, covering the snapshot, the guard on both paths, the deletion, the bump, a
moved operation, the 41% rule, a malformed snapshot, and both records.

```bash
node --test model.test.mjs
```

```text output
1..13
# tests 13
# suites 0
# pass 13
# fail 0
# cancelled 0
# skipped 0
# todo 0
```

Add `npm ci` and `node --test model.test.mjs` to the CI job. The first test fails with the same
lines `check.mjs` prints, and its message names the update command.

## When it goes wrong

The check fails on every run. Something other than the model reached the snapshot: the CLI's
stdout, or `.jostraca/jostraca.meta.log`. Build the snapshot from `res.apimodel.main.kit` alone.

The check passes on a broken document. The snapshot was regenerated from an empty model, with
the guard left out. Keep `guard` on the update path as well as the check.

A pull request updates the snapshot without a word. The update step is one command, so review
the snapshot diff like code, and require the `changes` output in the pull request description.

## When not to do this

Do not snapshot apidef's model if nothing you ship is generated from it. A spec diff such as
oasdiff covers the wire, and the model adds a file to review for no consumer.

Do not install apidef through a version range. The 8.32.4 to 8.33.1 step renamed an entity with no
change to the document, so pin an exact version, and treat its bump as a change to review.

Hey API is the better fit when the client is TypeScript and its names follow operation ids. The
generated code is then the only artifact, and a git diff of it is the whole test. Fern suits a team
that wants a hosted pipeline, where `fern check` guards the definition and the preview folder is
the thing to diff.

## Related how-tos

- [Diff two semantic models from successive spec versions](/howto/diff-two-semantic-models-across-versions)

- [Block a release when generated SDK files have drifted](/howto/block-a-release-when-generated-sdk-files-drift)

- [Detect breaking changes between two OpenAPI versions in CI](/howto/detect-breaking-changes-between-openapi-versions-in-ci)

- [Decide if a regenerated SDK is a minor or major release](/howto/regenerated-sdk-minor-or-major-release)

- [Verify an SDK, CLI and MCP server share one model version](/howto/verify-surfaces-share-one-model-version)

## Last verified

Verified 2026-11-22 against Node 22.23.5 and `@voxgig/apidef` 8.32.4 and 8.33.1, which the
captured commands run. `record.sh` ran oasdiff v1.39.0 on the Go 1.26 toolchain it fetched, and
`@hey-api/openapi-ts` 0.99.0 with TypeScript 5.9.3. Fern was not run, because generation needs a
login, a guest session or Docker; its commands come from its documentation. Vitest was not run.

[^1]: Vitest's [snapshot guide](https://vitest.dev/guide/snapshot) says that by default it does
not write snapshots when `process.env.CI` is set to a true value. There, a mismatch, a missing
snapshot or an obsolete one fails the run. The same `vitest` command therefore writes a missing
snapshot on a laptop and fails on it in CI, with one environment variable deciding which.

[^2]: oasdiff's [breaking changes guide](https://github.com/oasdiff/oasdiff/blob/v1.39.0/docs/BREAKING-CHANGES.md)
for v1.39.0 lists four uses of `--fail-on`. The last exits 1 only when a breaking change comes
without a major version increase in `info.version`, so a team can let a removal through by
editing one field of the document. The extractor's version is in no field of the document, so
no flag of that kind can see an apidef bump.