How-to › Ship an SDK

How to check whether two generator runs produce identical output#

Run a generator twice from empty directories, hash both trees, and learn how many bytes differ, which files carry them, and which setting removes them.

Audience
Platform team
Level
advanced
Topic
Choose a generator and generate an SDK
Verified

Your CI job regenerates the SDK and fails when the result differs from the committed one. It fails every night on files nobody touched: a README carries a build date, and eleven Java classes carry the instant they were generated. The team learns to ignore the job, so the first difference that matters is ignored with the rest.

What you get

You will end up with a count of the files and bytes that differ between two clean runs, the cause of each, and a check that fails when a run disagrees. This is for you if a generated SDK is committed or diffed in CI.

Short answer

Run the generator twice from two empty directories, on one document at one pinned version, and compare the trees byte for byte. Two sdkgen scaffolds differed in three record files under .sdk/log, by a timestamp each, and in no file of the SDK. orval and Kiota matched exactly. The Java clients of Swagger Codegen and OpenAPI Generator differed by a generation timestamp until hideGenerationTimestamp was set. Pin the version, remove the clock, then gate CI on the comparison.

You will need

Node 22 or later, git, and a generator you can run twice without an account. Verified 2026-11-19 against @voxgig/sdkgen 4.44.3, orval 8.48.0, Swagger Codegen 3.0.85, OpenAPI Generator 7.27.0, and Kiota 1.38.0, on a Bookshelf document with five operations. The two Java generators need a Java runtime. The comparison itself needs Node alone.

Approaches compared

ApproachWhen it fitsWhat it costs youWhen to pick something else
Kiota kiota-lock.jsonYou want a run’s inputs recorded beside the client: description hash, version, parametersThe lock proves the inputs matched and says nothing about the bytes writtenYou need the byte comparison, which the lock does not replace
OpenAPI Generator hideGenerationTimestampA Java client, whose @Generated annotation otherwise carries the instant of the runAn option to set per generator, and an npm wrapper that picks the jar when openapitools.json is absentYour target, such as typescript-fetch, already repeats byte for byte
orval --outdatedA TypeScript client, when you want the comparison from the generator, with nothing writtenPrettier output is compared too, and a project formatting with biome, oxfmt, or hooks cannot be checkedYou compare runs from two machines, not one run against the disk
sdkgen byte-stable outputAn sdkgen project, where check-drift deletes each target, regenerates, and reportsThree record files carry timestamps, and the toolchain is requested by range until a lockfile pins itYou want a check that leaves the tree untouched
Swagger Codegen hideGenerationTimestampA Java client from one versioned jar, with the option passed as an additional propertyNo check of its own, and a default that writes the clock into eight filesYou want the generator to run the comparison for you

Voxgig maintains sdkgen. It is one of five options here, not the recommendation.

Byte-for-byte output makes a CI diff mean something, and it costs you every value that came from the clock or the environment. With Swagger Codegen and OpenAPI Generator that is one option, at the price of a client that no longer says when it was built. With orval the check is built in, and it refuses a project that formats with biome or oxfmt. With sdkgen the SDK repeats and its records do not.

Run sdkgen twice from two empty directories

A second run in the same directory proves less than it seems to, because the first run’s files are still there to be kept. Start both runs from nothing. In generate.sh, clean makes two directories that hold the document and nothing else, and twice runs one command in each.

clean sdkgen
twice sdkgen npx --yes @voxgig/create-sdkgen@0.31.9 bookshelf --def api.json --target ts
twice sdkgen generate

runs.mjs hash then writes a manifest for each run, one SHA-256 and one path per file, in the format sha256sum prints.1 With --tracked it hashes what git would commit and skips what the ignore files name. runs.mjs compare reads two manifests, or two directories.

node runs.mjs compare runs/sdkgen/scaffold-1.sha256 runs/sdkgen/scaffold-2.sha256
349 files, 346 identical, 3 differ, 0 in one run only
  differs   .sdk/log/copies.jsonl        12 bytes on 2 lines  timestamp
  differs   .sdk/log/generated.jsonl      5 bytes on 1 line  timestamp
  differs   .sdk/log/scaffold.jsonl       6 bytes on 1 line  timestamp

The exit status is 1. Of 349 files, 346 are identical, including every file under ts/. The three that differ are sdkgen’s own records of what it copied and wrote, and each line of them opens with the instant it was appended. Masking that instant makes the two versions equal, which is what timestamp in the last column means. Four more log files differ too, and git ignores them.

Separate the generator from the clock and the registry

Two scaffolds measure more than the generator. They also measure the clock and whatever npm resolved that minute. Take those apart.

node table.mjs sdkgen
tool    run                              files  differ  bytes  cause
sdkgen  ts, two scaffolds                  349       3     23  timestamp
sdkgen  ts, one project generated twice    349       0      0  -
sdkgen  ts, clock a year ahead             349       1      1  content

One project generated twice differs in nothing, and the scaffold’s own npm run check-drift agrees, printing check-drift: clean. That is the claim in sdkgen’s documentation, byte-stable output from the same model, and it held. The third row moves the clock. next-year.cjs is loaded into the generator with Node’s --require option.

const RealDate = Date
const SHIFT = 366 * 24 * 60 * 60 * 1000
const now = () => RealDate.now() + SHIFT

globalThis.Date = new Proxy(RealDate, {
  construct: (target, args) => (args.length ? new target(...args) : new target(now())),
  apply: (target) => new target(now()).toString(),
  get: (target, key) => (key === 'now' ? now : Reflect.get(target, key)),
})

A year ahead, one file changed by one byte: the copyright year in the root LICENSE. That is a commit every January, and a red job on the first run of the year.

The registry is the larger risk. The scaffold command is pinned, but the toolchain it installs is asked for by range.

"typescript": ">=6.0",
"@voxgig/apidef": "~8.32.3",
"@voxgig/model": "~12.0.2",
"@voxgig/sdkgen": "~4.44.3",
"@voxgig/util": ">=0.5",
"@voxgig/struct": ">=0.2",
"jostraca": ">=0.42.0",

A later scaffold can install a different generator. The pin is the .sdk/package-lock.json the scaffold writes, which is one of the 349 committed files, and npm ci installs from it without changing it.

Measure orval and Swagger Codegen the same way

The same two commands work on any generator, because they read only files.

node table.mjs
tool               run                                 files  differ  bytes  cause
Kiota              typescript                              5       0      0  -
OpenAPI Generator  typescript-fetch                       11       0      0  -
OpenAPI Generator  java                                   43      12    144  timestamp
OpenAPI Generator  java, hideGenerationTimestamp=true     43       0      0  -
orval              axios client                            1       0      0  -
sdkgen             ts, two scaffolds                     349       3     23  timestamp
sdkgen             ts, one project generated twice       349       0      0  -
sdkgen             ts, clock a year ahead                349       1      1  content
Swagger Codegen    typescript-fetch                        9       0      0  -
Swagger Codegen    java                                   37       8     88  timestamp
Swagger Codegen    java, hideGenerationTimestamp=true     37       0      0  -

orval wrote one file, twice the same. It also ships the check: orval --outdated compares what it would generate with the files on disk and writes nothing. In the run for this page it exited 0 on untouched output, and 1 with outdated: client/bookshelf.ts after a one-byte edit. With the file deleted it printed missing and did not write it back.

Swagger Codegen has no such check, so the two trees are compared from outside. Its typescript-fetch client repeated exactly. Its Java client did not.

node runs.mjs compare runs/swagger-codegen-java/a.sha256 runs/swagger-codegen-java/b.sha256
37 files, 29 identical, 8 differ, 0 in one run only
  differs   README.md                                                11 bytes on 1 line  timestamp
  differs   src/main/java/io/swagger/client/ApiException.java        11 bytes on 1 line  timestamp
  differs   src/main/java/io/swagger/client/Configuration.java       11 bytes on 1 line  timestamp
  differs   src/main/java/io/swagger/client/Pair.java                11 bytes on 1 line  timestamp
  differs   src/main/java/io/swagger/client/StringUtil.java          11 bytes on 1 line  timestamp
  differs   src/main/java/io/swagger/client/auth/ApiKeyAuth.java     11 bytes on 1 line  timestamp
  differs   src/main/java/io/swagger/client/auth/OAuth.java          11 bytes on 1 line  timestamp
  differs   src/main/java/io/swagger/client/model/Book.java          11 bytes on 1 line  timestamp

Seven of the eight files carry a line like this one, and the README prints the same instant as its build date.

@javax.annotation.Generated(value = "io.swagger.codegen.v3.generators.java.JavaClientCodegen", date = "2026-11-19T20:59:14.254354453Z[Etc/UTC]")

The option that removes it is in the output of config-help -l java, and it is passed as --additional-properties hideGenerationTimestamp=true. With it, two runs matched in all 37 files. OpenAPI Generator’s Java client behaves the same way and takes the same option.2

Check it worked

Eighteen tests hold each row of the table to its recorded runs and exercise the comparison on trees built for the purpose.

node --test runs.test.mjs
1..18
# tests 18
# suites 0
# pass 18
# fail 0
# cancelled 0
# skipped 0
# todo 0

Then run it on your own generator. Two clean runs that agree print a first line ending 0 differ, 0 in one run only, and node runs.mjs compare exits 0.

When it goes wrong

The comparison passes locally and fails in CI. The two machines ran different generators. The npm wrapper for OpenAPI Generator downloads the last stable version the first time it runs and records it in openapitools.json. Without that file in the repository, each clean checkout chooses again. Commit it.

Every file differs. Line endings or a formatter changed between runs, not the generator. Pin the formatter with the generator, and compare on one operating system first.

A handful of files differ by a few bytes each. Read the cause column. A timestamp has an option to remove it, or a record file to leave out of the comparison. Anything else is worth a bug report with both versions attached.

The check is green in December and red in January. Something reads the year. Run once with the clock moved, as next-year.cjs does, before the calendar does it for you.

When not to do this

Do not demand byte-identical output from a tree you never commit or diff. If the SDK is built and published in one job, a timestamp in an annotation costs nothing, and it tells a reader when the client was built.

Do not strip a timestamp by post-processing when the generator has an option for it. A sed over generated Java is a second generator to maintain.

Do not treat sdkgen’s result as the whole answer for sdkgen. The SDK repeated here, but its record files and its license year did not, and check-drift has to delete and regenerate to tell you. orval’s --outdated is the better tool when the check must not write. Swagger Codegen is the easier generator to pin, as one versioned jar, where the sdkgen scaffold installed 101 npm packages.

Do not compare a run against itself. A second run in the same directory can pass on files the first run left behind.

Last verified

Verified 2026-11-19 against Node 22.23.5, @voxgig/create-sdkgen 0.31.9, @voxgig/sdkgen 4.44.3, orval 8.48.0, Swagger Codegen 3.0.85, OpenAPI Generator 7.27.0, and Kiota 1.38.0. Every output block is what the command preceding it printed. All five generators were run twice by generate.sh, and the page’s commands read the manifests and the differing files it saved.

Footnotes

  1. The GNU manual introduces sha256sum and its three siblings as a legacy interface to the more modern cksum utility. For the format they print, it sends the reader to cksum as well, where the layout is filed as untagged. ↩︎ Back to text

  2. The noise is by specification. The reference page for javax.annotation.Generated gives the annotation a date element for the date the source was generated, requires ISO 8601, and offers 2001-07-04T12:08:56.235-0700 as its example. Both generators fill it in from the wall clock. Setting SOURCE_DATE_EPOCH, the variable reproducible-builds.org defines so that build tools can agree on a date, changed nothing in either run here. ↩︎ Back to text

Read this page as markdown · All how-to guides

Generate the client instead of writing it#

Retries, timeouts, pagination and auth are the same problems in every client. Voxgig generates them from your OpenAPI description, in 23 languages, from one model.

Listen to Fireside, the Voxgig podcast

Since 2018, Richard Rodger has talked with developer advocates, event organizers, and the people who build developer-facing companies about the specifics of the work.