# How to judge a generator by reading a reference SDK it produced

> Read an SDK each generator produced, find which files it really wrote, score what users call, and regenerate one with the release you would install.

Source: https://voxgig.com/howto/judge-a-generator-by-its-reference-sdk

- Audience: api-producer
- Level: intermediate
- Languages: go, python
- Verified: 2026-11-21
- Published: 2026-11-21

## Short answer

Before you run a generator, read an SDK it produced and score what users notice: the list call, the errors, the credentials, a non-entity endpoint, and the runtime share. First find which files the generator wrote, from its marker, its ignore file such as .fernignore, or its record such as sdkgen's .sdk/log/generated.jsonl. Then regenerate one reference with the release you would install and diff it.

---
## You will need

Node 22 or later and git. The sample's `fetch.sh` clones each reference at a pinned commit, runs
the survey, and copies every file the page quotes, with its licence, into `refs/`. Everything the
page runs reads those committed files, offline.

## Approaches compared

| Approach | When it fits | What it costs you | When to pick something else |
| --- | --- | --- | --- |
| [Fern, read through the Cohere Python SDK](https://buildwithfern.com/learn/sdks/overview/custom-code) | You weigh Fern and want a large, maintained SDK to read | Its client classes are in `.fernignore`, and Replay patches edit marked files | You want to see Fern output untouched by its customer |
| [oapi-codegen, read through examples/client](https://github.com/oapi-codegen/oapi-codegen) | A Go client, and a reference you can regenerate from two files | A two-operation example on `main`, made by a build the release does not match | You need to see pagination, auth or errors on a real API |
| [sdkgen, read through voxgig-solardemo-sdk](https://github.com/voxgig/sdkgen/blob/main/docs/reference/project-layout.md) | Entity-shaped clients in many languages from one model | A record to read instead of markers, and a reference made by an older release | Your API is a set of actions, which the reference shows folded into create |
| [Speakeasy, read through the Mistral Python client](https://www.speakeasy.com/docs/sdks/customize/code/custom-code/genignore) | A commercial generator with a large public SDK to read | `.genignore` and hook files mark the hand-kept code, and a run needs an account | You cannot sign up, or you need to regenerate the reference yourself |

Voxgig maintains sdkgen and both of its reference SDKs. It is one of four rows here, read with the
same scripts as the other three.

Treat each reference as its generator's best case. Mistral and Cohere are large and maintained, so they
show what a team adds around the output. The oapi-codegen example is small and pure, so it shows
the output and little else. The sdkgen references sit between them, and are older than the release
you would install.

## Find which files the generator wrote

Three kinds of evidence say who wrote a file: a marker in its first lines, an ignore file that
hands paths to people, and a record the generator keeps. `survey.mjs` read all three in each clone,
leaving tests out, and `report.mjs` prints what it found.

```bash
node report.mjs provenance
```

```text output
voxgig-solardemo-sdk @ c40d675, sdkgen
  release named in the repo: @voxgig/sdkgen 4.34.5, 4.34.7 (.sdk/log/copies.jsonl)
  go/: 82 source files, 18986 lines; evidence: record .sdk/log/generated.jsonl
  marked 21, recorded 82, generated 82 (100% of lines), runtime 87% of generated lines
  hand-kept: 0 listed in the ignore file, 0 unmarked and not listed
  .sdk templates no copy record names: 20
voxgig-elementdemo-sdk @ faf8de6, sdkgen
  release named in the repo: @voxgig/sdkgen 4.42.0, 4.44.0 (.sdk/log/copies.jsonl)
  go/: 85 source files, 20636 lines; evidence: record .sdk/log/generated.jsonl
  marked 21, recorded 85, generated 85 (100% of lines), runtime 82% of generated lines
  hand-kept: 0 listed in the ignore file, 0 unmarked and not listed
  .sdk templates no copy record names: 32
mistral-client-python @ e344499, Speakeasy
  release named in the repo: speakeasyVersion 1.807.0, generationVersion 2.956.1 (.speakeasy/gen.lock)
  src/mistralai/: 1397 source files, 178651 lines; evidence: marker Code generated by Speakeasy
  marked 1332, recorded -, generated 1332 (93% of lines), runtime 2% of generated lines
  hand-kept: 57 listed in the ignore file, 8 unmarked and not listed
    src/mistralai/extra/batch/client.py (1000 lines, ignore file)
    src/mistralai/extra/observability/otel.py (673 lines, ignore file)
    src/mistralai/extra/voice_agents/transport.py (655 lines, ignore file)
  edits inside marked files: persistentEdits enabled: "false" (.speakeasy/gen.yaml)
cohere-python @ f1f21ac, Fern
  release named in the repo: fernapi/fern-python-sdk 5.41.10, cliVersion 5.55.0 (.fern/metadata.json)
  src/cohere/: 361 source files, 45319 lines; evidence: marker auto-generated by Fern
  marked 333, recorded -, generated 333 (89% of lines), runtime 10% of generated lines
  hand-kept: 27 listed in the ignore file, 1 unmarked and not listed
    src/cohere/oci_client.py (1236 lines, ignore file)
    src/cohere/manually_maintained/cohere_aws/client.py (1035 lines, ignore file)
    src/cohere/client.py (578 lines, ignore file)
  edits inside marked files: 3 Replay patches in .fern/replay.lock, touching src/cohere/core/client_wrapper.py in scope
oapi-codegen-examples-client @ d27c033, oapi-codegen
  release named in the repo: github.com/oapi-codegen/oapi-codegen/v2 version v2.0.0-00010101000000-000000000000 (file header)
  examples/client/: 2 source files, 400 lines; evidence: marker Code generated by
  marked 1, recorded -, generated 1 (99% of lines), runtime - of generated lines
  hand-kept: 0 listed in the ignore file, 1 unmarked and not listed
    examples/client/generate.go (3 lines)
```

Read the hand-kept lines first, because they are what you must not credit. Cohere's ignore file
names the client module in its first four lines:

```text title="refs/cohere-python/.fernignore"
4.0.0-5.0.0-migration-guide.md
banner.png
README.md
src/cohere/client.py
```

Both Cohere entry points, `client.py` and `client_v2.py`, are in `.fernignore`. In Mistral, `.genignore` hands
`src/mistralai/extra/*` to people, and the eight unmarked files sit in `_hooks/`, where Speakeasy
generates the registration file once and then leaves it to its customer.[^1]

A marker is not proof of untouched output. Fern's [Replay](https://buildwithfern.com/learn/sdks/overview/custom-code)
stores hand commits as patches and reapplies them to marked files on the next generate, and
Cohere's three patches edit `core/client_wrapper.py`. Mistral's `gen.yaml` turns the equivalent
Speakeasy setting off.

sdkgen marks only 21 of the 82 Go files, so for its references the record is the evidence. Its
[project layout reference](https://github.com/voxgig/sdkgen/blob/main/docs/reference/project-layout.md)
describes `.sdk/log/generated.jsonl` as the record of what every generate wrote. A recorded file
can still come from the project: elementdemo's `elementcard` feature is in the record, and
`refs/voxgig-elementdemo-sdk/.sdk/model/feature/elementcard.aontu` calls it "A CUSTOM feature,
defined in this project's own `.sdk`". Count the `.sdk` templates no copy record names before you
credit the release.

The runtime share is the fifth question. It is the share of generated lines in each SDK's runtime
directories, as `REFERENCES` in `survey.mjs` lists them. sdkgen's Go reference spends 87% of its
lines on runtime for two entities; Mistral spends 2%, because most of its lines are model types.

## Score the four calls your users write

`scorecard.json` answers the other four questions for oapi-codegen, Speakeasy and sdkgen, each
answer with the exact text it rests on. `scorecard.mjs` prints the answers and checks every quote
against its file in `refs/`.

```bash
node scorecard.mjs
```

```text output
voxgig-solardemo-sdk (sdkgen)
  list call            an entity method; a []any of entities, each read with Data()
  errors               (value, error) from every operation; the README names no error type per status
  auth                 no credential in the constructor table; apikey is an option of the secrets feature
  non-entity endpoint  POST .../terraform is a create point on Planet, chosen with $action
mistral-client-python (Speakeasy)
  list call            a method per operation in a namespace; next() fetches the following page
  errors               one base class, MistralError, with a subclass for each declared error response
  auth                 an api_key argument to the client
  non-entity endpoint  a verb method in a namespace, the same shape as every other call
oapi-codegen-examples-client (oapi-codegen)
  list call            the example has no list operation; every call is a method named from its operationId
  errors               no error type; a typed field per documented status, and the status is yours to check
  auth                 a request editor function you write and pass in
  non-entity endpoint  no distinction: an operation is a method, whatever its path
evidence: 12 of 12 quotes found in 5 files
```

The answers show taste. Speakeasy gives every operation the same shape, actions included, with a
typed error for each declared error response. sdkgen gives resources a five-verb entity and folds an action
into `create`, which reads well for `Moon` and badly for `terraform`. Its Go list returns `[]any`,
so a caller asserts types. oapi-codegen hands back the response and leaves the status to you.

A reference shows only what its document exercises. The solardemo document declares no security
scheme, so its README cannot show credentials beyond a feature option.

## Regenerate one reference with the release you would install

A reference is a file at a commit, made by some release. Run the release you would install on the
reference's own document and compare.

```bash
node report.mjs regen
```

```text output
oapi-codegen v2.8.2 on examples/client at tag v2.8.2: client.gen.go: 2 lines differ
oapi-codegen v2.8.2 on examples/client at main: client.gen.go: 14 lines differ
sdkgen 4.44.3 on voxgig-solardemo-sdk's own document, go/ compared with the reference:
  identical 61, differ 31 (1159 lines), only in the reference 21, only in the run 0
    test/omni/omni.go (244 lines)
    utility/struct/voxgigstruct.go (150 lines)
    test/moon_entity_test.go (119 lines)
  only in the reference, by first two path elements: feature/secrets, feature/secrets_feature.go, test/feature
```

At the v2.8.2 tag the two lines are the header, one version string on each side.[^2] On `main`,
the example uses `runtime.JSONUnmarshal`, which v2.8.2 does not write, so reading `main` shows you a
release nobody can install yet.

sdkgen's reference was made by 4.34.x, and 4.44.3 differs in 31 Go files. The 21 files only in the
reference are the `secrets` feature: the run passed `-f secrets`, and 4.44.3 wrote none of its
source. A second run with `main: kit: feature: secrets: active: true` in
`.sdk/model/project.aontu` wrote it, so the feature stays off until the overlay activates it.

## Read the oapi-codegen and Speakeasy references the same way

oapi-codegen ran here. Its example carries the marker on `client.gen.go`, and the configuration
beside it: `cfg.yaml`, and `generate.go` holding the `//go:generate` line. That makes it the
easiest reference of the four to verify, because two committed files regenerate it. Its
[README](https://github.com/oapi-codegen/oapi-codegen) recommends committing the generated code,
so its own examples have output to read. It beats sdkgen for a reader who needs to see that
the reference is exactly what the tool writes.

Speakeasy was not run, because generation needs an account. Its reference is read from three
files. The marker sits on each generated file. `.genignore` is documented as removing matched
files from generation and granting ownership of them. `.speakeasy/gen.lock` names the
`speakeasyVersion` and `generationVersion` that produced the output. A marker on every generated file lets
you judge any single file on its own; sdkgen's partial markers send you to the record.

## Check it worked

The tests pin the survey logic and each figure the page quotes from it.

```bash
node --test judge.test.mjs
```

```text output
1..6
# tests 6
# suites 0
# pass 6
# fail 0
# cancelled 0
# skipped 0
# todo 0
```

## When it goes wrong

You credit glue to the generator. The polished entry class is in the ignore file. Read
`.fernignore` or `.genignore` before any source file, and score only what the generator wrote.

The survey shows no hand-kept files, and you still find edits. A Replay patch or a persistent edit
lives inside a marked file. Read `.fern/replay.lock` or the `persistentEdits` setting, and list
the files its patches name.

Your first run looks nothing like the reference. The reference came from `main` or from an older
release. Regenerate it with the release you would install, as `report.mjs regen` does, and score
that output instead.

## When not to do this

Do not choose a generator from a reference alone. Each one here is its generator's best case on a
document that suits it, so pair the reading with a run on your own document.

Do not score sdkgen from voxgig-solardemo-sdk without regenerating it. Voxgig maintains both the
generator and the reference, which was made ten minor versions before 4.44.3. Of its 113 files
under `go/`, 31 differ from what that release writes and 21 more were not written at all.

Do not stop at the oapi-codegen example when your SDKs are not Go-only. It is pure output that
regenerates from two files, but it covers one language, so it cannot show the TypeScript or
Python clients you would also need.

Do not read the Mistral client as a sample of what any Speakeasy account produces. Its hook files
rely on SDK hooks, which the hooks page makes available to Business and Enterprise users only.

## Related how-tos

- [Judge whether an unofficial SDK is safe to depend on](/howto/judge-an-unofficial-sdk-before-adoption)

- [Check whether two generator runs produce identical output](/howto/check-two-generator-runs-are-identical)

- [Fail a build when a generator customization becomes a fork](/howto/audit-generator-customisations-for-drift)

- [Generate your first SDK with sdkgen](/howto/first-sdk-with-sdkgen)

- [Tell an entity-shaped SDK from an endpoint-shaped one](/howto/measure-entity-versus-endpoint-shaped-output)

## Last verified

Verified 2026-11-21 against Node 22.23.5, `@voxgig/create-sdkgen` 0.31.9, `@voxgig/sdkgen` 4.44.3,
`@voxgig/apidef` 8.32.4 and oapi-codegen v2.8.2. The references were cloned at the commits
`fetch.sh` pins and read from their files, without running their code. Every output block is what the
command preceding it printed. Speakeasy and Fern were not run; their behavior comes from their
documentation and from the files in the two references.

[^1]: Speakeasy's [SDK hooks page](https://www.speakeasy.com/docs/sdks/customize/code/sdk-hooks)
says the registration file is generated once and will not be overwritten, after which you have
full control and ownership of it. The same page says hooks are available for Business and
Enterprise users only, so a free account cannot reproduce the design of Mistral's eight hook files.

[^2]: The header on the tag's copy reads `version v2.0.0-00010101000000-000000000000`, because the
example is generated by the generator built from its own source tree. The
[Go modules reference](https://go.dev/ref/mod#pseudo-versions) uses the form
`vX.0.0-yyyymmddhhmmss-abcdefabcdef` when there is no known base version. Here the timestamp is
midnight on the first of January in year one, and the commit is twelve zeros.