# How to score SDK generators on your API with a repeatable rubric

> Record what each SDK generator does with your own OpenAPI document, keep the raw measurements beside per-team weights, and rescore without a generator run.

Source: https://voxgig.com/howto/score-sdk-generators-with-a-repeatable-rubric

- Audience: platform-team
- Level: intermediate
- Verified: 2026-11-21
- Published: 2026-11-21

## Short answer

Run each candidate generator twice on your own OpenAPI document and build what it writes. Record the evidence: operations reached, languages built, identical runs, runtime dependencies, license, and probes for the schemas and auth scheme your API uses. Score that evidence with one weights file per team, and make a must-have probe a gate rather than points. On the Depot document here, sdkgen, oapi-codegen and openapi-typescript reached all eight operations and differed on the probes.

---
## You will need

An OpenAPI 3 document for your API, Node 22 or later, and Go. oapi-codegen v2.8.2 needs Go 1.25,
and its runtime module v1.9.1 needs Go 1.26; the `go` command fetches each toolchain itself when
[`GOTOOLCHAIN=auto`](https://go.dev/doc/toolchain) is in effect. The sample's document, `api.json`,
describes a parcel API with eight operations, a `oneOf` with a discriminator, a bearer scheme, and a
cursor parameter. Verified 2026-11-21.

## Approaches compared

| Approach | When it fits | What it costs you | When to pick something else |
| --- | --- | --- | --- |
| [Hand-written SDK](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API) | One language, a small API, and ergonomics no generator produces | Every column of the rubric becomes a review task, repeated for each language you add | A second language, or a document that changes monthly |
| [oapi-codegen](https://github.com/oapi-codegen/oapi-codegen) | Go only, and you want one generated file to review | Go alone, three runtime modules, and one method per operation for callers to learn | You ship clients in more than one language |
| [openapi-typescript and openapi-fetch](https://openapi-ts.dev/introduction) | TypeScript only, and you want types with a small runtime | A path-keyed call style, and auth left to a middleware you write | Callers want named methods, or you need another language |
| [sdkgen](https://github.com/voxgig/sdkgen) | Several languages from one model, with entity-shaped clients | A model to read, a large generated tree, and gaps where the model drops schemas | One language, or your API leans on `oneOf` responses |

Voxgig maintains sdkgen. It is one of four approaches on this page, and its row lists its costs.

The two single-language generators are cheaper to review, because each writes one file, and they
typed the event and quote fields the probes look for. sdkgen built TypeScript and Go from one model
whose manifests declare no runtime package, at the price of 187 generated files and two failed probes.

## Write probes from your own document

A probe is a check your document needs and Petstore cannot ask for. Write one per feature your API
depends on, as regular expressions that must each match a line of generated code.

```json title="probes.json"
{
  "api": [
    { "name": "oneOf variants typed", "patterns": ["\\bfacility\\b", "\\bsignedBy\\b"] },
    { "name": "array response typed", "patterns": ["\\bcarrier\\b", "\\bpriceCents\\b"] },
    { "name": "bearer scheme in code", "patterns": ["['\"`]Bearer['\"`]"] }
  ],
  "petstore": [
    { "name": "x-next paging header typed", "patterns": ["x-next"] }
  ]
}
```

`facility` and `signedBy` are fields of the two variants of `TrackingEvent`. `carrier` and
`priceCents` belong to the array that `quoteRates` returns. `probe.mjs` removes `//` and `/* */`
comments before it matches, because a probe that counts a comment passes for a generator that
wrote nothing.

## Run every candidate twice and record the evidence

`record.sh` scaffolds or runs each generator twice into empty directories, builds the first run,
and writes `evidence/`. The sdkgen step:

```bash title="record.sh"
  for RUN in 1 2; do
    status "$DOC sdkgen run $RUN"
    D="$WORK/$DOC/sdkgen-run$RUN"
    rm -rf "$D" && mkdir -p "$D" && cp "$SRC" "$D/$DOC.json"
    (cd "$D" && npx --yes "$CREATE_SDKGEN" "$NAME" --def "$DOC.json" --target ts --target go > scaffold.log 2>&1)
    (cd "$D/$NAME-sdk/.sdk" && npm run generate > ../../generate.log 2>&1)
  done
```

Each run's file hashes go into `run1.sha256` and `run2.sha256`. An operation counts as reached when
the output has a handle for its method and path. In sdkgen that is a point in the extracted model,
in oapi-codegen a `*Client` method named after the `operationId`, and in `openapi-typescript` a
path entry typed with `operations[...]`. Runtime dependencies are what the generated manifests declare, so code a
generator copies into its output does not count. For sdkgen the comparison covers the
`ts/` and `go/` trees. The record files under `.sdk/log` and `.jostraca` differed between the two
scaffolds, so a comparison of the whole project reports a difference the SDK does not have. The
scoring scripts read only `evidence/`, so a team can reweight offline.

## Read the raw measurements before any total

```bash
node measure.mjs api
```

```text output
api.json: 8 operations, 3 probes
measured                sdkgen              oapi-codegen        openapi-typescript
version                 4.44.3              v2.8.2              7.13.0
operations reached      8/8                 8/8                 8/8
languages built         2/2 go,typescript   1/1 go              1/1 typescript
two runs identical      yes, 187 files      yes, 1 file         yes, 1 file
names a caller learns   3 entities          8 methods           5 paths
runtime dependencies    0                   3                   2
license                 MIT                 Apache-2.0          MIT
oneOf variants typed    no                  yes                 yes
array response typed    no                  yes                 yes
bearer scheme in code   yes                 no                  no
```

Coverage ties, and so does repeatability. Every row is recomputed from the committed `evidence/`,
so a team that doubts a cell reruns `record.sh` and compares. The probes separate the three. sdkgen's model kept no
fields for `TrackingEvent` and took the `rate` entity's fields from the request body, so neither
the event fields nor the quote fields reached its code. It reached `cancelShipment` as a second
point of `create`, selected by an `$action` key:

```aontu title="evidence/api/sdkgen/entity/shipment.aontu"
  op: {
    create: {
      name: "create"
      points: [
        {
          co: {
            id: "POST /shipments/{shipmentId}/cancel"
            source: "openapi3"
            version: 2
          }
```

The same run on `@voxgig/sdkgen` 4.45.0 with `@voxgig/apidef` 8.33.1, the highest releases on npm
that day, renamed `tracking_event` to `event` and failed the same two probes. No probe covers
pagination. sdkgen's model records the cursor as `body.nextCursor`, and the scaffold added no
`paging` feature to either target.

## Check unions and auth in oapi-codegen and openapi-fetch

oapi-codegen reads a YAML file passed with `-config`. Its
[README](https://github.com/oapi-codegen/oapi-codegen#usage) says the remaining command-line options
should be considered deprecated, so `record.sh` writes `package`, `output`, and `generate` with
`models: true` and `client: true`. The `oneOf` became a union type with `AsScanEvent`,
`FromScanEvent`, `MergeScanEvent` and a `Discriminator` method, which is why it passed the first
probe. No auth code was generated for the bearer scheme. A caller adds the header with a
`RequestEditorFn`, or with `NewSecurityProviderBearerToken` from its
[`securityprovider` package](https://github.com/oapi-codegen/oapi-codegen/blob/main/pkg/securityprovider/securityprovider.go).

`openapi-typescript` writes types only,[^2] and `openapi-fetch` makes the calls with
`client.GET('/shipments')`. The union came out as
`components["schemas"]["ScanEvent"] | components["schemas"]["DeliveryEvent"]`. Its
[auth guide](https://openapi-ts.dev/openapi-fetch/middleware-auth) says the library works with any
Authorization setup, and sets the header in a middleware's `onRequest`, which is code you own. The sample's
`callers/api.caller.ts` calls all eight operations, and `tsc --strict` accepted it.

## Reweight per team, and gate on the must-haves

Each weights file lists the languages a team needs, the licenses it accepts, its budget, its
must-haves, and a weight for every column. The two files here are examples for a team to replace,
not a recommendation. The platform team needs TypeScript and Go:

```bash
node score.mjs api weights/platform.json
```

```text output
weights/platform.json: platform team shipping TypeScript and Go clients
column (weight)         sdkgen              oapi-codegen        openapi-typescript
coverage (3)            1.00                1.00                1.00
builds (2)              1.00                1.00                1.00
repeatable (2)          1.00                1.00                1.00
shape (1)               0.63                0.00                0.38
runtime (1)             1.00                0.25                0.33
footprint (1)           0.31                1.00                1.00
languages (3)           1.00                0.50                0.50
probes (3)              0.33                0.67                0.67
license (1)             1.00                1.00                1.00
price (1)               1.00                1.00                1.00
total                   82.9                76.4                78.9
rank: sdkgen, openapi-typescript, oapi-codegen
```

Under these weights sdkgen ranks first, from the languages and runtime columns, and the total hides
that it failed the event probe. The web team renders every tracking event, so it makes that probe a
must:

```bash
node score.mjs api weights/web-team.json | tail -3
```

```text output
total                   out                 73.6                90.7
out: sdkgen fails a must, oneOf variants typed
rank: openapi-typescript, oapi-codegen
```

Under the web team's file sdkgen is out, removed by the must before any weight applied, and
`openapi-typescript` ranks first. Publish the raw table beside every total, so a reader can redo
the judgement call the weights made.

## Check it worked

```bash
node --test rubric.test.mjs
```

```text output
1..10
# tests 10
# suites 0
# pass 10
# fail 0
# cancelled 0
# skipped 0
# todo 0
```

The tests fail if an operation drops out of any output, or if a probe counts a comment. They also
fail if a recorded probe disagrees with a rerun on the committed sources, or if two differing runs
read as identical.
`score` also refuses a weights file that leaves out a column, so a missing weight cannot read as 0.

## When it goes wrong

The ranking flips when the document changes. Petstore asks none of the Depot probes, so under the
same weights it ranks the three in another order, with sdkgen last:

```bash
node score.mjs petstore weights/platform.json | tail -1
```

```text output
rank: openapi-typescript, oapi-codegen, sdkgen
```

The generators and the weights were the same in both runs, and only the document changed. Score on
the document you ship, and write a probe
for each `oneOf`, scheme, and paging rule it declares.

A probe passes on a comment. A generator that mentions `Bearer` in a doc comment and applies
nothing reads as supporting the scheme. Strip comments before matching, as `probe.mjs` does.

Two runs look different because the comparison included log files. Compare the generated trees,
and list what you left out.

## When not to do this

Do not build a rubric to choose between two generators when one of them cannot produce your
language. The language column decides, and the rest is ceremony.

Do not read sdkgen's lead under the platform weights as a verdict. It comes from the languages and
runtime columns, and a team whose API returns `oneOf` events should gate on that probe first.
oapi-codegen is the better fit for a Go-only service, because it writes one reviewable file and
passed both schema probes. `openapi-typescript` with `openapi-fetch` suits a TypeScript web app that
needs every union typed and can own its auth middleware.

Do not treat a grep probe as proof of behavior. It shows that a name reached the code, so test a
must-have with a call against a stand-in server before you sign off.

## Related how-tos

- [Generate your first SDK with sdkgen](/howto/first-sdk-with-sdkgen)

- [Check whether two generator runs produce identical output](/howto/check-two-generator-runs-are-identical)

- [Tell an entity-shaped SDK from an endpoint-shaped one](/howto/measure-entity-versus-endpoint-shaped-output)

- [Judge a generator by reading a reference SDK it produced](/howto/judge-a-generator-by-its-reference-sdk)

- [Compare an AI-written API client with generated clients](/howto/compare-an-ai-written-client-with-generator-output)

## Last verified

Verified 2026-11-21 against Node 22.23.5, `@voxgig/create-sdkgen` 0.31.9, `@voxgig/sdkgen` 4.44.3,
`@voxgig/apidef` 8.32.4, oapi-codegen v2.8.2 with its runtime v1.9.1, `openapi-typescript` 7.13.0,
`openapi-fetch` 0.17.0 and TypeScript 5.9.3. The generators ran once through `record.sh`, which
wrote the committed `evidence/`, and again in a copy, which wrote an identical tree. Every output
block is what the command preceding it printed from that evidence. The sheet scores only these three
generators, because OpenAPI Generator, Kiota, Speakeasy and Fern, which the page's brief also
names, were not run for it.

[^1]: The Petstore document comes from the OpenAPI Initiative's
[example descriptions](https://learn.openapis.org/examples/), whose footer licenses the work under
Creative Commons Attribution 4.0 International. The file's own `info.license` block names MIT,
which the [specification](https://spec.openapis.org/oas/v3.0.3.html#info-object) defines as the
license information for the exposed API. The MIT license therefore covers a pet shop that does not
exist, while the file describing it is under CC BY 4.0.

[^2]: The `openapi-typescript` [governance file](https://github.com/openapi-ts/openapi-typescript/blob/main/GOVERNANCE.md)
describes it as a 100% volunteer open source project that no legal entity backs financially, and
says it accepts sponsorship to maintain it. The price input in `inputs.json` records what a user
pays, which is zero, and is a different number from what the project costs its maintainers.