# How to audit an SDK generator's language coverage

> Generate one operation for each target you plan to ship, score every output on one checklist, and list where it contradicts the published matrix.

Source: https://voxgig.com/howto/audit-generator-language-coverage-claims

- Audience: api-producer
- Level: intermediate
- Languages: typescript, python, go
- Verified: 2026-11-21
- Published: 2026-11-21

## Short answer

Generate one operation from your own OpenAPI document, for every target you plan to ship, with each generator you are considering, such as sdkgen, OpenAPI Generator and `openapi-python-client`. Score each output on one checklist. It compiles unedited, the error body is typed, pagination is handled or left to you, the manifest's runtime floor holds when tested, and the module has commits of its own. Then list every disagreement with the published matrix.

---
## You will need

An OpenAPI 3 document, Node 22 or later, Python with [uv](https://docs.astral.sh/uv/), Go, and
Java 21 for OpenAPI Generator. Verified 2026-11-21 against `@voxgig/create-sdkgen` 0.31.9,
`@voxgig/sdkgen` 4.44.3, OpenAPI Generator 7.27.0 and `openapi-python-client` 0.30.0. The
document here, `api.json`, holds one operation, `GET /books`, with a `cursor` and a `limit`, a
`BookPage` response carrying `next`, and a 400 answered with an `Error` schema.

## Approaches compared

| Approach | When it fits | What it costs you | When to pick something else |
| --- | --- | --- | --- |
| [Fern](https://buildwithfern.com/learn/sdks/overview/introduction) | You want a hosted product with a quickstart for each listed language | Generation needs a login, a guest session, or Docker, and its overview grades no language | Your document cannot go to a hosted service and Docker is unavailable |
| [OpenAPI Generator](https://openapi-generator.tech/docs/generators/) | You want a stability label per generator and a public history to check it against | A label says stable or beta, not when the module last had a change of its own | You want one feature model shared by every target |
| [`openapi-python-client`](https://github.com/openapi-generators/openapi-python-client) | Python is the only target, and you want the error body typed per status | One language, and its README says it does not support all OpenAPI features | You ship TypeScript or Go from the same document |
| [sdkgen](https://github.com/voxgig/sdkgen) | You want one model and one feature set behind every target you add | It grades none of its targets, so every score here had to be measured | You need a vendor-graded matrix before you commit |
| [Speakeasy](https://www.speakeasy.com/docs/sdks/languages/maturity) | You want a maturity level per target with written definitions of each level | Generation needs an account, so a GA label cannot be scored before signing up | You want to measure output offline first |

Voxgig maintains sdkgen. It is one of five generators here, scored on the same checklist as the
two others that ran.

Speakeasy publishes the most precise claims, GA for all three targets with each level defined,
and is the one you can test least before you have an account. OpenAPI Generator labels every
generator and keeps its history public, but its label lags its activity. sdkgen publishes no
maturity label. Its [features reference](https://github.com/voxgig/sdkgen/blob/main/docs/reference/features.md#language-parity)
says every bundled language target implements every feature and lists `ts`, `py` and `go` among
the targets that run its shared test corpus, a claim this checklist does not test.

## Generate one operation for three targets

Keep the operation small and give it the three things a checklist can fail: query parameters for
a cursor, a page envelope, and a documented error body. `generate.sh` runs each generator once on
that document and records every check under `evidence/`, once, outside the gate. The sdkgen part scaffolds three targets,
builds each, and then tests the floor each manifest states on that runtime's oldest admitted
version.

```bash title="generate.sh"
npx --yes @voxgig/create-sdkgen@0.31.9 bookshelf --def api.json -t ts -t py -t go >/dev/null 2>&1
(cd bookshelf-sdk/.sdk && npm run generate >/dev/null 2>&1)
S=bookshelf-sdk
(cd $S/ts && npm install --no-audit --no-fund >/dev/null 2>&1)
rec sdkgen-ts-compile $S/ts npm run build
rec sdkgen-py-compile $S/py python3 -c 'import bookshelf_sdk'
rec sdkgen-go-compile $S/go go build ./...
npm install --prefix node2213 --no-audit --no-fund node@22.13.0 >/dev/null 2>&1
rec sdkgen-ts-floor $S/ts "$WORK/node2213/node_modules/node/bin/node" -e "require('./dist/BookshelfSDK.js'); console.log('loaded on', process.version)"
rec sdkgen-py-floor-3.8 $S/py "$(py 3.8)" -c 'import bookshelf_sdk'
rec sdkgen-py-floor-3.9 $S/py "$(py 3.9)" -c 'import bookshelf_sdk'
rec sdkgen-py-floor-3.10 $S/py "$(py 3.10)" -c 'import bookshelf_sdk; print("imported")'
rec sdkgen-go-floor $S/go env GOTOOLCHAIN=go1.21.13 go build ./...
```

`rec` keeps each command's exit code and last line of output. OpenAPI Generator runs its
`typescript-fetch`, `python` and `go` generators the same way. Only the floor runs need extra
installs: uv fetches each Python, and [`GOTOOLCHAIN`](https://go.dev/doc/toolchain) makes Go
fetch an older toolchain.[^1]

## Score every output on one checklist

`audit.mjs` reads only the committed records and manifests, so the scorecard re-runs offline.

```bash
node audit.mjs
```

```text output
target                             compiles error for the 400           pagination     manifest floor   label  last change 12 mo (own)
sdkgen ts                          yes      one SDK error, body untyped left to caller node >=22.13     none   2026-11-19  305 (43)
sdkgen py                          yes      one SDK error, body untyped left to caller python >=3.8     none   2026-11-19  240 (7)
sdkgen go                          yes      one SDK error, body untyped left to caller go 1.21          none   2026-11-19  250 (17)
OpenAPI Generator typescript-fetch yes      raw Response, body untyped  left to caller none             stable 2026-11-18  60 (47)
OpenAPI Generator python           yes      Error schema typed          left to caller python >=3.11    stable 2026-10-28  63 (46)
OpenAPI Generator go               yes      Error schema typed          left to caller go 1.23.0        stable 2026-11-10  26 (17)
openapi-python-client              yes      Error schema typed          left to caller python ^3.11     none   2026-11-07  21 (-)

floor runs
  sdkgen ts                          22.13   exit 0  loaded on v22.13.0
  sdkgen py                          3.8     exit 1  TypeError: 'type' object is not subscriptable
  sdkgen py                          3.9     exit 1  TypeError: unsupported operand type(s) for |: 'type' and 'NoneType'
  sdkgen py                          3.10    exit 0  imported
  sdkgen go                          1.21.13 exit 0  
  OpenAPI Generator python           3.10    exit 0  imported
  OpenAPI Generator go               1.23.0  exit 0  
  OpenAPI Generator go               1.21.13 exit 1  go: go.mod requires go >= 1.23.0 (running go 1.21.13; GOTOOLCHAIN=go1.21.13)
  openapi-python-client              3.10    exit 1  ImportError: cannot import name 'Self' from 'typing' (pythons/cpython-3.10-linux-x86_64-gnu/lib/python3.10/typing.py)

a target outside the three: OpenAPI Generator lua, labelled beta by its own list command
  last change 2026-10-24, 5 commits in 12 months, 0 touching lua alone
  last commit naming lua: 8608103c9fd 2023-09-05 add name mapping features to lua client generator (#16511)

disagreements and gaps
  sdkgen py: manifest says python >=3.8, fails on 3.8: TypeError: 'type' object is not subscriptable
  sdkgen py: manifest says python >=3.8, fails on 3.9: TypeError: unsupported operand type(s) for |: 'type' and 'NoneType'
  sdkgen py: manifest says python >=3.8, its dependency requests>=2.33 needs python >=3.10
  OpenAPI Generator typescript-fetch: no manifest written, so no runtime range stated
  sdkgen: grades none of its targets, so only the measurement says how good each is
  openapi-python-client: grades none of its targets, so only the measurement says how good each is
  Speakeasy: publishes typescript GA, python GA, go GA; not run here, so unmeasured
  Fern: publishes typescript, python, go; not run here, so unmeasured
```

Every output compiled unedited, so the first item on the checklist separates nothing here. The
other four do.

The error column is where the targets differ most. OpenAPI Generator's `python` and `go` output
decode the 400 into the document's `Error` model. Its `typescript-fetch` output generates a
`ModelError` type and still hands you a raw `Response`. It also writes no `package.json`, and so
states no Node floor, because its [`npmName` option](https://openapi-generator.tech/docs/generators/typescript-fetch/)
is required for a full package. sdkgen raises one `BookshelfError` per
target, with a status and a code, and generated no type for the `Error` schema in any of the
three.

Pagination is left to the caller everywhere. The sdkgen model records the cursor, as
`next: "body.next"` in `.sdk/model/entity/book.aontu`, but no generated code reads it until you
add its `paging` feature.

The floor is the column a matrix never shows. sdkgen's `pyproject.toml` admits 3.8; the code uses
`list[str]` and `str | None`, which need 3.9 and 3.10.[^2] Its one dependency,
[requests 2.33.0](https://pypi.org/project/requests/2.33.0/), needs 3.10 as well. The newest
release, `@voxgig/sdkgen` 4.45.0, wrote the same `pyproject.toml`, and its package failed the same
two imports.

## Read the history behind a label

The last change date misleads on its own. OpenAPI Generator's `lua` client, labelled beta on its
[generator page](https://openapi-generator.tech/docs/generators/lua/), last changed on 2026-10-24.
Each of its five commits in the 12 months ending 2026-11-21 also touched other generators, and the
last commit whose subject names Lua was made on 2023-09-05. The `own` count, commits that touched no
other target, is the number that says whether anyone works on a module.

`generate.sh` read those numbers on 2026-11-21 from a clone of each repository made with
`git clone --filter=blob:none --no-checkout`, which fetches the history without the file contents.
`history.py` then runs `git log --since --until --name-only` for each target's paths and checks
the files of every commit for other targets.

sdkgen's three targets show the opposite shape: hundreds of commits in the year, most of them
shared. `openapi-python-client` is one target in one repository, so every commit is its own.

## Hold Speakeasy and the Python-only client to the same checklist

`openapi-python-client` ran here and is in the scorecard. Its `pyproject.toml` asks for
`python = "^3.11"`, and the floor holds: the import fails on 3.10 on `typing.Self`. `sync()`
returns `BookPage | Error | None`, so a 400 arrives as a typed `Error` value. A status the document
does not list returns `None`, or raises `errors.UnexpectedStatus` when the client is built with
`raise_on_unexpected_status=True`. Its
[README](https://github.com/openapi-generators/openapi-python-client) states that it does not
support all OpenAPI features, which is a gap it publishes rather than one you find.

Speakeasy was not run, because its [quickstart](https://www.speakeasy.com/docs/sdks/create-client-sdks)
prompts for an account before it generates. What it publishes is checkable once you have one: the
[maturity matrix](https://www.speakeasy.com/docs/sdks/languages/maturity) defines GA as a fully
supported release, and its [pagination page](https://www.speakeasy.com/docs/sdks/customize/runtime/pagination)
documents `x-speakeasy-pagination`. Add both to the checklist row for each target, and add its
output to `generate.sh` once you can run it.

Fern was not run either. Its [SDK commands](https://buildwithfern.com/learn/cli-api-reference/cli-reference/sdk-commands)
generate on Fern's servers after a login or on a `--guest` session, or in Docker with `--local`.
Its overview, written for the `sdk-config.yml` format, lists each language with a version and no
grade. Its [pricing page](https://buildwithfern.com/pricing) puts Go on the Enterprise plan and
only Python and TypeScript on the free one.

## Check it worked

The test fails if the audit stops reporting a floor the code misses, or starts reporting a
failure below the floor that the manifest correctly refuses.

```bash
node --test audit.test.mjs
```

```text output
1..10
# tests 10
# suites 0
# pass 10
# fail 0
# cancelled 0
# skipped 0
# todo 0
# duration_ms 73.075082
```

## When it goes wrong

A floor run fails on a version the manifest excludes. Go 1.21.13 refuses OpenAPI Generator's
module because its `go.mod` says `go 1.23.0`. That is the manifest working, so test the floor
itself and the version below it, and score only a failure at a version the manifest admits.

The import works on your machine and fails for users. The check ran under the newest interpreter
you have. Install the floor version with `uv python install 3.8` and import there.

A module looks active because its last change was last month. Count the commits that touched
that module alone. A sweep across every generator moves the date without anyone reading the
target.

## When not to do this

Do not run this audit for a target you will not ship. Each target costs a runtime install per
floor version, and the result decays as each generator releases.

Do not treat sdkgen's missing grades as a sign every target is equal: the Python manifest in the scorecard
claimed two Python versions its code cannot load. Do not choose sdkgen for a target you cannot
test yourself.

Do not discard Speakeasy because it was not measured here. Choose it when a vendor's written
definition of GA matters more to you than measuring before signing up.
Choose `openapi-python-client` when Python is the only target and a typed error per status is the
property you need most.

## Related how-tos

- [Decide which SDK language to add next](/howto/decide-which-sdk-language-to-add-next)

- [Evaluate a third-tier SDK target before you ship it](/howto/evaluate-a-third-tier-sdk-target)

- [Set support tiers for your SDK languages](/howto/set-support-tiers-for-sdk-languages)

- [Generate your first SDK with sdkgen](/howto/first-sdk-with-sdkgen)

- [Judge a generator by reading a reference SDK it produced](/howto/judge-a-generator-by-its-reference-sdk)

## Last verified

Verified 2026-11-21 against Node 22.23.5, `@voxgig/create-sdkgen` 0.31.9, `@voxgig/sdkgen`
4.44.3, `@voxgig/apidef` 8.32.4, OpenAPI Generator 7.27.0, `openapi-python-client` 0.30.0,
Python 3.13.18, Go 1.24.7 and OpenJDK 21.0.13. The generators, the builds, and the floor runs ran
once, in `generate.sh`. The floor runs used Node 22.13.0, Python 3.8.20, 3.9.25, 3.10.20 and 3.11.15
from uv, and Go 1.21.13 and 1.23.0 through `GOTOOLCHAIN`. `@voxgig/sdkgen` 4.45.0 ran once on the
Python floor.
The gate re-runs only `audit.mjs` and its test, over the records that run committed. Every output
block is what the command preceding it printed. Speakeasy and Fern were not run; their claims come
from their documentation.

[^1]: Go's [toolchain documentation](https://go.dev/doc/toolchain) says that before Go 1.21 a
toolchain treated the `go` line as an advisory requirement, and that Go 1.21 made it mandatory.
sdkgen's module says `go 1.21`, the first version for which the line it writes is binding rather
than a suggestion.

[^2]: The two errors are two proposals, one per release. [PEP 585](https://peps.python.org/pep-0585/)
let `list[str]` stand as a type from Python 3.9, and [PEP 604](https://peps.python.org/pep-0604/)
let `str | None` stand for an optional from 3.10. The 3.8 interpreter stops at the first and the
3.9 interpreter at the second, so the traceback dates the syntax to the release.