How to evaluate a third-tier SDK target before you ship it#

The generator’s table lists Ruby, so the docs site lists Ruby, and the first customer to install the gem finds that retries raise NotImplementedError. The gem built and the README rendered without anybody running anything against it. The table listed a target. It never said the target was the same product as the TypeScript one.

What you get

You will end up with a scorecard for any candidate target, produced by the same six steps every time. A file records the score beside the generator’s own claim for the language. This is for you if a language is about to appear on your docs site because it appeared in a table.

Short answer

Generate the candidate, compile it, load it, and run your shared cross-language corpus against it. Validate its packaging manifest with the ecosystem’s own checker. Count the stub methods, the TODO comments, and the runtime features with a real body. Write the result beside what the generator claims for that language. A target that compiles is not a target you support.

You will need

Node 22 or later, Ruby 3.2 or later, a generated candidate target, and a cross-language test corpus your flagship target already passes. Verified 2026-09-24 against Node 22.22.2 and Ruby 3.3.6. The two targets here are stand-ins written for this page: a JavaScript flagship that passes everything, and a Ruby candidate with the defects a third-tier target has. Ruby plays the candidate because it is on the runner. The evaluator does not care which generator produced them.

Voxgig maintains sdkgen. This page compares its ungraded list of bundled targets with the stability labels OpenAPI Generator publishes and the maturity levels Speakeasy publishes.

Approaches compared

The three claims differ in kind, not only in level. OpenAPI Generator labels each generator STABLE, BETA, EXPERIMENTAL or DEPRECATED, which grades the templates.¹ Speakeasy grades each target GA, Beta or Alpha and adds a feature-support level beside it.² sdkgen lists twenty-two bundled targets and grades none of them, which is not a claim at all and is the reason to measure. None of the three is a number you produced against your own API.

Run the same six steps on every target

Each target directory carries a target.json naming how to compile it, load it, run the corpus and find its manifest. The evaluator reads that and nothing else about the language.

export function evaluate(language, dir = join(HERE, 'targets', language)) {
  const target = JSON.parse(readFileSync(join(dir, 'target.json'), 'utf8'))
  const result = { language, tier: target.tier, steps: {} }
  const compile = run(target.compile, dir)
  result.steps.compile = { ok: compile.ok, detail: compile.ok ? target.compile.join(' ') : compile.out.split('\n')[0] }
  if (!compile.ok) return { ...result, verdict: 'does not compile' }
  const load = run(target.load, dir)
  result.steps.load = { ok: load.ok, detail: load.ok ? target.load.slice(-1)[0] : load.out.split('\n')[0] }
  result.steps.corpus = corpus(target.corpus, dir)
  result.steps.manifest = manifest(dir, target.manifest)
  const source = target.sources.map((f) => readFileSync(join(dir, f), 'utf8')).join('\n')
  result.steps.stubs = stubs(source, language)
  result.steps.features = features(source, language)
  const c = result.steps.corpus
  const supported = load.ok && c.passed === c.total && result.steps.manifest.ok && !result.steps.stubs.stubbed.length && !result.steps.features.missing.length
  return { ...result, verdict: supported ? 'supported' : 'compiles, not supported' }
}

Compile is node --check for the flagship and ruby -c for the candidate, and a failure there ends the evaluation, because nothing after it means anything. The verdict has two values on the far side of compiling, and only one of them is supported.

node evaluate.mjs ruby js

target: ruby (candidate)
  compile    ok      ruby -c lib/meterco.rb
  load       ok      require 'meterco'
  corpus     2 of 4  FAIL: get by id encodes the path: path /meters/mtr 1, expected /meters/mtr%201; create carries an idempotency key: NotImplementedError: idempotency_key is not implemented for ruby
  manifest   FAIL    meterco.gemspec: missing value for attribute summary
  stubs      3       with_retry, idempotency_key, log_request raise not implemented; 2 TODO comments
  features   3 of 6  missing with_retry, idempotency_key, log_request
  verdict    compiles, not supported

target: js (flagship)
  compile    ok      node --check src/client.mjs
  load       ok      import('./src/client.mjs').then(() => {})
  corpus     4 of 4  every case passed
  manifest   ok      package.json: name, version, license, exports and files present
  stubs      0       none; 0 TODO comments
  features   6 of 6  all implemented
  verdict    supported

claims for ruby, recorded 2026-09-24
  OpenAPI Generator  STABLE                       a stability label per generator
  Speakeasy          GA, feature support level 1  a maturity level and a feature level per target
  sdkgen             bundled, no grade            22 bundled targets, none graded
  measured           2 of 4 corpus cases, 3 stubs, 3 of 6 features, manifest invalid
wrote evaluation.json

Read the candidate’s block top to bottom. It compiles and loads, which is where a table stops. The corpus then fails two of four cases: the path is not URL-encoded, and creating a meter raises because the idempotency method is a stub. The manifest fails the ecosystem’s own check. Three of six runtime features are stubs. The verdict line is the one to copy into the decision.

Count a feature only when it has a body

A search for the method cannot tell a stub from an implementation, because a stub has a name, a signature, and a docstring too. The count has to read the body.

export function stubs(source, language) {
  const defs = language === 'ruby'
    ? [...source.matchAll(/^\s*def (\w+)[^\n]*\n([\s\S]*?)^\s*end$/gm)]
    : [...source.matchAll(/^\s*(?:async\s+)?\*?(\w+)\([^)]*\)\s*\{\n([\s\S]*?)^\s*\}$/gm)]
  const stubbed = defs.filter(([, , body]) => /NotImplementedError|not implemented|throw new Error\(['"]TODO/i.test(body)).map(([, name]) => name)
  const todos = (source.match(/\bTODO\b/g) ?? []).length
  return { stubbed, todos }
}

/** A feature counts as implemented when its marker method exists and is not a stub. */
export function features(source, language) {
  const marker = FEATURES.markers[language]
  const { stubbed } = stubs(source, language)
  const defined = (name) => (language === 'ruby' ? new RegExp(`^\\s*def ${name}\\b`, 'm') : new RegExp(`^\\s*(?:async\\s+)?\\*?${name}\\(`, 'm')).test(source)
  const implemented = FEATURES.features.filter((f) => defined(marker[f]) && !stubbed.includes(marker[f]))
  const missing = FEATURES.features.filter((f) => !implemented.includes(f)).map((f) => marker[f])
  return { implemented, missing, total: FEATURES.features.length }
}

features.json names the six runtime features your SDK promises and the method that carries each one in every language, so the count is against your list and not the generator’s. sdkgen’s feature list has nineteen entries. The count is the same measurement with a longer list, and a longer list is more places for a stub to hide.

Let the ecosystem judge the manifest

The evaluator does not know what a valid gemspec is. RubyGems does, and Gem::Specification#validate is the check gem build runs.

export function manifest(dir, file) {
  const path = join(dir, file)
  if (!existsSync(path)) return { ok: false, detail: `${file} is missing` }
  if (file.endsWith('.gemspec')) {
    const r = run(['ruby', '-e', `begin; Gem::Specification.load(${JSON.stringify(file)}).validate; puts 'valid'; rescue Gem::InvalidSpecificationException => e; puts e.message; exit 1; end`], dir)
    return { ok: r.ok, detail: `${file}: ${r.out.split('\n').filter((l) => !l.startsWith('WARNING')).join(' ').trim()}` }
  }
  const pkg = JSON.parse(readFileSync(path, 'utf8'))
  const missing = ['name', 'version', 'license', 'exports', 'files'].filter((k) => !(k in pkg))
  return { ok: !missing.length, detail: missing.length ? `${file}: missing ${missing.join(', ')}` : `${file}: name, version, license, exports and files present` }
}

The candidate’s gemspec fails on summary, which the specification reference lists among the five required attributes.³ The npm side is a field check, because npm’s own validation happens at publish time. Use the ecosystem’s validator wherever one runs offline, and say in the report which manifests got a real check and which got a list of keys.

Check it worked

node --test evaluate.test.mjs

1..7
# tests 7
# suites 0
# pass 7
# fail 0
# cancelled 0
# skipped 0
# todo 0
# duration_ms 1258.166003

Two tests carry the page’s claim. One asserts that the candidate compiles, loads, and is still compiles, not supported. The other copies the candidate to a scratch directory, appends a stray end to its source, and asserts the verdict is does not compile with the compile step as the only step recorded. A third edits the scratch gemspec to add a summary and a license, and asserts RubyGems then accepts it, so the failure the page shows is one the reader can fix.

When it goes wrong

The corpus reports 0 of 0 and the candidate’s other numbers look fine. The runner died before its summary line. In Ruby, NotImplementedError descends from ScriptError, not from StandardError, so a bare rescue does not catch it,⁴ and the first stubbed method took the whole runner down. The runner rescues both classes, and the evaluator reads passed n of m rather than counting ok lines, so a runner that dies scores zero instead of scoring the cases it reached.

A feature counts as implemented and a customer finds it empty. The count matched the method name. Read the body, as stubs does, and put the generator’s own phrase for a stub in the pattern, because each generator has one.

The candidate passes the corpus and fails in production. The corpus checks request shapes, not transport. A target can build every request correctly and still have no timeout, which is why features are counted separately from cases.

The label and the measurement disagree. OpenAPI Generator marks its perl and ocaml generators STABLE, and a stable template still produces output that has never met your API. Record both, and let the measurement decide.

When not to do this

Do not put a target on the docs site because it compiles. Deciding who fixes a bug in it at two in the morning is the evaluation’s last step, and a target with no name against it is not supported whatever the scorecard says. Write the name in the same file as the score.

Do not read sdkgen’s bundled list as a support matrix. It publishes twenty-two bundled targets and grades none of them, and the same pipeline and corpus behind every target is the reason the measurement is cheap, not a reason to skip it. The evaluation here is the grading sdkgen leaves to you.

Do not run the evaluation once. A generator release, a feature added to the flagship, or a new corpus case moves the score. A score from the release before last is a table entry with a date on it. Run it in the build for every target you ship.

Do not treat a passing corpus as idiomatic. The corpus says the requests are right. Whether a Ruby developer would call the client Ruby is a different review, and this page does not do it.

Related how-tos

• Judge whether an unofficial SDK is safe to depend on

• Fail a build when a generator customization becomes a fork

• Generate your first SDK with sdkgen

Last verified

Verified 2026-09-24 against Node 22.22.2 and Ruby 3.3.6. Both output blocks are what the preceding command printed. No generator was run, because the two targets are stand-ins written for this page rather than generator output. The three claims in claims.json were read from the pages they cite on 2026-09-24.