How to design the case shape of a language-neutral test corpus#

The Go port fails an ambiguous case the TypeScript port passes, so nobody can say which port is wrong. The case has an expected field in one file and an out field in the next. One runner reads null as “expect nothing,” the other reads it as “expect null,” and the second runner was written from a guess about the first. A corpus meant to settle arguments has started one.

What you get

You will end up with a case shape written as a JSON Schema, and a corpus that CI refuses when a case breaks it. Two runners, one in Node and one in Python, read the same file without guessing. This is for you if you maintain a library with ports and want one set of tests to hold them together.

Short answer

Give every case a stable id, a doc line, exactly one in, and exactly one of out or err. Write that rule as a JSON Schema and validate the whole corpus in CI with Ajv. Reserve three strings for what JSON cannot say: __UNDEF__ for no argument, __NULL__ for an expected null, and __EXISTS__ for any value. Never use JSON null as an expectation, because a runner cannot tell it from no expectation.

You will need

Node 22 or later and Ajv for the validator, Python 3.11 or later for the second runner, and a function with a port in another language. The one here parses a Retry-After header, whose two forms RFC 9110 defines as a count of seconds or an HTTP-date. Verified 2026-09-25 against Node 22.22.2, Ajv 8.20.0 and Python 3.11.15. The three corpora in the table are read from their own repositories and not run here.

Approaches compared

Grouping shared setup cuts repetition and makes cases order-dependent, because a runner has to carry the group’s schema to every test under it. Paired files diff cleanly and stand alone, and double the file count while telling you nothing about why an invalid input is invalid. The shape below takes the flat list from CommonMark, the explicit expectation from the JSON Schema suite, and adds the one field neither has: the error a failing input must produce.

Write the shape down before the second runner

The shape is a JSON Schema, and the corpus is an array of cases that must match it. The interesting keywords are the ones that say what a case may not carry.

    "required": [
      "id",
      "doc",
      "in"
    ],
    "properties": {
      "id": {
        "description": "Stable, unique, and the name every runner reports a failure under.",
        "type": "string",
        "pattern": "^[a-z0-9]+(-[a-z0-9]+)*$"
      },
      "doc": {
        "description": "Why this case exists, in one sentence a failing runner can print.",
        "type": "string",
        "minLength": 12
      },
      "in": {
        "description": "The one input. Any JSON value, or the sentinel \"__UNDEF__\" for no argument."
      },
      "out": {
        "description": "The expected value. Any JSON value except null: write \"__NULL__\" to expect null and \"__EXISTS__\" to accept any value.",
        "not": {
          "type": "null"
        }
      },
      "err": {
        "description": "The expected error code, when the call must fail.",
        "type": "string",
        "pattern": "^[a-z]+(_[a-z]+)*$"
      }
    },
    "additionalProperties": false,
    "oneOf": [
      {
        "required": [
          "out"
        ],
        "not": {
          "required": [
            "err"
          ]

Four decisions are in that block. additionalProperties: false is the one that stops a second input field from appearing under a new name, which is how expected and out came to coexist in the opening paragraph. The oneOf makes an expectation mandatory and single. The id pattern is what every runner will name a test after, so it allows nothing a test framework would mangle. And out refuses JSON null outright, which is the pitfall this page exists for.¹

Validate the corpus in CI

Ajv compiles the schema once and reports every violation when allErrors is set. The one rule the schema cannot express is that ids are unique across the file, so the validator checks that itself.

node validate.mjs cases.json bad-cases.json

cases.json: ok
bad-cases.json: 6 problems
  /0/out must NOT be valid
  /1 must match exactly one schema in oneOf
  /2 must match exactly one schema in oneOf
  /3/id must match pattern "^[a-z0-9]+(-[a-z0-9]+)*$"
  /4 must NOT have additional properties: input
  /6/id duplicates /5/id: delay-seconds

Each line names the case by its position and the field that broke the rule. The first is a case whose out is JSON null; the last is a duplicate id the schema never saw. Wire the command into CI and a corpus edit that breaks the shape fails the pull request that made it, before any runner in any language gets to guess.

Say what JSON cannot

JSON has null and nothing else in that direction. It has no way to write “no argument” and no way to write “anything at all” as values.² Three reserved strings carry those meanings, and the runner is where they are decoded.

const UNDEF = '__UNDEF__'
const NULL = '__NULL__'
const EXISTS = '__EXISTS__'

// Key order is not part of a JSON value, so objects compare with their keys sorted, as
// run_cases.py compares them with sort_keys.
const canonical = (value) => JSON.stringify(value, (_, v) => (v && typeof v === 'object' && !Array.isArray(v)
  ? Object.fromEntries(Object.entries(v).sort(([a], [b]) => (a < b ? -1 : a > b ? 1 : 0)))
  : v))

/** Run `fn` against every case; returns the failures, each naming the case id. */
export function runCases(cases, fn) {
  const failures = []
  for (const c of cases) {
    let result
    let error
    try {
      result = c.in === UNDEF ? fn() : fn(c.in)
    } catch (e) {
      error = e
    }
    if ('err' in c) {
      if (!error) failures.push({ id: c.id, doc: c.doc, why: `expected error ${c.err}, got ${JSON.stringify(result)}` })
      else if (error.code !== c.err) failures.push({ id: c.id, doc: c.doc, why: `expected error ${c.err}, got ${error.code ?? error.message}` })
      continue
    }
    if (error) { failures.push({ id: c.id, doc: c.doc, why: `threw ${error.code ?? error.message}` }); continue }
    const ok = c.out === EXISTS ? result !== undefined
      : c.out === NULL ? result === null
      : canonical(result) === canonical(c.out)
    if (!ok) failures.push({ id: c.id, doc: c.doc, why: `expected ${JSON.stringify(c.out)}, got ${JSON.stringify(result)}` })
  }
  return failures
}

That is the whole runner, and the Python one in run_cases.py is the same thirty lines with different punctuation. Both read cases.json, twelve cases for the Retry-After parser.

node run-cases.mjs && python3 run_cases.py

node runner: 12 cases, 12 passed, 0 failed
python runner: 12 cases, 12 passed, 0 failed

The corpus earned its keep before the ports agreed. The first Node draft handed anything that was not digits to Date.parse, which accepts "-5" as a year, rolls the thirty-first of February into March, and reads a numeric zone. The Python port refuses all three. Point the same runner at that draft, kept as retry-after-lenient.mjs, and the corpus names each case and prints its doc line.

node run-cases.mjs cases.json ./retry-after-lenient.mjs

node runner (./retry-after-lenient.mjs): 12 cases, 9 passed, 3 failed
  negative-seconds: expected error retry_after_invalid, got 0
    The grammar allows digits only, so a sign is an error.
  http-date-impossible: expected error retry_after_invalid, got 259200
    The 31st of February does not exist, and a parser that rolls it into March is guessing.
  http-date-numeric-zone: expected error retry_after_invalid, got 60
    The ports read one date form and a numeric zone is not it, so neither accepts what the other refuses.

The fix was to accept only the IMF-fixdate form, the one RFC 9110 has a sender generate, and only a date that prints back the same. The cases stay in the corpus so the next port cannot make the same choices. Two more came from the ports themselves. Python’s round sends half a second to the even number where Math.round goes up, and its \d reads Arabic-Indic digits as seconds. RFC 9110 also has a recipient accept two obsolete date forms, which neither port does. A port that takes them on adds their cases first, so the other ports hear of it.

Check it worked

The test that matters is the one that shows a null expectation being refused, because a schema that lets it through has silently allowed two meanings for one value.

test('a JSON null expectation is refused, so __NULL__ is the only way to expect null', () => {
  assert.deepEqual(one(bad[0]), ['/0/out must NOT be valid'])
  assert.deepEqual(one({ ...bad[0], out: '__NULL__' }), [])
})

node --test shape.test.mjs

# tests 10
# suites 0
# pass 10
# fail 0
# cancelled 0
# skipped 0
# todo 0

Of the other nine, five pin the oneOf, the id pattern, the extra field, the duplicate check, and the sentinel decoding. One checks the shipped corpus, and one checks that objects compare without regard to key order, as they do in the Python runner. Two more pin the lenient port’s three failures, and a naive runner that passes a broken implementation because it read null as “no expectation.”

When it goes wrong

Two ports pass and one of them is wrong. A case has "out": null, and one runner compares it while the other skips cases whose expectation is null. Refuse JSON null in out, as the schema here does, and write __NULL__ when null is the answer. The last test in shape.test.mjs shows a function that never returns null passing the naive runner.

Two cases have the same id and the schema passed. JSON Schema can require unique items, not unique values of one field across items, so a duplicate id is a runner-side check. The validator here does it in six lines; put the same six in every runner that names tests by id.

A case passes alone and fails in the file. Cases share setup through a group, and a runner ran them in a different order or without the group. Keep every case self-contained, and if setup must be shared, copy it into each case and let the file be longer.

A runner treats a sentinel as a plain string. A function that happens to return the text "__NULL__" passes an __EXISTS__ case and fails a __NULL__ one, or the reverse. Decode the sentinels before comparing, and keep the three strings out of the domain the corpus tests.

When not to do this

Do not design a shape for one language. A field that holds a regular expression, a date, or a big integer looks portable in JSON and is not. JavaScript and Go disagree about integers past 2^53, and every regular expression dialect differs. Carry those as strings with a stated format, and test the decoding in each runner.

Do not group cases under shared setup because it saves lines. The JSON Schema Test Suite does it, and every implementer of that suite writes the loop that re-applies the schema to each test. Repetition in a corpus is cheap; a runner that must reconstruct context is not.

Do not borrow toml-test’s tagged encoding unless your inputs have a format of their own. Its {"type": "integer", "value": "42"} exists because TOML has types JSON lacks. For a function that takes and returns JSON values, plain values are the shape, and a wrapper is a second grammar every runner must learn.

Do not put expected errors in a file’s absence. toml-test marks an invalid input by the directory it sits in and says nothing about which error it expects, so a decoder that fails for the wrong reason still passes.³ Name the error code.

Related how-tos

• Validate JSON model output before you use it

• Verify every error response matches one schema

Last verified

Verified 2026-09-25 against Node 22.22.2, Ajv 8.20.0 and Python 3.11.15. Every output block is what the command preceding it printed, in the code directory. The three corpora compared were read from their repositories and their specification rather than run here.