# How to design the case shape of a language-neutral test corpus > Fix the fields every case carries before you write a second runner: a stable id, a doc line, one input, one expectation, and sentinels for what JSON cannot say. Source: https://voxgig.com/howto/design-the-case-shape-of-a-test-corpus - Audience: library-maintainer - Level: intermediate - Verified: 2026-09-25 - Published: 2026-09-25 ## Short answer Give every case a stable id, a doc line, exactly one in, and exactly one of out or err. Write that rule as a JSON Schema and validate the whole corpus in CI with Ajv. Reserve three strings for what JSON cannot say: __UNDEF__ for no argument, __NULL__ for an expected null, and __EXISTS__ for any value. Never use JSON null as an expectation, because a runner cannot tell it from no expectation. --- ## You will need Node 22 or later and Ajv for the validator, Python 3.11 or later for the second runner, and a function with a port in another language. The one here parses a `Retry-After` header, whose two forms [RFC 9110](https://www.rfc-editor.org/rfc/rfc9110.html#name-retry-after) defines as a count of seconds or an HTTP-date. Verified 2026-09-25 against Node 22.22.2, Ajv 8.20.0 and Python 3.11.15. The three corpora in the table are read from their own repositories and not run here. ## Approaches compared | Approach | When it fits | What it costs you | When to pick something else | | --- | --- | --- | --- | | [CommonMark spec tests](https://spec.commonmark.org/0.31.2/) | The expected output is text, and the specification itself is where the examples live | A script that extracts cases from prose, example numbers that shift when the text does, and one output format for every case | The cases are not illustrations of a document somebody is also reading | | [JSON Schema Test Suite](https://github.com/json-schema-org/JSON-Schema-Test-Suite) | Many inputs share one piece of setup, and the answer is a boolean | Cases nested under a shared schema cannot be run alone or reordered without carrying the group, and every runner rebuilds the nesting | The expectation is a value rather than a verdict, or cases must stand alone | | [toml-test](https://github.com/toml-lang/toml-test) | The input has its own file format and the runner is a separate process | Two files per case, a tagged JSON encoding that turns every value into a string, and invalid cases that carry no expected error at all | Inputs are small JSON values, where a second file per case doubles the noise | Grouping shared setup cuts repetition and makes cases order-dependent, because a runner has to carry the group's schema to every test under it. Paired files diff cleanly and stand alone, and double the file count while telling you nothing about why an invalid input is invalid. The shape below takes the flat list from CommonMark, the explicit expectation from the JSON Schema suite, and adds the one field neither has: the error a failing input must produce. ## Write the shape down before the second runner The shape is a JSON Schema, and the corpus is an array of cases that must match it. The interesting keywords are the ones that say what a case may not carry. ```json title="case-shape.schema.json" "required": [ "id", "doc", "in" ], "properties": { "id": { "description": "Stable, unique, and the name every runner reports a failure under.", "type": "string", "pattern": "^[a-z0-9]+(-[a-z0-9]+)*$" }, "doc": { "description": "Why this case exists, in one sentence a failing runner can print.", "type": "string", "minLength": 12 }, "in": { "description": "The one input. Any JSON value, or the sentinel \"__UNDEF__\" for no argument." }, "out": { "description": "The expected value. Any JSON value except null: write \"__NULL__\" to expect null and \"__EXISTS__\" to accept any value.", "not": { "type": "null" } }, "err": { "description": "The expected error code, when the call must fail.", "type": "string", "pattern": "^[a-z]+(_[a-z]+)*$" } }, "additionalProperties": false, "oneOf": [ { "required": [ "out" ], "not": { "required": [ "err" ] ``` Four decisions are in that block. `additionalProperties: false` is the one that stops a second input field from appearing under a new name, which is how `expected` and `out` came to coexist in the opening paragraph. The `oneOf` makes an expectation mandatory and single. The `id` pattern is what every runner will name a test after, so it allows nothing a test framework would mangle. And `out` refuses JSON `null` outright, which is the pitfall this page exists for.[^1] ## Validate the corpus in CI [Ajv](https://ajv.js.org/guide/getting-started.html) compiles the schema once and reports every violation when `allErrors` is set. The one rule the schema cannot express is that ids are unique across the file, so the validator checks that itself. ```bash node validate.mjs cases.json bad-cases.json ``` ```text output cases.json: ok bad-cases.json: 6 problems /0/out must NOT be valid /1 must match exactly one schema in oneOf /2 must match exactly one schema in oneOf /3/id must match pattern "^[a-z0-9]+(-[a-z0-9]+)*$" /4 must NOT have additional properties: input /6/id duplicates /5/id: delay-seconds ``` Each line names the case by its position and the field that broke the rule. The first is a case whose `out` is JSON `null`; the last is a duplicate `id` the schema never saw. Wire the command into CI and a corpus edit that breaks the shape fails the pull request that made it, before any runner in any language gets to guess. ## Say what JSON cannot JSON has `null` and nothing else in that direction. It has no way to write "no argument" and no way to write "anything at all" as values.[^2] Three reserved strings carry those meanings, and the runner is where they are decoded. ```js title="run-cases.mjs" const UNDEF = '__UNDEF__' const NULL = '__NULL__' const EXISTS = '__EXISTS__' // Key order is not part of a JSON value, so objects compare with their keys sorted, as // run_cases.py compares them with sort_keys. const canonical = (value) => JSON.stringify(value, (_, v) => (v && typeof v === 'object' && !Array.isArray(v) ? Object.fromEntries(Object.entries(v).sort(([a], [b]) => (a < b ? -1 : a > b ? 1 : 0))) : v)) /** Run `fn` against every case; returns the failures, each naming the case id. */ export function runCases(cases, fn) { const failures = [] for (const c of cases) { let result let error try { result = c.in === UNDEF ? fn() : fn(c.in) } catch (e) { error = e } if ('err' in c) { if (!error) failures.push({ id: c.id, doc: c.doc, why: `expected error ${c.err}, got ${JSON.stringify(result)}` }) else if (error.code !== c.err) failures.push({ id: c.id, doc: c.doc, why: `expected error ${c.err}, got ${error.code ?? error.message}` }) continue } if (error) { failures.push({ id: c.id, doc: c.doc, why: `threw ${error.code ?? error.message}` }); continue } const ok = c.out === EXISTS ? result !== undefined : c.out === NULL ? result === null : canonical(result) === canonical(c.out) if (!ok) failures.push({ id: c.id, doc: c.doc, why: `expected ${JSON.stringify(c.out)}, got ${JSON.stringify(result)}` }) } return failures } ``` That is the whole runner, and the Python one in `run_cases.py` is the same thirty lines with different punctuation. Both read `cases.json`, twelve cases for the `Retry-After` parser. ```bash node run-cases.mjs && python3 run_cases.py ``` ```text output node runner: 12 cases, 12 passed, 0 failed python runner: 12 cases, 12 passed, 0 failed ``` The corpus earned its keep before the ports agreed. The first Node draft handed anything that was not digits to `Date.parse`, which accepts `"-5"` as a year, rolls the thirty-first of February into March, and reads a numeric zone. The Python port refuses all three. Point the same runner at that draft, kept as `retry-after-lenient.mjs`, and the corpus names each case and prints its `doc` line. ```bash node run-cases.mjs cases.json ./retry-after-lenient.mjs ``` ```text output node runner (./retry-after-lenient.mjs): 12 cases, 9 passed, 3 failed negative-seconds: expected error retry_after_invalid, got 0 The grammar allows digits only, so a sign is an error. http-date-impossible: expected error retry_after_invalid, got 259200 The 31st of February does not exist, and a parser that rolls it into March is guessing. http-date-numeric-zone: expected error retry_after_invalid, got 60 The ports read one date form and a numeric zone is not it, so neither accepts what the other refuses. ``` The fix was to accept only the `IMF-fixdate` form, the one RFC 9110 has a sender generate, and only a date that prints back the same. The cases stay in the corpus so the next port cannot make the same choices. Two more came from the ports themselves. Python's `round` sends half a second to the even number where `Math.round` goes up, and its `\d` reads Arabic-Indic digits as seconds. RFC 9110 also has a recipient accept two obsolete date forms, which neither port does. A port that takes them on adds their cases first, so the other ports hear of it. ## Check it worked The test that matters is the one that shows a `null` expectation being refused, because a schema that lets it through has silently allowed two meanings for one value. ```js title="shape.test.mjs" test('a JSON null expectation is refused, so __NULL__ is the only way to expect null', () => { assert.deepEqual(one(bad[0]), ['/0/out must NOT be valid']) assert.deepEqual(one({ ...bad[0], out: '__NULL__' }), []) }) ``` ```bash node --test shape.test.mjs ``` ```text output # tests 10 # suites 0 # pass 10 # fail 0 # cancelled 0 # skipped 0 # todo 0 ``` Of the other nine, five pin the `oneOf`, the `id` pattern, the extra field, the duplicate check, and the sentinel decoding. One checks the shipped corpus, and one checks that objects compare without regard to key order, as they do in the Python runner. Two more pin the lenient port's three failures, and a naive runner that passes a broken implementation because it read `null` as "no expectation." ## When it goes wrong Two ports pass and one of them is wrong. A case has `"out": null`, and one runner compares it while the other skips cases whose expectation is `null`. Refuse JSON `null` in `out`, as the schema here does, and write `__NULL__` when null is the answer. The last test in `shape.test.mjs` shows a function that never returns null passing the naive runner. Two cases have the same id and the schema passed. JSON Schema can require unique items, not unique values of one field across items, so a duplicate id is a runner-side check. The validator here does it in six lines; put the same six in every runner that names tests by id. A case passes alone and fails in the file. Cases share setup through a group, and a runner ran them in a different order or without the group. Keep every case self-contained, and if setup must be shared, copy it into each case and let the file be longer. A runner treats a sentinel as a plain string. A function that happens to return the text `"__NULL__"` passes an `__EXISTS__` case and fails a `__NULL__` one, or the reverse. Decode the sentinels before comparing, and keep the three strings out of the domain the corpus tests. ## When not to do this Do not design a shape for one language. A field that holds a regular expression, a date, or a big integer looks portable in JSON and is not. JavaScript and Go disagree about integers past 2^53, and every regular expression dialect differs. Carry those as strings with a stated format, and test the decoding in each runner. Do not group cases under shared setup because it saves lines. The JSON Schema Test Suite does it, and every implementer of that suite writes the loop that re-applies the schema to each test. Repetition in a corpus is cheap; a runner that must reconstruct context is not. Do not borrow toml-test's tagged encoding unless your inputs have a format of their own. Its `{"type": "integer", "value": "42"}` exists because TOML has types JSON lacks. For a function that takes and returns JSON values, plain values are the shape, and a wrapper is a second grammar every runner must learn. Do not put expected errors in a file's absence. toml-test marks an invalid input by the directory it sits in and says nothing about which error it expects, so a decoder that fails for the wrong reason still passes.[^3] Name the error code. ## Related how-tos - [Validate JSON model output before you use it](/howto/validate-json-model-output-in-typescript) - [Verify every error response matches one schema](/howto/verify-every-error-response-matches-one-schema) ## Last verified Verified 2026-09-25 against Node 22.22.2, Ajv 8.20.0 and Python 3.11.15. Every output block is what the command preceding it printed, in the code directory. The three corpora compared were read from their repositories and their specification rather than run here. [^1]: The JSON Schema Test Suite's own [README](https://raw.githubusercontent.com/json-schema-org/JSON-Schema-Test-Suite/main/README.md) draws the line this page draws. A test case is a schema with a description and an array of tests; a test is a description, an instance, and a boolean. It ships a [`test-schema.json`](https://raw.githubusercontent.com/json-schema-org/JSON-Schema-Test-Suite/main/test-schema.json) that formalizes the layout, down to a `specification` array of references. Each entry names a section of a JSON Schema draft, an RFC, an ISO standard, ECMA-262 or the Perl regex documentation, or carries a quote. A corpus that validates its own shape against a schema is the natural condition of a corpus for schemas, and a good habit for the rest. [^2]: [RFC 8259](https://www.rfc-editor.org/rfc/rfc8259.html) gives JSON exactly one literal for absence, `null`, and gives an object member no way to be present without a value. So "call with no argument" and "the answer is null" both want the same character sequence. The three sentinels here are the same three [omni](https://github.com/voxgig/omni#readme) documents for its own spec files: `__NULL__`, `__UNDEF__`, and `__EXISTS__`. A corpus that borrows the names can be read by its runners. [^3]: The toml-test [README](https://raw.githubusercontent.com/toml-lang/toml-test/master/README.md) sets the rule for a decoder in two lines: on invalid input, exit non-zero; on valid input, print the tagged JSON. The tagged JSON uses objects, arrays, and strings only, so an integer is written as the string `"42"` inside an object that names its type. A test suite for a configuration format with integers, floats, and four kinds of date declined to use any of JSON's numbers. The reason is the one this page keeps meeting: a number in JSON does not say what kind of number it is.