How to verify every error response matches one schema#

An agent calling your API retries a 503 forever because the body was upstream unavailable in plain text, and its error handler expected an object with a type.¹ Your description says every failure is a problem document. One route disagrees, a mismatch the build could not catch because nothing in it ever compared the two.

What you get

You will end up with two checks that together cover the ways an error contract leaks: a lint over the description, and a conformance check against the running service. This is for you if you publish a problem schema and want it to be true.

Short answer

Run two checks. A lint over the description fails when a declared 4XX or 5XX carries anything but application/problem+json, which is cheap and catches the paperwork. A runtime check calls each failing path and compares the status, the media type, and the required members. Only that half catches a status the service returns and the description never mentions.

You will need

An OpenAPI 3 description with a problem schema, and Node 22 or later. The media type and the required members come from RFC 9457, and the response declarations live in the Responses Object.

Approaches compared

Each of these sees a different part of the problem. The lint reads your intent. The fuzzer finds statuses your intent never covered. Hand-written cases are the only ones that check whether the type URI and the title say something a client can act on. Two of the three are enough, and the lint should be one of them because it costs nothing. Order them by how cheap they are to run: the lint on every commit, the hand-written cases with the rest of the suite, the fuzzer on a schedule that opens an issue.

Lint what the description declares

The rule is short, and the second half of it matters more than the first.

for (const [status, response] of Object.entries(responses)) {
  if (!FAILURE.test(status)) continue
  const types = Object.keys(response.content ?? {})
  if (!types.includes('application/problem+json')) {
    findings.push(`${verb.toUpperCase()} ${path} ${status}: declares ${types.join(', ') || 'no body'}, not application/problem+json`)
  }
}

A default response is the trap.² It makes every status conform on paper, because any code the service returns is covered by something. Declare each status you actually return, and keep default for the genuinely unexpected. A description whose only failure entry is default tells a client generator nothing, and the generated client will have no typed errors at all.

Keep the list of statuses somewhere the service and the description both read from. A constant in the code, checked by the lint, is enough. Two hand-maintained lists drift in the same week they are written.

Check what the service returns

The runtime half needs a running service and three assertions per call.

if (!declared[status]) findings.push(`${status} is not declared, only default covers it`)
if (type !== 'application/problem+json') findings.push(`content type is ${type}`)

Check the media type separately from the body. A route that returns the right JSON under application/json passes any check that only parses the body. It still breaks a client that switches on the media type before deciding how to read the payload.

Drive the check from a list of failing requests you maintain beside the description. That list is also the answer when a security review asks which failures are tested.

Check it worked

Run both halves over one description and one service.

node demo.mjs

-- static: what the description declares
  GET /meters/{id} 404: declares application/json, not application/problem+json
  GET /meters/{id} 503: returned by the service, declared only by default
-- runtime: what the service actually returns
  /meters?limit=none     400  conforms
  /meters?burst=yes      429  conforms
  /meters/missing        404  content type is application/json
  /meters/offline        503  503 is not declared, only default covers it; content type is text/plain; body is not JSON

The two halves overlap on the 404 and diverge on the 503. The lint sees the 404 because the description is wrong about it. The runtime check sees the 503 fully, because only a real call reveals a plain text body where a problem document was promised.³ Neither check alone covers both.

The two conforming rows matter as much as the failing ones. They show the checks can pass, which is the property a gate needs before anyone will trust its failures.

node --test conform.test.mjs

1..6
# tests 6
# suites 0
# pass 6
# fail 0
# cancelled 0
# skipped 0
# todo 0
# duration_ms 279.939139

When it goes wrong

The lint passes, but clients still see odd bodies. A framework is handling an error before your code does, and returning its own default page. Add a catch-all that converts anything unhandled into a problem document, and test it by throwing on purpose. A route that throws a string rather than an Error is the usual way past a handler that checks the type first.

A gateway rewrites the body. Timeouts and rate limits answered by a proxy carry the proxy’s format, not yours. Configure the error response at that layer too, or document the difference.

Every status conforms and the fields are useless. The schema requires type, title and status, and all three are present and generic. Assert on the values for the failures you care about, not only on the shape. Check the declared types while you are there: a body of {"type": 42, "title": null, "status": "400"} carries every required name, and a check that counts keys calls it conforming. That is the exact body a generated client’s error handler breaks on.

A generated client has one error class. The description hides every failure behind default. Declare the statuses, and the generator will give you the types.

The checks pass in CI and fail in production. The service behind the load balancer is not the one the tests started, and a layer between them is answering some failures itself. Run the conformance list against a deployed environment once per release, not only against a local process.

When not to do this

Do not fail a deploy on a fuzzer run. Generated requests find real problems and they also find paths that are slow, flaky, or dependent on data. Run the fuzzer on a schedule, open an issue, and keep the deploy gate on the lint and the hand-written cases.

Do not put the conformance check behind an integration environment you rarely run. A check that runs weekly tells you which week broke it. Run it against a service started by the test process.

Do not treat a problem document as a place for a stack trace. The members are for a client, and detail reaching a caller with internal paths in it is an information leak wearing a schema.

Related how-tos

• Return validation errors a client can map to a form

• Validate responses against the OpenAPI document at runtime

Last verified

Verified 2026-09-14 against Node 22.22.2. Both output blocks are what the preceding command printed.