Your OpenAPI document promises that GET /orders/{orderId} answers 200 with an Order, but the
service renamed a field in that Order two releases ago. No test compares the two: the document
is read by people, and the suite asserts what the code does rather than what the document says.
The first client generated from the document breaks on its first call.
What you get
You will end up with a contract suite that Specmatic runs from your document and a directory of examples. It fails when the service disagrees with the document or an operation has no example. This is for you if you own the service you describe.
Short answer
Put one JSON example per documented response in a directory named orders_examples beside
orders.yaml, start the service, and run specmatic test --strict --testBaseURL=<url> orders.yaml. Specmatic sends each example’s request and checks the response against the
document. Set successCriteria to minCoveragePercentage: 100 with enforce: true in
specmatic.yaml, so a documented response with no example is not tested, lowers coverage, and
fails the run.
You will need
Node 22 or later for the service and the scripts, a Java runtime for Specmatic, and an OpenAPI 3
document with a service you can start in CI. Specmatic ships as a jar inside the
specmatic npm package, so npm ci fetches it and the
java on your PATH runs it.1 Verified 2026-10-21 against Node 22.22.0, OpenJDK 21.0.12.1 and
Specmatic 2.55.3.
The sample service is server.mjs, an in-memory orders API with two operations, started on a
free port by every script here. Nothing here makes a network call: disableTelemetry in
specmatic.yaml stops the usage report Specmatic
posts at exit by default.
Approaches compared
| Approach | When it fits | What it costs you | When to pick something else |
|---|---|---|---|
| Pact provider verification | Consumers you know publish the interactions they rely on | A pact per consumer, fetched from a broker, a URL, or a path, and a handler for every provider state a pact names | No consumer writes pacts, or the document is the contract you want enforced |
| Schemathesis | You want generated requests that probe the schema’s edges rather than curated ones | A Python toolchain or its Docker image, many generated requests per operation, and a different set on every run unless you pin --seed | Every case must be a fixed request a reviewer can read in a diff |
| Specmatic | The document is the contract and every case should be a readable example | A JVM at test time, one JSON file per response to keep in step with the service, and a wrapper that masks the exit code | You have no document, or you want the requests nobody thought to write |
Schemathesis finds the request nobody wrote down, at the cost of a Python toolchain and a different set of requests on every run unless you pin a seed. Pact verifies only what a consumer asked for, so it says nothing about an operation no consumer uses yet. Specmatic sits between them: each case is a file a reviewer can read, and each file is a thing that goes stale when the service moves.
Keep one example per documented response
Specmatic reads examples from a directory named after the document, orders_examples beside
orders.yaml, and picks it up by convention.
A directory elsewhere needs --examples=<dir>. One file holds one request and the response it
should get, and its filename is the example’s name in every report.
{
"http-request": {
"method": "GET",
"path": "/orders/ord_nope"
},
"http-response": {
"status": 404,
"headers": { "Content-Type": "application/json" },
"body": { "title": "no such order" }
}
}The four files cover the four documented responses. A valid order produces the 201 and an order
with quantity: 0 the 400; an id the service seeds produces the 200 and an id it never issues the
404. The 400 example breaks the request schema on purpose, and Specmatic’s documentation says it
does not enforce contract validity
for an example meant to produce a 400. Every other file has to agree with the document, which
specmatic examples validate --spec-file orders.yaml checks without a running service.
Run the suite in strict mode against a fresh service
contract-test.mjs starts server.mjs on a free port, runs the jar’s test command against it,
and keeps the output lines that carry no timestamp and no absolute path.
// Runs one Specmatic command and resolves with its exit code and output. A JVM killed by
// a signal closes with code null, and process.exitCode = null exits 0, so that counts as 1.
export function specmatic(args, { cwd = HERE } = {}) {
return new Promise((resolve, reject) => {
const child = spawn('java', ['-jar', JAR, ...args], { cwd, stdio: ['ignore', 'pipe', 'pipe'] })
let output = ''
child.stdout.on('data', (chunk) => { output += chunk })
child.stderr.on('data', (chunk) => { output += chunk })
child.on('error', reject)
child.on('close', (code) => resolve({ code: code ?? 1, output, lines: summarise(output) }))
})
}
// The contract test: a fresh service, then `specmatic test` pointed at it. The document
// and specmatic.yaml are read from cwd, so a caller can run the same suite over a copy.
export async function contractTest({ options = ['--strict'], serverEnv = {}, cwd = HERE } = {}) {
const server = await startServer(serverEnv)
try {
return await specmatic(['test', `--testBaseURL=${server.url}`, ...options, 'orders.yaml'], { cwd })
} finally {
server.stop()
}
}node contract-test.mjsScenario: POST /orders -> 201 with the request from the example 'place_order_201' has SUCCEEDED
Scenario: POST /orders -> 400 with the request from the example 'place_order_400' has SUCCEEDED
Scenario: GET /orders/(orderId:string) -> 200 with the request from the example 'get_order_200' has SUCCEEDED
Scenario: GET /orders/(orderId:string) -> 404 with the request from the example 'get_order_404' has SUCCEEDED
|----------------------------------------------------------------------------------------------------------------|
| SPECMATIC API COVERAGE SUMMARY |
|----------------------------------------------------------------------------------------------------------------|
| coverage | path | method | requestContentType | response | responseContentType | remarks | result |
|----------|-------------------|--------|--------------------|----------|---------------------|---------|--------|
| 100% | /orders | POST | application/json | 201 | application/json | covered | 1p |
| | | | | 400 | application/json | covered | 1p |
| 100% | /orders/{orderId} | GET | NA | 200 | application/json | covered | 1p |
| | | | | 404 | application/json | covered | 1p |
|----------------------------------------------------------------------------------------------------------------|
| * = Operation not eligible for coverage |
| ! = Operation excluded from the run due to a filter |
| p = passed tests |
| f = failed tests |
| w = wip tests |
| |
| 100% API Coverage reported from 4 operations eligible for coverage |
| 100% Absolute Coverage (includes excluded operations that were not tested) |
|----------------------------------------------------------------------------------------------------------------|
Tests run: 4, Successes: 4, Failures: 0, WIP: 0, Errors: 0Each scenario line names the operation, the status, and the example file the request came from.
The --strict flag is what makes the list complete. Without it, Specmatic
generates a positive test
for an operation with no example, filling the request from the schema, so the gap is covered by a
request nobody reviewed. The table is the coverage report. A row reads covered when a response
matched it, not tested when no test ran against it, and not implemented when a test ran and
the service did not answer as the document says.
The harness runs the jar through java -jar rather than npx specmatic, for a reason shown
below.
Fail the build when a response has no example
Coverage is the lever. Under --strict an operation with no example is skipped, so it is not
covered, and the report configuration
turns coverage below a threshold into a failed run.
version: 3
specmatic:
settings:
general:
# Specmatic posts usage analytics at exit unless told not to.
disableTelemetry: true
governance:
report:
formats:
- html
outputDirectory: build/reports/specmatic
# Under --strict an operation with no example is not tested, so it is not
# covered, so a run that skips one fails this threshold.
successCriteria:
minCoveragePercentage: 100
maxMissedOperationsInSpec: 0
enforce: truemissing-example.mjs copies the document, this configuration, and three of the four examples into
a temporary directory, then runs the same suite from there.
node missing-example.mjsScenario: POST /orders -> 201 with the request from the example 'place_order_201' has SUCCEEDED
Skipping Scenario: POST /orders -> 400 (requestContentType application/json, responseContentType application/json)
- T00004: Generative Disabled
Summary: This operation was skipped because it requires schema resiliency to be enabled
- T00003: Examples Required in Strict Mode
Summary: Strict mode requires at least one example, but none were found for this operation
Scenario: GET /orders/(orderId:string) -> 200 with the request from the example 'get_order_200' has SUCCEEDED
Scenario: GET /orders/(orderId:string) -> 404 with the request from the example 'get_order_404' has SUCCEEDED
|-------------------------------------------------------------------------------------------------------------------|
| SPECMATIC API COVERAGE SUMMARY |
|-------------------------------------------------------------------------------------------------------------------|
| coverage | path | method | requestContentType | response | responseContentType | remarks | result |
|----------|-------------------|--------|--------------------|----------|---------------------|------------|--------|
| 50% | /orders | POST | application/json | 201 | application/json | covered | 1p |
| | | | | 400 | application/json | not tested | |
| 100% | /orders/{orderId} | GET | NA | 200 | application/json | covered | 1p |
| | | | | 404 | application/json | covered | 1p |
|-------------------------------------------------------------------------------------------------------------------|
| * = Operation not eligible for coverage |
| ! = Operation excluded from the run due to a filter |
| p = passed tests |
| f = failed tests |
| w = wip tests |
| |
| 75% API Coverage reported from 4 operations eligible for coverage |
| 75% Absolute Coverage (includes excluded operations that were not tested) |
|-------------------------------------------------------------------------------------------------------------------|
Failed the following API Coverage Report success criteria:
Total API coverage: 75% is less than the specified minimum threshold of 100%.
One or more API Coverage report's success criteria were not met
Tests run: 3, Successes: 3, Failures: 0, WIP: 0, Errors: 0
exit code 1Three tests passed and the run exited 1, which is the behavior a gate needs: the 400 response was
never exercised, so its row reads not tested and coverage is 75%. The two codes under the skipped
scenario come from Specmatic’s rule catalog.2 T00003 is
the strict-mode skip. T00004 says the negative tests Specmatic could generate for a 4xx are off,
since they need its schema resiliency mode. An example is therefore the one way a 4xx row gets
covered here.
Read the report when an example goes stale
The example for the 200 names ord_1001, an order the service seeds at startup. drift.mjs
starts the service without it and runs the same strict suite.
node drift.mjsScenario: POST /orders -> 201 with the request from the example 'place_order_201' has SUCCEEDED
Scenario: POST /orders -> 400 with the request from the example 'place_order_400' has SUCCEEDED
Scenario: GET /orders/(orderId:string) -> 200 with the request from the example 'get_order_200' has FAILED
>> RESPONSE.STATUS
R0002: HTTP status mismatch
Summary: The HTTP status code does not match the expected status code defined in the specification
Specification expected status 200 but response contained status 404
Scenario: GET /orders/(orderId:string) -> 404 with the request from the example 'get_order_404' has SUCCEEDEDThe run goes on to the table, where the 200 row reads not implemented with a result of 1f,
and ends with Tests run: 4, Successes: 3, Failures: 1, WIP: 0, Errors: 0 and exit code 1. The
scenario line names the operation and the example file, and R0002 names the mismatch, so the
report answers which operation and which example without a log. Specmatic also writes an HTML
report under build/reports/specmatic on every run.3
The fix is never to loosen the example. Either the service seeds the record the example names, or
a setup step creates it before the run, which is what
Dredd hooks do for the same kind of suite. Review
the example files in the same pull request as the document, where a renamed field or a retired
fixture id is a visible diff before it is an R0002.
Check it worked
contract.test.mjs runs the strict suite and asserts the exit code, the totals line, and the
coverage line, then runs examples validate and asserts that all four files are valid. A third
asserts that the harness exits 1 when java dies of SIGKILL.
test('every documented response has an example, and every example passes against the service', async () => {
const { code, lines } = await contractTest()
assert.equal(code, 0, lines.join('\n'))
assert.ok(lines.includes('Tests run: 4, Successes: 4, Failures: 0, WIP: 0, Errors: 0'), lines.join('\n'))
assert.ok(lines.some((l) => l.includes('100% API Coverage reported from 4 operations')), lines.join('\n'))
assert.ok(!lines.some((l) => /T00003/.test(l)), 'no operation was skipped for want of an example')
})node --test contract.test.mjs1..3
# tests 3
# suites 0
# pass 3
# fail 0
# cancelled 0
# skipped 0
# todo 0
# duration_ms 8139.442258All three tests pass. The second checks the example files against the document with no service
running, so an example that breaks the document fails without a server.
Tests run: 4, Successes: 4 is the literal to look for.
When it goes wrong
The build stays green after a failed contract. pitfall.mjs runs the same failing suite as
drift.mjs through npx specmatic, the bin the npm package installs, instead of the jar.
node pitfall.mjsScenario: GET /orders/(orderId:string) -> 200 with the request from the example 'get_order_200' has FAILED
Tests run: 4, Successes: 3, Failures: 1, WIP: 0, Errors: 0
npx specmatic exit code 0The jar exited 1 on the same run. The wrapper receives that code and drops it on the way out,4
so a CI step built on npx specmatic test passes whatever the contract says. Run
java -jar node_modules/specmatic/specmatic.jar test ..., as contract-test.mjs does, or parse
the Tests run: line yourself.
A 4xx row reads not tested without --strict too. Without the flag Specmatic generates a
positive test for an operation with no example and skips the negative ones with T00002,
Examples Required. Add an example per error response: the request may break the schema, and the response must
match it.
The run passes with a wrong value in the response. Specmatic validates the response against the
document’s schema, not against the body written in the example. With the example’s quantity
changed to 9 while the service kept answering 2, the scenario still passed. Contract tests prove
shape, so a value that matters needs a test that asserts it.
When not to do this
Do not use Specmatic to police an API you do not own. Its examples describe the provider’s behavior from the provider’s side, and the service has to be yours to start in CI. For a vendor API, the scheduled runs in contract test an API you do not own fit better.
Do not read a green strict run as a statement about values. The suite proves that each documented response exists, carries the documented status, and matches the schema. A field with the right type and the wrong number passes.
Do not let the examples become a second copy of your fixtures. Each one names records the service
must hold, so a suite that needs ord_1001 to exist has a setup problem the examples cannot
solve. Seed the service for the run.
Do not pick Specmatic for the cases nobody thought of. Its generated requests are random values
that fit the schema, and its negative tests need the resiliency mode. Schemathesis is built for
that search and prints a curl command that reproduces each failure.
Related how-tos
Last verified
Verified 2026-10-21 against Node 22.22.0, OpenJDK 21.0.12.1 and Specmatic 2.55.3. Every output
block is what the command preceding it printed, after contract-test.mjs dropped the lines that
carry a timestamp, an absolute path, or a report-writing notice. The observation about response
values in the section on what goes wrong comes from a run whose output is not shown, with the
example’s quantity edited. Schemathesis and Pact were read from their documentation, not run.
Footnotes
-
The npm registry lists the package’s unpacked size as 85,710,486 bytes across 64 files, and the README calls the module a thin wrapper over the Specmatic executable jar. The jar ships inside the same tarball, which is where the 85 megabytes go, so
npm cifetches a Java program and leaves finding a Java runtime to you. ↩︎ Back to text -
Specmatic numbers its reasons. The rule catalog files
T00002asExamples Required,T00003asExamples Required in Strict Mode, andT00004asGenerative Disabled, with an entry for each under the anchor the console prints beside it.R0002,HTTP status mismatch, sits in a second series for what the response did. A failure therefore arrives with its citation attached. ↩︎ Back to text -
Near the end of every run the console prints
Generating CTRF report using coverage report specifications..., and the reports reference files the Common Test Report Format under the commercial edition. The run here, on the open source license, wrote the HTML report atbuild/reports/specmatic/test/html/index.html, the only file it wrote even withctrfadded toformats. The CTRF file the console announces belongs to the paid edition, so the announcement is the part of it that ships. ↩︎ Back to text -
The command handler in the wrapper’s
bin, in thespecmatic-noderepository, ends withprocess.exitCode = err.code. The error it receives is built asnew Error('Command exited with non zero code: ' + code), which carries the number in its message and has nocodeproperty. The assignment therefore storesundefined, and Node exits 0. The number leaves the JVM and stops in the wrapper. ↩︎ Back to text