How to serve schema-constrained output with vLLM#

Your agents call one vLLM endpoint for JSON, and one morning the replies stop parsing. The server was upgraded, but the client still sends guided_json, which this release ignores. The server warned once, at the first such request, so today’s log is clean. In the same week a schema that decoded in staging fails with a 400 in production, where the backend is pinned and staging’s was not.

What you get

You will end up with a schema preflight, a client that sends the schema in the field this release reads, and a harness that validates every reply before counting it. This is for you if you run vLLM for agents that need valid JSON.

Short answer

Pin --structured-outputs-config.backend to xgrammar or guidance on vllm serve, so a schema the engine cannot take fails the request with a 400 instead of moving to another backend under auto. Send the schema as structured_outputs or response_format; the removed guided_json family is ignored with a warning. Preflight every schema with llguidance offline, and validate every reply against the schema before you read the server’s throughput.

You will need

A GPU host already running vLLM, and Python 3.11 or later for the harness. Verified 2026-10-21 against Python 3.13.16, openai 3.35.1, jsonschema 4.26.0, and llguidance 1.9.1. vLLM 0.32.0 and SGLang 0.5.22 were read in their documentation and source, not run, because there is no GPU here. The server in every output block is a stand-in on 127.0.0.1. It answers vLLM’s request shapes the way the source does and compiles schemas with the same llguidance engine, so the counts are real and the speeds are not.

Approaches compared

XGrammar compiles a grammar ahead of time and caches it, so a repeated schema is cheap and a stream of new ones pays the compile each time. llguidance skips the compile and charges per token instead, about fifty microseconds a mask by its own measure, which is why it suits schemas that differ per request. SGLang offers the same two engines behind one flag, at the price of a second server with its own request fields.

Compile every schema before it reaches a GPU

llguidance is the engine behind vLLM’s guidance backend, and its validator needs no tokenizer and no model. Run every schema through it in CI, before a request can reach the server.

def check(schema):
    """'' when the engine compiles the schema, otherwise its first error message."""
    grammar = llguidance.LLMatcher.grammar_from_json_schema(schema)
    is_error, messages = llguidance.LLMatcher.validate_grammar_with_warnings(grammar)
    return messages[0] if is_error else ""

The four schemas here are an order, an invoice with multipleOf: 0.01 on its total, a contact with a pattern and a maxLength on one field, and a tag list with uniqueItems.

PYTHONPATH=vendor python3 preflight.py schemas

schema    guidance engine (llguidance 1.9.1)
contact   ok
invoice   ok
order     ok
tags      rejected: Unimplemented keys: ["uniqueItems"]
1 of 4 schemas would be refused with the backend pinned to guidance

The engine takes three of the four and names the keyword it refuses. XGrammar draws the line elsewhere. vLLM’s own pre-check for that backend, in backend_xgrammar.py, refuses multipleOf on a number, uniqueItems and contains on an array, and a pattern or format combined with a length bound. It also refuses several propertyNames and patternProperties combinations, and every refusal carries one message, The provided JSON schema contains features not supported by xgrammar. in the 400 body. So the invoice and the contact pass llguidance and fail XGrammar, and the tag list fails both.¹ Under auto, vLLM tries XGrammar first and, when that check fails, hands the request to guidance, or to outlines when guidance’s own check refuses it, as sampling_params.py spells out. With a backend named, the same code comments xgrammar with no fallback and the request fails.

Pin the backend and send the schema where this release reads it

Name the backend on the server. The vllm serve option takes either dotted keys or one JSON string, and its fields are documented on the StructuredOutputsConfig page. This command runs on the GPU host and was not run here:

vllm serve Qwen/Qwen3-8B --structured-outputs-config '{"backend": "xgrammar", "disable_any_whitespace": true}'

On the client, the schema travels in one of two fields. structured_outputs is vLLM’s own, sent through the openai client’s extra_body because the client has no parameter for it; response_format with json_schema is the OpenAI shape, which vLLM maps onto the same constraint.

def request_kwargs(schema, shape):
    """The keyword arguments that put the schema on the wire in the given shape."""
    if shape == "structured_outputs":
        return {"extra_body": {"structured_outputs": {"json": schema}}}
    if shape == "response_format":
        name = schema.get("title", "reply")
        return {"response_format": {"type": "json_schema", "json_schema": {"name": name, "schema": schema}}}
    if shape == "guided_json":
        return {"extra_body": {"guided_json": schema}}
    raise ValueError(f"unknown shape {shape!r}")

The third shape is the trap. The guided_* fields were removed, and the request model keeps a list of their names so it can log, once, that they were ignored. The demo sends the order schema in all three shapes to the stand-in and validates each reply.

PYTHONPATH=vendor python3 shapes.py

shape               field on the wire             server        reply valid
structured_outputs  structured_outputs            constrained   True
response_format     response_format               constrained   True
guided_json         guided_json                   ignored       False
  server log: Request contains the removed guided-decoding field(s) ['guided_json'], which are ignored; output will NOT be constrained.

The first column is what the openai client put on the wire. The last is what a request with the old field gets back: a 200 with prose in it, and a warning in a log the client never sees.

Count a reply only after it validates

The harness sends each schema several times, validates every reply with jsonschema before counting it, and counts a 400 as a rejected schema rather than a skipped request.

    for _ in range(requests):
        try:
            content, usage = complete(client, model, schema, prompt, shape)
        except BadRequestError as err:
            counts["rejected"] += 1
            counts["reason"] = str(err.body.get("message", err.message) if isinstance(err.body, dict) else err.message)
            continue
        counts["tokens"] += usage.completion_tokens if usage else 0
        try:
            value = json.loads(content)
            validator.validate(value)
            counts["valid"] += 1
        except (json.JSONDecodeError, jsonschema.ValidationError):
            counts["invalid"] += 1

Against the stand-in, which behaves as a server pinned to guidance, the table proves the accounting. Against the GPU host, pass --base-url and --model, and the last line becomes completion tokens per second.

PYTHONPATH=vendor python3 measure.py --standin

shape: structured_outputs, requests per schema: 5
schema     valid  invalid  rejected  tokens  time
contact        5        0         0      90  duration_ms: 56.2
invoice        5        0         0      80  duration_ms: 15.9
order          5        0         0     140  duration_ms: 13.7
tags           0        0         5       0  duration_ms: 14.4
          400: Grammar error: Unimplemented keys: ["uniqueItems"]
tokens per second: not measured, the stand-in has no model behind it

The tag schema shows what a pinned backend does with an unsupported schema: every one of its five requests is refused, with the engine’s reason under the row. That is the loud failure you want in a test run. Measure throughput with the schemas your agents send, not with a toy. The per-token cost of the guidance engine and the compile cost of XGrammar both scale with the grammar, and the XGrammar cache only helps when the same grammar comes back.²

Check it worked

Thirteen checks run the harness’s own logic against the stand-in. Each raises on failure rather than using assert, which python -O strips.

PYTHONPATH=vendor python3 test_harness.py

ok  preflight accepts order, invoice, and contact
ok  preflight rejects tags and names the keyword
ok  structured_outputs travels in extra_body
ok  response_format carries the schema under json_schema
ok  guided_json is a removed field, not a parameter
ok  the server received the structured_outputs field
ok  the server received the response_format field
ok  the server received the guided_json field
ok  a constrained reply that matches its schema counts as valid
ok  an unconstrained reply counts as invalid, never valid
ok  the server warned that the removed field was ignored
ok  a refused schema counts as rejected with the engine's reason
ok  a refused schema is never counted valid
13 checks passed

The two that matter most are an unconstrained reply counts as invalid, never valid and a refused schema is never counted valid, because a harness that gets either wrong reports a healthy endpoint that is not constraining anything.

When it goes wrong

A schema that works in staging returns 400 in production. Staging ran auto, so the request moved to another backend when the first refused it, and production pins one. Pin the same backend in both, and run the preflight for that backend in CI so the refusal happens before a deploy.

Replies stop being JSON after a vLLM upgrade. The client still sends a guided_* field, the server ignores it, and the warning appears once in the server log. Switch to structured_outputs or response_format, and keep the harness in CI so the invalid column catches the next one.

Throughput falls when a new agent joins. Its schemas differ on every request, so XGrammar compiles a grammar per request and its cache, which evicts by least recent use, stops hitting. Move that workload to guidance, which compiles nothing up front, or give the agent a fixed set of schemas.

When not to do this

Do not run auto where you need to know which engine decoded a reply. The selection happens per request, is logged nowhere the client can see, and changes with each release, as the config’s own docstring says. A pinned backend fails a schema out loud, which is the behavior a test can hold on to.

Do not take a 200 as a constrained reply. The removed guided_* fields produce exactly that, a 200 with a warning in a log you are not reading. Validate on the client, every reply, even on the day the server is known to be right.

Do not treat the grammar as validation of meaning. A grammar guarantees shape, so an invoice total of 0.00 and a contact at nobody@example.com are both valid output. Validators for the values that matter still run after the parse.

Do not benchmark a backend with one small schema. The engine’s cost is a function of the grammar and of how often it changes, and a toy schema measures neither. Run the harness with the schemas your agents send and with the traffic shape they produce.³

Related how-tos

• Validate JSON model output before you use it

• Compare agent task success rates across model providers

Last verified

Verified 2026-10-21 against Python 3.13.16, openai 3.35.1, jsonschema 4.26.0, and llguidance 1.9.1. Every output block is what the command preceding it printed. vLLM 0.32.0 and SGLang 0.5.22 were not run. Their backends, flags, request fields, fallback order, and warning text were read from their documentation and source at those versions. The stand-in reproduces that behavior for the three request shapes with the llguidance engine. XGrammar’s refusals are quoted from vLLM’s pre-check and were not exercised, because the package needs a GPU runtime this machine lacks.