The latest community version appears to be buggy

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

In the latest version of the community build, this error occurs with a recipe that previously worked fine.
" ERROR 06-15 16:12:26 [scheduler.py:1547] Unexpected: grammar rejected tokens [57590, 3158, 198, 248069] for request chatcmpl-a942ac30a1c89397-aff7ab7b. Terminating request." . This error occurs when I use ool-eval-bench --base-url http://0.0.0.0:8888 --seed 42 --perf --spec-bench. nvidia/Qwen3.6-35B-A3B-NVFP4 (alias: JL-Spark)

✓ Warm-up complete (546 ms)
🔍 Engine: vLLM 0.22.1rc1.dev511+gc621af169.d20260614

2

Yes it’s buggy, all my models are broken, Qwen3.6 reply half time, it stops after reasoning, an other model dosen’t work anymore each time I write a message Ive a tokenizer error.

3

There was a pull request merged upstream that enabled structured outputs for many common models during most tool calling scenarios. However, we’ve since partially reverted that in [Frontend] Skip structural tags for auto tool_choice without strict mode by sfeng33 · Pull Request #45600 · vllm-project/vllm · GitHub so that it only enables when a client sends in strict=True for their function definitions when tool_choice=auto (the default) is used.

I and some of the maintainers discovered similar things you describe here, along with some places we need to improve the constraints being used to guide the model outputs, over the weekend. It just took until today to get that reverted where it’s not enabled for default paths.

So, let me know if that doesn’t get this working again for you once you get a build with that change in it.

4

Thanks for your and the vLLM team’s attention to this! I had the same failure with DFlash draft models and after a rebuild tonight I can confirm it is resolved - at least for tool calling.

However, we still want to be able to strictly enforce structured output for block drafting models which I think needs some more work.

5

It’s now fixed since 4 hours, just run ./build-and-copy.sh --rebuild-vlm --cleanup to be certain

6

Oops I didnt see the recent messages 😀

8

It’s better. but still get sometimes : Failed to advance FSM for request chatcmpl-82595a2eccf3c7ec-85b19bf7 for tokens 198. Please file an issue.
(EngineCore pid=169) ERROR 06-16 15:46:58 [scheduler.py:1579] Unexpected: grammar rejected tokens [198, 248069, 271, 22] for request chatcmpl-82595a2eccf3c7ec-85b19bf7. Terminating request.
(APIServer pid=94) ERROR 06-16 15:46:58 [serving.py:195] Request chatcmpl-82595a2eccf3c7ec failed with an internal error during generation
(APIServer pid=94) INFO: 127.0.0.1:59280 - “POST /v1/chat/completions HTTP/1.1” 500 Internal Server Error.
An some errors about bad or not valid json output

Engine: vLLM 0.23.1rc1.dev53+gc69c73418.d20260616

9

@lewald_jens I have been having a similar issue, using “pi” with the “@official/qwen3.6-35b-a3b-fp8-mtp-vllm” recipe. “pi” would show errors like:

 Thinking...

 Error: Internal server error

 Thinking...

 Error: Internal server error

 Thinking...

The errors (in sparkrun logs) looked like this:

EngineCore pid=413) ERROR 06-16 20:24:27 [backend_xgrammar.py:162] Failed to advance FSM for request chatcmpl-8e9a44e133936941-8cfe297f for tokens 248069. Please file an issue.
(EngineCore pid=413) ERROR 06-16 20:24:27 [scheduler.py:1547] Unexpected: grammar rejected tokens [13, 198, 248069] for request chatcmpl-8e9a44e133936941-8cfe297f. Terminating request.

With a little help from chatgpt, I fixed the issue by removing two lines from the official recipe (in my local cache):

$ diff qwen3.6-35b-a3b-fp8-mtp-vllm.yaml-ORIG ./.cache/sparkrun/registries/_url_70b481ce5ca0/official-recipes/qwen3.6/vllm/qwen3.6-35b-a3b-fp8-mtp-vllm.yaml
26d25
<   speculative_config: '{"method": "mtp", "num_speculative_tokens": 2}'
46d44
<     --speculative-config '{speculative_config}' \

After stopping and re-running the recipe (with sparkrun), the errors are gone.

John

10

The MTP might be the problem. If so, it’s in the implementation for that in the vLLM. After all, it was working perfectly until recently.

11

The grammar rejection relates to specific prompt formatting for that model specifically?

12

Yeah, I think you’re right. It started for me after I switched to the MTP model, because huggingface showed “MTP enables ~1.5-2x faster inference with no accuracy loss.” (whatever that means!)
*
source:* unsloth/Qwen3.6-35B-A3B-MTP-GGUF · Hugging Face

13

FWIW, I had a few surprises and issues when moving from vllm 0.22 → 0.23. My nemotron-3-super-120b-a12b-nvfp4 model went from starting and running ok on a single node to needing 2 nodes to start. but its kicking butt at 25 tok/s so maybe I won’t complain.

14

Try setting VLLM_ENFORCE_STRICT_TOOL_CALLING=false as an environment variable before starting your vLLM to see if that gets rid of these grammar errors. There was a change recently to wire in the xgrammar library by default any time tool_choice=required is set in requests or strict=True is set in a tool definition. Depending on what client/workload you’re running, that could be triggering this.

I’ve advocated for turning this back off by default, so if this helps this error I can go back upstream and advocate for that once again.

15

Its better but not same as before. i got a lot of json output errors. ● TC-64 Simple Schema Compliance ❌ FAIL 0/2 4.9s ttft=4,353ms Output is not valid JSON.
● TC-65 Tool → Structured Output ✅ PASS 2/2 3.5s ttft=1,456ms t2 Called get_weather, then produced
schema-compliant JSON with correct data.
● TC-66 Nested Schema (Array of Objects) ⚠️ PARTIAL 1/2 9.6s ttft=4,731ms t2 Called get_contacts
correctly but output is not valid JSON.
● TC-67 Enum Constraint + Analysis ⚠️ PARTIAL 1/2 4.7s ttft=1,247ms t2 Called get_stock_price but
output is not valid JSON.
● TC-68 Schema Violation Resistance ✅ PASS 2/2 3.1s ttft=2,800ms Produced schema-compliant JSON
without the forbidden extra fields, despite the user requesting them.
● TC-69 Multi-Tool → Complex Schema ⚠️ PARTIAL 1/2 7.6s ttft=3,235ms t2 Called both tools but output
is not valid JSON.
This errors are new for me and the test.

16

Yes, so what’s happening is with VLLM_ENFORCE_STRICT_TOOL_CALLING=false it disables all guiding of the model output so tool_choice=required or structured outputs essentially become a no-op.

There’s a complex set of scenarios at play here with multiple groups trying to fix this upstream but the TL;DR is we used to force all models to just output JSON directly but models like Qwen 3.6 aren’t natively trained in that format and instead have their own XML tool call format. There are attempts to fix that, to natively guide the model into its own format (instead of JSON), but the first pass at the proper grammars to enforce the model generation in its native format trigger bugs in the speculative decoding and reasoning → tool call transition boundary for advanced models.

There are people pushing on this upstream already, but if you have specific issues you want to open in vLLM’s github repo we can more closely track and fix this specific interaction of Qwen 3.6 schema enforcement with spec decoding. It’s not just a problem with Qwen models but a problem we have with multiple models as they transition from reasoning to not reasoning and how that interacts with where we start enforcing the output schema when spec decoding is in use.