Nvcr.io/nvidia/vllm:26.07-py3 — tool-calling requests 500 due to xgrammar/transformers version

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

Environment:

  • Container: nvcr.io/nvidia/vllm:26.07-py3
  • vLLM: 0.24.0+092c4842.nv26.7.59534043
  • xgrammar: 0.2.0 (as shipped)
  • Hardware: DGX Spark (GB10)

Symptom: Any request with tools/tool_choice set returns a 500, regardless of model:

{"error": {"message": "cannot import name 'normalize_tool_choice' from 'xgrammar'", "type": "InternalServerError", "code": 500}}

Requests without tools work fine — this only breaks tool-calling specifically.

Root cause: This container’s vLLM build calls xgrammar.normalize_tool_choice, which doesn’t exist in xgrammar==0.2.0 (the version pinned in the image) — it was added in xgrammar 0.2.4. So the container ships a vLLM build that’s outrun its own bundled dependency.

Workaround we verified works: pip install -U xgrammar inside the container bumps it to 0.2.4 and fixes the import — but pip’s resolver then silently downgrades transformers from the 5.6.1 this vLLM build actually requires down to 4.57.6 as a side effect, which would break other things. Re-pinning transformers==5.6.1 afterward keeps the import fixed. Built this into a two-line derived Dockerfile:

FROM nvcr.io/nvidia/vllm:26.07-py3
RUN pip install -q -U xgrammar && pip install -q transformers==5.6.1

With that, tool-calling works cleanly end-to-end — verified with real requests producing correct structured tool_calls.

One caveat I can’t fully vouch for: xgrammar==0.2.4 itself declares transformers<5,>=4.38.0 as a dependency constraint, so forcing transformers==5.6.1 back in is technically outside what xgrammar claims to support. Basic tool-calling checked out fine in our testing, but I haven’t exhaustively verified every xgrammar code path against transformers 5.x.

Ask: Could xgrammar be bumped to >=0.2.4 in the next 26.xx container build (with transformers re-verified against it), so this doesn’t need a manual patch?

2

Thank you for the detailed report and for including a clear reproducer, root-cause analysis, and workaround.

We were able to match this to an internal issue tracking tool-calling failures in the 26.07 vLLM container. The issue is in the tool-calling dependency path and can lead to the 500 error you observed when tools / tool_choice are used.

Thank you as well for sharing the derived-image workaround. That is useful for investigation, but because it changes dependency constraints, we cannot yet treat it as an officially validated fix across all related code paths.

We have an internal ticket tracking this issue and will update this thread once we have further information.

In the meantime, if others hit the same problem, please include the exact container tag, pip list output for the affected environment, and whether you tested only tool calling or any other structured-output / guided-decoding paths after applying dependency overrides.

3

Thanks for looking into it. Here’s the full detail:

Exact container tag: nvcr.io/nvidia/vllm:26.07-py3 (base, affected). Our patched derivative is tagged nvcr.io/nvidia/vllm:26.07-py3-toolfix, built via:

FROM nvcr.io/nvidia/vllm:26.07-py3
RUN pip install -q -U xgrammar && pip install -q transformers==5.6.1

Relevant pip list (patched environment):

vllm                  0.24.0+092c4842.nv26.7.59534043
xgrammar              0.2.4
transformers          5.6.1
torch                 2.13.0a0+9186a08b2c.nv26.7.59513937
apache-tvm-ffi        0.1.12   (vllm wants ==0.1.9, unresolved — separate minor conflict, hasn't caused issues in our testing)
flashinfer-python     0.6.14+d0510b70.nv26.7.cu.59527636

Full pip list (202 packages) is attached.

nvcr-vllm-26.07-py3-toolfix-pip-list.txt (9.7 KB)

Structured-output paths tested beyond tool-calling: good catch, we’d only checked tool-calling initially. Went back and tested two more paths against the patched image before replying:

  • response_format: {"type": "json_schema", ...} — clean, schema-conforming JSON output, finish_reason: "stop", no errors.
  • guided_regex — output correctly constrained to the regex pattern, no errors.

Both worked cleanly post-patch. Haven’t tested guided_grammar (CFG) or guided_choice specifically — can if useful.

Hardware/model for all of the above: DGX Spark (GB10), tested against both nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8 and Qwen/Qwen3.6-35B-A3B-FP8. One side note unrelated to xgrammar: Qwen3.6-35B-A3B-FP8 fails to load entirely on this same patched image with a DeepGEMM assertion (RuntimeError: Assertion error .../layout.hpp:59: Unknown SF transformation) during FP8 MoE weight-layout conversion — separate issue, happy to open that as its own thread if it’s not already known.

4

Thanks guys I had the same problem. I decided to roll back to 26.06 instead :)

5

Thanks for the detailed report. We’ve reproduced this internally on the public nvcr.io/nvidia/vllm:26.07-py3 image and linked it to our internal tracking.

Our latest investigation shows this is not limited to the initial normalize_tool_choice import failure. Tool calling in 26.07 is affected by a deeper compatibility issue in the tool-calling path, so simply upgrading xgrammar inside the container is not a reliable fix for tool_choice="required" or forced named tool choice.

At the moment, if you need working tool calling, the safest option is to use 26.06 rather than 26.07.

We’re continuing to investigate and will share another update here when we have a validated fix path. Thanks again for reporting this and for the clear repro details.

6

quick fix by patch image:

# ./vllm/Dockerfile.26.07
FROM nvcr.io/nvidia/vllm:26.07-py3

RUN python -m pip install --no-cache-dir --no-deps \
      "xgrammar==0.2.4"

# compose.yml
vllm:
  build:
    context: .
    dockerfile: ./vllm/Dockerfile.26.07
  image: dgx-spark-ai-vllm:26.07-xgrammar-fix
7

@Neill,

Confirming on a Gigabyte AI TOP ATOM (GB10), same container and vLLM build. Also a regression: identical flags worked on 26.05.post1-py3 (vLLM 0.21.0 / xgrammar 0.1.34).

Slightly tighter variant of the workaround — --no-deps avoids the transformers downgrade entirely rather than correcting it afterward:

FROM nvcr.io/nvidia/vllm:26.07-py3
RUN pip install --no-cache-dir --no-deps xgrammar==0.2.4

Can provide full pip list if that’s helpful, otherwise the relevant packages in the resulting image are:

apache-tvm-ffi    0.1.7
torch             2.13.0a0+9186a08b2c.nv26.7.59513937
transformers      5.6.1
vllm              0.24.0+092c4842.nv26.7.59534043
xgrammar          0.2.4

Currently have tested tool calling only (agent mode via AnythingLLM, --tool-call-parser qwen3_coder). Have not exercised guided-decoding or structured-output paths.

Unrelated observation while investigating: vLLM’s metadata in this image declares apache-tvm-ffi==0.1.9, but 0.1.7 is what’s installed in the stock container. That’s present before any patching and doesn’t appear to affect tool calling — noting it only in case it’s relevant to the dependency audit.