How to run DGX Spark using the harmony format for gpt-oss

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

We would like to run tensor-rt using the harmony format, as specified here:

docker pull nvcr.io/nvidia/tensorrt-llm/release:gpt-oss-dev
docker run --gpus all -it --rm -v $(pwd):/workspace nvcr.io/nvidia/tensorrt-llm/release:gpt-oss-dev

Based on your instructions:

docker run --rm -it --gpus all \
  nvcr.io/nvidia/tensorrt-llm/release:spark-single-gpu-dev \
  python -c "import tensorrt_llm; print(f'TensorRT-LLM version: {tensorrt_llm.__version__}')"

We are supposed to use the “release:spark-single-gpu-dev“ as opposed to “release:gpt-oss-dev” ?

2

Yes you should use the release:spark-single-gpu-dev tag. Please check out our TensorRT-LLM playbook for more information and steps

4

Could you please read the question carefully, this is for the “harmony format” by OpenAI. The question is how do we set that up

5

Please read their reply carefully, it tells you how in the playbook.

dgx-spark-playbooks/nvidia/trt-llm/

7

Hi

I think the tensorrt-llm image should support harmony format if you follow our instructions. Do you get an error or some bad output when you try submitting a query with harmony format?

8

We do get an error unfortunately. A higher priority issue for us is to get the two sparks to serve models efficiently. I’ve opened a github issue here:

Could you please take a look. We get stuck at this point:

```bash
[10/22/2025-14:53:33] [TRT-LLM] [RANK 0] [I] [Autotuner] Cache size after warmup is 28
[10/22/2025-14:53:33] [TRT-LLM] [RANK 0] [I] [Autotuner] Autotuning process ends
[10/22/2025-14:53:33] [TRT-LLM] [RANK 0] [I] Creating CUDA graph instances for 3 batch sizes.
[10/22/2025-14:53:33] [TRT-LLM] [RANK 0] [I] Run generation only CUDA graph warmup for batch size=4, draft_len=0
[10/22/2025-14:53:34] [TRT-LLM] [RANK 0] [I] Run generation only CUDA graph warmup for batch size=2, draft_len=0
[10/22/2025-14:53:34] [TRT-LLM] [RANK 0] [I] Run generation only CUDA graph warmup for batch size=1, draft_len=0
[10/22/2025-14:53:35] [TRT-LLM] [RANK 0] [I] Memory used after loading model weights (inside torch) in memory usage profiling: 33.22 GiB
[10/22/2025-14:53:35] [TRT-LLM] [RANK 0] [I] Memory used after loading model weights (outside torch) in memory usage profiling: 74.81 GiB
```

and then it hangs for 90+ minutes. The details are laid out in the github issue. Thanks!

9

I see you started another thread on that issue. We will respond and work with you on that thread. I will investigate this harmony issue and get back to you

10

If you try the last step in our playbook what response do you get?

curl -s http://localhost:8355/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "'"$MODEL_HANDLE"'",
    "messages": [{"role": "user", "content": "Paris is great because"}],
    "max_tokens": 64
  }'

This is OpenAI style query which TRT-LLM will translate into harmony format

11

The response format from that request is the OpenAI response format, which not the same as the Harmony format, which is described in detail here: