Hi everyone!
Has anyone managed to get the trt-llm playbook to run on two sparks (or GB10) with nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc5?
I’ve got GB10’s and I can’t get it to work where it blows up on ptxas fatal : Value ‘sm_121a’ is not defined for option ‘gpu-name’ I’ve also tried 1.3.0rc10 but then I get different errors.
Full disclosure I’m using nvidia/Qwen3-8B-NVFP4 to make loading go quicker.
I need a little more detail to help triage. What step did you run. Can you share more logs that may point to the issue?
I had to deviate from the example a bit:
docker run -d --rm \
--name trtllm-multinode \
--gpus '"device=all"' \
--network host \
--ulimit memlock=-1 \
--ulimit stack=67108864 \
--device /dev/infiniband:/dev/infiniband \
-e UCX_NET_DEVICES="enp1s0f0np0" \
-e NCCL_SOCKET_IFNAME="enp1s0f0np0" \
-e OMPI_MCA_btl_tcp_if_include="enp1s0f0np0" \
-e OMPI_MCA_orte_default_hostfile="/etc/openmpi-hostfile" \
-e OMPI_MCA_rmaps_ppr_n_pernode="1" \
-e OMPI_ALLOW_RUN_AS_ROOT="1" \
-e OMPI_ALLOW_RUN_AS_ROOT_CONFIRM="1" \
-e CPATH=/usr/local/cuda/include \
-e TRITON_PTXAS_PATH=/usr/local/cuda/bin/ptxas \
-v ~/.cache/huggingface/:/root/.cache/huggingface/ \
-v ~/.ssh:/tmp/.ssh:ro \
-v ~/offlinefiles:/tmp/offlinefiles \
-e HF_ENDPOINT="*SNIP*" \
myregistryhost/nvidia/tensorrt-llm/release:1.3.0rc5 \
sh -c "cat /tmp/offlinefiles/trtllm-mn-entrypoint.sh | sh"
docker cp ~/openmpi-hostfile trtllm-multinode:/etc/openmpi-hostfile
export TRTLLM_MN_CONTAINER=trtllm-multinode
docker exec -it \
-e MODEL="nvidia/Qwen3-8B-NVFP4" \
-e HF_TOKEN=$HF_TOKEN \
-it $TRTLLM_MN_CONTAINER bash -c '
mpirun -x HF_ENDPOINT=$HF_ENDPOINT trtllm-llmapi-launch trtllm-serve $MODEL \
--tp_size 2 \
--backend pytorch \
--max_num_tokens 32768 \
--max_batch_size 4 \
--extra_llm_api_options /tmp/offlinefiles/qwen3.yml \
--port 8355'
Naturally, I’ve got the container started on both hosts before running trtllm-serve on the primary.
I’m using the 8B Qwen3 instead of the nvidia/Qwen3-235B-A22B-FP4 (there is no FP4 BTW) which should be somewhat similar, but quick to load and have less memory pressure.
I’ll spare everyone the entire lengthy error output, but the likely important lines are:
triton.runtime.errors.PTXASError: PTXAS error: Internal Triton PTX codegen error
ptxas stderr:
ptxas fatal : Value ‘sm_121a’ is not defined for option ‘gpu-name’
Repro command: /usr/local/lib/python3.12/dist-packages/triton/backends/nvidia/bin/ptxas -lineinfo -v --gpu-name=sm_121a /tmp/tmpnw_bdlb2.ptx -o /tmp/tmpnw_bdlb2.ptx.o
PTXAS appears to detect the GPU correctly, but doesn’t support it. I haven’t figured out how to force it to use something other than sm_121a.
I’ve also tried using:
My GB10 has it’s system updates up-to-date as of April 7.
The container’s bundled ptxas is different from the required version for GB10 systems, which is why we inject the local path -e TRITON_PTXAS_PATH=/usr/local/cuda/bin/ptxas \
However, smaller models may require runtime kernel compilation which will fail since you don’t forward the path to the mpirun command.
To fix this, add -x TRITON_PTXAS_PATH to your mpirun command.
That solved the issue with Qwen3-8B-NVFP4. Thank you.
As an FYI when trying to run nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-FP8 it complains about not having enough RAM (although there is plenty between two nodes)
[04/10/2026-17:34:03] [TRT-LLM] [RANK 0] [I] max_seq_len is not specified, using inferred value 262144
[04/10/2026-17:34:03] [TRT-LLM] [RANK 0] [I] Using Sampler: TorchSampler
[04/10/2026-17:34:03] [TRT-LLM] [RANK 0] [I] [create_py_executor] Created execution_stream: <torch.cuda.Stream device=cuda:0 cuda_stream=0xe9d798833990>
[04/10/2026-17:34:04] [TRT-LLM] [RANK 0] [E] Failed to initialize executor on rank 0: Executor creation failed due to insufficient GPU memory.
The following component could not be created: Initial KV Cache (temporary for KV cache size estimation)
Total GPU memory (GiB): 121.63
Free GPU memory before component creation attempt (GiB): 3.91
Previously created components and free GPU memory before/after creation (GiB):
model: 112.50 / 3.89
sampler: 3.89 / 3.91
I watched the startup closely, and ran this several times since it seems to hang onto the cache and then push into swap:
sudo sh -c ‘sync; echo 3 > /proc/sys/vm/drop_caches’
Ran with:
export MODEL_HANDLE=“nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-FP8”
docker exec -it -e MODEL_HANDLE=“$MODEL_HANDLE” $TRTLLM_MN_CONTAINER bash -c ’
mpirun -x TRITON_PTXAS_PATH -x CPATH -x HF_ENDPOINT=$HF_ENDPOINT --allow-run-as-root trtllm-llmapi-launch
trtllm-serve serve “$MODEL_HANDLE”
–tp_size 2
–host 127.0.0.1
–port 8355
–backend pytorch
–extra_llm_api_options /tmp/offlinefiles/nemotronsuper-llm-api-config.yml
–reasoning_parser nano-v3
–max_num_tokens 8192
–tool_parser qwen3_coder’
extra_llm_api_options:
kv_cache_config:
dtype: fp8
enable_block_reuse: false
free_gpu_memory_fraction: 0.8
cuda_graph_config:
max_batch_size: 4
enable_padding: true
enable_chunked_prefill: true
stream_interval: 10