When I run the tensorrt-llm docker image and serve llama 4 DGX Spark crashes. Power cycle to recover.
docker run --rm -it --ipc host --gpus all --ulimit memlock=-1 --ulimit stack=67108864 -p 9000:8000 -v ~/.cache/huggingface:/root/.cache/huggingface nvcr.io/nvidia/tensorrt-llm/release:1.2.0rc4
trtllm-serve “nvidia/Llama-4-Scout-17B-16E-Instruct-NVFP4” --host 0.0.0.0
Can you provide any logs from the run and also submit an nvidia-bug-report
We’ve been investigating the FlashInfer MoE FP4 cubin issue on DGX Spark (GB10/SM121) which may be related to crashes in TRT-LLM. The root cause we found: the FlashInfer batched GEMM cubins in the artifactory are compiled for SM100 only, and cuModuleLoadData() fails when trying to load them on SM121 since SM100 and SM121 are different major architecture families (Blackwell B-series vs GB10).
SM120 cubins would be forward-compatible with SM121, but none are currently published. We’ve filed a request on the TRT-LLM repo to add SM120 cubins: [Feature]: Compile trtllm-gen FMHA cubins for SM120/SM121 (consumer Blackwell) · Issue #11799 · NVIDIA/TensorRT-LLM · GitHub
Workaround: use the Triton MoE path (VLLM_USE_FLASHINFER_MOE_FP4=0) instead of the FlashInfer cubin path.
Please request this from the flashinfer folks as well, [Bug]: Qwen3.5 NVFP4 models crash on ARM64 GB10 DGX Spark (CUDA illegal instruction during generation) · Issue #35519 · vllm-project/vllm
I’d be willing to bet it’s impacting any inferencing with flashinfer+nvfp4+sm120+sm121 and have been desperately trying to get nvidia to prioritize it.