Abysmal performance and triton crashes

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

Dear community,

I made the HUGE mistake of trading an AGX Thor development system for a DGX Spark.

On the Thor I could see up 4500 token/s for prefill (Qwen3.6-35B-A3B-NVFP4) after vllm and pytorch optimized builds from scratch. This was on JP 7.2 and current CUDA.

On the DGX Spark I can see 150 - 600 token/s for prefill. There is something seriously wrong here.

Trying vllm with Triton I see crash boom bang burn. See log snippet attached.

I am beyond frustrated. I can see the performance that I get from the DGX Spark on an Orin AGX which has a custom compiled JP7.2 environment. The AGX Orin is on Ampere.

I get it. I am to blame. I gave away a perfectly running Thor for an unfinished gizmo that severely lacks in what I actually wanted to use it for, and Nvidia has raised prices by about 100% for the Thor, making it just not feasible to go out and just buy another one. I am miffed.

How can I at least get vllm to perform reasonably ?

(and no, I am not a docker person. There are licensing issues with docker related stuff that make any mention of the word “docker” heresy at the company I work at.)

Thank you for your insights.

@Nvidia could you at least get Cuda 13.2 for the DGX plus native sm121a support integrated into your tool chain so that stuff starts working ?

(EngineCore pid=396964) WARNING 08-20 15:10:07 [qwen_gdn_linear_attn.py:1085] File “/opt/vllm/src/lib/python3.12/site-packages/triton/compiler/compiler.py”, line 80, in make_ir
(EngineCore pid=396964) WARNING 08-20 15:10:07 [qwen_gdn_linear_attn.py:1085] return ast_to_ttir(self.fn, self, context=context, options=options, codegen_fns=codegen_fns,
(EngineCore pid=396964) WARNING 08-20 15:10:07 [qwen_gdn_linear_attn.py:1085] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=396964) WARNING 08-20 15:10:07 [qwen_gdn_linear_attn.py:1085] triton.compiler.errors.CompilationError: at 37:14:
(EngineCore pid=396964) WARNING 08-20 15:10:07 [qwen_gdn_linear_attn.py:1085] else:
(EngineCore pid=396964) WARNING 08-20 15:10:07 [qwen_gdn_linear_attn.py:1085] bos, eos = i_b * T, i_b * T + T
(EngineCore pid=396964) WARNING 08-20 15:10:07 [qwen_gdn_linear_attn.py:1085]
(EngineCore pid=396964) WARNING 08-20 15:10:07 [qwen_gdn_linear_attn.py:1085] if HEAD_FIRST:
(EngineCore pid=396964) WARNING 08-20 15:10:07 [qwen_gdn_linear_attn.py:1085] p_s = tl.make_block_ptr(
(EngineCore pid=396964) WARNING 08-20 15:10:07 [qwen_gdn_linear_attn.py:1085] s + bos * H + i_h * T, (T,), (1,), (i_t * BT,), (BT,), (0,)
(EngineCore pid=396964) WARNING 08-20 15:10:07 [qwen_gdn_linear_attn.py:1085] )
(EngineCore pid=396964) WARNING 08-20 15:10:07 [qwen_gdn_linear_attn.py:1085] p_o = tl.make_block_ptr(
(EngineCore pid=396964) WARNING 08-20 15:10:07 [qwen_gdn_linear_attn.py:1085] o + bos * H + i_h * T, (T,), (1,), (i_t * BT,), (BT,), (0,)
(EngineCore pid=396964) WARNING 08-20 15:10:07 [qwen_gdn_linear_attn.py:1085] )
(EngineCore pid=396964) WARNING 08-20 15:10:07 [qwen_gdn_linear_attn.py:1085] else:
(EngineCore pid=396964) WARNING 08-20 15:10:07 [qwen_gdn_linear_attn.py:1085] p_s = tl.make_block_ptr(s + bos * H + i_h, (T,), (H,), (i_t * BT,), (BT,), (0,))
(EngineCore pid=396964) WARNING 08-20 15:10:07 [qwen_gdn_linear_attn.py:1085] ^
(EngineCore pid=396964) WARNING 08-20 15:10:07 [qwen_gdn_linear_attn.py:1085] Block pointers have been removed in favor of the tensor descriptor API
(EngineCore pid=396964) WARNING 08-20 15:10:07 [qwen_gdn_linear_attn.py:1085] GDN prefill kernel warmup (T=64) failed for layer language_model.model.layers.61.linear_attn. First inference may OOM due to autotuner.
(EngineCore pid=396964) WARNING 08-20 15:10:07 [qwen_gdn_linear_attn.py:1085] Traceback (most recent call last):
(EngineCore pid=396964) WARNING 08-20 15:10:07 [qwen_gdn_linear_attn.py:1085] File “/opt/vllm/src/lib/python3.12/site-packages/triton/language/core.py”, line 42, in wrapper
(EngineCore pid=396964) WARNING 08-20 15:10:07 [qwen_gdn_linear_attn.py:1085] return fn(*args, **kwargs)
(EngineCore pid=396964) WARNING 08-20 15:10:07 [qwen_gdn_linear_attn.py:1085] ^^^^^

2

for nvidia/Qwen3.6-35B-A3B-NVFP4 you can easily get 5000-6000 prefill and 90-110+tks decode using most of the recipes in sparkrun but then again you have the no docker constraint. I don’t see why you wont be able to setup vllm and get those speeds as well, might be a little involved though.

3

I am very open for suggestions on how to rebuild triton, python, vllm to give me that kind of performance.

I am getting loads of sm_121a not implemented from ptxas, and similar errors. I am extremely frustrated by now, especially given the fact that my AGX Orin beats the Spark in inference. This should not happen as far as I am concerned …

4

Have you tried using sparkrun or @eugr_nv’s spark-vllm-docker images to get a validated setup tested first? That would give you a solid baseline for comparison, and those models are definitely available with either option and essentially turnkey to deploy.

5

no, I have not. As I said, docker is a huge red flag and basically unusable within our organization.

6

Would Podman (essentially a drop-in replacement for Docker which runs Rootless and Daemonless out of the box) be an alternative?

7

I could give it a try, potentially. However I would appreciate the underlying software issue to be fixed. Why is building triton / pytorch out of the box not working properly for sm_121a ? I have no problems recompiling those, I think for a gizmo that was advertized as the AI super computer on the desktop, a lot of stuff is not working properly.

The Spark was advertized as Blackwell. It should be able to work fine with NVFP4 models and provide reasonable throughput.

I am perfectly fine compiling pytorch / triton / vllm with the proper flags and then selecting the right back ends to speed up performance. I understand that docker, etc is extremely convenient for most folks not subjected to docker license policies. I would like to see the GB10 support in the full software stack, Cuda 13.2 officially supported, plus the current kernel, …

If there are wheels for pytorch, triton, vllm, … which give me reasonable performance for the GB10 kindly point me in their direction.

I will look at the eugr repository for inspiration and try to rebuild locally.