Hard thermal shutdown during prefill on long context using latest spark-vllm-docker

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

I just updated my docker image to the latest b12x version of vllm, and now the main FE gold Spark in my 2-node cluster (DGX+GX10) does a hard shutdown during long prefill. This is 100% repeatable. When rebooting the cluster, the chat history saved in Open WebUI must be re-cached, so with long chat sessions vllm needs to do prefill for the chat history before it can pick the conversation back up. This hammers the GPU on the Spark, but I noticed even with 98% GPU usage at ~2400MHz and power spikes to 97W, the fans never spin up to 100%. I’m using the latest EC firmware, 0x03000508, but I tried to roll back to 0x02 and that didn’t fix the problem.

The only thing that got me back up-and-running was using the fan control kernel module from GitHub - christopherowen/dgx-spark-fan-control: Userland fan control for NVIDIA DGX Spark, with additive RPM floors, a temperature-based curve, Secure Boot signing, and DKMS support. · GitHub. I had to set the fan to 100% at all times to prevent the machine from shutting down. But now the fan is on all the time, which it doesn’t really need to be since token generation is limited by memory bandwidth and therefore the machine doesn’t get as hot. But I can’t anticipate when long contexts will need to be processed, and it’s those instances where the machine is at risk of shutting off.

Is there going to be any fix for this in the future or do I need to take other action? This is really frustrating since the GX10 doesn’t seem to have any problem with this at all.

2

I’ve got similar experience and applied the same fix but using the supplied custom curve – so fans are certainly not always running at 100%. In my case the CPU was overheating with NVIDIA’s default cooling profile, not the GPU… Some more details are available in this thread: 7.0.0-1019-nvidia CPU overheating

3

Thanks. Yeah, I’m using the dgx-spark-fan-control systemd service now to adjust the fan dynamically, and it seems OK so far. I’ll stick with this for a while and see how it goes.

I’m not sure why we need to rely on community mods for this, though. I think this should be fixed in the firmware (Nvidia devs listening?).

4

More information on this. I ran sudo ./partnerdiag --field, and the machine shut down before the test even completed. So I can’t even diagnose the problem because the diag tool won’t complete. The last entry in fieldiag.log before the machine hard shuts down is:

Testing PowerStress -\|/-\|/-\|/-\|/-\|/-\|

5

If unit shuts down during field diagnostic, then you are eligible for RMA. If it is an FE Spark contact NVIDIA Customer support, otherwise contact your OEM

6

Thanks. Yes, it’s the gold Nvidia model. I already started the RMA process so at least the issue can be documented before the warranty is up. Hopefully I can get this resolved quickly!