Dear NVIDIA Developer Support Team, We are reporting a critical stability issue encountered while deploying large-scale models (120B+) on a SparkDGX (Blackwell GB10) multi-node cluster. The system exhibits consistent crashes during high-load tasks, involving both local VRAM allocation failures and …

Can you contact Support to get RMA ticket opened?

First thing: sudo fwupdmgr enable-remote lvfs-testing sudo fwupdmgr refresh sudo fwupdmgr update Most important: Once complete (or if no updates found), you need to: Power off you spark ( unplug the cable from the spark itself and unplug the power brick from the wall) Wait for at least 60 seco…

@blue8 - Thank you for the detailed report and configuration info you’ve provided. To help narrow this down, I have some questions about reproducing the error. First question - has this specific configuration (SparkDGX GB10 dual-node with 120B-class models and similar parallelism) run stably in th…

Following up with a 580.159.03 (May 5 2026 ERD/UDA release, LP #2151130) data point on the same GB10 dual-DGX-Spark hardware. Setup: 2x DGX Spark (GB10 Grace-Blackwell, 128 GB unified, Ubuntu 24.04, linux-image 6.17.0-1018-nvidia HWE) NVIDIA open kernel module 580.159.03 (apt full-upgrade from 5…

We hit the same NCCL-timeout-mid-workload pattern on GB10 multi-node, though on the training side (FSDP) rather than inference. Two things moved it for us. NCCL config was the bigger one. NCCL_NET_GDR_LEVEL=0 specifically — the published Blackwell guidance points to GDR level 5 and that actively br…

Confirming the same signature on a second 2× DGX Spark deployment — and importantly, on the newer driver 580.159.03 (not 580.142). Mitigation reduces the rate but the leak is still terminal: we hit it after ~20-30 min of light serving. Setup: 2× DGX Spark (GB10, SM121), CX-7 between nodes, Ubuntu 2…

@vgoklani Thanks for the cross-deployment confirmation. The newer driver (580.159.03) and the light-serving ~20-30 min onset are both useful here. They differ from where we hit it: training-side FSDP on 580.95.05. Different workload and driver, same terminal behavior actually argues this is UMA-stru…

Another GB10 data point — hard lockup requiring a physical power-cycle under sustained 2-node Qwen3.5-122B TP=2 Confirming I’m hitting this on a 2× DGX Spark setup. In my case the box goes completely unresponsive and only a physical power-cycle brings it back — not a hang I can recover from remotel…

Can someone from @nvidia please respond??? Is anyone looking into this? Where do we file the bug ticket??? These sparks are useless. We can’t train with them, there isn’t enough SRAM for our custom kernels. Now we can’t even run inference across two sparks without having deadlocks, WHICH REQUIRE H…

[image] miken: 13–30 min into serving, the machine becomes totally unresponsive — headless, no SSH, nothing on the network, nothing in the kernel log (journald just stops mid-line; no Xid, MCE/EDAC, thermal trip, OOM, or panic). A physical power-cycle is the only way back. I can’t tell whether i…

Multi-Node Inference Crash on Blackwell GB10: Memory Allocation (0x51) & NCCL Timeouts (Tested on Qwen 122B & Nemotron 120B)

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10

j0n May 23, 2026, 6:56am 7

Paging @eugr_nv as I believe he is looking into the NV_ERR_NO_MEMORY issue, which seems to be affecting more people recently across the forum (and beyond).

Topic		Replies	Views
DGX Spark regression: kernel 7.0.0-1019-nvidia causes NCCL/RoCE ibv_reg_mr_iova2 ENOMEM; 6.17.0-1032 works DGX Spark / GB10 kernel , deepseek , dgx-spark-issue	73	6726	September 22, 2026
NCCL all-reduce deadlock on dual DGX Spark after successful channel establishment — affects both vLLM and TRT-LLM DGX Spark / GB10 nemotron	21	1473	April 17, 2026
DGX Spark Shutdown around 95°C during nanoChat Pretraining (20-30 min) DGX Spark / GB10	21	2391	March 23, 2026
Day 1 with DGX Spark (Asus version) DGX Spark / GB10	29	2992	February 7, 2026
7.0.0-1019-nvidia CPU overheating DGX Spark / GB10	12	618	October 3, 2026
DGX Spark Multi-Node LLM Inference Report for Qwen3-235B model DGX Spark / GB10 nim , llama	34	3538	May 1, 2026
Two-Spark cluster with vLLM using tensor-parallel-size 2 causes one node to drop while the other's GPU goes 100% forever DGX Spark / GB10	36	2762	February 13, 2026
DGX Spark stability / out of RAM / overheating DGX Spark / GB10 llama , dgx-spark-issue	51	4425	September 7, 2026
DGX Spark GB10 hard-resets after vLLM silently exits 255 serving Qwen3.8-27B-FP8 — no OOM, Xid, or traceback DGX Spark / GB10 agentic-ai , dgx-spark-issue	5	518	August 26, 2026
Spark-vllm-docker runs out of memory loading Qwen3.5-397B-A17B-int4-AutoRound DGX Spark / GB10	36	1528	May 13, 2026

Multi-Node Inference Crash on Blackwell GB10: Memory Allocation (0x51) & NCCL Timeouts (Tested on Qwen 122B & Nemotron 120B)

Related topics