DGX Spark regression: kernel 7.0.0-1019-nvidia causes NCCL/RoCE ibv_reg_mr_iova2 ENOMEM; 6.17.0-1032 works

[mlau1]

I’d like to preface that I have no idea what I’m doing and all text below is generated from AI. This all could be user error but if its not maybe it’ll be useful.

I am seeing a reproducible RDMA/NCCL regression on a 2-node GB10 cluster after the NVIDIA HWE kernel upgraded from 6.17.0-1032-nvidia to 7.0.0-1019-nvidia.

Platform

• 2x NVIDIA GB10 systems
• DGX OS / Ubuntu 24.04 ARM64
• ConnectX-7 RoCE v2
• NVIDIA driver 580.173.02
• CUDA 13.0
• vLLM tensor parallel size 2

Working kernel

6.17.0-1032-nvidia
package: 6.17.0-1032.32

Failing kernel

7.0.0-1019-nvidia
package: 7.0.0-1019.19~24.04.2

The kernel was upgraded through the normal NVIDIA HWE meta-packages:

linux-image-nvidia-hwe-24.04
linux-headers-nvidia-hwe-24.04
linux-modules-nvidia-580-open-nvidia-hwe-24.04
linux-tools-nvidia-hwe-24.04
linux-nvidia-hwe-24.04

RoCE configuration

NCCL_NET=IB
NCCL_IB_DISABLE=0
NCCL_IB_HCA=roceP2p1s0f1
NCCL_SOCKET_IFNAME=enP2p1s0f1np1
NCCL_IB_ROCE_VERSION_NUM=2
NCCL_IB_ADDR_FAMILY=AF_INET
NCCL_IB_GID_INDEX=5

Raw RoCE/RDMA testing is healthy. ib_write_bw over the fabric reaches approximately 108 Gb/s.

Workload

deepseek-ai/DeepSeek-V4-Flash-Vision-Exp
2-node TP=2
vLLM
stock container:
ghcr.io/anemll/dspark-vllm-gx10:0.1.1
GPU_MEMORY_UTILIZATION_TEXT=0.835

On 7.0.0-1019-nvidia, the model loads but fails during vLLM profiling / tensor-parallel communication with:

Call to ibv_reg_mr_iova2 failed with error Cannot allocate memory

NCCL then reports a system error and the other rank eventually sees TCPStore/broken-pipe errors.

I tested multiple NCCL versions, including 2.28.9, 2.30.7, and 2.31.2, and the full workload can reproduce the same RDMA memory-registration failure.

I then performed a kernel-only A/B test.

I restored the standard container/configuration and booted both nodes back to:

6.17.0-1032-nvidia

With no other relevant configuration changes, the same workload succeeds:

container: healthy
Minimal chat request succeeded.
boot-shape-warmup: 47/47 requests ok

7.0.0-1019-nvidia
  -> ibv_reg_mr_iova2 ENOMEM / NCCL failure

6.17.0-1032-nvidia
  -> healthy startup / chat succeeds / 47 of 47 warmup requests succeed

Has anyone else seen an RDMA memory-registration regression with linux-nvidia 7.0.0-1019 on GB10/ConnectX-7?

I was using this: GitHub - MiaAI-Lab/DeepSeek-v4-Flash-DSpark-2x-DGX-Spark: DeepSeek-v4-Flash 0731 recipe for 2x DGX Sparks · GitHub

[anoop13]

I’m seeing the same thing. Reverting the kernel version fixes it.

[hiroshiya]

Have you checked if nccl-tests still work?

If you’ve never run nccl-tests before, there is a guide: NCCL for Multiple Sparks | DGX Spark

[chrisb03]

I was seeing similar NCCL issues with kernel: 7.0.0-1019-nvidia

I rolled back to the previous kernel I was using: 6.17.0-1031-nvidia

…and that corrected the problem for me as well..

[giles8]

How do I revert the kernel back to the previous version ?

I edited /etc/default/grub modified these lines:

#GRUB_DEFAULT=0
GRUB_DEFAULT="Advanced options for DGX OS GNU/Linux>DGX OS GNU/Linux, with Linux 6.17.0-1032-nvidia"

Then ran “sudo update-grub”

You need to make sure you select a grub option that exists on your system (check /boot/grub/grub.cfg)

[serg.piter]

Similar NCCL and MEMORY errors on 7.0.0-1019

Possible root cause from my prospective is a bug in CMA allocation that came with 7.0.0-1019

user@unaffected6.17.0-1032:~$ cat /proc/meminfo | grep -iE “CmaTotal|CmaFree|HugePages_Total|Hugetlb|VmallocUsed|DirectMap”
VmallocUsed: 863232 kB
CmaTotal: 131072 kB
CmaFree: 69320 kB
HugePages_Total: 0
Hugetlb: 0 kB

user@affected7.0.0-1019:~$ cat /proc/meminfo | grep -iE “CmaTotal|CmaFree|HugePages_Total|Hugetlb|VmallocUsed|DirectMap”
VmallocUsed: 577528 kB
CmaTotal: 0 kB
CmaFree: 135416 kB
HugePages_Total: 0
Hugetlb: 0 kB

CmaTotal is not announced correctly as it was in 6.17.0-1032 (latest unaffected)

I hope it will help.

[OllieJW]

I am seeing NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359 during CUDA graph capture and NCCL initialization on multi‑node workloads, and came to the forums to see if others were experiencing the same. Kernel: 7.0.0‑1019‑nvidia Driver: 580.173.02 CUDA: 13.0 NCCL: 2.31.2 and RoCE firmware: 28.45.4028. I will look at rolling back - this was super disruptive as it regressed the models I had successfully served in the past and stopped work from progressing.

[J-Grove]

Also upgraded the kernel today to this version and seeing the same issue.. no longer able to serve models that previously worked.

[fidecastro]

Just to add yet another person seeing this, and I confirm the same diagnosis with Grok on my Sparks.

[OllieJW]

How are you all rolling back to a previous kernel? Is renaming /etc/default/grub.d/no-grubmenu.cfg and running grub-set-default "Advanced options for DGX OS GNU/Linux>DGX OS GNU/Linux, with Linux 6.17.0-1032-nvidia" the sustainable way to do this?

[kenleo_lucas]

Does Nvidia try to destroy every local model player’s kitchen to follow Dario’s shouting? Be careful, this could be a conspiracy.

[eugr_nv]

I’ve raised this internally, hopefully we’ll have a resolution shortly.

[dbsci]

All, ymmv / read the script (or let your agent do it) and be careful, but if you run this script with sudo, it should basically help you do the rollback. No warranty or guarantees provided!

rollback-script.sh.txt (2.7 KB)

[kenleo_lucas]

I updated the kernel by DGX Dashboard and triggered the breakdown.
Then I did this to get it back:

sudo grep -E "1032|7.0.0-1019" /boot/grub/grub.cfg      # check first

sudo sed -i 's|^#\?GRUB_DEFAULT=.*|GRUB_DEFAULT="Advanced options for DGX OS GNU/Linux>DGX OS GNU/Linux, with Linux 6.17.0-1032-nvidia"|' /etc/default/grub
sudo update-grub
sudo reboot

[hypermac.6502]

I was getting an error during startup, but thanks to the info shared here, I rolled back and got it working normally again.

Thank you all.

[gurvann]

Hi

Confirming the regression and the fix on a dual-node setup (2x ASUS Ascent GX10 / GB10, DGX OS 24.04 ARM64, ConnectX-7 RoCE v2, direct 200G QSFP link, driver 580.173.02, CUDA 13.0).

After updating via DGX Dashboard on Sep 12, both nodes went from 6.17.0-1032-nvidia to 7.0.0-1019-nvidia, and vLLM tensor-parallel-2 stopped starting with the exact same signature:

misc/ibvwrap.cc:217 (wrap_ibv_reg_mr_iova2) NCCL WARN

Call to ibv_reg_mr_iova2 failed with error Cannot allocate memory

RuntimeError: NCCL error: unhandled system error

Things I ruled out before finding this thread, in case it saves someone time:

- memlock: unlimited both inside the container and via /etc/security/limits.d, verified with ulimit -l

- NCCL_IB_GID_INDEX=3 pinned, valid RoCEv2 IPv4 GID present on the cabled HCA

- pinning a single HCA instead of both rails

- /dev/infiniband exposed inside the container

- vm.max_map_count already at 1048576

- MTU 9000 validated end to end (ping -M do -s 8972, 0% loss)

None of these changed anything. The kernel is the only variable.

The GRUB_DEFAULT fix posted above works. Two notes for anyone doing this headless:

- GRUB_DEFAULT=saved combined with grub-set-default did NOT work here, the saved entry was written to grubenv but ignored at boot. Only the explicit submenu>menuentry path worked.

- Same fix is also documented in NVIDIA’s own docs for DGX Spark (Aerial CUDA-Accelerated RAN install guide), which is a good sign it is the supported way on this platform.

Measured impact of the NCCL_IB_DISABLE=1 TCP fallback, in case anyone is considering it instead of reverting the kernel: generation throughput dropped from 46-70 tok/s to 23 tok/s on DeepSeek-V4-Flash TP=2, and DSpark speculative acceptance fell from ~55% to ~30%. It is a stopgap, not a workaround.

Kernel is now held with apt-mark hold on linux-nvidia-hwe-24.04, linux-image-nvidia-hwe-24.04 and linux-headers-nvidia-hwe-24.04 until a fixed kernel ships.

[OllieJW]

Thank you.

[ForsakenSilver]

I can reproduce the same issue on a 4-node GB10 TP4 switchless ring running DeepSeek-V4-Flash-0731.

With 7.0.0-1019-nvidia, NCCL/RoCE intermittently failed with:

Call to ibv_reg_mr_iova2 failed with error Cannot allocate memory

Even a 1 MiB NCCL AllReduce could trigger it. When the model did start, prefill was only about 1,430-1,490 tok/s.

After booting all four nodes into 6.17.0-1032-nvidia, the error disappeared. The same ring reached about 187 Gb/s AllReduce bus bandwidth, and prefill increased to 2,590-2,830 tok/s. Across the full benchmark matrix, PP improved by 87.9% and TTFT dropped by 47.1%.

I also repeated the test on my 2-node TP2 direct-connect setup. TP2 did not show the same regression: prefill on 6.17 was within 1% of the previous result, with no clear decode improvement.

So in my case, the regression affects TP4 much more severely, possibly because of the additional RDMA registrations or HCA/collective pressure. The physical ring itself was not the main bottleneck.

[james.park4]

I hit it this morning. they should pull the update. How did this get past QA?

[NVES]

DGX Spark Update Advisory

We are investigating reports of an issue affecting multi-node NCCL/RoCE workloads after updating to kernel 7.0.0-1019-nvidia.

While we investigate, we recommend that DGX Spark customers using multi-node/RoCE configurations hold off on updating to kernel 7.0.0-1019, including updates via DGX Dashboard.

If you have already updated and are experiencing issues, please share your configuration and symptoms in this thread.

We will provide an update as soon as we have more information.