DGX Spark regression: kernel 7.0.0-1019-nvidia causes NCCL/RoCE ibv_reg_mr_iova2 ENOMEM; 6.17.0-1032 works

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
, ,
43

While we are working on the update, here are the steps to mitigate this issue without reverting the kernel:

  1. Edit /etc/default/grub in your favorite editor (e.g. vim: sudo vi /etc/default/grub). Add kho=off to GRUB_CMDLINE_LINUX_DEFAULT line so it looks like this: GRUB_CMDLINE_LINUX_DEFAULT="console=tty0 console=ttyS0,921600 kho=off" (leave all other values as they are on your system, you just need to add kho=off). Save the file.
  2. Run sudo update-grub.
  3. Reboot: sudo reboot.
  4. Check that KHO is off: sudo ls /sys/kernel/debug/kho - it should give you an error: ls: cannot access '/sys/kernel/debug/kho': No such file or directory