DGX Spark: nvidia-peermem fails to load with "Invalid argument" on 6.17.0-1014-nvidia after DOCA/MLNX OFED DKMS installation

Accelerated Computing Aerial Forum
1

Hi NVIDIA Aerial team,

We are setting up NVIDIA Aerial CUDA-Accelerated RAN on DGX Spark.

The PTP path is now working correctly. The Dell switch is locked to the VIAVI Qg 2 Grandmaster, and the DGX Spark ptp4l/phc2sys services are also locked:

  • ptp4l rms: ~3 ns
  • phc2sys rms: ~11 ns
  • PTP GM identity: fcaf6a.fffe.02e8d2

We also resolved the previous DKMS loop caused by both 6.17.0-1014-nvidia and 6.17.0-1018-nvidia being installed. The system is now running:

uname -r
6.17.0-1014-nvidia

The following packages are now installed successfully:

  • doca-ofed 3.2.1-044000
  • mlnx-ofed-kernel-dkms 25.10.OFED.25.10.1.7.1.1-1
  • iser-dkms 25.10.OFED.25.10.1.7.1.1-1
  • isert-dkms 25.10.OFED.25.10.1.7.1.1-1
  • srp-dkms 25.10.OFED.25.10.1.7.1.1-1

DKMS status shows that the MLNX OFED modules are installed for 6.17.0-1014-nvidia. The NVIDIA driver is also installed:

  • NVIDIA driver: 590.48.01
  • CX7 firmware: 28.47.1088
  • DOCA version seen in container: 3.2.1025

The nvidia-peermem module exists and matches the running kernel:

modinfo nvidia-peermem
filename: /lib/modules/6.17.0-1014-nvidia/updates/dkms/nvidia-peermem.ko.zst
version: 590.48.01
vermagic: 6.17.0-1014-nvidia SMP preempt mod_unload modversions aarch64

However, nvidia-peermem fails to load:

sudo modprobe -v nvidia-peermem
insmod /lib/modules/6.17.0-1014-nvidia/updates/dkms/nvidia-peermem.ko.zst
modprobe: ERROR: could not insert ‘nvidia_peermem’: Invalid argument

We also tested both parameter combinations:

sudo modprobe -v nvidia-peermem persistent_api_support=1 peerdirect_support=0
sudo modprobe -v nvidia-peermem peerdirect_support=1 persistent_api_support=0

Both return the same error:

modprobe: ERROR: could not insert ‘nvidia_peermem’: Invalid argument

As a result, nvidia.service fails because /usr/local/bin/nvidia.sh contains:

modprobe nvidia-peermem 2>/dev/null || lsmod | grep -q nvidia_peermem || { echo “[ERROR] Failed to load nvidia-peermem”; FAILED=1; }

Could you please confirm whether nvidia-peermem is expected to load on DGX Spark with the following combination?

  • Kernel: 6.17.0-1014-nvidia
  • NVIDIA driver: 590.48.01
  • DOCA OFED: 3.2.1 / 25.10 OFED
  • MLNX OFED DKMS: 25.10.OFED.25.10.1.7.1.1-1
  • CX7 firmware: 28.47.1088
  • Platform: DGX Spark / aarch64

Questions:

  1. Is nvidia-peermem required for the current Aerial Testbed / ACAR 26-1 workflow on DGX Spark?
  2. Is this driver/kernel/OFED combination officially supported?
  3. Is there a known issue where nvidia-peermem returns EINVAL / Invalid argument on DGX Spark?
  4. Is there a required module parameter or service ordering dependency before loading nvidia-peermem?
  5. Should nvidia.service treat nvidia-peermem load failure as fatal on DGX Spark, or can Aerial continue without it?

Thanks.

2

Hi @xudong.zhao

  1. nvidia-peermem is not supported on dgx-spark
  2. yes, the drive/kernel/OFED combinations you listed are officially supported.
  3. N/A: nvidia-peermem is not supported, so it is not needed.
  4. N/A
  5. Aerial will continue without it. The error message can be ignored. It should be removed from the service.