HI , I understand while using latest kernel ( above 6.2) which has P2P DMA , we dont need to patch nvme driver/use nvidia-fs/install mofed . But what settings/config are needed to force GDS Apps to use nvidia-fs and what are the things needed to be configure?
1.NVIDIA-FS ( must)
2.MOFED - is it necessary ?
3.Any nvidia.ko modules parameter needed?
like NVreg_RegistryDwords="RMForceStaticBar1=1;RmForceDisableIomapWC=1; required for P2PDMA
4.cufile.json parameters?
@alok.prasad7 to use nvidia-fs.ko for NVMe please set
“use_pci_p2pdma”: false .
Make sure the NVMe entry in gdscheck shows nvfs
NVMe : nvfs, compat
if the OS is not BaseOs, then the NVMe patches to support GDS should be installed with DOCA
@kmodukuri i have vanilla kernel installed (6.12 from kernel.org) , DOCA package will fail as its not the Base kernel /default kernel that comes with ubuntu 24.04 , can you point me what exact package from DOCA needs to be patched for NVME.
the instructions for Ubuntu 24.04
The actual package that contains the name patches is
mlnx-nvme-dkms_25.10.OFED.25.10.1.7.1.1-1_all.deb
if you extract the local deb you should see in
./usr/share/doca-host-3.2.1-044000-25.10-ubuntu2404/repo/pool/mlnx-nvme-dkms_25.10.OFED.25.10.1.7.1.1-1_all.deb
@kmodukuri Thanks for pointing me the exact patches. any other patches also do i need?
Also had related to nvidia driver module NVreg_RegistryDwords="RMForceStaticBar1=1;RmForceDisableIomapWC=1; is it required
For nvidia-fs or its only for P2PDMA.
Is it feasible to make this patches work without installing MOFED/DOCA OFED
what i see is that patches are actually files under <linux_kernel_src>/source/drivers/nvme/
So if i replace common, host and target it should be fine (hope there is no compilation or other dependency)
The NVreg_RegistryDwords="RMForceStaticBar1=1;RmForceDisableIomapWC=1; is required if you want to enable PCI_P2PDMA support.
The NVMe patches include all patches from mellanox along with GDS patches for the specific DOCA install.
@kmodukuri one final query , does nvidia-fs uses leverage nvidia-peermem.ko or its directly using nv-p2p.c calls.In short do i need to install nvidia-peermem too? btw still need have to try patchin nvme module from doca to my kernel.
nvidia-peermem/DMABuf is NOT needed for NVMe/NVMeOF driver.
nvidia-peermem is our legacy implementation to support userspace RDMA path (WekaFS and GPFS). with newer kernels DMABuf is used instead of nvidia-peermem. if you want to know the differences please see the section GPUDirect RDMA and GPUDirect Storage — NVIDIA GPU Operator 24.6.2 documentation