Hi Team I am trying to enable GDS as NVMe supported on Dell R750XA server . Below is my OS configuration [root@node002 src]# cat /etc/centos-release Rocky Linux release 8.6 (Green Obsidian) [root@node002 src]# [root@node002 src]# uname -r 4.18.0-477.27.1.el8_8.x86_64 [root@node002 src]# dnf lis…

Not sure from the errors why the NVMe DMA is failing. Can you please look at the below questions. what drive is this ? lspci -nn what is the exact MOFED version ? ofed_info -s ? Is this a local nvme drive or a NVMeOF drive ? nvme list is IOMMU set to ON ? $ gdscheck -p what is the PCIe topolog…

@kmodukuri , anything you can share on above error , we are almost close in fixing the issue :)

Can you try a simple test with /usr/local/cuda-12.3/gds/tools/gdsio_verify -f /mnt/gds/gdstest -n 1 -m 0 -s 4k -o 0 -d 0 -t 0 /usr/local/cuda-12.3/gds/tools/gdsio -f /mnt/gds/test -d 0 -w 1 -s 4k -x 0 -i 4K -I 1 -V

@kmodukuri , please refer the below requested output [root@gds ~]# /usr/local/cuda-12.3/gds/tools/gdsio_verify -f /mnt/gds/gdstest -n 1 -m 0 -s 4k -o 0 -d 0 -t 0 gpu index :0,file :/mnt/gds/gdstest, gpu buffer alignment :0, gpu buffer offset :0, gpu devptr offset :0, file offset :0, io_requested :4…

@kmodukuri , anything on above ?

@kmodukuri , anything that can be shared on above issue , please let me know if you need more information

[image] kmodukuri: lspci -nn [root@gds ~]# lspci -nn 00:00.0 System peripheral [0880]: Intel Corporation Ice Lake Memory Map/VT-d [8086:09a2] (rev 04) 00:00.1 System peripheral [0880]: Intel Corporation Ice Lake Mesh 2 PCIe [8086:09a4] (rev 04) 00:00.2 System peripheral [0880]: Intel Corporat…

[image] kmodukuri: ofed_info -s ? This is a local NVMe drive . [root@gds ~]# ofed_info -s MLNX_OFED_LINUX-5.8-3.0.7.0:

[image] kmodukuri: gdscheck -p IOMMU is off [root@gds ~]# /usr/local/cuda-12.3/gds/tools/gdscheck.py -p GDS release version: 1.8.1.2 nvidia_fs version: 2.18 libcufile version: 2.12 Platform: x86_64 ============ ENVIRONMENT: ============ ===================== DRIVER CONFIGURATION: =…

[image] kmodukuri: lspci -tvvv ? GDS mount NVMe ssd → Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/PM9A3/980PRO OS drive is other NVMe device , please refer the lsblk and lspci outputs below [root@gds ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS nvme1n1 259:0 0 …

@kmodukuri , please check the above requested outputs. Below is some more information. [root@gds ~]# lsmod | grep nvme nvme 65536 6 nvme_core 135168 8 nvme mlx_compat 20480 13 rdma_cm,ib_ipoib,mlxdevm,nvme,iw_cm,nvme_core,ib_umad,ib_core,rdma_ucm,ib_uv…

NVMe Driver not registered with nvidia-fs - GDS NVMe unsupported on Rocky 8.6

Accelerated Computing GPU-Accelerated Libraries

karanveersingh5623 December 7, 2023, 4:05am 45

@kmodukuri , any pointers you can share on this issue . Is it purely related to NVMe device and PCIe bus ?

Topic		Replies	Views
NVMe Driver not registered with nvidia-fs on Ubuntu 20.04 GPU-Accelerated Libraries gds	5	1002	March 22, 2024
Nvme unsupported from "gdscheck" in GPU direct storage GPU-Accelerated Libraries nvme	3	3809	April 17, 2023
Gdscheck says gds supported but nvidiafs shows error Storage gds	3	237	September 24, 2025
How do I use Nvidia GDS with NVME without compatability mode? GPU-Accelerated Libraries cuda , gds	4	2401	November 22, 2023
Why is my NVMe storage unsupported in GDS? (Ubuntu 22.04 + Tesla V100) GPU-Accelerated Libraries	0	187	November 13, 2025
GDS read/write error:use gdsio -I 1 to write GPU-Accelerated Libraries gds	4	156	September 22, 2025
Is GDS actually enabled on my system? gdscheck shows NVMe Unsupported but cuFile I/O works CUDA Setup and Installation gds , system-setup	1	187	February 23, 2026
Failing to use gds on A4000 Storage gds	5	261	March 19, 2025
How to verify GPUDirect Storage's P2P DMA is working correctly for local attached NVMe SSD? GPU-Accelerated Libraries nvme , gds	0	468	August 28, 2024
How to Configure and force GDS to use nvidia-fs path instead P2PDMA GPU-Accelerated Libraries gds	8	193	February 12, 2026

NVMe Driver not registered with nvidia-fs - GDS NVMe unsupported on Rocky 8.6

Related topics