Is GDS actually enabled on my system? gdscheck shows NVMe Unsupported but cuFile I/O works

Infrastructure & Networking DOCA Storage
,
1

Hi,

I’m trying to verify whether GPUDirect Storage (GDS) is fully enabled on my system.
I installed the NVIDIA Open Kernel driver, CUDA toolkit, and GDS (nvidia-fs + libcufile). gdscheck reports a supported platform, but the device section still shows NVMe: Unsupported, which is confusing because gdsio achieves high GPU-direct throughput.

Here are the details.

1. gdscheck output

GDS release version: 1.13.0.11
nvidia_fs version:  2.26
libcufile version: 2.12
Platform: x86_64

=====================
DRIVER CONFIGURATION:
=====================
NVMe P2PDMA        : Unsupported
NVMe               : Unsupported
NVMeOF             : Unsupported
SCSI               : Unsupported
...

Userspace RDMA     : Unsupported
--rdma library        : Not Loaded (libcufile_rdma.so)
--rdma devices        : Not configured

...
GPU index 0 NVIDIA RTX A6000 bar:1 bar size (MiB):256 supports GDS
IOMMU State: Disabled

Platform verification succeeded

2. Filesystem & hardware info

My NVMe is mounted here:

/main-partition-samsung (ext4)

nvidia_fs is loaded:

nvidia_fs  274432  0
nvidia    11644928  3 nvidia_uvm,nvidia_fs,nvidia_modeset

Kernel:

Linux 6.8.0-85-generic

Driver supports CUDA 12.8:

Cuda Driver Version Installed: 12080

3. GDS I/O performance (gdsio)

Even though gdscheck shows NVMe as unsupported, gdsio gives very high throughput (3.8 GiB/s using GPU memory):

IoType: WRITE XferType: GPUD Threads: 4
Throughput: 3.827373 GiB/sec
Avg_Latency: 4050 usecs

This suggests that cuFile is taking the GDS path successfully.

4. My questions

(1) Is GDS actually enabled/active on this system, despite NVMe: Unsupported in gdscheck?

(2) Does “NVMe: Unsupported” indicate lack of peer-to-peer DMA capability (P2PDMA), or that the NVMe driver didn’t register with nvidia-fs?

(3) What exactly does gdscheck require for “NVMe: Supported”?

Is it only dependent on platform topology (ACS/IOMMU), or also NVMe firmware/driver specifics?

(4) Since gdsio achieves high GPU-direct throughput, is it safe to assume that cuFile → nvidia-fs → NVMe direct path is active?

Happy to provide:

  • /proc/driver/nvidia-fs/devices

  • /proc/driver/nvidia-fs/mounts

  • PCIe topology

  • NVMe controller details (lspci -vvv)

  • Kernel logs

Thanks in advance for any clarification!