Hi,
I’m trying to verify whether GPUDirect Storage (GDS) is fully enabled on my system.
I installed the NVIDIA Open Kernel driver, CUDA toolkit, and GDS (nvidia-fs + libcufile). gdscheck reports a supported platform, but the device section still shows NVMe: Unsupported, which is confusing because gdsio achieves high GPU-direct throughput.
Here are the details.
1. gdscheck output
GDS release version: 1.13.0.11
nvidia_fs version: 2.26
libcufile version: 2.12
Platform: x86_64
=====================
DRIVER CONFIGURATION:
=====================
NVMe P2PDMA : Unsupported
NVMe : Unsupported
NVMeOF : Unsupported
SCSI : Unsupported
...
Userspace RDMA : Unsupported
--rdma library : Not Loaded (libcufile_rdma.so)
--rdma devices : Not configured
...
GPU index 0 NVIDIA RTX A6000 bar:1 bar size (MiB):256 supports GDS
IOMMU State: Disabled
Platform verification succeeded
2. Filesystem & hardware info
My NVMe is mounted here:
/main-partition-samsung (ext4)
nvidia_fs is loaded:
nvidia_fs 274432 0
nvidia 11644928 3 nvidia_uvm,nvidia_fs,nvidia_modeset
Kernel:
Linux 6.8.0-85-generic
Driver supports CUDA 12.8:
Cuda Driver Version Installed: 12080
3. GDS I/O performance (gdsio)
Even though gdscheck shows NVMe as unsupported, gdsio gives very high throughput (3.8 GiB/s using GPU memory):
IoType: WRITE XferType: GPUD Threads: 4
Throughput: 3.827373 GiB/sec
Avg_Latency: 4050 usecs
This suggests that cuFile is taking the GDS path successfully.
4. My questions
(1) Is GDS actually enabled/active on this system, despite NVMe: Unsupported in gdscheck?
(2) Does “NVMe: Unsupported” indicate lack of peer-to-peer DMA capability (P2PDMA), or that the NVMe driver didn’t register with nvidia-fs?
(3) What exactly does gdscheck require for “NVMe: Supported”?
Is it only dependent on platform topology (ACS/IOMMU), or also NVMe firmware/driver specifics?
(4) Since gdsio achieves high GPU-direct throughput, is it safe to assume that cuFile → nvidia-fs → NVMe direct path is active?
Happy to provide:
-
/proc/driver/nvidia-fs/devices -
/proc/driver/nvidia-fs/mounts -
PCIe topology
-
NVMe controller details (
lspci -vvv) -
Kernel logs
Thanks in advance for any clarification!