GDSIO Not working with T600 and samsung NVME

Infrastructure & Networking DOCA Storage
1

Hi I am trying to run GDSIO to evaluate GPUDirect Storage(GDS)
Below are my config/setup

Hardware

NVIDIA TU117GL/T600 ( 7.5 Compute capability)
NVME -
Both on same NUMA Node.

nvidia-smi topo -nvme
NVMe0
GPU0 NODE

Software

Driver Version: 580.126.09 CUDA Version: 13.0
nvidia_fs v2.27.3 - ( though documentation says this is not required after cuda 12.8 and WITH P2pdma support.)

System/kernel

vanilla kernel 6.12.0
cat /proc/kallsyms | grep -i p2pdma_pgmap_ops
0000000000000000 d p2pdma_pgmap_ops
ls /sys/kernel/debug/dma_buf/–this is present.
iommu and pt disabled.

Test 1

 // allow compat mode, this will enable use of cuFile posix read/writes
                            "allow_compat_mode": false,
 // allow p2pdma, this will enable use of cuFile without nvme patches
                            "use_pci_p2pdma": true,
//Allow NVMe to use p2pdma mode
                            "use_pci_p2pdma" : true

 GDS release version: 1.16.1.26
 nvidia_fs version:  2.27 nvidia_fs minimum version: 2.12
 Platform: x86_64
 ============
 ENVIRONMENT:
 ============
 =====================
 DRIVER CONFIGURATION:
 =====================
 NVMe               : Unsupported
 NVMeOF             : Unsupported
 SCSI               : Unsupported
 ScaleFlux CSD      : Unsupported
 NVMesh             : Unsupported
 DDN EXAScaler      : Unsupported
 IBM Spectrum Scale : Unsupported
 NFS                : Unsupported
 BeeGFS             : Unsupported
 ScaTeFS            : Unsupported
 VIRTIOFS           : Unsupported
 WekaFS             : Unsupported
 Userspace RDMA     : Unsupported
 --Mellanox PeerDirect : Disabled
 --rdma library        : Not Loaded (libcufile_rdma.so)
 --rdma devices        : Not configured
 --rdma_device_status  : Up: 0 Down: 0
 =====================
 CUFILE CONFIGURATION:
 =====================
 properties.use_pci_p2pdma : true
 properties.use_compat_mode : false
 properties.force_compat_mode : false
 properties.gds_rdma_write_support : true
 properties.use_poll_mode : false
 properties.poll_mode_max_size_kb : 4
 properties.max_batch_io_size : 128
 properties.max_batch_io_timeout_msecs : 5
 properties.max_direct_io_size_kb : 16384
 properties.max_device_cache_size_kb : 131072
 properties.per_buffer_cache_size_kb : 1024
 properties.max_device_pinned_mem_size_kb : 33554432
 properties.posix_pool_slab_size_kb : 4 1024 16384
 properties.posix_pool_slab_count : 128 64 64
 properties.rdma_peer_affinity_policy : RoundRobin
 properties.rdma_dynamic_routing : 0
 fs.generic.posix_unaligned_writes : false
 fs.lustre.posix_gds_min_kb: 0
 fs.lustre.use_pci_p2pdma : false
 fs.beegfs.posix_gds_min_kb: 0
 fs.beegfs.use_pci_p2pdma : false
 fs.scatefs.posix_gds_min_kb: 0
 fs.scatefs.use_pci_p2pdma : false
 fs.weka.rdma_write_support: false
 fs.nfs.use_pci_p2pdma : false
 fs.gpfs.gds_write_support: false
 fs.gpfs.gds_async_support: true
 fs.gpfs.use_pci_p2pdma : false
 fs.virtiofs.use_pci_p2pdma : false
 profile.cufile_stats : 0
 miscellaneous.api_check_aggressive : false
 execution.max_io_threads : 4
 execution.max_io_queue_depth : 128
 execution.parallel_io : true
 execution.min_io_threshold_size_kb : 8192
 execution.max_request_parallelism : 4
 properties.force_odirect_mode : false
 properties.prefer_iouring : false
 block.raid.use_pci_p2pdma : false
 block.nvme.use_pci_p2pdma : true
 block.nvmeof.use_pci_p2pdma : false
 =========
 GPU INFO:
 =========
 GPU index 0 NVIDIA T600 bar:1 bar size (MiB):256 supports GDS, IOMMU State: Disabled
 ==============
 PLATFORM INFO:
 ==============
 IOMMU: disabled
 Nvidia Driver Info Status: Supported(Nvidia Open Driver Installed)
 Cuda Driver Version Installed:  13000
 Platform: PowerEdge R740, Arch: x86_64(Linux 6.12.0)
 Platform verification succeeded

when i run gdsio it fails.

/usr/local/cuda/gds/tools/gdsio -x 0 -d 0 -s 8K -i 4K -f /nvme/gds_test -I 1
file register error: GPUDirect Storage not supported on current file filename :/nvme/gds_test

Test 2

all same config but with compat mode

  // allow compat mode, this will enable use of cuFile posix read/writes
                            "allow_compat_mode": true,

/usr/local/cuda/gds/tools/gdscheck.py -p
 GDS release version: 1.16.1.26
 nvidia_fs version:  2.27 nvidia_fs minimum version: 2.12
 Platform: x86_64
 ============
 ENVIRONMENT:
 ============
 =====================
 DRIVER CONFIGURATION:
 =====================
 NVMe               : compat
 NVMeOF             : compat
 SCSI               : compat
 ScaleFlux CSD      : compat
 NVMesh             : compat
 DDN EXAScaler      : compat
 IBM Spectrum Scale : compat
 NFS                : compat
 BeeGFS             : compat
 ScaTeFS            : compat
 VIRTIOFS           : compat
 WekaFS             : compat
 Userspace RDMA     : Unsupported
 --Mellanox PeerDirect : Disabled
 --rdma library        : Not Loaded (libcufile_rdma.so)
 --rdma devices        : Not configured
 --rdma_device_status  : Up: 0 Down: 0
 =====================
 CUFILE CONFIGURATION:
 =====================
 properties.use_pci_p2pdma : true
 properties.use_compat_mode : true
 properties.force_compat_mode : false
 properties.gds_rdma_write_support : true
 properties.use_poll_mode : false
 properties.poll_mode_max_size_kb : 4
 properties.max_batch_io_size : 128
 properties.max_batch_io_timeout_msecs : 5
 properties.max_direct_io_size_kb : 16384
 properties.max_device_cache_size_kb : 131072
 properties.per_buffer_cache_size_kb : 1024
 properties.max_device_pinned_mem_size_kb : 33554432
 properties.posix_pool_slab_size_kb : 4 1024 16384
 properties.posix_pool_slab_count : 128 64 64
 properties.rdma_peer_affinity_policy : RoundRobin
 properties.rdma_dynamic_routing : 0
 fs.generic.posix_unaligned_writes : false
 fs.lustre.posix_gds_min_kb: 0
 fs.lustre.use_pci_p2pdma : false
 fs.beegfs.posix_gds_min_kb: 0
 fs.beegfs.use_pci_p2pdma : false
 fs.scatefs.posix_gds_min_kb: 0
 fs.scatefs.use_pci_p2pdma : false
 fs.weka.rdma_write_support: false
 fs.nfs.use_pci_p2pdma : false
 fs.gpfs.gds_write_support: false
 fs.gpfs.gds_async_support: true
 fs.gpfs.use_pci_p2pdma : false
 fs.virtiofs.use_pci_p2pdma : false
 profile.cufile_stats : 0
 miscellaneous.api_check_aggressive : false
 execution.max_io_threads : 4
 execution.max_io_queue_depth : 128
 execution.parallel_io : true
 execution.min_io_threshold_size_kb : 8192
 execution.max_request_parallelism : 4
 properties.force_odirect_mode : false
 properties.prefer_iouring : false
 block.raid.use_pci_p2pdma : false
 block.nvme.use_pci_p2pdma : true
 block.nvmeof.use_pci_p2pdma : false
 =========
 GPU INFO:
 =========
 GPU index 0 NVIDIA T600 bar:1 bar size (MiB):256 supports GDS, IOMMU State: Disabled
 ==============
 PLATFORM INFO:
 ==============
 IOMMU: disabled
 Nvidia Driver Info Status: Supported(Nvidia Open Driver Installed)
 Cuda Driver Version Installed:  13000
 Platform: PowerEdge R740, Arch: x86_64(Linux 6.12.0)
 Platform verification succeeded

/usr/local/cuda/gds/tools/gdsio -x 0 -d 0 -s 8K -i 4K -f /nvme/gds_test -I 1
IoType: WRITE XferType: GPUD Threads: 1 DataSetSize: 8/8(KiB) IOSize: 4(KiB) Throughput: 0.010915 GiB/sec, Avg_Latency: 59.500000 usecs ops: 2 total_time 0.000699 secs**
**
How can i make non compat ( true gpudirect to work ) with changes in above config?

2

cat cufile.log
25-01-2026 17:43:16:161 [pid=3801 tid=3801] ERROR cufio-fs:79 mount option not found in mount table data device: /dev/nvme0n1p1
25-01-2026 17:43:16:161 [pid=3801 tid=3801] NOTICE cufio-fs:484 dumping volume attributes: DEVNAME:/dev/nvme0n1p1,ID_FS_TYPE:ext4,ID_FS_USAGE:filesystem,UDEV_PCI_BRIDGE:0000:85:02.0,device/transport:pcie,fsid:268f665d427e66f00x,numa_node:1,queue/logical_block_size:4096,wwid:eui.0025385721b1384f,
25-01-2026 17:43:16:161 [pid=3801 tid=3801] ERROR cufio:293 cuFileHandleRegister error, file checks failed
25-01-2026 17:43:16:161 [pid=3801 tid=3801] ERROR cufio:337 cuFileHandleRegister error: GPUDirect Storage not supported on current file

3

Few more logs with debug enabled
echo 1 > /sys/module/nvidia_fs/parameters/dbg_enabled ; CUFILE_LOGGING_LEVEL=TRACE /usr/local/cuda/gds/tools/gdsio -x 0 -d 0 -s 8K -i 4K -f /tmp/tmp1/gds_test -I 1

cat /proc/driver/nvidia-fs/stats
GDS Version: 1.16.0.49
NVFS statistics(ver: 4.0)
NVFS Driver(version: 2.27.3)
Mellanox PeerDirect Supported: False
IO stats: Disabled, peer IO stats: Disabled
Logging level: debug

Active Shadow-Buffer (MiB): 0
Active Process: 0
Reads : err=0 io_state_err=0
Sparse Reads : n=0 io=0 holes=0 pages=0
Writes : err=0 io_state_err=0 pg-cache=0 pg-cache-fail=0 pg-cache-eio=0
Mmap : n=8 ok=8 err=0 munmap=8
Bar1-map : n=8 ok=8 err=0 free=8 callbacks=0 active=0 delay-frees=0
Error : cpu-gpu-pages=0 sg-ext=0 dma-map=0 dma-ref=0
Ops : Read=0 Write=0 BatchIO=0
GPU 0000:d8:00.0 uuid:1f2e9cf0-1b12-8a03-0857-bb6c6ebf8ba3 : Registered_MiB=0 Cache_MiB=0 max_pinned_MiB=0

Important lines from cufile.log
26-01-2026 07:10:56:621 [pid=5231 tid=5231] TRACE cufio-cuda:663 Allocated 65536 bytes on the GPU: 0 to check for P2PDMA Support
26-01-2026 07:10:56:621 [pid=5231 tid=5231] INFO cufio-cuda:669 Failed to get cuda p2p device address for ptr 0x7f593f000000 length: 65536 errornum: 801
26-01-2026 07:10:56:731 [pid=5231 tid=5231] DEBUG cufio-drv:102 gpu attribute pci_p2pdma not supported errornum: 801 error: CUDA_ERROR_NOT_SUPPORTED
26-01-2026 07:10:56:731 [pid=5231 tid=5231] INFO cufio-drv:132 Is PCIP2PDMA Supported: 1 kernel Support: 1 pci_p2pdma_supported_cuda: 1 checkIfAllGPUsSupportP2PDMA(): 0

cufile_debug.txt (100.3 KB)

dmesg.txt (2.6 KB)

4

Few things i also need clarification
1.BAR1 Size for GDS to work?
2.is DOCA-OFED/MOFED Necessary for GDS ( Installation using sudo ./mlnxofedinstall --with-nvmf --with-nfsrdma --enable-gds --add-kernel-support --dkms), since i am suing 6.12 with P2PDMA and DMABUF
i suppose its not required?
3.same as above i dont need to patch nvme kernel module right?
4.

5

I was able to run Same in H200 But it required nvidia driver to be loaded with

options nvidia NVreg_RegistryDwords="RMForceStaticBar1=1;RmForceDisableIomapWC=1;"

and also nvme drive to remounter in data=ordered

/usr/local/cuda/gds/tools/gdsio -x 0 -d 0 -s 8K -i 4K -f /boot/gds_test -I 1
IoType: WRITE XferType: GPUD Threads: 1 DataSetSize: 8/8(KiB) IOSize: 4(KiB) Throughput: 0.028468 GiB/sec, Avg_Latency: 39.000000 usecs ops: 2 total_time 0.000268 secs

So question does this kernel module parameter supported by T600 , if not what is the workaround with P2PDMA without using nvidia-fs