Nvme unsupported from "gdscheck" in GPU direct storage

Hi all,

I am using cuda-11.4 and trying to use GDS (GPU direct storage) 1.0.1.3.

I wanted to use nvme device, but when I run gdscheck.py, I see that nvme says unsupported.
Also when I run the test code, it shows this log.

$  sudo   ./gdsio -f /mnt/nvme/blob -d 0 -n 0 -w 2 -s 1G -i 4K -x 0 -I  1
 

07-10-2021 12:35:54:807 [pid=355177 tid=355177] ERROR  cufio-topo-nvfs:72 pci device not present in topology device attribute table: 0000:75:00.0
 07-10-2021 12:35:54:808 [pid=355177 tid=355177] ERROR  cufio-topo-nvfs:72 pci device not present in topology device attribute table: 0000:75:00.0
 07-10-2021 12:35:54:818 [pid=355177 tid=355177] ERROR  cufio-fs:192 NVMe Driver not registered with nvidia-fs!!!
 07-10-2021 12:35:54:819 [pid=355177 tid=355177] ERROR  cufio-fs:788 block device not supported by cufile, bdev : /dev/nvme0n1p1

Could anyone help how should I debug this?

:/usr/local/cuda-11.4/gds$ sudo ./tools/gdscheck.py -p
 GDS release version: 1.0.1.3
 nvidia_fs version:  2.7 libcufile version: 2.4
 ============
 ENVIRONMENT:
 ============
 =====================
 DRIVER CONFIGURATION:
 =====================
 NVMe               : Unsupported
 NVMeOF             : Unsupported
 SCSI               : Unsupported
 ScaleFlux CSD      : Unsupported
 NVMesh             : Unsupported
 DDN EXAScaler      : Unsupported
 IBM Spectrum Scale : Unsupported
 NFS                : Unsupported
 WekaFS             : Unsupported
 Userspace RDMA     : Unsupported
 --Mellanox PeerDirect : Disabled
 --rdma library        : Not Loaded (libcufile_rdma.so)
 --rdma devices        : Not configured
 --rdma_device_status  : Up: 0 Down: 0
 =====================
 CUFILE CONFIGURATION:
 =====================
 properties.use_compat_mode : true
 properties.gds_rdma_write_support : true
 properties.use_poll_mode : false
 properties.poll_mode_max_size_kb : 4
 properties.max_batch_io_timeout_msecs : 5
 properties.max_direct_io_size_kb : 16384
 properties.max_device_cache_size_kb : 131072
 properties.max_device_pinned_mem_size_kb : 33554432
 properties.posix_pool_slab_size_kb : 4 1024 16384 
 properties.posix_pool_slab_count : 128 64 32 
 properties.rdma_peer_affinity_policy : RoundRobin
 properties.rdma_dynamic_routing : 0
 fs.generic.posix_unaligned_writes : false
 fs.lustre.posix_gds_min_kb: 0
 fs.weka.rdma_write_support: false
 profile.nvtx : false
 profile.cufile_stats : 0
 miscellaneous.api_check_aggressive : false
 =========
 GPU INFO:
 =========
 GPU index 0 NVIDIA A10 bar:1 bar size (MiB):32768 supports GDS
 GPU index 1 NVIDIA A10 bar:1 bar size (MiB):32768 supports GDS
 GPU index 2 NVIDIA A10 bar:1 bar size (MiB):32768 supports GDS
 GPU index 3 NVIDIA A10 bar:1 bar size (MiB):32768 supports GDS
 ==============
 PLATFORM INFO:
 ==============
 IOMMU: disabled
 Platform verification succeeded



$ lsmod  | grep nvme
nvmet                  86016  0
nvme_tcp               32768  0
nvme_fabrics           24576  1 nvme_tcp
nvme                   49152  0
nvme_core             102400  3 nvme_tcp,nvme,nvme_fabrics