NVIDIA GDS does not run PCIe P2P in xfer=0

Hi,

I’m trying to use Nvidia GDS to transfer data between an NVMe storage and an Nvidia GTX 3060, with no success.

My setup is:
PCIe RP ↔ PCIe switch ↔ Nvidia 3060
PCIe switch ↔ NVMe

My setup enumerates on PCIe under Linux, and I’m able to transfer data from the GPU to the storage, but it is always redirected via the PCIe root port. There is no direct route of data going from the GPU to the storage or vice versa. I’m using a PCIe protocol analyzer to confirm this.

I’m using the following manual:
NVIDIA GPUDirect Storage Benchmarking and Configuration Guide - NVIDIA Docs.

The command I’m using in gds is:
gdsio -D /mnt/test_dir -d 0 -w 1 -s 4K -i 4K -x 0 -I 1 -T 5

So far I’ve disabled ACS on PCIe and confirmed that IOMMU is disabled. Does anyone know what other configurations are required?

Hi @snotna ,

Could you paste the output of the following commands:

  1. /usr/local/cuda-12.x/gds/tools/gdscheck.py -p
  2. lsmod | grep nvidia to verify whether nvidia-fs is loaded. This would be needed for the direct path
  3. ls -l /dev/disk/by-path to identify nvme device’s pcie domain:bus:device.func
  4. lsblk and mount output to see how you are mounting the nvme to /mnt
  5. lspci -tv | egrep -i “nvidia | nvme” to see your PCIe tree in order to identify existence of direct path
  6. nvidia-smi
  7. lspci -D | grep -i nvidia
  8. lspci -D | grep -i nvme

Thanks,
Prashant

just noticed from your original question RTX 3060 is a consumer class GPU.
We only support GDS on data center models from Tesla and Quadro GPU family.

Hi,

I tried to re-run using a Quadro P400 GPU which supports CUDA compute engine 6.1 which supports CUDA SDK 12.x. Still I cannot see PCIe peer-to-peer traffic. All traffic is going through the RP.

I am aligned to the versions mentioned here:

Perhaps I need a different device?

Hi,
GDS is supported for Quadro GPU family. However, for the RP issue, there could be several other factors which can cause this. It would be helpful if you provide the details which prashantp requested earlier about your configuration.