Difference between NVME vs NVME P2PDMA

Infrastructure & Networking DOCA Storage
1

When using gdscheck -p

 GDS release version: 1.13.0.11                                                                                                                                                                                    
 nvidia_fs version:  2.24 libcufile version: 2.12                                                                                                                                                                  
 Platform: x86_64                                                                                                                                                                                                  
 ============                                                                                                                                                                                                      
 ENVIRONMENT:                                                                                                                                                                                                      
 ============                                                                                                                                                                                                      
 =====================                                                                                                                                                                                             
 DRIVER CONFIGURATION:                                                                                                                                                                                             
 =====================                                                                                                                                                                                             
 NVMe P2PDMA        : Unsupported                                                                                                                                                                                  
 NVMe               : Supported

What’s the difference between those two? Isn’t GDS supposed to use P2P DMA for direct transfer to and from GPU and NVME?

I tried looking at the docs but couldn’t find the difference.
Can anyone please explain this to me?

Thanks.

1 Like
2

I also find it confused. But I guess P2PDMA is a term from Linux Kernel: PCI Peer-to-Peer DMA Support — The Linux Kernel documentation

1 Like
3

Yes maybe.
From here

As of CUDA 12.8, GDS supports additional peer-to-peer DMA for NVMe devices using upstream kernel PCI P2PDMA infrastructure for x86_64 platforms (local or remote with SNAP). This feature will no longer depend on nvidia-fs.ko or custom patches to nvme.ko.

This is the only relevant information I can see. Maybe NVMe P2PDMA means it uses Linux’s PCI P2P calls?

P.S. Does anyone know how to inform Nvidia that the linked doc doesn’t actually link to the guide in the paragraph?

NVMe Multipathing are currently not enabled with PCI P2PDMA in the upstream kernel and will not work without a specialized patch. See <which guide> for configuration details
4

If you only want transfer data between GPU and NVMe, your system should work as NVMe : Supported as you can check gdsio for basic io cases.

I understand the NVMe P2PDMA is only a new feature to workwith copying data between multiple NVMe drives. So more like an advanced case.

1 Like
5

Ah!
Multiple drives, can you link to doc that refer this.
I’m a student learning about this tech, so just digging around.

6

Was my guess only. The term P2PDMA is somewhere else: https://www.snia.org/sites/default/files/SDC/2018/presentations/Storage_Architecture/Bates_Stephen_Accelerating_Storage_with_NVM_Express_SSDs_and_P2PDMA.pdf

1 Like
7

Yes ‘NVME P2PDMA’ in gdscheck refers to using upstream PCI P2PDMA for x86_64 and GraceHopper based platforms.

Thanks for pointing out the missing link in the documentation. we will update shortly.

1 Like