Difference between Device Memory Programming and Nvidia PeerDirect

[k.bodi2]

Hey there,

so we have either

• Device Memory Programming or
• Nvidia PeerDirect

but what’s the difference?

[k.bodi2]

Any help please?

[ipavis]

Hi There,
Thanks for you question

Device Memory Programming (DMP) and NVIDIA PeerDirect are both really nice technologies that help improve data transfer in high-performance computing. Let me break it down for you:

Device Memory Programming (DMP) is an API that allows applications to use on-chip memory (like a special memory area on your graphics card or accelerator) for faster data transfers. By doing this, it reduces the time it takes to send and receive data compared to using regular system memory.

On the other hand, NVIDIA PeerDirect is a technology developed by NVIDIA. It enables direct communication between devices (like GPUs or accelerators) using high-speed InfiniBand or RoCE connections. The best part is, it allows these devices to share computing power and use the interconnect at the same time, all without the need to copy data between them.

In simple terms, DMP helps reduce latency by using special on-chip memory, while PeerDirect lets devices directly access each other’s memory for faster communication. Both technologies make high-performance computing even more efficient and speedy!

Remember, DMP can work with various devices, while PeerDirect is specific to NVIDIA’s GPUs and accelerators. Hope this explanation helps you understand these awesome technologies better!

Thanks,
Ilan.

[k.bodi2]

Thank you for clarification! That was important to understand. From my understanding, GPUDirect RDMA is based on DMP - is this correct? If one wants to utilize DMP for other devices than a GPU, one would need a ConnectX-5 NIC at least, right? As stated in the MLX_OFED manuals.

[ipavis]

You’re welcome! I’m glad I could help clarify things for you.

Regarding GPUDirect RDMA and DMP:

GPUDirect RDMA and Device Memory Programming (DMP) are related but different technologies.

Regarding the ConnectX-5 NIC:

You are correct that some features and functionalities related to RDMA and GPU communication may depend on the hardware and network interconnect used.

As for DMP, the exact hardware requirements can vary depending on the implementation and vendor. In some cases, using DMP on devices other than GPUs may require specific network interface cards (NICs) with on-chip memory support, such as Mellanox ConnectX-5 NICs and above.

Thanks,
Ilan.