The relationship between GPUDirect RDMA, GPUDirect P2P, NVidia IPC, NCCL, and NVSHMEM

@Curefab Yes, I read that gds-nvidia-fs a month ago. Testing effort and quality assurance I am able to do myself + invite close friends for testing if needed. From what I understood, I don’t want RDMA. RDMA is only for Ethernet to GPU transfer using sockets (in your code you have UDP, TCPv4 or TCPv6). You know I want GPU to NVMe and you call this GPUDirect Storage while GPUDirect RDMA is completely different implementation.

You know a lot has been said already here. Just give me cuFile.cu and I show you how people with slight autism works :D. I think it is time to stop showing muscle and show real actions. That’s why I’m here :-)

READ HERE: Remote direct memory access - Wikipedia

RDMA is for remote DMA between TWO computers using ETHERNET!

I never mentioned this. I specifically require not using RDMA and not using TWO computers. I will create RAM DISK or use NVMe and PCI Express transport.

Yes, you have gds-nvidia-fs on Github, but I don’t need that, I would roll my own implementation. There seems to be a lot of GDS stuff on Github, you have even Enterprise NVIDIA driver for Quadro and RTX (4000’s) and here: RTX 4090 vs RTX 4000 [1-Benchmark Showdown].

My RTX 4090 for desktop outperforms your RTX 4000 for workstation by whooping 152% and there is no single parameter where Quadro RTX 4000 would be better than RTX 4090 (nothing regarding performance and nothing regarding API version).

It seems you have everything on Github, just one piece is missing: cuFile. It is like a fairy tale about lost treasure.

And on Download website for Enterprise GPU Driver you mention that your Enterprise driver has Windows support for GPUDirect

Then in release notes you mention TCC:

And that TCC does not support DirectX or OpenGL and it is, in industry we say “headless” driver rather that “non-display” driver. So basically I would not maybe need cuFile and I would need to create that Windows service you mention you already have and that Windows service would send CUDA API calls to Ubuntu in WSL2 using WSLg (like passthrough or direct api).

So while we are talking here, and it is fun, I enjoy it, not that I could say no :-) but I want to point out I came here with 5 different ways how to accomplish this task. Even that Microsoft released DeepSpeed compiled for Windows two days ago. Intel is also great company. Only you are currently lacking behind the market in this, but I’m not here to criticize, I’m hear to offer helping hand and solutions. But you need to stop worrying - I even don’t know what is it about - I offered to sign NDA and my work would be licensed to you if you want to be more protective about SDK.

Small rant I should do in the end: You mention you have like hundreds or thousands of expensive people focusing only on software. Well I’m also retrogamer! I have Geforce 256 DDR for playing Unreal on awesome computer with 21’’ EIZO CRT display and 3D glasses (in set with GF 256 by ASUS) and when you mention what you mention, I start thinking that that NVIDIA Control Panel did not change like…in…25 years??? Haha, that made my day, I was laughing here on the chair. You might be willing to answer that it is not true and now you have NVIDIA App. Well, but NVIDIA App cannot create custom resolution, cannot change colors from RGB 8bit to 422 10bit, which is often needed to be able to turn on HDR over HDMI. Yes, on display port you can have HDR even using RGB 8bit. But 10bit is often better for HDR. So my point is still valid in 2025.

Well, it is 4 AM in Europe! Good morning!

and have a nice day - maybe use text to speach (TTS) so you don’t need to read. I recommend Azure TTS with Chinese mainland Xiaoxiao - her english accent is very funny and tone exciting and agitating like cheerleading for whole community - it is funny to let her read scientific papers :D

As you answered to my message and addressed Nvidia with ‘you’: I am just a user trying to help.
So I am assuming the very first sentence is addressed to me, the rest of the post in general/to Nvidia.

Oh, I apologize. Does it mean I’m like talking with myself and NV actually ignores all of this :D ? You are from Germany. We live quite close, I’m from Prague / Ostrava.

As EU and German citizen, have you heard about this German law?

  • Many EU countries allow reverse engineering for academic purposes (e.g., Germany’s §69d UrhG).

    Under the EU Copyright Directive (2001/29/EC), reverse engineering for interoperability is permitted if:

      • Done independently (no direct code reference)
      • Necessary to achieve interoperability

        Digital Markets Act (DMA)

        EULA terms are unfair under:

        • EU Unfair Contract Terms Directive (93/13/EEC)

        • Art. 3(1): Terms causing “significant imbalance” to consumers

          Proposal of **European GPU Compute Act** mandating:
          - - Open driver interfaces
          - - Ban on anti-interoperability clauses