# Direct GPU \<-\> GPU communication does not seem to work properly

**URL:** <https://forums.developer.nvidia.com/t/direct-gpu-gpu-communication-does-not-seem-to-work-properly/283264>\
**Category:** CUDA Programming and Performance\
**Created:** [February 19, 2024, 2:25pm UTC](https://forums.developer.nvidia.com/t/direct-gpu-gpu-communication-does-not-seem-to-work-properly/283264 "2024-02-19T14:25:07Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![morgangiraud](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/morgangiraud/32/40828_2.png) [@morgangiraud](https://forums.developer.nvidia.com/u/morgangiraud)\
**Post date:** [February 19, 2024, 2:25pm UTC](https://forums.developer.nvidia.com/t/direct-gpu-gpu-communication-does-not-seem-to-work-properly/283264/1 "2024-02-19T14:25:07Z")

</div>

Hello there!

I’m reaching out because I’m having trouble understanding if I should or should not have P2P capabilities in my system.  
I’m coming at this from a pytorch bug where you can find more detailled discussion there:

- [Direct P2P GPU \<-\> GPU communication with torch.to does not seem to work - PyTorch Forums](https://discuss.pytorch.org/t/direct-p2p-gpu-gpu-communication-with-torch-to-does-not-seem-to-work/197145)
- [Direct P2P GPU \<-\> GPU communication with `torch.to` does not seem to work. · Issue #119638 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/issues/119638)

Summary so far:

- Disabling ACS and IOMMU has no impact
- With nvidia-driver `535.*` and `550.*` I don’t have any P2P capabilities
- With nvidia-driver `545.*`, I do have P2P capabilities and the nvidia P2P script is indeed improving the bandwidth
- NCCL tests from [GitHub - NVIDIA/nccl-tests: NCCL Tests](https://github.com/nvidia/nccl-tests) are hanging with driver `545.*` and are working when I disable P2P for NCCL with `NCCL_P2P_DISABLE=1`
- Last test, I used [GitHub - NVIDIA/nvbandwidth: A tool for bandwidth measurements on NVIDIA GPUs.](https://github.com/NVIDIA/nvbandwidth) with driver `545.*` and I got the following error during the verification step:

```auto
Running device_to_device_memcpy_read_ce.
 Invalid value when checking the pattern at <0x7fefec000000>
 Current offset [0/67108864]

```

**So the data is not copied properly but driver `545.*` still thinks it can do it.**

Do you know how to theoretically know if I should have P2P capabilities or not? Did I miss anything obvious to test?

---

<div class="post-metadata">

**Author:** ![Robert\_Crovella](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/robert_crovella/32/14043_2.png) [@Robert\_Crovella](https://forums.developer.nvidia.com/u/Robert_Crovella)\
**Post date:** [February 19, 2024, 5:23pm UTC](https://forums.developer.nvidia.com/t/direct-gpu-gpu-communication-does-not-seem-to-work-properly/283264/2 "2024-02-19T17:23:45Z")

</div>

There isn’t any way to know if you should have P2P capability, except for the tests provided by NVIDIA in the form of `cudaDeviceCanAccessPeer()`.

And of course bugs are always possible. I don’t know what GPUs you have although I can see they are cc8.9 GPUs with probably 16GB. The lowest level cc8.9 enterprise GPU I am familiar with is the L4, which has 24GB of memory, so I imagine these are GeForce GPUs (RTX 40-series, Ada generation, RTX 4080 perhaps). In recent years [NVIDIA does not support P2P on most GeForce GPUs that I am familiar with](https://forums.developer.nvidia.com/t/standard-nvidia-cuda-tests-fail-with-dual-rtx-4090-linux-box/233202/28) unless a NVLink bridge is installed. And not all GeForce GPUs support NVLink bridges.

---

<div class="post-metadata">

**Author:** ![rs277](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@rs277](https://forums.developer.nvidia.com/u/rs277)\
**Post date:** [February 19, 2024, 6:02pm UTC](https://forums.developer.nvidia.com/t/direct-gpu-gpu-communication-does-not-seem-to-work-properly/283264/3 "2024-02-19T18:02:58Z")

</div>

From the OP’s first link:  
`_CudaDeviceProperties(name='NVIDIA Graphics Device', major=8, minor=9, total_memory=15868MB, multi_processor_count=66)`

So possibly RTX 4070 Ti [Super.](https://www.techpowerup.com/gpu-specs/geforce-rtx-4070-ti-super.c4187)

---

<div class="post-metadata">

**Author:** ![Robert\_Crovella](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/robert_crovella/32/14043_2.png) [@Robert\_Crovella](https://forums.developer.nvidia.com/u/Robert_Crovella)\
**Post date:** [February 19, 2024, 6:11pm UTC](https://forums.developer.nvidia.com/t/direct-gpu-gpu-communication-does-not-seem-to-work-properly/283264/4 "2024-02-19T18:11:29Z")

</div>

Yep, agreed, based on the SM count.

---

<div class="post-metadata">

**Author:** ![morgangiraud](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/morgangiraud/32/40828_2.png) [@morgangiraud](https://forums.developer.nvidia.com/u/morgangiraud)\
**Post date:** [February 19, 2024, 6:30pm UTC](https://forums.developer.nvidia.com/t/direct-gpu-gpu-communication-does-not-seem-to-work-properly/283264/5 "2024-02-19T18:30:49Z")

</div>

Hi, thanks for taking the time to answer.  
I’ve been wondering this past week if the problem was coming from the mobo/pcie config or the drivers.

Indeed, the setup is:

- 2 GPU geforce RTX 4070 TI SUPER 16Go
- Motherboard MSI X670e Carbon Wifi
- CPU AMD ryzen 7900x

So based on your input, it seems that there is a bug in driver `545.*` which makes the GPUs believe they can do PHB P2P when they actually can’t which explains why everything is failing when I have this driver installed.  
Should I report this bug somewhere?

---

<div class="post-metadata">

**Author:** ![morgangiraud](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/morgangiraud/32/40828_2.png) [@morgangiraud](https://forums.developer.nvidia.com/u/morgangiraud)\
**Post date:** [February 19, 2024, 6:35pm UTC](https://forums.developer.nvidia.com/t/direct-gpu-gpu-communication-does-not-seem-to-work-properly/283264/6 "2024-02-19T18:35:31Z")

</div>

Also, do you agree that the solution is then to update my drivers to `550.*` to fix this issue if I don’t want to wait for a new patch on the `545.*` version?

---

<div class="post-metadata">

**Author:** ![Robert\_Crovella](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/robert_crovella/32/14043_2.png) [@Robert\_Crovella](https://forums.developer.nvidia.com/u/Robert_Crovella)\
**Post date:** [February 20, 2024, 5:05pm UTC](https://forums.developer.nvidia.com/t/direct-gpu-gpu-communication-does-not-seem-to-work-properly/283264/7 "2024-02-20T17:05:29Z")

</div>

> [@morgangiraud](#):
>
> Should I report this bug somewhere?

I suspect it is probably already fixed if the R550 drivers do not show the issue. It’s possible that a later R545 driver also has the fix.

Anyone can file a bug if they wish at any time. The instructions are linked to a sticky post at the top of this sub-forum.

Yes, if that were my system, I would use the R550 driver.

---

<div class="post-metadata">

**Author:** ![morgangiraud](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/morgangiraud/32/40828_2.png) [@morgangiraud](https://forums.developer.nvidia.com/u/morgangiraud)\
**Post date:** [February 20, 2024, 5:54pm UTC](https://forums.developer.nvidia.com/t/direct-gpu-gpu-communication-does-not-seem-to-work-properly/283264/8 "2024-02-20T17:54:20Z")

</div>

Perfect. Thanks for your help!

---

<div class="post-metadata">

**Author:** ![Yuki\_Ni](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/yuki_ni/32/14047_2.png) [@Yuki\_Ni](https://forums.developer.nvidia.com/u/Yuki_Ni)\
**Post date:** [February 22, 2024, 2:46pm UTC](https://forums.developer.nvidia.com/t/direct-gpu-gpu-communication-does-not-seem-to-work-properly/283264/10 "2024-02-22T14:46:02Z")

</div>

[Public] Hi Morgan ,

Thanks for filing a bug ticket . This is a known issue to us . Your observation is correct that the P2P report was once broken on all R545 series . R535 , R550 and go on for later releases , it is fixed .  
We feel sorry for the breakage . But we have no more plan to backport fixes to R545 , please move forward with R550 drivers .

Hope this helps.

Best,  
Yuki

---

<div class="post-metadata">

**Author:** ![system](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/system/32/68080_2.png) [@system](https://forums.developer.nvidia.com/u/system)\
**Post date:** [March 7, 2024, 2:46pm UTC](https://forums.developer.nvidia.com/t/direct-gpu-gpu-communication-does-not-seem-to-work-properly/283264/11 "2024-03-07T14:46:11Z")

</div>

Closed on March 7, 2024, 2:46pm UTC

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.
