Direct GPU <-> GPU communication does not seem to work properly

There isn’t any way to know if you should have P2P capability, except for the tests provided by NVIDIA in the form of cudaDeviceCanAccessPeer().

And of course bugs are always possible. I don’t know what GPUs you have although I can see they are cc8.9 GPUs with probably 16GB. The lowest level cc8.9 enterprise GPU I am familiar with is the L4, which has 24GB of memory, so I imagine these are GeForce GPUs (RTX 40-series, Ada generation, RTX 4080 perhaps). In recent years NVIDIA does not support P2P on most GeForce GPUs that I am familiar with unless a NVLink bridge is installed. And not all GeForce GPUs support NVLink bridges.