NVLink and Quadro RTX 5000

Graphics / Linux Linux
1

Hello, I am working on a machine running Ubuntu 20.04, that has 2 Quadro RTX 5000 GPUs. I have connected these cards with a 3-slot NVLink and am trying to figure out whether it is working. According to what is written here:

https://www.nvidia.com/content/dam/en-zz/Solutions/design-visualization/quadro-product-literature/176-0356-000-q8-nvlink-qsg-90x75mm-r6.pdf

"Note: To check the status of your NVLink for each of the two GPUs, use the following commands:

nvidia-smi nvlink -i 0 -s
nvidia-smi nvlink -i 1 -s

If configured correctly, the output should be the same for both GPUs:

Link 0: active
Link 1: active"

However, in my case I instead get the following:

nvidia-smi nvlink -i 0 -s
GPU 0: Quadro RTX 5000 (UUID: GPU-51f3288e-0275-ec17-9b0e-85e073c1c0ee)
Link 0: 25.781 GB/s

nvidia-smi nvlink -i 1 -s
GPU 1: Quadro RTX 5000 (UUID: GPU-ddfb59b1-02b4-6723-8ece-82f921b41988)
Link 0: 25.781 GB/s

Given that the above values match the expected NVLink 2.0 bandwidth for my card (NVLink - Wikipedia), I would be inclined to assume that the link is working, but can you please confirm?

Also, why am I getting “Link 0” for both cards, instead of “Link 0” and “Link 1” as in your PDF?

Finally, is there any linux executable to test NVLink installation/performance?

Thanks and best regards,

Luca

2

The rtx 5000 only has one sublink, so the output is correct. e.g. the rtx 6000 has two links, the rtx a6000 four.
nvidia-smi nvlink -s already is the installation test, if you want some other performance test, you could use the p2p samples from the cuda samples: https://docs.nvidia.com/cuda/cuda-samples/index.html

1 Like
3

Thanks - from p2pBandwidthLatencyTest I get:

[P2P (Peer-to-Peer) GPU Bandwidth Latency Test]
Device: 0, Quadro RTX 5000, pciBusID: 4f, pciDeviceID: 0, pciDomainID:0
Device: 1, Quadro RTX 5000, pciBusID: bd, pciDeviceID: 0, pciDomainID:0
Device=0 CAN Access Peer Device=1
Device=1 CAN Access Peer Device=0
(…)

so indeed it looks like the two GPUs are seeing each other!

1 Like
4

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.