No P2P - Dual NVlinked RTX 2080 TI setup on HP workstation - NVLink not working, no SLI option in control panel

Accelerated Computing CUDA CUDA Setup and Installation
1

We have some HP G4 Z4 Core i9 7900X workstations. These are fine since they allow compact dual slot dual gpu setup plus additional pcie 8x slot for special purpose hardware. We require this for some demo setups.

Equipped them with 2x slim RTX 2080 Ti from Gigabyte plus Quadro RTX HB nvlink bridge. Buying Quadro RTX is not necessary and quite price intensive since Geforce fulfils our requirements.

Both Geforce RTX are X16 PCIEv3 connected and individuially work fine with CUDA.

System is certified by HP for Quadro SLI, but it is not mentioned that Geforce would not work… Geforce SLI cannot be enabled / does not show up in the NVIDIA Control panel.

Additionally there comes the missing P2P over PCIE on RTX cards into place which is only available with NVLINK!!! But since NVlink cannot be activated, xfer between the cards is incedibly slow.

With nvidia-smi the links are shown as enabled and even SLI capable - see info below.

So I thinke there is no technical reason refusing this option. Is there any beta driver or tweak available just to enable this upon our own risk?

Thanks, in hope of a solution,
Enrico

C:\Program Files\NVIDIA Corporation\NVSMI>nvidia-smi.exe nvlink -s
GPU 0: GeForce RTX 2080 Ti (UUID: GPU-e91516bc-6040-c0fd-e07e-50a1b08f12fd)
Link 0: 25.781 GB/s
Link 1: 25.781 GB/s
GPU 1: GeForce RTX 2080 Ti (UUID: GPU-6396582e-aeb1-a471-068c-07d510b519be)
Link 0: 25.781 GB/s
Link 1: 25.781 GB/s

C:\Program Files\NVIDIA Corporation\NVSMI>nvidia-smi.exe nvlink -c
GPU 0: GeForce RTX 2080 Ti (UUID: GPU-e91516bc-6040-c0fd-e07e-50a1b08f12fd)
Link 0, P2P is supported: true
Link 0, Access to system memory supported: true
Link 0, P2P atomics supported: true
Link 0, System memory atomics supported: true
Link 0, SLI is supported: true
Link 0, Link is supported: false
Link 1, P2P is supported: true
Link 1, Access to system memory supported: true
Link 1, P2P atomics supported: true
Link 1, System memory atomics supported: true
Link 1, SLI is supported: true
Link 1, Link is supported: false
GPU 1: GeForce RTX 2080 Ti (UUID: GPU-6396582e-aeb1-a471-068c-07d510b519be)
Link 0, P2P is supported: true
Link 0, Access to system memory supported: true
Link 0, P2P atomics supported: true
Link 0, System memory atomics supported: true
Link 0, SLI is supported: true
Link 0, Link is supported: false
Link 1, P2P is supported: true
Link 1, Access to system memory supported: true
Link 1, P2P atomics supported: true
Link 1, System memory atomics supported: true
Link 1, SLI is supported: true
Link 1, Link is supported: false

2

You mention a Quadro NVLink bridge with GeForce products.

NVLink bridges are not generally interchangeable.

(There are actually several different kinds of Quadro RTX NVLink bridge, so not sure which you have.)

What I can say is that if I were trying to set this up with two 2080Ti, I would purchase the bridge that is designed to work with 2080/2080Ti

Of course I cannot guarantee that you don’t have some other issue with your setup.

3

Basically you are right, but there are no 2 slot bridges available for Geforce. So i followed the test on this website:

Here the Quadro RTX 6000 HB bridge was tested as compatible - since Quadro RTX 6000 and Geforce RTX 2080Ti are somewhat very similar re interfaces and hardware, possibly it may be even the same GPU chip but differently enabled (both based on TU-102). So we bought this bridge and also tested this in a a custom build Asus ROG X299 system here to verify - there the setup works flawlessly with SLI.

But my point is why this does not work on the HP Z4 G4 system. My thinking is that there it needs just some “enabling” by Nvidia-driver etc. The Workstation bios does not offer such an option to allow SLI, but this may not be required since Quadro SLI with RTX 6000 should be available by default.

The nvidia-smi.exe outputs show basic opreration of the link, but the option to enable it in the driver is just missing.

So what do you think?

Thanks,
Enrico (SeeReal Technologies)

4

Certainly I would make sure to try the latest driver for your GPU, and also double-check the device manager to make sure there are no issues reported.

Other than that I can’t explain it.

5

Driver is latest, even tried the latest gaming and studio driver. Both cards are shown in nvidia control panel and device manager. Both cards can be individually driven by cuda. But no P2P functionality.

Below is the output of P2P cuda test example. You see the cards are both working but not peering.

It seems that I’m not alone with this issue, in HP forums similar issues were mentioned that Quadro works but Geforce not. But I dont understand the point why this is like this.

I would really appreciate it, when this could be discussed with the driver team. If you need some more info about the system please tell me. Technically I’m sure there is all fine.

Thanks in advance, Enrico

test results:

[P2P (Peer-to-Peer) GPU Bandwidth Latency Test]
Device: 0, GeForce RTX 2080 Ti, pciBusID: 15, pciDeviceID: 0, pciDomainID:0
Device: 1, GeForce RTX 2080 Ti, pciBusID: 21, pciDeviceID: 0, pciDomainID:0
Device=0 CANNOT Access Peer Device=1
Device=1 CANNOT Access Peer Device=0

***NOTE: In case a device doesn’t have P2P access to other one, it falls back to normal memcopy procedure.
So you can see lesser Bandwidth (GB/s) and unstable Latency (us) in those cases.

P2P Connectivity Matrix
D\D 0 1
0 1 0
1 0 1
Unidirectional P2P=Disabled Bandwidth Matrix (GB/s)
D\D 0 1
0 520.75 8.62
1 8.91 519.45
Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)
D\D 0 1
0 519.00 8.89
1 8.98 519.51
Bidirectional P2P=Disabled Bandwidth Matrix (GB/s)
D\D 0 1
0 430.20 14.66
1 15.00 530.74
Bidirectional P2P=Enabled Bandwidth Matrix (GB/s)
D\D 0 1
0 512.52 14.22
1 15.07 529.37
P2P=Disabled Latency Matrix (us)
GPU 0 1
0 2.96 59.67
1 60.79 3.36

CPU 0 1
0 2.48 54.92
1 45.44 2.15
P2P=Enabled Latency (P2P Writes) Matrix (us)
GPU 0 1
0 4.61 57.14
1 58.90 4.41

CPU 0 1
0 2.31 47.70
1 47.44 2.16

6

Dear Robert,

any news?

Thanks,
Enrico

7

same situation but without any progress in about half a year