Hi!
I was wondering if you know if Two/Dual GeForce RTX 3090s and NVLink have Ubuntu support? At least Blender has support for NVLink:
https://wiki.blender.org/wiki/Reference/Release_Notes/2.90/Cycles
I asked the Blender developers and they also said Ubuntu supports NVLink, which made me happy :)
What about you, NVIDIA, do you confirm?
Thank you!
FWIW, I have an NVLink and two RTX 3090s running Ubuntu, and I haven’t been able to get the NVLink to work.
When I run ‘nvidia-smi nvlink -c’ I get just:
GPU 0: GeForce RTX 3090 (UUID: GPU-f634707c-5e3b-b685-ad8c-30e3b916f204)
GPU 1: GeForce RTX 3090 (UUID: GPU-17056259-f618-8028-163a-0f62bf47aae7)
When I run ‘nvidia-smi nvlink -s’ I see:
GPU 0: GeForce RTX 3090 (UUID: GPU-f634707c-5e3b-b685-ad8c-30e3b916f204)
Link 0: <inactive>
Link 1: <inactive>
Link 2: <inactive>
Link 3: <inactive>
GPU 1: GeForce RTX 3090 (UUID: GPU-17056259-f618-8028-163a-0f62bf47aae7)
Link 0: <inactive>
Link 1: <inactive>
Link 2: <inactive>
Link 3: <inactive>
Guidance would be great.
Aw shucks… that doesn’t keep my hopes up :(
Which is weird, I asked from Blender institute and they said NVLink should work fine in Ubuntu with 2 x RTX 3090…
Hmph…
Of course it should work. @ben.hendrickson did you properly seat the bridge, is it lit? Did you try turning off iommu? Please run nvidia-bug-report.sh as root and attach the resulting nvidia-bug-report.log.gz file to your post.
What kind of bridge were you using? A 4-slot 3090 bridge or a 2/3-slot Quadro bridge?
@generix I’m running into the same exact issue, here’s my nvidia-bug-report.sh dump any help is much appreciated. I am not running IOMMU and my slot is in fitted nicely. I am using the 3-slot A6000 bridge on my 2 3090 FEs. Here’s some further talk on that same setup used by others.
Logs: https://drive.google.com/file/d/1Kmz5ut5r7GWpYQQIZGPzOiOYJm3hCY33/view?usp=sharing
Additional notes: NVIDIA K8s GPU operator isn’t being used so you can ignore that
I’ve checked this already in another thread where the user had the same setup (2x3090+A6000 bridge). No chance, the bridge just isn’t detected. Unfortunately, there are also no reports to be found from someone using the original 3090 bridge anywhere so it’s really inconclusive whether this is a general driver-wise problem with nvlink on 3090s or the A6000 bridge is incompatible.
I haven’t searched the cuda forums since then (~2 months) but also no official info from nvidia was to be found.
Now I got my 2 x 3090 (2 x Gigabyte GeForce RTX 3090 AORUS XTREME) and AORUS 4-slot NVLINK for series 30.
Can’t get NVLINK to work in Windows or Ubuntu :(( - But otherwise I like my setup a lot :) Renders blazing fast in Blender.
I’m asking Gigabyte if they’d have a clue about their cards and accessories.
Thanks for reporting back, what’s the output of
nvidia-smi nvlink -c
in your case? What’s the brand/model of your mainboard? Is it SLI capable?
I actually found one report of 2x3090+2 slot A6000 bridge working on a 4-Way SLI capable workstation mainboard.
mj@mj:~$ nvidia-smi nvlink -c
GPU 0: GeForce RTX 3090 (UUID: GPU-17eb117c-3ea1-28c3-c1bd-ab4984cbe49a)
GPU 1: GeForce RTX 3090 (UUID: GPU-ebad44cf-29c4-6a2b-031a-3942e384d171)
Asus Prime TRX40-Pro is my motherboard, SLI capable yes.
Interesting, hmmm…
Let’s see what Gigabyte will answer anyhow
Scratch my post, that combo wasn’t working either, here’s that combo:
https://forums.developer.nvidia.com/t/using-an-a6000-2-slot-nvlink-bracket-on-geforce-3090/169118
So while this company claims to have tested 3090’s + A6000 2-slot bridge on Windows and found it working:
https://www.pugetsystems.com/labs/articles/NVIDIA-NVLink-2021-Update-and-Compatibility-Chart-2074/
Nobody seems to have managed to make it work with any of the bridges (2,3,4-slot A6000,3090), neither on Linux nor on Windows.
https://forums.developer.nvidia.com/t/making-work-nvlink-on-windows-10-with-dual-3090/175086
regardless of SLI-capable board or not.
OTOH, nvlink on A6000 just works plug-and-play:
https://forums.developer.nvidia.com/t/two-rtx-a6000-nvlink-link-is-supported-false/177285/4
Yes, hard to say why they’re even selling those NVLink bridges if they don’t seem to work. Must have worked when they tested them in their own labs, I think.
UPDATE:
my 2x3090 FEs with 3-slot A6000 bridge using Ubuntu 20.04 works as expected! my fitment was not tight and I realized that after re-installing the bridge and I heard a click sound! I am so happy it works
Wonderful to hear that! I didn’t hear a click sound even though I thought I had a tight fit - will observe this next time, thanks!
Updated:
Rig : EYPC 7352 x86 + 2x3090 with a 4 slot NVLINK
Environment : Ubuntu22.04, nvidia-driver-530 (530.30.02) CUDA12.1
Before checking nvidia-smi nvlink -s or nvdiam-smi topo -m, you need to turn on the Persistent-Mode with nvidia-smi -pm 1
And sudo reboot, then you have it! NVlink is enabled!
nvdiam-smi topo -m should output NV# between GPU0 and GPU1 now!