PCIE 3.0 on Linux?

Accelerated Computing CUDA CUDA Programming and Performance
1

We have purchased a HP ML 350p G8 server and a GTX 680 to do some CUDA testing. Everything runs fine, except the bandwidth D->H or H->D is only pcie 2.0 speeds. (Running the standard bandwidthTest confirms this). We have tried both the 4.2 devdriver and the 5.0 preview devdriver. We are running RHEL 6.2

The output from lspci shows that the card is not being recognized / activated as a pcie 3.0 device:

21:00.0 VGA compatible controller: NVIDIA Corporation GK104 [GeForce GTX 680] (rev a1) (prog-if 00 [VGA controller])

...

LnkCap:	Port #0, Speed 5GT/s, Width x16, ASPM L0s L1, Latency L0 <512ns, L1 <4us

Any ideas on how to get this to work properly?

2

For what its worth I’m having trouble getting PCIE 3.0 working on the Xeon E5 platform under Windows 7 too. In my case I’m using a Supermicro X9DR3-F motherboard and GTX 670 video cards.

I’ve been posting here: [url=“The Official NVIDIA Forums | NVIDIA”]The Official NVIDIA Forums | NVIDIA

So far I’ve discovered that you have to enable Gen3 for each PCIE slot individually in the BIOS and also that nVidia seems to be treating this platform as unvalidated (like the X79) which means that on Windows you need to make a change to the registry. There might be a similar change you can make on Linux or you may just be stuck. Finally, I’ve found that the Supermicro X9DR3-F is currently unstable when running these video cards in Gen3 mode. This might be a problem with the current BIOS or with the motherboard hardware or it might be a problem with the platform in general.

Please keep posting updates.

3

I’ll let you know if I find anything. The HP bios does not have any settings that relate to PCIE except for one that doesn’t appear to have any affect on 2 vs 3. We do have some other cards showing as 8GT/s so I suspect we’re getting the run around from the Nvidia driver, which is beyond frustrating.

Need to find out if there’s something like the registry hack for the nvidia linux driver.

4

Well this is a stupid workaround, but it does work.

I loaded the 295.33 drivers and the card works at 8GT/s

D->H is 11.5GB/s
H->D is 12GB/s

COME ON NVIDIA fix your stupid driver!

We should at least have the option to activate pcie 3.0 on platforms we believe it should work on.

5

Thanks for the update. What is the exact make and model of GTX 680 you are using please?

6

We’re using the PNY board:

VCGGTX680XPB

7

I was about to post a new thread on this. It appears that NVIDIA has disabled PCI-E 3.0 support in drivers post 295.33 including the latest 302.07 BETA driver. I am running a CUDA app on an x79 board with three GTX 680s and ran into this limitation. The application I am running is as much as 27% slower set at PCI-E 2.0 5 GT/s compared to running natively at PCI-E 3.0 8 GT/s.

I find this to be extremely frustrating as I am now stuck using an older driver until this issue is resolved. This also limits my options on buying any newer cards post GTX 680. A main reason I bought your Kepler products is because of PCI-E 3.0 support.

NVIDIA, can you please re-enable PCI-E 3.0 support in your latest Linux driver? At the very least, can you give us an option to enable PCI-E 3.0 when loading the kernel module? For example something like this:

modprobe nvidia ENABLE_PCI3=1

Also, please do not make this an nvidia-settings only option as I am not running X11 and that program is useless to me and probably many other people that are running CUDA in the shell only. Please help us with this issue. Thanks.

8

NVIDIA released a tool to enable PCI-Express 3.0 on X79 systems:

http://nvidia.custhelp.com/app/answers/detail/a_id/3135/session/L3RpbWUvMTM0MDIyMzU2OC9zaWQvaDEzbE45X2s=

The text indicates that it is a DOS executable (but who knows), but you might be able to run it from a FreeDOS boot image without Windows.

9

That is a Windows program that sets a value in the registry. We need something like this compiled for running on Linux that will set the link speed to 8 GT/s or an argument that can be added to the kernel module for the nvidia driver.

10

And for me, it’s not an x79 board so the solution should be something generic.

11

Sorry for resurrecting this old thread, but I had been looking for a solution myself and stumbled upon many unanswered questions. The solutions suggested here didn’t work.

On Ubuntu 16.04, I also had to run update-initramfs (or the equivalent for your distro to update the initramfs/initrd). More details here: nvidia_367 kernel module options ignored at boot, ubuntu 16.04, /etc/modprobe.d

12

There’s an argument that enables PCI-E 3.0 under Linux as long as your motherboard supports it. See my old post here:

[url]https://devtalk.nvidia.com/default/topic/533200/linux/gtx-titan-drivers-for-linux-32-64-bit-release-/post/3753244/#3753244[/url]

13

I had found your post and tried it but it didn’t work for me on Ubuntu 16.04. The only thing that worked was setting the module option via a file in modprobe.d/ and the issuing update-initramfs