In my server application, Using RHEL I would like my system to have persistent settings to configure PCIe Gen 4.0 and max PCIe buswidth x16 for all L40S GPUs .
These settings have been confirmed to be achievable with the supermicro server configuration.
The following forum post suggests PCIe Gen 3 can be forced:
The NVreg_EnablePCIeGen3 setting appears to force PCIe Gen3 mode.
Q1. Is there a way to force the PCIe Gen 4.0 and PCIe buswidth 16 ?
I would expect the nvidia-smi --query-gpu=$csv_params --format=csv to look similar to the following:
(Note. This is my 5th edit of this post because I have been unsuccessful to upload any images to the NVIDIA site, after several edit requests. After each upload the image does NOT appear. So describing the image here with words..).
The Image displays the CSV file with following same values for all GPUs 0 through 7 in my system.
pcie.link.current=4
pcie.link.gen.gpucurrent=4
pcie.link.gen.max=4
pcie.link.gen.gpumax=4
pcie.link.gen.hostmax=5
pcie.link.width.current=16
pcie.link.width.max=16
In other words:
During system operation, the following "xxx.current" values change dynamically for each system boot and during system operation: pcie.link.current, pcie.link.gen.gpucurrent, and pcie.link.width.current.
However, I want to make these values FIXED over the life of the system.
Q2. Is there a way to also force the PCIe Gen and bus width to be maximum supported by the host hardware and the GPU hardware, and leave at the those negotiated settings ?
Q3. Do you recommend a specific PCIe Active State Power Management (ASPM) setting such as "off" or "performance" for my high performance server application?
Related links:
https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/6/html/power_management_guide/aspm
https://forums.developer.nvidia.com/t/enabling-pcie-3-0-with-nvreg-enablepciegen3-on-titan/29542
https://resources.nvidia.com/en-us-l40s/l40s-datasheet-28413
We don’t want to save power and want to have maximum performance.
The devices are running code all the time and there are brief moments in time, the GPU is idle.
We’ve seen many instances where p2pBandwidthLatencyTest tool indicates half the bandwidth is available for at least one GPUx. All traffic to and from the GPUx is half the value reported by all the other GPUs.
PCIe bus width gets reduced from 16x to 8x following message gets logged by the kernel:
messages:Jul 7 13:17:26 rhel88-hostname kernel: pci 0000:c0:00.0: 126.024 Gb/s available PCIe bandwidth, limited by 16.0 GT/s PCIe x8 link at 0000:be:01.0 (capable of 252.048 Gb/s with 16.0 GT/s PCIe x16 link)
The particular GPUx pcie.link.width.current = 8, while all the other GPUs show pcie.link.width.current = 16.
Once the system gets into this state, the GPU remains in this state. We close all processes to the GPU, but the width remains limited to 8. A reboot of the system, does not always fix this issue.
We’ve pulled the GPUx card that has the issue, ensured the slot was clean, etc , but problem still pops up.
Since we’ve recently pair-swapped of all the cards (0/1, 2/3, etc) in the respective PCIe slots, we have not seen the PCIe limited to 8 issue. However, have major concerns the bus width issue is going to pop-up again!!!
As far as pcie.link.gen…
We’ve also seen pcie.link.gen.current=4 and pcie.link.gen.gpucurrent=4 , change to pcie.link.gen.current=1 and pcie.link.gen.gpucurrent=1 , and back to pcie.link.gen.current=4 and pcie.link.gen.gpucurrent=4 .. There seems to be a correlation that p2pBandwith has lower results when initially at pcie.link.gen.current=1 and pcie.link.gen.gpucurrent=1.
So just to clarify your response to my questions:
A1 “No, current driver does not have the ability to set user desired GPU PCIe Gen4.0 and Width 16x”
A2. “No, current driver does not have the ability to keep the GPU PCIe bus at a fixed maximum negotiated generation and bus width as limited by the GPU and PCIe bus hardware & software configuration.”
Q3. (Earlier) Do you recommend a specific PCIe Active State Power Management (ASPM) setting such as “off” or “performance” for my high performance server application?
A3. ??? Can you please provide answer??)
Thank you for providing some feedback related to performance measurement and answering questions 1 & 2.
Q3. (Earlier & added clarification) While using NVIDIA GPUs and with priority of highest bandwidth and lowest latency performance. Do you recommend a specific PCIe Active State Power Management (ASPM) setting such as “off” or “performance” for my high performance server application?
A3. ??? Can you please provide answer??)
Q4. Is there some linux operating system commands or configuration settings that can be used to force the PCIe gen 4 and bus width 16x ? Possibly use pcie_set_speed.sh provided here PCIe Set Speed [Alex Forencich] or troubleshooting steps here: https://unix.stackexchange.com/questions/42361/force-re-negotiation-of-pcie-speed-on-linux . On my system, the pcie_set_speed.sh was unsuccessful at forcing the setting.
I’ve also confirmed BIOS settings are configured correctly to support the 16x operation.
On Q3, I do not know, but given your situation, I’d probably go with “Disable”.
Q4. I’m not sure any OS side command would be effective, as the the card is negotiating speed/width with the PCIe root controller, based on it’s own power saving algorithm.
I started to look at this but after developing a huge respect for those that deal with this stuff, dropped out early.