How to enable PCIe relaxed ordering for GPU Direct RDMA p2p load stores?

ryan.walton · March 7, 2025, 3:17am

PCIe relaxed ordering is a mode in which the strict execution order of TLPs is relaxed. This mode has benefits for high throughput applications. This page indicates that Nvidia GPUs can enable relaxed ordering:

github.com/NVIDIA/open-gpu-kernel-modules

src/nvidia/interface/nvrm_registry.h

main


      
          #define NV_REG_STR_RM_PCIE_LINK_SPEED_ALLOW_GEN5_ENABLE                   (0x00000001)
          #define NV_REG_STR_RM_PCIE_LINK_SPEED_ALLOW_GEN5_DISABLE                  (0x00000002)
          #define NV_REG_STR_RM_PCIE_LINK_SPEED_ALLOW_GEN6                          9:8
          #define NV_REG_STR_RM_PCIE_LINK_SPEED_ALLOW_GEN6_DEFAULT                  (0x00000000)
          #define NV_REG_STR_RM_PCIE_LINK_SPEED_ALLOW_GEN6_ENABLE                   (0x00000001)
          #define NV_REG_STR_RM_PCIE_LINK_SPEED_ALLOW_GEN6_DISABLE                  (0x00000002)
          #define NV_REG_STR_RM_PCIE_LINK_SPEED_LOCK_AT_LOAD                        31:31
          #define NV_REG_STR_RM_PCIE_LINK_SPEED_LOCK_AT_LOAD_DISABLE                (0x00000000)
          #define NV_REG_STR_RM_PCIE_LINK_SPEED_LOCK_AT_LOAD_ENABLE                 (0x00000001)
          
          //
          // Type DWORD
          //
          //
          // This can be used as a per-device regkey or not, in which case the setting
          // will apply to all devices. If this key is supplied as both a per-device and
          // non-per-device regkey, the non-per-device option will apply first to all
          // devices, and then the per-device key settings will apply, overriding the
          // settings for the relevant devices.
          //
          // Encoding : 0 - Disable PCIe Relaxed Ordering TLP header bit setting. This is

How can this setting be enabled?

Topic		Replies	Views
TX1 PCIe relaxed ordering requirement Jetson TX1	16	2302	October 18, 2021
GPU RDMA Memory Ordering Limitations RDMA Software For GPU	2	38	February 28, 2025
GPUDirect RDMA performance CUDA Programming and Performance	2	2170	March 26, 2013
RDMA latency stability and PCIE version recognize WinOF Driver	2	821	April 27, 2023
PCIe peer-to-peer between NVIDIA GPU and generic hardware? RDMA Software For GPU kernel	3	695	January 30, 2024
PCIe not being recognized - TX2 Jetson TX2	20	3103	March 16, 2020
460.73.01, Failed to acquire modesetting permission Linux	2	2512	April 24, 2021
Can I change the link order of the PCIe Lannes? Jetson TX2 pcie	3	309	October 18, 2021
PCIe memory bus prioritization over GPU Jetson AGX Orin pcie	7	628	April 16, 2024
PCIe 1x1 2x1 on custom board Jetson TX2	7	772	October 18, 2021

How to enable PCIe relaxed ordering for GPU Direct RDMA p2p load stores?

Related topics