How to enable PCIe relaxed ordering for GPU Direct RDMA p2p load stores?

Infrastructure & Networking Software And Drivers RDMA Software For GPU
1

PCIe relaxed ordering is a mode in which the strict execution order of TLPs is relaxed. This mode has benefits for high throughput applications. This page indicates that Nvidia GPUs can enable relaxed ordering:

How can this setting be enabled?

2

Hi @ryan.walton

For Linux Systems
To enable PCIe relaxed ordering on Linux, you can use the kernel module parameter system:
1. Temporary enabling (until next reboot):

sudo modprobe -r nvidia  # Remove the current module if loaded
sudo modprobe nvidia NVreg_SetPCIERelaxedOrdering=1  # Enable
  1. Permanent enabling (persists across reboots):
    Create a configuration file in /etc/modprobe.d/:

echo "options nvidia NVreg_SetPCIERelaxedOrdering=1" | sudo tee /etc/modprobe.d/nvidia-relaxed-ordering.conf

  1. To force-enable regardless of topology compatibility:

echo "options nvidia NVreg_SetPCIERelaxedOrdering=2" | sudo tee /etc/modprobe.d/nvidia-relaxed-ordering.conf

Verifying the Setting
You can verify if relaxed ordering is enabled by using the lspci command:

sudo lspci -vv | grep -i "RlxdOrd"

If relaxed ordering is enabled, you should see RlxdOrd+ in the output. If disabled, you’ll see RlxdOrd-.