When PCIe ATS is enabled, ConnectX-6 Dx cannot convert DMA addresses

Infrastructure & Networking BlueField
1

Using a PCie protocol analyzer to capture the TLP, it is impossible to capture the ATS request message send by CX6, and CX6 uses an unconverted address to send DMA WR/RD TLP（As shown in the figure dma_at_is_0.jpg）。In addition, we can catch the ATS Invalidation message sent by RC/TA to CX6。
By the way, we use VFIO to pass through the CX6 to QEMU for network transmission.

dma_at_is_0
dma_at_is_01920×549 151 KB

2

Hello @gulan,

Thank you for posting your query on our community. Please ensure you are following the below procedure to enable ATS:

  1. Ensure that the firmware is upgraded to version xx.40.1000 or later.
  2. Enable IOMMU in kernel parameters: iommu=on
  3. Use the following commands to enable ATS in the firmware:
    # mlxconfig -d <device> s ATS_ENABLED=1
  4. After enabling ATS, perform a system reboot for the changes to take effect.
  5. Verify that ATS is enabled by checking the device configuration:
    # mlxconfig -d <device> q | grep ATS_ENABLED
    Expected output:
    ATS_ENABLED True(1)
  6. Check the lspci output for ATS capabilities:
    # lspci -s <BDF> -vvvxxx | grep ATS
    Expected output:
    Capabilities: [480 v1] Address Translation Service (ATS)
    ATSCap: Invalidate Queue Depth: 00
    ATSCtl: Enable+, Smallest Translation Unit: 00
  7. Please note, that to activate ATS, a GPU direct solution must be implemented.

Thanks,
Bhargavi

1 Like