GPU <--> Xilinx FPGA failing, "Unsupported Request"

Infrastructure & Networking Software And Drivers RDMA Software For GPU
1

Hi,

I’m working on developing peer-to-peer communication between a Xilinx FPGA and a Turing GPU. x86_64 Intel host, with Broadcom (PLX) PCIe switches, RHEL9, NVIDIA driver 550.120.

I am encountering issues when trying to perform read and writes mastered by the FPGA to the GPU. I have followed the GPU Direct documentation online, and I’ve verified that I’m properly pinning Cuda allocated GPU memory into the GPU BAR.

I can do the following:

  • Master read and writes from the FPGA to the x86 host
  • Master read and writes from the GPU to the FPGA BAR
  • Master read and writes from the x86 host to the GPU, by mapping the section of the GPU BAR that holds the Cuda allocated buffer into the x86 host’s virtual memory.

However, when I try to master transactions from the FPGA to the GPU, I run into issues.

  • For writes mastered by the FPGA to the GPU, the GPU memory remains untouched.
  • For reads mastered by the FPGA to the GPU, I get “Unsupported Request” failure codes in the TLP completion packets.

I have data link layer data recordings captured by the FPGA for when this issue is occurring that I can post if it helps solve the issue.

I have made sure to disable the Intel “Access Control Service” in BIOS. The GPU is connected by a Broadcom (PLX) PCIe switch that should have and MMU functionality disabled.

What could be going on here?

Thanks
Ryan