RDMA: GPU DMA Link Layer backpressure

Seeing high TLP watermarks from FPGA

  • Periodic sustained TLP of 511 for 5+ seconds
  • Requester (RQ) Not Ready for 352us
  • GPU seems to be slow at returning DMA post credits to FPGA, causing credit starvation.

Observed Issue:

  • Partial frame data loss.
  • Performed frame capture and compared with good frame. Found some data within the frame is missing.
  • Tried waiting12ms from FPGA interrupted and still missing data within frame.

NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition

  • NVIDIA-SMI 570.172.08
  • Driver Version: 570.172.08
  • CUDA Version: 12.8

Frame data is ~8MB at 60Hz.

  • FPGA directly into GPU
  • Host SW allocates, pins and get GPU page addresses to program into FPGA page table.

Intel CPU is the root complex (no bridge)

  • IOMMU=pt
  • ASPM=off
  • GPU Config:
    • Set GPU & Mem Clock to max
    • Max Power
    • Tried to force into P0 but stuck at P1 - I read that this is the new norm??

Thank you for any help / suggestions

hi

Seems this issue is not networking issue, suggest ask in RTX pro forum RTX PRO - NVIDIA Developer Forums

Thank you @shim1

Cross Posted: 386990