Seeing high TLP watermarks from FPGA
- Periodic sustained TLP of 511 for 5+ seconds
- Requester (RQ) Not Ready for 352us
- GPU seems to be slow at returning DMA post credits to FPGA, causing credit starvation.
Observed Issue:
- Partial frame data loss.
- Performed frame capture and compared with good frame. Found some data within the frame is missing.
- Tried waiting12ms from FPGA interrupted and still missing data within frame.
NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition
- NVIDIA-SMI 570.172.08
- Driver Version: 570.172.08
- CUDA Version: 12.8
Frame data is ~8MB at 60Hz.
- FPGA directly into GPU
- Host SW allocates, pins and get GPU page addresses to program into FPGA page table.
Intel CPU is the root complex (no bridge)
- IOMMU=pt
- ASPM=off
- GPU Config:
- Set GPU & Mem Clock to max
- Max Power
- Tried to force into P0 but stuck at P1 - I read that this is the new norm??
Thank you for any help / suggestions