The IB adapters are Mellanox Technologies MT27800 Family [ConnectX-5], updated with last firmware version (16.26.1040)
The driver is mlx5_core, version MLNX_OFED_LINUX-4.6-1.0.1.1-ubuntu18.04-x86_64.
I am running the test on Ubuntu 18.04 LTS with kernel 4.15.0-72-generic.
The experiment consists of reading a buffer in an FPGA DDR memory, located in server1, from a remote server (server0) using RDMA.
In order to run this experiment I have followed the information provided in the link
https://community.mellanox.com/s/article/howto-implement-peerdirect-client-using-mlnx-ofed
To do that, I have extended the FPGA driver with mmap. In the next figure, the memory map can be seen. Although I am showing here a 32-bit non-prefetchable BAR, I have tried also with a 64-bit prefetchable BAR. The result has been the same.
Once the setup has been completed, I have run the ib_read_bw perfool. Next you can find the command line and the output of the command. Also, I provide the error reported by the mlx5_core driver.
Both Mellanox and FPGA devices are attached to the same root PCIe, but FPGA device is hidden behind an additional PCI switch, as it can be seen in next figure.
