I am experiencing a wrong behavior when trying RDMA between two computers connected directly through IB adapters. Could it be due to the fact that the third-party PCI device is behind a PCI switch?

The IB adapters are Mellanox Technologies MT27800 Family [ConnectX-5], updated with last firmware version (16.26.1040)

The driver is mlx5_core, version MLNX_OFED_LINUX-4.6-

I am running the test on Ubuntu 18.04 LTS with kernel 4.15.0-72-generic.

The experiment consists of reading a buffer in an FPGA DDR memory, located in server1, from a remote server (server0) using RDMA.

In order to run this experiment I have followed the information provided in the link


To do that, I have extended the FPGA driver with mmap. In the next figure, the memory map can be seen. Although I am showing here a 32-bit non-prefetchable BAR, I have tried also with a 64-bit prefetchable BAR. The result has been the same.

Once the setup has been completed, I have run the ib_read_bw perfool. Next you can find the command line and the output of the command. Also, I provide the error reported by the mlx5_core driver.

Both Mellanox and FPGA devices are attached to the same root PCIe, but FPGA device is hidden behind an additional PCI switch, as it can be seen in next figure.

Hello Rafael,

Thank you for posting your question on the Mellanox Community.

Unfortunately this question is outside of the scope of the community forum.

For more information about this issue you are facing please contact support@mellanox.com with a valid support contract and we can work on resolving this issue through that support channel.

Also as a side note a newer version of the firmware you are using for the ConnectX-5 firmware has come out and can be downloaded here:


Thank you,

~Mellanox Technical Support