How to use VF on host

Infrastructure & Networking Software And Drivers Virtualization For Infiniband And Ethernet
1

I would like to test the performance of SR-IOV, so I create multiple VFs on a CX-5 adapter. Then I assigned each VF an IP address and tested the connection with another machine, each VF worked well for Ethernet (tested with ping command). However, when I tested these VFs with rping or ibv_pingpong for RoCEv2, neither of them worked.

However, the VF that was passed through to a VM worked well on all tests (it can communicate with another server with both Ethernet and RoCEv2). I do not have any idea about how to debug this problem? Does anyone know the reason? Thank you.

For details, refer to https://forums.developer.nvidia.com/t/passthrough-the-whole-mlnx5-to-a-vm/273047/10?u=luckyoung

2

hi

Suspect issue caused by RDMA device not init successfully or some permission setting of VF was wrong.
This link maybe can help you:
https://docs.nvidia.com/networking/display/mlnxofedv590560/single+root+io+virtualization+(sr-iov)

Thank you
Meng, Shi

3

Hi shim,

Thank you for your answer. Actually, this article is what I have followed. I will try it again and will let you know the result.

Bests,
Luckyoung

4

I had the same issue and figured out a workaround. The root cause is that after setting up VFs, the IB device’s node_guid in the host stays zero in ibv_devices. When I pass the VF to a guest it gets a node_guid and ROCE works.

To work around this I use devlink to set a MAC address on the port, then I unload mlx5_ib and reload the module:

devlink port # find your VF in the list here and copy the id
devlink port function set pci/0000:10:00.1/65537 hw_addr 72:e4:64:e0:87:a0
rmmod mlx5_ib
modprobe mlx5_ib
ibv_devices # should show a non-zero node_guid for the VF now

after this the VF has a node_guid in ibv_devices. This seems like a bug in the driver.