Unable to configure SR-IOV on Connect-IB

Hi,

I have followed all the instruction in this article.HowTo Configure SR-IOV for Connect-IB/ConnectX-4 with KVM (InfiniBand) https://community.mellanox.com/s/article/howto-configure-sr-iov-for-connect-ib-connectx-4-with-kvm--infiniband-x but I am getting following error while Setting the desired number of VFs using “echo 6 > /sys/class/infiniband/mlx5_0/device/mlx5_num_vfs”

-bash: echo: write error: Invalid argument

Any help would be greatly appreciated.

intel_iommu=on and iommu=pt are added to kernel bootime parameter

cat /proc/cmdline

BOOT_IMAGE=/boot/vmlinuz-3.10.0-514.el7.x86_64 root=UUID=51777676-1b13-40a7-aed8-12e9609e4b31 ro intel_pstate=disable console=tty0 console=ttyS0,115200n8 net.ifnames=0 crashkernel=auto rhgb quiet intel_iommu=on iommu=pt

mlxconfig -d /dev/mst/mt4113_pciconf0 q

Device #1:


Device type: ConnectIB

PCI device: /dev/mst/mt4113_pciconf0

Configurations: Next Boot

ROCE_NEXT_PROTOCOL 254

NUM_OF_VFS 6

SRIOV_EN True(1)

Hi,

Kindly note that SR-IOV protocol need to be enabled in 4 different places:

  1. Firmware level - according to mlxconfig output it is enabled.

  2. System BIOS - need to verify the “Virtualization Technology” option is enabled.

  3. Operation System at grub.conf - we can see that “intel_iommu=on” exists

In Connect-IB you must also set FPP_EN=1

  1. Driver - Set the desired number of VFs by invoking:

echo 4 > /sys/class/infiniband/mlx5_0/device/mlx5_num_vfs

cat /sys/class/infiniband/mlx5_0/device/mlx5_num_vfs

  1. The command to enable SRIOV support for ConnectX4 , Connect-IB and ConnectX-5 on an MLNX-OS based subnet manage is:

switch(config)# ib sm virt enable

In addition, if after applying the above settings you still encounter the same issue

I suggest reviewing the release notes of the latest OFED 4.3 : http://www.mellanox.com/related-docs/prod_software/Mellanox_OFED_Linux_Release_Notes_4_3-1_0_1_0.pdf http://www.mellanox.com/related-docs/prod_software/Mellanox_OFED_Linux_Release_Notes_4_3-1_0_1_0.pdf

And check if there are known issues with SRIOV with older OFED versions , if yes please try and upgrade the OFED version accordingly and check if the issue resolved .

  1. Try using the latest MFT 4.9.0-38 and try configuring FPP_EN.

Thanks,

Samer