I have followed all the instruction in this article.HowTo Configure SR-IOV for Connect-IB/ConnectX-4 with KVM (InfiniBand) https://community.mellanox.com/s/article/howto-configure-sr-iov-for-connect-ib-connectx-4-with-kvm--infiniband-x but I am getting following error while Setting the desired number of VFs using “echo 6 > /sys/class/infiniband/mlx5_0/device/mlx5_num_vfs”
-bash: echo: write error: Invalid argument
Any help would be greatly appreciated.
intel_iommu=on and iommu=pt are added to kernel bootime parameter
BOOT_IMAGE=/boot/vmlinuz-3.10.0-514.el7.x86_64 root=UUID=51777676-1b13-40a7-aed8-12e9609e4b31 ro intel_pstate=disable console=tty0 console=ttyS0,115200n8 net.ifnames=0 crashkernel=auto rhgb quiet intel_iommu=on iommu=pt
mlxconfig -d /dev/mst/mt4113_pciconf0 q
Device type: ConnectIB
PCI device: /dev/mst/mt4113_pciconf0
Configurations: Next Boot
Kindly note that SR-IOV protocol need to be enabled in 4 different places:
Firmware level - according to mlxconfig output it is enabled.
System BIOS - need to verify the “Virtualization Technology” option is enabled.
Operation System at grub.conf - we can see that “intel_iommu=on” exists
In Connect-IB you must also set FPP_EN=1
- Driver - Set the desired number of VFs by invoking:
echo 4 > /sys/class/infiniband/mlx5_0/device/mlx5_num_vfs
- The command to enable SRIOV support for ConnectX4 , Connect-IB and ConnectX-5 on an MLNX-OS based subnet manage is:
switch(config)# ib sm virt enable
In addition, if after applying the above settings you still encounter the same issue
I suggest reviewing the release notes of the latest OFED 4.3 : http://www.mellanox.com/related-docs/prod_software/Mellanox_OFED_Linux_Release_Notes_4_3-1_0_1_0.pdf http://www.mellanox.com/related-docs/prod_software/Mellanox_OFED_Linux_Release_Notes_4_3-1_0_1_0.pdf
And check if there are known issues with SRIOV with older OFED versions , if yes please try and upgrade the OFED version accordingly and check if the issue resolved .
- Try using the latest MFT 4.9.0-38 and try configuring FPP_EN.