Passthrough the whole MLNX5 to a VM

Infrastructure & Networking Software And Drivers SoC And SmartNIC
3

Thank you for your reply!

  1. Why are you trying to bind the PF to vfio-pci? What is the use case?
    Just like I said in this topic Using NEO-Host on vm with SR-IOV. I have tried to use NEO-HOST in a VM with a VF. I installed everything as far as I know. But NEO-HOST does not work. I checked the log and noticed that it uses MST to get device information. However, I cannot get anything with mst status. So I guess maybe it is the limitation of VF. That is why I’m trying to bind PF to vfio-pci so that I can passthrough PF to the VM.

  2. Are you using MLNX_OFED driver versus mlx5 OS Inbox?
    I am not totally sure this question. I have installed the MLNX_OFED successfully.

  3. Are you able to assign a VF pcie passthrough or SRIOV using our mlx5 driver stack?
    Are vfio* modules loaded (lsmod)?
    Virtualization enabled via BIOS?
    Is GRUB configured with iommu=on, iomm=pt?
    Is the PCIe device correct? (lspci -Dk | grep -i mel)
    Does the path exist? (ls -l /sys/bus/pci/drivers/vfio-pci)
    The answers to all these questions are yes. As I have said, I have already passed through a VF to the VM successfully and ibv_*_pingpong works well on the VM.

  4. Any hints in dmesg?
    This is the output in dmesg when I unbind the device from mlx5_core driver.

[2242079.400456] mlx5_core 0000:1a:00.0: E-Switch: Disable: mode(LEGACY), nvfs(4), active vports(5)
[2242081.284288] mlx5_core 0000:1a:00.0: E-Switch: cleanup
[2242082.307516] mlx5_core 0000:1a:00.0: driver left SR-IOV enabled after remove

This is the output of vfio-pci operation:

$ sudo bash -c "echo 0000:1a:00.0 > /sys/bus/pci/drivers/vfio-pci/bind"
bash: line 1: echo: write error: No such device