Connect-X3 how to create VFs on multiple cards

Infrastructure & Networking Adapters and Cables Ethernet Adapter Cards
#1

I’ve been struggling to get VFs working with two Connect-X3 card in the same chassis. The chassis is a NUMA system with one card associated with each processor. The 4 PFs are showing up in Linux, but I need to have 3 VFs per physical PF, Can somebody please advise on how to configure the Mellanox driver for this?

I can get the driver to correctly create the VFs for the first card alright, but I can’t get it to create any VFs at all on the 2nd card

Here’s the contents of /etc/modprobe.d/mlx4.conf with the variations I’ve tried in comments:

lab@rpdsim:/etc/modprobe.d$ cat mlx4.conf

# mlx4_core gets automatically loaded, load mlx4_en also (LP: #1115710)

softdep mlx4_core post: mlx4_en

#This line kind of works, but only creates VFs for the 0000:82:00.0 card

#options mlx4_core log_num_mgm_entry_size=-7 enable_sys_tune=1 mlx4_en_only_mode=1 num_vfs=3,3,0 probe_vf=3,3,0

#This line doesn’t work at all

#options mlx4_core log_num_mgm_entry_size=-7 enable_sys_tune=1 mlx4_en_only_mode=1 num_vfs=0000:06:00.0,0000:82:00.0;3;3 probe_vf=0000:06:00.0,0000:82:00.0;3;3

#This line doesn’t work at all

#options mlx4_core log_num_mgm_entry_size=-7 enable_sys_tune=1 mlx4_en_only_mode=1 num_vfs=0000:82:00.0;3 probe_vf=0000:82:00.0;3

#This line doesn’t work at all

#options mlx4_core log_num_mgm_entry_size=-7 enable_sys_tune=1 mlx4_en_only_mode=1 num_vfs=0000:82:00.3-5;3 probe_vf=0000:82:00.3-5;3

#This line doesn’t work at all

#options mlx4_core log_num_mgm_entry_size=-7 enable_sys_tune=1 mlx4_en_only_mode=1 num_vfs=0000:82:00.3-5;3;3;0 probe_vf=0000:82:00.3-5;3;3;0

#This line kind of works, but only creates VFs for the 0000:82:00.0 card

#options mlx4_core log_num_mgm_entry_size=-7 enable_sys_tune=1 mlx4_en_only_mode=1 num_vfs=0000:82:00.0-3 probe_vf=0000:82:00.0-3

#This line kind of works, but only creates VFs for the 0000:82:00.0 card

#options mlx4_core log_num_mgm_entry_size=-7 enable_sys_tune=1 mlx4_en_only_mode=1 num_vfs=0000:06:00.0-3,0000:82:00.0-3 probe_vf=0000:06:00.0-3,0000:82:00.0-3

#This line doesn’t work at all

options mlx4_core log_num_mgm_entry_size=-7 enable_sys_tune=1 mlx4_en_only_mode=1 num_vfs=0000:06:00.0-3 probe_vf=0000:06:00.0-3

Thanks

Donal

#2

Turns out I had neglected to configure SRIOV on the 2nd card, via ‘sudo mlxconfig -d /dev/mst/mt4099_pciconf0 set SRIOV_EN=1 NUM_OF_VFS=4’

It worked fine after this was applied