Hello,
Hoping for some configuration help. We are using a Connect-X4 card, dual port QSFP28. It is connected with a 100G CR-4 DAC cable to an Arista switch.
Image type: FS3
FW Version: 12.28.2006
FW Release Date: 15.9.2020
Product Version: 12.28.2006
Rom Info: type=UEFI version=14.21.17 cpu=AMD64
type=PXE version=3.6.102 cpu=AMD64
Description: UID GuidsNumber
Base GUID: 3cecef15d95ca518 4
Base MAC: 3cecef5ca518 4
Image VSD: N/A
Device VSD: N/A
PSID: SM_2001000001033
Our goal is to treat this link as 2 50G ports instead of 1 100G port.
I have changed the configuration option for NUM_OF_PF to 4, which when rebooted, resulted in the system presenting 4 physical functions.
[ 356.805337] mlx5_core 0000:02:00.0 enp2s0f0np0: Link up
[ 420.093327] mlx5_core 0000:02:00.1 enp2s0f1np1: Link up
[ 424.132549] mlx5_core 0000:02:00.2 enp2s0f2np2: Link up
[ 427.681081] mlx5_core 0000:02:00.3 enp2s0f3np3: Link up
However, from inspecting on the switch side, it seems that only ever the first quad comes up
swdp01(config-if-Et20/3)#show int Et20/1-4 status
Port Name Status Vlan Duplex Speed Type Flags Encapsulation
Et20/1 MLAG Member data connected in Po8 full 50G 100GBASE-CR4
Et20/3 notconnect 1 full 50G 100GBASE-CR4
When I bounce the first split interface on the switch side, I see the pair of both PF mapped to port 1 bounce as well.
[ 1187.751725] mlx5_core 0000:02:00.0 enp2s0f0np0: Link down
[ 1187.752562] mlx5_core 0000:02:00.2 enp2s0f2np2: Link down
[ 1187.753266] bond1: (slave enp2s0f0np0): speed changed to 0 on port 1
[ 1187.755334] bond1: (slave enp2s0f0np0): link status definitely down, disabling slave
[ 1187.755340] bond1: now running without any active interface!
[ 1187.755443] br-vxlan: port 1(bond1.57) entered disabled state
[ 1198.900903] mlx5_core 0000:02:00.0 enp2s0f0np0: Link up
[ 1198.901721] mlx5_core 0000:02:00.2 enp2s0f2np2: Link up
Is there an mlxconfig or ini reconfiguration which can be done to make it so that the 2 PFs on each port are independent/different lanes of the CR-4 DAC?
Ultimately I want the card to negotiate 2 discrete 50G or even 25G links for the PFs so that we can have independent interface configuration on the switch side for each of them. How would I get the CX-4 to treat them as separate rather than mirroring state of the first PF on a port?
This is something similar to intel epct, where you can split the ports into different links, here’s an example of that tool
root@comp-1-1:~# ./epct64e -nic=1 -get
Ethernet Port Configuration Tool
EPCT version: v1.41.03.03
Copyright 2019 - 2024 Intel Corporation.
Available Port Options:
==========================================================================
Port Quad 0 Quad 1
Option Option (Gbps) L0 L1 L2 L3 L4 L5 L6 L7
======= ============================= ================ ================
2x1x100 -> 100 - - - 100 - - -
2x50 -> 50 - 50 - - - - -
4x25 -> 25 25 25 25 - - - -
Active 2x2x25 -> 25 25 - - 25 25 - -
8x10 -> 10 10 10 10 10 10 10 10
100 -> 100 - - - - - - -