SN2100 MLAG link down and stuck in passive mode

Infrastructure & Networking Switches and Gateways
1

We have 2 brand new SN2100 running Onyx 3.9.0914 which I’m trying to connect to a pair of Nexus switches. On the Nexus side our netops configured a vPC, on the Mellanox side I followed the guide here: https://mymellanox.force.com/mellanoxcommunity/s/article/how-to-configure-mlag-on-mellanox-switches#jive_content_id_MLAG_Interface and also read through https://community.mellanox.com/s/article/mellanox-mlag-and-cisco-vpc-configuration-and-troubleshooting

Whatever I try, the links stay down, I have tried shutting down and no shutdown on the mlag-port-channel, the members, and mlag.

EDIT: Extra detail and move switch config to here instead of reply

We are using 40G to 4x10G breakout cables. 2x10G from each breakout to each Nexus Switch

Switch 1 abridged config:

Running database “initial”

Generated at 2020/07/06 17:05:49 +0000

Hostname: VTC-PS01-140B02

Product release: 3.9.0914

Running-config temporary prefix mode setting

no cli default prefix-modes enable

Interface Split configuration

interface ethernet 1/3 module-type qsfp-split-4 force

MLAG protocol

protocol mlag

Interface Ethernet configuration

interface mlag-port-channel 90

interface port-channel 1

interface ethernet 1/3/1-1/3/4 speed 10G force

interface ethernet 1/3/1-1/3/4 mtu 9216 force

interface mlag-port-channel 90 mtu 9216 force

interface ethernet 1/1-1/2 channel-group 1 mode active

interface ethernet 1/3/1-1/3/4 mlag-channel-group 90 mode active

interface mlag-port-channel 90 switchport mode trunk

interface mlag-port-channel 90 no shutdown

LAG configuration

lacp

port-channel load-balance ethernet source-destination-mac

VLAN configuration

vlan 202

vlan 205-207

vlan 218

vlan 227

vlan 231

vlan 4000

interface mlag-port-channel 90 switchport trunk allowed-vlan none

interface mlag-port-channel 90 switchport trunk allowed-vlan add 1

interface mlag-port-channel 90 switchport trunk allowed-vlan add 202

interface mlag-port-channel 90 switchport trunk allowed-vlan add 205-207

interface mlag-port-channel 90 switchport trunk allowed-vlan add 218

interface mlag-port-channel 90 switchport trunk allowed-vlan add 227

interface mlag-port-channel 90 switchport trunk allowed-vlan add 231

STP configuration

no spanning-tree

L3 configuration

ip routing vrf default

interface vlan 4000

interface vlan 4000 ip address 192.168.100.2/24 primary

DCBX PFC configuration

dcb priority-flow-control enable force

interface port-channel 1 dcb priority-flow-control mode on force

MLAG configurations

mlag-vip mlag-vip1 ip 10.96.97.236 /25 force

no mlag shutdown

mlag system-mac 00:00:5E:00:01:5D

interface port-channel 1 ipl 1

interface vlan 4000 ipl 1 peer-address 192.168.100.4

Network interface configuration

no interface mgmt0 dhcp

interface mgmt0 ip address 10.96.97.227 /25

Other IP configuration

hostname VTC-PS01-140B02

ip domain-list redacted

ip name-server 10.96.110.2

ip route vrf default 0.0.0.0/0 10.96.97.129

Other IPv6 configuration

no ipv6 enable

Switch 2 abridged config:

Running database “initial”

Generated at 2020/07/06 17:13:09 +0000

Hostname: VTC-PS01-140B04

Product release: 3.9.0914

Running-config temporary prefix mode setting

no cli default prefix-modes enable

Interface Split configuration

interface ethernet 1/3 module-type qsfp-split-4 force

MLAG protocol

protocol mlag

Interface Ethernet configuration

interface mlag-port-channel 90

interface port-channel 1

interface ethernet 1/3/1-1/3/4 speed 10G force

interface ethernet 1/3/1-1/3/4 mtu 9000 force

interface mlag-port-channel 90 mtu 9000 force

interface ethernet 1/1-1/2 channel-group 1 mode active

interface ethernet 1/3/1-1/3/4 mlag-channel-group 90 mode active

interface mlag-port-channel 90 switchport mode trunk

interface mlag-port-channel 90 no shutdown

LAG configuration

lacp

port-channel load-balance ethernet source-destination-mac

VLAN configuration

vlan 202

vlan 205-207

vlan 218

vlan 227

vlan 231

vlan 4000

interface mlag-port-channel 90 switchport trunk allowed-vlan none

interface mlag-port-channel 90 switchport trunk allowed-vlan add 1

interface mlag-port-channel 90 switchport trunk allowed-vlan add 202

interface mlag-port-channel 90 switchport trunk allowed-vlan add 205-207

interface mlag-port-channel 90 switchport trunk allowed-vlan add 218

interface mlag-port-channel 90 switchport trunk allowed-vlan add 227

interface mlag-port-channel 90 switchport trunk allowed-vlan add 231

STP configuration

no spanning-tree

L3 configuration

ip routing vrf default

interface vlan 4000

interface vlan 4000 ip address 192.168.100.4/24 primary

DCBX PFC configuration

dcb priority-flow-control enable force

interface port-channel 1 dcb priority-flow-control mode on force

MLAG configurations

mlag-vip mlag-vip1 ip 10.96.97.236 /25 force

no mlag shutdown

mlag system-mac 00:00:5E:00:01:5D

interface port-channel 1 ipl 1

interface vlan 4000 ipl 1 peer-address 192.168.100.2

Network interface configuration

no interface mgmt0 dhcp

interface mgmt0 ip address 10.96.97.228 /25

Other IP configuration

hostname VTC-PS01-140B04

ip domain-list redacted

ip name-server 10.96.110.2

ip route vrf default 0.0.0.0/0 10.96.97.129

Other IPv6 configuration

no ipv6 enable

2

Hi Darien,

I understand that ports 1/3/1-1/3/4 on both switches are the ones connected to the nexus switches.

By default the split ports are at speed of 25G, this is the expected speed that you want to link at or you need 10G speed on those ports?

You can change speed (need to remove the ports from MPO 90 before):

(config)# no interface ethernet 1/3/1-1/3/4 mlag-channel-group

(config)# interface ethernet 1/3/1-1/3/4 speed 10G force

(config)# interface ethernet 1/3/1-1/3/4 mlag-channel-group 90 mode active

3

Hi Eddie,

Yes the ports are 10G, I had already set them before adding to the channel group. I have edited the original post with that information.

4

Hi Darien,

Can you open a support case with support@mellanox.com ? Nvidia Mellanox Technical Support will continue working with you from the case.