Background:
MB: Supermicro X10DRi
SW: 2x Supermicro SSE-F3548R (MLAG - UP and running)
Cables: 1x Mellanox MCP2M00-A003E30L and 1x FS S28-PC025
OS: Ubuntu 18.04
Problem:
For my perspective everything seems to be fine and all interfaces are UP but no connection.
Host have also other separate 1G port (connected on separate SW) which is alright…after netplan apply bond and everything seems to be UP…pinging 25GB network card but it seems somehow throught 1G port becasue i can’t ping from 25GB SW.
After reboot still everything is UP but no connection either to separate 1G port :D
What could be wrong ?
/////////////////////////////////////
lspci | grep -i Mellanox
81:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
81:00.1 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
/////////////////////////////////////
mst status:
MST modules:
MST PCI module is not loaded
MST PCI configuration module loaded
MST devices:
/dev/mst/mt4117_pciconf0 - PCI configuration cycles access.
domain:bus:dev.fn=0000:81:00.0 addr.reg=88 data.reg=92 cr_bar.gw_offset=-1
Chip revision is: 00
/////////////////////////////////////
ibstat:
CA ‘mlx5_bond_0’
CA type: MT4117
Number of ports: 1
Firmware version: 14.29.2002
Hardware version: 0
Node GUID: 0x0c42a10300ba8d6e
System image GUID: 0x0c42a10300ba8d6e
Port 1:
State: Active
Physical state: LinkUp
Rate: 25
Base lid: 0
LMC: 0
SM lid: 0
Capability mask: 0x04010000
Port GUID: 0xf06504fffe689d2d
Link layer: Ethernet
/////////////////////////////////////
lshw -class network
*-network:0
description: Ethernet interface
product: MT27710 Family [ConnectX-4 Lx]
vendor: Mellanox Technologies
physical id: 0
bus info: pci@0000:81:00.0
logical name: ens5f0
version: 00
serial: f2:65:04:68:9d:2d
width: 64 bits
clock: 33MHz
capabilities: pciexpress vpd msix pm bus_master cap_list rom ethernet physical autonegotiation
configuration: autonegotiation=on broadcast=yes driver=mlx5_core driverversion=5.0-0 duplex=full firmware=14.29.2002 (MT_2420110034) latency=0 link=yes multicast=yes slave=yes
resources: irq:24 memory:c8000000-c9ffffff memory:fbe00000-fbefffff memory:cc000000-cc7fffff
*-network:1
description: Ethernet interface
product: MT27710 Family [ConnectX-4 Lx]
vendor: Mellanox Technologies
physical id: 0.1
bus info: pci@0000:81:00.1
logical name: ens5f1
version: 00
serial: f2:65:04:68:9d:2d
width: 64 bits
clock: 33MHz
capabilities: pciexpress vpd msix pm bus_master cap_list rom ethernet physical autonegotiation
configuration: autonegotiation=on broadcast=yes driver=mlx5_core driverversion=5.0-0 duplex=full firmware=14.29.2002 (MT_2420110034) latency=0 link=yes multicast=yes slave=yes
resources: irq:111 memory:ca000000-cbffffff memory:fbd00000-fbdfffff memory:cc800000-ccffffff
*-network
description: Ethernet interface
physical id: 3
logical name: bond-lan
serial: f2:65:04:68:9d:2d
capabilities: ethernet physical
configuration: autonegotiation=off broadcast=yes driver=bonding driverversion=3.7.1 duplex=full firmware=2 ip=10.20.32.199 link=yes master=yes multicast=yes
/////////////////////////////////////
ip a:
4: ens5f0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond-lan state UP group default qlen 1000
link/ether f2:65:04:68:9d:2d brd ff:ff:ff:ff:ff:ff
5: ens5f1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond-lan state UP group default qlen 1000
link/ether f2:65:04:68:9d:2d brd ff:ff:ff:ff:ff:ff
6: bond-lan: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether f2:65:04:68:9d:2d brd ff:ff:ff:ff:ff:ff
inet 10.20.32.199/24 brd 10.20.32.255 scope global bond-lan
valid_lft forever preferred_lft forever
inet6 fe80::f065:4ff:fe68:9d2d/64 scope link tentative
valid_lft forever preferred_lft forever
/////////////////////////////////////
SW ports are in 25GB, full-duplex
SMIS# sh interface port-channel 3
po3 up, line protocol is up (connected)
Bridge Port Type: Customer Bridge Port
Hardware Address is 0c:c4:7a:2c:d0:c5
MTU 1500 bytes,
LACP Port Admin Oper Port Port
Port State Priority Key Key Number State
Fx0/3 Bundle 128 3 3 0x3 0x3d
Reception Counters
Octets : 890
Unicast Packets : 0
Unicast Packets Rate : 0/Sec
Broadcast Packets : 0
Broadcast Packets Rate : 0/Sec
Multicast Packets : 9
Multicast Packets Rate : 0/Sec
Pause Frames : 0
Undersize Frames : 0
Oversize Frames : 0
CRC Error Frames : 0
Discarded Packets : 0
Error Packets : 0
Unknown Protocol : 0
Received Rate : 0 bps
Transmission Counters
Octets : 635
Unicast Packets : 0
Unicast Packets Rate : 0/Sec
Broadcast Packets : 0
Broadcast Packets Rate : 0/Sec
Multicast Packets : 5
Multicast Packets Rate : 0/Sec
Pause Frames : 0
Discarded Packets : 0
Error Packets : 0
Transmit Rate : 492 bps