Hi friends!
Got visual bug with 100gb ports after update to Cumulus ver.5.4.0
After update and reboot all services stopped and started Ok.
But after hard power off/On all updated Switches show strange solid amber indication on ports 100Gb.
Look at screen:
All services also started ok. No issues.
Ports have not yet been connected and not checked.
Confuses this chaotic garland of port indication…
What’s wrong?
Hello! You upgraded to CL 5.4.0 but this sounds like a new install since no ports are connected yet, what version did you upgrade from and was it a package upgrade via ‘apt’ or an image upgrade via ‘onie-install’?
Can you provide the output of net show interface or nv show interface and we can see if those ports are broken out or not? Also, for one of those ports showing amber LED please issue sudo l1-show swpXXsYY
Hello!
Captured output:
Linux cumulus 5.10.0-cl-1-amd64 #1 SMP Debian 5.10.162-1+cl5.4.0u1 (2023-01-20) x86_64
Welcome to NVIDIA Cumulus (R) Linux (R)
ZTP in progress. To disable, do ‘ztp -d’
cumulus@cumulus:mgmt:~$ net show interface
State Name Spd MTU Mode LLDP Summary
UP lo N/A 65536 Loopback IP: 127.0.0.1/8
lo IP: ::1/128
DN eth0 N/A 1500 Mgmt Master: mgmt(UP)
DN swp1 N/A 9216 Default
DN swp2 N/A 9216 Default
DN swp3 N/A 9216 Default
DN swp4 N/A 9216 Default
DN swp5 N/A 9216 Default
DN swp6 N/A 9216 Default
DN swp7 N/A 9216 Default
DN swp8 N/A 9216 Default
DN swp9 N/A 9216 Default
DN swp10 N/A 9216 Default
DN swp11 N/A 9216 Default
DN swp12 N/A 9216 Default
DN swp13 N/A 9216 Default
DN swp14 N/A 9216 Default
DN swp15 N/A 9216 Default
DN swp16 N/A 9216 Default
DN swp17 N/A 9216 Default
DN swp18 N/A 9216 Default
DN swp19 N/A 9216 Default
DN swp20 N/A 9216 Default
DN swp21 N/A 9216 Default
DN swp22 N/A 9216 Default
DN swp23 N/A 9216 Default
DN swp24 N/A 9216 Default
DN swp25 N/A 9216 Default
DN swp26 N/A 9216 Default
DN swp27 N/A 9216 Default
DN swp28 N/A 9216 Default
DN swp29 N/A 9216 Default
DN swp30 N/A 9216 Default
DN swp31 N/A 9216 Default
DN swp32 N/A 9216 Default
DN swp33 N/A 9216 Default
DN swp34 N/A 9216 Default
DN swp35 N/A 9216 Default
DN swp36 N/A 9216 Default
DN swp37 N/A 9216 Default
DN swp38 N/A 9216 Default
DN swp39 N/A 9216 Default
DN swp40 N/A 9216 Default
DN swp41 N/A 9216 Default
DN swp42 N/A 9216 Default
DN swp43 N/A 9216 Default
DN swp44 N/A 9216 Default
DN swp45 N/A 9216 Default
DN swp46 N/A 9216 Default
DN swp47 N/A 9216 Default
DN swp48 N/A 9216 Default
DN swp49 N/A 9216 Default
DN swp50 N/A 9216 Default
DN swp51 N/A 9216 Default
DN swp52 N/A 9216 Default
DN swp53 N/A 9216 Default
DN swp54 N/A 9216 Default
DN swp55 N/A 9216 Default
DN swp56 N/A 9216 Default
DN swp57 N/A 9216 Default
DN swp58 N/A 9216 Default
DN swp59 N/A 9216 Default
DN swp60 N/A 9216 Default
UP mgmt N/A 65575 VRF IP: 127.0.0.1/8
mgmt IP: ::1/128
cumulus@cumulus:mgmt:~$ nv show interface
Interface State Speed MTU Type Remote Host Remote Port Summary
--------- ----- ----- ----- -------- ----------- ----------- -----------…
eth0 down 1500 eth
lo up 65536 loopback IP Address:
127.0.0.1/8
IP Address:
::1/128
mgmt up 65575 vrf IP Address:
127.0.0.1/8
IP Address:
::1/128
swp1 down 9216 swp
swp2 down 9216 swp
swp3 down 9216 swp
swp4 down 9216 swp
swp5 down 9216 swp
swp6 down 9216 swp
swp7 down 9216 swp
swp8 down 9216 swp
swp9 down 9216 swp
swp10 down 9216 swp
swp11 down 9216 swp
swp12 down 9216 swp
swp13 down 9216 swp
swp14 down 9216 swp
swp15 down 9216 swp
swp16 down 9216 swp
swp17 down 9216 swp
swp18 down 9216 swp
swp19 down 9216 swp
swp20 down 9216 swp
swp21 down 9216 swp
swp22 down 9216 swp
swp23 down 9216 swp
swp24 down 9216 swp
swp25 down 9216 swp
swp26 down 9216 swp
swp27 down 9216 swp
swp28 down 9216 swp
swp29 down 9216 swp
swp30 down 9216 swp
swp31 down 9216 swp
swp32 down 9216 swp
swp33 down 9216 swp
swp34 down 9216 swp
swp35 down 9216 swp
swp36 down 9216 swp
swp37 down 9216 swp
swp38 down 9216 swp
swp39 down 9216 swp
swp40 down 9216 swp
swp41 down 9216 swp
swp42 down 9216 swp
swp43 down 9216 swp
swp44 down 9216 swp
swp45 down 9216 swp
swp46 down 9216 swp
swp47 down 9216 swp
swp48 down 9216 swp
swp49 down 9216 swp
swp50 down 9216 swp
swp51 down 9216 swp
swp52 down 9216 swp
swp53 down 9216 swp
swp54 down 9216 swp
swp55 down 9216 swp
swp56 down 9216 swp
swp57 down 9216 swp
swp58 down 9216 swp
swp59 down 9216 swp
swp60 down 9216 swp
cumulus@cumulus:mgmt:~$ sudo su
We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:
#1) Respect the privacy of others.
#2) Think before you type.
#3) With great power comes great responsibility.
[sudo] password for cumulus:
root@cumulus:mgmt:/home/cumulus# l1-show swp1
l1-show swp12
l1-show swp13
l1-show swp14
l1-show swp15
l1-show swp16
l1-show swp17
l1-show swp18
l1-show swp19
l1-show swp20
l1-show swp21
l1-show swp22
l1-show swp23
l1-show swp24
l1-show swp25
l1-show swp26
l1-show swp27
l1-show swp28
l1-show swp29
l1-show swp30
l1-show swp31
l1-show swp32
l1-show swp33
l1-show swp34
l1-show swp35
l1-show swp36
l1-show swp37
l1-show swp38
l1-show swp39
l1-show swp40
l1-show swp41
l1-show swp42
l1-show swp43
l1-show swp44
l1-show swp45
l1-show swp46
l1-show swp47
l1-show swp48
l1-show swp49
l1-show swp50
l1-show swp51
l1-show swp52
l1-show swp53
l1-show swp54
l1-show swp55
l1-show swp56
l1-show swp57
l1-show swp58
l1-show swp59
l1-show swp60
Port: swp1
Module Info
Vendor Name: None PN: None
Identifier: None Type:
Configured State
Admin: Admin Up Speed: Unknown! MTU: 9216
Autoneg: On FEC: Auto
Operational State
Link Status: Kernel: Down Hardware: Down
Speed: Kernel: Unknown! Hardware: N/A
Autoneg: On (Autodetect enabld) FEC: None (down)
TX Power (mW): None
RX Power (mW): None
Topo File Neighbor: None, None
LLDP Neighbor: None, None
Port Hardware State:
Compliance Code: N/A
Cable Type: N/A
Speed: N/A Autodetect: Enabled
Eyes: N/A Grade: 0
Troubleshooting Info: Cable is unplugged.
It seems to me that there are no problems, just need to check the status of port by connecting the host.
Hi Alex,
Can you gather the output of sudo l1-show swp49 ? I see the output for swp1 but not the QSFP ports (swp49-60).
If you don’t see anything in the l1-show for swp49, can you plug in an optic and without connecting a cable, see if the LEDs change? If they do not change, try connecting the cable to another port on the same or different switch.
Hello Rmckenna,
u38.txt (60.6 KB)
root@cumulus:mgmt:/home/cumulus# l1-show swp49
Port: swp49
Module Info
Vendor Name: None PN: None
Identifier: None Type:
Configured State
Admin: Admin Up Speed: Unknown! MTU: 9216
Autoneg: On FEC: Auto
Operational State
Link Status: Kernel: Down Hardware: Down
Speed: Kernel: Unknown! Hardware: N/A
Autoneg: On (Autodetect enabld) FEC: None (down)
TX Power (mW): None
RX Power (mW): None
Topo File Neighbor: None, None
LLDP Neighbor: None, None
Port Hardware State:
Compliance Code: N/A
Cable Type: N/A
Speed: N/A Autodetect: Enabled
Eyes: N/A, N/A, N/A, N/A Grade: 0, 0, 0, 0
Troubleshooting Info: Cable is unplugged.
None
root@cumulus:mgmt:/home/cumulus# l1-show swp50
Port: swp50
Module Info
Vendor Name: None PN: None
Identifier: None Type:
Configured State
Admin: Admin Up Speed: Unknown! MTU: 9216
Autoneg: On FEC: Auto
Operational State
Link Status: Kernel: Down Hardware: Down
Speed: Kernel: Unknown! Hardware: N/A
Autoneg: On (Autodetect enabld) FEC: None (down)
TX Power (mW): None
RX Power (mW): None
Topo File Neighbor: None, None
LLDP Neighbor: None, None
Port Hardware State:
Compliance Code: N/A
Cable Type: N/A
Speed: N/A Autodetect: Enabled
Eyes: N/A, N/A, N/A, N/A Grade: 0, 0, 0, 0
Troubleshooting Info: Cable is unplugged.
None
Hi Community,
Is there anyone here who encountered the same problem on the indication?
hi Alex,
What was the result after you plugged in the optics/cables?
Hi Rmckenna,
I’m not yet able to connect to the host to test.
So I see that the logs look good?
Hi Rmckenna,
Connected cable both ends into ports, looks good, if the cable is disconnected the amber indication returns…
cumulus login: cumulus
Password:
cumulus@cumulus:mgmt:~$ nv show interface swp49
operational applied
type swp swp
[acl]
evpn
multihoming
uplink off
lldp
dcbx-ets-config-tlv off
dcbx-ets-recomm-tlv off
dcbx-pfc-tlv off
[neighbor] cumulus
ptp
enable off
router
adaptive-routing
enable off
ospf
enable off
ospf6
enable off
pbr
[map]
pim
enable off
synce
enable off
ip
igmp
enable off
ipv4
forward on
ipv6
enable on
forward on
neighbor-discovery
enable on
[dnssl]
home-agent
enable off
[prefix]
[rdnss]
router-advertisement
enable on
fast-retransmit on
hop-limit 64
interval 600000
interval-option off
lifetime 1800
managed-config off
other-config off
reachable-time 0
retransmit-time 0
router-preference medium
vrrp
enable off
vrf default
[gateway]
link
auto-negotiate on on
duplex full full
speed 100G auto
fec auto auto
mtu 9216 9216
[breakout] 1x
state up up
stats
carrier-transitions 2
in-bytes 2.52 KB
in-drops 0
in-errors 0
in-pkts 14
out-bytes 2.52 KB
out-drops 0
out-errors 0
out-pkts 14
mac 9c:05:91:32:6b:10
pluggable
identifier QSFP28
vendor-name Mellanox
vendor-pn MFA1A00-C010
vendor-rev B3
vendor-sn MT2311FT00443
ifindex 52
cumulus@cumulus:mgmt:~$ nv show interface swp57
operational applied
type swp swp
[acl]
evpn
multihoming
uplink off
lldp
dcbx-ets-config-tlv off
dcbx-ets-recomm-tlv off
dcbx-pfc-tlv off
[neighbor] cumulus
ptp
enable off
router
adaptive-routing
enable off
ospf
enable off
ospf6
enable off
pbr
[map]
pim
enable off
synce
enable off
ip
igmp
enable off
ipv4
forward on
ipv6
enable on
forward on
neighbor-discovery
enable on
[dnssl]
home-agent
enable off
[prefix]
[rdnss]
router-advertisement
enable on
fast-retransmit on
hop-limit 64
interval 600000
interval-option off
lifetime 1800
managed-config off
other-config off
reachable-time 0
retransmit-time 0
router-preference medium
vrrp
enable off
vrf default
[gateway]
link
auto-negotiate on on
duplex full full
speed 100G auto
fec auto auto
mtu 9216 9216
[breakout] 1x
state up up
stats
carrier-transitions 2
in-bytes 2.52 KB
in-drops 0
in-errors 0
in-pkts 14
out-bytes 2.52 KB
out-drops 0
out-errors 0
out-pkts 14
mac 9c:05:91:32:6b:48
pluggable
identifier QSFP28
vendor-name Mellanox
vendor-pn MFA1A00-C010
vendor-rev B3
vendor-sn MT2311FT00443
ifindex 60
cumulus@cumulus:mgmt:~$
Hi Alex,
Thanks for the info! The ports don’t seem to have an issue coming online but that’s still strange. I don’t see anything wrong with the port from a software level, and no errors logged from hardware that we can see. You should open a support case for this issue so we can analyze the logs deeper and possibly RMA the switch if it’s a hardware error with the port LEDs.
