Switchd takes a long time to restart

Infrastructure & Networking Cumulus Linux System Configuration & Management
1

HI all. On startup commands “systemctl restart switchd”, system freezes for 3 minutes.
Build… Cumulus Linux 4.2.0
ASIC… Mellanox Spectrum MT52132
Product Name… MSN2410
2024-07-29T17:00:08.885258+03:00 cumulus systemd[1]: Started Cumulus Linux switch port setup.
2024-07-29T17:00:08.893552+03:00 cumulus systemd[1]: Starting Cumulus Linux Switch Daemon…
2024-07-29T17:00:09.254197+03:00 cumulus kernel: [973050.825291] sx_core: Successfully added path I2C to device 1
2024-07-29T17:00:09.254215+03:00 cumulus kernel: [973050.825298] sx_dpt_find_pci_dev(): lookup for sx_pci_dev_id: 0x100 pci addr 0:1:0
2024-07-29T17:00:09.254219+03:00 cumulus kernel: [973050.825300] sx_dpt_find_pci_dev(): CHECK vendor: 0x15b3 , device : 0xcb84 ,pci addr 3:0:0
2024-07-29T17:00:09.254222+03:00 cumulus kernel: [973050.825302] sx_core: Successfully added path PCIE to device 1 pci_dev_id 256
2024-07-29T17:00:09.254224+03:00 cumulus kernel: [973050.825304] sx_core: Successfully added path PCIE to device 1
2024-07-29T17:00:09.254226+03:00 cumulus kernel: [973050.825312] sx_core: Successfully set CR_ACCESS path for device 1 to PCIE
2024-07-29T17:00:09.254228+03:00 cumulus kernel: [973050.825322] sx_core: ioctl set pci profile called
2024-07-29T17:00:09.254230+03:00 cumulus kernel: [973050.825325] sx_core: ioctl set pci profile: set cqe version (1)
2024-07-29T17:00:09.258199+03:00 cumulus kernel: [973050.829316] sx_core: ioctl enable_swid called with swid 0
2024-07-29T17:00:09.322178+03:00 cumulus kernel: [973050.893967] sx_netdev: Net Device for swid 0 was already allocated
2024-07-29T17:00:09.340170+03:00 cumulus kernel: [973050.911621] mlxsw_minimal 2-0048: Firmware revision: 13.2007.1292
2024-07-29T17:00:09.340202+03:00 cumulus kernel: [973050.911669] i2c i2c-2: new_device: Instantiated device mlxsw_minimal at 0x48
2024-07-29T17:00:09.351929+03:00 cumulus hw-management: FAN speed is set to full speed
2024-07-29T17:00:18.393428+03:00 cumulus : /usr/sbin/smond : : Fan1(Fan Tray 1): state changed from ABSENT to OK
2024-07-29T17:00:18.460581+03:00 cumulus : /usr/sbin/smond : : Fan2(Fan Tray 1): state changed from ABSENT to OK
2024-07-29T17:00:18.527333+03:00 cumulus : /usr/sbin/smond : : Fan3(Fan Tray 2): state changed from ABSENT to OK
2024-07-29T17:00:18.592455+03:00 cumulus : /usr/sbin/smond : : Fan4(Fan Tray 2): state changed from ABSENT to OK
2024-07-29T17:00:18.656610+03:00 cumulus : /usr/sbin/smond : : Fan5(Fan Tray 3): state changed from ABSENT to OK
2024-07-29T17:00:18.722579+03:00 cumulus : /usr/sbin/smond : : Fan6(Fan Tray 3): state changed from ABSENT to OK
2024-07-29T17:00:18.788226+03:00 cumulus : /usr/sbin/smond : : Fan7(Fan Tray 4): state changed from ABSENT to OK
2024-07-29T17:00:18.854973+03:00 cumulus : /usr/sbin/smond : : Fan8(Fan Tray 4): state changed from ABSENT to OK
2024-07-29T17:00:18.970311+03:00 cumulus : /usr/sbin/smond : : Temp6(Asic Temp Sensor): state changed from ABSENT to OK
2024-07-29T17:03:25.249008+03:00 cumulus systemd[1]: Started Cumulus Linux Switch Daemon.
2024-07-29T17:03:25.255540+03:00 cumulus CRON[4501]: (root) END (systemctl restart switchd)
2024-07-29T17:03:25.256402+03:00 cumulus systemd[1]: Starting Cumulus Linux acltool…
2024-07-29T17:03:25.257422+03:00 cumulus CRON[4501]: pam_unix(cron:session): session closed for user root
2024-07-29T17:03:25.257795+03:00 cumulus systemd[1]: Condition check resulted in Cumulus Linux Port Watch Event Daemon being skipped.
2024-07-29T17:03:25.270071+03:00 cumulus mstpd: set_if_up: Port swp7 : up
2024-07-29T17:03:25.276149+03:00 cumulus mstpd: set_if_up: Port swp49 : up
2024-07-29T17:03:25.282117+03:00 cumulus mstpd: MSTP_OUT_set_state: bridge:swp49:0 entering learning state(Root)
2024-07-29T17:03:25.282298+03:00 cumulus mstpd: MSTP_OUT_set_state: bridge:swp49:0 entering forwarding state(Root)
2024-07-29T17:03:25.283101+03:00 cumulus mstpd: MSTP_OUT_set_state: bridge:swp7:0 entering learning state(Designated)
2024-07-29T17:03:25.287906+03:00 cumulus mstpd: MSTP_OUT_set_state: bridge:swp7:0 entering forwarding state(Designated)
2024-07-29T17:03:25.288119+03:00 cumulus mstpd: MSTP_OUT_flush_all_fids: bridge:swp49:0 Flushing forwarding database
2024-07-29T17:03:25.301989+03:00 cumulus mstpd: MSTP_OUT_flush_all_fids: bridge:swp49:0 Flushing forwarding database
2024-07-29T17:03:25.796790+03:00 cumulus cl-acltool[8776]: Reading rule file /etc/cumulus/acl/policy.d/00control_plane.rules …
2024-07-29T17:03:25.797131+03:00 cumulus cl-acltool[8776]: Processing rules in file /etc/cumulus/acl/policy.d/00control_plane.rules …
2024-07-29T17:03:26.008818+03:00 cumulus cl-acltool[8776]: Reading rule file /etc/cumulus/acl/policy.d/99control_plane_catch_all.rules …
2024-07-29T17:03:26.009064+03:00 cumulus cl-acltool[8776]: Processing rules in file /etc/cumulus/acl/policy.d/99control_plane_catch_all.rules …

How to remove freezing? Thanks.

2

Restarting Switchd is an intensive operation. Under some circumstances, this will be expected to take 3 minutes or more if the configuration is extremely complex with large port or vlan counts. As I reflect on the fact that the 2410 is in use, that particular switch has the smallest CPU in the fleet of switches and could be getting bogged down in CPU operations processing the switchd restart.

Without understanding more about your situation/config it’s hard to say that 3 minutes is unexpected for this device.

3

The configuration is minimal, IP address, VLAN on the ports and that’s it. l2 level. 1 switch connected.

4

On another switch, the same one, the service takes 30 seconds to reboot. But the truth is Cumulus 4.3 version