ConnectX-7 throughput drops to 25 Gbps after hot-plug/cable reconnection on Gigabyte GB10 (AI TOP ATOM)

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

Hello.

We setup a cluster of DGX Sparks connected through a MicroTik CRS804 DDQ switch, with the following topology:

  • Nodes: 4x Gigabyte GB10
  • NIC: NVIDIA ConnectX-7 (Firmware: 28.45.4028)
  • Switch: MikroTik CRS804-4XQ-IN (running at 200Gbps per port)
  • Switch configuration: The interfaces are fixed at 200G speeds, because if left at “autonegotiation”, they default at 100G.
  • Topology: Each node is directly connected via a 200Gbps cable to the switch.

When booting up normally, a 4-node nccl-tests all_gather operation performs perfectly, hitting the expected line-rate maximum of 192 Gbps (~24 GB/s).

However, if we physically unplug a network cable from one of the nodes and immediately reconnect it:

  • The 4-node all_gather throughput permanently drops to 25 Gbps (~3.2 GB/s).
  • If we isolate the remaining 3 nodes (excluding the one where the cable was plugged/unplugged), the all_gather performance returns to the expected 192 Gbps.
  • The degraded node only recovers its full performance after a complete system reboot.

The frustrating part is that the physical and link layer tools report absolutely no errors. Running mlxlink on the degraded node before and after the hot-plug shows an identical, successful auto-negotiation status pointing to a fully operational 200G link over 4 lanes:

root@aitopatom-a092:~# mlxlink -d /dev/mst/mt4129_pciconf0 -p 1

Operational Info
----------------
State                              : Active 
Physical state                     : ETH_AN_FSM_ENABLE 
Speed                              : 200G 
Width                              : 4x 
FEC                                : Standard_RS-FEC - (544,514) 
Loopback Mode                      : No Loopback 
Auto Negotiation                   : ON 

Supported Info
--------------
Enabled Link Speed (Ext.)          : 0x00003ff2 (200G_2X,200G_4X,100G_1X,100G_2X,100G_4X,50G_1X,50G_2X,40G,25G,10G,1G) 
Supported Cable Speed (Ext.)       : 0x00013ff2 (400G_4X,200G_2X,200G_4X,100G_1X,100G_2X,100G_4X,50G_1X,50G_2X,40G,25G,10G,1G) 

Troubleshooting Info
--------------------
Status Opcode                      : 0 
Group Opcode                       : N/A 
Recommendation                     : No issue was observed 

Tool Information
----------------
Firmware Version                   : 28.45.4028 
amBER Version                      : 6.4

Has anyone encountered this specific behavior with ConnectX-7 hot-plugging under Ethernet environments, or is there a known driver/firmware mismatch between this CX7 firmware version (28.45.4028) and specific top-of-rack switches when handling link-state transitions?

Does anyone have any idea why that happens? Any insight would be useful.

Thank you.

2

Anything useful reported by dmesg -wT when monitoring at the time of the disconnect/reconnect?

3

Hello. Excuse me for the late reply, but I did not have access to the cluster during the previous days.

Here is the dmesg output, starting from the moment we unplugged and re-plugged the cable.

[Wed May 27 11:44:18 2026] mlx5_core 0000:01:00.0: Port module event: module 0, Cable unplugged
[Wed May 27 11:44:18 2026] mlx5_core 0002:01:00.0: Port module event: module 0, Cable unplugged
[Wed May 27 11:44:18 2026] mlx5_core 0002:01:00.0 enP2p1s0f0np0: Link down
[Wed May 27 11:44:18 2026] mlx5_core 0000:01:00.0 enp1s0f0np0: Link down
[Wed May 27 11:44:18 2026] mlx5_core 0002:01:00.0 mlx5_2: Port: 1 Link DOWN
[Wed May 27 11:44:18 2026] mlx5_core 0000:01:00.0 mlx5_0: Port: 1 Link DOWN
[Wed May 27 11:44:19 2026] mlx5_core 0000:01:00.0: E-Switch: Unload vfs: mode(LEGACY), nvfs(0), necvfs(0), active vports(0)
[Wed May 27 11:44:19 2026] mlx5_core 0000:01:00.0: E-Switch: Disable: mode(LEGACY), nvfs(0), necvfs(0), active vports(0)
[Wed May 27 11:44:25 2026] mlx5_core 0000:01:00.0: Port module event: module 0, Cable plugged
[Wed May 27 11:44:25 2026] mlx5_core 0002:01:00.0: Port module event: module 0, Cable plugged
[Wed May 27 11:44:25 2026] mlx5_core 0000:01:00.0 enp1s0f0np0: Link up
[Wed May 27 11:44:25 2026] mlx5_core 0002:01:00.0 enP2p1s0f0np0: Link up
[Wed May 27 11:44:25 2026] mlx5_core 0002:01:00.0 mlx5_2: Port: 1 Link ACTIVE
[Wed May 27 11:44:27 2026] mlx5_core 0000:01:00.0: E-Switch: Disable: mode(LEGACY), nvfs(0), necvfs(0), active vports(0)
[Wed May 27 11:44:28 2026] mlx5_core 0000:01:00.0: E-Switch: cleanup
[Wed May 27 11:44:28 2026] mlx5_core 0000:01:00.1: E-Switch: Unload vfs: mode(LEGACY), nvfs(0), necvfs(0), active vports(0)
[Wed May 27 11:44:28 2026] mlx5_core 0000:01:00.1: E-Switch: Disable: mode(LEGACY), nvfs(0), necvfs(0), active vports(0)
[Wed May 27 11:44:38 2026] mlx5_core 0000:01:00.1: E-Switch: Disable: mode(LEGACY), nvfs(0), necvfs(0), active vports(0)
[Wed May 27 11:44:38 2026] mlx5_core 0000:01:00.1: E-Switch: cleanup
[Wed May 27 11:44:39 2026] mlx5_core 0002:01:00.0: E-Switch: Unload vfs: mode(LEGACY), nvfs(0), necvfs(0), active vports(0)
[Wed May 27 11:44:39 2026] mlx5_core 0002:01:00.0: E-Switch: Disable: mode(LEGACY), nvfs(0), necvfs(0), active vports(0)
[Wed May 27 11:44:48 2026] mlx5_core 0002:01:00.0: E-Switch: Disable: mode(LEGACY), nvfs(0), necvfs(0), active vports(0)
[Wed May 27 11:44:48 2026] mlx5_core 0002:01:00.0: E-Switch: cleanup
[Wed May 27 11:44:49 2026] mlx5_core 0002:01:00.1: E-Switch: Unload vfs: mode(LEGACY), nvfs(0), necvfs(0), active vports(0)
[Wed May 27 11:44:49 2026] mlx5_core 0002:01:00.1: E-Switch: Disable: mode(LEGACY), nvfs(0), necvfs(0), active vports(0)
[Wed May 27 11:44:57 2026] mlx5_core 0002:01:00.1: E-Switch: Disable: mode(LEGACY), nvfs(0), necvfs(0), active vports(0)
[Wed May 27 11:44:58 2026] mlx5_core 0002:01:00.1: E-Switch: cleanup
[Wed May 27 11:44:58 2026] cx7-pcie-hotplug MTKP0001:00: Cable removal
[Wed May 27 11:44:58 2026] pcieport 0000:00:00.0: AER: Correctable error message received from 0000:00:00.0
[Wed May 27 11:44:58 2026] pcieport 0002:00:00.0: AER: Correctable error message received from 0002:00:00.0
[Wed May 27 11:44:58 2026] pcieport 0000:00:00.0: PCIe Bus Error: severity=Correctable, type=Physical Layer, (Receiver ID)
[Wed May 27 11:44:58 2026] pcieport 0000:00:00.0:   device [10de:22ce] error status/mask=00000001/0000e000
[Wed May 27 11:44:58 2026] pcieport 0000:00:00.0:    [ 0] RxErr                  (First)
[Wed May 27 11:44:58 2026] pcieport 0002:00:00.0: PCIe Bus Error: severity=Correctable, type=Physical Layer, (Receiver ID)
[Wed May 27 11:44:58 2026] pcieport 0002:00:00.0:   device [10de:22ce] error status/mask=00000001/0000e000
[Wed May 27 11:44:58 2026] pcieport 0002:00:00.0:    [ 0] RxErr                  (First)
[Wed May 27 11:45:02 2026] cx7-pcie-hotplug MTKP0001:00: Cable plugin
[Wed May 27 11:45:03 2026] pcieport 0000:00:00.0: AER: Multiple Correctable error message received from 0000:ff:1f.7 (no details found
[Wed May 27 11:45:03 2026] pcieport 0000:00:00.0: AER: Multiple Uncorrectable (Fatal) error message received from 0000:ff:1f.7 (no details found
[Wed May 27 11:45:03 2026] pcieport 0002:00:00.0: AER: Multiple Correctable error message received from 0002:ff:1f.7 (no details found
[Wed May 27 11:45:03 2026] pcieport 0002:00:00.0: AER: Multiple Uncorrectable (Fatal) error message received from 0002:ff:1f.7 (no details found
[Wed May 27 11:45:05 2026] pci 0000:01:00.0: [15b3:1021] type 00 class 0x020000 PCIe Endpoint
[Wed May 27 11:45:05 2026] pci 0000:01:00.0: BAR 0 [mem 0x00000000-0x01ffffff 64bit pref]
[Wed May 27 11:45:05 2026] pci 0000:01:00.0: ROM [mem 0x00000000-0x000fffff pref]
[Wed May 27 11:45:05 2026] pci 0000:01:00.0: PME# supported from D3cold
[Wed May 27 11:45:05 2026] pci 0000:01:00.0: VF BAR 0 [mem 0x00000000-0x000fffff 64bit pref]
[Wed May 27 11:45:05 2026] pci 0000:01:00.0: VF BAR 0 [mem 0x00000000-0x007fffff 64bit pref]: contains BAR 0 for 8 VFs
[Wed May 27 11:45:05 2026] pci 0000:01:00.0: Adding to iommu group 7
[Wed May 27 11:45:05 2026] pcieport 0000:00:00.0: Max Payload Size set to  512/ 512 (was  512), Max Read Rq  512
[Wed May 27 11:45:05 2026] pci 0000:01:00.0: Max Payload Size set to  512/ 512 (was  128), Max Read Rq  512
[Wed May 27 11:45:05 2026] pci 0000:01:00.1: [15b3:1021] type 00 class 0x020000 PCIe Endpoint
[Wed May 27 11:45:05 2026] pci 0000:01:00.1: BAR 0 [mem 0x00000000-0x01ffffff 64bit pref]
[Wed May 27 11:45:05 2026] pci 0000:01:00.1: ROM [mem 0x00000000-0x000fffff pref]
[Wed May 27 11:45:05 2026] pci 0000:01:00.1: PME# supported from D3cold
[Wed May 27 11:45:05 2026] pci 0000:01:00.1: VF BAR 0 [mem 0x00000000-0x000fffff 64bit pref]
[Wed May 27 11:45:05 2026] pci 0000:01:00.1: VF BAR 0 [mem 0x00000000-0x007fffff 64bit pref]: contains BAR 0 for 8 VFs
[Wed May 27 11:45:05 2026] pci 0000:01:00.1: Adding to iommu group 8
[Wed May 27 11:45:05 2026] pcieport 0000:00:00.0: Max Payload Size set to  512/ 512 (was  512), Max Read Rq  512
[Wed May 27 11:45:05 2026] pci 0000:01:00.0: Max Payload Size set to  512/ 512 (was  512), Max Read Rq  512
[Wed May 27 11:45:05 2026] pci 0000:01:00.1: Max Payload Size set to  512/ 512 (was  128), Max Read Rq  512
[Wed May 27 11:45:05 2026] pci 0000:01:00.0: BAR 0 [mem 0x7500000000-0x7501ffffff 64bit pref]: assigned
[Wed May 27 11:45:05 2026] pci 0000:01:00.1: BAR 0 [mem 0x7502000000-0x7503ffffff 64bit pref]: assigned
[Wed May 27 11:45:05 2026] pci 0000:01:00.0: ROM [mem 0x67100000-0x671fffff pref]: assigned
[Wed May 27 11:45:05 2026] pci 0000:01:00.0: VF BAR 0 [mem 0x7504000000-0x75047fffff 64bit pref]: assigned
[Wed May 27 11:45:05 2026] pci 0000:01:00.1: ROM [mem 0x67200000-0x672fffff pref]: assigned
[Wed May 27 11:45:05 2026] pci 0000:01:00.1: VF BAR 0 [mem 0x7504800000-0x7504ffffff 64bit pref]: assigned
[Wed May 27 11:45:05 2026] mlx5_core 0000:01:00.0: enabling device (0000 -> 0002)
[Wed May 27 11:45:05 2026] mlx5_core 0000:01:00.0: firmware version: 28.45.4028
[Wed May 27 11:45:05 2026] mlx5_core 0000:01:00.0: 126.028 Gb/s available PCIe bandwidth (32.0 GT/s PCIe x4 link)
[Wed May 27 11:45:05 2026] mlx5_core 0000:01:00.0: Rate limit: 127 rates are supported, range: 0Mbps to 195312Mbps
[Wed May 27 11:45:05 2026] mlx5_core 0000:01:00.0: E-Switch: Total vports 10, per vport: max uc(128) max mc(2048)
[Wed May 27 11:45:05 2026] mlx5_core 0000:01:00.0: Flow counters bulk query buffer size increased, bulk_query_len(8)
[Wed May 27 11:45:05 2026] mlx5_core 0000:01:00.0: mlx5_pcie_event:325:(pid 928): PCIe slot power capability was not advertised.
[Wed May 27 11:45:05 2026] mlx5_core 0000:01:00.0: mlx5e: IPSec ESP acceleration enabled
[Wed May 27 11:45:05 2026] mlx5_core 0000:01:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0 enhanced)
[Wed May 27 11:45:05 2026] mlx5_core 0000:01:00.0 enp1s0f0np0: renamed from eth0
[Wed May 27 11:45:05 2026] mlx5_core 0000:01:00.1: enabling device (0000 -> 0002)
[Wed May 27 11:45:05 2026] mlx5_core 0000:01:00.1: firmware version: 28.45.4028
[Wed May 27 11:45:05 2026] mlx5_core 0000:01:00.1: 126.028 Gb/s available PCIe bandwidth (32.0 GT/s PCIe x4 link)
[Wed May 27 11:45:06 2026] mlx5_core 0000:01:00.1: Rate limit: 127 rates are supported, range: 0Mbps to 195312Mbps
[Wed May 27 11:45:06 2026] mlx5_core 0000:01:00.1: E-Switch: Total vports 10, per vport: max uc(128) max mc(2048)
[Wed May 27 11:45:06 2026] mlx5_core 0000:01:00.1: Flow counters bulk query buffer size increased, bulk_query_len(8)
[Wed May 27 11:45:06 2026] mlx5_core 0000:01:00.1: Port module event: module 1, Cable unplugged
[Wed May 27 11:45:06 2026] mlx5_core 0000:01:00.1: mlx5_pcie_event:325:(pid 11): PCIe slot power capability was not advertised.
[Wed May 27 11:45:06 2026] mlx5_core 0000:01:00.1: mlx5e: IPSec ESP acceleration enabled
[Wed May 27 11:45:06 2026] mlx5_core 0000:01:00.0 enp1s0f0np0: Link down
[Wed May 27 11:45:06 2026] mlx5_core 0000:01:00.1: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0 enhanced)
[Wed May 27 11:45:06 2026] mlx5_core 0000:01:00.1 enp1s0f1np1: renamed from eth0
[Wed May 27 11:45:06 2026] pci 0002:01:00.0: [15b3:1021] type 00 class 0x020000 PCIe Endpoint
[Wed May 27 11:45:06 2026] pci 0002:01:00.0: BAR 0 [mem 0x00000000-0x01ffffff 64bit pref]
[Wed May 27 11:45:06 2026] pci 0002:01:00.0: ROM [mem 0x00000000-0x000fffff pref]
[Wed May 27 11:45:06 2026] pci 0002:01:00.0: PME# supported from D3cold
[Wed May 27 11:45:06 2026] pci 0002:01:00.0: VF BAR 0 [mem 0x00000000-0x000fffff 64bit pref]
[Wed May 27 11:45:06 2026] pci 0002:01:00.0: VF BAR 0 [mem 0x00000000-0x007fffff 64bit pref]: contains BAR 0 for 8 VFs
[Wed May 27 11:45:06 2026] pci 0002:01:00.0: Adding to iommu group 10
[Wed May 27 11:45:06 2026] pcieport 0002:00:00.0: Max Payload Size set to  512/ 512 (was  512), Max Read Rq  512
[Wed May 27 11:45:06 2026] pci 0002:01:00.0: Max Payload Size set to  512/ 512 (was  128), Max Read Rq  512
[Wed May 27 11:45:06 2026] pci 0002:01:00.1: [15b3:1021] type 00 class 0x020000 PCIe Endpoint
[Wed May 27 11:45:06 2026] pci 0002:01:00.1: BAR 0 [mem 0x00000000-0x01ffffff 64bit pref]
[Wed May 27 11:45:06 2026] pci 0002:01:00.1: ROM [mem 0x00000000-0x000fffff pref]
[Wed May 27 11:45:06 2026] pci 0002:01:00.1: PME# supported from D3cold
[Wed May 27 11:45:06 2026] pci 0002:01:00.1: VF BAR 0 [mem 0x00000000-0x000fffff 64bit pref]
[Wed May 27 11:45:06 2026] pci 0002:01:00.1: VF BAR 0 [mem 0x00000000-0x007fffff 64bit pref]: contains BAR 0 for 8 VFs
[Wed May 27 11:45:06 2026] pci 0002:01:00.1: Adding to iommu group 11
[Wed May 27 11:45:06 2026] pcieport 0002:00:00.0: Max Payload Size set to  512/ 512 (was  512), Max Read Rq  512
[Wed May 27 11:45:06 2026] pci 0002:01:00.0: Max Payload Size set to  512/ 512 (was  512), Max Read Rq  512
[Wed May 27 11:45:06 2026] pci 0002:01:00.1: Max Payload Size set to  512/ 512 (was  128), Max Read Rq  512
[Wed May 27 11:45:06 2026] pci 0002:01:00.0: BAR 0 [mem 0x3d00000000-0x3d01ffffff 64bit pref]: assigned
[Wed May 27 11:45:06 2026] pci 0002:01:00.1: BAR 0 [mem 0x3d02000000-0x3d03ffffff 64bit pref]: assigned
[Wed May 27 11:45:06 2026] pci 0002:01:00.0: ROM [mem 0x5d100000-0x5d1fffff pref]: assigned
[Wed May 27 11:45:06 2026] pci 0002:01:00.0: VF BAR 0 [mem 0x3d04000000-0x3d047fffff 64bit pref]: assigned
[Wed May 27 11:45:06 2026] pci 0002:01:00.1: ROM [mem 0x5d200000-0x5d2fffff pref]: assigned
[Wed May 27 11:45:06 2026] pci 0002:01:00.1: VF BAR 0 [mem 0x3d04800000-0x3d04ffffff 64bit pref]: assigned
[Wed May 27 11:45:06 2026] mlx5_core 0002:01:00.0: enabling device (0000 -> 0002)
[Wed May 27 11:45:06 2026] mlx5_core 0002:01:00.0: firmware version: 28.45.4028
[Wed May 27 11:45:06 2026] mlx5_core 0002:01:00.0: 126.028 Gb/s available PCIe bandwidth (32.0 GT/s PCIe x4 link)
[Wed May 27 11:45:06 2026] mlx5_core 0000:01:00.0: Port module event: module 0, Cable plugged
[Wed May 27 11:45:06 2026] mlx5_core 0000:01:00.1: mlx5_pcie_event:329:(pid 2144): Detected insufficient power on the PCIe slot (27W).
[Wed May 27 11:45:06 2026] mlx5_core 0000:01:00.0: mlx5_pcie_event:329:(pid 928): Detected insufficient power on the PCIe slot (27W).
[Wed May 27 11:45:07 2026] mlx5_core 0002:01:00.0: Rate limit: 127 rates are supported, range: 0Mbps to 195312Mbps
[Wed May 27 11:45:07 2026] mlx5_core 0002:01:00.0: E-Switch: Total vports 10, per vport: max uc(128) max mc(2048)
[Wed May 27 11:45:07 2026] mlx5_core 0002:01:00.0: Flow counters bulk query buffer size increased, bulk_query_len(8)
[Wed May 27 11:45:07 2026] mlx5_core 0002:01:00.0: Port module event: module 0, Cable plugged
[Wed May 27 11:45:07 2026] mlx5_core 0002:01:00.0: mlx5_pcie_event:329:(pid 928): Detected insufficient power on the PCIe slot (27W).
[Wed May 27 11:45:07 2026] mlx5_core 0002:01:00.0: mlx5e: IPSec ESP acceleration enabled
[Wed May 27 11:45:07 2026] mlx5_core 0000:01:00.1 enp1s0f1np1: Link down
[Wed May 27 11:45:07 2026] mlx5_core 0000:01:00.0 enp1s0f0np0: Link up
[Wed May 27 11:45:07 2026] mlx5_core 0000:01:00.0 mlx5_0: Port: 1 Link ACTIVE
[Wed May 27 11:45:07 2026] mlx5_core 0002:01:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0 enhanced)
[Wed May 27 11:45:07 2026] mlx5_core 0002:01:00.0 enP2p1s0f0np0: renamed from eth0
[Wed May 27 11:45:07 2026] mlx5_core 0002:01:00.1: enabling device (0000 -> 0002)
[Wed May 27 11:45:07 2026] mlx5_core 0002:01:00.1: firmware version: 28.45.4028
[Wed May 27 11:45:07 2026] mlx5_core 0002:01:00.1: 126.028 Gb/s available PCIe bandwidth (32.0 GT/s PCIe x4 link)
[Wed May 27 11:45:07 2026] mlx5_core 0002:01:00.1: Rate limit: 127 rates are supported, range: 0Mbps to 195312Mbps
[Wed May 27 11:45:07 2026] mlx5_core 0002:01:00.1: E-Switch: Total vports 10, per vport: max uc(128) max mc(2048)
[Wed May 27 11:45:07 2026] mlx5_core 0002:01:00.1: Flow counters bulk query buffer size increased, bulk_query_len(8)
[Wed May 27 11:45:07 2026] mlx5_core 0002:01:00.1: Port module event: module 1, Cable unplugged
[Wed May 27 11:45:07 2026] mlx5_core 0002:01:00.1: mlx5_pcie_event:329:(pid 2328): Detected insufficient power on the PCIe slot (27W).
[Wed May 27 11:45:07 2026] mlx5_core 0002:01:00.0 enP2p1s0f0np0: Link up
[Wed May 27 11:45:07 2026] mlx5_core 0002:01:00.1: mlx5e: IPSec ESP acceleration enabled
[Wed May 27 11:45:07 2026] mlx5_core 0002:01:00.0 mlx5_2: Port: 1 Link ACTIVE
[Wed May 27 11:45:08 2026] mlx5_core 0002:01:00.1: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0 enhanced)
[Wed May 27 11:45:08 2026] mlx5_core 0002:01:00.1 enP2p1s0f1np1: renamed from eth0
[Wed May 27 11:45:08 2026] mlx5_core 0002:01:00.1 enP2p1s0f1np1: Link down

Right before the removal, we had ran the test at ~200Gbps, and after this change it fell down to 25Gbps.
(The insufficient/27W warning appeared even before removing the cable, when we were still able to achieve the 200Gbps speed…)

4

The log is very useful and I’m pretty sure will give enough information to troubleshoot this.

It looks like when you replug the cable, the device is being enumerated with inadequate power (Detected insufficient power on the PCIe slot (27W)), causing it to throttle. This situation is described here ASUS Ascent GX10 ConnectX-7: Why You’re Getting 13 Gbps Instead of 200 Gbps (And How to Fix It) | Chronara. The article suggests a command-line update procedure which I can’t really recommend but, as I believe NVIDIA have made improvements in this area, have you installed all updates available through the Dashboard (official update mechanism)?

5

See also ConnectX-7 Inter-Spark Link Capped at ~13 Gbps (Expected 200 Gbps) — PCIe Power Throttling (27W).

6

Thank you for your reply.

We have installed all the latest firmware updates. When running:

fwupdmgr refresh --force
fwupdmgr upgrade

we get the message that we have the latest possible versions everywhere.

Compared to the versions mentioned in the article from Chronara, our setup is newer:

  • Kernel version: 6.17.0-1018-nvidia (newer than the suggested 6.17.0-1014-nvidia)
  • NVIDIA driver: 580.159.03 (newer than the suggested 580.142)
  • linux-modules-580

Additionally, ethtool gives the following information

driver: mlx5_core
version: 26.01-1.0.0
firmware-version: 28.45.4028 (NVD0000000087)
bus-info: 0000:01:00.0
supports-statistics: yes
supports-test: yes
supports-priv-flags: yes

I noticed that the mlx5_core version is 26.01-1.0.0 (with firmware version 28.45.4028 / NVD0000000087), which I believe is the latest available version, referred in other forum articles.

Regarding the 27W warning, it appears even when the connection is “healthy” (operating at 200Gbps) across all 4 ports. Specifically, I see it in the dmesg log when the GB10 boots, even before moving the cable.

Overall, I suspect this has something to do with the NIC hardware. When it detects a disconnection, it automatically throttles the link for subsequent use. Since a reboot resolves the issue, our current workaround is to keep the cluster up and avoid hot-plugging cables while the system is live. 🙂 We are just left scratching our heads as to why this behavior occurs in the first place.

7

A puzzle indeed!

If the hot plug use case is more important than power saving, you could try disabling hotplug support per @aniculescu’s suggestion here:

(Maybe back up the file first, although the contents of the file are quite amusing!)

Hopefully this just disables the hot plug power saving feature… if it disables hot plug altogether then that’s obviously not going to work for you 😂