One of Four DGX Sparks Shows ~35% Lower NCCL Bandwidth — Can't Figure Out Why

Ok. The firmware for the mlx NICs was’nt changed then:

From your post in December 2025.

ASUS boxes did receive firmware updates also, but I noticed that there was no hint for the mlx NICs. So I was wondering if these updates were incomplete. But I assume that a different part was updated that caused the effect of bandwitdh degradation.

This what I got with my ASUS firmware update bundle:

  • GX10 Embedded Controller Firmware
  • GX10 SoC FW System
  • GX10 USB-C PD FW Controller
root@vroomfondel:~# fwupdmgr get-history
ASUSTeK COMPUTER INC. GX10
│
├─Embedded Controller:
│ │   Previous version:   0x02000003
│ │   Update State:       Success
│ │   Last modified:      2026-03-14 10:19
│ │   Device Flags:       • Internal device
│ │                       • Updatable
│ │                       • System requires external power source
│ │                       • Supported on remote server
│ │                       • Needs a reboot after installation
│ │                       • Reported to remote server
│ │                       • Device is usable for the duration of the update
│ │                       • Signed Payload
│ │
│ └─GX10 Embedded Controller Update:
│       New version:      0x02000004
│       Remote ID:        lvfs
│       Release ID:       135862
│       Summary:          GX10 Embedded Controller Firmware Update
│       License:          Proprietary
│       Size:             518.9 kB
│       Created:          2026-01-29
│       Urgency:          High
│       Tested by Asus:
│         Tested:         2026-01-29
│         Distribution:   ubuntu 24.04
│         Old version:    0x00000001
│         Version[fwupd]: 1.9.30
│       Vendor:           Asus
│       Duration:         30 seconds
│       Release Flags:    • Trusted metadata
│                         • Tested by trusted vendor
│       Description:
│       This update improves the performance and stability of the Device.
│
│       • Integrated in GA2.0 OTA2 RC3.
│       Checksum:         a3bd27ec2d97a97b9e8e2bf3087fcb6e8ef23666236c3ae41ac564978e86a99f
│
├─UEFI Device Firmware:
│ │   Previous version:   0x03000004
│ │   Update State:       Success
│ │   Last modified:      2026-03-14 10:19
│ │   Device Flags:       • Internal device
│ │                       • Updatable
│ │                       • System requires external power source
│ │                       • Supported on remote server
│ │                       • Needs a reboot after installation
│ │                       • Reported to remote server
│ │                       • Device is usable for the duration of the update
│ │                       • Signed Payload
│ │
│ └─GX10 SoC FW System Update:
│       New version:      0x03000005
│       Remote ID:        lvfs
│       Release ID:       135915
│       Summary:          GX10 SoC Firmware Update
│       License:          Proprietary
│       Size:             30.4 MB
│       Created:          2026-01-30
│       Urgency:          High
│       Tested by Asus:
│         Tested:         2026-01-30
│         Distribution:   ubuntu 24.04
│         Old version:    0x03000004
│         Version[fwupd]: 1.9.30
│       Vendor:           Asus
│       Duration:         30 seconds
│       Release Flags:    • Trusted metadata
│                         • Tested by trusted vendor
│       Description:
│       This update improves the performance and stability of the Device.
│
│       • Upgrade GA2.0 OTA2 RC3.
│       Checksum:         0544e24a7450c762312939191067269b4dbde673e398988e53ddefeed915af0e
│
└─UEFI Device Firmware:
  │   Previous version:   0x00000500
  │   Update State:       Success
  │   Last modified:      2026-03-14 10:44
  │   Device Flags:       • Internal device
  │                       • Updatable
  │                       • System requires external power source
  │                       • Supported on remote server
  │                       • Needs a reboot after installation
  │                       • Reported to remote server
  │                       • Device is usable for the duration of the update
  │                       • Signed Payload
  │
  └─GX10 USB-C PD FW Controller Update:
        New version:      0x00000507
        Remote ID:        lvfs
        Release ID:       133405
        Summary:          GX10 USB-C PD FW Firmware Update
        License:          Proprietary
        Size:             1.1 MB
        Created:          2025-12-09
        Urgency:          High
        Tested by Asus:
          Tested:         2026-01-30
          Distribution:   ubuntu 24.04
          Old version:    0x00000500
          Version[fwupd]: 1.9.30
        Vendor:           Asus
        Duration:         30 seconds
        Release Flags:    • Trusted metadata
                          • Tested by trusted vendor
        Description:
        This update improves the performance and stability of the Device.

        • Isolated the SBU mux when the U4_RDY signal is low.
        • Added a workaround to the MSI Dongle EXIT_DP_MODE issue.
        Checksum:         c0044b32557fd69a19cc05838a7d62629df3edf003f787b4dacd990dd35c295f

I need to do a proper speed test to verify if this update was successful.