One of Four DGX Sparks Shows ~35% Lower NCCL Bandwidth — Can't Figure Out Why

Ok. The firmware for the mlx NICs was’nt changed then:

From your post in December 2025.

ASUS boxes did receive firmware updates also, but I noticed that there was no hint for the mlx NICs. So I was wondering if these updates were incomplete. But I assume that a different part was updated that caused the effect of bandwitdh degradation.

This what I got with my ASUS firmware update bundle:

  • GX10 Embedded Controller Firmware
  • GX10 SoC FW System
  • GX10 USB-C PD FW Controller
root@vroomfondel:~# fwupdmgr get-history
ASUSTeK COMPUTER INC. GX10
│
├─Embedded Controller:
│ │   Previous version:   0x02000003
│ │   Update State:       Success
│ │   Last modified:      2026-03-14 10:19
│ │   Device Flags:       • Internal device
│ │                       • Updatable
│ │                       • System requires external power source
│ │                       • Supported on remote server
│ │                       • Needs a reboot after installation
│ │                       • Reported to remote server
│ │                       • Device is usable for the duration of the update
│ │                       • Signed Payload
│ │
│ └─GX10 Embedded Controller Update:
│       New version:      0x02000004
│       Remote ID:        lvfs
│       Release ID:       135862
│       Summary:          GX10 Embedded Controller Firmware Update
│       License:          Proprietary
│       Size:             518.9 kB
│       Created:          2026-01-29
│       Urgency:          High
│       Tested by Asus:
│         Tested:         2026-01-29
│         Distribution:   ubuntu 24.04
│         Old version:    0x00000001
│         Version[fwupd]: 1.9.30
│       Vendor:           Asus
│       Duration:         30 seconds
│       Release Flags:    • Trusted metadata
│                         • Tested by trusted vendor
│       Description:
│       This update improves the performance and stability of the Device.
│
│       • Integrated in GA2.0 OTA2 RC3.
│       Checksum:         a3bd27ec2d97a97b9e8e2bf3087fcb6e8ef23666236c3ae41ac564978e86a99f
│
├─UEFI Device Firmware:
│ │   Previous version:   0x03000004
│ │   Update State:       Success
│ │   Last modified:      2026-03-14 10:19
│ │   Device Flags:       • Internal device
│ │                       • Updatable
│ │                       • System requires external power source
│ │                       • Supported on remote server
│ │                       • Needs a reboot after installation
│ │                       • Reported to remote server
│ │                       • Device is usable for the duration of the update
│ │                       • Signed Payload
│ │
│ └─GX10 SoC FW System Update:
│       New version:      0x03000005
│       Remote ID:        lvfs
│       Release ID:       135915
│       Summary:          GX10 SoC Firmware Update
│       License:          Proprietary
│       Size:             30.4 MB
│       Created:          2026-01-30
│       Urgency:          High
│       Tested by Asus:
│         Tested:         2026-01-30
│         Distribution:   ubuntu 24.04
│         Old version:    0x03000004
│         Version[fwupd]: 1.9.30
│       Vendor:           Asus
│       Duration:         30 seconds
│       Release Flags:    • Trusted metadata
│                         • Tested by trusted vendor
│       Description:
│       This update improves the performance and stability of the Device.
│
│       • Upgrade GA2.0 OTA2 RC3.
│       Checksum:         0544e24a7450c762312939191067269b4dbde673e398988e53ddefeed915af0e
│
└─UEFI Device Firmware:
  │   Previous version:   0x00000500
  │   Update State:       Success
  │   Last modified:      2026-03-14 10:44
  │   Device Flags:       • Internal device
  │                       • Updatable
  │                       • System requires external power source
  │                       • Supported on remote server
  │                       • Needs a reboot after installation
  │                       • Reported to remote server
  │                       • Device is usable for the duration of the update
  │                       • Signed Payload
  │
  └─GX10 USB-C PD FW Controller Update:
        New version:      0x00000507
        Remote ID:        lvfs
        Release ID:       133405
        Summary:          GX10 USB-C PD FW Firmware Update
        License:          Proprietary
        Size:             1.1 MB
        Created:          2025-12-09
        Urgency:          High
        Tested by Asus:
          Tested:         2026-01-30
          Distribution:   ubuntu 24.04
          Old version:    0x00000500
          Version[fwupd]: 1.9.30
        Vendor:           Asus
        Duration:         30 seconds
        Release Flags:    • Trusted metadata
                          • Tested by trusted vendor
        Description:
        This update improves the performance and stability of the Device.

        • Isolated the SBU mux when the U4_RDY signal is low.
        • Added a workaround to the MSI Dongle EXIT_DP_MODE issue.
        Checksum:         c0044b32557fd69a19cc05838a7d62629df3edf003f787b4dacd990dd35c295f

I need to do a proper speed test to verify if this update was successful.

well, there is this new thing in the version string: (NVD0000000087)

My 8 units are all updated.

Asus Ascent

│ Current version: ERFM12.0

├─Embedded Controller:

│ Current version: 0x02000004

│ Update State: Success

├─MT2910 Family [ConnectX-7]:

│ Current version: 01

├─MT2910 Family [ConnectX-7]:

│ Current version: 01

├─MT2910 Family [ConnectX-7]:

│ Current version: 01

├─MT2910 Family [ConnectX-7]:

│ Current version: 01

├─UEFI Device Firmware:

│ Current version: 0x03000005

│ Update State: Success

├─UEFI Device Firmware:

│ Current version: 0x00000507

│ Update State: Success

└─UEFI dbx:

  Current version:    20230501

I have 4xDGX Spark FE’s. I’d love some help determining whether or not I received an update for this issue. I executed the manual updates, then I executed the dashboard updates via the DGX Dashboard. I unplugged all power supplies from both the units and the power plugs. Here is the output I have from fwupdmgr get-history, etc. Are there any details on how/what the fix was? Was it actually hardware/firmware or software/driver related or both? Unfortunately I cannot find release notes. Any help/guidance would be appreciated.

NVIDIA NVIDIA_DGX_Spark
│
├─Embedded Controller:
│ │ Device ID:
│ │ Previous version: 0x00000001
│ │ Update State: Success
│ │ Last modified: 2026-02-17 07:01
│ │ GUID:
│ │ Device Flags: • Internal device
│ │ • Updatable
│ │ • System requires external power source
│ │ • Supported on remote server
│ │ • Needs a reboot after installation
│ │ • Reported to remote server
│ │ • Device is usable for the duration of the update
│ │ • Signed Payload
│ │
│ └─DGX Spark EC FW Embedded Controller Update:
│ New version: 0x02004e12
│ Remote ID: lvfs
│ Release ID: 135056
│ Summary: DGX Spark Embedded Controller Firmware Update
│ License: Proprietary
│ Size: 519.3 kB
│ Created: 2026-01-13
│ Urgency: High
│ Vendor: NVIDIA
│ Duration: 30 seconds
│ Release Flags: • Trusted metadata
│ Description:
│ This update improves the performance and stability of the Embedded Controller in DGX Spark
│ Checksum: 55f0d41a5a93b39bb6a0afa3765b6b746ddcc667f4e9047972f02f2d57cc1073
│
├─UEFI Device Firmware:
│ │ Device ID:
│ │ Previous version: 0x00000500
│ │ Update State: Success
│ │ Last modified: 2026-03-08 21:31
│ │ GUID:
│ │ Device Flags: • Internal device
│ │ • Updatable
│ │ • System requires external power source
│ │ • Supported on remote server
│ │ • Needs a reboot after installation
│ │ • Reported to remote server
│ │ • Device is usable for the duration of the update
│ │ • Signed Payload
│ │
│ └─DGX Spark USB-C PD FW Controller Update:
│ New version: 0x00000507
│ Remote ID: lvfs
│ Release ID: 133185
│ Summary: DGX Spark USB-C Power Delivery Controller Firmware Update
│ License: Proprietary
│ Size: 1.1 MB
│ Created: 2025-12-04
│ Urgency: High
│ Vendor: NVIDIA
│ Duration: 30 seconds
│ Release Flags: • Trusted metadata
│ Description:
│ This update improves the performance and stability of the USB Power Delivery Controller in DGX Spark for USB-C Ports
│ Checksum: 575ee2e586119a941c83659b6d73b47ffe761a64bb783668803a3625beceeb6f
│
└─UEFI Device Firmware:
│ Device ID:
│ Previous version: 0x02009418
│ Update State: Success
│ Last modified: 2026-03-16 20:50
│ GUID:
│ Device Flags: • Internal device
│ • Updatable
│ • System requires external power source
│ • Supported on remote server
│ • Needs a reboot after installation
│ • Reported to remote server
│ • Device is usable for the duration of the update
│ • Signed Payload
│
└─DGX Spark SoC FW System Update:
New version: 0x0200941a
Remote ID: lvfs
Release ID: 136978
Summary: DGX Spark SoC Firmware Update
License: Proprietary
Size: 30.4 MB
Created: 2026-02-24
Urgency: High
Tested by NVIDIA:
Tested: 2026-03-02
Distribution: ubuntu 24.04
Old version: 0x02009009
Version[fwupd]: 1.9.33
Vendor: NVIDIA
Duration: 30 seconds
Release Flags: • Trusted metadata
• Tested by trusted vendor
Description:
This update improves the performance and stability of the System-on-Chip Firmware including UEFI and GPU in DGX Spark
Checksum: d4b05b4b9455d4c139361ce3ff521b2cad53e7de90d9637f2f76419231081a2b

sudo fwupdmgr get-devices --show-all-devices 2>/dev/null | grep -E ‘├─|└─|Current version|Update State’
├─Unknown Device:
├─Embedded Controller:
│ Current version: 0x02004e12
│ Update State: Success
├─MT2910 Family [ConnectX-7]:
│ Current version: 01
├─MT2910 Family [ConnectX-7]:
│ Current version: 01
├─MT2910 Family [ConnectX-7]:
│ Current version: 01
├─MT2910 Family [ConnectX-7]:
│ Current version: 01
├─MZALC4T0HBL1-00B07:
│ Current version: NXHB202Q
├─UEFI Device Firmware:
│ Current version: 0x0200941a
│ Update State: Success
├─UEFI Device Firmware:
│ Current version: 0x00000507
│ Update State: Success
└─UEFI dbx:
Current version: 20230501

[ 3.420549] mlx5_core 0002:01:00.0: firmware version: 28.45.4028
[ 3.420571] mlx5_core 0002:01:00.0: 126.028 Gb/s available PCIe bandwidth (32.0 GT/s PCIe x4 link)
[ 4.384849] mlx5_core 0002:01:00.1: mlx5_pcie_event:326:(pid 433): Detected insufficient power on the PCIe slot (27W).

Doesn’t look like you have that update.

Run:

sudo fwupdmgr refresh --force
sudo fwupdmgr upgrade

I can’t seem to get it to pull anything new. Thanks for the help!

sudo fwupdmgr refresh --force
sudo fwupdmgr upgrade
Updating lvfs
Downloading… [********************************** ]
Successfully downloaded new metadata: 3 local devices supported
Devices with no available firmware updates:
• MZALC4T0HBL1-00B07
• UEFI dbx
Devices with the latest available firmware version:
• Embedded Controller
• UEFI Device Firmware
• UEFI Device Firmware

To check if you have received the update, run sudo dmidecode -t 45 | grep -A2 "UEFI" and check if you have UEFI Firmware Version 1.107.26

To see the latest published firmware use LVFS: Search Results and if your Spark is behind run fwupdmgr upgrade

Firmware Component Name: UEFI
Firmware Version: R:1.107.26
Firmware ID: Not Specified

Thank you

Thank you