ConnectX PCIe Slot Power Message

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

NVIDIA Report.docx (13.5 KB)
Hello NVIDIA Team,

We have completed a full health audit of a dual NVIDIA DGX Spark cluster and would like engineering clarification regarding a ConnectX power-related message.

Environment:

  • Dual NVIDIA DGX Spark systems

  • Dual-rail RoCE configured and operational

  • NCCL operational

  • torchrun operational

  • Distributed workloads operational

Validated Results:

  • Health Audit completed successfully

  • Phase 1 through Phase 10: PASS

  • Post-reboot validation: PASS

  • Recovery validation: PASS

  • Thermal stability: PASS

  • Power stability: PASS

  • No XID errors observed

  • No NVRM failures observed

Performance Baseline:

  • Sustained dual-rail NCCL throughput: 23.2053 GB/s

  • No measurable regression after reboot

Observed Message:

mlx5_core:
Detected insufficient power on the PCIe slot (27W)

This message is observed on both DGX Spark systems.

Observed Impact:

  • No demonstrated impact on NCCL

  • No demonstrated impact on RoCE

  • No demonstrated impact on torchrun

  • No demonstrated impact on distributed workloads

  • Systems remain fully operational

Questions:

  1. Is this expected behavior on DGX Spark systems?

  2. Is this a known firmware or platform reporting artifact?

  3. Is the ConnectX adapter operating in any reduced-power or degraded mode?

  4. Is any functionality disabled because of this condition?

  5. Is any firmware, BIOS, BMC, ConnectX, or platform update recommended?

  6. Can NVIDIA provide the official interpretation of this message for DGX Spark customers?

We are intentionally preserving the currently validated operational baseline and would like guidance before making any configuration or firmware changes.

Thank you.
PS, For more information and assistance, please see the attached file

2

That message has been discussed before and is considered benign

3

Thank you for letting me know.

4

hi @giles8

Hi

I have a similar log in dmesg onto 2 DGX spark recently received

[96075.102746] mlx5_core 0000:01:00.1: mlx5_pcie_event:322:(pid 14525): PCIe slot power capability was not advertised.
[96075.756987] mlx5_core 0000:01:00.1: mlx5_pcie_event:326:(pid 14203): Detected insufficient power on the PCIe slot (27W).
[96075.782486] mlx5_core 0000:01:00.0: mlx5_pcie_event:326:(pid 14203): Detected insufficient power on the PCIe slot (27W).
[96075.782612] mlx5_core 0002:01:00.0: mlx5_pcie_event:326:(pid 14720): Detected insufficient power on the PCIe slot (27W).
[96076.463349] mlx5_core 0002:01:00.1: mlx5_pcie_event:326:(pid 14666): Detected insufficient power on the PCIe slot (27W).

and iperf3 test shows a transmit up to 15 Gbps and receive at 105Gbps

Where on 2 others olders DGX sparks, which do not have this “power detection” can transmit and receive at 105Gbps.

Do you have more details on the issue or logs

5

Don’t rely on iperf3, use the benchmarking procedure here: spark-vllm-docker/docs/NETWORKING.md at main · eugr/spark-vllm-docker · GitHub

It uses ib_write_bw (you set one as a receiver and the other as a sender).

I get the:

[ 3.408559] mlx5_core 0002:01:00.1: mlx5_pcie_event:326:(pid 163): Detected insufficient power on the PCIe slot (27W).

Message, but I still get:

bytes iterations BW peak[Gb/sec] BW average[Gb/sec] MsgRate[Mpps]

65536 20000 111.55 111.54 0.212739

Note that the 111.55Gbps is the test only on 1 channel, and you have 2 channels, so can expect full throughput.

There are many reasons why iperf3 will not be able to accurately determine the line speed.