Hello,
I have just unboxed a new pair of DGX Sparks. It all went according to plan until I tried to bring up the 200Gb link between them. After some discussions with Gemini it turns out the PD bug is rearings its ugly face for the PCIe bus as well and in this case both usnits are suffering and no matter how many times I unplug and replug the power adaptors, the ConnectX-6 cards will not receive the required power:
mashie@ceres:~$ sudo dmesg | grep -i “power”
[ 0.040781] thermal_sys: Registered thermal governor ‘power_allocator’
[ 0.061036] ACPI: _SB_.P0RR: New power resource
[ 0.061077] ACPI: _SB_.R0RR: New power resource
[ 0.061115] ACPI: _SB_.P2RR: New power resource
[ 0.061152] ACPI: _SB_.R2RR: New power resource
[ 0.061191] ACPI: _SB_.P4RR: New power resource
[ 0.061228] ACPI: _SB_.R4RR: New power resource
[ 0.061266] ACPI: _SB_.P6RR: New power resource
[ 0.061303] ACPI: _SB_.R6RR: New power resource
[ 0.061342] ACPI: _SB_.P7RR: New power resource
[ 0.061380] ACPI: _SB_.R7RR: New power resource
[ 0.061419] ACPI: _SB_.P8RR: New power resource
[ 0.061455] ACPI: _SB_.R8RR: New power resource
[ 0.061494] ACPI: _SB_.P9RR: New power resource
[ 0.061532] ACPI: _SB_.R9RR: New power resource
[ 0.064737] ACPI: _SB_.USB5.RHUB.PRT2.PWFR: New power resource
[ 0.066869] ACPI: _SB_.PBRR: New power resource
[ 0.066906] ACPI: _SB_.RBRR: New power resource
[ 0.066945] ACPI: _SB_.PCRR: New power resource
[ 0.066982] ACPI: _SB_.RCRR: New power resource
[ 0.067019] ACPI: _SB_.PDRR: New power resource
[ 0.067057] ACPI: _SB_.RDRR: New power resource
[ 0.067093] ACPI: _SB_.PERR: New power resource
[ 0.067130] ACPI: _SB_.RERR: New power resource
[ 0.067168] ACPI: _SB_.PFRR: New power resource
[ 1.251579] input: Power Button as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0C:00/input/input0
[ 1.251648] ACPI: button: Power Button [PWRB]
[ 1.485219] PM: genpd: Disabling unused power domains
[ 2.623077] mlx5_core 0000:01:00.0: mlx5_pcie_event:326:(pid 165): Detected insufficient power on the PCIe slot (27W).
[ 3.235882] mlx5_core 0000:01:00.1: mlx5_pcie_event:326:(pid 364): Detected insufficient power on the PCIe slot (27W).
[ 3.802188] mlx5_core 0002:01:00.0: mlx5_pcie_event:326:(pid 404): Detected insufficient power on the PCIe slot (27W).
[ 4.382611] mlx5_core 0002:01:00.1: mlx5_pcie_event:326:(pid 12): Detected insufficient power on the PCIe slot (27W).
[ 16.061981] wlP9s9: Limiting TX power to 23 (23 - 0) dBm as advertised by 84:78:48:02:0b:13
mashie@ceres:~$ sudo lspci -vvv | grep PowerLimit
Slot #0, PowerLimit 0W; Interlock- NoCompl+
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W
Slot #2, PowerLimit 0W; Interlock- NoCompl+
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W
Slot #4, PowerLimit 0W; Interlock- NoCompl+
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W
Slot #7, PowerLimit 0W; Interlock- NoCompl+
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0W
pcilib: sysfs_read_vpd: read failed: No such device
Slot #9, PowerLimit 0W; Interlock- NoCompl+
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W
mashie@ceres:~$
mashie@vesta:~$ sudo dmesg | grep -i “power”
[ 0.042298] thermal_sys: Registered thermal governor ‘power_allocator’
[ 0.063075] ACPI: _SB_.P0RR: New power resource
[ 0.063117] ACPI: _SB_.R0RR: New power resource
[ 0.063157] ACPI: _SB_.P2RR: New power resource
[ 0.063196] ACPI: _SB_.R2RR: New power resource
[ 0.063236] ACPI: _SB_.P4RR: New power resource
[ 0.063275] ACPI: _SB_.R4RR: New power resource
[ 0.063317] ACPI: _SB_.P6RR: New power resource
[ 0.063357] ACPI: _SB_.R6RR: New power resource
[ 0.063397] ACPI: _SB_.P7RR: New power resource
[ 0.063435] ACPI: _SB_.R7RR: New power resource
[ 0.063475] ACPI: _SB_.P8RR: New power resource
[ 0.063514] ACPI: _SB_.R8RR: New power resource
[ 0.063554] ACPI: _SB_.P9RR: New power resource
[ 0.063594] ACPI: _SB_.R9RR: New power resource
[ 0.067003] ACPI: _SB_.USB5.RHUB.PRT2.PWFR: New power resource
[ 0.069241] ACPI: _SB_.PBRR: New power resource
[ 0.069280] ACPI: _SB_.RBRR: New power resource
[ 0.069322] ACPI: _SB_.PCRR: New power resource
[ 0.069362] ACPI: _SB_.RCRR: New power resource
[ 0.069403] ACPI: _SB_.PDRR: New power resource
[ 0.069443] ACPI: _SB_.RDRR: New power resource
[ 0.069483] ACPI: _SB_.PERR: New power resource
[ 0.069522] ACPI: _SB_.RERR: New power resource
[ 0.069563] ACPI: _SB_.PFRR: New power resource
[ 1.221141] input: Power Button as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0C:00/input/input0
[ 1.221183] ACPI: button: Power Button [PWRB]
[ 1.465354] PM: genpd: Disabling unused power domains
[ 2.609878] mlx5_core 0000:01:00.0: mlx5_pcie_event:326:(pid 165): Detected insufficient power on the PCIe slot (27W).
[ 3.166722] mlx5_core 0000:01:00.1: mlx5_pcie_event:326:(pid 371): Detected insufficient power on the PCIe slot (27W).
[ 3.712735] mlx5_core 0002:01:00.0: mlx5_pcie_event:326:(pid 387): Detected insufficient power on the PCIe slot (27W).
[ 4.332550] mlx5_core 0002:01:00.1: mlx5_pcie_event:326:(pid 371): Detected insufficient power on the PCIe slot (27W).
[ 16.054921] wlP9s9: Limiting TX power to 23 (23 - 0) dBm as advertised by 84:78:48:02:0b:13
mashie@vesta:~$ sudo lspci -vvv | grep PowerLimit
Slot #0, PowerLimit 0W; Interlock- NoCompl+
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W
Slot #2, PowerLimit 0W; Interlock- NoCompl+
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W
Slot #4, PowerLimit 0W; Interlock- NoCompl+
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W
Slot #7, PowerLimit 0W; Interlock- NoCompl+
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0W
Slot #9, PowerLimit 0W; Interlock- NoCompl+
pcilib: sysfs_read_vpd: read failed: No such device
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W
mashie@vesta:~$
All software and firmware are on the latest versions.
Any ideas besides a big hammer to try next?