ConnectX-3 ASPM support?

Infrastructure & Networking Software And Drivers Mellanox OFED
1

I find my CX311/CX312 refuses to enable ASPM even though the link-capability claims ASPM support:

LnkCap: Port #8, Speed 8GT/s, Width x4, ASPM L0s, Exit Latency L0s unlimited
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-

I suspect this prevents the CPU package from entering deep power saving mode ( was ~92% PC8 without the NIC, and ~93% PC3 with the NIC) and contributes to much higher measured power consumption (~7.5W) than the rated 3.8W typical.

Question: is there any way (driver / firmware config) to get the NIC to run in ASPM mode?

Question2: does SR-IOV play any role here? (I also played around with SR-IOV enabled and disabled and find no direct impact on the power. But I do noticed that the VF is listed as ASPM not supported, which makes me worry…)

2

I tried some dirty trick from this post to manually force enable the ASPM L0S mode.
Unfortunately this lead to floods of error message in the kernel log:

[ 597.337196] pcieport 0000:00:01.0: PCIe Bus Error: severity=Corrected, type=Data Link Layer, (Transmitter ID)
[ 597.337197] pcieport 0000:00:01.0: device [8086:1901] error status/mask=00001000/00002000
[ 597.337198] pcieport 0000:00:01.0: [12] Timeout
[ 597.339789] pcieport 0000:00:01.0: AER: Multiple Corrected error received: 0000:00:01.0
[ 597.339825] pcieport 0000:00:01.0: PCIe Bus Error: severity=Corrected, type=Data Link Layer, (Transmitter ID)
[ 597.339826] pcieport 0000:00:01.0: device [8086:1901] error status/mask=00001100/00002000
[ 597.339826] pcieport 0000:00:01.0: [ 8] Rollover
[ 597.339827] pcieport 0000:00:01.0: [12] Timeout
[ 597.339829] pcieport 0000:00:01.0: AER: Multiple Corrected error received: 0000:00:01.0
[ 597.339854] pcieport 0000:00:01.0: AER: can’t find device of ID0008

Does this indicate that the ASPM support is not working at all?

3

did you ever figure this out?

I am also trying to find out why this ConnectX-3 dual SFP+ card uses 12 watts with nothing plugged onto the system. I thought I read it should be 3 watt per port

4

Hi,
I am also very interested to figure this out. Maybe somebody from support team @Mellanox_Support can chime in.

Regards!