Disabling Power Saving Modes on Mellanox BlueField DPU

Infrastructure & Networking BlueField
, , , ,
1

Hello NVIDIA Community,

I’m currently working with a Mellanox BlueField DPU and I’m facing some challenges regarding the power saving features that seem to be affecting the network performance.

I have used ethtool to explore and adjust network card settings, but haven’t found any specific options related to power saving (like EEE - Energy Efficient Ethernet).

$ sudo ethtool -i enp3s0f0np0
driver: mlx5_core
version: 5.8-4.1.5
firmware-version: 18.24.0302 (MSF0000000005)
expansion-rom-version:
bus-info: 0000:03:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes

ethtool --show-features:

  • There were no EEE settings or similar power management features available to toggle.
$ sudo ethtool --set-eee enp3s0f0np0 eee off

Cannot get EEE settings: Operation not supported

I can find some configurations related to power but I don’t know what they mean:

$ sudo mlxconfig -d /dev/mst/mt41682_pciconf0 query

Device #1:
----------

Device type:    BlueField       
Name:           N/A             
Description:    N/A             
Device:         /dev/mst/mt41682_pciconf0

Configurations:                                      Next Boot      
         DISABLE_SLOT_POWER_LIMITER                  True(1)         
         ADVANCED_POWER_SETTINGS                     True(1)

More info:

$ sudo lshw -c network

  *-network:0
       description: Ethernet interface
       product: MT416842 BlueField integrated ConnectX-5 network controller
       vendor: Mellanox Technologies
       physical id: 0
       bus info: pci@0000:03:00.0
       logical name: enp3s0f0np0
       logical name: /dev/fb0
       version: 00
       serial: b8:59:9f:42:6d:24
       capacity: 56Gbit/s
       width: 64 bits
       clock: 33MHz
       capabilities: pciexpress vpd msix pm bus_master cap_list rom ethernet physical 1000bt-fd 10000bt-fd 25000bt-fd 40000bt-fd 56000bt-fd autonegotiation fb
       configuration: autonegotiation=on broadcast=yes depth=32 driver=mlx5_core driverversion=5.8-4.1.5 duplex=full firmware=18.24.0302 (MSF0000000005) ip=192.168.3.2 latency=0 link=yes mode=1024x768 multicast=yes promiscuous=yes visual=truecolor xres=1024 yres=768
       resources: iomemory:3000-2fff irq:84 memory:30002000000-30003ffffff memory:96200000-962fffff memory:30006000000-300073fffff
  *-network:1
       description: Ethernet interface
       product: MT416842 BlueField integrated ConnectX-5 network controller
       vendor: Mellanox Technologies
       physical id: 0.1
       bus info: pci@0000:03:00.1
       logical name: enp3s0f1np1
       version: 00
       serial: b8:59:9f:42:6d:25
       capacity: 56Gbit/s
       width: 64 bits
       clock: 33MHz
       capabilities: pciexpress vpd msix pm bus_master cap_list rom ethernet physical 1000bt-fd 10000bt-fd 25000bt-fd 40000bt-fd 56000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=mlx5_core driverversion=5.8-4.1.5 firmware=18.24.0302 (MSF0000000005) ip=192.168.3.3 latency=0 link=yes multicast=yes promiscuous=yes
       resources: iomemory:3000-2fff irq:149 memory:30004000000-30005ffffff memory:96300000-963fffff memory:30007400000-300087fffff

Given these details, I would appreciate any guidance on:

  1. How to check if any embedded power-saving modes are active that aren’t visible via ethtool.
  2. Steps to disable these power-saving features to prioritize performance, particularly for latency-sensitive applications.

Has anyone else encountered similar challenges? Any advice on managing power settings specifically for Mellanox network cards in performance-critical environments would be very helpful.

Thank you in advance for your insights!

2

After reading Linux Kernel Doc and RedHat Article

I think I found a solution:

sudo vim /sys/class/net/<my device>/power/control

Change auto to on in order to disable RDPM

3

Hello,
Thank you for reaching out.
Based on your previous comment, I assume your questions are answered and the power control command solved the issue?

If the issue still persists or any other questions arise, please open a case at: enterprisesupport@nvidia.com, and it will be handled according to entitlement.

Best Regards,
Jonathan.

4 
sudo vim /sys/class/net/<my device>/power/control

This change is based on the Linux kernel.

Is there any configuration regarding power provided by the Mellanox driver?

By the way, what’s the meaning of ADVANCED_POWER_SETTINGS in mlxconfig?
I cannot find it in Nvidia document