Increase RX queue size on ConnectX-6 Dx | Ubuntu 22.04

Infrastructure & Networking Software And Drivers Mellanox OFED
1

Hi,

I experience huge package loss using a ConnectX-6 Dx with jumbo frames MTU 9014.
Incoming are about 300k packets per second which the RQ of the default Ubuntu 22.04 6.2.0-39 generic kernel can not handle (I can see >1M dropped packets per second on cat /proc/net/dev).

The vendor of the system suggests me to increase the number of receive queue slots using

ethtool -g → to check for maximum value

and then

ethtool -G rx to increase the number of RX queue slots to the maximum.
As of Ubuntu kernel 6.2.0-39-generic this maximum is set to 8192.

I then rechecked for the respective value in the unix kernel which indeed is set to

#define MLX5E_PARAMS_MINIMUM_LOG_SQ_SIZE                0x6
#define MLX5E_PARAMS_DEFAULT_LOG_SQ_SIZE                0xa
#define MLX5E_PARAMS_MAXIMUM_LOG_SQ_SIZE                0xd

#define MLX5E_PARAMS_MINIMUM_LOG_RQ_SIZE (1 + MLX5E_LOG_MAX_RX_WQE_BULK)
#define MLX5E_PARAMS_DEFAULT_LOG_RQ_SIZE                0xa
#define MLX5E_PARAMS_MAXIMUM_LOG_RQ_SIZE		0xd

However, changing these values and recompiling the kernel didn’t lead to a larger number of RQ queue slots when checking with ethtool -g .

What can I do to increase the receive queue slots correctly?
Is there maybe even a better way to deal with this traffic?

Best, Jonas
Mercedes-Benz

2

Hi Jonas

Please install/use MLNX-OFED, other than Unbuntu In-Box kernel driver.

/HyungKwang

3

Just did. I still had to increase the rx queue size to 8192 using

ethtool -G rx 8192

But doing this resolved all packet loss.
Thanks for your help.

~ Jonas