Currently I am uninterested in performance.
I tried compiling my kernel by setting CONFIG_NET_RX_BUSY_POLL=n and compiled the kernel only to be enabled by default.
I also tried ethtool -C adaptive-rx off rx-frames 1 rx-usecs 0. However, the result didnt show any increase in interrupt rate.
What is the best way to increase interrupt rate and achieve 1:1 packet to interrupt ratio and disable NAPI for Mellanox?
I would suggest the following with the default kernel
ethtool -C ens1f0 adaptive-rx off adaptive-tx off rx-frames 1 rx-usecs 1 rx-frames-irq 1 rx-usecs-irq 1
or this
ethtool -C ens1f0 adaptive-rx off adaptive-tx off rx-frames 0 rx-frames-irq 0 rx-usecs 1 rx-usecs-irq 1
On my setup the number of interrupts tripled after applying this change.
In addition, I used Mellanox OFED. In the case of inbox behaviour might be different.
Thank you Aleksey for your suggestion.
What do you mean by the default kernel?
Kernel that comes with OS.
Hi, the default kernel is not able to recognize the Mellanox NICs. Should we install the drivers for the recognition of NICs?
I am worried it would replace the drivers of the kernel where the NIcs are being recognized without any issues.
I did install the drivers on the default kernel and was unable to notice any difference in interrupt rate?
Are yo able to see the difference when using non-Mellanox card? What is the test you are using? How to you measure the number of interrupts? Are you using Mellanox OFED? What is the HCA P/N - “lspci -d 15b3:” - modern kernel are able to recognize Mellanox cards.
lspci -d 15b3:
02:00.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]
02:00.1 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]
I do have any non mellanox card with me right now, so I am unable to confirm the same.
I didn’t install any Mellanox OFED tools. I am using the in-built drivers provided.
I am measuring interrupt rate by collecting /proc/interrupts data into a file and processing it. I am also using mpstat - “mpstat -I SUM 1 50000000” to provide the interrupt data.
I wasn’t able to get 1:1, however I succeeded to increase it 10 times by doing this:
Tune the system to run on maximum CPU speed, disable idling, etc. You may find Mellanox Tuning guide on our web site
Increase MTU to 9000. You need more packets to have higher interrupt rate
Disable pause frames
Set interrupts of the device on corresponding CPU ( see tuning guide)
Play with ‘ethtool’ coalesce setting to see what you can get
Again, I’m using Mellanox OFED, when using inbox driver, you might open a discussion with OS vendor support. Vendor might have different implementation of the driver.
Here is the output from my host
No modification:
Linux 3.10.0-693.el7.x86_64 07/12/2019 x86_64 (12 CPU)
06:16:20 PM CPU intr/s
06:16:21 PM all 70870.00
06:16:22 PM all 66880.00
06:16:23 PM all 65389.00
06:16:24 PM all 67802.00
With modifications:
06:10:42 PM all 801222.00
06:10:43 PM all 799500.00
06:10:44 PM all 780986.00
06:10:45 PM all 775810.00
06:10:46 PM all 786453.00
06:10:47 PM all 808909.00
06:10:48 PM all 820274.00
Hi,
Thank you for the info. Could you kindly let me know how to disable CPU idling? I am unsure of what you mean by disable idling etc. Is it the section 3.3 of the Mellanox tuning guide?
Grub configuration will do the trick. Be aware that it can increase power consumption
intel_pstate=disable intel_idle.max_cstate=0 processor.max_cstate=1 idle=poll