BlueField3 Msg Rate Test

Infrastructure & Networking DOCA Getting Started & Resources
1

I was trying to test the msg rate.
With netperf, use 64 points to test and i got this. It’s much far away from the MAX.
What can I do now?

image
image865×174 42.2 KB

2

Hello 1054919371,

Thank you for posting your inquiry to the NVIDIA Developer Forums.

Some troubleshooting/tuning recommendations follow - if these do not resolve your issue, and you have valid support entitlement, we highly recommend engaging NVEX Support for further assistance on this issue.

1. System Tuning

  • Use tuned-adm profile network-throughput to set optimal network parameters

  • Disable IRQ balancing: systemctl stop irqbalance.service

  • Optimize CPU governor for performance

2. Network Interface Optimization

  • Increase interface queue size: ethtool -G <interface> rx 8192 tx 8192

  • Disable adaptive coalescing: ethtool -C <interface> adaptive-rx off adaptive-tx off

  • Optimize interrupt moderation: ethtool -C <interface> rx-usecs 0 rx-frames 0

  • Set appropriate queue count: ethtool -L <interface> combined 31

3. IRQ Affinity Optimization

  • Use set_irq_affinity_cpulist.sh to assign IRQs to specific CPUs

4. RSS Configuration for BlueField-3

  • Implement proper RSS rules to distribute traffic across multiple queues

  • Ensure 1:1 ratio between streams and RX queues

  • Use multiple IP addresses for better queue utilization

  • Apply TC rules with set_tc_rules.sh p0 <interface_name>

5. Netperf Testing Methodology

  • Start with a single stream baseline test, then scale up

  • Use: netperf -H <server_ip> -t UDP_STREAM -l 60 -- -m 64

  • For multiple streams, launch parallel netperf instances

  • Test with different message sizes (8, 64, 128, 512 bytes)

Best regards,
NVIDIA Enterprise Experience