BF3 Offload Limitations and Capacity Tuning for Improved Performance

Infrastructure & Networking BlueField
1

Hello Team,

I am currently evaluating a BlueField-3 (BF3) deployment utilizing OVS-DPDK and OVS-DOCA. I am interested in understanding the inherent hardware limitations of the BF3 and would like advice on how to adjust the offload capacity for enhancing overall packet forwarding performance.

Details:

  • Platform: BlueField-3 (BF3)
  • Software: OVS-DPDK and OVS-DOCA
  • Scenario:
    • The OVS bridge is configured with 200K unique IPs for both source and destination, totaling 400K flows, which are set to drop on a match.
    • Cisco TRex is generating traffic with 65,000 unique server IPs corresponding to 1 client IP, resulting in 65,000 concurrent sessions.
    • As these server and client IP addresses do not overlap with those configured in the OVS bridge, megaflow merging is prevented, forcing each session to use an independent offloaded resource.
  • Observed Issue:
    When generating traffic at 64-byte packets and 5 Mpps, only about 45K flows are successfully offloaded. The remaining flows are dropped due to slow path limitations. A test using OVS-DOCA with flow-limit=200K produced similar results.

Request:
Could you please clarify the hardware limitations of the BF3 in this context and provide recommendations on tuning the offload capacity to optimize performance? Any additional insights on potential configuration adjustments (e.g., DOCA flow table settings, E-switch size modifications, or TCAM configurations) would be highly appreciated.

Thank you for your assistance. I look forward to your guidance on this matter.

Best regards,
Dickens Yeh

root@localhost:/home/ubuntu# ovs-appctl dpctl/offload-stats-show | grep offload
     Total                 Enqueued offloads:       0
     Total                 Inserted offloads:   50986
     [  0]                 Enqueued offloads:       0
     [  0]                 Inserted offloads:    5110
     [  1]                 Enqueued offloads:       0
     [  1]                 Inserted offloads:   45876
     [  2]                 Enqueued offloads:       0
     [  2]                 Inserted offloads:       0
     [  3]                 Enqueued offloads:       0
     [  3]                 Inserted offloads:       0
     [  4]                 Enqueued offloads:       0
     [  4]                 Inserted offloads:       0
     [  5]                 Enqueued offloads:       0
     [  5]                 Inserted offloads:       0
     [  6]                 Enqueued offloads:       0
     [  6]                 Inserted offloads:       0
     [  7]                 Enqueued offloads:       0
     [  7]                 Inserted offloads:       0
     [  8]                 Enqueued offloads:       0
     [  8]                 Inserted offloads:       0
     [  9]                 Enqueued offloads:       0
     [  9]                 Inserted offloads:       0
root@localhost:/home/ubuntu# ovs-appctl dpctl/dump-flows -m type=offloaded | grep -v 'flow-dump' | wc -l
45876
2

Hi @dickens.yeh,

Thank you for posting your query on our community and sharing information regarding your BlueField-3 deployment using OVS-DPDK and OVS-DOCA.
To help you optimize offload performance and better understand the hardware resource limitations, I recommend reviewing the latest NVIDIA DOCA documentation, which includes specific guidance on flow table tuning, hardware offload configuration, and performance best practices.

OVS‑DOCA Hardware Acceleration (DOCA 3.0.0):

https://docs.nvidia.com/doca/archive/3-0-0/OVS-DOCA%2BHardware%2BAcceleration/index.html

If you need further assistance, I would recommend to submit a support ticket for further troubleshooting. The support ticket can be opened by emailing " Networking-support@nvidia.com ".
Please note that an active support contract would be required for the same. For contracts information, please feel free to reach out to our contracts team at " Networking-Contracts@nvidia.com "

Thanks,
Bhargavi