Hello NVIDIA team,
We are conducting networking research and plan to use SHAMPO in a DPDK-based system. Since the upstream DPDK mlx5 PMD does not expose SHAMPO, we added minimal DevX SHAMPO support to DPDK 25.11.3.
We found that one SHAMPO RQ is limited to about 20.0 Mpps on a 400 GbE ConnectX-7. The same NIC reaches 32.51 Mpps (400 Gb/s) with one regular MPRQ or two SHAMPO RQs. We would like to determine whether this is an expected per-SHAMPO-RQ limit or an error in our programming.
Setup
- Receiver: ConnectX-7 MCX75310AAS-NEA_Ax
- Firmware:
28.47.2682(MT_0000000838) - MLNX_OFED / mlx5_core:
25.10-1.7.1 - Kernel:
5.15.0-181-generic - Intel Xeon Gold 5512U; NIC and polling core on NUMA node 0
- DPDK
v25.11.3, plus our minimal SHAMPO patch - RX IOVA mode PA, 2 MB hugepages, flow control disabled
The directly connected sender is another 400 GbE ConnectX-7 PF. It uses one DPDK core and TCP TSO to offer a stable 400 Gb/s without TX failures.
To isolate SHAMPO hardware performance from other software processing, we use a raw CQ sink. It only checks CQ ownership/errors, counts CQEs, advances the linked-list RQ producer from wqe_id, and updates the CQ/RQ doorbells. It does not read packet data or perform software GRO or application processing.
RQ Configuration
- One TCP flow, MTU 1500, MSS 1460
- One RX polling core; CQE compression and RX vector path disabled
- 32 linked-list striding WQEs, 256 KiB each
- 256-byte data stride, 1,024 strides per WQE; total data buffer 8 MiB
- 64 KiB reservation, at most 64 packets per reservation
- Header ring: 8,192 × 256-byte entries
- Static MTT mkeys for both data and headers
- Timeout 1024, extended match criteria, stride-aligned no-match handling
- Required/allocated CQEs: 8,320 / 16,384
We also matched the regular MPRQ data-WQE geometry (16 × 512 KiB WQEs, each with 256 × 2,048-byte strides), tested WQE sizes from 256 KiB to 1 MiB, and doubled the CQ depth to 32,768. None materially changed the single-RQ rate of approximately 20 Mpps.
Results
Processed throughput is calculated from the packet-CQE rate and the full on-wire packet size, including Ethernet overhead. The physical ingress rate is approximately 400 Gb/s in all three tests. SHAMPO requires CQE compression to be disabled. We therefore also disabled it for the regular-MPRQ control, and the PF compression counters remained zero in every test.
| RX path | Flows / RQs / RX cores | CQE compression | Packet-CQE rate | Processed throughput | PF buffer discard |
|---|---|---|---|---|---|
| SHAMPO raw sink (our patch) | 1 / 1 / 1 | Off | 20.03 Mpps | 246.44 Gb/s | 38.34% |
| SHAMPO raw sink (our patch) | 2 / 2 / 1 | Off | 32.5 Mpps aggregate | 399.97 Gb/s | 0% |
Upstream DPDK regular MPRQ (testpmd rxonly) |
1 / 1 / 1 | Off | 32.5 Mpps | 399.97 Gb/s | 0% |
The sender delivers a stable 400 Gb/s in all three tests. With one SHAMPO RQ, the receiver consumes only 20.03 million packet CQEs/s, and the remaining packets appear as PF rx_prio0_buf_discard. With two SHAMPO RQs, the same CPU core polls both RQs serially and consumes the full 32.51 million packet CQEs/s in aggregate. This makes a CPU bottleneck in the polling path unlikely. A regular MPRQ also receives the full 400 Gb/s with one RQ and one RX core, indicating that the NIC and the general uncompressed-CQE path can sustain the offered rate.
In the single-SHAMPO-RQ test, most polls find no ready CQE. CQE/WQE/buffer errors and compressed CQEs are zero.
Questions
- Is approximately 20 million packet CQEs/s an expected per-SHAMPO-RQ limit on ConnectX-7?
- If not, is any part of the RQ geometry, SHAMPO attributes, static MTT mkeys, or producer update described above incorrect? Are there other SHAMPO configuration requirements or tuning knobs we may have overlooked?
We can run any additional experiments or provide any further information that would help with the investigation.
We would greatly appreciate any guidance you can provide.