Managing Large Packet Volume than Completion Queue size

Infrastructure & Networking Software And Drivers Mellanox OFED
,
1

Hello Fellow Forum Members,

I’m currently encountering a challenge concerning the reception of a large volume of packets while needing to maintain a limited size for the completion queue (which must be smaller than the number of packets to be received).

Every time I set the size of the completion queue to be less than the number of packets to be received, I encounter an issue where I get stuck in an infinite loop .Of the following .tar.gz file code I am referring to is raw_qp_recv

raw_qp.tar.gz (7.6 KB)

In the provided code, I’ve set the size of the work request queue to 20k and the size of the completion queue to 100k. If I aim to receive the same number of packets as there are completion queue entries, there are no errors. However, if I try to receive more packets than the completion queue can accommodate, I end up stuck in an infinite loop at line number 473.

I would greatly appreciate any suggestions, experiences, or guidance you can offer on this matter.

Thank you in advance for your assistance.

2

Hello,

I can see recv side cq=wr=512, that no issue.

define RQ_NUM_DESC 512
cq_attr_ex.cqe = RQ_NUM_DESC;
qp.cap.max_recv_wr = RQ_NUM_DESC,

I don’t think “package nu#” etc related with your issue. If CQ not enough etc, you will get CQ over run complete error. Where is the “stuck”, have you add debug print? ibv_poll?

And I notice you use verbs exp API, that will be not supported long time ago on new driver.

If you want process flow etc, I recommend you use dpdk or devx.

https://doc.dpdk.org/guides/platform/mlx5.html

4

Hello Xiaofengl,
I’m facing an issue where I need to receive more than 10 million packets. However, I’m unable to set the parameters cq=wr=10 million as the system doesn’t allow it. I’ve tried setting wr to 20,000 (the maximum allowed by my system) and cq to 4 million (also the maximum), but it only allows me to receive 4 million packets, which is equal to the cq size. Could you please advise me on how to proceed if I need to receive 10 or 20 million data packets? Thanks!

5

Imagine if I configured the write request (wr) to 20,000 and the completion queue (cq) to 50,000. Then, when attempting to receive 70,000 packets, an error occurred: "ibv_poll_next: no space left on device