We’ve done so many tests on “dns_filter” app and we really wish it can help us to control the flow of RoCE v2. After some tests on it,we find that only when turning off the offloading of OVS and changing the dst_port(4791) of the flow,can we use the dns_filter app to forward this kind of offloading traffic.However the speed of RoCE v2 fell down to 30MB/S from 2700MB/s after turning off the offloading function of OVS. We really don’t want low speed of RoCE v2 ,so is it possible for us to control the flow of RoCE v2 by “Flow Programming” when offloading the OVS? Really need your help,please.
the problem for the speed decrease from disabling offloading comes from the CPU handling the flow rules and/or the OVS decission making. Which means, the packets will come in the interface, sent to the DPU, processed by the CPU, sent back to the eSwitch and forwarded to the target (if the target is not the DPU).
(see also https://docs.nvidia.com/doca/archive/doca-v1.2/pdf/modes-of-operation.pdf Chapter 2, figure)
For RoCEv2 packets as you noted you can directly accept port 4791 with an offloaded flow rule in the eSwitch which then forwards all packets to the RDMA engine.
However since (to my knowledge) the eSwitch in the current generations of Bluefields do not support selecting RoCEv2 headers and also cannot change them, you should not be able to modify those.
In any case, if you were to be able to e.g. forward RoCE packets to another destination, not only do you need to change the IP, but also identifiers for the QP, PSN etc which might be different from destination to destination.
But I also would be interested in selectors and modifiers for RDMA fields coming in the future.
So glad to see your reply!But I think I didn’t explain our problem clearly.Actually We take the RoCE v2 packets as the normal UDP packets when handling them since it is based on UDP protocal.So as you can see in this picture we found on the DOCA SDK Documents,we changed the dst_port of RoCE v2 flow from 4791 to 4790 in ovs-br1 and recover the dst_port of it from 4790 to 4791 in ovs-br2.And if we don’t do it like this,we cannot use flow programming to forward RDMA flow.The problem is that only when we turned off the offloading of ovs-br1 and ovs-br2,the test of ib_send_bw (perftest) could be finished completely with a low sending speed like 30MB/s.After we used ovs-tcpdump to capture the packets of it ,we found lots of packets were sended twice or more. Anyway, thanks for your reply ,sincerely.
And after a lot of tests on DPU,I think I have found the solution by myself.**We can use dpdk-l2fwd or dpdk-testpmd to forward the RDMA traffic from pf0hpf to p0.**And I also think these two applications are more simple and easier for us to learn.Finally I still wonder why we can’t get the RDMA traffic from SF and why we use SF and OvS in doca examples like doca_dns_filter or doca_url_filter while we can get the target traffic directly from p0 or pf0hpf.And is it possible for us to use p0 and pf0hpf directly when starting doca examples? If so ,how to do that?
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.