Know/control dual-port RNIC concurrency

Is there a way to know if the device driver can handle 2 RDMA operations launched on the 2 different ports of a dual-port RNIC in a simultaneous fashion (i.e., it will not handle them sequentially due to some locks/synchronization mechanisms)?
How to know the stuff protected by these locks (ex: some shared resources on the RNIC)?

No lock or sync needed.

RNIC (IB/RoCE link layer) handle send/recv QP with HARDWARE. No driver sync needed, driver just polling completion Q on HARDWARE know transmit finished.
All package had been DMA to peer side user space memory.

Simple!? It done by hardware.

Thank you for your reply. Then, having 2 ports is just I assume to allow some kind of hardware isolation?
But if that is the case, can I launch an Rdma operation on port 0 and then launch another Rdma operation on port 1 and be sure that they will not compete for Hardware resources (i.e., will be totally parallel). But I am sure that this parallelism will not be 100% any more if I increased the number of RDMA operations on each port to be 10K. In that case, I assume the only benefit I get from having a 2-port RNIC is replication (i.e., basically if a port is not working because of a hardware malfunction, I can use the other one). Is that correct?

Increase “RDMA operations” 10K QPs will 100% parallel processing in hardware, only issue is CPU need polling 10K CQ, that spend software resource.

I try to use 100K QPs in my labs no issue before.

2 port NIC not only replicate, many other use case.

1 Like

There are also RDMA API can use shared CQ, for polling. Anyway, many way to improve performance on software side.

1 Like

Thank you so much again for your reply.

According to your reply, when you said 10K QPs, do you usually has each QP assigned to a different device context to ensure better parallelism in a multi-threaded application?
And how do you do it when using rdma communication manager because to the best of my understanding, it automatically assign you to a device context. And in a multi-threaded scenario, it usually assigns the same device context, unlike when you have multiple processes.

The way I try to overcome this is to manually call ibv_open_device to get a unique context and then manually change the context returned from the rdma communication manager but not sure this is correct?

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.