DOCA RDMA CM: Can multiple RDMA contexts listen on the same device/port in one process?

Infrastructure & Networking Software And Drivers RDMA Software For GPU
1

I am designing a multithreaded RDMA server using DOCA RDMA and RDMA CM, and I would like to clarify the intended usage model for doca_rdma_start_listen_to_port() / RDMA CM listening.

My intended architecture is:

process
 ├─ shared doca_dev
 ├─ worker thread 0
 │   ├─ doca_pe_0
 │   └─ doca_rdma_0 / doca_ctx_0
 ├─ worker thread 1
 │   ├─ doca_pe_1
 │   └─ doca_rdma_1 / doca_ctx_1
 └─ worker thread N
     ├─ doca_pe_N
     └─ doca_rdma_N / doca_ctx_N

Each worker thread would own its own doca_pe and its own doca_rdma context, because DOCA Core documentation says that doca_pe and doca_ctx are not thread-safe and should be progressed/used by the owning thread.

My questions are:

  1. Is it supported to create multiple doca_rdma instances in the same process using the same doca_dev, where each RDMA instance is connected to a different doca_pe in a different worker thread?

  2. Is doca_rdma_set_max_num_connections() a per-doca_rdma-instance limit, or does it reserve/configure some global limit on the underlying device?

For example, if I create 4 RDMA instances and call:

doca_rdma_set_max_num_connections(rdma0, 1024);
doca_rdma_set_max_num_connections(rdma1, 1024);
doca_rdma_set_max_num_connections(rdma2, 1024);
doca_rdma_set_max_num_connections(rdma3, 1024);

does this mean each context can manage up to 1024 connections, subject to global device resource limits?

  1. For RDMA CM server mode, can multiple doca_rdma contexts call doca_rdma_start_listen_to_port() on the same device and same TCP/RDMA CM port from different worker threads?

For example:

worker 0: doca_rdma_start_listen_to_port(rdma0, 7471)
worker 1: doca_rdma_start_listen_to_port(rdma1, 7471)
worker 2: doca_rdma_start_listen_to_port(rdma2, 7471)
worker 3: doca_rdma_start_listen_to_port(rdma3, 7471)

Is this supported as a kind of multi-context listener/load-balancing model, or should I expect bind/listen conflicts because all contexts use the same device and port?

  1. If multiple DOCA RDMA listeners on the same port are not supported, what is the recommended architecture for a multithreaded DOCA RDMA server with one public RDMA CM port?

For my use case, the preferred architecture would be:

one public RDMA CM port
+ multiple worker threads
+ each worker has its own doca_pe and doca_rdma context
+ accepted connections are distributed across workers
2

Based on the published DOCA RDMA documentation, the documented RDMA CM flow is a server RDMA instance listening on a specific port and then handling connection requests/events for multiple connections.

Also, doca_rdma_set_max_num_connections() is described as configuring the maximum number of connections supported by an RDMA instance, so this is per-instance, subject to overall device/resource limits.

For the same device + same port + multiple listening contexts scenario, this is not described in the documentation as a supported model. The safer recommended approach is one listener per port, accept multiple connections on that listener, and distribute the work across your worker threads/PEs as needed.

Reference:

If you need confirmation for a specific production design, please open an NVIDIA Enterprise Support case with your DOCA version and topology details.