How to Properly Set CM Private Data Using `doca_rdma_connection_set_user_data` or `doca_rdma_connect_to_addr`?

Infrastructure & Networking DOCA Getting Started & Resources
1

Dear Community,

I am following the documentation on Connecting Using RDMA CM Connection Flow. In Step 3 of “A typical connection flow would be as follows,” it is mentioned:

The RDMA client can set the connection user data to include in each connection using doca_rdma_connection_set_user_data(), and retrieve it from a connection using doca_rdma_connection_get_user_data().

I understand this is conceptually similar to how struct rdma_conn_param uses const void *private_data and int8_t private_data_len in the RDMA Core Library. However, when examining the detailed definition of
DOCA_EXPERIMENTAL doca_error_t doca_rdma_connection_set_user_data(struct doca_rdma_connection *rdma_connection, union doca_data connection_user_data);
in /opt/mellanox/doca/include/doca_rdma.h, I noticed there is no field analogous to private_data_len.

Similarly, I found that
DOCA_EXPERIMENTAL doca_error_t doca_rdma_connect_to_addr(struct doca_rdma *rdma, struct doca_rdma_addr *addr, union doca_data connection_user_data);
also lacks a setting for private_data_len.

Could you please clarify how to correctly set CM Private Data without such a field? Is there a specific mechanism within DOCA to ensure the appropriate length of the user data is accounted for during the connection setup?

Additionally, I noticed that the image associated with the configuration flow in the Exporting and Connecting RDMA section is missing or broken.

Thank you for your assistance and insights.

2

Hi, In doca, the user data is not a variable length as it is with the RDMA CM in the core library. It is a single unsigned 64b value that is taken directly as specified by the user. For convenience the data can be specified as a pointer or as a u64 integer (which is why it is a union).

Thanks,
-matt

3

Thank you for your response. I appreciate your guidance!

I observed that in the sample file /opt/mellanox/doca/samples/doca_gpunetio/gpunetio_rdma_client_server_write/host/gpunetio_rdma_client_server_write_sample.c, the union doca_data is used as connection_data:

// define  
struct rdma_resources resources = {0};  
union doca_data connection_data;  

// use in function  `doca_error_t rdma_write_client(struct rdma_config *cfg)`
connection_data.ptr = (void *)&resources;  
result = doca_rdma_connect_to_addr(resources.rdma, resources.cm_addr, connection_data);

The relevant struct definitions are as follows:

union doca_data {  
    void *ptr;    /**< Data as a pointer */  
    uint64_t u64; /**< Data as a 64-bit unsigned integer */  
};  

struct rdma_resources {  
    struct rdma_config *cfg;       /* RDMA samples configuration parameters */  
    struct doca_dev *doca_device;  /* DOCA device */  
    struct doca_gpu *gpudev;       /* DOCA GPU device */  
    struct doca_rdma *rdma;        /* DOCA RDMA instance */  
    struct doca_gpu_dev_rdma *gpu_rdma; /* DOCA RDMA instance GPU handler */  
    struct doca_ctx *rdma_ctx;     /* DOCA context to be used with DOCA RDMA */  
    struct doca_pe *pe;            /* DOCA progress engine -- needed by server only */  
    const void *connection_details;/* Remote peer connection details */  
    size_t conn_det_len;           /* Remote peer connection details data length */  

    /* rdma_cm resources */  
    struct doca_rdma_addr *cm_addr;         /* CM server address to connect by a client */  
    struct doca_rdma_connection *connection; /* The RDMA_CM connection instance */  
    bool connection_established;           /* Indicate whether connection is established */  
    bool connection_error;                 /* Indicate connection error */  
    bool server_listen_active;             /* Indicate if server listen_to_port is active */  

    struct doca_rdma_connection *connection2; /* The RDMA_CM connection instance */  
    bool connection2_established;          /* Indicate whether connection is established */  
    bool connection2_error;                /* Indicate connection error */  
};

I am wondering if it is possible to use the connection_details and conn_det_len members of struct rdma_resources to set the CM Private Data. However, I suspect that this might not work as intended.

When I tested this, I observed via Wireshark that the CM ConnectRequest packet’s PrivateData was still all zeros:

connection_data.ptr = (void *)&resources;  
const char private_data[] = "test";  
resources.connection_details = private_data; // struct rdma_conn_param : private_data  
resources.conn_det_len = sizeof(private_data); // struct rdma_conn_param : private_data_len  
DOCA_LOG_INFO("Client Set CM connection_data %uB at %p:%s", resources.conn_det_len, resources.connection_details, resources.connection_details);  
result = doca_rdma_connect_to_addr(resources.rdma, resources.cm_addr, connection_data);

Could you please confirm whether it is possible to use these fields (connection_details and conn_det_len) to properly set the CM Private Data, or if there is another recommended approach for this?

Thank you again for your time and assistance!

4

Unfortunately the connection details are DOCA specific and are not used as private data in the RDMA CM function calls.

Looking at the source code, it seems that private data is not currently possible via the DOCA RDMA CM calls. On the connect side, there is no way to pass any private data. On the accept side you can pass private data via doca_rdma_connection_accept(), however, it looks like that data is not processed/passed along once the connection is established and simply gets lost.

5

Thank you very much for your response and for taking the time to address my question! I truly appreciate your guidance and the insights you’ve provided.

To summarize, it seems that DOCA 2.9 still does not support the functionality of carrying Private Data during the RDMA CM connection establishment process.