Ibv_poll_cq() always gets CQE

[eric503630]

In my example, I use RDMA CM to connect nodes and deal with the connection in RDMA_CM_EVENT_ESTABLISHED.
here is my server’s code:

void on_connected(struct rdma_cm_id *id, struct resources *res){
  struct ibv_cq *cq=res->cq;
  struct ibv_wc wc;
  while(1){
    int numx=ibv_poll_cq(res->recv_cq,1,&wc);
    if (numx=0){
      continue;
    }
    if(numx<0){
      std::cout << "error" << std::endl;
    }
    if(wc.status==IBV_WC_SUCCESS){
      if(wc.opcode==IBV_WC_SEND){
        std::cout << "send completed" << std::endl;
      }
      if(wc.opcode==IBV_WC_RECV){
        // rdma_post_recv(id,NULL,res->buf,kMemSize,res->mr);
        std::cout << "recv completed mem in server: " << res->buf << std::endl;
        std::string data = "server server!";
        memcpy(res->buf, data.c_str(), data.size());
        std::cout << "mem in server: " << res->buf << std::endl;
        break;
      }
      std::cout << "once again" << std::endl;
    }
  }
  
  rdma_post_send(id,NULL,res->buf,kMemSize,res->mr,0);
}

In my client’s code, I only call ibv_post_send() once.
I think when I call ibv_poll_cq() one CQE will pop out. In my case, there will be no CQE in CQ. But I can always get one actuality. Is there some wrong when I use the API?

[chenh1]

Hi @eric503630,

1. Did you set the signaled flag when posting send WQE? If the signaled flag is being set, it would normally trigger a completion for the send operation.
2. Completion error checking is not being done on the server side, this should be addressed to check the completion status when getting completion.
3. rdma_post_recv must be called after polling for receive completion if you want to continue receiving traffic.
4. I am unsure why rdma_post_send is being called on the server side at this point. Did you post receive WQE on the client side previously?

Regards,
Chen

[eric503630]

Thanks for your answers.

1. I’m not sure whether I set the signaled flag. Here is my part of the code for the client.

void on_connected(struct rdma_cm_id *id, struct resources *res){
  struct ibv_cq *cq=res->cq;
  struct ibv_send_wr sr;
  struct ibv_sge sge;
  struct ibv_send_wr *bad_wr = nullptr;
  struct ibv_wc wc;
  std::string data = "client client!";
  memcpy(res->buf, data.c_str(), data.size());
  std::cout << "mem in client: " << res->buf << std::endl;
  memset(&sge, 0, sizeof(sge));
  sge.addr = (uintptr_t) res->buf;
  sge.length = kMemSize;
  sge.lkey = res->mr->lkey;
  memset(&sr, 0, sizeof(sr));
  sr.next = nullptr;
  sr.wr_id = (uintptr_t)id->context;
  sr.sg_list = &sge;
  sr.num_sge = 1;
  sr.opcode = IBV_WR_SEND;
  sr.send_flags = IBV_SEND_SIGNALED;
  ibv_post_send(res->qp, &sr, &bad_wr);
  std::cout << "send" << std::endl;
  while (1) {
    int num_wc = ibv_poll_cq(res->recv_cq, 1, &wc);
    if (num_wc < 0) {
        std::cout << num_wc << " poll error!" << std::endl;
    }
        if (wc.status != IBV_WC_SUCCESS) {
            std::cout << "error!" << std::endl;
        }
        if ( wc.opcode == IBV_WC_RECV ) {
            std::cout << "mem in client: " << res->buf << std::endl;
            break;
        }
  }
}

2. For now, I have not considered the error case. I will catch it and handle it.
3. Sorry for that. It’s just a comment. It’s meanless. Sorry to confuse you.
4. Yes, when the client handle RDMA_CM_EVENT_ADDR_RESOLVED, I call ibv_post_recv(). I think it is OK, right?