Seeking guidance on modifying RoCEv2 code for continuous reception of UDP data packets from an FPGA over Ethernet

I’m currently working on receiving UDP data packets over Ethernet using RoCEv2, but I’ve encountered a challenge in my code. I’m trying to continuously receive data packets emitted by an FPGA, utilizing RoCEv2. I attempted to modify the code for multicast given in following document ,to unicast for receiving data packet:

However, I’m stuck at a certain point.

Could someone please assist me by providing guidance on what all parameters need to be adjusted in multicast code? Any help or suggestions would be greatly appreciated!

Unfortunately, due to privacy reasons mandated by my institute, I cannot share further details about code. Thank you in advance for your assistance!

Hi,
You wrote “I attempted to modify the code for multicast given in following document, to unicast for receiving data packet”. Are you trying to implement receiving unicast or multicast packets emitted by an FPGA?
If unicast, why don’t you take some unicast sample, for example rc_pingpong from OFED examples (rdma-core/libibverbs/examples/rc_pingpong.c)?
Multicast sample from “RDMA_Aware_Programing” document is using RDMA-CM, so RDMA-CM connection should be established between 2 sides before you are starting to send packets from one side and receive by another side. Similar mc sample can be found also in OFED sources: rdma-core/librdmacm/examples/mckey.c
If you are looking for unicast example with RC QP over RoCE I suggest to check rc_pingpong.

Thank you for your assistance. Essentially, I’m endeavoring to receive UDP data packets from an FPGA, which is continuously transmitting them. However, I’m encountering an issue where the completion queue entry isn’t being polled successfully. The FPGA is emitting data packets to port 4660 and IP address 192.168.1.69. ie fpga is basically programmed to emit data packet.

Here’s the interface information:
enp1s0f1np1: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
ether 1c:34:da:72:93:73 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

I’ll attempt to resolve this using the rc ping pong method. Thanks again

hence i meant to say i have no access to code of fpga side

Hi adv,
If you are going to use rdma application sample creating RC/UD QP types used for RoCE traffic, you can’t receive any traffic sent from FPGA and not corresponding to the correct RoCEv2 packets format. For example, destined to RoCE application packets should have reserved for RoCEv2 UDP port 4791 and all the rest of IB headers with right fields, right qpn, etc. Arbitrary packets sent by FPGA can’t be received by RoCE application.
If you need to receive by QP some Ethernet packets with some SA,DA,VLAN, IP, UDP ports, you need to create raw QP type and add corresponding steering rule to catch corresponding packets from FPGA. I can give you working raw QP application doing this. In my application the receiver is catching all packets with given SA, DA and EterType IP (0x0800)
raw_qp.tar.gz (7.6 KB)
If you need only packets with some IP and UDP header, yo need to change in the code “.num_of_specs = 1” to “3” and add ibv_flow_spec_ipv4 and ibv_flow_spec_tcp_udp in addition to ibv_flow_spec_eth structure attributes to struct raw_eth_flow_attr from the attached sample.
Beast regards,
Michael.

I forgot to say, raw_qp application should run in su mode (sudo):

[michaelbe@l-csi-1331h raw_qp]$ ./raw_qp_recv -d mlx5_0 -L 1 -n 6
PD(0x1875870) created
CQ ex(0x18758d0) created
Couldn’t create QP, errno 1: Operation not permitted
ibv_destroy_cq(0x18758d0)
ibv_dealloc_pd(0x1875870)
ibv_close_device(0x7f8002ada150)

[michaelbe@l-csi-1330h raw_qp]$ sudo ./raw_qp_send -d mlx5_0 -n 6
We are done

[michaelbe@l-csi-1331h raw_qp]$ sudo ./raw_qp_recv -d mlx5_0 -L 1 -n 6
PD(0x1972870) created
CQ ex(0x19728d0) created
QP(0x1972b38) created: qp_num = 0xc00049
QP 0xc00049 modified to INIT and RTR state
Memory buffer(0x7efe2e934000) allocated
MR(0x1971a10) registered
6 packets receiving completed
ibv_destroy_flow(0x19715d0)
ibv_dereg_mr(0x1971a10)
Free memory buffer (0x7efe2e934000)
ibv_destroy_qp(0x1972b38)
ibv_destroy_cq(0x19728d0)
ibv_dealloc_pd(0x1972870)
ibv_close_device(0x7efe30591150)

Hello Michael,

I’ve attempted to implement the method you suggested for capturing UDP packets, but I’m encountering an issue where the completion queue isn’t being polled, leading to an infinite loop. Here’s the data packet I managed to print using the tcpdump -i enp1s0f0np0 udp -AA command:

16:24:11.631686 IP 192.168.1.68.22136 > applus.4660: UDP, length 1456
.4.r.r…E…|@…n…D…EVx.4…B.0….uP.0….uQ.0….uR.0….uS.0….uT.0….uU.0….uV.0….uW.0….uX.0….uY.0….uZ.0….u[.0….u.0….u].0….u^.0….u_.0….u.0..\.ua.0..\.ub.0..\.uc.0..\.ud.0..\.ue.0..\.uf.0..\.ug.0..\.uh.0..\.ui.0..\.uj.0..\.uk.0..\.ul.0..\.um.0..\.un.0..\.uo.0..\.up.0..\.uq.0..\.ur.0..\.us.0..\.ut.0..\.uu.0..\.uv.0..\.uw.0..\.ux.0..\.uy.0..\.uz.0..\.u{.0..\.u|.0..\.u}.0..\.u~.0..\.u..0..\.u..0..\.u..0.~\.u..0.}\.u..0.|\.u..0.{\.u..0.z\.u..0.y\.u..0.x\.u..0.w\.u..0.v\.u..0.u\.u..0.t\.u..0.s\.u..0.r\.u..0.q\.u..0.p\.u..0.o\.u..0.n\.u..0.m\.u..0.l\.u..0.k\.u..0.j\.u..0.i\.u..0.h\.u..0.g\.u..0.f\.u..0.e\.u..0.d\.u..0.c\.u..0.b\.u..0.a\.u..0..u…0._.u…0.^.u…0.].u…0.\.u…0.[.u…0.Z.u…0.Y.u…0.X.u…0.W.u…0.V.u…0.U.u…0.T.u…0.S.u…0.R.u…0.Q.u…0.P.u…0.O.u…0.N.u…0.M.u…0.L.u…0.K.u…0.J.u…0.I.u…0.H.u…0.G.u…0.F.u…0.E.u…0.D.u…0.C.u…0.B.u…0.A.u…0.@.u…0.?.u…0.>.u…0.=.u…0.<.u…0.;.u…0.:.u…0.9.u…0.8.u…0.7.u…0.6.u…0.5.u…0.4.u…0.3.u…0.2.u…0.1.u…0.0.u…0./.u…0….u…0.-.u…0.,.u…0.+.u…0.*.u…0.).u…0.(.u…0.'.u…0.&.u…0.%.u…0.$.u…0.#.u…0.".u…0.!.u…0. .u…0….u…0….u…0….u…0….u…0….u…0….u…0….u…0….u…0….u…0….u…0….u…0….u…0….u…0….u…0….u…0….u…0….u…0….u…0….u…0….u…0….u…0.
.u…0. .u…0….u…0….u…0….u…0….u…0….u…0….u…0….u…0….u…0….v…0….v…0….v…0….v…0….v…0….v.

Below is the steering rule I created to capture these packets:
struct raw_eth_flow_attr {
struct ibv_flow_attr attr;
struct ibv_flow_spec_eth spec_eth;
struct ibv_flow_spec_ipv4 spec_ipv4;
struct ibv_flow_spec_tcp_udp spec_udp;

} __attribute__((packed)) flow_attr = {

    .attr = {
    .comp_mask = 0,
    .type = IBV_FLOW_ATTR_NORMAL,
    .size = sizeof(flow_attr),
    .priority = 0,
    .num_of_specs = 3, // Increase to 3 for adding IPv4 and UDP specifications
    .port = usr_par.ib_port,
    .flags = 0,
    },

    .spec_eth = {
        .type = IBV_FLOW_SPEC_ETH,
        .size = sizeof(struct ibv_flow_spec_eth),
        .val = {
            .ether_type = htons(0x0800), // IPv4
            .vlan_tag = 0,
        },
        .mask = {
            .dst_mac = {0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF},
            .src_mac = {0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF},
            .ether_type = 0xFFFF,
            .vlan_tag = 0,
        }
    },

    .spec_ipv4 = {
        .type = IBV_FLOW_SPEC_IPV4,
        .size = sizeof(struct ibv_flow_spec_ipv4),
        .val = {
            .src_ip = inet_addr("192.168.1.68"), // Source IP Address
            .dst_ip = inet_addr("applus"),     // Destination IP Address
           
        },
        .mask = {
            .src_ip = inet_addr("255.255.255.255"), // All bits must match
            .dst_ip = inet_addr("255.255.255.255"), // All bits must match
            
        }
    },
    
    .spec_udp = {
        .type = IBV_FLOW_SPEC_UDP,
        .size = sizeof(struct ibv_flow_spec_tcp_udp),
        .val = {
            .dst_port = htons(4660), // Destination UDP Port
        },
        .mask = {
            .dst_port = 0xFFFF, // All bits must match
        }
    }
};
memcpy(flow_attr.spec_eth.val.dst_mac, usr_par.dst_mac, 6);
memcpy(flow_attr.spec_eth.val.src_mac, usr_par.src_mac, 6);

/* Create steering rule */
struct ibv_flow *eth_flow;
eth_flow = ibv_create_flow(qp, &flow_attr.attr);
if (!eth_flow) {
    fprintf(stderr, "Couldn't attach steering flow, errno %d: %s\n", errno, strerror(errno));
    main_ret = 1;
    goto clean_mr;
}

Could you please provide insight into why the completion queue isn’t being polled and how I can resolve this issue? Any assistance would be greatly appreciated

Hi adv,
I suggest to start from simple case and verify on your system that this works.
Thake the application as is and put in command line the right SA and DA. In my case the default were:
char dmac_arr = {0x01, 0x00, 0x5e, 0x00, 0x02, 0x14};
char smac_arr = {0xe4, 0x1d, 0x2d, 0x11, 0x22, 0x33};
In your case this can be something else.
After you are putting the right DA/SA in command line (or change the defaults in the code), verify that you are getting completion from CQ when steering rule is just one type IBV_FLOW_SPEC_ETH.
After you see the application working on my setup is also working as is on your setup, start to add slowly additional types: only layer 3 (IP), Check this is working, then add layer 4. By the way, I’m not sure if we need .src_ip = inet_addr(“192.168.1.68”) or .src_ip = htons(inet_addr(“192.168.1.68”)).
Can you capture tcpdump to a pcap format file and attach to the chat?
Best regards,
Michael.

Thank you, Michaelbe, for your assistance. My code is now functioning properly.
I have a question regarding the raw qp function: does it read the entire UDP packet or just the payload into the buffer? When I attempted to print the byte length using ibv_wc_read_byte_len(cq_ex), it returned 1498, whereas my payload size is only 1456. I can provide a TCP dump snapshot as well.

14:02:25.864869 IP 192.168.1.68.22136 > apexplus.4660: UDP, length 1456
0x0000: 4500 05cc 1697 4000 ff11 dbaf c0a8 0144
0x0010: c0a8 0145 5678 1234 05b8 0742 e9f7 cece
0x0020: 1608 3132 e9f7 cecd 1608 3133 e9f7 cecc
0x0030: 1608 3134 e9f7 cecb 1608 3135 e9f7 ceca
0x0040: 1608 3136 e9f7 cec9 1608 3137 e9f7 cec8
0x0050: 1608 3138 e9f7 cec7 1608 3139 e9f7 cec6
0x0060: 1608 313a e9f7 cec5 1608 313b e9f7 cec4
0x0070: 1608 313c e9f7 cec3 1608 313d e9f7 cec2
0x0080: 1608 313e e9f7 cec1 1608 313f e9f7 cec0
0x0090: 1608 3140 e9f7 cebf 1608 3141 e9f7 cebe
0x00a0: 1608 3142 e9f7 cebd 1608 3143 e9f7 cebc
0x00b0: 1608 3144 e9f7 cebb 1608 3145 e9f7 ceba
0x00c0: 1608 3146 e9f7 ceb9 1608 3147 e9f7 ceb8
0x00d0: 1608 3148 e9f7 ceb7 1608 3149 e9f7 ceb6
0x00e0: 1608 314a e9f7 ceb5 1608 314b e9f7 ceb4
0x00f0: 1608 314c e9f7 ceb3 1608 314d e9f7 ceb2
0x0100: 1608 314e e9f7 ceb1 1608 314f e9f7 ceb0
0x0110: 1608 3150 e9f7 ceaf 1608 3151 e9f7 ceae
0x0120: 1608 3152 e9f7 cead 1608 3153 e9f7 ceac
0x0130: 1608 3154 e9f7 ceab 1608 3155 e9f7 ceaa
0x0140: 1608 3156 e9f7 cea9 1608 3157 e9f7 cea8
0x0150: 1608 3158 e9f7 cea7 1608 3159 e9f7 cea6
0x0160: 1608 315a e9f7 cea5 1608 315b e9f7 cea4
0x0170: 1608 315c e9f7 cea3 1608 315d e9f7 cea2
0x0180: 1608 315e e9f7 cea1 1608 315f e9f7 cea0
0x0190: 1608 3160 e9f7 ce9f 1608 3161 e9f7 ce9e
0x01a0: 1608 3162 e9f7 ce9d 1608 3163 e9f7 ce9c
0x01b0: 1608 3164 e9f7 ce9b 1608 3165 e9f7 ce9a
0x01c0: 1608 3166 e9f7 ce99 1608 3167 e9f7 ce98
0x01d0: 1608 3168 e9f7 ce97 1608 3169 e9f7 ce96
0x01e0: 1608 316a e9f7 ce95 1608 316b e9f7 ce94
0x01f0: 1608 316c e9f7 ce93 1608 316d e9f7 ce92
0x0200: 1608 316e e9f7 ce91 1608 316f e9f7 ce90
0x0210: 1608 3170 e9f7 ce8f 1608 3171 e9f7 ce8e
0x0220: 1608 3172 e9f7 ce8d 1608 3173 e9f7 ce8c
0x0230: 1608 3174 e9f7 ce8b 1608 3175 e9f7 ce8a
0x0240: 1608 3176 e9f7 ce89 1608 3177 e9f7 ce88
0x0250: 1608 3178 e9f7 ce87 1608 3179 e9f7 ce86
0x0260: 1608 317a e9f7 ce85 1608 317b e9f7 ce84
0x0270: 1608 317c e9f7 ce83 1608 317d e9f7 ce82
0x0280: 1608 317e e9f7 ce81 1608 317f e9f7 ce80
0x0290: 1608 3180 e9f7 ce7f 1608 3181 e9f7 ce7e
0x02a0: 1608 3182 e9f7 ce7d 1608 3183 e9f7 ce7c
0x02b0: 1608 3184 e9f7 ce7b 1608 3185 e9f7 ce7a
0x02c0: 1608 3186 e9f7 ce79 1608 3187 e9f7 ce78
0x02d0: 1608 3188 e9f7 ce77 1608 3189 e9f7 ce76
0x02e0: 1608 318a e9f7 ce75 1608 318b e9f7 ce74
0x02f0: 1608 318c e9f7 ce73 1608 318d e9f7 ce72
0x0300: 1608 318e e9f7 ce71 1608 318f e9f7 ce70
0x0310: 1608 3190 e9f7 ce6f 1608 3191 e9f7 ce6e
0x0320: 1608 3192 e9f7 ce6d 1608 3193 e9f7 ce6c
0x0330: 1608 3194 e9f7 ce6b 1608 3195 e9f7 ce6a
0x0340: 1608 3196 e9f7 ce69 1608 3197 e9f7 ce68
0x0350: 1608 3198 e9f7 ce67 1608 3199 e9f7 ce66
0x0360: 1608 319a e9f7 ce65 1608 319b e9f7 ce64
0x0370: 1608 319c e9f7 ce63 1608 319d e9f7 ce62
0x0380: 1608 319e e9f7 ce61 1608 319f e9f7 ce60
0x0390: 1608 31a0 e9f7 ce5f 1608 31a1 e9f7 ce5e
0x03a0: 1608 31a2 e9f7 ce5d 1608 31a3 e9f7 ce5c
0x03b0: 1608 31a4 e9f7 ce5b 1608 31a5 e9f7 ce5a
0x03c0: 1608 31a6 e9f7 ce59 1608 31a7 e9f7 ce58
0x03d0: 1608 31a8 e9f7 ce57 1608 31a9 e9f7 ce56
0x03e0: 1608 31aa e9f7 ce55 1608 31ab e9f7 ce54
0x03f0: 1608 31ac e9f7 ce53 1608 31ad e9f7 ce52
0x0400: 1608 31ae e9f7 ce51 1608 31af e9f7 ce50
0x0410: 1608 31b0 e9f7 ce4f 1608 31b1 e9f7 ce4e
0x0420: 1608 31b2 e9f7 ce4d 1608 31b3 e9f7 ce4c
0x0430: 1608 31b4 e9f7 ce4b 1608 31b5 e9f7 ce4a
0x0440: 1608 31b6 e9f7 ce49 1608 31b7 e9f7 ce48
0x0450: 1608 31b8 e9f7 ce47 1608 31b9 e9f7 ce46
0x0460: 1608 31ba e9f7 ce45 1608 31bb e9f7 ce44
0x0470: 1608 31bc e9f7 ce43 1608 31bd e9f7 ce42
0x0480: 1608 31be e9f7 ce41 1608 31bf e9f7 ce40
0x0490: 1608 31c0 e9f7 ce3f 1608 31c1 e9f7 ce3e
0x04a0: 1608 31c2 e9f7 ce3d 1608 31c3 e9f7 ce3c
0x04b0: 1608 31c4 e9f7 ce3b 1608 31c5 e9f7 ce3a
0x04c0: 1608 31c6 e9f7 ce39 1608 31c7 e9f7 ce38
0x04d0: 1608 31c8 e9f7 ce37 1608 31c9 e9f7 ce36
0x04e0: 1608 31ca e9f7 ce35 1608 31cb e9f7 ce34
0x04f0: 1608 31cc e9f7 ce33 1608 31cd e9f7 ce32
0x0500: 1608 31ce e9f7 ce31 1608 31cf e9f7 ce30
0x0510: 1608 31d0 e9f7 ce2f 1608 31d1 e9f7 ce2e
0x0520: 1608 31d2 e9f7 ce2d 1608 31d3 e9f7 ce2c
0x0530: 1608 31d4 e9f7 ce2b 1608 31d5 e9f7 ce2a
0x0540: 1608 31d6 e9f7 ce29 1608 31d7 e9f7 ce28
0x0550: 1608 31d8 e9f7 ce27 1608 31d9 e9f7 ce26
0x0560: 1608 31da e9f7 ce25 1608 31db e9f7 ce24
0x0570: 1608 31dc e9f7 ce23 1608 31dd e9f7 ce22
0x0580: 1608 31de e9f7 ce21 1608 31df e9f7 ce20
0x0590: 1608 31e0 e9f7 ce1f 1608 31e1 e9f7 ce1e
0x05a0: 1608 31e2 e9f7 ce1d 1608 31e3 e9f7 ce1c
0x05b0: 1608 31e4 e9f7 ce1b 1608 31e5 e9f7 ce1a
0x05c0: 1608 31e6 e9f7 ce19 1608 31e7

Could you please explain what might be happening? I need to ensure there is no packet loss, so I have to check the payload data. Your help would be greatly appreciated.

this is command that i have used for tcp dump
sudo tcpdump -i enp1s0f0np0 udp -x

Hi adv,
raw packet means absolutely raw. The received and written to memory packet by raw QP includes L2 (DA,SA,EthType) 14bytes + IP header 20 byte + UDP header 8 bytes + payload 1456 bytes, not including CRC.
14+20+8+1456=1498
You need to allocate additional space for L2, L3, L4 headers.
You can check the sender code. We are putting to the buffer the data starting L2 header for sending. The same is on the receiver side.
Best regards,
Michael.

Hello Michaelbe,

I have a couple of questions regarding packet processing. Firstly, is there a specific verb that directly extracts the payload from raw packets? Currently, I find myself having to loop twice to parse the payload from the buffer containing raw data packets.

Secondly, I’m curious about the necessity of allocating buffers in multiples of 1024. I’m encountering issues when attempting to assign a buffer to num_packets * (size_of_raw_packets).

Thirdly, I’m attempting to run a program to receive approximately 10 GB of data, but I’m encountering difficulties allocating a buffer of that size. Despite having 12 GB of RAM available, the function does not allow me to allocate the required buffer size. Could there be additional memory allocation beyond the buffer?

Lastly, I’d like to express my gratitude as I’ve been stuck on this issue for quite some time. Thank you for your assistance.

Along side the memory error ,when i am trying to revive approx 10 GB ie 1,00,00,000 packets then i am getting this error

sudo ./cqarray -d mlx5_0 -L 1 -n 10000000
PD(0x55d492442c00) created
Couldn’t create CQ ex, errno 22: Invalid argument
ibv_dealloc_pd(0x55d492442c00)
ibv_close_device(0x7f4abe41c150)