Arieal SDK (23-3) setup: E2E test issue: RU data not received by DU server

Hello,

I am currently trying to setup Aerial SDK on 2 DELL R740 + Nvidia Tesla A100 servers to verify the 5G L1 test cases.

I am following the instructions of env setup of “Aerial SDK documentation Release 23-3” and test the UL test case : TC7258
It is not passed since I can not observe any received data on DU host.

observation:

  1. on the RU host(host1: DEL R740), it print out the UL data transmission rate in the ru.log
    04:29:24.147117 WRN 635 0 [RU] Cell 0 UL 0.58 Mbps 105 Slots | DL_C_ON 0.00% DL_U_ON 0.00% UL_C_ON 0.00% |Seconds 1
    04:29:25.147110 WRN 635 0 [RU] Cell 0 UL 0.55 Mbps 100 Slots | DL_C_ON 0.00% DL_U_ON 0.00% UL_C_ON 0.00% |Seconds 2
    04:29:26.147112 WRN 635 0 [RU] Cell 0 UL 0.55 Mbps 100 Slots | DL_C_ON 0.00% DL_U_ON 0.00% UL_C_ON 0.00% |Seconds 3
    04:29:27.147109 WRN 635 0 [RU] Cell 0 UL 0.55 Mbps 100 Slots | DL_C_ON 0.00% DL_U_ON 0.00% UL_C_ON 0.00% |Seconds 4
    04:29:28.147116 WRN 635 0 [RU] Cell 0 UL 0.55 Mbps 100 Slots | DL_C_ON 0.00% DL_U_ON 0.00% UL_C_ON 0.00% |Seconds 5
    04:29:29.147116 WRN 635 0 [RU] Cell 0 UL 0.55 Mbps 100 Slots | DL_C_ON 0.00% DL_U_ON 0.00% UL_C_ON 0.00% |Seconds 6

    04:31:17.147110 WRN 635 0 [RU] Cell 0 UL 0.55 Mbps 100 Slots | DL_C_ON 0.00% DL_U_ON 0.00% UL_C_ON 100.00% |Seconds 114
    04:31:18.147113 WRN 635 0 [RU] Cell 0 UL 0.55 Mbps 100 Slots | DL_C_ON 0.00% DL_U_ON 0.00% UL_C_ON 100.00% |Seconds 115
    04:31:19.147112 WRN 635 0 [RU] Cell 0 UL 0.55 Mbps 100 Slots | DL_C_ON 0.00% DL_U_ON 0.00% UL_C_ON 100.00% |Seconds 116
    04:31:20.147112 WRN 635 0 [RU] Cell 0 UL 0.55 Mbps 100 Slots | DL_C_ON 0.00% DL_U_ON 0.00% UL_C_ON 100.00% |Seconds 117

  2. for the Aerial SDK(L1) on the DU host(host2: DEL R740+A100), it looks like there is no UL data received.

    Line 17283: 04:31:08.160004 WRN 3091 0 [SCF.PHY] Cell 0 | DL 0.00 Mbps 0 Slots | UL 0.00 Mbps 100 Slots CRC 100 ( 15699) | Tick 314000
    Line 17384: 04:31:09.160003 WRN 3091 0 [SCF.PHY] Cell 0 | DL 0.00 Mbps 0 Slots | UL 0.00 Mbps 100 Slots CRC 100 ( 15799) | Tick 316000
    Line 17485: 04:31:10.160004 WRN 3091 0 [SCF.PHY] Cell 0 | DL 0.00 Mbps 0 Slots | UL 0.00 Mbps 100 Slots CRC 100 ( 15899) | Tick 318000
    Line 17586: 04:31:11.160003 WRN 3091 0 [SCF.PHY] Cell 0 | DL 0.00 Mbps 0 Slots | UL 0.00 Mbps 100 Slots CRC 100 ( 15999) | Tick 320000
    Line 17687: 04:31:12.160003 WRN 3091 0 [SCF.PHY] Cell 0 | DL 0.00 Mbps 0 Slots | UL 0.00 Mbps 100 Slots CRC 100 ( 16099) | Tick 322000
    Line 17788: 04:31:13.160004 WRN 3091 0 [SCF.PHY] Cell 0 | DL 0.00 Mbps 0 Slots | UL 0.00 Mbps 100 Slots CRC 100 ( 16199) | Tick 324000
    Line 17889: 04:31:14.160003 WRN 3091 0 [SCF.PHY] Cell 0 | DL 0.00 Mbps 0 Slots | UL 0.00 Mbps 100 Slots CRC 100 ( 16299) | Tick 326000
    Line 17990: 04:31:15.160004 WRN 3091 0 [SCF.PHY] Cell 0 | DL 0.00 Mbps 0 Slots | UL 0.00 Mbps 100 Slots CRC 100 ( 16399) | Tick 328000
    Line 18091: 04:31:16.160003 WRN 3091 0 [SCF.PHY] Cell 0 | DL 0.00 Mbps 0 Slots | UL 0.00 Mbps 100 Slots CRC 100 ( 16499) | Tick 330000
    Line 18192: 04:31:17.160003 WRN 3091 0 [SCF.PHY] Cell 0 | DL 0.00 Mbps 0 Slots | UL 0.00 Mbps 100 Slots CRC 100 ( 16599) | Tick 332000
    Line 18293: 04:31:18.160004 WRN 3091 0 [SCF.PHY] Cell 0 | DL 0.00 Mbps 0 Slots | UL 0.00 Mbps 100 Slots CRC 100 ( 16699) | Tick 334000
    Line 18394: 04:31:19.160003 WRN 3091 0 [SCF.PHY] Cell 0 | DL 0.00 Mbps 0 Slots | UL 0.00 Mbps 100 Slots CRC 100 ( 16799) | Tick 336000

  3. for the TestMac(L2) on the DU host(host2: DEL R740+A100), it looks like there is no UL data received.
    04:31:08.160006 WRN 3087 0 [MAC.FAPI] Cell 0 | DL 0.00 Mbps 0 Slots | UL 0.00 Mbps 100 Slots | Prmb 0 | HARQ 0 | SR 0 | CSI1 0 | CSI2 0 | SRS 0 | ERR 0 | INV 200 | Slots 314000
    04:31:09.160006 WRN 3087 0 [MAC.FAPI] Cell 0 | DL 0.00 Mbps 0 Slots | UL 0.00 Mbps 100 Slots | Prmb 0 | HARQ 0 | SR 0 | CSI1 0 | CSI2 0 | SRS 0 | ERR 0 | INV 200 | Slots 316000
    04:31:10.160007 WRN 3087 0 [MAC.FAPI] Cell 0 | DL 0.00 Mbps 0 Slots | UL 0.00 Mbps 100 Slots | Prmb 0 | HARQ 0 | SR 0 | CSI1 0 | CSI2 0 | SRS 0 | ERR 0 | INV 200 | Slots 318000
    04:31:11.160006 WRN 3087 0 [MAC.FAPI] Cell 0 | DL 0.00 Mbps 0 Slots | UL 0.00 Mbps 100 Slots | Prmb 0 | HARQ 0 | SR 0 | CSI1 0 | CSI2 0 | SRS 0 | ERR 0 | INV 200 | Slots 320000
    04:31:12.160006 WRN 3087 0 [MAC.FAPI] Cell 0 | DL 0.00 Mbps 0 Slots | UL 0.00 Mbps 100 Slots | Prmb 0 | HARQ 0 | SR 0 | CSI1 0 | CSI2 0 | SRS 0 | ERR 0 | INV 200 | Slots 322000
    04:31:13.160007 WRN 3087 0 [MAC.FAPI] Cell 0 | DL 0.00 Mbps 0 Slots | UL 0.00 Mbps 100 Slots | Prmb 0 | HARQ 0 | SR 0 | CSI1 0 | CSI2 0 | SRS 0 | ERR 0 | INV 200 | Slots 324000
    04:31:14.160006 WRN 3087 0 [MAC.FAPI] Cell 0 | DL 0.00 Mbps 0 Slots | UL 0.00 Mbps 100 Slots | Prmb 0 | HARQ 0 | SR 0 | CSI1 0 | CSI2 0 | SRS 0 | ERR 0 | INV 200 | Slots 326000
    04:31:15.160007 WRN 3087 0 [MAC.FAPI] Cell 0 | DL 0.00 Mbps 0 Slots | UL 0.00 Mbps 100 Slots | Prmb 0 | HARQ 0 | SR 0 | CSI1 0 | CSI2 0 | SRS 0 | ERR 0 | INV 200 | Slots 328000
    04:31:16.160006 WRN 3087 0 [MAC.FAPI] Cell 0 | DL 0.00 Mbps 0 Slots | UL 0.00 Mbps 100 Slots | Prmb 0 | HARQ 0 | SR 0 | CSI1 0 | CSI2 0 | SRS 0 | ERR 0 | INV 200 | Slots 330000
    04:31:17.160006 WRN 3087 0 [MAC.FAPI] Cell 0 | DL 0.00 Mbps 0 Slots | UL 0.00 Mbps 100 Slots | Prmb 0 | HARQ 0 | SR 0 | CSI1 0 | CSI2 0 | SRS 0 | ERR 0 | INV 200 | Slots 332000
    04:31:18.160007 WRN 3087 0 [MAC.FAPI] Cell 0 | DL 0.00 Mbps 0 Slots | UL 0.00 Mbps 100 Slots | Prmb 0 | HARQ 0 | SR 0 | CSI1 0 | CSI2 0 | SRS 0 | ERR 0 | INV 200 | Slots 334000

    --------------------------------------------------debugging analyze-----------------------------------------------------------
    I checked the NIC on the RU server(host1: DEL R740), there are C-Plane messages received from DU server, but no UL packets send to DU server:
    DU-NIC:b8:ce:f6:8e:44:a6
    RU-NIC: 08:c0:eb:53:ca:a8

------------------------Tcpdump for RU NIC----------------------------------

19 0.036682 Mellanox_8e:44:a6 → IEEEI&MS_00:00:00 PTPv2 60 Delay_Req Message
20 0.036692 Mellanox_53:ca:a8 → IEEEI&MS_00:00:00 PTPv2 68 Delay_Resp Message
21 0.037497 Mellanox_8e:44:a6 → 20:04:9b:9e:27:a3 O-RAN-FH-C 60 C-Plane, Type: 1 (Most channels), Id: 0 (PRB: 0-59)
22 0.037497 Mellanox_8e:44:a6 → 20:04:9b:9e:27:a3 O-RAN-FH-C 60 C-Plane, Type: 1 (Most channels), Id: 0 (PRB: 0-59)
23 0.037497 Mellanox_8e:44:a6 → 20:04:9b:9e:27:a3 O-RAN-FH-C 60 C-Plane, Type: 1 (Most channels), Id: 0 (PRB: 0-59)
24 0.037497 Mellanox_8e:44:a6 → 20:04:9b:9e:27:a3 O-RAN-FH-C 60 C-Plane, Type: 1 (Most channels), Id: 0 (PRB: 0-59)
25 0.047496 Mellanox_8e:44:a6 → 20:04:9b:9e:27:a3 O-RAN-FH-C 60 C-Plane, Type: 1 (Most channels), Id: 0 (PRB: 0-59)
26 0.047496 Mellanox_8e:44:a6 → 20:04:9b:9e:27:a3 O-RAN-FH-C 60 C-Plane, Type: 1 (Most channels), Id: 0 (PRB: 0-59)
27 0.047496 Mellanox_8e:44:a6 → 20:04:9b:9e:27:a3 O-RAN-FH-C 60 C-Plane, Type: 1 (Most channels), Id: 0 (PRB: 0-59)
28 0.047496 Mellanox_8e:44:a6 → 20:04:9b:9e:27:a3 O-RAN-FH-C 60 C-Plane, Type: 1 (Most channels), Id: 0 (PRB: 0-59)
//PTP sync messages and C-Plane messages from DU

------------------------Tcpdump for DU NIC----------------------------------
31 0.400296 Mellanox_53:ca:a8 → IEEEI&MS_00:00:00 PTPv2 60 Follow_Up Message
32 0.459719 Mellanox_8e:44:a6 → IEEEI&MS_00:00:00 PTPv2 58 Delay_Req Message
33 0.459762 Mellanox_53:ca:a8 → IEEEI&MS_00:00:00 PTPv2 68 Delay_Resp Message
34 0.462783 Mellanox_53:ca:a8 → IEEEI&MS_00:00:00 PTPv2 60 Sync Message
35 0.462795 Mellanox_53:ca:a8 → IEEEI&MS_00:00:00 PTPv2 60 Follow_Up Message
36 0.462808 Mellanox_53:ca:a8 → IEEEI&MS_00:00:00 PTPv2 78 Announce Message
37 0.475780 Mellanox_8e:44:a6 → IEEEI&MS_00:00:00 PTPv2 58 Delay_Req Message
// only PTP sync messages, no UL packets


It looks like the UL packets are sending by ru_emulator (on host1 RU server), but for some reason, those packes are not reaching the NIC on the RU server.
my understanding is : the ru_emulator will pass those UL packets to DPDK and DPDK will be responsable for delivering them to NIC.
But I am not sure how to step forward to debug the DPDK.
Can any experts give a hand on it?

Thank you very much.

Junsong W