Hi Aerial team,
I want to run OAI L2/L3 and Aerial L1(26.1) on DGX Spark. PTP is synced on both DU and RU (screenshots attached).
Using Benetel RU for the setup. I captured packets on aerial00 port while the stack is running. I’m able to see only Uplink U-plane messages. No Downlink messages are there.
Attached the running logs of oai-gnb and nvcubb.
oai-gnb-aerial-dgx_logs.txt (20.4 KB)
nv-cubb_logs.txt (76.2 KB)
config files of oai, cuphycontroller, l2adapter, dockercompose file also attached. sfn slot doesn’t match unpacked one! PHY 0 L2->L1 0.0 vs L1->L2 0.3 error comes always. And in nvcubb log also the slot changes from healthy to unhealthy frequently. Cores 4-19 were isolated. RU timing parameters also matched. The same RU worked fine with OAI 7.2 split and other setups also.
l2_adapter_config_P5G_DGX.yaml.txt (4.9 KB)
gnb-vnf.sa.band78.273prb.aerial_benetel.conf.txt (8.0 KB)
cuphycontroller_P5G_WNC_DGX_BENETEL.yaml.txt (22.3 KB)
docker-compose.yaml.txt (1.8 KB)
Hi @sathishn ,
According to the provided nv-cubb_logs.txt, L1 didn’t receive any U-plane packets. This would be a network configuration issue.
- According to the Wireshark screenshot, the destination MAC address is not the one for ConnectX-7. Was the destination MAC address at the O-RU correct?
- Aerial’s NIC flow rule looks at not only the MAC address but also VLAN TCI (ID + PCP + DEI). According to the screenshot, PRI (PCP) was set to 0, but the PCP is set to 7 in the provided cuphycontroller YAML. Does the O-RU have the same value, i.e., 7? If you set it to 7 at the O-RU, your network switch may change it. You should use 0 in Aerial and the O-RU in the case.
- How did you capture the packets? If you captured this using tcpdump specifying the interface
aerial00, the packets went to the kernel. It was supposed to be directly forwarded to the memory by the RDMA and to be able to capture the packet specifying mlx5_0. For the details of the tcpdump for RDMA packets, please take a look at this topic.
Thank you,
Nori
Hi @nhashimoto , thanks for the reply.
MAC addresses and VLAN IDs are matched on both ends.
In Benetel RU, there is no option to set PCP. So I configured the switch to modify PCP to 7 from all packets(ingress and egress) from DU and RU. And I compiled the custom tcpdump and able to see both DL and UL traffic on mlx5_0 port(aerial 00).
But in RU, all packets are showing late and Tx 0 only. Moreover in 4x4 scenario, the RX_TOTAL count is also very less as compared to other setups. The count should be roughly x4 of whatever is coming. If I change the ru-type to 3 in L1 config file, the count is decreasing more.
The same RU and timing parameters worked perfectly fine in OAI 7.2 and other setups. I tried keeping PCP as 0 in L1 config file. same issue.
Attached the logs of gnb and nv-cubb. The “Dyn index -1 setting as unhealthy!” error also coming in some runs. SFN mismatch error also there. I made tuned-adm profile to realtime as well.
oai-gnb-aerial-dgx_logs.txt (28.5 KB)
nv-cubb_logs.txt (81.1 KB)
Hi @sathishn,
Can you replace the following values with the ones in the bottom, which we have used for a Benetel O-RU?
### BENETEL Timing Parameters according to OAI
T1a_max_cp_ul_ns: 336000
T1a_min_cp_ul_ns: 285000
T1a_max_cp_dl_ns: 470000
T1a_min_cp_dl_ns: 419000
T1a_max_up_ns: 345000 # T1a_max_cp_dl - Tcp_adv_dl
T1a_min_up_ns: 294000
Ta4_min_ns: 0 #20000
Ta4_max_ns: 200000 #150000 # Ta3_max_up + T34max
Tcp_adv_dl_ns: 125000
T1a_max_up_ns: 280000
T1a_max_cp_ul_ns: 405000
Ta4_min_ns: 50000
Ta4_max_ns: 331000
Tcp_adv_dl_ns: 125000
Hi @nhashimoto
I tried changing the parameters given by you. Mobile is able to detect the cell but Dataplane is not flowing. Issue is same with the parameters Im using too. Mobile detects the cell but RACH not happening.
Only LCID1 one is showing in gNB logs. Control and data bearers(LCID 2 and 4) are not established. RACH process is not successful at the gNB.
nv-cubb_logs.txt (60.0 KB)
oai-gnb-aerial-dgx_logs.txt (147.5 KB)
According to the nv-cubb log, there is no error related to PRACH in L1.
Next, you should check if the UE/O-RU is sending/receiving PRACH.
If your radio environment has no or less noise, you can see some higher magnitudes in U-plane IQ samples when the UE is transmitting PRACH.
Im testing in a lab environment. The distance between RU and UE is hardly 10 meters. trying to capture FAPI messages. but getting the error. nvipc_pcap in /var/log/aerial is showing 0 bytes. I already added nvidia-fapi lua file in wireshark.
./opt/nvidia/cuBB/build.aarch64/cuPHY-CP/gt_common_libs/nvIPC/tests/pcap/pcap_collect nvIPC .
[C]: Usage: ./pcap_collect
[C]: Current run: ./pcap_collect name=nvIPC dest_path=.
[I]: shmlogger_collect_ex: save /var/log/aerial/nvIPC_pcap and /dev/shm/nvIPC_pcap logs to ./nvIPC_pcap
[E][AERIAL_SYSTEM_API_EVENT]: ipc_shm_open: shm_open nvIPC_pcap failed error -1
[E][AERIAL_NVIPC_API_EVENT]: nv_ipc_shm_open: primary=0 name=nvIPC_pcap size=8388680 Failed
[E][AERIAL_SYSTEM_API_EVENT]: ipc_shm_close: close shm_fd failed
[E][AERIAL_NVIPC_API_EVENT]: shmlogger_open: nv_ipc_shm_open failed
[I]: shmlogger_collect_ex: no /dev/shm/nvIPC_pcap, logger may have been closed normally
how can I capture FAPI packets and see? The problem is gNB is not receiving any MSG3 from UE.
below is the l2adapter file im using.
l2_adapter_config_P5G_DGX_BENETEL.txt (4.9 KB)
This command will search for the exact file name specified by the first argument. In this case, it will search for nvIPC_pcap in /dev/shm/ and /var/log/aerial.
./opt/nvidia/cuBB/build.aarch64/cuPHY-CP/gt_common_libs/nvIPC/tests/pcap/pcap_collect nvIPC .
By default, it is nvipc_pcap. So, you should run:
./opt/nvidia/cuBB/build.aarch64/cuPHY-CP/gt_common_libs/nvIPC/tests/pcap/pcap_collect nvipc .
Can you share this output?
ls -l /dev/shm/*pcap /var/log/aerial/*pcap 2>/dev/null
The problem is gNB is not receiving any MSG3 from UE. below is the l2adapter file im using.
Do you mean you confirmed that L2 received a RACH.indication from L1, but MSG3 on PUSCH failed?
Thank you.
ls -l /dev/shm/*pcap /var/log/aerial/*pcap 2>/dev/null
-rwxr-xr-x 1 root root 545259592 Sep 29 09:29 /dev/shm/nvipc_pcap
-rw------- 1 root root 32 Sep 29 09:29 /dev/shm/sem.nvipc_pcap
-rw-r–r-- 1 root root 0 Sep 29 09:29 /var/log/aerial/nvipc_pcap
Please run this command:
./opt/nvidia/cuBB/build.aarch64/cuPHY-CP/gt_common_libs/nvIPC/tests/pcap/pcap_collect nvipc .
./pcap_collect nvipc .
[C]: Usage: ./pcap_collect
[C]: Current run: ./pcap_collect name=nvipc dest_path=.
[I]: shmlogger_collect_ex: save /var/log/aerial/nvipc_pcap and /dev/shm/nvipc_pcap logs to ./nvipc_pcap
[E][AERIAL_SYSTEM_API_EVENT]: ipc_shm_open: shm_open nvipc_pcap failed error -1
[E][AERIAL_NVIPC_API_EVENT]: nv_ipc_shm_open: primary=0 name=nvipc_pcap size=8388680 Failed
[E][AERIAL_SYSTEM_API_EVENT]: ipc_shm_close: close shm_fd failed
[E][AERIAL_NVIPC_API_EVENT]: shmlogger_open: nv_ipc_shm_open failed
[I]: shmlogger_collect_ex: no /dev/shm/nvipc_pcap, logger may have been closed normally
still the same issue. Running inside the nv-cubb container only.
I think it’s just a permission issue because “root” owns all the files. Can you run the same command with sudo?
ls -l /dev/shm/\*pcap /var/log/aerial/\*pcap 2>/dev/null
-rwxr-xr-x 1 root root 545259592 Sep 29 09:29 /dev/shm/nvipc_pcap
-rw------- 1 root root 32 Sep 29 09:29 /dev/shm/sem.nvipc_pcap
-rw-r–r-- 1 root root 0 Sep 29 09:29 /var/log/aerial/nvipc_pcap
We are able to open the nvipc_pcap file. In UL_TTI.REQUEST, we see only a PRACH PDU. Other UL PDUs are not present. Is this because msg3 failure is happening? (refer to the attached log file). We see MSg3 failed due to max HARQ limit reached. Also on the Benetel RU, the TX_Total (kpi.sh) is most of the times 0 (occasionally some counters show up)
msg3_failure.log (1000 Bytes)
Can you find any RACH.indication message?
If L2 allocates PUSCH for Msg3, you should find the message and some PUSCH PDUs after it.
Also, if L1 receives PUSCH PDUs, you should find CRC.indication.
CRC.indication has UL SINR. It helps you understand whether low SINR was the cause of the failure.
In our test with a Benetel O-RU, we used the following values, which are parameters in the cuphycontroller YAML file. Please try this as well.
fs_offset_dl: 6
fs_offset_ul: 7
We are able to see PUSCH and PUCCH PDUs. The UL SINR we see in the CRC.indication is 0.062 dB. We also tried with the given fs_offset_dl and fs_offset_ul. Still we see that no DRBs getting activated. However SRBs are occasionally getting activated and we see LCID 1 (Sometimes msg4 is getting scheduled and sometimes not).
oai_logs_lcid_1.txt (147.1 KB)
Please refer to the attached log file and CRC.indication msg screenshot from wireshark.
Also we are using the below file as the wireshark dissector but it errors out on CONFIG.Request as well as several parameters in other packets. It seems to downloaded from ngc (It is .lua but uploaded as .txt due to extension restrictions).
aerial-fapi-dissector_26-1.txt (222.6 KB)
OAI supports only SCF FAPI v10.02. This is written in this document. I think your dissector preference specifies v10.04. Please check it again with v10.02.
Hi @nhashimoto .
We are able to see ACK for MSG4. But immediately after that, the UE again trying RACH. RSRP out of range is coming. and “SCF_ERROR_CODE_L1_MISSING_UL_IQ” error comes after MSG4 ACK.
Attached the logs.
oai-gnb-aerial-dgx_logs.txt (227.5 KB)
nv-cubb_logs.txt (60.8 KB)
cuphycontroller_P5G_WNC_DGX_BENETEL.yaml.txt (10.3 KB)
This error corresponds to the error below in nv-cubb_logs.txt. The cubb error means that the DU server didn’t receive UL U-plane packets continuously.
10740.761612 [W] 3464294752: nr_error_ind_cb: [VNF] Received error code 0x44 (SCF_ERROR_CODE_L1_MISSING_UL_IQ)
06:05:49.201291 ERR UlPhyDriver05 0 [AERIAL_CUPHY_API_EVENT] [DRV.FUNC_UL] SFN 123.18 Slot Map 385 cell index 0 Dyn index -1 setting as unhealthy!
This error also indicates that UL U-plane packets for PUSCH didn’t arrive.
3913 Cell 0 Order kernel sem_idx_rx 796 last_sem_idx_rx 747 sem_idx_order 796 last_sem_idx_order 747 PUSCH PRBs 0/2184 PRACH PRBs 0/0. First packet received timeout after 1500000 ns F118S6S1 done = 0 current_time=10803178892864,last_timeout_log_time=10790824067296,total_rx_pkts=56
Could you confirm if your fronthaul network connection has enough bandwidth and O-RU’s fronthaul parameters for PUSCH, such as eAxC IDs, VLAN IDs, PCP, etc., match with those configured in the cuphycontroller YAML file?
RSRP out of range is coming.
I think fs_offset doesn’t match between the O-RU and DU. According to this document, it can be confirmed on the O-RU console. Please try to see the current values of Tx and Rx and configure the matched values in the cuphycontroller YAML file.
Thank you.
fs_offset parameter is related to M-plane configuration of the RU. But in our setup, we completely disabled the M-plane as we dont have any DHCP server on DU side. Is M-plane mandatory to change fs_offset values in RU?
Is M-plane mandatory to change fs_offset values in RU?
Please ask the O-RU vendor. We don’t have much experience with Benetel O-RUs.
