Segmentation fault at aerial l1 while running with oai l2-l3

shivank.chaudhary · August 8, 2024, 4:10pm

Hello,
I’m trying to run aerial l1 with oai l2-l3
setup details:
aerial version: 24-2
oai branch: develop

error at oai side:

[C]: [nvipc] Waiting for nvipc server to start ...
[C]: [nvipc] Waiting for nvipc server to start ... 
[C]: [nvipc] nvipc unix socket client connected
[C]: [nvipc] Received peer event_fd: 102
[C]: Share event_fd succeed: efd_tx=102, efd_rx=100
[C]: [nvipc][core 13 ] nvipc unix socket exit
[C]: Start initialize nvipc client
[C]: shm_ipc_open: forward_enable=0 fw_max_msg_buf_count=0 fw_max_data_buf_count=0
[C]: create_shm_nv_ipc_interface: OK
[NFAPI_VNF]   nvIPC_Init: create IPC interface successful
[UTIL]   threadCreate() for vnf_nvipc_aerial: creating thread with affinity ffffffff, priority 50 
[VNF] pnf connection indication idx:1 
Try to send first CONFIG.request  
708307618745 [W]  562431872: pack_nr_tlv: Warning pack_tlv tag 0x1ed8 does not match expected 0x1027  
708307618751 [W]  562431872: pack_nr_tlv: Warning pack_tlv tag 0x83b0 does not match expected 0x1027
Entering ITTI signals handler
TYPE <CTRL-C> TO TERMINATE 
[NFAPI_VNF]   Received CONFIG.response, gNB is ready! 
708307619571 [E]  483935808: nfapi_vnf_pnf_list_find: nfapi_vnf_pnf_list_find : curr->p5_idx:1 p5_idx:1

at aerial side:

====> cuPHYController initialized, L1 is ready!
13:27:37.073595 WRN phy_main 0 [CTL.STARTUP_TIMES] {TI PERCENTAGE} <cuphycontroller main> Start Main:0.3,Parse Cuphycontroller YAML:0.0,Init nvlog:0.3,Cuda Set Device:19.4,Cuphy PTI Init:0.1,Init PHYDriver:68.7,Init cuphydriver:0.0,Make PHYDriverProxy:0.0,Init SCF FAPI:0.0,Create PHY_group:11.2,Start PHY_group:0.0, (total: 4455076.5us),

13:27:37.073599 WRN phy_main 0 [CTL.STARTUP_TIMES] {TI DURATION} <cuphycontroller main> Start Main:11272.981,Parse Cuphycontroller YAML:589.793,Init nvlog:13016.387,Cuda Set Device:865617.901,Cuphy PTI Init:5114.295,Init PHYDriver:3061305.077,Init cuphydriver:0.690,Make PHYDriverProxy:8.650,Init SCF FAPI:6.850,Create PHY_group:497883.123,Start PHY_group:260.705,

13:27:37.073601 WRN phy_main 0 [CTL.STARTUP_TIMES] {TI TIMESTAMPS} <cuphycontroller main> Start Main:1723123652618502281,Parse Cuphycontroller YAML:1723123652629775262,Init nvlog:1723123652630365055,Cuda Set Device:1723123652643381442,Cuphy PTI Init:1723123653508999343,Init PHYDriver:1723123653514113638,Init cuphydriver:1723123656575418715,Make PHYDriverProxy:1723123656575419405,Init SCF FAPI:1723123656575428055,Create PHY_group:1723123656575434905,Start PHY_group:1723123657073318028,End Main:1723123657073578733,

13:27:37.079147 WRN msg_processing 0 [L2A.MODULE] Thread thread_func on CPU 15 initialized fmtlog
13:27:37.755941 WRN 137 0 [NVIPC.EFD] [nvipc] nvipc unix socket server connected
13:27:37.756047 WRN 137 0 [NVIPC.EFD] [nvipc] Received peer event_fd: 207
13:27:37.956188 WRN 137 0 [NVIPC.EFD] Share event_fd succeed: efd_tx=207, efd_rx=216
13:27:37.956191 WRN 137 0 [NVIPC.EFD] [nvipc][core 17 ] nvipc unix socket exit
./aerial_l1_entrypoint.sh: line 37:    51 Segmentation fault      (core dumped) "$cuBB_Path"/build/cuPHY-CP/cuphycontroller/examples/cuphycontroller_scf "$argument"

please find the detailed log file attached.

Thanks
aerial.txt (35.8 KB)
oai.txt (17.3 KB)

shivank.chaudhary · August 8, 2024, 4:11pm

also this the docker-compose used:

services:
  nv-cubb:
    container_name: nv-cubb
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities:
                - gpu
    network_mode: host
    shm_size: 4096m
    privileged: true
    stdin_open: true
    tty: true
    volumes:
      - /lib/modules:/lib/modules
      - /dev/hugepages:/dev/hugepages
      - /usr/src:/usr/src
      - ./aerial_l1_entrypoint.sh:/opt/nvidia/cuBB/aerial_l1_entrypoint.sh
      - /var/log/aerial:/var/log/aerial
      - ../../../cmake_targets/share:/opt/cuBB/share
    userns_mode: host
    ipc: "shareable"
    image: aerial-l1:e2e-TV-half-build
    environment:
      - cuBB_SDK=/opt/nvidia/cuBB
    #command: bash -c "sudo rm -rf /tmp/phy.log && sudo chmod +x /opt/nvidia/cuBB/aerial_l1_entrypoint.sh && /opt/nvidia/cuBB/aerial_l1_entrypoint.sh"
    command: bash -c "sleep 100000"
    healthcheck:
      test: ["CMD-SHELL",'grep -q "L1 is ready!" /tmp/phy.log && echo 0 || echo 1']
      interval: 20s
      timeout: 5s
      retries: 5
  oai-gnb-aerial:
    cpuset: "13,14,15,16,17,18,19,20"
    image: oai-gnb-aerial:develop
    depends_on:
      nv-cubb:
        condition: service_healthy
    privileged: true
    ipc: "container:nv-cubb"
    environment:
      USE_ADDITIONAL_OPTIONS: --log_config.global_log_options level,nocolor,time
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities:
                - gpu
    network_mode: host
    shm_size: 4096m
    stdin_open: true
    tty: true
    volumes:
      - ../../conf_files/gnb-vnf.sa.band78.273prb.aerial.conf:/opt/oai-gnb/etc/gnb.conf
    container_name: oai-gnb-aerial
    command: bash -c "chrt -f 99 /opt/oai-gnb/bin/nr-softmodem -O /opt/oai-gnb/etc/gnb.conf"
    healthcheck:
      test: /bin/bash -c "ps aux | grep -v grep | grep -c softmodem"
      interval: 10s
      timeout: 5s
      retries: 5

bkecicioglu · August 8, 2024, 4:38pm

Hi @shivank.chaudhary ,

cuphycontroller does not see the NIC according to the logs below:

AL: Probe PCI driver: gpu_cuda (10de:24b0) device: 0000:8a:00.0 (socket 0)
[18:11:55:314149][52][DOCA][ERR][doca_dev.cpp:1118][doca_dev_as_devinfo] Failed to convert device to devinfo. parameter dev=NULL
[18:11:55:314161][52][DOCA][ERR][doca_eth_txq.c:3518][doca_eth_txq_cap_get_wait_on_time_offload_supported] Failed to get wait_on_time_offload_supported: parameter devinfo=NULL
[18:11:55:314175][52][DOCA][ERR][doca_dpdk.cpp:526][doca_dpdk_port_probe] Invalid input: dev=(nil), devargs=0x39fda1476e0
Invalid port_id=255
18:11:55.314123 ERR phy_init 0 [AERIAL_DPDK_API_EVENT] [FH.NIC] Matching device not found.
18:11:55.314146 ERR phy_init 0 [AERIAL_DPDK_API_EVENT] [FH.NIC] open_doca_device_with_pci returned Requested Resource Not Found
18:11:55.314165 ERR phy_init 0 [AERIAL_DPDK_API_EVENT] [FH.NIC] doca_eth_txq_get_wait_on_time_offload_supported returned Invalid input
18:11:55.314180 ERR phy_init 0 [AERIAL_DPDK_API_EVENT] [FH.NIC] doca_dpdk_port_probe returned Invalid input
18:11:55.314180 ERR phy_init 0 [AERIAL_DPDK_API_EVENT] [FH.NIC] get_dpdk_port_id_doca_dev returned Invalid input
18:11:55.314210 ERR phy_init 0 [AERIAL_DPDK_API_EVENT] [FH.NIC] Failed to get device info for NIC 0000:51:00.0: No such device
18:11:55.314287 ERR phy_init 0 [AERIAL_ORAN_FH_EVENT] [FH.LIB] Exception! basic_string: construction from null is not valid
18:11:55.314291 ERR phy_init 0 [AERIAL_ORAN_FH_EVENT] [DRV.FH] Failed to add NIC 0000:51:00.0

Can you please check the NIC configuration in your cuphycontroller configuration yaml file and make sure the NIC address is matching with your system?

You can find these steps in Aerial documentation under “Running cuBB end-to-end” in section “Updating Configuration Files for End-to-End”

Thank you.

shivank.chaudhary · August 8, 2024, 4:54pm

to resolve this I tried aerial with the following NICs:

Ethernet Controller E810-XXV for SFP 159b' if=ens2f0 drv=ice unused=vfio-pci

Ethernet Controller X710/X557-AT 10GBASE-T 1589' if=enp193s0f0 drv=i40e unused=vfio-pci

but it’s not getting detected

shivank.chaudhary · August 8, 2024, 4:57pm

Thanks for the reply @bkecicioglu
one thing I want to clarify,
is this error:

[NFAPI_VNF]   Received CONFIG.response, gNB is ready! 
708307619571 [E]  483935808: nfapi_vnf_pnf_list_find: nfapi_vnf_pnf_list_find : curr->p5_idx:1 p5_idx:1

related to nic not getting on-boarded or is it related to something else?

bkecicioglu · August 8, 2024, 5:10pm

Can you please provide the output of the following commands?

nvidia-smi

nvidia-smi topo -m

sudo lshw -c network -businfo

Thanks.

bkecicioglu · August 8, 2024, 5:13pm

This does not seem to be related. It is related to the FAPI interface.

shivank.chaudhary · August 8, 2024, 5:22pm

sure

nvidia-smi
Thu Aug  8 17:21:21 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 555.42.02              Driver Version: 555.42.02      CUDA Version: 12.5     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA A100 80GB PCIe          Off |   00000000:01:00.0 Off |                    0 |
| N/A   35C    P0             45W /  300W |      44MiB /  81920MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA A100 80GB PCIe          Off |   00000000:A1:00.0 Off |                    0 |
| N/A   40C    P0             48W /  300W |      40MiB /  81920MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A   1789871      C   nvidia-cuda-mps-server                         30MiB |
|    1   N/A  N/A   1789871      C   nvidia-cuda-mps-server                         30MiB |
+-----------------------------------------------------------------------------------------+

nvidia-smi topo -m
        GPU0    GPU1    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      SYS     0-15,32-47      0               N/A
GPU1    SYS      X      16-31,48-63     1               N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

 sudo lshw -c network -businfo
[sudo] password for ubuntu:
Bus info          Device        Class          Description
==========================================================
pci@0000:21:00.0  enp33s0f0     network        I350 Gigabit Network Connection
pci@0000:21:00.1  enp33s0f1     network        I350 Gigabit Network Connection
pci@0000:21:00.2  enp33s0f2     network        I350 Gigabit Network Connection
pci@0000:21:00.3  enp33s0f3     network        I350 Gigabit Network Connection
pci@0000:c1:00.0  enp193s0f0    network        Ethernet Controller X710/X557-AT 10GBASE-T
pci@0000:c1:00.1  enp193s0f1    network        Ethernet Controller X710/X557-AT 10GBASE-T
pci@0000:c1:00.2  enp193s0f2    network        Ethernet Controller X710/X557-AT 10GBASE-T
pci@0000:c1:00.3  enp193s0f3    network        Ethernet Controller X710/X557-AT 10GBASE-T

bkecicioglu · August 8, 2024, 8:09pm

Hi @shivank.chaudhary,

The NIC cards you have in your system are not supported by Aerial. CX6DX NIC is recommended for A100 GPU. Please also see https://docs.nvidia.com/aerial/aerial-ran-colab-ota/current/text/installation_guide/procure_the_hardware.html for the HW manifest information

Thank you.

shivank.chaudhary · August 9, 2024, 4:17am

@bkecicioglu thanks a lot for the help.
For oai I tried 2024-w15, 2024-w21, develop and 2024-w21+Arc1.5 branches,
but the maximum progress I got with develop. (error Mentioned above in thread)
Is there any other branch you would like to suggest?

jixu · August 9, 2024, 4:45am

@shivank.chaudhary
develop and 2024-w21+Arc1.5 branches is the latest release for ARC-OTA. We recommend you using this one.

shivank.chaudhary · August 10, 2024, 5:45am

Hi @jixu
I’m now using cubb 24-01
and at the time of running, it throws this error:

aerial@gpu-node:/opt/nvidia/cuBB$ sudo ./aerial_l1_entrypoint.sh P5G_FXN_R750
Started cuphycontroller on CPU core 38
AERIAL_LOG_PATH unset
Using default log path
Log file set to /tmp/phy.log
Aerial metrics backend address: 127.0.0.1:8081
05:41:21.291983 WRN phy_init 0 [CTL.SCF] Config file: /opt/nvidia/cuBB/cuPHY-CP/cuphycontroller/config/cuphycontroller_P5G_FXN_R750.yaml
05:41:21.292598 WRN phy_init 0 [CTL.SCF] low_priority_core=14
05:41:21.293000 WRN phy_init 0 [NVLOG.CPP] Using /opt/nvidia/cuBB/cuPHY/nvlog/config/nvlog_config.yaml for nvlog configuration
YAML invalid key: ul_order_timeout_gpu_log_enable Using default value of 0 to YAML_PARAM_UL_ORDER_TIMEOUT_GPU_LOG_ENABLE
YAML invalid key: ue_mode Using default value of 0 to YAML_PARAM_UE_MODE
YAML invalid key: enable_l1_param_sanity_check Using default value of 0 to YAML_PARAM_ENABLE_L1_PARAM_SANITY_CHECK
YAML invalid key: disable_empw Using default value of 0 to YAML_PARAM_DISABLE_EMPW
YAML invalid key: ul_rx_pkt_tracing_level Using default value of 0 to YAML_PARAM_UL_RX_PKT_TRACING_LEVEL
YAML invalid key: enable_h2d_copy_thread Using default value of 0 to YAML_PARAM_ENABLE_H2D_COPY_THREAD
YAML invalid key: h2d_copy_thread_cpu_affinity Using default value of 29 to YAML_PARAM_H2D_COPY_THREAD_CPU_AFFINITY
YAML invalid key: h2d_copy_thread_sched_priority Using default value of 0 to YAML_PARAM_H2D_COPY_THREAD_SCHED_PRIORITY
YAML invalid key: aggr_obj_non_avail_th Using default value of 5 to YAML_PARAM_AGGR_OBJ_NON_AVAIL_TH
YAML invalid key: sendCPlane_timing_error_th_ns Using default value of 50 us to YAML_PARAM_SENDCPLANE_TIMING_ERROR_TH_NS
YAML invalid key: pusch_waitTimeOutPreEarlyHarqUs Using default value of 1100 to PUSCH-WAITTIMEOUTPREEHQUS
YAML invalid key: pusch_waitTimeOutPostEarlyHarqUs Using default value of 1500 to PUSCH-WAITTIMEOUTPOSTEHQUS
YAML invalid key: puxch_polarDcdrListSz Using default value of 1 to PUSCH_POLAR_DCDR_LIST_SZ
YAML invalid key: split_ul_cuda_streams Using default value of 0 to YAML_PARAM_SPLIT_UL_CUDA_STREAMS
YAML invalid key: serialize_pucch_pusch Using default value of 0 to YAML_PARAM_SERIALIZE_PUCCH_PUSCH
YAML invalid key: ul_order_max_rx_pkts Using default value of 0 to UL_ORDER_MAX_RX_PKTS
YAML invalid key: ul_order_timeout_log_interval_ns Using default value of 1s to YAML_PARAM_UL_ORDER_TIMEOUT_LOG_INTERVAL_NS
05:41:21.305102 ERR phy_init 0 [AERIAL_CONFIG_EVENT] [CTL.YAML] cuphycontroller config. yaml does not have mps_sm_ul_order key; defaulting to 16.
05:41:21.305103 ERR phy_init 0 [AERIAL_CONFIG_EVENT] [CTL.YAML] cuphycontroller config. yaml does not have mps_sm_gpu_comms key; defaulting to old count of 8 SMs.
05:41:21.305343 WRN phy_init 0 [CTL.YAML] cell_id 1 nic_index :0
05:41:21.305411 WRN phy_init 0 [CTL.YAML] pusch_nMaxPrb not set in config file, using default of 273 PRB allocation
05:41:21.305415 WRN phy_init 0 [CTL.YAML] pusch_nMaxRx not set in config file, using default value of 0
05:41:21.305418 WRN phy_init 0 [CTL.YAML] ul_u_plane_tx_offset_ns not set in config file, using default of 280 us
05:41:21.305424 WRN phy_init 0 [CTL.YAML] cell_id 2 nic_index :0
05:41:21.305450 WRN phy_init 0 [CTL.YAML] pusch_nMaxPrb not set in config file, using default of 273 PRB allocation
05:41:21.305453 WRN phy_init 0 [CTL.YAML] pusch_nMaxRx not set in config file, using default value of 0
05:41:21.305456 WRN phy_init 0 [CTL.YAML] ul_u_plane_tx_offset_ns not set in config file, using default of 280 us
05:41:21.305466 WRN phy_init 0 [CTL.YAML] cell_id 3 nic_index :0
05:41:21.305491 WRN phy_init 0 [CTL.YAML] pusch_nMaxPrb not set in config file, using default of 273 PRB allocation
05:41:21.305494 WRN phy_init 0 [CTL.YAML] pusch_nMaxRx not set in config file, using default value of 0
05:41:21.305497 WRN phy_init 0 [CTL.YAML] ul_u_plane_tx_offset_ns not set in config file, using default of 280 us
05:41:21.305502 WRN phy_init 0 [CTL.YAML] cell_id 4 nic_index :0
05:41:21.305527 WRN phy_init 0 [CTL.YAML] pusch_nMaxPrb not set in config file, using default of 273 PRB allocation
05:41:21.305529 WRN phy_init 0 [CTL.YAML] pusch_nMaxRx not set in config file, using default value of 0
05:41:21.305532 WRN phy_init 0 [CTL.YAML] ul_u_plane_tx_offset_ns not set in config file, using default of 280 us
05:41:21.305550 WRN phy_init 0 [CTL.YAML] Num Slots: 8
05:41:21.305551 WRN phy_init 0 [CTL.YAML] Enable UL cuPHY Graphs: 1
05:41:21.305551 WRN phy_init 0 [CTL.YAML] Enable DL cuPHY Graphs: 1
05:41:21.305551 WRN phy_init 0 [CTL.YAML] Accurate TX scheduling clock resolution (ns): 500
05:41:21.305551 WRN phy_init 0 [CTL.YAML] DPDK core: 14
05:41:21.305552 WRN phy_init 0 [CTL.YAML] Prometheus core: -1
05:41:21.305552 WRN phy_init 0 [CTL.YAML] UL cores:
05:41:21.305552 WRN phy_init 0 [CTL.YAML]       - 5
05:41:21.305552 WRN phy_init 0 [CTL.YAML]       - 7
05:41:21.305552 WRN phy_init 0 [CTL.YAML] DL cores:
05:41:21.305552 WRN phy_init 0 [CTL.YAML]       - 9
05:41:21.305552 WRN phy_init 0 [CTL.YAML]       - 11
05:41:21.305552 WRN phy_init 0 [CTL.YAML]       - 13
05:41:21.305552 WRN phy_init 0 [CTL.YAML] Debug worker: -1
05:41:21.305552 WRN phy_init 0 [CTL.YAML] Data Lake core: -1
05:41:21.305552 WRN phy_init 0 [CTL.YAML] SRS starting Section ID: 3072
05:41:21.305552 WRN phy_init 0 [CTL.YAML] PRACH starting Section ID: 2048
05:41:21.305552 WRN phy_init 0 [CTL.YAML] MPS SM PUSCH: 108
05:41:21.305552 WRN phy_init 0 [CTL.YAML] MPS SM PUCCH: 16
05:41:21.305552 WRN phy_init 0 [CTL.YAML] MPS SM PRACH: 16
05:41:21.305552 WRN phy_init 0 [CTL.YAML] MPS SM UL ORDER: 16
05:41:21.305553 WRN phy_init 0 [CTL.YAML] MPS SM PDSCH: 82
05:41:21.305553 WRN phy_init 0 [CTL.YAML] MPS SM PDCCH: 28
05:41:21.305553 WRN phy_init 0 [CTL.YAML] MPS SM PBCH: 14
05:41:21.305553 WRN phy_init 0 [CTL.YAML] MPS SM GPU_COMMS: 8
05:41:21.305553 WRN phy_init 0 [CTL.YAML] PDSCH fallback: 0
05:41:21.305553 WRN phy_init 0 [CTL.YAML] Massive MIMO enable: 0
05:41:21.305553 WRN phy_init 0 [CTL.YAML] Enable SRS : 0
05:41:21.305553 WRN phy_init 0 [CTL.YAML] ul_order_timeout_gpu_log_enable: 0
05:41:21.305553 WRN phy_init 0 [CTL.YAML] ue_mode: 0
05:41:21.305554 WRN phy_init 0 [CTL.YAML] Aggr Obj Non-availability threshold: 5
05:41:21.305554 WRN phy_init 0 [CTL.YAML] sendCPlane_timing_error_th_ns: 50000
05:41:21.305554 WRN phy_init 0 [CTL.YAML] ul_order_timeout_gpu_log_enable: 0
05:41:21.305554 WRN phy_init 0 [CTL.YAML] GPU-initiated comms DL: 1
05:41:21.305554 WRN phy_init 0 [CTL.YAML] Cell group: 1
05:41:21.305554 WRN phy_init 0 [CTL.YAML] Cell group num: 1
05:41:21.305554 WRN phy_init 0 [CTL.YAML] puxchPolarDcdrListSz: 1
05:41:21.305555 WRN phy_init 0 [CTL.YAML] split_ul_cuda_streams: 0
05:41:21.305555 WRN phy_init 0 [CTL.YAML] serialize_pucch_pusch: 0
05:41:21.305555 WRN phy_init 0 [CTL.YAML] Number of Cell Configs: 4
05:41:21.305555 WRN phy_init 0 [CTL.YAML] L2Adapter config file: /opt/nvidia/cuBB/cuPHY-CP/cuphycontroller/config/l2_adapter_config_P5G_R750.yaml
05:41:21.305555 WRN phy_init 0 [CTL.YAML] Cell name: O-RU 0
05:41:21.305555 WRN phy_init 0 [CTL.YAML]       MU: 1
05:41:21.305555 WRN phy_init 0 [CTL.YAML]       ID: 1
05:41:21.305555 WRN phy_init 0 [CTL.YAML] Cell name: O-RU 1
05:41:21.305555 WRN phy_init 0 [CTL.YAML]       MU: 1
05:41:21.305555 WRN phy_init 0 [CTL.YAML]       ID: 2
05:41:21.305556 WRN phy_init 0 [CTL.YAML] Cell name: O-RU 2
05:41:21.305556 WRN phy_init 0 [CTL.YAML]       MU: 1
05:41:21.305556 WRN phy_init 0 [CTL.YAML]       ID: 3
05:41:21.305556 WRN phy_init 0 [CTL.YAML] Cell name: O-RU 3
05:41:21.305556 WRN phy_init 0 [CTL.YAML]       MU: 1
05:41:21.305556 WRN phy_init 0 [CTL.YAML]       ID: 4
05:41:21.305556 WRN phy_init 0 [CTL.YAML] Number of MPlane Configs: 4
05:41:21.305556 WRN phy_init 0 [CTL.YAML]       Mplane ID: 1
05:41:21.305556 WRN phy_init 0 [CTL.YAML]       VLAN ID: 2
05:41:21.305556 WRN phy_init 0 [CTL.YAML]       Source Eth Address: 00:00:00:00:00:00
05:41:21.305556 WRN phy_init 0 [CTL.YAML]       Destination Eth Address: 6c:ad:ad:00:04:6c
05:41:21.305556 WRN phy_init 0 [CTL.YAML]       NIC port: 0000:cc:00.1
05:41:21.305556 WRN phy_init 0 [CTL.YAML]       RU Type: 1
05:41:21.305556 WRN phy_init 0 [CTL.YAML]       U-plane TXQs: 1
05:41:21.305556 WRN phy_init 0 [CTL.YAML]       DL compression method: 1
05:41:21.305556 WRN phy_init 0 [CTL.YAML]       DL iq bit width: 9
05:41:21.305556 WRN phy_init 0 [CTL.YAML]       UL compression method: 1
05:41:21.305556 WRN phy_init 0 [CTL.YAML]       UL iq bit width: 9
05:41:21.305556 WRN phy_init 0 [CTL.YAML]
05:41:21.305556 WRN phy_init 0 [CTL.YAML]       Flow list SSB/PBCH:
05:41:21.305557 WRN phy_init 0 [CTL.YAML]               0
05:41:21.305557 WRN phy_init 0 [CTL.YAML]               1
05:41:21.305557 WRN phy_init 0 [CTL.YAML]               2
05:41:21.305557 WRN phy_init 0 [CTL.YAML]               3
05:41:21.305557 WRN phy_init 0 [CTL.YAML]       Flow list PDCCH:
05:41:21.305557 WRN phy_init 0 [CTL.YAML]               0
05:41:21.305557 WRN phy_init 0 [CTL.YAML]               1
05:41:21.305557 WRN phy_init 0 [CTL.YAML]               2
05:41:21.305557 WRN phy_init 0 [CTL.YAML]               3
05:41:21.305557 WRN phy_init 0 [CTL.YAML]       Flow list PDSCH:
05:41:21.305557 WRN phy_init 0 [CTL.YAML]               0
05:41:21.305558 WRN phy_init 0 [CTL.YAML]               1
05:41:21.305558 WRN phy_init 0 [CTL.YAML]               2
05:41:21.305558 WRN phy_init 0 [CTL.YAML]               3
05:41:21.305558 WRN phy_init 0 [CTL.YAML]       Flow list CSIRS:
05:41:21.305558 WRN phy_init 0 [CTL.YAML]               0
05:41:21.305558 WRN phy_init 0 [CTL.YAML]               1
05:41:21.305558 WRN phy_init 0 [CTL.YAML]               2
05:41:21.305558 WRN phy_init 0 [CTL.YAML]               3
05:41:21.305558 WRN phy_init 0 [CTL.YAML]       Flow list PUSCH:
05:41:21.305558 WRN phy_init 0 [CTL.YAML]               0
05:41:21.305558 WRN phy_init 0 [CTL.YAML]               1
05:41:21.305558 WRN phy_init 0 [CTL.YAML]               2
05:41:21.305558 WRN phy_init 0 [CTL.YAML]               3
05:41:21.305558 WRN phy_init 0 [CTL.YAML]       Flow list PUCCH:
05:41:21.305558 WRN phy_init 0 [CTL.YAML]               0
05:41:21.305558 WRN phy_init 0 [CTL.YAML]               1
05:41:21.305558 WRN phy_init 0 [CTL.YAML]               2
05:41:21.305558 WRN phy_init 0 [CTL.YAML]               3
05:41:21.305558 WRN phy_init 0 [CTL.YAML]       Flow list SRS:
05:41:21.305558 WRN phy_init 0 [CTL.YAML]               0
05:41:21.305558 WRN phy_init 0 [CTL.YAML]               1
05:41:21.305558 WRN phy_init 0 [CTL.YAML]               2
05:41:21.305558 WRN phy_init 0 [CTL.YAML]               3
05:41:21.305558 WRN phy_init 0 [CTL.YAML]       Flow list PRACH:
05:41:21.305558 WRN phy_init 0 [CTL.YAML]               4
05:41:21.305558 WRN phy_init 0 [CTL.YAML]               5
05:41:21.305558 WRN phy_init 0 [CTL.YAML]               6
05:41:21.305558 WRN phy_init 0 [CTL.YAML]               7
05:41:21.305558 WRN phy_init 0 [CTL.YAML]       PUSCH TV: /opt/nvidia/cuBB/testVectors/cuPhyChEstCoeffs.h5
05:41:21.305559 WRN phy_init 0 [CTL.YAML]       SRS TV: /opt/nvidia/cuBB/testVectors/cuPhyChEstCoeffs.h5
05:41:21.305559 WRN phy_init 0 [CTL.YAML]       Section_3 time offset: 58369
05:41:21.305559 WRN phy_init 0 [CTL.YAML]       nMaxRxAnt: 4
05:41:21.305559 WRN phy_init 0 [CTL.YAML]       PUSCH PRBs Stride: 273
05:41:21.305559 WRN phy_init 0 [CTL.YAML]       PRACH PRBs Stride: 12
05:41:21.305559 WRN phy_init 0 [CTL.YAML]       SRS PRBs Stride: 273
05:41:21.305559 WRN phy_init 0 [CTL.YAML]       PUSCH nMaxPrb: 273
05:41:21.305559 WRN phy_init 0 [CTL.YAML]       PUSCH nMaxRx: 0
05:41:21.305559 WRN phy_init 0 [CTL.YAML]       UL Gain Calibration: 48.68
05:41:21.305559 WRN phy_init 0 [CTL.YAML]       Lower guard bw: 845
05:41:21.305559 WRN phy_init 0 [CTL.YAML]       Mplane ID: 2
05:41:21.305559 WRN phy_init 0 [CTL.YAML]       VLAN ID: 2
05:41:21.305559 WRN phy_init 0 [CTL.YAML]       Source Eth Address: 00:00:00:00:00:00
05:41:21.305559 WRN phy_init 0 [CTL.YAML]       Destination Eth Address: 6c:ad:ad:00:04:68
05:41:21.305559 WRN phy_init 0 [CTL.YAML]       NIC port: 0000:cc:00.1
05:41:21.305559 WRN phy_init 0 [CTL.YAML]       RU Type: 1
05:41:21.305559 WRN phy_init 0 [CTL.YAML]       U-plane TXQs: 1
05:41:21.305559 WRN phy_init 0 [CTL.YAML]       DL compression method: 1
05:41:21.305559 WRN phy_init 0 [CTL.YAML]       DL iq bit width: 9
05:41:21.305559 WRN phy_init 0 [CTL.YAML]       UL compression method: 1
05:41:21.305559 WRN phy_init 0 [CTL.YAML]       UL iq bit width: 9
05:41:21.305559 WRN phy_init 0 [CTL.YAML]
05:41:21.305559 WRN phy_init 0 [CTL.YAML]       Flow list SSB/PBCH:
05:41:21.305559 WRN phy_init 0 [CTL.YAML]               0
05:41:21.305559 WRN phy_init 0 [CTL.YAML]               1
05:41:21.305559 WRN phy_init 0 [CTL.YAML]               2
05:41:21.305559 WRN phy_init 0 [CTL.YAML]               3
05:41:21.305559 WRN phy_init 0 [CTL.YAML]       Flow list PDCCH:
05:41:21.305559 WRN phy_init 0 [CTL.YAML]               0
05:41:21.305559 WRN phy_init 0 [CTL.YAML]               1
05:41:21.305559 WRN phy_init 0 [CTL.YAML]               2
05:41:21.305559 WRN phy_init 0 [CTL.YAML]               3
05:41:21.305559 WRN phy_init 0 [CTL.YAML]       Flow list PDSCH:
05:41:21.305559 WRN phy_init 0 [CTL.YAML]               0
05:41:21.305559 WRN phy_init 0 [CTL.YAML]               1
05:41:21.305559 WRN phy_init 0 [CTL.YAML]               2
05:41:21.305559 WRN phy_init 0 [CTL.YAML]               3
05:41:21.305559 WRN phy_init 0 [CTL.YAML]       Flow list CSIRS:
05:41:21.305559 WRN phy_init 0 [CTL.YAML]               0
05:41:21.305560 WRN phy_init 0 [CTL.YAML]               1
05:41:21.305560 WRN phy_init 0 [CTL.YAML]               2
05:41:21.305560 WRN phy_init 0 [CTL.YAML]               3
05:41:21.305560 WRN phy_init 0 [CTL.YAML]       Flow list PUSCH:
05:41:21.305560 WRN phy_init 0 [CTL.YAML]               0
05:41:21.305560 WRN phy_init 0 [CTL.YAML]               1
05:41:21.305560 WRN phy_init 0 [CTL.YAML]               2
05:41:21.305560 WRN phy_init 0 [CTL.YAML]               3
05:41:21.305560 WRN phy_init 0 [CTL.YAML]       Flow list PUCCH:
05:41:21.305560 WRN phy_init 0 [CTL.YAML]               0
05:41:21.305560 WRN phy_init 0 [CTL.YAML]               1
05:41:21.305560 WRN phy_init 0 [CTL.YAML]               2
05:41:21.305560 WRN phy_init 0 [CTL.YAML]               3
05:41:21.305560 WRN phy_init 0 [CTL.YAML]       Flow list SRS:
05:41:21.305560 WRN phy_init 0 [CTL.YAML]               0
05:41:21.305560 WRN phy_init 0 [CTL.YAML]               1
05:41:21.305560 WRN phy_init 0 [CTL.YAML]               2
05:41:21.305560 WRN phy_init 0 [CTL.YAML]               3
05:41:21.305560 WRN phy_init 0 [CTL.YAML]       Flow list PRACH:
05:41:21.305560 WRN phy_init 0 [CTL.YAML]               4
05:41:21.305560 WRN phy_init 0 [CTL.YAML]               5
05:41:21.305560 WRN phy_init 0 [CTL.YAML]               6
05:41:21.305560 WRN phy_init 0 [CTL.YAML]               7
05:41:21.305560 WRN phy_init 0 [CTL.YAML]       PUSCH TV: /opt/nvidia/cuBB/testVectors/cuPhyChEstCoeffs.h5
05:41:21.305560 WRN phy_init 0 [CTL.YAML]       SRS TV: /opt/nvidia/cuBB/testVectors/cuPhyChEstCoeffs.h5
05:41:21.305560 WRN phy_init 0 [CTL.YAML]       Section_3 time offset: 58369
05:41:21.305560 WRN phy_init 0 [CTL.YAML]       nMaxRxAnt: 4
05:41:21.305560 WRN phy_init 0 [CTL.YAML]       PUSCH PRBs Stride: 273
05:41:21.305560 WRN phy_init 0 [CTL.YAML]       PRACH PRBs Stride: 12
05:41:21.305560 WRN phy_init 0 [CTL.YAML]       SRS PRBs Stride: 273
05:41:21.305560 WRN phy_init 0 [CTL.YAML]       PUSCH nMaxPrb: 273
05:41:21.305560 WRN phy_init 0 [CTL.YAML]       PUSCH nMaxRx: 0
05:41:21.305560 WRN phy_init 0 [CTL.YAML]       UL Gain Calibration: 48.68
05:41:21.305560 WRN phy_init 0 [CTL.YAML]       Lower guard bw: 845
05:41:21.305560 WRN phy_init 0 [CTL.YAML]       Mplane ID: 3
05:41:21.305560 WRN phy_init 0 [CTL.YAML]       VLAN ID: 2
05:41:21.305560 WRN phy_init 0 [CTL.YAML]       Source Eth Address: 00:00:00:00:00:00
05:41:21.305560 WRN phy_init 0 [CTL.YAML]       Destination Eth Address: 6c:ad:ad:00:02:00
05:41:21.305560 WRN phy_init 0 [CTL.YAML]       NIC port: 0000:cc:00.1
05:41:21.305560 WRN phy_init 0 [CTL.YAML]       RU Type: 1
05:41:21.305560 WRN phy_init 0 [CTL.YAML]       U-plane TXQs: 1
05:41:21.305560 WRN phy_init 0 [CTL.YAML]       DL compression method: 1
05:41:21.305561 WRN phy_init 0 [CTL.YAML]       DL iq bit width: 9
05:41:21.305561 WRN phy_init 0 [CTL.YAML]       UL compression method: 1
05:41:21.305561 WRN phy_init 0 [CTL.YAML]       UL iq bit width: 9
05:41:21.305561 WRN phy_init 0 [CTL.YAML]
05:41:21.305561 WRN phy_init 0 [CTL.YAML]       Flow list SSB/PBCH:
05:41:21.305561 WRN phy_init 0 [CTL.YAML]               0
05:41:21.305561 WRN phy_init 0 [CTL.YAML]               1
05:41:21.305561 WRN phy_init 0 [CTL.YAML]               2
05:41:21.305561 WRN phy_init 0 [CTL.YAML]               3
05:41:21.305561 WRN phy_init 0 [CTL.YAML]       Flow list PDCCH:
05:41:21.305561 WRN phy_init 0 [CTL.YAML]               0
05:41:21.305561 WRN phy_init 0 [CTL.YAML]               1
05:41:21.305561 WRN phy_init 0 [CTL.YAML]               2
05:41:21.305561 WRN phy_init 0 [CTL.YAML]               3
05:41:21.305561 WRN phy_init 0 [CTL.YAML]       Flow list PDSCH:
05:41:21.305561 WRN phy_init 0 [CTL.YAML]               0
05:41:21.305561 WRN phy_init 0 [CTL.YAML]               1
05:41:21.305561 WRN phy_init 0 [CTL.YAML]               2
05:41:21.305561 WRN phy_init 0 [CTL.YAML]               3
05:41:21.305561 WRN phy_init 0 [CTL.YAML]       Flow list CSIRS:
05:41:21.305561 WRN phy_init 0 [CTL.YAML]               0
05:41:21.305561 WRN phy_init 0 [CTL.YAML]               1
05:41:21.305561 WRN phy_init 0 [CTL.YAML]               2
05:41:21.305561 WRN phy_init 0 [CTL.YAML]               3
05:41:21.305561 WRN phy_init 0 [CTL.YAML]       Flow list PUSCH:
05:41:21.305561 WRN phy_init 0 [CTL.YAML]               0
05:41:21.305561 WRN phy_init 0 [CTL.YAML]               1
05:41:21.305561 WRN phy_init 0 [CTL.YAML]               2
05:41:21.305561 WRN phy_init 0 [CTL.YAML]               3
05:41:21.305561 WRN phy_init 0 [CTL.YAML]       Flow list PUCCH:
05:41:21.305561 WRN phy_init 0 [CTL.YAML]               0
05:41:21.305561 WRN phy_init 0 [CTL.YAML]               1
05:41:21.305561 WRN phy_init 0 [CTL.YAML]               2
05:41:21.305561 WRN phy_init 0 [CTL.YAML]               3
05:41:21.305561 WRN phy_init 0 [CTL.YAML]       Flow list SRS:
05:41:21.305561 WRN phy_init 0 [CTL.YAML]               0
05:41:21.305561 WRN phy_init 0 [CTL.YAML]               1
05:41:21.305561 WRN phy_init 0 [CTL.YAML]               2
05:41:21.305561 WRN phy_init 0 [CTL.YAML]               3
05:41:21.305561 WRN phy_init 0 [CTL.YAML]       Flow list PRACH:
05:41:21.305561 WRN phy_init 0 [CTL.YAML]               4
05:41:21.305561 WRN phy_init 0 [CTL.YAML]               5
05:41:21.305561 WRN phy_init 0 [CTL.YAML]               6
05:41:21.305561 WRN phy_init 0 [CTL.YAML]               7
05:41:21.305561 WRN phy_init 0 [CTL.YAML]       PUSCH TV: /opt/nvidia/cuBB/testVectors/cuPhyChEstCoeffs.h5
05:41:21.305561 WRN phy_init 0 [CTL.YAML]       SRS TV: /opt/nvidia/cuBB/testVectors/cuPhyChEstCoeffs.h5
05:41:21.305562 WRN phy_init 0 [CTL.YAML]       Section_3 time offset: 58369
05:41:21.305562 WRN phy_init 0 [CTL.YAML]       nMaxRxAnt: 4
05:41:21.305562 WRN phy_init 0 [CTL.YAML]       PUSCH PRBs Stride: 273
05:41:21.305562 WRN phy_init 0 [CTL.YAML]       PRACH PRBs Stride: 12
05:41:21.305562 WRN phy_init 0 [CTL.YAML]       SRS PRBs Stride: 273
05:41:21.305562 WRN phy_init 0 [CTL.YAML]       PUSCH nMaxPrb: 273
05:41:21.305562 WRN phy_init 0 [CTL.YAML]       PUSCH nMaxRx: 0
05:41:21.305562 WRN phy_init 0 [CTL.YAML]       UL Gain Calibration: 48.68
05:41:21.305562 WRN phy_init 0 [CTL.YAML]       Lower guard bw: 845
05:41:21.305562 WRN phy_init 0 [CTL.YAML]       Mplane ID: 4
05:41:21.305562 WRN phy_init 0 [CTL.YAML]       VLAN ID: 2
05:41:21.305562 WRN phy_init 0 [CTL.YAML]       Source Eth Address: 00:00:00:00:00:00
05:41:21.305562 WRN phy_init 0 [CTL.YAML]       Destination Eth Address: 6c:ad:ad:00:04:70
05:41:21.305562 WRN phy_init 0 [CTL.YAML]       NIC port: 0000:cc:00.1
05:41:21.305562 WRN phy_init 0 [CTL.YAML]       RU Type: 1
05:41:21.305562 WRN phy_init 0 [CTL.YAML]       U-plane TXQs: 1
05:41:21.305562 WRN phy_init 0 [CTL.YAML]       DL compression method: 1
05:41:21.305562 WRN phy_init 0 [CTL.YAML]       DL iq bit width: 9
05:41:21.305562 WRN phy_init 0 [CTL.YAML]       UL compression method: 1
05:41:21.305562 WRN phy_init 0 [CTL.YAML]       UL iq bit width: 9
05:41:21.305562 WRN phy_init 0 [CTL.YAML]
05:41:21.305562 WRN phy_init 0 [CTL.YAML]       Flow list SSB/PBCH:
05:41:21.305562 WRN phy_init 0 [CTL.YAML]               0
05:41:21.305562 WRN phy_init 0 [CTL.YAML]               1
05:41:21.305562 WRN phy_init 0 [CTL.YAML]               2
05:41:21.305562 WRN phy_init 0 [CTL.YAML]               3
05:41:21.305562 WRN phy_init 0 [CTL.YAML]       Flow list PDCCH:
05:41:21.305562 WRN phy_init 0 [CTL.YAML]               0
05:41:21.305562 WRN phy_init 0 [CTL.YAML]               1
05:41:21.305562 WRN phy_init 0 [CTL.YAML]               2
05:41:21.305562 WRN phy_init 0 [CTL.YAML]               3
05:41:21.305562 WRN phy_init 0 [CTL.YAML]       Flow list PDSCH:
05:41:21.305562 WRN phy_init 0 [CTL.YAML]               0
05:41:21.305562 WRN phy_init 0 [CTL.YAML]               1
05:41:21.305562 WRN phy_init 0 [CTL.YAML]               2
05:41:21.305562 WRN phy_init 0 [CTL.YAML]               3
05:41:21.305562 WRN phy_init 0 [CTL.YAML]       Flow list CSIRS:
05:41:21.305562 WRN phy_init 0 [CTL.YAML]               0
05:41:21.305562 WRN phy_init 0 [CTL.YAML]               1
05:41:21.305562 WRN phy_init 0 [CTL.YAML]               2
05:41:21.305562 WRN phy_init 0 [CTL.YAML]               3
05:41:21.305562 WRN phy_init 0 [CTL.YAML]       Flow list PUSCH:
05:41:21.305562 WRN phy_init 0 [CTL.YAML]               0
05:41:21.305562 WRN phy_init 0 [CTL.YAML]               1
05:41:21.305562 WRN phy_init 0 [CTL.YAML]               2
05:41:21.305562 WRN phy_init 0 [CTL.YAML]               3
05:41:21.305562 WRN phy_init 0 [CTL.YAML]       Flow list PUCCH:
05:41:21.305562 WRN phy_init 0 [CTL.YAML]               0
05:41:21.305562 WRN phy_init 0 [CTL.YAML]               1
05:41:21.305562 WRN phy_init 0 [CTL.YAML]               2
05:41:21.305562 WRN phy_init 0 [CTL.YAML]               3
05:41:21.305562 WRN phy_init 0 [CTL.YAML]       Flow list SRS:
05:41:21.305562 WRN phy_init 0 [CTL.YAML]               0
05:41:21.305562 WRN phy_init 0 [CTL.YAML]               1
05:41:21.305562 WRN phy_init 0 [CTL.YAML]               2
05:41:21.305562 WRN phy_init 0 [CTL.YAML]               3
05:41:21.305562 WRN phy_init 0 [CTL.YAML]       Flow list PRACH:
05:41:21.305562 WRN phy_init 0 [CTL.YAML]               4
05:41:21.305562 WRN phy_init 0 [CTL.YAML]               5
05:41:21.305562 WRN phy_init 0 [CTL.YAML]               6
05:41:21.305562 WRN phy_init 0 [CTL.YAML]               7
05:41:21.305562 WRN phy_init 0 [CTL.YAML]       PUSCH TV: /opt/nvidia/cuBB/testVectors/cuPhyChEstCoeffs.h5
05:41:21.305562 WRN phy_init 0 [CTL.YAML]       SRS TV: /opt/nvidia/cuBB/testVectors/cuPhyChEstCoeffs.h5
05:41:21.305562 WRN phy_init 0 [CTL.YAML]       Section_3 time offset: 58369
05:41:21.305562 WRN phy_init 0 [CTL.YAML]       nMaxRxAnt: 4
05:41:21.305562 WRN phy_init 0 [CTL.YAML]       PUSCH PRBs Stride: 273
05:41:21.305562 WRN phy_init 0 [CTL.YAML]       PRACH PRBs Stride: 12
05:41:21.305562 WRN phy_init 0 [CTL.YAML]       SRS PRBs Stride: 273
05:41:21.305562 WRN phy_init 0 [CTL.YAML]       PUSCH nMaxPrb: 273
05:41:21.305562 WRN phy_init 0 [CTL.YAML]       PUSCH nMaxRx: 0
05:41:21.305562 WRN phy_init 0 [CTL.YAML]       UL Gain Calibration: 48.68
05:41:21.305562 WRN phy_init 0 [CTL.YAML]       Lower guard bw: 845
EAL: Detected CPU lcores: 64
EAL: Detected NUMA nodes: 2
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/cuphycontroller/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Probe PCI driver: gpu_cuda (10de:20b5) device: 0000:01:00.0 (socket 0)
[05:41:24:506784][504][DOCA][ERR][doca_eth_txq.c:2274][doca_eth_txq_get_wait_on_time_offload_supported] Failed to get wait_on_time_offload_supported: parameter dev=NULL
[05:41:24:506807][504][DOCA][ERR][doca_dpdk.cpp:465][doca_dpdk_port_probe] Invalid input: dev=(nil), devargs=0x530e1df0720
Invalid port_id=255
05:41:24.506765 ERR phy_init 0 [AERIAL_DPDK_API_EVENT] [FH.NIC] Matching device not found.
05:41:24.506779 ERR phy_init 0 [AERIAL_DPDK_API_EVENT] [FH.NIC] open_doca_device_with_pci returned Requested Resource Not Found
05:41:24.506799 ERR phy_init 0 [AERIAL_DPDK_API_EVENT] [FH.NIC] doca_eth_txq_get_wait_on_time_offload_supported returned Invalid input
05:41:24.506814 ERR phy_init 0 [AERIAL_DPDK_API_EVENT] [FH.NIC] doca_dpdk_port_probe returned Invalid input
05:41:24.506814 ERR phy_init 0 [AERIAL_DPDK_API_EVENT] [FH.NIC] get_dpdk_port_id_doca_dev returned Invalid input
05:41:24.506838 ERR phy_init 0 [AERIAL_DPDK_API_EVENT] [FH.NIC] Failed to get device info for NIC 0000:c1:00.0: No such device
05:41:24.506899 ERR phy_init 0 [AERIAL_ORAN_FH_EVENT] [FH.LIB] Exception! basic_string::_M_construct null not valid
05:41:24.506901 ERR phy_init 0 [AERIAL_ORAN_FH_EVENT] [DRV.FH] Failed to add NIC 0000:c1:00.0
terminate called after throwing an instance of 'pd_exc_h'
  what():  Invalid pointer: StaticConversion can't return nullptr
./aerial_l1_entrypoint.sh: line 37:   504 Aborted                 (core dumped) "$cuBB_Path"/build/cuPHY-CP/cuphycontroller/examples/cuphycontroller_scf "$argument"
aerial@gpu-node:/opt/nvidia/cuBB$

Thanks

bkecicioglu · August 10, 2024, 5:32pm

Hi @shivank.chaudhary,

The error message is the same as the previous logs you shared.

EAL: Probe PCI driver: gpu_cuda (10de:20b5) device: 0000:01:00.0 (socket 0)
[05:41:24:506784][504][DOCA][ERR][doca_eth_txq.c:2274][doca_eth_txq_get_wait_on_time_offload_supported] Failed to get wait_on_time_offload_supported: parameter dev=NULL
[05:41:24:506807][504][DOCA][ERR][doca_dpdk.cpp:465][doca_dpdk_port_probe] Invalid input: dev=(nil), devargs=0x530e1df0720
Invalid port_id=255
05:41:24.506765 ERR phy_init 0 [AERIAL_DPDK_API_EVENT] [FH.NIC] Matching device not found.
05:41:24.506779 ERR phy_init 0 [AERIAL_DPDK_API_EVENT] [FH.NIC] open_doca_device_with_pci returned Requested Resource Not Found
05:41:24.506799 ERR phy_init 0 [AERIAL_DPDK_API_EVENT] [FH.NIC] doca_eth_txq_get_wait_on_time_offload_supported returned Invalid input
05:41:24.506814 ERR phy_init 0 [AERIAL_DPDK_API_EVENT] [FH.NIC] doca_dpdk_port_probe returned Invalid input
05:41:24.506814 ERR phy_init 0 [AERIAL_DPDK_API_EVENT] [FH.NIC] get_dpdk_port_id_doca_dev returned Invalid input
05:41:24.506838 ERR phy_init 0 [AERIAL_DPDK_API_EVENT] [FH.NIC] Failed to get device info for NIC 0000:c1:00.0: No such device
05:41:24.506899 ERR phy_init 0 [AERIAL_ORAN_FH_EVENT] [FH.LIB] Exception! basic_string::_M_construct null not valid
05:41:24.506901 ERR phy_init 0 [AERIAL_ORAN_FH_EVENT] [DRV.FH] Failed to add NIC 0000:c1:00.0

The reason for this message is the lack of NIC card.

Please see here .

Thank you.

shivank.chaudhary · August 11, 2024, 12:56pm

@bkecicioglu can I use this Network card:

pci@0000:3b:00.0  ens2f0np0        network        MT27800 Family [ConnectX-5]
pci@0000:3b:00.1  ens2f1np1        network        MT27800 Family [ConnectX-5]

--
3b:00.0 Ethernet controller [0200]: Mellanox Technologies MT27800 Family [ConnectX-5] [15b3:1017]
        Subsystem: Mellanox Technologies MT27800 Family [ConnectX-5] [15b3:0007]
        Kernel driver in use: mlx5_core
        Kernel modules: mlx5_core
3b:00.1 Ethernet controller [0200]: Mellanox Technologies MT27800 Family [ConnectX-5] [15b3:1017]
        Subsystem: Mellanox Technologies MT27800 Family [ConnectX-5] [15b3:0007]
        Kernel driver in use: mlx5_core
        Kernel modules: mlx5_core

is it supported?

Thanks

bkecicioglu · August 12, 2024, 6:58pm

Hi @shivank.chaudhary ,

It is not supported. We recommend using CX6DX NIC (please see here).

Thank you.

shivank.chaudhary · August 23, 2024, 4:03pm

Hi @bkecicioglu @jixu
I changed the nic now, its connect x6 MCX654105A-HCAT
and these are the logs of the aerial:

aerial@gpu-node:/opt/nvidia/cuBB$ sudo ./build/cuPHY-CP/cuphycontroller/examples/cuphycontroller_scf P5G_FXN_R750
Started cuphycontroller on CPU core 1
AERIAL_LOG_PATH unset
Using default log path
Log file set to /tmp/phy.log
Aerial metrics backend address: 127.0.0.1:8081
16:00:06.383055 WRN phy_init 0 [CTL.SCF] Config file: /opt/nvidia/cuBB/cuPHY-CP/cuphycontroller/config/cuphycontroller_P5G_FXN_R750.yaml
16:00:06.383587 WRN phy_init 0 [CTL.SCF] low_priority_core=17
16:00:06.383948 WRN phy_init 0 [NVLOG.CPP] Using /opt/nvidia/cuBB/cuPHY/nvlog/config/nvlog_config.yaml for nvlog configuration
YAML invalid key: ul_order_timeout_gpu_log_enable Using default value of 0 to YAML_PARAM_UL_ORDER_TIMEOUT_GPU_LOG_ENABLE
YAML invalid key: ue_mode Using default value of 0 to YAML_PARAM_UE_MODE
YAML invalid key: enable_l1_param_sanity_check Using default value of 0 to YAML_PARAM_ENABLE_L1_PARAM_SANITY_CHECK
YAML invalid key: disable_empw Using default value of 0 to YAML_PARAM_DISABLE_EMPW
YAML invalid key: ul_rx_pkt_tracing_level Using default value of 0 to YAML_PARAM_UL_RX_PKT_TRACING_LEVEL
YAML invalid key: enable_h2d_copy_thread Using default value of 0 to YAML_PARAM_ENABLE_H2D_COPY_THREAD
YAML invalid key: h2d_copy_thread_cpu_affinity Using default value of 29 to YAML_PARAM_H2D_COPY_THREAD_CPU_AFFINITY
YAML invalid key: h2d_copy_thread_sched_priority Using default value of 0 to YAML_PARAM_H2D_COPY_THREAD_SCHED_PRIORITY
YAML invalid key: aggr_obj_non_avail_th Using default value of 5 to YAML_PARAM_AGGR_OBJ_NON_AVAIL_TH
YAML invalid key: sendCPlane_timing_error_th_ns Using default value of 50 us to YAML_PARAM_SENDCPLANE_TIMING_ERROR_TH_NS
YAML invalid key: pusch_waitTimeOutPreEarlyHarqUs Using default value of 1100 to PUSCH-WAITTIMEOUTPREEHQUS
YAML invalid key: pusch_waitTimeOutPostEarlyHarqUs Using default value of 1500 to PUSCH-WAITTIMEOUTPOSTEHQUS
YAML invalid key: puxch_polarDcdrListSz Using default value of 1 to PUSCH_POLAR_DCDR_LIST_SZ
YAML invalid key: split_ul_cuda_streams Using default value of 0 to YAML_PARAM_SPLIT_UL_CUDA_STREAMS
YAML invalid key: serialize_pucch_pusch Using default value of 0 to YAML_PARAM_SERIALIZE_PUCCH_PUSCH
YAML invalid key: ul_order_max_rx_pkts Using default value of 0 to UL_ORDER_MAX_RX_PKTS
YAML invalid key: ul_order_timeout_log_interval_ns Using default value of 1s to YAML_PARAM_UL_ORDER_TIMEOUT_LOG_INTERVAL_NS
16:00:06.396146 ERR phy_init 0 [AERIAL_CONFIG_EVENT] [CTL.YAML] cuphycontroller config. yaml does not have mps_sm_ul_order key; defaulting to 16.
16:00:06.396146 ERR phy_init 0 [AERIAL_CONFIG_EVENT] [CTL.YAML] cuphycontroller config. yaml does not have mps_sm_gpu_comms key; defaulting to old count of 8 SMs.
16:00:06.396456 WRN phy_init 0 [CTL.YAML] cell_id 1 nic_index :0
16:00:06.396523 WRN phy_init 0 [CTL.YAML] pusch_nMaxPrb not set in config file, using default of 273 PRB allocation
16:00:06.396527 WRN phy_init 0 [CTL.YAML] pusch_nMaxRx not set in config file, using default value of 0
16:00:06.396530 WRN phy_init 0 [CTL.YAML] ul_u_plane_tx_offset_ns not set in config file, using default of 280 us
16:00:06.396537 WRN phy_init 0 [CTL.YAML] cell_id 2 nic_index :0
16:00:06.396564 WRN phy_init 0 [CTL.YAML] pusch_nMaxPrb not set in config file, using default of 273 PRB allocation
16:00:06.396567 WRN phy_init 0 [CTL.YAML] pusch_nMaxRx not set in config file, using default value of 0
16:00:06.396569 WRN phy_init 0 [CTL.YAML] ul_u_plane_tx_offset_ns not set in config file, using default of 280 us
16:00:06.396577 WRN phy_init 0 [CTL.YAML] cell_id 3 nic_index :0
16:00:06.396602 WRN phy_init 0 [CTL.YAML] pusch_nMaxPrb not set in config file, using default of 273 PRB allocation
16:00:06.396605 WRN phy_init 0 [CTL.YAML] pusch_nMaxRx not set in config file, using default value of 0
16:00:06.396608 WRN phy_init 0 [CTL.YAML] ul_u_plane_tx_offset_ns not set in config file, using default of 280 us
16:00:06.396613 WRN phy_init 0 [CTL.YAML] cell_id 4 nic_index :0
16:00:06.396639 WRN phy_init 0 [CTL.YAML] pusch_nMaxPrb not set in config file, using default of 273 PRB allocation
16:00:06.396642 WRN phy_init 0 [CTL.YAML] pusch_nMaxRx not set in config file, using default value of 0
16:00:06.396644 WRN phy_init 0 [CTL.YAML] ul_u_plane_tx_offset_ns not set in config file, using default of 280 us
16:00:06.396662 WRN phy_init 0 [CTL.YAML] Num Slots: 8
16:00:06.396662 WRN phy_init 0 [CTL.YAML] Enable UL cuPHY Graphs: 1
16:00:06.396662 WRN phy_init 0 [CTL.YAML] Enable DL cuPHY Graphs: 1
16:00:06.396663 WRN phy_init 0 [CTL.YAML] Accurate TX scheduling clock resolution (ns): 500
16:00:06.396663 WRN phy_init 0 [CTL.YAML] DPDK core: 17
16:00:06.396663 WRN phy_init 0 [CTL.YAML] Prometheus core: -1
16:00:06.396663 WRN phy_init 0 [CTL.YAML] UL cores:
16:00:06.396663 WRN phy_init 0 [CTL.YAML]       - 5
16:00:06.396663 WRN phy_init 0 [CTL.YAML]       - 7
16:00:06.396663 WRN phy_init 0 [CTL.YAML] DL cores:
16:00:06.396664 WRN phy_init 0 [CTL.YAML]       - 9
16:00:06.396664 WRN phy_init 0 [CTL.YAML]       - 11
16:00:06.396664 WRN phy_init 0 [CTL.YAML]       - 13
16:00:06.396664 WRN phy_init 0 [CTL.YAML] Debug worker: -1
16:00:06.396664 WRN phy_init 0 [CTL.YAML] Data Lake core: -1
16:00:06.396664 WRN phy_init 0 [CTL.YAML] SRS starting Section ID: 3072
16:00:06.396664 WRN phy_init 0 [CTL.YAML] PRACH starting Section ID: 2048
16:00:06.396664 WRN phy_init 0 [CTL.YAML] MPS SM PUSCH: 108
16:00:06.396664 WRN phy_init 0 [CTL.YAML] MPS SM PUCCH: 16
16:00:06.396664 WRN phy_init 0 [CTL.YAML] MPS SM PRACH: 16
16:00:06.396664 WRN phy_init 0 [CTL.YAML] MPS SM UL ORDER: 16
16:00:06.396664 WRN phy_init 0 [CTL.YAML] MPS SM PDSCH: 82
16:00:06.396664 WRN phy_init 0 [CTL.YAML] MPS SM PDCCH: 28
16:00:06.396664 WRN phy_init 0 [CTL.YAML] MPS SM PBCH: 14
16:00:06.396664 WRN phy_init 0 [CTL.YAML] MPS SM GPU_COMMS: 8
16:00:06.396664 WRN phy_init 0 [CTL.YAML] PDSCH fallback: 0
16:00:06.396664 WRN phy_init 0 [CTL.YAML] Massive MIMO enable: 0
16:00:06.396665 WRN phy_init 0 [CTL.YAML] Enable SRS : 0
16:00:06.396665 WRN phy_init 0 [CTL.YAML] ul_order_timeout_gpu_log_enable: 0
16:00:06.396665 WRN phy_init 0 [CTL.YAML] ue_mode: 0
16:00:06.396666 WRN phy_init 0 [CTL.YAML] Aggr Obj Non-availability threshold: 5
16:00:06.396666 WRN phy_init 0 [CTL.YAML] sendCPlane_timing_error_th_ns: 50000
16:00:06.396667 WRN phy_init 0 [CTL.YAML] ul_order_timeout_gpu_log_enable: 0
16:00:06.396667 WRN phy_init 0 [CTL.YAML] GPU-initiated comms DL: 1
16:00:06.396667 WRN phy_init 0 [CTL.YAML] Cell group: 1
16:00:06.396667 WRN phy_init 0 [CTL.YAML] Cell group num: 1
16:00:06.396667 WRN phy_init 0 [CTL.YAML] puxchPolarDcdrListSz: 1
16:00:06.396667 WRN phy_init 0 [CTL.YAML] split_ul_cuda_streams: 0
16:00:06.396667 WRN phy_init 0 [CTL.YAML] serialize_pucch_pusch: 0
16:00:06.396667 WRN phy_init 0 [CTL.YAML] Number of Cell Configs: 4
16:00:06.396667 WRN phy_init 0 [CTL.YAML] L2Adapter config file: /opt/nvidia/cuBB/cuPHY-CP/cuphycontroller/config/l2_adapter_config_P5G_R750.yaml
16:00:06.396667 WRN phy_init 0 [CTL.YAML] Cell name: O-RU 0
16:00:06.396668 WRN phy_init 0 [CTL.YAML]       MU: 1
16:00:06.396668 WRN phy_init 0 [CTL.YAML]       ID: 1
16:00:06.396668 WRN phy_init 0 [CTL.YAML] Cell name: O-RU 1
16:00:06.396668 WRN phy_init 0 [CTL.YAML]       MU: 1
16:00:06.396668 WRN phy_init 0 [CTL.YAML]       ID: 2
16:00:06.396668 WRN phy_init 0 [CTL.YAML] Cell name: O-RU 2
16:00:06.396668 WRN phy_init 0 [CTL.YAML]       MU: 1
16:00:06.396668 WRN phy_init 0 [CTL.YAML]       ID: 3
16:00:06.396668 WRN phy_init 0 [CTL.YAML] Cell name: O-RU 3
16:00:06.396668 WRN phy_init 0 [CTL.YAML]       MU: 1
16:00:06.396668 WRN phy_init 0 [CTL.YAML]       ID: 4
16:00:06.396668 WRN phy_init 0 [CTL.YAML] Number of MPlane Configs: 4
16:00:06.396668 WRN phy_init 0 [CTL.YAML]       Mplane ID: 1
16:00:06.396668 WRN phy_init 0 [CTL.YAML]       VLAN ID: 2
16:00:06.396669 WRN phy_init 0 [CTL.YAML]       Source Eth Address: 00:00:00:00:00:00
16:00:06.396669 WRN phy_init 0 [CTL.YAML]       Destination Eth Address: 6c:ad:ad:00:04:6c
16:00:06.396669 WRN phy_init 0 [CTL.YAML]       NIC port: 0000:cc:00.1
16:00:06.396669 WRN phy_init 0 [CTL.YAML]       RU Type: 1
16:00:06.396669 WRN phy_init 0 [CTL.YAML]       U-plane TXQs: 1
16:00:06.396669 WRN phy_init 0 [CTL.YAML]       DL compression method: 1
16:00:06.396669 WRN phy_init 0 [CTL.YAML]       DL iq bit width: 9
16:00:06.396669 WRN phy_init 0 [CTL.YAML]       UL compression method: 1
16:00:06.396669 WRN phy_init 0 [CTL.YAML]       UL iq bit width: 9
16:00:06.396669 WRN phy_init 0 [CTL.YAML]
16:00:06.396669 WRN phy_init 0 [CTL.YAML]       Flow list SSB/PBCH:
16:00:06.396669 WRN phy_init 0 [CTL.YAML]               0
16:00:06.396670 WRN phy_init 0 [CTL.YAML]               1
16:00:06.396670 WRN phy_init 0 [CTL.YAML]               2
16:00:06.396670 WRN phy_init 0 [CTL.YAML]               3
16:00:06.396670 WRN phy_init 0 [CTL.YAML]       Flow list PDCCH:
16:00:06.396670 WRN phy_init 0 [CTL.YAML]               0
16:00:06.396670 WRN phy_init 0 [CTL.YAML]               1
16:00:06.396670 WRN phy_init 0 [CTL.YAML]               2
16:00:06.396670 WRN phy_init 0 [CTL.YAML]               3
16:00:06.396670 WRN phy_init 0 [CTL.YAML]       Flow list PDSCH:
16:00:06.396670 WRN phy_init 0 [CTL.YAML]               0
16:00:06.396670 WRN phy_init 0 [CTL.YAML]               1
16:00:06.396670 WRN phy_init 0 [CTL.YAML]               2
16:00:06.396670 WRN phy_init 0 [CTL.YAML]               3
16:00:06.396670 WRN phy_init 0 [CTL.YAML]       Flow list CSIRS:
16:00:06.396671 WRN phy_init 0 [CTL.YAML]               0
16:00:06.396671 WRN phy_init 0 [CTL.YAML]               1
16:00:06.396671 WRN phy_init 0 [CTL.YAML]               2
16:00:06.396671 WRN phy_init 0 [CTL.YAML]               3
16:00:06.396671 WRN phy_init 0 [CTL.YAML]       Flow list PUSCH:
16:00:06.396671 WRN phy_init 0 [CTL.YAML]               0
16:00:06.396671 WRN phy_init 0 [CTL.YAML]               1
16:00:06.396671 WRN phy_init 0 [CTL.YAML]               2
16:00:06.396671 WRN phy_init 0 [CTL.YAML]               3
16:00:06.396671 WRN phy_init 0 [CTL.YAML]       Flow list PUCCH:
16:00:06.396671 WRN phy_init 0 [CTL.YAML]               0
16:00:06.396671 WRN phy_init 0 [CTL.YAML]               1
16:00:06.396671 WRN phy_init 0 [CTL.YAML]               2
16:00:06.396671 WRN phy_init 0 [CTL.YAML]               3
16:00:06.396671 WRN phy_init 0 [CTL.YAML]       Flow list SRS:
16:00:06.396671 WRN phy_init 0 [CTL.YAML]               0
16:00:06.396671 WRN phy_init 0 [CTL.YAML]               1
16:00:06.396671 WRN phy_init 0 [CTL.YAML]               2
16:00:06.396671 WRN phy_init 0 [CTL.YAML]               3
16:00:06.396671 WRN phy_init 0 [CTL.YAML]       Flow list PRACH:
16:00:06.396671 WRN phy_init 0 [CTL.YAML]               4
16:00:06.396671 WRN phy_init 0 [CTL.YAML]               5
16:00:06.396671 WRN phy_init 0 [CTL.YAML]               6
16:00:06.396671 WRN phy_init 0 [CTL.YAML]               7
16:00:06.396671 WRN phy_init 0 [CTL.YAML]       PUSCH TV: /opt/nvidia/cuBB/testVectors/cuPhyChEstCoeffs.h5
16:00:06.396671 WRN phy_init 0 [CTL.YAML]       SRS TV: /opt/nvidia/cuBB/testVectors/cuPhyChEstCoeffs.h5
16:00:06.396671 WRN phy_init 0 [CTL.YAML]       Section_3 time offset: 58369
16:00:06.396672 WRN phy_init 0 [CTL.YAML]       nMaxRxAnt: 4
16:00:06.396672 WRN phy_init 0 [CTL.YAML]       PUSCH PRBs Stride: 273
16:00:06.396672 WRN phy_init 0 [CTL.YAML]       PRACH PRBs Stride: 12
16:00:06.396672 WRN phy_init 0 [CTL.YAML]       SRS PRBs Stride: 273
16:00:06.396672 WRN phy_init 0 [CTL.YAML]       PUSCH nMaxPrb: 273
16:00:06.396672 WRN phy_init 0 [CTL.YAML]       PUSCH nMaxRx: 0
16:00:06.396672 WRN phy_init 0 [CTL.YAML]       UL Gain Calibration: 48.68
16:00:06.396672 WRN phy_init 0 [CTL.YAML]       Lower guard bw: 845
16:00:06.396672 WRN phy_init 0 [CTL.YAML]       Mplane ID: 2
16:00:06.396672 WRN phy_init 0 [CTL.YAML]       VLAN ID: 2
16:00:06.396672 WRN phy_init 0 [CTL.YAML]       Source Eth Address: 00:00:00:00:00:00
16:00:06.396672 WRN phy_init 0 [CTL.YAML]       Destination Eth Address: 6c:ad:ad:00:04:68
16:00:06.396672 WRN phy_init 0 [CTL.YAML]       NIC port: 0000:cc:00.1
16:00:06.396672 WRN phy_init 0 [CTL.YAML]       RU Type: 1
16:00:06.396672 WRN phy_init 0 [CTL.YAML]       U-plane TXQs: 1
16:00:06.396672 WRN phy_init 0 [CTL.YAML]       DL compression method: 1
16:00:06.396672 WRN phy_init 0 [CTL.YAML]       DL iq bit width: 9
16:00:06.396672 WRN phy_init 0 [CTL.YAML]       UL compression method: 1
16:00:06.396672 WRN phy_init 0 [CTL.YAML]       UL iq bit width: 9
16:00:06.396672 WRN phy_init 0 [CTL.YAML]
16:00:06.396672 WRN phy_init 0 [CTL.YAML]       Flow list SSB/PBCH:
16:00:06.396672 WRN phy_init 0 [CTL.YAML]               0
16:00:06.396672 WRN phy_init 0 [CTL.YAML]               1
16:00:06.396672 WRN phy_init 0 [CTL.YAML]               2
16:00:06.396672 WRN phy_init 0 [CTL.YAML]               3
16:00:06.396672 WRN phy_init 0 [CTL.YAML]       Flow list PDCCH:
16:00:06.396672 WRN phy_init 0 [CTL.YAML]               0
16:00:06.396672 WRN phy_init 0 [CTL.YAML]               1
16:00:06.396672 WRN phy_init 0 [CTL.YAML]               2
16:00:06.396672 WRN phy_init 0 [CTL.YAML]               3
16:00:06.396672 WRN phy_init 0 [CTL.YAML]       Flow list PDSCH:
16:00:06.396672 WRN phy_init 0 [CTL.YAML]               0
16:00:06.396672 WRN phy_init 0 [CTL.YAML]               1
16:00:06.396672 WRN phy_init 0 [CTL.YAML]               2
16:00:06.396672 WRN phy_init 0 [CTL.YAML]               3
16:00:06.396672 WRN phy_init 0 [CTL.YAML]       Flow list CSIRS:
16:00:06.396673 WRN phy_init 0 [CTL.YAML]               0
16:00:06.396673 WRN phy_init 0 [CTL.YAML]               1
16:00:06.396673 WRN phy_init 0 [CTL.YAML]               2
16:00:06.396673 WRN phy_init 0 [CTL.YAML]               3
16:00:06.396673 WRN phy_init 0 [CTL.YAML]       Flow list PUSCH:
16:00:06.396673 WRN phy_init 0 [CTL.YAML]               0
16:00:06.396673 WRN phy_init 0 [CTL.YAML]               1
16:00:06.396673 WRN phy_init 0 [CTL.YAML]               2
16:00:06.396673 WRN phy_init 0 [CTL.YAML]               3
16:00:06.396673 WRN phy_init 0 [CTL.YAML]       Flow list PUCCH:
16:00:06.396673 WRN phy_init 0 [CTL.YAML]               0
16:00:06.396673 WRN phy_init 0 [CTL.YAML]               1
16:00:06.396673 WRN phy_init 0 [CTL.YAML]               2
16:00:06.396673 WRN phy_init 0 [CTL.YAML]               3
16:00:06.396673 WRN phy_init 0 [CTL.YAML]       Flow list SRS:
16:00:06.396673 WRN phy_init 0 [CTL.YAML]               0
16:00:06.396673 WRN phy_init 0 [CTL.YAML]               1
16:00:06.396673 WRN phy_init 0 [CTL.YAML]               2
16:00:06.396673 WRN phy_init 0 [CTL.YAML]               3
16:00:06.396673 WRN phy_init 0 [CTL.YAML]       Flow list PRACH:
16:00:06.396673 WRN phy_init 0 [CTL.YAML]               4
16:00:06.396673 WRN phy_init 0 [CTL.YAML]               5
16:00:06.396673 WRN phy_init 0 [CTL.YAML]               6
16:00:06.396673 WRN phy_init 0 [CTL.YAML]               7
16:00:06.396673 WRN phy_init 0 [CTL.YAML]       PUSCH TV: /opt/nvidia/cuBB/testVectors/cuPhyChEstCoeffs.h5
16:00:06.396673 WRN phy_init 0 [CTL.YAML]       SRS TV: /opt/nvidia/cuBB/testVectors/cuPhyChEstCoeffs.h5
16:00:06.396673 WRN phy_init 0 [CTL.YAML]       Section_3 time offset: 58369
16:00:06.396673 WRN phy_init 0 [CTL.YAML]       nMaxRxAnt: 4
16:00:06.396673 WRN phy_init 0 [CTL.YAML]       PUSCH PRBs Stride: 273
16:00:06.396673 WRN phy_init 0 [CTL.YAML]       PRACH PRBs Stride: 12
16:00:06.396673 WRN phy_init 0 [CTL.YAML]       SRS PRBs Stride: 273
16:00:06.396673 WRN phy_init 0 [CTL.YAML]       PUSCH nMaxPrb: 273
16:00:06.396673 WRN phy_init 0 [CTL.YAML]       PUSCH nMaxRx: 0
16:00:06.396673 WRN phy_init 0 [CTL.YAML]       UL Gain Calibration: 48.68
16:00:06.396673 WRN phy_init 0 [CTL.YAML]       Lower guard bw: 845
16:00:06.396673 WRN phy_init 0 [CTL.YAML]       Mplane ID: 3
16:00:06.396673 WRN phy_init 0 [CTL.YAML]       VLAN ID: 2
16:00:06.396673 WRN phy_init 0 [CTL.YAML]       Source Eth Address: 00:00:00:00:00:00
16:00:06.396673 WRN phy_init 0 [CTL.YAML]       Destination Eth Address: 6c:ad:ad:00:02:00
16:00:06.396673 WRN phy_init 0 [CTL.YAML]       NIC port: 0000:cc:00.1
16:00:06.396673 WRN phy_init 0 [CTL.YAML]       RU Type: 1
16:00:06.396673 WRN phy_init 0 [CTL.YAML]       U-plane TXQs: 1
16:00:06.396673 WRN phy_init 0 [CTL.YAML]       DL compression method: 1
16:00:06.396673 WRN phy_init 0 [CTL.YAML]       DL iq bit width: 9
16:00:06.396673 WRN phy_init 0 [CTL.YAML]       UL compression method: 1
16:00:06.396673 WRN phy_init 0 [CTL.YAML]       UL iq bit width: 9
16:00:06.396673 WRN phy_init 0 [CTL.YAML]
16:00:06.396673 WRN phy_init 0 [CTL.YAML]       Flow list SSB/PBCH:
16:00:06.396673 WRN phy_init 0 [CTL.YAML]               0
16:00:06.396673 WRN phy_init 0 [CTL.YAML]               1
16:00:06.396673 WRN phy_init 0 [CTL.YAML]               2
16:00:06.396673 WRN phy_init 0 [CTL.YAML]               3
16:00:06.396673 WRN phy_init 0 [CTL.YAML]       Flow list PDCCH:
16:00:06.396673 WRN phy_init 0 [CTL.YAML]               0
16:00:06.396673 WRN phy_init 0 [CTL.YAML]               1
16:00:06.396673 WRN phy_init 0 [CTL.YAML]               2
16:00:06.396673 WRN phy_init 0 [CTL.YAML]               3
16:00:06.396673 WRN phy_init 0 [CTL.YAML]       Flow list PDSCH:
16:00:06.396673 WRN phy_init 0 [CTL.YAML]               0
16:00:06.396673 WRN phy_init 0 [CTL.YAML]               1
16:00:06.396673 WRN phy_init 0 [CTL.YAML]               2
16:00:06.396673 WRN phy_init 0 [CTL.YAML]               3
16:00:06.396673 WRN phy_init 0 [CTL.YAML]       Flow list CSIRS:
16:00:06.396674 WRN phy_init 0 [CTL.YAML]               0
16:00:06.396674 WRN phy_init 0 [CTL.YAML]               1
16:00:06.396674 WRN phy_init 0 [CTL.YAML]               2
16:00:06.396674 WRN phy_init 0 [CTL.YAML]               3
16:00:06.396674 WRN phy_init 0 [CTL.YAML]       Flow list PUSCH:
16:00:06.396674 WRN phy_init 0 [CTL.YAML]               0
16:00:06.396674 WRN phy_init 0 [CTL.YAML]               1
16:00:06.396674 WRN phy_init 0 [CTL.YAML]               2
16:00:06.396674 WRN phy_init 0 [CTL.YAML]               3
16:00:06.396674 WRN phy_init 0 [CTL.YAML]       Flow list PUCCH:
16:00:06.396674 WRN phy_init 0 [CTL.YAML]               0
16:00:06.396674 WRN phy_init 0 [CTL.YAML]               1
16:00:06.396674 WRN phy_init 0 [CTL.YAML]               2
16:00:06.396674 WRN phy_init 0 [CTL.YAML]               3
16:00:06.396674 WRN phy_init 0 [CTL.YAML]       Flow list SRS:
16:00:06.396674 WRN phy_init 0 [CTL.YAML]               0
16:00:06.396674 WRN phy_init 0 [CTL.YAML]               1
16:00:06.396674 WRN phy_init 0 [CTL.YAML]               2
16:00:06.396674 WRN phy_init 0 [CTL.YAML]               3
16:00:06.396674 WRN phy_init 0 [CTL.YAML]       Flow list PRACH:
16:00:06.396674 WRN phy_init 0 [CTL.YAML]               4
16:00:06.396674 WRN phy_init 0 [CTL.YAML]               5
16:00:06.396674 WRN phy_init 0 [CTL.YAML]               6
16:00:06.396674 WRN phy_init 0 [CTL.YAML]               7
16:00:06.396674 WRN phy_init 0 [CTL.YAML]       PUSCH TV: /opt/nvidia/cuBB/testVectors/cuPhyChEstCoeffs.h5
16:00:06.396674 WRN phy_init 0 [CTL.YAML]       SRS TV: /opt/nvidia/cuBB/testVectors/cuPhyChEstCoeffs.h5
16:00:06.396674 WRN phy_init 0 [CTL.YAML]       Section_3 time offset: 58369
16:00:06.396674 WRN phy_init 0 [CTL.YAML]       nMaxRxAnt: 4
16:00:06.396674 WRN phy_init 0 [CTL.YAML]       PUSCH PRBs Stride: 273
16:00:06.396674 WRN phy_init 0 [CTL.YAML]       PRACH PRBs Stride: 12
16:00:06.396674 WRN phy_init 0 [CTL.YAML]       SRS PRBs Stride: 273
16:00:06.396674 WRN phy_init 0 [CTL.YAML]       PUSCH nMaxPrb: 273
16:00:06.396674 WRN phy_init 0 [CTL.YAML]       PUSCH nMaxRx: 0
16:00:06.396674 WRN phy_init 0 [CTL.YAML]       UL Gain Calibration: 48.68
16:00:06.396674 WRN phy_init 0 [CTL.YAML]       Lower guard bw: 845
16:00:06.396674 WRN phy_init 0 [CTL.YAML]       Mplane ID: 4
16:00:06.396674 WRN phy_init 0 [CTL.YAML]       VLAN ID: 2
16:00:06.396674 WRN phy_init 0 [CTL.YAML]       Source Eth Address: 00:00:00:00:00:00
16:00:06.396674 WRN phy_init 0 [CTL.YAML]       Destination Eth Address: 6c:ad:ad:00:04:70
16:00:06.396674 WRN phy_init 0 [CTL.YAML]       NIC port: 0000:cc:00.1
16:00:06.396674 WRN phy_init 0 [CTL.YAML]       RU Type: 1
16:00:06.396674 WRN phy_init 0 [CTL.YAML]       U-plane TXQs: 1
16:00:06.396674 WRN phy_init 0 [CTL.YAML]       DL compression method: 1
16:00:06.396674 WRN phy_init 0 [CTL.YAML]       DL iq bit width: 9
16:00:06.396674 WRN phy_init 0 [CTL.YAML]       UL compression method: 1
16:00:06.396674 WRN phy_init 0 [CTL.YAML]       UL iq bit width: 9
16:00:06.396674 WRN phy_init 0 [CTL.YAML]
16:00:06.396674 WRN phy_init 0 [CTL.YAML]       Flow list SSB/PBCH:
16:00:06.396674 WRN phy_init 0 [CTL.YAML]               0
16:00:06.396674 WRN phy_init 0 [CTL.YAML]               1
16:00:06.396674 WRN phy_init 0 [CTL.YAML]               2
16:00:06.396674 WRN phy_init 0 [CTL.YAML]               3
16:00:06.396674 WRN phy_init 0 [CTL.YAML]       Flow list PDCCH:
16:00:06.396674 WRN phy_init 0 [CTL.YAML]               0
16:00:06.396675 WRN phy_init 0 [CTL.YAML]               1
16:00:06.396675 WRN phy_init 0 [CTL.YAML]               2
16:00:06.396675 WRN phy_init 0 [CTL.YAML]               3
16:00:06.396675 WRN phy_init 0 [CTL.YAML]       Flow list PDSCH:
16:00:06.396675 WRN phy_init 0 [CTL.YAML]               0
16:00:06.396675 WRN phy_init 0 [CTL.YAML]               1
16:00:06.396675 WRN phy_init 0 [CTL.YAML]               2
16:00:06.396675 WRN phy_init 0 [CTL.YAML]               3
16:00:06.396675 WRN phy_init 0 [CTL.YAML]       Flow list CSIRS:
16:00:06.396675 WRN phy_init 0 [CTL.YAML]               0
16:00:06.396675 WRN phy_init 0 [CTL.YAML]               1
16:00:06.396675 WRN phy_init 0 [CTL.YAML]               2
16:00:06.396675 WRN phy_init 0 [CTL.YAML]               3
16:00:06.396675 WRN phy_init 0 [CTL.YAML]       Flow list PUSCH:
16:00:06.396675 WRN phy_init 0 [CTL.YAML]               0
16:00:06.396675 WRN phy_init 0 [CTL.YAML]               1
16:00:06.396675 WRN phy_init 0 [CTL.YAML]               2
16:00:06.396675 WRN phy_init 0 [CTL.YAML]               3
16:00:06.396675 WRN phy_init 0 [CTL.YAML]       Flow list PUCCH:
16:00:06.396675 WRN phy_init 0 [CTL.YAML]               0
16:00:06.396675 WRN phy_init 0 [CTL.YAML]               1
16:00:06.396675 WRN phy_init 0 [CTL.YAML]               2
16:00:06.396675 WRN phy_init 0 [CTL.YAML]               3
16:00:06.396675 WRN phy_init 0 [CTL.YAML]       Flow list SRS:
16:00:06.396675 WRN phy_init 0 [CTL.YAML]               0
16:00:06.396675 WRN phy_init 0 [CTL.YAML]               1
16:00:06.396675 WRN phy_init 0 [CTL.YAML]               2
16:00:06.396675 WRN phy_init 0 [CTL.YAML]               3
16:00:06.396675 WRN phy_init 0 [CTL.YAML]       Flow list PRACH:
16:00:06.396675 WRN phy_init 0 [CTL.YAML]               4
16:00:06.396675 WRN phy_init 0 [CTL.YAML]               5
16:00:06.396675 WRN phy_init 0 [CTL.YAML]               6
16:00:06.396675 WRN phy_init 0 [CTL.YAML]               7
16:00:06.396675 WRN phy_init 0 [CTL.YAML]       PUSCH TV: /opt/nvidia/cuBB/testVectors/cuPhyChEstCoeffs.h5
16:00:06.396675 WRN phy_init 0 [CTL.YAML]       SRS TV: /opt/nvidia/cuBB/testVectors/cuPhyChEstCoeffs.h5
16:00:06.396675 WRN phy_init 0 [CTL.YAML]       Section_3 time offset: 58369
16:00:06.396675 WRN phy_init 0 [CTL.YAML]       nMaxRxAnt: 4
16:00:06.396675 WRN phy_init 0 [CTL.YAML]       PUSCH PRBs Stride: 273
16:00:06.396675 WRN phy_init 0 [CTL.YAML]       PRACH PRBs Stride: 12
16:00:06.396675 WRN phy_init 0 [CTL.YAML]       SRS PRBs Stride: 273
16:00:06.396675 WRN phy_init 0 [CTL.YAML]       PUSCH nMaxPrb: 273
16:00:06.396675 WRN phy_init 0 [CTL.YAML]       PUSCH nMaxRx: 0
16:00:06.396675 WRN phy_init 0 [CTL.YAML]       UL Gain Calibration: 48.68
16:00:06.396675 WRN phy_init 0 [CTL.YAML]       Lower guard bw: 845
EAL: Detected CPU lcores: 64
EAL: Detected NUMA nodes: 2
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/cuphycontroller/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: VFIO support initialized
EAL: Probe PCI driver: gpu_cuda (10de:20b5) device: 0000:01:00.0 (socket 0)
[16:00:08:582961][358][DOCA][ERR][doca_eth_txq.c:2274][doca_eth_txq_get_wait_on_time_offload_supported] Failed to get wait_on_time_offload_supported: parameter dev=NULL
[16:00:08:582988][358][DOCA][ERR][doca_dpdk.cpp:465][doca_dpdk_port_probe] Invalid input: dev=(nil), devargs=0x41be888cd20
Invalid port_id=255
16:00:08.582933 ERR phy_init 0 [AERIAL_DPDK_API_EVENT] [FH.NIC] Matching device not found.
16:00:08.582954 ERR phy_init 0 [AERIAL_DPDK_API_EVENT] [FH.NIC] open_doca_device_with_pci returned Requested Resource Not Found
16:00:08.582976 ERR phy_init 0 [AERIAL_DPDK_API_EVENT] [FH.NIC] doca_eth_txq_get_wait_on_time_offload_supported returned Invalid input
16:00:08.582997 ERR phy_init 0 [AERIAL_DPDK_API_EVENT] [FH.NIC] doca_dpdk_port_probe returned Invalid input
16:00:08.582997 ERR phy_init 0 [AERIAL_DPDK_API_EVENT] [FH.NIC] get_dpdk_port_id_doca_dev returned Invalid input
16:00:08.583027 ERR phy_init 0 [AERIAL_DPDK_API_EVENT] [FH.NIC] Failed to get device info for NIC 0000:c1:00.0: No such device
16:00:08.583097 ERR phy_init 0 [AERIAL_ORAN_FH_EVENT] [FH.LIB] Exception! basic_string::_M_construct null not valid
16:00:08.583101 ERR phy_init 0 [AERIAL_ORAN_FH_EVENT] [DRV.FH] Failed to add NIC 0000:c1:00.0
terminate called after throwing an instance of 'pd_exc_h'
  what():  Invalid pointer: StaticConversion can't return nullptr
Aborted

Thanks

shivank.chaudhary · August 23, 2024, 4:04pm

ubuntu@gpu-node:~$ ./dpdk-devbind.py -s

Network devices using kernel driver
===================================
0000:21:00.0 'I350 Gigabit Network Connection 1521' if=enp33s0f0 drv=igb unused=vfio-pci *Active*
0000:21:00.1 'I350 Gigabit Network Connection 1521' if=enp33s0f1 drv=igb unused=vfio-pci
0000:21:00.2 'I350 Gigabit Network Connection 1521' if=enp33s0f2 drv=igb unused=vfio-pci
0000:21:00.3 'I350 Gigabit Network Connection 1521' if=enp33s0f3 drv=igb unused=vfio-pci
0000:c1:00.0 'MT28908 Family [ConnectX-6] 101b' if= drv=mlx5_core unused=vfio-pci

No 'Baseband' devices detected
==============================

Crypto devices using kernel driver
==================================
0000:03:00.5 'Device 14ca' drv=ccp unused=vfio-pci
0000:89:00.5 'Device 14ca' drv=ccp unused=vfio-pci

nhashimoto · August 23, 2024, 4:12pm

Hi @shivank.chaudhary ,

Please check if the issue is the same as [DOCA] Failed to register user memory. Got errno: Bad address - Accelerated Computing / Aerial Forum (private) - NVIDIA Developer Forums.

Thank you.

bkecicioglu · August 23, 2024, 5:26pm

Hi @shivank.chaudhary ,

We have verified CX6-DX nic with Aerial but not CX6. The error you are seeing originates from the following:

[16:00:08:582961][358][DOCA][ERR][doca_eth_txq.c:2274][doca_eth_txq_get_wait_on_time_offload_supported

The cuphycontroller checks for doca capabilities and it sees a failure (not supported) for on time scheduling feature support.

Thank you.

shivank.chaudhary · August 24, 2024, 11:55am

Hello @nhashimoto @bkecicioglu
This is the driver my system is using:

ubuntu@gpu-node:~$ cat /proc/driver/nvidia/version
NVRM version: NVIDIA UNIX Open Kernel Module for x86_64  555.42.02  Release Build  (dvs-builder@U16-I3-A03-02-4)  Mon May 13 17:10:33 UTC 2024
GCC version:  gcc version 11.4.0 (Ubuntu 11.4.0-1ubuntu1~22.04)

Also, please share the datasheet link for this Connect x6 Dx nic.
I can’t find the same model mentioned in the aerial documents.

Thank you very much for the help.

Topic		Replies	Views
Aerial L1/ Openair DU L2 FAPI Compatibiity SCF_FAPI_10_04 Aerial Forum aerial-cuda-accelerated-ran , aerial-ran-colab	17	175	August 29, 2024
Prerequisites for Aerial CUDA-Accelerated RAN Aerial Forum aerial-cuda-accelerated-ran	17	409	November 23, 2024
ERR UlPhyDriver03 0 [AERIAL_CUPHY_API_EVENT] [DRV.FUNC_UL] Aerial Forum aerial-cuda-accelerated-ran	8	44	May 22, 2025
Introducing NVIDIA Aerial Research Cloud for Innovations in 5G and 6G Technical Blog	23	1853	February 24, 2025
Aerila cuBB end-to-end test cuphycontroller init fail Aerial Forum aerial-cuda-accelerated-ran	5	58	August 27, 2025
AERIAL ARC With Foxconn RPQN-7800E for N78 Aerial Forum aerial-ran-colab	44	330	November 4, 2024
Problem with memory allocation inside NVidia Aerial Aerial Forum aerial-cuda-accelerated-ran	7	129	May 14, 2024
[DRV.FUNC_UL] Slot Map 103, SFN 104 Slot 19 Order kernel timeout error or Exit error for cell index 0 Dyn index -1! Aerial Forum aerial-cuda-accelerated-ran	7	176	July 24, 2024
Failed to find DPDK GPU device ID for CUDA device Aerial Forum aerial-cuda-accelerated-ran	6	42	October 13, 2025
Question about running cuPHYController Aerial Forum aerial-cuda-accelerated-ran	6	146	October 30, 2024

Segmentation fault at aerial l1 while running with oai l2-l3

Related topics