Error Running cuphycontroller

Accelerated Computing Aerial Forum
1

When I run the cuphycontroller using the /opt/nvidia/cuBB/cuPHY-CP/cuphycontroller/config/cuphycontroller_F08_CG1.yaml config I get the following error:

$ ./runl1.sh 1
Number of cells: 1
Change directory to /opt/nvidia/cuBB/build.aarch64…
Update cuphycontroller_F08_CG1.yaml with cell count…
Set MPS environment log and pipe directories…
Stop any existing running MPS servers…
Cannot find MPS control daemon process
Start new MPS server…
Change directory to /opt/nvidia/cuBB/build.aarch64/cuPHY-CP/cuphycontroller/examples…
Start cuphycontroller with F08_CG1…
Started cuphycontroller on CPU core 10
AERIAL_LOG_PATH unset
Using default log path
Log file set to /tmp/phy.log
Aerial metrics backend address: 127.0.0.1:8082
15:16:23.675977 WRN phy_init 0 [CTL.SCF] Config file: /opt/nvidia/cuBB/cuPHY-CP/cuphycontroller/config/cuphycontroller_F08_CG1.yaml
15:16:23.676639 WRN phy_init 0 [CTL.SCF] low_priority_core=20
15:16:23.676900 WRN phy_init 0 [NVLOG.CPP] Using /opt/nvidia/cuBB/cuPHY/nvlog/config/nvlog_config.yaml for nvlog configuration
YAML invalid key: enable_l1_param_sanity_check Using default value of 0 to YAML_PARAM_ENABLE_L1_PARAM_SANITY_CHECK
YAML invalid key: ul_order_max_rx_pkts Using default value of 0 to UL_ORDER_MAX_RX_PKTS
YAML invalid key: pusch_nMaxLdpcHetConfigs Using default value of 32 to PUSCH-N-MAX-LDPC-HET-CONFIGS
15:16:23.688559 ERR phy_init 0 [AERIAL_CONFIG_EVENT] [CTL.YAML] cuphycontroller config. yaml does not have cpu_init_comms key; defaulting to 0.
15:16:23.688713 WRN phy_init 0 [CTL.YAML] cell_id 1 nic_index :0
15:16:23.688825 WRN phy_init 0 [CTL.YAML] Num Slots: 8
15:16:23.688826 WRN phy_init 0 [CTL.YAML] Enable UL cuPHY Graphs: 1
15:16:23.688826 WRN phy_init 0 [CTL.YAML] Enable DL cuPHY Graphs: 1
15:16:23.688826 WRN phy_init 0 [CTL.YAML] Accurate TX scheduling clock resolution (ns): 500
15:16:23.688826 WRN phy_init 0 [CTL.YAML] DPDK core: 21
15:16:23.688827 WRN phy_init 0 [CTL.YAML] Prometheus core: -1
15:16:23.688827 WRN phy_init 0 [CTL.YAML] UL cores:
15:16:23.688827 WRN phy_init 0 [CTL.YAML] - 7
15:16:23.688827 WRN phy_init 0 [CTL.YAML] - 8
15:16:23.688827 WRN phy_init 0 [CTL.YAML] DL cores:
15:16:23.688828 WRN phy_init 0 [CTL.YAML] - 9
15:16:23.688828 WRN phy_init 0 [CTL.YAML] - 10
15:16:23.688828 WRN phy_init 0 [CTL.YAML] - 11
15:16:23.688828 WRN phy_init 0 [CTL.YAML] Debug worker: -1
15:16:23.688828 WRN phy_init 0 [CTL.YAML] Data Lake core: -1
15:16:23.688829 WRN phy_init 0 [CTL.YAML] SRS starting Section ID: 3072
15:16:23.688829 WRN phy_init 0 [CTL.YAML] PRACH starting Section ID: 2048
15:16:23.688829 WRN phy_init 0 [CTL.YAML] USE GREEN CONTEXTS: 0
15:16:23.688829 WRN phy_init 0 [CTL.YAML] MPS SM PUSCH: 82
15:16:23.688829 WRN phy_init 0 [CTL.YAML] MPS SM PUCCH: 20
15:16:23.688829 WRN phy_init 0 [CTL.YAML] MPS SM PRACH: 2
15:16:23.688829 WRN phy_init 0 [CTL.YAML] MPS SM UL ORDER: 20
15:16:23.688829 WRN phy_init 0 [CTL.YAML] MPS SM PDSCH: 102
15:16:23.688830 WRN phy_init 0 [CTL.YAML] MPS SM PDCCH: 10
15:16:23.688830 WRN phy_init 0 [CTL.YAML] MPS SM PBCH: 2
15:16:23.688830 WRN phy_init 0 [CTL.YAML] MPS SM GPU_COMMS: 16
15:16:23.688830 WRN phy_init 0 [CTL.YAML] PDSCH fallback: 0
15:16:23.688830 WRN phy_init 0 [CTL.YAML] Massive MIMO enable: 0
15:16:23.688830 WRN phy_init 0 [CTL.YAML] Enable SRS : 0
15:16:23.688831 WRN phy_init 0 [CTL.YAML] ul_order_timeout_gpu_log_enable: 0
15:16:23.688831 WRN phy_init 0 [CTL.YAML] ue_mode: 0
15:16:23.688831 WRN phy_init 0 [CTL.YAML] Aggr Obj Non-availability threshold: 5
15:16:23.688831 WRN phy_init 0 [CTL.YAML] sendCPlane_timing_error_th_ns: 0
15:16:23.688831 WRN phy_init 0 [CTL.YAML] pusch_aggr_per_ctx: 3
15:16:23.688832 WRN phy_init 0 [CTL.YAML] prach_aggr_per_ctx: 2
15:16:23.688833 WRN phy_init 0 [CTL.YAML] pucch_aggr_per_ctx: 4
15:16:23.688833 WRN phy_init 0 [CTL.YAML] srs_aggr_per_ctx: 2
15:16:23.688833 WRN phy_init 0 [CTL.YAML] max_harq_pools: 384
15:16:23.688833 WRN phy_init 0 [CTL.YAML] ul_input_buffer_per_cell: 10
15:16:23.688833 WRN phy_init 0 [CTL.YAML] ul_input_buffer_per_cell_srs: 4
15:16:23.688833 WRN phy_init 0 [CTL.YAML] ul_order_timeout_gpu_log_enable: 0
15:16:23.688833 WRN phy_init 0 [CTL.YAML] GPU-initiated comms DL: 1
15:16:23.688833 WRN phy_init 0 [CTL.YAML] CPU-initiated comms : 0
15:16:23.688833 WRN phy_init 0 [CTL.YAML] Cell group: 1
15:16:23.688833 WRN phy_init 0 [CTL.YAML] Cell group num: 1
15:16:23.688834 WRN phy_init 0 [CTL.YAML] puxchPolarDcdrListSz: 8
15:16:23.688834 WRN phy_init 0 [CTL.YAML] split_ul_cuda_streams: 0
15:16:23.688834 WRN phy_init 0 [CTL.YAML] serialize_pucch_pusch: 0
15:16:23.688834 WRN phy_init 0 [CTL.YAML] Number of Cell Configs: 1
15:16:23.688834 WRN phy_init 0 [CTL.YAML] L2Adapter config file: /opt/nvidia/cuBB/cuPHY-CP/cuphycontroller/config/l2_adapter_config_F08_CG1.yaml
15:16:23.688835 WRN phy_init 0 [CTL.YAML] Cell name: O-RU 0
15:16:23.688835 WRN phy_init 0 [CTL.YAML] MU: 1
15:16:23.688835 WRN phy_init 0 [CTL.YAML] ID: 1
15:16:23.688835 WRN phy_init 0 [CTL.YAML] Number of MPlane Configs: 1
15:16:23.688835 WRN phy_init 0 [CTL.YAML] Mplane ID: 1
15:16:23.688835 WRN phy_init 0 [CTL.YAML] VLAN ID: 2
15:16:23.688836 WRN phy_init 0 [CTL.YAML] Source Eth Address: 00:00:00:00:00:00
15:16:23.688836 WRN phy_init 0 [CTL.YAML] Destination Eth Address: 20:04:9b:9e:27:a3
15:16:23.688836 WRN phy_init 0 [CTL.YAML] NIC port: 0000:01:00.0
15:16:23.688836 WRN phy_init 0 [CTL.YAML] RU Type: 3
15:16:23.688836 WRN phy_init 0 [CTL.YAML] U-plane TXQs: 1
15:16:23.688836 WRN phy_init 0 [CTL.YAML] DL compression method: 1
15:16:23.688836 WRN phy_init 0 [CTL.YAML] DL iq bit width: 9
15:16:23.688836 WRN phy_init 0 [CTL.YAML] UL compression method: 1
15:16:23.688836 WRN phy_init 0 [CTL.YAML] UL iq bit width: 9
15:16:23.688836 WRN phy_init 0 [CTL.YAML]
15:16:23.688837 WRN phy_init 0 [CTL.YAML] Flow list SSB/PBCH:
15:16:23.688837 WRN phy_init 0 [CTL.YAML] 8
15:16:23.688837 WRN phy_init 0 [CTL.YAML] 0
15:16:23.688837 WRN phy_init 0 [CTL.YAML] 1
15:16:23.688837 WRN phy_init 0 [CTL.YAML] 2
15:16:23.688838 WRN phy_init 0 [CTL.YAML] Flow list PDCCH:
15:16:23.688838 WRN phy_init 0 [CTL.YAML] 8
15:16:23.688838 WRN phy_init 0 [CTL.YAML] 0
15:16:23.688838 WRN phy_init 0 [CTL.YAML] 1
15:16:23.688838 WRN phy_init 0 [CTL.YAML] 2
15:16:23.688838 WRN phy_init 0 [CTL.YAML] Flow list PDSCH:
15:16:23.688838 WRN phy_init 0 [CTL.YAML] 8
15:16:23.688838 WRN phy_init 0 [CTL.YAML] 0
15:16:23.688838 WRN phy_init 0 [CTL.YAML] 1
15:16:23.688838 WRN phy_init 0 [CTL.YAML] 2
15:16:23.688839 WRN phy_init 0 [CTL.YAML] Flow list CSIRS:
15:16:23.688839 WRN phy_init 0 [CTL.YAML] 8
15:16:23.688839 WRN phy_init 0 [CTL.YAML] 0
15:16:23.688839 WRN phy_init 0 [CTL.YAML] 1
15:16:23.688839 WRN phy_init 0 [CTL.YAML] 2
15:16:23.688839 WRN phy_init 0 [CTL.YAML] Flow list PUSCH:
15:16:23.688839 WRN phy_init 0 [CTL.YAML] 8
15:16:23.688839 WRN phy_init 0 [CTL.YAML] 0
15:16:23.688839 WRN phy_init 0 [CTL.YAML] 1
15:16:23.688839 WRN phy_init 0 [CTL.YAML] 2
15:16:23.688839 WRN phy_init 0 [CTL.YAML] Flow list PUCCH:
15:16:23.688840 WRN phy_init 0 [CTL.YAML] 8
15:16:23.688840 WRN phy_init 0 [CTL.YAML] 0
15:16:23.688840 WRN phy_init 0 [CTL.YAML] 1
15:16:23.688840 WRN phy_init 0 [CTL.YAML] 2
15:16:23.688840 WRN phy_init 0 [CTL.YAML] Flow list SRS:
15:16:23.688840 WRN phy_init 0 [CTL.YAML] 8
15:16:23.688840 WRN phy_init 0 [CTL.YAML] 0
15:16:23.688840 WRN phy_init 0 [CTL.YAML] 1
15:16:23.688840 WRN phy_init 0 [CTL.YAML] 2
15:16:23.688840 WRN phy_init 0 [CTL.YAML] Flow list PRACH:
15:16:23.688840 WRN phy_init 0 [CTL.YAML] 15
15:16:23.688840 WRN phy_init 0 [CTL.YAML] 7
15:16:23.688841 WRN phy_init 0 [CTL.YAML] 0
15:16:23.688841 WRN phy_init 0 [CTL.YAML] 1
15:16:23.688841 WRN phy_init 0 [CTL.YAML] PUSCH TV: /opt/nvidia/cuBB/testVectors/cuPhyChEstCoeffs.h5
15:16:23.688841 WRN phy_init 0 [CTL.YAML] SRS TV: /opt/nvidia/cuBB/testVectors/cuPhyChEstCoeffs.h5
15:16:23.688841 WRN phy_init 0 [CTL.YAML] Section_3 time offset: 58369
15:16:23.688841 WRN phy_init 0 [CTL.YAML] nMaxRxAnt: 4
15:16:23.688841 WRN phy_init 0 [CTL.YAML] PUSCH PRBs Stride: 273
15:16:23.688841 WRN phy_init 0 [CTL.YAML] PRACH PRBs Stride: 12
15:16:23.688842 WRN phy_init 0 [CTL.YAML] SRS PRBs Stride: 273
15:16:23.688842 WRN phy_init 0 [CTL.YAML] PUSCH nMaxPrb: 136
15:16:23.688842 WRN phy_init 0 [CTL.YAML] PUSCH nMaxRx: 4
15:16:23.688842 WRN phy_init 0 [CTL.YAML] UL Gain Calibration: 48.68
15:16:23.688842 WRN phy_init 0 [CTL.YAML] Lower guard bw: 845
15:16:24.092133 ERR phy_init 0 [AERIAL_CUPHYDRV_API_EVENT] [DRV.EXCP] /opt/nvidia/cuBB/cuPHY-CP/cuphydriver/src/common/cuphydriver_api.cpp l1_init line 127 exception: GDRcopy open failed
15:16:24.092150 ERR phy_init 0 [AERIAL_CUPHYDRV_API_EVENT] [CTL.DRV] Error l1_init
15:16:24.092152 ERR phy_init 0 [AERIAL_CUPHYDRV_API_EVENT] [CTL.SCF] pc_init_phydriver error -1
15:16:24.092156 ERR phy_init 0 [AERIAL_SYSTEM_API_EVENT] [NVLOG.EXIT_HANDLER] Triggering L1 application exit from exit handler
15:16:24.092157 WRN phy_init 0 [DRV.API] Trigging L1 exit handler

If I run the system checks

$ pip3 install psutil
$ cd $cuBB_SDK/cuPHY/util/cuBB_system_checks
$ sudo -E python3 ./cuBB_system_checks.py

It does report that the gdrcopy module is loaded. What else can I check or try to determine the root cause of my issue?

2

This is resolved as well. When using a pod Kind in Kubernetes it works - if using a deployment kind seem to run into issues. I have not sorted out the issues but its not NVIDIA code related.