Question about running cuPHYController

Accelerated Computing Aerial Forum
1

Hi! I have problem with runnig cuPHYController on out GH200 server. The text below is the result log of “sudo -E ./cuphycontroller_scf F08_CG1”. In summary, I have repeated kernel timeout error on slots. Can you find some problem on the result log?

Started cuphycontroller on CPU core 70
AERIAL_LOG_PATH unset
Using default log path
Log file set to /tmp/phy.log
Aerial metrics backend address: 127.0.0.1:8081
05:41:45.003763 WRN phy_init 0 [CTL.SCF] Config file: /opt/nvidia/cuBB/cuPHY-CP/cuphycontroller/config/cuphycontroller_F08_CG1.yaml
05:41:45.004388 WRN phy_init 0 [CTL.SCF] low_priority_core=23
05:41:45.005824 WRN phy_init 0 [NVLOG.CPP] Using /opt/nvidia/cuBB/cuPHY/nvlog/config/nvlog_config.yaml for nvlog configuration
YAML invalid key: enable_l1_param_sanity_check Using default value of 0 to YAML_PARAM_ENABLE_L1_PARAM_SANITY_CHECK
YAML invalid key: ul_order_max_rx_pkts Using default value of 0 to UL_ORDER_MAX_RX_PKTS
YAML invalid key: pusch_nMaxLdpcHetConfigs Using default value of 32 to PUSCH-N-MAX-LDPC-HET-CONFIGS
05:41:45.017391 ERR phy_init 0 [AERIAL_CONFIG_EVENT] [CTL.YAML] cuphycontroller config. yaml does not have cpu_init_comms key; defaulting to 0.
05:41:45.017543 WRN phy_init 0 [CTL.YAML] cell_id 1 nic_index :0
05:41:45.017648 WRN phy_init 0 [CTL.YAML] Num Slots: 8
05:41:45.017648 WRN phy_init 0 [CTL.YAML] Enable UL cuPHY Graphs: 1
05:41:45.017648 WRN phy_init 0 [CTL.YAML] Enable DL cuPHY Graphs: 1
05:41:45.017648 WRN phy_init 0 [CTL.YAML] Accurate TX scheduling clock resolution (ns): 500
05:41:45.017648 WRN phy_init 0 [CTL.YAML] DPDK core: 23
05:41:45.017649 WRN phy_init 0 [CTL.YAML] Prometheus core: -1
05:41:45.017649 WRN phy_init 0 [CTL.YAML] UL cores: 
05:41:45.017649 WRN phy_init 0 [CTL.YAML] 	- 5
05:41:45.017649 WRN phy_init 0 [CTL.YAML] 	- 6
05:41:45.017650 WRN phy_init 0 [CTL.YAML] DL cores: 
05:41:45.017650 WRN phy_init 0 [CTL.YAML] 	- 11
05:41:45.017650 WRN phy_init 0 [CTL.YAML] 	- 12
05:41:45.017650 WRN phy_init 0 [CTL.YAML] 	- 13
05:41:45.017650 WRN phy_init 0 [CTL.YAML] Debug worker: -1
05:41:45.017651 WRN phy_init 0 [CTL.YAML] Data Lake core: -1
05:41:45.017651 WRN phy_init 0 [CTL.YAML] SRS starting Section ID: 3072
05:41:45.017651 WRN phy_init 0 [CTL.YAML] PRACH starting Section ID: 2048
05:41:45.017651 WRN phy_init 0 [CTL.YAML] USE GREEN CONTEXTS: 0
05:41:45.017651 WRN phy_init 0 [CTL.YAML] MPS SM PUSCH: 82
05:41:45.017651 WRN phy_init 0 [CTL.YAML] MPS SM PUCCH: 20
05:41:45.017651 WRN phy_init 0 [CTL.YAML] MPS SM PRACH: 2
05:41:45.017651 WRN phy_init 0 [CTL.YAML] MPS SM UL ORDER: 20
05:41:45.017651 WRN phy_init 0 [CTL.YAML] MPS SM PDSCH: 102
05:41:45.017651 WRN phy_init 0 [CTL.YAML] MPS SM PDCCH: 10
05:41:45.017652 WRN phy_init 0 [CTL.YAML] MPS SM PBCH: 2
05:41:45.017652 WRN phy_init 0 [CTL.YAML] MPS SM GPU_COMMS: 16
05:41:45.017652 WRN phy_init 0 [CTL.YAML] PDSCH fallback: 0
05:41:45.017652 WRN phy_init 0 [CTL.YAML] Massive MIMO enable: 0
05:41:45.017652 WRN phy_init 0 [CTL.YAML] Enable SRS : 0
05:41:45.017652 WRN phy_init 0 [CTL.YAML] ul_order_timeout_gpu_log_enable: 0
05:41:45.017652 WRN phy_init 0 [CTL.YAML] ue_mode: 0
05:41:45.017652 WRN phy_init 0 [CTL.YAML] Aggr Obj Non-availability threshold: 5
05:41:45.017653 WRN phy_init 0 [CTL.YAML] sendCPlane_timing_error_th_ns: 0
05:41:45.017653 WRN phy_init 0 [CTL.YAML] pusch_aggr_per_ctx: 3
05:41:45.017654 WRN phy_init 0 [CTL.YAML] prach_aggr_per_ctx: 2
05:41:45.017654 WRN phy_init 0 [CTL.YAML] pucch_aggr_per_ctx: 4
05:41:45.017654 WRN phy_init 0 [CTL.YAML] srs_aggr_per_ctx: 2
05:41:45.017654 WRN phy_init 0 [CTL.YAML] max_harq_pools: 384
05:41:45.017654 WRN phy_init 0 [CTL.YAML] ul_input_buffer_per_cell: 10
05:41:45.017655 WRN phy_init 0 [CTL.YAML] ul_input_buffer_per_cell_srs: 4
05:41:45.017655 WRN phy_init 0 [CTL.YAML] ul_order_timeout_gpu_log_enable: 0
05:41:45.017655 WRN phy_init 0 [CTL.YAML] GPU-initiated comms DL: 1
05:41:45.017655 WRN phy_init 0 [CTL.YAML] CPU-initiated comms : 0
05:41:45.017655 WRN phy_init 0 [CTL.YAML] Cell group: 1
05:41:45.017655 WRN phy_init 0 [CTL.YAML] Cell group num: 1
05:41:45.017655 WRN phy_init 0 [CTL.YAML] puxchPolarDcdrListSz: 8
05:41:45.017655 WRN phy_init 0 [CTL.YAML] split_ul_cuda_streams: 0
05:41:45.017655 WRN phy_init 0 [CTL.YAML] serialize_pucch_pusch: 0
05:41:45.017655 WRN phy_init 0 [CTL.YAML] Number of Cell Configs: 1
05:41:45.017655 WRN phy_init 0 [CTL.YAML] L2Adapter config file: /opt/nvidia/cuBB/cuPHY-CP/cuphycontroller/config/l2_adapter_config_F08_CG1.yaml
05:41:45.017656 WRN phy_init 0 [CTL.YAML] Cell name: O-RU 0
05:41:45.017656 WRN phy_init 0 [CTL.YAML] 	MU: 1
05:41:45.017656 WRN phy_init 0 [CTL.YAML] 	ID: 1
05:41:45.017656 WRN phy_init 0 [CTL.YAML] Number of MPlane Configs: 1
05:41:45.017656 WRN phy_init 0 [CTL.YAML] 	Mplane ID: 1
05:41:45.017656 WRN phy_init 0 [CTL.YAML] 	VLAN ID: 2
05:41:45.017656 WRN phy_init 0 [CTL.YAML] 	Source Eth Address: 9c:63:c0:d8:ea:b6
05:41:45.017657 WRN phy_init 0 [CTL.YAML] 	Destination Eth Address: 9c:63:c0:d8:eb:9a
05:41:45.017657 WRN phy_init 0 [CTL.YAML] 	NIC port: 0000:01:00.0
05:41:45.017657 WRN phy_init 0 [CTL.YAML] 	RU Type: 3
05:41:45.017657 WRN phy_init 0 [CTL.YAML] 	U-plane TXQs: 1
05:41:45.017657 WRN phy_init 0 [CTL.YAML] 	DL compression method: 1
05:41:45.017657 WRN phy_init 0 [CTL.YAML] 	DL iq bit width: 9
05:41:45.017657 WRN phy_init 0 [CTL.YAML] 	UL compression method: 1
05:41:45.017657 WRN phy_init 0 [CTL.YAML] 	UL iq bit width: 9
05:41:45.017658 WRN phy_init 0 [CTL.YAML] 
05:41:45.017658 WRN phy_init 0 [CTL.YAML] 	Flow list SSB/PBCH: 
05:41:45.017658 WRN phy_init 0 [CTL.YAML] 		8
05:41:45.017658 WRN phy_init 0 [CTL.YAML] 		0
05:41:45.017658 WRN phy_init 0 [CTL.YAML] 		1
05:41:45.017658 WRN phy_init 0 [CTL.YAML] 		2
05:41:45.017658 WRN phy_init 0 [CTL.YAML] 	Flow list PDCCH: 
05:41:45.017659 WRN phy_init 0 [CTL.YAML] 		8
05:41:45.017659 WRN phy_init 0 [CTL.YAML] 		0
05:41:45.017659 WRN phy_init 0 [CTL.YAML] 		1
05:41:45.017659 WRN phy_init 0 [CTL.YAML] 		2
05:41:45.017659 WRN phy_init 0 [CTL.YAML] 	Flow list PDSCH: 
05:41:45.017659 WRN phy_init 0 [CTL.YAML] 		8
05:41:45.017659 WRN phy_init 0 [CTL.YAML] 		0
05:41:45.017659 WRN phy_init 0 [CTL.YAML] 		1
05:41:45.017659 WRN phy_init 0 [CTL.YAML] 		2
05:41:45.017659 WRN phy_init 0 [CTL.YAML] 	Flow list CSIRS: 
05:41:45.017659 WRN phy_init 0 [CTL.YAML] 		8
05:41:45.017659 WRN phy_init 0 [CTL.YAML] 		0
05:41:45.017659 WRN phy_init 0 [CTL.YAML] 		1
05:41:45.017660 WRN phy_init 0 [CTL.YAML] 		2
05:41:45.017660 WRN phy_init 0 [CTL.YAML] 	Flow list PUSCH: 
05:41:45.017660 WRN phy_init 0 [CTL.YAML] 		8
05:41:45.017660 WRN phy_init 0 [CTL.YAML] 		0
05:41:45.017660 WRN phy_init 0 [CTL.YAML] 		1
05:41:45.017660 WRN phy_init 0 [CTL.YAML] 		2
05:41:45.017660 WRN phy_init 0 [CTL.YAML] 	Flow list PUCCH: 
05:41:45.017660 WRN phy_init 0 [CTL.YAML] 		8
05:41:45.017660 WRN phy_init 0 [CTL.YAML] 		0
05:41:45.017660 WRN phy_init 0 [CTL.YAML] 		1
05:41:45.017660 WRN phy_init 0 [CTL.YAML] 		2
05:41:45.017660 WRN phy_init 0 [CTL.YAML] 	Flow list SRS: 
05:41:45.017660 WRN phy_init 0 [CTL.YAML] 		8
05:41:45.017660 WRN phy_init 0 [CTL.YAML] 		0
05:41:45.017660 WRN phy_init 0 [CTL.YAML] 		1
05:41:45.017661 WRN phy_init 0 [CTL.YAML] 		2
05:41:45.017661 WRN phy_init 0 [CTL.YAML] 	Flow list PRACH: 
05:41:45.017661 WRN phy_init 0 [CTL.YAML] 		15
05:41:45.017661 WRN phy_init 0 [CTL.YAML] 		7
05:41:45.017661 WRN phy_init 0 [CTL.YAML] 		0
05:41:45.017661 WRN phy_init 0 [CTL.YAML] 		1
05:41:45.017661 WRN phy_init 0 [CTL.YAML] 	PUSCH TV: /opt/nvidia/cuBB/testVectors/cuPhyChEstCoeffs.h5
05:41:45.017661 WRN phy_init 0 [CTL.YAML] 	SRS TV: /opt/nvidia/cuBB/testVectors/cuPhyChEstCoeffs.h5
05:41:45.017661 WRN phy_init 0 [CTL.YAML] 	Section_3 time offset: 58369
05:41:45.017661 WRN phy_init 0 [CTL.YAML] 	nMaxRxAnt: 4
05:41:45.017661 WRN phy_init 0 [CTL.YAML] 	PUSCH PRBs Stride: 273
05:41:45.017661 WRN phy_init 0 [CTL.YAML] 	PRACH PRBs Stride: 12
05:41:45.017662 WRN phy_init 0 [CTL.YAML] 	SRS PRBs Stride: 273
05:41:45.017662 WRN phy_init 0 [CTL.YAML] 	PUSCH nMaxPrb: 273
05:41:45.017662 WRN phy_init 0 [CTL.YAML] 	PUSCH nMaxRx: 4
05:41:45.017662 WRN phy_init 0 [CTL.YAML] 	UL Gain Calibration: 48.68
05:41:45.017662 WRN phy_init 0 [CTL.YAML] 	Lower guard bw: 845
05:41:45.209720 WRN phy_init 0 [DRV.CTX] CUDA_DEVICE_MAX_CONNECTIONS 8
05:41:45.209726 WRN phy_init 0 [DRV.CTX] use_green_contexts 0
EAL: Detected CPU lcores: 72
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/cuphycontroller/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: VFIO support initialized
EAL: Probe PCI driver: gpu_cuda (10de:2342) device: 0009:01:00.0 (socket 0)
EAL: Probe PCI driver: mlx5_pci (15b3:a2dc) device: 0000:01:00.0 (socket 0)
gRPC Server listening on 0.0.0.0:50051
05:41:48.392492 WRN phy_init 0 [NVIPC.RING] nv_ipc_ring_open: name=R_cell_cfg ring_len=16 buf_size=8 OK
05:41:48.392498 WRN phy_init 0 [NVIPC.RING] nv_ipc_ring_open: name=R_generic_async_test ring_len=16 buf_size=8 OK
05:41:48.392499 WRN phy_init 0 [NVIPC.RING] nv_ipc_ring_open: name=R_cpu_stall ring_len=16 buf_size=8 OK
05:41:48.392500 WRN phy_init 0 [NVIPC.RING] nv_ipc_ring_open: name=R_fapi_delay_test ring_len=16 buf_size=8 OK
05:41:48.392502 WRN phy_init 0 [NVIPC.RING] nv_ipc_ring_open: name=R_ul_u_plane_drop_requests ring_len=32 buf_size=8 OK
05:41:48.392503 WRN phy_init 0 [NVIPC.RING] nv_ipc_ring_open: name=R_sfn_slot_sync_requests ring_len=16 buf_size=8 OK
05:41:48.403782 WRN phy_init 0 [CTL.SCF] 
====> PhyDriver initialized!
05:41:48.410197 WRN phy_init 0 [NVIPC.SHM] nvipc server initialized
05:41:48.410278 WRN phy_init 0 [NVIPC.SHM] shm_ipc_open: forward_enable=0 fw_max_msg_buf_count=0 fw_max_data_buf_count=0
05:41:48.410279 WRN phy_init 0 [NVIPC.SHM] create_shm_nv_ipc_interface: OK
05:41:48.410280 WRN phy_init 0 [L2A.TRANSPORT] phy_mac_transport[0] created. phy_cell_map=0 mac_cell_map=0 mapped=false
05:41:48.410281 WRN phy_init 0 [L2A.TRANSPORT] init: loaded 1 nvipc instance
05:41:48.425680 WRN 12118 0 [NVIPC.EFD] [nvipc][core 23 ] share event_fd thread info: sched_policy=0 sched_priority=0
05:41:48.425743 WRN 12118 0 [NVIPC.EFD] [nvipc] Listening for nvipc client to connect ...
05:41:48.428427 WRN phy_main 0 [CTL.SCF] cuPHYController configured for 1 cells
05:41:48.428428 WRN phy_main 0 [CTL.SCF] 
====> cuPHYController initialized, L1 is ready!
05:41:48.428441 WRN phy_main 0 [CTL.STARTUP_TIMES] {TI PERCENTAGE} <cuphycontroller main> Start Main:0.0,Parse Cuphycontroller YAML:0.0,Init nvlog:0.4,Cuda Set Device:4.6,Cuphy PTI Init:0.8,Init PHYDriver:93.5,Init cuphydriver:0.0,Make PHYDriverProxy:0.0,Init SCF FAPI:0.0,Create PHY_group:0.7,Start PHY_group:0.0, (total: 3425436.8us),

05:41:48.428444 WRN phy_main 0 [CTL.STARTUP_TIMES] {TI DURATION} <cuphycontroller main> Start Main:730.584,Parse Cuphycontroller YAML:680.218,Init nvlog:13261.276,Cuda Set Device:158573.177,Cuphy PTI Init:26307.483,Init PHYDriver:3201241.625,Init cuphydriver:0.576,Make PHYDriverProxy:9.376,Init SCF FAPI:1.888,Create PHY_group:24453.965,Start PHY_group:176.670,

05:41:48.428447 WRN phy_main 0 [CTL.STARTUP_TIMES] {TI TIMESTAMPS} <cuphycontroller main> Start Main:1728970905002990510,Parse Cuphycontroller YAML:1728970905003721094,Init nvlog:1728970905004401312,Cuda Set Device:1728970905017662588,Cuphy PTI Init:1728970905176235765,Init PHYDriver:1728970905202543248,Init cuphydriver:1728970908403784873,Make PHYDriverProxy:1728970908403785449,Init SCF FAPI:1728970908403794825,Create PHY_group:1728970908403796713,Start PHY_group:1728970908428250678,End Main:1728970908428427348,

05:41:48.429136 WRN msg_processing 0 [L2A.MODULE] Thread thread_func on CPU 29 initialized fmtlog
05:41:49.393043 WRN UlPhyDriver06 0 [DRV.WORKER] Thread UlPhyDriver06(worker_default) on CPU 6 initialized fmtlog
05:41:49.393095 WRN UlPhyDriver05 0 [DRV.WORKER] Thread UlPhyDriver05(worker_default) on CPU 5 initialized fmtlog
05:41:49.393184 WRN DlPhyDriver13 0 [DRV.WORKER] Thread DlPhyDriver13(worker_default) on CPU 13 initialized fmtlog
05:41:49.393253 WRN DlPhyDriver12 0 [DRV.WORKER] Thread DlPhyDriver12(worker_default) on CPU 12 initialized fmtlog
05:41:49.393280 WRN DlPhyDriver11 0 [DRV.WORKER] Thread DlPhyDriver11(worker_default) on CPU 11 initialized fmtlog
05:49:44.716998 WRN 12118 0 [NVIPC.EFD] [nvipc] nvipc unix socket server connected
05:49:44.717038 WRN 12118 0 [NVIPC.EFD] [nvipc] Received peer event_fd: 497
05:49:44.917121 WRN 12118 0 [NVIPC.EFD] Share event_fd succeed: efd_tx=497, efd_rx=504
05:49:44.917123 WRN 12118 0 [NVIPC.EFD] [nvipc][core 23 ] nvipc unix socket exit
05:49:45.736070 WRN msg_processing 0 [SCF.PHY] CONFIG_TLV_INDICATION_INSTANCES_PER_SLOT indPerSlotPtr[0] = 0
05:49:45.736071 WRN msg_processing 0 [SCF.PHY] CONFIG_TLV_INDICATION_INSTANCES_PER_SLOT indPerSlotPtr[1] = 0
05:49:45.736071 WRN msg_processing 0 [SCF.PHY] CONFIG_TLV_INDICATION_INSTANCES_PER_SLOT indPerSlotPtr[2] = 0
05:49:45.736071 WRN msg_processing 0 [SCF.PHY] CONFIG_TLV_INDICATION_INSTANCES_PER_SLOT indPerSlotPtr[3] = 0
05:49:45.736071 WRN msg_processing 0 [SCF.PHY] CONFIG_TLV_INDICATION_INSTANCES_PER_SLOT indPerSlotPtr[4] = 0
05:49:45.736071 WRN msg_processing 0 [SCF.PHY] CONFIG_TLV_INDICATION_INSTANCES_PER_SLOT indPerSlotPtr[5] = 0
05:49:45.736071 WRN msg_processing 0 [SCF.PHY] txPortTlvPresent absent. Setting nTxAnt to 4
05:49:45.736071 WRN msg_processing 0 [SCF.PHY] txPortTlvPresent absent. Setting nRxAnt to 4
05:49:45.736081 WRN msg_processing 0 [SCF.PHY] PHY Cell Id = 41, M-Plane Id= 1
05:49:45.742629 WRN msg_processing 0 [DRV.PUSCH] tvStatPrms: PUSCH enableDeviceGraphLaunch=1 enableCsiP2Fapiv3 = 0 nMaxLdpcHetConfigs = 32
05:49:45.742630 WRN msg_processing 0 [DRV.PUSCH] Early-HARQ is enabled. Timeout values for wait kernels are 5000us and 5000us
05:49:45.742633 WRN msg_processing 0 [DRV.PUSCH] static_params.nMaxPrb: 273
05:49:45.742634 WRN msg_processing 0 [DRV.PUSCH] static_params.nMaxRx: 4
05:49:45.750253 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 55.367 MiB for cuPHY PUSCH channel object (0x5d0654c0000).
05:49:45.750253 WRN msg_processing 0 [CUPHY.PUSCH_RX] PuschRx: Running with eqCoeffAlgo 1
05:49:45.753849 WRN msg_processing 0 [DRV.PUSCH] tvStatPrms: PUSCH enableDeviceGraphLaunch=1 enableCsiP2Fapiv3 = 0 nMaxLdpcHetConfigs = 32
05:49:45.753849 WRN msg_processing 0 [DRV.PUSCH] Early-HARQ is enabled. Timeout values for wait kernels are 5000us and 5000us
05:49:45.753850 WRN msg_processing 0 [DRV.PUSCH] static_params.nMaxPrb: 273
05:49:45.753850 WRN msg_processing 0 [DRV.PUSCH] static_params.nMaxRx: 4
05:49:45.760313 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 55.367 MiB for cuPHY PUSCH channel object (0x5d065530000).
05:49:45.760313 WRN msg_processing 0 [CUPHY.PUSCH_RX] PuschRx: Running with eqCoeffAlgo 1
05:49:45.763917 WRN msg_processing 0 [DRV.PUSCH] tvStatPrms: PUSCH enableDeviceGraphLaunch=1 enableCsiP2Fapiv3 = 0 nMaxLdpcHetConfigs = 32
05:49:45.763917 WRN msg_processing 0 [DRV.PUSCH] Early-HARQ is enabled. Timeout values for wait kernels are 5000us and 5000us
05:49:45.763918 WRN msg_processing 0 [DRV.PUSCH] static_params.nMaxPrb: 273
05:49:45.763918 WRN msg_processing 0 [DRV.PUSCH] static_params.nMaxRx: 4
05:49:45.770966 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 55.367 MiB for cuPHY PUSCH channel object (0x5d065f80000).
05:49:45.770966 WRN msg_processing 0 [CUPHY.PUSCH_RX] PuschRx: Running with eqCoeffAlgo 1
05:49:45.774548 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 3.437 MiB for cuPHY PUCCH channel object (0x5d0653ad200).
05:49:45.776870 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 3.437 MiB for cuPHY PUCCH channel object (0x5d0653ae000).
05:49:45.779445 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 3.437 MiB for cuPHY PUCCH channel object (0x5d0653aee00).
05:49:45.781728 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 3.437 MiB for cuPHY PUCCH channel object (0x5d066b60000).
05:49:45.783660 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.182 MiB for cuPHY PRACH channel object (0x5d06667e400).
05:49:45.784446 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.182 MiB for cuPHY PRACH channel object (0x5d06667e780).
05:49:45.789944 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 98.130 MiB for cuPHY PDSCH channel object (0x5d065456c00).
05:49:45.795059 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 98.130 MiB for cuPHY PDSCH channel object (0x5d065458400).
05:49:45.799631 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 98.130 MiB for cuPHY PDSCH channel object (0x5d065459c00).
05:49:45.804394 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 98.130 MiB for cuPHY PDSCH channel object (0x5d06545b400).
05:49:45.809649 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 98.130 MiB for cuPHY PDSCH channel object (0x5d06545cc00).
05:49:45.814338 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 98.130 MiB for cuPHY PDSCH channel object (0x5d06545e400).
05:49:45.819260 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 98.130 MiB for cuPHY PDSCH channel object (0x5d060470000).
05:49:45.824308 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 98.130 MiB for cuPHY PDSCH channel object (0x5d060471800).
05:49:45.829406 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 98.130 MiB for cuPHY PDSCH channel object (0x5d060473000).
05:49:45.834584 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 98.130 MiB for cuPHY PDSCH channel object (0x5d060474800).
05:49:45.835776 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.001 MiB for cuPHY SSB channel object (0x5d061346080).
05:49:45.835962 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.001 MiB for cuPHY SSB channel object (0x5d061346380).
05:49:45.836091 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.001 MiB for cuPHY SSB channel object (0x5d061346680).
05:49:45.836220 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.001 MiB for cuPHY SSB channel object (0x5d061346980).
05:49:45.836352 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.001 MiB for cuPHY SSB channel object (0x5d061346c80).
05:49:45.836469 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.001 MiB for cuPHY SSB channel object (0x5d061346f80).
05:49:45.836605 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.001 MiB for cuPHY SSB channel object (0x5d061347280).
05:49:45.836726 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.001 MiB for cuPHY SSB channel object (0x5d061347580).
05:49:45.836844 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.001 MiB for cuPHY SSB channel object (0x5d061347880).
05:49:45.836958 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.001 MiB for cuPHY SSB channel object (0x5d061347b80).
05:49:45.837241 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.563 MiB for cuPHY PDCCH channel object (0x5d066556e00).
05:49:45.837397 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.563 MiB for cuPHY PDCCH channel object (0x5d066557300).
05:49:45.837510 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.563 MiB for cuPHY PDCCH channel object (0x5d066556900).
05:49:45.837709 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.563 MiB for cuPHY PDCCH channel object (0x5d066557800).
05:49:45.837827 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.563 MiB for cuPHY PDCCH channel object (0x5d066557d00).
05:49:45.837924 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.563 MiB for cuPHY PDCCH channel object (0x5d066558200).
05:49:45.838099 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.563 MiB for cuPHY PDCCH channel object (0x5d066558700).
05:49:45.838307 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.563 MiB for cuPHY PDCCH channel object (0x5d066558c00).
05:49:45.838406 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.563 MiB for cuPHY PDCCH channel object (0x5d066559100).
05:49:45.838522 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.563 MiB for cuPHY PDCCH channel object (0x5d066559600).
05:49:45.838637 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.093 MiB for cuPHY CSIRS channel object (0x5d06134c980).
05:49:45.838749 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.093 MiB for cuPHY CSIRS channel object (0x5d06134cc80).
05:49:45.838857 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.093 MiB for cuPHY CSIRS channel object (0x5d06134cf80).
05:49:45.838953 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.093 MiB for cuPHY CSIRS channel object (0x5d06134d280).
05:49:45.839171 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.093 MiB for cuPHY CSIRS channel object (0x5d06134d580).
05:49:45.839264 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.093 MiB for cuPHY CSIRS channel object (0x5d06134f080).
05:49:45.839363 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.093 MiB for cuPHY CSIRS channel object (0x5d06134f380).
05:49:45.839466 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.093 MiB for cuPHY CSIRS channel object (0x5d06134f680).
05:49:45.839556 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.093 MiB for cuPHY CSIRS channel object (0x5d06134f980).
05:49:45.839822 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.093 MiB for cuPHY CSIRS channel object (0x5d06134fc80).
05:49:45.839834 WRN msg_processing 0 [DRV.API] Update cell: mplane_id=1 dl_grid_sz=273 
05:49:45.839835 WRN msg_processing 0 [DRV.API] Update cell: mplane_id=1 ul_grid_sz=273 
05:49:45.841747 WRN timer_thread 0 [L2A.TICK] Thread slot_indication_thread_sleep_method initialized fmtlog
05:49:45.841749 WRN timer_thread 0 [L2A.TICK] PTP Configs: gps_alpha: 0 gps_beta: 0
05:49:50.520021 WRN timer_thread 0 [SCF.PHY] Cell  0 | DL    0.00 Mbps    0 Slots | UL    0.00 Mbps    0 Slots CRC   0 (     0) | Tick 0
05:49:50.524514 ERR UlPhyDriver05 0 [AERIAL_CUPHY_API_EVENT] [DRV.FUNC_UL] Slot Map 0, SFN 0 Slot 4 Order kernel timeout error or Exit error for cell index 0 Dyn index 0!
05:49:50.526033 ERR msg_processing 0 [AERIAL_CUPHYDRV_API_EVENT] [DRV.API] SFN 0.15 No available Aggr PUSCH objects
05:49:50.526042 WRN msg_processing 0 [SCF.PHY] L1 Enqueue Error on SFN 0.15 for cell-id 0
05:49:50.526908 ERR UlPhyDriver06 0 [AERIAL_CUPHY_API_EVENT] [DRV.FUNC_UL] Slot Map 0, SFN 0 Slot 4 PUSCH Pre Early Harq Wait kernel timeout!
05:49:50.526977 ERR UlPhyDriver05 0 [AERIAL_CUPHY_API_EVENT] [DRV.FUNC_UL] Slot Map 0, SFN 0 Slot 4 PUSCH Post Early Harq Wait kernel timeout!
05:49:50.527570 ERR UlPhyDriver06 0 [AERIAL_CUPHY_API_EVENT] [DRV.FUNC_UL] Slot Map 1, SFN 0 Slot 5 Order kernel timeout error or Exit error for cell index 0 Dyn index 0!
05:49:50.531036 ERR msg_processing 0 [AERIAL_CUPHYDRV_API_EVENT] [DRV.API] SFN 1.5 No available Aggr PUSCH objects
05:49:50.531043 WRN msg_processing 0 [SCF.PHY] L1 Enqueue Error on SFN 1.5 for cell-id 0
05:49:50.532010 ERR UlPhyDriver05 0 [AERIAL_CUPHY_API_EVENT] [DRV.FUNC_UL] Slot Map 1, SFN 0 Slot 5 PUSCH Pre Early Harq Wait kernel timeout!
05:49:50.532063 ERR UlPhyDriver06 0 [AERIAL_CUPHY_API_EVENT] [DRV.FUNC_UL] Slot Map 1, SFN 0 Slot 5 PUSCH Post Early Harq Wait kernel timeout!
05:49:50.532068 WRN UlPhyDriver05 0 [DRV.FH] sendCPlane_timingCheck : sendCPlane Timing error for ULC. Too late to abort UL tasks start_tx_time 1728971390526664000 current_time 1728971390532067951 start_ch_task_time 1728971390526000000
05:49:50.532068 ERR UlPhyDriver05 0 [AERIAL_CUPHYDRV_API_EVENT] [DRV.FUNC_UL] UL C-plane send timing error for cell index 0 error type 2 Map 2,Error detection too late. No UL Task Abort!
05:49:50.532068 ERR UlPhyDriver05 0 [AERIAL_L2ADAPTER_EVENT] [SCF.PHY] Send SLOT error indication from L1 SFN=0 slot=14 for msg_id=0x07
Question about test vector
2

Hi @root1031 ,

Is this a test using testMAC/ru-emulator or with a real O-RU? If it is the former, was the RU-emulator running before the testMAC started?

Thanks.

3

Hi @bkecicioglu !! Thanks for your reply!!

  • This is a test using testMAC/ru-emulator.
  • I ran RU-emulator first, and ran cuphycontroller and testmac. I wrote the cli used below. Also I have testMAC error running while running this test.
# cuphycontroller
cd /opt/nvidia/cuBB/build/cuPHY-CP/cuphycontroller/examples
sudo -E .∕cuphycontroller_scf F08_CG1
# testmac
cd /opt/nvidia/cuBB/build/cuPHY-CP/testMAC/testMAC$ 
sudo ./test_mac F08 1C 59
# RU emulator
cd /opt/nvidia/cuBB/build/cuPHY-CP/ru-emulator/ru_emulator
sudo ./ru_emulator F08 1C 59

Thank you!

4

Hi @root1031,

Order kernel is responsible for re-ordering incoming (UL) packets. Order kernel wait error timeouts can happen when packets are not received on time.

Please check nic_interface and peerethaddr settings in the ru_emulator config.yaml file as described in section titled “Updating configuration files for end-to-end” of cuBB Quickstart Guide.

You can also try to run DL only channel to see if the settings are ok for a simple test. You can do this as follows:

# cuphycontroller
cd /opt/nvidia/cuBB/build/cuPHY-CP/cuphycontroller/examples
sudo -E .∕cuphycontroller_scf F08_CG1
# RU emulator
cd /opt/nvidia/cuBB/build/cuPHY-CP/ru-emulator/ru_emulator
sudo ./ru_emulator F08 1C 59 --channels PBCH
# testmac
cd /opt/nvidia/cuBB/build/cuPHY-CP/testMAC/testMAC$ 
sudo ./test_mac F08 1C 59 --channels PBCH

In this example, only PBCH will be transmitted.

Thanks.

5

Thanks, @bkecicioglu !!

I fixed the config file and Order kernel wait error timeouts does not happen anymore.
But I got some different error after running “sudo -E ./cuphycontroller_scf F08_CG1”. In short, Peer 0 was not registered and UL C-plane send error happened. Can you find the reason why?

Thank you !!

Below is the error log.

Started cuphycontroller on CPU core 70
AERIAL_LOG_PATH unset
Using default log path
Log file set to /tmp/phy.log
Aerial metrics backend address: 127.0.0.1:8081
05:16:40.424322 WRN phy_init 0 [CTL.SCF] Config file: /opt/nvidia/cuBB/cuPHY-CP/cuphycontroller/config/cuphycontroller_F08_CG1.yaml
05:16:40.424968 WRN phy_init 0 [CTL.SCF] low_priority_core=23
05:16:40.425175 WRN phy_init 0 [NVLOG.CPP] Using /opt/nvidia/cuBB/cuPHY/nvlog/config/nvlog_config.yaml for nvlog configuration
YAML invalid key: enable_l1_param_sanity_check Using default value of 0 to YAML_PARAM_ENABLE_L1_PARAM_SANITY_CHECK
YAML invalid key: ul_order_max_rx_pkts Using default value of 0 to UL_ORDER_MAX_RX_PKTS
YAML invalid key: pusch_nMaxLdpcHetConfigs Using default value of 32 to PUSCH-N-MAX-LDPC-HET-CONFIGS
05:16:40.436735 ERR phy_init 0 [AERIAL_CONFIG_EVENT] [CTL.YAML] cuphycontroller config. yaml does not have cpu_init_comms key; defaulting to 0.
05:16:40.436874 WRN phy_init 0 [CTL.YAML] cell_id 1 nic_index :0
05:16:40.436970 WRN phy_init 0 [CTL.YAML] Num Slots: 8
05:16:40.436971 WRN phy_init 0 [CTL.YAML] Enable UL cuPHY Graphs: 1
05:16:40.436971 WRN phy_init 0 [CTL.YAML] Enable DL cuPHY Graphs: 1
05:16:40.436971 WRN phy_init 0 [CTL.YAML] Accurate TX scheduling clock resolution (ns): 500
05:16:40.436971 WRN phy_init 0 [CTL.YAML] DPDK core: 23
05:16:40.436971 WRN phy_init 0 [CTL.YAML] Prometheus core: -1
05:16:40.436972 WRN phy_init 0 [CTL.YAML] UL cores: 
05:16:40.436972 WRN phy_init 0 [CTL.YAML] 	- 5
05:16:40.436972 WRN phy_init 0 [CTL.YAML] 	- 6
05:16:40.436972 WRN phy_init 0 [CTL.YAML] DL cores: 
05:16:40.436972 WRN phy_init 0 [CTL.YAML] 	- 11
05:16:40.436973 WRN phy_init 0 [CTL.YAML] 	- 12
05:16:40.436973 WRN phy_init 0 [CTL.YAML] 	- 13
05:16:40.436973 WRN phy_init 0 [CTL.YAML] Debug worker: -1
05:16:40.436973 WRN phy_init 0 [CTL.YAML] Data Lake core: -1
05:16:40.436973 WRN phy_init 0 [CTL.YAML] SRS starting Section ID: 3072
05:16:40.436973 WRN phy_init 0 [CTL.YAML] PRACH starting Section ID: 2048
05:16:40.436973 WRN phy_init 0 [CTL.YAML] USE GREEN CONTEXTS: 0
05:16:40.436974 WRN phy_init 0 [CTL.YAML] MPS SM PUSCH: 82
05:16:40.436974 WRN phy_init 0 [CTL.YAML] MPS SM PUCCH: 20
05:16:40.436974 WRN phy_init 0 [CTL.YAML] MPS SM PRACH: 2
05:16:40.436974 WRN phy_init 0 [CTL.YAML] MPS SM UL ORDER: 20
05:16:40.436974 WRN phy_init 0 [CTL.YAML] MPS SM PDSCH: 102
05:16:40.436974 WRN phy_init 0 [CTL.YAML] MPS SM PDCCH: 10
05:16:40.436974 WRN phy_init 0 [CTL.YAML] MPS SM PBCH: 2
05:16:40.436975 WRN phy_init 0 [CTL.YAML] MPS SM GPU_COMMS: 16
05:16:40.436975 WRN phy_init 0 [CTL.YAML] PDSCH fallback: 0
05:16:40.436975 WRN phy_init 0 [CTL.YAML] Massive MIMO enable: 0
05:16:40.436975 WRN phy_init 0 [CTL.YAML] Enable SRS : 0
05:16:40.436975 WRN phy_init 0 [CTL.YAML] ul_order_timeout_gpu_log_enable: 0
05:16:40.436975 WRN phy_init 0 [CTL.YAML] ue_mode: 0
05:16:40.436975 WRN phy_init 0 [CTL.YAML] Aggr Obj Non-availability threshold: 5
05:16:40.436976 WRN phy_init 0 [CTL.YAML] sendCPlane_timing_error_th_ns: 0
05:16:40.436976 WRN phy_init 0 [CTL.YAML] pusch_aggr_per_ctx: 3
05:16:40.436977 WRN phy_init 0 [CTL.YAML] prach_aggr_per_ctx: 2
05:16:40.436977 WRN phy_init 0 [CTL.YAML] pucch_aggr_per_ctx: 4
05:16:40.436977 WRN phy_init 0 [CTL.YAML] srs_aggr_per_ctx: 2
05:16:40.436977 WRN phy_init 0 [CTL.YAML] max_harq_pools: 384
05:16:40.436977 WRN phy_init 0 [CTL.YAML] ul_input_buffer_per_cell: 10
05:16:40.436978 WRN phy_init 0 [CTL.YAML] ul_input_buffer_per_cell_srs: 4
05:16:40.436978 WRN phy_init 0 [CTL.YAML] ul_order_timeout_gpu_log_enable: 0
05:16:40.436978 WRN phy_init 0 [CTL.YAML] GPU-initiated comms DL: 1
05:16:40.436978 WRN phy_init 0 [CTL.YAML] CPU-initiated comms : 0
05:16:40.436978 WRN phy_init 0 [CTL.YAML] Cell group: 1
05:16:40.436978 WRN phy_init 0 [CTL.YAML] Cell group num: 1
05:16:40.436978 WRN phy_init 0 [CTL.YAML] puxchPolarDcdrListSz: 8
05:16:40.436978 WRN phy_init 0 [CTL.YAML] split_ul_cuda_streams: 0
05:16:40.436978 WRN phy_init 0 [CTL.YAML] serialize_pucch_pusch: 0
05:16:40.436978 WRN phy_init 0 [CTL.YAML] Number of Cell Configs: 1
05:16:40.436978 WRN phy_init 0 [CTL.YAML] L2Adapter config file: /opt/nvidia/cuBB/cuPHY-CP/cuphycontroller/config/l2_adapter_config_F08_CG1.yaml
05:16:40.436979 WRN phy_init 0 [CTL.YAML] Cell name: O-RU 0
05:16:40.436979 WRN phy_init 0 [CTL.YAML] 	MU: 1
05:16:40.436979 WRN phy_init 0 [CTL.YAML] 	ID: 1
05:16:40.436979 WRN phy_init 0 [CTL.YAML] Number of MPlane Configs: 1
05:16:40.436980 WRN phy_init 0 [CTL.YAML] 	Mplane ID: 1
05:16:40.436980 WRN phy_init 0 [CTL.YAML] 	VLAN ID: 2
05:16:40.436980 WRN phy_init 0 [CTL.YAML] 	Source Eth Address: 9c:63:c0:d8:ea:b6
05:16:40.436980 WRN phy_init 0 [CTL.YAML] 	Destination Eth Address: 9c:63:c0:d8:eb:9a
05:16:40.436980 WRN phy_init 0 [CTL.YAML] 	NIC port: 0000:01:00.1
05:16:40.436980 WRN phy_init 0 [CTL.YAML] 	RU Type: 3
05:16:40.436980 WRN phy_init 0 [CTL.YAML] 	U-plane TXQs: 1
05:16:40.436980 WRN phy_init 0 [CTL.YAML] 	DL compression method: 1
05:16:40.436981 WRN phy_init 0 [CTL.YAML] 	DL iq bit width: 9
05:16:40.436981 WRN phy_init 0 [CTL.YAML] 	UL compression method: 1
05:16:40.436981 WRN phy_init 0 [CTL.YAML] 	UL iq bit width: 9
05:16:40.436981 WRN phy_init 0 [CTL.YAML] 
05:16:40.436981 WRN phy_init 0 [CTL.YAML] 	Flow list SSB/PBCH: 
05:16:40.436982 WRN phy_init 0 [CTL.YAML] 		8
05:16:40.436982 WRN phy_init 0 [CTL.YAML] 		0
05:16:40.436982 WRN phy_init 0 [CTL.YAML] 		1
05:16:40.436982 WRN phy_init 0 [CTL.YAML] 		2
05:16:40.436982 WRN phy_init 0 [CTL.YAML] 	Flow list PDCCH: 
05:16:40.436982 WRN phy_init 0 [CTL.YAML] 		8
05:16:40.436982 WRN phy_init 0 [CTL.YAML] 		0
05:16:40.436982 WRN phy_init 0 [CTL.YAML] 		1
05:16:40.436982 WRN phy_init 0 [CTL.YAML] 		2
05:16:40.436983 WRN phy_init 0 [CTL.YAML] 	Flow list PDSCH: 
05:16:40.436983 WRN phy_init 0 [CTL.YAML] 		8
05:16:40.436983 WRN phy_init 0 [CTL.YAML] 		0
05:16:40.436983 WRN phy_init 0 [CTL.YAML] 		1
05:16:40.436983 WRN phy_init 0 [CTL.YAML] 		2
05:16:40.436983 WRN phy_init 0 [CTL.YAML] 	Flow list CSIRS: 
05:16:40.436983 WRN phy_init 0 [CTL.YAML] 		8
05:16:40.436983 WRN phy_init 0 [CTL.YAML] 		0
05:16:40.436983 WRN phy_init 0 [CTL.YAML] 		1
05:16:40.436983 WRN phy_init 0 [CTL.YAML] 		2
05:16:40.436983 WRN phy_init 0 [CTL.YAML] 	Flow list PUSCH: 
05:16:40.436983 WRN phy_init 0 [CTL.YAML] 		8
05:16:40.436984 WRN phy_init 0 [CTL.YAML] 		0
05:16:40.436984 WRN phy_init 0 [CTL.YAML] 		1
05:16:40.436984 WRN phy_init 0 [CTL.YAML] 		2
05:16:40.436984 WRN phy_init 0 [CTL.YAML] 	Flow list PUCCH: 
05:16:40.436984 WRN phy_init 0 [CTL.YAML] 		8
05:16:40.436984 WRN phy_init 0 [CTL.YAML] 		0
05:16:40.436984 WRN phy_init 0 [CTL.YAML] 		1
05:16:40.436984 WRN phy_init 0 [CTL.YAML] 		2
05:16:40.436984 WRN phy_init 0 [CTL.YAML] 	Flow list SRS: 
05:16:40.436984 WRN phy_init 0 [CTL.YAML] 		8
05:16:40.436984 WRN phy_init 0 [CTL.YAML] 		0
05:16:40.436984 WRN phy_init 0 [CTL.YAML] 		1
05:16:40.436984 WRN phy_init 0 [CTL.YAML] 		2
05:16:40.436984 WRN phy_init 0 [CTL.YAML] 	Flow list PRACH: 
05:16:40.436984 WRN phy_init 0 [CTL.YAML] 		15
05:16:40.436985 WRN phy_init 0 [CTL.YAML] 		7
05:16:40.436985 WRN phy_init 0 [CTL.YAML] 		0
05:16:40.436985 WRN phy_init 0 [CTL.YAML] 		1
05:16:40.436985 WRN phy_init 0 [CTL.YAML] 	PUSCH TV: /opt/nvidia/cuBB/testVectors/cuPhyChEstCoeffs.h5
05:16:40.436985 WRN phy_init 0 [CTL.YAML] 	SRS TV: /opt/nvidia/cuBB/testVectors/cuPhyChEstCoeffs.h5
05:16:40.436985 WRN phy_init 0 [CTL.YAML] 	Section_3 time offset: 58369
05:16:40.436985 WRN phy_init 0 [CTL.YAML] 	nMaxRxAnt: 4
05:16:40.436985 WRN phy_init 0 [CTL.YAML] 	PUSCH PRBs Stride: 273
05:16:40.436985 WRN phy_init 0 [CTL.YAML] 	PRACH PRBs Stride: 12
05:16:40.436985 WRN phy_init 0 [CTL.YAML] 	SRS PRBs Stride: 273
05:16:40.436985 WRN phy_init 0 [CTL.YAML] 	PUSCH nMaxPrb: 273
05:16:40.436986 WRN phy_init 0 [CTL.YAML] 	PUSCH nMaxRx: 4
05:16:40.436986 WRN phy_init 0 [CTL.YAML] 	UL Gain Calibration: 48.68
05:16:40.436986 WRN phy_init 0 [CTL.YAML] 	Lower guard bw: 845
05:16:40.648976 WRN phy_init 0 [DRV.CTX] CUDA_DEVICE_MAX_CONNECTIONS 8
05:16:40.648982 WRN phy_init 0 [DRV.CTX] use_green_contexts 0
EAL: Detected CPU lcores: 72
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/cuphycontroller/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: VFIO support initialized
EAL: Probe PCI driver: gpu_cuda (10de:2342) device: 0009:01:00.0 (socket 0)
EAL: Probe PCI driver: mlx5_pci (15b3:a2dc) device: 0000:01:00.0 (socket 0)
gRPC Server listening on 0.0.0.0:50051
05:16:43.771412 WRN phy_main 0 [NVIPC.RING] nv_ipc_ring_open: name=R_cell_cfg ring_len=16 buf_size=8 OK
05:16:43.771418 WRN phy_main 0 [NVIPC.RING] nv_ipc_ring_open: name=R_generic_async_test ring_len=16 buf_size=8 OK
05:16:43.771419 WRN phy_main 0 [NVIPC.RING] nv_ipc_ring_open: name=R_cpu_stall ring_len=16 buf_size=8 OK
05:16:43.771420 WRN phy_main 0 [NVIPC.RING] nv_ipc_ring_open: name=R_fapi_delay_test ring_len=16 buf_size=8 OK
05:16:43.771422 WRN phy_main 0 [NVIPC.RING] nv_ipc_ring_open: name=R_ul_u_plane_drop_requests ring_len=32 buf_size=8 OK
05:16:43.771423 WRN phy_main 0 [NVIPC.RING] nv_ipc_ring_open: name=R_sfn_slot_sync_requests ring_len=16 buf_size=8 OK
05:16:43.782398 WRN phy_main 0 [CTL.SCF] 
====> PhyDriver initialized!
05:16:43.788007 WRN phy_main 0 [NVIPC.SHM] nvipc server initialized
05:16:43.788082 WRN phy_main 0 [NVIPC.SHM] shm_ipc_open: forward_enable=0 fw_max_msg_buf_count=0 fw_max_data_buf_count=0
05:16:43.788082 WRN phy_main 0 [NVIPC.SHM] create_shm_nv_ipc_interface: OK
05:16:43.788084 WRN phy_main 0 [L2A.TRANSPORT] phy_mac_transport[0] created. phy_cell_map=0 mac_cell_map=0 mapped=false
05:16:43.788085 WRN phy_main 0 [L2A.TRANSPORT] init: loaded 1 nvipc instance
05:16:43.789992 WRN 1476 0 [NVIPC.EFD] [nvipc][core 23 ] share event_fd thread info: sched_policy=0 sched_priority=0
05:16:43.790043 WRN 1476 0 [NVIPC.EFD] [nvipc] Listening for nvipc client to connect ...
05:16:43.806461 WRN phy_main 0 [CTL.SCF] cuPHYController configured for 1 cells
05:16:43.806463 WRN phy_main 0 [CTL.SCF] 
====> cuPHYController initialized, L1 is ready!
05:16:43.806477 WRN phy_main 0 [CTL.STARTUP_TIMES] {TI PERCENTAGE} <cuphycontroller main> Start Main:0.0,Parse Cuphycontroller YAML:0.0,Init nvlog:0.4,Cuda Set Device:5.3,Cuphy PTI Init:0.8,Init PHYDriver:92.8,Init cuphydriver:0.0,Make PHYDriverProxy:0.0,Init SCF FAPI:0.0,Create PHY_group:0.7,Start PHY_group:0.0, (total: 3382952.5us),

05:16:43.806481 WRN phy_main 0 [CTL.STARTUP_TIMES] {TI DURATION} <cuphycontroller main> Start Main:767.608,Parse Cuphycontroller YAML:704.345,Init nvlog:12004.775,Cuda Set Device:178797.000,Cuphy PTI Init:26147.798,Init PHYDriver:3140469.116,Init cuphydriver:0.608,Make PHYDriverProxy:8.608,Init SCF FAPI:1.728,Create PHY_group:23844.047,Start PHY_group:206.846,

05:16:43.806483 WRN phy_main 0 [CTL.STARTUP_TIMES] {TI TIMESTAMPS} <cuphycontroller main> Start Main:1729055800423509503,Parse Cuphycontroller YAML:1729055800424277111,Init nvlog:1729055800424981456,Cuda Set Device:1729055800436986231,Cuphy PTI Init:1729055800615783231,Init PHYDriver:1729055800641931029,Init cuphydriver:1729055803782400145,Make PHYDriverProxy:1729055803782400753,Init SCF FAPI:1729055803782409361,Create PHY_group:1729055803782411089,Start PHY_group:1729055803806255136,End Main:1729055803806461982,

05:16:43.807119 WRN msg_processing 0 [L2A.MODULE] Thread thread_func on CPU 29 initialized fmtlog
05:16:44.771939 WRN UlPhyDriver06 0 [DRV.WORKER] Thread UlPhyDriver06(worker_default) on CPU 6 initialized fmtlog
05:16:44.772078 WRN UlPhyDriver05 0 [DRV.WORKER] Thread UlPhyDriver05(worker_default) on CPU 5 initialized fmtlog
05:16:44.772106 WRN DlPhyDriver13 0 [DRV.WORKER] Thread DlPhyDriver13(worker_default) on CPU 13 initialized fmtlog
05:16:44.772179 WRN DlPhyDriver12 0 [DRV.WORKER] Thread DlPhyDriver12(worker_default) on CPU 12 initialized fmtlog
05:16:44.772197 WRN DlPhyDriver11 0 [DRV.WORKER] Thread DlPhyDriver11(worker_default) on CPU 11 initialized fmtlog
05:16:47.182483 WRN 1476 0 [NVIPC.EFD] [nvipc] nvipc unix socket server connected
05:16:47.182519 WRN 1476 0 [NVIPC.EFD] [nvipc] Received peer event_fd: 497
05:16:47.382602 WRN 1476 0 [NVIPC.EFD] Share event_fd succeed: efd_tx=497, efd_rx=504
05:16:47.382604 WRN 1476 0 [NVIPC.EFD] [nvipc][core 23 ] nvipc unix socket exit
05:16:48.201140 WRN msg_processing 0 [SCF.PHY] CONFIG_TLV_INDICATION_INSTANCES_PER_SLOT indPerSlotPtr[0] = 0
05:16:48.201140 WRN msg_processing 0 [SCF.PHY] CONFIG_TLV_INDICATION_INSTANCES_PER_SLOT indPerSlotPtr[1] = 0
05:16:48.201141 WRN msg_processing 0 [SCF.PHY] CONFIG_TLV_INDICATION_INSTANCES_PER_SLOT indPerSlotPtr[2] = 2
05:16:48.201141 WRN msg_processing 0 [SCF.PHY] CONFIG_TLV_INDICATION_INSTANCES_PER_SLOT indPerSlotPtr[3] = 0
05:16:48.201141 WRN msg_processing 0 [SCF.PHY] CONFIG_TLV_INDICATION_INSTANCES_PER_SLOT indPerSlotPtr[4] = 0
05:16:48.201141 WRN msg_processing 0 [SCF.PHY] CONFIG_TLV_INDICATION_INSTANCES_PER_SLOT indPerSlotPtr[5] = 0
05:16:48.201141 WRN msg_processing 0 [SCF.PHY] txPortTlvPresent absent. Setting nTxAnt to 4
05:16:48.201141 WRN msg_processing 0 [SCF.PHY] txPortTlvPresent absent. Setting nRxAnt to 4
05:16:48.201151 WRN msg_processing 0 [SCF.PHY] PHY Cell Id = 41, M-Plane Id= 1
05:16:48.201171 ERR msg_processing 0 [AERIAL_CUPHYDRV_API_EVENT] [DRV.FH] Peer 0 not registered

05:16:48.207610 WRN msg_processing 0 [DRV.PUSCH] tvStatPrms: PUSCH enableDeviceGraphLaunch=1 enableCsiP2Fapiv3 = 0 nMaxLdpcHetConfigs = 32
05:16:48.207611 WRN msg_processing 0 [DRV.PUSCH] Early-HARQ is enabled. Timeout values for wait kernels are 5000us and 5000us
05:16:48.207614 WRN msg_processing 0 [DRV.PUSCH] static_params.nMaxPrb: 273
05:16:48.207614 WRN msg_processing 0 [DRV.PUSCH] static_params.nMaxRx: 4
05:16:48.215164 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 55.367 MiB for cuPHY PUSCH channel object (0x497a54c0000).
05:16:48.215165 WRN msg_processing 0 [CUPHY.PUSCH_RX] PuschRx: Running with eqCoeffAlgo 1
05:16:48.218641 WRN msg_processing 0 [DRV.PUSCH] tvStatPrms: PUSCH enableDeviceGraphLaunch=1 enableCsiP2Fapiv3 = 0 nMaxLdpcHetConfigs = 32
05:16:48.218642 WRN msg_processing 0 [DRV.PUSCH] Early-HARQ is enabled. Timeout values for wait kernels are 5000us and 5000us
05:16:48.218642 WRN msg_processing 0 [DRV.PUSCH] static_params.nMaxPrb: 273
05:16:48.218642 WRN msg_processing 0 [DRV.PUSCH] static_params.nMaxRx: 4
05:16:48.225052 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 55.367 MiB for cuPHY PUSCH channel object (0x497a5530000).
05:16:48.225053 WRN msg_processing 0 [CUPHY.PUSCH_RX] PuschRx: Running with eqCoeffAlgo 1
05:16:48.228624 WRN msg_processing 0 [DRV.PUSCH] tvStatPrms: PUSCH enableDeviceGraphLaunch=1 enableCsiP2Fapiv3 = 0 nMaxLdpcHetConfigs = 32
05:16:48.228624 WRN msg_processing 0 [DRV.PUSCH] Early-HARQ is enabled. Timeout values for wait kernels are 5000us and 5000us
05:16:48.228625 WRN msg_processing 0 [DRV.PUSCH] static_params.nMaxPrb: 273
05:16:48.228625 WRN msg_processing 0 [DRV.PUSCH] static_params.nMaxRx: 4
05:16:48.235501 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 55.367 MiB for cuPHY PUSCH channel object (0x497a5f80000).
05:16:48.235501 WRN msg_processing 0 [CUPHY.PUSCH_RX] PuschRx: Running with eqCoeffAlgo 1
05:16:48.239043 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 3.437 MiB for cuPHY PUCCH channel object (0x497a53ad200).
05:16:48.241328 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 3.437 MiB for cuPHY PUCCH channel object (0x497a53ae000).
05:16:48.243829 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 3.437 MiB for cuPHY PUCCH channel object (0x497a53aee00).
05:16:48.246148 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 3.437 MiB for cuPHY PUCCH channel object (0x497a6b60000).
05:16:48.248047 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.182 MiB for cuPHY PRACH channel object (0x497a667e400).
05:16:48.248815 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.182 MiB for cuPHY PRACH channel object (0x497a667e780).
05:16:48.254311 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 98.130 MiB for cuPHY PDSCH channel object (0x497a55c6c00).
05:16:48.259275 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 98.130 MiB for cuPHY PDSCH channel object (0x497a55c8400).
05:16:48.263848 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 98.130 MiB for cuPHY PDSCH channel object (0x497a55c9c00).
05:16:48.268542 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 98.130 MiB for cuPHY PDSCH channel object (0x497a55cb400).
05:16:48.273718 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 98.130 MiB for cuPHY PDSCH channel object (0x497a55ccc00).
05:16:48.278302 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 98.130 MiB for cuPHY PDSCH channel object (0x497a55ce400).
05:16:48.283170 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 98.130 MiB for cuPHY PDSCH channel object (0x4978c470000).
05:16:48.288106 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 98.130 MiB for cuPHY PDSCH channel object (0x4978c471800).
05:16:48.293151 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 98.130 MiB for cuPHY PDSCH channel object (0x4978c473000).
05:16:48.298230 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 98.130 MiB for cuPHY PDSCH channel object (0x4978c474800).
05:16:48.299399 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.001 MiB for cuPHY SSB channel object (0x4978d346080).
05:16:48.299584 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.001 MiB for cuPHY SSB channel object (0x4978d346380).
05:16:48.299713 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.001 MiB for cuPHY SSB channel object (0x4978d346680).
05:16:48.299837 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.001 MiB for cuPHY SSB channel object (0x4978d346980).
05:16:48.299959 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.001 MiB for cuPHY SSB channel object (0x4978d346c80).
05:16:48.300069 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.001 MiB for cuPHY SSB channel object (0x4978d346f80).
05:16:48.300184 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.001 MiB for cuPHY SSB channel object (0x4978d347280).
05:16:48.300299 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.001 MiB for cuPHY SSB channel object (0x4978d347580).
05:16:48.300413 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.001 MiB for cuPHY SSB channel object (0x4978d347880).
05:16:48.300522 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.001 MiB for cuPHY SSB channel object (0x4978d347b80).
05:16:48.300812 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.563 MiB for cuPHY PDCCH channel object (0x497a6556900).
05:16:48.300962 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.563 MiB for cuPHY PDCCH channel object (0x497a6556e00).
05:16:48.301068 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.563 MiB for cuPHY PDCCH channel object (0x497a6557300).
05:16:48.301268 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.563 MiB for cuPHY PDCCH channel object (0x497a6557800).
05:16:48.301389 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.563 MiB for cuPHY PDCCH channel object (0x497a6557d00).
05:16:48.301490 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.563 MiB for cuPHY PDCCH channel object (0x497a6558200).
05:16:48.301679 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.563 MiB for cuPHY PDCCH channel object (0x497a6558700).
05:16:48.301889 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.563 MiB for cuPHY PDCCH channel object (0x497a6558c00).
05:16:48.301985 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.563 MiB for cuPHY PDCCH channel object (0x497a6559100).
05:16:48.302101 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.563 MiB for cuPHY PDCCH channel object (0x497a6559600).
05:16:48.302215 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.093 MiB for cuPHY CSIRS channel object (0x4978d34c980).
05:16:48.302325 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.093 MiB for cuPHY CSIRS channel object (0x4978d34cc80).
05:16:48.302429 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.093 MiB for cuPHY CSIRS channel object (0x4978d34cf80).
05:16:48.302523 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.093 MiB for cuPHY CSIRS channel object (0x4978d34d280).
05:16:48.302730 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.093 MiB for cuPHY CSIRS channel object (0x4978d34d580).
05:16:48.302822 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.093 MiB for cuPHY CSIRS channel object (0x4978d34f080).
05:16:48.302917 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.093 MiB for cuPHY CSIRS channel object (0x4978d34f380).
05:16:48.303017 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.093 MiB for cuPHY CSIRS channel object (0x4978d34f680).
05:16:48.303102 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.093 MiB for cuPHY CSIRS channel object (0x4978d34f980).
05:16:48.303365 WRN msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.093 MiB for cuPHY CSIRS channel object (0x4978d34fc80).
05:16:48.303376 WRN msg_processing 0 [DRV.API] Update cell: mplane_id=1 dl_grid_sz=273 
05:16:48.303377 WRN msg_processing 0 [DRV.API] Update cell: mplane_id=1 ul_grid_sz=273 
05:16:48.305295 WRN timer_thread 0 [L2A.TICK] Thread slot_indication_thread_sleep_method initialized fmtlog
05:16:48.305298 WRN timer_thread 0 [L2A.TICK] PTP Configs: gps_alpha: 0 gps_beta: 0
05:16:49.080020 WRN timer_thread 0 [SCF.PHY] Cell  0 | DL    0.00 Mbps    0 Slots | UL    0.00 Mbps    0 Slots CRC   0 (     0) | Tick 0
05:16:49.081089 ERR UlPhyDriver05 0 [AERIAL_CUPHYDRV_API_EVENT] [DRV.FH] Peer 0 not registered

05:16:49.081090 ERR UlPhyDriver05 0 [AERIAL_CUPHYDRV_API_EVENT] [DRV.FUNC_UL] UL C-plane send error for cell index 0,error type 3 Map 0 Abort UL Tasks!

05:16:49.081093 ERR UlPhyDriver05 0 [AERIAL_L2ADAPTER_EVENT] [SCF.PHY] Send SLOT error indication from L1 SFN=0 slot=4 for msg_id=0x07
05:16:49.081102 ERR UlPhyDriver05 0 [AERIAL_L2ADAPTER_EVENT] [SCF.PHY] Send SLOT error indication from L1 SFN=0 slot=4 for msg_id=0x07
05:16:49.081405 WRN UlPhyDriver05 0 [DRV.FUNC_UL] task_work_function_ul_aggr_1_pusch Task aborted for Slot Map 0
05:16:49.081405 WRN UlPhyDriver06 0 [DRV.FUNC_UL] task_work_function_ul_aggr_1_orderKernel Task aborted for Slot Map 0
05:16:49.081495 ERR UlPhyDriver05 0 [AERIAL_CUPHYDRV_API_EVENT] [DRV.FH] Peer 0 not registered

05:16:49.081495 ERR UlPhyDriver05 0 [AERIAL_CUPHYDRV_API_EVENT] [DRV.FUNC_UL] UL C-plane send error for cell index 0,error type 3 Map 1 Abort UL Tasks!

05:16:49.081496 ERR UlPhyDriver05 0 [AERIAL_L2ADAPTER_EVENT] [SCF.PHY] Send SLOT error indication from L1 SFN=0 slot=5 for msg_id=0x07
05:16:49.081499 ERR UlPhyDriver05 0 [AERIAL_L2ADAPTER_EVENT] [SCF.PHY] Send SLOT error indication from L1 SFN=0 slot=5 for msg_id=0x07
05:16:49.081903 WRN UlPhyDriver05 0 [DRV.FUNC_UL] task_work_function_ul_aggr_1_orderKernel Task aborted for Slot Map 1
05:16:49.081903 WRN UlPhyDriver06 0 [DRV.FUNC_UL] task_work_function_ul_aggr_1_pusch Task aborted for Slot Map 1
05:16:49.082072 ERR DlPhyDriver11 0 [AERIAL_CUPHYDRV_API_EVENT] [DRV.FH] Peer 0 not registered

05:16:49.082075 ERR DlPhyDriver11 0 [AERIAL_SYSTEM_API_EVENT] [NVLOG.EXIT_HANDLER] Triggering L1 application exit from exit handler
05:16:49.082078 WRN DlPhyDriver11 0 [DRV.API] Trigging L1 exit handler
05:16:49.082081 ERR DlPhyDriver11 0 [AERIAL_L2ADAPTER_EVENT] [SCF.PHY] Send SLOT error indication from L1 SFN=0 slot=7 for msg_id=0x07
05:16:49.082091 WRN DlPhyDriver11 0 [DRV.API] L1 exit handler: end of step 1
05:16:49.082094 WRN DlPhyDriver11 0 [DRV.API] L1 exit handler: end of step 2
05:16:49.082095 WRN DlPhyDriver11 0 [DRV.API] L1 exit handler: step 3 (CUDA_ENABLE_COREDUMP_ON_EXCEPTION unset)
Segmentation fault
type or paste code here
6

Hi @root1031,

Can you please enable INFO level logs for the tags FH.PEER, FH.FLOW and DRV.FH and test again?

There is still an issue with the NIC settings but it is not clear what it is.

Thank you.

7

I solved it by changing nic setting physically!! Thank you

1 Like