Mlx5_common: Fail to create MR for address (0x7ee1d4000000)

I’m trying run the Aerial cuBB Phase4 test (testMAC+cuPHy+RU Emulator) , and the following error occurs:
16:41:21.197602 CON phy_init 0 [DRV.FH] Dynamic beam id start: 16528, end: 32767, num of beam ids per slots: 2192, number of consecutive slots with unique of beam ids: 7
16:41:21.878543 CON phy_init 0 [FH.DOCA] doca_gpu_dmabuf_fd returned 0 (DOCA_SUCCESS is 0)
16:41:21.878555 CON phy_init 0 [FH.DOCA] Mapping transmit queue buffer (0x7ee2fe000000 size 2147483648B dmabuf fd 204) with dmabuf mode
PCAP_LOGGER Initialized
mlx5_common: Fail to create MR for address (0x7ee1d4000000)
mlx5_common: Device 0000:38:00.0 unable to DMA map
16:41:22.567108 ERR phy_init 0 [AERIAL_ORAN_FH_EVENT] [FH.LIB] Exception! Failed to DMA map addr 0x7ee1d4000000 for NIC 0000:38:00.0: Operation not permitted
16:41:22.567137 ERR phy_init 0 [AERIAL_ORAN_FH_EVENT] [DRV.FH] aerial_fh::register_memory error
16:41:22.567138 ERR phy_init 0 [AERIAL_CUPHYDRV_API_EVENT] [DRV.CTX] registerMem error
gRPC Server listening on 0.0.0.0:50051

The environment information:
root@worker2:~# lsmod | grep nvidia_peermem
nvidia_peermem 16384 0
nvidia 11427840 133 nvidia_uvm,nvidia_peermem,gdrdrv,nvidia_modeset
ib_uverbs 163840 8 nvidia_peermem,rdma_ucm,mlx5_ib
root@worker2:~# ofed_info -s
OFED-internal-25.04-0.6.1:
root@worker2:~# nvidia-smi topo -m
GPU0 NIC0 NIC1 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X NODE NODE 0-31 0 N/A
NIC0 NODE X PIX
NIC1 NODE PIX X

Legend:

X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks

NIC Legend:

NIC0: mlx5_0
NIC1: mlx5_1

What’s the reason of this error?

Hi @kemalong,

Can you share the complete phy.log?

What is your system configuration? What is the GPU, CPU and the NIC you are using?

Please try the following for initial check:

  # 1. Check IOMMU status
dmesg | grep -i "IOMMU\|DMAR"

# 2. Check ACS status
lspci -vvv 2>/dev/null | grep -i "ACSCtl" | head -20

# 3. Check locked memory limits
ulimit -l

# 4. Check nvidia-peermem status
cat /sys/kernel/mm/memory_hotplug/online_type 2>/dev/null
dmesg | grep -i "peermem\|nv_peer"

# 5. Check GPU BAR configuration
nvidia-smi -q -d MEMORY | grep -A5 "BAR"

# 6. Verify GPUDirect RDMA works at all
# (if you have the perftest package)
# Run: ib_write_bw -d mlx5_0 --use_cuda=0

Thank you for your reply!

System Configuration:

Two Intel SPR-EE Servers Connected with Mallenox NICs (MT2892 Family, ConectX-6 Dx), one server for cuphycontroller and testMAC, the other one for RU Emulator.

CPU: Intel(R) Xeon(R) Gold 6430

GPU: NVIDIA L4 Tensor Core GPU

NIC: MT2892 Family, ConectX-6 Dx

Initial Check:

1) IOMMU & DMAR:

root@worker2:\~# dmesg | grep -i “IOMMU|DMAR”
\[    0.000000\] Command line: BOOT_IMAGE=/vmlinuz-5.15.0-1042-nvidia-lowlatency root=/dev/mapper/ubuntu–vg-ubuntu–lv ro intel_iommu=off iommu=off pci=realloc=off pci=pcie_bus_safe default_hugepagesz=1G hugepagesz=1G hugepages=64 hugepagesz=2M hugepages=64 tsc=reliable processor.max_cstate=0 audit=0 idle=poll rcu_nocb_poll nosoftlockup irqaffinity=0 isolcpus=managed_irq,domain,4-64 nohz_full=4-64 rcu_nocbs=4-64 earlycon module_blacklist=nouveau acpi_power_meter.force_cap_on=y numa_balancing=disable init_on_alloc=0 preempt=none psi=0 acpi=force reboot=pci
\[    0.023201\] ACPI: DMAR 0x00000000773E9000 000448 (v01 XF     EGLSTRM  00000001 ACPI 00040000)
\[    0.023306\] ACPI: Reserving DMAR table memory at \[mem 0x773e9000-0x773e9447\]
\[    2.950668\] Kernel command line: BOOT_IMAGE=/vmlinuz-5.15.0-1042-nvidia-lowlatency root=/dev/mapper/ubuntu–vg-ubuntu–lv ro intel_iommu=off iommu=off pci=realloc=off pci=pcie_bus_safe default_hugepagesz=1G hugepagesz=1G hugepages=64 hugepagesz=2M hugepages=64 tsc=reliable processor.max_cstate=0 audit=0 idle=poll rcu_nocb_poll nosoftlockup irqaffinity=0 isolcpus=managed_irq,domain,4-64 nohz_full=4-64 rcu_nocbs=4-64 earlycon module_blacklist=nouveau acpi_power_meter.force_cap_on=y numa_balancing=disable init_on_alloc=0 preempt=none psi=0 acpi=force reboot=pci
\[    3.011051\] DMAR: IOMMU disabled
\[    4.118648\] DMAR: Host address width 46
\[    4.118652\] DMAR: DRHD base: 0x000000dbffc000 flags: 0x0
\[    4.118664\] DMAR: dmar0: reg_base_addr dbffc000 ver 6:0 cap 9ed008c40780466 ecap 3ee9e86f050df
\[    4.118673\] DMAR: DRHD base: 0x000000e37fc000 flags: 0x0
\[    4.118680\] DMAR: dmar1: reg_base_addr e37fc000 ver 6:0 cap 9ed008c40780466 ecap 3ee9e86f050df
\[    4.118687\] DMAR: DRHD base: 0x000000eaffc000 flags: 0x0
\[    4.118694\] DMAR: dmar2: reg_base_addr eaffc000 ver 6:0 cap 9ed008c40780466 ecap 3ee9e86f050df
\[    4.118701\] DMAR: DRHD base: 0x000000f23fc000 flags: 0x0
\[    4.118708\] DMAR: dmar3: reg_base_addr f23fc000 ver 6:0 cap 9ed008c40780466 ecap 3ee9e86f050df
\[    4.118714\] DMAR: DRHD base: 0x000000f97fc000 flags: 0x0
\[    4.118725\] DMAR: dmar4: reg_base_addr f97fc000 ver 6:0 cap 9ed008c40780466 ecap 3ee9e86f050df
\[    4.118732\] DMAR: DRHD base: 0x000000d47fc000 flags: 0x0
\[    4.118739\] DMAR: dmar5: reg_base_addr d47fc000 ver 6:0 cap 9ed008c40780466 ecap 3ee9e86f050df
\[    4.118746\] DMAR: DRHD base: 0x000000f9ffc000 flags: 0x0
\[    4.118755\] DMAR: dmar6: reg_base_addr f9ffc000 ver 6:0 cap 9ed008c40780466 ecap 3ef9ea6f050df
\[    4.118762\] DMAR: DRHD base: 0x000000fa7fc000 flags: 0x0
\[    4.118770\] DMAR: dmar7: reg_base_addr fa7fc000 ver 6:0 cap 9ed008c40780466 ecap 3ef9ea6f050df
\[    4.118777\] DMAR: DRHD base: 0x000000faffc000 flags: 0x0
\[    4.118784\] DMAR: dmar8: reg_base_addr faffc000 ver 6:0 cap 9ed008c40780466 ecap 3ef9ea6f050df
\[    4.118791\] DMAR: DRHD base: 0x000000fb7fc000 flags: 0x0
\[    4.118798\] DMAR: dmar9: reg_base_addr fb7fc000 ver 6:0 cap 9ed008c40780466 ecap 3ef9ea6f050df
\[    4.118805\] DMAR: DRHD base: 0x000000a6ffc000 flags: 0x0
\[    4.118812\] DMAR: dmar10: reg_base_addr a6ffc000 ver 6:0 cap 9ed008c40780466 ecap 3ee9e86f050df
\[    4.118819\] DMAR: DRHD base: 0x000000afffc000 flags: 0x0
\[    4.118826\] DMAR: dmar11: reg_base_addr afffc000 ver 6:0 cap 9ed008c40780466 ecap 3ee9e86f050df
\[    4.118833\] DMAR: DRHD base: 0x000000b8ffc000 flags: 0x0
\[    4.118840\] DMAR: dmar12: reg_base_addr b8ffc000 ver 6:0 cap 9ed008c40780466 ecap 3ee9e86f050df
\[    4.118846\] DMAR: DRHD base: 0x000000c1bfc000 flags: 0x0
\[    4.118853\] DMAR: dmar13: reg_base_addr c1bfc000 ver 6:0 cap 9ed008c40780466 ecap 3ee9e86f050df
\[    4.118860\] DMAR: DRHD base: 0x000000ca7fc000 flags: 0x0
\[    4.118867\] DMAR: dmar14: reg_base_addr ca7fc000 ver 6:0 cap 9ed008c40780466 ecap 3ee9e86f050df
\[    4.118873\] DMAR: DRHD base: 0x000000caffc000 flags: 0x0
\[    4.118880\] DMAR: dmar15: reg_base_addr caffc000 ver 6:0 cap 9ed008c40780466 ecap 3ef9ea6f050df
\[    4.118887\] DMAR: DRHD base: 0x000000cb7fc000 flags: 0x0
\[    4.118894\] DMAR: dmar16: reg_base_addr cb7fc000 ver 6:0 cap 9ed008c40780466 ecap 3ef9ea6f050df
\[    4.118901\] DMAR: DRHD base: 0x000000cbffc000 flags: 0x0
\[    4.118908\] DMAR: dmar17: reg_base_addr cbffc000 ver 6:0 cap 9ed008c40780466 ecap 3ef9ea6f050df
\[    4.118914\] DMAR: DRHD base: 0x000000cc7fc000 flags: 0x0
\[    4.118921\] DMAR: dmar18: reg_base_addr cc7fc000 ver 6:0 cap 9ed008c40780466 ecap 3ef9ea6f050df
\[    4.118928\] DMAR: DRHD base: 0x0000009dffc000 flags: 0x1
\[    4.118935\] DMAR: dmar19: reg_base_addr 9dffc000 ver 6:0 cap 9ed008c40780466 ecap 3ee9e86f050df
\[    4.118942\] DMAR: RMRR base: 0x0000006e7c2000 end: 0x0000006e7c4fff
\[    4.118948\] DMAR: ATSR flags: 0x0
\[    4.118952\] DMAR: RHSA base: 0x0000009dffc000 proximity domain: 0x0
\[    4.118958\] DMAR: RHSA base: 0x000000a6ffc000 proximity domain: 0x0
\[    4.118962\] DMAR: RHSA base: 0x000000afffc000 proximity domain: 0x0
\[    4.118965\] DMAR: RHSA base: 0x000000b8ffc000 proximity domain: 0x0
\[    4.118969\] DMAR: RHSA base: 0x000000c1bfc000 proximity domain: 0x0
\[    4.118973\] DMAR: RHSA base: 0x000000ca7fc000 proximity domain: 0x0
\[    4.118977\] DMAR: RHSA base: 0x000000caffc000 proximity domain: 0x0
\[    4.118981\] DMAR: RHSA base: 0x000000cb7fc000 proximity domain: 0x0
\[    4.118985\] DMAR: RHSA base: 0x000000cbffc000 proximity domain: 0x0
\[    4.118988\] DMAR: RHSA base: 0x000000cc7fc000 proximity domain: 0x0
\[    4.118992\] DMAR: RHSA base: 0x000000dbffc000 proximity domain: 0x1
\[    4.118997\] DMAR: RHSA base: 0x000000e37fc000 proximity domain: 0x1
\[    4.119000\] DMAR: RHSA base: 0x000000eaffc000 proximity domain: 0x1
\[    4.119004\] DMAR: RHSA base: 0x000000f23fc000 proximity domain: 0x1
\[    4.119008\] DMAR: RHSA base: 0x000000f97fc000 proximity domain: 0x1
\[    4.119012\] DMAR: RHSA base: 0x000000d47fc000 proximity domain: 0x1
\[    4.119016\] DMAR: RHSA base: 0x000000f9ffc000 proximity domain: 0x1
\[    4.119020\] DMAR: RHSA base: 0x000000fa7fc000 proximity domain: 0x1
\[    4.119024\] DMAR: RHSA base: 0x000000faffc000 proximity domain: 0x1
\[    4.119028\] DMAR: RHSA base: 0x000000fb7fc000 proximity domain: 0x1
\[    4.119031\] DMAR: SATC flags: 0x0
\[    4.119038\] DMAR-IR: IOAPIC id 8 under DRHD base  0x9dffc000 IOMMU 19
\[    4.119043\] DMAR-IR: HPET id 0 under DRHD base 0x9dffc000
\[    4.119048\] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
\[    4.125509\] DMAR-IR: Enabled IRQ remapping in x2apic mode
\[   10.727894\] iommu: Default domain type: Translated
\[   10.727894\] iommu: DMA domain TLB invalidation policy: lazy mode

2) ACSCtl:

root@worker2:\~# lspci -vvv 2>/dev/null | grep -i “ACSCtl” | head -20
ACSCtl: SrcValid+ TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
ACSCtl: SrcValid+ TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
ACSCtl: SrcValid+ TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
ACSCtl: SrcValid+ TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
ACSCtl: SrcValid+ TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
ACSCtl: SrcValid+ TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
ACSCtl: SrcValid+ TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
ACSCtl: SrcValid+ TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
ACSCtl: SrcValid+ TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
ACSCtl: SrcValid+ TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-

3) Locked Memory:

root@worker2:\~# ulimit -l
32975068

4) nvidia-peermem:

root@worker2:\~# lsmod | grep nvidia_peermem
nvidia_peermem         16384  0
nvidia              11427840  9 nvidia_uvm,nvidia_peermem,gdrdrv,nvidia_modeset
ib_uverbs             163840  3 nvidia_peermem,rdma_ucm,mlx5_ib
root@worker2:\~# dmesg | grep -i “peermem|nv_peer”

5) GPU BAR:

root@worker2:\~# nvidia-smi -q -d MEMORY | grep -A5 “BAR”
BAR1 Memory Usage
Total                             : 32768 MiB
Used                              : 1 MiB
Free                              : 32767 MiB
Conf Compute Protected Memory Usage
Total                             : 0 MiB

6) perftest:
Node-02 is the server for RU Emulator (without GPU),worker2 is the server for cuphycontroller and testMAC (with L4 GPU):

root@Node-02:/home/ai-ran/perftest# ./ib_write_bw -d mlx5_0 

************************************
* Waiting for client to connect... *
************************************
---------------------------------------------------------------------------------------
                    RDMA_Write BW Test
 Dual-port       : OFF          Device         : mlx5_0
 Number of qps   : 1            Transport type : IB
 Connection type : RC           Using SRQ      : OFF
 PCIe relax order: ON           Lock-free      : OFF
 ibv_wr* API     : ON           Using Enhanced Reorder      : OFF
 CQ Moderation   : 1
 CQE Poll Batch  : Dynamic
 Mtu             : 2048[B]
 Link type       : Ethernet
 GID index       : 1
 Max inline data : 0[B]
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x0349 PSN 0x688c18 RKey 0x1ac000 VAddr 0x007fe86cdd5000
 GID: 254:128:00:00:00:00:00:00:186:206:246:255:254:254:99:184
 remote address: LID 0000 QPN 0x2238 PSN 0xb3fc2e RKey 0x201620 VAddr 0x007f957e810000
 GID: 254:128:00:00:00:00:00:00:186:206:246:255:254:254:98:244
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[MiB/sec]    BW average[MiB/sec]   MsgRate[Mpps]
 65536      5000             1135.97            1134.92              0.018159
---------------------------------------------------------------------------------------

root@worker2:~# ib_write_bw -d mlx5_0 --use_cuda=0 10.110.162.243
Perftest doesn't supports CUDA tests with inline messages: inline size set to 0
initializing CUDA
Listing all CUDA devices in system:
CUDA device 0: PCIe address is 27:00

Picking device No. 0
[pid = 1734272, dev = 0] device name = [NVIDIA L4]
creating CUDA Ctx
making it the current CUDA Ctx
CUDA device integrated: 0
allocated GPU buffer of a 131072 address at 0x5584bf5e8090 for type CUDA_MEM_DEVICE
---------------------------------------------------------------------------------------
                    RDMA_Write BW Test
 Dual-port       : OFF          Device         : mlx5_0
 Number of qps   : 1            Transport type : IB
 Connection type : RC           Using SRQ      : OFF
 PCIe relax order: ON           Lock-free      : OFF
 ibv_wr* API     : ON           Using Enhanced Reorder      : OFF
 TX depth        : 128
 CQ Moderation   : 1
 CQE Poll Batch  : Dynamic
 Mtu             : 2048[B]
 Link type       : Ethernet
 GID index       : 1
 Max inline data : 0[B]
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x2238 PSN 0xb3fc2e RKey 0x201620 VAddr 0x007f957e810000
 GID: 254:128:00:00:00:00:00:00:186:206:246:255:254:254:98:244
 remote address: LID 0000 QPN 0x0349 PSN 0x688c18 RKey 0x1ac000 VAddr 0x007fe86cdd5000
 GID: 254:128:00:00:00:00:00:00:186:206:246:255:254:254:99:184
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[MiB/sec]    BW average[MiB/sec]   MsgRate[Mpps]
Conflicting CPU frequency values detected: 1201.000000 != 2600.000000. CPU Frequency is not max.
 65536      5000             1135.97            1134.92              0.018159
---------------------------------------------------------------------------------------
deallocating GPU buffer 00007f957e800000
destroying current CUDA Ctx

When I run the Simulator E2E, a segmation fault is also occurs, is this related to the aforementioned error?

17:16:08.259640 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.093 MiB for cuPHY CSIRS channel object (0x3de74e80380).
17:16:08.259653 CON msg_processing 0 [DRV.API] Update cell: mplane_id=1 dl_grid_sz=273 
17:16:08.259654 CON msg_processing 0 [DRV.API] Update cell: mplane_id=1 ul_grid_sz=273 
17:16:08.259670 CON msg_processing 0 [SCF.PHY] on_config_request: create_cell_configs for cell_id=0 phy_cell_id=41 returned error_code=0
17:16:08.259686 CON msg_processing 0 [SCF.PHY] send_cell_config_response: Send CONFIG.response: cell_id=0 error_code=0x0
17:16:08.259721 CON msg_processing 0 [L2A.TRANSPORT] wrapper: configured_cells_mask=0x1 expected_mask=0x1 all_cells_configured=true - all cells configured
17:16:08.259748 CON msg_processing 0 [SCF.PHY] on_cell_start_request: Cell 0 Received START.request, sending START.response... phy_cell_id=41
17:16:08.259907 CON msg_processing 0 [L2A.TRANSPORT] wrapper: configured_cells_mask=0x1 expected_mask=0x1 all_cells_configured=true - all cells configured
17:16:08.268268 CON timer_thread 0 [L2A.TICK] Thread slot_indication_thread_sleep_method initialized fmtlog
17:16:08.268275 CON timer_thread 0 [L2A.TICK] PTP Configs: gps_alpha: 0 gps_beta: 0
17:16:08.268282 CON timer_thread 0 [L2A.TICK] Start time: tick=1769534168268271359, seconds since epoch = 1769534168, nanoseconds = 268271359
17:16:08.268283 CON timer_thread 0 [L2A.TICK] FIRST tick scheduled for: tick=1769534170680000000, seconds since epoch = 1769534170, nanoseconds = 680000000
/opt/nvidia/cuBB/testBenches/phase4_test_scripts/run2_cuPHYcontroller.sh: line 270:  7912 Segmentation fault      sudo -E $WITH_TIMEOUT $GDB_SCRIPT "$cuBB_SDK/$BUILD_DIR/cuPHY-CP/cuphycontroller/examples/cuphycontroller_scf" $CONTROLLER_MODE

PHY Log:
phy.log (40.7 KB)

Can you please check if you have the following setting in your cuphycontroller yaml configuration file?

gpu_init_comms_via_cpu: 1

Thank you very much! Yes, I have this setting in the configuration file, as


When I use “gpu_init_comms_via_cpu: 0”, the segmentation issue has been addressed, but the following new errors occur:

root@c_aerial_root:/opt/nvidia/cuBB# $cuBB_SDK/testBenches/phase4_test_scripts/run2_cuPHYcontroller.sh $RUN2_CUPHYCONTROLLER_PARAMS
stopping MPS for green contexts
Stopping MPS...
Cannot find MPS control daemon process
/opt/nvidia/cuBB build.10_04.x86_64 F08_GL4
  /opt/nvidia/cuBB/build.10_04.x86_64/cuPHY-CP/cuphycontroller/examples/cuphycontroller_scf F08_GL4
F08_GL4
Started cuphycontroller on CPU core 19
05:47:42.336117 CON phy_init 0 [CTL.SCF] Config file: /opt/nvidia/cuBB/cuPHY-CP/cuphycontroller/config/cuphycontroller_F08_GL4.yaml
05:47:42.338573 CON phy_init 0 [CTL.SCF] low_priority_core=0
05:47:42.338596 CON phy_init 0 [APP.CONFIG] Current TAI offset: 0s
05:47:42.338651 CON phy_init 0 [NVLOG.CPP] Using /opt/nvidia/cuBB/cuPHY/nvlog/config/nvlog_config.yaml for nvlog configuration
05:47:42.354302 CON phy_init 0 [NVLOG.CPP] Output log file path /tmp/phy.log
Aerial metrics backend address: 127.0.0.1:60000
YAML invalid key: dlc_alloc_cplane_bfw_txq Using default value of 0 for YAML_PARAM_DLC_ALLOC_CPLANE_BFW_TXQ
05:47:42.368004 CON phy_init 0 [CTL.YAML] cuphycontroller config. yaml does not have ul_order_timeout_gpu_srs_ns key; defaulting to 5200000 ns
05:47:42.368021 WRN phy_init 0 [CTL.YAML] cuphycontroller config. yaml does not have ul_order_kernel_mode key; defaulting to 0 (Ping-Pong mode).
05:47:42.368111 WRN phy_init 0 [CTL.YAML] YAML invalid key: enable_l1_param_sanity_check Using default value of 0 to YAML_PARAM_ENABLE_L1_PARAM_SANITY_CHECK
05:47:42.368189 WRN phy_init 0 [CTL.YAML] YAML invalid key: pusch_weighted_average_cfo Using default value of 0 to PUSCH-WEIGHTED-AVERAGE-CFO
05:47:42.368217 WRN phy_init 0 [CTL.YAML] YAML invalid key: pucch_aggr_per_ctx Using default value of 4 to YAML_PARAM_PUCCH_AGGR_PER_CTX
05:47:42.368226 WRN phy_init 0 [CTL.YAML] YAML invalid key: srs_aggr_per_ctx Using default value of 2 to YAML_PARAM_SRS_AGGR_PER_CTX
05:47:42.368235 WRN phy_init 0 [CTL.YAML] YAML invalid key: ul_input_buffer_per_cell_srs Using default value of 4 to YAML_PARAM_UL_INPUT_BUFFER_NUM_PER_CELL_SRS
05:47:42.368244 WRN phy_init 0 [CTL.YAML] YAML invalid key: max_ru_unhealthy_ul_slots Using default value of 0 for YAML_PARAM_MAX_RU_UNHEALTHY_UL_SLOTS
05:47:42.368270 WRN phy_init 0 [CTL.YAML] YAML invalid key: ul_pcap_capture_enable Using default value of 0 for YAML_PARAM_UL_PCAP_CAPTURE_ENABLE
05:47:42.368280 WRN phy_init 0 [CTL.YAML] YAML invalid key: pcap_logger_ul_cplane_enable PCAP Logging parameters are set incorrectly, so logger is disabled by default
05:47:42.368333 WRN phy_init 0 [CTL.YAML] YAML invalid key: enable_tx_notification Using default value of 0 for YAML_PARAM_ENABLE_TX_NOTIFICATION
05:47:42.368456 WRN phy_init 0 [CTL.YAML] YAML invalid key: pusch_nMaxLdpcHetConfigs Using default value of 32 to PUSCH-N-MAX-LDPC-HET-CONFIGS
05:47:42.368538 CON phy_init 0 [CTL.YAML] cell_id 1 nic_index :0
05:47:42.368820 CON phy_init 0 [CTL.YAML] t1a_min_cp_dl_ns/t1a_max_cp_dl_ns not set in config file, using default t1a_min_cp_dl_ns = 419 us and t1a_max_cp_dl_ns = 669 us
05:47:42.368827 CON phy_init 0 [CTL.YAML] t1a_min_cp_ul_ns not set in config file, using default of 285 us
05:47:42.368947 CON phy_init 0 [CTL.YAML] Num Slots: 8
05:47:42.368948 CON phy_init 0 [CTL.YAML] Enable UL cuPHY Graphs: 1
05:47:42.368948 CON phy_init 0 [CTL.YAML] Enable DL cuPHY Graphs: 1
05:47:42.368948 CON phy_init 0 [CTL.YAML] Accurate TX scheduling clock resolution (ns): 500
05:47:42.368949 CON phy_init 0 [CTL.YAML] DPDK core: 0
05:47:42.368949 CON phy_init 0 [CTL.YAML] Prometheus core: -1
05:47:42.368950 CON phy_init 0 [CTL.YAML] UL cores: 
05:47:42.368950 CON phy_init 0 [CTL.YAML]       - 2
05:47:42.368950 CON phy_init 0 [CTL.YAML]       - 3
05:47:42.368951 CON phy_init 0 [CTL.YAML] DL cores: 
05:47:42.368951 CON phy_init 0 [CTL.YAML]       - 4
05:47:42.368951 CON phy_init 0 [CTL.YAML]       - 5
05:47:42.368951 CON phy_init 0 [CTL.YAML]       - 6
05:47:42.368951 CON phy_init 0 [CTL.YAML]       - 7
05:47:42.368951 CON phy_init 0 [CTL.YAML] Debug worker: 9
05:47:42.368952 CON phy_init 0 [CTL.YAML] Data Lake core: -1
05:47:42.368952 CON phy_init 0 [CTL.YAML] SRS starting Section ID: 3072
05:47:42.368952 CON phy_init 0 [CTL.YAML] PRACH starting Section ID: 2048
05:47:42.368952 CON phy_init 0 [CTL.YAML] USE GREEN CONTEXTS: 1
05:47:42.368952 CON phy_init 0 [CTL.YAML] USE BATCHED MEMCPY: 1
05:47:42.368952 CON phy_init 0 [CTL.YAML] MPS SM PUSCH: 32
05:47:42.368953 CON phy_init 0 [CTL.YAML] MPS SM PUCCH: 8
05:47:42.368954 CON phy_init 0 [CTL.YAML] MPS SM PRACH: 4
05:47:42.368954 CON phy_init 0 [CTL.YAML] MPS SM UL ORDER: 6
05:47:42.368954 CON phy_init 0 [CTL.YAML] MPS SM PDSCH: 40
05:47:42.368954 CON phy_init 0 [CTL.YAML] MPS SM PDCCH: 4
05:47:42.368955 CON phy_init 0 [CTL.YAML] MPS SM PBCH: 2
05:47:42.368955 CON phy_init 0 [CTL.YAML] MPS SM GPU_COMMS: 16
05:47:42.368955 CON phy_init 0 [CTL.YAML] MPS SM SRS: 14
05:47:42.368955 CON phy_init 0 [CTL.YAML] UL Order Kernel Mode: 0
05:47:42.368955 CON phy_init 0 [CTL.YAML] PDSCH fallback: 0
05:47:42.368955 CON phy_init 0 [CTL.YAML] Massive MIMO enable: 0
05:47:42.368956 CON phy_init 0 [CTL.YAML] mMIMO_enable feature 0
05:47:42.368956 CON phy_init 0 [CTL.YAML] Enable SRS : 0
05:47:42.368956 CON phy_init 0 [CTL.YAML] ul_order_timeout_gpu_log_enable: 1
05:47:42.368956 CON phy_init 0 [CTL.YAML] ue_mode: 0
05:47:42.368956 CON phy_init 0 [CTL.YAML] Aggr Obj Non-availability threshold: 5
05:47:42.368956 CON phy_init 0 [CTL.YAML] sendCPlane_timing_error_th_ns: 0
05:47:42.368956 CON phy_init 0 [CTL.YAML] pusch_aggr_per_ctx: 4
05:47:42.368957 CON phy_init 0 [CTL.YAML] prach_aggr_per_ctx: 2
05:47:42.368957 CON phy_init 0 [CTL.YAML] pucch_aggr_per_ctx: 4
05:47:42.368957 CON phy_init 0 [CTL.YAML] srs_aggr_per_ctx: 2
05:47:42.368957 CON phy_init 0 [CTL.YAML] max_harq_pools: 128
05:47:42.368957 CON phy_init 0 [CTL.YAML] max_harq_tx_count_bundled: 10
05:47:42.368957 CON phy_init 0 [CTL.YAML] max_harq_tx_count_non_bundled: 4
05:47:42.368957 CON phy_init 0 [CTL.YAML] ul_input_buffer_per_cell: 5
05:47:42.368958 CON phy_init 0 [CTL.YAML] ul_input_buffer_per_cell_srs: 4
05:47:42.368958 CON phy_init 0 [CTL.YAML] max_ru_unhealthy_ul_slots: 0
05:47:42.368958 CON phy_init 0 [CTL.YAML] srs_chest_algo_type: 0
05:47:42.368958 CON phy_init 0 [CTL.YAML] srs_chest_tol2_normalization_algo_type: 1
05:47:42.368958 CON phy_init 0 [CTL.YAML] srs_chest_tol2_constant_scaler: 32768
05:47:42.368959 CON phy_init 0 [CTL.YAML] bfw_power_normalization_alg_selector: 1
05:47:42.368959 CON phy_init 0 [CTL.YAML] bfw_beta_prescaler: 16384
05:47:42.368959 CON phy_init 0 [CTL.YAML] total_num_srs_chest_buffers: 3072
05:47:42.368959 CON phy_init 0 [CTL.YAML] send_static_bfw_wt_all_cplane: 1
05:47:42.368959 CON phy_init 0 [CTL.YAML] ul_pcap_capture_enable: 0
05:47:42.368960 CON phy_init 0 [CTL.YAML] ul_pcap_capture_thread_cpu_affinity: 0
05:47:42.368960 CON phy_init 0 [CTL.YAML] ul_pcap_capture_thread_sched_priority: 0
05:47:42.368960 CON phy_init 0 [CTL.YAML] pcap_logger_ul_cplane_enable: 0
05:47:42.368960 CON phy_init 0 [CTL.YAML] pcap_logger_dl_cplane_enable: 0
05:47:42.368960 CON phy_init 0 [CTL.YAML] pcap_logger_thread_cpu_affinity: 0
05:47:42.368961 CON phy_init 0 [CTL.YAML] pcap_logger_thread_sched_prio: 0
05:47:42.368961 CON phy_init 0 [CTL.YAML] pcap_logger_file_save_dir: .
05:47:42.368961 CON phy_init 0 [CTL.YAML] static_beam_id_start: 1
05:47:42.368961 CON phy_init 0 [CTL.YAML] static_beam_id_end: 16527
05:47:42.368962 CON phy_init 0 [CTL.YAML] dynamic_beam_id_start: 16528
05:47:42.368962 CON phy_init 0 [CTL.YAML] dynamic_beam_id_end: 32767
05:47:42.368962 CON phy_init 0 [CTL.YAML] ul_order_timeout_gpu_log_enable: 1
05:47:42.368962 CON phy_init 0 [CTL.YAML] pusch_workCancelMode: 2
05:47:42.368962 CON phy_init 0 [CTL.YAML] GPU-initiated comms DL: 1
05:47:42.368962 CON phy_init 0 [CTL.YAML] GPU-initiated comms (via CPU): 0
05:47:42.368962 CON phy_init 0 [CTL.YAML] CPU-initiated comms : 0
05:47:42.368963 CON phy_init 0 [CTL.YAML] Cell group: 1
05:47:42.368963 CON phy_init 0 [CTL.YAML] Cell group num: 1
05:47:42.368963 CON phy_init 0 [CTL.YAML] puxchPolarDcdrListSz: 8
05:47:42.368963 CON phy_init 0 [CTL.YAML] split_ul_cuda_streams: 0
05:47:42.368963 CON phy_init 0 [CTL.YAML] serialize_pucch_pusch: 0
05:47:42.368963 CON phy_init 0 [CTL.YAML] bfw_c_plane_chaining_mode: 1
05:47:42.368963 CON phy_init 0 [CTL.YAML] fh_stats_dump_cpu_core: -1
05:47:42.368964 CON phy_init 0 [CTL.YAML] fix_beta_dl: 0
05:47:42.368964 CON phy_init 0 [CTL.YAML] pdump_client_thread: -1
05:47:42.368964 CON phy_init 0 [CTL.YAML] profiler_sec: 0
05:47:42.368964 CON phy_init 0 [CTL.YAML] datalake_address: localhost
05:47:42.368964 CON phy_init 0 [CTL.YAML] dpdk_file_prefix: cuphycontroller
05:47:42.368964 CON phy_init 0 [CTL.YAML] puschrxChestFactorySettingsFilename: 
05:47:42.368965 CON phy_init 0 [CTL.YAML] accu_tx_sched_disable: 0
05:47:42.368965 CON phy_init 0 [CTL.YAML] cplane_disable: 0
05:47:42.368965 CON phy_init 0 [CTL.YAML] disable_empw: 0
05:47:42.368965 CON phy_init 0 [CTL.YAML] dlc_alloc_cplane_bfw_txq: 0
05:47:42.368965 CON phy_init 0 [CTL.YAML] dlc_bfw_enable_divide_per_cell: 1
05:47:42.368966 CON phy_init 0 [CTL.YAML] dpdk_verbose_logs: 0
05:47:42.368966 CON phy_init 0 [CTL.YAML] enable_cpu_task_tracing: 1
05:47:42.368966 CON phy_init 0 [CTL.YAML] enable_dl_cqe_tracing: 0
05:47:42.368966 CON phy_init 0 [CTL.YAML] enable_l1_param_sanity_check: 0
05:47:42.368966 CON phy_init 0 [CTL.YAML] enable_ok_tb: 0
05:47:42.368966 CON phy_init 0 [CTL.YAML] enable_prepare_tracing: 1
05:47:42.368966 CON phy_init 0 [CTL.YAML] mCh_segment_proc_enable: 0
05:47:42.368967 CON phy_init 0 [CTL.YAML] pmu_metrics: 3
05:47:42.368967 CON phy_init 0 [CTL.YAML] puschCfo: 1
05:47:42.368967 CON phy_init 0 [CTL.YAML] puschDftSOfdm: 0
05:47:42.368967 CON phy_init 0 [CTL.YAML] puschEnablePerPrgChEst: 0
05:47:42.368967 CON phy_init 0 [CTL.YAML] puschRssi: 1
05:47:42.368967 CON phy_init 0 [CTL.YAML] puschSelectChEstAlgo: 1
05:47:42.368967 CON phy_init 0 [CTL.YAML] puschSelectEqCoeffAlgo: 3
05:47:42.368967 CON phy_init 0 [CTL.YAML] puschSinr: 2
05:47:42.368968 CON phy_init 0 [CTL.YAML] puschTbSizeCheck: 1
05:47:42.368968 CON phy_init 0 [CTL.YAML] puschTdi: 1
05:47:42.368968 CON phy_init 0 [CTL.YAML] puschTo: 1
05:47:42.368968 CON phy_init 0 [CTL.YAML] pusch_deviceGraphLaunchEn: 1
05:47:42.368968 CON phy_init 0 [CTL.YAML] ulc_alloc_cplane_bfw_txq: 1
05:47:42.368968 CON phy_init 0 [CTL.YAML] ulc_bfw_enable_divide_per_cell: 0
05:47:42.368968 CON phy_init 0 [CTL.YAML] ul_rx_pkt_tracing_level: 0
05:47:42.368969 CON phy_init 0 [CTL.YAML] ul_rx_pkt_tracing_level_srs: 0
05:47:42.368969 CON phy_init 0 [CTL.YAML] ul_warmup_frame_count: 8
05:47:42.368969 CON phy_init 0 [CTL.YAML] forcedNumCsi2Bits: 0
05:47:42.368969 CON phy_init 0 [CTL.YAML] pusch_waitTimeOutPostEarlyHarqUs: 3000
05:47:42.368969 CON phy_init 0 [CTL.YAML] pusch_waitTimeOutPreEarlyHarqUs: 3000
05:47:42.368969 CON phy_init 0 [CTL.YAML] validation: 0
05:47:42.368971 CON phy_init 0 [CTL.YAML] cqe_trace_slot_mask: 196800
05:47:42.368971 CON phy_init 0 [CTL.YAML] datalake_samples: 1000000
05:47:42.368971 CON phy_init 0 [CTL.YAML] num_ok_tb_slot: 0
05:47:42.368971 CON phy_init 0 [CTL.YAML] pusch_nMaxLdpcHetConfigs: 32
05:47:42.368971 CON phy_init 0 [CTL.YAML] sendCPlane_dlbfw_backoff_th_ns: 300000
05:47:42.368972 CON phy_init 0 [CTL.YAML] sendCPlane_ulbfw_backoff_th_ns: 300000
05:47:42.368972 CON phy_init 0 [CTL.YAML] ul_order_max_rx_pkts: 256
05:47:42.368972 CON phy_init 0 [CTL.YAML] ul_order_rx_pkts_timeout_ns: 50000
05:47:42.368972 CON phy_init 0 [CTL.YAML] ul_order_timeout_cpu_ns: 8000000
05:47:42.368972 CON phy_init 0 [CTL.YAML] ul_order_timeout_gpu_ns: 8000000
05:47:42.368972 CON phy_init 0 [CTL.YAML] ul_order_timeout_gpu_srs_ns: 5200000
05:47:42.368972 CON phy_init 0 [CTL.YAML] ul_order_timeout_log_interval_ns: 0
05:47:42.368973 CON phy_init 0 [CTL.YAML] ul_srs_aggr3_task_launch_offset_ns: 500000
05:47:42.368973 CON phy_init 0 [CTL.YAML] workers_sched_priority: 95
05:47:42.368973 CON phy_init 0 [CTL.YAML] cqe_trace_cell_mask: 1
05:47:42.368974 CON phy_init 0 [CTL.YAML] Number of Cell Configs: 1
05:47:42.368974 CON phy_init 0 [CTL.YAML] L2Adapter config file: /opt/nvidia/cuBB/cuPHY-CP/cuphycontroller/config/l2_adapter_config_F08_CG1.yaml
05:47:42.368974 CON phy_init 0 [CTL.YAML] Cell name: O-RU 0
05:47:42.368974 CON phy_init 0 [CTL.YAML]       MU: 1
05:47:42.368975 CON phy_init 0 [CTL.YAML]       ID: 1
05:47:42.368975 CON phy_init 0 [CTL.YAML] Number of MPlane Configs: 1
05:47:42.368975 CON phy_init 0 [CTL.YAML]       Mplane ID: 1
05:47:42.368976 CON phy_init 0 [CTL.YAML]       VLAN ID: 2
05:47:42.368976 CON phy_init 0 [CTL.YAML]       Source Eth Address: b8:ce:f6:fe:62:f4
05:47:42.368977 CON phy_init 0 [CTL.YAML]       Destination Eth Address: 20:04:9b:9e:27:a3
05:47:42.368978 CON phy_init 0 [CTL.YAML]       NIC port: 0000:38:00.0
05:47:42.368978 CON phy_init 0 [CTL.YAML]       RU Type: 3
05:47:42.368978 CON phy_init 0 [CTL.YAML]       U-plane TXQs: 1
05:47:42.368978 CON phy_init 0 [CTL.YAML]       DL compression method: 1
05:47:42.368978 CON phy_init 0 [CTL.YAML]       DL iq bit width: 9
05:47:42.368978 CON phy_init 0 [CTL.YAML]       UL compression method: 1
05:47:42.368979 CON phy_init 0 [CTL.YAML]       UL iq bit width: 9
05:47:42.368979 CON phy_init 0 [CTL.YAML] 
05:47:42.368979 CON phy_init 0 [CTL.YAML]       Flow list SSB/PBCH: 
05:47:42.368979 CON phy_init 0 [CTL.YAML]               32512
05:47:42.368980 CON phy_init 0 [CTL.YAML]               32513
05:47:42.368980 CON phy_init 0 [CTL.YAML]               32514
05:47:42.368980 CON phy_init 0 [CTL.YAML]               32515
05:47:42.368980 CON phy_init 0 [CTL.YAML]               32516
05:47:42.368980 CON phy_init 0 [CTL.YAML]               32517
05:47:42.368980 CON phy_init 0 [CTL.YAML]               32518
05:47:42.368980 CON phy_init 0 [CTL.YAML]               32519
05:47:42.368980 CON phy_init 0 [CTL.YAML]               32520
05:47:42.368981 CON phy_init 0 [CTL.YAML]               32521
05:47:42.368981 CON phy_init 0 [CTL.YAML]               32522
05:47:42.368981 CON phy_init 0 [CTL.YAML]               32523
05:47:42.368981 CON phy_init 0 [CTL.YAML]               32524
05:47:42.368981 CON phy_init 0 [CTL.YAML]               32525
05:47:42.368981 CON phy_init 0 [CTL.YAML]               32526
05:47:42.368981 CON phy_init 0 [CTL.YAML]               32527
05:47:42.368981 CON phy_init 0 [CTL.YAML]       Flow list PDCCH: 
05:47:42.368982 CON phy_init 0 [CTL.YAML]               32512
05:47:42.368982 CON phy_init 0 [CTL.YAML]               32513
05:47:42.368982 CON phy_init 0 [CTL.YAML]               32514
05:47:42.368982 CON phy_init 0 [CTL.YAML]               32515
05:47:42.368982 CON phy_init 0 [CTL.YAML]               32516
05:47:42.368982 CON phy_init 0 [CTL.YAML]               32517
05:47:42.368982 CON phy_init 0 [CTL.YAML]               32518
05:47:42.368982 CON phy_init 0 [CTL.YAML]               32519
05:47:42.368982 CON phy_init 0 [CTL.YAML]               32520
05:47:42.368983 CON phy_init 0 [CTL.YAML]               32521
05:47:42.368983 CON phy_init 0 [CTL.YAML]               32522
05:47:42.368983 CON phy_init 0 [CTL.YAML]               32523
05:47:42.368983 CON phy_init 0 [CTL.YAML]               32524
05:47:42.368983 CON phy_init 0 [CTL.YAML]               32525
05:47:42.368983 CON phy_init 0 [CTL.YAML]               32526
05:47:42.368983 CON phy_init 0 [CTL.YAML]               32527
05:47:42.368983 CON phy_init 0 [CTL.YAML]       Flow list PDSCH: 
05:47:42.368983 CON phy_init 0 [CTL.YAML]               32512
05:47:42.368984 CON phy_init 0 [CTL.YAML]               32513
05:47:42.368984 CON phy_init 0 [CTL.YAML]               32514
05:47:42.368984 CON phy_init 0 [CTL.YAML]               32515
05:47:42.368984 CON phy_init 0 [CTL.YAML]               32516
05:47:42.368984 CON phy_init 0 [CTL.YAML]               32517
05:47:42.368984 CON phy_init 0 [CTL.YAML]               32518
05:47:42.368984 CON phy_init 0 [CTL.YAML]               32519
05:47:42.368984 CON phy_init 0 [CTL.YAML]               32520
05:47:42.368984 CON phy_init 0 [CTL.YAML]               32521
05:47:42.368985 CON phy_init 0 [CTL.YAML]               32522
05:47:42.368985 CON phy_init 0 [CTL.YAML]               32523
05:47:42.368985 CON phy_init 0 [CTL.YAML]               32524
05:47:42.368985 CON phy_init 0 [CTL.YAML]               32525
05:47:42.368985 CON phy_init 0 [CTL.YAML]               32526
05:47:42.368985 CON phy_init 0 [CTL.YAML]               32527
05:47:42.368985 CON phy_init 0 [CTL.YAML]       Flow list CSIRS: 
05:47:42.368985 CON phy_init 0 [CTL.YAML]               32512
05:47:42.368986 CON phy_init 0 [CTL.YAML]               32513
05:47:42.368986 CON phy_init 0 [CTL.YAML]               32514
05:47:42.368986 CON phy_init 0 [CTL.YAML]               32515
05:47:42.368986 CON phy_init 0 [CTL.YAML]               32516
05:47:42.368986 CON phy_init 0 [CTL.YAML]               32517
05:47:42.368986 CON phy_init 0 [CTL.YAML]               32518
05:47:42.368986 CON phy_init 0 [CTL.YAML]               32519
05:47:42.368986 CON phy_init 0 [CTL.YAML]               32520
05:47:42.368986 CON phy_init 0 [CTL.YAML]               32521
05:47:42.368986 CON phy_init 0 [CTL.YAML]               32522
05:47:42.368987 CON phy_init 0 [CTL.YAML]               32523
05:47:42.368987 CON phy_init 0 [CTL.YAML]               32524
05:47:42.368987 CON phy_init 0 [CTL.YAML]               32525
05:47:42.368987 CON phy_init 0 [CTL.YAML]               32526
05:47:42.368987 CON phy_init 0 [CTL.YAML]               32527
05:47:42.368987 CON phy_init 0 [CTL.YAML]       Flow list PUSCH: 
05:47:42.368987 CON phy_init 0 [CTL.YAML]               32512
05:47:42.368987 CON phy_init 0 [CTL.YAML]               32513
05:47:42.368988 CON phy_init 0 [CTL.YAML]               32514
05:47:42.368988 CON phy_init 0 [CTL.YAML]               32515
05:47:42.368988 CON phy_init 0 [CTL.YAML]               32516
05:47:42.368988 CON phy_init 0 [CTL.YAML]               32517
05:47:42.368988 CON phy_init 0 [CTL.YAML]               32518
05:47:42.368988 CON phy_init 0 [CTL.YAML]               32519
05:47:42.368988 CON phy_init 0 [CTL.YAML]               32520
05:47:42.368988 CON phy_init 0 [CTL.YAML]               32521
05:47:42.368989 CON phy_init 0 [CTL.YAML]               32522
05:47:42.368989 CON phy_init 0 [CTL.YAML]               32523
05:47:42.368989 CON phy_init 0 [CTL.YAML]               32524
05:47:42.368989 CON phy_init 0 [CTL.YAML]               32525
05:47:42.368989 CON phy_init 0 [CTL.YAML]               32526
05:47:42.368989 CON phy_init 0 [CTL.YAML]               32527
05:47:42.368989 CON phy_init 0 [CTL.YAML]       Flow list PUCCH: 
05:47:42.368989 CON phy_init 0 [CTL.YAML]               32512
05:47:42.368990 CON phy_init 0 [CTL.YAML]               32513
05:47:42.368990 CON phy_init 0 [CTL.YAML]               32514
05:47:42.368990 CON phy_init 0 [CTL.YAML]               32515
05:47:42.368990 CON phy_init 0 [CTL.YAML]               32516
05:47:42.368990 CON phy_init 0 [CTL.YAML]               32517
05:47:42.368990 CON phy_init 0 [CTL.YAML]               32518
05:47:42.368990 CON phy_init 0 [CTL.YAML]               32519
05:47:42.368990 CON phy_init 0 [CTL.YAML]               32520
05:47:42.368990 CON phy_init 0 [CTL.YAML]               32521
05:47:42.368990 CON phy_init 0 [CTL.YAML]               32522
05:47:42.368991 CON phy_init 0 [CTL.YAML]               32523
05:47:42.368991 CON phy_init 0 [CTL.YAML]               32524
05:47:42.368991 CON phy_init 0 [CTL.YAML]               32525
05:47:42.368991 CON phy_init 0 [CTL.YAML]               32526
05:47:42.368991 CON phy_init 0 [CTL.YAML]               32527
05:47:42.368991 CON phy_init 0 [CTL.YAML]       Flow list SRS: 
05:47:42.368991 CON phy_init 0 [CTL.YAML]               32576
05:47:42.368991 CON phy_init 0 [CTL.YAML]               32577
05:47:42.368992 CON phy_init 0 [CTL.YAML]               32578
05:47:42.368992 CON phy_init 0 [CTL.YAML]               32579
05:47:42.368992 CON phy_init 0 [CTL.YAML]               32580
05:47:42.368992 CON phy_init 0 [CTL.YAML]               32581
05:47:42.368992 CON phy_init 0 [CTL.YAML]               32582
05:47:42.368992 CON phy_init 0 [CTL.YAML]               32583
05:47:42.368992 CON phy_init 0 [CTL.YAML]               32584
05:47:42.368992 CON phy_init 0 [CTL.YAML]               32585
05:47:42.368992 CON phy_init 0 [CTL.YAML]               32586
05:47:42.368992 CON phy_init 0 [CTL.YAML]               32587
05:47:42.368993 CON phy_init 0 [CTL.YAML]               32588
05:47:42.368993 CON phy_init 0 [CTL.YAML]               32589
05:47:42.368993 CON phy_init 0 [CTL.YAML]               32590
05:47:42.368993 CON phy_init 0 [CTL.YAML]               32591
05:47:42.368993 CON phy_init 0 [CTL.YAML]               32592
05:47:42.368993 CON phy_init 0 [CTL.YAML]               32593
05:47:42.368993 CON phy_init 0 [CTL.YAML]               32594
05:47:42.368993 CON phy_init 0 [CTL.YAML]               32595
05:47:42.368993 CON phy_init 0 [CTL.YAML]               32596
05:47:42.368993 CON phy_init 0 [CTL.YAML]               32597
05:47:42.368994 CON phy_init 0 [CTL.YAML]               32598
05:47:42.368994 CON phy_init 0 [CTL.YAML]               32599
05:47:42.368994 CON phy_init 0 [CTL.YAML]               32600
05:47:42.368994 CON phy_init 0 [CTL.YAML]               32601
05:47:42.368994 CON phy_init 0 [CTL.YAML]               32602
05:47:42.368994 CON phy_init 0 [CTL.YAML]               32603
05:47:42.368994 CON phy_init 0 [CTL.YAML]               32604
05:47:42.368994 CON phy_init 0 [CTL.YAML]               32605
05:47:42.368994 CON phy_init 0 [CTL.YAML]               32606
05:47:42.368994 CON phy_init 0 [CTL.YAML]               32607
05:47:42.368994 CON phy_init 0 [CTL.YAML]               32608
05:47:42.368995 CON phy_init 0 [CTL.YAML]               32609
05:47:42.368995 CON phy_init 0 [CTL.YAML]               32610
05:47:42.368995 CON phy_init 0 [CTL.YAML]               32611
05:47:42.368995 CON phy_init 0 [CTL.YAML]               32612
05:47:42.368995 CON phy_init 0 [CTL.YAML]               32613
05:47:42.368995 CON phy_init 0 [CTL.YAML]               32614
05:47:42.368995 CON phy_init 0 [CTL.YAML]               32615
05:47:42.368995 CON phy_init 0 [CTL.YAML]               32616
05:47:42.368995 CON phy_init 0 [CTL.YAML]               32617
05:47:42.368995 CON phy_init 0 [CTL.YAML]               32618
05:47:42.368996 CON phy_init 0 [CTL.YAML]               32619
05:47:42.368996 CON phy_init 0 [CTL.YAML]               32620
05:47:42.368996 CON phy_init 0 [CTL.YAML]               32621
05:47:42.368996 CON phy_init 0 [CTL.YAML]               32622
05:47:42.368996 CON phy_init 0 [CTL.YAML]               32623
05:47:42.368996 CON phy_init 0 [CTL.YAML]               32624
05:47:42.368996 CON phy_init 0 [CTL.YAML]               32625
05:47:42.368996 CON phy_init 0 [CTL.YAML]               32626
05:47:42.368996 CON phy_init 0 [CTL.YAML]               32627
05:47:42.368996 CON phy_init 0 [CTL.YAML]               32628
05:47:42.368997 CON phy_init 0 [CTL.YAML]               32629
05:47:42.368997 CON phy_init 0 [CTL.YAML]               32630
05:47:42.368997 CON phy_init 0 [CTL.YAML]               32631
05:47:42.368997 CON phy_init 0 [CTL.YAML]               32632
05:47:42.368997 CON phy_init 0 [CTL.YAML]               32633
05:47:42.368997 CON phy_init 0 [CTL.YAML]               32634
05:47:42.368997 CON phy_init 0 [CTL.YAML]               32635
05:47:42.368997 CON phy_init 0 [CTL.YAML]               32636
05:47:42.368997 CON phy_init 0 [CTL.YAML]               32637
05:47:42.368998 CON phy_init 0 [CTL.YAML]               32638
05:47:42.368998 CON phy_init 0 [CTL.YAML]               32639
05:47:42.368998 CON phy_init 0 [CTL.YAML]       Flow list PRACH: 
05:47:42.368998 CON phy_init 0 [CTL.YAML]               32544
05:47:42.368998 CON phy_init 0 [CTL.YAML]               32545
05:47:42.368998 CON phy_init 0 [CTL.YAML]               32546
05:47:42.368998 CON phy_init 0 [CTL.YAML]               32547
05:47:42.368998 CON phy_init 0 [CTL.YAML]               32548
05:47:42.368999 CON phy_init 0 [CTL.YAML]               32549
05:47:42.368999 CON phy_init 0 [CTL.YAML]               32550
05:47:42.368999 CON phy_init 0 [CTL.YAML]               32551
05:47:42.368999 CON phy_init 0 [CTL.YAML]               32552
05:47:42.368999 CON phy_init 0 [CTL.YAML]               32553
05:47:42.368999 CON phy_init 0 [CTL.YAML]               32554
05:47:42.368999 CON phy_init 0 [CTL.YAML]               32555
05:47:42.368999 CON phy_init 0 [CTL.YAML]               32556
05:47:42.368999 CON phy_init 0 [CTL.YAML]               32557
05:47:42.368999 CON phy_init 0 [CTL.YAML]               32558
05:47:42.369000 CON phy_init 0 [CTL.YAML]               32559
05:47:42.369000 CON phy_init 0 [CTL.YAML]       PUSCH TV: /opt/nvidia/cuBB/testVectors/cuPhyChEstCoeffs.h5
05:47:42.369000 CON phy_init 0 [CTL.YAML]       SRS TV: /opt/nvidia/cuBB/testVectors/cuPhyChEstCoeffs.h5
05:47:42.369000 CON phy_init 0 [CTL.YAML]       Section_3 time offset: 58369
05:47:42.369000 CON phy_init 0 [CTL.YAML]       nMaxRxAnt: 4
05:47:42.369000 CON phy_init 0 [CTL.YAML]       PUSCH PRBs Stride: 273
05:47:42.369001 CON phy_init 0 [CTL.YAML]       PRACH PRBs Stride: 12
05:47:42.369001 CON phy_init 0 [CTL.YAML]       SRS PRBs Stride: 273
05:47:42.369001 CON phy_init 0 [CTL.YAML]       PUSCH nMaxPrb: 136
05:47:42.369001 CON phy_init 0 [CTL.YAML]       PUSCH nMaxRx: 4
05:47:42.369001 CON phy_init 0 [CTL.YAML]       UL Gain Calibration: 48.68
05:47:42.369001 CON phy_init 0 [CTL.YAML]       Lower guard bw: 845
05:47:42.369007 CON phy_init 0 [CTL.SCF] Updated CUDA_DEVICE_MAX_CONNECTIONS to 32 for green contexts
05:47:43.065831 CON phy_init 0 [CTL.SCF] Network interface for PCIe address 0000:38:00.0 : aerial00
05:47:43.065892 CON phy_init 0 [APP.UTILS]           PHC clock: 1769665663.65887764
05:47:43.065925 CON phy_init 0 [APP.UTILS]           CLOCK_TAI: 1769665663.65917523
05:47:43.065926 CON phy_init 0 [APP.UTILS]      CLOCK_REALTIME: 1769665663.65917476
05:47:43.065927 CON phy_init 0 [APP.UTILS] TAI/REALTIME offset: 0 seconds
05:47:43.071549 WRN phy_init 0 [DRV.WORKER] Unable to set pmu_type=3 for non Grace system.  Disabling pmu_metrics.
05:47:43.071561 WRN phy_init 0 [DRV.WORKER] Unable to set pmu_type=3 for non Grace system.  Disabling pmu_metrics.
05:47:43.071562 WRN phy_init 0 [DRV.WORKER] Unable to set pmu_type=3 for non Grace system.  Disabling pmu_metrics.
05:47:43.071565 WRN phy_init 0 [DRV.WORKER] Unable to set pmu_type=3 for non Grace system.  Disabling pmu_metrics.
05:47:43.071565 WRN phy_init 0 [DRV.WORKER] Unable to set pmu_type=3 for non Grace system.  Disabling pmu_metrics.
05:47:43.071565 WRN phy_init 0 [DRV.WORKER] Unable to set pmu_type=3 for non Grace system.  Disabling pmu_metrics.
05:47:43.071566 WRN phy_init 0 [DRV.WORKER] Unable to set pmu_type=3 for non Grace system.  Disabling pmu_metrics.
05:47:43.072993 CON phy_init 0 [DRV.CTX] CUDA_DEVICE_MAX_CONNECTIONS 32
05:47:43.072994 CON phy_init 0 [DRV.CTX] use_green_contexts 1
05:47:43.073007 CON phy_init 0 [DRV.CTX] Initial GPU resources retrieved via cuDeviceGetDevResource() have type 1 and SM count 58.
05:47:43.073402 CON phy_init 0 [DRV.CTX] PUSCH green context with SM count of 32.
05:47:43.073993 CON phy_init 0 [DRV.CTX] PUCCH green context with SM count of 8.
05:47:43.074264 CON phy_init 0 [DRV.CTX] PRACH green context with SM count of 10.
05:47:43.074546 CON phy_init 0 [DRV.CTX] PDSCH green context with SM count of 40.
05:47:43.074820 CON phy_init 0 [DRV.CTX] DL Ctrl green context with SM count of 8.
05:47:43.075368 CON phy_init 0 [DRV.CTX] UL order green context with SM count of 6.
05:47:43.075643 CON phy_init 0 [DRV.CTX] GPU comm green context with SM count of 16.
EAL: Detected CPU lcores: 64
EAL: Detected NUMA nodes: 2
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/cuphycontroller/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: 64 hugepages of size 2097152 reserved, but no mounted hugetlbfs found for that size
EAL: VFIO support initialized
EAL: Probe PCI driver: gpu_cuda (10de:27b8) device: 0000:27:00.0 (socket 0)
EAL: Probe PCI driver: mlx5_pci (15b3:101d) device: 0000:38:00.0 (socket 0)
05:47:43.342517 CON phy_init 0 [DRV.FH] Dynamic beam id start: 16528, end: 32767, num of beam ids per slots: 2192, number of consecutive slots with unique of beam ids: 7 
05:47:44.004308 CON phy_init 0 [FH.DOCA] doca_gpu_dmabuf_fd returned 0 (DOCA_SUCCESS is 0)
05:47:44.004319 CON phy_init 0 [FH.DOCA] Mapping transmit queue buffer (0x7fd0be000000 size 4294967296B dmabuf fd 79) with dmabuf mode
PCAP_LOGGER Initialized
mlx5_common: Fail to create MR for address (0x7fe218000000)
mlx5_common: Device 0000:38:00.0 unable to DMA map
05:47:45.391021 ERR phy_init 0 [AERIAL_ORAN_FH_EVENT] [FH.LIB] Exception! Failed to DMA map addr 0x7fe218000000 for NIC 0000:38:00.0: Operation not permitted
05:47:45.391050 ERR phy_init 0 [AERIAL_ORAN_FH_EVENT] [DRV.FH] aerial_fh::register_memory error
05:47:45.391051 ERR phy_init 0 [AERIAL_CUPHYDRV_API_EVENT] [DRV.CTX] registerMem error
gRPC Server listening on 0.0.0.0:50051
05:47:45.600693 CON phy_init 0 [CTL.SCF] ====> PhyDriver initialized!
05:47:45.601195 CON phy_init 0 [NVIPC:YAML] nv_ipc_parse_yaml_node: PCAP msg_filter: enable all messages: 0x00-0xFF
05:47:45.601227 CON phy_init 0 [NVIPC:YAML] nv_ipc_parse_yaml_node: PCAP cell_filter: enable all cells 0-255
05:47:45.601640 CON phy_init 0 [NVIPC.DEBUG] nv_ipc_debug_open: prefix=nvipc fapi_type=1 nvipc_version=25.3.0 OK
05:47:45.651619 CON 17277 0 [NVIPC.DEBUG] pcap_shm_caching_thread_func: thread [nvipc_pcap_shm] started on CPU core [10] ...
05:47:45.680748 CON phy_init 0 [NVIPC.SHM] nvipc server initialized
05:47:45.680837 CON phy_init 0 [NVIPC.SHM] shm_ipc_open: forward_enable=0 fw_max_msg_buf_count=1365 fw_max_data_buf_count=341
05:47:45.680840 CON phy_init 0 [NVIPC.SHM] create_shm_nv_ipc_interface: OK. buf_size: cpu_msg=15000 cpu_data=576000 cpu_large=4096000
05:47:45.680852 CON phy_init 0 [L2A.TRANSPORT] phy_mac_transport[0] created. phy_cell_map.size=0 mac_cell_map.size=0 mapped=false cell_num=1
05:47:45.680858 CON phy_init 0 [L2A.TRANSPORT] init: loaded 1 nvipc instance, total_cell_num=1
05:47:45.680994 CON phy_init 0 [L2A.MODULE] PHY_module: transport_num=1 total_cell_num=1
05:47:45.712248 CON 17278 0 [NVIPC.EFD] [nvipc][core 00 ] share event_fd thread info: sched_policy=0 sched_priority=0
05:47:45.712292 CON 17278 0 [NVIPC.EFD] [nvipc] Listening for nvipc client to connect ...
05:47:46.605202 CON UlPhyDriver03 0 [DRV.WORKER] Thread UlPhyDriver03(worker_default) on CPU 3 initialized fmtlog
05:47:46.605245 CON UlPhyDriver02 0 [DRV.WORKER] Thread UlPhyDriver02(worker_default) on CPU 2 initialized fmtlog
05:47:46.605275 CON DlPhyDriver06 0 [DRV.WORKER] Thread DlPhyDriver06(worker_default) on CPU 6 initialized fmtlog
05:47:46.605286 CON DlPhyDriver07 0 [DRV.WORKER] Thread DlPhyDriver07(worker_default) on CPU 7 initialized fmtlog
05:47:46.605358 CON DlPhyDriver05 0 [DRV.WORKER] Thread DlPhyDriver05(worker_default) on CPU 5 initialized fmtlog
05:47:46.605389 CON DlPhyDriver04 0 [DRV.WORKER] Thread DlPhyDriver04(worker_default) on CPU 4 initialized fmtlog
05:47:46.605420 CON DebugWorker09 0 [DRV.WORKER] Thread DebugWorker09(worker_default) on CPU 9 initialized fmtlog
Lookup table populated with 13104 entries
05:47:49.654924 CON phy_main 0 [CTL.SCF] cuPHYController configured for 1 cells
05:47:49.654926 CON phy_main 0 [CTL.SCF] ====> cuPHYController initialized, L1 is ready!
05:47:49.654941 CON phy_main 0 [CTL.STARTUP_TIMES] {TI PERCENTAGE} <cuphycontroller main> Start Main:0.1,Parse Cuphycontroller YAML:0.0,Init nvlog:0.4,Cuda Set Device:5.8,Cuphy PTI Init:3.7,Init PHYDriver:34.6,Init cuphydriver:0.0,Make PHYDriverProxy:0.0,Init SCF FAPI:0.0,Create PHY_group:55.3,Start PHY_group:0.0, (total: 7325830.9us),

05:47:49.654947 CON phy_main 0 [CTL.STARTUP_TIMES] {TI DURATION} <cuphycontroller main> Start Main:6971.126,Parse Cuphycontroller YAML:2551.700,Init nvlog:30389.284,Cuda Set Device:427809.241,Cuphy PTI Init:269112.276,Init PHYDriver:2534766.365,Init cuphydriver:0.365,Make PHYDriverProxy:23.665,Init SCF FAPI:5.072,Create PHY_group:4053933.687,Start PHY_group:268.080,

05:47:49.654950 CON phy_main 0 [CTL.STARTUP_TIMES] {TI TIMESTAMPS} <cuphycontroller main> Start Main:1769665662329095886,Parse Cuphycontroller YAML:1769665662336067012,Init nvlog:1769665662338618712,Cuda Set Device:1769665662369007996,Cuphy PTI Init:1769665662796817237,Init PHYDriver:1769665663065929513,Init cuphydriver:1769665665600695878,Make PHYDriverProxy:1769665665600696243,Init SCF FAPI:1769665665600719908,Create PHY_group:1769665665600724980,Start PHY_group:1769665669654658667,End Main:1769665669654926747,

05:47:49.654972 CON 17280 0 [L2A.MODULE] cell_update_thread_func: OAM thread affinity set to cpu core 0
05:47:49.662663 CON msg_processing 0 [L2A.MODULE] Thread thread_func on CPU 8 initialized fmtlog
05:48:09.653290 CON 17278 0 [NVIPC.EFD] [nvipc] nvipc unix socket server connected
05:48:09.653324 CON 17278 0 [NVIPC.EFD] [nvipc] Received peer event_fd: 86
05:48:09.853397 CON 17278 0 [NVIPC.EFD] Share event_fd succeed: efd_tx=86, efd_rx=94
05:48:09.853398 CON 17278 0 [NVIPC.EFD] [nvipc] Listening for nvipc client to connect ...
05:48:10.685929 CON msg_processing 0 [SCF.PHY] on_config_request: CONFIG.req received for cell_id=0 numTLVs=69 state=0
05:48:10.686183 CON msg_processing 0 [SCF.PHY] update_phy_stat_configs_l1: PHY Cell Id = 41, M-Plane Id= 1
05:48:10.702046 CON msg_processing 0 [DRV.PUSCH] tvStatPrms: PUSCH enableDeviceGraphLaunch=1 enableCsiP2Fapiv3 = 1 nMaxLdpcHetConfigs = 32
05:48:10.702049 CON msg_processing 0 [DRV.PUSCH] Timeout values for wait kernels are 3000us and 3000us
05:48:10.702051 CON msg_processing 0 [DRV.PUSCH] static_params.nMaxPrb: 136
05:48:10.702051 CON msg_processing 0 [DRV.PUSCH] static_params.nMaxRx: 4
05:48:10.711661 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 49.593 MiB for cuPHY PUSCH channel object (0x2cc01440000).
05:48:10.711678 CON msg_processing 0 [CUPHY.PUSCH_RX] PuschRx: Running with eqCoeffAlgo 3
05:48:10.724417 CON msg_processing 0 [DRV.PUSCH] tvStatPrms: PUSCH enableDeviceGraphLaunch=1 enableCsiP2Fapiv3 = 1 nMaxLdpcHetConfigs = 32
05:48:10.724418 CON msg_processing 0 [DRV.PUSCH] Timeout values for wait kernels are 3000us and 3000us
05:48:10.724419 CON msg_processing 0 [DRV.PUSCH] static_params.nMaxPrb: 136
05:48:10.724419 CON msg_processing 0 [DRV.PUSCH] static_params.nMaxRx: 4
05:48:10.731137 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 49.593 MiB for cuPHY PUSCH channel object (0x2cc0144c000).
05:48:10.731142 CON msg_processing 0 [CUPHY.PUSCH_RX] PuschRx: Running with eqCoeffAlgo 3
05:48:10.743761 CON msg_processing 0 [DRV.PUSCH] tvStatPrms: PUSCH enableDeviceGraphLaunch=1 enableCsiP2Fapiv3 = 1 nMaxLdpcHetConfigs = 32
05:48:10.743762 CON msg_processing 0 [DRV.PUSCH] Timeout values for wait kernels are 3000us and 3000us
05:48:10.743762 CON msg_processing 0 [DRV.PUSCH] static_params.nMaxPrb: 136
05:48:10.743762 CON msg_processing 0 [DRV.PUSCH] static_params.nMaxRx: 4
05:48:10.750828 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 49.593 MiB for cuPHY PUSCH channel object (0x2cc01458000).
05:48:10.750833 CON msg_processing 0 [CUPHY.PUSCH_RX] PuschRx: Running with eqCoeffAlgo 3
05:48:10.763433 CON msg_processing 0 [DRV.PUSCH] tvStatPrms: PUSCH enableDeviceGraphLaunch=1 enableCsiP2Fapiv3 = 1 nMaxLdpcHetConfigs = 32
05:48:10.763433 CON msg_processing 0 [DRV.PUSCH] Timeout values for wait kernels are 3000us and 3000us
05:48:10.763434 CON msg_processing 0 [DRV.PUSCH] static_params.nMaxPrb: 136
05:48:10.763434 CON msg_processing 0 [DRV.PUSCH] static_params.nMaxRx: 4
05:48:10.769913 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 49.593 MiB for cuPHY PUSCH channel object (0x2cc01464000).
05:48:10.769918 CON msg_processing 0 [CUPHY.PUSCH_RX] PuschRx: Running with eqCoeffAlgo 3
05:48:10.770601 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 3.344 MiB for cuPHY PUCCH channel object (0x2cc01399a00).
05:48:10.771119 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 3.344 MiB for cuPHY PUCCH channel object (0x2cc0139a800).
05:48:10.771542 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 3.344 MiB for cuPHY PUCCH channel object (0x2cc0139b600).
05:48:10.771909 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 3.344 MiB for cuPHY PUCCH channel object (0x2cc0139c400).
05:48:10.772763 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.182 MiB for cuPHY PRACH channel object (0x2cc02b6a340).
05:48:10.773501 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.182 MiB for cuPHY PRACH channel object (0x2cc02b6a6c0).
05:48:10.776756 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 98.130 MiB for cuPHY PDSCH channel object (0x2cc02952400).
05:48:10.779776 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 98.130 MiB for cuPHY PDSCH channel object (0x2cc02953c00).
05:48:10.782786 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 98.130 MiB for cuPHY PDSCH channel object (0x2cc02955400).
05:48:10.785686 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 98.130 MiB for cuPHY PDSCH channel object (0x2cc02956c00).
05:48:10.788794 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 98.130 MiB for cuPHY PDSCH channel object (0x2cc02958400).
05:48:10.792174 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 98.130 MiB for cuPHY PDSCH channel object (0x2cc02959c00).
05:48:10.795376 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 98.130 MiB for cuPHY PDSCH channel object (0x2cc0295b400).
05:48:10.799154 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 98.130 MiB for cuPHY PDSCH channel object (0x2cc0295cc00).
05:48:10.802541 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 98.130 MiB for cuPHY PDSCH channel object (0x2cc0295e400).
05:48:10.805893 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 98.130 MiB for cuPHY PDSCH channel object (0x2cc08960000).
05:48:10.805982 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.001 MiB for cuPHY SSB channel object (0x2cc02bb3c00).
05:48:10.806023 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.001 MiB for cuPHY SSB channel object (0x2cc02bb5100).
05:48:10.806060 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.001 MiB for cuPHY SSB channel object (0x2cc02bb4d80).
05:48:10.806114 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.001 MiB for cuPHY SSB channel object (0x2cc02bb5480).
05:48:10.806155 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.001 MiB for cuPHY SSB channel object (0x2cc02bb5800).
05:48:10.806192 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.001 MiB for cuPHY SSB channel object (0x2cc02bb5b80).
05:48:10.806236 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.001 MiB for cuPHY SSB channel object (0x2cc02bb5f00).
05:48:10.806271 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.001 MiB for cuPHY SSB channel object (0x2cc02bb6280).
05:48:10.806308 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.001 MiB for cuPHY SSB channel object (0x2cc02bb6600).
05:48:10.806349 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.001 MiB for cuPHY SSB channel object (0x2cc02bb6980).
05:48:10.806425 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 1.607 MiB for cuPHY PDCCH channel object (0x2cc02d2c800).
05:48:10.806573 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 1.607 MiB for cuPHY PDCCH channel object (0x2cc02d2cd00).
05:48:10.806661 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 1.607 MiB for cuPHY PDCCH channel object (0x2cc02d2d200).
05:48:10.806782 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 1.607 MiB for cuPHY PDCCH channel object (0x2cc02d2d700).
05:48:10.806899 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 1.607 MiB for cuPHY PDCCH channel object (0x2cc02d2dc00).
05:48:10.807130 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 1.607 MiB for cuPHY PDCCH channel object (0x2cc02d2e100).
05:48:10.807245 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 1.607 MiB for cuPHY PDCCH channel object (0x2cc02d2e600).
05:48:10.807357 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 1.607 MiB for cuPHY PDCCH channel object (0x2cc02d2eb00).
05:48:10.807471 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 1.607 MiB for cuPHY PDCCH channel object (0x2cc02d2f000).
05:48:10.807616 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 1.607 MiB for cuPHY PDCCH channel object (0x2cc02d2f500).
05:48:10.807714 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.093 MiB for cuPHY CSIRS channel object (0x2cc0815e480).
05:48:10.807749 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.093 MiB for cuPHY CSIRS channel object (0x2cc0815e780).
05:48:10.807825 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.093 MiB for cuPHY CSIRS channel object (0x2cc0815ea80).
05:48:10.807857 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.093 MiB for cuPHY CSIRS channel object (0x2cc0815f080).
05:48:10.807888 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.093 MiB for cuPHY CSIRS channel object (0x2cc0815f380).
05:48:10.807918 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.093 MiB for cuPHY CSIRS channel object (0x2cc0815f680).
05:48:10.807945 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.093 MiB for cuPHY CSIRS channel object (0x2cc0815f980).
05:48:10.807988 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.093 MiB for cuPHY CSIRS channel object (0x2cc0815fc80).
05:48:10.808021 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.093 MiB for cuPHY CSIRS channel object (0x2cc08e80080).
05:48:10.808048 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.093 MiB for cuPHY CSIRS channel object (0x2cc08e80380).
05:48:10.808061 CON msg_processing 0 [DRV.API] Update cell: mplane_id=1 dl_grid_sz=273 
05:48:10.808062 CON msg_processing 0 [DRV.API] Update cell: mplane_id=1 ul_grid_sz=273 
05:48:10.808077 CON msg_processing 0 [SCF.PHY] on_config_request: create_cell_configs for cell_id=0 phy_cell_id=41 returned error_code=0
05:48:10.808093 CON msg_processing 0 [SCF.PHY] send_cell_config_response: Send CONFIG.response: cell_id=0 error_code=0x0
05:48:10.808128 CON msg_processing 0 [L2A.TRANSPORT] wrapper: configured_cells_mask=0x1 expected_mask=0x1 all_cells_configured=true - all cells configured
05:48:10.808156 CON msg_processing 0 [SCF.PHY] on_cell_start_request: Cell 0 Received START.request, sending START.response... phy_cell_id=41
05:48:10.808319 CON msg_processing 0 [L2A.TRANSPORT] wrapper: configured_cells_mask=0x1 expected_mask=0x1 all_cells_configured=true - all cells configured
05:48:10.816657 CON timer_thread 0 [L2A.TICK] Thread slot_indication_thread_sleep_method initialized fmtlog
05:48:10.816663 CON timer_thread 0 [L2A.TICK] PTP Configs: gps_alpha: 0 gps_beta: 0
05:48:10.816670 CON timer_thread 0 [L2A.TICK] Start time: tick=1769665690816659508, seconds since epoch = 1769665690, nanoseconds = 816659508
05:48:10.816672 CON timer_thread 0 [L2A.TICK] FIRST tick scheduled for: tick=1769665693240000000, seconds since epoch = 1769665693, nanoseconds = 240000000
05:48:13.240042 CON timer_thread 0 [SCF.PHY] Cell  0 | DL    0.00 Mbps    0 Slots | UL    0.00 Mbps    0 Slots CRC   0 (     0) | Tick 0
05:48:13.245416 ERR UlPhyDriver03 0 [AERIAL_CUPHYDRV_API_EVENT] [DRV.FUNC_UL] ERROR: SFN 0.5 AGGR 3 task waiting for order kernel for more than 2000000 ns for Slot Map 1. Attempting PUSCH pipeline termination
05:48:13.246111 ERR msg_processing 0 [AERIAL_CUPHYDRV_API_EVENT] [DRV.API] SFN 0.15 No available UL input buffers st3 for cell 41
05:48:13.246118 WRN msg_processing 0 [SCF.PHY] send_phy_l1_enqueue_error_indication: L1 enqueue error SFN 0.15 cell_id=0
05:48:13.246120 ERR msg_processing 0 [AERIAL_NVIPC_API_EVENT] [SCF.PHY] Send Err.ind for SFN 0.15 cell_id=0 msg_id=0x81 err_code=0x33
05:48:13.246573 CON UlPhyDriver03 0 [DRV.FUNC_UL] SFN 0.5 AGGR 3 task PUSCH order kernel timeout waiting finished
05:48:13.246994 WRN UlPhyDriver02 0 [DRV.FH] sendCPlane_timingCheck : sendCPlane Timing error for ULC start_tx_time 1769665693246465000 current_time 1769665693246994292
05:48:13.246994 WRN UlPhyDriver02 0 [DRV.FH] sendCPlane_timingCheck: Error at oframe_id_ 0 osfid_ 7 oslotid_ 0
05:48:13.246994 ERR UlPhyDriver02 0 [AERIAL_CUPHYDRV_API_EVENT] [DRV.FUNC_UL] UL C-plane send error for cell index 0,error type 2 Map 2 Abort UL Tasks!

05:48:13.246996 ERR UlPhyDriver02 0 [AERIAL_CUPHYDRV_API_EVENT] [DRV.FUNC_UL] Calling ul_tx_error_fn 1491

05:48:13.246998 ERR UlPhyDriver02 0 [AERIAL_NVIPC_API_EVENT] [SCF.PHY] Send Err.ind for SFN 0.14 cell_id=0 msg_id=0x81 err_code=0x37
05:48:13.247006 ERR UlPhyDriver02 0 [AERIAL_CUPHYDRV_API_EVENT] [DRV.FUNC_UL] Task ul_aggr_1_cplane 0 aborted the tasklist for an error
05:48:13.247011 WRN UlPhyDriver02 0 [DRV.FUNC_UL] task_work_function_ul_aggr_1_orderKernel Task aborted for Slot Map 2
05:48:13.247012 WRN UlPhyDriver02 0 [DRV.FUNC_UL] task_work_function_ul_aggr_1_pucch_pusch Task aborted for Slot Map 2
05:48:13.247912 WRN UlPhyDriver03 0 [DRV.FUNC_UL] task_work_function_ul_aggr_3 Task aborted for Slot Map 2
05:48:13.248400 WRN UlPhyDriver02 0 [DRV.FUNC_UL] task_work_function_ul_aggr_3_early_uci_ind Task aborted for Slot Map 2
05:48:14.240009 CON timer_thread 0 [SCF.PHY] Cell  0 | DL  558.75 Mbps 1599 Slots | UL   79.35 Mbps  398 Slots CRC   1 (     1) | Tick 2000
05:48:15.240009 CON timer_thread 0 [SCF.PHY] Cell  0 | DL  558.90 Mbps 1600 Slots | UL   79.91 Mbps  400 Slots CRC   0 (     1) | Tick 4000
05:48:16.240009 CON timer_thread 0 [SCF.PHY] Cell  0 | DL  558.90 Mbps 1600 Slots | UL   79.91 Mbps  400 Slots CRC   0 (     1) | Tick 6000
05:48:16.523673 ERR DebugWorker09 0 [AERIAL_CUPHYDRV_API_EVENT] [DRV.DLBUF] ERROR compression: DLBuffer Object Cell 41 waiting for compression start for more than 1500000 ns
05:48:16.523673 ERR DebugWorker09 0 [AERIAL_CUPHYDRV_API_EVENT] [DRV.FUNC_DL] Task DL 1 Map 129 got COMPRESSION DL wait start error
05:48:17.240008 CON timer_thread 0 [SCF.PHY] Cell  0 | DL  558.90 Mbps 1600 Slots | UL   79.91 Mbps  400 Slots CRC   0 (     1) | Tick 8000
05:48:18.240009 CON timer_thread 0 [SCF.PHY] Cell  0 | DL  558.90 Mbps 1600 Slots | UL   79.91 Mbps  400 Slots CRC   0 (     1) | Tick 10000
05:48:19.240008 CON timer_thread 0 [SCF.PHY] Cell  0 | DL  558.90 Mbps 1600 Slots | UL   79.74 Mbps  399 Slots CRC   0 (     1) | Tick 12000
05:48:20.240009 CON timer_thread 0 [SCF.PHY] Cell  0 | DL  558.90 Mbps 1600 Slots | UL   80.07 Mbps  401 Slots CRC   0 (     1) | Tick 14000
3839 Cell 0 Order kernel sem_idx_rx 2808 last_sem_idx_rx 2669 sem_idx_order 2808 last_sem_idx_order 2669 PUSCH PRBs 0/26688 PRACH PRBs 0/0.  First packet received timeout after 4000000 ns F2S7S0 done = 0 current_time=1769665705428621312,last_timeout_log_time=0,total_rx_pkts=748
05:48:20.949915 ERR UlPhyDriver02 0 [AERIAL_CUPHYDRV_API_EVENT] [DRV.FUNC_UL] ERROR: SFN 770.14 AGGR 3 task waiting for order kernel for more than 2000000 ns for Slot Map 10. Attempting PUSCH pipeline termination
05:48:20.950412 ERR UlPhyDriver03 0 [AERIAL_CUPHYDRV_API_EVENT] [DRV.FUNC_UL] ERROR: SFN 770.15 AGGR 3 task waiting for order kernel for more than 2000000 ns for Slot Map 11. Attempting PUSCH pipeline termination
05:48:20.951106 ERR msg_processing 0 [AERIAL_CUPHYDRV_API_EVENT] [DRV.API] SFN 771.5 No available UL input buffers st3 for cell 41
05:48:20.951110 WRN msg_processing 0 [SCF.PHY] send_phy_l1_enqueue_error_indication: L1 enqueue error SFN 771.5 cell_id=0
05:48:20.951112 ERR msg_processing 0 [AERIAL_NVIPC_API_EVENT] [SCF.PHY] Send Err.ind for SFN 771.5 cell_id=0 msg_id=0x81 err_code=0x33
05:48:20.951349 ERR UlPhyDriver02 0 [AERIAL_CUPHY_API_EVENT] [DRV.FUNC_UL] SFN 770.14 Slot Map 10 Order kernel timeout error (exit condition 3) for cell index 0 Dyn index 0! Attempting PUSCH pipeline termination
05:48:20.951356 CON UlPhyDriver02 0 [DRV.FUNC_UL] SFN 770.14 AGGR 3 task PUSCH order kernel timeout waiting finished
05:48:20.951504 WRN UlPhyDriver02 0 [DRV.FH] sendCPlane_timingCheck : sendCPlane Timing error for ULC start_tx_time 1769665700951465000 current_time 1769665700951504723
05:48:20.951505 WRN UlPhyDriver02 0 [DRV.FH] sendCPlane_timingCheck: Error at oframe_id_ 3 osfid_ 2 oslotid_ 0
05:48:20.951505 ERR UlPhyDriver02 0 [AERIAL_CUPHYDRV_API_EVENT] [DRV.FUNC_UL] UL C-plane send error for cell index 0,error type 2 Map 12 Abort UL Tasks!

05:48:20.951506 ERR UlPhyDriver02 0 [AERIAL_CUPHYDRV_API_EVENT] [DRV.FUNC_UL] Calling ul_tx_error_fn 1491

05:48:20.951508 ERR UlPhyDriver02 0 [AERIAL_NVIPC_API_EVENT] [SCF.PHY] Send Err.ind for SFN 771.4 cell_id=0 msg_id=0x81 err_code=0x37
05:48:20.951512 ERR UlPhyDriver02 0 [AERIAL_CUPHYDRV_API_EVENT] [DRV.FUNC_UL] Task ul_aggr_1_cplane 0 aborted the tasklist for an error
05:48:20.951516 WRN UlPhyDriver02 0 [DRV.FUNC_UL] task_work_function_ul_aggr_1_orderKernel Task aborted for Slot Map 12
05:48:20.951517 WRN UlPhyDriver02 0 [DRV.FUNC_UL] task_work_function_ul_aggr_1_pucch_pusch Task aborted for Slot Map 12
05:48:20.952416 CON UlPhyDriver03 0 [DRV.FUNC_UL] SFN 770.15 AGGR 3 task PUSCH order kernel timeout waiting finished
05:48:20.952912 WRN UlPhyDriver03 0 [DRV.FUNC_UL] task_work_function_ul_aggr_3 Task aborted for Slot Map 12
05:48:20.953403 WRN UlPhyDriver02 0 [DRV.FUNC_UL] task_work_function_ul_aggr_3_early_uci_ind Task aborted for Slot Map 12
05:48:21.240008 CON timer_thread 0 [SCF.PHY] Cell  0 | DL  558.90 Mbps 1600 Slots | UL   79.11 Mbps  398 Slots CRC   2 (     3) | Tick 16000
05:48:22.240009 CON timer_thread 0 [SCF.PHY] Cell  0 | DL  558.90 Mbps 1600 Slots | UL   79.91 Mbps  400 Slots CRC   0 (     3) | Tick 18000
05:48:23.240008 CON timer_thread 0 [SCF.PHY] Cell  0 | DL  558.90 Mbps 1600 Slots | UL   79.91 Mbps  400 Slots CRC   0 (     3) | Tick 20000
05:48:24.240009 CON timer_thread 0 [SCF.PHY] Cell  0 | DL  558.90 Mbps 1600 Slots | UL   79.91 Mbps  400 Slots CRC   0 (     3) | Tick 22000
05:48:25.240008 CON timer_thread 0 [SCF.PHY] Cell  0 | DL  558.90 Mbps 1600 Slots | UL   79.91 Mbps  400 Slots CRC   0 (     3) | Tick 24000
05:48:26.240008 CON timer_thread 0 [SCF.PHY] Cell  0 | DL  558.90 Mbps 1600 Slots | UL   79.91 Mbps  400 Slots CRC   0 (     3) | Tick 26000
05:48:27.240008 CON timer_thread 0 [SCF.PHY] Cell  0 | DL  558.90 Mbps 1600 Slots | UL   79.91 Mbps  400 Slots CRC   0 (     3) | Tick 28000
05:48:28.240009 CON timer_thread 0 [SCF.PHY] Cell  0 | DL  558.90 Mbps 1600 Slots | UL   79.91 Mbps  400 Slots CRC   0 (     3) | Tick 30000
05:48:28.284694 CON msg_processing 0 [L2A.TICK] TTI Generator refcount = 0, stopping timer thread.
05:48:28.284986 CON msg_processing 0 [L2A.TICK] TTI Generator refcount = 0, stopped timer thread
^C05:48:34.535112 CON phy_main 0 [CTL.SCF] Signal 2 received, preparing to exit...
PhyDriverCtx destructor starting
flushing counters completed
05:48:34.635199 CON phy_main 0 [DRV.CTX] PhyDriverCtx destructor starting
05:48:34.635205 CON phy_main 0 [DRV.UL_PACKET_SUMMARY] slot,cell_0_early,cell_0_ontime,cell_0_late,
05:48:34.635206 CON phy_main 0 [DRV.UL_PACKET_SUMMARY] Slot 4 | 0,46527,61473 |
05:48:34.635207 CON phy_main 0 [DRV.UL_PACKET_SUMMARY] Slot 5 | 0,38888,141112 |
05:48:34.635207 CON phy_main 0 [DRV.UL_PACKET_SUMMARY] Slot 14 | 0,44392,63320 |
05:48:34.635208 CON phy_main 0 [DRV.UL_PACKET_SUMMARY] Slot 15 | 0,38846,140674 |
05:48:34.635208 CON phy_main 0 [DRV.UL_PACKET_SUMMARY] Slot 24 | 0,45321,62391 |
05:48:34.635208 CON phy_main 0 [DRV.UL_PACKET_SUMMARY] Slot 25 | 0,38587,140933 |
05:48:34.635209 CON phy_main 0 [DRV.UL_PACKET_SUMMARY] Slot 34 | 0,44916,62796 |
05:48:34.635209 CON phy_main 0 [DRV.UL_PACKET_SUMMARY] Slot 35 | 0,39425,140095 |
05:48:34.635209 CON phy_main 0 [DRV.UL_PACKET_SUMMARY] Slot 44 | 0,45643,62069 |
05:48:34.635210 CON phy_main 0 [DRV.UL_PACKET_SUMMARY] Slot 45 | 0,38577,140943 |
05:48:34.635210 CON phy_main 0 [DRV.UL_PACKET_SUMMARY] Slot 54 | 0,44650,63042 |
05:48:34.635210 CON phy_main 0 [DRV.UL_PACKET_SUMMARY] Slot 55 | 0,39132,140388 |
05:48:34.635210 CON phy_main 0 [DRV.UL_PACKET_SUMMARY] Slot 64 | 0,46042,61382 |
05:48:34.635211 CON phy_main 0 [DRV.UL_PACKET_SUMMARY] Slot 65 | 0,38141,140899 |
05:48:34.635211 CON phy_main 0 [DRV.UL_PACKET_SUMMARY] Slot 74 | 0,44343,63369 |
05:48:34.635211 CON phy_main 0 [DRV.UL_PACKET_SUMMARY] Slot 75 | 0,39129,140391 |
05:48:34.635220 CON phy_main 0 [DRV.UL_PACKET_SUMMARY] Slot 4 | 0.00,43.08,56.92 |
05:48:34.635221 CON phy_main 0 [DRV.UL_PACKET_SUMMARY] Slot 5 | 0.00,21.60,78.40 |
05:48:34.635222 CON phy_main 0 [DRV.UL_PACKET_SUMMARY] Slot 14 | 0.00,41.21,58.79 |
05:48:34.635223 CON phy_main 0 [DRV.UL_PACKET_SUMMARY] Slot 15 | 0.00,21.64,78.36 |
05:48:34.635224 CON phy_main 0 [DRV.UL_PACKET_SUMMARY] Slot 24 | 0.00,42.08,57.92 |
05:48:34.635224 CON phy_main 0 [DRV.UL_PACKET_SUMMARY] Slot 25 | 0.00,21.49,78.51 |
05:48:34.635225 CON phy_main 0 [DRV.UL_PACKET_SUMMARY] Slot 34 | 0.00,41.70,58.30 |
05:48:34.635226 CON phy_main 0 [DRV.UL_PACKET_SUMMARY] Slot 35 | 0.00,21.96,78.04 |
05:48:34.635227 CON phy_main 0 [DRV.UL_PACKET_SUMMARY] Slot 44 | 0.00,42.38,57.62 |
05:48:34.635227 CON phy_main 0 [DRV.UL_PACKET_SUMMARY] Slot 45 | 0.00,21.49,78.51 |
05:48:34.635228 CON phy_main 0 [DRV.UL_PACKET_SUMMARY] Slot 54 | 0.00,41.46,58.54 |
05:48:34.635229 CON phy_main 0 [DRV.UL_PACKET_SUMMARY] Slot 55 | 0.00,21.80,78.20 |
05:48:34.635230 CON phy_main 0 [DRV.UL_PACKET_SUMMARY] Slot 64 | 0.00,42.86,57.14 |
05:48:34.635230 CON phy_main 0 [DRV.UL_PACKET_SUMMARY] Slot 65 | 0.00,21.30,78.70 |
05:48:34.635231 CON phy_main 0 [DRV.UL_PACKET_SUMMARY] Slot 74 | 0.00,41.17,58.83 |
05:48:34.635232 CON phy_main 0 [DRV.UL_PACKET_SUMMARY] Slot 75 | 0.00,21.80,78.20 |
05:48:34.635413 CON phy_main 0 [DRV.CTX] flushing counters completed
05:48:36.835613 CON phy_main 0 [FH.NIC] NIC 0000:38:00.0 stats:
05:48:36.835618 CON phy_main 0 [FH.NIC]   tx_packets: 1460668
05:48:36.835621 CON phy_main 0 [FH.NIC]   rx_packets: 0
05:48:36.835622 CON phy_main 0 [FH.NIC]   tx_bytes: 723118784
05:48:36.835623 CON phy_main 0 [FH.NIC]   rx_bytes: 0
05:48:36.835623 CON phy_main 0 [FH.NIC]   tx_errors: 0
05:48:36.835624 CON phy_main 0 [FH.NIC]   rx_errors: 0
05:48:36.835625 CON phy_main 0 [FH.NIC]   rx_missed: 0
05:48:36.835625 CON phy_main 0 [FH.NIC]   rx_nombuf: 0
PhyDriverCtx destructor completed successfully
05:48:37.463603 CON phy_main 0 [DRV.CTX] Total CPU regular memory 81023728 B / 79124 KiB / 77 MiB
05:48:37.463605 CON phy_main 0 [DRV.CTX] Total CPU pinned memory 179525068 B / 175317 KiB / 171 MiB
05:48:37.463605 CON phy_main 0 [DRV.CTX] Total GPU regular memory 2960551744 B / 2891163 KiB / 2823 MiB
05:48:37.463605 CON phy_main 0 [DRV.CTX] Total GPU pinned memory 77755584 B / 75933 KiB / 74 MiB
05:48:37.463617 CON phy_main 0 [DRV.CTX] PhyDriverCtx destructor completed successfully
05:48:37.463759 CON phy_main 0 [CTL.SCF] ====> PhyDriver finalized! Exit from signal_handler after 1s delay...
EXIT successfully at: /opt/nvidia/cuBB/cuPHY-CP/cuphycontroller/examples/cuphycontroller_scf.cpp line 73 

Now I have two problems:

  1. Will the error “mlx5_common: Fail to create MR for address (0x7ee1d4000000)” affect the Simulator E2E test?
  2. What‘s the meaning of the setting “gpu_init_comms_via_cpu: 1” and “gpu_init_comms_via_cpu: 0”?

gpu_init_comms_via_cpu setting controls how GPU-initiated network communications work - specifically whether data goes directly from GPU to NIC (GPUDirect RDMA) or through CPU memory as an intermediary.

gpu_init_comms_via_cpu: 0 (Default - Direct GPUDirect RDMA):

GPU Memory → [PCIe P2P] → NIC → Network

  • Requires nvidia-peermem / dmabuf working

  • Requires P2P support between GPU and NIC

  • Lower latency, higher throughput

  • This is what’s failing on the L4 system

gpu_init_comms_via_cpu: 1 (CPU Fallback):

GPU Memory → [PCIe] → CPU Memory → [PCIe] → NIC → Network

  • Uses DOCA_GPU_MEM_TYPE_CPU_GPU (CPU-accessible GPU memory)

  • Does not require GPUDirect RDMA / nvidia-peermem

  • Slightly higher latency, but works on platforms without P2P support

Please also see the yaml file for Grace+L4 system (cuphycontroller_F08_GL4.yaml), which has gpu_init_comms_via_cpu: 1 .

This is the setting you should use for your system.

@kemalong Can you share the output of the following commands?

#confirm the cx6-dx device id
sudo mst status -v

# Here replace your nic device id with mt4125_pciconf0 
sudo mlxconfig -d /dev/mst/mt4125_pciconf0 q | grep "CQE_COMPRESSION\|PROG_PARSE_GRAPH\|FLEX_PARSER_PROFILE_ENABLE\|REAL_TIME_CLOCK_ENABLE\|ACCURATE_TX_SCHEDULER"

The output is as follows:


The error “mlx5_common: Fail to create MR for address (0x7ee1d4000000)” has been addressed. I should have misconfigured a certain parameter in the cuphycontroller_F08_GL4.yaml. I’m still confirming it. After I replaced the yaml file with the original one in the repository, the problem has been addressed. The new log is as follows:

root@c_aerial_root:/opt/nvidia/cuBB/testBenches# $cuBB_SDK/testBenches/phase4_test_scripts/run2_cuPHYcontroller.sh $RUN2_CUPHYCONTROLLER_PARAMS
restarting MPS
Stopping MPS...
Cannot find MPS control daemon process
Starting MPS...
/opt/nvidia/cuBB build.perf.x86_64 F08_GL4
  /opt/nvidia/cuBB/build.perf.x86_64/cuPHY-CP/cuphycontroller/examples/cuphycontroller_scf F08_GL4
F08_GL4
Started cuphycontroller on CPU core 11
08:14:31.749566 CON phy_init 0 [CTL.SCF] Config file: /opt/nvidia/cuBB/cuPHY-CP/cuphycontroller/config/cuphycontroller_F08_GL4.yaml
08:14:31.751292 CON phy_init 0 [CTL.SCF] low_priority_core=0
08:14:31.751318 CON phy_init 0 [APP.CONFIG] Current TAI offset: 0s
08:14:31.751380 CON phy_init 0 [NVLOG.CPP] Using /opt/nvidia/cuBB/cuPHY/nvlog/config/nvlog_config.yaml for nvlog configuration
08:14:31.766963 CON phy_init 0 [NVLOG.CPP] Output log file path /tmp/phy.log
Aerial metrics backend address: 127.0.0.1:60000
YAML invalid key: dlc_alloc_cplane_bfw_txq Using default value of 0 for YAML_PARAM_DLC_ALLOC_CPLANE_BFW_TXQ
YAML invalid key: ulc_alloc_cplane_bfw_txq Using default value of 0 for YAML_PARAM_ULC_ALLOC_CPLANE_BFW_TXQ
08:14:31.779855 CON phy_init 0 [CTL.YAML] cuphycontroller config. yaml does not have ul_order_timeout_gpu_srs_ns key; defaulting to 5200000 ns
08:14:31.779857 CON phy_init 0 [CTL.YAML] cuphycontroller config. yaml does not have ul_srs_aggr3_task_launch_offset_ns key; defaulting to 500000 ns
08:14:31.779869 WRN phy_init 0 [CTL.YAML] cuphycontroller config. yaml does not have ul_order_kernel_mode key; defaulting to 0 (Ping-Pong mode).
08:14:31.782816 WRN phy_init 0 [CTL.YAML] YAML invalid key: enable_l1_param_sanity_check Using default value of 0 to YAML_PARAM_ENABLE_L1_PARAM_SANITY_CHECK
08:14:31.782895 WRN phy_init 0 [CTL.YAML] YAML invalid key: pusch_weighted_average_cfo Using default value of 0 to PUSCH-WEIGHTED-AVERAGE-CFO
08:14:31.782924 WRN phy_init 0 [CTL.YAML] YAML invalid key: pucch_aggr_per_ctx Using default value of 4 to YAML_PARAM_PUCCH_AGGR_PER_CTX
08:14:31.782933 WRN phy_init 0 [CTL.YAML] YAML invalid key: srs_aggr_per_ctx Using default value of 2 to YAML_PARAM_SRS_AGGR_PER_CTX
08:14:31.782942 WRN phy_init 0 [CTL.YAML] YAML invalid key: ul_input_buffer_per_cell_srs Using default value of 4 to YAML_PARAM_UL_INPUT_BUFFER_NUM_PER_CELL_SRS
08:14:31.782951 WRN phy_init 0 [CTL.YAML] YAML invalid key: max_ru_unhealthy_ul_slots Using default value of 0 for YAML_PARAM_MAX_RU_UNHEALTHY_UL_SLOTS
08:14:31.782971 WRN phy_init 0 [CTL.YAML] YAML invalid key: bfw_beta_prescaler Using default value of 16384 for YAML_PARAM_BFW_BETA_PRESCALER 16384
08:14:31.782986 WRN phy_init 0 [CTL.YAML] YAML invalid key: ul_pcap_capture_enable Using default value of 0 for YAML_PARAM_UL_PCAP_CAPTURE_ENABLE
08:14:31.782996 WRN phy_init 0 [CTL.YAML] YAML invalid key: pcap_logger_ul_cplane_enable PCAP Logging parameters are set incorrectly, so logger is disabled by default
08:14:31.783063 WRN phy_init 0 [CTL.YAML] YAML invalid key: bfw_c_plane_chaining_mode Using default value of 0 for YAML_PARAM_BFW_C_PLANE_CHAINING_MODE
08:14:31.783074 WRN phy_init 0 [CTL.YAML] YAML invalid key: enable_tx_notification Using default value of 0 for YAML_PARAM_ENABLE_TX_NOTIFICATION
08:14:31.783223 WRN phy_init 0 [CTL.YAML] YAML invalid key: pusch_nMaxLdpcHetConfigs Using default value of 32 to PUSCH-N-MAX-LDPC-HET-CONFIGS
08:14:31.783286 CON phy_init 0 [CTL.YAML] cell_id 1 nic_index :0
08:14:31.783412 CON phy_init 0 [CTL.YAML] ta4_min_ns_srs/ta4_max_ns_srs not set in config file, using default ta4_min_ns_srs = 621 us and ta4_max_ns_srs = 1831 us
08:14:31.783420 CON phy_init 0 [CTL.YAML] t1a_min_cp_dl_ns/t1a_max_cp_dl_ns not set in config file, using default t1a_min_cp_dl_ns = 419 us and t1a_max_cp_dl_ns = 669 us
08:14:31.783426 CON phy_init 0 [CTL.YAML] t1a_min_cp_ul_ns not set in config file, using default of 285 us
08:14:31.783520 CON phy_init 0 [CTL.YAML] Num Slots: 8
08:14:31.783521 CON phy_init 0 [CTL.YAML] Enable UL cuPHY Graphs: 1
08:14:31.783522 CON phy_init 0 [CTL.YAML] Enable DL cuPHY Graphs: 1
08:14:31.783522 CON phy_init 0 [CTL.YAML] Accurate TX scheduling clock resolution (ns): 500
08:14:31.783523 CON phy_init 0 [CTL.YAML] DPDK core: 0
08:14:31.783523 CON phy_init 0 [CTL.YAML] Prometheus core: -1
08:14:31.783523 CON phy_init 0 [CTL.YAML] UL cores: 
08:14:31.783524 CON phy_init 0 [CTL.YAML]       - 2
08:14:31.783524 CON phy_init 0 [CTL.YAML]       - 3
08:14:31.783524 CON phy_init 0 [CTL.YAML] DL cores: 
08:14:31.783524 CON phy_init 0 [CTL.YAML]       - 4
08:14:31.783524 CON phy_init 0 [CTL.YAML]       - 5
08:14:31.783524 CON phy_init 0 [CTL.YAML]       - 6
08:14:31.783525 CON phy_init 0 [CTL.YAML]       - 7
08:14:31.783525 CON phy_init 0 [CTL.YAML] Debug worker: 9
08:14:31.783525 CON phy_init 0 [CTL.YAML] Data Lake core: -1
08:14:31.783525 CON phy_init 0 [CTL.YAML] SRS starting Section ID: 3072
08:14:31.783526 CON phy_init 0 [CTL.YAML] PRACH starting Section ID: 2048
08:14:31.783527 CON phy_init 0 [CTL.YAML] USE GREEN CONTEXTS: 0
08:14:31.783527 CON phy_init 0 [CTL.YAML] USE BATCHED MEMCPY: 1
08:14:31.783527 CON phy_init 0 [CTL.YAML] MPS SM PUSCH: 32
08:14:31.783527 CON phy_init 0 [CTL.YAML] MPS SM PUCCH: 8
08:14:31.783528 CON phy_init 0 [CTL.YAML] MPS SM PRACH: 4
08:14:31.783528 CON phy_init 0 [CTL.YAML] MPS SM UL ORDER: 12
08:14:31.783528 CON phy_init 0 [CTL.YAML] MPS SM PDSCH: 40
08:14:31.783528 CON phy_init 0 [CTL.YAML] MPS SM PDCCH: 4
08:14:31.783528 CON phy_init 0 [CTL.YAML] MPS SM PBCH: 2
08:14:31.783528 CON phy_init 0 [CTL.YAML] MPS SM GPU_COMMS: 16
08:14:31.783528 CON phy_init 0 [CTL.YAML] MPS SM SRS: 4
08:14:31.783529 CON phy_init 0 [CTL.YAML] UL Order Kernel Mode: 0
08:14:31.783529 CON phy_init 0 [CTL.YAML] PDSCH fallback: 0
08:14:31.783529 CON phy_init 0 [CTL.YAML] Massive MIMO enable: 0
08:14:31.783529 CON phy_init 0 [CTL.YAML] mMIMO_enable feature 0
08:14:31.783529 CON phy_init 0 [CTL.YAML] Enable SRS : 0
08:14:31.783530 CON phy_init 0 [CTL.YAML] ul_order_timeout_gpu_log_enable: 1
08:14:31.783530 CON phy_init 0 [CTL.YAML] ue_mode: 0
08:14:31.783530 CON phy_init 0 [CTL.YAML] Aggr Obj Non-availability threshold: 5
08:14:31.783530 CON phy_init 0 [CTL.YAML] sendCPlane_timing_error_th_ns: 0
08:14:31.783530 CON phy_init 0 [CTL.YAML] pusch_aggr_per_ctx: 4
08:14:31.783530 CON phy_init 0 [CTL.YAML] prach_aggr_per_ctx: 2
08:14:31.783531 CON phy_init 0 [CTL.YAML] pucch_aggr_per_ctx: 4
08:14:31.783531 CON phy_init 0 [CTL.YAML] srs_aggr_per_ctx: 2
08:14:31.783531 CON phy_init 0 [CTL.YAML] max_harq_pools: 512
08:14:31.783531 CON phy_init 0 [CTL.YAML] max_harq_tx_count_bundled: 10
08:14:31.783531 CON phy_init 0 [CTL.YAML] max_harq_tx_count_non_bundled: 4
08:14:31.783531 CON phy_init 0 [CTL.YAML] ul_input_buffer_per_cell: 10
08:14:31.783531 CON phy_init 0 [CTL.YAML] ul_input_buffer_per_cell_srs: 4
08:14:31.783532 CON phy_init 0 [CTL.YAML] max_ru_unhealthy_ul_slots: 0
08:14:31.783532 CON phy_init 0 [CTL.YAML] srs_chest_algo_type: 0
08:14:31.783532 CON phy_init 0 [CTL.YAML] srs_chest_tol2_normalization_algo_type: 1
08:14:31.783533 CON phy_init 0 [CTL.YAML] srs_chest_tol2_constant_scaler: 32768
08:14:31.783533 CON phy_init 0 [CTL.YAML] bfw_power_normalization_alg_selector: 1
08:14:31.783533 CON phy_init 0 [CTL.YAML] bfw_beta_prescaler: 16384
08:14:31.783533 CON phy_init 0 [CTL.YAML] total_num_srs_chest_buffers: 6144
08:14:31.783533 CON phy_init 0 [CTL.YAML] send_static_bfw_wt_all_cplane: 1
08:14:31.783534 CON phy_init 0 [CTL.YAML] ul_pcap_capture_enable: 0
08:14:31.783534 CON phy_init 0 [CTL.YAML] ul_pcap_capture_thread_cpu_affinity: 0
08:14:31.783534 CON phy_init 0 [CTL.YAML] ul_pcap_capture_thread_sched_priority: 0
08:14:31.783534 CON phy_init 0 [CTL.YAML] pcap_logger_ul_cplane_enable: 0
08:14:31.783534 CON phy_init 0 [CTL.YAML] pcap_logger_dl_cplane_enable: 0
08:14:31.783534 CON phy_init 0 [CTL.YAML] pcap_logger_thread_cpu_affinity: 0
08:14:31.783535 CON phy_init 0 [CTL.YAML] pcap_logger_thread_sched_prio: 0
08:14:31.783535 CON phy_init 0 [CTL.YAML] pcap_logger_file_save_dir: .
08:14:31.783535 CON phy_init 0 [CTL.YAML] static_beam_id_start: 1
08:14:31.783536 CON phy_init 0 [CTL.YAML] static_beam_id_end: 16527
08:14:31.783536 CON phy_init 0 [CTL.YAML] dynamic_beam_id_start: 16528
08:14:31.783536 CON phy_init 0 [CTL.YAML] dynamic_beam_id_end: 32767
08:14:31.783536 CON phy_init 0 [CTL.YAML] ul_order_timeout_gpu_log_enable: 1
08:14:31.783536 CON phy_init 0 [CTL.YAML] pusch_workCancelMode: 2
08:14:31.783536 CON phy_init 0 [CTL.YAML] GPU-initiated comms DL: 1
08:14:31.783537 CON phy_init 0 [CTL.YAML] GPU-initiated comms (via CPU): 1
08:14:31.783537 CON phy_init 0 [CTL.YAML] CPU-initiated comms : 0
08:14:31.783537 CON phy_init 0 [CTL.YAML] Cell group: 1
08:14:31.783537 CON phy_init 0 [CTL.YAML] Cell group num: 1
08:14:31.783537 CON phy_init 0 [CTL.YAML] puxchPolarDcdrListSz: 8
08:14:31.783537 CON phy_init 0 [CTL.YAML] split_ul_cuda_streams: 0
08:14:31.783538 CON phy_init 0 [CTL.YAML] serialize_pucch_pusch: 0
08:14:31.783538 CON phy_init 0 [CTL.YAML] bfw_c_plane_chaining_mode: 0
08:14:31.783538 CON phy_init 0 [CTL.YAML] fh_stats_dump_cpu_core: -1
08:14:31.783538 CON phy_init 0 [CTL.YAML] fix_beta_dl: 1
08:14:31.783538 CON phy_init 0 [CTL.YAML] pdump_client_thread: -1
08:14:31.783538 CON phy_init 0 [CTL.YAML] profiler_sec: 0
08:14:31.783538 CON phy_init 0 [CTL.YAML] datalake_address: localhost
08:14:31.783539 CON phy_init 0 [CTL.YAML] dpdk_file_prefix: cuphycontroller
08:14:31.783539 CON phy_init 0 [CTL.YAML] puschrxChestFactorySettingsFilename: 
08:14:31.783539 CON phy_init 0 [CTL.YAML] accu_tx_sched_disable: 0
08:14:31.783539 CON phy_init 0 [CTL.YAML] cplane_disable: 0
08:14:31.783539 CON phy_init 0 [CTL.YAML] disable_empw: 0
08:14:31.783540 CON phy_init 0 [CTL.YAML] dlc_alloc_cplane_bfw_txq: 0
08:14:31.783540 CON phy_init 0 [CTL.YAML] dlc_bfw_enable_divide_per_cell: 0
08:14:31.783540 CON phy_init 0 [CTL.YAML] dpdk_verbose_logs: 0
08:14:31.783540 CON phy_init 0 [CTL.YAML] enable_cpu_task_tracing: 1
08:14:31.783540 CON phy_init 0 [CTL.YAML] enable_dl_cqe_tracing: 0
08:14:31.783540 CON phy_init 0 [CTL.YAML] enable_l1_param_sanity_check: 0
08:14:31.783541 CON phy_init 0 [CTL.YAML] enable_ok_tb: 0
08:14:31.783541 CON phy_init 0 [CTL.YAML] enable_prepare_tracing: 1
08:14:31.783541 CON phy_init 0 [CTL.YAML] mCh_segment_proc_enable: 0
08:14:31.783541 CON phy_init 0 [CTL.YAML] pmu_metrics: 3
08:14:31.783541 CON phy_init 0 [CTL.YAML] puschCfo: 1
08:14:31.783541 CON phy_init 0 [CTL.YAML] puschDftSOfdm: 0
08:14:31.783542 CON phy_init 0 [CTL.YAML] puschEnablePerPrgChEst: 0
08:14:31.783542 CON phy_init 0 [CTL.YAML] puschRssi: 1
08:14:31.783542 CON phy_init 0 [CTL.YAML] puschSelectChEstAlgo: 1
08:14:31.783542 CON phy_init 0 [CTL.YAML] puschSelectEqCoeffAlgo: 3
08:14:31.783542 CON phy_init 0 [CTL.YAML] puschSinr: 2
08:14:31.783542 CON phy_init 0 [CTL.YAML] puschTbSizeCheck: 1
08:14:31.783542 CON phy_init 0 [CTL.YAML] puschTdi: 1
08:14:31.783543 CON phy_init 0 [CTL.YAML] puschTo: 1
08:14:31.783543 CON phy_init 0 [CTL.YAML] pusch_deviceGraphLaunchEn: 1
08:14:31.783543 CON phy_init 0 [CTL.YAML] ulc_alloc_cplane_bfw_txq: 0
08:14:31.783543 CON phy_init 0 [CTL.YAML] ulc_bfw_enable_divide_per_cell: 0
08:14:31.783543 CON phy_init 0 [CTL.YAML] ul_rx_pkt_tracing_level: 0
08:14:31.783543 CON phy_init 0 [CTL.YAML] ul_rx_pkt_tracing_level_srs: 0
08:14:31.783543 CON phy_init 0 [CTL.YAML] ul_warmup_frame_count: 8
08:14:31.783545 CON phy_init 0 [CTL.YAML] forcedNumCsi2Bits: 0
08:14:31.783545 CON phy_init 0 [CTL.YAML] pusch_waitTimeOutPostEarlyHarqUs: 3000
08:14:31.783545 CON phy_init 0 [CTL.YAML] pusch_waitTimeOutPreEarlyHarqUs: 3000
08:14:31.783546 CON phy_init 0 [CTL.YAML] validation: 0
08:14:31.783546 CON phy_init 0 [CTL.YAML] cqe_trace_slot_mask: 196800
08:14:31.783546 CON phy_init 0 [CTL.YAML] datalake_samples: 1000000
08:14:31.783546 CON phy_init 0 [CTL.YAML] num_ok_tb_slot: 0
08:14:31.783546 CON phy_init 0 [CTL.YAML] pusch_nMaxLdpcHetConfigs: 32
08:14:31.783546 CON phy_init 0 [CTL.YAML] sendCPlane_dlbfw_backoff_th_ns: 300000
08:14:31.783546 CON phy_init 0 [CTL.YAML] sendCPlane_ulbfw_backoff_th_ns: 300000
08:14:31.783547 CON phy_init 0 [CTL.YAML] ul_order_max_rx_pkts: 256
08:14:31.783547 CON phy_init 0 [CTL.YAML] ul_order_rx_pkts_timeout_ns: 50000
08:14:31.783547 CON phy_init 0 [CTL.YAML] ul_order_timeout_cpu_ns: 8000000
08:14:31.783547 CON phy_init 0 [CTL.YAML] ul_order_timeout_gpu_ns: 8000000
08:14:31.783547 CON phy_init 0 [CTL.YAML] ul_order_timeout_gpu_srs_ns: 5200000
08:14:31.783547 CON phy_init 0 [CTL.YAML] ul_order_timeout_log_interval_ns: 0
08:14:31.783547 CON phy_init 0 [CTL.YAML] ul_srs_aggr3_task_launch_offset_ns: 500000
08:14:31.783547 CON phy_init 0 [CTL.YAML] workers_sched_priority: 95
08:14:31.783548 CON phy_init 0 [CTL.YAML] cqe_trace_cell_mask: 1
08:14:31.783548 CON phy_init 0 [CTL.YAML] Number of Cell Configs: 1
08:14:31.783548 CON phy_init 0 [CTL.YAML] L2Adapter config file: /opt/nvidia/cuBB/cuPHY-CP/cuphycontroller/config/l2_adapter_config_F08_CG1.yaml
08:14:31.783549 CON phy_init 0 [CTL.YAML] Cell name: O-RU 0
08:14:31.783549 CON phy_init 0 [CTL.YAML]       MU: 1
08:14:31.783549 CON phy_init 0 [CTL.YAML]       ID: 1
08:14:31.783550 CON phy_init 0 [CTL.YAML] Number of MPlane Configs: 1
08:14:31.783550 CON phy_init 0 [CTL.YAML]       Mplane ID: 1
08:14:31.783550 CON phy_init 0 [CTL.YAML]       VLAN ID: 2
08:14:31.783551 CON phy_init 0 [CTL.YAML]       Source Eth Address: b8:ce:f6:fe:62:f4
08:14:31.783552 CON phy_init 0 [CTL.YAML]       Destination Eth Address: 20:04:9b:9e:27:a3
08:14:31.783552 CON phy_init 0 [CTL.YAML]       NIC port: 0000:38:00.0
08:14:31.783552 CON phy_init 0 [CTL.YAML]       RU Type: 3
08:14:31.783553 CON phy_init 0 [CTL.YAML]       U-plane TXQs: 1
08:14:31.783553 CON phy_init 0 [CTL.YAML]       DL compression method: 1
08:14:31.783553 CON phy_init 0 [CTL.YAML]       DL iq bit width: 9
08:14:31.783553 CON phy_init 0 [CTL.YAML]       UL compression method: 1
08:14:31.783553 CON phy_init 0 [CTL.YAML]       UL iq bit width: 9
08:14:31.783554 CON phy_init 0 [CTL.YAML] 
08:14:31.783554 CON phy_init 0 [CTL.YAML]       Flow list SSB/PBCH: 
08:14:31.783554 CON phy_init 0 [CTL.YAML]               8
08:14:31.783554 CON phy_init 0 [CTL.YAML]               0
08:14:31.783554 CON phy_init 0 [CTL.YAML]               1
08:14:31.783555 CON phy_init 0 [CTL.YAML]               2
08:14:31.783555 CON phy_init 0 [CTL.YAML]       Flow list PDCCH: 
08:14:31.783555 CON phy_init 0 [CTL.YAML]               8
08:14:31.783555 CON phy_init 0 [CTL.YAML]               0
08:14:31.783555 CON phy_init 0 [CTL.YAML]               1
08:14:31.783555 CON phy_init 0 [CTL.YAML]               2
08:14:31.783556 CON phy_init 0 [CTL.YAML]       Flow list PDSCH: 
08:14:31.783556 CON phy_init 0 [CTL.YAML]               8
08:14:31.783556 CON phy_init 0 [CTL.YAML]               0
08:14:31.783556 CON phy_init 0 [CTL.YAML]               1
08:14:31.783556 CON phy_init 0 [CTL.YAML]               2
08:14:31.783556 CON phy_init 0 [CTL.YAML]       Flow list CSIRS: 
08:14:31.783556 CON phy_init 0 [CTL.YAML]               8
08:14:31.783557 CON phy_init 0 [CTL.YAML]               0
08:14:31.783557 CON phy_init 0 [CTL.YAML]               1
08:14:31.783557 CON phy_init 0 [CTL.YAML]               2
08:14:31.783557 CON phy_init 0 [CTL.YAML]       Flow list PUSCH: 
08:14:31.783557 CON phy_init 0 [CTL.YAML]               8
08:14:31.783557 CON phy_init 0 [CTL.YAML]               0
08:14:31.783557 CON phy_init 0 [CTL.YAML]               1
08:14:31.783558 CON phy_init 0 [CTL.YAML]               2
08:14:31.783558 CON phy_init 0 [CTL.YAML]       Flow list PUCCH: 
08:14:31.783558 CON phy_init 0 [CTL.YAML]               8
08:14:31.783558 CON phy_init 0 [CTL.YAML]               0
08:14:31.783558 CON phy_init 0 [CTL.YAML]               1
08:14:31.783558 CON phy_init 0 [CTL.YAML]               2
08:14:31.783558 CON phy_init 0 [CTL.YAML]       Flow list SRS: 
08:14:31.783559 CON phy_init 0 [CTL.YAML]               8
08:14:31.783559 CON phy_init 0 [CTL.YAML]               0
08:14:31.783559 CON phy_init 0 [CTL.YAML]               1
08:14:31.783559 CON phy_init 0 [CTL.YAML]               2
08:14:31.783559 CON phy_init 0 [CTL.YAML]       Flow list PRACH: 
08:14:31.783559 CON phy_init 0 [CTL.YAML]               15
08:14:31.783560 CON phy_init 0 [CTL.YAML]               7
08:14:31.783560 CON phy_init 0 [CTL.YAML]               0
08:14:31.783560 CON phy_init 0 [CTL.YAML]               1
08:14:31.783560 CON phy_init 0 [CTL.YAML]       PUSCH TV: /opt/nvidia/cuBB/testVectors/cuPhyChEstCoeffs.h5
08:14:31.783560 CON phy_init 0 [CTL.YAML]       SRS TV: /opt/nvidia/cuBB/testVectors/cuPhyChEstCoeffs.h5
08:14:31.783560 CON phy_init 0 [CTL.YAML]       Section_3 time offset: 58369
08:14:31.783560 CON phy_init 0 [CTL.YAML]       nMaxRxAnt: 4
08:14:31.783561 CON phy_init 0 [CTL.YAML]       PUSCH PRBs Stride: 273
08:14:31.783561 CON phy_init 0 [CTL.YAML]       PRACH PRBs Stride: 12
08:14:31.783561 CON phy_init 0 [CTL.YAML]       SRS PRBs Stride: 12
08:14:31.783561 CON phy_init 0 [CTL.YAML]       PUSCH nMaxPrb: 273
08:14:31.783561 CON phy_init 0 [CTL.YAML]       PUSCH nMaxRx: 4
08:14:31.783561 CON phy_init 0 [CTL.YAML]       UL Gain Calibration: 48.68
08:14:31.783561 CON phy_init 0 [CTL.YAML]       Lower guard bw: 845
08:14:35.091025 CON phy_init 0 [CTL.SCF] Network interface for PCIe address 0000:38:00.0 : aerial00
08:14:35.091086 CON phy_init 0 [APP.UTILS]           PHC clock: 1769674475.91082288
08:14:35.091120 CON phy_init 0 [APP.UTILS]           CLOCK_TAI: 1769674475.91113967
08:14:35.091121 CON phy_init 0 [APP.UTILS]      CLOCK_REALTIME: 1769674475.91113910
08:14:35.091122 CON phy_init 0 [APP.UTILS] TAI/REALTIME offset: 0 seconds
08:14:35.096843 WRN phy_init 0 [DRV.WORKER] Unable to set pmu_type=3 for non Grace system.  Disabling pmu_metrics.
08:14:35.096858 WRN phy_init 0 [DRV.WORKER] Unable to set pmu_type=3 for non Grace system.  Disabling pmu_metrics.
08:14:35.096859 WRN phy_init 0 [DRV.WORKER] Unable to set pmu_type=3 for non Grace system.  Disabling pmu_metrics.
08:14:35.096864 WRN phy_init 0 [DRV.WORKER] Unable to set pmu_type=3 for non Grace system.  Disabling pmu_metrics.
08:14:35.096865 WRN phy_init 0 [DRV.WORKER] Unable to set pmu_type=3 for non Grace system.  Disabling pmu_metrics.
08:14:35.096865 WRN phy_init 0 [DRV.WORKER] Unable to set pmu_type=3 for non Grace system.  Disabling pmu_metrics.
08:14:35.096865 WRN phy_init 0 [DRV.WORKER] Unable to set pmu_type=3 for non Grace system.  Disabling pmu_metrics.
08:14:35.098184 CON phy_init 0 [DRV.CTX] CUDA_DEVICE_MAX_CONNECTIONS 8
08:14:35.098184 CON phy_init 0 [DRV.CTX] use_green_contexts 0
EAL: Detected CPU lcores: 64
EAL: Detected NUMA nodes: 2
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/cuphycontroller/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: 64 hugepages of size 2097152 reserved, but no mounted hugetlbfs found for that size
EAL: VFIO support initialized
EAL: Probe PCI driver: gpu_cuda (10de:27b8) device: 0000:27:00.0 (socket 0)
EAL: Probe PCI driver: mlx5_pci (15b3:101d) device: 0000:38:00.0 (socket 0)
08:14:36.810353 CON phy_init 0 [DRV.FH] Dynamic beam id start: 16528, end: 32767, num of beam ids per slots: 2192, number of consecutive slots with unique of beam ids: 7 
[08:14:37:446767][6687][DOCA][ERR][doca_gpunetio.cpp:461][doca_gpu_dmabuf_fd] cuMemGetHandleForAddressRange returned 1.
08:14:37.550692 CON phy_init 0 [FH.DOCA] doca_gpu_dmabuf_fd returned 0 (DOCA_SUCCESS is 0)
08:14:37.550703 CON phy_init 0 [FH.DOCA] Mapping transmit queue buffer (0x7f923e000000 size 4294967296B dmabuf fd 203) with dmabuf mode
PCAP_LOGGER Initialized
gRPC Server listening on 0.0.0.0:50051
08:14:42.301590 CON phy_init 0 [CTL.SCF] ====> PhyDriver initialized!
08:14:42.302859 CON phy_init 0 [NVIPC:YAML] nv_ipc_parse_yaml_node: PCAP msg_filter: enable all messages: 0x00-0xFF
08:14:42.302891 CON phy_init 0 [NVIPC:YAML] nv_ipc_parse_yaml_node: PCAP cell_filter: enable all cells 0-255
08:14:42.303305 CON phy_init 0 [NVIPC.DEBUG] nv_ipc_debug_open: prefix=nvipc fapi_type=1 nvipc_version=25.3.0 OK
08:14:42.311184 CON 6754 0 [NVIPC.DEBUG] pcap_shm_caching_thread_func: thread [nvipc_pcap_shm] started on CPU core [10] ...
08:14:42.384301 CON phy_init 0 [NVIPC.SHM] nvipc server initialized
08:14:42.384410 CON phy_init 0 [NVIPC.SHM] shm_ipc_open: forward_enable=0 fw_max_msg_buf_count=1365 fw_max_data_buf_count=341
08:14:42.384414 CON phy_init 0 [NVIPC.SHM] create_shm_nv_ipc_interface: OK. buf_size: cpu_msg=15000 cpu_data=576000 cpu_large=4096000
08:14:42.384427 CON phy_init 0 [L2A.TRANSPORT] phy_mac_transport[0] created. phy_cell_map.size=0 mac_cell_map.size=0 mapped=false cell_num=1
08:14:42.384432 CON phy_init 0 [L2A.TRANSPORT] init: loaded 1 nvipc instance, total_cell_num=1
08:14:42.384579 CON phy_init 0 [L2A.MODULE] PHY_module: transport_num=1 total_cell_num=1
08:14:42.406354 CON 6755 0 [NVIPC.EFD] [nvipc][core 00 ] share event_fd thread info: sched_policy=0 sched_priority=0
08:14:42.406398 CON 6755 0 [NVIPC.EFD] [nvipc] Listening for nvipc client to connect ...
08:14:43.305782 CON UlPhyDriver02 0 [DRV.WORKER] Thread UlPhyDriver02(worker_default) on CPU 2 initialized fmtlog
08:14:43.305786 CON DlPhyDriver07 0 [DRV.WORKER] Thread DlPhyDriver07(worker_default) on CPU 7 initialized fmtlog
08:14:43.305786 CON UlPhyDriver03 0 [DRV.WORKER] Thread UlPhyDriver03(worker_default) on CPU 3 initialized fmtlog
08:14:43.305802 CON DlPhyDriver06 0 [DRV.WORKER] Thread DlPhyDriver06(worker_default) on CPU 6 initialized fmtlog
08:14:43.305833 CON DlPhyDriver05 0 [DRV.WORKER] Thread DlPhyDriver05(worker_default) on CPU 5 initialized fmtlog
08:14:43.305865 CON DebugWorker09 0 [DRV.WORKER] Thread DebugWorker09(worker_default) on CPU 9 initialized fmtlog
08:14:43.305874 CON DlPhyDriver04 0 [DRV.WORKER] Thread DlPhyDriver04(worker_default) on CPU 4 initialized fmtlog
Lookup table populated with 13104 entries
08:14:46.365318 CON phy_main 0 [CTL.SCF] cuPHYController configured for 1 cells
08:14:46.365320 CON phy_main 0 [CTL.SCF] ====> cuPHYController initialized, L1 is ready!
08:14:46.365336 CON phy_main 0 [CTL.STARTUP_TIMES] {TI PERCENTAGE} <cuphycontroller main> Start Main:0.0,Parse Cuphycontroller YAML:0.0,Init nvlog:0.2,Cuda Set Device:1.7,Cuphy PTI Init:20.9,Init PHYDriver:49.3,Init cuphydriver:0.0,Make PHYDriverProxy:0.0,Init SCF FAPI:0.0,Create PHY_group:27.8,Start PHY_group:0.0, (total: 14623914.7us),

08:14:46.365343 CON phy_main 0 [CTL.STARTUP_TIMES] {TI DURATION} <cuphycontroller main> Start Main:6940.327,Parse Cuphycontroller YAML:2997.062,Init nvlog:32218.934,Cuda Set Device:251339.094,Cuphy PTI Init:3056222.970,Init PHYDriver:7210468.115,Init cuphydriver:0.359,Make PHYDriverProxy:22.419,Init SCF FAPI:5.222,Create PHY_group:4063445.613,Start PHY_group:254.618,

08:14:46.365345 CON phy_main 0 [CTL.STARTUP_TIMES] {TI TIMESTAMPS} <cuphycontroller main> Start Main:1769674471741406153,Parse Cuphycontroller YAML:1769674471748346480,Init nvlog:1769674471751343542,Cuda Set Device:1769674471783562476,Cuphy PTI Init:1769674472034901570,Init PHYDriver:1769674475091124540,Init cuphydriver:1769674482301592655,Make PHYDriverProxy:1769674482301593014,Init SCF FAPI:1769674482301615433,Create PHY_group:1769674482301620655,Start PHY_group:1769674486365066268,End Main:1769674486365320886,

08:14:46.365368 CON 6757 0 [L2A.MODULE] cell_update_thread_func: OAM thread affinity set to cpu core 0
08:14:46.373118 CON msg_processing 0 [L2A.MODULE] Thread thread_func on CPU 8 initialized fmtlog
08:15:15.728508 CON 6755 0 [NVIPC.EFD] [nvipc] nvipc unix socket server connected
08:15:15.728545 CON 6755 0 [NVIPC.EFD] [nvipc] Received peer event_fd: 209
08:15:15.928618 CON 6755 0 [NVIPC.EFD] Share event_fd succeed: efd_tx=209, efd_rx=217
08:15:15.928619 CON 6755 0 [NVIPC.EFD] [nvipc] Listening for nvipc client to connect ...
08:15:16.761420 CON msg_processing 0 [SCF.PHY] on_config_request: CONFIG.req received for cell_id=0 numTLVs=69 state=0
08:15:16.761686 CON msg_processing 0 [SCF.PHY] update_phy_stat_configs_l1: PHY Cell Id = 41, M-Plane Id= 1
08:15:16.782737 CON msg_processing 0 [DRV.PUSCH] tvStatPrms: PUSCH enableDeviceGraphLaunch=1 enableCsiP2Fapiv3 = 1 nMaxLdpcHetConfigs = 32
08:15:16.782740 CON msg_processing 0 [DRV.PUSCH] Timeout values for wait kernels are 3000us and 3000us
08:15:16.782742 CON msg_processing 0 [DRV.PUSCH] static_params.nMaxPrb: 273
08:15:16.782743 CON msg_processing 0 [DRV.PUSCH] static_params.nMaxRx: 4
08:15:16.811965 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 54.710 MiB for cuPHY PUSCH channel object (0x501954c6000).
08:15:16.811983 CON msg_processing 0 [CUPHY.PUSCH_RX] PuschRx: Running with eqCoeffAlgo 3
08:15:16.826186 CON msg_processing 0 [DRV.PUSCH] tvStatPrms: PUSCH enableDeviceGraphLaunch=1 enableCsiP2Fapiv3 = 1 nMaxLdpcHetConfigs = 32
08:15:16.826187 CON msg_processing 0 [DRV.PUSCH] Timeout values for wait kernels are 3000us and 3000us
08:15:16.826187 CON msg_processing 0 [DRV.PUSCH] static_params.nMaxPrb: 273
08:15:16.826187 CON msg_processing 0 [DRV.PUSCH] static_params.nMaxRx: 4
08:15:16.834659 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 54.710 MiB for cuPHY PUSCH channel object (0x501954d2000).
08:15:16.834664 CON msg_processing 0 [CUPHY.PUSCH_RX] PuschRx: Running with eqCoeffAlgo 3
08:15:16.855367 CON msg_processing 0 [DRV.PUSCH] tvStatPrms: PUSCH enableDeviceGraphLaunch=1 enableCsiP2Fapiv3 = 1 nMaxLdpcHetConfigs = 32
08:15:16.855368 CON msg_processing 0 [DRV.PUSCH] Timeout values for wait kernels are 3000us and 3000us
08:15:16.855369 CON msg_processing 0 [DRV.PUSCH] static_params.nMaxPrb: 273
08:15:16.855369 CON msg_processing 0 [DRV.PUSCH] static_params.nMaxRx: 4
08:15:16.863448 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 54.710 MiB for cuPHY PUSCH channel object (0x501954de000).
08:15:16.863453 CON msg_processing 0 [CUPHY.PUSCH_RX] PuschRx: Running with eqCoeffAlgo 3
08:15:16.877427 CON msg_processing 0 [DRV.PUSCH] tvStatPrms: PUSCH enableDeviceGraphLaunch=1 enableCsiP2Fapiv3 = 1 nMaxLdpcHetConfigs = 32
08:15:16.877427 CON msg_processing 0 [DRV.PUSCH] Timeout values for wait kernels are 3000us and 3000us
08:15:16.877428 CON msg_processing 0 [DRV.PUSCH] static_params.nMaxPrb: 273
08:15:16.877428 CON msg_processing 0 [DRV.PUSCH] static_params.nMaxRx: 4
08:15:16.885405 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 54.710 MiB for cuPHY PUSCH channel object (0x501954ea000).
08:15:16.885410 CON msg_processing 0 [CUPHY.PUSCH_RX] PuschRx: Running with eqCoeffAlgo 3
08:15:16.888830 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 3.344 MiB for cuPHY PUCCH channel object (0x50195399a00).
08:15:16.890764 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 3.344 MiB for cuPHY PUCCH channel object (0x5019539a800).
08:15:16.891670 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 3.344 MiB for cuPHY PUCCH channel object (0x5019539b600).
08:15:16.892537 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 3.344 MiB for cuPHY PUCCH channel object (0x5019539c400).
08:15:16.896277 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.182 MiB for cuPHY PRACH channel object (0x50196d0a500).
08:15:16.897810 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.182 MiB for cuPHY PRACH channel object (0x50196d0a880).
08:15:16.904122 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 98.130 MiB for cuPHY PDSCH channel object (0x50196a43000).
08:15:16.907904 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 98.130 MiB for cuPHY PDSCH channel object (0x50196a46c00).
08:15:16.911670 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 98.130 MiB for cuPHY PDSCH channel object (0x50196a48400).
08:15:16.915366 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 98.130 MiB for cuPHY PDSCH channel object (0x50196a49c00).
08:15:16.919245 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 98.130 MiB for cuPHY PDSCH channel object (0x50196a4b400).
08:15:16.923202 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 98.130 MiB for cuPHY PDSCH channel object (0x50196a4cc00).
08:15:16.927267 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 98.130 MiB for cuPHY PDSCH channel object (0x50196a4e400).
08:15:16.932040 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 98.130 MiB for cuPHY PDSCH channel object (0x5019c850000).
08:15:16.936112 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 98.130 MiB for cuPHY PDSCH channel object (0x5019c851800).
08:15:16.940222 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 98.130 MiB for cuPHY PDSCH channel object (0x5019c853000).
08:15:16.942664 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.001 MiB for cuPHY SSB channel object (0x50196d55100).
08:15:16.942938 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.001 MiB for cuPHY SSB channel object (0x50196d53c00).
08:15:16.943124 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.001 MiB for cuPHY SSB channel object (0x50196d54d80).
08:15:16.943291 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.001 MiB for cuPHY SSB channel object (0x50196d55480).
08:15:16.943450 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.001 MiB for cuPHY SSB channel object (0x50196d55800).
08:15:16.943613 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.001 MiB for cuPHY SSB channel object (0x50196d55b80).
08:15:16.943793 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.001 MiB for cuPHY SSB channel object (0x50196d55f00).
08:15:16.943958 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.001 MiB for cuPHY SSB channel object (0x50196d56280).
08:15:16.944118 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.001 MiB for cuPHY SSB channel object (0x50196d56600).
08:15:16.944275 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.001 MiB for cuPHY SSB channel object (0x50196d56980).
08:15:16.944600 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 1.607 MiB for cuPHY PDCCH channel object (0x50196edc800).
08:15:16.944851 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 1.607 MiB for cuPHY PDCCH channel object (0x50196edcd00).
08:15:16.945178 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 1.607 MiB for cuPHY PDCCH channel object (0x50196edd200).
08:15:16.945423 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 1.607 MiB for cuPHY PDCCH channel object (0x50196edd700).
08:15:16.946522 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 1.607 MiB for cuPHY PDCCH channel object (0x50196eddc00).
08:15:16.946925 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 1.607 MiB for cuPHY PDCCH channel object (0x50196ede100).
08:15:16.947187 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 1.607 MiB for cuPHY PDCCH channel object (0x50196ede600).
08:15:16.947447 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 1.607 MiB for cuPHY PDCCH channel object (0x50196edeb00).
08:15:16.948512 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 1.607 MiB for cuPHY PDCCH channel object (0x50196edf000).
08:15:16.948804 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 1.607 MiB for cuPHY PDCCH channel object (0x50196edf500).
08:15:16.949048 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.093 MiB for cuPHY CSIRS channel object (0x5019d35e480).
08:15:16.949204 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.093 MiB for cuPHY CSIRS channel object (0x5019d35e780).
08:15:16.949378 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.093 MiB for cuPHY CSIRS channel object (0x5019d35ea80).
08:15:16.949508 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.093 MiB for cuPHY CSIRS channel object (0x5019d35ed80).
08:15:16.949630 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.093 MiB for cuPHY CSIRS channel object (0x5019d35f080).
08:15:16.949738 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.093 MiB for cuPHY CSIRS channel object (0x5019d35f380).
08:15:16.949878 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.093 MiB for cuPHY CSIRS channel object (0x5019d35f680).
08:15:16.950019 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.093 MiB for cuPHY CSIRS channel object (0x5019d35f980).
08:15:16.950146 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.093 MiB for cuPHY CSIRS channel object (0x5019d35fc80).
08:15:16.950270 CON msg_processing 0 [CUPHY.MEMFOOT] cuphyMemoryFootprint - GPU allocation: 0.093 MiB for cuPHY CSIRS channel object (0x5019d510080).
08:15:16.950284 CON msg_processing 0 [DRV.API] Update cell: mplane_id=1 dl_grid_sz=273 
08:15:16.950285 CON msg_processing 0 [DRV.API] Update cell: mplane_id=1 ul_grid_sz=273 
08:15:16.950301 CON msg_processing 0 [SCF.PHY] on_config_request: create_cell_configs for cell_id=0 phy_cell_id=41 returned error_code=0
08:15:16.950319 CON msg_processing 0 [SCF.PHY] send_cell_config_response: Send CONFIG.response: cell_id=0 error_code=0x0
08:15:16.950356 CON msg_processing 0 [L2A.TRANSPORT] wrapper: configured_cells_mask=0x1 expected_mask=0x1 all_cells_configured=true - all cells configured
08:15:16.950398 CON msg_processing 0 [SCF.PHY] on_cell_start_request: Cell 0 Received START.request, sending START.response... phy_cell_id=41
08:15:16.950911 CON msg_processing 0 [L2A.TRANSPORT] wrapper: configured_cells_mask=0x1 expected_mask=0x1 all_cells_configured=true - all cells configured
08:15:16.959348 CON timer_thread 0 [L2A.TICK] Thread slot_indication_thread_sleep_method initialized fmtlog
08:15:16.959355 CON timer_thread 0 [L2A.TICK] PTP Configs: gps_alpha: 0 gps_beta: 0
08:15:16.959362 CON timer_thread 0 [L2A.TICK] Start time: tick=1769674516959350821, seconds since epoch = 1769674516, nanoseconds = 959350821
08:15:16.959363 CON timer_thread 0 [L2A.TICK] FIRST tick scheduled for: tick=1769674520120000000, seconds since epoch = 1769674520, nanoseconds = 120000000
/opt/nvidia/cuBB/testBenches/phase4_test_scripts/run2_cuPHYcontroller.sh: line 270:  6685 Segmentation fault      sudo -E $WITH_TIMEOUT $GDB_SCRIPT "$cuBB_SDK/$BUILD_DIR/cuPHY-CP/cuphycontroller/examples/cuphycontroller_scf" $CONTROLLER_MODE

Here, the segmentation fault still occurs, the detailed configuration and log are attached.
config.yaml.txt (24.3 KB)
cuphycontroller_F08_GL4.yaml.txt (30.1 KB)
phy.log (33.0 KB)
ru.log (396.8 KB)
test_mac_config.yaml.txt (2.7 KB)
testmac.log (15.1 KB)

Hi, bkecicioglu
I have confirmed that the error “mlx5_common: Fail to create MR for address (0x7ee1d4000000)” is caused by the configuration “bfw_c_plane_chaining_mode: 2” in the cuphycontroller yaml file. When I removed this configuration, the problem was resolved. What’s the meaning of this configuration? Why does it cause this issue?

It seems there is an issue with nvidia-peermem module on your system. Can you check the following:

# Check module parameters
cat /sys/module/nvidia_peermem/parameters/* 2>/dev/null || echo "No parameters exposed"

# Look for any peermem-related errors in dmesg
dmesg | grep -i -E "peermem|nv_peer|nvidia.*peer|ib_register|mlx5.*mr"

# gdrdrv is another path for GPU direct access
lsmod | grep gdrdrv
dmesg | grep -i gdr

# Check if gdrdrv device exists
ls -la /dev/gdrdrv

# Check GPU peer-to-peer capabilities
nvidia-smi -q | grep -i -A5 "Peer to Peer"

# Check if GPU supports peer memory mapping
nvidia-smi -q | grep -i -A10 "GPU Memory"

The results are as follows. It seems there are no problems.

root@worker2:~# cat /sys/module/nvidia_peermem/parameters/* 2>/dev/null || echo "No parameters exposed"
0
1
root@worker2:~# dmesg | grep -i -E "peermem|nv_peer|nvidia.*peer|ib_register|mlx5.*mr"
root@worker2:~# lsmod | grep gdrdrv
gdrdrv                 28672  0
nvidia              11427840  9 nvidia_uvm,nvidia_peermem,gdrdrv,nvidia_modeset
root@worker2:~# dmesg | grep -i gdr
root@worker2:~# ls -la /dev/gdrdrv
crw-rw-rw- 1 root root 503, 0 Jan 27 09:21 /dev/gdrdrv
root@worker2:~# nvidia-smi -q | grep -i -A5 "Peer to Peer"
root@worker2:~# nvidia-smi -q | grep -i -A10 "GPU Memory"
    GPU Memory Power Readings 
        Average Power Draw                : N/A
        Instantaneous Power Draw          : N/A
    Module Power Readings
        Average Power Draw                : N/A
        Instantaneous Power Draw          : N/A
        Current Power Limit               : N/A
        Requested Power Limit             : N/A
        Default Power Limit               : N/A
        Min Power Limit                   : N/A
        Max Power Limit                   : N/A

Hi @kemalong ,

Here are the possible configurations for this parameter:

Value Mode Memory Allocation FH Registration Description
0 NO_CHAINING CPU pinned only None Single packet per message, no DMA registration needed
1 CPU_CHAINING CPU pinned only Pinned buffer registered Chained mbufs for large BFW, CPU pinned memory DMA-mapped to NIC
2 GPU_CHAINING CPU pinned + GPU device Both registered Chained mbufs using GPU memory, both buffers DMA-mapped to NIC
pdctx->registerBufferToFh(bfw_coeff_buffer_dev.addr(), pdctx->getFhProxy()->getBfwCoeffSize());

This attempts to DMA-map cudaMalloc()-allocated GPU memory via rte_dev_dma_map(), which requires the nvidia-peermem path. This is where the mlx5_common: Fail to create MR error occurs.

Hi, bkecicioglu,
I have reinstalled the DOCA OFED、CUDA Driver and GDRCopy, and also have connected the GPU and NIC with PCIe switch, the problem still can not be resolved. According to the following output, can we rule out the possibility that it’s a problem with the nvidia-peerm module?

root@worker2:~# cat /sys/module/nvidia_peermem/parameters/* 2>/dev/null || echo "No parameters exposed"
0
1
root@worker2:~# dmesg | grep -i -E "peermem|nv_peer|nvidia.*peer|ib_register|mlx5.*mr"
[ 3244.559709]  ib_register_peer_memory_client+0x10a/0x240 [ib_uverbs]
[ 3244.559718]  nv_mem_client_init+0x64/0x1000 [nv_peer_mem]
[ 3379.402506]  ib_register_peer_memory_client+0x10a/0x240 [ib_uverbs]
[ 3379.402517]  nv_mem_client_init+0x43/0x1000 [nv_peer_mem]
[ 3381.519814]  ib_register_peer_memory_client+0x10a/0x240 [ib_uverbs]
[ 3381.519825]  nv_mem_client_init+0x43/0x1000 [nv_peer_mem]
root@worker2:~# lsmod | grep gdrdrv
gdrdrv                 28672  0
nvidia              11427840  54 nvidia_uvm,nvidia_peermem,gdrdrv,nvidia_modeset
root@worker2:~# dmesg | grep -i gdr
[  115.931411] gdrdrv:gdrdrv_init:loading gdrdrv version 2.5 built for opensource NVIDIA driver
[  115.931414] gdrdrv:gdrdrv_init:device registered with major number 503
[  115.931415] gdrdrv:gdrdrv_init:dbg traces disabled, info traces disabled
[  115.931416] gdrdrv:gdrdrv_init:Persistent mapping will be used
[ 2060.531015] gdrdrv:gdrdrv_cleanup:unregistering major number 503
[ 2887.042027] gdrdrv:gdrdrv_init:loading gdrdrv version 2.5 built for opensource NVIDIA driver
[ 2887.042030] gdrdrv:gdrdrv_init:device registered with major number 502
[ 2887.042031] gdrdrv:gdrdrv_init:dbg traces disabled, info traces disabled
[ 2887.042032] gdrdrv:gdrdrv_init:Persistent mapping will be used
root@worker2:~# ls -la /dev/gdrdrv
crw-rw-rw- 1 root root 502, 0 Jan 28 09:39 /dev/gdrdrv
root@worker2:~# nvidia-smi -q | grep -i -A5 "Peer to Peer"
root@worker2:~# nvidia-smi -q | grep -i -A10 "GPU Memory"
    GPU Memory Power Readings 
        Average Power Draw                : N/A
        Instantaneous Power Draw          : N/A
    Module Power Readings
        Average Power Draw                : N/A
        Instantaneous Power Draw          : N/A
        Current Power Limit               : N/A
        Requested Power Limit             : N/A
        Default Power Limit               : N/A
        Min Power Limit                   : N/A
        Max Power Limit                   : N/A
--
            Used GPU Memory               : 28 MiB
    Capabilities
        EGM                               : disabled

root@worker2:~# nvidia-smi  topo -m
	GPU0	NIC0	NIC1	CPU Affinity	NUMA Affinity	GPU NUMA ID
GPU0	 X 	PIX	PIX	0-31	0		N/A
NIC0	PIX	 X 	PIX				
NIC1	PIX	PIX	 X 				

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: mlx5_0
  NIC1: mlx5_1

All the modules and configurations seem to be correct. Do you have any other ideas for the next step? Many thanks.

By the way, is the following DOCA error related to to DMA MAP error?

12:09:44.209346 CON phy_init 0 [DRV.CTX] use_green_contexts 0
EAL: Detected CPU lcores: 64
EAL: Detected NUMA nodes: 2
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/cuphycontroller/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: 64 hugepages of size 2097152 reserved, but no mounted hugetlbfs found for that size
EAL: VFIO support initialized
EAL: Probe PCI driver: gpu_cuda (10de:27b8) device: 0000:29:00.0 (socket 0)
EAL: Probe PCI driver: mlx5_pci (15b3:101d) device: 0000:2a:00.0 (socket 0)
12:09:45.934749 CON phy_init 0 [DRV.FH] Dynamic beam id start: 16528, end: 32767, num of beam ids per slots: 2192, number of consecutive slots with unique of beam ids: 7 
[12:09:46:591500][13244][DOCA][ERR][doca_gpunetio.cpp:461][doca_gpu_dmabuf_fd] cuMemGetHandleForAddressRange returned 1.
[12:09:46:674263][13244][DOCA][ERR][doca_gpunetio.cpp:461][doca_gpu_dmabuf_fd] cuMemGetHandleForAddressRange returned 1.
[12:09:46:755057][13244][DOCA][ERR][doca_gpunetio.cpp:461][doca_gpu_dmabuf_fd] cuMemGetHandleForAddressRange returned 1.
[12:09:46:835373][13244][DOCA][ERR][doca_gpunetio.cpp:461][doca_gpu_dmabuf_fd] cuMemGetHandleForAddressRange returned 1.
[12:09:46:915608][13244][DOCA][ERR][doca_gpunetio.cpp:461][doca_gpu_dmabuf_fd] cuMemGetHandleForAddressRange returned 1.
[12:09:46:996067][13244][DOCA][ERR][doca_gpunetio.cpp:461][doca_gpu_dmabuf_fd] cuMemGetHandleForAddressRange returned 1.
12:09:47.138007 CON phy_init 0 [FH.DOCA] doca_gpu_dmabuf_fd returned 0 (DOCA_SUCCESS is 0)
12:09:47.138017 CON phy_init 0 [FH.DOCA] Mapping transmit queue buffer (0x7fa27e000000 size 2147483648B dmabuf fd 243) with dmabuf mode

@kemalong

would you please confirm that, you have gotten cuphycontroller started, but the it crashed as soon as you started the E2E test with RU-emulator. right?

if so, would you please share the info below?

lscpu

nvidia-smi topo -m

nvidia-smi

cat /proc/cmdline

l2_adapter_config_F08_CG1.yaml

what type of system is used for RU-emulator?

Hi jixu,

Thank you for your response. By adopting the following configurations, the simulator E2E can run normally now:

  1. The GPU and the NIC are connected with a PCIe switch, i.e.,
  2. Usegpu_init_comms_via_cpu: 0 in the cuphycontroller_F08_GL4.yaml.
    With the above configurations, testMAC、cuphycontroller and RU Emulator have the same traffic,and RU reported that all the slots are on time.

However, I still confused with the following problems:

Problem 1:
When I use gpu_init_comms_via_cpu: 1 in the cuphycontroller_F08_GL4.yaml, then the cuphycontroller occurs the following error:


and when the testMAC running, it suffers a segmentaion fault, as:

the gdb result is as:

Problem 2:
When I use bfw_c_plane_chaining_mode: 2 in the cuphycontroller_F08_GL4.yaml, then the folowing DMA MAP error ouccurs, but the cuphycontroller seems work normally. When I removed the bfw_c_plane_chaining_mode: 2, the the DMA MAP error disappear.

Could you please give some explanations about the above results? Will these errors affects the subsequent OTA test with real O-RU?

The information you need is provided as follows:

root@Node-01:~# lscpu
Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         46 bits physical, 57 bits virtual
  Byte Order:            Little Endian
CPU(s):                  64
  On-line CPU(s) list:   0-63
Vendor ID:               GenuineIntel
  Model name:            Intel(R) Xeon(R) Gold 6430
    CPU family:          6
    Model:               143
    Thread(s) per core:  1
    Core(s) per socket:  32
    Socket(s):           2
    Stepping:            8
    Frequency boost:     enabled
    CPU max MHz:         1201.0000
    CPU min MHz:         800.0000
    BogoMIPS:            2400.00
    Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc ar
                         t arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdc
                         m pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cat_l2 cdp_l3 invpcid_sin
                         gle intel_ppin cdp_l2 ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm r
                         dt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm
                         _mbm_total cqm_mbm_local split_lock_detect avx_vnni avx512_bf16 wbnoinvd dtherm ida arat pln pts avx512vbmi umip pku ospke waitpkg avx512_vbmi2 gfni vaes vpclmulqdq
                          avx512_vnni avx512_bitalg tme avx512_vpopcntdq la57 rdpid bus_lock_detect cldemote movdiri movdir64b enqcmd fsrm md_clear serialize tsxldtrk pconfig arch_lbr amx_b
                         f16 avx512_fp16 amx_tile amx_int8 flush_l1d arch_capabilities
Virtualization features: 
  Virtualization:        VT-x
Caches (sum of all):     
  L1d:                   3 MiB (64 instances)
  L1i:                   2 MiB (64 instances)
  L2:                    128 MiB (64 instances)
  L3:                    120 MiB (2 instances)
NUMA:                    
  NUMA node(s):          2
  NUMA node0 CPU(s):     0-31
  NUMA node1 CPU(s):     32-63
Vulnerabilities:         
  Gather data sampling:  Not affected
  Itlb multihit:         Not affected
  L1tf:                  Not affected
  Mds:                   Not affected
  Meltdown:              Not affected
  Mmio stale data:       Not affected
  Retbleed:              Not affected
  Spec rstack overflow:  Not affected
  Spec store bypass:     Mitigation; Speculative Store Bypass disabled via prctl and seccomp
  Spectre v1:            Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:            Mitigation; Enhanced IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS SW sequence
  Srbds:                 Not affected
  Tsx async abort:       Not affected
root@Node-01:~# nvidia-smi topo -m
	GPU0	NIC0	NIC1	CPU Affinity	NUMA Affinity	GPU NUMA ID
GPU0	 X 	PIX	PIX	0-31	0		N/A
NIC0	PIX	 X 	PIX				
NIC1	PIX	PIX	 X 				

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: mlx5_0
  NIC1: mlx5_1
root@Node-01:~# nvidia-smi
Mon Feb  2 11:35:03 2026       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 575.57.08              Driver Version: 575.57.08      CUDA Version: 12.9     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA L4                      On  |   00000000:29:00.0 Off |                    0 |
| N/A   51C    P0             28W /   72W |     119MiB /  23034MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A          961788      C   nvidia-cuda-mps-server                   28MiB |
|    0   N/A  N/A         1598223      C   nvidia-cuda-mps-server                   28MiB |
|    0   N/A  N/A         2625044      C   nvidia-cuda-mps-server                   28MiB |
+-----------------------------------------------------------------------------------------+
root@Node-01:~# 
root@Node-01:~# cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-5.15.0-1042-nvidia-lowlatency root=/dev/mapper/ubuntu--vg-ubuntu--lv ro intel_iommu=on iommu=pt vfio_pci.enable_sriov=1 vfio_pci.disable_idle_d3=1 pci=realloc=off pci=pcie_bus_safe default_hugepagesz=1G hugepagesz=1G hugepages=64 hugepagesz=2M hugepages=64 tsc=reliable processor.max_cstate=0 audit=0 idle=poll rcu_nocb_poll nosoftlockup irqaffinity=0 isolcpus=managed_irq,domain,4-64 nohz_full=4-64 rcu_nocbs=4-64 earlycon module_blacklist=nouveau acpi_power_meter.force_cap_on=y numa_balancing=disable init_on_alloc=0 preempt=none psi=0 acpi=force reboot=pci

l2_adapter_config_F08_CG1.txt (4.3 KB)
The RU Emulator is deployed on a x86 platform with Hygon C86 CPU:

root@Node-02:~# lscpu
Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         48 bits physical, 48 bits virtual
  Byte Order:            Little Endian
CPU(s):                  128
  On-line CPU(s) list:   0-127
Vendor ID:               HygonGenuine
  Model name:            Hygon C86 7285 32-core Processor
    CPU family:          24
    Model:               1
    Thread(s) per core:  2
    Core(s) per socket:  32
    Socket(s):           2
    Stepping:            1
    Frequency boost:     enabled
    CPU max MHz:         2000.0000
    CPU min MHz:         1200.0000
    BogoMIPS:            3999.42
    Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_g
                         ood nopl nonstop_tsc cpuid extd_apicid amd_dcm aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp
                         _legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb hw_pstate ssbd i
                         bpb vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 xsaves clzero irperf xsaveerptr arat npt lbrv svm_lock nrip_save 
                         tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif overflow_recov succor smca sme sev sev_es
Virtualization features: 
  Virtualization:        AMD-V
Caches (sum of all):     
  L1d:                   2 MiB (64 instances)
  L1i:                   4 MiB (64 instances)
  L2:                    32 MiB (64 instances)
  L3:                    128 MiB (16 instances)
NUMA:                    
  NUMA node(s):          8
  NUMA node0 CPU(s):     0-7,64-71
  NUMA node1 CPU(s):     8-15,72-79
  NUMA node2 CPU(s):     16-23,80-87
  NUMA node3 CPU(s):     24-31,88-95
  NUMA node4 CPU(s):     32-39,96-103
  NUMA node5 CPU(s):     40-47,104-111
  NUMA node6 CPU(s):     48-55,112-119
  NUMA node7 CPU(s):     56-63,120-127
Vulnerabilities:         
  Gather data sampling:  Not affected
  Itlb multihit:         Not affected
  L1tf:                  Not affected
  Mds:                   Not affected
  Meltdown:              Not affected
  Mmio stale data:       Not affected
  Retbleed:              Mitigation; untrained return thunk; SMT vulnerable
  Spec rstack overflow:  Not affected
  Spec store bypass:     Mitigation; Speculative Store Bypass disabled via prctl and seccomp
  Spectre v1:            Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:            Mitigation; Retpolines, IBPB conditional, STIBP disabled, RSB filling, PBRSB-eIBRS Not affected
  Srbds:                 Not affected
  Tsx async abort:       Not affected

If you need more information, please let me know.

Hi @kemalong,

Glad to hear the simulator is running with gpu_init_comms_via_cpu: 0. Let me address the two remaining issues.

Problem 1 (gpu_init_comms_via_cpu: 1 → segfault)

Can you provide the following for analyzing the crash in aerial_fh::GpuComm::cpu_send → ring_cpu_doorbell :

  1. The full phy.log from a run with gpu_init_comms_via_cpu: 1 (not the one from post #8 — a fresh run with only this flag changed)
  2. Your DOCA version: doca_version or dpkg -l | grep doca
  3. The complete GDB backtrace — the screenshot is truncated. Please run with set pagination off and capture all frames

Problem 2 (bfw_c_plane_chaining_mode: 2 → mlx5 MR failure)

This is a GPUDirect RDMA failure on the DPDK/mlx5 path, which is separate from the DOCA dmabuf path used by gpu_init_comms_via_cpu: 0. The fact that one works but the other doesn’t suggests nvidia-peermem specifically may not be functioning correctly. To confirm, please run:

# 1. Check GPU BAR1 size — needs to be large enough for GPUDirect RDMA
nvidia-smi -q | grep -i BAR1

# 2. Check ACS across the full PCIe hierarchy
lspci -vvv | grep -i -A5 ACS

# 3. Verify GPUDirect RDMA actually works end-to-end
gdrcopy_sanity

# 4. Check dmesg immediately after loading nvidia-peermem
dmesg | grep -i -E "peer_mem|nv_mem|ib_register_peer"

Also, could you share the make/model of the PCIe switch connecting the GPU and NIC? Some switches require explicit P2P routing configuration.

One observation: nvidia-smi -q | grep "Peer to Peer" returned no output in your diagnostics (post #12). On a system with working GPUDirect RDMA this section is typically present. This is worth investigating independently of the Aerial stack.