cuBB end-to-end loopback test issue: RU Emulator fails to create C-plane flow rule on BF3 loopback in cuBB 26-1

Accelerated Computing Aerial Forum
1

Hi NVIDIA Aerial team,

I am trying to run the cuBB end-to-end loopback test on a single GH200 server, following the loopback cuBB test flow. The RU Emulator fails during initialization when creating the C-plane flow rule on the BlueField-3 NIC.

Environment

  • Platform: GH200 / aarch64

  • Aerial cuBB image: nvcr.io/nvidia/aerial/aerial-cuda-accelerated-ran:26-1-cubb

  • Host source path: /home/rain/aerial-cuda-accelerated-ran

  • Container path: /opt/nvidia/cuBB

  • NIC used for loopback: BlueField-3

    • aerial00 → 0002:01:00.0

    • aerial01 → 0002:01:00.1

  • BF3 firmware:

    • 32.47.1088

    • PSID: MT_0000000884

  • BFB installed:

    • bf-fwbundle-3.2.1-34_25.11-prod.bfb

    • bfb-install completed successfully

    • Output included:

      • INFO[MISC]: Installation finished

      • INFO[UEFI]: In Enhanced NIC mode

      • Binding pf0 (0002:01:00.0)

      • Binding pf1 (0002:01:00.1)

  • Cold boot / full power cycle was performed after the BFB installation and mlxconfig changes.

Physical link status

The two BF3 ports are connected with a QSFP112 DAC cable. Both ports are up at 200G:

aerial00:
  bus-info: 0002:01:00.0
  Speed: 200000Mb/s
  Link detected: yes

aerial01:
  bus-info: 0002:01:00.1
  Speed: 200000Mb/s
  Link detected: yes

BF3 mlxconfig

Both /dev/mst/mt41692_pciconf0 and /dev/mst/mt41692_pciconf0.1 show the expected settings:

FLEX_PARSER_PROFILE_ENABLE        4
PROG_PARSE_GRAPH                  True(1)
INTERNAL_CPU_MODEL                EMBEDDED_CPU(1)
INTERNAL_CPU_PAGE_SUPPLIER        EXT_HOST_PF(1)
INTERNAL_CPU_ESWITCH_MANAGER      EXT_HOST_PF(1)
INTERNAL_CPU_IB_VPORT0            EXT_HOST_PF(1)
INTERNAL_CPU_OFFLOAD_ENGINE       DISABLED(1)
ACCURATE_TX_SCHEDULER             True(1)
CQE_COMPRESSION                   AGGRESSIVE(1)
REAL_TIME_CLOCK_ENABLE            True(1)
LINK_TYPE_P1                      ETH(2)
LINK_TYPE_P2                      ETH(2)

PTP status

ptp4l and phc2sys are running.

ptp4l selected /dev/ptp2, which corresponds to aerial00.

phc2sys is using aerial00:

/usr/sbin/phc2sys -s aerial00 -c CLOCK_REALTIME -n 24 -O 0 -R 256 -u 256

The phc2sys RMS value is around 5–8.

Test case

I am running the 6001 PUCCH loopback case.

Setup commands:

export cuBB_SDK=/opt/nvidia/cuBB
cd $cuBB_SDK/testBenches/phase4_test_scripts

export du_port=aerial00
export ru_port=aerial01

./setup1_DU.sh -y nrSim_SCF_CG1_6001 \
  --ru-host-type=_LOOPBACK \
  --du-eth0=$du_port

./setup2_RU.sh --ru-eth0=$ru_port

./test_config_nrSim.sh --channels PUCCH

Then I run:

./run1_RU.sh

Error

The RU Emulator fails during initialization:

EAL: Probe PCI driver: mlx5_pci (15b3:a2dc) device: 0002:01:00.1 (socket 0)

[RU] PUCCH enabled!
[RU] Config file: /opt/nvidia/cuBB/cuPHY-CP/ru-emulator/config/ru_emulator_config_nrSim_SCF_CG1_6001.yaml
[RU] Launch pattern file: /opt/nvidia/cuBB/testVectors/multi-cell/launch_pattern_nrSim_6001.yaml

ERR [AERIAL_ORAN_FH_EVENT] [FH.LIB] Exception! Failed to create C-plane flow rule for Peer 58:a2:e1:68:5c:b4 on NIC 0002:01:00.1: hardware refuses to create flow
ERR [AERIAL_RU_EMULATOR_EVENT] [RU] Throwing exception: Failed to add Peer 0
ERR [AERIAL_RU_EMULATOR_EVENT] [RU] Exception caught at initialization phase: Failed to add Peer 0

The relevant dmesg line is:

mlx5_core 0002:01:00.1: mlx5_cmd_out_err:839:
CREATE_FLOW_GROUP(0x933) op_mod(0x0) failed,
status bad parameter(0x3), syndrome (0x8b69a9), err(-22)

Sometimes after this main error, RU Emulator also aborts with:

fmt::v10::format_error: string pointer is null

I assume this is a secondary error during exception handling. The main issue seems to be the C-plane flow rule creation failure.

What I already tried

  1. Verified that aerial00/aerial01 are the BF3 ports and are linked at 200G.

  2. Verified BF3 mlxconfig settings on both mt41692_pciconf0 and mt41692_pciconf0.1.

  3. Installed the BF3 BFB bundle successfully via rshim0.

  4. Performed a full cold boot / power cycle after BFB installation and mlxconfig changes.

  5. Verified that rshim0 is attached.

  6. Verified that ptp4l and phc2sys are running.

  7. Reversed the DU/RU ports:

    • du_port=aerial01

    • ru_port=aerial00
      The same type of failure still occurred.

  8. Reduced the RU config to only Cell1 to rule out too many cell/flow rules.
    The same failure still occurred.

Question

Could you help clarify what CREATE_FLOW_GROUP failed, status bad parameter, syndrome 0x8b69a9 means in this context?

Is there any additional BF3 / eCPRI parser / flow steering / DOCA / DPDK configuration required for RU Emulator C-plane flow rule creation in cuBB 26-1 on GH200 loopback?

Also, is there any known issue with the RU Emulator C-plane flow rule on BF3 firmware 32.47.1088 / BFB 3.2.1-34_25.11?

image
image1203×1278 180 KB

I also attached a photo of the physical cabling. The red box marks the DAC cable connection used for the BF3 loopback test.

2

Hi @chengjunjin20011 ,

The issue can be DOCA 3.2.1-specific issue. We have also observed the same error message when we migrate to DOCA 3.2.1. However, if the situation is the same as what we observed, it means you are using an older Aerial source code. Could you please check if your source code is the original one without any changes?
Particularly, could you share the output of the following commands?

grep -n "dv_flow_en\|txq_inline_max" /opt/nvidia/cuBB/cuPHY-CP/aerial-fh-driver/lib/nic.cpp
dpkg -l | grep -i doca-host
ofed_info -s
cat /home/rain/aerial-cuda-accelerated-ran/aerial_sdk_version

Thank you.