After setting mlxconfig -d <device> s PCI_BUS00_HIERARCHY_TYPE=2, the DPU system failed to boot

I installed the BlueField3 Ubuntu.bfb and plugged it into the pcie switch rc port. After setting

mlxconfig -d <device> s PCI_BUS00_HIERARCHY_TYPE=2

the DPU system failed to boot.

bmc:
root@dpu-bmc:~# echo 'SW_RESET 1' > /dev/rshim0/misc
root@dpu-bmc:~# cat /dev/rshim0/misc 
DISPLAY_LEVEL   2 (0:basic, 1:advanced, 2:log)
BOOT_MODE       1 (0:rshim, 1:emmc, 2:emmc-boot-swap)
BOOT_TIMEOUT    150 (seconds)
DROP_MODE       0 (0:normal, 1:drop)
SW_RESET        0 (1: reset)
DEV_NAME        usb-2.1
DEV_INFO        BlueField-3(Rev 1)
OPN_STR         N/A
UP_TIME         40(s)
---------------------------------------
             Log Messages
---------------------------------------
 INFO[PSC]: PSC BL1 START
 INFO[BL2]: start
 INFO[BL2]: boot mode (emmc)
 INFO[BL2]: VDDQ adjustment complete
 INFO[BL2]: VDD: 1118 mV
 INFO[BL2]: DDR POST passed
 INFO[BL2]: UEFI loaded
 INFO[BL31]: start
 INFO[BL31]: lifecycle GA Secured
 INFO[BL31]: VDD: 881 mV
 INFO[BL31]: runtime
 INFO[UEFI]: eMMC init
 INFO[UEFI]: eMMC probed
 INFO[UEFI]: UPVS valid
 INFO[UEFI]: PCIe enum start

dpu-log
Nvidia BlueField-3 rev1 BL1 V1.0
NOTICE:  Running as 9009D3B600CVAA system
NOTICE:  BL2: v2.2(release):4.5.0-37-g8408d48
NOTICE:  BL2: Built : 23:21:33, Dec  6 2023
NOTICE:  BL2 built for hw (ver 2)
NOTICE:  # Finished initializing DDR MSS0
NOTICE:  # Finished initializing DDR MSS1
NOTICE:  DDR POST passed.
NOTICE:  BL31: v2.2(release):4.5.0-37-g8408d48
NOTICE:  BL31: Built : 23:21:33, Dec  6 2023
NOTICE:  BL31 built for hw (ver 2), lifecycle GA Secured
UEFI firmware (version BlueField:4.5.0-46-gf57517d-BId12993 built at 23:31:04 on Dec  6 2023)


cat /var/log/obmc-console.log
Nvidia BlueField-3 rev1 BL1 V1.0
INFO: psc supervisor init.
INFO: psc_irq_init...
INFO: force_crs_enable=0 pcr.lock0 = 1, time = 84366
INFO: enter idle task.
NOTICE:  Running as 9009D3B600CVAA system
NOTICE:  BL2: v2.2(release):4.5.0-37-g8408d48
NOTICE:  BL2: Built : 23:21:33, Dec  6 2023
NOTICE:  BL2 built for hw (ver 2)
NOTICE:  # Finished initializing DDR MSS0
NOTICE:  # Finished initializing DDR MSS1
NOTICE:  DDR POST passed.
INFO: mailbox rx: channel = 2, code = 0x43544c44
NOTICE:  BL31: v2.2(release):4.5.0-37-g8408d48
NOTICE:  BL31: Built : 23:21:33, Dec  6 2023
NOTICE:  BL31 built for hw (ver 2), lifecycle GA Secured
I/TC: 
I/TC: OP-TEE version: 3.10.0-20-ge0ab597 (gcc version 8.3.0 (GCC)) #1 Wed Dec  6 23:19:33 UTC 2023 aarch64
I/TC: Primary CPU initializing
INFO: mailbox rx: channel = 2, code = 0x41544b52
I/TC: Primary CPU switching to normal world boot
UEFI firmware (version BlueField:4.5.0-46-gf57517d-BId12993 built at 23:31:04 on Dec  6 2023)

Restart log stops here,is bios PCIe enum failed?
How can I change it back or restart it?

First time powering up, stop here.

DISPLAY_LEVEL   2 (0:basic, 1:advanced, 2:log)
BOOT_MODE       1 (0:rshim, 1:emmc, 2:emmc-boot-swap)
BOOT_TIMEOUT    150 (seconds)
DROP_MODE       0 (0:normal, 1:drop)
SW_RESET        0 (1: reset)
DEV_NAME        usb-2.1
DEV_INFO        BlueField-3(Rev 1)
OPN_STR         N/A
UP_TIME         551(s)
---------------------------------------
             Log Messages
---------------------------------------
 INFO[PSC]: PSC BL1 START
 INFO[BL2]: start
 INFO[BL2]: boot mode (emmc)
 INFO[BL2]: VDDQ adjustment complete
 INFO[BL2]: VDD: 1120 mV
 INFO[BL2]: DDR POST passed
 INFO[BL2]: UEFI loaded
 INFO[BL31]: start
 INFO[BL31]: lifecycle GA Secured
 INFO[BL31]: VDD: 878 mV
 INFO[BL31]: runtime
 INFO[UEFI]: eMMC init
 INFO[UEFI]: eMMC probed
 INFO[UEFI]: UPVS valid
 INFO[UEFI]: PCIe enum start
 INFO[UEFI]: PCIe enum end
 INFO[UEFI]: DPU-BMC RF credentials not found
 INFO[UEFI]: exit Boot Service
root@dpu-bmc:~# 
root@dpu-bmc:~# 
root@dpu-bmc:~# 
root@dpu-bmc:~# 
root@dpu-bmc:~# 
root@dpu-bmc:~# 
root@dpu-bmc:~# 
root@dpu-bmc:~# cat /var/log/obmc-console.log
000000
[   24.585132] mlx5_core 0000:03:00.1: print_health_info:511:(pid 0): assert_exit_ptr 0x207217a4
[   24.585132] mlx5_core 0000:03:00.1: print_health_info:511:(pid 0): assert_exit_ptr 0x207217a4
[   24.602195] mlx5_core 0000:03:00.1: print_health_info:512:(pid 0): assert_callra 0x2072178c
[   24.602195] mlx5_core 0000:03:00.1: print_health_info:512:(pid 0): assert_callra 0x2072178c
[   24.618915] mlx5_core 0000:03:00.1: print_health_info:513:(pid 0): fw_ver 32.40.1000
[   24.618915] mlx5_core 0000:03:00.1: print_health_info:513:(pid 0): fw_ver 32.40.1000
[   24.634411] mlx5_core 0000:03:00.1: print_health_info:515:(pid 0): time 0
[   24.634411] mlx5_core 0000:03:00.1: print_health_info:515:(pid 0): time 0
[   24.647992] mlx5_core 0000:03:00.1: print_health_info:516:(pid 0): hw_id 0x0001021c
[   24.647992] mlx5_core 0000:03:00.1: print_health_info:516:(pid 0): hw_id 0x0001021c
[   24.663312] mlx5_core 0000:03:00.1: print_health_info:517:(pid 0): rfr 0
[   24.663312] mlx5_core 0000:03:00.1: print_health_info:517:(pid 0): rfr 0
[   24.676740] mlx5_core 0000:03:00.1: print_health_info:518:(pid 0): severity 3 (ERROR)
[   24.676740] mlx5_core 0000:03:00.1: print_health_info:518:(pid 0): severity 3 (ERROR)
[   24.692410] mlx5_core 0000:03:00.1: print_health_info:519:(pid 0): irisc_index 3
[   24.692410] mlx5_core 0000:03:00.1: print_health_info:519:(pid 0): irisc_index 3
[   24.707221] mlx5_core 0000:03:00.1: print_health_info:520:(pid 0): synd 0x1: firmware internal error
[   24.707221] mlx5_core 0000:03:00.1: print_health_info:520:(pid 0): synd 0x1: firmware internal error
[   24.725502] mlx5_core 0000:03:00.1: print_health_info:522:(pid 0): ext_synd 0x8e00
[   24.725502] mlx5_core 0000:03:00.1: print_health_info:522:(pid 0): ext_synd 0x8e00
[   24.740649] mlx5_core 0000:03:00.1: print_health_info:523:(pid 0): raw fw_ver 0x202803e8
[   24.740649] mlx5_core 0000:03:00.1: print_health_info:523:(pid 0): raw fw_ver 0x202803e8
[   46.352859] watchdog: BUG: soft lockup - CPU#10 stuck for 26s! [mlnx_bf_configu:951]
[   46.352859] watchdog: BUG: soft lockup - CPU#10 stuck for 26s! [mlnx_bf_configu:951]
[   70.352859] watchdog: BUG: soft lockup - CPU#10 stuck for 48s! [mlnx_bf_configu:951]
[   70.352859] watchdog: BUG: soft lockup - CPU#10 stuck for 48s! [mlnx_bf_configu:951]
[   78.972873] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 10-... } 15033 jiffies s: 197 root: 0x400/.
[   78.972873] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 10-... } 15033 jiffies s: 197 root: 0x400/.
[   78.994489] rcu: blocking rcu_node structures (internal RCU debug):
[   78.994489] rcu: blocking rcu_node structures (internal RCU debug):
[   82.416860] rcu: INFO: rcu_sched self-detected stall on CPU
[   82.416860] rcu: INFO: rcu_sched self-detected stall on CPU
[   82.428012] rcu: 	10-....: (14991 ticks this GP) idle=9f5/1/0x4000000000000002 softirq=789/789 fqs=6324 
[   82.428012] rcu: 	10-....: (14991 ticks this GP) idle=9f5/1/0x4000000000000002 softirq=789/789 fqs=6324 
[  110.352859] watchdog: BUG: soft lockup - CPU#10 stuck for 85s! [mlnx_bf_configu:951]
[  110.352859] watchdog: BUG: soft lockup - CPU#10 stuck for 85s! [mlnx_bf_configu:951]
[  134.352859] watchdog: BUG: soft lockup - CPU#10 stuck for 108s! [mlnx_bf_configu:951]
[  134.352859] watchdog: BUG: soft lockup - CPU#10 stuck for 108s! [mlnx_bf_configu:951]
[  158.352859] watchdog: BUG: soft lockup - CPU#10 stuck for 130s! [mlnx_bf_configu:951]
[  158.352859] watchdog: BUG: soft lockup - CPU#10 stuck for 130s! [mlnx_bf_configu:951]
[  182.352859] watchdog: BUG: soft lockup - CPU#10 stuck for 152s! [mlnx_bf_configu:951]
[  182.352859] watchdog: BUG: soft lockup - CPU#10 stuck for 152s! [mlnx_bf_configu:951]
[  206.352859] watchdog: BUG: soft lockup - CPU#10 stuck for 175s! [mlnx_bf_configu:951]
[  206.352859] watchdog: BUG: soft lockup - CPU#10 stuck for 175s! [mlnx_bf_configu:951]
[  230.352859] watchdog: BUG: soft lockup - CPU#10 stuck for 197s! [mlnx_bf_configu:951]
[  230.352859] watchdog: BUG: soft lockup - CPU#10 stuck for 197s! [mlnx_bf_configu:951]
[  244.856890] INFO: task kworker/3:1:112 blocked for more than 120 seconds.
[  244.856890] INFO: task kworker/3:1:112 blocked for more than 120 seconds.
[  244.870494]       Tainted: G           O L    5.15.0-1032-bluefield #34-Ubuntu
[  244.870494]       Tainted: G           O L    5.15.0-1032-bluefield #34-Ubuntu
[  244.884947] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  244.884947] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  244.900686] INFO: task kworker/3:2:408 blocked for more than 120 seconds.
[  244.900686] INFO: task kworker/3:2:408 blocked for more than 120 seconds.
[  244.914273]       Tainted: G           O L    5.15.0-1032-bluefield #34-Ubuntu
[  244.914273]       Tainted: G           O L    5.15.0-1032-bluefield #34-Ubuntu
[  244.928722] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  244.928722] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  254.352859] watchdog: BUG: soft lockup - CPU#10 stuck for 219s! [mlnx_bf_configu:951]
[  254.352859] watchdog: BUG: soft lockup - CPU#10 stuck for 219s! [mlnx_bf_configu:951]
[  262.428860] rcu: INFO: rcu_sched self-detected stall on CPU
[  262.428860] rcu: INFO: rcu_sched self-detected stall on CPU
[  262.440012] rcu: 	10-....: (59960 ticks this GP) idle=9f5/1/0x4000000000000002 softirq=789/789 fqs=25561 
[  262.440012] rcu: 	10-....: (59960 ticks this GP) idle=9f5/1/0x4000000000000002 softirq=789/789 fqs=25561 
[  269.432867] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 10-... } 62648 jiffies s: 197 root: 0x400/.
[  269.432867] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 10-... } 62648 jiffies s: 197 root: 0x400/.
[  269.454482] rcu: blocking rcu_node structures (internal RCU debug):
[  269.454482] rcu: blocking rcu_node structures (internal RCU debug):
[  290.352859] watchdog: BUG: soft lockup - CPU#10 stuck for 253s! [mlnx_bf_configu:951]
[  290.352859] watchdog: BUG: soft lockup - CPU#10 stuck for 253s! [mlnx_bf_configu:951]
[  314.352859] watchdog: BUG: soft lockup - CPU#10 stuck for 275s! [mlnx_bf_configu:951]
[  314.352859] watchdog: BUG: soft lockup - CPU#10 stuck for 275s! [mlnx_bf_configu:951]
[  338.352859] watchdog: BUG: soft lockup - CPU#10 stuck for 298s! [mlnx_bf_configu:951]
[  338.352859] watchdog: BUG: soft lockup - CPU#10 stuck for 298s! [mlnx_bf_configu:951]
[  362.352859] watchdog: BUG: soft lockup - CPU#10 stuck for 320s! [mlnx_bf_configu:951]
[  362.352859] watchdog: BUG: soft lockup - CPU#10 stuck for 320s! [mlnx_bf_configu:951]
[  365.692871] INFO: task kworker/0:1:17 blocked for more than 120 seconds.
[  365.692871] INFO: task kworker/0:1:17 blocked for more than 120 seconds.
[  365.706306]       Tainted: G           O L    5.15.0-1032-bluefield #34-Ubuntu
[  365.706306]       Tainted: G           O L    5.15.0-1032-bluefield #34-Ubuntu
[  365.720759] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  365.720759] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  365.736518] INFO: task kworker/8:2:219 blocked for more than 120 seconds.
[  365.736518] INFO: task kworker/8:2:219 blocked for more than 120 seconds.
[  365.750102]       Tainted: G           O L    5.15.0-1032-bluefield #34-Ubuntu
[  365.750102]       Tainted: G           O L    5.15.0-1032-bluefield #34-Ubuntu
[  365.764559] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  365.764559] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  365.780282] INFO: task kworker/3:2:408 blocked for more than 241 seconds.
[  365.780282] INFO: task kworker/3:2:408 blocked for more than 241 seconds.
[  365.793864]       Tainted: G           O L    5.15.0-1032-bluefield #34-Ubuntu
[  365.793864]       Tainted: G           O L    5.15.0-1032-bluefield #34-Ubuntu
[  365.808322] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  365.808322] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  365.824043] INFO: task modprobe:1458 blocked for more than 120 seconds.
[  365.824043] INFO: task modprobe:1458 blocked for more than 120 seconds.
[  365.837279]       Tainted: G           O L    5.15.0-1032-bluefield #34-Ubuntu
[  365.837279]       Tainted: G           O L    5.15.0-1032-bluefield #34-Ubuntu
[  365.851731] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  365.851731] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  386.352859] watchdog: BUG: soft lockup - CPU#10 stuck for 342s! [mlnx_bf_configu:951]
[  386.352859] watchdog: BUG: soft lockup - CPU#10 stuck for 342s! [mlnx_bf_configu:951]
[  410.352859] watchdog: BUG: soft lockup - CPU#10 stuck for 365s! [mlnx_bf_configu:951]
[  410.352859] watchdog: BUG: soft lockup - CPU#10 stuck for 365s! [mlnx_bf_configu:951]
[  434.352859] watchdog: BUG: soft lockup - CPU#10 stuck for 387s! [mlnx_bf_configu:951]
[  434.352859] watchdog: BUG: soft lockup - CPU#10 stuck for 387s! [mlnx_bf_configu:951]
[  442.440860] rcu: INFO: rcu_sched self-detected stall on CPU
[  442.440860] rcu: INFO: rcu_sched self-detected stall on CPU
[  442.452015] rcu: 	10-....: (104929 ticks this GP) idle=9f5/1/0x4000000000000002 softirq=789/789 fqs=44747 
[  442.452015] rcu: 	10-....: (104929 ticks this GP) idle=9f5/1/0x4000000000000002 softirq=789/789 fqs=44747 
[  449.656866] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 10-... } 107704 jiffies s: 197 root: 0x400/.
[  449.656866] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 10-... } 107704 jiffies s: 197 root: 0x400/.
[  449.678651] rcu: blocking rcu_node structures (internal RCU debug):
[  449.678651] rcu: blocking rcu_node structures (internal RCU debug):
[  470.352859] watchdog: BUG: soft lockup - CPU#10 stuck for 421s! [mlnx_bf_configu:951]
[  470.352859] watchdog: BUG: soft lockup - CPU#10 stuck for 421s! [mlnx_bf_configu:951]
[  486.524879] INFO: task kworker/0:1:17 blocked for more than 241 seconds.
[  486.524879] INFO: task kworker/0:1:17 blocked for more than 241 seconds.
[  486.538327]       Tainted: G           O L    5.15.0-1032-bluefield #34-Ubuntu
[  486.538327]       Tainted: G           O L    5.15.0-1032-bluefield #34-Ubuntu
[  486.552779] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  486.552779] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  486.568526] INFO: task kworker/8:2:219 blocked for more than 241 seconds.
[  486.568526] INFO: task kworker/8:2:219 blocked for more than 241 seconds.
[  486.582106]       Tainted: G           O L    5.15.0-1032-bluefield #34-Ubuntu
[  486.582106]       Tainted: G           O L    5.15.0-1032-bluefield #34-Ubuntu
[  486.596554] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  486.596554] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  486.612279] INFO: task kworker/3:2:408 blocked for more than 362 seconds.
[  486.612279] INFO: task kworker/3:2:408 blocked for more than 362 seconds.
[  486.625870]       Tainted: G           O L    5.15.0-1032-bluefield #34-Ubuntu
[  486.625870]       Tainted: G           O L    5.15.0-1032-bluefield #34-Ubuntu
[  486.640319] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  486.640319] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  486.656043] INFO: task modprobe:1458 blocked for more than 241 seconds.
[  486.656043] INFO: task modprobe:1458 blocked for more than 241 seconds.
[  486.669276]       Tainted: G           O L    5.15.0-1032-bluefield #34-Ubuntu
[  486.669276]       Tainted: G           O L    5.15.0-1032-bluefield #34-Ubuntu
[  486.683725] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  486.683725] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  494.352859] watchdog: BUG: soft lockup - CPU#10 stuck for 443s! [mlnx_bf_configu:951]
[  494.352859] watchdog: BUG: soft lockup - CPU#10 stuck for 443s! [mlnx_bf_configu:951]
[  518.352859] watchdog: BUG: soft lockup - CPU#10 stuck for 465s! [mlnx_bf_configu:951]
[  518.352859] watchdog: BUG: soft lockup - CPU#10 stuck for 465s! [mlnx_bf_configu:951]
[  542.352859] watchdog: BUG: soft lockup - CPU#10 stuck for 488s! [mlnx_bf_configu:951]
[  542.352859] watchdog: BUG: soft lockup - CPU#10 stuck for 488s! [mlnx_bf_configu:951]
root@dpu-bmc:~# 
root@dpu-bmc:~# 
root@dpu-bmc:~# 
root@dpu-bmc:~# cat /var/log/obmc-console.log.1 
t64x1  gen()  4465 MB/s
[    9.580866] raid6: int64x1  gen()  4465 MB/s
[    9.656862] raid6: int64x1  xor()  2460 MB/s
[    9.656862] raid6: int64x1  xor()  2460 MB/s
[    9.665398] raid6: using algorithm neonx2 gen() 10137 MB/s
[    9.665398] raid6: using algorithm neonx2 gen() 10137 MB/s
[    9.676365] raid6: .... xor() 8369 MB/s, rmw enabled
[    9.676365] raid6: .... xor() 8369 MB/s, rmw enabled
[    9.686299] raid6: using neon recovery algorithm
[    9.686299] raid6: using neon recovery algorithm
[    9.697529] xor: measuring software checksum speed
[    9.697529] xor: measuring software checksum speed
[    9.707728]    8regs           : 16634 MB/sec
[    9.707728]    8regs           : 16634 MB/sec
[    9.717042]    32regs          : 16599 MB/sec
[    9.717042]    32regs          : 16599 MB/sec
[    9.726132]    arm64_neon      : 25656 MB/sec
[    9.726132]    arm64_neon      : 25656 MB/sec
[    9.734843] xor: using function: arm64_neon (25656 MB/sec)
[    9.734843] xor: using function: arm64_neon (25656 MB/sec)
[    9.746873] async_tx: api initialized (async)
[    9.746873] async_tx: api initialized (async)
done.
Begin: Running /scripts/init-premount ... done.
Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done.
Begin: Running /scripts/local-premount ... [    9.831721] Btrfs loaded, crc32c=crc32c-generic, zoned=yes, fsverity=yes
[    9.831721] Btrfs loaded, crc32c=crc32c-generic, zoned=yes, fsverity=yes
Scanning for Btrfs filesystems
Sat Jan  3 06:09:54 UTC 1970
done.
Begin: Will now check root file system ... fsck from util-linux 2.37.2
[/usr/sbin/fsck.ext4 (1) -- /dev/nvme0n1p2] fsck.ext4 -a -C0 /dev/nvme0n1p2 
OS0: recovering journal
OS0: clean, 135993/7815168 files, 2184098/31245649 blocks
done.
[   10.502224] EXT4-fs (nvme0n1p2): mounted filesystem with ordered data mode. Opts: (null). Quota mode: none.
[   10.502224] EXT4-fs (nvme0n1p2): mounted filesystem with ordered data mode. Opts: (null). Quota mode: none.
done.
Begin: Running /scripts/local-bottom ... done.
Begin: Running /scripts/init-bottom ... done.
[   10.672066] systemd[1]: Inserted module 'autofs4'
[   10.672066] systemd[1]: Inserted module 'autofs4'
[   10.693876] systemd[1]: systemd 249.11-0ubuntu3.11 running in system mode (+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY -P11KIT -QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[   10.693876] systemd[1]: systemd 249.11-0ubuntu3.11 running in system mode (+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY -P11KIT -QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[   10.756472] systemd[1]: Detected architecture arm64.
[   10.756472] systemd[1]: Detected architecture arm64.
[   10.768175] systemd[1]: No hostname configured, using default hostname.
[   10.768175] systemd[1]: No hostname configured, using default hostname.
[   10.781555] systemd[1]: Hostname set to <localhost>.
[   10.781555] systemd[1]: Hostname set to <localhost>.
[   11.452269] systemd[1]: /lib/systemd/system/strongswan.service:14: Standard output type syslog is obsolete, automatically updating to journal. Please update your unit file, and consider removing the setting altogether.
[   11.452269] systemd[1]: /lib/systemd/system/strongswan.service:14: Standard output type syslog is obsolete, automatically updating to journal. Please update your unit file, and consider removing the setting altogether.
[   11.503283] systemd[1]: Configuration file /lib/systemd/system/kubelet.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway.
[   11.503283] systemd[1]: Configuration file /lib/systemd/system/kubelet.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway.
[   11.582992] systemd[1]: Queued start job for default target Graphical Interface.
[   11.582992] systemd[1]: Queued start job for default target Graphical Interface.
[   11.626251] systemd[1]: Created slice Slice /system/modprobe.
[   11.626251] systemd[1]: Created slice Slice /system/modprobe.
[   11.638858] systemd[1]: Created slice Slice /system/serial-getty.
[   11.638858] systemd[1]: Created slice Slice /system/serial-getty.
[   11.652001] systemd[1]: Created slice Slice /system/systemd-fsck.
[   11.652001] systemd[1]: Created slice Slice /system/systemd-fsck.
[   11.664950] systemd[1]: Created slice User and Session Slice.
[   11.664950] systemd[1]: Created slice User and Session Slice.
[   11.676729] systemd[1]: Started ntp-systemd-netif.path.
[   11.676729] systemd[1]: Started ntp-systemd-netif.path.
[   11.687496] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.
[   11.687496] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.
[   11.703681] systemd[1]: Started Forward Password Requests to Wall Directory Watch.
[   11.703681] systemd[1]: Started Forward Password Requests to Wall Directory Watch.
[   11.719339] systemd[1]: Set up automount Arbitrary Executable File Formats File System Automount Point.
[   11.719339] systemd[1]: Set up automount Arbitrary Executable File Formats File System Automount Point.
[   11.738423] systemd[1]: Reached target Local Encrypted Volumes.
[   11.738423] systemd[1]: Reached target Local Encrypted Volumes.
[   11.750523] systemd[1]: Reached target Slice Units.
[   11.750523] systemd[1]: Reached target Slice Units.
[   11.760439] systemd[1]: Reached target Swaps.
[   11.760439] systemd[1]: Reached target Swaps.
[   11.769284] systemd[1]: Reached target Local Verity Protected Volumes.
[   11.769284] systemd[1]: Reached target Local Verity Protected Volumes.
[   11.782662] systemd[1]: Listening on Device-mapper event daemon FIFOs.
[   11.782662] systemd[1]: Listening on Device-mapper event daemon FIFOs.
[   11.796118] systemd[1]: Listening on LVM2 poll daemon socket.
[   11.796118] systemd[1]: Listening on LVM2 poll daemon socket.
[   11.807981] systemd[1]: Listening on multipathd control socket.
[   11.807981] systemd[1]: Listening on multipathd control socket.
[   11.845844] systemd[1]: Listening on RPCbind Server Activation Socket.
[   11.845844] systemd[1]: Listening on RPCbind Server Activation Socket.
[   11.859438] systemd[1]: Listening on Syslog Socket.
[   11.859438] systemd[1]: Listening on Syslog Socket.
[   11.869667] systemd[1]: Listening on fsck to fsckd communication Socket.
[   11.869667] systemd[1]: Listening on fsck to fsckd communication Socket.
[   11.883490] systemd[1]: Listening on initctl Compatibility Named Pipe.
[   11.883490] systemd[1]: Listening on initctl Compatibility Named Pipe.
[   11.897124] systemd[1]: Listening on Journal Audit Socket.
[   11.897124] systemd[1]: Listening on Journal Audit Socket.
[   11.908504] systemd[1]: Listening on Journal Socket (/dev/log).
[   11.908504] systemd[1]: Listening on Journal Socket (/dev/log).
[   11.920785] systemd[1]: Listening on Journal Socket.
[   11.920785] systemd[1]: Listening on Journal Socket.
[   11.931206] systemd[1]: Listening on Network Service Netlink Socket.
[   11.931206] systemd[1]: Listening on Network Service Netlink Socket.
[   11.944446] systemd[1]: Listening on udev Control Socket.
[   11.944446] systemd[1]: Listening on udev Control Socket.
[   11.955621] systemd[1]: Listening on udev Kernel Socket.
[   11.955621] systemd[1]: Listening on udev Kernel Socket.
[   11.985949] systemd[1]: Mounting Huge Pages File System...
[   11.985949] systemd[1]: Mounting Huge Pages File System...
[   11.998519] systemd[1]: Mounting POSIX Message Queue File System...
[   11.998519] systemd[1]: Mounting POSIX Message Queue File System...
[   12.012699] systemd[1]: Mounting Kernel Debug File System...
[   12.012699] systemd[1]: Mounting Kernel Debug File System...
[   12.025829] systemd[1]: Mounting Kernel Trace File System...
[   12.025829] systemd[1]: Mounting Kernel Trace File System...
[   12.039749] systemd[1]: Starting Journal Service...
[   12.039749] systemd[1]: Starting Journal Service...
[   12.049828] systemd[1]: Condition check resulted in Kernel Module supporting RPCSEC_GSS being skipped.
[   12.049828] systemd[1]: Condition check resulted in Kernel Module supporting RPCSEC_GSS being skipped.
[   12.070561] systemd[1]: Starting Set the console keyboard layout...
[   12.070561] systemd[1]: Starting Set the console keyboard layout...
[   12.084964] systemd[1]: Starting Create List of Static Device Nodes...
[   12.084964] systemd[1]: Starting Create List of Static Device Nodes...
[   12.099731] systemd[1]: Starting Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling...
[   12.099731] systemd[1]: Starting Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling...
[   12.120839] systemd[1]: Condition check resulted in LXD - agent being skipped.
[   12.120839] systemd[1]: Condition check resulted in LXD - agent being skipped.
[   12.136932] systemd[1]: Starting Load Kernel Module configfs...
[   12.136932] systemd[1]: Starting Load Kernel Module configfs...
[   12.150402] systemd[1]: Starting Load Kernel Module drm...
[   12.150402] systemd[1]: Starting Load Kernel Module drm...
[   12.163231] systemd[1]: Starting Load Kernel Module efi_pstore...
[   12.163231] systemd[1]: Starting Load Kernel Module efi_pstore...
[   12.177088] systemd[1]: Starting Load Kernel Module fuse...
[   12.177088] systemd[1]: Starting Load Kernel Module fuse...
[   12.188675] systemd[1]: Condition check resulted in File System Check on Root Device being skipped.
[   12.188675] systemd[1]: Condition check resulted in File System Check on Root Device being skipped.
[   12.208977] systemd[1]: Starting Load Kernel Modules...
[   12.208977] systemd[1]: Starting Load Kernel Modules...
[   12.221265] systemd[1]: Starting Remount Root and Kernel File Systems...
[   12.221265] systemd[1]: Starting Remount Root and Kernel File Systems...
[   12.228894] EXT4-fs (nvme0n1p2): re-mounted. Opts: (null). Quota mode: none.
[   12.228894] EXT4-fs (nvme0n1p2): re-mounted. Opts: (null). Quota mode: none.
[   12.250452] systemd[1]: Starting Coldplug All udev Devices...
[   12.250452] systemd[1]: Starting Coldplug All udev Devices...
[   12.264853] systemd[1]: Started Journal Service.
[   12.264853] systemd[1]: Started Journal Service.
[   12.304994] alua: device handler registered
[   12.304994] alua: device handler registered
[   12.307024] systemd-journald[570]: Received client request to flush runtime journal.
[   12.307024] systemd-journald[570]: Received client request to flush runtime journal.
[   23.164883] mlx5_core 0000:03:00.0: poll_health:997:(pid 0): device's health compromised - reached miss count
[   23.164883] mlx5_core 0000:03:00.0: poll_health:997:(pid 0): device's health compromised - reached miss count
[   23.184775] mlx5_core 0000:03:00.0: print_health_info:504:(pid 0): Health issue observed, firmware internal error, severity(3) ERROR:
[   23.184775] mlx5_core 0000:03:00.0: print_health_info:504:(pid 0): Health issue observed, firmware internal error, severity(3) ERROR:
[   23.208802] mlx5_core 0000:03:00.0: print_health_info:508:(pid 0): assert_var[0] 0x00000000
[   23.208802] mlx5_core 0000:03:00.0: print_health_info:508:(pid 0): assert_var[0] 0x00000000
[   23.225517] mlx5_core 0000:03:00.0: print_health_info:508:(pid 0): assert_var[1] 0x00000000
[   23.225517] mlx5_core 0000:03:00.0: print_health_info:508:(pid 0): assert_var[1] 0x00000000
[   23.242232] mlx5_core 0000:03:00.0: print_health_info:508:(pid 0): assert_var[2] 0x00000000
[   23.242232] mlx5_core 0000:03:00.0: print_health_info:508:(pid 0): assert_var[2] 0x00000000
[   23.258945] mlx5_core 0000:03:00.0: print_health_info:508:(pid 0): assert_var[3] 0x00000000
[   23.258945] mlx5_core 0000:03:00.0: print_health_info:508:(pid 0): assert_var[3] 0x00000000
[   23.275660] mlx5_core 0000:03:00.0: print_health_info:508:(pid 0): assert_var[4] 0x00000000
[   23.275660] mlx5_core 0000:03:00.0: print_health_info:508:(pid 0): assert_var[4] 0x00000000
[   23.292373] mlx5_core 0000:03:00.0: print_health_info:508:(pid 0): assert_var[5] 0x00000000
[   23.292373] mlx5_core 0000:03:00.0: print_health_info:508:(pid 0): assert_var[5] 0x00000000
[   23.309093] mlx5_core 0000:03:00.0: print_health_info:511:(pid 0): assert_exit_ptr 0x207217a4
[   23.309093] mlx5_core 0000:03:00.0: print_health_info:511:(pid 0): assert_exit_ptr 0x207217a4
[   23.326156] mlx5_core 0000:03:00.0: print_health_info:512:(pid 0): assert_callra 0x2072178c
[   23.326156] mlx5_core 0000:03:00.0: print_health_info:512:(pid 0): assert_callra 0x2072178c
[   23.342875] mlx5_core 0000:03:00.0: print_health_info:513:(pid 0): fw_ver 32.40.1000
[   23.342875] mlx5_core 0000:03:00.0: print_health_info:513:(pid 0): fw_ver 32.40.1000
[   23.358376] mlx5_core 0000:03:00.0: print_health_info:515:(pid 0): time 0
[   23.358376] mlx5_core 0000:03:00.0: print_health_info:515:(pid 0): time 0
[   23.371959] mlx5_core 0000:03:00.0: print_health_info:516:(pid 0): hw_id 0x0001021c
[   23.371959] mlx5_core 0000:03:00.0: print_health_info:516:(pid 0): hw_id 0x0001021c
[   23.387289] mlx5_core 0000:03:00.0: print_health_info:517:(pid 0): rfr 0
[   23.387289] mlx5_core 0000:03:00.0: print_health_info:517:(pid 0): rfr 0
[   23.400694] mlx5_core 0000:03:00.0: print_health_info:518:(pid 0): severity 3 (ERROR)
[   23.400694] mlx5_core 0000:03:00.0: print_health_info:518:(pid 0): severity 3 (ERROR)
[   23.416373] mlx5_core 0000:03:00.0: print_health_info:519:(pid 0): irisc_index 3
[   23.416373] mlx5_core 0000:03:00.0: print_health_info:519:(pid 0): irisc_index 3
[   23.431180] mlx5_core 0000:03:00.0: print_health_info:520:(pid 0): synd 0x1: firmware internal error
[   23.431180] mlx5_core 0000:03:00.0: print_health_info:520:(pid 0): synd 0x1: firmware internal error
[   23.449460] mlx5_core 0000:03:00.0: print_health_info:522:(pid 0): ext_synd 0x8e00
[   23.449460] mlx5_core 0000:03:00.0: print_health_info:522:(pid 0): ext_synd 0x8e00
[   23.464607] mlx5_core 0000:03:00.0: print_health_info:523:(pid 0): raw fw_ver 0x202803e8
[   23.464607] mlx5_core 0000:03:00.0: print_health_info:523:(pid 0): raw fw_ver 0x202803e8
[   24.440882] mlx5_core 0000:03:00.1: poll_health:997:(pid 0): device's health compromised - reached miss count
[   24.440882] mlx5_core 0000:03:00.1: poll_health:997:(pid 0): device's health compromised - reached miss count
[   24.460761] mlx5_core 0000:03:00.1: print_health_info:504:(pid 0): Health issue observed, firmware internal error, severity(3) ERROR:
[   24.460761] mlx5_core 0000:03:00.1: print_health_info:504:(pid 0): Health issue observed, firmware internal error, severity(3) ERROR:
[   24.484799] mlx5_core 0000:03:00.1: print_health_info:508:(pid 0): assert_var[0] 0x00000000
[   24.484799] mlx5_core 0000:03:00.1: print_health_info:508:(pid 0): assert_var[0] 0x00000000
[   24.501513] mlx5_core 0000:03:00.1: print_health_info:508:(pid 0): assert_var[1] 0x00000000
[   24.501513] mlx5_core 0000:03:00.1: print_health_info:508:(pid 0): assert_var[1] 0x00000000
[   24.518240] mlx5_core 0000:03:00.1: print_health_info:508:(pid 0): assert_var[2] 0x00000000
[   24.518240] mlx5_core 0000:03:00.1: print_health_info:508:(pid 0): assert_var[2] 0x00000000
[   24.534965] mlx5_core 0000:03:00.1: print_health_info:508:(pid 0): assert_var[3] 0x00000000
[   24.534965] mlx5_core 0000:03:00.1: print_health_info:508:(pid 0): assert_var[3] 0x00000000
[   24.551700] mlx5_core 0000:03:00.1: print_health_info:508:(pid 0): assert_var[4] 0x00000000
[   24.551700] mlx5_core 0000:03:00.1: print_health_info:508:(pid 0): assert_var[4] 0x00000000
[   24.568418] mlx5_core 0000:03:00.1: print_health_info:508:(pid 0): assert_var[5] 0x00000000
[   24.568418] mlx5_core 0000:03:00.1: print_health_info:508:(pid 0): assert_var[5] 0x00root@dpu-bmc:~# 

According to the part number “9009D3B600CVAA”, this is a BF3 card and not a BF3 DPU controller.
Bifurcation is only supported on BF3 DPU controller cards.
IE: PCIe Bifurcation Configuration Options - NVIDIA Docs

The FW configuration needs to be reset via mlxconfig from the host, either revert to all default or revert this setting back to its original follow by a cold boot of the host.

You have also the option to change the BF3 card from DPU mode to NIC, and see if the host is booting.
You can then reset the firmware configuration using “mlxconfig -d reset” either from the DPU or from the host, and see if it’s operating correctly

Hi spruitt, thanks for your reply!
BF3 Supplier PN: 900-9D3B6-00CV-AA0 support bifurcation,Did I install the wrong software?
PCIe goldfinger is rc mode,host cannot discover DPU,Can only be operated through BMC.
Set NicMode failed

user@user:~/BlueField$ curl -k -u root:HelloNvidia3D! -H 'content-type: application/json' -d '{ "Attributes": { "NicMode": "NicMode" } }' -X PATCH https://192.168.1.118/redfish/v1/Systems/Bluefield/Bios/Settings
{
  "Attributes@Message.ExtendedInfo": [
    {
      "@odata.type": "#Message.v1_1_1.Message",
      "Message": "The value 'NicMode' for the property Attributes is not in the list of acceptable values.",
      "MessageArgs": [
        "NicMode",
        "Attributes"
      ],
      "MessageId": "Base.1.15.0.PropertyValueNotInList",
      "MessageSeverity": "Warning",
      "Resolution": "Choose a value from the enumeration list that the implementation can support and resubmit the request if the operation failed."
    }
  ]
}

Reinstall BFB failed(65 minutes elapsed)

user@user:~/BlueField$ curl -k -u root:HelloNvidia3D! -H "Content-Type: application/json" -X POST -d '{"TransferProtocol":"SCP", "ImageURI":"192.168.1.126//home/user/BlueField/DOCA_2.5.0_BSP_4.5.0_Ubuntu_22.04-1.23-10.prod.bfb","Targets":["redfish/v1/UpdateService/FirmwareInventory/DPU_OS"], "Username":"root"}' https://192.168.1.118/redfish/v1/UpdateService/Actions/UpdateService.SimpleUpdate
{
  "@odata.id": "/redfish/v1/TaskService/Tasks/1",
  "@odata.type": "#Task.v1_4_3.Task",
  "Id": "1",
  "TaskState": "Running",
  "TaskStatus": "OK"
}
      "MessageId": "Update.1.0.TransferringToComponent",
      "Resolution": "Transfer is still in progress (60 minutes elapsed): Please wait",
      "Severity": "OK"
    },
    {
      "@odata.type": "#MessageRegistry.v1_4_1.MessageRegistry",
      "Message": "Image 'DOCA_2.5.0_BSP_4.5.0_Ubuntu_22.04-1.23-10.prod.bfb' is being transferred to '/dev/rshim0/boot'.",
      "MessageArgs": [
        "DOCA_2.5.0_BSP_4.5.0_Ubuntu_22.04-1.23-10.prod.bfb",
        "/dev/rshim0/boot"
      ],
      "MessageId": "Update.1.0.TransferringToComponent",
      "Resolution": "Transfer is still in progress (65 minutes elapsed): Please wait",
      "Severity": "OK"
    }
  ],
  "Name": "Task 1",
  "Payload": {
    "HttpHeaders": [
      "Host: 192.168.1.118",
      "User-Agent: curl/7.81.0",
      "Accept: */*",
      "Content-Length: 209"
    ],
    "HttpOperation": "POST",
    "JsonBody": "{\n  \"ImageURI\": \"192.168.1.126//home/user/BlueField/DOCA_2.5.0_BSP_4.5.0_Ubuntu_22.04-1.23-10.prod.bfb\",\n  \"Targets\": [\n    \"redfish/v1/UpdateService/FirmwareInventory/DPU_OS\"\n  ],\n  \"TransferProtocol\": \"SCP\",\n  \"Username\": \"root\"\n}",
    "TargetUri": "/redfish/v1/UpdateService/Actions/UpdateService.SimpleUpdate"
  },
  "PercentComplete": 0,
  "StartTime": "1970-01-02T05:20:41+00:00",
  "TaskMonitor": "/redfish/v1/TaskService/Tasks/1/Monitor",
  "TaskState": "Running",
  "TaskStatus": "OK"
}