Xavier nx (Jetpack5.1.1) system crash after "reboot" command

Hi Teams,
We are working on software reboot test with our own xavier nx platform(Jetpack5.1.1). The device experienced random crashes during our test. And those crash situation could be reproduced on xavier nx evk with Jetpack5.1.1 also.
1.system crash at MB2 boot stage
System crash after “reboot” command and system boot into MB2 stage.

[0000.916] I> Welcome to MB2(TBoot-BPMP) (version: default.t194-mobile-74494172)
[0000.917] I> DMA Heap @ [0x526fa000 - 0x52ffa000]
[0000.917] I> Default Heap @ [0xd486400 - 0xd48a400]
[0000.919] E> DEVICE_PROD: Invalid value data = 70020000, size = 0.
[0000.924] W> device prod register failed
[0000.928] I> gpio framework initialized
[0000.931] I> tegrabl_gpio_driver_register: register 'nvidia,tegra194-gpio' driver
[0000.939] I> tegrabl_gpio_driver_register: register 'nvidia,tegra194-gpio-aon' driver
[0000.947] I> No valid sdcard_params in mb1_bct
[0000.951] I> Boot_device: QSPI_FLASH instance: 0
[0000.955] I> qspi flash-0 params source = boot args
[0000.961] I> QSPI-0l initialized successfully
[0000.971] I> Found 41 partitions in QSPI_FLASH (instance 0)
[0000.972] I> Active Boot chain : 0
[0001.654] I> Relocating BR-BCT
[0001.655]  > DEVICE_PROD: device prod is not initialized.
[0001.966] I> Relocating OP-TEE dtb from: 0x6bfff240 to 0x70050000, size: 1008
[0001.967] I> [0] START: 0x80000000, SIZE: 0x2f000000
[0001.968] I> [1] START: 0xaf010000, SIZE: 0x189f0000
[0001.968] I> [2] START: 0xc7b00000, SIZE: 0xc0000
[0001.969] I> [3] START: 0xca000000, SIZE: 0x800000
[0001.969] I> dram_block larger than 80000000
[0001.972] I> [4] START: 0x100000000, SIZE: 0x180000000
[0001.983] I> Setting NS memory ranges to OP-TEE dtb finished.
[0001.986] I> found decompressor handler: lz4
[0002.215] I> EKB detected (length: 0x410) @ VA:0x526ff400
[0002.217] I> Setting EKB blob info to OPTEE dtb finished.
ÿäNOTICE:  BL31: v2.6(release):07eea4970
NOTICE:  BL31: Built : 07:55:15, Mar 19 2023
ERROR:   MPIDR 0x80000000: exception reason=0 syndrome=0xbe000000
ERROR:   **************************************
ERROR:   RAS Error in L2, ERRSELR_EL1=0x200:
ERROR:   	Status = 0xfc00640d
ERROR:   	IERR = SCF to L2 Decode Error Read: 0x64
ERROR:   SERR =nIlhegal adddesl (eofdwa e Eauxt): 0xd
ERROR:  o	Overflow (there maL be more
rrx3rs0 - Un or ec ab e ER O :   	 ncorrectable0(this i0 0atal)4000f1ERRO:   	MISC0 = 0x200 00000100000
ERROR0   	MISC1 = 00800c0000000
ExR1R:   	ADDR = 0x800  00=0a305507040
ERROR:   ********4*f***x********************** 
=E ROR: 0 RAS error handled!
ERROR:   sde
_dispatch_event returned -1
 0x00000E00RROR0   MPID8 0x8x000000: exception reason== syndrome=0xbe000000
000ERROR:   x*************************************
ERROR:   RAS Error in L2, ERRSELR EL1 0x2 0: ERROR:   	0tatus = 0xfc00640d
ERROR:   	I0RR = SCF to L2 D code Error Rea=: 0064
ERROR:   SE00 = I0le00gal addr ss (sof ware f ult= ): 0xd
E0ROR:   	Over0lo1b24w
 (ther  may be mo    err ors) - Uncorrecta0le
ERROR: 0 	Uncorrectable (this is fatal)
ER ROR:   	MIS0C0 = 0x2000000000100000x
 ERROR:    	MIS 1 = 0x0x5c0000000
ERROR:   	ADDR = 0x800000a325567160
ERR R:   **** *********************************
00RROR:   RAS error  andled! =RROR:   sdei_dispatch_eve0t returned -1
4 ER O :     IDR 0x80 000000 0xcepti0n reason=0 syndrome
0x1e000000 
  ERROR:   *******0************0*****************0
x1  R OR    RA  E ror i   2, ER=SELR_EL1=0x000:
ERROR:   	St0tus =10xfc00640d
ER OR:   	IERR = SCF0to L2 Decode E0ror Read: 0x64
E8ROR:   SERR =  llegal adxess (softw0re fa0lt0: 0xd4
ERROR: 0 	Overflow (th9re may be more errors) - Uncorrectable
ERR0OR:   	Unco0rrectable (2this is fatal)
ERROR:    	MISC0 = 0x200000000100000
ERROR:   	MISC1 = 0x805c0000006
ERROR:   	ADDR = 0x800000a305567000
ERROR:   ************************ *************
00ERROR:   RAS 0rror handled!
ERROR:   sdei_dispatch_event retu0ne00d -1
000000
x24 ERROR:   MPIDR 0x80 00000: exception reason=0 s0ndrome=0xbe000000
   ERROR:   *******0************************0***0*
ERROR:   RAS Error in L2, ERRSELR_EL1=0x2000
ERROR:   0Status = 0xfc00640d 
ERROR:   	IERR = SCF to L2 Decode Error Read: 0x64
ERROR:   SERR = Illegal address (software fault): 0xd
ERROR:   	Overflow (there may be more errors) - Un00rrecta0le
ERROR:
  	Uncorrectable (th s is=fa0al)
ERROR:   	MISC00= 0x2000000001c0000
_RROR:   	MISC1 = 0x805c0000002
ERROR:   5ADDR = 0x800000a3055670c0
ERROR:   ****=* ****************0**0***********
0RROR:   RAS error  h ndled!
E RROR: x sde0i00ispatch_e2vent ret
rned -1
         = 0x00000000000003c0
mair_el3       = 0x00000000004404ff
spsr_el3       = 0x00000000400002cIDR 0x80000000:0exxeptio0 reas0n=0 0yn0rome=4xb00000001734
ttbr0_el3      = 0x000ERROR0   **0*********1**d*****8*****
*e**s**r*_EeRROR: 3 RAS Err r  n  2, ERRSE R_EL1=0x200:0ERROR0  0	Stat0s0= 0xfc00630d
ER0OR:   0IERR = SCF to L2 Decode Er or W ite  0 63 
E RO=:   SERR = Illegal address (softw0re fault): 0xd
ERROR:   	Overflow (there may be more errors) - Uncorre_table
ERROR    	Uncorrectable0(this is fatal)
0RROR:   	MISC0 = 0x80000
ERROR:   	MISC1 = 0xf40000030
ERR0R:  0	ADDR = 0x800000_305567240
ERR R:   ****************0*************0*******
rERROR:   RAS error handled!
ERROR:   sdei0disp0tch_event preturned -1 
     = 0x0000000000000000
sctlr_el1      = 0x0000000030d50820
actlr_el1      = 0x0000000000000001
Icpacr_/el1      = 0x0000000000000000
csselr_el1     = 0x00000000T00000000
sp_el1         = 0x0000000000000000
esr_el1        = 0x0000000000000000
ttbr0_el1      = 0x0000000000000000
ttbr1_Cel1      = 0x0000000000000000
mair_e:l1       = 0x0000000000000000
amair_el1      = 0x0000000000000000
tcr_el1        = 0x000000000000 0000
tpidr_el1      = 0x0000000000000000
tpidr_el0      = 0x0000000000000000
tpidrro_el0    = 0x0000000000000000
par_el1        = 0x0000000000000000
mpidr_el1      = 0x0000000080000200
afsr0_el1      = 0x0000000000000000
afsr1_el1      = 0x0000000000000000
contextidr_ROR0   MP0DR 0x800000000 exce
ion reason=0 syndrome=0xbe 00 00 
   = 0x0000000000000000
cntp_ctl_elERR R:   **=** **0**x*****0*0*****0*****0******0
ERR0R:  0RA0 E
rror tn L2_ ERRSELR_EL1=_xe00l
ERROR:   =Stat0s = 0xf400600d
E0ROR0   	IERR0= SC0 0o L2 Decode
rror nrtte: 0x63
ERROR:  0SERR = Illegal addres0 (software fa0lt): 0xd
ERROR:   	Uncorrectable  this is fatal)
ERROR:   	MISC0 = 0x80000
ERROR:  t	MISC1 l 0xf4000003f
ERROR:   	ADDR = 0x800000a3055670c0
ERRlOR:    *********************0*****************
c0RROR: s RASeerror handled!
ERROR:   sdxi_dispatch_event retu0ned -1
dacr32_el2     = 0x0000000000000000
ifsr32_el2     = 0x0000000000000000
actlr_el1      = 0x0000000000000001
gicc_hppir     = 0x00000000000003ff
gicc_ahppir    = 0x00000000000003ff
gicc_ctlr      = 0x0000000000000000
gicd_ispendr regs (Offsets 0x200 - 0x278)
 Offset:			value
0000000000000200:		0x0000000000000000
0000000000000204:		0x0000000000000000
0000000000000208:		0x0000000000000000
000000000000020c:		0x0000000000000000
0000000000000210:		0x0000000000000000
0000000000000214:		0x0000000000000000
0000000000000218:		0x0000000000000000
000000000000021c:		0x0000000000000000
0000000000000220:		0x0000000000000000
0000000000000224:		0x0000000000000000
0000000000000228:		0x0000000000000000
000000000000022c:		0x0000000000000000
0000000000000230:		0x0000000000000000
0000000000000234:		0x0000000000000000
0000000000000238:		0x0000000000000000
000000000000023c:		0x0000000000000000
0000000000000240:		0x0000000000000000
0000000000000244:		0x0000000000000000
0000000000000248:		0x0000000000000000
000000000000024c:		0x0000000000000000
0000000000000250:		0x0000000000000000
0000000000000254:		0x0000000000000000
0000000000000258:		0x0000000000000000
000000000000025c:		0x0000000000000000
0000000000000260:		0x0000000000000000
0000000000000264:		0x0000000000000000
0000000000000268:		0x0000000000000000
000000000000026c:		0x0000000000000000
0000000000000270:		0x0000000000000000
0000000000000274:		0x0000000000000000
0000000000000278:		0x0000000000000000
000000000000027c:		0x0000000000000000
xception in EL3.
x30            = 0x000000004000f184
x0             = 0x0000000000000045
x1             = 0x000000000c198000
x2             = 0x0000000080000000
x3             = 0x00000000fb005b33
x4             = 0x000000004001dd8a
x5             = 0x0000000000000040
x6             = 0x0000000000000000
x7             = 0x0000000000000002
x8             = 0x0000000000000008
x9             = 0x0000000000000000
x10            = 0x0000000000000e3c
x11            = 0x00000000000004a4
x12            = 0x00000000cb15ad78
x13            = 0x000000000000000a
x14            = 0x00000000ffffffff
x15            = 0x0000000000000020
x16            = 0x00000000cb0501e0
x17            = 0x0000000000000000
x18            = 0x00000000cb15ae00
x19            = 0x000000004001c960
x20            = 0x00000000ffffff80
x21            = 0x0000000000000045
x22            = 0x00000000400167c0
x23            = 0x000000004001a1a0
x24            = 0x0000000000000000
x25            = 0x0000000000000471
x26            = 0x0000000000000006
x27            = 0x0000000000000001
x28            = 0x0000000040016b40
x29            = 0x00000000400169f0
scr_el3        = 0x0000000000000e3c
sctlr_el3      = 0x0000000030cd183f
cptr_el3       = 0x0000000000000000
tcr_el3        = 0x0000000080823518
daif           = 0x00000000000003c0
mair_el3       = 0x00000000004404ff
spsr_el3       = 0x00000000800002cc
elr_el3        = 0x0000000040000ef4
ttbr0_el3      = 0x000000004001dd81
esr_el3        = 0x00000000be000000
far_el3        = 0x0000000000000000
spsr_el1       = 0x0000000000000000
elr_el1        = 0x0000000000000000
spsr_abt       = 0x0000000000000000
spsr_und       = 0x0000000000000000
spsr_irq       = 0x0000000000000000
spsr_fiq       = 0x0000000000000000
sctlr_el1      = 0x0000000030d8180d
actlr_el1      = 0x0000000000000001
cpacr_el1      = 0x0000000000000000
csselr_el1     = 0x0000000000000000
sp_el1         = 0x00000000cb113720
esr_el1        = 0x0000000000000000
ttbr0_el1      = 0x00000000cb147000
ttbr1_el1      = 0x0000000000000000
mair_el1       = 0x00000000ff00ff04
amair_el1      = 0x0000000000000000
tcr_el1        = 0x0000000280803f1a
tpidr_el1      = 0x0000000000000000
tpidr_el0      = 0x0000000000000000
tpidrro_el0    = 0x0000000000000000
par_el1        = 0xff000000cb13b980
mpidr_el1      = 0x0000000080000000
afsr0_el1      = 0x0000000000000000
afsr1_el1      = 0x0000000000000000
contextidr_el1 = 0x0000000000000000
vbar_el1       = 0x00000000cb042000
cntp_ctl_el0   = 0x0000000000000000
cntp_cval_el0  = 0x0000000000000000
cntv_ctl_el0   = 0x0000000000000000
cntv_cval_el0  = 0x0000000000000000
cntkctl_el1    = 0x0000000000000000
sp_el0         = 0x00000000400169f0
isr_el1        = 0x0000000000000000
dacr32_el2     = 0x0000000000000000
ifsr32_el2     = 0x0000000000000000
actlr_el1      = 0x0000000000000001
gicc_hppir     = 0x00000000000003ff
gicc_ahppir    = 0x00000000000003ff
gicc_ctlr      = 0x00000000000001e9
gicd_ispendr regs (Offsets 0x200 - 0x278)
 Offset:			value
0000000000000200:		0x0000000000000000
0000000000000204:		0x0000000000000000
0000000000000208:		0x0000000000000000
000000000000020c:		0x0000000000000000
0000000000000210:		0x0000000000000000
0000000000000214:		0x0000000000000000
0000000000000218:		0x0000000000000000
000000000000021c:		0x0000000000000000
0000000000000220:		0x0000000000000000
0000000000000224:		0x0000000000000000
0000000000000228:		0x0000000000000000
000000000000022c:		0x0000000000000000
0000000000000230:		0x0000000000000000
0000000000000234:		0x0000000000000000
0000000000000238:		0x0000000000000000
000000000000023c:		0x0000000000000000
0000000000000240:		0x0000000000000000
0000000000000244:		0x0000000000000000
0000000000000248:		0x0000000000000000
000000000000024c:		0x0000000000000000
0000000000000250:		0x0000000000000000
0000000000000254:		0x0000000000000000
0000000000000258:		0x0000000000000000
000000000000025c:		0x0000000000000000
0000000000000260:		0x0000000000000000
0000000000000264:		0x0000000000000000
0000000000000268:		0x0000000000000000
000000000000026c:		0x0000000000000000
0000000000000270:		0x0000000000000000
0000000000000274:		0x0000000000000000
0000000000000278:		0x0000000000000000
000000000000027c:		0x0000000000000000ÿâ
bpmp: init
bpmp: tag is 128431eec76692047e1ac1ebc0392266
sku_dt_init: not sku 0x00
clk_early initialized
mail_early initialized
fuse initialized
hwwdt initialized
t194_ec_get_ec_list: found 45 ecs
ec initialized
vmon_setup_monitors: found 3 monitors
vmon initialized
adc initialized
fmon_populate_monitors: found 73 monitors
fmon initialized
mc initialized
reset initialized
nvhs initialized
uphy_early initialized
emc_early initialized
392 clocks registered
clk initialized
io_dpd initialized
thermal initialized
thermal_mrq initialized
i2c initialized
vrmon_dt_init: vrmon node not found
vrmon_chk_boot_state: found 0 rail monitors
vrmet_mrq initialized
ec_mrq initialized
fmon_mrq initialized
clk_mrq initialized
avfs_mrq initialized
mail_mrq initialized
i2c_mrq initialized
tag_mrq initialized
console_mrq initialized
mrq initialized
clk_sync_fmon_post initialized
ec_swd_poll_timer_cb: poll interval 109 above target 60
ec_swd_poll_timer_cb: failed to restart expired WDT after poll interval 109 (last 3081 start 3189 done 3190)
clk_dt_late initialized
noc_late initialized
pm_post initialized
dbells initialized
dmce initialized
cvc initialized
avfs_clk_mach_post initialized
avfs_clk_platform_post initialized
cvc_late initialized
regulator_post initialized
rm initialized
console_late initialized
clk_dt_post initialized
mc_reg initialized
pg_post initialized
profile initialized
fuse_late initialized
extras_post initialized
bpmp: init complete
entering main console loop
] ec_swd_poll_timer_cb: failed to restart expired WDT after poll interval 47 (last 3189 start 3236 done 3236)
ec_swd_poll_timer_cb: failed to restart expired WDT after poll interval 47 (last 3236 start 3283 done 3283)
ec_swd_poll_timer_cb: failed to restart expired WDT after poll interval 47 (last 3283 start 3330 done 3330)
ec_swd_poll_timer_cb: failed to restart expired WDT after poll interval 47 (last 3330 start 3377 done 3377)
ec_swd_poll_timer_cb: failed to restart expired WDT after poll interval 47 (last 3377 start 3424 done 3424) 

reboot_MB2_fail.log (26.5 KB)

2.system kernel panic with kernel initializing stage
kernel panic occurred when kernel initializing.

[    5.709021] tegra194-cpufreq ccplex: probed with BWMGR
[    5.714330] sdhci-tegra 3460000.sdhci: BWMGR client registration for eMC Successful
[    5.714944] irq: IRQ268: trimming hierarchy from :pmc@c360000
[    5.726933] usb-conn-gpio 3520000.xusb_padctl:ports:usb2-0:connector: repeated role: 0
[    5.728326] tegra194-isp5 13e10000.host1x:isp@14800000: initialized
[    5.741909] mmc0: CQHCI version 5.10
[    5.748339] tegra194-vi5 15c10000.vi: initialized
[    5.750328] tegradc 15200000.display: disp0 connected to head0->sor1
[    5.751863] nvethernet 2490000.ethernet: Adding to iommu group 27
[    5.755502] generic_infoframe_type: 0x87
[    5.762032] nvethernet 2490000.ethernet: failed to read skip mac reset flag, default 0
[    5.765420] tegradc 15200000.display: DT parsed successfully
[    5.773750] nvethernet 2490000.ethernet: failed to read MDIO address
[    5.773759] nvethernet 2490000.ethernet: setting to default DMA bit mask
[    5.779342] tegradc 15200000.display: Display dc.(____ptrval____) registered with id=0
[    5.785838] nvethernet 2490000.ethernet: set default TXQ to TC mapping
[    5.793766] tegra_nvdisp_bandwidth_register_max_config: max config iso bw = 15681600 KB/s
[    5.800380] nvethernet 2490000.ethernet: Setting default PTP RX queue
[    5.807069] tegra_nvdisp_bandwidth_register_max_config: max config EMC floor = 933000000 Hz
[    5.815080] nvethernet 2490000.ethernet: Failed to read DMA Tx ring size, using default [1024]
[    5.815086] nvethernet 2490000.ethernet: Failed to read DMA Rx ring size, using default [1024]
[    5.815111] nvethernet 2490000.ethernet: missing nvidia,pad_auto_cal_pu_offset, setting default 0
[    5.819956] mmc0: SDHCI controller on 3460000.sdhci [3460000.sdhci] using ADMA 64-bit
[    5.821744] tegra_nvdisp_bandwidth_register_max_config: max config hubclk = 300000000 Hz
[    5.829674] nvethernet 2490000.ethernet: missing nvidia,pad_auto_cal_pd_offset, setting default 0
[    5.838772] tegradc 15200000.display: vblank syncpt # 11 for dc 0
[    5.854802] nvethernet 2490000.ethernet: failed to get eqos_rx_m clk
[    5.855479] tegradc 15200000.display: vpulse3 syncpt # 12 for dc 0
[    5.863545] nvethernet 2490000.ethernet: failed to get eqos_rx_input clk
[    5.871729] tegradc 15200000.display: Bootloader disp_param detected. Detected mode: 8x4 (on 0x0mm) pclk=148350937
[    5.878757] usb 1-2: new high-speed USB device number 2 using tegra-xusb
[    5.880408] nvethernet 2490000.ethernet: failed to get eqos_tx_divider clk
[    5.881756] nvethernet 2490000.ethernet: Ethernet MAC address: 48:b0:2d:60:16:ec
[    5.887536] tegradc 15200000.display: hdmi: invalid prod list prod_list_hdmi_board
[    5.899306] tegradc 15200000.display: hdmi: tegra_hdmi_tmds_range_read(bd) failed
[    5.909684] CPU:0, Error: cbb-noc@2300000, irq=15
[    5.916616] **************************************
[    5.916620] CPU:0, Error:cbb-noc
[    5.916624] 	Error Logger		: 0
[    5.916633] 	ErrLog0			: 0x80030000
[    5.916636] 	  Transaction Type	: RD  - Read, Incrementing
[    5.916638] 	  Error Code		: SLV
[    5.916641] 	  Error Source		: Target
[    5.916648] 	  Error Description	: Target error detected by CBB slave
[    5.923210] nvethernet 2490000.ethernet: Macsec not supported/Not enabled in DT
[    5.930279] 	  AXI2APB_5 bridge error: RDFIFOF - Read Response FIFO Full interrupt
[    5.930295] 	  Packet header Lock	: 0
[    6.010068] 	  Packet header Len1	: 3
[    6.013757] 	  NOC protocol version	: version >= 2.7
[    6.018728] 	ErrLog1			: 0x35242c
[    6.022055] 	ErrLog2			: 0x0
[    6.024683] 	  RouteId		: 0x35242c
[    6.028353] 	  InitFlow		: ccroc_p2ps/I/ccroc_p2ps
[    6.032749] 	  Targflow		: host1x_p2pm/T/host1x_p2pm
[    6.037633] 	  TargSubRange		: 18
[    6.041128] 	  SeqId			: 0
[    6.043758] 	ErrLog3			: 0x4045c
[    6.047253] 	ErrLog4			: 0x0
[    6.050157] 	  Address accessed	: 0x15b4045c
[    6.054603] 	ErrLog5			: 0xb09f851
[    6.057671] 	  Non-Modify		: 0x1
[    6.061341] 	  AXI ID		: 0x16
[    6.064490] 	  Master ID		: CCPLEX
[    6.067905] 	  Security Group(GRPSEC): 0x7e
[    6.072103] 	  Cache			: 0x1 -- Bufferable 
[    6.076304] 	  Protection		: 0x2 -- Unprivileged, Non-Secure, Data Access
[    6.082872] 	  FALCONSEC		: 0x0
[    6.086021] 	  Virtual Queuing Channel(VQC): 0x0
[    6.090918] 	**************************************
[    6.095964] ------------[ cut here ]------------
[    6.100455] kernel BUG at drivers/soc/tegra/cbb/tegra194-cbb.c:2057!
[    6.106761] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP
[    6.112267] Modules linked in:
[    6.115245] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.10.104-tegra #1
[    6.121634] Hardware name: Unknown NVIDIA Jetson Xavier NX Developer Kit/NVIDIA Jetson Xavier NX Developer Kit, BIOS 3.1-32827747 03/19/2023
[    6.134151] pstate: 60400089 (nZCv daIf +PAN -UAO -TCO BTYPE=--)
[    6.140450] pc : tegra194_cbb_err_isr+0x19c/0x1b0
[    6.144913] lr : tegra194_cbb_err_isr+0x11c/0x1b0
[    6.149908] sp : ffff800010003df0
[    6.153051] x29: ffff800010003df0 x28: 0000000000000001 
[    6.158732] x27: 0000000000000080 x26: ffffc285b93e8b60 
[    6.163915] x25: ffffc285b9d3be38 x24: 0000000000000001 
[    6.169578] x23: ffffc285b96d7000 x22: ffffc285b9b5ecb0 
[    6.174917] x21: 000000000000000f x20: 0000000000000005 
[    6.180429] x19: ffffc285b9b5eca0 x18: 0000000000000010 
[    6.185596] x17: ffffc285b9491540 x16: 0000000000000068 
[    6.190951] x15: ffffc285b99d2bf0 x14: 0720072007200720 
[    6.196446] x13: 0720072007200720 x12: 0720072007200720 
[    6.202042] x11: 0720072007200720 x10: 0720072007200720 
[    6.207557] x9 : 0720072007200720 x8 : 07200720072a072a 
[    6.212810] x7 : 072a072a072a072a x6 : c0000000ffffefff 
[    6.218597] x5 : 0000000000057fa8 x4 : ffffc285b99e7968 
[    6.224003] x3 : 00000000ffffffff x2 : ffffc285b7e6e170 
[    6.229084] x1 : ffffc285b99d2680 x0 : 0000000100010001 
[    6.234679] Call trace:
[    6.236877]  tegra194_cbb_err_isr+0x19c/0x1b0
[    6.241422]  __handle_irq_event_percpu+0x68/0x2a0
[    6.245968]  handle_irq_event_percpu+0x40/0xa0
[    6.250432]  handle_irq_event+0x50/0xf0
[    6.254193]  handle_fasteoi_irq+0xc0/0x170
[    6.257978]  generic_handle_irq+0x40/0x60
[    6.262244]  __handle_domain_irq+0x70/0xd0
[    6.266186]  efi_header_end+0xb0/0xf0
[    6.269685]  el1_irq+0xd0/0x180
[    6.272663]  cpuidle_enter_state+0xb8/0x410
[    6.277118]  cpuidle_enter+0x40/0x60
[    6.280618]  call_cpuidle+0x44/0x80
[    6.283875]  do_idle+0x208/0x270
[    6.287530]  cpu_startup_entry+0x2c/0x70
[    6.291039]  rest_init+0xdc/0xe8
[    6.294276]  arch_call_rest_init+0x18/0x20
[    6.298473]  start_kernel+0x514/0x54c
[    6.302065] Code: a9446bf9 a94573fb a8c67bfd d65f03c0 (d4210000) 
[    6.308289] ---[ end trace 6308f0d78059cb83 ]---
[    6.312993] Kernel panic - not syncing: Oops - BUG: Fatal exception in interrupt
[    6.320517] SMP: stopping secondary CPUs
[    6.324123] Kernel Offset: 0x4285a7cb0000 from 0xffff800010000000
[    6.330490] PHYS_OFFSET: 0xfffff76240000000
[    6.334606] CPU features: 0x8240002,03802a30
[    6.339065] Memory Limit: none
[    6.342220] ---[ end Kernel panic - not syncing: Oops - BUG: Fatal exception in interrupt ]---
\FF\E2
[0000.025] W> RATCHET: MB1 binary ratchet value 4 is larger than ratchet level 2 from HW fuses.
[0000.033] I> MB1 (prd-version: 2.6.0.0-t194-41334769-cab45716)
[0000.038] I> Boot-mode: Coldboot
[0000.041] I> Platform: Silicon
[0000.044] I> Chip revision : A02P
[0000.047] I> Bootrom patch version : 15 (correctly patched)
[0000.052] I> ATE fuse revision : 0x200
[0000.056] I> Ram repair fuse : 0x0
[0000.059] I> Ram Code : 0x0
[0000.061] I> rst_source: 0x2, rst_level: 0x1
[0000.066] I> Boot-device: QSPI (instance: 0)
[0000.070] I> Qspi flash params source = brbct

kernel_panic_when_kernel_init_without_LAN.log (3.8 MB)

3.kernel panic after “reboot” command

[    8.722934] systemd[1]: Finished Load/Save Random Seed.
[    8.727080] systemd[1]: Mounted Kernel Configuration File System.
[    8.757382] systemd[1]: Finished Create System Users.
[   10.099362] using random self ethernet address
[   10.107918] using random host ethernet address
[   10.944109] using random self ethernet address
[   10.946227] using random host ethernet address
[   11.831624] imx219 9-0010: imx219_board_setup: error during i2c read probe (-121)
[   11.840391] imx219 9-0010: board setup failed
[   11.842700] imx219: probe of 9-0010 failed with error -121
[   11.880292] imx219 10-0010: imx219_board_setup: error during i2c read probe (-121)
[   11.888100] imx219 10-0010: board setup failed
[   11.893780] imx219: probe of 10-0010 failed with error -121

Ubuntu 20.04.5 LTS ubuntu ttyTCU0

ubuntu login: [   28.163840] Bridge firewalling registered
[   29.336490] nvidia: loading out-of-tree module taints kernel.
[  104.831960] watchdog: watchdog0: watchdog did not stop!
[  105.328280] Trying to unregister non-registered hwtime source
[  105.331613] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000040
[  105.332905] Mem abort info:
[  105.333989]   ESR = 0x96000004
[  105.335036]   EC = 0x25: DABT (current EL), IL = 32 bits
[  105.336180]   SET = 0, FnV = 0
[  105.337314]   EA = 0, S1PTW = 0
[  105.338423] Data abort info:
[  105.339469]   ISV = 0, ISS = 0x00000004
[  105.340551]   CM = 0, WnR = 0
[  105.341612] user pgtable: 4k pages, 48-bit VAs, pgdp=0000000118039000
[  105.342837] [0000000000000040] pgd=0000000000000000, p4d=0000000000000000
[  105.344065] Internal error: Oops: 96000004 [#1] PREEMPT SMP
[  105.345224] Modules linked in: fuse xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c xt_addrtype iptable_filter br_netfilter lzo_rle lzo_compress zram bnep overlay hid_logitech_hidpp snd_soc_tegra186_dspk snd_soc_tegra210_ope snd_soc_tegra210_iqc snd_soc_tegra210_admaif snd_soc_tegra186_asrc snd_soc_tegra186_arad snd_soc_tegra210_mvc snd_soc_tegra210_afc snd_soc_tegra210_dmic snd_soc_tegra210_mixer snd_soc_tegra210_adx snd_soc_tegra_pcm snd_soc_tegra210_amx snd_soc_tegra210_sfc snd_soc_tegra210_i2s aes_ce_blk crypto_simd cryptd aes_ce_cipher ghash_ce sha2_ce sha256_arm64 sha1_ce input_leds snd_soc_spdif_tx snd_soc_tegra_machine_driver hid_logitech_dj rtk_btusb btusb btrtl btbcm btintel leds_gpio max77620_thermal snd_soc_tegra210_adsp snd_soc_tegra_utils snd_soc_simple_card_utils tegra_bpmp_thermal snd_soc_tegra210_ahub tegra210_adma nvadsp userspace_alert nv_imx219 snd_hda_codec_hdmi snd_hda_tegra
[  105.345554]  snd_hda_codec snd_hda_core spi_tegra114 ramoops reed_solomon realtek rtl8822ce cfg80211 loop binfmt_misc ina3221 pwm_fan nvgpu nvmap ip_tables x_tables [last unloaded: mtd]
[  105.402205] CPU: 3 PID: 89 Comm: kworker/u12:1 Tainted: G           O      5.10.104-tegra #1
[  105.410852] Hardware name: Unknown NVIDIA Jetson Xavier NX Developer Kit/NVIDIA Jetson Xavier NX Developer Kit, BIOS 3.1-32827747 03/19/2023
[  105.423468] Workqueue: events_power_efficient phy_state_machine
[  105.429497] pstate: 60c00009 (nZCv daif +PAN +UAO -TCO BTYPE=--)
[  105.435538] pc : netif_carrier_off+0x1c/0x90
[  105.440251] lr : netif_carrier_off+0x1c/0x90
[  105.444453] sp : ffff8000109bbd00
[  105.447879] x29: ffff8000109bbd00 x28: ffffbbbdf1ef6000 
[  105.453663] x27: ffff263b4001bc70 x26: ffff263b4001bc20 
[  105.458904] x25: 0000000000000000 x24: 0000000000000000 
[  105.464673] x23: ffff263b5943a4f8 x22: ffff263b41058e80 
[  105.470005] x21: 0000000000000000 x20: 0000000000000000 
[  105.475260] x19: 0000000000000000 x18: 000000000000001b 
[  105.480855] x17: 0000000000000007 x16: 000000000000000e 
[  105.486112] x15: 0000000000000004 x14: 0000000000000019 
[  105.491470] x13: 0000000000000000 x12: ffff263b644a0628 
[  105.496885] x11: ffff263b644a0490 x10: 0000000000000a80 
[  105.502384] x9 : ffff263b410c457c x8 : fefefefefefefeff 
[  105.507986] x7 : 0000000000000018 x6 : ffff263b410c456c 
[  105.513313] x5 : 0000746e65696369 x4 : 000000000000002f 
[  105.518482] x3 : 0000000000000000 x2 : ffffbbbdf039e170 
[  105.523818] x1 : 0000000000000000 x0 : ffffbbbdf039e170 
[  105.529155] Call trace:
[  105.531638]  netif_carrier_off+0x1c/0x90
[  105.535397]  phy_link_change+0x70/0x80
[  105.539565]  phy_state_machine+0x184/0x220
[  105.543592]  process_one_work+0x1c4/0x4a0
[  105.547615]  worker_thread+0x54/0x430
[  105.551312]  kthread+0x148/0x170
[  105.554529]  ret_from_fork+0x10/0x24
[  105.558037] Code: aa0003f3 d50320ff aa1e03e0 97c1ce85 (f9402260) 
[  105.563922] ---[ end trace f781be445dae501a ]---
[  105.582906] Kernel panic - not syncing: Oops: Fatal exception
[  105.584696] Kernel Offset: 0x3bbde01e0000 from 0xffff800010000000
[  105.586452] PHYS_OFFSET: 0xffffd9c5c0000000
[  105.589649] CPU features: 0x8240002,03802a30
[  105.591376] Memory Limit: none
[  105.604831] ---[ end Kernel panic - not syncing: Oops: Fatal exception ]---
ÿâ
[0000.025] W> RATCHET: MB1 binary ratchet value 4 is larger than ratchet level 2 from HW fuses.
[0000.033] I> MB1 (prd-version: 2.6.0.0-t194-41334769-cab45716)
[0000.038] I> Boot-mode: Coldboot
[0000.041] I> Platform: Silicon

kernel_panic_after_reboot.log (75.7 KB)

please upgrade to rel-35.4.1. The first issue would be resolved.

For the 2nd issue, please also refer to the discussion in this post.

Thanks for the reply. We have no plans to upgrade L4T35.3.1 to L4T35.4.1 in the short term. So, Is there any way to solved this problem(first issue) on L4T35.3.1?

Not possible.

Oh sorry. I just notice it is Xavier but not Orin.

You could try to disable optee. By flashing the mon_only binary (tos-mon-only_t194.img) instead of the original one.

hi,where is mon_only binary (tos-mon-only_t194.img) that can instead of the original one?

It is in your BSP directory Linux_for_Tegra/bootloader directory on your x86 host PC.