Xavier AGX "Halt: spinning forever..." on boot

Hi, I am using the Xavier AGX 32GB developer kit. I am having booting issues after modifying extlinux.conf in /boot/extLinux after adding another kernel boot option. It goes into a “Halt: spinning forever…” after attempting to load extlinux.conf. See the boot messages below.

What is causing this error? I was able to add other alternate kernels (before this one) with no issues. Is there a limit on the number of alternate kernel boot options? How do I recover the device from here?

¦¦¦¦Shutdown state requested 1
Rebooting system …
¦¦
[0000.054] W> RATCHET: MB1 binary ratchet value 4 is too large than ratchet level 2 from HW fuses.
[0000.062] I> MB1 (prd-version: 1.5.1.3-t194-41334769-d2a21c57)
[0000.067] I> Boot-mode: Coldboot
[0000.070] I> Chip revision : A02P
[0000.073] I> Bootrom patch version : 15 (correctly patched)
[0000.079] I> ATE fuse revision : 0x200
[0000.082] I> Ram repair fuse : 0x1
[0000.085] I> Ram Code : 0x2
[0000.088] I> rst_source : 0xb
[0000.090] I> rst_level : 0x1
[0000.094] I> Boot-device: eMMC
[0000.109] I> sdmmc DDR50 mode
[0000.113] W> No valid slot number is found in scratch register
[0000.118] W> Return default slot: _a
[0000.122] I> Active Boot chain : 0
[0000.125] I> Boot-device: eMMC
[0000.129] W> MB1_PLATFORM_CONFIG: device prod data is empty in MB1 BCT.
[0000.135] I> Temperature = 42500
[0000.138] W> Skipping boost for clk: BPMP_CPU_NIC
[0000.142] W> Skipping boost for clk: BPMP_APB
[0000.146] W> Skipping boost for clk: AXI_CBB
[0000.150] W> Skipping boost for clk: AON_CPU_NIC
[0000.154] W> Skipping boost for clk: CAN1
[0000.158] W> Skipping boost for clk: CAN2
[0000.163] I> Boot-device: eMMC
[0000.165] I> Boot-device: eMMC
[0000.175] I> Sdmmc: HS400 mode enabled
[0000.179] I> ECC region[0]: Start:0x0, End:0x0
[0000.183] I> ECC region[1]: Start:0x0, End:0x0
[0000.187] I> ECC region[2]: Start:0x0, End:0x0
[0000.192] I> ECC region[3]: Start:0x0, End:0x0
[0000.196] I> ECC region[4]: Start:0x0, End:0x0
[0000.200] I> Non-ECC region[0]: Start:0x80000000, End:0x100000000
[0000.206] I> Non-ECC region[1]: Start:0x0, End:0x0
[0000.210] I> Non-ECC region[2]: Start:0x0, End:0x0
[0000.214] I> Non-ECC region[3]: Start:0x0, End:0x0
[0000.219] I> Non-ECC region[4]: Start:0x0, End:0x0
[0000.224] E> FAILED: Thermal config
[0000.232] E> FAILED: MEMIO rail config
[0000.250] I> Boot-device: eMMC
[0000.260] I> sdmmc bdev is already initialized
[0000.340] I> MB1 done

¦¦¦¦main enter
SPE VERSION #: R01.00.14 Created: Sep 19 2018 @ 11:03:21
HW Function test
Start Scheduler.
in late init
¦¦
[0000.349] I> Welcome to MB2(TBoot-BPMP) (version: 00.00.2018.32-mobile-2dfe4beb)
[0000.349] I> DMA Heap @ [0x526fa000 - 0x52ffa000]
[0000.350] I> Default Heap @ [0xd486400 - 0xd48a400]
[0000.351] E> DEVICE_PROD: Invalid value data = 70020000, size = 0.
[0000.356] W> device prod register failed
[0000.360] I> Boot-device: eMMC
[0000.363] I> Boot_device: SDMMC_BOOT instance: 3
[0000.369] I> sdmmc-3 params source = boot args
[0000.372] I> sdmmc bdev is already initialized
[0000.376] I> sdmmc-3 params source = boot args
[0000.383] I> Found 17 partitions in SDMMC_BOOT (instance 3)
[0000.390] I> Found 42 partitions in SDMMC_USER (instance 3)
[0000.392] W> No valid slot number is found in scratch register
[0000.397] W> Return default slot: _a
[0000.400] I> Active Boot chain : 0
[0000.404] I> parsing oem signed section of bpmp-fw header done
[0000.409] I> bpmp-fw binary init read from storage
[0000.415] I> oem authentication of bpmp-fw header done
[0000.422] I> bpmp-fw binary done read from storage
[0000.424] I> bpmp-fw: Authentication init Done
[0000.428] I> parsing oem signed section of cpubl header done
[0000.434] I> cpubl binary init read from storage
[0000.438] I> bpmp-fw: Authentication Finalize Done
[0000.443] I> oem authentication of cpubl header done
[0000.447] I> cpubl binary done read from storage
[0000.452] I> cpubl: Authentication init Done
[0000.457] I> parsing oem signed section of rce header done
[0000.461] I> rce binary init read from storage
[0000.466] I> Relocating BR-BCT
[0000.469] I> cpubl: Authentication Finalize Done
[0000.473] I> oem authentication of rce header done
[0000.478] I> rce binary done read from storage
[0000.482] I> rce: Authentication init Done
[0000.487] I> parsing oem signed section of ape header done
[0000.491] I> ape binary init read from storage
[0000.496] I> rce: Authentication Finalize Done
[0000.500] I> oem authentication of ape header done
[0000.504] I> ape binary done read from storage
[0000.509] I> ape: Authentication init Done
[0000.515] I> parsing oem signed section of tos header done
[0000.518] I> tos binary init read from storage
[0000.522] I> ape: Authentication Finalize Done
[0000.527] I> oem authentication of tos header done
[0000.531] I> tos binary done read from storage
[0000.535] I> tos: Authentication init Done
[0000.540] I> parsing oem signed section of bpmp-fw-dtb header done
[0000.545] I> bpmp-fw-dtb binary init read from storage
[0000.550] I> tos: Authentication Finalize Done
[0000.556] I> oem authentication of bpmp-fw-dtb header done
[0000.563] I> bpmp-fw-dtb binary done read from storage
[0000.565] I> bpmp-fw-dtb: Authentication init Done
[0000.570] I> parsing oem signed section of cpubl-dtb header done
[0000.576] I> cpubl-dtb binary init read from storage
[0000.580] I> bpmp-fw-dtb: Authentication Finalize Done
[0000.618] I> oem authentication of cpubl-dtb header done
[0000.618] I> cpubl-dtb binary done read from storage
[0000.619] I> cpubl-dtb: Authentication init Done
[0000.621] I> parsing oem signed section of eks header done
[0000.621] I> eks binary init read from storage
[0000.622] I> cpubl-dtb: Authentication Finalize Done
[0000.623] I> oem authentication of eks header done
[0000.626] I> eks binary done read from storage
[0000.631] I> eks: Authentication init Done
[0000.635] I> eks: Authentication Finalize Done
[0000.639] I> EKB detected (length: 0x410) @ VA:0x5270a400
¦¦NOTICE: BL31: v1.3(release):41d46a9cf
NOTICE: BL31: Built : 21:14:52, Jun 25 2020
ipc-unittest-main: 1519: Welcome to IPC unittest!!!
ipc-unittest-main: 1531: waiting forever
ipc-unittest-srv: 329: Init unittest services!!!
hwkey-agent: 40: hwkey-agent is running!!
hwkey-agent: 182: key_mgnt_processing …
hwkey-agent: 157: Init hweky-agent services!!
platform_bootstrap_epilog: trusty bootstrap complete
¦¦

welcome to lk
calling constructors
initializing heap
creating bootstrap completion thread
top of bootstrap2()
initializing platform
bpmp: platform_init
tag is 57f8a77779f848bf2ecf21dabee5645f
tag_show initialized
dt initialized
mail initialized
chipid initialized
fuse initialized
sku initialized
speedo initialized
ec_get_ec_list: found 45 ecs
ec initialized
ec_mrq initialized
vmon_populate_monitors: found 3 monitors
vmon initialized
adc initialized
fmon_populate_monitors: found 73 monitors
fmon initialized
fmon_mrq initialized
reset initialized
nvhs initialized
392 clocks registered
WARNING: pll_c4 has no dyn ramp
clk_mrq_init: mrq handler registered
clk initialized
nvlink initialized
io_dpd initialized
io_dpd initialized
thermal initialized
i2c5 controller initialized
initialized i2c mrq handling
i2c initialized
regulator initialized
avfs_clk_platform initialized
soctherm initialized
aotag initialized
powergate initialized
dvs initialized
pm initialized
pg_late initialized
strap initialized
tag initialized
emc initialized
clk_dt initialized
avfs_ccplex_platform initialized
tj_max: dt node not found
tj_init initialized
uphy_mrq_init: mrq handler registered
uphy_dt initialized
uphy initialized
safereg_init: period 80 ms
ec_late initialized
mrq initialized
¦¦
[0001.143] I> Welcome to Cboot
¦¦fmon_post initialized
¦¦[0001.143] I> Cboot Version: t194-18fdfe28
[0001.144] I> CPU-BL Params @ 0xf2820000
[0001.144] I> 0) Base:0x00000000 Size:0x00000000
[0001.147] I> 1) Base:0xf1100000 Size:0x00100000
[0001.152] I> 2) Base:0xf2000000 Size:0x00200000
[0001.156] I> 3) Base:0xf1200000 Size:0x00200000
[0001.161] I> 4) Base:0xf1000000 Size:0x00100000
¦¦clk_set_parent failed for clk can1, parent pll_aon (-22)
clk_set_parent failed for clk can2, parent pll_aon (-22)
clk_set_parent failed for clk dmic5, parent pll_aon (-22)
clk_set_parent failed for clk i2c2, parent pll_aon (-22)
clk_set_parent failed for clk i2c8, parent pll_aon (-22)
clk_set_parent failed for clk spi2, parent pll_aon (-22)
clk_set_parent failed for clk pwm4, parent pll_aon (-22)
clk_dt_late initialized
machine_check initialized
pm_post initialized
dbells initialized
avfs_clk_platform_post initialized
dmce initialized
cvc initialized
ccplex_avfs_hw_init: nafll_cluster0: not monitored
ccplex_avfs_hw_init: nafll_cluster1: not monitored
ccplex_avfs_hw_init: nafll_cluster2: not monitored
ccplex_avfs_hw_init: nafll_cluster3: not monitored
avfs_clk_mach_post initialized
regulator_post initialized
rm initialized
sc7_diag initialized
thermal_test initialized
serial_late initialized
clk_post initialized
clk_dt_post initialized
mc_reg initialized
pg_post initialized
dyn_modules initialized
sku_debugfs initialized
speedo_debugfs initialized
adc_debugfs initialized
clk_debugfs initialized
¦¦[0001.165] I> 5) Base:0xf0f00000 Size:0x00100000
[0001.270] I> 6) Base:0xf3800000 Size:0x00400000
[0001.275] I> 7) Base:0xf1c00000 Size:0x00400000
[0001.279] I> 8) Base:0xf0e00000 Size:0x00100000
[0001.284] I> 9) Base:0xf0d00000 Size:0x00100000
[0001.288] I> 10) Base:0xf3000000 Size:0x00800000
[0001.293] I> 11) Base:0x40000000 Size:0x00040000
[0001.297] I> 12) Base:0xf0c00000 Size:0x00100000
[0001.301] I> 13) Base:0x40046000 Size:0x00002000¦¦emc_debugfs initialized
dvs_debugfs initialized
¦¦
¦¦fmon_debugfs initialized
vmon_debugfs initialized
pg_debugfs initialized
profile_fs initialized
debugfs_cons initialized
mail_fs initialized
profile initialized
cvc_debugfs initialized
dmce_debugfs initialized
ec_debugfs initialized
rm_debugfs initialized
soctherm_debug initialized
gr_reader initialized
mods initialized
dt_fs initialized
debugfs_mrq initialized
debug_mrq initialized
debug_safereg initialized
initializing target
calling apps_init()
starting app shell
entering main console loop
] ¦¦[0001.311] I> 14) Base:0x40048000 Size:0x00002000
[0001.361] I> 15) Base:0xac000000 Size:0x00004000
[0001.366] I> 16) Base:0x4004a000 Size:0x00002000
[0001.370] I> 17) Base:0xf0b00000 Size:0x00100000
[0001.374] I> 18) Base:0x4004c000 Size:0x00002000
[0001.379] I> 19) Base:0xf2200000 Size:0x00600000
[0001.383] I> 20) Base:0x4004e000 Size:0x00002000
[0001.388] I> 21) Base:0xf09d0000 Size:0x0000c000
[0001.392] I> 22) Base:0x00000000 Size:0x00000000
[0001.397] I> 23) Base:0xf09e0000 Size:0x00020000
[0001.401] I> 24) Base:0xf6000000 Size:0x02000000
[0001.406] I> 25) Base:0x40050000 Size:0x00002000
[0001.410] I> 26) Base:0x40040000 Size:0x00006000
[0001.415] I> 27) Base:0xf1800000 Size:0x00400000
[0001.419] I> 28) Base:0xf4c00000 Size:0x01400000
[0001.424] I> 29) Base:0xf1400000 Size:0x00400000
[0001.428] I> 30) Base:0xf0a00000 Size:0x00100000
[0001.433] I> 31) Base:0x00000000 Size:0x00000000
[0001.437] I> 32) Base:0xf8000000 Size:0x08000000
[0001.441] I> 33) Base:0x00000000 Size:0x00000000
[0001.446] I> 34) Base:0xf3c00000 Size:0x01000000
[0001.450] I> 35) Base:0xab000000 Size:0x01000000
[0001.455] I> 36) Base:0xa0000000 Size:0x0b000000
[0001.459] I> 37) Base:0xf2800000 Size:0x00800000
[0001.464] I> 38) Base:0x80000000 Size:0x20000000
[0001.468] I> 39) Base:0xb0000000 Size:0x08000000
[0001.473] I> 40) Base:0x00000000 Size:0x00000000
[0001.477] I> 41) Base:0x00000000 Size:0x00000000
[0001.482] I> 42) Base:0x00000000 Size:0x00000000
[0001.486] I> 43) Base:0x00000000 Size:0x00000000
[0001.491] I> 44) Base:0x00000000 Size:0x00000000
[0001.495] I> 45) Base:0x00000000 Size:0x00000000
[0001.499] GIC-SPI Target CPU: 0
[0001.502] Interrupts Init done
[0001.505] calling constructors
[0001.508] initializing heap
[0001.511] I> Heap: [0xa06905a8 … 0xab000000]
[0001.515] initializing threads
[0001.518] initializing timers
[0001.521] creating bootstrap completion thread
[0001.525] top of bootstrap2()
[0001.528] CPU: MIDR: 0x4E0F0040, MPIDR: 0x80000000
[0001.533] initializing platform
[0001.535] E> DEVICE_PROD: Invalid value data = 0, size = 0.
[0001.541] W> device prod register failed
[0001.545] I> Bl_dtb @0xaaf00000
[0001.551] W> “plugin-manager” doesn’t exist, creating
[0001.553] W> “ids” doesn’t exist, creating
[0001.557] W> “connection” doesn’t exist, creating
[0001.561] W> “configs” doesn’t exist, creating
[0001.573] I> Find /i2c@3160000’s alias i2c0
[0001.573] I> Reading eeprom i2c=0 address=0x50
[0001.599] I> Device at /i2c@3160000:0x50
[0001.599] I> Reading eeprom i2c=0 address=0x56
[0001.624] I> Device at /i2c@3160000:0x56
[0001.625] I> Find /i2c@3180000’s alias i2c2
[0001.625] I> Reading eeprom i2c=2 address=0x54
[0001.627] E> I2C: slave not found in slaves.
[0001.627] E> I2C: Could not write 0 bytes to slave: 0x00a8 with repeat start true.
[0001.628] E> I2C_DEV: Failed to send register address 0x00000000.
[0001.629] E> I2C_DEV: Could not read 256 registers of size 1 from slave 0xa8 at 0x00000000 via instance 2.
[0001.638] E> eeprom: Failed to read I2C slave device
[0001.643] I> Eeprom read failed 0x3526070d
[0001.647] I> Reading eeprom i2c=2 address=0x57
[0001.651] E> I2C: slave not found in slaves.
[0001.655] E> I2C: Could not write 0 bytes to slave: 0x00ae with repeat start true.
[0001.663] E> I2C_DEV: Failed to send register address 0x00000000.
[0001.668] E> I2C_DEV: Could not read 256 registers of size 1 from slave 0xae at 0x00000000 via instance 2.
[0001.678] E> eeprom: Failed to read I2C slave device
[0001.683] I> Eeprom read failed 0x3526070d
[0001.687] I> Reading eeprom i2c=2 address=0x52
[0001.691] E> I2C: slave not found in slaves.
[0001.695] E> I2C: Could not write 0 bytes to slave: 0x00a4 with repeat start true.
[0001.703] E> I2C_DEV: Failed to send register address 0x00000000.
[0001.708] E> I2C_DEV: Could not read 256 registers of size 1 from slave 0xa4 at 0x00000000 via instance 2.
[0001.718] E> eeprom: Failed to read I2C slave device
[0001.723] I> Eeprom read failed 0x3526070d
[0001.728] I> Find /i2c@c240000’s alias i2c1
[0001.731] I> Reading eeprom i2c=1 address=0x52
[0001.736] E> I2C: slave not found in slaves.
[0001.739] E> I2C: Could not write 0 bytes to slave: 0x00a4 with repeat start true.
[0001.747] E> I2C_DEV: Failed to send register address 0x00000000.
[0001.753] E> I2C_DEV: Could not read 256 registers of size 1 from slave 0xa4 at 0x00000000 via instance 1.
[0001.762] E> eeprom: Retry to read I2C slave device.
[0001.767] E> I2C: slave not found in slaves.
[0001.771] E> I2C: Could not write 0 bytes to slave: 0x00a4 with repeat start true.
[0001.779] E> I2C_DEV: Failed to send register address 0x00000000.
[0001.784] E> I2C_DEV: Could not read 256 registers of size 1 from slave 0xa4 at 0x00000000 via instance 1.
[0001.794] E> eeprom: Failed to read I2C slave device
[0001.799] I> Eeprom read failed 0x3526070d
[0001.803] I> Reading eeprom i2c=1 address=0x50
[0001.807] E> I2C: slave not found in slaves.
[0001.811] E> I2C: Could not write 0 bytes to slave: 0x00a0 with repeat start true.
[0001.819] E> I2C_DEV: Failed to send register address 0x00000000.
[0001.824] E> I2C_DEV: Could not read 256 registers of size 1 from slave 0xa0 at 0x00000000 via instance 1.
[0001.834] E> eeprom: Retry to read I2C slave device.
[0001.839] E> I2C: slave not found in slaves.
[0001.843] E> I2C: Could not write 0 bytes to slave: 0x00a0 with repeat start true.
[0001.851] E> I2C_DEV: Failed to send register address 0x00000000.
[0001.856] E> I2C_DEV: Could not read 256 registers of size 1 from slave 0xa0 at 0x00000000 via instance 1.
[0001.866] E> eeprom: Failed to read I2C slave device
[0001.871] I> Eeprom read failed 0x3526070d
[0001.875] I> create_pm_ids: id: 2888-0004-400-L, len: 15
[0001.880] I> config: mem-type:00,power-config:00,misc-config:00,modem-config:00,touch-config:00,display-config:00, len: 93
[0001.891] I> create_pm_ids: id: 2822-0000-700-K, len: 15
[0001.896] I> config: mem-type:00,power-config:00,misc-config:00,modem-config:00,touch-config:00,display-config:00, len: 93
[0001.907] I> Adding plugin-manager/ids/2888-0004-400=/i2c@3160000:module@0x50
[0001.915] W> “i2c@3160000” doesn’t exist, creating
[0001.919] W> “module@0x50” doesn’t exist, creating
[0001.923] I> Adding plugin-manager/ids/2822-0000-700=/i2c@3160000:module@0x56
[0001.930] W> “module@0x56” doesn’t exist, creating
[0001.937] I> Adding plugin-manager/cvm
[0001.939] W> “chip-id” doesn’t exist, creating
[0001.943] I> Adding plugin-manager/chip-id/A02P
[0001.947] I> Plugin-manager override starting
[0001.953] I> node /plugin-manager/fragement-tegra-wdt-en matches
[0001.961] I> node /plugin-manager/fragement-soft-wdt matches
[0001.971] I> node /plugin-manager/fragment-pcie-c5-rp matches
[0001.975] I> node /plugin-manager/fragment-tegra-ufs-lane10 matches
[0001.986] I> Disable plugin-manager status in FDT
[0001.986] I> Plugin-manager override finished successfully
[0001.987] I> gpio framework initialized
[0001.989] I> tegrabl_gpio_driver_register: register ‘nvidia,tegra194-gpio’ driver
[0001.996] I> tegrabl_gpio_driver_register: register ‘nvidia,tegra194-gpio-aon’ driver
[0002.003] I> tegrabl_tca9539_init: i2c bus: 1, slave addr: 0x46
[0002.010] E> fetch_driver_phandle_from_dt: failed to get node with compatible ti,tca9539
[0002.019] E> fetch_driver_phandle_from_dt: failed to get node with compatible nxp,tca9539
[0002.025] W> tegrabl_tca9539_init: failed to fetch phandle from dt
[0002.031] I> tegrabl_tca9539_init: i2c bus: 1, slave addr: 0x44
[0002.038] E> fetch_driver_phandle_from_dt: failed to get node with compatible ti,tca9539
[0002.046] E> fetch_driver_phandle_from_dt: failed to get node with compatible nxp,tca9539
[0002.053] W> tegrabl_tca9539_init: failed to fetch phandle from dt
[0002.060] I> fixed regulator driver initialized
[0002.070] I> register ‘maxim’ power off handle
[0002.071] I> virtual i2c enabled
[0002.072] I> registered ‘maxim,max20024’ pmic
[0002.075] I> tegrabl_gpio_driver_register: register ‘max20024-gpio’ driver
[0002.081] I> Boot-device: eMMC
[0002.084] I> Boot_device: SDMMC_BOOT instance: 3
[0002.093] I> sdmmc-3 params source = boot args
[0002.093] I> create_pm_ids: id: 2888-0004-400-L, len: 15
[0002.098] I> config: mem-type:00,power-config:00,misc-config:00,modem-config:00,touch-config:00,display-config:00, len: 93
[0002.109] I> create_pm_ids: id: 2822-0000-700-K, len: 15
[0002.114] I> config: mem-type:00,power-config:00,misc-config:00,modem-config:00,touch-config:00,display-config:00, len: 93
[0002.125] I> sdmmc bdev is already initialized
[0002.130] I> sdmmc-3 params source = boot args
[0002.160] I> Found 17 partitions in SDMMC_BOOT (instance 3)
[0002.173] I> Found 42 partitions in SDMMC_USER (instance 3)
[0002.182] I> enabling ‘vdd-hdmi-5v0’ regulator
[0002.187] I> regulator ‘vdd-hdmi-5v0’ already enabled
[0002.188] I> hdmi cable connected
[0002.190] W> set volts not configured for ‘vdd-1v0’
[0002.192] W> set volts not configured for ‘vdd-1v8-hs’
[0002.195] E> invalid display type
[0002.199] E> invalid display type
[0002.200] E> cannot find any other nvdisp nodes
[0002.216] I> edid read success
[0002.228] I> edid read success
[0002.228] I> width = 640, height = 480, frequency = 25174825
[0002.229] I> width = 640, height = 480, frequency = 25174825
[0002.229] I> width = 640, height = 480, frequency = 25174825
[0002.230] I> width = 2560, height = 1440, frequency = 296730000
[0002.230] I> width = 1920, height = 1080, frequency = 148500000
[0002.234] I> width = 720, height = 480, frequency = 27000000
[0002.240] I> width = 640, height = 480, frequency = 25174825
[0002.245] I> width = 720, height = 480, frequency = 26973026
[0002.251] I> width = 720, height = 480, frequency = 26973026
[0002.256] I> width = 720, height = 576, frequency = 26973026
[0002.262] I> width = 720, height = 576, frequency = 26973026
[0002.267] I> width = 1280, height = 720, frequency = 74175824
[0002.273] I> width = 1280, height = 720, frequency = 74175824
[0002.278] I> width = 1920, height = 1080, frequency = 148351648
[0002.284] I> width = 1920, height = 1080, frequency = 148351648
[0002.290] I> Best mode Width = 1920, Height = 1080, freq = 148351648
[0002.300] I> hdmi_enable, starting HDMI initialisation
[0002.305] I> hdmi_enable, HDMI initialisation complete
[0002.316] I> Load in CBoot Boot Options partition and parse it
[0002.322] E> Error -9 when finding node with path /boot-configuration
[0002.323] E> tegrabl_cbo_parse_info: “boot-configuration” not found in CBO file.
[0002.325] I> Hit any key to stop autoboot: 4 3 2 1
[0004.333] initializing target
[0004.333] calling apps_init()
[0004.334] starting app kernel_boot_app
[0004.353] I> found decompressor handler: lz4-legacy
[0004.354] I> decompressing BMP blob …
[0004.357] I> Kernel type = Normal
[0004.358] I> Loading kernel-bootctrl from partition
[0004.358] I> Loading partition kernel-bootctrl at 0xa4ac0000 from device(0x1)
[0004.365] W> tegrabl_get_kernel_bootctrl: magic number(0x00000000) is invalid
[0004.365] W> tegrabl_get_kernel_bootctrl: use default dummy boot control data
[0004.366] I> ########## SD boot ##########
[0004.370] I> No sdcard
[0004.372] I> -0 params source =
[0004.375] E> Blockdev open: exit error
[0004.378] E> SD boot failed, err: 724238353
[0004.382] I> ########## USB boot ##########
[0004.391] W> No valid slot number is found in scratch register
[0004.392] W> Return default slot: _a
[0004.409] I> USB Firmware Version: 60.06 release
[0004.464] I> regulator of usb2-0 already enabled
[0004.472] I> regulator of usb2-1 already enabled
[0004.479] I> regulator of usb2-2 already enabled
[0004.488] I> enabling ‘vdd-5v-sata’ regulator
[0005.556] I> USB 2.0 port 4 new low-speed USB device detected
[0005.557] W> Change usb2 root port id to 1
[0005.557] W> Change usb2 root port id to 4
[0005.558] W> WARNING: event and command not matching, cmd_trb_ptr = 0xa0e9f800, cmd_ring.dma = 0xa0e9f840
[0005.559] E> slot id is 1
[0005.560] W> handle_command_completion_event: WARNING: Command was not successfully completed (0x11)
[0006.569] W> xusbh_wait_irq: Timed out! status = 0x00
[0007.573] W> xusbh_wait_irq: Timed out! status = 0x00
[0007.574] W> failed to enumerate usb device
[0007.574] E> failed to start xhci controller
[0007.574] E> Error in init of XUSB host driver, err: 7979000c
[0007.575] E> Failed to initialize device 5-0
[0007.575] E> USB boot failed, err: 2037973004
[0007.575] I> ########## Fixed storage boot ##########
[0007.580] I> Already published: 00010003
[0007.584] I> Look for boot partition
[0007.587] I> Fallback: assuming 0th partition is boot partition
[0007.593] I> Detect filesystem
[0007.620] I> Loading extlinux.conf …
[0007.621] I> rootfs path: /sdmmc_user/boot/extlinux/extlinux.conf
[0007.657]
[0007.658] -----------------------------------------------
[0007.659] Synchronous Exception: DATA ABORT (FAR: 6)
[0007.660] -----------------------------------------------
[0007.662] PAR_ELX: 0x80f
[0007.662]
[0007.663] ESR 0x96000007: ec 0x25, il 0x1, iss 0x7
[0007.664] -----------------------------------------------
[0007.665] [Stack Trace]
[0007.666]
[0007.666] => pc:0xA0644764, sp:0xA0E9F360
[0007.670] => pc:0xA063EE30, sp:0xA0E9F590
[0007.673] => pc:0xA063F160, sp:0xA0E9F620
[0007.677] => pc:0xA060F450, sp:0xA0E9F690
[0007.681] => pc:0xA060F5EC, sp:0xA0E9F730
[0007.685] => pc:0xA060F600, sp:0xA0E9F7A0
[0007.689] => pc:0xA060F284, sp:0xA0E9F7E0
[0007.693] => pc:0xA060E7F8, sp:0xA0E9F7F0
[0007.696] => pc:0xA060E7CC, sp:0xA0E9F800
[0007.700] -----------------------------------------------
[0007.706] iframe 0xa0e9f270:
[0007.708] x0 0x 6 x1 0x a4ab863b x2 0x 7 x3 0x 6b
[0007.717] x4 0x 0 x5 0x 70 x6 0x a x7 0xffffffffffffffff
[0007.727] x8 0x d0 x9 0x 1 x10 0x 2f x11 0x a064d328
[0007.736] x12 0x a0690000 x13 0x a0691808 x14 0x a0692b50 x15 0x 200
[0007.745] x16 0x 1 x17 0x 438 x18 0x 0 x19 0x a0690080
[0007.754] x20 0x 5 x21 0x a0692298 x22 0x a0e9f778 x23 0x a0e9f718
[0007.763] x24 0x a4ab8e94 x25 0x a0655000 x26 0x a4ab863b x27 0x 30
[0007.772] x28 0x 6 x29 0x a0e9f590 lr 0x a0640484 sp 0x a0e9f360
[0007.781] elr 0x a0644764
[0007.784] spsr 0x 80000209
[0007.787] -----------------------------------------------
[0007.793] panic (caller 0xa0601238): die
[0007.796] HALT: spinning forever…

Is there a reason you cannot use serial console to boot one of the other kernels? Also, I don’t actually see a kernel load stage…so if there were notes on a problem during initial boot which would help, then it is missing. You might want to get a full serial console log of boot and shutdown (I could have misunderstood this log, but if it includes not just shutdown, and also includes boot, then you might want to add that detail).

Sorry for the confusion. The serial console log is showing shutdown and boot. See the logs leading up to “Rebooting system…” from the previous posted log, below:

nluong@nluong-desktop:~$ sudo shutdown -r now
[sudo] password for nluong:
[244909.990215] watchdog: watchdog0: watchdog did not stop!
[244909.996498] systemd-shutdow: 49 output lines suppressed due to ratelimiting
[244911.094480] reboot: Restarting system
¦¦¦¦Shutdown state requested 1
Rebooting system …

The reason I cannot use serial console to boot one of the other kernels is because it goes into the “HALT: spinning forever…” state before I have the option to. Under normal behavior, it loads extlinux.conf and then gives boot options. However, at [0007.620] it attempts to load extlinux.conf, but then a Synchronous Exception occurs leading to the spinning forever.

Still looking into what caused this problem, and how to recover from here. Any insight is greatly appreciated.

What did you add to extlinux.conf?

Unfortunately I cannot share the exact extlinux.conf file since the AGX will not boot. But I added something like this:

LABEL KernelIMX2
    	MENU LABEL Kernel-IMX2
    	LINUX /boot/ImageIMX2
    	INITRD /boot/initrd
    	APPEND ${cbootargs}

Note that this was the 6th boot option and was not the primary/default boot option.

I don’t remember the cbootargs off the top of my head, but it was identical to the other 5 kernels which did not cause this problem.

Then it seems this 6th kernel has something wrong and causes system hang. What change did you add to it?

In /hardware/nvidia/platform/t19x/galen/kernel-dts/tegra194-p2888-0001-p2822-0000.dtsi, I replaced

#include "common/tegra194-p2822-camera-modules.dtsi"
#include "t19x-common-modules/tegra194-camera-plugin-manager.dtsi"

with

#include "common/tegra194-p2822-0000-camera-imx185_v1-a00.dtsi"
#include "common/tegra194-p2822-0000-camera-vivid.dtsi"

I was under the impression that extlinux.conf allows you to add “bad” kernels and still lets you choose to boot a good kernel. Is this not the case?

If you still have the other entries in extlinux.conf, and a serial console to pick the other entries at the correct moment in boot, then you can use any entry you want and get around the failing kernel.

One stipulation is that if you replaced any of the original boot content, e.g., if the kernel Image original there is overwritten, or if the initrd previously there is overwritten, then the above falls apart since using older content depends on the content still being there. Are you able to interrupt boot at the right moment via serial console?

Yes, the other entries are still in extlinux.conf. However, part of the problem is that the system hangs before the kernel selection option.

I am confident I did not overwrite the original kernel image, or initrd. The moved image file name does not conflict with the original image file name.

I can “Hit any key to stop autoboot”. But, for the reasons already stated I cannot select a kernel.

It sounds little weird that this happens when loading extlinux.conf.

Could you share more detail about your software/board and do some more tests to verify what is the exact way to trigger this error?

For example, adding entry one by one in extlinux.conf and then reboot to see which one is causing the error.

I am using the Jetson AGX Xavier Developer Kit 32GB with Jetpack 4.4

I looked into it some more. I recovered the AGX by putting it into recovery and then reflashing Jetpack 4.4 using SDK Manager (of course I lost all the data that was on the AGX).

It seems extlinux.conf has an issue when there are more than 5 kernel options, regardless of the image files in /boot/. When I create 6 identical boot options (primary, kernel2, kernel3, …, kernel6), the system hangs. When there is only 5 (primary, kernel2, …, kernel5), it does not. See the extlinux.conf file that causes system hang below:

TIMEOUT 30
DEFAULT primary

MENU TITLE L4T boot options

LABEL primary
      MENU LABEL primary kernel
      LINUX /boot/Image
      INITRD /boot/initrd
      APPEND ${cbootargs} quiet root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 console=ttyTCU0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 rootfstype=ext4

LABEL kernel2
      MENU LABEL kernel2
      LINUX /boot/Image
      INITRD /boot/initrd
      APPEND ${cbootargs} quiet root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 console=ttyTCU0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 rootfstype=ext4

LABEL kernel3
      MENU LABEL kernel3
      LINUX /boot/Image
      INITRD /boot/initrd
      APPEND ${cbootargs} quiet root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 console=ttyTCU0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 rootfstype=ext4

LABEL kernel4
      MENU LABEL kernel4
      LINUX /boot/Image
      INITRD /boot/initrd
      APPEND ${cbootargs} quiet root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 console=ttyTCU0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 rootfstype=ext4

LABEL kernel5
      MENU LABEL kernel5
      LINUX /boot/Image
      INITRD /boot/initrd
      APPEND ${cbootargs} quiet root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 console=ttyTCU0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 rootfstype=ext4

LABEL kernel6
      MENU LABEL kernel6
      LINUX /boot/Image
      INITRD /boot/initrd
      APPEND ${cbootargs} quiet root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 console=ttyTCU0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 rootfstype=ext4

# When testing a custom kernel, it is recommended that you create a backup of
# the original kernel and add a new entry to this file so that the device can
# fallback to the original kernel. To do this:
#
# 1, Make a backup of the original kernel
#      sudo cp /boot/Image /boot/Image.backup
#
# 2, Copy your custom kernel into /boot/Image
#
# 3, Uncomment below menu setting lines for the original kernel
#
# 4, Reboot

# LABEL backup
#    MENU LABEL backup kernel
#    LINUX /boot/Image.backup
#    INITRD /boot/initrd
#    APPEND ${cbootargs}

Also note I am able to boot the old 6th “problem” kernel when there are less than 6 options.

I don’t know if there is a limit of 5 entries, but you could be right about it. If that is the case, then you could clone the rootfs, edit the loopback mounted clone, and then flash the clone back to the Jetson (only the edit would differ from what you already have).

Hi @nicholasluong ,

I think 5 entry is sufficient for common use case. Does your usecase need more than 5 labels in extlinux.conf?

Hi @WayneWWW ,

Yes, you are right. 5 entry is sufficient for common use case. I don’t need more than 5, but as I was modifying and testing kernels I added entries instead of replacing entries. Now I know that the limit is 5, this will not be an issue. Thanks everyone for the help.

1 Like