Get back old kernel after new compiled kernel not booting

Hello,

I have a Jetson Orin Nano on which I tried to compile a new kernel. I followed the steps in the doc, unfortunately when replacing the new Image file in /boot/Image, the new kernel does not boot, and actually screen goes dark.

Is it possible to restore the older kernel image, or at least boot from a USB drive for instance and login to the system to restore it (I kept a copy of every file) ?

Thank you

Hi,

because Orin Nano can only boot from external media, you may just insert whatever the rootfs is installed in to your host PC, and replace the \boot\Image with the default one.

However, I’d suggest name the kernel image with different names, and modify \boot\extlinux\extlinux.conf to uncomment the backup entries, so even when one of the setting fails to boot, you still have another working one.

Thanks DaveYYY,

I tried actually connecting the Jetson to my Host PC (running standard Ubuntu 22.04) with USB-C, and it’s not recognized. How would you proceed to boot the jetson with external media ?

Thanks

Hi,

please directly connect the storage device you are using to the host PC.
(whether it’s an SD card/USB drive/NVMe SSD…)
Connecting the Jetson device to the host PC with USB-C is not going to show the rootfs for you.

I actually made a USB bootable ubuntu, and it is not booting from the jetson. Is there any specific way of preparing the usb ?

Hi,

I don’t know what you are doing here.

I don’t know how you made the USB, but Jetson is not going to boot with the default Ubuntu system that lacks our BSP.

Can you please just try this?

Or if you insist on booting from the USB drive, flash it with this command:

sudo ./tools/kernel_flash/l4t_initrd_flash.sh --external-device sda1 \
  -c tools/kernel_flash/flash_l4t_external.xml -p "-c bootloader/t186ref/cfg/flash_t234_qspi.xml" \
  --showlogs --network usb0 jetson-orin-nano-devkit internal

https://docs.nvidia.com/jetson/archives/r35.3.1/DeveloperGuide/text/IN/QuickStart.html

@DaveYYY thanks for the answer.

What I meant is, is it possible to create a bootable USB drive, that the Jetson will boot from.
My issue now is that the kernel is not booting, I don’t know how to access the filesystem to restore the old kernel Image.

As I told you, I found a doc where they suggest to connect the Jetson to USB C to USB-A in the host, start in recovery mode, and flash it from there, which did not work.

Can you please elaborate, and suggest a correct way of how I can do to restore the old image ? You suggested flashing the NVMe SSD, is there another more conveniant way of doing ? By the way, here is the device I’m using : https://www.seeedstudio.com/reComputer-J3011-p-5590.html

Thank you again for your help

Hi,

Yes, and just use the command I gave you with some modification in the board config file. It won’t affect the file system that is already on your device.

I’ve said, just take out the storage device and plug it in to your host PC to read the content.
Is it really that difficult for you? Or you are not allowed to/you cannot open the box of the device?

No, I suggested flashing a USB drive.

Again, can you please just try this?

Hi @DaveYYY

Thank you for your answer.
Indeed, I don’t have the authorization to open the device, so I’m really looking for a way without.

I actually tried the command :

sudo ./tools/kernel_flash/l4t_initrd_flash.sh --external-device sdc1 -c tools/kernel_flash/flash_l4t_external.xml -p "-c bootloader/t186ref/cfg/flash_t234_qspi.xml" --showlogs --network usb0 jetson-orin-nano-devkit internal

When it gets to flashing step, it’s crashing. I’m probably missing something. I can do that from a regular Ubuntu distro right ?

Here are the logs:

[  10.8472 ] ERROR: carveout /misc/carveout/aux_info@CARVEOUT_UNUSED5/ is not supported
[  10.8485 ] ERROR: carveout /misc/carveout/aux_info@CARVEOUT_UNUSED5/ is not supported
[  10.8500 ] ERROR: carveout /misc/carveout/aux_info@CARVEOUT_UNUSED5/ is not supported
[  10.8513 ] ERROR: carveout /misc/carveout/aux_info@CARVEOUT_MCE_COVERAGE/ is not supported
[  10.8523 ] ERROR: carveout /misc/carveout/aux_info@CARVEOUT_MCE_COVERAGE/ is not supported
[  10.8530 ] ERROR: carveout /misc/carveout/aux_info@CARVEOUT_MCE_COVERAGE/ is not supported
[  10.8537 ] ERROR: /misc/tsc_controls/tsc_locking_config is not supported
[  10.8542 ] ERROR: /misc/tsc_controls/tsc_locking_diff_configuration is not supported
[  10.8548 ] ERROR: /misc/tsc_controls/tsc_locking_ref_frequency_configuration is not supported
[  10.8555 ] ERROR: /misc/tsc_controls/tsc_locking_control is not supported
[  10.8559 ] ERROR: /misc/tsc_controls/tsc_locking_adjust_configuration is not supported
[  10.8565 ] ERROR: /misc/tsc_controls/tsc_locking_fast_adjust_configuration is not supported
[  10.8571 ] ERROR: /misc/tsc_controls/tsc_locking_adjust_delta_control is not supported
[  10.8578 ] ERROR: /misc/tsc_controls/tsc_capture_control_ptx is not supported
[  10.8585 ] ERROR: /misc/tsc_controls/tsc_capture_config_ptx is not supported
[  10.8598 ] ERROR: /misc/tsc_controls/tsc_stscrsr is not supported
[  10.8604 ] ERROR: /misc/tsc_controls/tsc_locking_adjust_num_control is not supported

[  10.8610 ] Parsing config file :tegra234-mb1-bct-pinmux-p3767-dp-a03_cpp.dtb 
[  10.8615 ] Added Platform Config 0 data with size :- 2416

[  10.8619 ] Parsing config file :tegra234-mb1-bct-padvoltage-p3767-dp-a03_cpp.dtb 
[  10.8635 ] WARNING: unknown node 'g2'
[  10.8635 ] WARNING: unknown node 'g2'
[  10.8635 ] WARNING: unknown node 'g9'
[  10.8635 ] WARNING: unknown node 'g9'
[  10.8635 ] Added Platform Config 2 data with size :- 24
[  10.8635 ] 
[  10.8635 ] Parsing config file :tegra234-mb1-bct-pmic-p3767-0000-a02_cpp.dtb 
[  10.8635 ] Added Platform Config 4 data with size :- 288
[  10.8635 ] 
[  10.8635 ] Parsing config file :tegra234-mb1-bct-reset-p3767-0000_cpp.dtb 
[  10.8636 ] Added Platform Config 3 data with size :- 52
[  10.8636 ] 
[  10.8636 ] Parsing config file :tegra234-mb1-bct-prod-p3767-0000_cpp.dtb 
[  10.8636 ] WARNING: unknown property 'major'
[  10.8636 ] WARNING: unknown property 'minor'
[  10.8636 ] Added Platform Config 5 data with size :- 500
[  10.8636 ] 
[  10.8636 ] Parsing config file :tegra234-mb1-bct-gpioint-p3767-0000_cpp.dtb 
[  10.8636 ] WARNING: unknown property 'major'
[  10.8636 ] WARNING: unknown property 'minor'
[  10.8636 ] Added Platform Config 7 data with size :- 380
[  10.8636 ] 
[  10.8636 ] Parsing config file :tegra234-mb1-bct-uphylane-si_cpp.dtb 
[  10.8636 ] Added Platform Config 8 data with size :- 24
[  10.8637 ] 
[  10.8637 ] Parsing config file :tegra234-mb1-bct-device-p3767-0000_cpp.dtb 
[  10.8637 ] Added Platform Config 9 data with size :- 100
[  10.8637 ] 
[  10.8637 ] Parsing config file :tegra234-mb1-bct-cprod-p3767-0000_cpp.dtb 
[  10.8637 ] ModuleCount 0 NumProdNames 0
[  10.8637 ] Added Platform Config 6 data with size :- 16
[  10.8637 ] 
[  10.8637 ] Parsing config file :tegra234-mb1-bct-ratchet-p3767-0000_cpp.dtb 
[  10.8637 ] 
[  10.8638 ] Updating mb1-bct with firmware information
[  10.8643 ] tegrabct_v2 --chip 0x23 0 --mb1bct mb1_cold_boot_bct_MB1.bct --updatefwinfo flash.xml.bin
[  10.8660 ] tegrahost_v2 --chip 0x23 0 --align mb1_cold_boot_bct_MB1_aligned.bct
[  10.8669 ] tegrahost_v2 --chip 0x23 0 --magicid MBCT --ratchet_blob ratchet_blob.bin --appendsigheader mb1_cold_boot_bct_MB1_aligned.bct zerosbk
[  10.8672 ] adding BCH for mb1_cold_boot_bct_MB1_aligned.bct
[  11.2548 ] tegrasign_v3.py --key None --list mb1_cold_boot_bct_MB1_aligned_sigheader.bct_list.xml --pubkeyhash pub_key.key --sha sha512
[  11.2549 ] Assuming zero filled SBK key
[  11.4770 ] Warning: /home/spider/Téléchargements/jetson/Linux_for_Tegra/bootloader/v3_cmac_129802.out is not found
[  11.5549 ] Traceback (most recent call last):
  File "/home/spider/Téléchargements/jetson/Linux_for_Tegra/bootloader/tegrasign_v3.py", line 499, in tegrasign
    retVal = sign_files_in_list(p_keylist, internal)
  File "/home/spider/Téléchargements/jetson/Linux_for_Tegra/bootloader/tegrasign_v3_internal.py", line 287, in sign_files_in_list
    retVal = sign_files_internal(p_keylist, child, pkh, mont, sha_type, iv)
  File "/home/spider/Téléchargements/jetson/Linux_for_Tegra/bootloader/tegrasign_v3_internal.py", line 119, in sign_files_internal
    write_file(hash_fh, buff_hash)
  File "/home/spider/Téléchargements/jetson/Linux_for_Tegra/bootloader/tegrasign_v3_util.py", line 679, in write_file
    file_handle.write(data)
TypeError: a bytes-like object is required, not 'str'

[  11.5549 ] Encounter exception when signing
[  11.5549 ] a bytes-like object is required, not 'str'
[  11.5549 ] ********* Error. Quitting. *********
Traceback (most recent call last):
  File "/home/spider/Téléchargements/jetson/Linux_for_Tegra/bootloader/./tegraflash.py", line 1383, in <module>
    tegraflash_run_commands()
  File "/home/spider/Téléchargements/jetson/Linux_for_Tegra/bootloader/./tegraflash.py", line 1217, in tegraflash_run_commands
    interpreter.onecmd(command)
  File "/usr/lib/python3.10/cmd.py", line 217, in onecmd
    return func(arg)
  File "/home/spider/Téléchargements/jetson/Linux_for_Tegra/bootloader/./tegraflash.py", line 730, in do_sign
    self.chip_inst.tegraflash_sign(exports)
  File "/home/spider/Téléchargements/jetson/Linux_for_Tegra/bootloader/tegraflash_impl_t234.py", line 2584, in tegraflash_sign
    self.tegraflash_sign_images()
  File "/home/spider/Téléchargements/jetson/Linux_for_Tegra/bootloader/tegraflash_impl_t234.py", line 541, in tegraflash_sign_images
    self.tegraflash_generate_ratchet_blob()
  File "/home/spider/Téléchargements/jetson/Linux_for_Tegra/bootloader/tegraflash_impl_t234.py", line 532, in tegraflash_generate_ratchet_blob
    self.tegraflash_generate_mb1_bct(True)
  File "/home/spider/Téléchargements/jetson/Linux_for_Tegra/bootloader/tegraflash_impl_t234.py", line 1567, in tegraflash_generate_mb1_bct
    self.tegrabct_values['--mb1_cold_boot_bct'] = self.tegraflash_oem_sign_file(
  File "/home/spider/Téléchargements/jetson/Linux_for_Tegra/bootloader/tegraflash_impl_t234.py", line 1652, in tegraflash_oem_sign_file
    with open(sign_xml_file, 'rt') as file:
FileNotFoundError: [Errno 2] No such file or directory: 'mb1_cold_boot_bct_MB1_aligned_sigheader.bct_list_signed.xml'
Error: /home/spider/Téléchargements/jetson/Linux_for_Tegra/bootloader/signed/flash.idx is not found
Error: failed to relocate images to /home/spider/Téléchargements/jetson/Linux_for_Tegra/tools/kernel_flash/images
Cleaning up...

Thank you again @DaveYYY

Hi,

please try again under paths that do not contain non-English characters.
Also, make sure sdc1 indeed points to the USB drive you are using, so stuff on sda1 and sdb1 won’t get lost.

@DaveYYY ,

Indeed, changed the location of the folder. It went to the end now. However, it says that there is no device to flash.

Using bpmp-dtb concatenated with odmdata in blob for t23x
./tegraflash.py --bl uefi_jetson_with_dtb_sigheader.bin.encrypt --bct br_bct_BR.bct --securedev  --bldtb tegra234-p3767-0000-p3768-0000-a0.dtb --applet rcm_2_encrypt.rcm --applet_softfuse rcm_1_encrypt.rcm --cmd "rcmboot"  --cfg secureflash.xml --chip 0x23 --mb1_bct mb1_bct_MB1_sigheader.bct.encrypt --mem_bct mem_rcm_sigheader.bct.encrypt --mb1_cold_boot_bct mb1_cold_boot_bct_MB1_sigheader.bct.encrypt --mb1_bin mb1_t234_prod_aligned_sigheader.bin.encrypt --psc_bl1_bin psc_bl1_t234_prod_aligned_sigheader.bin.encrypt --mem_bct_cold_boot mem_coldboot_sigheader.bct.encrypt  --bins "psc_fw pscfw_t234_prod_sigheader.bin.encrypt; mts_mce mce_flash_o10_cr_prod_sigheader.bin.encrypt; mb2_applet applet_t234_sigheader.bin.encrypt; mb2_bootloader mb2_t234_with_mb2_cold_boot_bct_MB2_sigheader.bin.encrypt; xusb_fw xusb_t234_prod_sigheader.bin.encrypt; dce_fw display-t234-dce_sigheader.bin.encrypt; nvdec nvdec_t234_prod_sigheader.fw.encrypt; bpmp_fw bpmp_t234-TE990M-A1_prod_sigheader.bin.encrypt; bpmp_fw_dtb tegra234-bpmp-3767-0000-a02-3509-a02_with_odm_sigheader.dtb.encrypt; sce_fw camera-rtcpu-sce_sigheader.img.encrypt; rce_fw camera-rtcpu-t234-rce_sigheader.img.encrypt; ape_fw adsp-fw_sigheader.bin.encrypt; spe_fw spe_t234_sigheader.bin.encrypt; tos tos-optee_t234_sigheader.img.encrypt; eks eks_t234_sigheader.img.encrypt; kernel boot.img; kernel_dtb tegra234-p3767-0000-p3768-0000-a0.dtb"    --secondary_gpt_backup  --bct_backup  --boot_chain A 
saving flash command in flashcmd.txt

*** no-flash flag enabled. Exiting now... *** 

User can run above saved command in factory environment without 
providing pkc and sbk keys to flash a device

Example:

    $ cd bootloader 
    $ sudo bash ./flashcmd.txt

Save initrd flashing command parameters to /home/spider/jetson/Linux_for_Tegra/tools/kernel_flash/initrdflashparam.txt
/tmp/tmp.g8BwVB9gBJ /home/spider/jetson/Linux_for_Tegra
writing boot image config in bootimg.cfg
extracting kernel in zImage
extracting ramdisk in initrd.img
/tmp/tmp.g8BwVB9gBJ/initrd /tmp/tmp.g8BwVB9gBJ /home/spider/jetson/Linux_for_Tegra
56174 blocs
83147 blocs
/tmp/tmp.g8BwVB9gBJ /home/spider/jetson/Linux_for_Tegra
flashimg0=boot0.img
/home/spider/jetson/Linux_for_Tegra
Success
Cleaning up...
Finish generating flash package.
No devices to flash

My usb stcik is in /dev/sdc1.

Thank you

Hi,

I just noticed that according to the vendors’ tutorial page, you should use p3509-a02+p3767-0000 as the board config file, instead of jetson-orin-nano-devkit.

@DaveYYY ,

Same issue.

./tegraflash.py --bl uefi_jetson_with_dtb_sigheader.bin.encrypt --bct br_bct_BR.bct --securedev  --bldtb tegra234-p3767-0000-p3509-a02.dtb --applet rcm_2_encrypt.rcm --applet_softfuse rcm_1_encrypt.rcm --cmd "rcmboot"  --cfg secureflash.xml --chip 0x23 --mb1_bct mb1_bct_MB1_sigheader.bct.encrypt --mem_bct mem_rcm_sigheader.bct.encrypt --mb1_cold_boot_bct mb1_cold_boot_bct_MB1_sigheader.bct.encrypt --mb1_bin mb1_t234_prod_aligned_sigheader.bin.encrypt --psc_bl1_bin psc_bl1_t234_prod_aligned_sigheader.bin.encrypt --mem_bct_cold_boot mem_coldboot_sigheader.bct.encrypt  --bins "psc_fw pscfw_t234_prod_sigheader.bin.encrypt; mts_mce mce_flash_o10_cr_prod_sigheader.bin.encrypt; mb2_applet applet_t234_sigheader.bin.encrypt; mb2_bootloader mb2_t234_with_mb2_cold_boot_bct_MB2_sigheader.bin.encrypt; xusb_fw xusb_t234_prod_sigheader.bin.encrypt; dce_fw display-t234-dce_sigheader.bin.encrypt; nvdec nvdec_t234_prod_sigheader.fw.encrypt; bpmp_fw bpmp_t234-TE990M-A1_prod_sigheader.bin.encrypt; bpmp_fw_dtb tegra234-bpmp-3767-0000-a02-3509-a02_with_odm_sigheader.dtb.encrypt; sce_fw camera-rtcpu-sce_sigheader.img.encrypt; rce_fw camera-rtcpu-t234-rce_sigheader.img.encrypt; ape_fw adsp-fw_sigheader.bin.encrypt; spe_fw spe_t234_sigheader.bin.encrypt; tos tos-optee_t234_sigheader.img.encrypt; eks eks_t234_sigheader.img.encrypt; kernel boot.img; kernel_dtb tegra234-p3767-0000-p3509-a02.dtb"    --secondary_gpt_backup  --bct_backup  --boot_chain A 
saving flash command in flashcmd.txt

*** no-flash flag enabled. Exiting now... *** 

User can run above saved command in factory environment without 
providing pkc and sbk keys to flash a device

Example:

    $ cd bootloader 
    $ sudo bash ./flashcmd.txt

Save initrd flashing command parameters to /home/spider/jetson/Linux_for_Tegra/tools/kernel_flash/initrdflashparam.txt
/tmp/tmp.jp5n8foEk8 /home/spider/jetson/Linux_for_Tegra
writing boot image config in bootimg.cfg
extracting kernel in zImage
extracting ramdisk in initrd.img
/tmp/tmp.jp5n8foEk8/initrd /tmp/tmp.jp5n8foEk8 /home/spider/jetson/Linux_for_Tegra
56174 blocs
83147 blocs
/tmp/tmp.jp5n8foEk8 /home/spider/jetson/Linux_for_Tegra
flashimg0=boot0.img
/home/spider/jetson/Linux_for_Tegra
Success
Cleaning up...
Finish generating flash package.
No devices to flash

I just want to emphasize that I partionned the device to GPT, format it in ext4, before that.
Is there anything I should do prior ? Or am I doing somethng wrong here.

Also,

I want to precize that I tried something else in parallel, I tried just copying the /rootfs content to the USB drive.
I could see it in the boot screen of the Jetson, but it could’nt boot from it.

Hi,

please always put the full log, and don’t crop it.
We also need the UART log during flash.

By the way, are you using a real Ubuntu PC to flash the device?
Please avoid using a VM in case you do.

Or you may try manipulating the rootfs on the NVMe SSD by booting into initrd:
https://docs.nvidia.com/jetson/archives/r35.3.1/DeveloperGuide/text/SD/FlashingSupport.html#cloning-rootfs-with-initrd

Hi,

Here are the logs.

And yes, I used a real PC and not a VM.

Thank you @DaveYYY
logs.txt (196.7 KB)

Hi,

again, we need UART log here.
Or you don’t know how to dump it?

If you still cannot get it to work, try either flashing again with SDK Manager, or this:

@DaveYYY ,

How would you get them in linux ? (not really familiar with that).

Also, I think I’m missing a step here. I saw in another doc, that you must turn the device in recovery mode and connect it to the Host machine. Sorry, but I’m really lost here, this is first time using Jetson devices.

Thank you !