Jetson Orin NX - board can't flash after partial fuse burn

Autonomous Machines Jetson & Embedded Systems Jetson Orin NX
,
1

Hi,

System:

  • Orin NX 16G on Forecr carrier
  • JetPack r35.5.0

What was done:

  1. I burnt my orin nx following fuses from a clean board:
  • PKC0/1/2 with RSA3K keys
  • SBK
  • OEM K1/K2
  • Boot Security Info to 0x9 (for RSA3K+SBK)

(Note that no security mode was burnt)

  1. Burn script returns success, and when I booted to my system and ran nv_fuse_read.sh I saw the PKC0/1/2 and Boot Security Info with the values I burnt.

  2. From now on, I can’t flash anything to the jetson in recovery and it always fail on passing the bct_br with the following:

[ 0.6556 ] tegrarcm_v2 --new_session --chip 0x23 0 --uid --download bct_br br_bct_BR.bct --download mb1 mb1_t234_prod_aligned_sigheader_encrypt.bin.signed --download psc_bl1 psc_bl1_t234_prod_aligned_sigheader_encrypt.bin.signed --download bct_mb1 mb1_bct_MB1_sigheader_encrypt.bct.signed
[ 0.6559 ] BR_CID: REDACTED
[ 0.7387 ] Sending bct_br
[ 0.8193 ] ERROR: might be timeout in USB write.
Error: Return value 3

I’ve checked different USB cables/systems/disabled autosuspend, and I’ve used this USB cable for long flash sequences before without any issue.
I don’t think it’s a USB cable issue, seems to me like the Orin rejects my signed+encrypted payloads from now on.
The recovery states that it works in SBKPKC mode and I pass my keys to the scripts. The issue happens with l4t_initrd_flash.sh or more concisely when I run odmfuse.sh / odmfuseread.sh.
I tried all the 3 PKCs I burnt, same issue.
The serial console is completly empty during this flash issue.

Provided logs for your reference.
my_system__orinnx_fuse_read_secured.log (17.1 KB)
my_system__orinnx_fuse_for_real.log (90.4 KB)
fuse_burn_serial_snippets.log (52.8 KB)

Thank you for your help.
Nadir.

3

hello ncs1,

had you kept the fuse configuration file, which is an XML file that contains fuse data.
may I know what’s your commands to flash a target.
besides, may I also know what’s your host environment setup.

4

Hi,

This is the xml file used: (sent it as .txt to bypass the upload restrictions)
redacted_fuse.xml.txt (469 Bytes)

To burn used the following:

sudo ./odmfuse.sh -X fuse.xml -i 0x23 --auth NS jetson-orin-nano-devkit

All the flash commands fail now, since run the bcr on the target to extract info.

sudo ./odmfuseread.sh -i 0x23 -k RSA1.key -S SBK.key jetson-orin-nano-devkit

For just qspi now,

sudo ./tools/kernel_flash/l4t_initrd_flash.sh -u RSA1.key -v SBK.key -p “-c bootloader/t186ref/cfg/flash_t234_qspi.xml” --showlogs --network usb0 --no-flash jetson-orin-nano-devkit internal

The following command which was suggested in another forum post, resulted in the same error:

sudo ADDITIONAL_DTB_OVERLAY_OPT=“BootOrderNvme.dtbo” ./tools/kernel_flash/l4t_initrd_flash.sh --no-flash --external-device nvme0n1p1 -c tools/kernel_flash/flash_l4t_external.xml -p “-c bootloader/t186ref/cfg/flash_t234_qspi.xml” -u rsa_priv-3k.pem -v sbk.key --showlogs --network usb0 jetson-orin-nano-devkit internal

My host environment is native ubuntu 24.04 for flashing. I work with L4T downloads directly and no GUI.

Asserted the same behavior exactly on ubuntu 20.04 system.

Thanks.

5

hello ncs1,

could you please also examine with tegrakeyhash utility for your PKC keys.
for example, ./tegrakeyhash --pkc <Key>

6

Hi,

I used the following to generate the fuse hash for the xml:
tegrasign_v3.py --pubkeyhash RSA1_pub.pem RSA1_pub.hash --key RSA1_priv.pem
Used the stdout where ‘tegra-fuse format (big endian)’ is outputted and asserted that the hexdump of RSA1_pub.hash is the same like it.

I’m trying to run:
tegrakeyhash --pkc RSA1_pub.pem --chip 0x23
but I get Invalid key format.

Update:
For some reason, tegrasign_v3.py command above clobber the RSA1_pub.pem, when I generate the -pubout from the priv again the file was a valid pem certificate and tegrakeyhash above ran correctly.
The tegra-fuse-format it outputted was the same as above.

Thanks.

7

hello ncs1,

as mentioned in developer guide, Prerequisites Secure Boot.

An X86 host running Ubuntu 18.04 LTS, or 20.04 LTS.

for ubuntu-22 upwards, you may add -traditional options for using the traditional PKCS#1 format to generate a key.

FYI,
we’ve confirmed by burning fuses (flash and boot-up successfully) with Orin-NX-8GB.
here’re our steps for your reference,
(1) $ sudo ./odmfuse.sh -X fuses.xml -i 0x23 jetson-orin-nano-devkit
(2) $ sudo ADDITIONAL_DTB_OVERLAY_OPT="BootOrderNvme.dtbo" ./tools/kernel_flash/l4t_initrd_flash.sh --no-flash --external-device nvme0n1p1 -c tools/kernel_flash/flash_l4t_external.xml -p "-c bootloader/t186ref/cfg/flash_t234_qspi.xml" -u rsa_priv-3k.pem -v sbk.key --showlogs --network usb0 jetson-orin-nano-devkit internal
(3) $ sudo ./tools/kernel_flash/l4t_initrd_flash.sh --flash-only -u rsa_priv-3k.pem -v sbk.key jetson-agx-orin-devkit internal

8

Hi,

  1. I’ve converted my priv.pem to traditional format and asserted that tegrasign returned the same hash on it.
  2. Tried to flash using the tradiational pem, same issue.
  3. I followed the steps you wrote here as seen in my previous comments, same issue.
  4. Please note that I tried to flash from ub20.04 system too and it failed the same.

What else can we assert/do ?

Thanks.

9

hello ncs1,

since you’re using customize OEM K1 fuse key value, had you update EKS image?
please visit jetson-linux-r3550 for downloading [Driver Package (BSP) Sources] package.
you should extract nvidia-jetson-optee-source.tbz2 package.
please refer to optee/samples/hwkey-agent/host/tool/gen_ekb/example.sh to generate a new EKS image, i.e. eks_t234.img

you may see-also Topic 270934 for steps to update EKS image accordingly.

10

Hi JerryChang,

  • generated eks_t234.img for my oemk1 already and my tests were against it.
  • in any way, I don’t think that the eks/oemk1 are related so early in the flash process as my issue here.

Thanks

11

hello ncs1,

let’s give it a try by adding skip eeprom check to bypass parsing board information.
you should given SKIP_EEPROM_CHECK=1 and also the board info into flash pipeline.
for instance,
here’s sample command-line for Jetson AGX Orin platform,
$ sudo SKIP_EEPROM_CHECK=1 BOARDID=3701 FAB=300 BOARDSKU=0004 BOARDREV=C.2 CHIP_SKU=00:00:00:D2 ./flash.sh jetson-agx-orin-devkit internal

since you’re using Orin NX,
please assign BOARDID=3767 BOARDSKU=0000 into flash command-line for testing.

12

Hi,

Used the following:

sudo SKIP_EEPROM_CHECK=1 BOARDID=3767 BOARDSKU=0000 FAB=300 BOARDREV='N.1' CHIPREV=1 CHIP_SKU=D3 ./flash.sh -u RSA1_TRADITIONAL -v SBK_KEY jetson-orin-nano-devkit internal

Same result as before (after long tar and build offline):

(long build before…)
[ 0.3115 ] Sending bct_br
[ 0.3916 ] Sending mb1
[ 0.3923 ] ERROR: might be timeout in USB write.
Error: Return value 3

Thanks

13

Just my thoughts on this, as we also some what similar faced flashing issue( though we haven’t dealt with any security/keys etc ).

If you are using custom carrier board and your carrier board dont have EEPROM, set its size to 0 and rebuild DTB files.

Refer this link:
https://docs.nvidia.com/jetson/archives/r36.3/DeveloperGuide/HR/JetsonModuleAdaptationAndBringUp/JetsonAgxOrinSeries.html#modifying-the-eeprom

16

Hi nagesh_accord,

The dtb eeprom size is already set to 0, was taken care of by forecr bsp

Thanks.

1 Like
17

hello ncs1,

here’re several questions,
please helps me to understand this issue more clearly.
for instance,
(1) it is a customize board, it’s okay to flash before Jetson security enabled.
(2) may I double confirm what’s your original flash command-line.
(3) it should be no issue with image creating, but USB communication failure only for image flashing, right?
(4) such -traditional options should be added for PKC key creation, which means a key used for burning fuse. key cannot change, since you’ve already burn the fuse.

18

Hi JerryChang,

  1. I burnt the fuses written above, without the security mode - which I intended to burn in a separate step. Please assert that it should be valid.
  2. Any command will fail now in the first steps involving the board, i.e. ‘send bct_br/mb1’
    Flash command:
    sudo ./tools/kernel_flash/l4t_initrd_flash.sh -u RSA1.key -v SBK.key -p “-c bootloader/t186ref/cfg/flash_t234_qspi.xml” --showlogs --network usb0 --no-flash jetson-orin-nano-devkit internal
    Faster check is:
    sudo ./odmfuseread.sh -i 0x23 -k RSA1.key -S SBK.key jetson-orin-nano-devkit
  3. In any USB recovery communication with the board will receive USB timeout from here onwards. i.e. ‘send bct_br/mb1’
  4. -traditional is just the .pem format, no ?
    I’ve converted my private keys to traditional format using:
    openssl pkey -in ./PRIV_NEW.pem -traditional >./PRIV_TRADITIONAL.pem
    I ran tegrasign_v3.py on the PRIV_TRADITIONAL.pem and asserted that the same hash was outputted as burnt.
  • The board will boot all the way to the old rootfs, if it helps you in any way.

Thanks.

19

hello ncs1,

is this correct board config?
you should flashing with a specific board config since you’ve working with a customize board.

let’s check whether you’ve burn the fuse variables correctly.
please examine those properties within… /sys/devices/platform/tegra-fuse

20

Hi JerryChang,

  1. forecr considered jetson-orin-nano-devkit to be the correct id, trying p3509-a02+p3767-0000 result the same. Please note that odmfuseread doesn’t need specific board and the issue is much earlier.
  2. This is my read from inside the target, please note that sysfs tegra-fuse is misleading (bootsecinfo returned 0, PKCs are clobbered), executing nv_fuse_read returned the correct results.
root@ORIN_NX_FUSED:/home/nvidia# sudo dd if=/sys/bus/nvmem/devices/fuse/nvmem of=boot_security_info bs=4 count=1 skip=$((0x5a))
1+0 records in
1+0 records out
4 bytes copied, 0.000237864 s, 16.8 kB/s
root@ORIN_NX_FUSED:/home/nvidia# hexdump boot_security_info
0000000 0009 0000
0000004
root@ORIN_NX_FUSED:/home/nvidia# hexdump boot_security_info
0000000  \t  \0  \0  \0
0000004
root@ORIN_NX_FUSED:/home/nvidia# sudo cat /sys/devices/platform/tegra-fuse/boot_security_info
0x00000000
root@ORIN_NX_FUSED:/home/nvidia# sudo cat /sys/devices/platform/tegra-fuse/public_key
0xcadfd9f2fc2934bbed00bd8f2e79f2f5772c49706dbef32dc4ee4c0518fc6c9c00000000000090000000000000000000000000000000105b507e30baa2f04ec6

nv_fuse_read call:

revoke_pk_h0: 0x00000000
revoke_pk_h1: 0x00000000
pk_h1: CORRECT PKC1
odminfo: 0x00000000
pk_h2: CORRECT PKC2
odmid: 0x0000000000000000
system_fw_field_ratchet1: 0x00000000
system_fw_field_ratchet0: 0x00000000
system_fw_field_ratchet3: 0x00000000
system_fw_field_ratchet2: 0x00000000
optin_enable: 0x00000000
public_key_hash: CORRECT PKC0
ecid: 0x847263291307FA02
reserved_odm2: 0x00000000
reserved_odm3: 0x00000000
reserved_odm0: 0x00000000
reserved_odm1: 0x00000000
reserved_odm6: 0x00000000
reserved_odm7: 0x00000000
reserved_odm4: 0x00000000
reserved_odm5: 0x00000000
boot_security_info: 0x00000009
security_mode: 0x00000000
odm_lock: 0x00000000

Thanks

21

hello ncs1,

such RSA1.key in the command-line should be the the key of your public_key_hash PKC0, is it correct?

22

Hi,

Correct. (As said, tried against all 3 keys + traditional formats)

Thanks.

23

hello ncs1,

may I still confirm what’s your original flash command-line to flash your target (i.e. Orin NX 16G on Forecr carrier) before Jetson security has enabled.