Xavier NX (eMMC) and TX2 NX Secureboot Fusing renders devices unbootable

Hello,

We are experiencing some difficulties when enabling secureboot on both Xavier NX (eMMC) and TX2 NX. The manufacturing process of our OEM product involves network booting an image, performing various preflight checks, installing the image to the eMMC, installing an updated BUP and burning the fuses. We DO NOT have access to the USB interface at this time so we cannot use ./flash.sh or ./odmfuse.sh. What we have done instead is pregenerate a signed, encrypted BUP using meta-tegra and use the --noburn option of odmfuse to produce an XML file which then is used to burn the fuses in the sysfs. We have confirmed that the sha256 digest of the BUP components matches the output of ./flash.sh from the latest sdkmanager release. We’ve also confirmed the components in the BUP match the public key hash value and can be correctly decrypted with our SBK.

After fusing, neither of the platforms have any serial output and appear to be booting in RCM mode but no commands appear to be working and always result in Boot Rom communication failed:

$ sudo ./flash.sh -x 0x19 -y SBKPKC -u ${KEY_DIR}/pkc.key -v ${KEY_DIR}/sbk.key jetson-xavier-nx-devkit-emmc mmcblk0p1
###############################################################################
# L4T BSP Information:
# R32 , REVISION: 6.1
###############################################################################
sbk_keyfile is present, but no user_keyfile; set user_keyfile to zero keyfile
# Target Board Information:
# Name: jetson-xavier-nx-devkit-emmc, Board Family: t186ref, SoC: Tegra 194, 
# OpMode: production, Boot Authentication: SBKPKC, 
# Disk encryption: disabled ,
###############################################################################
<snip>
[   0.0997 ] Boot Rom communication
[   0.1015 ] tegrarcm_v2 --chip 0x19 0 --rcm rcm_list_signed.xml --skipuid
[   0.1021 ] RCM version 0X13
[   0.1027 ] Boot Rom communication failed
[   5.1696 ] 
Error: Return value 3
Command tegrarcm_v2 --chip 0x19 0 --rcm rcm_list_signed.xml --skipuid
Reading board information failed.

or using the meta-tegra flash helper:

./tegraflash.py --bl nvtboot_recovery_cpu_t194.bin_sigheader.encrypt.signed --bct br_bct_BR.bct --applet rcm_2_signed.rcm --applet_softfuse rcm_1_signed.rcm --cmd "secureflash;reboot"  --cfg secureflash.xml --chip 0x19 --mb1_bct mb1_bct_MB1.bct_sigheader.encrypt.signed --mem_bct mem_rcm.bct_sigheader.encrypt.signed --mb1_cold_boot_bct mb1_cold_boot_bct_MB1.bct_sigheader.encrypt.signed --mem_bct_cold_boot mem_coldboot_sigheader.bct.signed  --bins <snip>"
saving flash command in flashcmd.txt
*** Flashing target device started. ***
Welcome to Tegra Flash
version 1.0.0
Type ? or help for help and q or quit to exit
Use ! to execute system commands
 
[   0.0049 ] Parsing partition layout
[   0.0067 ] tegraparser_v2 --pt secureflash.xml.tmp
[   0.0077 ] 
[   0.0077 ] Boot Rom communication
[   0.0095 ] tegrarcm_v2 --chip 0x19 0 --rcm rcm_1_signed.rcm --rcm rcm_2_signed.rcm
[   0.0101 ] BR_CID: 0xd8021911647e25c80c0000000c018280
[   0.0108 ] Bootrom returned error 19
[   0.0110 ] Boot Rom communication failed
[   0.0110 ] 
Error: Return value 19
Command tegrarcm_v2 --chip 0x19 0 --rcm rcm_1_signed.rcm --rcm rcm_2_signed.rcm

I’ve attached full output from flash, odmfuseread, and the tegra194-flash-helper in case anything else can be gleaned from that.

After setting the fuse values in the image we’ve confirmed their values (before rebooting and losing communication). Given an odmfuse.xml for p3668-0000 that looks like this: (SBK censored, PKH replaced)

<genericfuse MagicId="0x45535546" version="1.0.0">
<fuse name="SecureBootKey" size="16" value="0123456789abcdef0123456789abcdef" />
<fuse name="PublicKeyHash" size="32" value="99b0fed05eab60e28885a1ba66d248ce66c263c5b17e1e7524369ccfa15364c7" />
<fuse name="BootSecurityInfo" size="4" value="0x5" />
<fuse name="SecurityMode" size="4" value="0x1" />
</genericfuse>

we set the values in the order:

  1. BootSecurityInfo
  2. PublicKeyHash
  3. SecureBootKey
  4. SecurityMode

and when checking the values we get:

$ cat /sys/devices/platform/tegra-fuse/boot_security_info
0x00000005
$ cat /sys/devices/platform/tegra-fuse/public_key
0x99b0fed05eab60e28885a1ba66d248ce66c263c5b17e1e7524369ccfa15364c7
$ cat /sys/devices/platform/tegra-fuse/secure_boot_key
0x0123456789abcdef0123456789abcdef
$ cat /sys/devices/platform/tegra-fuse/odm_production_mode
0x00000001

When using TX2 NX there’s an extra step between 3 and 4 to update the ECC fuse by reading /sys/devices/platform/tegra-fuse/calc_h2 and setting /sys/devices/platform/tegra-fuse/odm_h2 to the lower 16bits:

$ cat /sys/devices/platform/tegra-fuse/calc_h2
0x00001234
$ echo "0x1234" > /sys/devices/platform/tegra-fuse/odm_h2

We also initial tested with --disable-jtag and --debug_authentication 0x1f as arguments to odmfuse.sh but removed those in case they were the cause of our problem.

Given all that information we have the following questions:

  1. Is burning the fuses in a running image on the device supported in any way and what steps would we need to change to make this work? We assumed it was supported because the driver allows burning but if that’s not the case this should be made more clear.

  2. Is there any way to recover these handful of nonbooting units if we have the SBK and PKC available to us? If not is there a way we can RMA these devices (possibly using our nvpartner support) and get unfused replacements? There were some other threads that seemed to indicate this was possible but I want to confirm this is still the case.

meta-tegra-tegraflash-helper.txt (67.5 KB)
sdkmanager-flash.txt (4.1 KB)
sdkmanager-odmreadfuses.txt (3.7 KB)

1 Like

hello anthony.squires,

how did you fuse the target? we’ve only test the fuse burning through micro-usb.

Hello I am having the same issue
I’ve burned the odm fuses using ./odmfuse.sh

sudo ./odmfuse.sh -i 0x19 -p -k /home/ubuntu/secure_booting_keys/rsa_priv.pem --KEK0 /home/ubuntu/secure_booting_keys/KEK0 --KEK1 /home/ubuntu/secure_booting_keys/KEK1 --KEK2 /home/ubuntu/secure_booting_keys/KEK2 -S /home/ubuntu/secure_booting_keys/SBK jetson-xavier-nx-devkit-emmc  

and returned success

*** The fuse configuration is saved in bootloader/odmfuse_pkc.xml
*** The ODM fuse has been burned successfully.
*** done.

but I am trying to flash using

sudo ./flash.sh -u /home/ubuntu/secure_booting_keys/rsa_priv.pem -v /home/ubuntu/secure_booting_keys/SBK jetson-xavier-nx-devkit-emmc mmcblk0p1

I am getting

###############################################################################
# L4T BSP Information:
# R32 , REVISION: 5.1
###############################################################################
sbk_keyfile is present, but no user_keyfile; set user_keyfile to zero keyfile
# Target Board Information:
# Name: jetson-xavier-nx-devkit-emmc, Board Family: t186ref, SoC: Tegra 194, 
# OpMode: production, Boot Authentication: SBKPKC, 
# Disk encryption: disabled ,
###############################################################################
.
.
.
.
[   0.6248 ] Boot Rom communication
[   0.6255 ] tegrarcm_v2 --chip 0x19 0 --rcm rcm_list_signed.xml --skipuid
[   0.6261 ] RCM version 0X16
[   0.6290 ] Boot Rom communication failed
[   5.6316 ] 
Error: Return value 3
Command tegrarcm_v2 --chip 0x19 0 --rcm rcm_list_signed.xml --skipuid
Reading board information failed.

Please advice
thanks

hello HusamAlqaza,

you’ve add --test option, which did not burn the target.

furthermore,
we had confirmed fuse and flashing works on Xavier NX,
please also check the steps for reference,
thanks

$ sudo BOARDID=3668 BOARDSKU=0001 FAB=100 BOARDREV=H.0 ./odmfuse.sh --noburn -j -i 0x19 -c PKC -p -k rsa_priv.pem -S sbk.key --KEK2 kek2.key jetson-xavier-nx-devkit-emmc
$ tar xpvf fuseblob.tbz2
$ cd bootloader/
$ sudo ./fusecmd.sh
$ cd ../
$ sudo BOARDID=3668 BOARDSKU=0001 FAB=100 BOARDREV=H.0 ./flash.sh --no-flash -u rsa_priv.pem -v sbk.key jetson-xavier-nx-devkit-emmc mmcblk0p1
$ cd bootloader/
$ sudo bash ./flashcmd.txt

Hello JerryChang

No I actually ran it without ---test
and you can see in the flash command the information states the device is protected via SBKPKC

I also tried to read the fuses and returns the same error

sudo ./odmfuseread.sh -i 0x19 -k /home/ubuntu/secure_booting_keys/rsa_priv.pem -S /home/ubuntu/secure_booting_keys/SBK jetson-xavier-nx-devkit-emmc

result

BK jetson-xavier-nx-devkit-emmc
copying soft_fuses(/home/ubuntu/nvidia/nvidia_sdk/JetPack_4.5.1_Linux_JETSON_XAVIER_NX/Linux_for_Tegra/bootloader/t186ref/BCT/tegra194-mb1-soft-fuses-l4t.cfg)... done.
copying soft_fuses(/home/ubuntu/nvidia/nvidia_sdk/JetPack_4.5.1_Linux_JETSON_XAVIER_NX/Linux_for_Tegra/bootloader/t186ref/BCT/tegra194-mb1-soft-fuses-l4t.cfg)... done.
./tegraflash.py --chip 0x19 --applet "/home/ubuntu/nvidia/nvidia_sdk/JetPack_4.5.1_Linux_JETSON_XAVIER_NX/Linux_for_Tegra/bootloader/mb1_t194_prod.bin" --skipuid --soft_fuses tegra194-mb1-soft-fuses-l4t.cfg --bins "mb2_applet nvtboot_applet_t194.bin" --cmd "dump eeprom boardinfo cvm.bin;reboot recovery" --encrypt_key "/home/ubuntu/secure_booting_keys/SBK" --key "/home/ubuntu/secure_booting_keys/rsa_priv.pem" 
Welcome to Tegra Flash
version 1.0.0
Type ? or help for help and q or quit to exit
Use ! to execute system commands
 
[   0.0032 ] Generating RCM messages
[   0.0040 ] tegrasign_v2 --key /home/ubuntu/secure_booting_keys/SBK --file /home/ubuntu/nvidia/nvidia_sdk/JetPack_4.5.1_Linux_JETSON_XAVIER_NX/Linux_for_Tegra/bootloader/mb1_t194_prod.bin --offset 4096
[   0.0047 ] Key is a SBK key
[   0.0049 ] Key Size is 16 bytes
[   0.0145 ] 
[   0.0152 ] tegrahost_v2 --chip 0x19 0 --magicid MB1B --appendsigheader /home/ubuntu/nvidia/nvidia_sdk/JetPack_4.5.1_Linux_JETSON_XAVIER_NX/Linux_for_Tegra/bootloader/mb1_t194_prod_encrypt.bin zerosbk
[   0.0158 ] Header already present for /home/ubuntu/nvidia/nvidia_sdk/JetPack_4.5.1_Linux_JETSON_XAVIER_NX/Linux_for_Tegra/bootloader/mb1_t194_prod_encrypt.bin
[   0.0186 ] 
[   0.0193 ] tegrasign_v2 --key /home/ubuntu/secure_booting_keys/rsa_priv.pem --getmode mode.txt
[   0.0200 ] PKC key in Open SSL format
[   0.0202 ] Key size is 384 bytes
[   0.0203 ] Valid PKC key
[   0.0239 ] 
[   0.0247 ] tegrasign_v2 --key /home/ubuntu/secure_booting_keys/rsa_priv.pem --file /home/ubuntu/nvidia/nvidia_sdk/JetPack_4.5.1_Linux_JETSON_XAVIER_NX/Linux_for_Tegra/bootloader/mb1_t194_prod_encrypt_sigheader.bin --offset 2960 --length 1136 --pubkeyhash pub_key.key --getmontgomeryvalues montgomery.bin
[   0.0253 ] PKC key in Open SSL format
[   0.0255 ] Key size is 384 bytes
[   0.0256 ] Valid PKC key
[   0.0293 ] Saving pkc public key  in pub_key.key
[   0.1773 ] Saving Montgomery values  in montgomery.bin
[   0.1784 ] 
[   0.1793 ] tegrahost_v2 --chip 0x19 0 --pubkeyhash pub_key.key --setmontgomeryvalues montgomery.bin --updatesigheader /home/ubuntu/nvidia/nvidia_sdk/JetPack_4.5.1_Linux_JETSON_XAVIER_NX/Linux_for_Tegra/bootloader/mb1_t194_prod_encrypt_sigheader.bin /home/ubuntu/nvidia/nvidia_sdk/JetPack_4.5.1_Linux_JETSON_XAVIER_NX/Linux_for_Tegra/bootloader/mb1_t194_prod_encrypt_sigheader.sig oem-rsa
[   0.1827 ] 
[   0.1837 ] tegrabct_v2 --chip 0x19 0 --sfuse tegra194-mb1-soft-fuses-l4t.cfg.pdf sfuse.bin
[   0.1845 ] 
[   0.1852 ] tegrarcm_v2 --listrcm rcm_list.xml --chip 0x19 0 --sfuses sfuse.bin --download rcm /home/ubuntu/nvidia/nvidia_sdk/JetPack_4.5.1_Linux_JETSON_XAVIER_NX/Linux_for_Tegra/bootloader/mb1_t194_prod_encrypt_sigheader.bin 0 0
[   0.1859 ] RCM 0 is saved as rcm_0.rcm
[   0.1883 ] RCM 1 is saved as rcm_1.rcm
[   0.1883 ] RCM 2 is saved as rcm_2.rcm
[   0.1883 ] List of rcm files are saved in rcm_list.xml
[   0.1883 ] 
[   0.1883 ] Signing RCM messages
[   0.1891 ] tegrasign_v2 --key /home/ubuntu/secure_booting_keys/rsa_priv.pem --list rcm_list.xml --pubkeyhash pub_key.key --getmontgomeryvalues montgomery.bin
[   0.1897 ] PKC key in Open SSL format
[   0.1898 ] Key size is 384 bytes
[   0.1899 ] Valid PKC key
[   0.1934 ] Saving pkc public key  in pub_key.key
[   0.6247 ] Saving Montgomery values  in montgomery.bin
[   0.6257 ] 
[   0.6257 ] Copying signature to RCM mesages
[   0.6264 ] tegrarcm_v2 --chip 0x19 0 --updatesig rcm_list_signed.xml --pubkeyhash pub_key.key --setmontgomeryvalues montgomery.bin
[   0.6275 ] 
[   0.6276 ] Boot Rom communication
[   0.6283 ] tegrarcm_v2 --chip 0x19 0 --rcm rcm_list_signed.xml --skipuid
[   0.6289 ] RCM version 0X16
[   0.6312 ] Boot Rom communication failed
[   5.6642 ] 
**Error: Return value 3**
**Command tegrarcm_v2 --chip 0x19 0 --rcm rcm_list_signed.xml --skipuid**
**Reading board information failed.**

please advice

hello HusamAlqaza,

there’ll be connection/disconnection for serval times during board flashing,
this may due to the board communication stability, may I know what’s your environment setups, for example, is it a laptop or desktop, are you using a virtual machine?
thanks

Hello JerryChang
Thanks for your fast response

the host machine is DELL laptop with ubuntu 18.04

uname -a
Linux ubuntu 5.4.0-99-generic #112~18.04.1-Ubuntu SMP Thu Feb 3 14:09:57 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

and the last update I ran the suggested command

sudo BOARDID=3668 BOARDSKU=0001 FAB=100 BOARDREV=H.0 ./flash.sh --no-flash -u rsa_priv.pem -v sbk.key jetson-xavier-nx-devkit-emmc mmcblk0p1

and finished successfully

saving flash command in flashcmd.txt

*** no-flash flag enabled. Exiting now... *** 

User can run above saved command in factory environment without 
providing pkc and sbk keys to flash a device

Example:

    $ cd bootloader 
    $ sudo bash ./flashcmd.txt

and created ./flashcmd.txt under bootloader dir

then I ran

 sudo bash ./flashcmd.txt

and here is the result

Welcome to Tegra Flash
version 1.0.0
Type ? or help for help and q or quit to exit
Use ! to execute system commands
 
[   0.0069 ] Parsing partition layout
[   0.0077 ] tegraparser_v2 --pt secureflash.xml.tmp
[   0.0089 ] 
[   0.0090 ] Boot Rom communication
[   0.0097 ] tegrarcm_v2 --chip 0x19 0 --rcm rcm_1_signed.rcm --rcm rcm_2_signed.rcm
[   0.0104 ] BR_CID: 0xd8021911640560822000000008020200
[   0.0113 ] Bootrom returned error 22
[   0.0276 ] Boot Rom communication failed
[   0.0277 ] 
Error: Return value 22
Command tegrarcm_v2 --chip 0x19 0 --rcm rcm_1_signed.rcm --rcm rcm_2_signed.rcm

hello HusamAlqaza,

please put the target enter forced-recovery and checking $ lsusb for its device name.
you should keep the power-supply connected for image flashing, you may also try another cable or another usb port for verification,
thanks

Hello JerryChang

I did the following

  1. poweroff
  2. connect new USB cable to another port
  3. connect power cable
  4. press rec + rest buttons to enter the recovery mode
  5. $ lsub shows Bus 001 Device 038: ID 0955:7e19 NVidia Corp
$ dmesg
 [59163.570662] usb 1-3: USB disconnect, device number 36
[59164.787738] usb 1-3: new high-speed USB device number 37 using xhci_hcd
[59164.936762] usb 1-3: New USB device found, idVendor=0955, idProduct=7e19, bcdDevice= 1.02
[59164.936768] usb 1-3: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[59164.936770] usb 1-3: Product: APX
[59164.936773] usb 1-3: Manufacturer: NVIDIA Corp.
  1. ran $ sudo bash ./flashcmd.txt
    result
Welcome to Tegra Flash
version 1.0.0
Type ? or help for help and q or quit to exit
Use ! to execute system commands
 
[   0.0068 ] Parsing partition layout
[   0.0076 ] tegraparser_v2 --pt secureflash.xml.tmp
[   0.0088 ] 
[   0.0088 ] Boot Rom communication
[   0.0095 ] tegrarcm_v2 --chip 0x19 0 --rcm rcm_1_signed.rcm --rcm rcm_2_signed.rcm
[   0.0102 ] BR_CID: 0xd8021911640560822000000008020200
[   0.0109 ] Bootrom returned error 22
[   0.0260 ] Boot Rom communication failed
[   0.0260 ] 
Error: Return value 22
Command tegrarcm_v2 --chip 0x19 0 --rcm rcm_1_signed.rcm --rcm rcm_2_signed.rcm

please let me know if you need any extra information

Please advice

Hi Jerry, Sorry for the late reply but we managed to figure out the issue and get our secureboot enable working.

On our production line we are using the sysfs fuse-burn driver by writing the hex values to /sys/devices/platform/tegra-fuse with it’s source located at drivers/soc/tegra/fuse/fuse-burn.c in the kernel. We managed to figure out the problem by using odmfuse.sh over USB recovery and comparing values and changes. To be clear, odmfuse.sh works over USB but we need a fully automated process that does not involve putting the custom appliance into recovery mode.

What we found was the problem outside of our processes was that the H2 fuse calculation was incorrect for t186 and missing for t194. If it’s allowed I can add the patch that allowed us to exactly match the process odmfuse.sh takes.

With regards to the bricked SoMs, are we just stuck with them or is the some process where we can recover them?

hello anthony.squires,

are you able to execute odmfuseread.sh to read the fuse info from the target board?

@JerryChang
any update about my issue

1 Like

hello HusamAlqaza,

is this a fuse board? may I know what’s the fuse variable you’ve burned.
do you have other Jetson platforms that able to flash with your host machine?

I attached an odmfuseread.sh output in the original post on one of the bad SoMs. It doesn’t get past the initial RCM communication similar to what @HusamAlqaza is seeing. Using the same carrier, cable and host machine as those I can read back the fuses on the working SoMs with no issues.

1 Like

There is no update from you for a period, assuming this is not an issue any more.
Hence we are closing this topic. If need further support, please open a new one.
Thanks

hello anthony.squires,

is this still an opened discussion thread, may I know the topic-id.
could you please also share the fuse types, and also the original odm fuse command you’ve used.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.