Jetson TX2 failed flash Was: initial setup, cert verification failed on nvidia r32.7 repositories

fahnoe · May 9, 2022, 9:26pm

Hello,

SDK Manager 1.8.0.10363
JetPack 4.6.2 r32.7

Just getting started with an old TX2 developer kit that’s never been used. The host system (a physical server) is running ubuntu 18.04.6 and the TX2 has been flashed via sdkmanager, however the subsequent verification of packages failed because the TX2 did not have Internet access. In order to provide access I needed to change the IP which I did via the dhcp server and then rebooting the TX2. It comes up fine and now has Internet access.

In retrying the verification, now the problem is due to errors in certificate verification for the nvidia repositories. Assuming that once the system had received all of its packages the proper keyrings would be in place, so as a temporary hack I added the allow-insecure=yes option to the apt sources file on the TX2:

# sed -i.orig -e 's/deb https/deb [ allow-insecure=yes ] https/' /etc/apt/sources.list.d/nvidia-l4t-apt-source.list

# grep -v '^#' /etc/apt/sources.list.d/nvidia-l4t-apt-source.list

deb [ allow-insecure=yes ] https://repo.download.nvidia.com/jetson/common r32.7 main
deb [ allow-insecure=yes ] https://repo.download.nvidia.com/jetson/t186 r32.7 main

This resolves some of the errors, but now I see that there are additional nvidia repositories with the same problem, however I don’t find apt sources for these:

E: Failed to fetch https://repo.download.nvidia.com/jetson/common/dists/r32.7/main/binary-arm64/Packages  Certificate verification failed: The certificate is NOT trusted. The certificate issuer is unknown.  Could not handshake: Error in the certificate verification. [IP: 96.17.149.8 443]
E: Failed to fetch https://repo.download.nvidia.com/jetson/t186/dists/r32.7/main/binary-arm64/Packages  Certificate verification failed: The certificate is NOT trusted. The certificate issuer is unknown.  Could not handshake: Error in the certificate verification. [IP: 96.17.149.8 443]

This much fiddling also says to me that I’m fighting someone else’s battles, in other words the problem is farther upstream in the nvidia repositories themselves (I think).

–Larry

The complete output from apt update is as follows

# apt update; echo Status: $?
Ign:1 https://repo.download.nvidia.com/jetson/common r32.7 InRelease
Hit:2 http://ports.ubuntu.com/ubuntu-ports bionic InRelease
Hit:3 http://ports.ubuntu.com/ubuntu-ports bionic-updates InRelease            
Ign:4 https://repo.download.nvidia.com/jetson/t186 r32.7 InRelease             
Hit:5 http://ports.ubuntu.com/ubuntu-ports bionic-backports InRelease          
Ign:6 https://repo.download.nvidia.com/jetson/common r32.7 Release             
Hit:7 http://ports.ubuntu.com/ubuntu-ports bionic-security InRelease           
Ign:8 https://repo.download.nvidia.com/jetson/t186 r32.7 Release               
Ign:9 https://repo.download.nvidia.com/jetson/common r32.7/main all Packages   
Ign:10 https://repo.download.nvidia.com/jetson/common r32.7/main arm64 Packages
Ign:11 https://repo.download.nvidia.com/jetson/common r32.7/main Translation-en
Ign:12 https://repo.download.nvidia.com/jetson/common r32.7/main Translation-en_US
Ign:13 https://repo.download.nvidia.com/jetson/common r32.7/main arm64 DEP-11 Metadata
Ign:14 https://repo.download.nvidia.com/jetson/common r32.7/main all DEP-11 Metadata
Ign:15 https://repo.download.nvidia.com/jetson/common r32.7/main DEP-11 48x48 Icons
Ign:16 https://repo.download.nvidia.com/jetson/common r32.7/main DEP-11 64x64 Icons
Ign:17 https://repo.download.nvidia.com/jetson/t186 r32.7/main arm64 Packages
Ign:18 https://repo.download.nvidia.com/jetson/t186 r32.7/main all Packages
Ign:19 https://repo.download.nvidia.com/jetson/t186 r32.7/main Translation-en_US
Ign:20 https://repo.download.nvidia.com/jetson/t186 r32.7/main Translation-en
Ign:21 https://repo.download.nvidia.com/jetson/t186 r32.7/main all DEP-11 Metadata
Ign:22 https://repo.download.nvidia.com/jetson/t186 r32.7/main arm64 DEP-11 Metadata
Ign:23 https://repo.download.nvidia.com/jetson/t186 r32.7/main DEP-11 48x48 Icons
Ign:24 https://repo.download.nvidia.com/jetson/t186 r32.7/main DEP-11 64x64 Icons
Ign:9 https://repo.download.nvidia.com/jetson/common r32.7/main all Packages
Ign:10 https://repo.download.nvidia.com/jetson/common r32.7/main arm64 Packages
Ign:11 https://repo.download.nvidia.com/jetson/common r32.7/main Translation-en
Ign:12 https://repo.download.nvidia.com/jetson/common r32.7/main Translation-en_US
Ign:13 https://repo.download.nvidia.com/jetson/common r32.7/main arm64 DEP-11 Metadata
Ign:14 https://repo.download.nvidia.com/jetson/common r32.7/main all DEP-11 Metadata
Ign:15 https://repo.download.nvidia.com/jetson/common r32.7/main DEP-11 48x48 Icons
Ign:16 https://repo.download.nvidia.com/jetson/common r32.7/main DEP-11 64x64 Icons
Ign:17 https://repo.download.nvidia.com/jetson/t186 r32.7/main arm64 Packages
Ign:18 https://repo.download.nvidia.com/jetson/t186 r32.7/main all Packages
Ign:19 https://repo.download.nvidia.com/jetson/t186 r32.7/main Translation-en_US
Ign:20 https://repo.download.nvidia.com/jetson/t186 r32.7/main Translation-en
Ign:21 https://repo.download.nvidia.com/jetson/t186 r32.7/main all DEP-11 Metadata
Ign:22 https://repo.download.nvidia.com/jetson/t186 r32.7/main arm64 DEP-11 Metadata
Ign:23 https://repo.download.nvidia.com/jetson/t186 r32.7/main DEP-11 48x48 Icons
Ign:24 https://repo.download.nvidia.com/jetson/t186 r32.7/main DEP-11 64x64 Icons
Ign:9 https://repo.download.nvidia.com/jetson/common r32.7/main all Packages
Ign:10 https://repo.download.nvidia.com/jetson/common r32.7/main arm64 Packages
Ign:11 https://repo.download.nvidia.com/jetson/common r32.7/main Translation-en
Ign:12 https://repo.download.nvidia.com/jetson/common r32.7/main Translation-en_US
Ign:13 https://repo.download.nvidia.com/jetson/common r32.7/main arm64 DEP-11 Metadata
Ign:14 https://repo.download.nvidia.com/jetson/common r32.7/main all DEP-11 Metadata
Ign:15 https://repo.download.nvidia.com/jetson/common r32.7/main DEP-11 48x48 Icons
Ign:16 https://repo.download.nvidia.com/jetson/common r32.7/main DEP-11 64x64 Icons
Ign:17 https://repo.download.nvidia.com/jetson/t186 r32.7/main arm64 Packages
Ign:18 https://repo.download.nvidia.com/jetson/t186 r32.7/main all Packages
Ign:19 https://repo.download.nvidia.com/jetson/t186 r32.7/main Translation-en_US
Ign:20 https://repo.download.nvidia.com/jetson/t186 r32.7/main Translation-en
Ign:21 https://repo.download.nvidia.com/jetson/t186 r32.7/main all DEP-11 Metadata
Ign:22 https://repo.download.nvidia.com/jetson/t186 r32.7/main arm64 DEP-11 Metadata
Ign:23 https://repo.download.nvidia.com/jetson/t186 r32.7/main DEP-11 48x48 Icons
Ign:24 https://repo.download.nvidia.com/jetson/t186 r32.7/main DEP-11 64x64 Icons
Ign:9 https://repo.download.nvidia.com/jetson/common r32.7/main all Packages
Ign:10 https://repo.download.nvidia.com/jetson/common r32.7/main arm64 Packages
Ign:11 https://repo.download.nvidia.com/jetson/common r32.7/main Translation-en
Ign:12 https://repo.download.nvidia.com/jetson/common r32.7/main Translation-en_US
Ign:13 https://repo.download.nvidia.com/jetson/common r32.7/main arm64 DEP-11 Metadata
Ign:14 https://repo.download.nvidia.com/jetson/common r32.7/main all DEP-11 Metadata
Ign:15 https://repo.download.nvidia.com/jetson/common r32.7/main DEP-11 48x48 Icons
Ign:16 https://repo.download.nvidia.com/jetson/common r32.7/main DEP-11 64x64 Icons
Ign:17 https://repo.download.nvidia.com/jetson/t186 r32.7/main arm64 Packages
Ign:18 https://repo.download.nvidia.com/jetson/t186 r32.7/main all Packages
Ign:19 https://repo.download.nvidia.com/jetson/t186 r32.7/main Translation-en_US
Ign:20 https://repo.download.nvidia.com/jetson/t186 r32.7/main Translation-en
Ign:21 https://repo.download.nvidia.com/jetson/t186 r32.7/main all DEP-11 Metadata
Ign:22 https://repo.download.nvidia.com/jetson/t186 r32.7/main arm64 DEP-11 Metadata
Ign:23 https://repo.download.nvidia.com/jetson/t186 r32.7/main DEP-11 48x48 Icons
Ign:24 https://repo.download.nvidia.com/jetson/t186 r32.7/main DEP-11 64x64 Icons
Ign:9 https://repo.download.nvidia.com/jetson/common r32.7/main all Packages
Ign:10 https://repo.download.nvidia.com/jetson/common r32.7/main arm64 Packages
Ign:11 https://repo.download.nvidia.com/jetson/common r32.7/main Translation-en
Ign:12 https://repo.download.nvidia.com/jetson/common r32.7/main Translation-en_US
Ign:13 https://repo.download.nvidia.com/jetson/common r32.7/main arm64 DEP-11 Metadata
Ign:14 https://repo.download.nvidia.com/jetson/common r32.7/main all DEP-11 Metadata
Ign:15 https://repo.download.nvidia.com/jetson/common r32.7/main DEP-11 48x48 Icons
Ign:16 https://repo.download.nvidia.com/jetson/common r32.7/main DEP-11 64x64 Icons
Ign:17 https://repo.download.nvidia.com/jetson/t186 r32.7/main arm64 Packages
Ign:18 https://repo.download.nvidia.com/jetson/t186 r32.7/main all Packages
Ign:19 https://repo.download.nvidia.com/jetson/t186 r32.7/main Translation-en_US
Ign:20 https://repo.download.nvidia.com/jetson/t186 r32.7/main Translation-en
Ign:21 https://repo.download.nvidia.com/jetson/t186 r32.7/main all DEP-11 Metadata
Ign:22 https://repo.download.nvidia.com/jetson/t186 r32.7/main arm64 DEP-11 Metadata
Ign:23 https://repo.download.nvidia.com/jetson/t186 r32.7/main DEP-11 48x48 Icons
Ign:24 https://repo.download.nvidia.com/jetson/t186 r32.7/main DEP-11 64x64 Icons
Ign:9 https://repo.download.nvidia.com/jetson/common r32.7/main all Packages
Ign:10 https://repo.download.nvidia.com/jetson/common r32.7/main arm64 Packages
Ign:11 https://repo.download.nvidia.com/jetson/common r32.7/main Translation-en
Ign:12 https://repo.download.nvidia.com/jetson/common r32.7/main Translation-en_US
Ign:13 https://repo.download.nvidia.com/jetson/common r32.7/main arm64 DEP-11 Metadata
Ign:14 https://repo.download.nvidia.com/jetson/common r32.7/main all DEP-11 Metadata
Ign:15 https://repo.download.nvidia.com/jetson/common r32.7/main DEP-11 48x48 Icons
Ign:16 https://repo.download.nvidia.com/jetson/common r32.7/main DEP-11 64x64 Icons
Ign:17 https://repo.download.nvidia.com/jetson/t186 r32.7/main arm64 Packages
Ign:18 https://repo.download.nvidia.com/jetson/t186 r32.7/main all Packages
Ign:19 https://repo.download.nvidia.com/jetson/t186 r32.7/main Translation-en_US
Ign:20 https://repo.download.nvidia.com/jetson/t186 r32.7/main Translation-en
Ign:21 https://repo.download.nvidia.com/jetson/t186 r32.7/main all DEP-11 Metadata
Ign:22 https://repo.download.nvidia.com/jetson/t186 r32.7/main arm64 DEP-11 Metadata
Ign:23 https://repo.download.nvidia.com/jetson/t186 r32.7/main DEP-11 48x48 Icons
Ign:24 https://repo.download.nvidia.com/jetson/t186 r32.7/main DEP-11 64x64 Icons
Ign:9 https://repo.download.nvidia.com/jetson/common r32.7/main all Packages
Err:10 https://repo.download.nvidia.com/jetson/common r32.7/main arm64 Packages
  Certificate verification failed: The certificate is NOT trusted. The certificate issuer is unknown.  Could not handshake: Error in the certificate verification. [IP: 96.17.149.8 443]
Ign:11 https://repo.download.nvidia.com/jetson/common r32.7/main Translation-en
Ign:12 https://repo.download.nvidia.com/jetson/common r32.7/main Translation-en_US
Ign:13 https://repo.download.nvidia.com/jetson/common r32.7/main arm64 DEP-11 Metadata
Ign:14 https://repo.download.nvidia.com/jetson/common r32.7/main all DEP-11 Metadata
Ign:15 https://repo.download.nvidia.com/jetson/common r32.7/main DEP-11 48x48 Icons
Ign:16 https://repo.download.nvidia.com/jetson/common r32.7/main DEP-11 64x64 Icons
Err:17 https://repo.download.nvidia.com/jetson/t186 r32.7/main arm64 Packages
  Certificate verification failed: The certificate is NOT trusted. The certificate issuer is unknown.  Could not handshake: Error in the certificate verification. [IP: 96.17.149.8 443]
Ign:18 https://repo.download.nvidia.com/jetson/t186 r32.7/main all Packages
Ign:19 https://repo.download.nvidia.com/jetson/t186 r32.7/main Translation-en_US
Ign:20 https://repo.download.nvidia.com/jetson/t186 r32.7/main Translation-en
Ign:21 https://repo.download.nvidia.com/jetson/t186 r32.7/main all DEP-11 Metadata
Ign:22 https://repo.download.nvidia.com/jetson/t186 r32.7/main arm64 DEP-11 Metadata
Ign:23 https://repo.download.nvidia.com/jetson/t186 r32.7/main DEP-11 48x48 Icons
Ign:24 https://repo.download.nvidia.com/jetson/t186 r32.7/main DEP-11 64x64 Icons
Reading package lists... Done
W: The repository 'https://repo.download.nvidia.com/jetson/common r32.7 Release' does not have a Release file.
N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use.
N: See apt-secure(8) manpage for repository creation and user configuration details.
W: The repository 'https://repo.download.nvidia.com/jetson/t186 r32.7 Release' does not have a Release file.
N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use.
N: See apt-secure(8) manpage for repository creation and user configuration details.
E: Failed to fetch https://repo.download.nvidia.com/jetson/common/dists/r32.7/main/binary-arm64/Packages  Certificate verification failed: The certificate is NOT trusted. The certificate issuer is unknown.  Could not handshake: Error in the certificate verification. [IP: 96.17.149.8 443]
E: Failed to fetch https://repo.download.nvidia.com/jetson/t186/dists/r32.7/main/binary-arm64/Packages  Certificate verification failed: The certificate is NOT trusted. The certificate issuer is unknown.  Could not handshake: Error in the certificate verification. [IP: 96.17.149.8 443]
E: Some index files failed to download. They have been ignored, or old ones used instead.
Status: 100

linuxdev · May 10, 2022, 1:57pm

I don’t know if it is related to this or not, but it could be:

https://developer.nvidia.com/blog/updating-the-cuda-linux-gpg-repository-key/?mkt_tok=MTU2LU9GTi03NDIAAAGENMEeAgIWA-MGHZS0UZWutc2f4TR2NGlQyzvI5M5rdKpzIoFudk13-PhiqvY2wTOft1gE85KaeGDXuJINKdRzMo-BnXhOi304O24_oKWA1n5HXgnW

There was a repo key update (described in the above URL), and perhaps that would solve the problem.

fahnoe · May 10, 2022, 5:15pm

Thanks for that article. May well be related in that there might have been TLS cert changes as well as the repo key updates. However after a bit more investigation I can see another potential culprit: the TX2 was routing out through a firewall that was doing HTTPS inspection and thus would have seen a self-signed cert that it wouldn’t know to trust. I changed routing so that it went out via a different firewall (no HTTPS inspection), removed the allow-insecure=yes option hack and the apt update now completes successfully.

Although the TX2 is now happy with the nvidia repos, because the original verification had not completed fully, I decided to reflash lest there be some step that had been missed. I put the TX2 into recovery mode and chose Manual Setup - Jetson TX2 in Step 03. The flash failed with the following (shown in the GUI’s TERMINAL tab):

10:54:11 ERROR: Can not find matching device in recovery mode.
10:54:11 DEBUG: running command < true >
10:54:11 INFO: exec_command: true
10:54:11 INFO: command finished successfully
10:54:11 DEBUG: running command < /tmp/device_mode_host_setup.sh >
10:54:11 INFO: exec_command: /tmp/device_mode_host_setup.sh
10:54:11 INFO: Found route default enp2s0, setting up device mode
10:54:11 INFO: [host] [sudo] password for fahnoe:
10:54:11 INFO: net.ipv4.ip_forward = 1
10:54:11 INFO: Device mode on host successfully set with DNS 8.8.8.8!
10:54:11 INFO: command finished successfully
10:54:11 DEBUG: running command < true >
10:54:11 INFO: exec_command: true
10:54:11 INFO: command finished successfully
10:54:11 DEBUG: Flash Jetson TX2 - flash: using adapter to install NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS to /home/fahnoe/nvidia/nvidia_sdk/JetPack_4.6.2_Linux_JETSON_TX2_TARGETS
10:54:11 INFO: Flash Jetson TX2 - flash: change working directory to /home/fahnoe/nvidia/nvidia_sdk/JetPack_4.6.2_Linux_JETSON_TX2_TARGETS/Linux_for_Tegra
10:54:11 INFO: Flash Jetson TX2 - flash: current working directory is /home/fahnoe/nvidia/nvidia_sdk/JetPack_4.6.2_Linux_JETSON_TX2_TARGETS/Linux_for_Tegra
10:54:11 INFO: Flash Jetson TX2 - flash: **********************
10:54:12 INFO: Flash Jetson TX2 - flash: exec_command: /tmp/tmp_NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP.sh
10:54:12 INFO: Flash Jetson TX2 - flash: sudo ./nvsdkmanager_flash.sh --nv-auto-config --username fahnoe
10:54:12 INFO: Flash Jetson TX2 - flash: [OEMPreconfig] SDKM_INSTALL_ERROR Password for L4T new user fahnoe:
10:54:12 INFO: Flash Jetson TX2 - flash: *** Checking ONLINE mode ... OK.
10:54:12 INFO: Flash Jetson TX2 - flash: *** Checking target board connection ... 1 connections found.
10:54:12 INFO: Flash Jetson TX2 - flash: *** Reading ECID ... FUSELEVEL=fuselevel_production hwchipid=0x18 bootauth=NS
10:54:12 INFO: Flash Jetson TX2 - flash: *** Reading EEPROM ... "/home/fahnoe/nvidia/nvidia_sdk/JetPack_4.6.2_Linux_JETSON_TX2_TARGETS/Linux_for_Tegra/bootloader/tegraflash.py" --chip 0x18 --applet "/home/fahnoe/nvidia/nvidia_sdk/JetPack_4.6.2_Linux_JETSON_TX2_TARGETS/Linux_for_Tegra/bootloader/mb1_recovery_prod.bin" --skipuid --cmd "dump eeprom boardinfo cvm.bin; dump eeprom baseinfo bbd.bin; reboot recovery"
10:54:12 INFO: Flash Jetson TX2 - flash: [ 0.0144 ] tegrarcm_v2 --listrcm rcm_list.xml --chip 0x18 0 --download rcm /home/fahnoe/nvidia/nvidia_sdk/JetPack_4.6.2_Linux_JETSON_TX2_TARGETS/Linux_for_Tegra/bootloader/mb1_recovery_prod.bin 0 0
10:54:12 INFO: Flash Jetson TX2 - flash: [ 0.0300 ] Assuming zero filled SBK key
10:54:12 INFO: Flash Jetson TX2 - flash: [ 0.0981 ] tegrarcm_v2 --chip 0x18 0 --updatesig rcm_list_signed.xml
10:54:12 INFO: Flash Jetson TX2 - flash: [ 0.1051 ] tegrarcm_v2 --chip 0x18 0 --rcm rcm_list_signed.xml --skipuid
10:54:12 INFO: Flash Jetson TX2 - flash: [ 0.1074 ] RCM version 0X180001
10:54:14 INFO: Flash Jetson TX2 - flash: [ 2.2500 ] tegrarcm_v2 --isapplet
10:54:14 INFO: Flash Jetson TX2 - flash: [ 2.2844 ] tegradevflash_v2 --iscpubl
10:54:15 INFO: Flash Jetson TX2 - flash: [ 3.2963 ] tegrarcm_v2 --isapplet
10:54:15 INFO: Flash Jetson TX2 - flash: [ 3.3040 ] tegradevflash_v2 --iscpubl
[...]
10:54:43 INFO: Flash Jetson TX2 - flash: [ 30.7872 ] tegradevflash_v2 --iscpubl
10:54:44 INFO: Flash Jetson TX2 - flash: [ 31.7963 ] tegrarcm_v2 --isapplet
10:54:44 INFO: Flash Jetson TX2 - flash: [ 31.8054 ] tegradevflash_v2 --iscpubl
10:54:44 ERROR: Flash Jetson TX2 - flash: Error: None of the bootloaders are running on device. Check the UART log.
10:54:44 ERROR: Flash Jetson TX2 - flash: command terminated with error
10:54:44 SUMMARY: DateTime Target Setup - target: Depends on failed component

So naturally this has me rather concerned… The first ERROR: Can not find matching device in recovery mode. doesn’t make sense because the board was in recovery mode and sdkmanager was able to see it. Further, I’ve noticed that the TX2 only seems to show up in the host’s lsusb output if it is in recovery mode and I verified that it was showing prior to launching sdkmanager.

I cannot find anything resembling a UART log (per the trailing message above), but the only thing on the serial console was the final shutdown prior to being put in recovery mode:

[    0.000000] Boot CPU: AArch64 Processor [411fd073]
[    0.000000] OF: fdt:memory scan node memory@80000000, reg size 80,
[    0.000000] OF: fdt: - 80000000 ,  70000000
[    0.000000] OF: fdt: - f0200000 ,  185600000
[    0.000000] OF: fdt: - 275e00000 ,  200000
[    0.000000] OF: fdt: - 276600000 ,  200000
[    0.000000] OF: fdt: - 277000000 ,  200000
[    0.000000] earlycon: uart8250 at MMIO32 0x0000000003100000 (options '')
[    0.000000] bootconsole [uart8250] enabled
[    1.798324] cgroup: cgroup2: unknown option "nsdelegate"
[    2.819017] using random self ethernet address
[    2.825561] using random host ethernet address
[    3.255393] random: crng init done
[    3.258801] random: 7 urandom warning(s) missed due to ratelimiting
[    3.413832] using random self ethernet address
[    3.422785] using random host ethernet address
[    3.861330] CPU1: shutdown
[    3.928615] CPU2: shutdown
[    5.051370] Bridge firewalling registered

Ubuntu 18.04.6 LTS elroy ttyS0

elroy login: fahnoe
Password: 
Last login: Tue May 10 10:43:31 CDT 2022 from 192.168.3.65 on pts/0
Welcome to Ubuntu 18.04.6 LTS (GNU/Linux 4.9.253-tegra aarch64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage
This system has been minimized by removing packages and content that are
not required on a system that users do not log into.

To restore this content, you can run the 'unminimize' command.

40 updates can be applied immediately.
37 of these updates are standard security updates.
To see these additional updates run: apt list --upgradable

fahnoe@elroy:~$ sudo /sbin/shutdown -P now
[sudo] password for fahnoe: 
[ 1981.690095] systemd-shutdow: 34 output lines suppressed due to ratelimiting
[ 1981.922026] CPU3: shutdown
[ 1981.965995] CPU4: shutdown
[ 1982.005958] CPU5: shutdown
[ 1982.027087] reboot: Power down

At this point with the board in recovery mode the console does not respond, and with the above error I’m not enthusiastic about pressing reset or power-cycling the board. I would further point out that the TX2 has been rebooted several times prior to this attempt at flashing so I know its bootloader was functional.

Digging further in the logs from sdkmanager I find what seems to be the relevant portion of the failed flashing process in sdkm-2022-05-10-10-48-59.log including the fact that the board is in recovery mode:

10:51:34.960 - info: All required files were downloaded
10:54:11.343 - info: Start to check If Jetson device Ready To Flash Manual for board...
10:54:11.344 - info: Device info before flashing:
Jetson TX2 | 002/004 | 2-7 | Recovery

10:54:11.347 - info: Start to check if in device recovery mode...
10:54:11.490 - info: Found matching device in recovery mode.
10:54:11.491 - error: Can not find matching device in recovery mode.
10:54:11.513 - info: create device_mode_host_setup.sh under /tmp
10:54:11.524 - info: ***** execute device_mode_host_setup.sh *****
10:54:11.540 - info: exec_command: true
10:54:11.550 - info: 
10:54:11.551 - info: command finished successfully
10:54:11.577 - info: exec_command: /tmp/device_mode_host_setup.sh
10:54:11.606 - info: Found route default enp2s0, setting up device mode
10:54:11.622 - info: [host] [sudo] password for fahnoe: 
10:54:11.634 - info: [HOST PASSWORD]
10:54:11.635 - info: 
10:54:11.761 - info: net.ipv4.ip_forward = 1
10:54:11.829 - info: Device mode on host successfully set with DNS 8.8.8.8!
10:54:11.841 - info: 
10:54:11.842 - info: command finished successfully
10:54:11.850 - info: Event: Device connection retry is complete: completed. Dialog Name: pre-flash.
            Error: . User retried: true
10:54:11.851 - info: Dialog was closed: pre-flash dialog. Duration: 00:02:40
10:54:11.886 - info: memInfo [total: 7.79GB, free: 3.45GB, available: 6.83GB]
10:54:11.887 - info: Event: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS - install started
10:54:11.890 - info: install timeout for NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP: 1000s
10:54:11.932 - info: exec_command: true
10:54:11.939 - info: 
10:54:11.939 - info: command finished successfully
10:54:11.967 - info: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS: [host] [ Disk Avail:50.66 GB ]
10:54:11.967 - info: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS: Installing component 'Flash Jetson TX2' (NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP)
10:54:11.968 - info: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS: change working directory to /home/fahnoe/nvidia/nvidia_sdk/JetPack_4.6.2_Linux_JETSON_TX2_TARGETS/Lin
ux_for_Tegra
10:54:11.978 - info: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS: [ Component Install Started ]
10:54:11.978 - info: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS: current working directory is /home/fahnoe/nvidia/nvidia_sdk/JetPack_4.6.2_Linux_JETSON_TX2_TARGETS/Li
nux_for_Tegra
10:54:11.989 - info: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS: exec_command [host]:
10:54:11.990 - info: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS: **********************
10:54:11.991 - info: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS: #!/bin/bash
10:54:11.991 - info: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS: set -e
10:54:11.992 - info: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS: set -e
10:54:11.992 - info: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS: unset LC_ALL
10:54:11.992 - info: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS: echo sudo ./nvsdkmanager_flash.sh  --nv-auto-config --username fahnoe
10:54:11.993 - info: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS: sudo ./nvsdkmanager_flash.sh  --nv-auto-config --username fahnoe
10:54:11.993 - info: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS: sleep 20
10:54:11.994 - info: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS: **********************
10:54:12.003 - info: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS: exec_command: /tmp/tmp_NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP.sh
10:54:12.012 - info: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS: sudo ./nvsdkmanager_flash.sh --nv-auto-config --username fahnoe
10:54:12.065 - info: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS: [OEMPreconfig] SDKM_INSTALL_ERROR Password for L4T new user fahnoe: 
10:54:12.066 - info: [OEMPreconfig PASSWORD]
10:54:12.090 - info: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS: 
10:54:12.164 - info: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS: *** Checking ONLINE mode ... OK.
10:54:12.213 - info: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS: *** Checking target board connection ... 1 connections found.
10:54:12.291 - info: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS: *** Reading ECID ... FUSELEVEL=fuselevel_production hwchipid=0x18 bootauth=NS
10:54:12.303 - info: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS: *** Reading EEPROM ... "/home/fahnoe/nvidia/nvidia_sdk/JetPack_4.6.2_Linux_JETSON_TX2_TARGETS/Linux_for_Tegra/bootloader/tegraflash.py" --chip 0x18 --applet "/home/fahnoe/nvidia/nvidia_sdk/JetPack_4.6.2_Linux_JETSON_TX2_TARGETS/Linux_for_Tegra/bootloader/mb1_recovery_prod.bin" --skipuid --cmd "dump eeprom boardinfo cvm.bin; dump eeprom baseinfo bbd.bin; reboot recovery" 
10:54:12.484 - info: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS: Welcome to Tegra Flash
10:54:12.485 - info: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS: version 1.0.0
10:54:12.486 - info: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS: Type ? or help for help and q or quit to exit
10:54:12.486 - info: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS: Use ! to execute system commands
10:54:12.487 - info: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS:  
10:54:12.487 - info: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS: [   0.0106 ] Generating RCM messages
10:54:12.487 - info: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS: [   0.0144 ] tegrarcm_v2 --listrcm rcm_list.xml --chip 0x18 0 --download rcm /home/fahnoe/nvidia/nvidia_sdk/JetPack_4.6.2_Linux_JETSON_TX2_TARGETS/Linux_for_Tegra/bootloader/mb1_recovery_prod.bin 0 0
10:54:12.543 - info: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS: [   0.0166 ] RCM 0 is saved as rcm_0.rcm
10:54:12.544 - info: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS: [   0.0191 ] RCM 1 is saved as rcm_1.rcm
10:54:12.544 - info: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS: [   0.0192 ] List of rcm files are saved in rcm_list.xml
10:54:12.544 - info: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS: [   0.0192 ] 
10:54:12.545 - info: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS: [   0.0192 ] Signing RCM messages
10:54:12.545 - info: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS: [   0.0298 ] tegrasign_v3.py --key None --list rcm_list.xml --pubkeyhash pub_key.key
10:54:12.545 - info: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS: [   0.0300 ] Assuming zero filled SBK key
10:54:12.567 - info: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS: [   0.0942 ] Copying signature to RCM mesages
10:54:12.568 - info: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS: [   0.0981 ] tegrarcm_v2 --chip 0x18 0 --updatesig rcm_list_signed.xml
10:54:12.579 - info: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS: [   0.1011 ] 
10:54:12.580 - info: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS: [   0.1011 ] Boot Rom communication
10:54:12.580 - info: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS: [   0.1051 ] tegrarcm_v2 --chip 0x18 0 --rcm rcm_list_signed.xml --skipuid
10:54:12.594 - info: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS: [   0.1074 ] RCM version 0X180001
10:54:14.721 - info: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS: [   0.1129 ] Boot Rom communication completed
10:54:14.722 - info: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS: [   1.2440 ] 
10:54:14.723 - info: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS: [   2.2500 ] tegrarcm_v2 --isapplet
10:54:14.757 - info: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS: [   2.2548 ] 
10:54:14.758 - info: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS: [   2.2844 ] tegradevflash_v2 --iscpubl
10:54:15.766 - info: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS: [   2.2874 ] CPU Bootloader is not running on device.
[...]
10:54:44.282 - info: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS: [  31.8054 ] tegradevflash_v2 --iscpubl
10:54:44.290 - info: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS: [  31.8079 ] CPU Bootloader is not running on device.
10:54:44.291 - info: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS: [  31.8108 ] 
10:54:44.307 - error: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS: Error: None of the bootloaders are running on device. Check the UART log.
10:54:44.311 - info: Event: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS - error is: Error: None of the bootloaders are running on device. Check the UART log.
10:54:44.363 - error: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS: --- Error: Reading board information failed.
10:54:44.363 - info: Event: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS - error is: --- Error: Reading board information failed.
10:54:44.363 - info: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS: [ Component Install Finished with Error ]
10:54:44.364 - info: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS: [host] [ 0.00 B released. Disk Avail: 50.66 GB ]
10:54:44.364 - info: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS: [ NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP Install took 32s ]
10:54:44.365 - info: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS: 
10:54:44.365 - error: command error code: 11
10:54:44.366 - info: Event: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS - error code is: 1011
10:54:44.366 - error: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS: command terminated with error
10:54:44.366 - info: Event: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS - error is: command terminated with error
10:54:44.377 - info: Event: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS - install ended - failure
10:54:44.378 - error: Session first failure occurred while Installing. Root cause component: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS. Root cause error code: (1011: Failed to execute commands with GenericInstaller)
10:54:44.380 - info: Summary: NV_L4T_FLASH_TX2_WITH_OS_IMAGE_COMP@JETSON_TX2_TARGETS: Installation failed.

Any words of wisdom are gratefully appreciated! I’m going to hold off on resetting or power-cycling until I either find the answer or see some additional wisdom here, so offering thanks in advance.

–Larry

fahnoe · May 10, 2022, 9:40pm

Following up on this, after more reading on other threads with problems flashing it appeared that failed flashing can be retried–I was concerned that if I power-cycled the TX2 I’d end up with a brick. So, I power-cycled, verified that it would still boot properly, then powered down, put it into recovery mode and reran the sdkmanager (using Manual Setup - Jetson TX2 in step 03). All phases of flashing and subsequent software installation appear to have completed successfully.

Summary notes:

Having self-signed TLS certs without trust in ca-certificates can cause problems
Uncertain as to what caused the flashing failure, perhaps I retried a step within sdkmanager and it got confused? Running it straight through this last time resulted in a successful installation.
Though I don’t know if it was related to the flashing problem (the initial flashing was successful yesterday), I did disable the USB autosuspend on the host: echo -1 > /sys/module/usbcore/parameters/autosuspend

–Larry

linuxdev · May 11, 2022, 7:55pm

The above would be true if the Jetson did not fully boot in the non-recovery mode. Recovery mode uses hardware to turn into a custom USB device (visible via lsusb), whereas fully booted this depends on a program running to emulate a USB device (if it doesn’t boot far enough for that program to run, then it won’t appear as a device…or if the program were disabled there would be no visible device for lsusb to see).

I don’t know what the cause would be of a failed flash where it complains of not finding the device, at least not if it is at the flash stage. Normally flash would be in recovery mode at the start, but as flash completes, then the Jetson would reboot. Once rebooted and first boot account setup is complete, then ssh is used over the network (which is what enp2s0 is mentioned for) to install further optional software. In that case flash would have completed, the Jetson would have rebooted, and then there would be some sort of network error.

The lack of serial console response says something more is wrong. In the case of a device tree failing to install I would start by asking if this is a developer kit, or if it is instead a module on a third party carrier board? If the former, then the device tree should be correct, but if the latter, then you’d need the flash software provided by the third party manufacturer for that module/carrier combination. If this is a dev kit, then I’d consider deleting your JetPack/SDK Manager install and downloading it again.

fahnoe · May 12, 2022, 7:01pm

The good news is that my third attempt at installation worked without issue.

I appreciate your time & thoughts! For a while I was worried that I’d bricked it, but as mentioned upon reading through more of the threads about flashing problems these devices appear to be pretty forgiving of problems during the flashing stage. In answer to your question: yes, this is a standard NVIDIA dev board that I’m finally getting around to working with.

–Larry

linuxdev · May 13, 2022, 5:54pm

Just some follow-up trivia…

It is hard to actually “brick” a Jetson. One normally thinks of bricking a PC motherboard from something like a botched CMOS BIOS flash, where the flash goes wrong, and the motherboard does not have a backup copy of the BIOS in case of failure. In that case the ability to install a new BIOS is gone, and the only way to fix it would be to remove the CMOS memory from motherboard, flash it on some sort of programmer, and then put it back. Jetsons don’t have a BIOS, and the logic to make it possible to program memory (eMMC for a regular module, QSPI NOR memory for an SD card model dev kit) is hard wired in and loss of all memory won’t interfere with flashing. Flashing a Jetson is essentially a simultaneous flash of the operating system, boot loader, and memory which performs the equivalent of the CMOS BIOS. No backup required since BIOS function is not needed to flash BIOS function. Most failures are from some catastrophic hardware failure, e.g., power surge.

system · June 1, 2022, 4:16am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.