SDK Manager install fails

I am running sdkmanager inside Docker (--cli) to install to my Jetson Xavier NX. It gets to 99% of creating the Linux image, and successfully downloads all of the parts, but then it all turns red. The “terminal log” section is extremely hard to scroll through, and I am not convinced it still has all of the failures.

Is there a log file somewhere that I could look at?

Here is a screen capture.

Running sdkmanager-2.0.0.11402-Ubuntu_20.04 in Docker, it definitely finds the device at USB 4-1. The container has -v /dev/bus/usb:/dev/bus/usb to mount the USB bus from the host into the container.

The sdkmanager command is: sdkmanager --cli --action install --login-type devzone --product Jetson --target-os Linux --version 5.1.2 --host --target JETSON_XAVIER_NX_TARGETS --flash

Help?

Update: I logged into the container, found ~/.nvsdkm/logs/JetPack_5.1.2_Linux, which has the following:

NV_CUDA_CROSS_HOST_COMP.log
NV_CUDA_HOST_COMP.log
NV_L4T_DRIVERS_COMP.log
NV_L4T_FILE_SYSTEM_AND_OS_COMP.log
NV_L4T_NVIDIA_NSIGHT_GRAPHICS_HOST_COMP.log
NV_L4T_NVIDIA_NSIGHT_PERFSDK_HOST_COMP.log
NV_L4T_NVIDIA_NSIGHT_SYSTEMS_HOST_COMP.log
NV_NVIDIA_NSIGHT_COMPUTE_HOST_COMP.log
NV_NVSCI_HOST_COMP.log

The word error (any case) appears only once, in NV_L4T_FILE_SYSTEM_AND_OS_COMP.log, towards the end (last ~12 lines):

Root file system directory is /home/nvidia/nvidia/nvidia_sdk/JetPack_5.1.2_Linux_JETSON_XAVIER_NX_TARGETS/Linux_for_Tegra/rootfs
Copying public debian packages to rootfs
Skipping installation of nvidia-l4t-dgpu-apt-source_35.4.1-20230801124926_arm64.deb ....
Skipping installation of nvidia-l4t-dgpu-config_35.4.1-20230801124926_arm64.deb ....
Skipping installation of nvidia-l4t-factory-service_35.4.1-20230801124926_arm64.deb ....
Start L4T BSP package installation
QEMU binary is not available, looking for QEMU from host system
Found /usr/bin/qemu-aarch64-static
Installing QEMU binary in rootfs
/home/nvidia/nvidia/nvidia_sdk/JetPack_5.1.2_Linux_JETSON_XAVIER_NX_TARGETS/Linux_for_Tegra/rootfs /home/nvidia/nvidia/nvidia_sdk/JetPack_5.1.2_Linux_JETSON_XAVIER_NX_TARGETS/Linux_for_Tegra
Installing BSP Debian packages in /home/nvidia/nvidia/nvidia_sdk/JetPack_5.1.2_Linux_JETSON_XAVIER_NX_TARGETS/Linux_for_Tegra/rootfs
13:32:39.511 - Error: [exec_command]: /bin/bash -c /tmp/tmp_NV_L4T_FILE_SYSTEM_AND_OS_COMP..sh; [error]: chroot: failed to run command 'dpkg': Exec format error


13:32:39.518 - Info: [ Package Install Finished with Error ]
13:32:39.528 - Info: [host] [ 10.52 GB used. Disk Avail: 57.80 GB ]
13:32:39.532 - Info: [ NV_L4T_FILE_SYSTEM_AND_OS_COMP Install took 39m27s ]

What is wrong?

I didn’t have binfmt_misc enabled, so I enabled it and tried again. I still get an error at the same place, but slightly different. From the log:

QEMU binary is not available, looking for QEMU from host system
Found /usr/bin/qemu-aarch64-static
Installing QEMU binary in rootfs
/home/nvidia/nvidia/nvidia_sdk/JetPack_5.1.2_Linux_JETSON_XAVIER_NX_TARGETS/Linux_for_Tegra/rootfs /home/nvidia/nvidia/nvidia_sdk/JetPack_5.1.2_Linux_JETSON_XAVIER_NX_TARGETS/Linux_for_Tegra
Installing BSP Debian packages in /home/nvidia/nvidia/nvidia_sdk/JetPack_5.1.2_Linux_JETSON_XAVIER_NX_TARGETS/Linux_for_Tegra/rootfs
18:18:16.212 - Error: [exec_command]: /bin/bash -c /tmp/tmp_NV_L4T_FILE_SYSTEM_AND_OS_COMP..sh; [error]: /home/nvidia/nvidia/nvidia_sdk/JetPack_5.1.2_Linux_JETSON_XAVIER_NX_TARGETS/Linux_for_Tegra/nv_tegra/nv-apply-debs.sh: line 271:  1866 Aborted                 LC_ALL=C PYTHONHASHSEED=0 chroot . dpkg -i --path-include="/usr/share/doc/*" "${pre_deb_list[@]}"

And from the terminal of the CLI sdkmanager:

│info: QEMU binary is not available, looking for QEMU from host system                                                                                                                       │
│info: Found /usr/bin/qemu-aarch64-static                                                                                                                                                    │
│info: Installing QEMU binary in rootfs                                                                                                                                                      │
│info: /home/nvidia/nvidia/nvidia_sdk/JetPack_5.1.2_Linux_JETSON_XAVIER_NX_TARGETS/Linux_for_Tegra/rootfs /home/nvidia/nvidia/nvidia_sdk/JetPack_5.1.2_Linux_JETSON_XAVIER_NX_TARGETS/Linux_ │
│for_Tegra                                                                                                                                                                                   │
│info: Installing BSP Debian packages in /home/nvidia/nvidia/nvidia_sdk/JetPack_5.1.2_Linux_JETSON_XAVIER_NX_TARGETS/Linux_for_Tegra/rootfs                                                  │
│error: qemu: qemu_thread_create: Operation not permitted                                                                                                                                    │
│error: /home/nvidia/nvidia/nvidia_sdk/JetPack_5.1.2_Linux_JETSON_XAVIER_NX_TARGETS/Linux_for_Tegra/nv_tegra/nv-apply-debs.sh: line 271:  1866 Aborted                 LC_ALL=C PYTHONHASHSE │
│ED=0 chroot . dpkg -i --path-include="/usr/share/doc/*" "${pre_deb_list[@]}"                                                                                                                │
│error: [exec_command]: /bin/bash -c /tmp/tmp_NV_L4T_FILE_SYSTEM_AND_OS_COMP..sh; [error]: /home/nvidia/nvidia/nvidia_sdk/JetPack_5.1.2_Linux_JETSON_XAVIER_NX_TARGETS/Linux_for_Tegra/nv_te │
│gra/nv-apply-debs.sh: line 271:  1866 Aborted                 LC_ALL=C PYTHONHASHSEED=0 chroot . dpkg -i --path-include="/usr/share/doc/*" "${pre_deb_list[@]}"                             │
│info:                                                                                                                                                                                       │
│info:                                                                                                                                                                                       │
│info: [ Package Install Finished with Error ]                                                                                                                                               │
1 Like

@avi24 ,

error: qemu: qemu_thread_create: Operation not permitted

This seems to be the cause.
Could you share the host OS version and installed docker version?
Will the solution(upgrading docker version to 20.10.10+) work for you:

Hi @changbins ; thanks for jumping in to help.

Could you share the host OS version and installed docker version?

Docker was 20.10.6, which I realized after posting here is pretty old.
I just bumped docker to latest available (24.0.6) and am rerunning. I will report back here what happens.

As for host OS, it is a minimally built custom image. Very little on it except for kernel, minimal init, containerd, runc and docker. kernel is 5.10.104. I should get that into the 6.x series, but I doubt that is the issue.

Managed to get to the same point, failed again, but with a different error message:

│info: QEMU binary is not available, looking for QEMU from host system                                                                                                                       │
│info: Found /usr/bin/qemu-aarch64-static                                                                                                                                                    │
│info: Installing QEMU binary in rootfs                                                                                                                                                      │
│error: mknod:                                                                                                                                                                               │
│error: /home/nvidia/nvidia/nvidia_sdk/JetPack_5.1.2_Linux_JETSON_XAVIER_NX_TARGETS/Linux_for_Tegra/rootfs/dev/random                                                                        │
│error: : File exists                                                                                                                                                                        │
│error: [exec_command]: /bin/bash -c /tmp/tmp_NV_L4T_FILE_SYSTEM_AND_OS_COMP..sh; [error]:                                                                                                   │
│info:                                                                                                                                                                                       │```

It looks like it doesn’t like the output of previous runs? I tried removing /home/nvidia/nvidia/nvidia_sdk/JetPack_5.1.2_Linux_JETSON_XAVIER_NX_TARGETS/Linux_for_Tegra/rootfs/ entirely and running again. Will report here.

I got past the previous stage. It now is asking if I am ready to flash the device, which config, to which storage (I picked eMMC), and to enter a username and password to flash. This is good.

Now I get an error:

│info: Device mode on host successfully set with DNS 8.8.8.8!                                                                                                                                │
│info:                                                                                                                                                                                       │
│info: command finished successfully                                                                                                                                                         │
│debug: running command < true >                                                                                                                                                             │
│info: exec_command: true                                                                                                                                                                    │
│info: command finished successfully                                                                                                                                                         │
│debug: using adapter to install NV_L4T_FLASH_JETSON_LINUX_COMP@JETSON_XAVIER_NX_TARGETS to /home/nvidia/nvidia/nvidia_sdk/JetPack_5.1.2_Linux_JETSON_XAVIER_NX_TARGETS                      │
│info:                                                                                                                                                                                       │
│info: [host] [ Disk Avail:56.88 GB ]                                                                                                                                                        │
│info: Installing component 'Flash Jetson Linux' (NV_L4T_FLASH_JETSON_LINUX_COMP)                                                                                                            │
│info: change working directory to /home/nvidia/nvidia/nvidia_sdk/JetPack_5.1.2_Linux_JETSON_XAVIER_NX_TARGETS/Linux_for_Tegra                                                               │
│info: [ Component Install Started ]                                                                                                                                                         │
│info: current working directory is /home/nvidia/nvidia/nvidia_sdk/JetPack_5.1.2_Linux_JETSON_XAVIER_NX_TARGETS/Linux_for_Tegra                                                              │
│info: exec_command [host]:                                                                                                                                                                  │
│info: **********************                                                                                                                                                                │
│info: set -e                                                                                                                                                                                │
│info: unset LC_ALL                                                                                                                                                                          │
│info: echo sudo ./nvsdkmanager_flash.sh  --nv-auto-config --username zededa                                                                                                                 │
│info: sudo ./nvsdkmanager_flash.sh  --nv-auto-config --username zededa                                                                                                                      │
│info: sleep 20                                                                                                                                                                              │
│info: if [[ ' --nv-auto-config --username zededa' == *'--nv-auto-config'* ]]; then                                                                                                          │
│info: sleep 40                                                                                                                                                                              │
│info: fi                                                                                                                                                                                    │
│info: **********************                                                                                                                                                                │
│info: exec_command: /tmp/tmp_NV_L4T_FLASH_JETSON_LINUX_COMP..sh                                                                                                                             │
│info: sudo ./nvsdkmanager_flash.sh --nv-auto-config --username zededa                                                                                                                       │
│info: [OEMPreconfig] SDKM_INSTALL_ERROR Password for L4T new user zededa:                                                                                                                   │
│info:                                                                                                                                                                                       │
│info: *** Checking ONLINE mode ... OK.                                                                                                                                                      │
│info: *** Checking target board connection ... 1 connections found.                                                                                                                         │
│error: *** Reading ECID ... SDKM_INSTALL_ERROR *** Error: ECID read failed.                                                                                                                 │
│error: Put the target board in RCM mode and retry.                                                                                                                                          │
│error: [exec_command]: /bin/bash -c /tmp/tmp_NV_L4T_FLASH_JETSON_LINUX_COMP..sh; [error]: Put the target board in RCM mode and retry.                                                       │
│info:                                                                                                                                                                                       │
│info:                                                                                                                                                                                       │
│info: [ Component Install Finished with Error ]                                                                                                                                             │
...
│error: command terminated with error                                                                                                                                                        │
│info: Jetson board is in a bad state and cannot be recovered. Please manually put board to recovery mode again and retry.. The Jetson target is in a bad state and cannot be flashed.       │
│Please manually put the target into recovery mode and then retry flashing.                                                                                                                  │

The device is in recovery mode. I checked both by running --list-connected all:

Jetson:
Device Name | USB Port | USB Bus/Device | USB Specification | Network Interface | MAC Address | Status
--------------------------------------------------------------------------------------------------------
            | 4-1      | 004/003        | 2.00              | null              | null        | Recovery

and by running lsusb per this doc:

root@c959ff2ae04b:/home/nvidia# lsusb
Bus 004 Device 003: ID 0955:7e19 NVIDIA Corp. APX
Bus 004 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 003 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub

│error: *** Reading ECID … SDKM_INSTALL_ERROR *** Error: ECID read failed. │
│error: Put the target board in RCM mode and retry.

│info: Jetson board is in a bad state and cannot be recovered. Please manually put board to recovery mode again and retry… The Jetson target is in a bad state and cannot be flashed. │
│Please manually put the target into recovery mode and then retry flashing.

Although the Jetson board appears to be in recovery mode according to the lsusb output, actually it is in a bad state and cannot be used for flashing. Can you please try to manually put the board to recovery mode again and retry?

By “manually”, do you mean jumpering together the 2nd and 3rd pins? I thought I had, but I will disconnect them and push it back in; maybe it wasn’t in firmly enough to make contact?

I had thought that if --list-connected all showed it as in recovery mode, then it is in recovery mode, i.e. pins properly connected? Are there different types of “recovery mode”?

Definitely got the pins in more tightly this time. Ran it again. Some old, some new errors:

│info: Device info:                                                                                                                                                                          │
│004/002 | 4-1 | 2.00 | Recovery                                                                                                                                                             │
│                                                                                                                                                                                            │
│info: Start to check If Jetson device Ready To Flash Recovery...                                                                                                                            │
│debug: running command < true >                                                                                                                                                             │
│info: exec_command: true                                                                                                                                                                    │
│info:                                                                                                                                                                                       │
│info: command finished successfully                                                                                                                                                         │
│debug: running command < /tmp/device_mode_host_setup.undefined.sh >                                                                                                                         │
│info: exec_command: /tmp/device_mode_host_setup.undefined.sh                                                                                                                                │
│info: Found route default eth0, setting up device mode                                                                                                                                      │
│error: getsockopt failed strangely: Operation not permitted                                                                                                                                 │
│error: iptables v1.8.4 (legacy):                                                                                                                                                            │
│error: can't initialize iptables table `filter': Permission denied (you must be root)                                                                                                       │
│error: Perhaps iptables or your kernel needs to be upgraded.                                                                                                                                │
│error: getsockopt failed strangely: Operation not permitted                                                                                                                                 │
│error: sysctl:                                                                                                                                                                              │
│error: setting key "net.ipv4.ip_forward", ignoring                                                                                                                                          │
│error: : Read-only file system                                                                                                                                                              │
│info: Device mode on host successfully set with DNS 8.8.8.8!                                                                                                                                │
│info:                                                                                                                                                                                       │
│info: command finished successfully                                                                                                                                                         │
│info: Found file /home/nvidia/Downloads/nvidia/sdkm_downloads/Tegra_Linux_Sample-Root-Filesystem_R35.4.1_aarch64.tbz2 with correct checksum, skip downloading.                              │
│debug: running command < true >                                                                                                                                                             │
│info: verifying checksum of /home/nvidia/Downloads/nvidia/sdkm_downloads/cuda-repo-l4t-11-4-local_11.4.19-1_arm64.deb                                                                       │
│info: exec_command: true                                                                                                                                                                    │
│info: command finished successfully                                                                                                                                                         │
│debug: using adapter to install NV_L4T_FLASH_JETSON_LINUX_COMP@JETSON_XAVIER_NX_TARGETS to /home/nvidia/nvidia/nvidia_sdk/JetPack_5.1.2_Linux_JETSON_XAVIER_NX_TARGETS                      │
│info:                                                                                                                                                                                       │
│info: [host] [ Disk Avail:57.11 GB ]                                                                                                                                                        │
│info: Installing component 'Flash Jetson Linux' (NV_L4T_FLASH_JETSON_LINUX_COMP)                                                                                                            │
│info: change working directory to /home/nvidia/nvidia/nvidia_sdk/JetPack_5.1.2_Linux_JETSON_XAVIER_NX_TARGETS/Linux_for_Tegra                                                               │
│info: [ Component Install Started ]                                                                                                                                                         │
│info: current working directory is /home/nvidia/nvidia/nvidia_sdk/JetPack_5.1.2_Linux_JETSON_XAVIER_NX_TARGETS/Linux_for_Tegra                                                              │
│info: exec_command [host]:                                                                                                                                                                  │
│info: **********************                                                                                                                                                                │
│info: set -e                                                                                                                                                                                │
│info: unset LC_ALL                                                                                                                                                                          │
│info: echo sudo ./nvsdkmanager_flash.sh  --nv-auto-config --username zededa                                                                                                                 │
│info: sudo ./nvsdkmanager_flash.sh  --nv-auto-config --username zededa                                                                                                                      │
│info: sleep 20                                                                                                                                                                              │
│info: if [[ ' --nv-auto-config --username zededa' == *'--nv-auto-config'* ]]; then                                                                                                          │
│info: sleep 40                                                                                                                                                                              │
│info: fi                                                                                                                                                                                    │
│info: **********************                                                                                                                                                                │
│info: exec_command: /tmp/tmp_NV_L4T_FLASH_JETSON_LINUX_COMP..sh                                                                                                                             │
│info: sudo ./nvsdkmanager_flash.sh --nv-auto-config --username zededa                                                                                                                       │
│error: ERROR: please run "apt install whois" to install required package                                                                                                                    │
│info: Usage:                                                                                                                                                                                │
│info: nv_preseed.sh -u <username> [-p <password>]                                                                                                                                           │
│info: nv_preseed.sh -h                                                                                                                                                                      │
│info:   -u | --username - username for new user account                                                                                                                                     │
│info:   -p | --password - hash data of password for new user account                                                                                                                        │
│info:   -h | --help - print this usage                                                                                                                                                      │
│info: *** Checking ONLINE mode ... OK.                                                                                                                                                      │
│info: *** Checking target board connection ... 1 connections found.                                                                                                                         │
│error: *** Reading ECID ... SDKM_INSTALL_ERROR *** Error: ECID read failed.                                                                                                                 │
│error: Put the target board in RCM mode and retry.                                                                                                                                          │
│error: [exec_command]: /bin/bash -c /tmp/tmp_NV_L4T_FLASH_JETSON_LINUX_COMP..sh; [error]: *** Error: ECID read failed.                                                                      │
│error: Put the target board in RCM mode and retry.                                                                                                                                          │
│error: [exec_command]: /bin/bash -c /tmp/tmp_NV_L4T_FLASH_JETSON_LINUX_COMP..sh; [error]: *** Error: ECID read failed.                                                                      │
│info: Put the target board in RCM mode and retry.                                                                                                                                           │
│info:                                                                                                                                                                                       │
│info:                                                                                                                                                                                       │
│info: [ Component Install Finished with Error ]                                                                                                                                             │
│info: [host] [ 8.00 KB used. Disk Avail: 57.11 GB ]                                                                                                                                         │
│info: [ NV_L4T_FLASH_JETSON_LINUX_COMP Install took 3s ]                                                                                                                                    │
│info:                                                                                                                                                                                       │
│error: command terminated with error                                                                                                                                                        │
│info: Jetson board is in a bad state and cannot be recovered. Please manually put board to recovery mode again and retry.. The Jetson target is in a bad state and cannot be flashed.       │
│Please manually put the target into recovery mode and then retry flashing.                                                                                                                  │

A lot of it seems to centre around ECID read failed.

I do not know what that means, or how to fix it?

I also tried doing it manually using flash.sh, it didn’t even get as far:

nvidia@7bdf16f61e08:~/nvidia/nvidia_sdk/JetPack_5.1.2_Linux_JETSON_XAVIER_NX_TARGETS/Linux_for_Tegra$ sudo ./flash.sh jetson-xavier-nx-devkit-emmc mmcblk0p1
###############################################################################
# L4T BSP Information:
# R35 , REVISION: 4.1
# User release: 0.0
###############################################################################
Error: probing the target board failed.
       Make sure the target board is connected through
       USB port and is in recovery mode.

I dug even deeper, tracing each script one step at a time. I eventually came down to the failure coming from this binary, called from the script:

$ /home/nvidia/nvidia/nvidia_sdk/JetPack_5.1.2_Linux_JETSON_XAVIER_NX_TARGETS/Linux_for_Tegra/bootloader/tegrarcm_v2 --uid
USB communication failed.Check if device is in recovery

I know for sure that the device is in recovery, and I can see it in lsusb and --list-connected all. I am at a loss as to what this means.

Hi avi24,
Can you please try the following steps?

  1. Power off the board.
  2. Power on the board.
  3. Manually put the board into recovery mode.
  4. Try to flash the board again.

Although the board seems to be in recovery mode from lsusb command, however, it has passed the state when ECID can be read. In other words, it is not ready for flashing.
At this moment, we need to manually go through the whole process, to put the board to recovery mode again. This means to power off or reset the board, then put it into recovery mode again. The purpose is to reset board status, so that ECID can be read when retrying flash next time.

Based on your comment, should I be powering on and then putting it in recovery mode? I have pins, not button, so I usually:

  1. Power down by removing the barrel power connector.
  2. Put in recovery mode by jumping together pins 9 &10 (GND & FC REC).
  3. Power on by putting in the barrel power connector (green LED light goes on).

I usually just leave the USB cable connected to my computer the whole time, since with no power, nothing is running through it.

by the way, what is ECID? I actually do not know.

Hi avi24,
I don’t have Orin Nano board. But your above three steps look good to me. USB cable can be kept connected as is.
ECID is a chip unique id.

Hi @EdwardZhou

It isn’t an Orin Nano, it is a Xavier NX.

Thanks for explaining the ECID.

your above three steps look good to me

Then why does tegrarcm_v2 always fail to find it in recovery mode?

Is there a way I can debug what it is doing? Even low-level would be fine, although I am not about to buy a USB analyzer; I imagine it should be readable from within the OS?

After you got the flashing failure, did you try to use the three steps to put the board into recovery mode again and reflash? I have no more details about the low-level analysis. If you still have an issue with flashing, I’ll forward your question to our TSE team for more investigation.

I did. I will try it again right now.

Same thing. It fails at the same point, calling tegrarcm_v2 --uid, whether I use flash.sh or sdkmanager, and when I call it manually, same thing. It doesn’t matter if I call tegrarcm_v2 --uid just like that, or if I explicitly give it --instance.

If I give it an invalid instance, I get No device on specified bus-port[4-12], but if I give it a valid one (in my case, it always ends up as 4-1), or if I give it none, I get the same result:

USB communication failed.Check if device is in recovery

When I actually connect the device (or power it up), I see the following or similar in dmesg:

[43098.357535] usb 4-1: new high-speed USB device number 12 using ehci-pci

Hi avi24,
I’ve forwarded your question to TSE team for further investigation. Thanks for your patience.