Jetpack 5.1.3 , Ubuntu 18, 20.04 flashing fails, info: ...RTNETLINK answers: File exists

Jetpack 5.1.3 , Ubuntu 18, 20.04 containers stuck on final flashing step

20.04 prompts me to wait or continue,

18.04 times out and shows “cleaning up”

docker run --network host --privileged -v /dev/bus/usb:/dev/bus/usb/ -v /dev:/dev -v /media/$USER:/media/nvidia:slave -it --rm sdkmanager:latest --cli --action install --login-type devzone --product Jetson --target-os Linux --version 5.1.3 --host --target JETSON_XAVIER_NX_TARGETS --select 'Jetson Linux' --select 'Jetson Runtime Components' --select 'Jetson SDK Components' --select 'Developer Tools' --flash --license accept --stay-logged-in true

│info: ****************************************************                                                                              │
│info: *                                                  *                                                                              │
│info: *  Step 2: Boot the device with flash initrd image *                                                                              │
│info: *                                                  *                                                                              │
│info: ****************************************************                                                                              │
│info: ~/nvidia/nvidia_sdk/JetPack_5.1.3_Linux_JETSON_XAVIER_NX_TARGETS/Linux_for_Tegra/temp_initrdflash/bootloader0 ~/nvidia/nvidia_sdk │
│/JetPack_5.1.3_Linux_JETSON_XAVIER_NX_TARGETS/Linux_for_Tegra                                                                           │
│info: ./tegraflash.py --bl nvtboot_recovery_cpu_t194_sigheader.bin.encrypt --bct br_bct_BR.bct --securedev  --bldtb tegra194-p3668-all- │
│p3509-0000.dtb --applet rcm_2_encrypt.rcm --applet_softfuse rcm_1_encrypt.rcm --cmd "rcmboot"  --cfg secureflash.xml --chip 0x19        │
│--mb1_bct mb1_bct_MB1_sigheader.bct.encrypt --mem_bct mem_rcm_sigheader.bct.encrypt --mb1_cold_boot_bct mb1_cold_boot_bct_MB1_sigheader │
│.bct.encrypt --mem_bct_cold_boot mem_coldboot_sigheader.bct.encrypt  --bins "mb2_bootloader nvtboot_recovery_t194_sigheader.bin.encrypt │
│; mts_preboot preboot_c10_prod_cr_sigheader.bin.encrypt; mts_mce mce_c10_prod_cr_sigheader.bin.encrypt; mts_proper mts_c10_prod_cr_sigh │
│eader.bin.encrypt; bpmp_fw bpmp-2_t194_sigheader.bin.encrypt; bpmp_fw_dtb tegra194-a02-bpmp-p3668-a00_lz4_sigheader.dtb.encrypt;        │
│spe_fw spe_t194_sigheader.bin.encrypt; tos tos-optee_t194_sigheader.img.encrypt; eks eks_t194_sigheader.img.encrypt; kernel boot0.img;  │
│kernel_dtb kernel_tegra194-p3668-0000-p3509-0000.dtb; bootloader_dtb tegra194-p3668-all-p3509-0000_sigheader.dtb.encrypt"    --bct_back │
│up  --instance 1-9   

… many lines removed

│info: ***************************************                                                                                           │
│info: *                                     *                                                                                           │
│info: *  Step 3: Start the flashing process *                                                                                           │
│info: *                                     *                                                                                           │
│info: ***************************************                                                                                           │
│info: Waiting for target to boot-up...                                                                                                  │
│info: Waiting for target to boot-up...                                                                                                  │
│info: Waiting for target to boot-up...                                                                                                  │
│info: Waiting for target to boot-up...                                                                                                  │
│info: Waiting for target to boot-up...                                                                                                  │
│info: Waiting for target to boot-up...                                                                                                  │
│info: Waiting for target to boot-up...                                                                                                  │
│info: Waiting for target to boot-up...                                                                                                  │
│info: Waiting for target to boot-up...                                                                                                  │
│info: Waiting for device to expose ssh ...RTNETLINK answers: File exists                                                                │
│info: RTNETLINK answers: File exists                                                                                                    │
│info: ...RTNETLINK answers: File exists                                                                                                 │
│info: RTNETLINK answers: File exists                                                                                                    │
└──────────────────────────────────────

Further investigation

Installing to SD card , which just had a wipefs -a fails, As it does with a pre flashed image on it.

│                                                                                                                                            ││info: **********************                                                                                                                 │
│                                                                                                                                            ││info: exec_command: /tmp/tmp_NV_L4T_FLASH_JETSON_LINUX_COMP..sh                                                                              │
│                                                                                                                                            ││info: sudo ./nvsdkmanager_flash.sh                                                                                                           │
│                                                                                                                                            ││info: Defaulting to autoflash                                                                                                                │
│                                                                                                                                            ││info: *** Checking ONLINE mode ... OK.                                                                                                       │
│                                                                                                                                            ││info: *** Checking target board connection ... 0 connections found.                                                                          │
│                                                                                                                                            ││error: *** Error: No Jetson device found.                                                                                                    │
│                                                                                                                                            ││error: [exec_command]: /bin/bash -c /tmp/tmp_NV_L4T_FLASH_JETSON_LINUX_COMP..sh; [error]: *** Error: No Jetson device found.       

Same issue with 5.1.2

log_default_jp_sdkm.log (66.3 KB)

Are you sure the SDKM log is the full one you can get?
Serial console log shows that the USB device mode has already been turned on, so SSH connection should have been there.

Also, device tree shows that this is P3668 SKU1, which is the eMMC module without the SD card slot, then how are you able to flash with an SD card:

[ 0.069139] DTS File Name: /dvs/git/dirty/git-master_linux/kernel/kernel-5.10/arch/arm64/boot/dts/…/…/…/…/…/…/hardware/nvidia/platform/t19x/jakku/kernel-dts/tegra194-p3668-0001-p3509-0000.dts

Are you sure the SDKM log is the full one you can get?

Definitely not, the one it dumped into downloads. I just found it and used. I’ve tried these exact steps with the following combinations of parameters

  • With docker network host
  • With docker network bridge
  • WIth NVME
  • With sdcard
  • With my own username password
  • With default username password

SD card fails with different errors as seen in the second copy paste above.

It always shows the device when connected via USB and if I start the container with recovery mode on (the instructions dont mention this but its required on my host at least)

Serial console log shows that the USB device mode has already been turned on, so SSH connection should have been there.

I scrolled up by holding up arrow (hah) and the only STDERR observed were expected outputs from dd and so on , Everything else appears to be correct. I tried to find the source code for this to review and determine what its even trying to do at this failure point. Are you able to help me there?

Also, device tree shows that this is P3668 SKU1, which is the eMMC module without the SD card slot, then how are you able to flash with an SD card:

Not sure the context here but I’ve previously run it with SD cards by just flashing an SD card and inserting, it worked fine. I did some extensive testing with it. I only have one Xavier, No mistakes there. Has an SD card slot and previously worked with an SD card. (I ran PWM tests on an ESC and with an oscilloscope)

It definitely has an SD card slot. I also tried a wipefs -a on the sdcard prior to a fresh attempt.

I just need you to provide the complete log and not a screenshot.

For the SD card slot, I’m talking about the one on the module itself.
If the SD card slot is on the carrier board, then that means this is a custom board and you should ask the vendor for the right BSP.

It’s not a custom board:

This is exactly the listing and exactly what received when I purchased in 2020.

https://www.amazon.com/gp/product/B086874Q5R

The SD card slot is not on the carrier board , The NVME slot is though.

Please still give the full SDKM log.

Spent another hour trying to make this work/get you logs, Apparently it has a fake --export-logs param that does literally nothing.

Can you tell me which logs you want. I have no idea why someone programmed a UI around this tool, preventing redirecting STDOUT to a log. What a decision that was. Or why like any normal unit test, it doesn’t summarize every failure itself at the end. It’s like NVIDIA doesn’t want people to use their hardware.

a@b:~$ cd …;docker run --network host --privileged -v /dev/bus/usb:/dev/bus/usb/ -v /dev:/dev -v /media/$USER:/media/nvidia:slave -it --rm sdkmanager:2.1.0.11660-Ubuntu_20.04 --cli --action install --login-type devzone --product Jetson --target-os Linux --version 5.1.3 --host --target JETSON_XAVIER_NX_TARGETS --select ‘Jetson Linux’ --select ‘Jetson Runtime Components’ --select ‘Jetson SDK Components’ --select ‘Developer Tools’ --flash --license accept --stay-logged-in true --export-logs logs

===== INSTALLATION FAILED =====
- CUDA on Host: Installed
- CUDA Cross Compile Package on Host: Installed
- NvSci: Installed
- VPI on Host: Installed
- Nsight Systems: Installed
- Nsight Graphics: Installed
- Nsight Compute: Installed
- Nsight Perf SDK: Installed
- Drivers for Jetson: Installed
- File System and OS: Installed
- Flash Jetson Linux: Failed
- DateTime Target Setup: DependencyFailure
- CUDA Runtime: DependencyFailure
- CuDNN Runtime: DependencyFailure
- TensorRT Runtime: DependencyFailure
- OpenCV Runtime: DependencyFailure
- CuPVA Runtime: DependencyFailure
- VPI Runtime: DependencyFailure
- NVIDIA Container Runtime with Docker integration (Beta): DependencyFailure
- Multimedia API: DependencyFailure
- CUDA Toolkit for L4T: DependencyFailure
- cuDNN on Target: DependencyFailure
- TensorRT on Target: DependencyFailure
- OpenCV: DependencyFailure
- VPI on Target: DependencyFailure
- Nsight Systems: DependencyFailure
- Nsight Graphics: DependencyFailure
- Nsight Compute: DependencyFailure

===== Installation failed - Total 28 components =====
===== 10 succeeded, 1 failed, 0 up-to-date, 17 skipped =====

a@b:~$ ls logs #empty

Forgive me for thinking export logs would export the logs…

Ok overriding the entrypoint with /bin/bash the SDKM is actually usable. Please update documentation to instruct people to do it this way or there is literally no existing interface to view the logs.

docker run --network host --privileged -v /dev/bus/usb:/dev/bus/usb/ -v /dev:/dev -v /media/$USER:/media/nvidia:slave -it --rm --entrypoint /bin/bash sdkmanager:2.1.0.11660-Ubuntu_20.04

flash.log (189.9 KB)
sdkm.log (1.6 MB)

Can you please try this method instead?
https://docs.nvidia.com/jetson/archives/r35.5.0/DeveloperGuide/SD/FlashingSupport.html#flashing-to-an-nvme-drive

Race condition there. I already read that doc and saw it needs to boot. It wont boot.

Whether the sdkm flashed internal emmc im not sure but it certainly failed flashing the sdcard.

Either way, i need it to boot to do this.

To set up an NVMe drive manually for booting1. Confirm that the device can boot successfully from eMMC. If it cannot, correct the problem by flashing to eMMC first.

You can first flash to the SD card and make sure it’s bootable before you deal with NVMe.

Ive tried the sdcard as many times as i tried the nvme.

The second paste in OP above shows that failure.

I assure you ive lost 20+ hours trying everything. Read every doc long before i posted.

I initally tried just an sdcard flash directly to the sdcard. No boot. I tried again with your recommended flashing software instead of dd. no boot.

I don’t really know what error you are reffering to with the SD card.

Actually you don’t need either docker or SDK Manager.
Manually download the BSP package here:
https://developer.nvidia.com/downloads/embedded/l4t/r35_release_v5.0/release/jetson_linux_r35.5.0_aarch64.tbz2
https://developer.nvidia.com/downloads/embedded/l4t/r35_release_v5.0/release/tegra_linux_sample-root-filesystem_r35.5.0_aarch64.tbz2

Follow this link to setup:
https://docs.nvidia.com/jetson/archives/r35.5.0/DeveloperGuide/IN/QuickStart.html

, and flash the SD card with:

sudo ./flash.sh jetson-xavier-nx-devkit internal

I’m sorry you don’t understand what I’m referring to in the original post I clearly noted that I have tried flashing the SD card which is why I’m trying to explain to you that I don’t think it will work I followed the exact guide on how to do this I followed the instructions specifically for these Xavier NX development kit which instructed me to download a jp513-xnx-sd-card-image.zip

Flashing the sdcard with this image using etcher as per instructions results in nvidia logo, green light on. No boot.

Neither did it flash the sdcard using sdkm. In the second terminal paste in the original post and in the explanation i have noted this error.

Please stop telling me to flash the sdcard. I have .

Please address my comments.

You didn’t tell anything about whether you are using SDKM or the SD card image at all…

Do this

sudo ./tools/kernel_flash/l4t_initrd_flash.sh --external-device nvme0n1p1 \
      -c tools/kernel_flash/flash_l4t_external.xml \
      -p "-c bootloader/t186ref/cfg/flash_l4t_t194_qspi_p3668.xml" \
      --showlogs --network usb0 jetson-xavier-nx-devkit internal

, and show both log on the host and log from the serial console.

You didn’t tell anything about whether you are using SDKM or the SD card image at all…

I did, I also confirmed the same with 5.1.2 If the context wasn’t clear from posting about SDKM, I was posting about SDKM.

“As it does with a pre flashed image on it” - means I also did a manual flash myself.

Are you saying this documentation is wrong with your advice not to follow it and to instead take alternate steps?

Perhaps this might explain , why nothing is working. Because, the documentation is known wrong?

sudo ./tools/kernel_flash/l4t_initrd_flash.sh --external-device nvme0n1p1 \
      -c tools/kernel_flash/flash_l4t_external.xml \
      -p "-c bootloader/t186ref/cfg/flash_l4t_t194_qspi_p3668.xml" \
      --showlogs --network usb0 jetson-xavier-nx-devkit internal

Can I recommend, Perhaps, Fixing that?

The guide here is for Jetson Orin Nano devkit. Can you try this link for your Jetson Xavier NX?
Jetson Xavier NX Developer Kit - Get Started | NVIDIA Developer

I think you can download SD card image for JP 5.1.3 from here
JetPack SDK | NVIDIA Developer

Was about to post this evidence I flashed the card so I was believed, this time it booted after a 40 second delay.

hooray.

/dev/sdc1: UUID="39aee467-ca2c-49ad-bbd6-38a5b40831d9" BLOCK_SIZE="4096" TYPE="ext4" PARTLABEL="APP" PARTUUID="4d67cfb9-ddc9-4ff1-89f9-5413cfc72fe2"
/dev/sdc2: PARTLABEL="kernel" PARTUUID="f0025ee8-2170-4d26-8785-3cff9b525839"
/dev/sdc3: PARTLABEL="kernel-dtb" PARTUUID="a415e277-b0a8-4554-9c82-95c65a31019b"
/dev/sdc4: PARTLABEL="reserved_for_chain_A_user" PARTUUID="cf6c4c75-1d40-4e4b-83b0-c560bee55f69"
/dev/sdc5: PARTLABEL="secure-os_b" PARTUUID="b58735a5-5a2e-411a-af64-fc895b548c6a"
/dev/sdc6: PARTLABEL="eks_b" PARTUUID="a8e219ca-e5fa-4be2-a07e-1098ca1732d1"
/dev/sdc7: PARTLABEL="adsp-fw_b" PARTUUID="aabf27a7-bc5a-4703-8d91-4ebc35f96efe"
/dev/sdc8: PARTLABEL="rce-fw_b" PARTUUID="cd1515df-bc3c-43d8-9d73-b53822c6ad3a"
/dev/sdc9: PARTLABEL="sce-fw_b" PARTUUID="1183a9e7-0b7b-4e4f-bb78-32f6468b06a7"
/dev/sdc10: PARTLABEL="bpmp-fw_b" PARTUUID="a87fbc41-3032-4b72-8d2e-805d81d421e7"
/dev/sdc11: PARTLABEL="bpmp-fw-dtb_b" PARTUUID="6b171212-6649-47ab-b53c-c969cd426a72"
/dev/sdc12: PARTLABEL="kernel_b" PARTUUID="543f31dd-c0f9-4bbc-ba07-a98924996bc3"
/dev/sdc13: PARTLABEL="kernel-dtb_b" PARTUUID="b6e3bed1-b317-4875-ab05-bd87c50d17de"
/dev/sdc14: PARTLABEL="reserved_for_chain_B_user" PARTUUID="d214f0a0-b06d-4cc3-99c7-98081a1dcbc1"
/dev/sdc15: PARTLABEL="recovery" PARTUUID="a0fb25c8-9e6a-4052-9a4b-0ade78737407"
/dev/sdc16: PARTLABEL="recovery-dtb" PARTUUID="bd849a1b-0f9c-46e8-bf9e-821452047859"
/dev/sdc17: PARTLABEL="RECROOTFS" PARTUUID="40933a3b-ba68-42ab-8d59-8d696028003b"
/dev/sdc18: UUID="CFE8-5BF9" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="esp" PARTUUID="2f7cd2ba-7821-4aec-8329-89688a703d96"
/dev/sdc19: PARTLABEL="recovery_alt" PARTUUID="0d96df79-c6f0-4e9f-9554-f055991ff6c3"
/dev/sdc20: PARTLABEL="recovery-dtb_alt" PARTUUID="fe5f1718-622f-4cb3-a912-a146e618745d"
/dev/sdc21: PARTLABEL="esp_alt" PARTUUID="ee936638-2a06-4199-a266-dc9551cef865"
/dev/sdc22: PARTLABEL="UDA" PARTUUID="ed584243-c56a-427c-a383-60b8083fe711"

Thank you very much for the help. Please convert SDKMS --cli to normal shell output so we can debug easier . The GUI is not helpful. It’s literally not a CLI.

Your documentation specifically linked me to that page for the Xavier NX,

Follow it yourself and you will find this. No BS.

It also booted for the first time using that pages instruction now…

I am glad that it boots. Do you have any other questions?