Flashing / cloning Jetson TX2 fails with JetPack 4.2/4.3

Autonomous Machines Jetson & Embedded Systems Jetson TX2
#1

Hello.
We have 2 Jetson TX2 with auvidea J120 board. They are running our custom video streaming programs and we need to clone them to another boards without change.
In the past i managed to clone 1 to another with this simple manual.
Jetson/TX2 Cloning - eLinux.org

When i tried to replicate it now to another new 3 (Jetson TX2 + Auvidea J120) boards.

I tried to do i according to manual, but i quickly got this error.

failedclone
failedclone1317×781 269 KB

So i have found here
Flash TX2 failed with Error: Return value 13 - Jetson & Embedded Systems / Jetson TX2 - NVIDIA Developer Forums
that first i need to flash jetson with same releases of JetPack (propably with auvidea package) as they are on device that is being cloned.

I managed to do it in the past with auvidea manual provided with firmwares (dowload JetPack, copy kernel_out, then apply binaries and run flash.sh)
But right now it doesnt work.
Where i try to flash JetPack 4.2 or 4.3 with or without coresponding auvidea package it fails (stucks forever) at

failedflash
failedflash1478×963 285 KB

It doesnt matter if i do it via sdk manager or just flash.sh command.
It never finishes successfully.
The only way i am able to make jetson work (at least somehow)is by uploading newest JetPack 4.4.1 via SDK manager (without auvidea package). This is only thing that works.
And Jetson is able to run after this.
Then when i the try to clone older jetson which was built with JP4.2 and corresponding auvidea package with flash.sh from JetPAck 4.4 folder on host machine, it finishes fine, but after i try to run the Jetson TX2 it keeps rebooting on NVIDIA logo.
Can anyone please help me to solve my problems?
I only need to clone my older jetson image to new boards.
I tried with Ubuntu 16.04, 18.04, 20.04 (all installed ubuntus) and sdk latest 14 and previous 13.x and all finished with the same error.

#2

And please be aware that i am quite a noob at theses things, so if you try to help me, do it the way i might be able to understand what you are suggesting :-D

#3

The actual flash does not require anything special for the carrier board, but to boot, you must use the firmware/device tree content provided by the carrier board manufacturer. Actual boot will fail without that.

A desktop PC has a CMOS BIOS/UEFI setup. This sets up clocks, routing to various devices, so on, and then provides a uniform interface for bootloaders. Embedded systems do not have this, and those other partitions (aside from the rootfs) are essentially a custom CMOS BIOS plus bootloader.

Jetsons which already have that non-rootfs content in place will succeed if a matching rootfs (there are version dependencies) from a clone is placed without harming the non-rootfs content (or if the non-rootfs content is added during the flash of the clone).

Actual flash should not stop in the middle of a flash, but unless you use your third party carrier board’s software, it isn’t possible to say exactly what went wrong.

One thing known to cause flashes to stop in the middle is if the host is a VM. Right now I think the only reliable host PC o/s to use for flash is a non-VM Ubuntu 18.04.

#4

Ok. so that explaines why it wont boot without auvidea firmware. But i also tried mutliple times with it, and it doest work that way either.
And all ubuntus i tried were installed on pc. None of them was VM.
But all on the same pc if that might be a problem.

#5

Usually when a flash halts it is related to losing USB. One case for that is a VM, but since you are not using a VM, then we can move on to other causes.

The dev kits have a big advantage most people won’t ever think about: They come with a quality micro-B USB cable intended for data, and it isn’t just a charger cable with 2 strands of copper for data as an afterthought. I’d say about 3 out of 4 “charger” cables will fail under sustained data activity. This would more or less be the same symptoms as a VM when it loses USB.

Sometimes I hear about a NUC appliance not working, and they seem to have some issue with their USB ports. On rare occasion someone has an older laptop with a bad port, and switching to another computer helps. Often having only the Jetson on the port for flash helps in comparison to a HUB with other devices. A HUB might help (due to signal quality), but in that case you’d want only the Jetson to be on the HUB.

#6

I undertand, however the root of the problem is propably somewhere else.

  1. I tried 3 different USB cables (i actually was able to flash with them JetPack 4.4.1 as i wrote in first post.). They are not cheap chinese cables. But no i dont have yours USB cable, that is sure.
  2. I am not using any HUB.
  3. I tried ports on my laptops both sides
#7

On your host PC be sure to monitor “dmesg --follow” while flashing, see what might show up. I’m running low on ideas other than testing on yet another host PC just to see if the problem definitely sticks to the new laptop or not.

#8

Ok. Ill do that. Ill report any changes

#9

Ok. So i tried with different computer and this is in dmesg --follow messages when it freezes
[ 4259.203221] usb 2-4: USB disconnect, device number 11
[ 4261.152802] usb 2-4: new high-speed USB device number 13 using xhci_hcd
[ 4261.301764] usb 2-4: New USB device found, idVendor=0955, idProduct=7c18, bcdDevice= 0.00
[ 4261.301769] usb 2-4: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[ 4261.301773] usb 2-4: Product: APX
[ 4261.301775] usb 2-4: Manufacturer: NVIDIA Corp.
[ 4382.641644] usb 2-4: USB disconnect, device number 13
[ 4384.389529] usb 2-4: new high-speed USB device number 14 using xhci_hcd
[ 4384.538453] usb 2-4: New USB device found, idVendor=0955, idProduct=7c18, bcdDevice= 0.00
[ 4384.538458] usb 2-4: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[ 4384.538461] usb 2-4: Product: APX
[ 4384.538464] usb 2-4: Manufacturer: NVIDIA Corp.
[ 4390.508031] usb 2-4: USB disconnect, device number 14
[ 4390.817573] usb 2-4: new high-speed USB device number 15 using xhci_hcd
[ 4390.966473] usb 2-4: New USB device found, idVendor=0955, idProduct=7c18, bcdDevice= 0.00
[ 4390.966476] usb 2-4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 4390.966477] usb 2-4: Product: APX
[ 4390.966478] usb 2-4: Manufacturer: NVIDIA Corp.
[ 4390.966479] usb 2-4: SerialNumber: 00000
[ 4394.808149] EXT4-fs (loop8): mounted filesystem with ordered data mode. Opts: (null)
[ 4529.945487] usb 2-4: USB disconnect, device number 15
[ 4530.253452] usb 2-4: new high-speed USB device number 16 using xhci_hcd
[ 4530.402400] usb 2-4: New USB device found, idVendor=0955, idProduct=7c18, bcdDevice= 0.00
[ 4530.402406] usb 2-4: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[ 4530.402409] usb 2-4: Product: APX
[ 4530.402412] usb 2-4: Manufacturer: NVIDIA Corp.
[ 4713.236479] INFO: task tegrarcm_v2:21121 blocked for more than 120 seconds.
[ 4713.236487] Not tainted 5.4.0-58-generic #64~18.04.1-Ubuntu
[ 4713.236490] “echo 0 > /proc/sys/kernel/hung_task_timeout_secs” disables this message.
[ 4713.236494] tegrarcm_v2 D 0 21121 20962 0x20020000
[ 4713.236499] Call Trace:
[ 4713.236513] __schedule+0x293/0x720
[ 4713.236519] schedule+0x33/0xa0
[ 4713.236523] schedule_timeout+0x15d/0x320
[ 4713.236529] ? __next_timer_interrupt+0xe0/0xe0
[ 4713.236535] wait_for_completion_timeout+0xb3/0x140
[ 4713.236541] ? wake_up_q+0x80/0x80
[ 4713.236547] usb_start_wait_urb+0xe3/0x180
[ 4713.236552] usb_bulk_msg+0xb8/0x160
[ 4713.236557] proc_bulk+0x257/0x3a0
[ 4713.236563] usbdev_do_ioctl+0x93d/0x1490
[ 4713.236569] ? handle_mm_fault+0xcb/0x210
[ 4713.236574] usbdev_compat_ioctl+0x10/0x20
[ 4713.236580] __ia32_compat_sys_ioctl+0x105/0x230
[ 4713.236587] do_int80_syscall_32+0x5b/0x130
[ 4713.236593] entry_INT80_compat+0x85/0x90
[ 4713.236597] RIP: 0023:0x807a434
[ 4713.236609] Code: Bad RIP value.
[ 4713.236611] RSP: 002b:00000000ffe4f298 EFLAGS: 00000283 ORIG_RAX: 0000000000000036
[ 4713.236615] RAX: ffffffffffffffda RBX: 0000000000000005 RCX: 00000000c0105502
[ 4713.236617] RDX: 00000000ffe4f2c0 RSI: 0000000000000010 RDI: 0000000009631688
[ 4713.236619] RBP: 00000000ffe4f320 R08: 0000000000000000 R09: 0000000000000000
[ 4713.236621] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[ 4713.236623] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[ 4834.069536] INFO: task tegrarcm_v2:21121 blocked for more than 241 seconds.
[ 4834.069544] Not tainted 5.4.0-58-generic #64~18.04.1-Ubuntu
[ 4834.069546] “echo 0 > /proc/sys/kernel/hung_task_timeout_secs” disables this message.
[ 4834.069550] tegrarcm_v2 D 0 21121 20962 0x20020000
[ 4834.069555] Call Trace:
[ 4834.069569] __schedule+0x293/0x720
[ 4834.069576] schedule+0x33/0xa0
[ 4834.069580] schedule_timeout+0x15d/0x320
[ 4834.069585] ? __next_timer_interrupt+0xe0/0xe0
[ 4834.069592] wait_for_completion_timeout+0xb3/0x140
[ 4834.069597] ? wake_up_q+0x80/0x80
[ 4834.069603] usb_start_wait_urb+0xe3/0x180
[ 4834.069608] usb_bulk_msg+0xb8/0x160
[ 4834.069613] proc_bulk+0x257/0x3a0
[ 4834.069619] usbdev_do_ioctl+0x93d/0x1490
[ 4834.069625] ? handle_mm_fault+0xcb/0x210
[ 4834.069630] usbdev_compat_ioctl+0x10/0x20
[ 4834.069636] __ia32_compat_sys_ioctl+0x105/0x230
[ 4834.069643] do_int80_syscall_32+0x5b/0x130
[ 4834.069648] entry_INT80_compat+0x85/0x90
[ 4834.069653] RIP: 0023:0x807a434
[ 4834.069664] Code: Bad RIP value.
[ 4834.069667] RSP: 002b:00000000ffe4f298 EFLAGS: 00000283 ORIG_RAX: 0000000000000036
[ 4834.069671] RAX: ffffffffffffffda RBX: 0000000000000005 RCX: 00000000c0105502
[ 4834.069673] RDX: 00000000ffe4f2c0 RSI: 0000000000000010 RDI: 0000000009631688
[ 4834.069675] RBP: 00000000ffe4f320 R08: 0000000000000000 R09: 0000000000000000
[ 4834.069677] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[ 4834.069678] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000

#11

Like that

IMG_20201216_142704
IMG_20201216_1427044000×3000 2.32 MB

#12

I would like to add some details.

  1. Half a year ago, i was able to flash jetson TX2 with auvidea j120 board from this pc.
  2. Only things i see that changed were:
    a, Propably older release of ubuntu 18.04.X (now have 18.04.5)
    b, Different version of SDK manager (previously 12.X, now tried with 13.X and 14.X because when i install 12.x it wont let me flash jetson without updating to newer version)
    c, different revisions of boards. (Previously was D00, now has D02 only)
    d, propably different revisions of AUVIDEA J120 boards (new is on the right)
    auvidea1
    auvidea11150×1533 413 KB
    auvidea2
    auvidea21143×1524 415 KB
#13

About this:

I saw a number of vague failures like this via internet searches. None of them seemed to be a “smoking gun” evidence, but there was a theme to it. Some people had upgraded their PCs or just installed and updated a new o/s (Linux of different flavors). There seemed to be a pattern of certain kernel upgrades causing this, and recompiled or further updated kernels seemed to fix it. I’m thinking this is a host PC issue, but only due to a recent kernel upgrade.

For Ubuntu, when booting, you should have a GRUB bootloader menu early on. There might be an “Advanced” menu, and within this the previous kernel could be picked. If you have this, could you boot to one kernel version older, and try again?

#15

Ok so i tried with kernel 4.18 Ubuntu 18.04.2 with the same result
Yesterday was Ubuntu 18.04.5 with kernel 5.4.0.42/5.4.0.58

Screenshot from 2020-12-17 11-28-42
Screenshot from 2020-12-17 11-28-421920×1080 593 KB

#16

The thing that i think is the weirdest. Is that via SDK manager uploading 4.4.1 JetPack works flawlesly.
But unfortunately Firmware – Auvidea](https://auvidea.eu/firmware/ dont have a firmware compatible with JetPack newer than 4.3

#17

I see in your screenshot a disconnect and reconnect, which is normal during flash. Only the software is not seeing the reconnect and simply halts and waits. When it does this again, can you try to unplug the TX2’s USB and then replug it? Perhaps a manual hot plug will pick it up.

It is really rare to see this on a full Ubuntu 18.04 install for host PC. The same thing is highly likely to occur on any kind of VM. I’m kind of surprised this is not a VM.

FYI, you will have to stick to releases supported by Auvidea. You won’t be able to mix and match.

#18

I already tried few times to disconnect and connect both with or without recovery button reset.
Didnt help at All ☹️

#19

So tried again.
Also tried new high quality USB cable,
Also tried 2 jetson modules with their own Auvidea J120 module, so i could rule out faulty jetson.
Same result.

#20

I actually made a video of whole flashing process. Maybe it will help someone understand my problem. Actually via SDK manager it stops at 99% forever and when updating manually. it also freezes at the last step forever.
JetsonTX2 problem - YouTube

#21

EDIT: Read the part about the PCIe errors before trying another flash. Need to know more about what is on the PCI bus.

Can you try without host “HOST COMPONENTS” , and without “Jetson SDK components” (e.g., CUDA, which can be installed later)? Try picking only “Jetson OS”.

I see a lot of errors in the actual flash step. The steps leading up to making it possible to flash (e.g., download of content) seem to be good. While converting the raw image file for the rootfs to sparse format there are many errors…there was a time in the past where the error message itself was a bug and it worked anyway, but this could still be relevant. On the host PC, from the “~/nvidia/nvidia_sdk/JetPack...version.../Linux_for_Tegra/” location (cd there first), see how much disk space you have:
df -H -T .

When you bring up the console on the host PC in the video, and dmesg shows the errors, am I correct that this is the dmesg of the host PC itself? What I see are PCIe errors, which are rather significant and (possibly) unrelated to the flash. The exception would be if the PCIe device failing is a USB card, and the card is used for the flash. This is a serious error no matter how you look at it, but it is something the Jetson and flash software have no control over. Even so, if the device is part of USB, then this would cause flash to fail. Can you provide more details on the host, especially anything related to USB and PCI?

The video is not of high enough resolution to see any details on the PCIe errors of the host PC, but if you were to simply log via this:
dmesg | tee host_dmesg_log.txt
…then you could post the log.

#22

I dont have acces to jetson for the weekend, but i have a full resolution video.
And from that I was able make a screenshots of those errors.
I hope those are those you were asking for.

pci errors
pci errors1330×1464 632 KB

And btw yes. thats all on host pc