Force recovery mode sequence changed (L4T 32.4.3)?

tmx3 · July 30, 2020, 8:14am

Recently, I upgraded my TX2 board to the latest Jetpack 4.4 (L4T 32.4.3), things seemed working well. Then without paying much attention, I ran terminal command to flash a new DTB
$ sudo ./flash.sh -r -k kernel-dtb jetson-tx2 mmcblk0p1
which failed. It caused the TX2 board failed to boot, only to show the NVidia logo on and off.

Then I decided to rerun the SDKManager. Unfortunately, it won’t allow me to finish. The flash hung at 99%.

At rerun, I noticed the button sequence for “Force Recovery Mode” with SDKManager is different from what I did to flash my DTB.

SDKManager requires a sequence:

Starting with the device powered off. Connect USB cable
Press and hold down the Force Recovery button.
Press the Power button.
Release the Power button, then release the Force Recovery button.

I pretty sure I used a sequence of old fashion, which is

Starting with the device powered off.Connect USB cable
Press Power button
Press and hold down the Force Recovery button
Press and release the Reset button
Then release the Force Recovery button.

Would mixing the different sequence of recovery mode cause any trouble?

I hope I didn’t kill my TX2 by doing that.
BTW, my SDK Manager version is 1.2.0.6738.

tmx3 · July 30, 2020, 10:15pm

Tried to re-flash the TX2 from SDK Manager on host PC a couple of times. No luck. SDK Manager shows a successful flash, but TX2 screen shows no response at power up. Screen remains dark. Interestingly though, the button sequence is still working to enter the force recovery mode. Host PC can see the change by the “lsusb” command. So, apparently somehow the board is still alive.

Checked the display and cable. No problem with the HDMI cable and the screen.

Really hate to give up on the TX2, which has been working well before flashing.
Any suggestions?

linuxdev · July 30, 2020, 10:27pm

Anything which powers up or resets power while the recovery button is connected will do the job at entering recovery mode. Odd combinations will not harm anything. Recovery mode itself does not do anything to the Jetson other than temporarily causing it to become a custom USB device the driver package understands. Unless the driver package alters something the recovery mode itself will make no changes (driver package is what JetPack/SDKM runs during the flash steps).

If flash completed, then chances are that the Jetson is just fine and only video display is at fault. There are many things which can go wrong with video, but quite often when this happens all of the other services are running on the Jetson.

Note that if you don’t perform the first boot login setup to create an account, then even if video works you won’t be able to log in. Similar for ssh or serial console…you can’t log in until there is an account to log in to.

Serial console is probably what you need next. See:
http://www.jetsonhacks.com/2017/03/24/serial-console-nvidia-jetson-tx2/

Basically, serial console has very few issues because it has so few dependencies. Networking can fail, GUI can fail, and all kinds of things can fail while serial console continues to work. Serial console can also be used to create that account on first boot.

More importantly, you can get a boot log via serial console. That log works even in early bootloader stages, and can provide information about what is working or failing before the Linux kernel ever loads. Any application running on your PC for serial console can log the complete boot.

As far as video is concerned, is your monitor purely HDMI, and was it connected during the flash? Adapters (such as from VGA) will cause HDMI failure. Not having a monitor/keyboard on first boot (which happens automatically after it says flash is completed…the unit then reboots itself and is no longer in recovery mode) can be an issue (and is one reason why it is nice to have serial console attached during a flash).

tmx3 · July 30, 2020, 11:23pm

linuxdev, thanks for the detailed reply.

The serial console sounds a good idea. I’ll see if I can make one. BTW, JetsonHacks is always the one of those superb videos to watch.

My monitor is pure HDMI. And I replaced the monitor and cable just in case. They are good. Regarding the keyboard and mouse, I had them connected via bluetooth normally. They have to be replace with USB based devices for flashing setup. Agree, as you mention, that’s another little bump to overcome.

linuxdev · July 30, 2020, 11:35pm

We have @Kangalow to thank for that!

For keyboard/mouse you’d want to use actual USB mouse/keyboard during a flash. I suspect there would be nothing but trouble to use Bluetooth during a flash.

tmx3 · August 10, 2020, 7:46pm

Got my serial console working. thanks for the guidance! In term of TX2, the good news is that the board is still alive. The bad news is that the board keeps rebooting itself.

From the console message, I saw the footprint of my DTB which is to configure six IMX219 cameras (apparently failed). What puzzles me is that I had reflashed that board a couple of times using SDK manager. I expected the system should have been wiped clean, is there a way to force that in SDK Manager?

For those viewers didn’t know what happened, my TX2 went dark after I tried to reflash the DTB in old way (something like L4T 28). Reflash the system with new Jetpak seems not helping.

Here is the screen shot from series console:

U-Boot 2016.07-ge6da093be3 (Jun 25 2020 - 21:14:32 -0700)

TEGRA186
Model: NVIDIA P2771-0000-500
DRAM: 7.8 GiB
MC: Tegra SD/MMC: 0, Tegra SD/MMC: 1
*** Warning - bad CRC, using default environment

In: serial
Out: serial
Err: serial
Net: eth0: ethernet@2490000
Hit any key to stop autoboot: 0
MMC: no card present
switch to partitions #0, OK
mmc0(part 0) is current device
Scanning mmc 0:1…
Found /boot/extlinux/extlinux.conf
Retrieving file: /boot/extlinux/extlinux.conf
857 bytes read in 133 ms (5.9 KiB/s)
1: primary kernel
Retrieving file: /boot/initrd
5565615 bytes read in 196 ms (27.1 MiB/s)
Retrieving file: /boot/Image
34330632 bytes read in 861 ms (38 MiB/s)
append: console=ttyS0,115200 root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 co
## Flattened Device Tree blob at 80000000
Booting using the fdt blob at 0x80000000
reserving fdt memory region: addr=80000000 size=10000
Using Device Tree in place at 0000000080000000, end 000000008003e3cd

Starting kernel …
[ 0.000000] Booting Linux on physical CPU 0x100
[ 0.000000] Linux version 4.9.140-tegra (buildbrain@mobile-u64-3357) (gcc ve0
[ 0.000000] Boot CPU: AArch64 Processor [411fd073]
[ 0.000000] OF: fdt:memory scan node memory@80000000, reg size 16416,
[ 0.000000] OF: fdt: - 80000000 , 70000000
[ 0.000000] OF: fdt: - f0200000 , 185600000
[ 0.000000] OF: fdt: - 275e00000 , 200000
[ 0.000000] OF: fdt: - 276600000 , 200000
[ 0.000000] OF: fdt: - 277000000 , 200000
[ 0.000000] earlycon: uart8250 at MMIO32 0x0000000003100000 (options ‘’)
[ 0.000000] bootconsole [uart8250] enabled
[ 0.795392] falcon 15340000.vic: failed to get IRQ
[ 0.798282] iommu_context_dev 13e10000.host1x:ctx0: iommu is not enabled for.
[ 0.798320] iommu_context_dev 13e10000.host1x:ctx1: iommu is not enabled for.
[ 0.798353] iommu_context_dev 13e10000.host1x:ctx2: iommu is not enabled for.
[ 0.798385] iommu_context_dev 13e10000.host1x:ctx3: iommu is not enabled for.
[ 0.798417] iommu_context_dev 13e10000.host1x:ctx4: iommu is not enabled for.
[ 0.798449] iommu_context_dev 13e10000.host1x:ctx5: iommu is not enabled for.
[ 0.798480] iommu_context_dev 13e10000.host1x:ctx6: iommu is not enabled for.
[ 0.798512] iommu_context_dev 13e10000.host1x:ctx7: iommu is not enabled for.
[ 1.915964] tegra-ahci 3507000.ahci-sata: controller reset failed (0xfffffff)
[ 2.081529] imx219 2-0038: Failed to find port index
[ 2.086515] imx219 2-0038: Failed to find port info.
[ 2.091493] imx219 2-0038: Failed to initialize imx219
[ 2.096644] imx219 2-0038: tegra camera driver registration failed
[ 2.103262] imx219 2-0039: imx219_power_get: unable to request reset_gpio (-)
[ 2.110599] imx219 2-0039: unable to power get
[ 2.115057] imx219 2-0039: tegra camera driver registration failed
[ 2.121633] imx219 2-003a: imx219_power_get: unable to request reset_gpio (-)
[ 2.128960] imx219 2-003a: unable to power get
[ 2.133429] imx219 2-003a: tegra camera driver registration failed
[ 2.140161] imx219 2-003b: imx219_power_get: unable to request reset_gpio (-)
[ 2.147490] imx219 2-003b: unable to power get
[ 2.151946] imx219 2-003b: tegra camera driver registration failed
[ 2.158517] imx219 2-003c: imx219_power_get: unable to request reset_gpio (-)
[ 2.165854] imx219 2-003c: unable to power get
[ 2.170312] imx219 2-003c: tegra camera driver registration failed
[ 2.176873] imx219 2-003d: imx219_power_get: unable to request reset_gpio (-)
[ 2.184199] imx219 2-003d: unable to power get
[ 2.188665] imx219 2-003d: tegra camera driver registration failed
[ 2.203873] arm-smmu 12000000.iommu: Unexpected {global,context} fault, thiss
[ 2.212402] arm-smmu 12000000.iommu: GFSR 0x80000002, GFSYNR0 0x00000
[ 2.222707] mc-err: vpr base=0:0, size=0, ctrl=1, override:(e01a8341, 1dc10c)
[ 2.231078] mc-err: (255) csw_bpmpw: MC request violates VPR requirements
[ 2.237871] mc-err: status = 0x00337094; addr = 0x3ffffffc0
[ 2.243622] mc-err: secure: yes, access-type: write
[ 2.248685] mc-err: unknown mcerr fault, int_status=0x00000000, ch_int_statu0
[ 2.259211] mc-err: unknown mcerr fault, int_status=0x00000000, ch_int_statu0
[ 2.269736] mc-err: unknown mcerr fault, int_status=0x00000000, ch_int_statu0
[ 2.280260] mc-err: Too many MC errors; throttling prints
[ 2.285796] arm-smmu 12000000.iommu: Unexpected {global,context} fault, thiss
[ 2.294320] arm-smmu 12000000.iommu: GFSR 0x80000002, GFSYNR0 0x00000
[ 2.306971] Unable to handle kernel paging request at virtual address ffffff0
[ 2.314924] Mem abort info:
[ 2.317739] ESR = 0x96000047
[ 2.320818] Exception class = DABT (current EL), IL = 32 bits
[ 2.326749] SET = 0, FnV = 0
[ 2.329816] EA = 0, S1PTW = 0
[ 2.332969] Data abort info:
[ 2.335851] ISV = 0, ISS = 0x00000047
[ 2.339697] CM = 0, WnR = 1
[ 2.342678] swapper pgtable: 4k pages, 39-bit VAs, pgd = ffffff800a1d9000
[ 2.349474] [ffffff800a3aa020] *pgd=00000002771fe003, *pud=00000002771fe003,0
[ 2.360508] Internal error: Oops: 96000047 [#1] PREEMPT SMP
[ 2.366080] Modules linked in:
[ 2.369153] CPU: 3 PID: 1 Comm: swapper/0 Not tainted 4.9.140-tegra #1
[ 2.375678] Hardware name: quill (DT)
[ 2.379342] task: ffffffc1ecb30000 task.stack: ffffffc1ecb38000
[ 2.385270] PC is at tegra_update_cpu_speed+0x140/0x178
[ 2.390494] LR is at tegra_update_cpu_speed+0x11c/0x178
[ 2.395719] pc : [] lr : [] pstate: 60405
[ 2.403112] sp : ffffffc1ecb3b9a0
[ 2.406427] x29: ffffffc1ecb3b9a0 x28: 00000000024000c0
[ 2.411764] x27: 0000000000000000 x26: ffffff8009e67f74
[ 2.417099] x25: ffffff800a1aa4f0 x24: ffffff800a3aa020
[ 2.422432] x23: ffffff800a1aa4f0 x22: 0000000000000000
[ 2.427766] x21: ffffff800a1aa558 x20: ffffffc1f703da98
[ 2.433100] x19: 0000000000000000 x18: 0000000000000000
[ 2.438434] x17: 0000000000000000 x16: 0000000000000000
[ 2.443768] x15: 0000000000000000 x14: 0000000000000001
[ 2.449100] x13: 0000000000000030 x12: 0000000000000038
[ 2.454432] x11: 000000000000000b x10: 0101010101010101
[ 2.459765] x9 : ffffffffffffffff x8 : ffffffc1e9f32800
[ 2.465099] x7 : 0000000000000000 x6 : 0000000000000000
[ 2.470432] x5 : 0000000000000001 x4 : 0000000000000000
[ 2.475765] x3 : ffffff8009831a98 x2 : 0000000000000000
[ 2.481096] x1 : 0000000000000000 x0 : ffffff800a3aa000
[ 2.486430]
[ 2.487925] Process swapper/0 (pid: 1, stack limit = 0xffffffc1ecb38000)
[ 2.494622] Call trace:
[ 2.497076] [] tegra_update_cpu_speed+0x140/0x178
[ 2.503344] [] tegra186_cpufreq_set_target+0x198/0x228
[ 2.510043] [] __cpufreq_driver_target+0x1d0/0x610
[ 2.516395] [] cpufreq_online+0x654/0x748
[ 2.521965] [] cpufreq_add_dev+0x84/0xc8
[ 2.527455] [] subsys_interface_register+0x90/0xd8
[ 2.533806] [] cpufreq_register_driver+0x190/0x278
[ 2.540163] [] tegra186_cpufreq_probe+0xa20/0xa90
[ 2.546430] [] platform_drv_probe+0x60/0xc0
[ 2.552175] [] driver_probe_device+0xd8/0x408
[ 2.558093] [] __driver_attach+0xdc/0x128
[ 2.563664] [] bus_for_each_dev+0x5c/0xa8
[ 2.569235] [] driver_attach+0x30/0x40
[ 2.574546] [] bus_add_driver+0x20c/0x2a8
[ 2.580118] [] driver_register+0x6c/0x110
[ 2.585688] [] __platform_driver_register+0x5c/0x68
[ 2.592127] [] tegra186_cpufreq_platform_driver_init+0x18/0
[ 2.599527] [] do_one_initcall+0x44/0x130
[ 2.605101] [] kernel_init_freeable+0x1a0/0x244
[ 2.611195] [] kernel_init+0x18/0x108
[ 2.616418] [] ret_from_fork+0x10/0x30
[ 2.621738] —[ end trace 8c0218b566354a00 ]—
[ 2.630776] note: swapper/0[1] exited with preempt_count 1
[ 2.636289] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0b
[ 2.636289]
[ 2.645427] SMP: stopping secondary CPUs
[ 2.649364] Kernel Offset: disabled
[ 2.652852] Memory Limit: none
[ 2.660370] Rebooting in 5 seconds…

linuxdev · August 11, 2020, 4:31pm

Your serial console program should be able to log the whole boot, which might be useful. Not sure if useful for this case.

A fresh flash would indeed go back to the default device tree. However, if you use the same content as that from which you flashed the device tree originally, it is quite possible the tree is still in place within that content and being flashed back again. You might need to remove all of the content in your “~/nvidia” directory (and perhaps also “~/.nvsdkm”, and then run SDKM again with fresh content being added. This would remove any of the custom device tree and use clean content.

tmx3 · August 12, 2020, 4:09am

Your solution works like a charm! I was able to flash the TX2 and bring back the board to live after cleaning up the data.

The serial console is helpful in isolating the problem. There were so many possibilities otherwise.

WayneWWW · August 12, 2020, 9:07am

The force recovery button order never changes.

If your board didn’t power on, then

press the recovery button
press the power button. (power LED would be ON)
release the power btn.
release the recovery button.

And back to your log. It looks like almost every component has issue…

Are you using a nvidia devkit or custom carrier board? Are you sure you are using pure jetpack release?

tmx3 · August 16, 2020, 6:13pm

@WayneWWW, thanks for the response.

I’m using the official nvidia devkit. And we design our CSI adapter board for six-channel cameras, which has been working well with previous L4T kernel on TX2. But we ran into problem after upgraded the system to R32.4.3. We are still working to compile the device tree and imx219.c driver. I would open a new thread later. Anyhow, that’s the status of current progress. With the serial console, we can rule out the power-on and flashing issue, it allows us to go back to DTB and debug further (otherwise I thought I might have fried the board by a wrong sequence). Thanks for the forum and helping hands from all you guys.

WayneWWW · August 17, 2020, 2:11am

Hi,

The imx219 sounds like another issue. Please file separate topic for it.

But for current topic, have you resolved it? I think you could create a base case from official dts. build it and flash it first.

Please check if any error for this base case.

tmx3 · August 17, 2020, 3:27am

Hi WayneWWW,
Yes, the case is solved as far as the force recovery mode is concerned. I have check-marked linuxdev’s response as the “Solution”.
As for imx219 DTB compilation, I’ll post a report or a call for help once we narrow down the key issues. Thanks for your guideline.