AGX Flasing Stuck at 99%

Hi all,

I have a commercial AGX module which recently had issues in it being unable to boot. I attempted to reflash the unit, but the SDK manager gets stuck at 99% and does not progress any further, even after letting it run overnight.

I have attempted to flash the AGX on both a third party carrier board and the NVIDIA carrier board shipped with the developer kit, with both having the same results. I also tried multiple, high-quality cables to no avail.The SDK manager is running on a Linux machine; not through a VM.

Any aid would be greatly appreciated.

-Ryan

Hi ryan74,

Please package the folder ~/.nvsdkm and attach it here to analyze.

Thanks

Hi kaycc,

File attached nvsdkm_AGX_FLASH.tar.gz (782.8 KB)

As Jetpack 4.6 released I also attempted the flash using that, still the same issue unfortunately .

Hi ryan74,

SDKM currently only support flashing NVIDIA carrier board. It doesn’t support flashing custom carrier board.
In the sdkm-2021-08-02-09-45-03.log file, I could see that the flashing is successful.

09:49:33.507 - info: NV_L4T_FLASH_XAVIER_WITH_OS_IMAGE_COMP@JETSON_AGX_XAVIER: *** The target t186ref has been flashed successfully. ***
09:49:33.520 - info: NV_L4T_FLASH_XAVIER_WITH_OS_IMAGE_COMP@JETSON_AGX_XAVIER: Reset the board to boot from internal eMMC.
09:49:33.520 - info: NV_L4T_FLASH_XAVIER_WITH_OS_IMAGE_COMP@JETSON_AGX_XAVIER:
09:49:53.509 - info: NV_L4T_FLASH_XAVIER_WITH_OS_IMAGE_COMP@JETSON_AGX_XAVIER: [ Component Install Finished Successfully ]
09:49:53.514 - info: NV_L4T_FLASH_XAVIER_WITH_OS_IMAGE_COMP@JETSON_AGX_XAVIER: [host] [ 872.00 KB used. Disk Avail: 348.45 GB ]
09:49:53.514 - info: NV_L4T_FLASH_XAVIER_WITH_OS_IMAGE_COMP@JETSON_AGX_XAVIER: [ NV_L4T_FLASH_XAVIER_WITH_OS_IMAGE_COMP Install took 4m11s ]
09:49:53.519 - info: cmd finished success SDKM_END_CODE_SUCCESS_2acef3ba-a841-4daa-ae3b-2648f99c42b0
09:49:53.519 - info: NV_L4T_FLASH_XAVIER_WITH_OS_IMAGE_COMP@JETSON_AGX_XAVIER: command finished successfully
09:49:53.524 - info: Event: NV_L4T_FLASH_XAVIER_WITH_OS_IMAGE_COMP@JETSON_AGX_XAVIER - install ended
09:49:53.525 - info: Summary: NV_L4T_FLASH_XAVIER_WITH_OS_IMAGE_COMP@JETSON_AGX_XAVIER: Install completed successfully.

After flashing, please manually set up username/password and other OEM configurations for Jetson board, then install target components like CUDA on Jetson board.

Many thanks for your comments Edward. I have tried flashing the same AGX module on both a custom carrier board and the Nvidia Developer kit, but I still encountered the same issue. It may be that the one which flashed successfully was a separate AGX.

I’ll retry the flash on a development kit and attach the log files again.

Hi EdwardZhou,

Kindly see attached file from when we attempted to flash the same AGX on an NVIDIA development kit carrier board. We used cables and power adapters as supplied with a separate AGX development kit, so they are of good quality. The relevant log file is named sdkm-2021-07-30-14-50-55.log nvsdkm.tar.gz (461.0 KB)

hello ryan74,

according to below,

12:59:43.581 - info: NV_L4T_FLASH_XAVIER_WITH_OS_IMAGE_COMP@JETSON_AGX_XAVIER: [ 78371.0254 ] CPU Bootloader is not running on device.
12:59:43.581 - info: NV_L4T_FLASH_XAVIER_WITH_OS_IMAGE_COMP@JETSON_AGX_XAVIER: [ 79386.8281 ]
12:59:43.581 - info: NV_L4T_FLASH_XAVIER_WITH_OS_IMAGE_COMP@JETSON_AGX_XAVIER: Error: None of the bootloaders are running on device. Check the UART log.

could you please setup serial console via port J501 to gather bootloader logs for reference,
thanks

Dear JerryChang

bootloader logs attached as obtained from the agx on the Nvidia development board; Seems that Task 76 has failed, leading the AGX to reboot in recovery mode automatically.

agx_bootloader_log.txt (2.5 KB)

hello ryan74,

suggest you enter the JetPack release folder and execute flash.sh to re-flash the board.
for example, it’s by default under ~/nvidia/nvidia_sdk/ for the release image.
please also check Basic Flash Script Usage for running image flashing.
thanks

Thanks for the heads up on the flash script, but it seems the error persists here as well when the AGX is mounted on the development kit carrier board.

Oddly enough, a different error is present when the AGX is mounted on the custom carrier board. I am unsure why this occurs, as the AGX has to be in recovery mode for the flash.sh script to even run.

Remounting the AGX on the dev kit results in the same USB communication failed error.

hello ryan74,

you should using correct flash configuration file that include your customize board settings.
hence, please do perform the flash script with developer kits to have confirmation,
for example, $ sudo ./flash.sh jetson-xavier mmcblk0p1

I understand you’re not using a VM, but, is your host machine a laptop or desktop? it sometimes the USB port issue when using a laptop, please have power-supply connected and try other ports for board communication.
thanks

Hi JerryChang,

Have tried your suggestion but I still get the same USB communication failed.Check if device is in recovery error.

My host machine is a desktop. I have tried other ports directly on the machine and also a USB hub, all resulting in the same issues.

hello ryan74,

please use the AGX Xavier on the NVIDIA carrier board as Jetson AGX Xavier developer kit,
here’re two ways to enter forced-recovery mode, please do these trails for verification,
(1) it’s normal approach to hold recovery button and press reset to restart the board, release recovery button after 2~3 seconds.
(2) you should unplug/plug the power-supply, hold the recovery button, press power button, release the recovery button once you see led on.

Hello JerryChang,

have tried your suggestions, and I keep getting the same USB communication failed.Check if device is in recovery error, when attempting to flash via the script.

hello ryan74,

could you please take a picture of your Jetson platform and the USB connections.

Hello JerryChang

Images as requested



I have the 19V power supply connected to the DC barrel jack, a USB C to J512 for flashing and a micro USB to J501 to read the boot logs

hello ryan74,

couple of questions,
is this Jetson AGX Xavier works before? what’s the last known good release it’s has been working on.
is this a fused Jetson AGX Xavier platform? (i.e. PKC or SBKPKC)

We received this back from a customer as it was unable to boot. When I received it it was able to boot up once and function normally, but after shutting it down I was unable to reboot it successfully.

It was running Jetpack 4.5.1. I am unsure if it is fused or not; is there a test point from where I can check?

hello ryan74,

please examine the fuse info from the target board with odmfuseread.sh.
you’re able to access SecureBoot Tools via https://developer.nvidia.com/embedded/linux-tegra.
thanks

Thanks for all the help guys. Unfortunately the customer has requested back the boards so I couldn’t troubleshoot the issue any more :/