Jetson Xavier NX (Developer Edition) not starting (blank screen)

Hello,

I have the Jetson Xavier NX (Developer Edition) which I used to run with the Jetpack 4.6 (Ubuntu 18.04). I recently upgraded to the Jetpack 5.0.1 (Ubuntu 20.04) to work on newer robotic frameworks (ROS2). Now, I experience issues when I try to start the Jetson.

Previously, it worked fine, but now, with the newly flashed SD card, the Jetson does not boot anymore. It does not show any splash screen with the Nvidia Logo, it does not respond to keyboard input or mouse input and the keyboard and mouse are not even powered. It is just a black screen, the monitor tells me there is no HDMI (also tested DP) input.

The same image works fine on a second Jetson Xavier NX, but this one here seems to be dead.

When I connect the micro USB interface to my computer, it does show the Nvidia device when using lsusb with the following output:

Bus 001 Device 030: ID 0955:7e19 NVIDIA Corp. APX

When I use the SDK Manager, I can also see the device when I connect it through the USB cable. But when I try to flash it again (initially I flashed by downloading the image and using Balena Etcher) with the SDK Manager, it fails at around 37% with the following log file: (only the last few lines)

13:37:57 ERROR: Can not find matching device in recovery mode.
13:37:57 DEBUG: running command < true >
13:37:57 INFO: command finished successfully
13:37:57 DEBUG: running command < /tmp/device_mode_host_setup.sh >
13:37:57 INFO: Found route default enp0s31f6, setting up device mode
13:37:57 INFO: net.ipv4.ip_forward = 1
13:38:12 INFO: Default google DNS is not working, trying to find a valid server in local host settings!
13:38:12 INFO: Device mode on host successfully set with DNS 10.236.9.10!
13:38:12 INFO: command finished successfully
13:38:13 DEBUG: running command < true >
13:38:13 INFO: command finished successfully
13:38:13 DEBUG: Flash Jetson Xavier NX - flash: using adapter to install NV_L4T_FLASH_XAVIER_NX_WITH_OS_IMAGE_COMP@JETSON_XAVIER_NX_TARGETS to /home/ubadmin/nvidia/nvidia_sdk/JetPack_5.0.1_DP_Linux_JETSON_XAVIER_NX_TARGETS
13:38:13 INFO: Flash Jetson Xavier NX - flash: change working directory to /home/ubadmin/nvidia/nvidia_sdk/JetPack_5.0.1_DP_Linux_JETSON_XAVIER_NX_TARGETS/Linux_for_Tegra
13:38:13 INFO: Flash Jetson Xavier NX - flash: sudo ./nvsdkmanager_flash.sh --nv-auto-config --username xavier
13:38:13 INFO: Flash Jetson Xavier NX - flash: [OEMPreconfig] SDKM_INSTALL_ERROR Password for L4T new user xavier:
13:38:13 INFO: Flash Jetson Xavier NX - flash: *** Checking ONLINE mode ... OK.
13:38:13 INFO: Flash Jetson Xavier NX - flash: *** Checking target board connection ... 1 connections found.
13:38:18 ERROR: Flash Jetson Xavier NX - flash: Put the target board in RCM mode and retry.
13:38:18 ERROR: Flash Jetson Xavier NX - flash: command terminated with error
13:38:18 SUMMARY: DateTime Target Setup - target: Depends on failed component

I am afraid that the board is not working anymore. With the second Jetson Xavier, when I plug it to a monitor, it shows the NVidia logo and it boots. (I needed to disable the hardware acceleration on the SD card image to make it boot properly though). With this broken Jetson, the power LED is lighting green, but nothing else. The fan is not spinning, but the heat sink gets hot.

I also swapped the carrier boards to see if the problem is the carrier board or the top module and the carrier board seems to work just fine, so it must be related to the module.

Are there any steps I can do to verify that the board is still usable and how can I recover it?

Is this your first time flash the board using sdkmanager?

Hi,

Yes, this is the first time this Xavier device is flashed using the SDK Manager, before I manually flashed the SD card with another program (Belena Etcher) and inserted the SD card afterwards.

Then the board is probably still alive. Please try to put your board into recovery mode manually first.

As the error log says.

13:38:18 ERROR: Flash Jetson Xavier NX - flash: Put the target board in RCM mode and retry.

I will do it as soon as I have access to it, probably on Friday.

Should I keep the jumper on the FC REC after connecting the board to power or remove it afterwards?

Connect the REC before your connect the power. You can remove it until the flash process is done.

Okay, I will try. Thanks already, I will let you know if it works or not.

I have something like this going on. What I found while trying to trouble shoot the issue is that when I have the USB to serial cable connected to screen on a linux desktop that the system was trying to boot to network or some other bs. There was an option than to use ESC out and select boot device. A real pain in the A$$ if you ask me.

This type of cable USB to TTL Serial Cable - Debug / Console Cable for Raspberry Pi : ID 954 : $9.95 : Adafruit Industries, Unique & fun DIY electronics and kits

After retrying, following the procedure and flashing in manual mode (I have a jumper on pin 9-10), I still have the following logs :

10:22:59 ERROR: Can not find matching device in recovery mode.
10:22:59 DEBUG: running command < true >
10:22:59 INFO: command finished successfully
10:22:59 DEBUG: running command < /tmp/device_mode_host_setup.sh >
10:22:59 INFO: exec_command: /tmp/device_mode_host_setup.sh
10:22:59 INFO: Found route default enp0s31f6, setting up device mode
10:22:59 INFO: [host] [sudo] password for ubadmin:
10:23:00 INFO: net.ipv4.ip_forward = 1
10:23:16 INFO: Default google DNS is not working, trying to find a valid server in local host settings!
10:23:16 INFO: Device mode on host successfully set with DNS 10.236.9.10!
10:23:16 INFO: command finished successfully
10:23:16 DEBUG: running command < true >
10:23:16 INFO: command finished successfully
10:23:16 DEBUG: Flash Jetson Xavier NX - flash: using adapter to install NV_L4T_FLASH_XAVIER_NX_WITH_OS_IMAGE_COMP@JETSON_XAVIER_NX_TARGETS to /home/ubadmin/nvidia/nvidia_sdk/JetPack_5.0.1_DP_Linux_JETSON_XAVIER_NX_TARGETS
10:23:16 INFO: Flash Jetson Xavier NX - flash: [ Component Install Started ]
10:23:16 INFO: Flash Jetson Xavier NX - flash: exec_command: /tmp/tmp_NV_L4T_FLASH_XAVIER_NX_WITH_OS_IMAGE_COMP.sh
10:23:16 INFO: Flash Jetson Xavier NX - flash: sudo ./nvsdkmanager_flash.sh --nv-auto-config --username xavier
10:23:16 INFO: Flash Jetson Xavier NX - flash: [OEMPreconfig] SDKM_INSTALL_ERROR Password for L4T new user xavier:
10:23:16 INFO: Flash Jetson Xavier NX - flash: *** Checking ONLINE mode ... OK.
10:23:16 INFO: Flash Jetson Xavier NX - flash: *** Checking target board connection ... 1 connections found.
10:23:22 ERROR: Flash Jetson Xavier NX - flash: Put the target board in RCM mode and retry.
10:23:22 ERROR: Flash Jetson Xavier NX - flash: command terminated with error
10:23:22 SUMMARY: DateTime Target Setup - target: Depends on failed component

SDK manager show the following error:

The target is in a bad state

The Jetson target is in a bad state and cannot be flashed. Please manually put the target into recovery mode and then retry flashing.

If I try to run the command by myself, I have the following :

sudo ./nvsdkmanager_flash.sh --nv-auto-config --username xavier
Password for L4T new user xavier: 
*** Checking ONLINE mode ... OK.
*** Checking target board connection ... 1 connections found.
*** Reading ECID ... *** Error: ECID read failed.
Put the target board in RCM mode and retry.

I have been investigating further with some colleagues and we discovered something interesting.

We also tested the exact same Jetson device with a flashed SD card with the Jetpack containing the Ubuntu 18.04 image and it is booting up just fine. So apparently, it is related to the software of the Jetpack and not (only) the hardware.

Question: does the installation of a system alter the hardware settings like firmware or anything else?

We also connected the UART pins to a UART-USB adapter and read the serial output. When we did this, somehow, we managed to get the splash screen with the Nvidia logo again, but then after showing the logo on a black background, it lights up quickly with a white screen and then it boes dark again.

Then, we also read the output of the serial port. This is now with a cleanly flashed SD card with the latest Jetpack containing Ubuntu 20.04, flashed with dd. The last lines of the output before is crashes are:

[   23.329411] x21: ffff800012014fe0 x20: 0000000000000005 
[   23.334577] hpd: switching from state 1 (Check Plug) to state 3 (Disabled)
[   23.342877] x19: ffff8000120150b8 x18: 0000000000000010 
[   23.342891] x17: ffff800010efc870 x16: ffff800010ef9c70 
[   23.342902] x15: ffff800011d92b30 x14: ffffffffffffffff 
[   23.414720] x13: ffff8000900039e7 x12: ffff8000100039f1 
[   23.420315] x11: 616c70736944203a x10: 6572203336353637 
[   23.425651] x9 : ffff800010003d00 x8 : 79616c707369642e 
[   23.430988] x7 : ffff8000107456d0 x6 : 000000003f2588cb 
[   23.436070] x5 : 00ffffffffffffff x4 : ffff800012980000 
[   23.441664] x3 : 00000000ffffffff x2 : ffff800010236d10 
[   23.446758] x1 : ffff800011d925c0 x0 : 0000000000010001 
[   23.452078] Call trace:
[   23.454972]  tegra194_cbb_error_isr+0x198/0x1b0
[   23.459273]  __handle_irq_event_percpu+0x60/0x2a0
[   23.463805]  handle_irq_event_percpu+0x3c/0xa0
[   23.468526]  handle_irq_event+0x4c/0xf0
[   23.472287]  handle_fasteoi_irq+0xbc/0x170
[   23.476313]  generic_handle_irq+0x3c/0x60
[   23.480354]  __handle_domain_irq+0x6c/0xc0
[   23.484280]  efi_header_end+0xa8/0xf0
[   23.488037]  el1_irq+0xd0/0x180
[   23.491032]  cpuidle_enter_state+0xb4/0x400
[   23.495211]  cpuidle_enter+0x3c/0x50
[   23.498456]  call_cpuidle+0x40/0x70
[   23.501969]  do_idle+0x1fc/0x260
[   23.505624]  cpu_startup_entry+0x2c/0x70
[   23.509131]  rest_init+0xd8/0xe4
[   23.512371]  arch_call_rest_init+0x14/0x1c
[   23.516824]  start_kernel+0x4c0/0x4f4
[   23.520174] Code: a9425bf5 a9446bf9 a8c67bfd d65f03c0 (d4210000) 
[   23.526282] ---[ end trace 83ea15fdedc9d963 ]---
[   23.530743] Kernel panic - not syncing: Oops - BUG: Fatal exception in interrupt
[   23.538283] SMP: stopping secondary CPUs
[   23.542133] Kernel Offset: 0x80000 from 0xffff800010000000
[   23.547710] PHYS_OFFSET: 0x80000000
[   23.551317] CPU features: 0x8240002,03002a30
[   23.555502] Memory Limit: none
[   23.558653] trusty-log panic notifier - trusty version Built: 08:57:16 Feb 19 2022 
[   23.567033] ---[ end Kernel panic - not syncing: Oops - BUG: Fatal exception in interrupt ]---

nvidia-jetson-xavier-nx-boot-uart-j14_clean_flashed_ubuntu2004.log (80.7 KB)

Now, we also discovered, when using the UART-USB prompt, we can stop the auto-boot of the board with the Ubuntu 20.04 image when it is trying to boot. When we then enter boot to boot the device, we get the following output:

[   23.594372] ERROR: mmcblk0p1 mount fail...
[   23.670398] usb 1-2.1: new high-speed USB device number 4 using tegra-xusb
[   23.708886] usb 1-2.1: New USB device found, idVendor=0bda, idProduct=5409, bcdDevice= 0.02
[   23.710366] usb 1-2.1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[   23.711789] usb 1-2.1: Product: USB2.1 Hub
[   23.713104] usb 1-2.1: Manufacturer: Generic
[   23.716437] hub 1-2.1:1.0: USB hub found
[   23.719285] hub 1-2.1:1.0: 5 ports detected
[   23.931199] usb 2-3.1: new SuperSpeed Gen 1 USB device number 3 using tegra-xusb
[   23.968794] usb 2-3.1: New USB device found, idVendor=0bda, idProduct=0409, bcdDevice= 0.02
[   23.970257] usb 2-3.1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[   23.971681] usb 2-3.1: Product: USB3.2 Hub
[   23.973028] usb 2-3.1: Manufacturer: Generic
[   23.981604] hub 2-3.1:1.0: USB hub found
[   23.986527] hub 2-3.1:1.0: 4 ports detected
[   24.038365] usb 1-2.1.1: new low-speed USB device number 5 using tegra-xusb
[   24.066606] usb 1-2.1.1: New USB device found, idVendor=413c, idProduct=2113, bcdDevice= 1.10
[   24.068046] usb 1-2.1.1: New USB device strings: Mfr=0, Product=2, SerialNumber=0
[   24.069416] usb 1-2.1.1: Product: Dell KB216 Wired Keyboard
[   24.077100] input: Dell KB216 Wired Keyboard as /devices/platform/3610000.xhci/usb1/1-2/1-2.1/1-2.1.1/1-2.1.1:1.0/0003:413C:2113.0001/input/input1
[   24.138595] hid-generic 0003:413C:2113.0001: input,hidraw0: USB HID v1.11 Keyboard [Dell KB216 Wired Keyboard] on usb-3610000.xhci-2.1.1/input0
[   24.143419] input: Dell KB216 Wired Keyboard Consumer Control as /devices/platform/3610000.xhci/usb1/1-2/1-2.1/1-2.1.1/1-2.1.1:1.1/0003:413C:2113.0002/input/input2
[   24.202580] input: Dell KB216 Wired Keyboard System Control as /devices/platform/3610000.xhci/usb1/1-2/1-2.1/1-2.1.1/1-2.1.1:1.1/0003:413C:2113.0002/input/input3
[   24.204450] hid-generic 0003:413C:2113.0002: input,hidraw1: USB HID v1.11 Device [Dell KB216 Wired Keyboard] on usb-3610000.xhci-2.1.1/input1
[   24.286382] usb 1-2.1.2: new low-speed USB device number 6 using tegra-xusb
[   24.311062] usb 1-2.1.2: New USB device found, idVendor=413c, idProduct=301a, bcdDevice= 1.00
[   24.312599] usb 1-2.1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[   24.314076] usb 1-2.1.2: Product: Dell MS116 USB Optical Mouse
[   24.315966] usb 1-2.1.2: Manufacturer: PixArt
[   24.322200] input: PixArt Dell MS116 USB Optical Mouse as /devices/platform/3610000.xhci/usb1/1-2/1-2.1/1-2.1.2/1-2.1.2:1.0/0003:413C:301A.0003/input/input4
[   24.325186] hid-generic 0003:413C:301A.0003: input,hidraw2: USB HID v1.11 Mouse [PixArt Dell MS116 USB Optical Mouse] on usb-3610000.xhci-2.1.2/input0
[   24.406374] usb 1-2.1.5: new high-speed USB device number 7 using tegra-xusb
[   24.443649] usb 1-2.1.5: New USB device found, idVendor=0bda, idProduct=1100, bcdDevice= 1.01
[   24.445251] usb 1-2.1.5: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[   24.446817] usb 1-2.1.5: Product: HID Device
[   24.448299] usb 1-2.1.5: Manufacturer: Realtek
[   24.456425] hid-generic 0003:0BDA:1100.0004: hiddev96,hidraw3: USB HID v1.11 Device [Realtek HID Device] on usb-3610000.xhci-2.1.5/input0
[   54.242370] vdd-1v8-sd: disabling
[   54.243908] vdd-3v3-cvb: disabling
[   54.245374] vdd-1v8-cvb: disabling
[   54.246878] vdd-epb-1v0: disabling
[   54.248317] vdd-fan: disabling
[   54.249735] vdd_sys_en: disabling
[   54.251177] avdd-cam-2v8: disabling

nvidia-jetson-xavier-nx-boot-uart-j14_clean_flashed_ubuntu2004_manual_boot.log (60.9 KB)

In this scenario, it still does not work, but it shows text on the screen connected to the Jetson Xavier HDMI port. It is actually the same text that is also visible on the serial prompt.

We need to have Ubuntu 20.04 for using ROS Noetic and ROS2 Foxy with a realsense camera. Using an Ubuntu 18.04 image and upgrading to Ubuntu 20.04 takes several hours to install and it is not working as well as a clean installation. The upgrade version will also not allow to use CUDA. (At least not when we did it 2 months ago). And using docker could be possible, but the additional setup and added complexity to operate the device is not interesting when it could work with a native installation.

So, is it the device or the image?

Now, with the SKD Manager, when I set the Xavier into recovery mode, I can see that it is recognised as recovery mode (the small “i” icon says, detected as recovery), it still fails during flashing, but now, I get another error message I think:


16:22:49 DEBUG: Flash Jetson Xavier NX - flash: using adapter to install NV_L4T_FLASH_XAVIER_NX_WITH_OS_IMAGE_COMP@JETSON_XAVIER_NX_TARGETS to /home/ubadmin/nvidia/nvidia_sdk/JetPack_5.0.1_DP_Linux_JETSON_XAVIER_NX_TARGETS
16:22:49 INFO: Flash Jetson Xavier NX - flash: [host] [ Disk Avail:1584.75 GB ]
16:22:49 INFO: Flash Jetson Xavier NX - flash: Installing component 'Flash Jetson Xavier NX' (NV_L4T_FLASH_XAVIER_NX_WITH_OS_IMAGE_COMP)
16:22:49 INFO: Flash Jetson Xavier NX - flash: change working directory to /home/ubadmin/nvidia/nvidia_sdk/JetPack_5.0.1_DP_Linux_JETSON_XAVIER_NX_TARGETS/Linux_for_Tegra
16:22:49 INFO: Flash Jetson Xavier NX - flash: [ Component Install Started ]
16:22:49 INFO: Flash Jetson Xavier NX - flash: current working directory is /home/ubadmin/nvidia/nvidia_sdk/JetPack_5.0.1_DP_Linux_JETSON_XAVIER_NX_TARGETS/Linux_for_Tegra
16:22:49 INFO: Flash Jetson Xavier NX - flash: exec_command [host]:
16:22:49 INFO: Flash Jetson Xavier NX - flash: **********************
16:22:49 INFO: Flash Jetson Xavier NX - flash: #!/bin/bash
16:22:49 INFO: Flash Jetson Xavier NX - flash: set -e
16:22:49 INFO: Flash Jetson Xavier NX - flash: set -e
16:22:49 INFO: Flash Jetson Xavier NX - flash: unset LC_ALL
16:22:49 INFO: Flash Jetson Xavier NX - flash: echo sudo ./nvsdkmanager_flash.sh
16:22:49 INFO: Flash Jetson Xavier NX - flash: sudo ./nvsdkmanager_flash.sh
16:22:49 INFO: Flash Jetson Xavier NX - flash: sleep 20
16:22:49 INFO: Flash Jetson Xavier NX - flash: **********************
16:22:49 INFO: Flash Jetson Xavier NX - flash: exec_command: /tmp/tmp_NV_L4T_FLASH_XAVIER_NX_WITH_OS_IMAGE_COMP.sh
16:22:49 INFO: Flash Jetson Xavier NX - flash: sudo ./nvsdkmanager_flash.sh
16:22:49 INFO: Flash Jetson Xavier NX - flash: Defaulting to autoflash
16:22:49 INFO: Flash Jetson Xavier NX - flash: *** Checking ONLINE mode ... OK.
16:22:49 INFO: Flash Jetson Xavier NX - flash: *** Checking target board connection ... 1 connections found.
16:22:55 ERROR: Flash Jetson Xavier NX - flash: *** Reading ECID ... SDKM_INSTALL_ERROR *** Error: ECID read failed.
16:22:55 ERROR: Flash Jetson Xavier NX - flash: Put the target board in RCM mode and retry.
16:22:55 INFO: Flash Jetson Xavier NX - flash: [ Component Install Finished with Error ]
16:22:55 INFO: Flash Jetson Xavier NX - flash: [host] [ 4.00 KB used. Disk Avail: 1584.75 GB ]
16:22:55 INFO: Flash Jetson Xavier NX - flash: [ NV_L4T_FLASH_XAVIER_NX_WITH_OS_IMAGE_COMP Install took 6s ]
16:22:55 ERROR: Flash Jetson Xavier NX - flash: command terminated with error
16:22:55 SUMMARY: Flash Jetson Xavier NX - flash: First Error: Jetson board is in a bad state and cannot be recovered. Please manually put board to recovery mode again and retry.. The Jetson target is in a bad state and cannot be flashed. Please manually put the target into recovery mode and then retry flashing.

What is the ECID?

Hi,

Instead of sdkmanager, could you directly use the flash.sh to flash the board?

I just want to recap again, when you flash the board, the software on it does not matter. The flash process will erase the whole emmc on the board before flash so what was preinstalled on it really does not matter.

I tried to flash it using the flash.sh script in the location /nvidia/nvidia_sdk/JetPack_5.0.1_DP_Linux_JETSON_XAVIER_NX_TARGETS/Linux_for_Tegra and I get the following result with exit code 1:

$ sudo ./flash.sh  jetson-xavier-nx-devkit mmcblk0p1
###############################################################################
# L4T BSP Information:
# R34 , REVISION: 1.1
###############################################################################
Error: probing the target board failed.
       Make sure the target board is connected through 
       USB port and is in recovery mode.

The board was set to recovery mode. Same result with external and internal.

Additionally, we tried seeing if the variable hwchipid called by get_fuse_level inside flash.sh gets set, it didn’t. Neither is the ./bootloader/tegrarcm_v2 capable of communicating with the device, despite specifying --usb-instance to correct bus-dev value. Running ./bootloader/tegrarcm_v2 --uid results in Failed to read UID and error code 8. Neither --boot nor --reboot and parameters are resulting in anything, the binary stays busy, but nothing happens. --skipuid doesn’t help either. Reading or writing to a partition hangs too.

The device is properly recognized as being in recovery mode as per lsusb:

Bus 001 Device 022: ID 0955:7e19 NVIDIA Corp. APX

The USB2UART converter stays quiet, since the device seems to be in recovery mode. If booted up normally is starts to output kernel ring buffer.

So it seems there is an attempt at communicating with the device, but it fails.

Could you use older jetpack and see if they can flash your board?

Is your host machine a virtual machine?

Host machine is a native installation of Ubuntu 20.04.
When flashing an SD card with Jetpack 4.6.1 with dd, it worked before (yesterday). Now, somehow it doesn’t. When flashing the card freshly with dd, the screen remains black and the UART console asks to log in as xavier@xavier-desktop.

When you mean to use an older Jetpack, do you also mean to use the flash.sh script? Then , we would need to use a PC with Ubuntu 18.04, right?

I re-flashed the device, this time with the Win32DiskImager on Windows 10 and the Jetson boots this time into the setup assistant and then it boots properly to the desktop with the Jetpack 4.6.1. Now, Ubuntu 18.04 is running just fine.