Jetson Xavier NX - won’t load GUI, can’t get it out of bash environment

Still fails and goes to bash

Flashed
sudo ./flash.sh jetson-xavier-nx-devkit mmcblk0p1

TIMEOUT 30
DEFAULT primary

MENU TITLE L4T boot options

LABEL primary
MENU LABEL primary kernel
LINUX /boot/Image
INITRD /boot/initrd
APPEND ${cbootargs} root=/dev/mmcblk1p1

NX-m0p1.txt (87.3 KB)

–Noticed some of the I2C problems at 0x50 and 0x57
0001.487] I> Find /i2c@3160000’s alias i2c0
[0001.489] I> Reading eeprom i2c=0 address=0x50
[0001.518] E> verify_cvm_eeprom_version: EEPROM incompatible version found
[0001.519] I> Eeprom read failed 0x35360018
[0001.519] I> Reading eeprom i2c=0 address=0x57
[0001.544] I> Device at /i2c@3160000:0x57
[0001.545] I> Find /i2c@c240000’s alias i2c1
[0001.545] I> Reading eeprom i2c=1 address=0x50
[0001.547] E> I2C: slave not found in slaves.
[0001.547] E> I2C: Could not write 0 bytes to slave: 0x00a0 with repeat start true.
[0001.548] E> I2C_DEV: Failed to send register address 0x00000000.
[0001.549] E> I2C_DEV: Could not read 256 registers of size 1 from slave 0xa0 at 0x00000000 via instance 1.
[0001.558] E> eeprom: Retry to read I2C slave device.
[0001.563] E> I2C: slave not found in slaves.
[0001.567] E> I2C: Could not write 0 bytes to slave: 0x00a0 with repeat start true.
[0001.575] E> I2C_DEV: Failed to send register address 0x00000000.
[0001.580] E> I2C_DEV: Could not read 256 registers of size 1 from slave 0xa0 at 0x00000000 via instance 1.
[0001.590] E> eeprom: Failed to read I2C slave device
[0001.594] I> Eeprom read failed 0x3526070d

The i2c error is expected as my previous comment.

Looks like using flash method is not able to work. Previous flash result with both mmcblk1p1 and root=/dev/mmcblk1p1 already tried to mount the filesystem from your sdcard, but it still crashed with kernel panic.

I still have some tricks to try but need you to rebuild the kernel. Not sure if you are willing to do that.

BTW, we don’t know if the i2c error there is the hardware or just some wrong number installed in the eeprom.

If this is hardware defect, then booting up the device with workaround does not really resolve the issue.

Hey Wayne, first off thanks for all the help.
I’d be happy to RMA but the warranty period is over (purchased 7/1/20) unless you have another way for me to RMA :)

I did review the code and the bug cause an i2c address iteration where it wrote to bus/address in a loop so I’m pretty confident it’s corrupted eeprom.

Anyway, would like to recover this unit if you have any other options in your bag of tricks.

Hi,

Firstly, need to confirm that did you ever have experience to rebuild the device tree of your board before?
If not, please download the source code from the archive of your release.

And follow the document to build the dtbs.

https://docs.nvidia.com/jetson/l4t/index.html#page/Tegra%20Linux%20Driver%20Package%20Development%20Guide/kernel_custom.html#

After the dtb is created, put tegra194-p3668-all-p3509-0000.dtb from your build folder to Linux_for_Tegra/kernel/dtb/ and run the flash command again. This will flash your dtb to the system.

Have not built the device tree before. I’ll get started on building dtbs and let you know. Thanks

Previous comment does not really resolve anything. It just teaches you where to find the source code and how to build the dtb.

The second part… the reason behind this issue is we have a mechanism called “plugin-manager” in the device tree file
→ hardware/nvidia/platform/t19x/jakku/kernel-dts/common/tegra194-plugin-manager-p3668.dtsi

The key lines are:

47 		fragement-tegra-sdhci-sd-dis {
48 			ids = ">=3668-0001-000";
49 			override@0 {
50 				target = <&sdhci_sd>;
51 				_overlay_ {
52 					status = "disabled";
53 				};
54 			};
55 		};
56 
57 		fragement-tegra-sdhci-emmc-dis {
58 			ids = ">=3668-0000-000";
59 			override@0 {
60 				target = <&sdhci_emmc>;
61 				_overlay_ {
62 					status = "disabled";
63 				};
64 			};
65 		};

The meaning behind these lines are: During the boot up, our board will use the board id to decide which sdhci controller will be enabled. If the NX module with emmc comes, we will enable the controller which is for emmc and disable the controller for sdcard., and similar logic when NX module with sdcard comes.

Now the problem for your board is, your eeprom is corrupted. So the plugin-manger cannot read the board id anymore and the mechanism does not work. That is why the sdhci controller for emmc is also enabled on your board. This shouldn’t happen on normal NX sdcard module.

So the solution for broken plugin-manager is… we just ignore this mechanism and turn on/off the corresponding sdhci controller manually.

Inside the file → hardware/nvidia/platform/t19x/jakku/kernel-dts/common/tegra194-p3668-common.dtsi

These lines will decide the on/off of the sdhci controller.

260 	sdhci_emmc: sdhci@3460000 {
261 		uhs-mask = <0x0>;
262 		nvidia,enable-hwcq;
263 		status = "okay";
264 	};
265 
266 	sdhci_sd: sdhci@3400000 {
267 		mmc-ocr-mask = <0x0>;
268 		cd-inverted;
269 		cd-gpios = <&tegra_main_gpio TEGRA194_MAIN_GPIO(G, 7) 0>;
270 		nvidia,cd-wakeup-capable;
271 		mmc-ocr-mask = <0>;
272 		cd-inverted;
273 		vmmc-supply = <&p3668_vdd_sdmmc1_sw>;
274 		status = "okay";
275 	};

From the naming, you can know that sdhci@3460000 is for emmc. Set this node status to disabled. Only leave the sdhci_sd to “okay”.

Hey Wayne, your amazing. Ran through the building kernel and replacing the dtb and sure enough the system booted into the install wizard and after setting up I’m able to reboot with the patched kernel. So I have a few questions now that we have an OS.

  1. Do I need to do anything to fix the eeprom. I noticed the eeprom type is incorrect:
    Dump from brick’d system
    tim@Xavier2:~/jetsonUtilities$ i2cdump -f -y 0 0x50
    0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
    00: 81 00 fc 00 54 0e 00 00 1f 00 00 00 00 00 00 00 ?.?.T?..?..
    10: 00 00 00 00 36 39 39 2d 31 33 36 36 38 2d 30 30 …699-13668-00
    20: 30 30 2d 32 30 30 20 48 2e 30 00 00 00 00 00 00 00-200 H.0…
    30: 00 00 ff ff ff ff ff ff ff ff ff ff ff ff ff ff …
    40: ff ff ff ff d1 8d 07 2d b0 48 31 34 32 32 34 32 …???-?H142242
    50: 30 30 33 36 31 38 30 00 00 00 00 00 00 00 00 00 0036180…
    60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
    70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
    80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
    90: 00 00 00 00 00 00 4e 56 43 42 1c 00 4d 31 00 00 …NVCB?.M1…
    a0: ff ff ff ff ff ff ff ff ff ff ff ff d1 8d 07 2d …???-
    b0: b0 48 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ?H…
    c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
    d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
    e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
    f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 64 …d

Dump from normal system
tim@Xavier:/dev$ i2cdump -f -y 0 0x50

 0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef

00: 01 00 fc 00 54 0e 00 00 02 47 00 00 00 00 00 00 ?.?.T?..?G…
10: 00 00 00 00 36 39 39 2d 31 33 36 36 38 2d 30 30 …699-13668-00
20: 30 30 2d 32 30 30 20 47 2e 30 00 00 00 00 00 00 00-200 G.0…
30: 00 00 ff ff ff ff ff ff ff ff ff ff ff ff ff ff …
40: ff ff ff ff 8e 9e 05 2d b0 48 31 34 32 31 30 32 …???-?H142102
50: 30 30 30 39 35 38 35 00 00 00 00 00 00 00 00 00 0009585…
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
90: 00 00 00 00 00 00 4e 56 43 42 1c 00 4d 31 00 00 …NVCB?.M1…
a0: ff ff ff ff ff ff ff ff ff ff ff ff 8e 9e 05 2d …???-
b0: b0 48 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ?H…
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 …
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 91 …?

  1. Can I now use a image without this patch. In other words, can I download a SD card image (baleana etcher with Jetpack Image) and use on this system.

I actually tried with a fresh SD card JP 4.6 and it hangs up at bash. Is there a guideline on how to restore the eeprom to resolve the boot conflict.

Lastly, I’ve included the process I went through so anyone else can follow this and recover their system. I’ve seen some other threads that have had similar issues without a detailed solution.

Note: these instructions assumes the SDK Manager was installed in the $HOME directory.

https://docs.nvidia.com/jetson/l4t/index.html#page/Tegra%20Linux%20Driver%20Package%20Development%20Guide/kernel_custom.html#
– another site Compiling Tegra Source Code | Jetson Tegra X1 and X2 | RidgeRun

$ sudo apt install git-core
– sdk manager must be installed and libraries downloaded
$ cd $HOME/nvidia/nvidia_sdk/JetPack_4.6_Linux_JETSON_XAVIER_NX_TARGETS/Linux_for_Tegra
~/nvidia/nvidia_sdk/JetPack_4.6_Linux_JETSON_XAVIER_NX_TARGETS/Linux_for_Tegra$
$HOME/nvidia/nvidia_sdk/JetPack_4.6_Linux_JETSON_XAVIER_NX_TARGETS/Linux_for_Tegra$ ./source_sync.sh
— when asked for tage enter: tegra-l4t-r32.6.1
$HOME/nvidia/nvidia_sdk/JetPack_4.6_Linux_JETSON_XAVIER_NX_TARGETS/Linux_for_Tegra$ sudo apt install build-essential bc

–download toolchain
https://docs.nvidia.com/jetson/l4t/index.html#page/Tegra%20Linux%20Driver%20Package%20Development%20Guide/xavier_toolchain.html#

Put in tar in downloads directory
$ cd ~
$ mkdir $HOME/l4t-gcc
$ cd $HOME/l4t-gcc
$ tar xf $HOME/Downloads/gcc-linaro-7.3.1-2018.05-x86_64_aarch64-linux-gnu.tar.xz

$ cd $HOME/nvidia/nvidia_sdk/JetPack_4.6_Linux_JETSON_XAVIER_NX_TARGETS/Linux_for_Tegra
~/nvidia/nvidia_sdk/JetPack_4.6_Linux_JETSON_XAVIER_NX_TARGETS/Linux_for_Tegra$ export TEGRA_KERNEL_OUT=kernel_out
~/nvidia/nvidia_sdk/JetPack_4.6_Linux_JETSON_XAVIER_NX_TARGETS/Linux_for_Tegra$ export CROSS_COMPILE=$HOME/l4t-gcc/gcc-linaro-7.3.1-2018.05-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu-
~/nvidia/nvidia_sdk/JetPack_4.6_Linux_JETSON_XAVIER_NX_TARGETS/Linux_for_Tegra$ export LOCALVERSION=-tegra
~/nvidia/nvidia_sdk/JetPack_4.6_Linux_JETSON_XAVIER_NX_TARGETS/Linux_for_Tegra$ cd $HOME/nvidia/nvidia_sdk/JetPack_4.6_Linux_JETSON_XAVIER_NX_TARGETS/Linux_for_Tegra/sources/kernel/kernel-4.9
~/nvidia/nvidia_sdk/JetPack_4.6_Linux_JETSON_XAVIER_NX_TARGETS/Linux_for_Tegra/sources/kernel$ mkdir -p $TEGRA_KERNEL_OUT

– Change sdhci controller

~/nvidia/nvidia_sdk/JetPack_4.6_Linux_JETSON_XAVIER_NX_TARGETS/Linux_for_Tegra/sources/kernel$ sudo vi $HOME/nvidia/nvidia_sdk/JetPack_4.6_Linux_JETSON_XAVIER_NX_TARGETS/Linux_for_Tegra/sources/hardware/nvidia/platform/t19x/jakku/kernel-dts/common/tegra194-p3668-common.dtsi
Inside the file → hardware/nvidia/platform/t19x/jakku/kernel-dts/common/tegra194-p3668-common.dtsi
These lines will decide the on/off of the sdhci controller.

– Locate this section

sdhci_emmc: sdhci@3460000 {
	uhs-mask = <0x0>;
	nvidia,enable-hwcq;
	status = "okay";
};

– modify to this
sdhci_emmc: sdhci@3460000 {
uhs-mask = <0x0>;
nvidia,enable-hwcq;
status = “disabled”;
};

~/nvidia/nvidia_sdk/JetPack_4.6_Linux_JETSON_XAVIER_NX_TARGETS/Linux_for_Tegra/sources/kernel/kernel-4.9$ make ARCH=arm64 O=$TEGRA_KERNEL_OUT tegra_defconfig
~/nvidia/nvidia_sdk/JetPack_4.6_Linux_JETSON_XAVIER_NX_TARGETS/Linux_for_Tegra/sources/kernel/kernel-4.9$ make ARCH=arm64 O=$TEGRA_KERNEL_OUT -j2 — j2 is number of processors to use
~/nvidia/nvidia_sdk/JetPack_4.6_Linux_JETSON_XAVIER_NX_TARGETS/Linux_for_Tegra/sources/kernel/kernel-4.9$ cp $TEGRA_KERNEL_OUT/arch/arm64/boot/dts/tegra194-p3668-all-p3509-0000.dtb $HOME/nvidia/nvidia_sdk/JetPack_4.6_Linux_JETSON_XAVIER_NX_TARGETS/Linux_for_Tegra/kernel/dtb/.

~/nvidia/nvidia_sdk/JetPack_4.6_Linux_JETSON_XAVIER_NX_TARGETS/Linux_for_Tegra/sources/kernel/kernel-4.9$ cd $HOME/nvidia/nvidia_sdk/JetPack_4.6_Linux_JETSON_XAVIER_NX_TARGETS/Linux_for_Tegra/
~/nvidia/nvidia_sdk/JetPack_4.6_Linux_JETSON_XAVIER_NX_TARGETS/Linux_for_Tegra$ vi jetson_xavier_nx_devkit.conf
add the line
SKIP_EEPROM_CHECK=“-n”;

source “${LDK_DIR}/p3668.conf.common”;
SKIP_EEPROM_CHECK=“-n”;

BLBlockSize=1048576;
EMMC_CFG=flash_l4t_t194_spi_sd_p3668.xml;
RECROOTFSSIZE=100MiB

~/nvidia/nvidia_sdk/JetPack_4.6_Linux_JETSON_XAVIER_NX_TARGETS/Linux_for_Tegra$ vi rootfs/boot/extlinux/extlinux.conf
Remove the “quiet” inside this file and this will enable more log in the serial consoel
add “root=/dev/mmcblk0p1” after the APPEND field

TIMEOUT 30
DEFAULT primary

MENU TITLE L4T boot options

LABEL primary
MENU LABEL primary kernel
LINUX /boot/Image
INITRD /boot/initrd
APPEND ${cbootargs} root=/dev/mmcblk1p1

Flash board
~/nvidia/nvidia_sdk/JetPack_4.6_Linux_JETSON_XAVIER_NX_TARGETS/Linux_for_Tegra$ sudo ./flash.sh jetson-xavier-nx-devkit mmcblk0p1

Glad that it can boot up now. You cannot use default sdcard image until you fix the content inside the eeprom.

Please try to correct the last field in your eeprom first. With the help of this document.

That byte is the crc8 checksum. I guess current value does not match the result after the calculation.

Hey Wayne, again thanks for all your help. Patched 3 bites in eeprom and system is fully recovered. Just booted with a fresh downloaded sd image. Please feel free to mark this closed.

1 Like

Hey tim472tb. We are experiencing the same issue as you with our Jetson Xavier NX. How exactly did you patch the 3 bites in the eeprom? Thank you in advance!

1 Like

Hey anvandare132,
I used i2cdump and i2cset
Dumping eeprom
i2cdump -f -y 0 0x50
Writing byte 9 with 0x48
i2cset -f -y 0 0x50 0x09 0x48

Here is a little program I used to verify checksum. It shows syntax for i2cset and lets you use i2cdump output to verify checksum.

eeprom.py (5.8 KB)

Okay so when I do and i2cdump, it should look exactly like eeprom_map_xavier2? Our looks quiet different.


Thank you so much for such a quick reply tim472tb!

Here is the link for the eeprom layout.
https://docs.nvidia.com/jetson/archives/l4t-archived/l4t-325/index.html#page/Tegra%20Linux%20Driver%20Package%20Development%20Guide/jetson_eeprom_layout.html

This is what I used as a guide for the eeprom layout to determine what I had corrupted. It looks like you may have a bit more corruption than I did. I ended up only having to change 3 bytes I think and my checksum passed. (Byes 50:00 was 81 I changed to 01, 50:08 1f->02 and 50:09 00->48.

It looks like you have corruption in all your address rows: in bytes(0:7), bytes(8:f) look like they are intact, so you have a little bigger challenge. The answer to your question is that yes, with the exception of the MAC related fields your eeprom map should look very much like the ones I provided in the code (specifically the eeprom_map_xavier2 which is the patched version)

Thank you for all the help tim472tb, finally patched our Nvidia Jetson Xavier NX back together. Wouldn’t have been possible without your help tim472tb thanks thanks thanks!

1 Like

Happy to hear it’s back. Glad I could help. It was really Wayne’s insights and all the valuable information he provided that helped us both recover the Xaviers.
Good luck :)

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.