How to recover a broken Jetson Nano installation

Autonomous Machines Jetson & Embedded Systems Jetson Nano
1

Hi to all, I bought a Jetson Nano board dev kit two weeks ago. I was able to install the os as well the software packages . After few days, removing the board from the case I broken the mmc card reader of nano board so the sd card is anymore fixed inside. The sd card too was damaged during my attempt to fix. So I tried to boot the system from an external usb storage. I flashed the USB Device from the Nvidia distribution then changed the extlinux.conf configuration to /dev/sda1 and connected the serial port to a pc.
First problem: the usb storage device is not automatically detected at startup. I have to manually do a usb start-> usb reset and then the storage is recognized.

Second problem: I do the boot command from console and it loads until I received the message: Please complete system configuration setup on the serial port provided by Jetson’s USB device mode connection

I connect the micro USB port when it is started but the only message that I receive is: System configuration setup wasn’t completed. Please reboot device and try again.
The HDMI port show a blank screen during all the process.

I think that I have to create a custom configuration of usb storage but I do not know if it exists and I can download it

Thanks for your answer
Massimo

2

You can run l4t-create-default-user.sh to your rootfs when you create the BSP on the usb drive.

3

Thanks Wayne, but how can run the script?

4

Then it is my question that how did you prepare your usb drive?

If you really follow the correct steps to prepare usb drive, then you shall have L4T BSP, the same script is in L4T BSP too.

5

Wayne I flashed the usb storage on a Mac using Balena Etcher. Then I mounted the disk in a ubuntu (intel ) VM machine to change the configuration of boot device. Then I connected the usb storage to one of the four USB ports of nano, a ttl to usb converter to enable console. At boot I interrupted the start (because the board does not detect the usb storage automatically).
Usb start
Usb reset (and now the storage is detected)
boot

Now the boot run , Linux image is loaded and, after a while, ask to complete the setup.
Because I do not have a sd card working (because the sd port is broken) I cannot create an usb storage copying the file system from the sd card (the sd card itself is broken). So I think that the only way to run the script that you suggested could be during the startup of board (when I give the boot command I mean). So there is a way to force the execution of the script at startup ?
Thanks
Massimo

6

Hi,

Our official BSP only supports to work on native x86 ubuntu. Thus, unless you have one, I cannot share the method to you.

Basically, the method is based on the BSP downloaded by sdkmanager. Our full development is all based on ubuntu.

We don’t use sdcard image method.

7

Yes Wayne , I own a x86 ubuntu 18.04 computer (up2 ) so I can apply your method if you share the instructions. Thanks in advance
Massimo

8

Please download the BSP from the sdkmanager first. Then follow " Flashing to a USB Drive" to prepare the image on usb drive. Before you copy the image to the usb drive, run l4t-create-default-user script so that the rootfs will have your pre-defined user account/password.

https://docs.nvidia.com/jetson/archives/l4t-archived/l4t-3261/index.html#page/Tegra%20Linux%20Driver%20Package%20Development%20Guide/flashing.html#wwpID0E01O0HA

9

Good morning Wayne, I followed your suggestions, so:
1)Installed SDKManager on Linux intel VM x86
2) Connectedi the Nvidia nano using the usb slot
3) Downloaded the Jetson OS drivers and Image
4) created a default user in the Linux_for_Tegra directory
5) then I created a usb storage following your instructions and flashed the usb using the :
sudo rsync -axHAWX --numeric-ids --info=progress2 --exclude=/proc ./tmp_system/ /mnt

Attached the usb to board the boot is correct, I see the Nvidia splash screen then the startup lines.
But when it starts the graphic environment the scene becomes blank.

Here the Xorg.0.log:

[ 1287.947] (–) Log file renamed from “/var/log/Xorg.pid-24867.log” to “/var/l
og/Xorg.0.log”
[ 1287.948]
X.Org X Server 1.19.6
Release Date: 2017-12-20
[ 1287.948] X Protocol Version 11, Revision 0
[ 1287.948] Build Operating System: Linux 4.15.0-163-generic aarch64 Ubuntu
[ 1287.948] Current Operating System: Linux tegra-ubuntu 4.9.201-tegra #1 SMP P
REEMPT Fri Jul 9 08:56:29 PDT 2021 aarch64
[ 1287.948] Kernel command line: tegraid=21.1.2.0.0 ddr_die=4096M@2048M section
=512M memtype=0 vpr_resize usb_port_owner_info=0 lane_owner_info=0 emc_max_dvfs=
0 touch_id=0@63 video=tegrafb no_console_suspend=1 console=ttyS0,115200n8 debug_
uartport=lsport,4 earlyprintk=uart8250-32bit,0x70006000 maxcpus=4 usbcore.old_sc
heme_first=1 lp0_vec=0x1000@0xff780000 core_edp_mv=1075 core_edp_ma=4000 gpt teg
ra_fbmem=0x800000@0x92cb4000 is_hdmi_initialised=1 earlycon=uart8250,mmio32,0x7
0006000 root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 console=ttyS0,115200n8
console=tty0 fbcon=map:0 net.ifnames=0 sdhci_tegra.en_boot_part_access=1 quiet r
oot=/dev/sda1 rw rootwait rootfstype=ext4 console=ttyS0,115200n8 console=tty0 fb
con=map:0 net.ifnames=0
[ 1287.948] Build Date: 14 December 2021 04:14:51PM
[ 1287.948] xorg-server 2:1.19.6-1ubuntu4.10 (For technical support please see
Enterprise open source support | Ubuntu)
[ 1287.948] Current version of pixman: 0.34.0
[ 1287.948] Current version of pixman: 0.34.0
[ 1287.948] Before reporting problems, check http://wiki.x.org
to make sure that you have the latest version.
[ 1287.948] Markers: (–) probed, () from config file, (==) default setting,
(++) from command line, (!!) notice, (II) informational,
(WW) warning, (EE) error, (NI) not implemented, (??) unknown.
[ 1287.949] (==) Log file: “/var/log/Xorg.0.log”, Time: Fri Dec 10 09:37:12 202
1
[ 1287.949] (==) Using config file: “/etc/X11/xorg.conf”
[ 1287.949] (==) Using system config directory “/usr/share/X11/xorg.conf.d”
[ 1287.949] (==) No Layout section. Using the first Screen section.
[ 1287.949] (==) No screen section available. Using defaults.
[ 1287.950] () |–>Screen “Default Screen Section” (0)
[ 1287.950] () | |–>Monitor “”
[ 1287.950] (==) No device specified for screen “Default Screen Section”.
Using the first device section listed.
[ 1287.950] () | |–>Device “Tegra0”
[ 1287.950] (==) No monitor specified for screen “Default Screen Section”.
Using a default monitor configuration.
[ 1287.950] (==) Automatically adding devices
[ 1287.950] (==) Automatically enabling devices
[ 1287.950] (==) Automatically adding GPU devices
[ 1287.950] (==) Automatically binding GPU devices
[ 1287.950] (==) Max clients allowed: 256, resource mask: 0x1fffff
[ 1451.534] (WW) The directory “/usr/share/fonts/X11/cyrillic” does not exist.
[ 1451.534] Entry deleted from font path.
[ 1451.534] (WW) The directory “/usr/share/fonts/X11/100dpi/” does not exist.
[ 1451.534] Entry deleted from font path.
[ 1451.534] (WW) The directory “/usr/share/fonts/X11/75dpi/” does not exist.
[ 1451.534] Entry deleted from font path.
[ 1451.534] (WW) The directory “/usr/share/fonts/X11/100dpi” does not exist.
[ 1451.534] Entry deleted from font path.
[ 1451.534] (WW) The directory “/usr/share/fonts/X11/75dpi” does not exist.
[ 1451.534] Entry deleted from font path.
[ 1451.534] (==) FontPath set to:
/usr/share/fonts/X11/misc,
/usr/share/fonts/X11/Type1,
built-ins
[ 1451.534] (==) ModulePath set to “/usr/lib/xorg/modules”
[ 1451.535] (II) The server relies on udev to provide the list of input devices
.
If no devices become available, reconfigure udev or disable AutoAddDevic
es.
[ 1451.535] (II) Loader magic: 0x556e1f9010
[ 1451.535] (II) Module ABI versions:
[ 1451.535] X.Org ANSI C Emulation: 0.4
[ 1451.535] X.Org Video Driver: 23.0
[ 1451.535] X.Org XInput driver : 24.1
[ 1451.535] X.Org Server Extension : 10.0
[ 1451.536] (++) using VT number 1

[ 1451.541] (II) systemd-logind: took control of session /org/freedesktop/login
1/session/c786
[ 1451.543] (II) no primary bus or device found
[ 1451.543] (WW) “dri” will not be loaded unless you’ve specified it to be load
ed elsewhere.
[ 1451.543] (II) “glx” will be loaded by default.
[ 1451.543] (II) LoadModule: “extmod”
[ 1451.543] (II) Module “extmod” already built-in
[ 1451.543] (II) LoadModule: “glx”
[ 1451.543] (II) Loading /usr/lib/xorg/modules/extensions/libglx.so
[ 1451.546] (II) Module glx: vendor=“X.Org Foundation”
[ 1451.546] compiled for 1.19.6, module version = 1.0.0
[ 1451.546] ABI class: X.Org Server Extension, version 10.0
[ 1451.546] (II) LoadModule: “nvidia”
[ 1451.546] (II) Loading /usr/lib/xorg/modules/drivers/nvidia_drv.so
[ 1451.547] (II) Module nvidia: vendor=“NVIDIA Corporation”
[ 1451.547] compiled for 4.0.2, module version = 1.0.0
[ 1451.547] Module class: X.Org Video Driver
[ 1451.547] (II) NVIDIA dlloader X Driver 32.7.2 Release Build (integ_stage_
rel) (buildbrain@mobile-u64-5494-d8000) Wed Apr 20 14:25:16 PDT 2022
[ 1451.547] (II) NVIDIA Unified Driver for all Supported NVIDIA GPUs
[ 1451.547] (WW) Falling back to old probe method for NVIDIA
[ 1451.548] (II) Loading sub module “fb”
[ 1451.548] (II) LoadModule: “fb”
[ 1451.548] (II) Loading /usr/lib/xorg/modules/libfb.so
[ 1451.548] (II) Module fb: vendor=“X.Org Foundation”
[ 1451.548] compiled for 1.19.6, module version = 1.0.0
[ 1451.548] ABI class: X.Org ANSI C Emulation, version 0.4
[ 1451.548] (II) Loading sub module “wfb”
[ 1451.548] (II) LoadModule: “wfb”
[ 1451.549] (II) Loading /usr/lib/xorg/modules/libwfb.so
[ 1451.549] (II) Module wfb: vendor=“X.Org Foundation”
[ 1451.549] compiled for 1.19.6, module version = 1.0.0
[ 1451.549] ABI class: X.Org ANSI C Emulation, version 0.4
[ 1451.549] (II) Loading sub module “ramdac”
[ 1451.549] (II) LoadModule: “ramdac”
[ 1451.549] (II) Module “ramdac” already built-in
[ 1451.550] (WW) VGA arbiter: cannot open kernel arbiter, no multi-card support
[ 1451.550] (II) NVIDIA(0): Creating default Display subsection in Screen section
[ 1592.076] (II) NVIDIA(0): Creating default Display subsection in Screen secti
on
“Default Screen Section” for depth/fbbpp 24/32
[ 1592.076] (==) NVIDIA(0): Depth 24, (==) framebuffer bpp 32
[ 1592.076] (==) NVIDIA(0): RGB weight 888
[ 1592.076] (==) NVIDIA(0): Default visual is TrueColor
[ 1592.076] (==) NVIDIA(0): Using gamma correction (1.0, 1.0, 1.0)
[ 1592.076] (DB) xf86MergeOutputClassOptions unsupported bus type 0
[ 1592.077] () NVIDIA(0): Option “AllowEmptyInitialConfiguration” “true”
[ 1592.077] () NVIDIA(0): Enabling 2D acceleration
[ 1592.077] (II) Loading sub module “glxserver_nvidia”
[ 1592.077] (II) LoadModule: “glxserver_nvidia”
[ 1592.077] (II) Loading /usr/lib/xorg/modules/extensions/libglxserver_nvidia.s
o
[ 1592.084] (II) Module glxserver_nvidia: vendor=“NVIDIA Corporation”
[ 1592.084] compiled for 4.0.2, module version = 1.0.0
[ 1592.084] Module class: X.Org Server Extension
[ 1592.084] (II) NVIDIA GLX Module 32.7.2 Release Build (integ_stage_rel) (
buildbrain@mobile-u64-5494-d8000) Wed Apr 20 14:24:23 PDT 2022
[ 1592.084] (EE) NVIDIA(GPU-0): Failed to initialize the NVIDIA graphics device
!
[ 1592.084] (EE) NVIDIA(0): Failing initialization of X screen 0
[ 1592.084] (II) UnloadModule: “nvidia”
[ 1592.084] (II) UnloadSubModule: “glxserver_nvidia”
[ 1592.084] (II) Unloading glxserver_nvidia
[ 1592.085] (II) UnloadSubModule: “wfb”
[ 1592.085] (II) UnloadSubModule: “fb”
[ 1592.085] (EE) Screen(s) found, but none have a usable configuration.
[ 1592.085] (EE)
Fatal server error:
[ 1592.085] (EE) no screens found(EE)
[ 1592.085] (EE)
Please consult the The X.Org Foundation support
at http://wiki.x.org
for help.
[ 1592.085] (EE) Please also check the log file at “/var/log/Xorg.0.log” for ad
ditional information.
[ 1592.085] (EE)
[ 1592.353] (EE) Server terminated with error (1). Closing log file.

I found many occurrences on google search similar to this log but no solutions.
the uname -r command shows: 4.9.201-tegra
the command lsmod returns no modules loaded

in dmesg I found:
[ 1512.862758] tegradc tegradc.0: hdmi: get YCC quant from EDID.
[ 1512.930344] extcon-disp-state extcon:disp-state: cable 47 state 1
[ 1512.930346] Extcon AUX1(HDMI) enable
[ 1512.946373] tegradc tegradc.0: sync windows ret = 246
[ 1513.383274] tegradc tegradc.0: blank - powerdown
[ 1513.429764] extcon-disp-state extcon:disp-state: cable 47 state 0
[ 1513.429765] Extcon AUX1(HDMI) disable
[ 1513.451353] tegradc tegradc.0: unblank
[ 1513.491841] tegradc tegradc.0: nominal-pclk:594000000 parent:594000000 div:1.
0 pclk:594000000 588060000~647460000
[ 1513.491912] tegradc tegradc.0: hdmi: tmds rate:594000K prod-setting:prod_c_hd
mi_300m_600m

This is the welcome screen:
Welcome to Ubuntu 18.04.6 LTS (GNU/Linux 4.9.201-tegra aarch64)

To restore this content, you can run the ‘unminimize’ command.

3 updates can be applied immediately.

the modules dir is ls /lib/modules/4.9.253-tegra/
Even if the system is 4.9.201-tegra I think that the modules are correct.
This is the disk:
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda1 59457476 4999620 51414636 9% /
none 1780188 0 1780188 0% /dev
tmpfs 2025540 0 2025540 0% /dev/shm
tmpfs 2025540 179664 1845876 9% /run
tmpfs 5120 4 5116 1% /run/lock
tmpfs 2025540 0 2025540 0% /sys/fs/cgroup
tmpfs 405108 0 405108 0% /run/user/1000

I do not know how to solve this annoying problem.
lspci:
00:01.0 PCI bridge: NVIDIA Corporation Device 0fae (rev a1)
00:02.0 PCI bridge: NVIDIA Corporation Device 0faf (rev a1)
01:00.0 Network controller: Intel Corporation Wireless 8265 / 8275 (rev 78)
02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 19)

By the way I have also the wireless board but I think the driver is not loaded or recognized at startup

A suggestion is welcome !
Massimo

10

What does this comment try to say? This is just the cause that make your desktop gone.

the modules dir is ls /lib/modules/4.9.253-tegra/
Even if the system is 4.9.201-tegra I think that the modules are correct.

The system version decides where the kernel module would be loaded. If your version is 4.9.100, then it will search /lib/modules/4.9.100 for the kernel modules. Kernel modules include the gpu driver and since it is gone, desktop fails to launch.

11

Yes Wayne, you are right, this is the reason why the system did not work. At the end I cloned the content of eMMC and now it’s working.
But I do not understand why when I generate the system.img.raw, even is the kernel is 4.9.201 the modules loaded are 4.9.253:
maxdod@ubuntu:~/img/linux/JetPack_4.6.1_Linux_JETSON_NANO_TARGETS/Linux_for_Tegra$ ls
apply_binaries.sh nvautoflash.sh p3542-0000+p3448-0003-qspi.conf
bootloader nvmassflashgen.sh p3542-0000+p3448-0003-qspi-sd.conf
build_l4t_bup.sh nvsdkmanager_flash.sh p3542.conf
flash.sh nv_tegra README_Autoflash.txt
jetson-nano-2gb-devkit.conf nv_tools README_Massflash.txt
jetson-nano-devkit.conf p2371.conf README_RAPID_BOOT.txt
jetson-nano-devkit-emmc.conf p2597-0000+p2180-1000-24x7.conf rootfs
jetson-nano-emmc.conf p2597-0000+p2180-1000.conf source
jetson-nano-qspi.conf p3448-0000.conf.common source_sync.sh
jetson-nano-qspi-sd.conf p3448-0000-max-spi.conf tmp
jetson-tx1.conf p3449-0000+p3448-0000-qspi.conf tools
jetson-tx1-devkit.conf p3449-0000+p3448-0000-qspi-sd.conf TX1_boot-firmware-redundancy.txt
kernel p3449-0000+p3448-0002.conf
l4t_generate_soc_bup.sh p3450.conf
maxdod@ubuntu:~/img/linux/JetPack_4.6.1_Linux_JETSON_NANO_TARGETS/Linux_for_Tegra$ pwd
/home/maxdod/img/linux/JetPack_4.6.1_Linux_JETSON_NANO_TARGETS/Linux_for_Tegra
maxdod@ubuntu:~/img/linux/JetPack_4.6.1_Linux_JETSON_NANO_TARGETS/Linux_for_Tegra$ cd tmp
maxdod@ubuntu:~/img/linux/JetPack_4.6.1_Linux_JETSON_NANO_TARGETS/Linux_for_Tegra/tmp$ ls
bin dev home lost+found mnt proc root sbin srv tmp var
boot etc lib media opt README.txt run snap sys usr
maxdod@ubuntu:~/img/linux/JetPack_4.6.1_Linux_JETSON_NANO_TARGETS/Linux_for_Tegra/tmp$ ls lib
aarch64-linux-gnu chroot-setup.sh cryptsetup klibc-8tsH1fECQOKildhKK_y5UcXbdII.so modprobe.d systemd
apparmor console-setup firmware ld-linux-aarch64.so.1 modules terminfo
bridge-utils cpp ifupdown linux-sound-base partman udev
brltty crda init lsb resolvconf
maxdod@ubuntu:~/img/linux/JetPack_4.6.1_Linux_JETSON_NANO_TARGETS/Linux_for_Tegra/tmp$ ls lib/modules
4.9.253-tegra

Massimo

12

Hi,

It means the kernel your device is loading is different from the file under rootfs.

4.9.210 indicates this is not a image from jetpack4.6.1 because default kernel version should be 4.9.253.

Thus, the real problem here is the kernel image is somehow the old one. I don’t know why since the info you posted is not sufficient.

BTW, there is no rule says the kernel module will be changed to older version according to your kernel. They are just bunches of files under /lib/modules. If the files are missing or file name is mismatched, then it won’t get loaded either.

13

Thanks Wayne, your help has been useful to solve the problem.
I thought that at boot, the Image loaded was that in the usb rootfs, but now I know that the Image loaded is that contained in the eMMC card. So I modified extlinux.conf to set /dev/sda1 as rootfs, but then I copied the Image from usb to eMMC boot directory (saving original Image as Image.backup, as described in file extlinux.conf). Now the operating system (4.9.253-tegra) boots fine and I have the environment ready to install ros and other packages of my interest.

Thanks again. Massimo

14

Hi,

You need to modify the uboot so that uboot will search the usb first instead of emmc. This order will decide which kernel to be used. I mean where to load extlinux.conf indicates where to load the kernel. Since you load extlinux from emmc, it means your kernel is from emmc too. That causes the mismatch (kernel from emmc, modules from usb).

https://docs.nvidia.com/jetson/archives/l4t-archived/l4t-3271/index.html#page/Tegra%20Linux%20Driver%20Package%20Development%20Guide/flashing.html#wwpID0E05I0HA

15

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.