Jetson Nano production module takes long time to boot when SD card is inserted

Hi linuxdev,

I am able to do all of this but I mount the sd card at a different location.

Is there a way I could see the logs meant to go on UART especially during boot. We do not have an hdmi, nor a possibility of serial communication to view something on minicom.

I use gtkterm instead of minicom, but both show the serial UART (serial console). Without serial communications and without HDMI the only thing left would be ssh, which of course only works if everything else is working. Are you using a custom carrier board without a serial UART header?

Also, what is the alternate mount location you tested with?

Yes, a custom carrier board without a UART header. We tried a hack by soldering wires to rx tx pins and reading out uart messages but that didnt work.

The alternate mount location is at /home/$USER/. We have a startup script that is supposed to mount the sd card onto this location. But with and without this script, the delay is the same.

Should I test mounting at /mnt particularly?

The “/mnt” location is more or less a traditional location for mounting various temporary media. The “/media” location is tied to a particular user upon GUI login. I would suggest testing the “/mnt” location.

I do actually like the attempt to solder to the UART. I don’t know if this was the right UART, but if you put a pair of oscilloscope probes on the TX and RX, and set to differential display (this should be a 3.3V or 1.8V UART output, and so a 1V/division should be about perfect to see if the UART has traffic). Then you should see activity during boot if this is the serial console UART. It may be impossible to debug what is going on without this, so it is worth trying again.

Can someone verify if the serial UART console of the Nano is 1.8V? Previous Jetsons were 3.3V, but I’m thinking 1.8V is used on the Nano. FYI, if you use the wrong voltage UART, then serial console would fail even if it were wired correctly.

Hi linuxdev,

I finally managed to get a carrier board with HDMI for the Jetson Nano production module.

I had to re-enable hdmi-display inside the kernel (which I had disabled before, thinking it might lead to the i2c timeouts observed).

However, it is not yet booting (guess it is expecting image on sd card), so I am still looking into it, thought I should update. There is one repeated behavior observed however, the appearance of the folowing logs, leading to “jumps” in timestamps and timeouts (again, delays only with SD card inserted). I could not get the console output, but the dmesg output (after boot) seemed to be the same. Please find attached the dmesg.txt file. The following logs appear a lot, I am not sure if these are due to HDMI:

[   21.366378] tegra-i2c 7000c700.i2c: i2c transfer timed out addr: 0x50
[   31.607004] tegra-i2c 7000c700.i2c: pio timed out addr: 0x50 tlen:28 rlen:128
[   31.732939] tegradc tegradc.0: unblank
[   31.733131] tegradc tegradc.1: blank - powerdown
[   32.256495] fuse init (API version 7.26)
[   41.846356] tegra-i2c 7000c700.i2c: pio timed out addr: 0x50 tlen:28 rlen:128
[   41.894734] tegra-i2c 7000c700.i2c: i2c transfer timed out addr: 0x50
[   52.086398] tegra-i2c 7000c700.i2c: pio timed out addr: 0x50 tlen:28 rlen:128
[   52.134389] tegra-i2c 7000c700.i2c: i2c transfer timed out addr: 0x50
[   62.326342] tegra-i2c 7000c700.i2c: pio timed out addr: 0x50 tlen:28 rlen:128

Also please find attached the patch I use for enabling the SD card.
dmesg.txt (64.1 KB)
sd_card.patch.txt (2.07 KB)

Hi,

Guess it was HDMI after all, which was responsible for those timeouts by i2c.

However, even after disabling HDMI (again), the observed delay for the Nano to show up on network after powering on remains high, with SD card inserted.

This does not seem to be visible in any kind of logs (syslog or dmesg).

Could it be that with the SD card inserted, the uboot from eMMC has a timeout during which it checks whether the SD card is bootable?

Something like the priority of SD card boot might be higher than boot from eMMC?

If this is the case, how do we ensure that the root is always picked from /dev/mmcblk0p1 (emmc) without any delay?

Keep in mind the production version requires flashing to the internal eMMC instead of SD card, and that a different carrier board requires a different device tree/BSP (flashing does not require the different BSP, but booting does). Also, regardless of kernel config, if a device tree claims hardware is present (or present at the wrong address), then there is a good chance that this too will break the i2c queries. I do not know in what way the SD card might be related to i2c, but it is obvious the display controller init is where the i2c goes wrong.

I am not positive about the SD card during boot, but the various Jetsons which have U-Boot do have an environment which typically goes through a list of boot devices, and for the first boot device found which has the correct files, an attempt is made to use that device. If a boot device is found, and if the boot content is not valid on that device, then it will cause failure to complete boot. Serial console within the U-Boot prompt allows editing of environment variables and could be used to force only internal eMMC to be checked.

The problem with the log you provided is that this does not look like a serial console log. The reason a serial console log is important is that this is the only logging you will get while in earlier boot stages. Serial console has access long before the Linux kernel itself ever loads. I suppose it is possible the early boot stages have no debug output, but can you confirm if the log was from a serial console rather than from Linux? If this was actually from the serial console, but leaving out information, then that too is a clue.

Hi linuxdev,

Those were not serial console logs, but plain dmesg and syslog outputs. Agreed that serial logs will be helpful. Any tips on how to capture them, except images?

I am still working on getting it to boot on the new carrier board we have, which has the HDMI.

There is no possibility of pre-boot stages being visible on HDMI. Is there no way you can get serial console logs?

Oh I get it now. For the serial logs, I will have to bring out the UART pins by soldering the ends of a TTL to USB converter (1.8 or 3.3, I’ll have to check). Then connect the USB male end to my host pc and observe the logs on mincom/picocom, etc.

I will get back to you in case that is successful (I tried it last time, it did not work, maybe I’ll have to check what’s going wrong with that).

The ssh would be ready after kernel boot into ubuntu. Thus, every delay in boot process may be the cause. Since syslog and dmesg does not show any delay, it is probably bootloader has something to check.

Or maybe some system update right before entering the system which I guess can only be seen through HDMI monitor, but should see delay on UART too.

Hi,

So we could manage to bring out the UART0 pins of Nano by soldering them to the ends of a TTL to USB converter, with the USB end connected to the host pc. I can see the login prompt on minicom by enabling the nvgetty service. However, I do not see the pre-boot messages. Any clue how to enable these?

Just to update.

We managed to get the serial messages over UART2.

Please find attached the serial console logs from power on to the moment I could SSH from another terminal.
serial_logs.txt (49.6 KB)

In that log, do you see this line?

Hit any key to stop autoboot:  0

If you have minicom running, and see the “hit any key…”, then hitting a key on the keyboard would drop you into a U-Boot prompt. This is where you can examine the environment which chooses boot device and might indicate something related to the SD card.

To continue boot, simply type “boot” and the enter key.

To print all environment variables, run the command “printenv”.

I’m actually looking at a TX2, but it should be very similar (I’m removing my command line references to the particular TX2 board). If you examine “bootcmd”, then this will be something like (you can do things like “echo $bootcmd”):

bootcmd=run distro_bootcmd

Then you can drill down:

# echo $distro_bootcmd
for target in ${boot_targets}; do run bootcmd_${target}; done

Notice that “$boot_targets” will be a space delimited list of boot devices, and that this is iterating through that list (this is where the TX2 probably differs from your Nano, but it is all a variable and easily edited):

# echo $boot_targets
mmc1 mmc0 usb0 pxe dhcp

This loop tests devices mmc1 through dhcp by prefixing the name with “bootcmd_”, and then runs that through macro expansion. For example:

# echo $bootcmd_mmc1
setenv devnum 1; run mmc_boot
# echo $bootcmd_mmc1
setenv devnum 1; run mmc_boot

Now you might be interested in edit commands. Keep in mind that if you change a variable, but do not save, then the change applies only to the one boot, and then the edit will revert upon reboot. You can test things without worry until you run “saveenv”. See “help saveenv” and “help edit”. You can quote space delimited variables. Here I’ll make up a variable which won’t do anything:

# edit deleteme
edit: hello world
# echo $deleteme
hello world

…and if you reboot, then “deleteme” will be gone.

An edit you might be interested in testing:

# echo $boot_targets
mmc1 mmc0 usb0 pxe dhcp
# edit boot_targets
edit: mmc0
# echo $boot_targets
mmc0

If this works, then you can reboot, edit again, and “saveenv”.

See also:
https://devtalk.nvidia.com/default/topic/1056592/jetson-tx2/it-take-too-long-time-on-uboot-quot-retrieving-file-quot-/post/5358159/#5358159

Hi linuxdev,

Thank you for the input.

Sorry, I should have mentioned earlier. The pins of the Nano are really narrow and close by and we were afraid we might short them, hence we could only bring out the Tx pin.

So I am just able to view the logs, not respond to it via my keyboard.

Besides we plan to put the units under production soon, so is there a way I could do these changes in the uboot itself and compile and place it on the Nano, so it is available for all units (if it works)?

There is data CRC error from sdhci driver right after your system boot up. Your system seems get stuck and then reboot again. Thus, it seems still sdcard bring up issue.

We just resolved similar issue for other users who also make custom carrier board with sdcard slot.
Please also refer to this thread.
https://devtalk.nvidia.com/default/topic/1067459/jetson-nano/sd-card-not-detected/2

Hi WayneWWW,

Thanks for the suggestion. I disabled nvidia,min-tap-delay and nvidia,max-tap-delay in the device tree, as per the discussion in the forum the link to which you provided.

The delay in SSH is not reduced, and I still get the CRC error for mmc1.

I think it might be helpful if I could get the bootloader source code and modify the arguments so that SD card is not checked at all, and boot always happens from mmc0.

Hi linuxdev and WayneWWW,

The issue got resolved. I just added a parameter:

root=/dev/mmcblk0p1

to CMDLINE_ADD variable inside the file p3448-0000.conf.common.

I hope this is a proper solution, is it possible you could confirm it? I am basically bypassing the check for rootfs on mmcblk1p1 (i.e., if SD card is present) I think, hence the CRC errors which were causing a delay in boot were skipped.

This wouldn’t fix any SD card issues, but the value is inherited throughout boot and would prevent the error from slowing down boot.

I don’t think that line resolved anything either… curious about why it helps.
Did you put your rootfs on sdcard?

Need to address again, how fast you could ssh to device depends on how long it takes to enter ubuntu system. When your UART console shows the prompt for password, ssh may become available. What makes your boot up time delay is the sdcard driver. It hits kernel panic so you system reboots.

I think it might be helpful if I could get the bootloader source code and modify the arguments so that SD card is not checked at all,

You don’t even need bootloader source because your error is in kernel. Could you enable my debug patch in previous link and show the log again?

Also, does your system able to detect sdcard and see the content?