Jetson Nano custom board not booting up

Hello,

How do we get the boot logs for Custom Carrier board? We have headless setup, we know that the booting has started but unable to run the startup script. how do we generate the boot logs. The PWR_EN pin gets 5V.

This is the method from the devkit.

For custom board, you should check the design guide to find out the UART pin and use that pin to dump the log.

Hello Wyne
As per the design guide we have used UART2_TXD and UART2_RXD for debug USB. we are able to see logs in an empty production module SOM (Tiny Linux), but unable to see logs in SOM flashed with L4T 32.5.1. we do not have a HDMI interface in our custom board.
we re following pinmux changes as described in the ā€œplatform adaption and bring-up documentā€ that has to be done for the custom board (as we do not have HDMI Interface and single camera, USB 3.1 X4, Ethernet).

the below link shows that
https://docs.nvidia.com/jetson/l4t/index.html#page/Tegra%20Linux%20Driver%20Package%20Development%20Guide/adaptation_and_bringup_nano.html#wwpID0E0EQ0HA , in pinmux spread sheet the columns ā€œto be filled by customers has to be editedā€

and the below link shows that
https://developer.download.nvidia.com/embedded/L4T/r32-2_Release_v1.0/Tegra_Linux_Driver_Package_Nano_Adaptation_Guide.pdf?afnc6PMxcKmg-aGTfFbVOVbUPbWlf1-rZg4eZhRszi3AKWc96jP-QwqjUsWEa7t5W5VtmiHDqXFwK3kbule_9ENXNBCZWIeZK1Cn7AOgswz9heDnQcegOn7vUUejO3WyzoF5A0ogToIbRWw-jfJe47BnPm36el5nXRHr5sa-wrpZlNY1WoCx2j0h5F2mta4Fl9vE
the ā€œcolumns through AE-AO must be editedā€

please suggest the exact column that has to be edited to reflect the changes.

It is not possible that log only shows in some specific release but not in rel-32.5.1. Please reflash it first and check the log from the beginning when you boot up the device.

The log will stop printing if you donā€™t do the first boot configuration. If you donā€™t know what I am talking about, use the devkit to learn what is the expected behavior first.

Hello,
I work with Vindhya. I just wanted to clarify a few more details.
Our custom board has the same electrical connections as that of dev-kit. Only difference is that we donā€™t have OTG, USB, HDMI/DP, camera and ethernet peripherals. as we donā€™t need these for our product.

As a testā€¦
We put an empty SOM on our custom board and through the serial console/debug (not OTG), we are able to get boot logs.
Empty_SOM.txt (5.2 KB)
This is an incomplete boot but we are able to see the logs for further debugging and we know that the peripherals are working

The next test we conducted is to simply put a SOM with the standard kernel flashed in it, not expecting it to work, but to just check if our serial console works (similar to tinylinux) but that seems to not be the case. We are not getting any logs in minicom. This has nothing to do with versionā€¦ as we have tried 3 different L4T versions. (32.4.4, 32.5.1 and 32.5.2)

We also cannot do oem-config because, the booting will not have reached that partā€¦ TBH we cannot even know that since our serial console is not giving any output.

One more clarificationā€¦ we have used FTDI232 to convert UART2_RXD, UART2_TXD signals to 5V and got it as a micro USB-B to be interfaced with a computer

We initially suspected that FTDI drivers may not be installed but through our tests, we have been able to confirm that the drivers are installed in the standard kernel. So that cannot be the issue.

Also, we have followed the platform adaptation process and disabled the peripherals but that flash failed and we are still not getting any logs to see where we failed. We are asking help so that we can debug the issues.

In summary, we know the hardware works but we need to change something on the software side to get the logsā€¦

I donā€™t quite understand the situation here.

What your so-called ā€œempty SOMā€ log already booted into kernel. Can you share the full boot log of such ā€œempty SOMā€ ? The uart log shall give out the log from bootloader.

Where did that empty SOM come from? You install something with your own BSP?
Only the uboot and kernel on jetson nano is public source, but your case sounds like even the bootloader log prior to the uboot is not able to get print from your uart.

1 Like

Hi,

Thanks for replying so promptly.

We have bought a few emmc SOMs to conduct our experiments. On 2 of them we have installed NVidia kernel but the others are as-is, untouched by usā€¦ This is our ā€œempty SOMā€. Not actually emptyā€¦ It must be a bootloader as you mentioned or an RTOS that NVidia uses for operation purposes, I assume, to conduct hardware tests. The logs mention it is TinyLinux. This is the only log we haveā€¦

<3>[ 13.100207] tegra-i2c 7000c700.i2c: pio timed out addr: 0x50 tlen:28 rlen:128
<3>[ 13.107612] tegra-i2c 7000c700.i2c: ā€” register dump for debugging ----
<3>[ 13.114573] tegra-i2c 7000c700.i2c: I2C_CNFG - 0x22c00
<3>[ 13.119973] tegra-i2c 7000c700.i2c: I2C_PACKET_TRANSFER_STATUS - 0x10001
<3>[ 13.126934] tegra-i2c 7000c700.i2c: I2C_FIFO_CONTROL - 0xe0
<3>[ 13.132764] tegra-i2c 7000c700.i2c: I2C_FIFO_STATUS - 0x800040
<3>[ 13.138858] tegra-i2c 7000c700.i2c: I2C_INT_MASK - 0x7d
<3>[ 13.144341] tegra-i2c 7000c700.i2c: I2C_INT_STATUS - 0x0
<3>[ 13.149912] tegra-i2c 7000c700.i2c: i2c transfer timed out addr: 0x50
<6>[ 13.156851] extcon-disp-state extcon:disp-state: cable 47 state 1
<6>[ 13.163205] Extcon AUX1(HDMI) enable
<6>[ 13.167770] tegradc tegradc.1: disp1 connected to head1->/host1x/sor
<6>[ 13.174457] tegradc tegradc.1: No lt-data, using default setting
<4>[ 13.174493] edid invalid
<6>[ 13.183524] tegradc tegradc.1: No hpd-gpio in DT
<6>[ 13.188427] tegradc tegradc.1: DT parsed successfully
<6>[ 13.193769] tegradc tegradc.1: Display dc.ffffff800f4c0000 registered with id=1
<6>[ 13.201319] tegradc tegradc.0: nominal-pclk:25174825 parent:25173906 div:1.0 pclk:25173906 24923052~27440532
<3>[ 13.212566] tegradc tegradc.1: dpd enable lookup fail:-19
<6>[ 13.223335] tegradc tegradc.1: probed
<6>[ 13.227620] tegradc tegradc.1: fb registered
<6>[ 13.235285] hpd: state 7 (Takeover from bootloader), hpd 0, pending_hpd_evt 1
<6>[ 13.242708] hpd: switching from state 7 (Takeover from bootloader) to state 0 (Reset)
<6>[ 13.244429] isp 54600000.isp: initialized
<6>[ 13.245788] isp 54680000.isp: initialized
<6>[ 13.259462] hpd: state 0 (Reset), hpd 0, pending_hpd_evt 0
<6>[ 13.265291] extcon-disp-state extcon:disp-state: cable 44 state 0 already set.
<6>[ 13.272782] Extcon DP: HPD disabled
<6>[ 13.276542] hpd: hpd_switch 0
<6>[ 13.279766] hpd: switching from state 0 (Reset) to state 1 (Check Plug)
<6>[ 13.285109] last reset is due to power on reset
<6>[ 13.285112] KERNEL: PMC reset status reg: 0x0
<6>[ 13.285196] BL: PMC reset status reg: 0x0
<6>[ 13.285198] BL: PMIC poweroff Event Recorder: 0x40
<6>[ 13.286950] clk_cbus_recalc_rate: no gbus parent
<6>[ 13.286955] clk_cbus_round_rate: no gbus parent
<6>[ 13.286957] clk_cbus_round_rate: no gbus parent
<6>[ 13.286964] clk_cbus_recalc_rate: no gbus parent
<6>[ 13.287160] clk_cbus_recalc_rate: no gbus parent
<6>[ 13.287163] clk_cbus_round_rate: no gbus parent
<6>[ 13.287165] clk_cbus_round_rate: no gbus parent
<6>[ 13.287168] clk_cbus_recalc_rate: no gbus parent
<6>[ 13.287184] tegra_dvfs: GPU-cap: registered
<6>[ 13.287280] tegra dvfs: vdd-cpu: nominal 1168mV, offset 708000uV, step 19200uV, scaling enabled
<6>[ 13.287283] tegra dvfs: vdd-core: nominal 1075mV, offset 1000000uV, step 12500uV, scaling enabled
<6>[ 13.287285] tegra dvfs: vdd-gpu: nominal 1038mV, offset 708000uV, step 10000uV, scaling enabled
<6>[ 13.325184] tegra_dvfs: vdd-gpu-vts: registered
<6>[ 13.334532] tegra_core_action core_dvfs_cdev_floor: Tegra CORE DVFS ā€˜floor cooling deviceā€™ registered
<6>[ 13.335342] tegra_core_action core_dvfs_cdev_cap: Tegra CORE DVFS ā€˜cap cooling deviceā€™ registered
<6>[ 13.336361] input: gpio-keys as /devices/gpio-keys/input/input0
<6>[ 13.336938] hctosys: unable to open rtc device (rtc1)
<6>[ 13.337966] vi 54080000.vi: vi_probe: ++
<6>[ 13.415733] vi 54080000.vi: initialized
<6>[ 13.418100] hpd: state 1 (Check Plug), hpd 0, pending_hpd_evt 0
<6>[ 13.418106] hpd: switching from state 1 (Check Plug) to state 3 (Disabled)
<6>[ 13.435748] vi 54080000.vi: subdev nvcsiā€“1 bound
<6>[ 13.441226] mmcblk mmc0:0001: Card claimed for testing.
<6>[ 13.448646] Disable partitions left on by BL
<6>[ 13.453204] disb
<6>[ 13.455658] bwmgr: missing cdev-type property
<6>[ 13.460505] tegra_soctherm 700e2000.soctherm: soctherm: trip temperature -2147483647 forced to -127000
<6>[ 13.470084] DRAM derating cdev registered.
<6>[ 13.475126] hvc_sysfs: hypervisor is not present
<6>[ 13.480563] vdd-fan: disabling
<6>[ 13.483875] vdd-usb-vbus: disabling
<6>[ 13.487638] vdd-usb-vbus2: disabling
<6>[ 13.491482] vddio-sdmmc-ap: disabling
<6>[ 13.495498] vddio-sdmmc3-ap: disabling
<6>[ 13.499591] vdd-3v3-sd: disabling
<6>[ 13.503172] avdd-io-edp-1v05: disabling
<6>[ 13.507272] vdd-usb-hub-en: disabling
<6>[ 13.511286] ALSA device list:
<6>[ 13.514522] No soundcards found.
<6>[ 13.521359] Freeing unused kernel memory: 8064K
Booting TinyLinux
Loading pcie-tegra-dw
Loading pci-tegra
Loading pcie-tegra-dw-ep
Found partition: /dev/mmcblk0p13
ls: /sys/bus/pci/devices/*/uevent: No such file or directory
Starting modulesā€¦ OK [0.1s]
Starting mountā€¦mount: mounting UUID=b39a7ef2-3c11-442b-bddc-519b1c9728c7 on /home failed: No such file or directory
FAILED [0.1s]
Starting syslogdā€¦ OK [0.0s]
Starting hostnameā€¦ OK [0.0s]
Starting net.loā€¦ OK [0.0s]
Starting bashā€¦ OK [0.0s]
Starting modsā€¦ OK [0.0s]
Starting consoleā€¦ ttyS0 OK [0.0s]
Starting adbā€¦ OK [1.0s]
Starting fanā€¦ OK [0.0s]
Starting randomā€¦ OK [0.0s]
[kernel 12.9s, init 0.5s, services 1.7s, total 15.2s]

[16:00:15] /home

I have attached the same boot log in the previous message.
It fails at a point where it is not able to mount some partition that it needs.

Just want to know

  1. Why is it the ā€œonly logā€ you have? Are they all getting flashed now?

  2. Is this ā€œempty SOMā€ test using the same carrier board? I mean the board that cannot dump any log now.

What is your method to flash kernel only? I mean you are sure you only flashed the kernel and you expecting it not working. How did you do that?

The uart log shall start to print even in bootloader stage. In that stage, the kernel has nothing to do with it.

Hi,

  1. Every time we boot from the ā€œempty SOMā€, we get the same/similar logsā€¦ that is all I meant. This is the one I have saved, with me. I said that especially since you asked for full logsā€¦ I hope that clarifies your doubt regarding the log. If you want us to do anything else, please let us know. This log is from an SOM placed on the custom carrier board (not dev-kit)

  2. sorry, I didnā€™t mean only the kernelā€¦ in hindsight that may be the wrong words used.
    the only flashing procedure we have followed is the nvidia L4T BSP packages based using SDK manager. the complete ubuntu OS flashing
    But once the flashing is done, we no longer get any messages on our custom board, just a blank screen.

If I run lsusb, the custom board is identified as follows:
Bus 001 Device 037: ID 0403:6001 Future Technology Devices International, Ltd FT232 USB-Serial (UART) IC

we are using gtkterm as the serial console.
we are running ubuntu 18.04 LTS on our laptops. The flashing process was also done through the same OS

anything else you want to know about the setup/environment, please ask

If you are saying the kernel has nothing to with it, Iā€™ll accept itā€¦ Can you point us in the right direction? what is the reason for not even getting bootloader logs? What should I revisit?

Hi,

The full log means I need to check the log prior to the kernel. But you only shared the log that is going to finish in kernel.

For example, tegradc in your log is kernel log. I donā€™t need to check kernel log at this moment. I need the bootloader log to see why it can print log at that time but fails after you flash with sdkm rel-32.4/32.5.

Hi, I tried regenerating the logs from when we connect the boardā€¦ I redirected the logs to a file Check if this has what you need.
This is the factory output/unflashed SOM, placed on our custom carrier board.

minicom.log (63.3 KB)

Is there any log prior to these lines?

i3ā€šĆŖU Ā½Ćā€¢Ć‰Āā€™ā€¦Ā„Ā±ĀJƍĀĀŖƁ5RĆ¾[0000.393] CPU clock enabled
[0000.397] Performing RAM repair
[0000.400] Updating A64 Warmreset Address to 0xa00002e9

I opened the port for the serial console as soon as I powered on the device because it didnā€™t allow me to open port before powering on (USB was always connected.

Iā€™ll try with a different software and see if I can get logs before that pointā€¦ It might take some time though.

The log will tell us what was the bootloader version installed on your module.

I guess it is some early release bootloader. If possible, you can also try to flash with early release like rel-32.2 or rel-32.3.1 to see if they can dump the log.

1 Like

logs_minicom.txt (62.0 KB)

This is from the very first log as far as I can seeā€¦ since I powered on after connecting.

Just another question, are you able to see any log coming out from uart when you are flashing the board?

You can use devkit to check what I am indicating first.

Hi,

We can get the logs through devkit but we cannot flash using our custom carrier board because we donā€™t have the micro-usb that the devkit has (ttyACM0). We get the logs through debug serial UART (ttyUSB0).

Will that be an issue?

Are you talking about your board doesnā€™t have physical micro usb port so that it cannot flash at all

Or you are just saying that there is no ttyACM0?

We donā€™t care about ttyACM0. When I say uart log, it is just log from uart pin, not usb.

The custom board doesnā€™t have a physical micro USB port through which we can flash. We have tried.

The only physical USB is connected to the serial console

What is your command to flash over the devkit?