TK1 boot failure and debug serial terminal stops working as well

Autonomous Machines Jetson & Embedded Systems Jetson TK1
1

The Tegra K1 only boots partially and after that the serial terminal (UART4) stops working as well (can’t accept command, etc), here is screen copy from terminal log.

Could anyone help take a look and check what’s wrong in following logs?

[ 6.052566] ***********************************************
[ 6.052570] *Tuning values
[ 6.052572] estimated values
[ 6.052577] T2T_Vmax -111, T2T_Vmin 188, 1’st_hole_Vmax -272, UI_Vmax 0
[ 6.052580] Calculated values
[ 6.052584] T2T_Vmax 66, 1’st_hole_Vmax 38, UI_Vmax 75
[ 6.052668] T2T_Vmin -84, 1’st_hole_Vmin 156, UI_Vmin 0
[ 6.052671] ***********************************
[ 6.052677] *final tuning windows
[ 6.052681] win[0]: 26 - 150
[ 6.052684] win[1]: 47 - -32
[ 6.052687] win[2]: 75 - 150
[ 6.052691] win[3]: 127 - -91
[ 6.052694] win[4]: 150 - 150
[ 6.052703] win[5]: 208 - -150
[ 6.052706] win[6]: 226 - -189
[ 6.052709] ********************************
[ 6.052713] best tap win - (26-150), best tap value 88 prev_best_tap 88
[ 6.052724] sdhci-tegra sdhci-tegra.3: sdhci_tegra_verify_best_tap: tuning freq 200000000hz, best tap 88
[ 6.067781] mmc0: new HS200 MMC card at address 0001
[ 6.068010] mmcblk mmc0:0001: Card claimed for testing.
[ 6.068209] mmcblk0: mmc0:0001 HAG2e 14.6 GiB
[ 6.071589] Console: switching to colour frame buffer device 240x67
[ 6.072135] mmcblk0: p1
[ 6.270305] tegradc tegradc.0: fb registered
[ 6.277778] hdmi_state_machine_worker (tid edb8d5c0): state 7 (Takeover from bootloader), hpd 0, pending_hpd_evt 1
[ 6.291515] hdmi_state_machine_set_state_l: switching from state 7 (Takeover from bootloader) to state 0 (Reset)
[ 6.305083] as3722-rtc as3722-rtc.1: setting system clock to 2000-01-01 00:10:06 UTC (946685406)
[ 6.317571] Disabling clocks left on by bootloader:
[ 6.325922] audio_2x
[ 6.331821] audio
[ 6.337367] audio4
[ 6.342996] audio3
[ 6.344766] hdmi_state_machine_worker (tid edb8d5c0): state 0 (Reset), hpd 0, pending_hpd_evt 0
[ 6.344771] hdmi_disable_l: audio_switch 0
[ 6.344773] hdmi_disable_l: hpd_switch 0
[ 6.344779] hdmi_state_machine_set_state_l: switching from state 0 (Reset) to state 1 (Check Plug)
[ 6.354766] hdmi_state_machine_worker (tid edb8d5c0): state 1 (Check Plug), hpd 0, pending_hpd_evt 0
[ 6.354771] hdmi_disable_l: audio_switch 0
[ 6.354774] hdmi_disable_l: hpd_switch 0
[ 6.354779] hdmi_state_machine_set_state_l: switching from state 1 (Check Plug) to state 3 (Disabled)
[ 6.428782] audio2
[ 6.434783] audio1
[ 6.440741] audio0
[ 6.446679] hdmi_audio
[ 6.452856] entropy
[ 6.458740] dsi2-fixed
[ 6.464831] dsi1-fixed
[ 6.470894] usb3
[ 6.476430] usb2
[ 6.481988] mipi-cal-fixed
[ 6.488403] i2c3
[ 6.493910] i2c2
[ 6.499374] nor
[ 6.504699] trace
[ 6.510160] sdmmc4_ddr
[ 6.516058] sdmmc3_ddr
[ 6.521949] sdmmc3
[ 6.527513] cclk_lp
[ 6.533126] pciex
[ 6.538604] pll_x_out0
[ 6.544498] pll_m_out1
[ 6.550428] tegra_dvfs: vdd_cpu connected to regulator
[ 6.558882] tegra_dvfs: vdd_core connected to regulator
[ 6.567399] tegra_dvfs: vdd_gpu connected to regulator
[ 6.576477] CPU rate: 696 MHz
[ 6.582910] tegra cooling device gpu_scaling failed to register
[ 6.876620] tegra_simon_init_gpu: Failed to find gpu thermal zone
[ 6.886809] tegra_simon_add_grader: Failed to initialize grader for gpu
[ 6.897515] tegra_simon_init_cpu: Failed to find cpu thermal zone
[ 6.907749] tegra_simon_add_grader: Failed to initialize grader for cpu
[ 6.918514] ALSA device list:
[ 6.925646] #0: HDA NVIDIA Tegra at 0x70038000 irq 113
[ 6.938377] EXT4-fs (mmcblk0p1): couldn’t mount as ext3 due to feature incompatibilities
[ 6.951367] EXT4-fs (mmcblk0p1): couldn’t mount as ext2 due to feature incompatibilities
[ 7.049807] EXT4-fs (mmcblk0p1): recovery complete
[ 7.060275] EXT4-fs (mmcblk0p1): mounted filesystem with ordered data mode. Opts: (null)
[ 7.072899] VFS: Mounted root (ext4 filesystem) on device 179:1.
[ 7.088109] devtmpfs: mounted
[ 7.095917] Freeing unused kernel memory: 508K (c0b1c000 - c0b9b000)
[ 7.355693] init: plymouth-upstart-bridge main process (108) terminated with status 1
[ 7.368918] init: plymouth-upstart-bridge main process ended, respawning
[ 7.419120] init: plymouth-upstart-bridge main process (118) terminated with status 1
[ 7.431545] init: plymouth-upstart-bridge main process ended, respawning
[ 7.444852] init: ureadahead main process (111) terminated with status 5
[ 7.494679] init: plymouth-upstart-bridge main process (121) terminated with status 1
[ 7.507093] init: plymouth-upstart-bridge main process ended, respawning
[ 9.444132] vi vi.1: vi_probe: couldn’t get regulator
[ 9.457332] vi vi.1: vi_probe: failed
[ 9.463586] vi vi.1: vi_create_debugfs: failed to create tegra_vi_1 directory
[ 9.475428] vi vi.1: vi_create_debugfs: could not create debugfs
[ 9.484278] vi vi.1: vi_probe: couldn’t get regulator
[ 9.495410] vi vi.1: vi_probe: failed
[ 9.502253] vi vi.1: vi_create_debugfs: failed to create tegra_vi_1 directory
[ 9.511190] vi vi.1: vi_create_debugfs: could not create debugfs
[ 9.519696] vi vi.1: vi_probe: couldn’t get regulator
[ 9.527394] vi vi.1: vi_probe: failed

  • Starting Mount filesystems on boot [ OK ]
  • Starting Signal sysvinit that the rootfs is mounted [ OK ]
  • Starting Clean /tmp directory [ OK ]
  • Starting Populate and link to /run filesystem [ OK ]
  • Stopping Populate and link to /run filesystem [ OK ]
  • Stopping Clean /tmp directory [ OK ]
  • Stopping Track if upstart is running in a container [ OK ]
  • Starting Initialize or finalize resolvconf [ OK ]
  • Starting Signal sysvinit that virtual filesystems are mounted [ OK ]
  • Starting Signal sysvinit that virtual filesystems are mounted [ OK ]
  • Starting Signal sysvinit that local filesystems are mounted [ OK ]
  • Starting Bridge udev events into upstart [ OK ]
  • Starting Signal sysvinit that remote filesystems are mounted [ OK ]
  • Stopping Mount filesystems on boot [ OK ]
  • Starting NFSv4 id <-> name mapper [ OK ]
  • Starting Enabling additional executable binary formats [ OK ]
  • Starting flush early job output to logs [ OK ]
  • Starting D-Bus system message bus [ OK ]
  • Starting device node and kernel event manager [ OK ]
  • Stopping flush early job output to logs [ OK ]
  • Starting NVIDIA specific init script [ OK ]
  • Starting load modules from /etc/modules [ OK ]
  • Starting cold plug devices [ OK ]
  • Starting log initial device creation [ OK ]
  • Starting SystemD login management service [ OK ]
  • Stopping NVIDIA specific init script [ OK ]
  • Stopping load modules from /etc/modules [ OK ]
  • Stopping rpcsec_gss daemon [ OK ]
  • Starting bluetooth daemon [ OK ]
  • Stopping cold plug devices [ OK ]
  • Starting system logging daemon [ OK ]
  • Stopping NVIDIA specific first-boot script [ OK ]
  • Stopping log initial device creation [ OK ]
  • Starting CUPS printing spooler/server [ OK ]
  • Starting userspace bootsplash [ OK ]
  • Starting configure network device security [ OK ]
  • Starting enable remaining boot-time encrypted block devices [ OK ]
  • Stopping userspace bootsplash [ OK ]
  • Starting Send an event to indicate plymouth is up [ OK ]
  • Starting mDNS/DNS-SD daemon [ OK ]
  • Starting Reload cups, upon starting avahi-daemon to make sure remote queues are populated [ OK ]
  • Starting cups-browsed - Bonjour remote printer browsing daemon [ OK ]
  • Stopping Send an event to indicate plymouth is up [ OK ]
  • Stopping Reload cups, upon starting avahi-daemon to make sure remote queues are populated [ OK ]
  • Starting System V initialisation compatibility [ OK ]
  • Starting modem connection manager [ OK ]
  • Starting configure network device security [ OK ]
  • Starting early crypto disks… [ OK ]
  • Starting configure virtual network devices [ OK ]
  • Setting sensors limits [ OK ]
  • Starting network connection manager [ OK ]
  • Setting up X socket directories… [ OK ]
  • Stopping System V initialisation compatibility [ OK ]
  • speech-dispatcher disabled; edit /etc/default/speech-dispatcher
  • Starting NTP server ntpd [ OK ]
    saned disabled; edit /etc/default/saned
  • Starting System V runlevel compatibility [ OK ]
  • Starting anac(h)ronistic cron [ OK ]
  • Starting save kernel messages [ OK ]
  • Starting OpenSSH server [ OK ]
  • Starting regular background program processing daemon [ OK ]
  • Starting crash report submission daemon [ OK ]
  • Stopping save kernel messages [ OK ]
  • Stopping System V runlevel compatibility [ OK ]
  • Stopping anac(h)ronistic cron [ OK ]

Ubuntu 14.04.3 LTS tegra-ubuntu ttyS0

tegra-ubuntu login: ubuntu (automatic login)

Last login: Sat Jan 1 00:59:59 UTC 2000 on ttyS0
Welcome to Ubuntu 14.04.3 LTS (GNU/Linux 3.10.40-eniac armv7l)

  • Documentation: https://help.ubuntu.com/

  • Starting Mount filesystems on boot [ OK ]

  • Starting Signal sysvinit that the rootfs is mounted [ OK ]

  • Starting Clean /tmp directory [ OK ]

  • Starting Populate and link to /run filesystem [ OK ]

  • Stopping Populate and link to /run filesystem [ OK ]

  • Stopping Clean /tmp directory [ OK ]

  • Stopping Track if upstart is running in a container [ OK ]

  • Starting Initialize or finalize resolvconf [ OK ]

  • Starting Signal sysvinit that virtual filesystems are mounted [ OK ]

  • Starting Signal sysvinit that virtual filesystems are mounted [ OK ]

  • Starting Signal sysvinit that local filesystems are mounted [ OK ]

  • Starting Bridge udev events into upstart [ OK ]

  • Starting Signal sysvinit that remote filesystems are mounted [ OK ]

  • Stopping Mount filesystems on boot [ OK ]

  • Starting NFSv4 id <-> name mapper [ OK ]

  • Starting Enabling additional executable binary formats [ OK ]

  • Starting flush early job output to logs [ OK ]

  • Starting D-Bus system message bus [ OK ]

  • Starting device node and kernel event manager [ OK ]

  • Stopping flush early job output to logs [ OK ]

  • Starting NVIDIA specific init script [ OK ]

  • Starting load modules from /etc/modules [ OK ]

  • Starting cold plug devices [ OK ]

  • Starting log initial device creation [ OK ]

  • Starting SystemD login management service [ OK ]

  • Stopping NVIDIA specific init script [ OK ]

  • Stopping load modules from /etc/modules [ OK ]

  • Stopping rpcsec_gss daemon [ OK ]

  • Starting bluetooth daemon [ OK ]

  • Stopping cold plug devices [ OK ]

  • Starting system logging daemon [ OK ]

  • Stopping NVIDIA specific first-boot script [ OK ]

  • Stopping log initial device creation [ OK ]

  • Starting CUPS printing spooler/server [ OK ]

  • Starting userspace bootsplash [ OK ]

  • Starting configure network device security [ OK ]

  • Starting enable remaining boot-time encrypted block devices [ OK ]

  • Stopping userspace bootsplash [ OK ]

  • Starting Send an event to indicate plymouth is up [ OK ]

  • Starting mDNS/DNS-SD daemon [ OK ]

  • Starting Reload cups, upon starting avahi-daemon to make sure remote queues are populated [ OK ]

  • Starting cups-browsed - Bonjour remote printer browsing daemon [ OK ]

  • Stopping Send an event to indicate plymouth is up [ OK ]

  • Stopping Reload cups, upon starting avahi-daemon to make sure remote queues are populated [ OK ]

  • Starting System V initialisation compatibility [ OK ]

  • Starting modem connection manager [ OK ]

  • Starting configure network device security [ OK ]

  • Starting early crypto disks… [ OK ]

  • Starting configure virtual network devices [ OK ]

  • Setting sensors limits [ OK ]

  • Starting network connection manager [ OK ]

  • Setting up X socket directories… [ OK ]

  • Stopping System V initialisation compatibility [ OK ]

  • speech-dispatcher disabled; edit /etc/default/speech-dispatcher

  • Starting NTP server ntpd [ OK ]
    saned disabled; edit /etc/default/saned

  • Starting System V runlevel compatibility [ OK ]

  • Starting anac(h)ronistic cron [ OK ]

  • Starting save kernel messages [ OK ]

  • Starting OpenSSH server [ OK ]

  • Starting regular background program processing daemon [ OK ]

  • Starting crash report submission daemon [ OK ]

  • Stopping save kernel messages [ OK ]

  • Stopping System V runlevel compatibility [ OK ]

  • Stopping anac(h)ronistic cron [ OK ]

2

I noticed that it recovered the root partition via journal…it considered that the root partition was corrupt or otherwise incorrect and repaired it…here’s what your log shows:

[ 6.938377] EXT4-fs (mmcblk0p1): couldn't mount as ext3 due to feature incompatibilities
[ 6.951367] EXT4-fs (mmcblk0p1): couldn't mount as ext2 due to feature incompatibilities
[ 7.049807] EXT4-fs (mmcblk0p1): recovery complete
[ 7.060275] EXT4-fs (mmcblk0p1): mounted filesystem with ordered data mode. Opts: (null)
[ 7.072899] VFS: Mounted root (ext4 filesystem) on device 179:1.

…note that the line about mounting ext2 is normal, it’s testing file system type ext2 before ext4, but then it says “recovery complete” prior to mounting.

Here’s a sample of how it should look:

[    8.773115] EXT4-fs (mmcblk0p1): couldn't mount as ext2 due to feature incompatibilities
[    8.793130] EXT4-fs (mmcblk0p1): mounted filesystem with ordered data mode. Opts: (null)
[    8.805300] VFS: Mounted root (ext4 filesystem) on device 179:1.

In the latter example there is no “recovery” required, the system was shut down cleanly before restart.

It’s reasonable odds that file system recovery has left out something important for booting. If what you have on the system is important, you could attempt to clone the root partition and then repair it or extract information on a host machine, then install the fixed clone, or simply do a fresh install and then put back in place whatever was important. See:
http://elinux.org/Jetson/Cloning

3

Thanks linexdev. I think the file system is fine and even a fresh install still shows the same problem that serial debug terminal stops working after booting. The serial debug port is supposed to most robust thing, it’s very strange to me it crashes.

4

For the serial port to fail some rather low level code has to fail, so yes, that is unusual. Does the fresh install show the line with “recovery complete”? If so, it may be flashing a corrupted system.img. If for any reason a system.img exists and is being reused rather than being generated the bad image would keep being flashed. To be sure you might want to search for system.img and system.img.raw in the boot directory and delete those and flash again…you may even want to delete the sample rootfs and be sure to use sudo to unpack it again and run the apply_binaries.sh script (unpacking as non-root could cause the flash image to be incorrect).

5

It’s re-flashed and still same crash. I have a working TK1 board and it produces the same log (recovery complete) without any failure on serial port after that.

Just did comparison the logs between working one and failed one, the last line seems give me some indication, we can see: " Stopping anac(h)ronistic cron " is printed on failed one. The anacron appears to crash for some reason.

6

To provide context, what are a couple of lines of matching working code just prior to the “anac(h)ronistic cron” line?

This would take significant time, but you might consider cloning the rootfs on the failed unit, mounting the clone on loopback, and then manually checking the sha1sum of each file in <loopback_mount_point>/etc/nv_tegra_release. Possibly cron has accessed something using invalid hardware access (admittedly, this is reaching and may not be worth the time). Having a clone of the partition though would make the file system accessible from your host even with the failed Jetson shut down.

7

Hi Linuxdev,

It turns out on our customized TK1 board, the JTAG_RST_N signal is somehow pulled-up, after we remove the pull-up, the failure symptom is gone. We don’t why pulling high JTAG_RST_N signal is causing problem, however Jetson design does have it pull-down.

Anyway thanks for all your help.

8

JTAG_TRST_N should be pulled down for normal boot, There is a 10kohm pull down resistor R8B1 on the schematic.
JTAG_TRST_N pull high will put TK1 chip into JTAG scandump mode.
Pull high or floating JTAG_RST will have unexpected issue.