microSD Card on a custom carrier board causes boot errors.

Robotics & Edge Computing Jetson & Embedded Systems Jetson TX1
1

Hello,

We’ve a custom carrier board for the TX. It uses a microSD card slot. We have connected the microSD card exactly as described in the JetsonTX1_TX2_Developer_Kit_Carrier_Board_Specification for the Jetson board.

1440×692 95.9 KB

However the microSD card doesn’t have a C_DETECT Pin. So we tied the SDCARD_CS# pin to ground.

I understand based on https://devtalk.nvidia.com/default/topic/1004742/?comment=5130398 that this will cause us to loose the HotPlug feature. We’re OK with that. We can insert the SDCard before boot and keep it there all the time.

I look at drivers/mmc/host/sdhci-tegra.c in the kernel sources and it seems that if the CD-GPIO is detected to be low, then it’s assumed that the SDCard is present regardless of the interrupt. So it should work.

Currently what we’re seeing is that a freshly flashed (from Jetpack 3.1) Jetson TX1 fails to boot up on this carrier board if a freshly formatted (FAT) SDCard is inserted. If SDCard isn’t inserted it boots up fine. Here is the console error.

[    5.115463] mmc1 tuning done saved tap delay=169
[    5.120074] mmc1: hw tuning done ...
[    5.123645] mmc1: tuning_window[0]: 0xffffffff
[    5.128081] mmc1: tuning_window[1]: 0xffffc7ff
[    5.132517] mmc1: tuning_window[2]: 0xffffffff
[    5.136954] mmc1: tuning_window[3]: 0x3fffffff
[    5.141389] mmc1: tuning_window[4]: 0xfffffffe
[    5.145825] mmc1: tuning_window[5]: 0xffffffff
[    5.150260] mmc1: tuning_window[6]: 0xffe3ffff
[    5.154695] mmc1: tuning_window[7]: 0x7fffffff
[    5.159130] sdhci: Tap value: 169 | Trim value: 2
[    5.163825] sdhci: SDMMC_VENDOR_INTR_STATUS[0x108]: 0x40000
[    5.169409] mmc1: new ultra high speed SDR50 SDHC card at address aaaa
[    5.176221] mmcblk0: mmc1:aaaa SL08G 7.40 GiB
[    5.196742]  mmcblk0: p1
[    5.201026] [OV5693]: probing v4l2 sensor.
[    5.205256] ov5693 6-0036: camera_common_regulator_get vana ERR: fffffffffffffdfb
[    5.212835] ov5693 6-0036: camera_common_regulator_get vif ERR: fffffffffffffdfb
[    5.254429] mmc0: queuing unknown CIS tuple 0x80 (5 bytes)
[    5.254509] FAT-fs (mmcblk0p1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
[    5.254521] VFS: Mounted root (vfat filesystem) on device 179:1.
[    5.254580] devtmpfs: error mounting -2
[    5.254909] Freeing unused kernel memory: 1208K (ffffffc001175000 - ffffffc0012a3000)
[    5.254929] Freeing alternatives memory: 76K (ffffffc0012a3000 - ffffffc0012b6000)
[    5.255119] Kernel panic - not syncing: No working init found.  Try passing init= option to kernel. See Linux Documentation/init.txt for guidance.
[    5.255123] CPU: 3 PID: 1 Comm: swapper/0 Tainted: G        W       4.4.38-tegra #1
[    5.255124] Hardware name: jetson_tx1 (DT)
[    5.255125] Call trace:
[    5.255132] [<ffffffc0000893ac>] dump_backtrace+0x0/0xe8
[    5.255136] [<ffffffc000089858>] show_stack+0x14/0x1c
[    5.255140] [<ffffffc00036cde0>] __dump_stack+0x20/0x28
[    5.255142] [<ffffffc00036ce84>] dump_stack+0x9c/0xd8
[    5.255145] [<ffffffc0000a4920>] panic+0xfc/0x274
[    5.255148] [<ffffffc000b3049c>] kernel_init+0xdc/0xe8
[    5.255150] [<ffffffc000084790>] ret_from_fork+0x10/0x40
[    5.255155] CPU0: stopping
[    5.255158] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G        W       4.4.38-tegra #1
[    5.255159] Hardware name: jetson_tx1 (DT)
[    5.255160] Call trace:
[    5.255164] [<ffffffc0000893ac>] dump_backtrace+0x0/0xe8
[    5.255167] [<ffffffc000089858>] show_stack+0x14/0x1c
[    5.255170] [<ffffffc00036cde0>] __dump_stack+0x20/0x28
[    5.255172] [<ffffffc00036ce84>] dump_stack+0x9c/0xd8
[    5.255175] [<ffffffc00008eeb0>] ipi_cpu_stop+0x4c/0x70
[    5.255177] [<ffffffc00008f63c>] handle_IPI+0x168/0x2b8
[    5.255179] [<ffffffc000080bd4>] gic_handle_irq+0xa0/0xb8
[    5.255181] [<ffffffc000083f44>] el1_irq+0x84/0x100
[    5.255184] [<ffffffc00080d704>] cpuidle_enter+0x18/0x20
[    5.255187] [<ffffffc0000e6758>] call_cpuidle+0x4c/0x58
[    5.255189] [<ffffffc0000e6830>] cpuidle_idle_call+0xcc/0x120
[    5.255192] [<ffffffc0000e6ae0>] cpu_idle_loop+0x25c/0x27c
[    5.255194] [<ffffffc0000e6b10>] convert_prio+0x0/0x3c
[    5.255196] [<ffffffc000b303b4>] rest_init+0x8c/0x98
[    5.255200] [<ffffffc001175a7c>] start_kernel+0x2dc/0x2e4
[    5.255201] [<0000000080b36000>] 0x80b36000
[    5.255204] CPU2: stopping
[    5.255207] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G        W       4.4.38-tegra #1
[    5.255208] Hardware name: jetson_tx1 (DT)
[    5.255209] Call trace:
[    5.255213] [<ffffffc0000893ac>] dump_backtrace+0x0/0xe8
[    5.255216] [<ffffffc000089858>] show_stack+0x14/0x1c
[    5.255218] [<ffffffc00036cde0>] __dump_stack+0x20/0x28
[    5.255220] [<ffffffc00036ce84>] dump_stack+0x9c/0xd8
[    5.255222] [<ffffffc00008eeb0>] ipi_cpu_stop+0x4c/0x70
[    5.255225] [<ffffffc00008f63c>] handle_IPI+0x168/0x2b8
[    5.255226] [<ffffffc000080bd4>] gic_handle_irq+0xa0/0xb8
[    5.255228] [<ffffffc000083f44>] el1_irq+0x84/0x100
[    5.255230] [<ffffffc00080d704>] cpuidle_enter+0x18/0x20
[    5.255233] [<ffffffc0000e6758>] call_cpuidle+0x4c/0x58
[    5.255236] [<ffffffc0000e6830>] cpuidle_idle_call+0xcc/0x120
[    5.255238] [<ffffffc0000e6ae0>] cpu_idle_loop+0x25c/0x27c
[    5.255240] [<ffffffc0000e6b10>] convert_prio+0x0/0x3c
[    5.255243] [<ffffffc00008f1fc>] secondary_start_kernel+0x16c/0x178
[    5.255244] [<0000000080080e8c>] 0x80080e8c
[    5.259923] CPU1: stopping
[    5.259927] CPU: 1 PID: 163 Comm: kworker/u8:4 Tainted: G        W       4.4.38-tegra #1
[    5.259928] Hardware name: jetson_tx1 (DT)
[    5.259931] Workqueue: kmmcd mmc_rescan
[    5.259932] Call trace:
[    5.259936] [<ffffffc0000893ac>] dump_backtrace+0x0/0xe8
[    5.259938] [<ffffffc000089858>] show_stack+0x14/0x1c
[    5.259941] [<ffffffc00036cde0>] __dump_stack+0x20/0x28
[    5.259943] [<ffffffc00036ce84>] dump_stack+0x9c/0xd8
[    5.259945] [<ffffffc00008eeb0>] ipi_cpu_stop+0x4c/0x70
[    5.259947] [<ffffffc00008f63c>] handle_IPI+0x168/0x2b8
[    5.259949] [<ffffffc000080bd4>] gic_handle_irq+0xa0/0xb8
[    5.259951] [<ffffffc000083f44>] el1_irq+0x84/0x100
[    5.259954] [<ffffffc0000a9c40>] irq_exit+0x70/0xd0
[    5.259957] [<ffffffc0000f7018>] __handle_domain_irq+0x88/0xac
[    5.259958] [<ffffffc000080ba0>] gic_handle_irq+0x6c/0xb8
[    5.259960] [<ffffffc000083f44>] el1_irq+0x84/0x100
[    5.259962] [<ffffffc0000f5bcc>] vprintk_emit+0x450/0x4a8
[    5.259964] [<ffffffc0000f5e0c>] vprintk_default+0x34/0x3c
[    5.259965] [<ffffffc0000f47e4>] printk+0xb8/0xc4
[    5.259969] [<ffffffc00081e858>] sdio_read_cis+0x1fc/0x2ac
[    5.259972] [<ffffffc00081ea44>] sdio_read_common_cis+0x10/0x18
[    5.259974] [<ffffffc00081cc2c>] mmc_sdio_init_card+0x2f4/0x464
[    5.259977] [<ffffffc00081d4f4>] mmc_attach_sdio+0xb0/0x2b8
[    5.259980] [<ffffffc0008139d4>] mmc_rescan_try_freq+0x48/0x7c
[    5.259981] [<ffffffc0008144c8>] mmc_rescan+0x1e0/0x254
[    5.259984] [<ffffffc0000bd9cc>] process_one_work+0x25c/0x438
[    5.259987] [<ffffffc0000be038>] worker_thread+0x168/0x290
[    5.259989] [<ffffffc0000c3b2c>] kthread+0xfc/0x104
[    5.259992] [<ffffffc000084790>] ret_from_fork+0x10/0x40
[    5.712503] Rebooting in 30 seconds.......

The mmcblk0: mmc1:aaaa SL08G 7.40 GiB is the 8GB SDCard I had inserted. So it seems to be recognised at the sdhci driver level.

Also the same SDCard works fine on a Jetson board.

Any ideas what the issue could be?

2

I am curious if your VFAT kernel driver is loaded as a module? Without hot plug it may mean that the card is probed or used early on before the driver is available if the driver is module format. Calling VFAT functions before they exist would be “a bad idea”.

3

Good point. This is the standard kernel that comes with Jetpack. Do you know if it already has VFAT driver compiled in?

I can try to flash the card as ext on Monday and see if it helps.

4

I see on R28.1 this is already integrated (not module format). Is this R28.1? Some previous releases also had an initrd involved which could change things. I see SD card support is also integrated, so if you haven’t built a custom kernel this would not be an issue. You could run this to verify:

zcat /proc/config.gz | grep VFAT

It would be quite an accomplishment to create a custom carrier board which uses the same device tree even though the kernel config itself doesn’t change. Does this same SD card slot fail if an unformatted SD card is inserted? Or one formated ext2 or ext3?

Is the root partition on eMMC as mmc0? Is there any other SD card slot? Can you use an external USB SD card reader and test if an external version causes panic?

5

We did change the device tree in a few places. But the original design goal was to follow the Jetson board as much as possible so as to reduce software bringup efforts.

I will check with other file system formatted SDCards and USB SD Card reader on Monday when I’ve access to the board again.

As for the naming of the partitions. I see some interesting behaviour. The eMMC (SDMMC4) is on the TX1 module and hence is completely untouched. The only SDHCI peripheral we can work with on the carrier board is SDCARD (SDMMC1) which is sdhci-tegra.0 peripheral.

For this controller, I see the following bit in the console log.

[    4.052596] sdhci-tegra sdhci-tegra.0: Got CD GPIO
[    4.052630] sdhci-tegra sdhci-tegra.0: Got WP GPIO
[    4.052666] sdhci-tegra sdhci-tegra.0: Parent select= pll_p rate=408000000
[    4.052688] sdhci-tegra sdhci-tegra.0: wakeup init done, cd_irq: 353
[    4.052699] sdhci-tegra sdhci-tegra.0: Client registration for eMC Successful

[    4.102193] mmc1: SDHCI controller on sdhci-tegra.0 [sdhci-tegra.0] using ADMA 64-bit with 64 bit addr
[    4.103520] [OV5693]: probing v4l2 sensor.

mmc0 on the other hand seems to be assigned to sdhci-tegra.1 which is supposed to be internally connected to WiFi. :S

[    4.913980] mmc0: SDHCI controller on sdhci-tegra.1 [sdhci-tegra.1] using ADMA 64-bit with 64 bit addr

1696×412 89 KB

Or am not right in assuming that sdhci-tegra.X refers to the interface number X in the SD/eMMC peripherals?

And to add to my confusion, U-Boot has a whole different set of naming conventions.

U-Boot 2016.07-g0ce7ca2 (Jul 20 2017 - 00:37:03 -0700)

TEGRA210
Model: NVIDIA P2371-2180
Board: NVIDIA P2371-2180
DRAM:  4 GiB
MMC:   Tegra SD/MMC: 0, Tegra SD/MMC: 1
*** Warning - bad CRC, using default environment

In:    serial
Out:   serial
Err:   serial
Net:   No ethernet found.
Hit any key to stop autoboot:  0
switch to partitions #0, OK
mmc1 is current device
Scanning mmc 1:1...
switch to partitions #0, OK
mmc0(part 0) is current device
Scanning mmc 0:1...
Found /boot/extlinux/extlinux.conf
Retrieving file: /boot/extlinux/extlinux.conf
285 bytes read in 74 ms (2.9 KiB/s)
p2371-2180 eMMC boot options
1:.primary kernel
Enter choice: 1:.primary kernel
Retrieving file: /boot/initrd
0 bytes read in 59 ms (0 Bytes/s)
Retrieving file: /boot/Image
20984368 bytes read in 527 ms (38 MiB/s)
append: root=/dev/mmcblk0p1 rw rootwait console=ttyS0,115200n8 console=tty0 OS=l4t fbcon=map:0 net.ifnames=0 androidboot.modem=none androidboot.serialno=03218160291930000201 androidboot.security=non-secure tegraid=21.1.2.0.0 ddr_die=2048M@2048M ddr_die=2048M@4096M section=256M memtype=0 vpr_resize usb_port_owner_info=0 lane_owner_info=0 emc_max_dvfs=0 touch_id=0@63 video=tegrafb no_console_suspend=1 debug_uartport=lsport,0 earlyprintk=uart8250-32bit,0x70006000 maxcpus=4 usbcore.old_scheme_first=1 lp0_vec=0x1000@0xff2bf000 nvdumper_reserved=0xff23f000 core_edp_mv=1125 core_edp_ma=4000 gpt android.kerneltype=normal androidboot.touch_vendor_id=0 androidboot.touch_panel_id=63 androidboot.touch_feature=0 androidboot.bootloader=00.00.2014.50-t210-fadd1be5 androidboot.verifiedbootstate=orange root=/dev/mmcblk0p1 rw rootwait
Retrieving file: /boot/tegra210-jetson-tx1-p2597-2180-a01-devkit.dtb
6

I couldn’t say for certain if one naming convention or another is correct in one place or another…I believe you are correct, but an invalid device tree (relative to what some other device uses for setup) could cause the problems you are seeing.

7

Hi notthetup，

Please refer to kernel/kernel-4.4/drivers/mmc/host/sdhci-tegra.c to see if any help.