I2C and MMC2 errors

Autonomous Machines Jetson & Embedded Systems Jetson TK1
1

Hello,

We have developed a new board based on the Jetson design. Our design is almost the same as the jetson board except for the memories which are a little bit different. The problem we are having is that when booting two errors specifically arise a i2c error and a fatal error at a mmc2 driver. THe mmc2 error says it is a timeout. Please if you have any clue how I can debug/fix this. The linux is the stock l4t 21.4. I have attached the log

[    1.740347] tegradc tegradc.0: nominal-pclk:154679000 parent:463500000 div:3.0 pclk:154500000 153132210~168600110
[    1.746212] gpio wake4 for gpio=111
[    2.745498] tegra-i2c tegra12-i2c.3: --- register dump for debugging ----
[    2.745529] tegra-i2c tegra12-i2c.3: I2C_CNFG - 0x2c00
[    2.745554] tegra-i2c tegra12-i2c.3: I2C_PACKET_TRANSFER_STATUS - 0xff0001
[    2.745580] tegra-i2c tegra12-i2c.3: I2C_FIFO_CONTROL - 0xe0
[    2.745604] tegra-i2c tegra12-i2c.3: I2C_FIFO_STATUS - 0x800040
[    2.745628] tegra-i2c tegra12-i2c.3: I2C_INT_MASK - 0xed
[    2.745651] tegra-i2c tegra12-i2c.3: I2C_INT_STATUS - 0x0
[    2.745673] tegra-i2c tegra12-i2c.3: msg->len - 1
[    2.745695] tegra-i2c tegra12-i2c.3: is_msg_write - 1
[    2.745717] tegra-i2c tegra12-i2c.3: next_msg->len - 128
[    2.745739] tegra-i2c tegra12-i2c.3: is_next_msg_write - 0
[    2.745762] tegra-i2c tegra12-i2c.3: buf_remaining - 128
[    2.745789] tegra-i2c tegra12-i2c.3: i2c transfer timed out, addr 0x0050, data 0x00
[    2.746844] tegradc tegradc.1: probed
[    2.751264] Console: switching to colour frame buffer device 80x30
[    2.755567] tegradc tegradc.1: fb registered
[    2.756990] hdmi_state_machine_worker (tid edb0a580): state 7 (Takeover from bootloader), hpd 0, pending_hpd_evt 1

[   10.232502] EXT4-fs (mmcblk0p1): mounted filesystem with ordered data mode. Opts: (null)
[   10.244107] VFS: Mounted root (ext4 filesystem) on device 179:1.
[   10.254937] devtmpfs: mounted
[   10.260048] Freeing unused kernel memory: 508K (c0b4c000 - c0bcb000)
[   10.506819] init: plymouth-upstart-bridge main process (120) terminated with status 1
[   10.518494] init: plymouth-upstart-bridge main process ended, respawning
[   10.570178] init: plymouth-upstart-bridge main process (130) terminated with status 1
[   10.581982] init: plymouth-upstart-bridge main process ended, respawning
[   10.595290] init: ureadahead main process (123) terminated with status 5
[   10.630341] init: plymouth-upstart-bridge main process (134) terminated with status 1
[   10.642380] init: plymouth-upstart-bridge main process ended, respawning
 * Starting Mount filesystems on boot[ OK ]

 * Starting Signal sysvinit that the rootfs is mounted[ OK ]

 * Starting Clean /tmp directory[ OK ]

 * Starting Populate and link to /run filesystem[ OK ]

 * Stopping Populate and link to /run filesystem[ OK ]

 * Stopping Clean /tmp directory[ OK ]

 * Stopping Track if upstart is running in a container[ OK ]

 * Starting Initialize or finalize resolvconf[ OK ]

 * Starting Signal sysvinit that virtual filesystems are mounted[ OK ]

 * Starting Signal sysvinit that virtual filesystems are mounted[ OK ]

 * Starting Signal sysvinit that local filesystems are mounted[ OK ]

 * Starting Bridge udev events into upstart[ OK ]

 * Starting Signal sysvinit that remote filesystems are mounted[ OK ]

 * Stopping Mount filesystems on boot[ OK ]

 * Starting NFSv4 id <-> name mapper[ OK ]

 * Starting flush early job output to logs[ OK ]

 * Starting D-Bus system message bus[ OK ]

 * Starting device node and kernel event manager[ OK ]

 * Stopping flush early job output to logs[ OK ]

 * Starting NVIDIA specific init script[ OK ]

 * Starting load modules from /etc/modules[ OK ]

 * Starting cold plug devices[ OK ]

 * Starting log initial device creation[ OK ]

 * Starting SystemD login management service[ OK ]

 * Stopping NVIDIA specific init script[ OK ]

 * Stopping load modules from /etc/modules[ OK ]

 * Starting bluetooth daemon[ OK ]

 * Stopping rpcsec_gss daemon[ OK ]

 * Starting system logging daemon[ OK ]

 * Stopping NVIDIA specific first-boot script[ OK ]

 * Starting configure network device security[ OK ]

 * Starting userspace bootsplash[ OK ]

 * Starting configure network device[ OK ]

 * Starting mDNS/DNS-SD daemon[ OK ]

 * Stopping userspace bootsplash[ OK ]

 * Starting Send an event to indicate plymouth is up[ OK ]

 * Starting Reload cups, upon starting avahi-daemon to make sure remote queues are populated[ OK ]

 * Stopping Send an event to indicate plymouth is up[ OK ]

 * Stopping Reload cups, upon starting avahi-daemon to make sure remote queues are populated[ OK ]

 * Starting configure network device security[ OK ]

 * Starting configure network device security[ OK ]

 * Starting configure network device security[ OK ]

 * Starting configure network device security[ OK ]

 * Starting configure network device security[ OK ]

 * Starting CUPS printing spooler/server[ OK ]

 * Starting configure network device[ OK ]

 * Starting configure network device[ OK ]

 * Starting configure network device[ OK ]

 * Starting configure network device[ OK ]

 * Stopping cold plug devices[ OK ]

 * Stopping log initial device creation[ OK ]

 * Starting configure network device security[ OK ]

 * Starting enable remaining boot-time encrypted block devices[ OK ]

 * Starting Mount network filesystems[ OK ]

 * Starting Upstart job to start rpcbind on boot only[ OK ]

 * Starting Failsafe Boot Delay[ OK ]

 * Stopping Upstart job to start rpcbind on boot only[ OK ]

 * Starting cups-browsed - Bonjour remote printer browsing daemon[ OK ]

 * Starting configure network device[ OK ]

 * Stopping Mount network filesystems[ OK ]

 * Starting configure virtual network devices[ OK ]

 * Starting RPC portmapper replacement[ OK ]

 * Starting NSM status monitor[ OK ]

 * Starting Bridge file events into upstart[ OK ]

 * Starting Bridge socket events into upstart[ OK ]

[   18.617853] mmc2: Timeout waiting for hardware interrupt.
[   18.641102] sdhci: =========== REGISTER DUMP (mmc2)===========
[   18.664557] sdhci: Sys addr: 0x00000000 | Version:  0x00000303
[   18.687475] sdhci: Blk size: 0x00000000 | Blk cnt:  0x00000000
[   18.710040] sdhci: Argument: 0x00000c00 | Trn mode: 0x00000000
[   18.733047] sdhci: Present:  0x01fb00f1 | Host ctl: 0x00000001
[   18.756583] sdhci: Power:    0x0000000f | Blk gap:  0x00000000
[   18.780661] sdhci: Wake-up:  0x00000000 | Clock:    0x00000405
[   18.805185] sdhci: Timeout:  0x00000000 | Int stat: 0x00000000
[   18.829612] sdhci: Int enab: 0x00ff0003 | Sig enab: 0x00fc0003
[   18.854132] sdhci: AC12 err: 0x00000000 | Slot int: 0x00000000
[   18.879152] sdhci: Caps:     0x376fd080 | Caps_1:   0x10002f73
[   18.904814] sdhci: Cmd:      0x0000341a | Max curr: 0x00000000
[   18.931072] sdhci: Host ctl2: 0x00003000
[   18.955392] sdhci: ADMA Err: 0x00000000 | ADMA Ptr: 0x00000000
[   18.982265] sdhci: ===========================================
2

One thing I just noticed from the first log…this particular i2c error seems to be from the query of the DDC/EDID channel on the monitor over HDMI. Does the cabling have any kind of adapter in it, is it purely HDMI? Have you tried this monitor and cable on a Jetson? If you run this monitor on a Jetson, add packages read-edid and edid-decode, then see what the output is of:

get-edid | parse-edid
get-edid | edid-decode

I’m wondering if perhaps simpler and more subtle is going on, rather than a hardware issue (e.g., using incompatible X ABI). If you monitor the serial console and boot with no monitor attached, does the mmc2 error change in any way?

3

Well as this is a custom board we have no hdmi header. Is there a way to disable this like in the boot parameters ?

4

What kind of video does it have…is it completely headless?

I think the DDC/EDID query is part of the X11 install. If you have no graphics at all, perhaps removing everything related would stop the query. A short description of this query might help.

“In the early days” (this makes me feel like Fred Flintstone) we only had VGA. VGA required setting X11 to use only standard modes via modelines and manual edits until information in a manual somewhere became available for other modes, in which case the same files were appended to (or copied from a database of known monitors and naming the specific monitor). Everything since that time has added an extra wire(s) to allow the video card to query the monitor and ask it what it is…this is your query from the first log.

The standard L4T (independent of Jetson hardware) depends on this information and does not make arrangements for missing or bad EDID, so old 15 pin VGA connectors will also fail. Because this is only for video graphics setup, anyone not using graphical mode should be able to just remove X11 or set it up with one of the variants which expects the monitor to not be there (EDIT: One example being VNC remote X11).

It still isn’t certain that this is the issue, but if you have a Jetson and can experiment with cutting out X11 packages and disabling graphical interface entirely, there is no reason you couldn’t cut down your custom system packages the same way on the custom board and get success (at least in terms of the first logged failure).

5

Oh yes it is completely headless. All the output you have seen was captured via the serial console.

6

Was there any attempt to remove graphics software, especially X11? Was your rootfs from the default sample rootfs?