Cannot connect SD card to Jetson TX2 NX, even with working card detect

jlucier · March 10, 2022, 6:31pm

Hey,

I have a custom carrier for the TX2 NX and I cannot for the life of me get the SD to show up in the file system. I believe that I have the pinmux configuration working correctly to enable the card detect but I have no idea what is wrong with the sdhci@3440000 section of the dts.

Originally I got the sd card to show up in mmcblk1 by setting gpio7 to output Drive 1 in the pinmux. gpio07 is connected to a load switch that turns the power to the card on. When booting with a card in the slot, it would show up in the filesystem. However, this also seemed to cause a intermittent kernel panic and boot loop half of the time.

To solve this problem I changed gpio08 to input pull up and added that pin to the cd-gpois in sdhci@3440000. Now what happens is that when I boot (or hot plug the card in) I get 3 instances of the error “mmc1: error -110 whilst initialising SD card” in the dmesg. I can see that the card detect pin is working in /sys/kernel/debug/gpio. But for the life of me I can tell what’s wrong with my dts. I have tried almost 30 different configurations of dts based on these forum posts.

Here’s relevant dmesg:

[    0.000000] Kernel command line: console=ttyS0,115200 root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 console=ttyS0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 isolcpus=1-2  video=tegrafb earlycon=uart8250,mmio32,0x3100000 nvdumper_reserved=0x1772e0000 gpt rootfs.slot_suffix= tegra_fbmem=0x800000@0x96085000 lut_mem=0x2008@0x96081000 usbcore.old_scheme_first=1 tegraid=18.1.2.0.0 maxcpus=6 no_console_suspend boot.slot_suffix= boot.ratchetvalues=0.2031647.1 vpr_resize bl_prof_dataptr=0x10000@0x175840000 sdhci_tegra.en_boot_part_access=1 quiet root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 console=ttyS0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 isolcpus=1-2 
[    0.401634] iommu: Adding device 3460000.sdhci to group 0
[    0.402061] iommu: Adding device 3440000.sdhci to group 1
[    0.501206] vddio-sdmmc1: 1800 <--> 3300 mV at 3300 mV 
[    0.964609] sd: No Scsi addr parsed to reserve index
[    1.220741] PPP BSD Compression module registered
[    1.450397] sdhci: Secure Digital Host Controller Interface driver
[    1.450399] sdhci: Copyright(c) Pierre Ossman
[    1.450401] sdhci-pltfm: SDHCI platform and OF driver helper
[    1.451381] sdhci-tegra 3460000.sdhci: Client registration for eMC Successful
[    1.453349] sdhci-tegra 3440000.sdhci: Got CD GPIO
[    1.456142] sdhci-tegra 3440000.sdhci: Client registration for eMC Successful
[    1.458986] sdhci-tegra 3440000.sdhci: wakeup init done, cdirq 286
[    1.459096] sdhci-tegra 3440000.sdhci: Failed getting OCR mask: 0
[    1.496215] mmc0: SDHCI controller on 3460000.sdhci [3460000.sdhci] using ADMA 64-bit with 64 bit addr
[    1.508300] mmc1: SDHCI controller on 3440000.sdhci [3440000.sdhci] using ADMA 64-bit with 64 bit addr
[    1.537904] mmc0: mmc_decode_ext_csd: CMDQ supported: depth: 31, cmdq_support: 1
[    1.551842] mmc0: periodic cache flush enabled
[    1.551848] mmc0: new HS400 Enhanced strobe MMC card at address 0001
[    1.552132] mmcblk0: mmc0:0001 DG4016 14.7 GiB 
[    1.556279] mmcblk0boot0: mmc0:0001 DG4016 partition 1 4.00 MiB
[    1.561309] mmcblk0boot1: mmc0:0001 DG4016 partition 2 4.00 MiB
[    1.562777] mmc1: error -110 whilst initialising SD card
[    1.568268] mmcblk0rpmb: mmc0:0001 DG4016 partition 3 4.00 MiB
[    1.575566]  mmcblk0: p1 p2 p3 p4 p5 p6 p7 p8 p9 p10 p11 p12 p13 p14 p15 p16 p17 p18 p19 p20 p21 p22 p23 p24 p25 p26 p27 p28 p29 p30 p31 p32 p33
[    1.675442] mmc1: error -110 whilst initialising SD card
[    1.788133] mmc1: error -110 whilst initialising SD card
[    1.910822] mmc1: error -110 whilst initialising SD card
[   11.361574] mmcblk mmc0:0001: Card claimed for testing.
[   11.406802] Root device found: mmcblk0p1
[   11.407876] Found dev node: /dev/mmcblk0p1
[   11.449405] EXT4-fs (mmcblk0p1): 1 orphan inode deleted
[   11.449409] EXT4-fs (mmcblk0p1): recovery complete
[   11.452486] EXT4-fs (mmcblk0p1): mounted filesystem with ordered data mode. Opts: (null)
[   11.454336] Rootfs mounted over mmcblk0p1
[   11.582603] cgroup: cgroup2: unknown option "nsdelegate"
[   11.695580] systemd[1]: File /lib/systemd/system/systemd-journald.service:36 configures an IP firewall (IPAddressDeny=any), but the local system does not support BPF/cgroup based firewalling.
[   11.869206] EXT4-fs (mmcblk0p1): re-mounted. Opts: (null)
[   13.674065] cdc_acm: USB Abstract Control Model driver for USB modems and ISDN adapters
[   42.065072] vddio-sdmmc1: disabling

Here is relevant dts:

	sdhci@3440000 {
		compatible = "nvidia,tegra186-sdhci";
		reg = <0x0 0x3440000 0x0 0x210>;
		interrupts = <0x0 0x40 0x4>;
		max-clk-limit = <0xc28cb00>;
		ddr-clk-limit = <0x2dc6c00>;
		tap-delay = <0xb>;
		trim-delay = <0x5>;
		nvidia,ddr-tap-delay = <0xb>;
		ddr-trim-delay = <0x5>;
		bus-width = <0x4>;
		ignore-pm-notify;
		mmc-ocr-mask = <3>;
		keep-power-in-suspend;
		cap-mmc-highspeed;
		cap-sd-highspeed;
		pwrdet-support;
		pinctrl-names = "sdmmc_e_33v_enable", "sdmmc_e_33v_disable";
		pinctrl-0 = <0x14>;
		pinctrl-1 = <0x15>;
		compad-vref-3v3 = <0x1>;
		compad-vref-1v8 = <0x2>;
		nvidia,min-tap-delay = <0x54>;
		nvidia,max-tap-delay = <0x88>;
		pll_source = "pll_p";
		resets = <0x10 0x23>;
		reset-names = "sdhci";
		clocks = <0x10 0x4c 0x10 0x10d 0x10 0x80>;
		clock-names = "sdmmc", "pll_p", "sdmmc_legacy_tm";
		iommus = <0x11 0x19>;
		nvidia,en-periodic-calib;
		status = "okay";
		vqmmc-supply = <0x16>;
		vmmc-supply = <0x25>;
		cd-gpios = <0x21 0x9c 0x0>;
		nvidia,vmmc-always-on;
		nvidia,sd-device;
		nvidia,cd-wakeup-capable;
		disable-wp;
		no-sdio;
		no-mmc;
		sd-uhs-sdr104;
		sd-uhs-sdr50;
		sd-uhs-sdr25;
		sd-uhs-sdr12;
		mmc-ddr-1_8v;
		mmc-hs200-1_8v;
		linux,phandle = <0xc7>;
		phandle = <0xc7>;

Here is /sys/kernel/debug/mmc1/ios:

clock:		0 Hz
vdd:		0 (invalid)
bus mode:	2 (push-pull)
chip select:	0 (don't care)
power mode:	0 (off)
bus width:	0 (1 bits)
timing spec:	0 (legacy)
signal voltage:	1 (1.80 V)
driver type:	0 (driver type B)

Full dts
sdhci_3440000_test24.dts (239.3 KB)
Full dmesg
dmesg_dump.txt (91.7 KB)

For the record I have tried many configurations with vmmc-supply pointing at the spmic sd3 regulator. Nothing happened in dmesg following a hot plug suggesting that it is not attempting to connect to the card. When I changed vmmc-supply to vdd-ac-bat I now get the “mmc1: error -110 whilst initialising SD card” four times when ever the card is plugged in.

Also, the OCR error in the dmesg goes away if mmc-ocr-mask is set to 3 and iommus is removed from the sdhci@3440000.

Can some one help please!??

jlucier · March 10, 2022, 6:36pm

Also here are some pictures of the SD card circuit!

And here are my pinmux dtsi files:
tegra18x-2_input_pullup-gpio-default.dtsi (1.7 KB)
tegra18x-2_input_pullup-padvoltage-default.dtsi (1.6 KB)
tegra18x-2_input_pullup-pinmux.dtsi (51.5 KB)

WayneWWW · March 11, 2022, 2:26am

Please move your vmmc-supply back to sd3 and share the dmesg.

The regulator voltage needs to meet the hardware spec. You cannot just give a random one like vdd-ac-bat which is 5v.

You need to assign one with 3.3v. Maybe write a new one with similar one as vdd-ac-bat but chcange the volt to 3.3v.

jlucier · March 11, 2022, 10:56pm

@WayneWWW Thanks for the feedback.

I tried the change as described and did not get different results. I’ve attached the DTS file I used, as well as the dmesg output I observe.
dmesg.log (92.6 KB)
sdhci_3440000_test25.dts (239.3 KB)

One thing we found interesting that seems to differ with other posts is the contents of /sys/kernel/debug/mmc1/ios (specifically the signal voltage being 1.8V):

nvidia@nvidia-desktop:~$ sudo cat /sys/kernel/debug/mmc1/ios
[sudo] password for nvidia: 
clock:		0 Hz
vdd:		0 (invalid)
bus mode:	2 (push-pull)
chip select:	0 (don't care)
power mode:	0 (off)
bus width:	0 (1 bits)
timing spec:	0 (legacy)
signal voltage:	1 (1.80 V)
driver type:	0 (driver type B)

Let me know what you think, thanks!

WayneWWW · March 12, 2022, 5:52am

Try to remove vqmmc-supply in your DT. Only leave vmmc.

jlucier · March 14, 2022, 10:09pm

I tried removing vqmmc-supply from the sdhci@3440000 section of my dts, and that didn’t change much. Normally when I boot with an SD card or hotplug one into our PCB, I will see an LED flicker (indicating the SD card has 3.3V power). With vqmmc-supply removed, I don’t ever see the LED light up at all. Still no matter what, the SD card does not show in lsblk.

Additionally, I see the following line many times repeatedly during boot (over UART) when an SD card is plugged in:

[    1.470729] mmc1: CMD CRC or end bit error, int mask 0xc0000

Should I be editing other sections as well? For instance the other sdhci@xxxxx sections? For reference this is our SD card and we’ve tried multiple physical cards which all work in other computers.

Let me know what you think, thanks!

WayneWWW · March 15, 2022, 3:33am

You can disable unnecessary sdhci node in your DT. Just do not disable the one that is running the emmc should be fine.

Again, if you remove anything, share us the log. No matter that log is changed or not.

Also, this kind of issue is not something new. If the same configuration from other people can work on their TX2 NX but not your board, maybe need to review the hardware schematic.

jlucier · March 15, 2022, 1:48pm

Sorry, here’s the dmesg log for your inspection. Nothing seems to happen at all relating to the SD with vqmmc-supply removed, not even when I hotplug.

no_vqmmc_dmesg.log (69.6 KB)

I understand that this issue is not new, and I understand your feedback. That being said, I don’t believe our designer has laid out the PCB incorrectly.

This process is very challenging for us, since we are not Firmware Engineers. Hunting down countless different forum posts and trying random DT edits from them is not an effective process for us to fix issues or determine why something is not working.

It would make our lives a lot easier if you could explain a few things for us:

Which DT section is controlling the SD card? Is it sdhci@3440000? Which one is for the mmc?
What is the difference between vmmc-supply and vqmmc-supply?
Is there any documentation on the device tree itself and how you can do basic things like setting up support for an SD card?
Do you have any additional resources you can point us to, or do you have any more suggestions for things to try?

If these questions seem dumb, that’s because we are “dumb”! We need things explained as though we know fairly little about what we’re doing, because we really don’t!

Thank you for your continued support.

WayneWWW · March 15, 2022, 3:46pm

Hi,

sdhci@34x0000 just follows the hardware pin. Since you should use sdmmc3 pin the on module.

sdhci@3400000 → sdmmc1
sdhci@3420000 → sdmmc2
sdhci@3440000 → sdmmc3

From upstream kernel document

vmmc-supply : phandle to the regulator device tree node, mentioned as the VCC/VDD supply in the eMMC/SD specs.
vqmmc-supply : phandle to the regulator device tree node, mentioned as the VCCQ/VDD_IO supply in the eMMC/SD specs. And vqmmc-supply is optional.

Sorry, there is no document. Only the driver code. And kernel driver documentation folder shall have some info regarding each property meaning. I don’t remember the actual location. You can grep “sdhci-tegra” or something like “nvidia,vmmc-always-on” and it should tell the path.
Basically, no matter jetson nano/ jetson NX or jetson TX2-NX, they are all sharing the same case. vmmc-supply and cd-gpios needs to match the board design. If your vmmc-supply is a always-on one, which does not control by any GPIO, then you also need to add “nvidia,vmmc-always-on;”

Here is a post where we did some check with this chao.zhang and his sdcard slot can work.

jlucier · March 22, 2022, 4:12pm

Thanks for the info. I’m still stuck here unfortunately. Can you tell me if I want the iommus section in the sdhci@34400000.

WayneWWW · March 22, 2022, 4:13pm

iommus does not matter to your problem.

WayneWWW · March 22, 2022, 4:22pm

I just read your hardware schematic.

Is your 3V3_SD ever controlled by a GPIO or not? I mean what is the SD_EN in the picture?

jlucier · March 22, 2022, 4:31pm

Hey, yeah so the SD_EN is controlled by gpio07. I have left that untouched in the pinmux configuration currently. When ever I have the dt configured to get the “mmc1: error -110 whilst initialising SD card” in dmesg. I see the LED on 3v3_SD turn on for a split second. With this configuration, the light will flash 4 times whenever the card is hot plugged and I will get 4 new instances of the error message.

Also, I’m looking at the dts from chao.zhang and I see that he has vmmc-supply = <0x17>;
In that dts, phandle 0x17 is this section:

		regulator@111 {
			compatible = "regulator-fixed";
			reg = <0x6f>;
			regulator-name = "vdd-fan";
			regulator-min-microvolt = <0x4c4b40>;
			regulator-max-microvolt = <0x4c4b40>;
			gpio = <0x25 0x8 0x0>;
			linux,phandle = <0x17>;
			phandle = <0x17>;
		};

This is a different section than what you told me to use and also appears to be a 5V regulator. Could you explain that to me please? Am I being stupid and misreading the phandle? Thanks!

WayneWWW · March 22, 2022, 4:37pm

Hi,

Ok, we find the first mistake. If your “sd_en” is controlled by a GPIO from jetson, then it is not a always-on power source.

Which means that “nvidia,vmmc-always-on” is not matching your hardware.

I am just wondering, could you just put load control as always on first? If you have “nvidia,vmmc-always-on” in your device tree, then that 3v3_SD needs to be always there.

WayneWWW · March 22, 2022, 4:37pm

And just forget about chao.zhang’s dts. Just read his “vmmc-supply = <&spmic_sd3>;”. I guess his dts file does not match his real result.

jlucier · March 22, 2022, 5:06pm

Okay so for the record here: I removed the “nvidia,vmmc-always-on” and set vmmc-suply to spmic_sd3 (0x13 in my dts) and I get the repeated error at boot of

mmc1: CMD CRC or end bit error, int mask 0xc0000

However nothing happens when I hot plug and the 3v3_SD light does not come on.

For kicks, I also tried the vmmc-supply that chao.zhang uses in his dts (vdd-fan) and I get the light flashing on hot plug with

mmc1: error -110 whilst initialising SD card

jlucier · March 22, 2022, 5:06pm

Are you suggesting that I go back to setting the gpio07 pin to output drive1 in the pinmux config?
Do you have any sense of why I would get the intermittent kernel panic on boot with that setting?

WayneWWW · March 22, 2022, 5:18pm

No, you still not get the point.

That user (chao.zhang) can use spmic_sd3 regulator + always on setting because his hardware design is a always on 3v3 sd.

But your case is not.

That is why I asked you to directly make your load switch to always give out 3v3 instead of using a gpio to control. This does not matter to the pinmux. I am talking about the hardware schematic.

If the power source is wrong, then your error is expected.

WayneWWW · March 22, 2022, 5:25pm

Take your “LED light” as example.

A “always-on” power source means when the jetson board is powered on, your 3V3_SD LED is on. The existence of sdcard does not matter.

jlucier · March 22, 2022, 5:29pm

Are you saying that a power enable line is not supported at all? There’s no device tree change I can make that will work? When I set that pin to output drive 1 in the pinmux it turns the power enable on reliably…