Slow SD card access speed (read+write) with Jetson Nano production module

Hi WayneWWW,

Yes, I still have the slow speed, as observed on the gnome-disks benchmark, and also when trying to write files to the sd card.

On the TX1s that I use, the speed is a lot higher (40MBps) vs 1.5MBps on Jetson Nano.

Hi jetson_user,

Based on the ios dump in your reply, we see the card is operating in SDR104 mode and the interface clock is set to maximum value.
Unless the system clocks are changed I don’t see any reason for perf regression
Could you please help share below information?

  • Which app is being used for perf measurement?
    → “gnome-disks benchmark”, right? Have you tried other tools?

  • If it is command line based tool, it would be better to share output of the perf command with and without the patch.
    To summarize, we need all the data in one file

  1. ios dump without previous patch #11 of device tree.
  2. output of perf command without the previous patch #11.
  3. ios dump with the patch in #11.
  4. output of perf command with the patch in #11.

Hi WayneWWW,

I see no patch under #11 (or maybe I am mistaken).

Do you mean the debug patch? Or the second patch I applied to include the four lines (sd-uhs*) under the sdmmc3 controller?

And the tool for benchmarking is a gui-based tool, started from command line (I SSH into Nano with a -X option). There is no text based output of this benchmarking tool (I assume by ‘perf’ you mean the gnome-disks benchmarking utility). In all use cases, I have observed an SD card read and write speed as 1.5 MBps, as a graphical visual, and a constant line over 100 samples, as if the speed is capped or something. This result is fluctuating over samples usually on other systems that we tested (e.g. TX1).

We did not try other benchmarking tools, but we tried copying large files to the SD card and the operation was noticeably slow.

Hi jetson_user,

Sorry, not clear enough. I meant below properties under sdmmc3 DT.

cap-mmc-highspeed;
cap-sd-highspeed;
sd-uhs-sdr104;
sd-uhs-sdr50;
sd-uhs-sdr25;
sd-uhs-sdr12;

Please take this one as what I said as “patch in #11”.

Also, if the benchmark tool is GUI based, please share the screenshot or the result.

btw, could you also try if running jetson_clocks would enhance the performance?

Hi WayneWWW,

Please find attached the outputs in terms of images (the names of the images are in accordance with the test conditions , like with and without the uhs patch for sdmmc3 and ios and benchmark outputs).

We already run jetson_clocks by default at startup, and the cpu and gpu frequencies are already fixed to max.

Please let me know your opinion.
OutputsWithAndWithoutUHSEnabled.zip (439 KB)

Hi jetson_user,

Could you also share the test result of using iozone?

./iozone -ecI -+n -L64 -S32 -s64m -r512k -i0 -i1 -l8 -u8 -m -t8 -F file1 file2 file3 file4 file5 file6 file7 file8

Hi WayneWWW,

This benchmark does not seem to have support for aarch64. Do you have specific link or version of iozone that I should install? I looked for one with aarch64 support and found an rpm file, which I am figuring out how to install. Am I proceeding in the wrong direction?

Besides, could you please let me know the purpose of this benchmark meanwhile? The thing is we are running out of time (deadline to fix this is coming Monday) and would need the most effective method to debug. Since the benchmark suite I am using (gnome-disks) is already indicative enough that the SD card access speeds are slow,and also our observation when we copy huge files to the SD card.

Hi WayneWWW,

Please find attached the outputs of the iozone command you gave. This is for both with the patch case, i.e, all uhs speeds enabled, and without. Please let me know if this is right.
iozone_output_without_all_uhs_speeds_enabled.txt (2 KB)
iozone_output_with_all_uhs_speeds_enabled.txt (2 KB)

Hi WayneWWW,

Do you have any updates on this one yet?

Hi jetson_user,

Please try with this patch. It is verified by other customer too.

diff --git a/kernel-dts/tegra210-porg-p3448-common.dtsi b/kernel-dts/tegra210-porg-p3448-common.dtsi
index a73561e..4c214ba 100644
--- a/kernel-dts/tegra210-porg-p3448-common.dtsi
+++ b/kernel-dts/tegra210-porg-p3448-common.dtsi
@@ -274,7 +274,6 @@
                mmc-ddr-1_8v;
                mmc-ocr-mask = <3>;
                uhs-mask = <0x0>;
-               max-clk-limit = <400000>;
                tap-delay = <3>;
        };

Although max-clk-limit is set to 204MHz from the soc dt, it is re-written to be restricted at 40KHz in the platform dtsi.

Hi WayneWWW,

Thanks for your information.

I remove max-clk-limit is in dtsi, and the clk output below.

$cat /sys/kernel/debug/mmc1/clock
9600000

Hi jetson_user,

Do you use dmesg -w command?
is it to show “Tuning done, restoring the best tap value :91” ?
Thank you!

Karis

Hi WayneWWW,

Thank you for the solution! We checked in the TX2 kernel and device tree source code (on which we did not have this slow SD card access issue) and found that the max-clk-limit was set to <208000000> for sdr104. Do you recommend this? If no, we shall just use the patch you provided, i.e., just delete the max-clk-limit.

Hi Karis,

I tested with the above solution and got clock speed as 204000000. Also, my dmesg -w output gives:

[    1.649105] mmc1: Tuning correction cannot be applied
[    1.649127] mmc1: hw tuning done ...
[    1.649177] mmc1: new ultra high speed SDR104 SDXC card at address e624
[    1.649475] mmcblk1: mmc1:e624 SN64G 59.5 GiB

Please find attached the complete dmesg output.
dmesg-w.txt (49.5 KB)

Hi jetson_user,

Thanks for your reply.

Sorry! I am not to explain clear.

I find a problem on our nano, if I use dmesg -w and unplug SDcard, it shows “Tuning done, restoring the best tap value :91” ,I would like to know, do you have the same problem.

Thank you!

Karis

Hi Karis,

Sorry didn’t get you earlier. Yes I have the same problem. I get the below logs with dmesg -w:

[  146.018945] mmc1: card e624 removed
[  146.019720] sdhci-tegra sdhci-tegra.2: Tuning done, restoring the best tap value : 23
[  146.075067] sdhci-tegra sdhci-tegra.2: Tuning done, restoring the best tap value : 23

The “Tuning done, restoring the best tap value : 23” logs are repeated.

Also, are you using a production version or dev version?

Hi jetson_user,

Thank you for your test.

The logs are repeated,We have the same problem.

I use our carrier board and nano module of emmc.

Hi jetson_user,

Please use my patch instead. I guess the speed problem is resolved,right?

Hi WayneWWW,

Yes the speed problem got resolved. Thanks!

I was just wondering whether there might be any regressions, especially after Karis’s observation above.

Hi WayneWWW,

We ran into some issues because of which I had to reopen this ticket, sorry. The thing is my dtb changes take into effect only after a hard reboot (power off + on). But when I do a soft reboot, the dtb changes related to SD card speed are not reflected and they show some older dtb values. For instance, when I do a soft reboot, and then check the ios, I get the following output:

cat /sys/kernel/debug/mmc1/ios
clock:		50000000 Hz
vdd:		21 (3.3 ~ 3.4 V)
bus mode:	2 (push-pull)
chip select:	0 (don't care)
power mode:	2 (on)
bus width:	2 (4 bits)
timing spec:	2 (sd high-speed)
signal voltage:	0 (3.30 V)
driver type:	0 (driver type B)

But I have changed the dtb already before flashing, so this should not be the case. Instead what I should get as an output is the following, which I also get with a hard reboot (power off+on).

cat /sys/kernel/debug/mmc1/ios    
clock:		204000000 Hz
vdd:		21 (3.3 ~ 3.4 V)
bus mode:	2 (push-pull)
chip select:	0 (don't care)
power mode:	2 (on)
bus width:	2 (4 bits)
timing spec:	6 (sd uhs SDR104)
signal voltage:	1 (1.80 V)
driver type:	0 (driver type B)

It seems an old dtb is loaded from somewhere which I need to prevent. How do I do this? For the soft reboot case, I also observe that the SD card access speeds are actually slower 4 times (like 23 MBps vs 85 MBps on a hard reboot), when tested with the gnome-disks benchmarking utility that I mentioned before.

Hi WayneWWW,

Any updates on this?