Moving rootfs from SD to SSD causes huge os slowdown

Robotics & Edge Computing Jetson & Embedded Systems Jetson Nano
,
1

Hello, I’m using jetson nano dev kit (older version with one camera port). I want to use external USB3 SSD as the main device for rootfs. I have used this repo with a few modifications JetsonHacksNano/rootOnUSB. After making it work and rebooting to SSD the entire os behaves like 10-15 times slower than on sd card. After searching a trying few solutions I still have no idea what exactly might be causing this. I have rebuilt initrd using fstab below.

edit: L4T version: 32.2.0

I have added this entry to my /boot/extlinux/extlinux.conf, along with modifying default profile value:

LABEL usb
      MENU LABEL usb rootfs primary kernel
      LINUX /boot/Image
      INITRD /boot/initrd
      APPEND ${cbootargs} root=/dev/sda1 rw rootwait rootfstype=ext4

My /etc/fstab, uuids taken from blkid:

UUID=af5c1f19-3ea8-4e31-9999-e3fadfc397e7      /                     ext4           defaults                                     0 1
UUID=092835c0-9d29-4b21-b0f1-b44091dbbcc9      none                  swap           sw                                           0 0
UUID=d0a54240-9799-4f63-a90b-1a3f5ca4eda8      /sdcard               ext4           defaults,noatime                             0 2
/sdcard/boot                                   /boot                 none           bind                                         0 0

hdparm testing:

jetson@jetson-desktop:~$ sudo hdparm -t /dev/sda
/dev/sda: Timing buffered disk reads: 338 MB in  3.01 seconds = 112.39 MB/sec
jetson@jetson-desktop:~$ sudo hdparm -t /dev/sda1
/dev/sda1: Timing buffered disk reads: 420 MB in  3.01 seconds = 139.44 MB/sec
jetson@jetson-desktop:~$ sudo hdparm -T /dev/sda
/dev/sda: Timing cached reads:   3288 MB in  2.00 seconds = 1645.99 MB/sec
jetson@jetson-desktop:~$ sudo hdparm -T /dev/sda1
/dev/sda1: Timing cached reads:   3190 MB in  2.00 seconds = 1596.91 MB/sec

iostat during user space loading:

Linux 4.9.140-tegra (jetson-desktop)    09/24/2020      _aarch64_       (4 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           4.32    0.00    6.37   40.54    0.00   48.76

Device             tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
loop0             0.14         1.58         0.00        328          0
loop1             0.02         0.04         0.00          8          0
mtdblock0         0.13         0.52         0.00        108          0
mmcblk0           2.02        39.94         0.04       8271          8
sda              44.42      1621.08        52.25     335677      10820

dmesg and syslog:
https://gist.github.com/Nightenom/12691a6410b46db452083c75a52a05be

systemd-analyze:
https://gist.github.com/Nightenom/0a7fe18e6ac2cfa8d9282dd1e3ebbb1a

3

Hi,

Please use this method to boot rootfs from ssd.

https://elinux.org/Jetson/L4T/Boot_From_External_Device

4

Thanks for the reply,
if I use UUID or PARTUUID as “root=” in kernel cmdline (using extlinux.conf) the kernel doesn’t seem to recognize such an option and fallbacks to /dev/mmcblk partition. The only difference I can notice is "40M" first sectors -> Press enter or "+32G" last sectors - does it actually make a difference that the partition starts at 40M instead of 2048?

5

I have flashed the latest publicly available image (JP 4.4 2020/07/07). I have flashed on sd card, used old initrd with USB firmware, added the same option as above in extlinux.conf, rsynced entire rootfs except /boot and /proc to SSD, edited fstab in the same way as above (on the SSD), disabled zram, added 8gb swap. It’s slightly better than with the older image but still enormously slower than rootfs on the sd card.
note: for ssd partition creation I used the manual you have provided.
edit: I have done the first setup settings directly on the SSD, I haven’t ever booted from the sd card
edit2: ubuntu apps are randomly occasionally freezing

6

Hi,

Please forget about sdcard image and directly follow the steps here with sdkmanager.
https://elinux.org/Jetson/L4T/Boot_From_External_Device

Also, please show us the serial console log for this part.

if I use UUID or PARTUUID as “root=” in kernel cmdline (using extlinux.conf) the kernel doesn’t seem to recognize such an option and fallbacks to /dev/mmcblk partition.

7

Ok, how do I use your guide, I have prepared sdk manager with my board detected and downloaded. The instructions are quite not clear:
1 ) is this partition for rootfs? assuming yes - so I can make it as big as the ssd is?
Does partition label have to be exactly “PARTLABEL” (without quotes)?
3) where do I find this file
4) why is this step named copy rootfs to sdcard when rootfs should be copied on the ssd?
5) what is flash.sh, external should be ssd or sd card?
the ssd should be connected to host or to nano?