Anyone got the Geekworm Jetson Nano NVMe M.2 SSD Shield T100 working reliably?

Robotics & Edge Computing Jetson & Embedded Systems Jetson Nano
1

I have two issues with this product. The first one I solved myself…

First problem:
I get frequently these errors when the device is under load. Sometimes it corrects itself, other times it doesn’t.

tegra-xusb 70090000.xusb: ERROR Transfer event for disabled endpoint or incorrect stream ring
tegra-xusb 70090000.xusb: @00000000ffebc730 ffdec000 00000000 04000000 03088001
sd 0:0:0:0: [sda] tag#27 uas_eh_abort_handler 0 uas-tag 28 inflight: CMD OUT
sd 0:0:0:0: [sda] tag#27 CDB: opcode=0x2a 2a 00 0d 50 a2 e4 00 02 b8 00
scsi host0: uas_eh_bus_reset_handler start
usb 2-1.2: reset SuperSpeed USB device number 3 using tegra-xusb
scsi host0: uas_eh_bus_reset_handler success

The fix for this issue I found after two days of struggling…
I had to add the following to the kernel command line to turn off the UAS interface and force the driver to use the Bulk Data Transfer interface.

usb_storage.quirks=0x152d:0x0562:u

Technically, UAS is supposed to be “better” especially with multiple drives but I saw no difference in throughput between the two.

The second issue I haven’t found a solution for… I can’t reboot the Nano and have
the shield still work. I have to completely power both the Nano and the shield off and
on again or the shield to operate correctly. If I don’t, the boot process hangs with
the following messages. The shield is NOT the root filesystem. Just an extra drive.

usb 1-2.2: Device not responding to setup address.
tegra-xusb 70090000.xusb: Upgrade port 0 to USB3.0
tegra-xusb 70090000.xusb: Upgrade port 1 to USB3.0
usb 1-2.2: Device not responding to setup address.
usb 1-2.2: device not accepting address 3, error -71
usb 2-1: new SuperSpeed USB device number 2 using tegra-xusb
usb 2-1: New USB device found, idVendor=0bda, idProduct=0411
usb 2-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
usb 2-1: Product: 4-Port USB 3.1 Hub
usb 2-1: Manufacturer: Generic
hub 2-1:1.0: USB hub found
hub 2-1:1.0: 4 ports detected
usb 1-2.2: new high-speed USB device number 4 using tegra-xusb
usb 1-2.2: Device not responding to setup address.
usb 2-1: usb_suspend_both: status 0
usb usb2: usb_suspend_both: status 0
usb 1-2.2: Device not responding to setup address.
usb 1-2.2: device not accepting address 4, error -71
usb 1-2-port2: attempt power cycle
usb 1-2.2: new high-speed USB device number 5 using tegra-xusb
usb 1-2.2: device descriptor read/64, error -110

This one’s a deal breaker. A device that requires a power cycle to work
is a big issue.

Anyone have it working or else having the same issues?

I know Geekworm is on this forum and I’ve also emailed them. I hope they can help.

2

Hi,

I have actually got 3 of the T100’s, because I wanted to make a cluster of Jetson Nano’s (I’m a datatechnician). I started my journey by trying the JetsonHacks booting from USB, but never got past the step where you copy the SD Card files to the USB, because I encountered the exact issue you describe.

The drives work initially, until they come under load, then “dmesg” will show a bunch of I/O errors, eventually remounting the disk as Read-Only.

So I used your solution with the kernel parameter - and that worked. So thank you for sharing that.

The other issue, with the drive behaving odd and not surviving a reboot, you describe. I encountered that as well. I haven’t figured a way around this. May I ask, do you power the USB drive with the bundled Y-split power cable?

I have an untested suspicion, which is that, when the T100 gets permanent power, this issue occurs.

It should be possible to run the T100 without the Y splitter though, as one of my Jetsons, run 2xT100’s and one of them is only powered by the USB cable.

3

Ah, so it’s not just me (sorry). :)

I’ve tried powering both ways and it didn’t change anything. Right now I’m powering via the USB cable.
I hope Geekworm has some insight into what’s going on.

4

Hehe none taken, and nope you’re not alone.

Hopefully there is a solution to this. I think we’re looking at a similar issue to this: I must replug usb device to be visible - Raspberry Pi Forums

Basically that the USB T100 adapter is put into a suspend state and cannot wake from that. I wonder if that is a driver issue.

5

I think there’s a guy who pretty much explains the issue here: Raspberry pi 3 not rebooting from usb - Raspberry Pi Forums

Read the last to posts from a guy called HawaiianPi

6

Interesting. At least on the Jetson’s the bootloader code is actually on the SD card so if a change is needed there is should be easy for us to install.

Can someone from Nvidia comment?

7

Hi,
Seems like enumeration fails. Please power-cycle the usb hub after booting and check if the device can be successfully enumerated.

Steps of executing power-cycle:
https://elinux.org/L4T_Jetson/r32.2.1_patch
[Jetson Nano]Power control

8

It took a bit but yes, power cycling the hub worked.

jetson1 r/usb# gcc -lusb-1.0 -o pcycle ./realtek_hub_power_cycle.c
jetson1 r/usb# ./pcycle
[  332.638928] usb 1-2.4: Device not responding to setup address.
[  334.498819] usb 1-2.4: Device not responding to setup address.
[  334.711375] usb 1-2.4: device not accepting address 7, error -71
[  336.442839] usb 1-2.4: Device not responding to setup address.
[  338.302751] usb 1-2.4: Device not responding to setup address.
[  338.515303] usb 1-2.4: device not accepting address 8, error -71
[  344.435345] usb 1-2.4: device descriptor read/64, error -110
[  360.051035] usb 1-2.4: device descriptor read/64, error -110
[  365.426930] usb 1-2.4: device descriptor read/64, error -110
[  381.042619] usb 1-2.4: device descriptor read/64, error -110
[  381.154550] usb 1-2-port4: unable to enumerate USB device
jetson1 r/usb# [  383.092637] usb 2-1.4: UAS is blacklisted for this device, using usb-storage instead
[  383.100445] usb 2-1.4: UAS is blacklisted for this device, using usb-storage instead
[  384.130940] scsi 0:0:0:0: Direct-Access     NVME SSD                  0205 PQ: 0 ANSI: 6
[  386.211442] sd 0:0:0:0: [sda] 500118192 512-byte logical blocks: (256 GB/238 GiB)
[  386.219695] sd 0:0:0:0: [sda] Write Protect is off
[  386.225259] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[  386.242612] sd 0:0:0:0: [sda] Attached SCSI disk

Now the question is… Can we get a kernel patch or xusb firmware patch? Can’t really run this in production especially if the USB device is going to be used for the root filesystem.

9

Hi,
Are you able to get help from the board vendor? It is more like an issue that Nano is trying to enumerate the device but the device is not ready yet.

usb 1-2.2: Device not responding to setup address.
usb 1-2.2: device not accepting address 4, error -71

Or maybe doing power-cycle in rc.local. Below is a reference of enabling rc.local:
https://devtalk.nvidia.com/default/topic/1065398/jetson-nano/omxh264enc-and-nvvidconv-not-working-unless-/post/5396372/#5396372

10

Thanks Dane, Unfortunately, rc.local is too late if you’re going to use the usb drive as the rootfs (which I plan to do shortly). Maybe I can build it into the kernel or even u-boot (without libusb of course).

I heard from Geekworm last night and they’re trying to reproduce the issue now.

11

Well, I’ve heard no further word from Geekworm so I’m going to leave a 1 star review on Amazon.

I’ve tried to power cycle the hub from the kernel xhci layer but to no avail. Dane, since Nvidia obviously modified xhci, xhci-hub and hub kernel modules as well as provided the xhci-tegra, xhci-hub and phy-tegra-usb modules, would it be possible to get some assistance in figuring out where we might hook into to power cycle the hub using the standard usb kernel API?

12

Hi,
The approach is from USB hub vendor and uses libusb interfaces. It looks OK and is not like a hack or workaround.
There are other servies in

/etc/systemd/system

Please check if you can put it in an early-initialized service.

13

Hi Nvidia,

It would be great if USB was available at boot. Even better if the Jetson Hack SD card pivot boot, wouldn’t be necessary. However for my part, I need USB to boot. So as OP, I’m looking for a solution for power-cycling the Hub.

I wonder if power-cycling could be done as a final step in the reboot process (when the OS is shutting down).

14

Dane, the issue for getting the device to work as a rootfs is that the power cycle has to be done after the USB controller has its firmware installed but BEFORE the realtek hub is enumerated and it has to be complete before any other devices are enumerated. I’ve tried doing it in the initrd but it takes too long for the power cycle and the re-unumeration of the devices to occur. In this case, the pivot root times out.

peterglad1985: That’s a great idea. Let me see if I can work something up to execute after the filesystems are remounted read-only or unmounted.

15

@peterglad1985 : The attached zip file contains a systemd shutdown script that unbinds all USB devices and power cycles the hub as the last thing systemd does before rebooting or halting.

Copy unbind_usb.shutdown to /usr/lib/systemd/system-shutdown.
Copy realtek_hub_power_cycle to /usr/sbin.
You may have to recompile it since I run Fedora. Something like…
gcc -o realtek_hub_power_cycle -lusb realtek_hub_power_cycle.c

Give it a shot and see how it works for you. Been pretty reliable for me.

EDIT: See a few posts down for the latest script.
https://devtalk.nvidia.com/default/topic/1066735/jetson-nano/anyone-got-the-geekworm-jetson-nano-nvme-m-2-ssd-shield-t100-working-reliably-/post/5407050/#5407050

16

Hi gtj, thanks for the sharing.

17

Here’s an updated script and instructions…

First, you do NOT need the realtek_hub_power_cycle program. It causes more problems than it’s worth because when it powers the hub back on the devices all register again just before the reboot. Instead, just running the script will unbind the devices which seems to clear everything out just before the reboot.

I’ve also updated the script with the paths that are Ubuntu compatible. The script should be placed in /lib/systemd/system-shutdown instead of /usr/lib/systemd/system-shutdown as it would be for Fedora. Sorry about that.
unbind_usb.zip (489 Bytes)

18

Dear DaneLLL & gtj,

“Anyone got the Geekworm Jetson NVMe M.2 SSD Shield T100 working reliably?”

I guess it depends on what reliably means…

  • :)

    • The hardware used for my system:

    • Toshiba Qosmio Laptop as host running Ubuntu 18.04 with SDK Manager
      • Nvidia Jetson Developer Kit as target
        • Geekworm Jetson Nano NVMe M.2 SSD Shield T100
          • Intel 660p NVMe SSD (1 Terabyte)

    Last night I used the SDK Manager to login live and fetch the Jetpack 2.3.2 Host and Target files. These procedural steps are general, so there might be some detail missing. Forgive in advance…

    Logged into SDK Manager and downloaded Host and Target apps, Unchecked box for flashing to SD card.
    Exited SDK Manager.

    Restarted SDK Manager to install offline.
    Flashed SD card with Jetpack 2.3.2
    Exited SDK Manager.

    The flash to SD Card (64 GB) was successful but only had an APP partition of 14 GB (to be solved later).

    Rebooted the Nano and set up login, license agreements, etc, and network ( Ethernet hard wire and noted ip address)

    Restarted SDK Manager with offline install and ran installs for target.

    When prompted to login to Nano, I used the ip address noted above instead of the default.

    Installs of Target apps completed ok after about 1 hour and 30 minutes (I took a nap):)

    Exited from SDK Manager.

    Installed Nomachine for ARM 64 on Nano. (I would like to know if this does damage to Nano use later), incompatible?).

    Installed Gparted on Nano. (I would like to know if this does damage to Nano use later, incompatible?).

    Ran nomachine from host to access nano remotely.

    nomachine needs to be installed on both host and target.

    Ran Gparted and

    • confirmed that Nano was mounted and running on \dev\mmcblck0.
    • confirmed that all 14 partitions had no hidden flags set but all did have flag for msftdata
    • confirmed that NVMe SSD was mounted on \dev\sda.

    (The SSD partition was ok and contained data from past install-configure efforts.)

    I plan to reformat the SSD and make any changes toward making it bootable.

    I am going to do a few more tests on this but this is as far as I have gotten.

    A few questions please,

    • How can the SD card partitions be safely resized on the Nano?
    • Why would Nvidia apparently be supporting SD card partitions and formats that do not use available space?
    • Is there a formalized list of update sources approved by Nvidia to help prevent software package conflicts? Is it safe to install synaptic package manager on Nano given use of Nvidia approved sources?
    • What is the most stable method to manage partitions on Nano? I am hoping LVM would be compatible.
    • Does Nvidia formally support booting from NVMe SSD on Nano? If not, I am afraid that future Nvidia updates will make booting from SSD impossible.
    • Do you have any suggestions or corrections to the procedures I used?

    Please reply back with answers, comments, suggestions to eventually get nano to boot from NVMe SSD.

    Thank You in advance,

    Kent Smith
    (I use GUI apps because I am too old to remember command line syntax!)

    Screenshot from 2019-11-30 14-51-39.png
    Screenshot from 2019-11-30 14-51-39.png1280×720 194 KB

    19

    That’s a nice NVMe!

    OK first things first. Using the SDK Manager to create the the SDCard has had issues with placing the root filesystem partition first so it can’t be expanded. I’d recommend downloading the SD Card image from https://developer.nvidia.com/jetson-nano-sd-card-image-r3223 and writing that directly to an SDCard. It’ll also be faster. Once burned to the card, the root filesystem partition will be partition 1 but will be physically placed at the end so it can be expanded. You can then use parted to expand the partition. When you boot the Nano from the card for the first time, the root filesystem will be expanded to fit the new partition size. BTW, the SDK Manager creates 14 partitions for some reason but the supplied SDCard image only has 12. 11 system partitions and the root filesystem.

    In your case though, there’s no need if you’re going to use the NVMe as your root filesystem.

    To answer some of your other questions…

    I’ve not use Nomachine before but I can’t see how it would be incompatible with the Nano.

    (g)parted is also no issue as long as you don’t disturb any of the 12 existing partitions except for resizing partition 1.

    The available space issue in your case is due to what I believe is a bug in the SDK Manager. It assumes you have a 16gb card and creates the partitions accordingly. To make better use of the space, follow the procedure I outlined above and you should be OK.

    You can use lvm2 but you have to be careful of 2 things… Your /boot/ directory MUST be in the “APP” partition mmcblk0p1 and you MUST NOT disturb those 11 system partitions. You’ll have to install the lvm2 package to get the utilities. You can use lvm on the NVMe device all you want.

    I’m not really a Debian/Ubuntu person so I can’t answer the package manager stuff.

    As for “booting” from the NVMe… That has to be separated into two distinct things. There’s the actual boot process including the bootloader and kernel, then there’s the location of the root filesystem. There is currently no way to “boot” a Nano DevKit from anything other than an SDCard. The onboard boot component will only look at the SDCard for the 12 partitions. Once the kernel is running though, it’s standard Linux stuff to make the NVMe the root filesystem. However, as I stated above, the /boot directory must still be in partition 1 of the SDCard. Here’s how my layout looks…

    /dev/mmcblk0
   /dev/mmcblk0p1  mounted on /mnt/mmcblk0p1
      /boot
   /dev/mmcblk0p2 .. 12 are the system partitions

/dev/sda1 (my NVMe)
   /bin
   /boot ---> symlink to /mnt/mmcblk0p1/boot
   /dev
   et cetera

    In /boot/extlinx/extlinux.conf, add the “root=/dev/sda1” kernel parameter to the APPEND entries.

    There’s a good article on setting up a USB device as the root filesystem here: Jetson Nano - Run From USB Drive - JetsonHacks

    20

    Dear gtj,

    Outstanding! You have saved me a lot of time. I appreciate your knowledge.

    This morning, I powered down the Nano and started it up as part of my testing.

    Oh boy, no sign of the NVMe SSD!

    Hmmmmm…I need to read what you have provided along with some other Ubuntu stuff.

    Here is what I found so far…

    There is some documentation about NVMe loading here

    http://manpages.ubuntu.com/manpages/disco/en/man7/systemd-boot.7.html

    If I’m reading correctly,

    Ubuntu 18.04 LTS does not have a /boot/efi/* entry but
    Ubuntu 19.04 does.

    There is reference to a systems file /loader/loader.conf that could contain NVMe device load instructions.

    I did not find any /loader/loader.conf entry in my Nano jetpack 2.3.2/LAT system.
    It may make some sense because Nvidia uses
    Ubuntu 18.04 as the model for jetpack 2.3.2

    If that is correct, then Nvidia does not support NVMe yet at all.

    Any Nvidia updates may undo NVMe device configuration and setup.

    Well enough of my guesses. I’m gonna read your stuff right now!

    Thanks gtj

    I have attached a picture of the NVMe SSD mount as shown by Gparted just to prove I was’t
    imagining things. :)

    Screenshot from 2019-11-30 14-52-06.png
    Screenshot from 2019-11-30 14-52-06.png1280×720 122 KB