Xavier nx bridge: USB OTG connected host identifies pcie connected nvme ssd as disk

Do note that ssh and scp and sftp over ethernet can be over USB since there is a virtual ethernet device on the USB. However, it sounds like the following are options (difficulties vary):

  • As a USB “Gadget”, this would just be a disk. Since the computers using this are Linux, you would want to format it as ext4.
  • If other computers are involved, especially Windows, then the old Network Neighborhood could be used (Linux can mount that too, but it is a lot of work). In that case you’d format as NTFS. This is network, but it looks like a local disk, and the USB can be used for the network.
  • If other Linux (or *NIX flavors such as BSD) are to all share this, then NFS could be used. This too is network, but it looks like a local disk, and the USB can be used for the network.
  • iSCSI. Only a single computer can use this, and it is over the network. Again, this can be over a USB ethernet. This is tricky to set up. In the case of gigabit or faster (which the USB is not), this is ideal…the best performance is possible, and often this works in some strange environments.

For the case of being available over USB and appearing as a disk, you are stuck with the USB Gadget. You need to partition the disk, and then format as ext4. With the disk mounted, what do you currently see from “lsblk -f” (you only need to show the results related to the SSD)?

This is possible, but it is not possible by default. There is no “out of the box” solution. There is a significant learning curve, but information on the topic is available since a lot of people have done this, and because the Gadget API was extensively refined starting several years ago. I will emphasize this is not out of the box.

Thank you for your replies.
So I want to make sure I understood correctly:

  1. I can use ssh,scp,sftp from my host by creating a virtual Ethernet interface and read/write data via USB.
    The virtual interface is actually a USB gadget.
    I saw an example here:
    How to create USB Ethernet gadget for Jetson through configfs - RidgeRun Developer Connection
    This solution is fairly easy to implement, just need to know the exact needed functions and configurations.

  2. I can use same USB gadget method to make the nvme partition identified as a disk by my host? (isn’t that like previous options just choosing other suitable functions and configurations?)

I checked and confirm that when my host is connected through usb I see the following eth interfaces on the nx side:

  1. usb0
  2. l4tbr0
    they have almost the same mac address.

I assume I need to assign to one of these interfaces an ip address and then try to ssh from my host to the nx (therefore gain access to my ssd partition)

I didn’t tried it yet since I want to go for the other “harder” option: make my ssd identified as a disk on my host and read/write its files like a regular disk (using the host GUI)

Can I achieve that by configuring a USB gadget with the right functions and configurations?

Thanks

About this:

I can use ssh,scp,sftp from my host by creating a virtual Ethernet interface and read/write data via USB.
The virtual interface is actually a USB gadget.
I saw an example here:
How to create USB Ethernet gadget for Jetson through configfs - RidgeRun Developer Connection
This solution is fairly easy to implement, just need to know the exact needed functions and configurations.

Yes, you can access those files via those network commands. There are clients which work via those commands if you prefer that over command line or scripting. PuTTY is one example, and that program runs on both Linux and Windows. Most people only know about PuTTY because it can serve as a serial console program. PuTTY can also use ssh, scp, and sftp, which means it looks like a file manager when told to, but uses those network protocols instead of directly operating on storage devices (at least at the remote end).

You can use command line over ssh to log in to a Jetson over any ethernet, including the virtual one. With this you have access to command line tools as if you are logged in on the remote system. You can for example format a disk, partition a disk, delete or copy files, so on. It is important to know if the system you are running the client on is also Linux, or if it is Windows, as this changes which clients are available. This is not the same as the content appearing as if it is a hard disk, but the functionality is quite convenient and easy to set up. PuTTY is one example compatible with several operating systems. What would not happen is that you plug in the device and new storage shows up.

Those network interfaces, on the Jetson side, are indeed part of the virtual USB device. That device mode USB interface appears to the other computer as if it is a router. Should the other end of the ethernet accept this virtual router, then the following IP addresses are automatically set up via DHCP:

  • For the Jetson, 192.168.55.100 (from either computer “ping 192.168.55.100” will ping the Jetson).
  • For the remote end, 192.168.55.1 (from either computer “ping 192.168.55.1” will ping that other computer).

Any of the above would avoid needing to set up the Gadget template, although it is not quite as convenient. Incidentally, network access via a client means it won’t care about permissions, filesystem types, so on; access would be based on the user credentials at each end.

Your understanding is correct that you would need USB gadget to make the Jetson’s SSD show up as a disk to the remote system. That virtual ethernet and a README file are examples of Gadget setup, whereby the README itself has its access set up exactly as you want…the example sets up a tiny read-only partition with a README file via the Gadget API. You would be performing the same steps, except you would be naming the SSD’s partition (which you need to set up) as read-write instead of naming a loopback device covering a file which pretends to be a partition.

Btw, you can do both Gadget API and ethernet ssh/scp/sftp access. The ethernet part is already there, you just have to run the client.

One possible catch on ethernet: The other end which accepts the ethernet might have security set up to force you to verify that you are ok with that network device. This is often the default, but not always the case.

Thank you for your reply and patience.

I want to go on the “ssd identifies as a disk” option.

I understand there is already a similar USB gadget - the README partition.

My questions are:

  1. Where are this gadget configuration and settings files located? I assume I will copy them and make the appropriate changes.
  2. Is there some kind of tutorial for “newbies” regarding the different configurations and different functions options available when creating a new USB gadget?
  3. Is there some kind of tutorial explaining how to do the exact thing I want to do?
    How to create a usb gadget which makes existing partition with ext4 file system to identify as a disk by the host? (I know you help me with that but maybe I can learn more from some guide someone already written)

In general, the demonstration of Gadget device mode includes both the virtual ethernet and the block device with the README file. You’ll perhaps want to copy that, and remove the ethernet part (so the original is not changed), followed by editing the block device (mass storage) content to point at your partition instead of creating and pointing at a loopback device (the “losetup” content “covers” a file in order to make the file look like a partition). That location is in:
/opt/nvidia/l4t-usb-device-mode
(more on that below)

Before you start though, can you verify the SSD’s partition you wish to use, and say if it is able to mount and umount to some temporary location? What do you see from the command “lsblk -f”? Mostly you will want to set up some sample file to see if it can be read, and place it on the partition, but you won’t want the partition mounted while actually working with it.

Regarding the example mass storage code, it is both simplified and made more complex because it is enabled as a service. Are you familiar with reading bash shell script code? Mostly it is just the same as what people would type in on command line (your login shell is in fact a bash shell…the same interpreter is reading the scripts we look at that also looks at what you type into the command line when operating normally).

The scripts in that location have a manual “stop” and “start” shell script, which will be obvious by their names. There is also a “service” setup, which is used during startup by the operating system to enable/disable/start/stop that same service (the service is a front end wrapper to the start and stop of the actual function, and you want to work only with a start and stop of function, and not with start and stop of a service). When the operating system starts or stops this it is given the service name “nv-l4t-usb-device-mode.service”, and it starts or stops both the mass storage and the virtual ethernet services. When you go to work on a copy you’ll need to initially not stop or start this as a service (leave the original service and files alone, and when working, just stop or start the edited nv-l4t-usb-device-mode.sh or stop.sh which avoids touching any files or resource from the original demonstration of mass storage).

A demonstration of the service is to query the status:
systemctl status nv-l4t-usb-device-mode
(you won’t be using this, but it is useful to see if the service is running)

Note the file “filesystem.img”. This is the file which pretending to be a partition. Any command you see related to networking you can remove from your copy, and any command related to “losetup” can also be removed (this is part of mass storage, but you are not emulating a partition, you have an actual partition). You will need to operate as root for all edits, so you’ll drop into a root shell via “sudo -s”.

I would first “cd /opt/nvidia”. Then I’d make a copy of the content under a new directory:

# Warning: This gives you permission to destroy the o/s:
sudo -s
cd /opt/nvidia
cp -adpR ./l4t-usb-device-mode ssd
cd ssd
rm dhcp*
rm *filesystem*
rm mac-addresses
rm nv-l4t-usb-device-mode.service

The above will remove much of the content which you won’t be using. Explore those files. Find the parts which are related to a virtual network device, and remove them or comment them out. Incidentally, in a bash script, you always leave the top two lines alone, whereby the second line is blank. So you won’t touch this:

#!/bin/bash

(known as the “shebang”)

Note that comments start with “#”. If you wonder if something does something, but don’t want to delete it, you can make those lines inert via prefixing with “#”. You don’t need to remove comments related to networking, but you could. The comments might be useful in figuring out what is related to mass storage versus networking. Sections with IP addresses, MAC addresses, routing, so on, can all be deleted. Parts related to fs_img or filesystem or partitioning will be edited. Parts related to “loop0” (or any particular “loop#”) will need to be edited to point at the actual partition (“loop0” is a virtual partition).

Incidentally, this lists block devices, and you’ll find “loop0” (which is “/dev/loop0”) listed:
lsblk -f

Eventually you will link the SSD you see in that command. loop0 will still be present because you are not editing the original demo, you’re adding your own edited version in a new directory.

During the editing you’ll have to start learning about the Gadget API from some of those documents listed in earlier replies. Feel free to ask questions, for example, if you want to know how to edit a particular line of code in a file, or what the hexadecimal means for USB parameters, so on. This will not be without a learning curve.

1 Like

I start working on this matter.

I noticed that after powering on the xavier nx my host doesn’t see any new disk. The L4T-README partition (/dev/loop0) is seen from the nx itself only and not by my host.

(JP 4.4)

This tends to be a USB issue. If you monitor “dmesg --follow” on the host PC, and if you plug in the correct cable on the NX and to the host, what shows up in the logs as a result of that plugin?

These are the messages during the flashing process:

[169899.831611] usb 1-10: New USB device found, idVendor=0955, idProduct=7e19, bcdDevice= 1.02
[169899.831617] usb 1-10: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[169899.831621] usb 1-10: Product: APX
[169899.831623] usb 1-10: Manufacturer: NVIDIA Corp.
[169926.046348] usb 1-10: USB disconnect, device number 7
[169926.354762] usb 1-10: new high-speed USB device number 8 using xhci_hcd
[169926.503638] usb 1-10: New USB device found, idVendor=0955, idProduct=7e19, bcdDevice= 1.02
[169926.503643] usb 1-10: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[169926.503646] usb 1-10: Product: APX
[169926.503649] usb 1-10: Manufacturer: NVIDIA Corp.
[169929.807116] EXT4-fs (loop30): mounted filesystem with ordered data mode. Opts: (null)
[170357.972249] usb 1-10: USB disconnect, device number 8
[178430.458091] usb 1-10: new high-speed USB device number 9 using xhci_hcd
[178430.606698] usb 1-10: New USB device found, idVendor=0955, idProduct=7e19, bcdDevice= 1.02
[178430.606699] usb 1-10: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[178430.606701] usb 1-10: Product: APX
[178430.606701] usb 1-10: Manufacturer: NVIDIA Corp.
[178434.923971] usb 1-10: USB disconnect, device number 9
[178435.230102] usb 1-10: new high-speed USB device number 10 using xhci_hcd
[178435.379152] usb 1-10: New USB device found, idVendor=0955, idProduct=7e19, bcdDevice= 1.02
[178435.379158] usb 1-10: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[178435.379161] usb 1-10: Product: APX
[178435.379164] usb 1-10: Manufacturer: NVIDIA Corp.
[178462.574313] usb 1-10: USB disconnect, device number 10
[178462.882218] usb 1-10: new high-speed USB device number 11 using xhci_hcd
[178463.031266] usb 1-10: New USB device found, idVendor=0955, idProduct=7e19, bcdDevice= 1.02
[178463.031272] usb 1-10: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[178463.031275] usb 1-10: Product: APX
[178463.031278] usb 1-10: Manufacturer: NVIDIA Corp.
[178466.735553] EXT4-fs (loop30): mounted filesystem with ordered data mode. Opts: (null)
[178894.866217] usb 1-10: USB disconnect, device number 11
[258158.964711] usb 1-10: new high-speed USB device number 12 using xhci_hcd
[258159.113643] usb 1-10: New USB device found, idVendor=0955, idProduct=7e19, bcdDevice= 1.02
[258159.113649] usb 1-10: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[258159.113652] usb 1-10: Product: APX
[258159.113655] usb 1-10: Manufacturer: NVIDIA Corp.
[258173.198654] usb 1-10: USB disconnect, device number 12
[258173.504742] usb 1-10: new high-speed USB device number 13 using xhci_hcd
[258173.653623] usb 1-10: New USB device found, idVendor=0955, idProduct=7e19, bcdDevice= 1.02
[258173.653629] usb 1-10: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[258173.653632] usb 1-10: Product: APX
[258173.653635] usb 1-10: Manufacturer: NVIDIA Corp.
[258177.188357] EXT4-fs (loop9): mounted filesystem with ordered data mode. Opts: (null)
[169899.831611] usb 1-10: New USB device found, idVendor=0955, idProduct=7e19, bcdDevice= 1.02
[169899.831617] usb 1-10: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[169899.831621] usb 1-10: Product: APX
[169899.831623] usb 1-10: Manufacturer: NVIDIA Corp.
[169926.046348] usb 1-10: USB disconnect, device number 7
[169926.354762] usb 1-10: new high-speed USB device number 8 using xhci_hcd
[169926.503638] usb 1-10: New USB device found, idVendor=0955, idProduct=7e19, bcdDevice= 1.02
[169926.503643] usb 1-10: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[169926.503646] usb 1-10: Product: APX
[169926.503649] usb 1-10: Manufacturer: NVIDIA Corp.
[169929.807116] EXT4-fs (loop30): mounted filesystem with ordered data mode. Opts: (null)
[170357.972249] usb 1-10: USB disconnect, device number 8
[178430.458091] usb 1-10: new high-speed USB device number 9 using xhci_hcd
[178430.606698] usb 1-10: New USB device found, idVendor=0955, idProduct=7e19, bcdDevice= 1.02
[178430.606699] usb 1-10: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[178430.606701] usb 1-10: Product: APX
[178430.606701] usb 1-10: Manufacturer: NVIDIA Corp.
[178434.923971] usb 1-10: USB disconnect, device number 9
[178435.230102] usb 1-10: new high-speed USB device number 10 using xhci_hcd
[178435.379152] usb 1-10: New USB device found, idVendor=0955, idProduct=7e19, bcdDevice= 1.02
[178435.379158] usb 1-10: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[178435.379161] usb 1-10: Product: APX
[178435.379164] usb 1-10: Manufacturer: NVIDIA Corp.
[178462.574313] usb 1-10: USB disconnect, device number 10
[178462.882218] usb 1-10: new high-speed USB device number 11 using xhci_hcd
[178463.031266] usb 1-10: New USB device found, idVendor=0955, idProduct=7e19, bcdDevice= 1.02
[178463.031272] usb 1-10: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[178463.031275] usb 1-10: Product: APX
[178463.031278] usb 1-10: Manufacturer: NVIDIA Corp.
[178466.735553] EXT4-fs (loop30): mounted filesystem with ordered data mode. Opts: (null)
[178894.866217] usb 1-10: USB disconnect, device number 11
[258158.964711] usb 1-10: new high-speed USB device number 12 using xhci_hcd
[258159.113643] usb 1-10: New USB device found, idVendor=0955, idProduct=7e19, bcdDevice= 1.02
[258159.113649] usb 1-10: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[258159.113652] usb 1-10: Product: APX
[258159.113655] usb 1-10: Manufacturer: NVIDIA Corp.
[258173.198654] usb 1-10: USB disconnect, device number 12
[258173.504742] usb 1-10: new high-speed USB device number 13 using xhci_hcd
[258173.653623] usb 1-10: New USB device found, idVendor=0955, idProduct=7e19, bcdDevice= 1.02
[258173.653629] usb 1-10: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[258173.653632] usb 1-10: Product: APX
[258173.653635] usb 1-10: Manufacturer: NVIDIA Corp.
[258177.188357] EXT4-fs (loop9): mounted filesystem with ordered data mode. Opts: (null)

After that nothing appears. In addition, the lsusb output is:

Bus 002 Device 002: ID 1058:2627 Western Digital Technologies, Inc.
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 005: ID 03f0:1a4a Hewlett-Packard
Bus 001 Device 004: ID 0403:6001 Future Technology Devices International, Ltd FT232 USB-Serial (UART) IC
Bus 001 Device 003: ID 9710:7840 MosChip Semiconductor MCS7820/MCS7840 2/4 port serial adapter
Bus 001 Device 002: ID 046d:0a38 Logitech, Inc. Headset H340
Bus 001 Device 006: ID 0403:6001 Future Technology Devices International, Ltd FT232 USB-Serial (UART) IC
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

Is there any kind of VM involved? I tend to wonder about that when I see the APX in the logs. VMs normally have USB issues, but if this is not a VM, then it helps to know that.

no VM.
Host: Ubuntu 18.05
Target: Xavier nx , emmc model (not evaluation kit) , JP 4.4

“not evaluation kit” is probably the issue (maybe not). Did you flash using the manufacturer’s software? I ask because many of the pins on the SoC and module can be programmed to have different functions. Unless the electrical layout of the third party carrier board is an exact match to that of the development kit, using the NVIDIA flash software would result in some of those pins not being set up correctly (the device tree is how this is set up, and the third party carrier board manufacturer is the one providing that alternate device tree…either in the form of a patch to NVIDIA’s software, or else in the form of their own software flash tools based on NVIDIA’s software). If something like the ID detect pin on the USB connector is wrong, or a power rail associated with device mode is wrong, then part of the functionality could be missing.

so I’ll start with the devkit first and after it works I’ll move on to the custom carrier board (I didn’t make any changes to the device tree and flashed manually using the flash.sh script)

Does the devkit has the USB controller enabling it to be a host and an end point (device)?

Most of the full sized Jetsons with a micro-OTG port work in both host and device mode. I’m not positive, but I think the NX only supports device mode on the micro-OTG port. A third party carrier board might support both modes, but it would require different wiring and a different device tree.

OK but its enough for me am I wrong? I need the NX on device mode in order for one of its partitions will appear as a disk on the host

You have a custom carrier board. The dev kit already has device mode set up, but some dev kits (I think including the NX) do not enable host mode (and it cannot be enabled on those because the wiring does not exist). I couldn’t give you the exact instructions on a custom board in order to make the OTG port behave in both host and device mode, but there is one specific point that is important: The OTG socket has an ID pin, and a power delivery pin (hosts provide power, devices consume power). The physical wiring must be present, and the device tree must tell the kernel where and how to find those pins (the how might include which power rail to use…a USB2 micro-OTG is always +5V, so it wouldn’t work well if you routed a 12V rail to it).

I apologize, maybe I misunderstood something:

  1. I tried connecting my ubuntu host to the custom carrier board and saw there is no README partition after flashing. Therefore I said I will verify that I see the README partition using a Xavier NX emmc model devkit (regardless to my custom carrier board nx module) just to see if the README partition appears. That’s why I asked if it can function as a USB device. You said it can only act as a device - but that’s what I need isn’t it? I want my ubuntu host to act as a USB host and the devkit to act as a USB device in order to see the README partition on my Ubuntu host. Am I confusing?

  2. Regarding the custom carrier board module - during the flashing process we saw on the Ubuntu host dmesg output that it recognized the custom carrier nx partitions and mounted them on itself (I sent the dmesg output). Isn’t that enough to indicate that there is no problem with the device tree and physical wiring existence?

Thank you for your patience.

You may also tell if the target partition that you want to share has a filesystem mounted on Jetson. If not, it would be as easy as:

# Unplug USB to host

sudo service nv-l4t-usb-device-mode stop

# Check that target has not a fs mounted:
sudo lsblk

# If mounted as /media/user/...  first unmount it such as:
sudo umount /dev/nvme0n1p2

# Edit file /opt/nvidia/l4t-usb-device-mode/nv-l4t-usb-device-mode-config.sh and change last line with devfs path to your partition such as: `fs_img="/dev/nvme0n1p2"`

# restart service
sudo service nv-l4t-usb-device-mode start

Note that this would require a directory named version to be deleted and created at root of the target filesystem (should only be an issue if this fs already has a version directory).
If it fails to restart, try rebooting.

Note that this read-only. For rw, you would edit file /opt/nvidia/l4t-usb-device-mode/nv-l4t-usb-device-mode-start.sh and change line 202 to :

echo 0 > "${func}/lun.0/ro"

This applies to the exported mass storage.

For the disk that will appear on Jeston, it is mounted read-only at line 269, for rw you would try to remove -r flag.

If you want to share an already mounted partition filesystem, it may be more complex.

The partition in question is not a “real” partition. It only exists when loopback has set it up as such (loopback covers a file; the loopback device is what shops up as a partition). @Honey_Patouceul just gave you the method of enabling/disabling and working with the loopback partition. Assuming it is still set up as default, this would likely show up as “loop0” when listing partitions, e.g., via lsblk. I am curious, what do you see when specifically looking at loop devices?
lsblk -f | grep loop

The USB making this visible as device mode is a separate topic versus loopback setup. This is where device tree becomes involved. The device tree does not set up, nor does it mount any loop or block device. The device tree does determine the wiring and setup of the USB lanes, and those lanes are the other half of the OTG port (or even regular host port) on the carrier board.

Half is the partition being available (virtual or not), the other half is the pipe making this available to the outside world (in this case USB). What does the lsblk show for the virtual loop devices (I’m assuming this is what is missing at first; once that is available, e.g., through the mentioned nv-l4t-usb-device-mode or an actual partition, it becomes a USB issue).

I can see the L4T-README partition on my Ubuntu host when I connect it to the devkit.

So first step is to create a new partition with new filesystem on my devkit and cause it to identify on my Ubuntu host just like the L4T-README partition does.

After that I’ll move on to the custom board.