usb device mode ethernet speed

Robotics & Edge Computing Jetson & Embedded Systems Jetson AGX Xavier
1

booting up xavier, I can see the USB-device-ethernet

Xavier side
l4tbr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.55.1 netmask 255.255.255.0 broadcast 192.168.55.255
inet6 fe80::1 prefixlen 128 scopeid 0x20
inet6 fe80::3cb4:a4ff:fe42:c92a prefixlen 64 scopeid 0x20
ether 52:25:9b:65:b8:81 txqueuelen 1000 (Ethernet)
RX packets 1561229 bytes 2341069941 (2.3 GB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 778082 bytes 40469865 (40.4 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

On my X86 host side,
sudo ifconfig enp0s20u5 192.168.55.2 netmask 255.255.255.0

enp0s20u5 Link encap:Ethernet HWaddr d6:10:02:48:fe:5f
inet addr:192.168.55.2 Bcast:192.168.55.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:593304 errors:0 dropped:0 overruns:0 frame:0
TX packets:1190175 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:30901238 (30.9 MB) TX bytes:1853900111 (1.8 GB)

The on Xavier side, run

nc -v -l 2222 > /dev/null

On host side, run

gtang@ws4:~$ dd if=/dev/zero bs=1024K count=512 | nc -v 192.168.55.3 2222
Connection to 192.168.55.3 2222 port [tcp/*] succeeded!
512+0 records in
512+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 7.98383 s, 67.2 MB/s

I can only got 67.2MB/s speed, that is 10% what I had hoped.
Is this the expected speed of Xavier USB 3.1 device mode can get?
Limitation is on hardware level, or a software/kernel issue?

I checked USB layer, seems it is running at correct mode. It is 5000Mb/s,
so I should get around 5000/8 = 625MB/s, instead of 67.2MB/s.

gtang@ws4:~$ lsusb -t
/: Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/6p, 5000M
|__ Port 5: Dev 3, If 0, Class=Communications, Driver=rndis_host, 5000M
|__ Port 5: Dev 3, If 1, Class=CDC Data, Driver=rndis_host, 5000M
|__ Port 5: Dev 3, If 2, Class=Communications, Driver=cdc_acm, 5000M
|__ Port 5: Dev 3, If 3, Class=CDC Data, Driver=cdc_acm, 5000M
|__ Port 5: Dev 3, If 4, Class=Mass Storage, Driver=usb-storage, 5000M
|__ Port 5: Dev 3, If 5, Class=Communications, Driver=cdc_ether, 5000M
|__ Port 5: Dev 3, If 6, Class=CDC Data, Driver=cdc_ether, 5000M
/: Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/14p, 480M
|__ Port 4: Dev 2, If 0, Class=Human Interface Device, Driver=usbhid, 12M
|__ Port 4: Dev 2, If 1, Class=Human Interface Device, Driver=usbhid, 12M
|__ Port 13: Dev 85, If 0, Class=Human Interface Device, Driver=usbhid, 12M
|__ Port 13: Dev 85, If 1, Class=Human Interface Device, Driver=usbhid, 12M
|__ Port 13: Dev 85, If 2, Class=Human Interface Device, Driver=usbhid, 12M
/: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/2p, 480M
|__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/8p, 480M
/: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/2p, 480M
|__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/6p, 480M

2

guo.tang,

I don’t think we cover such test. Could you elaborate why do you need to use usb-ethernet interface?

3

@wayneWWW

Medical device development normally takes several years (FDA approval etc). If we are generating 1080p@60Hz data in our system, and at the same time want to use Xavier to add AI inference ability into existing system. The bottleneck will be how to get data into Xavier.

  1. 19201080RGB@60Hz => 373MB/s data rate. This is over what Gigabit Ethernet can handle.
  2. Ideally the solution is using USB 3.0 device mode over TCP/IP protocal, if it can do 500MB/s. At the same time use that USB cable to power Xavier up.
  3. maybe we can go hdmi capture-> csi interface, but it needs hardware design.
  4. since we own device, we can actually send non video data into Xavier, like high dynamic range data, that requires a high speed link. Then we might need to design a pcie high speed communication card. hardware work.
  5. maybe we can H264/H265 compress the data before sending to Xavier, but if the old device computation hardware is weak, we cannot do that fast enough.

Any suggestions?

Thanks

4

@wayneWWW

The problem is

  1. How to hook Xavier up to an existing device that you don’t want to modify at all, like 3rd party device?
    maybe video capture interface. but it is not built into Xavier SOM.
  2. How to hook Xavier up to an existing device that you can change device software?
    Through 500MB/s USB 3.0 device mode. seems not fast enough, can Nvidia make it to 500MB/s?
5

guo.tang,

I wonder if improve the performance through jetson_clock.sh would help or not.

Also, since it is internet related issue, how about change something like buffer size?

6

guo.tang

Have you considered using a PCI-E Gigabit Ethernet card?

7

@jjhwadsworth

Gigabit Ethernet is 1000Mb/s = 125MB/s, not enough to handle 370MB/s data rate.

8

@guo.tang

Sorry I should have been more clear, I was meaning using a 4 port card so 4 x 125 = 500MB/s, I know Intel have some, though I can’t comment on their actual throughput.

9

A 10gps PCI-E network card might be what’s needed.

10

@jjhwadsworth @mechadeck

Any recommendation sites to buy 5G or 10G PEI-E card that can be used in Xavier?

11

@wayneWWWW

“Also, since it is internet related issue, how about change something like buffer size?”

Run the following script to tune TCP/IP stack on both host and Jetson side.

echo ‘net.core.wmem_max=12582912’ >> /etc/sysctl.conf
echo ‘net.core.rmem_max=12582912’ >> /etc/sysctl.conf
echo ‘net.ipv4.tcp_rmem= 10240 87380 12582912’ >> /etc/sysctl.conf
echo ‘net.ipv4.tcp_wmem= 10240 87380 12582912’ >> /etc/sysctl.conf
echo ‘net.ipv4.tcp_window_scaling = 1’ >> /etc/sysctl.conf
echo ‘net.ipv4.tcp_timestamps = 1’ >> /etc/sysctl.conf
echo ‘net.ipv4.tcp_sack = 1’ >> /etc/sysctl.conf
echo ‘net.ipv4.tcp_no_metrics_save = 1’ >> /etc/sysctl.conf
echo ‘net.core.netdev_max_backlog = 5000’ >> /etc/sysctl.conf

speed increased to 76.0 MB/s, still not what I had hoped.
increase MTU size from 1500 to 15000, speed actually decreased to 55MB/s

12

If the OTG USB is pretending to be an ethernet card, then it might have been told that it is 1Gb/s and any extra bandwidth available on USB3 would be programmed out just because it is intentionally negotiating to be gigabit. If you really want full speed, then you might have to write your own USB driver, but if latency doesn’t matter, then you could also experiment with bulk storage or isochronous mode gadget emulation.

An important question is what is the data you need to feed to the Jetson, and what kind of latency can it live with? Do you have more information on the nature of the data beyond average throughput?

A separate 10Gb/s add-on card is most likely the easiest thing to do…but I have no suggestion on a specific card.

13

@linuxdev

Latency less than 30ms should be fine to us.

“you could also experiment with bulk storage or isochronous mode gadget emulation”

– Could you point me to some sample codes link on how to get this working? Thanks!

14

The kernel refers to the convenience API used to simplify customizing a port in device mode as the “gadget” framework. You can google for “linux kernel gadget”.

The Xavier should already come with this, at least on the sample. Go to “/opt/nvidia/l4t-usb-device-mode/”. The shell script will demonstrate bulk storage mode already…the file “filesystem.img” there is a loopback formatted ext4 file system and is what that script presents to a host when in device mode. The networking details should also be there (a USB port can be more than one device at once…some of those other devices might in fact cut back on bandwidth you can achieve by competing).

Bulk mode is essentially a hard drive. There will be burps if something blocks momentarily, but if your consumer and producers are actually living programs capable of producing and consuming at the same rate, then odds are quite good you’ll get much better average throughput.

Isochronous mode is not demonstrated in the “/opt/” scripts, and I don’t know how much support the gadget framework will add. Isochronous is typically for real time audio and video and reserves bandwidth for constant streaming. This is probably the most difficult mode to use, but depending on your data, it might be worth investigating.

15

@linuxdev

mass storage is faster than ethernet over usb. When reading from Xavier from Host, I get about 200MB/s rate. When writing to it, the speed I see is like 2-7GB/s. Guess writing number is not accurate, might just write to cache.

One trouble is when writing, I cannot see the new file on Xavier side if the fileimage.img already mounted. I can see it if I mount it again.

sudo mount /dev/loop0 mnt

Any suggestions on how to see new file on Xavier side?

16

USB does not determine the speed of ethernet. Ethernet speed is auto negotiated to a standard. You would have to know what link speed was negotiated. USB won’t change what can be negotiated, but if something faster than USB is negotiated, then it will fail. A dedicated USB ethernet dongle would be programmed to only allow link speed negotiation which the USB can support, but this would be a result of an engineer manually intervening and programming this in (USB itself will have no knowledge of whether the ethernet is negotiating something it can handle or not handle).

If you install package “ethtool”, then you can check link negotiation information:

sudo apt-get install ethtool
# Adjust for the ethernet interface used, eth0 is just an example:
sudo ethtool eth0

I did respond here with more details on the gadget mode bulk storage:
https://devtalk.nvidia.com/default/topic/1042037/jetson-agx-xavier/question-about-l4t-usb-device-mode-in-writable-mode/post/5285513/#5285513

As an additional note, gadget is able to directly use a partition. Gadget won’t know the difference between a real partition and a file which has been covered by loopback to look like a partition. The gadget mode bulk storage device is apparently using the partition directly without first mounting it. You should be able to edit nv-l4t-usb-device-mode.sh to use the obtained loopback device for mounting, but I have not tried. If you edit this script, and as an example if it finds the loopback device to cover with is “/dev/loop0”, then after losetup covers the device you could add a mount command which names “/dev/loop0” as the partition to be mounted.

Running “losetup -f” prior to the losetup command which actually attaches the loopback device would allow you to see what device is about to be attached. Once the step which attaches is complete, then you would add the mount command naming that device. I have not tried this, so I suppose it is possible there would be a complaint that the device is busy.

17

use ethtool to query eth0 shows lots of information

nvidia@jetson-0423318029348:/opt/nvidia/l4t-usb-device-mode$ sudo ethtool eth0
Settings for eth0:
Supported ports: [ TP MII ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Supported pause frame use: Symmetric Receive-only
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Advertised pause frame use: Symmetric Receive-only
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Link partner advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Link partner advertised pause frame use: Symmetric
Link partner advertised auto-negotiation: Yes
Link partner advertised FEC modes: Not reported
Speed: 1000Mb/s
Duplex: Full
Port: MII
PHYAD: 0
Transceiver: external
Auto-negotiation: on
Supports Wake-on: g
Wake-on: d
Link detected: yes

But use it on ethernet-over-usb created device shows
few information, cannot figure out the speed negotiated.

nvidia@jetson-0423318029348:/opt/nvidia/l4t-usb-device-mode$ sudo ethtool usb0
Settings for usb0:
Link detected: yes
nvidia@jetson-0423318029348:/opt/nvidia/l4t-usb-device-mode$ sudo ethtool rndis0
Settings for rndis0:
Link detected: yes
nvidia@jetson-0423318029348:/opt/nvidia/l4t-usb-device-mode$ sudo ethtool l4tbr0
Settings for l4tbr0:
Link detected: yes

nvidia@jetson-0423318029348:/opt/nvidia/l4t-usb-device-mode$ ifconfig [51/1977]
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.0.38 netmask 255.255.254.0 broadcast 192.168.1.255
inet6 2603:3023:123:17e0:c31f:c8e5:ada6:c523 prefixlen 64 scopeid 0x0
inet6 2603:3023:123:17e0:85e6:1c8d:b118:ad67 prefixlen 64 scopeid 0x0
inet6 fe80::703:df47:1b81:4127 prefixlen 64 scopeid 0x20
ether 00:04:4b:cb:9a:0d txqueuelen 1000 (Ethernet)
RX packets 2207728 bytes 1218852902 (1.2 GB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 811572 bytes 430145237 (430.1 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 40

l4tbr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.55.1 netmask 255.255.255.0 broadcast 192.168.55.255
inet6 fe80::e00c:2eff:fe52:a04a prefixlen 64 scopeid 0x20
inet6 fe80::1 prefixlen 128 scopeid 0x20
ether 46:88:0c:31:c5:34 txqueuelen 1000 (Ethernet)
RX packets 382443 bytes 535515719 (535.5 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 102252 bytes 5322530 (5.3 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10
loop txqueuelen 1 (Local Loopback)
RX packets 6986 bytes 560174 (560.1 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 6986 bytes 560174 (560.1 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

rndis0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::4488:cff:fe31:c534 prefixlen 64 scopeid 0x20
ether 46:88:0c:31:c5:34 txqueuelen 1000 (Ethernet)
RX packets 357835 bytes 531246147 (531.2 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 126938 bytes 16959821 (16.9 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

usb0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::48d8:5fff:fe79:1d45 prefixlen 64 scopeid 0x20
ether 4a:d8:5f:79:1d:45 txqueuelen 1000 (Ethernet)
RX packets 24608 bytes 4269572 (4.2 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 4351 bytes 831786 (831.7 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

18

“eth0” is just one example. On the Xavier itself the sample would show “usb0” for the USB version of ethernet. When a type-B cable is connected between Jetson and your PC host the naming just depends on your host’s conventions (udev rules and the number of connectors might change naming). You’d need to run any query of ethernet interface against the specific device as it appears from that computer.

From the Xavier this would imply:

sudo ethtool usb0

(eth0 is the wired RJ-45 interface)

From your PC host would be more interesting because it may have interacted with the USB ethernet. If you were to set up usb0 on the host PC, for example, to have an IP address, then the address would be missing prior to connecting to the PC, but exist after the PC sets up the address. What you are interested in here is the final link speed after the port is configured for use by the host (it would be possible for the Xavier itself to be a host and device).

I am guessing your USB ethernet device was not connected to the PC host and measured from the PC host since it had no information.

19

On the Xavier side, ifconfig shows the following devices:

eth0, l4tbr0, lo, rndis0, usb0

nvidia@jetson-0423318029348:~$ sudo ethtool usb0
Settings for usb0:
Link detected: yes

nvidia@jetson-0423318029348:~$ sudo ethtool -i usb0
driver: g_ether
version: 29-May-2008
firmware-version: tegra-xudc
expansion-rom-version:
bus-info: gadget
supports-statistics: no
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

nvidia@jetson-0423318029348:~$ sudo ethtool -k usb0
Features for usb0:
rx-checksumming: off [fixed]
tx-checksumming: off
tx-checksum-ipv4: off [fixed]
tx-checksum-ip-generic: off [fixed]
tx-checksum-ipv6: off [fixed]
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: off [fixed]
scatter-gather: off
tx-scatter-gather: off [fixed]
tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: off
tx-tcp-segmentation: off [fixed]
tx-tcp-ecn-segmentation: off [fixed]
tx-tcp-mangleid-segmentation: off [fixed]
tx-tcp6-segmentation: off [fixed]
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: off [requested on]
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: off [fixed]
tx-vlan-offload: off [fixed]
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: off [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-gre-csum-segmentation: off [fixed]
tx-ipxip4-segmentation: off [fixed]
tx-ipxip6-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-udp_tnl-csum-segmentation: off [fixed]
tx-gso-partial: off [fixed]
tx-sctp-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
busy-poll: off [fixed]
hw-tc-offload: off [fixed]

Lots of information, but no speed information.

My X86 hostside can see Xavier ethernet-over-usb device
enp0s20u5 Link encap:Ethernet HWaddr b2:97:71:58:f0:58
inet addr:192.168.55.2 Bcast:192.168.55.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:769 errors:0 dropped:0 overruns:0 frame:0
TX packets:709 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:128161 (128.1 KB) TX bytes:163850 (163.8 KB)

can ping 192.168.55.1 without any problem.

gtang@ws4:~/temp/Jetson_Linux/Linux_for_Tegra/nv_tegra$ sudo ethtool enp0s20u5
Settings for enp0s20u5:
Current message level: 0x00000007 (7)
drv probe link
Link detected: yes

gtang@ws4:~/temp/Jetson_Linux/Linux_for_Tegra/nv_tegra$ sudo ethtool -i enp0s20u5
driver: rndis_host
version: 22-Aug-2005
firmware-version: RNDIS device
expansion-rom-version:
bus-info: usb-0000:00:14.0-5
supports-statistics: no
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

gtang@ws4:~/temp/Jetson_Linux/Linux_for_Tegra/nv_tegra$ sudo ethtool -i enp0s20u5i5
driver: cdc_ether
version: 22-Aug-2005
firmware-version: CDC Ethernet Device
expansion-rom-version:
bus-info: usb-0000:00:14.0-5
supports-statistics: no
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

Is Nvidia Xavier’s ethernet-over-usb source code available somewhere in Linux kernel? I unzip kernel driver, can only find some binary codes related to usb

./Linux_for_Tegra/nv_tegra/lib/firmware/tegra18x_xusb_firmware
./Linux_for_Tegra/nv_tegra/lib/firmware/tegra19x_xusb_firmware

Would like to see Xavier ethernet-over-usb be able to reach 2.5G or 5G speed, otherwise might need to plugin a pcie multi-gige card or pcie card.

20

I am surprised to not see a link speed from the PC side, but probably should not be surprised. What you want to do might be possible, but I can’t guarantee it. I suspect it isn’t a limitation of the tegra19x_xusb_firmware, but instead a limit of the gadget interface. I’m not sure how a gadget ethernet device deals with speed when the link can’t advertise.

Just as an experiment, on the host PC side, what happens if you force a slower speed you know can work (I am assuming “enp0s20u5”, but adjust for how your PC sees it)?

ethtool -s enp0s20u5 speed 100 duplex full autoneg off

Note: Auto negotiation does not exist according to ethtool, so it was never advertised…disabling should be the same as saying “do what you are already doing”. You could leave off “autoneg off” (or any of the parameters) and test without. I am curious how the link deals with attempts to set a rate (any rate).