USB speed seems too low

Robotics & Edge Computing Jetson Systems Jetson AGX Orin
, , ,
1

I have a new Jetson AGX Orin 64 GB dev kit, and I just flashed it with the latest a couple of weeks ago using the SDK Manager so I have Jetpack 6.2.1. I’m running off of an NVMe.

I’m running into an issue trying to get 30 fps from 4 webcams (Logitech C920). I’m using ROS2 Humble with the ‘usb_cam’ driver. For some reason I am only getting 15 fps, though sometimes I’ll get 30 fps on one of the cameras. Rebooting doesn’t fix it. I am even only asking for 640x360 resolution. I realize it could be a number of things but I am confused about the USB. When I do ‘lsusb -t’ with no cameras plugged in (only a keyboard and mouse plugged into the USB A ports by the video output), I get:

/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=tegra-xusb/4p, 10000M
    |__ Port 3: Dev 2, If 0, Class=Hub, Driver=hub/4p, 10000M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=tegra-xusb/4p, 480M
    |__ Port 3: Dev 2, If 0, Class=Wireless, Driver=rtk_btusb, 12M
    |__ Port 3: Dev 2, If 1, Class=Wireless, Driver=rtk_btusb, 12M
    |__ Port 4: Dev 3, If 0, Class=Hub, Driver=hub/4p, 480M
        |__ Port 1: Dev 13, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
        |__ Port 1: Dev 13, If 1, Class=Human Interface Device, Driver=usbhid, 1.5M
        |__ Port 2: Dev 14, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M

I’m new at this but I interpret this to say that there are 2 root hubs, one is 10 Gbps (USB3) and the other is 480Mbps (USB2). There are 6 USB ports exposed on the dev kit (4 A, 2 C), and it doesn’t seem to matter what I plug in where, everything is going under Bus 01, the USB 2 one. What gives? Here’s what i see when I unplug all USB devices (including the keyboard and mouse – I’ll use a sleep to give me time to unplug) with cameras plugged into the USB A ports:

/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=tegra-xusb/4p, 10000M
    |__ Port 3: Dev 2, If 0, Class=Hub, Driver=hub/4p, 10000M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=tegra-xusb/4p, 480M
    |__ Port 3: Dev 2, If 0, Class=Wireless, Driver=rtk_btusb, 12M
    |__ Port 3: Dev 2, If 1, Class=Wireless, Driver=rtk_btusb, 12M
    |__ Port 4: Dev 3, If 0, Class=Hub, Driver=hub/4p, 480M
        |__ Port 3: Dev 15, If 3, Class=Audio, Driver=snd-usb-audio, 480M
        |__ Port 3: Dev 15, If 1, Class=Video, Driver=uvcvideo, 480M
        |__ Port 3: Dev 15, If 2, Class=Audio, Driver=snd-usb-audio, 480M
        |__ Port 3: Dev 15, If 0, Class=Video, Driver=uvcvideo, 480M
        |__ Port 1: Dev 21, If 2, Class=Audio, Driver=snd-usb-audio, 480M
        |__ Port 1: Dev 21, If 0, Class=Video, Driver=uvcvideo, 480M
        |__ Port 1: Dev 21, If 3, Class=Audio, Driver=snd-usb-audio, 480M
        |__ Port 1: Dev 21, If 1, Class=Video, Driver=uvcvideo, 480M
        |__ Port 4: Dev 16, If 0, Class=Video, Driver=uvcvideo, 480M
        |__ Port 4: Dev 16, If 3, Class=Audio, Driver=snd-usb-audio, 480M
        |__ Port 4: Dev 16, If 1, Class=Video, Driver=uvcvideo, 480M
        |__ Port 4: Dev 16, If 2, Class=Audio, Driver=snd-usb-audio, 480M
        |__ Port 2: Dev 22, If 2, Class=Audio, Driver=snd-usb-audio, 480M
        |__ Port 2: Dev 22, If 0, Class=Video, Driver=uvcvideo, 480M
        |__ Port 2: Dev 22, If 3, Class=Audio, Driver=snd-usb-audio, 480M
        |__ Port 2: Dev 22, If 1, Class=Video, Driver=uvcvideo, 480M

Here’s what I get when I plug 3 of the cameras into a USB3 type C hub that also breaks out HDMI (which I understand does not work on the dev kit), ethernet, and a microSD and SD card reader (keyboard and mouse are plugged into two of the USB A ports):

/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=tegra-xusb/4p, 10000M
    |__ Port 2: Dev 8, If 0, Class=Hub, Driver=hub/4p, 5000M
        |__ Port 4: Dev 10, If 0, Class=Mass Storage, Driver=usb-storage, 5000M
        |__ Port 2: Dev 9, If 0, Class=Hub, Driver=hub/4p, 5000M
            |__ Port 1: Dev 11, If 0, Class=Vendor Specific Class, Driver=r8152, 5000M
    |__ Port 3: Dev 2, If 0, Class=Hub, Driver=hub/4p, 10000M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=tegra-xusb/4p, 480M
    |__ Port 1: Dev 25, If 0, Class=Hub, Driver=hub/5p, 480M
        |__ Port 5: Dev 27, If 0, Class=, Driver=, 480M
        |__ Port 2: Dev 26, If 0, Class=Hub, Driver=hub/4p, 480M
            |__ Port 3: Dev 29, If 3, Class=Audio, Driver=snd-usb-audio, 480M
            |__ Port 3: Dev 29, If 1, Class=Video, Driver=uvcvideo, 480M
            |__ Port 3: Dev 29, If 2, Class=Audio, Driver=snd-usb-audio, 480M
            |__ Port 3: Dev 29, If 0, Class=Video, Driver=uvcvideo, 480M
            |__ Port 4: Dev 28, If 0, Class=Video, Driver=uvcvideo, 480M
            |__ Port 4: Dev 28, If 3, Class=Audio, Driver=snd-usb-audio, 480M
            |__ Port 4: Dev 28, If 1, Class=Video, Driver=uvcvideo, 480M
            |__ Port 4: Dev 28, If 2, Class=Audio, Driver=snd-usb-audio, 480M
            |__ Port 2: Dev 30, If 1, Class=Video, Driver=uvcvideo, 480M
            |__ Port 2: Dev 30, If 2, Class=Audio, Driver=snd-usb-audio, 480M
            |__ Port 2: Dev 30, If 0, Class=Video, Driver=uvcvideo, 480M
            |__ Port 2: Dev 30, If 3, Class=Audio, Driver=snd-usb-audio, 480M
    |__ Port 3: Dev 2, If 0, Class=Wireless, Driver=rtk_btusb, 12M
    |__ Port 3: Dev 2, If 1, Class=Wireless, Driver=rtk_btusb, 12M
    |__ Port 4: Dev 3, If 0, Class=Hub, Driver=hub/4p, 480M
        |__ Port 3: Dev 31, If 3, Class=Audio, Driver=snd-usb-audio, 480M
        |__ Port 3: Dev 31, If 1, Class=Video, Driver=uvcvideo, 480M
        |__ Port 3: Dev 31, If 2, Class=Audio, Driver=snd-usb-audio, 480M
        |__ Port 3: Dev 31, If 0, Class=Video, Driver=uvcvideo, 480M
        |__ Port 1: Dev 23, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
        |__ Port 2: Dev 24, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
        |__ Port 2: Dev 24, If 1, Class=Human Interface Device, Driver=usbhid, 1.5M

I have also tried plugging one camera into a USBA / USBC adapter and using both of the USBC ports, and I never get a camera showing up anywhere besides Bus 01, stuck at 480 Mbps.

Any ideas? I am most likely just misunderstanding something, but I had the same issue on a Jetson Orin Nano dev kit recently, and after spending a bunch of time basically just swapping ports I eventually got it to log all 4 cameras at 30 fps… but when I went back and tried to replicate that I couldn’t. (And when it was working I didn’t think to check lsusb to see if it was any different.) So confused. I don’t have a powered USB hub I could try atm. Anyway. Thanks for any suggestions / insight!

2

*** Please note that this reply is generated by LLM automatically ***

The issue you’re experiencing with your Jetson AGX Orin 64 GB dev kit, specifically with achieving 30 fps from 4 Logitech C920 webcams, involves several factors including hardware, software, and configuration. Let’s break down the key points and potential solutions:

  1. Hardware and USB Configuration:

    • Your Jetson AGX Orin has multiple USB ports, including USB-A and USB-C ports. The lsusb -t command helps in understanding the USB topology, showing which devices are connected to which bus and at what speed.
    • The output indicates that all devices, including your cameras, are connected to the USB 2.0 bus (480 Mbps), which could be a bottleneck for achieving high frame rates, especially with multiple cameras.
    • Typically, USB 3.0 (or higher) is required for high-bandwidth applications like multiple camera feeds at high resolutions and frame rates.

  2. Camera and Driver Configuration:

    • You’re using the usbcam driver with ROS2 Humble. Ensure that the driver is properly configured for your cameras and that it supports the frame rate and resolution you’re aiming for.
    • The V4L2 driver, which is part of the camera software development solution, provides a standard interface for video capture devices. It might be useful to explore V4L2 configurations or utilities to check the camera’s capabilities and current settings.

  3. Bandwidth and Resource Limitations:

    • Even if you’re only asking for 640x360 resolution, the aggregate bandwidth required for four cameras at 30 fps could still be significant, potentially exceeding the capabilities of a single USB 2.0 bus.
    • Consider the bandwidth requirements of your application and whether the current hardware configuration can support it.

  4. Potential Solutions and Workarounds:

    • Use of a Powered USB Hub: While you’ve mentioned not having a powered USB hub, using one could potentially help, especially if it supports USB 3.0 or higher speeds. This could allow you to distribute the cameras across different buses, potentially alleviating bandwidth limitations.
    • Camera Configuration and Optimization: Review the camera settings and the usbcam driver configuration to ensure they are optimized for your application. This might involve reducing the resolution or frame rate of some cameras if not all need to run at the highest settings.
    • Alternative Connection Methods: If possible, explore using different connection methods for your cameras, such as Ethernet cameras or cameras with direct CSI (Camera Serial Interface) connections, which might offer better performance and less dependency on USB bandwidth.
    • Jetson AGX Orin Specific Configurations: Look into Jetson-specific configurations or optimizations that might help in managing USB bandwidth or improving camera performance. NVIDIA provides various tools and guides for optimizing performance on Jetson devices.

  5. Comparison with Previous Experience:

    • Your experience with the Jetson Orin Nano dev kit suggests that there might be specific configurations or port combinations that can achieve the desired performance. While it’s frustrating that you couldn’t replicate the success, it indicates that the hardware is capable of handling the task under the right conditions.

  6. Next Steps:

    • Experiment with Different Configurations: Try different camera configurations, such as reducing the frame rate or resolution, to see if that improves performance.
    • Investigate USB Hub Options: Consider acquiring a powered USB 3.0 hub to distribute the cameras across faster buses.
    • Consult NVIDIA and ROS2 Documentation: Look for specific guides or forums related to NVIDIA Jetson and ROS2 for camera applications. There might be community-provided solutions or optimizations that can help.

In summary, your issue involves a combination of hardware limitations, particularly the USB bandwidth, and potentially the software configuration of your cameras and drivers. Experimenting with different configurations, exploring optimizations specific to the Jetson AGX Orin and ROS2, and considering hardware upgrades like a powered USB 3.0 hub could help in achieving your desired frame rates.

*** Please give the thumbs up if you get answers you like. Or provide feedback to help us improve the answer. ***

3

Is your AGX Orin running on a third party carrier board? Or is it on a developer’s kit carrier board? If flashed with the wrong device tree for a given carrier board, then it is possible the issue is because of the device tree (but not necessarily). The fact that the USB3 root HUB shows up tends to mean the device tree new about the HUB, but what it doesn’t prove is that the wiring and routing of pins from the Jetson to the USB3 pins of the PHY are valid. Perhaps the device tree has wires specified as going to empty space rather than the USB3 pins.

That said, some history might be of interest. When USB developed from 1.0 to 1.1 and then to 2.0 a single controller chip was introduced each time, and that controller understood both legacy and newer protocols. With USB3 this changes, and the USB3 root HUB handles only USB3. Detection of USB legacy versus USB3 changes PHY routing to the correct root HUB.

If signal wires are missing (such as from device tree issues), then USB3 won’t be detected and it’ll remain USB2. If the wires do exist, but signal quality or power requirements cannot be met, then it’ll route to USB2 or slower. For devices trying to use isochronous mode there are further requirements; isochronous tries to reserve a specific bandwidth for real time cases; sometimes cameras use bullk mode, and in a few cases they try to use isochronous mode. Any camera which demands isochronous mode when bandwidth is unavailable will get demoted to a slower speed if supported (some cameras do not support being demoted to slower speeds and will simply disappear from the lsusb listing).

4

This is an official 64 GB AGX developer kit. I believe I selected everything correctly in the SDK Manager – if you can think of anything I can check, I can give it a try.

Update: I had a colleague check on his AGX dev kit and he’s seeing the same thing when he types in ‘lsusb -t’ – one bus running at 10Gbps, the other at 480Mbps. His Jetson was probably flashed a year or more ago so it seems like it’s not a Jetpack version issue.

6

A dev kit would use the default device tree. So it is most likely the device tree is valid and not the cause.

If you use a single camera, and you try each camera one at a time after boot is complete, how often does it show as USB3 versus USB2?

Are these cameras powered by the USB cable, or do the cameras have their own power source? Note that when self-powered a device has a higher power delivery available versus when running power from the USB cable. If two or more devices are powered from the cable, then all combined devices cannot exceed that power rating. This can affect multiple USB devices, and might cause some failures which are obvious, or even some less obvious failures.

The USB cable type also can have an effect on which USB3 revision you get, and in turn, how much power is available. Can you describe any/all cables and HUBs involved? USB-C is the best case, and going through a powered HUB is also likely the best case; direct connection to the USB port by a self-powered device is also a good test case. If it is a type-A cable and not a type-C cable, then you want to verify it has all of the requirements for USB3 (extra connector tabs; otherwise it is incapable of USB3 modes).

If you can find a case where a device ever runs as USB3, then you know the firmware is working and the ports are correctly routing. It doesn’t even have to be the camera, you could also find some other USB3 device to find a single case of something working.

Having this working on his Jetson versus yours with the same cameras/HUBs/cables is a good test. You could get a copy of these files on his Jetson and yours for comparison, and post them here (make sure to label if they come from the working versus failing Jetson case):

  • /etc/nv_tegra_release
  • /etc/nv_boot_control.conf
  • uname -r
7

Thanks for following up. I think I have found the USB2/3 issue, and it seems to come from my own confusion about how the USB “tree” works. I unplugged all but 1 of the C920 webcams and then also plugged in a high quality USB3 machine vision camera from Point Grey that I happened to have – it is a good camera with a high quality USB3 cable. It doesn’t come up as a webcam (needs its own drivers etc) and is too expensive for me to get 4 of but it helped me debug.

When I plug a USB2 camera into a specific port on the AGX dev kit – let’s say the USB A connector on the top right if you’re looking at the 40-pin connector – then it appears on the USB2 bus. If you disconnect that and then plug in a USB3 camera into the exact same spot, it shows up on the USB3 bus.

Here’s the USB2 camera plugged into the top-right USB port when looking at the 40-pin connector, you can see it appears on the USB2 bus (Bus 01):

/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=tegra-xusb/4p, 10000M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 5000M
        |__ Port 2: Dev 4, If 0, Class=Hub, Driver=hub/4p, 5000M
            |__ Port 1: Dev 6, If 0, Class=Vendor Specific Class, Driver=r8152, 5000M
        |__ Port 4: Dev 5, If 0, Class=Mass Storage, Driver=usb-storage, 5000M
    |__ Port 3: Dev 3, If 0, Class=Hub, Driver=hub/4p, 10000M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=tegra-xusb/4p, 480M
    |__ Port 2: Dev 2, If 0, Class=Hub, Driver=hub/5p, 480M
        |__ Port 2: Dev 5, If 0, Class=Hub, Driver=hub/4p, 480M
            |__ Port 4: Dev 9, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
            |__ Port 3: Dev 13, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
            |__ Port 3: Dev 13, If 1, Class=Human Interface Device, Driver=usbhid, 1.5M
        |__ Port 5: Dev 6, If 0, Class=, Driver=, 480M
    |__ Port 3: Dev 3, If 0, Class=Wireless, Driver=rtk_btusb, 12M
    |__ Port 3: Dev 3, If 1, Class=Wireless, Driver=rtk_btusb, 12M
    |__ Port 4: Dev 4, If 0, Class=Hub, Driver=hub/4p, 480M
        |__ Port 4: Dev 12, If 0, Class=Video, Driver=uvcvideo, 480M
        |__ Port 4: Dev 12, If 3, Class=Audio, Driver=snd-usb-audio, 480M
        |__ Port 4: Dev 12, If 1, Class=Video, Driver=uvcvideo, 480M
        |__ Port 4: Dev 12, If 2, Class=Audio, Driver=snd-usb-audio, 480M

And here is the exact same setup but I’ve swapped the USB2 camera for a USB3 camera, and it appears on the USB3 bus (Bus 02):

/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=tegra-xusb/4p, 10000M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 5000M
        |__ Port 2: Dev 4, If 0, Class=Hub, Driver=hub/4p, 5000M
            |__ Port 1: Dev 6, If 0, Class=Vendor Specific Class, Driver=r8152, 5000M
        |__ Port 4: Dev 5, If 0, Class=Mass Storage, Driver=usb-storage, 5000M
    |__ Port 3: Dev 3, If 0, Class=Hub, Driver=hub/4p, 10000M
        |__ Port 4: Dev 8, If 0, Class=Miscellaneous Device, Driver=, 5000M
        |__ Port 4: Dev 8, If 1, Class=Miscellaneous Device, Driver=, 5000M
        |__ Port 4: Dev 8, If 2, Class=Miscellaneous Device, Driver=, 5000M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=tegra-xusb/4p, 480M
    |__ Port 2: Dev 2, If 0, Class=Hub, Driver=hub/5p, 480M
        |__ Port 2: Dev 5, If 0, Class=Hub, Driver=hub/4p, 480M
            |__ Port 4: Dev 9, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
            |__ Port 3: Dev 13, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
            |__ Port 3: Dev 13, If 1, Class=Human Interface Device, Driver=usbhid, 1.5M
        |__ Port 5: Dev 6, If 0, Class=, Driver=, 480M
    |__ Port 3: Dev 3, If 0, Class=Wireless, Driver=rtk_btusb, 12M
    |__ Port 3: Dev 3, If 1, Class=Wireless, Driver=rtk_btusb, 12M
    |__ Port 4: Dev 4, If 0, Class=Hub, Driver=hub/4p, 480M

So I was expecting the tree to be more representative of the physical layout – e.g., each physical USB connector maps to one of the USB buses, and since all 6 of the physical connectors support USB3, I expected anything plugged into them to appear on that bus.

As an aside, after fighting with trying to get all 4 cameras to work at 30 fps and only seeing 15, I finally gave up, finished securing everything, and went out to do my flight testing… and lo and behold, all 4 cameras were working at the full 30 fps, every time, like clockwork. I made no software changes – the only thing I can think of is that I secured the cables down a little bit more tightly and possibly a bit more neatly to keep them from flopping around in flight. All I can figure is it was an EMI issue related to where the cables were. Note that I coiled all cables “over-under” before securing them which at least in theory should help somewhat with EMI (noise induced on one winding is canceled out by the same noise being induced on the next winding, it has the opposite polarity because the coil is flipped the other way around), though I think and it’s possible DisplayPort cable was routed slightly differently. Over-under coiling the cables was not sufficient though, it seemed to be when I just tidied up and tightened down all the cables a bit.

Thanks again for your help on this wild goose chase.

9

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.