Insufficient bandwidth when using multiple USB cameras

Autonomous Machines Jetson & Embedded Systems Jetson AGX Orin
, ,
1

Hello,
I`m trying to use 3 Basler Dart Cameras USB3 (Basler dart daA3840-45uc (S-Mount) - Area Scan Camera) simultaneously with a Orin developer kit.

I run the following command:
gst-launch-1.0 pylonsrc device-index=0 cam::ExposureTime=8333 cam::Gain=1.3 ! video/x-raw, width=3860, height=2178, framerate=30/1 ! queue ! videoconvert ! autovideosink

It runs fine, than in other terminal I run the same command just changing ‘device-index’ = 1, and after when I try to run the tird camera with the same command and ‘device-index’ = 2. I receive the following error in the first camera that I started:

ERROR: from element /GstPipeline:pipeline0/GstPylonSrc:pylonsrc0: Failed to create buffer.
Additional debug info:
…/ext/pylon/gstpylonsrc.c(841): gst_pylon_src_create (): /GstPipeline:pipeline0/GstPylonSrc:pylonsrc0:
Payload data has been discarded. Payload data can be discarded by the camera device if the available bandwidth is insufficient.
Execution ended after 0:00:53.543345529
Setting pipeline to NULL …
Freeing pipeline …

I woud like to have some insight into why is this happening, is the USB board limiting the bandwidth? Do I have an inneficient gstreamer pipeline?

Thank you!

3

Hi,
There are 4 USB3 type-A ports on Orin developer kit. Do you connect one camera to one port?

4

Yes, each camera is connected to a different usb port.

5

Something to consider: Use “lsusb -t” to see the “tree” view of how USB devices are connected. If two or more cameras end up at the same “root_hub”, then even if they appear to be using different ports, they will be competing with each other for bandwidth.

At the end of each line in the tree view of lsusb you will see something like this:

  • 480M (meaning USB2)
  • 5000M (meaning USB3.1 gen. 1)
  • 1000)M (USB3.1 gen. 2 or USB3.2)

(there are others for USB1.0 and USB1.1, e.g., keyboards and mice)

Up through USB2 a single controller would handle requests for older standards, e.g., the USB2 root_hub would handle cameras in USB2 mode, plus keyboards and mice in USB1.0 or USB1.1 mode. With USB3+ the older content (USB2 or slower) gets routed to their own controller for USB2 or slower, and the USB3 root_hub is dedicated. In the case of USB3 devices, even if they use the same HUB which runs keyboards/mice/slower items, the USB3 device routes to a dedicated USB3 controller. Meaning your tree view, depending on speed of attached devices, might go to multiple controllers. Or you might find USB3 devices on different ports route to one USB3 controller even though the physical ports differ, depending on what is available and how things are arranged (so I suggest actually making sure that when you think things are on different ports they use a different root_hub as well; different ports does mean different power delivery, so there is still a use for different ports even if they route to and share the same root_hub).

6

lsusb

Bus 002 Device 006: ID 2676:ba06 Basler AG 
Bus 002 Device 005: ID 2676:ba06 Basler AG 4-Port USB 3.0 Hub
Bus 002 Device 004: ID 2676:ba06 Basler AG 
Bus 002 Device 003: ID 2676:ba06 Basler AG 
Bus 002 Device 002: ID 0bda:0420 Realtek Semiconductor Corp. 4-Port USB 3.0 Hub
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 003: ID 0bda:5420 Realtek Semiconductor Corp. 4-Port USB 2.0 Hub
Bus 001 Device 002: ID 13d3:3549 IMC Networks Bluetooth Radio
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

lsusb -t

/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=tegra-xusb/4p, 10000M
    |__ Port 3: Dev 2, If 0, Class=Hub, Driver=hub/4p, 10000M
        |__ Port 1: Dev 3, If 2, Class=Miscellaneous Device, Driver=, 5000M
        |__ Port 1: Dev 3, If 0, Class=Miscellaneous Device, Driver=, 5000M
        |__ Port 1: Dev 3, If 1, Class=Miscellaneous Device, Driver=, 5000M
        |__ Port 2: Dev 4, If 1, Class=Miscellaneous Device, Driver=, 5000M
        |__ Port 2: Dev 4, If 2, Class=Miscellaneous Device, Driver=, 5000M
        |__ Port 2: Dev 4, If 0, Class=Miscellaneous Device, Driver=, 5000M
        |__ Port 3: Dev 5, If 0, Class=Miscellaneous Device, Driver=usbfs, 5000M
        |__ Port 3: Dev 5, If 1, Class=Miscellaneous Device, Driver=usbfs, 5000M
        |__ Port 3: Dev 5, If 2, Class=Miscellaneous Device, Driver=usbfs, 5000M
        |__ Port 4: Dev 6, If 0, Class=Miscellaneous Device, Driver=, 5000M
        |__ Port 4: Dev 6, If 1, Class=Miscellaneous Device, Driver=, 5000M
        |__ Port 4: Dev 6, If 2, Class=Miscellaneous Device, Driver=, 5000M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=tegra-xusb/4p, 480M
    |__ Port 3: Dev 2, If 0, Class=Wireless, Driver=rtk_btusb, 12M
    |__ Port 3: Dev 2, If 1, Class=Wireless, Driver=rtk_btusb, 12M
    |__ Port 4: Dev 3, If 0, Class=Hub, Driver=hub/4p, 480M

Please find attached the file with ‘lsusb -v’ for more info.
Other information I find could be relevant is ‘dmesg’ (also attached) here some parts:

[   11.787907] usb 2-3.2: new SuperSpeed Gen 1 USB device number 4 using tegra-xusb
[   11.819671] usb 2-3.2: New USB device found, idVendor=2676, idProduct=ba06, bcdDevice= 0.00
[   11.828264] usb 2-3.2: New USB device strings: Mfr=1, Product=2, SerialNumber=4
[   11.835793] usb 2-3.2: Product: daA3840-45uc
[   11.840199] usb 2-3.2: Manufacturer: Basler
[   11.844497] usb 2-3.2: SerialNumber: 40162469

This above is repeated for all cameras.

When I run the commands to run the cameras:

[  996.007541] capability: warning: gst-launch-1.0 uses 32-bit capabilities (legacy support in use)
[ 1106.343806] usb 2-3.4: usbfs: interface 0 claimed by usbfs while gst-launch-1.0 sets config #1

dmesg.txt (78.9 KB)
lsusb.txt (38.0 KB)

7

Note that in your tree view you could have two devices at 5000M for a single 10000M root_hub and it would work without significant issue. As soon as you add another device at 5000M you are far beyond the capability of the root_hub. I see 12 such devices, which is about 6 times the available bandwidth. The only way this will work is if the cameras are round-robin, and don’t transmit simultaneously, e.g., a multiplex scheme (though I don’t know if that would be possible without some method to independently pause and restart devices).

Technically though, this should not cause a device to not show up as existing at the PHY. It would result in all kinds of errors talking to the device, or else lots of dropped data, but I’d think lsusb would always show the devices (the exception is if there is a signal quality issue, which is different than dropping data due to lack of bus availability).

8

Hi,
The 4 posts are from single rootport so it is possible to hit bandwidth constraint for connecting to multiple cameras. It is similar to Jetson Nano/Xavier NX developer kit as discussed in
connected more than two usb cameras problem on deepstream-app (Jetson Nano Dev Kit) - #12 by DaneLLL

9

Ok, let me check if I understand, I have:

3860(width)*2178(height)*24(pixel depth)*15(fps) = 3,026,548,800 = 3 Gb/s
for the third camera I will hit 9 Gb/s that is very close to the limit of 10, this is why I’m getting this error?

If I buy a piece similar to this usb card, and assembly it on the Orin Dev Kit, it should solve my problems, right?

10

Hi,
The camera driver may request for more bandwidth. You may try this method and see if it works:
Failed to allocate memory for four usb cameras - #4 by fabian.solano

11

Beware that the 10Gb/s specification is only for theoretical raw bit rate. What you actually get after overhead (such as any encoding) is significantly less. You might find an actual throughput of 3.5 Gb/s on a 5 Gb/s specification is near 100% of capacity. If a camera says it uses USB3.1 gen. 1, and thus is marked in “lsusb -t” as “5000M”, then expect actual bandwidth to be significantly less than “5000M”. However, if one camera uses 5000M spec, then two 5000M spec cameras on one 10000M connection should work.

However, data also must pass through some path internal to the Jetson. Let’s say you got that PCIe USB card, which advertises (due to number of ports) 20 Gb/s. If all of that is handled by a single CPU core, then the CPU core itself might bottleneck. Frames might still be lost, but the cameras and USB side should be ok for up to 4 cameras listed as 5000M, or 2 cameras listed as 10000M.

The reason such a bottleneck is important is that most of the hardware interrupts (which trigger a driver based on a hard wire to the CPU and not based on a timer…the wire must be able to physically touch the CPU pins) will be forced through the first CPU core (call it CPU0). On an Intel based x86 PC there is a device called the I/O APIC (Asynchronous Programmable Interrupt Controller; AMD has a different mechanism, but same idea). That I/O APIC is able to send hardware interrupts to any CPU core. Without this mechanism only CPU0 can handle hardware IRQs (take a look at what IRQs show up in “/proc/interrupt”…other than timers used for software IRQs (which don’t use a hard wire), most of this must run through CPU0).

A good practice in hardware drivers is to handle only the hardware IRQ content as minimally as possible, and then create a software IRQ for parts which don’t directly need to talk to the hardware. There is no requirement for this though. Even if you do such a thing, there is a high likelihood that you’ll end up with “interrupt starvation” if you put a 20 Gb/s PCIe card in (I am assuming that this PCIe slot cannot randomly route items on it…each USB end point…to independent CPUs…likely it all goes to CPU0).

Add to this that there are other hardware IRQs being handled at the same time, e.g., ethernet.

Should the work load through that 20 Gb/s card manage to use DMA such that the first CPU core only touches a brief part of that data (and thus does not load down CPU0), then it might still work. I guess all of this is to say it isn’t as simple as getting a bigger USB HUB, but without that (such as the PCIe card) there isn’t much chance of using more cameras on a single USB root_hub. You have to try it to know for sure, but if you want 8 cameras at USB3.1 gen. 1, and use the 20 Gb/s PCIe-based HUB, it still won’t be enough if all cameras send at the same time without a multiplex scheme.

1 Like
13

@DaneLLL the patch didn’t work, I think it’s because the files in Jetson Linux 35.1 are different or due to the fact I’m using this pipeline:

" gst-launch-1.0 pylonsrc device-index=0 cam::ExposureTime=8333 cam::Gain=1.3 ! video/x-raw, width=3860, height=2178, framerate=30/1 ! queue ! videoconvert ! autovideosink"

@linuxdev Thanks for all the information, I think I will proceed and buy a bigger USB
HUB and see what happens, I need 4 cameras at 3860x2178, I can drop framerate to 15, I hope this solves my problem.

14

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.