Insufficient bandwidth when using multiple USB cameras

Beware that the 10Gb/s specification is only for theoretical raw bit rate. What you actually get after overhead (such as any encoding) is significantly less. You might find an actual throughput of 3.5 Gb/s on a 5 Gb/s specification is near 100% of capacity. If a camera says it uses USB3.1 gen. 1, and thus is marked in “lsusb -t” as “5000M”, then expect actual bandwidth to be significantly less than “5000M”. However, if one camera uses 5000M spec, then two 5000M spec cameras on one 10000M connection should work.

However, data also must pass through some path internal to the Jetson. Let’s say you got that PCIe USB card, which advertises (due to number of ports) 20 Gb/s. If all of that is handled by a single CPU core, then the CPU core itself might bottleneck. Frames might still be lost, but the cameras and USB side should be ok for up to 4 cameras listed as 5000M, or 2 cameras listed as 10000M.

The reason such a bottleneck is important is that most of the hardware interrupts (which trigger a driver based on a hard wire to the CPU and not based on a timer…the wire must be able to physically touch the CPU pins) will be forced through the first CPU core (call it CPU0). On an Intel based x86 PC there is a device called the I/O APIC (Asynchronous Programmable Interrupt Controller; AMD has a different mechanism, but same idea). That I/O APIC is able to send hardware interrupts to any CPU core. Without this mechanism only CPU0 can handle hardware IRQs (take a look at what IRQs show up in “/proc/interrupt”…other than timers used for software IRQs (which don’t use a hard wire), most of this must run through CPU0).

A good practice in hardware drivers is to handle only the hardware IRQ content as minimally as possible, and then create a software IRQ for parts which don’t directly need to talk to the hardware. There is no requirement for this though. Even if you do such a thing, there is a high likelihood that you’ll end up with “interrupt starvation” if you put a 20 Gb/s PCIe card in (I am assuming that this PCIe slot cannot randomly route items on it…each USB end point…to independent CPUs…likely it all goes to CPU0).

Add to this that there are other hardware IRQs being handled at the same time, e.g., ethernet.

Should the work load through that 20 Gb/s card manage to use DMA such that the first CPU core only touches a brief part of that data (and thus does not load down CPU0), then it might still work. I guess all of this is to say it isn’t as simple as getting a bigger USB HUB, but without that (such as the PCIe card) there isn’t much chance of using more cameras on a single USB root_hub. You have to try it to know for sure, but if you want 8 cameras at USB3.1 gen. 1, and use the 20 Gb/s PCIe-based HUB, it still won’t be enough if all cameras send at the same time without a multiplex scheme.

1 Like