Stream 4 Cameras with GStreamer

g.oro · May 6, 2020, 5:29pm

Hello,
i’m recently working on a TX2 to implement a 4 cameras streaming. Since the USB single Bus of the TX2 does not support the bandwidth of 4 FullHD cameras, I bought a 4 channel/4 USB controllers PCIe - USB hub (https://www.amazon.com/Express-SuperSpeed-Adapter-Dedicated-Channels/dp/B00HJZEA2S/ref=sr_1_fkmr0_1?ie=UTF8&qid=1512205237&sr=8-1-fkmr0&keywords=startech+hub+usb+pci) to connect all my cameras to the Jetson.
I power the Hub, driver are correctly mounted by the system and I can see all my cameras using the

lsusb

command.
At this point I create a Bash script which start the stream of the each camera on a different port:

v4l2src

to choose the device

omxh265enc

encoding to encode the video

udpsink

to strem the video at a specific IP.

If I start the script all seems to work well, since I receive perfectly all 4 cameras on my host. However, after some seconds, three video freeze and I cannot get any new frame from 3 on 4 cameras.
Initally I thought in performance issue but

htop

shows my only 30/40% of CPU usage.
I suppose also a host problem, but even if I stream 4 cameras from the Jetson and decode only 2 of them on my host, these freeze.
No Script issue since the same problem happens if I run 4 separate Gstreamer commands.

Anyone have idea of how resolve this?
Thanks in advance

snarky · February 1, 2020, 5:53am

Sounds like a driver bug, where some interrupt gets lost or somesuch.
If you start this, and then look at where the three cameras are stuck, what does it say?
Which system call? Can you get kernel stack traces?

DaneLLL · February 1, 2020, 5:53am

Thank @snarky for the suggestion.

Hi g.oro, can you also try 2 or 3 cameras?

g.oro · May 6, 2020, 5:29pm

Hi, thanks for responses

@DaneLLL With 2 cameras connected to the PCIe HUB same problem: frames stuck after some seconds. The Same does NOT happen with a regular USB HUB

@snarky using dmesg --follow I can see repeating the same msg:
“xhci_hcd 0000:05:00.0: WARN Successful completion on short TX: needs XHCI_TRUST_TX_LENGTH quirk?”
and it repeats until frames stuck

Connecting the same two cameras to the regular USB HUB this message not appears and the streaming works well. However, I need to make it works on the PCIe HUB

Thanks again

vidyas · February 1, 2020, 5:54am

Is it possible to share full log when the freeze happens?

DaneLLL · May 6, 2020, 5:30pm

Hi g.oro,
Do you run with external power connected?

https://www.amazon.com/Express-SuperSpeed-Adapter-Dedicated-Channels/dp/B00HJZEA2S/ref=sr_1_fkmr0_1?ie=UTF8&qid=1512205237&sr=8-1-fkmr0&keywords=startech+hub+usb+pci
It says:

Power high-powered USB devices as necessary, with the optional LP4 or SATA power connector

Ravik · February 1, 2020, 5:54am

Alternatively you can use

https://www.euresys.com/products/frame-grabbers/

DaneLLL · February 29, 2020, 1:18am

Hi g.oro,
PLease try the patch:

diff --git a/drivers/usb/host/xhci-pci.c b/drivers/usb/host/xhci-pci.c
index cf147cc..c0551d4 100644
--- a/drivers/usb/host/xhci-pci.c
+++ b/drivers/usb/host/xhci-pci.c
@@ -192,6 +192,8 @@ static void xhci_pci_quirks(struct device *dev, struct xhci_hcd *xhci)
        if (xhci->quirks & XHCI_RESET_ON_RESUME)
                xhci_dbg_trace(xhci, trace_xhci_dbg_quirks,
                                "QUIRK: Resetting on resume");
+
+    xhci->quirks |= XHCI_TRUST_TX_LENGTH;
 }

 #ifdef CONFIG_ACPI

It is suggested in https://github.com/Grawp/rtl8812au_rtl8821au/issues/28#issuecomment-249651461

g.oro · February 1, 2020, 5:55am

It works!
The patch seems fix the problem! Thanks a lot DaneLLL!

crr · February 1, 2020, 5:55am

Hello g. oro.

I have the same problem. See the end of this thread: https://devtalk.nvidia.com/default/topic/1026623/jetson-tx2/pcie-x4-only-658mb-s/?offset=8#5225811

Have you found a solution to run 4 USB cameras on a PCIe card?

sf2584 · February 4, 2020, 6:47pm

Hi, I met the same problem here. Can you show me how to install the patch?

DaneLLL · February 29, 2020, 1:43am

You need to build kernel by yourself. Please refer to https://developer.nvidia.com/embedded/dlc/l4t-documentation-28-1

sf2584 · February 29, 2020, 1:18am

Thanks, I took a look at the documentation, but still couldn’t figure out. Are you saying that I should build a external kernel modules? How can I apply the patch? I’m really new to this. Can you give me some implementation procedures to elaborate this? Thanks in advance!

carolyuu · May 6, 2020, 5:31pm

Hi sf2584,

You can check document steps from: Kernel Customization -> Building External Kernel Modules

You can refer this topic:
https://devtalk.nvidia.com/default/topic/1014729/jetson-tx2/how-to-compile-the-tx2-l4t-kernel-source-/post/5172142/#5172142

Download r28.1 source from: https://developer.nvidia.com/embedded/dlc/l4t-sources-28-1
Download toolchain from: https://developer.nvidia.com/embedded/dlc/l4t-gcc-toolchain-64-bit-28-1

CHuang1 · February 1, 2020, 6:13am

RaviKiranK,
Thanks for sharing frame grabber card for Jetson TX2.

logidelic · February 1, 2020, 6:29am

Is this still an issue with 28.2 ? I’m running into issues with limited USB transfer size on the TX2 and wondering if this is related to this issue…

vidyas · February 1, 2020, 6:29am

Did you already try patch mentioned in comment #8 ?

joe-dev · May 6, 2020, 5:39pm

@logidelic, I was hitting the limit of the USB2.0 bus as well. So based on this thread purchased https://www.amazon.com/Express-SuperSpeed-Adapter-Dedicated-Channels/dp/B00HJZEA2S/ref=sr_1_fkmr0_1?ie=UTF8&qid=1512205237&sr=8-1-fkmr0&keywords=startech+hub+usb+pci, simply plugged it in with the power off, and it has worked perfectly fine with 28.2