USB Crash when using four USB3.0 industry Cameras streaming

Autonomous Machines Jetson & Embedded Systems Jetson TX2
#1

we have a customed TX2 carry board with four USB3.0 ports.
one port connected to TX2 PEX_RFU_*X
one port connected to TX2 USB_SS0_*X
other two ports connected to TUSB8041A USB HUB which connected to TX2 USB_SS1_*X

The BSP is L4T R28.1.

when connected all fours USB3.0 industry camera and streaming with aravis(https://wiki.gnome.org/Projects/Aravis).

After about 30mins to several hours the USB HOST will crash and streaming stop.

Follow are the message when USB host crash.

ar 30 18:57:19 tegra-ubuntu kernel: [33771.321630] tegra-xusb-mbox 3538000.mailbox: Controller firmware hang
Mar 30 18:57:19 tegra-ubuntu kernel: [33771.328096] tegra-xusb-mbox 3538000.mailbox: XUSB_CFG_ARU_MBOX_OWNER 0x0
Mar 30 18:57:19 tegra-ubuntu kernel: [33771.334810] tegra-xusb-mbox 3538000.mailbox: XUSB_CFG_ARU_MBOX_CMD 0x80000000
Mar 30 18:57:19 tegra-ubuntu kernel: [33771.341954] tegra-xusb-mbox 3538000.mailbox: XUSB_CFG_ARU_MBOX_DATA_IN 0x0
Mar 30 18:57:19 tegra-ubuntu kernel: [33771.348834] tegra-xusb-mbox 3538000.mailbox: XUSB_CFG_ARU_MBOX_DATA_OUT 0x6000258
Mar 30 18:57:23 tegra-ubuntu kernel: [33774.683130] xhci-tegra 3530000.xhci: xHCI host not responding to stop endpoint command.
Mar 30 18:57:23 tegra-ubuntu kernel: [33774.691155] xhci-tegra 3530000.xhci: Assuming host is dying, halting host.
Mar 30 18:57:23 tegra-ubuntu rsyslogd-2007: action 'action 9' suspended, next retry is Fri Mar 30 18:58:53 2018 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Mar 30 18:57:23 tegra-ubuntu kernel: [33774.740049] xhci-tegra 3530000.xhci: Host not halted after 16000 microseconds.
Mar 30 18:57:23 tegra-ubuntu kernel: [33774.747294] xhci-tegra 3530000.xhci: Non-responsive xHCI host is not halting.
Mar 30 18:57:23 tegra-ubuntu kernel: [33774.754432] xhci-tegra 3530000.xhci: Completing active URBs anyway.
Mar 30 18:57:23 tegra-ubuntu kernel: [33774.762113] xhci-tegra 3530000.xhci: HC died; cleaning up
Mar 30 18:57:23 tegra-ubuntu kernel: [33774.762624] usb 2-3.2: usbfs: usb_submit_urb returned -108
Mar 30 18:57:23 tegra-ubuntu kernel: [33774.763026] usb 2-3.2: usbfs: usb_submit_urb returned -108
Mar 30 18:57:23 tegra-ubuntu kernel: [33774.763072] usb 2-3.1: usbfs: usb_submit_urb returned -108
Mar 30 18:57:23 tegra-ubuntu kernel: [33774.763389] usb 2-3.2: usbfs: usb_submit_urb returned -108
Mar 30 18:57:23 tegra-ubuntu kernel: [33774.763736] usb 2-2: usbfs: usb_submit_urb returned -108
Mar 30 18:57:23 tegra-ubuntu kernel: [33774.763748] usb 2-3.2: usbfs: usb_submit_urb returned -108
Mar 30 18:57:23 tegra-ubuntu kernel: [33774.764086] usb 2-3.2: usbfs: usb_submit_urb returned -108
Mar 30 18:57:23 tegra-ubuntu kernel: [33774.764434] usb 2-3.2: usbfs: usb_submit_urb returned -108
Mar 30 18:57:23 tegra-ubuntu kernel: [33774.764696] usb 2-3.1: usbfs: usb_submit_urb returned -108
Mar 30 18:57:23 tegra-ubuntu kernel: [33774.764778] usb 2-3.2: usbfs: usb_submit_urb returned -108
Mar 30 18:57:23 tegra-ubuntu kernel: [33774.765119] usb 2-3.2: usbfs: usb_submit_urb returned -108
Mar 30 18:57:23 tegra-ubuntu kernel: [33774.765342] usb 2-2: usbfs: usb_submit_urb returned -108
Mar 30 18:57:23 tegra-ubuntu kernel: [33774.765459] usb 2-3.2: usbfs: usb_submit_urb returned -108
Mar 30 18:57:23 tegra-ubuntu kernel: [33774.765803] usb 2-3.2: usbfs: usb_submit_urb returned -108
Mar 30 18:57:23 tegra-ubuntu kernel: [33774.766164] usb 2-3.2: usbfs: usb_submit_urb returned -108
Mar 30 18:57:23 tegra-ubuntu kernel: [33774.766282] usb 2-3.1: usbfs: usb_submit_urb returned -108
Mar 30 18:57:23 tegra-ubuntu kernel: [33774.766465] usb 2-3.2: usbfs: usb_submit_urb returned -108
Mar 30 18:57:23 tegra-ubuntu kernel: [33774.766837] usb 2-3.2: usbfs: usb_submit_urb returned -108
Mar 30 18:57:23 tegra-ubuntu kernel: [33774.766929] usb 2-2: usbfs: usb_submit_urb returned -108
Mar 30 18:57:23 tegra-ubuntu kernel: [33774.767164] usb 2-3.2: usbfs: usb_submit_urb returned -108
Mar 30 18:57:23 tegra-ubuntu kernel: [33774.767522] usb 2-3.2: usbfs: usb_submit_urb returned -108
Mar 30 18:57:23 tegra-ubuntu kernel: [33774.767848] usb 2-3.1: usbfs: usb_submit_urb returned -108
Mar 30 18:57:23 tegra-ubuntu kernel: [33774.767871] usb 2-3.2: usbfs: usb_submit_urb returned -108
Mar 30 18:57:23 tegra-ubuntu kernel: [33774.768189] usb 2-3.2: usbfs: usb_submit_urb returned -108
Mar 30 18:57:23 tegra-ubuntu kernel: [33774.768557] usb 2-3.2: usbfs: usb_submit_urb returned -108
Mar 30 18:57:23 tegra-ubuntu kernel: [33774.768658] usb 2-2: usbfs: usb_submit_urb returned -108
Mar 30 18:57:23 tegra-ubuntu kernel: [33774.768864] usb 2-3.2: usbfs: usb_submit_urb returned -108
Mar 30 18:57:23 tegra-ubuntu kernel: [33774.769218] usb 2-3.2: usbfs: usb_submit_urb returned -108
Mar 30 18:57:23 tegra-ubuntu kernel: [33774.769445] usb 2-3.1: usbfs: usb_submit_urb returned -108
Mar 30 18:57:23 tegra-ubuntu kernel: [33774.769573] usb 2-3.2: usbfs: usb_submit_urb returned -108
Mar 30 18:57:23 tegra-ubuntu kernel: [33774.769924] usb 2-3.2: usbfs: usb_submit_urb returned -108
Mar 30 18:57:23 tegra-ubuntu kernel: [33774.770265] usb 2-2: usbfs: usb_submit_urb returned -108
Mar 30 18:57:23 tegra-ubuntu kernel: [33774.770289] usb 2-3.2: usbfs: usb_submit_urb returned -108
Mar 30 18:57:23 tegra-ubuntu kernel: [33774.770607] usb 2-3.2: usbfs: usb_submit_urb returned -108
Mar 30 18:57:23 tegra-ubuntu kernel: [33774.770983] usb 2-3.2: usbfs: usb_submit_urb returned -108
Mar 30 18:57:23 tegra-ubuntu kernel: [33774.771062] usb 2-3.1: usbfs: usb_submit_urb returned -108
Mar 30 18:57:23 tegra-ubuntu kernel: [33774.771560] usb 2-2: usbfs: usb_submit_urb returned -108
TX2 Devkit USB controller firmware hang
#2 
[   13.317025] tegra-pcie 10003000.pcie-controller: link 0 down, retrying                                     
 [   13.720664] tegra-pcie 10003000.pcie-controller: link 0 down, retrying                                     
 [   13.980955] xhci-tegra 3530000.xhci: Firmware timestamp: 2017-03-31 06:21:56 UTC, Version: 55.07 release   
 [   13.983020] xhci-tegra 3530000.xhci: xHCI Host Controller                                                  
 [   13.983038] xhci-tegra 3530000.xhci: new USB bus registered, assigned bus number 1                         
 [   13.983810] xhci-tegra 3530000.xhci: hcc params 0x0184fd25 hci version 0x100 quirks 0x00010810             
 [   13.983828] xhci-tegra 3530000.xhci: irq 59, io mem 0x03530000
#3

Hi vin,
Does it happen by connecting three cameras to TX2 PEX_RFU_*X, TX2 PEX_SS0_*X, TX2 PEX_SS1_*X ?

#4

Hi Dane,

For there are no USB port directly connect to TX2 PEX_SS1_*X, I can not connect three camears to TX2 directly.

I have try an other vendor USB HUB connected to TX2 USB_SS0_*X the issue happend too.

#5

Hi vin,
Can you try a PICe to USB card such as https://devtalk.nvidia.com/default/topic/1027100/ ?

#6

Have you ever found a solution to this issue?
We have a very similar setup (Connecttech Rudi carrier board with Jetson TX2, 4 cameras from IDS streaming full HD resolution) and we are using a yocto build of meta-tegra based on L4T 32.2. We get those errors after 3-4 hours of duration test and then the 4 cameras disconnect from USB:

[15389.012344] tegra-xusb 3530000.xhci: controller firmware hang
[15389.018148] tegra-xusb 3530000.xhci: hcd_reinit is disabled or in progress
[15392.594214] tegra-xusb 3530000.xhci: xHCI host not responding to stop endpoint command.
[15392.602273] tegra-xusb 3530000.xhci: Assuming host is dying, halting host.
[15392.609190] tegra-xusb 3530000.xhci: xHCI host not responding to stop endpoint command.
[15392.617240] tegra-xusb 3530000.xhci: Assuming host is dying, halting host.
[15392.624186] tegra-xusb 3530000.xhci: xHCI host not responding to stop endpoint command.
[15392.632436] tegra-xusb 3530000.xhci: Assuming host is dying, halting host.
[15392.652451] tegra-xusb 3530000.xhci: Host not halted after 16000 microseconds.
[15392.659714] tegra-xusb 3530000.xhci: Non-responsive xHCI host is not halting.
[15392.666871] tegra-xusb 3530000.xhci: Completing active URBs anyway.
[15392.668600] tegra-xusb 3530000.xhci: Host not halted after 16000 microseconds.
[15392.680405] tegra-xusb 3530000.xhci: Non-responsive xHCI host is not halting.
[15392.685172] tegra-xusb 3530000.xhci: Host not halted after 16000 microseconds.
[15392.694799] tegra-xusb 3530000.xhci: Completing active URBs anyway.
[15392.701108] tegra-xusb 3530000.xhci: Non-responsive xHCI host is not halting.
[15392.708259] tegra-xusb 3530000.xhci: Completing active URBs anyway.
[15392.715552] tegra-xusb 3530000.xhci: HC died; cleaning up
[15392.715648] tegra-xusb 3530000.xhci: HC died; cleaning up
[15392.715668] tegra-xusb 3530000.xhci: HC died; cleaning up
[15392.715697] tegra-xusb 3530000.xhci: hcd_reinit is disabled or in progress
[15392.715702] tegra-xusb 3530000.xhci: hcd_reinit is disabled or in progress
[15392.715724] tegra-xusb 3530000.xhci: xHCI host not responding to stop endpoint command.
[15392.715732] tegra-xusb 3530000.xhci: Assuming host is dying, halting host.
[15392.715776] usb 1-1: USB disconnect, device number 4
[15392.716287] usb 1-3: USB disconnect, device number 2
[15392.716291] usb 1-3.2: USB disconnect, device number 3
[15392.717411] usb 2-3: USB disconnect, device number 2
[15392.717416] usb 2-3.2: USB disconnect, device number 3
[15392.717419] usb 2-3.2.1: USB disconnect, device number 8
[15392.752408] tegra-xusb 3530000.xhci: Host not halted after 16000 microseconds.
[15392.752410] tegra-xusb 3530000.xhci: Non-responsive xHCI host is not halting.
[15392.752411] tegra-xusb 3530000.xhci: Completing active URBs anyway.
[15392.752424] tegra-xusb 3530000.xhci: HC died; cleaning up
[15392.752430] tegra-xusb 3530000.xhci: hcd_reinit is disabled or in progress
[15392.752543] usb 2-3.2.2: USB disconnect, device number 9
[15392.829021] tegra-xusb 3530000.xhci: hcd_reinit is disabled or in progress
[15411.107277] tegra-xusb 3530000.xhci: Stopped the command ring failed, maybe the host is dead
[15411.152438] tegra-xusb 3530000.xhci: Host not halted after 16000 microseconds.
[15411.159647] tegra-xusb 3530000.xhci: Abort command ring failed
[15411.165561] usb 2-3.2.3: USB disconnect, device number 10
[15411.166049] tegra-xusb 3530000.xhci: HC died; cleaning up
[15411.166060] tegra-xusb 3530000.xhci: hcd_reinit is disabled or in progress
[15411.183603] usb 2-3.2.4: USB disconnect, device number 11
1 Like
#7

Hi etienne,
Please chekc if you can get help from Connecttech.

On default carrier board there is only one USB3 Type A port. For having enough bandwidth, we suggest use PCIe to USB card.