USB 3.0 port unstable on Jetson TX1?

Autonomous Machines Jetson & Embedded Systems Jetson TX1
#1

I’ve been trying out a few different USB (UVC) cameras on Jetson TX1. And I can reproduce the problem with all of them. For example, when I plugged a Logitech C920 into the USB 3.0 port of Jetson TX1 and ran the OpenCV “hog” sample (samples/gpu/hog) with live camera feed, the TX1 system would usually hard hang within 2~3 hours. If I ran the same OpenCV “hog” sample with a video file input, the TX1 system could work overnight without problem.

I’m using JetPack 2.0 on Jetson TX1. Has anyone experienced a similar problem?

#2

Hi jkjung,

Thanks for reporting the issue, we are currently investigating the case and we’ll let you know when we have an update.
Because the Jetson is mostly designed to be an embedded device, it tries to conserve power by automatically suspending inactive USB ports to save power. Not sure if the hang was caused by “temporarily” disabling USB.
You may need to disable USB auto suspend mode, this is a link to being sure lower power is not entered:
http://elinux.org/Jetson/Performance

Search for “usb auto-suspend” in the above link.

Thanks

#3

Hi jkjung,

We are working on reproducing the issue. In the meantime, I would request you to send us the sample along with the steps followed for running it, which could help us in replicating the scenario at our end.

Thanks

#4

I did more testing and found the problem only occurred when I used the UVC camera and the GPU (for computer vision tasks) simultaneously. I’m now trying to find an easy and deterministic way to reproduce the problem. I think I’ll post the repro steps in 1~2 days.

(Since the OpenCV4Tegra which comes with JetPack-2.1 does not support live camera input, I have to compile OpenCV from source to be able to repro the problem.)

Here’s some additional validation work I’ve tried on Jetson TX1:

  • Environment: Jetson TX1 with fresh Jetpack-2.1, with a Logitech C920r camera plugged into the USB 3.0 port
  • If I only exercise the camera, I cannot repro the problem. The following works fine overnight.
$ gst-launch-1.0 v4l2src device="/dev/video0" ! xvimagesink -e
  • If I run OpenCV gpu sample program with a video file as the source, I cannot repro the problem.
  • It’s only when I combine the above 2 things together that I can see the TX1 hanging problem.
#5

I have 2 Jetson TX1 boards on hand. I can now conclude that the USB hard-hanging problem is more easily reproduced on the older board (bought around Dec. 2015), but it’s very difficult to repro the problem on the newer board. I’ll post more information, such as S/N, about the boards later.

Here’s how I reproduce the problem on Jetson TX1.

  1. Download “OpenCV for Linux/Mac (Version 3.1)” from http://opencv.org/downloads.html

  2. Compile and install OpenCV-3.1.0 by following the official Jetson Installing_OpenCV guide.

# Add universal repository to Ubuntu
sudo apt-add-repository universe
sudo apt-get update

# Some general development libraries
sudo apt-get install -y build-essential make cmake cmake-curses-gui g++
# libav video input/output development libraries
sudo apt-get install -y libavformat-dev libavutil-dev libswscale-dev
# Video4Linux camera development libraries
sudo apt-get install -y libv4l-dev
# Eigen3 math development libraries
sudo apt-get install -y libeigen3-dev
# OpenGL development libraries (to allow creating graphical windows)
sudo apt-get install -y libglew1.6-dev
# GTK development libraries (to allow creating graphical windows)
sudo apt-get install -y libgtk2.0-dev

# Compile and install opencv
cd ~
mkdir src
cd src
unzip ~/Downloads/opencv-3.1.0.zip
cd opencv-3.1.0
mkdir build
cd build
cmake -DWITH_CUDA=ON -DCUDA_ARCH_BIN="5.3" -DCUDA_ARCH_PTX="" -DBUILD_TESTS=OFF -DBUILD_PERF_TESTS=OFF -DCUDA_FAST_MATH=ON -DCMAKE_INSTALL_PREFIX=/home/ubuntu/opencv-3.1.0 ..
make -j4 install
  1. Compile and run the OpenCV gpu “hog” sample program with Logitech C920r camera as the video source.
$ cd ~/src/opencv-3.1.0/samples/gpu
$ g++ -o hog -I /home/ubuntu/opencv-3.1.0/include -O2 -g -Wall hog.cpp -L /home/ubuntu/opencv-3.1.0/lib -lopencv_core -lopencv_imgproc -l opencv_flann -l opencv_imgcodecs -lopencv_videoio -lopencv_highgui -lopencv_ml -lopencv_video -lopencv_objdetect -lopencv_photo -lopencv_features2d -lopencv_calib3d -lopencv_stitching -lopencv_videostab -lopencv_shape -lopencv_cudaobjdetect -lopencv_cudawarping -lopencv_cudaimgproc
$ export LD_LIBRARY_PATH=/home/ubuntu/opencv-3.1.0/lib:$LD_LIBRARY_PATH
$ ./hog --camera 0
  1. When running the gpu hog program with UVC camera source, I can see xhci related messages in dmesg. And the older TX1 board usually hard-hangs within 2~3 hours. (Note the same test runs OK on Jetson TK1.)
[127773.590404] xhci_queue_intr_tx: 1 callbacks suppressed
[127792.598398] xhci_queue_intr_tx: 2 callbacks suppressed
[127812.262369] xhci_queue_intr_tx: 1 callbacks suppressed
[128383.842479] pwm_fan_driver pwm-fan.35:  Enabled vdd-fan
[130176.688334] pwm_fan_driver pwm-fan.35:  Disabled vdd-fan
[130653.318391] xhci_queue_intr_tx: 1 callbacks suppressed
[130695.318371] xhci_queue_intr_tx: 2 callbacks suppressed
[130736.310378] xhci_queue_intr_tx: 2 callbacks suppressed
[130976.562482] pwm_fan_driver pwm-fan.35:  Enabled vdd-fan
[136366.758388] xhci_queue_intr_tx: 5 callbacks suppressed
[137284.668365] pwm_fan_driver pwm-fan.35:  Disabled vdd-fan
#6

Here are the serial numbers of the 2 Jetson TX1 boards I have.

Board #1 (Old): S/N 0324715075330
Board #2 (New): S/N 0324715075757

The USB 3.0 port hanging problem can be reliably reproduced only on Board #1.

#7

jkjung, I also get random kernel panics when using a Point Grey USB3 camera with ROS.

[ 1575.379045] nodelet[21403]: unhandled level 3 translation fault (11) at 0x000003fc, esr 0x92000007
[ 1575.379052] pgd = ffffffc0dda9f000
[ 1575.382511] [000003fc] *pgd=000000015d8a3003, *pmd=000000015d523003, *pte=0000000000000000
[ 1575.390950]
[ 1575.390962] CPU: 0 PID: 21403 Comm: nodelet Not tainted 3.10.67-gcdddc52 #1
[ 1575.390968] task: ffffffc0deb32040 ti: ffffffc0def60000 task.ti: ffffffc0def60000
[ 1575.390976] PC is at 0xf6dfbb54
[ 1575.390981] LR is at 0xf6e3cfc3
[ 1575.390989] pc : [<00000000f6dfbb54>] lr : [<00000000f6e3cfc3>] pstate: 80010030
[ 1575.390992] sp : 00000000eb4f6cf0
[ 1575.390996] x12: 00000000f6e578a8
[ 1575.391004] x11: 00000000f1dfe7f8 x10: 00000000eb4f7654
[ 1575.391012] x9 : 0000000000000000 x8 : 000000000004ee68
[ 1575.391021] x7 : 0000000000000ee8 x6 : 0000000000000002
[ 1575.391028] x5 : 00000000000ce09c x4 : 0000000000000ee8
[ 1575.391035] x3 : 00000000000003fc x2 : 00000000f6c23b71
[ 1575.391042] x1 : 00000000eb4f6cfc x0 : 00000000000003f4
[ 1575.391048]
[ 1575.391058] Library at 0xf6dfbb54: 0xf6db5000 /usr/lib/arm-linux-gnueabihf/libstdc++.so.6.0.19
[ 1575.400033] Library at 0xf6e3cfc3: 0xf6db5000 /usr/lib/arm-linux-gnueabihf/libstdc++.so.6.0.19
[ 1575.408766] vdso base = 0xf708f000
#8

NaterGator, sure. Besides Logitech UVC cameras, I also experience the same kind of hard-hanging problem on TX1 when using Basler’s USB3 camera.

#9

Update: I also saw “unhandled translation fault” kernel oops when using Logitech UVC camera on the faulty Jetson TX1.

[361459.074010] xhci_queue_intr_tx: 2 callbacks suppressed
[361498.401997] xhci_queue_intr_tx: 1 callbacks suppressed
[361511.394109] xhci_queue_intr_tx: 3 callbacks suppressed
[367391.563060] watch[9674]: unhandled level 2 translation fault (11) at 0xfffffffe, esr 0x83000006
[367391.563169] pgd = ffffffc0aae11000
[367391.568272] [fffffffe] *pgd=000000011d904003, *pmd=0000000000000000
[367391.574867] 
[367391.574883] CPU: 3 PID: 9674 Comm: watch Not tainted 3.10.67-gcdddc52 #1
[367391.574892] task: ffffffc0fff64080 ti: ffffffc09da60000 task.ti: ffffffc09da60000
[367391.574907] PC is at 0xfffffffe
[367391.574913] LR is at 0xf7238cb3
[367391.574920] pc : [<00000000fffffffe>] lr : [<00000000f7238cb3>] pstate: 00000030
[367391.574926] sp : 00000000ff8ee61c
[367391.574932] x12: 000000000000000b 
[367391.574939] x11: 00000000ff8ee62c x10: 0000000000000000 
[367391.574951] x9 : 00000000f72aa000 x8 : 00000000ff8ee64c 
[367391.574961] x7 : 00000000f720f2af x6 : 00000000ff8ee63c 
[367391.574972] x5 : 00000000000159f8 x4 : 0000000000000000 
[367391.574983] x3 : 00000000f72abbb8 x2 : 00000000000166f0 
[367391.574993] x1 : 00000000ff8ee63c x0 : 00000000fffffff2 
[367391.575003] 
[367391.575024] Library at 0xf7238cb3: 0xf71c7000 /lib/arm-linux-gnueabihf/libc-2.19.so
[367391.582919] vdso base = 0xf7331000
[410845.586276] xhci_queue_intr_tx: 1 callbacks suppressed
[410851.922562] xhci_queue_intr_tx: 1 callbacks suppressed
#10

Hi,

Did you ever find how to solve this problem?

#11

Hi DavidSoto,

The issue you met is the same USB3.0 camera + OpenCV4Tegra hang issue as below?
“USB 3.0 port of Jetson TX1 and ran the OpenCV “hog” sample (samples/gpu/hog) with live camera feed, the TX1 system would usually hard hang within 2~3 hours. If I ran the same OpenCV “hog” sample with a video file input, the TX1 system could work overnight without problem.”

Or it’s other symptom?

If it’s OpenCV related, then we will have a new version updated within JetPack2.2 release, please stay tuned.

Thanks

#12

hi, im on a usb webcam, i run the videoCapture code, but it always give me -216 error, could not get any frame, i test the webcam with luvcview, it is working. im on Jetpack 2.1 with opencv4tegra. is there anything i could do to solve the problem? or is there any “for sure” c++ code that can run on Jetpack 2.1?

#13

I am new to Tegra and OpenCV4Tegra. I am having a lot of problems trying to get OpenCV4Tegra to work with the nvidia camera and logitech camera on the TX1.

This command, from the console will display video from the USB camera on the monitor…
$ gst-launch-1.0 v4l2src device="/dev/video0" ! xvimagesink -e

How can I use the gst pipeline from OpenCV4Tegra to access the camera? If I do the following:

VideoCapture cap (“v4l2src device=’/dev/video0’ ! xvimagesink -e”)

and then open the cap stream… I get no output. The program does not generate errors, but also does not open any frames from the camera. I have looked at the threads in this forum and it appears that the problem is OpenCV4Tegra on Jetpack 2.1 is not compiled to allow gst usage. Can somebody confirm this?

If this is true, then what good is OpenCV4tegra if I can’t open a video stream from a USB camera or the TX1 provided camera?

#14

Hi AutobotCV, you might want to take a look at this page for some testing info: http://elinux.org/Jetson/Installing_OpenCV. It’s for TK1 but the interface should be the same for TX1.

#15

Hi

The updated OpenCV4Tegra 2.4.13 was released within the JetPack 2.2.

Download Jetpack 2.2 bundle — https://developer.nvidia.com/embedded/jetpack
Access L4T components/docs — https://developer.nvidia.com/embedded/linux-tegra
Please see the Release Notes — http://developer.nvidia.com/embedded/dlc/l4t-release-notes-24-1

Thanks

#16

Hi, Using the above instructions for OpenCV 3.1.0, the installer is crashing in “make -j4 install” with

[ 56%] Building CXX object modules/imgproc/CMakeFiles/opencv_imgproc.dir/src/thresh.cpp.o

c++: internal compiler error: Segmentation fault (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-4.8/README.Bugs> for instructions.
make[2]: *** [modules/imgproc/CMakeFiles/opencv_imgproc.dir/src/thresh.cpp.o] Error 4
make[1]: *** [modules/imgproc/CMakeFiles/opencv_imgproc.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....

...

[ 56%] Built target opencv_cudaarithm
make: *** [all] Error 2

Any ideas what may be happening? Thanks!

#17

Hi Brian_,

Regarding your OpenCV 3.1.0 install issue, that might be the known issue, OpenCV compiling frequently crash in R24.1 64-bit environment. Please repeat compiling till it’s done, or to wait for next release.

Thanks

#18

Thanks kayccc, you were right, it was that same install issue. After about 100 times it finally made it through (the last one requiring a sudo).

#19

Hi, We are facing almost similar issue in Jetson TK1 (R21.4) when using USB3 (UVC) Camera and encoder together.
(Note: if using separately, there was no issue at all. And usb auto-suspend is -1).

[60654.282051] Host read timeout at address 4300
[60654.285137] MSELECT error detected! status=0x3
[60654.285154] ------------[ cut here ]------------
[60654.285156] kernel BUG at drivers/platform/tegra/hier_ictlr/hier_ictlr.c:54!
[60654.285158] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP ARM
[60654.285169] Modules linked in: dm_crypt dm_mod rfcomm bnep bluetooth rfkill nvhost_vi
[60654.285173] CPU: 0 PID: 3660 Comm: kworker/0:1 Tainted: G W 3.10.40-g868897a #1
[60654.285182] Workqueue: host_syncpt syncpt_thresh_cascade_fn
[60654.285184] task: d2e35080 ti: cfe3c000 task.ti: cfe3c000
[60654.285189] PC is at tegra_hier_ictlr_irq_handler+0x38/0x40
[60654.285192] LR is at tegra_hier_ictlr_irq_handler+0x38/0x40
[60654.285194] pc : [] lr : [] psr: 60030193
[60654.285194] sp : cfe3dc38 ip : 00000000 fp : c0d0a198
[60654.285196] r10: 00000000 r9 : 00000000 r8 : cfe3c000
[60654.285198] r7 : cfe3dcd8 r6 : 0000009f r5 : c0bcb910 r4 : d70ca780
[60654.285199] r3 : 00000000 r2 : 00010100 r1 : 00000000 r0 : 00000023
[60654.285202] Flags: nZCv IRQs off FIQs on Mode SVC_32 ISA ARM Segment kernel
[60654.285204] Control: 10c5387d Table: 8f99006a DAC: 00000015

We are using gstreamer-1.0 for streaming and encoding. Please help us.

#20

For the TK1 issue you may want to move or post in the TK1 forum:
https://devtalk.nvidia.com/default/board/162/jetson-tk1/

You’ll also want to post the camera model/info and lsusb -vvv for that camera (plus lsusb -t). The command producing the OOPS would be useful. Actual L4T version can be posted via “head -n 1 /etc/nv_tegra_release”.