opencv cannot read usb camera at 30fps

Wenbin_Xu · June 14, 2019, 3:10am

Hi,

My device is a TX2 with Jetpack 4.2.

I’ve built opencv 3.4.6 from source using the following configuration:

cmake -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/usr/local -D WITH_CUDA=ON -D CUDA_ARCH_BIN="6.2" -D CUDA_ARCH_PTX="" -D WITH_CUBLAS=ON -D ENABLE_FAST_MATH=ON -D CUDA_FAST_MATH=ON -D ENABLE_NEON=ON -D WITH_GSTREAMER=ON -D WITH_FFMPEG=ON  -D WITH_LIBV4L=ON -D BUILD_opencv_python2=ON -D BUILD_opencv_python3=ON -D BUILD_TESTS=OFF -D BUILD_PERF_TESTS=OFF -D BUILD_EXAMPLES=OFF -D WITH_QT=ON -D WITH_OPENGL=ON

I’ve built opencv on my desktop, with libv4l enabled, and it can get 1080p frames at 30fps. However, with libv4l enabled opencv on TX2, I can only get 30fps at 480P, 10fps at 720p and 5fps at 1080p.

Any ideas? Thanks!

WayneWWW · June 14, 2019, 6:07am

Hi,

Have you run jetson_clock?

Wenbin_Xu · June 14, 2019, 10:26am

Yes, I’ve run both jetson_clock and nvpmodel to utilize all 6 cpu cores.

miguel.taylor · June 14, 2019, 3:41pm

Hi,

I know it is a longshot, bit it is possible that you are capturing in an uncompressed format on TX2. You can check your camera’s supported formats on TX2 and Desktop to see if they are the same:

First, get your camera device id:

$ lsusb
Bus 002 Device 003: ID 0bda:0411 Realtek Semiconductor Corp. 
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 007: ID 0bda:0129 Realtek Semiconductor Corp. RTS5129 Card Reader Controller
Bus 001 Device 006: ID 8087:0a2a Intel Corp. 
Bus 001 Device 004: ID 1bcf:2b8a Sunplus Innovation Technology Inc. 
Bus 001 Device 009: ID 08a0:0850  
Bus 001 Device 005: ID 045e:07a5 Microsoft Corp. Wireless Receiver 1461C
Bus 001 Device 003: ID 0bda:5411 Realtek Semiconductor Corp. 
Bus 001 Device 002: ID 046d:0a4d Logitech, Inc. G430 Surround Sound Gaming Headset
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

In my case, it is the 08a0:0850. Then run lsusb verbose mode with your device:

sudo lsusb -v -d <device-id>

The output is a little difficult to follow, but basically, it lists all formats (FORMAT_UNCOMPRESSED, FORMAT_MJPEG, …) an then the available resolutions on that format:

VideoStreaming Interface Descriptor:
        bLength                            34
        bDescriptorType                    36
        bDescriptorSubtype                  5 (FRAME_UNCOMPRESSED)
        bFrameIndex                        18
        bmCapabilities                   0x01
          Still image supported
        wWidth                           1280
        wHeight                           720
        dwMinBitRate                 73728000
        dwMaxBitRate                147456000
        dwMaxVideoFrameBufferSize     1843200
        dwDefaultFrameInterval        2000000
        bFrameIntervalType                  2
        dwFrameInterval( 0)           1333333
        dwFrameInterval( 1)           2000000

This FRAME_UNCOMPRESSED is for 1280x720 on my camera. Getting the framerate is a bit tricky, each dwFrameInterval is an available framerate in units of 100 nano seconds, so 10000000/dwFrameInterval will give you the framerate in fps. In my camera, for 720p raw video, the maximum framerate is 7.5 fps (dwFrameInterval 0).

I hope this helps.

Wenbin_Xu · June 17, 2019, 2:07am

Hi miguel.taylor,

Thank you for your reply.

Here is my camera info in 1080P:

VideoStreaming Interface Descriptor:
        bLength                            30
        bDescriptorType                    36
        bDescriptorSubtype                  7 (FRAME_MJPEG)
        bFrameIndex                         3
        bmCapabilities                   0x00
          Still image unsupported
        wWidth                           1920
        wHeight                          1080
        dwMinBitRate                995469360
        dwMaxBitRate                995469360
        dwMaxVideoFrameBufferSize     4147789
        dwDefaultFrameInterval         333333
        bFrameIntervalType                  1
        dwFrameInterval( 0)            333333

VideoStreaming Interface Descriptor:
        bLength                            30
        bDescriptorType                    36
        bDescriptorSubtype                  5 (FRAME_UNCOMPRESSED)
        bFrameIndex                         3
        bmCapabilities                   0x00
          Still image unsupported
        wWidth                           1920
        wHeight                          1080
        dwMinBitRate                165888000
        dwMaxBitRate                165888000
        dwMaxVideoFrameBufferSize     4147200
        dwDefaultFrameInterval        2000000
        bFrameIntervalType                  1
        dwFrameInterval( 0)           2000000

Seems like it can support 1080P at 30fps at FRAME_MJPEG but not uncompressed form.

It is really confusing since I can get 1080P at 30fps on my desktop using the exact same USB camera.

When I used this camera on my desktop for the first time, I used pip-installed opencv and it can only produce 1080P frames at 5fps. After installed libv4l, and built opencv from source (enabling libv4l), it can produce 1080p at 30fps. But when it comes to TX2, even if I built opencv from source with libv4l enabled, I still cannot get 30fps.

WayneWWW · June 17, 2019, 2:53am

Would you mind sharing your opencv application (source)?
Could you try to use the mmapi sampe from jetpack and run a v4l2 sample to check if mmapi sample can give out 30fps?

Wenbin_Xu · June 17, 2019, 7:46am

Hi WayneWWW:

I downloaded opencv from this link: https://github.com/opencv/opencv/archive/3.4.6.zip and cmake with

cmake -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/usr/local -D WITH_CUDA=ON -D CUDA_ARCH_BIN="6.2" -D CUDA_ARCH_PTX="" -D WITH_CUBLAS=ON -D ENABLE_FAST_MATH=ON -D CUDA_FAST_MATH=ON -D ENABLE_NEON=ON -D WITH_GSTREAMER=ON -D WITH_LIBV4L=ON -D BUILD_opencv_python2=ON -D BUILD_opencv_python3=ON -D BUILD_TESTS=OFF -D BUILD_PERF_TESTS=OFF -D BUILD_EXAMPLES=OFF -D WITH_QT=ON -D WITH_OPENGL=ON ..

I tried to use mmapi and failed to record a view using the sample number 10, it always return this error message:

Set governor to performance before enabling profiler
PRODUCER: Creating output stream
PRODUCER: Launching consumer thread
Failed to query video capabilities: Inappropriate ioctl for device
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 4 
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4 
875967048
842091865
create video encoder return true
H264: Profile = 100, Level = 50 
PRODUCER: Starting repeat capture requests.
Segmentation fault (core dumped)

From the lsusb output, it seems opencv is always reading the camera in uncompressed form, even if I tried some methods online to set to MJPEG, which can support 30fps.

WayneWWW · June 17, 2019, 8:31am

Could you share what commands did you use for mmapi sample #10?

Wenbin_Xu · June 17, 2019, 8:42am

I just open a terminal, go to /usr/src/tegra_multimedia_api/samples/10_camera_recording,
run

make

then

./camera_recording -i 1 -r 1920x1080 -d 10

WayneWWW · June 17, 2019, 9:46am

Hi,

Please try this first.

gst-launch-1.0 v4l2src device=“/dev/video1” ! “video/x-raw, width=1920, height=1080” ! xvimagesink -e

DaneLLL · June 18, 2019, 1:22am

Hi,
For MJPEG source, please run 12_camera_v4l2_cuda.
10_camera_recording is for Bayer sensors.

Wenbin_Xu · June 18, 2019, 1:53am

This command gives only 5 fps output, with the mmapi demo can produce 30fps MJPEG frames.

Any further suggestions?

Thanks.

Wenbin_Xu · June 18, 2019, 2:00am

Sample 12 can produce 30fps 1080P frames while selecting MJPEG form. The reason I want to use opencv is that it is easy to use and my gpu utilization is almost full. I’m not sure if using cuda for getting frames will affect my deep learning models or not.

So I’d prefer to get 30fps 1080P frames from opencv’s VideoCapture with MJPEG form. Any possible solutions?

Thanks.

DaneLLL · June 18, 2019, 2:06am

Hi,

This is complete CPU based implementation and may not work better then using MMAPI. Anyway, one more thing you can try is to run ‘sudo jetson_clocks’. It enables CPU running at max performance.

Wenbin_Xu · June 18, 2019, 5:14am

I already tried jetson_clocks and it does not improve the fps. The problem now seems to be how to read MJPEG frames from USB camera instead of uncompressed frames.

DaneLLL · June 18, 2019, 6:12am

Hi,
You may go to https://answers.opencv.org/ to get further support on using OpenCV. There are more experienced OpenCV users.

Recommend you consider to use 12_camera_v4l2_cuda.

Wenbin_Xu · June 18, 2019, 9:02am

Ok, thanks so much for your help!