GStreamer pipeline is slow

Autonomous Machines Jetson & Embedded Systems Jetson Xavier NX
#1

Hi Experts,

I’m new to JetPack dev kit - I’m trying to capture a video from TheImagingSource grabber and run actions on the captured video using OpenCV. TheImagingSource has it’s own software TisCamera that works along side the GStreamer to communicate with the camera device. So based on their example I am able to run the video and show it on my screen through OpenCV but the process is very heavy for the CPU, the output video lags where it should run smoothly on 30fps.
I wonder if there is anything I can do to accelerate the process on GPU instead?

Current GStreamer pipeline is something like:

gst-launch-1.0 tcambin serial=1235165 ! video/x-raw,format=YUY2,width=1920,height=1080,framerate=30/1 ! appsink drop=1,max-buffer=5,emit-signal=1

Anyway to speed it up ?

Need help on H264 encoded 4K@30fps video framerate
#3

Hi,
We have power modes listed in developer guide. Please execute sudo nvpmodel -m 2 and sudo jetson_clocks to get max performance.
If the performance is still not expected, please run sudo tegrastats to check system loading and find the bottleneck.

#4

Thanks for your reply!
Unfortunately maxing out the settings did not change anything. I tried to change tcambin to v4lscr and I achieved 30FPS video but I need the video format to be BGRx instead of YUY2 to do post processing in OpenCV so I did conversion which drops the fps from 30 to ~15
gst-launch-1.0 v4l2src device=/dev/video0 ! 'video/x-raw,format=YUY2,width=1920,height=1080,framerate=30/1' ! videoconvert ! 'video/x-raw,format=BGRx' ! ximagesink sync=0

so I tried to use nvidconv to do the conversion but I get an error: Internal data stream error.
gst-launch-1.0 v4l2src device=/dev/video0 ! 'video/x-raw,format=YUY2,width=1920,height=1080,framerate=30/1' ! nvvidconv ! 'video/x-raw,format=BGRx' ! ximagesink sync=0

What am I doing wrong ?

#5
  • nvvidconv expects at least one among its input or output to be in NVMM memory.
  • xvimagesink cannot handle BGRx nor BGR, so you would add videoconvert that will convert into a suitable format.
  • opencv videoio through gstreamer API cannot support BGRx (see supported formats). So you would have to use videoconvert to BGR as well.

You may try:

gst-launch-1.0 -v v4l2src device=/dev/video0 ! 'video/x-raw,format=YUY2,width=1920,height=1080,framerate=30/1' ! nvvidconv ! 'video/x-raw(memory:NVMM),format=YUY2' ! nvvidconv ! 'video/x-raw,format=BGRx' ! videoconvert ! fpsdisplaysink video-sink=xvimagesink text-overlay=0

# or
gst-launch-1.0 -v v4l2src device=/dev/video0 ! 'video/x-raw,format=YUY2,width=1920,height=1080,framerate=30/1' ! nvvidconv ! 'video/x-raw(memory:NVMM),format=BGRx' ! nvvidconv ! 'video/x-raw,format=BGRx' ! videoconvert ! fpsdisplaysink video-sink=xvimagesink text-overlay=0

and for opencv use a pipeline such as:

gst_str="v4l2src device=/dev/video0 ! video/x-raw,format=YUY2,width=1920,height=1080,framerate=30/1 ! nvvidconv ! video/x-raw(memory:NVMM),format=BGRx ! nvvidconv ! video/x-raw,format=BGRx ! videoconvert ! video/x-raw, format=BGR ! appsink"

# C++
cv::VideoCapture(gst_str, cv::CAP_GSTREAMER); 

# Python
cv2.VideoCapture(gst_str, cv2.CAP_GSTREAMER)

Also note that cv::imshow is not very efficient on Jetson.

1 Like
#6

Thank you, the GStreamer pipeline works but the video stream is slow again. the camera is capable of outputting 30FPS while I get laggy ~20FPS. So I thought maybe the Jetson Xavier is just unable to handle the 1920x1080 video stream even on max performance mode ? What Can I do to improve my situation ?

#7

You may try to add option io-mode=2 to v4l2src plugin:

gst-launch-1.0 -v v4l2src device=/dev/video0 io-mode=2 ! 'video/x-raw,format=YUY2,width=1920,height=1080,framerate=30/1' ! nvvidconv ! 'video/x-raw(memory:NVMM),format=YUY2' ! nvvidconv ! 'video/x-raw,format=BGRx' ! videoconvert ! fpsdisplaysink video-sink=xvimagesink

Also check that your USB connection to camera is using USB3 (check with lsusb -t).

#8

I have rebuilt OpenCV with cuda enabled but the performance did not improve, then I added WITH_QT=ON and now its working flawlessly! Stable 30FPS and smooth image! I don’t know if QT can have impact on the performance but I’m happy with my current results. Thanks for your feedbacks!

2 Likes
#10

One more update to this - I just realized that OpenCV built on Qt uses QGraphicsView with OpenGL acceleration and that is the reason that the image window runs smoother than on GTK interface.