Increasing latency when using Gstreamer in OpenCV

I’m currently working on a project that uses 2 cameras for object detection: one camera on a raspberry pi to stream video over TCP using Gstreamer and the other is connected to the board itself. When I use the gstreamer pipeline over the command line, it seems to work fine with about a ~500ms latency.

Raspberry Pi Gstreamer Command:

raspivid -t 0 -h 480 -w 640 -fps 25 -hf -b 2000000 -o - | gst-launch-1.0 -v fdsrc ! h264parse ! rtph264pay config-interval=1 pt=96 ! gdppay ! tcpserversink host=192.168.0.52 port=5000

Jetson Viewing Command:

gst-launch-1.0 -v tcpclientsrc host=192.168.0.52 port=5000 ! gdpdepay ! rtph264depay ! avdec_h264 ! videoconvert ! autovideosink

The camera attached to the board is fine, but when I use these commands in OpenCV, I get a huge amount of latency that seems to add up over time. I’ve also tried only taking in input from the gstreamer pipeline, but it still has the increasing latency issue.

I’ve attached my build information and code for reference below.

Thank you for your time

General configuration for OpenCV 4.4.0 =====================================
Version control: 4.4.0

Extra modules:
Location (extra): /tmp/build_opencv/opencv_contrib/modules
Version control (extra): 4.4.0

Platform:
Timestamp: 2024-01-19T03:10:13Z
Host: Linux 5.10.104-g16215e6bc aarch64
CMake: 3.27.9
CMake generator: Unix Makefiles
CMake build tool: /usr/bin/make
Configuration: RELEASE

CPU/HW features:
Baseline: NEON FP16
required: NEON
disabled: VFPV3

C/C++:
Built as dynamic libs?: YES
C++ standard: 11
C++ Compiler: /usr/bin/c++ (ver 9.4.0)
C++ flags (Release): -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Winit-self -Wsuggest-override -Wno-delete-non-virtual-dtor -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections -fvisibility=hidden -fvisibility-inlines-hidden -O3 -DNDEBUG -DNDEBUG
C++ flags (Debug): -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Winit-self -Wsuggest-override -Wno-delete-non-virtual-dtor -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections -fvisibility=hidden -fvisibility-inlines-hidden -g -O0 -DDEBUG -D_DEBUG
C Compiler: /usr/bin/cc
C flags (Release): -fsigned-char -W -Wall -Werror=return-type -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Winit-self -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections -fvisibility=hidden -O3 -DNDEBUG -DNDEBUG
C flags (Debug): -fsigned-char -W -Wall -Werror=return-type -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Winit-self -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections -fvisibility=hidden -g -O0 -DDEBUG -D_DEBUG
Linker flags (Release): -Wl,–gc-sections -Wl,–as-needed
Linker flags (Debug): -Wl,–gc-sections -Wl,–as-needed
ccache: NO
Precompiled headers: NO
Extra dependencies: m pthread cudart_static dl rt nppc nppial nppicc nppidei nppif nppig nppim nppist nppisu nppitc npps cublas cudnn cufft -L/usr/local/cuda/lib64 -L/usr/lib/aarch64-linux-gnu
3rdparty dependencies:

OpenCV modules:
To be built: alphamat aruco bgsegm bioinspired calib3d ccalib core cudaarithm cudabgsegm cudacodec cudafeatures2d cudafilters cudaimgproc cudalegacy cudaobjdetect cudaoptflow cudastereo cudawarping cudev datasets dnn dnn_objdetect dnn_superres dpm face features2d flann freetype fuzzy gapi hdf hfs highgui img_hash imgcodecs imgproc intensity_transform line_descriptor ml objdetect optflow phase_unwrapping photo plot python2 python3 quality rapid reg rgbd saliency shape stereo stitching structured_light superres surface_matching text tracking video videoio videostab xfeatures2d ximgproc xobjdetect xphoto
Disabled: world
Disabled by dependency: -
Unavailable: cnn_3dobj cvv java js julia matlab ovis sfm ts viz
Applications: apps
Documentation: NO
Non-free algorithms: YES

GUI:
GTK+: YES (ver 3.24.20)
GThread : YES (ver 2.64.6)
GtkGlExt: NO
OpenGL support: NO
VTK support: NO

Media I/O:
ZLib: /usr/lib/aarch64-linux-gnu/libz.so (ver 1.2.11)
JPEG: /usr/lib/aarch64-linux-gnu/libjpeg.so (ver 80)
WEBP: build (ver encoder: 0x020f)
PNG: /usr/lib/aarch64-linux-gnu/libpng.so (ver 1.6.37)
TIFF: /usr/lib/aarch64-linux-gnu/libtiff.so (ver 42 / 4.1.0)
JPEG 2000: build Jasper (ver 1.900.1)
OpenEXR: build (ver 2.3.0)
HDR: YES
SUNRASTER: YES
PXM: YES
PFM: YES

Video I/O:
DC1394: YES (2.2.5)
FFMPEG: YES
avcodec: YES (58.54.100)
avformat: YES (58.29.100)
avutil: YES (56.31.100)
swscale: YES (5.5.100)
avresample: YES (4.0.0)
GStreamer: YES (1.16.3)
v4l/v4l2: YES (linux/videodev2.h)

Parallel framework: pthreads

Trace: YES (with Intel ITT)

Other third-party libraries:
Lapack: YES (/usr/lib/aarch64-linux-gnu/liblapack.so /usr/lib/aarch64-linux-gnu/libcblas.so /usr/lib/aarch64-linux-gnu/libatlas.so)
Eigen: YES (ver 3.3.7)
Custom HAL: YES (carotene (ver 0.0.1))
Protobuf: build (3.5.1)

NVIDIA CUDA: YES (ver 11.4, CUFFT CUBLAS FAST_MATH)
NVIDIA GPU arch: 53 62 72 87
NVIDIA PTX archs:

cuDNN: YES (ver 8.0)

OpenCL: YES (no extra features)
Include path: /tmp/build_opencv/opencv/3rdparty/include/opencl/1.2
Link libraries: Dynamic load

Python 2:
Interpreter: /usr/bin/python2.7 (ver 2.7.18)
Libraries: /usr/lib/aarch64-linux-gnu/libpython2.7.so (ver 2.7.18)
numpy: /usr/lib/python2.7/dist-packages/numpy/core/include (ver 1.16.5)
install path: lib/python2.7/dist-packages/cv2/python-2.7

Python 3:
Interpreter: /usr/bin/python3 (ver 3.8.10)
Libraries: /usr/lib/aarch64-linux-gnu/libpython3.8.so (ver 3.8.10)
numpy: /home/pete/.local/lib/python3.8/site-packages/numpy/core/include (ver 1.23.5)
install path: lib/python3.8/dist-packages/cv2/python-3.8

Python (for build): /usr/bin/python2.7

Java:
ant: NO
JNI: NO
Java wrappers: NO
Java tests: NO

Install to: /usr/local

import cv2 as cv
import threading

class camThread(threading.Thread):
    def __init__(self, previewName, camera, flipImage):
        threading.Thread.__init__(self)
        self.previewName = previewName
        self.camera = camera
        self.flipImage = flipImage
    def run(self):
        print("Starting " + self.previewName)
        camPreview(self.previewName, self.camera, self.flipImage)

def camPreview(previewName, camera, flipImage):
    if camera.isOpened():  # try to get the first frame
        rval, frame = camera.read()
    else:
        rval = False

    while rval:
        if flipImage:
            frame = cv.rotate(frame, cv.ROTATE_180)
        cv.imshow(previewName, frame)
        rval, frame = camera.read()
        key = cv.waitKey(30)
        if key == 27:  # exit on ESC
            break
    cv.destroyWindow(previewName)
    camera.release()

# print(cv.__version__)
# print(cv.getBuildInformation())


vc = cv.VideoCapture(0, cv.CAP_V4L2)
vc.set(cv.CAP_PROP_FRAME_WIDTH, 1280)
vc.set(cv.CAP_PROP_FRAME_HEIGHT, 720)

raspicap = cv.VideoCapture("tcpclientsrc host=192.168.0.52 port=5000 ! gdpdepay ! rtph264depay ! avdec_h264 ! videoconvert ! appsink", cv.CAP_GSTREAMER)

# Create two threads as follows

thread1 = camThread("Camera 1", vc, True)
thread1.start()

thread2 = camThread("Camera 2", raspicap, False)
thread2.start()

Hi,
There is synchronization mechanism in gstreamer. Please set sync=0 to disable it and see if it helps. Please also try hardware decoder nvv4l2decoder plugin.

Hi DaneLLL,

Thank you for helping so quickly.

Adding the sync=0 flag to the viewing command has drastically improved latency, but I’ve now run into two new issues:

  1. Running the multi-threaded code above causes the python script to load the first frame, then immediately freeze and become unresponsive.

  2. Trying to use nvv4l2decoder causes the gstreamer pipeline to stop during the prerolling step. However, I could be using it wrong. Here’s my current command for viewing:

gst-launch-1.0 -v tcpclientsrc host=192.168.0.52 port=5000 ! gdpdepay ! rtph264depay ! nvv4l2decoder ! videoconvert ! autovideosink sync=0

Hi,
Not sure if it helps but please try

... ! appsink drop=1

And please add h264parse:

... ! h264parse ! nvv4l2decoder ! ...

See if hardware decoder can work.

Hi,

Here is my current progress now.

  1. Adding in the drop=1 flag allows the two cameras to stream for maybe around ~1 second before once again crashing the script. The gstreamer viewing command is as follows:

raspicap = cv.VideoCapture(“tcpclientsrc host=192.168.0.52 port=5000 ! gdpdepay ! rtph264depay ! avdec_h264 ! videoconvert ! appsink sync=0”, cv.CAP_GSTREAMER)

  1. I still can’t get the hardware decoder to work on viewing. Here’s my current command

gst-launch-1.0 -v tcpclientsrc host=192.168.0.52 port=5000 ! gdpdepay ! rtph264depay ! h264parse ! nvv4l2decoder ! videoconvert ! autovideosink sync=0 drop=1

Hi,
Please launch the source in UDP or RTSP for a try. To see if UDP or RTSP works. You may refer to the commands in Jetson AGX Orin FAQ

Q: Is there any example of running RTSP streaming?
Q: Is there an example for running UDP streaming?

Hi DaneLLL,

My main issue is that my OpenCV keeps not responding after a while with the VideoCapture elements in openCV stop responding after a while with Gstreamer.

The latency issue has been fixed with sync=0 and drop=1 flags, but my main issue is with the crashing.

Is there a reason why openCV crashes with gstreamer videocapture elements?

Hi,
This looks to be constraint of OpenCV. Please try the suggestion:
Taking in two Gstreamer VideoCapture into OpenCV can only be done in a single thread - #3 by DaneLLL