I’m currently working on a project that uses 2 cameras for object detection: one camera on a raspberry pi to stream video over TCP using Gstreamer and the other is connected to the board itself. When I use the gstreamer pipeline over the command line, it seems to work fine with about a ~500ms latency.
Raspberry Pi Gstreamer Command:
raspivid -t 0 -h 480 -w 640 -fps 25 -hf -b 2000000 -o - | gst-launch-1.0 -v fdsrc ! h264parse ! rtph264pay config-interval=1 pt=96 ! gdppay ! tcpserversink host=192.168.0.52 port=5000
Jetson Viewing Command:
gst-launch-1.0 -v tcpclientsrc host=192.168.0.52 port=5000 ! gdpdepay ! rtph264depay ! avdec_h264 ! videoconvert ! autovideosink
The camera attached to the board is fine, but when I use these commands in OpenCV, I get a huge amount of latency that seems to add up over time. I’ve also tried only taking in input from the gstreamer pipeline, but it still has the increasing latency issue.
I’ve attached my build information and code for reference below.
Thank you for your time
General configuration for OpenCV 4.4.0 =====================================
Version control: 4.4.0Extra modules:
Location (extra): /tmp/build_opencv/opencv_contrib/modules
Version control (extra): 4.4.0Platform:
Timestamp: 2024-01-19T03:10:13Z
Host: Linux 5.10.104-g16215e6bc aarch64
CMake: 3.27.9
CMake generator: Unix Makefiles
CMake build tool: /usr/bin/make
Configuration: RELEASECPU/HW features:
Baseline: NEON FP16
required: NEON
disabled: VFPV3C/C++:
Built as dynamic libs?: YES
C++ standard: 11
C++ Compiler: /usr/bin/c++ (ver 9.4.0)
C++ flags (Release): -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Winit-self -Wsuggest-override -Wno-delete-non-virtual-dtor -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections -fvisibility=hidden -fvisibility-inlines-hidden -O3 -DNDEBUG -DNDEBUG
C++ flags (Debug): -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Winit-self -Wsuggest-override -Wno-delete-non-virtual-dtor -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections -fvisibility=hidden -fvisibility-inlines-hidden -g -O0 -DDEBUG -D_DEBUG
C Compiler: /usr/bin/cc
C flags (Release): -fsigned-char -W -Wall -Werror=return-type -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Winit-self -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections -fvisibility=hidden -O3 -DNDEBUG -DNDEBUG
C flags (Debug): -fsigned-char -W -Wall -Werror=return-type -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Winit-self -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections -fvisibility=hidden -g -O0 -DDEBUG -D_DEBUG
Linker flags (Release): -Wl,–gc-sections -Wl,–as-needed
Linker flags (Debug): -Wl,–gc-sections -Wl,–as-needed
ccache: NO
Precompiled headers: NO
Extra dependencies: m pthread cudart_static dl rt nppc nppial nppicc nppidei nppif nppig nppim nppist nppisu nppitc npps cublas cudnn cufft -L/usr/local/cuda/lib64 -L/usr/lib/aarch64-linux-gnu
3rdparty dependencies:OpenCV modules:
To be built: alphamat aruco bgsegm bioinspired calib3d ccalib core cudaarithm cudabgsegm cudacodec cudafeatures2d cudafilters cudaimgproc cudalegacy cudaobjdetect cudaoptflow cudastereo cudawarping cudev datasets dnn dnn_objdetect dnn_superres dpm face features2d flann freetype fuzzy gapi hdf hfs highgui img_hash imgcodecs imgproc intensity_transform line_descriptor ml objdetect optflow phase_unwrapping photo plot python2 python3 quality rapid reg rgbd saliency shape stereo stitching structured_light superres surface_matching text tracking video videoio videostab xfeatures2d ximgproc xobjdetect xphoto
Disabled: world
Disabled by dependency: -
Unavailable: cnn_3dobj cvv java js julia matlab ovis sfm ts viz
Applications: apps
Documentation: NO
Non-free algorithms: YESGUI:
GTK+: YES (ver 3.24.20)
GThread : YES (ver 2.64.6)
GtkGlExt: NO
OpenGL support: NO
VTK support: NOMedia I/O:
ZLib: /usr/lib/aarch64-linux-gnu/libz.so (ver 1.2.11)
JPEG: /usr/lib/aarch64-linux-gnu/libjpeg.so (ver 80)
WEBP: build (ver encoder: 0x020f)
PNG: /usr/lib/aarch64-linux-gnu/libpng.so (ver 1.6.37)
TIFF: /usr/lib/aarch64-linux-gnu/libtiff.so (ver 42 / 4.1.0)
JPEG 2000: build Jasper (ver 1.900.1)
OpenEXR: build (ver 2.3.0)
HDR: YES
SUNRASTER: YES
PXM: YES
PFM: YESVideo I/O:
DC1394: YES (2.2.5)
FFMPEG: YES
avcodec: YES (58.54.100)
avformat: YES (58.29.100)
avutil: YES (56.31.100)
swscale: YES (5.5.100)
avresample: YES (4.0.0)
GStreamer: YES (1.16.3)
v4l/v4l2: YES (linux/videodev2.h)Parallel framework: pthreads
Trace: YES (with Intel ITT)
Other third-party libraries:
Lapack: YES (/usr/lib/aarch64-linux-gnu/liblapack.so /usr/lib/aarch64-linux-gnu/libcblas.so /usr/lib/aarch64-linux-gnu/libatlas.so)
Eigen: YES (ver 3.3.7)
Custom HAL: YES (carotene (ver 0.0.1))
Protobuf: build (3.5.1)NVIDIA CUDA: YES (ver 11.4, CUFFT CUBLAS FAST_MATH)
NVIDIA GPU arch: 53 62 72 87
NVIDIA PTX archs:cuDNN: YES (ver 8.0)
OpenCL: YES (no extra features)
Include path: /tmp/build_opencv/opencv/3rdparty/include/opencl/1.2
Link libraries: Dynamic loadPython 2:
Interpreter: /usr/bin/python2.7 (ver 2.7.18)
Libraries: /usr/lib/aarch64-linux-gnu/libpython2.7.so (ver 2.7.18)
numpy: /usr/lib/python2.7/dist-packages/numpy/core/include (ver 1.16.5)
install path: lib/python2.7/dist-packages/cv2/python-2.7Python 3:
Interpreter: /usr/bin/python3 (ver 3.8.10)
Libraries: /usr/lib/aarch64-linux-gnu/libpython3.8.so (ver 3.8.10)
numpy: /home/pete/.local/lib/python3.8/site-packages/numpy/core/include (ver 1.23.5)
install path: lib/python3.8/dist-packages/cv2/python-3.8Python (for build): /usr/bin/python2.7
Java:
ant: NO
JNI: NO
Java wrappers: NO
Java tests: NOInstall to: /usr/local
import cv2 as cv
import threading
class camThread(threading.Thread):
def __init__(self, previewName, camera, flipImage):
threading.Thread.__init__(self)
self.previewName = previewName
self.camera = camera
self.flipImage = flipImage
def run(self):
print("Starting " + self.previewName)
camPreview(self.previewName, self.camera, self.flipImage)
def camPreview(previewName, camera, flipImage):
if camera.isOpened(): # try to get the first frame
rval, frame = camera.read()
else:
rval = False
while rval:
if flipImage:
frame = cv.rotate(frame, cv.ROTATE_180)
cv.imshow(previewName, frame)
rval, frame = camera.read()
key = cv.waitKey(30)
if key == 27: # exit on ESC
break
cv.destroyWindow(previewName)
camera.release()
# print(cv.__version__)
# print(cv.getBuildInformation())
vc = cv.VideoCapture(0, cv.CAP_V4L2)
vc.set(cv.CAP_PROP_FRAME_WIDTH, 1280)
vc.set(cv.CAP_PROP_FRAME_HEIGHT, 720)
raspicap = cv.VideoCapture("tcpclientsrc host=192.168.0.52 port=5000 ! gdpdepay ! rtph264depay ! avdec_h264 ! videoconvert ! appsink", cv.CAP_GSTREAMER)
# Create two threads as follows
thread1 = camThread("Camera 1", vc, True)
thread1.start()
thread2 = camThread("Camera 2", raspicap, False)
thread2.start()