Sorry, I would need some time for trying gstreamer from opencv under WIndows before giving any further advice…
Not sure when I’ll have time for this, but don’t expect an update before a few days, maybe more. If I find more I’ll let you know here.
For now my best advice would be to try sync=false for gstreamer sink it you don’t need synchronization.
Or use a Linux host as receiver. Not sure it would better perform, but would be easier to test for me.
Hi @Legion,
I’ve given this a try and found it was working as expected.
I can get about 350 ms latency with sync and about 150ms latency wiithout sync.
For this, on jetson (NX), I ran:
- in a first terminal display a clock:
while true; do date +%s.%N; done
you may zoom in this terminal from menu View/ZoomIn for adapting to your camera.
- in a second terminal, launch a CSI camera server to H264 with your 2Mb/s bitrate to RTP/UDP with:
gst-launch-1.0 nvarguscamerasrc ! nvvidconv ! omxh264enc bitrate=2000000 insert-vui=1 ! h264parse ! rtph264pay config-interval=1 ! udpsink host=224.1.1.1 port=5000 auto-multicast=true
On Windows host, I have installed gstreamer and added bin directory to PATH.
Built opencv-4.5.1 with Visual Studio 2019 Community, enabling gstreamer support as shown in the link you’ve provided above and installed. Added opencv install bin directory to PATH.
Then made a new console project, selected Release/x64, added opencv include path, lib path and added this code:
#include <iostream>
#include <opencv2/core.hpp>
#include <opencv2/highgui.hpp>
#include <opencv2/videoio.hpp>
int main()
{
std::cout << cv::getBuildInformation() << std::endl;
cv::VideoCapture cap("udpsrc uri=udp://224.1.1.1:5000 auto-multicast=true ! application/x-rtp, media=video, encoding-name=H264 ! rtpjitterbuffer latency=200 ! rtph264depay ! decodebin ! videoconvert ! video/x-raw, format=BGR ! appsink", cv::CAP_GSTREAMER);
if (!cap.isOpened()) {
std::cerr << "Failed to open capture. Aborting\n";
return(-1);
}
std::cout << "Cam opened" << std::endl;
unsigned int width = cap.get(cv::CAP_PROP_FRAME_WIDTH);
unsigned int height = cap.get(cv::CAP_PROP_FRAME_HEIGHT);
unsigned int pixels = width * height;
float fps = cap.get(cv::CAP_PROP_FPS);
float pixrate = (float)pixels * fps;
std::cout << " Framing: " << width << " x " << height << " @" << fps << " FPS" << ", " << pixrate << " Pixels/s " << std::endl;
cv::Mat frame_in(width, height, CV_8UC3);
for (;;)
{
if (!cap.read(frame_in)) {
std::cout << "Capture read error" << std::endl;
break;
}
cv::imshow("Preview", frame_in);
cv::waitKey(1);
}
cap.release();
return 0;
}
then added dependant libraries into linker properties entries such as:
opencv_core451.lib;opencv_videoio451.lib;opencv_highgui451.lib;%(AdditionalDependencies)
and built and run. It gave:
General configuration for OpenCV 4.5.1-dev =====================================
Version control: 4.5.1-199-g767127c92e
Extra modules:
Location (extra): C:/opencv/opencv_contrib/modules
Version control (extra): 4.5.1-60-g4e85f8c6
Platform:
Timestamp: 2021-02-11T21:03:37Z
Host: Windows 10.0.19042 AMD64
CMake: 3.19.4
CMake generator: Visual Studio 16 2019
CMake build tool: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/MSBuild/Current/Bin/MSBuild.exe
MSVC: 1928
Configuration: Debug Release
CPU/HW features:
Baseline: SSE SSE2 SSE3
requested: SSE3
Dispatched code generation: SSE4_1 SSE4_2 FP16 AVX AVX2 AVX512_SKX
requested: SSE4_1 SSE4_2 AVX FP16 AVX2 AVX512_SKX
SSE4_1 (17 files): + SSSE3 SSE4_1
SSE4_2 (2 files): + SSSE3 SSE4_1 POPCNT SSE4_2
FP16 (1 files): + SSSE3 SSE4_1 POPCNT SSE4_2 FP16 AVX
AVX (5 files): + SSSE3 SSE4_1 POPCNT SSE4_2 AVX
AVX2 (31 files): + SSSE3 SSE4_1 POPCNT SSE4_2 FP16 FMA3 AVX AVX2
AVX512_SKX (7 files): + SSSE3 SSE4_1 POPCNT SSE4_2 FP16 FMA3 AVX AVX2 AVX_512F AVX512_COMMON AVX512_SKX
C/C++:
Built as dynamic libs?: YES
C++ standard: 11
C++ Compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.28.29333/bin/Hostx64/x64/cl.exe (ver 19.28.29337.0)
C++ flags (Release): /DWIN32 /D_WINDOWS /W4 /GR /D _CRT_SECURE_NO_DEPRECATE /D _CRT_NONSTDC_NO_DEPRECATE /D _SCL_SECURE_NO_WARNINGS /Gy /bigobj /Oi /fp:precise /EHa /wd4127 /wd4251 /wd4324 /wd4275 /wd4512 /wd4589 /MP /MD /O2 /Ob2 /DNDEBUG
C++ flags (Debug): /DWIN32 /D_WINDOWS /W4 /GR /D _CRT_SECURE_NO_DEPRECATE /D _CRT_NONSTDC_NO_DEPRECATE /D _SCL_SECURE_NO_WARNINGS /Gy /bigobj /Oi /fp:precise /EHa /wd4127 /wd4251 /wd4324 /wd4275 /wd4512 /wd4589 /MP /MDd /Zi /Ob0 /Od /RTC1
C Compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.28.29333/bin/Hostx64/x64/cl.exe
C flags (Release): /DWIN32 /D_WINDOWS /W3 /D _CRT_SECURE_NO_DEPRECATE /D _CRT_NONSTDC_NO_DEPRECATE /D _SCL_SECURE_NO_WARNINGS /Gy /bigobj /Oi /fp:precise /MP /MD /O2 /Ob2 /DNDEBUG
C flags (Debug): /DWIN32 /D_WINDOWS /W3 /D _CRT_SECURE_NO_DEPRECATE /D _CRT_NONSTDC_NO_DEPRECATE /D _SCL_SECURE_NO_WARNINGS /Gy /bigobj /Oi /fp:precise /MP /MDd /Zi /Ob0 /Od /RTC1
Linker flags (Release): /machine:x64 /INCREMENTAL:NO
Linker flags (Debug): /machine:x64 /debug /INCREMENTAL
ccache: NO
Precompiled headers: YES
Extra dependencies:
3rdparty dependencies:
OpenCV modules:
To be built: aruco bgsegm bioinspired calib3d ccalib core datasets dnn dnn_objdetect dnn_superres dpm face features2d flann fuzzy gapi hfs highgui img_hash imgcodecs imgproc intensity_transform line_descriptor mcc ml objdetect optflow phase_unwrapping photo plot quality rapid reg rgbd saliency shape stereo stitching structured_light superres surface_matching text tracking ts video videoio videostab wechat_qrcode xfeatures2d ximgproc xobjdetect xphoto
Disabled: java_bindings_generator python_bindings_generator python_tests world
Disabled by dependency: -
Unavailable: alphamat cnn_3dobj cudaarithm cudabgsegm cudacodec cudafeatures2d cudafilters cudaimgproc cudalegacy cudaobjdetect cudaoptflow cudastereo cudawarping cudev cvv freetype hdf java julia matlab ovis python2 python3 sfm viz
Applications: tests perf_tests apps
Documentation: NO
Non-free algorithms: NO
Windows RT support: NO
GUI:
Win32 UI: YES
VTK support: NO
Media I/O:
ZLib: build (ver 1.2.11)
JPEG: build-libjpeg-turbo (ver 2.0.6-62)
WEBP: build (ver encoder: 0x020f)
PNG: build (ver 1.6.37)
TIFF: build (ver 42 - 4.0.10)
JPEG 2000: build (ver 2.3.1)
OpenEXR: build (ver 2.3.0)
HDR: YES
SUNRASTER: YES
PXM: YES
PFM: YES
Video I/O:
DC1394: NO
FFMPEG: YES (prebuilt binaries)
avcodec: YES (58.91.100)
avformat: YES (58.45.100)
avutil: YES (56.51.100)
swscale: YES (5.7.100)
avresample: YES (4.0.0)
GStreamer: YES (1.18.3)
DirectShow: YES
Media Foundation: YES
DXVA: YES
Parallel framework: Concurrency
Trace: YES (with Intel ITT)
Other third-party libraries:
Intel IPP: 2020.0.0 Gold [2020.0.0]
at: C:/opencv/build/3rdparty/ippicv/ippicv_win/icv
Intel IPP IW: sources (2020.0.0)
at: C:/opencv/build/3rdparty/ippicv/ippicv_win/iw
Lapack: NO
Eigen: NO
Custom HAL: NO
Protobuf: build (3.5.1)
OpenCL: YES (NVD3D11)
Include path: C:/opencv/opencv/3rdparty/include/opencl/1.2
Link libraries: Dynamic load
Python (for build): NO
Java:
ant: NO
JNI: NO
Java wrappers: NO
Java tests: NO
Install to: C:/opencv/build/install
-----------------------------------------------------------------
[ WARN:0] global C:\opencv\opencv\modules\videoio\src\cap_gstreamer.cpp (961) cv::GStreamerCapture::open OpenCV | GStreamer warning: Cannot query video position: status=1, value=10, duration=-1
Cam opened
Framing: 1920 x 1080 @30 FPS, 6.2208e+07 Pixels/s
Then used the CSI camera from Jetson (IMX219) to record Jetson display showing clock and checked it was displayed in Windows. Then made a screen shot of both displays with my phone, compared the recorded clock on Jetson and the displayed clock on Windows, it was about 350ms.
If you don’t need sync, you can try on Jetson:
gst-launch-1.0 nvarguscamerasrc ! nvvidconv ! omxh264enc bitrate=2000000 insert-vui=1 ! h264parse ! rtph264pay config-interval=1 ! udpsink host=224.1.1.1 port=5000 auto-multicast=true sync=false
and in opencv from Windows:
cv::VideoCapture cap("udpsrc uri=udp://224.1.1.1:5000 auto-multicast=true ! application/x-rtp, media=video, encoding-name=H264 ! rtpjitterbuffer latency=0 ! rtph264depay ! decodebin ! videoconvert ! video/x-raw, format=BGR ! appsink sync=false", cv::CAP_GSTREAMER);
Then I get about 150ms glass to glass latency
If you reproduce this, do you get significantly different numbers ?
This may vary depending on Nano vs NX and your host vs mine, and maybe monitors as well, but I would expect so much difference.
It’s working! Thank you a lot!
I have like 60ms latency (on low res camera) but it’s the best I get until now.
Thank you once more :)
Hello again, I have yet one question - What I need to change to stream two videos at once? I tried changing the port and the last number of host IP, but it doesn’t seem to work?
Also I seem to have some problem with my network, I switched to wi-fi on my Jetson, and the stream stopped working on my PC, I tried connecting on Apple Laptop and there it’s working (VLC). Do I have some bad settings n PC now?
(I decided to continue asking here cause it’s connected and would make it easier for other users in the future)
You may either use different payloads (pt property of rtph*pay at sender side, payload-type cap at receiver side), or use different UDP ports or both. For example, sending H264 to pt 96 and port 5000 while streaming H265 to pt 97 and port 5001:
All these are supposed to be run a jetson. From another host, you would adjust for decoding to display (you may remove nvvidconv).
# Sender
gst-launch-1.0 nvarguscamerasrc ! nvvidconv ! tee name=t ! queue ! omxh264enc insert-vui=1 ! h264parse ! rtph264pay config-interval=1 pt=96 ! udpsink host=224.1.1.1 port=5000 auto-multicast=true t. ! queue ! omxh265enc insert-vui=1 ! h265parse ! rtph265pay config-interval=1 pt=97 ! udpsink host=224.1.1.1 port=5001 auto-multicast=true
# H264 receiver on PT 96, port 5000:
gst-launch-1.0 -v udpsrc uri=udp://224.1.1.1:5000 auto-multicast=true ! application/x-rtp, media=video, encoding-name=H264, payload-type=96 ! rtph264depay ! decodebin ! nvvidconv ! videoconvert ! xvimagesink
# H265 receiver on PT 97, port 5001:
gst-launch-1.0 -v udpsrc uri=udp://224.1.1.1:5001 auto-multicast=true ! application/x-rtp, media=video, encoding-name=H265, payload-type=97 ! rtph265depay ! decodebin ! nvvidconv ! videoconvert ! xvimagesink
For wifi, you would create a separate topic. I cannot advice.