How to stream camera by UDP H264

Sorry, I would need some time for trying gstreamer from opencv under WIndows before giving any further advice…
Not sure when I’ll have time for this, but don’t expect an update before a few days, maybe more. If I find more I’ll let you know here.
For now my best advice would be to try sync=false for gstreamer sink it you don’t need synchronization.
Or use a Linux host as receiver. Not sure it would better perform, but would be easier to test for me.

Hi @Legion,

I’ve given this a try and found it was working as expected.
I can get about 350 ms latency with sync and about 150ms latency wiithout sync.

For this, on jetson (NX), I ran:

  • in a first terminal display a clock:
while true; do date +%s.%N; done

you may zoom in this terminal from menu View/ZoomIn for adapting to your camera.

  • in a second terminal, launch a CSI camera server to H264 with your 2Mb/s bitrate to RTP/UDP with:
gst-launch-1.0 nvarguscamerasrc ! nvvidconv ! omxh264enc bitrate=2000000 insert-vui=1 ! h264parse ! rtph264pay config-interval=1 ! udpsink host=224.1.1.1 port=5000 auto-multicast=true

On Windows host, I have installed gstreamer and added bin directory to PATH.
Built opencv-4.5.1 with Visual Studio 2019 Community, enabling gstreamer support as shown in the link you’ve provided above and installed. Added opencv install bin directory to PATH.

Then made a new console project, selected Release/x64, added opencv include path, lib path and added this code:

#include <iostream>
#include <opencv2/core.hpp>
#include <opencv2/highgui.hpp>
#include <opencv2/videoio.hpp>

int main()
{
    std::cout << cv::getBuildInformation() << std::endl;

    cv::VideoCapture cap("udpsrc uri=udp://224.1.1.1:5000 auto-multicast=true ! application/x-rtp, media=video, encoding-name=H264 ! rtpjitterbuffer latency=200 ! rtph264depay ! decodebin ! videoconvert ! video/x-raw, format=BGR ! appsink", cv::CAP_GSTREAMER);
    if (!cap.isOpened()) {
        std::cerr << "Failed to open capture. Aborting\n";
        return(-1);
    }
    std::cout << "Cam opened" << std::endl;

    unsigned int width = cap.get(cv::CAP_PROP_FRAME_WIDTH);
    unsigned int height = cap.get(cv::CAP_PROP_FRAME_HEIGHT);
    unsigned int pixels = width * height;
    float fps = cap.get(cv::CAP_PROP_FPS);
    float pixrate = (float)pixels * fps;
    std::cout << " Framing: " << width << " x " << height << " @" << fps << " FPS" << ", " << pixrate << " Pixels/s " <<  std::endl;

    cv::Mat frame_in(width, height, CV_8UC3);

    for (;;)
    {
        if (!cap.read(frame_in)) {
            std::cout << "Capture read error" << std::endl;
            break;
        }
        cv::imshow("Preview", frame_in);
        cv::waitKey(1);
    }

    cap.release();
    return 0;
}

then added dependant libraries into linker properties entries such as:

opencv_core451.lib;opencv_videoio451.lib;opencv_highgui451.lib;%(AdditionalDependencies)

and built and run. It gave:

General configuration for OpenCV 4.5.1-dev =====================================
  Version control:               4.5.1-199-g767127c92e

  Extra modules:
    Location (extra):            C:/opencv/opencv_contrib/modules
    Version control (extra):     4.5.1-60-g4e85f8c6

  Platform:
    Timestamp:                   2021-02-11T21:03:37Z
    Host:                        Windows 10.0.19042 AMD64
    CMake:                       3.19.4
    CMake generator:             Visual Studio 16 2019
    CMake build tool:            C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/MSBuild/Current/Bin/MSBuild.exe
    MSVC:                        1928
    Configuration:               Debug Release

  CPU/HW features:
    Baseline:                    SSE SSE2 SSE3
      requested:                 SSE3
    Dispatched code generation:  SSE4_1 SSE4_2 FP16 AVX AVX2 AVX512_SKX
      requested:                 SSE4_1 SSE4_2 AVX FP16 AVX2 AVX512_SKX
      SSE4_1 (17 files):         + SSSE3 SSE4_1
      SSE4_2 (2 files):          + SSSE3 SSE4_1 POPCNT SSE4_2
      FP16 (1 files):            + SSSE3 SSE4_1 POPCNT SSE4_2 FP16 AVX
      AVX (5 files):             + SSSE3 SSE4_1 POPCNT SSE4_2 AVX
      AVX2 (31 files):           + SSSE3 SSE4_1 POPCNT SSE4_2 FP16 FMA3 AVX AVX2
      AVX512_SKX (7 files):      + SSSE3 SSE4_1 POPCNT SSE4_2 FP16 FMA3 AVX AVX2 AVX_512F AVX512_COMMON AVX512_SKX

  C/C++:
    Built as dynamic libs?:      YES
    C++ standard:                11
    C++ Compiler:                C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.28.29333/bin/Hostx64/x64/cl.exe  (ver 19.28.29337.0)
    C++ flags (Release):         /DWIN32 /D_WINDOWS /W4 /GR  /D _CRT_SECURE_NO_DEPRECATE /D _CRT_NONSTDC_NO_DEPRECATE /D _SCL_SECURE_NO_WARNINGS /Gy /bigobj /Oi  /fp:precise     /EHa /wd4127 /wd4251 /wd4324 /wd4275 /wd4512 /wd4589 /MP  /MD /O2 /Ob2 /DNDEBUG
    C++ flags (Debug):           /DWIN32 /D_WINDOWS /W4 /GR  /D _CRT_SECURE_NO_DEPRECATE /D _CRT_NONSTDC_NO_DEPRECATE /D _SCL_SECURE_NO_WARNINGS /Gy /bigobj /Oi  /fp:precise     /EHa /wd4127 /wd4251 /wd4324 /wd4275 /wd4512 /wd4589 /MP  /MDd /Zi /Ob0 /Od /RTC1
    C Compiler:                  C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.28.29333/bin/Hostx64/x64/cl.exe
    C flags (Release):           /DWIN32 /D_WINDOWS /W3  /D _CRT_SECURE_NO_DEPRECATE /D _CRT_NONSTDC_NO_DEPRECATE /D _SCL_SECURE_NO_WARNINGS /Gy /bigobj /Oi  /fp:precise     /MP   /MD /O2 /Ob2 /DNDEBUG
    C flags (Debug):             /DWIN32 /D_WINDOWS /W3  /D _CRT_SECURE_NO_DEPRECATE /D _CRT_NONSTDC_NO_DEPRECATE /D _SCL_SECURE_NO_WARNINGS /Gy /bigobj /Oi  /fp:precise     /MP /MDd /Zi /Ob0 /Od /RTC1
    Linker flags (Release):      /machine:x64  /INCREMENTAL:NO
    Linker flags (Debug):        /machine:x64  /debug /INCREMENTAL
    ccache:                      NO
    Precompiled headers:         YES
    Extra dependencies:
    3rdparty dependencies:

  OpenCV modules:
    To be built:                 aruco bgsegm bioinspired calib3d ccalib core datasets dnn dnn_objdetect dnn_superres dpm face features2d flann fuzzy gapi hfs highgui img_hash imgcodecs imgproc intensity_transform line_descriptor mcc ml objdetect optflow phase_unwrapping photo plot quality rapid reg rgbd saliency shape stereo stitching structured_light superres surface_matching text tracking ts video videoio videostab wechat_qrcode xfeatures2d ximgproc xobjdetect xphoto
    Disabled:                    java_bindings_generator python_bindings_generator python_tests world
    Disabled by dependency:      -
    Unavailable:                 alphamat cnn_3dobj cudaarithm cudabgsegm cudacodec cudafeatures2d cudafilters cudaimgproc cudalegacy cudaobjdetect cudaoptflow cudastereo cudawarping cudev cvv freetype hdf java julia matlab ovis python2 python3 sfm viz
    Applications:                tests perf_tests apps
    Documentation:               NO
    Non-free algorithms:         NO

  Windows RT support:            NO

  GUI:
    Win32 UI:                    YES
    VTK support:                 NO

  Media I/O:
    ZLib:                        build (ver 1.2.11)
    JPEG:                        build-libjpeg-turbo (ver 2.0.6-62)
    WEBP:                        build (ver encoder: 0x020f)
    PNG:                         build (ver 1.6.37)
    TIFF:                        build (ver 42 - 4.0.10)
    JPEG 2000:                   build (ver 2.3.1)
    OpenEXR:                     build (ver 2.3.0)
    HDR:                         YES
    SUNRASTER:                   YES
    PXM:                         YES
    PFM:                         YES

  Video I/O:
    DC1394:                      NO
    FFMPEG:                      YES (prebuilt binaries)
      avcodec:                   YES (58.91.100)
      avformat:                  YES (58.45.100)
      avutil:                    YES (56.51.100)
      swscale:                   YES (5.7.100)
      avresample:                YES (4.0.0)
    GStreamer:                   YES (1.18.3)
    DirectShow:                  YES
    Media Foundation:            YES
      DXVA:                      YES

  Parallel framework:            Concurrency

  Trace:                         YES (with Intel ITT)

  Other third-party libraries:
    Intel IPP:                   2020.0.0 Gold [2020.0.0]
           at:                   C:/opencv/build/3rdparty/ippicv/ippicv_win/icv
    Intel IPP IW:                sources (2020.0.0)
              at:                C:/opencv/build/3rdparty/ippicv/ippicv_win/iw
    Lapack:                      NO
    Eigen:                       NO
    Custom HAL:                  NO
    Protobuf:                    build (3.5.1)

  OpenCL:                        YES (NVD3D11)
    Include path:                C:/opencv/opencv/3rdparty/include/opencl/1.2
    Link libraries:              Dynamic load

  Python (for build):            NO

  Java:
    ant:                         NO
    JNI:                         NO
    Java wrappers:               NO
    Java tests:                  NO

  Install to:                    C:/opencv/build/install
-----------------------------------------------------------------


[ WARN:0] global C:\opencv\opencv\modules\videoio\src\cap_gstreamer.cpp (961) cv::GStreamerCapture::open OpenCV | GStreamer warning: Cannot query video position: status=1, value=10, duration=-1
Cam opened
 Framing: 1920 x 1080 @30 FPS, 6.2208e+07 Pixels/s

Then used the CSI camera from Jetson (IMX219) to record Jetson display showing clock and checked it was displayed in Windows. Then made a screen shot of both displays with my phone, compared the recorded clock on Jetson and the displayed clock on Windows, it was about 350ms.

If you don’t need sync, you can try on Jetson:

gst-launch-1.0 nvarguscamerasrc ! nvvidconv ! omxh264enc bitrate=2000000 insert-vui=1 ! h264parse ! rtph264pay config-interval=1 ! udpsink host=224.1.1.1 port=5000 auto-multicast=true sync=false

and in opencv from Windows:

 cv::VideoCapture cap("udpsrc uri=udp://224.1.1.1:5000 auto-multicast=true ! application/x-rtp, media=video, encoding-name=H264 ! rtpjitterbuffer latency=0 ! rtph264depay ! decodebin ! videoconvert ! video/x-raw, format=BGR ! appsink sync=false", cv::CAP_GSTREAMER);

Then I get about 150ms glass to glass latency

If you reproduce this, do you get significantly different numbers ?
This may vary depending on Nano vs NX and your host vs mine, and maybe monitors as well, but I would expect so much difference.

1 Like

It’s working! Thank you a lot!
I have like 60ms latency (on low res camera) but it’s the best I get until now.

Thank you once more :)

Hello again, I have yet one question - What I need to change to stream two videos at once? I tried changing the port and the last number of host IP, but it doesn’t seem to work?

Also I seem to have some problem with my network, I switched to wi-fi on my Jetson, and the stream stopped working on my PC, I tried connecting on Apple Laptop and there it’s working (VLC). Do I have some bad settings n PC now?

(I decided to continue asking here cause it’s connected and would make it easier for other users in the future)

You may either use different payloads (pt property of rtph*pay at sender side, payload-type cap at receiver side), or use different UDP ports or both. For example, sending H264 to pt 96 and port 5000 while streaming H265 to pt 97 and port 5001:
All these are supposed to be run a jetson. From another host, you would adjust for decoding to display (you may remove nvvidconv).

# Sender
gst-launch-1.0 nvarguscamerasrc ! nvvidconv ! tee name=t !  queue ! omxh264enc insert-vui=1 ! h264parse ! rtph264pay config-interval=1 pt=96 ! udpsink host=224.1.1.1 port=5000 auto-multicast=true    t. ! queue ! omxh265enc insert-vui=1 ! h265parse ! rtph265pay config-interval=1 pt=97 !  udpsink host=224.1.1.1 port=5001 auto-multicast=true

# H264 receiver on PT 96, port 5000:
gst-launch-1.0 -v udpsrc uri=udp://224.1.1.1:5000 auto-multicast=true ! application/x-rtp, media=video, encoding-name=H264, payload-type=96 ! rtph264depay ! decodebin ! nvvidconv ! videoconvert ! xvimagesink

# H265 receiver on PT 97, port 5001:
gst-launch-1.0 -v udpsrc uri=udp://224.1.1.1:5001 auto-multicast=true ! application/x-rtp, media=video, encoding-name=H265, payload-type=97 ! rtph265depay ! decodebin ! nvvidconv ! videoconvert ! xvimagesink

For wifi, you would create a separate topic. I cannot advice.