How to stream camera by UDP H264

Hello,
I need to live stream my webcam, I want to have access to that stream from every PC in the same network - so I thought that UDP is the perfect solution for that.

After some testing, reinstalling etc… I get that to work (I do not know if that work like UDP) :

./test-launch "v4l2src device=/dev/video0 ! nvvidconv ! nvv4l2h264enc insert-sps-pps=1 insert-vui=1 ! h264parse ! rtph264pay name=pay0"

using that program from FAQ, and receiving with :

sudo gst-launch-1.0 uridecodebin uri=rtsp://127.0.0.1:8554/test ! nvoverlaysink

The problem is that when I try to receive with VLC it stops working or load indefinitely (on Jetson, don’t need to work here) and give “Connection failed” + “Your input can’t be opened” (on my Windows PC where I really need it)
I tried using .sdp

v=0
m=video 8554 RTP/AVP 96
c=IN IP4 127.0.0.1
a=rtpmap:96 MP4V-ES/90000

With no success.

I tried giving my VLC on PC different IPs - 127.0.0.1 and all IPs that are given by “ipconfig” (with :8554), all give me the same error :<

I tried using FFmpeg or VLC commands to stream but found no working example ( FFmpegStream , FFmpegStream2, VLC )

It would be perfect if It was possible to do that with OpenCV cv::Mat but I’m frustrated, so anything that works is fine :P

1 Like

You may be confusing RTSP with RTP over UDP.
test-launch will run a RSTP server. Not sure, but several clients may connect and read from it. With VLC, you would just provide the URI starting with rtsp:.

Another way is using RTP/UDP with multicast address (you may check this). In such case you would use a sdp file giving multicast address for VLC.

Ok, the second solution worked and I can see the video on my PC! :D
But it’s delayed like 2 seconds, can I somehow speed this up?

PS: What if I want to play it in a different application or inside an OpenCV program on my PC how to do that, I can’t use .sdp file after all?

Hi,
For UDP streaming, you may refer to

That doesn’t seem right? :/
I do not want to know my PC IP beforehand on Jetson, I want any PC to connect whenever.

I think most of the latency is due to VLC itself. You may ask to VLC users forum. Quickly tried some options as here but just reduced latency from 5s to 2s with still available options.

From gstreamer receiver, you may try using rtpjitterbuffer that may try to adjust latency (in ms) sending upstream events:

# From a Jetson
gst-launch-1.0 udpsrc uri=udp://224.1.1.1:5000 auto-multicast=true ! application/x-rtp, media=video, encoding-name=H264 ! rtpjitterbuffer latency=300 ! rtph264depay ! h264parse ! omxh264dec ! nvvidconv ! xvimagesink

# From any other Linux host with ffmpeg/gstreamer libs
gst-launch-1.0 udpsrc uri=udp://224.1.1.1:5000 auto-multicast=true ! application/x-rtp, media=video, encoding-name=H264 ! rtpjitterbuffer latency=300 ! rtph264depay ! h264parse ! avdec_h264 ! xvimagesink

Oh after trying that I get great improvement.

Can I receive that stream any other way (other than VLC)- on Windows 10? I got some application from the client to test, but I can’t receive it thru it - I wonder is that my fault or application. So I would like to test it some other way
I tried using OpenCV - VideoCapture(“udp://224.1.1.1:5000”) onmy PC but I get :

[ INFO:0] global C:\build\master_winpack-build-win64-vc14\opencv\modules\videoio\src\videoio_registry.cpp (197) cv::`anonymous-namespace'::VideoBackendRegistry::VideoBackendRegistry VIDEOIO: Enabled backends(8, sorted by priority): FFMPEG(1000); GSTREAMER(990); INTEL_MFX(980); MSMF(970); DSHOW(960); CV_IMAGES(950); CV_MJPEG(940); UEYE(930)
[ INFO:0] global C:\build\master_winpack-build-win64-vc14\opencv\modules\videoio\src\backend_plugin.cpp (396) cv::impl::getPluginCandidates Found 3 plugin(s) for FFMPEG
[ INFO:0] global C:\build\master_winpack-build-win64-vc14\opencv\modules\videoio\src\backend_plugin.cpp (175) cv::impl::DynamicLib::libraryLoad load H:\Moje\Pulpit\Praca\Git\TestProjects\OpenCVTest\opencv_videoio_ffmpeg451_64d.dll => FAILED
[ INFO:0] global C:\build\master_winpack-build-win64-vc14\opencv\modules\videoio\src\backend_plugin.cpp (175) cv::impl::DynamicLib::libraryLoad load opencv_videoio_ffmpeg451_64d.dll => FAILED
[ INFO:0] global C:\build\master_winpack-build-win64-vc14\opencv\modules\videoio\src\backend_plugin.cpp (175) cv::impl::DynamicLib::libraryLoad load opencv_videoio_ffmpeg451_64.dll => OK
[ INFO:0] global C:\build\master_winpack-build-win64-vc14\opencv\modules\videoio\src\backend_plugin.cpp (245) cv::impl::PluginBackend::PluginBackend Video I/O: initialized 'FFmpeg OpenCV Video I/O plugin': built with OpenCV 4.5 (ABI/API = 0/0), current OpenCV version is '4.5.1' (ABI/API = 0/1)
[ INFO:0] global C:\build\master_winpack-build-win64-vc14\opencv\modules\videoio\src\backend_plugin.cpp (256) cv::impl::PluginBackend::PluginBackend Video I/O: NOTE: plugin is supported, but there is API version mismath: plugin API level (0) != OpenCV API level (1)
[ INFO:0] global C:\build\master_winpack-build-win64-vc14\opencv\modules\videoio\src\backend_plugin.cpp (259) cv::impl::PluginBackend::PluginBackend Video I/O: NOTE: some functionality may be unavailable due to lack of support by plugin implementation
[ INFO:0] global C:\build\master_winpack-build-win64-vc14\opencv\modules\videoio\src\backend_plugin.cpp (262) cv::impl::PluginBackend::PluginBackend Video I/O: plugin is ready to use 'FFmpeg OpenCV Video I/O plugin'
[ INFO:0] global C:\build\master_winpack-build-win64-vc14\opencv\modules\videoio\src\backend_plugin.cpp (396) cv::impl::getPluginCandidates Found 2 plugin(s) for GSTREAMER
[ INFO:0] global C:\build\master_winpack-build-win64-vc14\opencv\modules\videoio\src\backend_plugin.cpp (175) cv::impl::DynamicLib::libraryLoad load H:\Moje\Pulpit\Praca\Git\TestProjects\OpenCVTest\opencv_videoio_gstreamer451_64d.dll => FAILED
[ INFO:0] global C:\build\master_winpack-build-win64-vc14\opencv\modules\videoio\src\backend_plugin.cpp (175) cv::impl::DynamicLib::libraryLoad load opencv_videoio_gstreamer451_64d.dll => FAILED
[ INFO:0] global C:\build\master_winpack-build-win64-vc14\opencv\modules\videoio\src\backend_plugin.cpp (396) cv::impl::getPluginCandidates Found 2 plugin(s) for INTEL_MFX
[ INFO:0] global C:\build\master_winpack-build-win64-vc14\opencv\modules\videoio\src\backend_plugin.cpp (175) cv::impl::DynamicLib::libraryLoad load H:\Moje\Pulpit\Praca\Git\TestProjects\OpenCVTest\opencv_videoio_intel_mfx451_64d.dll => FAILED
[ INFO:0] global C:\build\master_winpack-build-win64-vc14\opencv\modules\videoio\src\backend_plugin.cpp (175) cv::impl::DynamicLib::libraryLoad load opencv_videoio_intel_mfx451_64d.dll => FAILED
[ INFO:0] global C:\build\master_winpack-build-win64-vc14\opencv\modules\videoio\src\backend_plugin.cpp (396) cv::impl::getPluginCandidates Found 2 plugin(s) for MSMF
[ INFO:0] global C:\build\master_winpack-build-win64-vc14\opencv\modules\videoio\src\backend_plugin.cpp (175) cv::impl::DynamicLib::libraryLoad load H:\Moje\Pulpit\Praca\Git\TestProjects\OpenCVTest\opencv_videoio_msmf451_64d.dll => OK
[ INFO:0] global C:\build\master_winpack-build-win64-vc14\opencv\modules\videoio\src\backend_plugin.cpp (245) cv::impl::PluginBackend::PluginBackend Video I/O: initialized 'Microsoft Media Foundation OpenCV Video I/O plugin': built with OpenCV 4.5 (ABI/API = 0/0), current OpenCV version is '4.5.1' (ABI/API = 0/1)
[ INFO:0] global C:\build\master_winpack-build-win64-vc14\opencv\modules\videoio\src\backend_plugin.cpp (256) cv::impl::PluginBackend::PluginBackend Video I/O: NOTE: plugin is supported, but there is API version mismath: plugin API level (0) != OpenCV API level (1)
[ INFO:0] global C:\build\master_winpack-build-win64-vc14\opencv\modules\videoio\src\backend_plugin.cpp (259) cv::impl::PluginBackend::PluginBackend Video I/O: NOTE: some functionality may be unavailable due to lack of support by plugin implementation
[ INFO:0] global C:\build\master_winpack-build-win64-vc14\opencv\modules\videoio\src\backend_plugin.cpp (262) cv::impl::PluginBackend::PluginBackend Video I/O: plugin is ready to use 'Microsoft Media Foundation OpenCV Video I/O plugin'
[ INFO:0] global C:\build\master_winpack-build-win64-vc14\opencv\modules\videoio\src\cap_images.cpp (282) cv::icvExtractPattern Pattern: udp://%03d.1.1.1:5000 @ 224
[ INFO:1] global C:\build\master_winpack-build-win64-vc14\opencv\modules\videoio\src\backend_plugin.cpp (181) cv::impl::DynamicLib::libraryRelease unload H:\Moje\Pulpit\Praca\Git\TestProjects\OpenCVTest\opencv_videoio_msmf451_64d.dll
[ INFO:2] global C:\build\master_winpack-build-win64-vc14\opencv\modules\videoio\src\backend_plugin.cpp (181) cv::impl::DynamicLib::libraryRelease unload opencv_videoio_ffmpeg451_64.dll

I installed FFmpeg on my PC.

I don’t know how to do that, any idea?

You would try to specify the video capture backend. Seems ffmpeg is available, so you would specify:

VideoCapture("udp://224.1.1.1:5000", cv::CAP_FFMPEG);

Seems Windows Media Foundation backend is also available, but I have no experience with it.

You may also have gstreamer installed on Win10, but here again I cannot advise.

I tried with CAP_FFMPEG but get the same messages :<

I fixed that OpenCV issue with FFmpeg (some issues with Path and compilation mode), but I still can’t connect.
.isOpened() returns false for me now. Do I specify something in the wrong way?
After all, VLC has that .sdp file with some specification, maybe I need to pass them in OpenCV as well. Or maybe that streaming command needs to be specified some other way to let me handle that way on PC?

Sorry I missed that (I’m more familiar with gstreamer for such case).
You would use the sdp file as well :

setenv ("OPENCV_FFMPEG_CAPTURE_OPTIONS", "protocol_whitelist;file,rtp,udp", 1);
cap = cv::VideoCapture("test.sdp", cv::CAP_FFMPEG);

I get:

[rtp @ 0000014dc1f83bc0] Protocol 'rtp' not on whitelist 'file,crypto,data'!

I have no setenv() so I tried this and it seems that’s a problem, any idea?

Can I somehow stream my video, so I could receive it using only IP and port?

I tried using
ffplay myFile.sdp -protocol_whitelist file,udp,rtp -fflags nobuffer
which seems to be command equivalent and it’s working but that delay is back. I do not know how to set those parameters to lower the delay.

I just inform so maybe it will help somehow - I still need to receive that in OpenCV.
If I can install Gstramer on my Windows and work normally with it in OpenCV I can, so such solutions are also welcome ;P

Still I would be perfect if I could stream just to IP / Port, with no .sdp file involved at all.

Is there no solution?

I have poor experience of video and image processing with Windows, so I cannot help that much. I’d suggest you install gstreamer for your host CPU architecture. You would need the runtime binaries, but you may also install development package for rebuilding opencv with gstreamer support.

When you have the runtime installed, you may be able to try from a command shell:

c:\gstreamer\1.0\msvc_x86_64\bin\gst-launch-1.0.exe  udpsrc uri=udp://224.1.1.1:5000 auto-multicast=true ! application/x-rtp, media=video, encoding-name=H264 ! rtpjitterbuffer latency=300 ! rtph264depay ! decodebin ! d3dvideosink

and adjust latency to your case.
When ok, from opencv built with gstreamer support you would just create a videoCapture with gstreamer backend

cv::VideoCapture cap("udpsrc uri=udp://224.1.1.1:5000 auto-multicast=true ! application/x-rtp, media=video, encoding-name=H264 ! rtpjitterbuffer latency=300 ! rtph264depay ! decodebin ! videoconvert ! video/x-raw, format=BGR ! appsink", cv::CAP_GSTREAMER);

Ok, I installed (following that) and tried that command - is working, unfortunetlelly no matter what latancy I set I still gat quite a big lag.
When I try to use OpenCV version I get

[ WARN:0] global F:\Code\opencv_4.5.1\opencv-4.5.1\modules\videoio\src\cap_gstreamer.cpp (734) cv::GStreamerCapture::open OpenCV | GStreamer warning: Error opening bin: no element "udpsrc"
[ WARN:0] global F:\Code\opencv_4.5.1\opencv-4.5.1\modules\videoio\src\cap_gstreamer.cpp (501) cv::GStreamerCapture::isPipelinePlaying OpenCV | GStreamer warning: GStreamer: pipeline have not been created

And .isOpened() give me false.
I’m don’t know if I installed something wrong?

Seems the gstreamer pipeline failed to be created because udpsrc plugin was not found. Did that work from shell as suggested in my previous post ? You may check setting the environment variables such as PATH.
This is more a general topic about using gstreamer in opencv from Windows, not Jetson-specific, so you may get better help asking to opencv forum.

Yeah, that shell command worked - I have a big delay but it successfully receives the stream.

Here, I added that to my PATH as instructed:
image

Ok, I will also ask on OpenCV Forum about that.

I would also like to ask, can I somehow set stream speed from jetson (I need it to be ~2mbit/s)

You would try to set bitrate option (and more for image quality) in encoder. See:

gst-inspect-1.0 omxh264enc
gst-inspect-1.0 nvv4l2h264enc

You may also try increasing kernel buffersize with buffer-size option of udpsrc:

gst-inspect-1.0 udpsrc

@Honey_Patouceul
Maybe you know some other solution that I may try, I searched so much that don’t even know what to search for anymore, do you maybe have any other hint for me?

I tried asking elsewhere but without success - you are my best bet right now, sorry to bother you again.