Gstreamer UDP sink using python

magnus.gabell · May 27, 2021, 3:19pm

Hi,

I am trying to get a udp stream pipe and a receiving pipe to work in python. First trying to get it to work from Xavier to Windows. Then translate it to python on the linux (later in windows but start to capture in windows using gst-launch-1.0)

Working:

Server (Linux):
gst-launch-1.0 videotestsrc ! jpegenc ! rtpjpegpay ! udpsink host=192.168.2.100 port=5000

Client(Windows 10)
gst-launch-1.0 udpsrc port=5000 ! application/x-rtp, encoding-name=JPEG,payload=26 ! rtpjpegdepay ! jpegdec ! autovideosink

Not Working:

Now I want to move server gst to python. Same Client gstreamer from windows console.
My source is a v4l2src and I split it to appsink and updsink: (I want image to screen and send UDP)
I also want very small overhead and do not want to overload CPU on jetson and keep 50-60FPS. The
problem by creating a separate gst after reading the image is that it will be slower and pass through CPU-buffer (I believe)

v4l2src device=/dev/video0 ! video/x-raw , width=(int){} , height=(int){} , format=(string){} , framerate=(fraction)60/1 ! timeoverlay ! nvvidconv ! video/x-raw(memory:NVMM), width=(int){}, height=(int){}, format=I420 ! tee name=t t. ! queue ! nvvidconv ! video/x-raw, format=BGRx ! videoconvert ! video/x-raw, format=BGR, width=(int)400 , height=(int)400 ! appsink t. ! queue ! jpegenc ! rtpjpegpay ! udpsink host=192.168.2.100 port=5000

This is where it stops. So I cant send this above from python to gst-launch-1.0 in windows terminal on a second computer. The initial command in console/terminal works with the simple test source.

Any ideas how I can modify this to get python to send my camera image to windows?

Honey_Patouceul · May 27, 2021, 5:27pm

You may try something like this (assuming your V4L cam supports 1920x1080@60fps):

#!/usr/bin/env python

import signal
import sys
import time
import gi
gi.require_version('Gst', '1.0')
from gi.repository import Gst, GObject

# Signal handler stopping the pipeline before exit
def signal_handler(sig, frame):
    p.set_state(Gst.State.NULL)
    sys.exit(0)



GObject.threads_init()
Gst.init(None)

gst_str = "v4l2src device=/dev/video0 ! video/x-raw,width=1920,height=1080, framerate=60/1 ! timeoverlay ! nvvidconv ! video/x-raw(memory:NVMM),format=I420 ! tee name=t \
   t. ! queue ! nvvidconv ! video/x-raw, format=BGRx,width=400,height=400,pixel-aspect-ratio=1/1 ! videoconvert ! video/x-raw, format=BGR ! appsink \
   t. ! queue ! nvjpegenc ! jpegparse ! rtpjpegpay ! udpsink host=127.0.0.1 port=5000"


# Create the pipeline
p = Gst.parse_launch (gst_str)

# Register signal handler for proper termination if receiving SIGINT such as Ctrl-C
signal.signal(signal.SIGINT, signal_handler)

# Start the pipeline
p.set_state(Gst.State.READY)
p.set_state(Gst.State.PAUSED)
p.set_state(Gst.State.PLAYING)


# Run for 10s 
time.sleep(10)

# Done. Stop the pipeline before clean up on exit.
p.set_state(Gst.State.NULL)
exit(0)

timeoverlay and videoconvert may generate some CPU load for high resolutions.

magnus.gabell · May 27, 2021, 5:48pm

Thanx. Will try. my cam support 1600x1300, 60fps
Its the Arducam ov2311

magnus.gabell · May 28, 2021, 11:56am

Hi,
So I tried your code straight up and I got some issues. I modified the gst_str a bit cause the input format is GRAY8.
Other then that I get the below warnings and no resulting stream.

gst_str = “v4l2src device=/dev/video0 ! video/x-raw,width=1600,height=1300, format=(string)GRAY8, framerate=60/1 ! timeoverlay ! nvvidconv ! video/x-raw(memory:NVMM),width=(int)1600, height=(int)1300, format=I420 ! tee name=t
t. ! queue ! nvvidconv ! video/x-raw, format=BGRx,width=1600,height=1300,pixel-aspect-ratio=1/1 ! videoconvert ! video/x-raw, format=BGR ! appsink
t. ! queue ! nvjpegenc ! jpegparse ! rtpjpegpay ! udpsink host=127.0.0.1 port=5000”

I then added debugging and this is what I get:

0:00:00.253146347 10910 0x429b6d0 WARN default video.c:94:gst_video_calculate_display_ratio: overflow in multiply
0:00:00.253350510 10910 0x429b6d0 WARN basetransform gstbasetransform.c:1355:gst_base_transform_setcaps: transform could not transform video/x-raw, width=(int)1600, height=(int)1300, format=(string)GRAY8, framerate=(fraction)60/1, pixel-aspect-ratio=(fraction)1/2147483647, colorimetry=(string)1:4:7:1, interlace-mode=(string)progressive in anything we support
0:00:00.254493506 10910 0x429b6d0 WARN default video.c:94:gst_video_calculate_display_ratio: overflow in multiply
0:00:00.254567331 10910 0x429b6d0 WARN basetransform gstbasetransform.c:1355:gst_base_transform_setcaps: transform could not transform video/x-raw, width=(int)1600, height=(int)1300, format=(string)GRAY8, framerate=(fraction)60/1, pixel-aspect-ratio=(fraction)1/2147483647, colorimetry=(string)1:4:7:1, interlace-mode=(string)progressive in anything we support
0:00:00.254609988 10910 0x429b6d0 WARN GST_PADS gstpad.c:4226:gst_pad_peer_query:timeoverlay0:src could not send sticky events
0:00:00.256989644 10910 0x429b6d0 WARN v4l2bufferpool gstv4l2bufferpool.c:790:gst_v4l2_buffer_pool_start:v4l2src0:pool:src Uncertain or not enough buffers, enabling copy threshold
0:00:00.483938493 10910 0x429b6d0 WARN v4l2src gstv4l2src.c:911:gst_v4l2src_create: Timestamp does not correlate with any clock, ignoring driver timestamps
0:00:00.485585977 10910 0x429b6d0 WARN default video.c:94:gst_video_calculate_display_ratio: overflow in multiply
0:00:00.485741564 10910 0x429b6d0 WARN basetransform gstbasetransform.c:1355:gst_base_transform_setcaps: transform could not transform video/x-raw, width=(int)1600, height=(int)1300, format=(string)GRAY8, framerate=(fraction)60/1, pixel-aspect-ratio=(fraction)1/2147483647, colorimetry=(string)1:4:7:1, interlace-mode=(string)progressive in anything we support
0:00:00.537567749 10910 0x429b6d0 WARN default video.c:94:gst_video_calculate_display_ratio: overflow in multiply
0:00:00.537721160 10910 0x429b6d0 WARN basetransform gstbasetransform.c:1355:gst_base_transform_setcaps: transform could not transform video/x-raw, width=(int)1600, height=(int)1300, format=(string)GRAY8, framerate=(fraction)60/1, pixel-aspect-ratio=(fraction)1/2147483647, colorimetry=(string)1:4:7:1, interlace-mode=(string)progressive in anything we support
0:00:00.537839178 10910 0x429b6d0 WARN basesrc gstbasesrc.c:3055:gst_base_src_loop: error: Internal data stream error.
0:00:00.537872106 10910 0x429b6d0 WARN basesrc gstbasesrc.c:3055:gst_base_src_loop: error: streaming stopped, reason not-negotiated (-4)
0:00:00.539063871 10910 0x429b6d0 WARN default video.c:94:gst_video_calculate_display_ratio: overflow in multiply
0:00:00.539146176 10910 0x429b6d0 WARN basetransform gstbasetransform.c:1355:gst_base_transform_setcaps: transform could not transform video/x-raw, width=(int)1600, height=(int)1300, format=(string)GRAY8, framerate=(fraction)60/1, pixel-aspect-ratio=(fraction)1/2147483647, colorimetry=(string)1:4:7:1, interlace-mode=(string)progressive in anything we support
0:00:00.540212370 10910 0x429b6d0 WARN default video.c:94:gst_video_calculate_display_ratio: overflow in multiply
0:00:00.540326260 10910 0x429b6d0 WARN basetransform gstbasetransform.c:1355:gst_base_transform_setcaps: transform could not transform video/x-raw, width=(int)1600, height=(int)1300, format=(string)GRAY8, framerate=(fraction)60/1, pixel-aspect-ratio=(fraction)1/2147483647, colorimetry=(string)1:4:7:1, interlace-mode=(string)progressive in anything we support
0:00:00.541447527 10910 0x429b6d0 WARN default video.c:94:gst_video_calculate_display_ratio: overflow in multiply
0:00:00.541529192 10910 0x429b6d0 WARN basetransform gstbasetransform.c:1355:gst_base_transform_setcaps: transform could not transform video/x-raw, width=(int)1600, height=(int)1300, format=(string)GRAY8, framerate=(fraction)60/1, pixel-aspect-ratio=(fraction)1/2147483647, colorimetry=(string)1:4:7:1, interlace-mode=(string)progressive in anything we support

Honey_Patouceul · May 28, 2021, 12:16pm

I don’t see any issue when trying with videotestsrc.
Since your source is monochrome, I’ve also removed color conversion for appsink.
You may first :

gst_str = "videotestsrc ! video/x-raw,format=GRAY8,width=1600,height=1300,framerate=60/1 ! timeoverlay ! nvvidconv ! video/x-raw(memory:NVMM) ! tee name=t \
  t. ! queue ! nvvidconv ! video/x-raw,format=GRAY8,width=400,height=400,pixel-aspect-ratio=1/1 ! appsink \
  t. ! queue ! nvjpegenc ! jpegparse ! rtpjpegpay ! udpsink host=127.0.0.1 port=5000"

and if working replace videotestsrc with your v4l2src:

gst_str = "v4l2src device=/dev/video0 ! video/x-raw,format=GRAY8,width=1600,height=1300,framerate=60/1 ! timeoverlay ! nvvidconv ! video/x-raw(memory:NVMM) ! tee name=t \
  t. ! queue ! nvvidconv ! video/x-raw,format=GRAY8,width=400,height=400,pixel-aspect-ratio=1/1 ! appsink \
  t. ! queue ! nvjpegenc ! jpegparse ! rtpjpegpay ! udpsink host=127.0.0.1 port=5000"

magnus.gabell · May 28, 2021, 12:23pm

Honey_Patouceul:

gst_str = "v4l2src device=/dev/video0 ! video/x-raw,format=GRAY8,width=1600,height=1300,framerate=60/1 ! timeoverlay ! nvvidconv ! video/x-raw(memory:NVMM) ! tee name=t \
  t. ! queue ! nvvidconv ! video/x-raw,format=GRAY8,width=400,height=400,pixel-aspect-ratio=1/1 ! appsink \
  t. ! queue ! nvjpegenc ! jpegparse ! rtpjpegpay ! udpsink host=127.0.0.1 port=5000"

Hi, The testsrc works. Its when adding my camerasrc that messes up the things.
For reference… This is wha tI use to send to appsink. And that works.

v4l2src device=/dev/video0 ! video/x-raw , width=(int){} , height=(int){} , format=(string){} , framerate=(fraction)60/1 ! nvvidconv ! video/x-raw(memory:NVMM), width=(int){}, height=(int){}, format=I420 ! nvvidconv ! video/x-raw, format=BGRx ! videoconvert ! video/x-raw, format=BGR, width=(int){} , height=(int){} ! appsink

0:00:00.279580582 12709 0x23119990 ERROR nvvidconv gstnvvconv.c:2218:gst_nvvconv_fixate_caps: calculation of the outputscaled size error
0:00:00.279730343 12709 0x23119990 WARN basetransform gstbasetransform.c:1355:gst_base_transform_setcaps: transform could not transform video/x-raw, format=(string)GRAY8, width=(int)1600, height=(int)1300, framerate=(fraction)60/1, pixel-aspect-ratio=(fraction)1/2147483647, colorimetry=(string)1:4:7:1, interlace-mode=(string)progressive in anything we support
0:00:00.281206867 12709 0x23119990 ERROR nvvidconv gstnvvconv.c:2218:gst_nvvconv_fixate_caps: calculation of the outputscaled size error
0:00:00.281330804 12709 0x23119990 WARN basetransform gstbasetransform.c:1355:gst_base_transform_setcaps: transform could not transform video/x-raw, format=(string)GRAY8, width=(int)1600, height=(int)1300, framerate=(fraction)60/1, pixel-aspect-ratio=(fraction)1/2147483647, colorimetry=(string)1:4:7:1, interlace-mode=(string)progressive in anything we support
0:00:00.281393013 12709 0x23119990 WARN GST_PADS gstpad.c:4226:gst_pad_peer_query:timeoverlay0:src could not send sticky events
0:00:00.288356972 12709 0x23119990 WARN v4l2bufferpool gstv4l2bufferpool.c:790:gst_v4l2_buffer_pool_start:v4l2src0:pool:src Uncertain or not enough buffers, enabling copy threshold
0:00:00.517407523 12709 0x23119990 WARN v4l2src gstv4l2src.c:911:gst_v4l2src_create: Timestamp does not correlate with any clock, ignoring driver timestamps
0:00:00.518978575 12709 0x23119990 ERROR nvvidconv gstnvvconv.c:2218:gst_nvvconv_fixate_caps: calculation of the outputscaled size error
0:00:00.519171249 12709 0x23119990 WARN basetransform gstbasetransform.c:1355:gst_base_transform_setcaps: transform could not transform video/x-raw, format=(string)GRAY8, width=(int)1600, height=(int)1300, framerate=(fraction)60/1, pixel-aspect-ratio=(fraction)1/2147483647, colorimetry=(string)1:4:7:1, interlace-mode=(string)progressive in anything we support
0:00:00.578449287 12709 0x23119990 ERROR nvvidconv gstnvvconv.c:2218:gst_nvvconv_fixate_caps: calculation of the outputscaled size error
0:00:00.578570888 12709 0x23119990 WARN basetransform gstbasetransform.c:1355:gst_base_transform_setcaps: transform could not transform video/x-raw, format=(string)GRAY8, width=(int)1600, height=(int)1300, framerate=(fraction)60/1, pixel-aspect-ratio=(fraction)1/2147483647, colorimetry=(string)1:4:7:1, interlace-mode=(string)progressive in anything we support
0:00:00.578700009 12709 0x23119990 WARN basesrc gstbasesrc.c:3055:gst_base_src_loop: error: Internal data stream error.
0:00:00.578736329 12709 0x23119990 WARN basesrc gstbasesrc.c:3055:gst_base_src_loop: error: streaming stopped, reason not-negotiated (-4)
0:00:00.579734001 12709 0x23119990 ERROR nvvidconv gstnvvconv.c:2218:gst_nvvconv_fixate_caps: calculation of the outputscaled size error
0:00:00.579813970 12709 0x23119990 WARN basetransform gstbasetransform.c:1355:gst_base_transform_setcaps: transform could not transform video/x-raw, format=(string)GRAY8, width=(int)1600, height=(int)1300, framerate=(fraction)60/1, pixel-aspect-ratio=(fraction)1/2147483647, colorimetry=(string)1:4:7:1, interlace-mode=(string)progressive in anything we support
0:00:00.581110620 12709 0x23119990 ERROR nvvidconv gstnvvconv.c:2218:gst_nvvconv_fixate_caps: calculation of the outputscaled size error
0:00:00.581205149 12709 0x23119990 WARN basetransform gstbasetransform.c:1355:gst_base_transform_setcaps: transform could not transform video/x-raw, format=(string)GRAY8, width=(int)1600, height=(int)1300, framerate=(fraction)60/1, pixel-aspect-ratio=(fraction)1/2147483647, colorimetry=(string)1:4:7:1, interlace-mode=(string)progressive in anything we support
0:00:00.582124068 12709 0x23119990 ERROR nvvidconv gstnvvconv.c:2218:gst_nvvconv_fixate_caps: calculation of the outputscaled size error
0:00:00.582198756 12709 0x23119990 WARN basetransform gstbasetransform.c:1355:gst_base_transform_setcaps: transform could not transform video/x-raw, format=(string)GRAY8, width=(int)1600, height=(int)1300, framerate=(fraction)60/1, pixel-aspect-ratio=(fraction)1/2147483647, colorimetry=(string)1:4:7:1, interlace-mode=(string)progressive in anything we support

Honey_Patouceul · May 28, 2021, 12:27pm

Seems the difference is the rescaling to 400x400. You may try to remove it for checking.
Furthermore, is 1600x1300@60fps a native mode of your camera ?

v4l2-ctl -d0 --list-formats-ext

magnus.gabell · May 28, 2021, 12:29pm

Yes. 1600x1300 is native 60FPS (according to supplier)

ioctl: VIDIOC_ENUM_FMT
Index : 0
Type : Video Capture
Pixel Format: ‘GREY’
Name : 8-bit Greyscale
Size: Discrete 1600x1300
Size: Discrete 1600x1080
Size: Discrete 1920x1080

Index       : 1
Type        : Video Capture
Pixel Format: 'Y10 '
Name        : 10-bit Greyscale
	Size: Discrete 1600x1300

Index       : 2
Type        : Video Capture
Pixel Format: 'Y16 '
Name        : 16-bit Greyscale
	Size: Discrete 1600x1300

magnus.gabell · May 28, 2021, 12:38pm

Thanks for your support. Your small short program helped me.
I got it working somehow:

G_STREAM_TO_UPD = v4l2src device=/dev/video0 ! video/x-raw , width=(int){} , height=(int){} , format=(string){} , framerate=(fraction)60/1 ! timeoverlay ! nvvidconv ! video/x-raw(memory:NVMM), width=(int){}, height=(int){}, format=I420 ! tee name=t t. ! queue! nvjpegenc ! rtpjpegpay ! udpsink host=127.0.0.1 port=5000 t. ! queue! nvvidconv ! video/x-raw, format=BGRx ! videoconvert ! video/x-raw, format=BGR, width=(int){} , height=(int){} ! appsink

Dont ask me how. But perhaps when rewriting it had to do with some parsing of characters or something.

Anyways. Now it works!

Honey_Patouceul · May 28, 2021, 12:38pm

I don’t see frame intervals… Be sure to use list-formats-ext

What gives:

# Without rescale:
gst-launch-1.0 -v v4l2src device=/dev/video0 ! video/x-raw,format=GRAY8,width=1600,height=1300,framerate=60/1 ! timeoverlay ! nvvidconv ! video/x-raw(memory:NVMM) ! nvvidconv ! video/x-raw,format=GRAY8 ! xvimagesink

# With rescale:
gst-launch-1.0 -v v4l2src device=/dev/video0 ! video/x-raw,format=GRAY8,width=1600,height=1300,framerate=60/1 ! timeoverlay ! nvvidconv ! video/x-raw(memory:NVMM) ! nvvidconv ! video/x-raw,format=GRAY8,width=400,height=400,pixel-aspect-ratio=1/1 ! xvimagesink

# With 2 steps rescaling:
gst-launch-1.0 -v v4l2src device=/dev/video0 ! video/x-raw,format=GRAY8,width=1600,height=1300,framerate=60/1 ! timeoverlay ! nvvidconv ! video/x-raw(memory:NVMM) ! nvvidconv ! video/x-raw(memory:NVMM),format=GRAY8,width=800,height=650,pixel-aspect-ratio=1/1 ! nvvidconv ! video/x-raw,format=GRAY8,width=400,height=400,pixel-aspect-ratio=1/1 ! xvimagesink

magnus.gabell · May 28, 2021, 12:39pm

thanks. This will also help me further!

Many thanks!

magnus.gabell · May 28, 2021, 1:02pm

magnus@magnus-desktop:~$ v4l2-ctl -d0 --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
Index : 0
Type : Video Capture
Pixel Format: ‘GREY’
Name : 8-bit Greyscale
Size: Discrete 1600x1300
Size: Discrete 1600x1080
Size: Discrete 1920x1080

Index       : 1
Type        : Video Capture
Pixel Format: 'Y10 '
Name        : 10-bit Greyscale
	Size: Discrete 1600x1300

Index       : 2
Type        : Video Capture
Pixel Format: 'Y16 '
Name        : 16-bit Greyscale
	Size: Discrete 1600x1300

magnus.gabell · June 7, 2021, 8:57am

For reference:
This works:

GStreamer woring on Client:
gst-launch-1.0 udpsrc port=5000 ! application/x-rtp, encoding-name=JPEG,payload=26 ! rtpjpegdepay ! jpegdec ! autovideosink

Server:

gst-launch-1.0 v4l2src ! ‘video/x-raw, width=(int)1600, height=(int)1300, framerate=(fraction)60/1, format=(string)GRAY8’ ! nvvidconv ! nvjpegenc ! jpegparse ! rtpjpegpay ! udpsink host=192.168.1.xxx port=5000

magnus.gabell · June 7, 2021, 9:20am

Not completely working.
I run this on server where camera is and it works great:

gst-launch-1.0 v4l2src ! ‘video/x-raw, width=(int)1600, height=(int)1300, framerate=(fraction)30/1, format=(string)GRAY8’ ! nvvidconv ! ‘video/x-raw(memory:NVMM), format=I420’ ! nvjpegenc ! jpegparse ! rtpjpegpay ! udpsink host=192.168.1.xxx port=5000 -e -v

On the client side I save the images to a folder:

gst-launch-1.0 -v udpsrc port=5000 ! application/x-rtp, encoding-name=JPEG,payload=26 ! identity sync=true ! rtpjpegdepay ! jpegdec ! queue ! multifilesink location=c:\temp\Image01_%09d.jpg

This multifilesink saves the images but I cant open them. It says something like: “The fileformat does not seem to be supported”. So I dont know how to save them so that the system understands its a jpeg file

The Imagesink in previous replies works like a charm and I seem to get something. Any thoughts?

br,
Magnus

Honey_Patouceul · June 7, 2021, 10:51am

If you use jpegdec, the frames are decoded video, no longer jpeg images.
Removing jpegdec from the receiver should allow to save correct jpeg files.

Topic		Replies	Views
Simple Gstream USB Camera Live View Jetson AGX Orin gstreamer	12	2973	July 25, 2023
Gstreamer TCPserversink 2-3 seconds latency Jetson Xavier NX camera , opencv , gstreamer	12	18231	October 3, 2021
Help improving gstreamer pipeline on Xavier AGX Jetson AGX Xavier gstreamer	12	1415	October 18, 2021
GStreamer pipeline hangs after hours due to errors Jetson Xavier NX gstreamer	17	5693	October 18, 2021
DeepStream, Python USB cam sample hangs DeepStream SDK	38	1373	October 12, 2021
recording video stream from usb cameras Jetson Nano	13	4794	October 18, 2021
Gstreamer decode live video stream with the delay difference between gst-launch-1.0 command and appsink callback Jetson TX1	11	11634	October 18, 2021
USB CAM problem DeepStream SDK camera , usb , gstreamer	13	1884	October 12, 2021
Receive RTP (Pipeline demo) DeepStream SDK gstreamer	23	1642	August 14, 2023
Testing usb cam on deepstream test1 usb-cam app without pgie config file DeepStream SDK camera , jetson , deepstream , jetson-nano	12	46	September 11, 2024

Gstreamer UDP sink using python

Working:

Not Working:

Related topics