Bugged frames using Gstreamer and vpiPerspectiveWarp

Hi,
I’m sending a video stream with gstreamer in h264, it is working fine, but if I translate (move the image on the x axis) it displays fine on the NX (with the translation). The application that receive the stream contains bugged frames, this behaviour is totally random. Here is a the kind of frame that we receive that alternates with the correct ones.

After much investigation on this issue, we think that it comes from the vpiPerspectiveWarp function. It may create frames that don’t suit our gstreamer pipeline. we are using VPI_INTERP_LINEAR and VPI_BORDER_ZERO on the vpiSubmitPerspectiveWarp function.
We tried to resize the frame to match the size defined in our Gstreamer pipeline, we also adjusted the nvvidconv gstreamer plugins with left=0, right=960, top=0, bottom=540. Both of these methods didn’t succeed.
Here is our pipeline :

std::string gst_out = "appsrc is-live=1 ! video/x-raw, width=960, height=540 ! videoconvert ! videoscale ! nvvidconv left=0 right = 960 top = 0 bottom=540 ! video/x-raw(memory:NVMM), width=960, height=540 ! nvv4l2h264enc insert-sps-pps=1 idrinterval=15 maxperf-enable=1 control-rate=1 peak-bitrate=4100000 ratecontrol-enable=1 preset-level=1 ! queue leaky=0 ! h264parse ! rtph264pay ! udpsink host=IP port=PORT sync=0 async=0"

Jetpack 4.5 + Opencv 4.1.1.
Thank you

One possibility may be the socket kernel buffer max size being a bit short. Does the following improve ?

std::string gst_out = "appsrc is-live=1 ! video/x-raw, width=960, height=540 ! queue ! videoconvert ! videoscale ! nvvidconv left=0 right = 960 top = 0 bottom=540 ! video/x-raw(memory:NVMM), width=960, height=540 ! nvv4l2h264enc insert-sps-pps=1 idrinterval=15 maxperf-enable=1 control-rate=1 peak-bitrate=4100000 ratecontrol-enable=1 preset-level=1 ! queue leaky=0 ! h264parse ! rtph264pay ! udpsink host=IP port=PORT buffer-size=33554432 sync=0 async=0"

If not working, you may tell what format your app is sending and try specifying this in appsrc SRC caps.
You may not need the videoscale and full crop. You may tell if this gives additional errors:

std::string gst_out = "appsrc is-live=1 ! video/x-raw, width=960, height=540 ! queue ! videoconvert ! nvvidconv ! nvv4l2h264enc insert-sps-pps=1 idrinterval=15 maxperf-enable=1 control-rate=1 peak-bitrate=4100000 ratecontrol-enable=1 preset-level=1 ! queue leaky=0 ! h264parse ! rtph264pay ! udpsink host=IP port=PORT buffer-size=33554432 sync=0 async=0"

Hi Honey_Patouceul,
Thank you for your fast reply,
I tried both pipelines and it didn’t improve my issue.

you may tell what format your app is sending and try specifying this in appsrc SRC caps.

I am not sure about how to do this, do i have to specify the format parameter ? Arguments seems weird to me.

$ gst-inspect-1.0 appsrc

Factory Details:
Rank none (0)
Long-name AppSrc
Klass Generic/Source
Description Allow the application to feed buffers to a pipeline
Author David Schleef ds@schleef.org, Wim Taymans wim.taymans@gmail.com

Plugin Details:
Name app
Description Elements used to communicate with applications
Filename /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstapp.so
Version 1.14.5
License LGPL
Source module gst-plugins-base
Source release date 2019-05-29
Binary package GStreamer Base Plugins (Ubuntu)
Origin URL gst-plugins-base1.0 package : Ubuntu

GObject
±—GInitiallyUnowned
±—GstObject
±—GstElement
±—GstBaseSrc
±—GstAppSrc

Implemented Interfaces:
GstURIHandler

Pad Templates:
SRC template: ‘src’
Availability: Always
Capabilities:
ANY

Element has no clocking capabilities.

URI handling capabilities:
Element can act as source.
Supported URI protocols:
appsrc

Pads:
SRC: ‘src’
Pad Template: ‘src’

Element Properties:
name : The name of the object
flags: readable, writable
String. Default: “appsrc0”
parent : The parent of the object
flags: readable, writable
Object of type “GstObject”
blocksize : Size in bytes to read per buffer (-1 = default)
flags: readable, writable
Unsigned Integer. Range: 0 - 4294967295 Default: 4096
num-buffers : Number of buffers to output before sending EOS (-1 = unlimited)
flags: readable, writable
Integer. Range: -1 - 2147483647 Default: -1
typefind : Run typefind before negotiating (deprecated, non-functional)
flags: readable, writable, deprecated
Boolean. Default: false
do-timestamp : Apply current stream time to buffers
flags: readable, writable
Boolean. Default: false
caps : The allowed caps for the src pad
flags: readable, writable
Caps (NULL)
size : The size of the data stream in bytes (-1 if unknown)
flags: readable, writable
Integer64. Range: -1 - 9223372036854775807 Default: -1
stream-type : the type of the stream
flags: readable, writable
Enum “GstAppStreamType” Default: 0, “stream”
(0): stream - GST_APP_STREAM_TYPE_STREAM
(1): seekable - GST_APP_STREAM_TYPE_SEEKABLE
(2): random-access - GST_APP_STREAM_TYPE_RANDOM_ACCESS
max-bytes : The maximum number of bytes to queue internally (0 = unlimited)
flags: readable, writable
Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 200000
format : The format of the segment events and seek
flags: readable, writable
Enum “GstFormat” Default: 2, “bytes”
(0): undefined - GST_FORMAT_UNDEFINED
(1): default - GST_FORMAT_DEFAULT
(2): bytes - GST_FORMAT_BYTES
(3): time - GST_FORMAT_TIME
(4): buffers - GST_FORMAT_BUFFERS
(5): percent - GST_FORMAT_PERCENT
block : Block push-buffer when max-bytes are queued
flags: readable, writable
Boolean. Default: false
is-live : Whether to act as a live source
flags: readable, writable
Boolean. Default: false
min-latency : The minimum latency (-1 = default)
flags: readable, writable
Integer64. Range: -1 - 9223372036854775807 Default: -1
max-latency : The maximum latency (-1 = unlimited)
flags: readable, writable
Integer64. Range: -1 - 9223372036854775807 Default: -1
emit-signals : Emit need-data, enough-data and seek-data signals
flags: readable, writable
Boolean. Default: true
min-percent : Emit need-data when queued bytes drops below this percent of max-bytes
flags: readable, writable
Unsigned Integer. Range: 0 - 100 Default: 0
current-level-bytes : The number of currently queued bytes
flags: readable
Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 0
duration : The duration of the data stream in nanoseconds (GST_CLOCK_TIME_NONE if unknown)
flags: readable, writable
Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 18446744073709551615

Element Signals:
“need-data” : void user_function (GstElement* object,
guint arg0,
gpointer user_data);
“enough-data” : void user_function (GstElement* object,
gpointer user_data);
“seek-data” : gboolean user_function (GstElement* object,
guint64 arg0,
gpointer user_data);

Element Actions:
“push-buffer” : GstFlowReturn user_function (GstElement* object,
GstBuffer* arg0);
“push-buffer-list” : GstFlowReturn user_function (GstElement* object,
GstBufferList* arg0);
“push-sample” : GstFlowReturn user_function (GstElement* object,
GstSample* arg0);
“end-of-stream” : GstFlowReturn user_function (GstElement* object);

Camera info :
v4l2-ctl --list-formats-ext -d /dev/video0 :
ioctl: VIDIOC_ENUM_FMT
Index : 0
Type : Video Capture
Pixel Format: ‘YUYV’
Name : YUYV 4:2:2
Size: Discrete 1920x1080
Interval: Discrete 0.017s (60.000 fps)

Hi,
Not sure how you use VPI functions. Do you call the functions in appsrc? If possible, please share a test sample so that we can reproduce it and do investigation.

And please share the release version for reference

$ head -1 /etc/nv_tegra_release 

Hi DaneLLL,
Sorry for late reply as I wasn’t in the office.
Here is what you requested :
$ head -1 /etc/nv_tegra_release
head: cannot open ‘1’ for reading: No such file or directory
==> /etc/nv_tegra_release <==
#R32 (release), REVISION: 5.0, GCID: 25531747, BOARD: t186ref, EABI: aarch64, DATE: Fri Jan 15 23:21:05 UTC 2021
$

For more precision, i’m working on a production module with a Quark board from Connecttech. My Jetpack version is 4.5 and I can’t upgrade it, since the BSP can’t support superior version.

Hi,
For further investigation we would need to reproduce the issue. There are existing samples in

/opt/nvidia/vpi1/ 

Please check if you can share a patch to either sample so that we can give it a try.

Hi DaneLLL,
I reproduced my usage of perspective warp from the vpi_sample_perspective_warp, with minimal variables and it is working as expected.
So the issue doesn’t come from the function or my usage with it. I am thinking about threads conflict, or lack of performance from the receiver.
Thank you

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.