Reshaping input data shape in gstdsexample

I wish to divide the input image/data into two parts horizontally and then stitch them vertically. I have modified the code of deepstream reference app gstdsexample plugin for my use. I saw a similar thing is being done in get_converted_mat() function in gstdsexample. I am comfortable performing operations on OpenCV mat and using OpenCV functions. So my question is if I just perform my operations on dsexample->cvmat after it has been converted to RGB format in get_converted_mat() function, Will my final rendered output will also reflect the changes made or do I need to do something else?

I am not that comfortable with gstreamer so kindly be as descriptive as possible. Thanks

Hi,
Please check architecture of deepstream-app

So your source is jpeg files. assume the resolution is 1920x1080, you would like to composite it into 960x2160 before nvinfer? Or keep 1920x1080 till nvdsosd and do composite before nveglglessink?

My source is rtsp streams.I want to do something like this.
Let’s say I have a 720x 360 input stream. After passing from nvinfer, I access it via gstdsexample just before nvtracker,nvosd and nveglglesink. I use the deepstream reference app code for that. Now I want to crop the image into two parts of 720x180 each and then stich them together to get a 1440x180 reshaped data and then pass it to nvdtracker, nvdosd etc and ultimately render it on screen. I was planning to use OpenCV for cropping and stitching purposes. I found that, in get_converted_mat function of gstdsexample, a OpenCV mat buffer is being associated with the input data. So, if I perform my operations on just the cvmat buffer, will I get the desired result? Kindly guide me on how to get the desired result.

@DaneLLL Any update on the cropping issue ?

Hi,
The case is a bit complex and we have not thought of any solution now.
If we perform it after nvinfer, the metadata may not match with the video frames.

Compositing 720x360 into 1440x180 have to be done before nvinfer. Trying to work out a solution for this. If you have further thoughts about this solution, please also share it for discussion. Thanks,

@DaneLLL What metadata exactly are we concerned with ? The bbox dimensions ? I can change them accordingly via gstdsexample lib. No, I don’t want it to be done before nvinfer. After nvinfer anywhere is fine. Any other metadata should I be concerned with? and which can’t be handled/modified via dsexample plugin ? .

@DaneLLL I am planning to use OpenCV operations on dsexample->cvmat to perform stitching and other things. I am following this thread. As for metadata, I was thinking of calculating the new bbox coordinates myself in dsexample and replace the existing values. I don’t know about any other metadata which needs to be changed. KIndly help with this issue.

Hi,
If there is an object at [x,y,w,h]=[10,190,64,64] in 720x360. It becomes [730,10,64,64] after compositing into 1440x180. If you don’t handle this and keep the original matadata, nvtracker may not work properly.

And dsexample plugin does not support compositing 720x360 to 1440x180. This is not do-able in current implementation.

@DaneLLL Yeah, precisely. What if I change the metadata corresponding to the the bounding box dimensions in place inside dsexample. Also, can’t we use OpenCV operations on dsexample->cvmat to reshape the OpenCV mat.

Hi,
One possible solution is to run like
source(720x360) ! tee name=t t. ! queue ! nvvideoconvert(crop top 720x180) ! mx.sink_0 t. ! queue ! nvvideoconvert(crop bottom 720x180) ! mx.sink_1 nvstreammux width=720 height=180 batch-size=2 name=mx ! nvinfer ! nvtracker ! nvmultistreamtiler width=1440 height=180 rows=2 columns=1 ! ...

It has to divide into two sources before nvinfer. This is different from your requirement but for your reference.

Hi @DaneLLL, Can you please point out what issues might arise if we crop and resize dsexample->cvmat , cvtcolor it to dsexample->interbuf (RGBA) and then map it back to ip_surf by nvbufsurfacetransform. If it’s only related to bounding boxes related metadata, can’t we modify them as well in gstdsexample. The above given solution for the time being is not feasible to me. I would prefer not changing the pipeline before nvinfer. Also, my gstreamer skills are beginner’s at best, I am more comfortable with using OpenCV functions. Kindly help.

Hi,
The default implementation of dsexample does not support inputing 720x360 and outputing 1440x180, since it utilizes transform_ip callback. For supporting the case, you need to use transform callback. Please check
https://developer.gnome.org/gstreamer-libs/stable/GstBaseTransform.html#GstBaseTransformClass

There is no samples at hand and may see if other users can share experience.

Hi @DaneLLL
Can you share with me at least the broader points on how to achieve the desired behaviour and point to some sample codes. I see that transform_ip is for inplace operations on buffer. As the total buffer size isn’t changing, whether it’s 1440x180 or 720x360, perhaps the whole pipeline will not get affected by the transformation. Anyhow, as my knowledge on the subject is limited, any help from anyone in solving this issue is appreciated.

Hi,

I am afraid I cannot. We don’t have existing sample for this. May need other users to share guidance.

HI @DaneLLL,
If not reshaping of input data , can I at least replace the existing data of the buffer with some other data. Let’s say I get a 480x360 RGBA buffer in gstdsexample. I replace the buffer using some other 480x360 RGBA buffer using OpenCV operations. Is that possible? I f yes what’s the best way to do it? I am using gstdsexample and I am trying to use dsexample->cvmat to perform opencv functions on it. Lastly, I am remapping the buffer to ip_surf using NvBufSurfaceTransfor,m.

Hi,
It is possible. In the callback
gst_dsexample_transform_ip (GstBaseTransform * btrans, GstBuffer * inbuf);
inbuf is the input buffer. You can overwrite it with NvBufSurf APIs.

There is a similar reference: