Gst-nvvideoconvert losing metadata

[vcmike]

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU)
NVIDIA GTX3080
• DeepStream Version
nvcr.io/nvidia/deepstream:6.0-devel
• NVIDIA GPU Driver Version (valid for GPU only)
NVIDIA-SMI 495.46 Driver Version: 495.46 CUDA Version: 11.5
• Issue Type( questions, new requirements, bugs)

I have found an issue where the NvDsBatchMeta->NvDsFrameMeta->NvDsObjectMeta->parent pointers are null if accessed from any element downstream of a Gst-nvvideoconvert element where the GstCaps of the downstream element is not on the GPU (i.e. not video/x-raw(memory:NVMM)). This makes it very hard to correlate nested detection inferences.

For example:

• If the GstCaps of the element immediately following the Gst-nvvideoconvert is a video/x-raw then the all the rest of the NvDsBatchMeta will be present when read via gst_buffer_get_nvds_batch_meta function but all NvDsBatchMeta->NvDsFrameMeta->NvDsObjectMeta->parent pointers will be null.
• If the GstCaps of the element immediately following the Gst-nvvideoconvert is any (so the Gst-nvvideoconvert element does not actually copy the data from the GPU) then the relevant NvDsBatchMeta->NvDsFrameMeta->NvDsObjectMeta->parent pointers will be not-null and the parent object can be read.

It seems like there is an issue with Gst-nvvideoconvert that is causing this loss of data when copying from the GPU back to the CPU.

• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)

Here is a modified version of the back-to-back-detectors example from the deepstream_reference_apps repository.

Modifications:

• added an avenc_mpeg4 element after the nvvideoconvert to force the nvvideoconvert element to copy the data from the GPU to the CPU.
• removed the nsdsosd and nveglglessink elements.
• added a probe to the the sink of the nvvideoconvert (before the GPU->CPU operation) to prove the NvDsBatchMeta->NvDsFrameMeta->NvDsObjectMeta->parent pointers are still populated.
• added a probe to the the sink of the avenc_mpeg4 (after the GPU->CPU operation) to prove the NvDsBatchMeta->NvDsFrameMeta->NvDsObjectMeta->parent pointers are no longer present.

Here is sample code for reproduction (link will expire in 2 weeks):
https://pastebin.com/iLHsLswt

[Fiona.Chen]

NvDsBatchMeta contains important information for the batched data which is based on GPU buffer(there are information for HW buffers) . When you use nvvideoconvert to convert the GPU buffer to CPU buffer, we suppose you want to do something which are not batch data related (use other open source gstreamer plugins), so some batch information are not copied to CPU buffers.

Why do you need NvDsBatchMeta for CPU buffers? What is the background of the requirement? Why can’t you get NvDsBatchMeta before converting to CPU buffers?

[vcmike]

@Fiona.Chen

NvDsBatchMeta

I am able to get the NvDsBatchMeta before copying to the CPU buffer but this does not fit with the gstreamer architecture where the buffer is the state not a global/shared state. The bigger question is why do you remove only the parent pointers when you copy across everything else … including other frame level NvDsObjectMeta objects?

I am copying to the CPU so I can take crops from the frame based on the detections (hence needing to move data from the GPU to CPU) as well as do further data processing based on the NvDsBatchMeta (and in particular the NvDsObjectMeta objects).

The problem is that due to the data structure of the NvDsBatchMeta without the parent pointers it is not possible to correlate secondary inferences back to a primary inference.

[Fiona.Chen]

We already have the sample of how to get cropped objects from GPU buffer. You don’t need to do this after copy the buffer to CPU. Please refer to the sample of deepstream-image-meta-test.

[vcmike]

Thank you @Fiona.Chen . I have successfully implemented the nvds_obj_enc_process pattern.

I still think you should investigate why Gst-nvvideoconvert is dropping the pointers but at least I have a solution for now.