Saving meta data matching SmartRecord files

Yeah it would take custom elements. At least one to perform the meta conversion (an element with a subtitle src pad as well as video) and another to parse it for playback, but there are already subtitle overlays elements (either nvidia or stock GStreamer) that you could be modified to do this as you note.

In the case of nvdsosd it might be more difficult since you would have to regenerate and attach Nvidia’s metadata. At the moment, neither nvstreammux or nvdsosd are not open source so it’d just be guessing at how that’s done.

It would work. I think you would either have to modify an existing GStreamer subtitle element (and license it under the LGPL) or roll your own metadata parser to de-serialize and re-attach NvDsDisplayMeta for the nvidia osd. The former is possibly a better solution since your player can run on anything that GStreamer can, and you don’t have to guess at how the metadata is supposed to be attached. It would be really nice if nvdsosd and nvstreammux were open source, Nvidia.

1 Like

In the case of nvdsosd it might be more difficult since you would have to regenerate and attach Nvidia’s metadata. At the moment, neither nvstreammux or nvdsosd are not open source so it’d just be guessing at how that’s done.

Attaching DeepStream metadata is not that difficult, we are doing it based on this sample:

/opt/nvidia/deepstream/deepstream-4.0/sources/apps/sample_apps/deepstream-gst-metadata-test/deepstream_gst_metadata.c

And the resulting buffer works fine with nvdsosd

Thanks Miquel, that is good to know. It solves the question on how to redraw the saved meta data.

What is your thoughts on saving the actual meta data to align with SmartRecord. Is the SmartRecord “bin surgery” feasible to add elements to save the meta data, or do you have any other suggestions? It would be best of Nvida could just support saving meta data with SmartRecord, or open source it altogether.

Regards

I haven’t do this before, but I can think of 3 options:

  1. Same as @mdegans, serializing the metadata in JSON format and saving it in the subtitle track. We have used the JSON serialization trick before to sent the metadata as a signal between applications and as input for a meta transfer element.
  2. Also using JSON but saving it by extending JPEG/EXIF or using one of its fields. jifmux can be used as reference.
  3. I know you can save data in the Supplemental enhancement information (SEI) H264 NAL unit. I don’t know the limitations of this, but assuming it works, you could add the whole metadata memory in a hacked encoder and extract it in the decoder.
1 Like

Thanks for pointing that out. From the readme:

This sample also explains how nvv4l2decoder can propagate gst metadata attached before it.
To demonstrate this, h264parse_src_pad_buffer_probe() attaches parser frame number as metadata to gstreamer buffer on src pad of h264parse element. Refer to h264parse_src_pad_buffer_probe() function in source code.

nvstreammux component, upon receiving gst buffer at its sinkpad, transforms
gst meta attached by decoder and h264parse elements into nvds user metadata at frame level. In this way we maintain the association of frame metadata attached before
and after nvstreammux element.

So that’s useful for sure. It looks like some meta can be attached before nvstreammux, however it says in the source:

The decoder can not attach to NvDsBatchMeta metadata because batch level metadata is created by nvstreammux component.`

Do you know if nvosd requires NvDsBatchMeta to draw NvDsDisplayMeta? If so, there can always be a nvstreammux element in the middle since it looks like custom meta can be made to survive the trip through. So worst case scenario, something like this on the player end?

source ! demux ! subtitle_to_custom_meta ! ... ! nvstreammux ! ... ! custom_meta_to_display_meta ! nvdsosd ! ... ! sink

with some other stuff thrown in there (tiler maybe), multiple source bins

Yes, NvDsBatchMeta is needed. What we do to add all the base meta structures is to pass the buffer through an nvstreammux and then add the other meta fields. We tried creating the NvDsBatchMeta in our element but we couldn’t figure how to add it with the current DeepStream meta API.

1 Like

Time synchronization code needs to be written by user so that the corresponding bounding box can be overlaid on the corresponding frame. One of the ways to achieve this would be - While saving the file to the disk specify NTP timestamp as part of the filename and then while achieving sync with metadata from database, use NTP timestamp field from metadata to match with the one extracted from filename.

Hey Jason, streammux can support different resolution input streams.

1 Like