Rtsp stream & meta data synchronization

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU)
• DeepStream Version
• JetPack Version (valid for Jetson only)
• TensorRT Version
• NVIDIA GPU Driver Version (valid for GPU only)
• Issue Type( questions, new requirements, bugs)
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

Jetson Orin AGX 64 Gbyte
Deepstream 6.4
Jetpack 6.0 DP

How can we synchronize metadata (object bounding boxes, classes etc) with an outgoing rtsp stream - e.g. for annotation later down the pipeline?

Is there a way to add e.g. the frame index to an outgoing RTSP stream, so that each frame there can be uniquely associated with metadata (boxes, object classes etc) for it?

the same question has been raised here in 2021:

Has there been any development on this since then?

As in Deepstream - synchronize rtsp output and message brokers - Intelligent Video Analytics / DeepStream SDK - NVIDIA Developer Forums, the metadata and RTSP streams are outside of DeepStream pipeline. You may need to synchronize them by timepstamp or other ways.

RTSP is live streaming, frame index is no meaning.
Please check with RTSP spec.

The stream starts at the same time I am sending messages (with a frame index), so if frames get transmitted, of course that index could be useful to synchronize them - so curious how one would go about an synchronize the messages and the rtsp stream?

This is your customized protocol, it is not included in the RTSP stream. Please implement your own customization in your own way. DeepStream will not conflict to your customization.

Do you know if this module will be availble for the Jetson Orin AGX?

https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvdsmetautils.html?highlight=SEI#id5

It might be useful for adding metadata to the rtsp stream, but in another post, it was said that this isnt part of JetPack?

It is not supported in Orin now. We may add the feature in the future.

Please refer to How to get the SEI payload in Deepstream pipeline - Intelligent Video Analytics / DeepStream SDK - NVIDIA Developer Forums for how to extract the SEI payload by GStreamer APIs.