What is the recommended way to pass Nvdsmetadata between different branches of a pipeline split using tee?
This is what my pipeline looks like:
source->nvstreammux->gsttee->|->nvvideoconvert->custom-plugin
|->custom-dbsink->etc…
Custom-dbsink plugin is supposed to get metadata from custom plugin and write to db, right now it does not receive metadata from custom-plugin
Why the tee out of curiosity? Can you not do:
source ! nvstreammux ! nvvideoconvert ! custom-plugin ! custom-dbsink
… if the sink needs to have the metadata from the plugin?
Is there a way of passing different crops of the video to custom-plugin / custom-dbsink with the pipeline u mentioned?
custom-plugin uses a crop of the original src video while the dbsink needs to save the full frame of the video thus we have split the pipeline as such,
The nvvideoconvert right before the custom-plugin handles this cropping.
Couple questions.
Reason I ask is because you might be able to save a lot of resources by just remuxing your video stream into a file container and storing your crops as frame numbers/timecodes and crop coordinates.
Cropping as I do not want to process the entire frame in custom plugin.
The full frame is stored as jpeg, source is typically from an rtsp stream. Also the db will be queried by a web page to display the frames thus I do not store the video file along with the crop coordinates.
Re: cropping… Why don’t you want to process the entire frame? If your idea is to perform a secondary inference, the nvinfer element operates in secondary mode on the metadata (eg. Bounding boxes) from the first. So for example the primary network can detect cars and the secondary can classify the make and model. IIRC DeepStream test 2 does this.
Re: Jpeg… It isn’t a very effecient codec for storing video. It’s a spatial codec – good for photos – not video. You’ll spend a fortune on storage compared to h264/5. You can of course do it, but I suspect you’ll run into unexpected trouble this way.
If you want to view the video from the web, that’s possible, as is displaying the metadata live on top, either by sending it with the video to the browser or drawing boxes beforehand. Think YouTube and subtitles. You can even use a subtitle track to store your metadata if you want. Qtmux and matroskamux support this among others. DJI uses subtitles to store it’s drone metadata. GoPro has an open metadata format as well but you will have to make a new gstreamer element for that if you go that route.
You can of course choose to roll your own thing your way, but Nvidia provides various out of the box solutions for what it seems like you want to do. You might try getting some ideas from how their examples are designed and see if any fit your use case.
So in conclusion there’s no way to pass metadata across the tee branches or sync between them such that both can see the metadata right?
Sure there is. It’s C and you have the source code. However keep in mind after a tee element there need to be queues, and queues are thread boundaries, so if you need to synchronize common resources across threads you’ll probably need to use standard GLib synchronisation primitives (correct me if I’m wrong, Nvidia).
There may by some other ways, such as some resynchronization elements but I am unsure of any off hand. Nvidia staff may have some ideas. There is always a way. I was merely trying to recommend alternatives that are tested, supported, and might save you some work.
Please don’t take my advice as “you have to do it this way”. You are of course free to modify the program the way you want.
Thanks for taking the time to answer my questions!
Hi
you can crop the objects in the custom plugin without modifying the original frame and then work with it. If you need to attach the cropped object as Metadata, you can refer to DeepStream user metadata sample. you can refer to nvinfer or dsexample source code where objects are cropped and processed internally without modifying the original frame.
Thanks!