For our application we need to output softlabels (score/probability for every class per detection). But after experimenting and looking at the documentation, it seems to only support hardlabels (single highest score per detection), at least according to the NvDsObjectMeta — Deepstream Deepstream Version: 8.0 documentation which only shows one confidence field.
Questions:
Are softlabel outputs supported out-of-the box by NvInfer and NvTracker?
Is there a recommended approach/example?
One approach could be to add a custom parser to NvInfer which stores all the scores in the meta, but then how will this used by the tracker?
Generally, the model output tensor contains the scores/probabilities for each classes, and I think using the output tensor instead of NvDsObjectMeta would better suit your needs.
No.
If you absolutely must do this, you could consider implementing a custom low-level tracker library.
However, you can describe your goal in more detail; there might be a better approach.
Of course, our robotic perception system is interested in the scores of all classes per track because it triggers specific behaviours for each class.
For example, for any detection:
class A, score > 50% → do something
class B, score > 20% → do another thing
Generally, the model output tensor contains the scores/probabilities for each classes, and I think using the output tensor instead of NvDsObjectMeta would better suit your needs.
So it sounds like the trackers do not support softlabels. This is not a major issue if they do not track based on the softlabels, as long as the tracker works/is robust. But we do need the softlabels per track eventually at the end of the pipeline.
My understanding now is, if we provide the hardlabel/final classification to the tracker it should work. Then we can extract the softlabels/raw tensors, pass those via NvDsUserMeta or some buffer, then at the end of the pipeline in the sink, extract from the buffer using something like pyds.gst_buffer_get_nvds_batch_meta().
Does that sound reasonable/possible?
If you absolutely must do this, you could consider implementing a custom low-level tracker library.
One of the main values of deepstream for us is leveraging the included trackers. We’ve implemented our own tracker and would like to move away from this if possible.
Please let me know if you need anymore information,