Using yolo-pose and yolo-seg together in one python app

[md44]

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) GPU
• DeepStream Version 7.0
• JetPack Version (valid for Jetson only)
• TensorRT Version
• NVIDIA GPU Driver Version (valid for GPU only)
• Issue Type( questions, new requirements, bugs)
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

Hi,

I would like to use both yolo-seg and yolo-pose as stages before the tracker, is it possible?
what exactly I need to set in the config files in order to do that?
how do I access the mask in the prob function?
I see that both pose and seg are using obj_meta.mask_params, so how to make each use its own?

thanks

[junshengy]

I think it is possible, you can refer to this sample.

NVIDIA-AI-IOT/deepstream_reference_apps/blob/master/back-to-back-detectors/back_to_back_detectors.c

Just configure it as back-to-back instance segmentation.

[Fiona.Chen]

@md44

I would like to use both yolo-seg and yolo-pose as stages before the tracker, is it possible?

nvtracker can only track objects with bboxes. If the yolo-seg or yolo-pose can output bboxes to nvtracker, you can put them before nvtracker.

@md44

I see that both pose and seg are using obj_meta.mask_params, so how to make each use its own?

Where did you see such code?

[md44]

for segmentation here:

NVIDIA-AI-IOT/deepstream_python_apps/blob/216ce3a08a61566cb3abcffdc0ddae30a57eb6b0/apps/deepstream-segmask/deepstream_segmask.py#L132C22-L132C36

for pose esimation here:

marcoslucianops/DeepStream-Yolo-Pose/blob/9cf44737d2effcaecea599d122bb2f553493a17d/deepstream.py#L99C1-L99C53

[Fiona.Chen]

@md44

how do I access the mask in the prob function?

DeepStream uses Metadata in DeepStream to store and transfer information through the elements after nvstreammux plugin in the pipeline. You can get the meta data in the pad probe functions. There are lots of DeepStream sample applications, almost every sample application has the sample code of getting the batch meta, frame meta, object meta,…

The python samples are also available: NVIDIA-AI-IOT/deepstream_python_apps: DeepStream SDK Python bindings and sample applications (github.com)

[md44]

Thank you for your reply, the problem is as I showed in my last reply it seems that both pose estimator and segmenter get their data from the same object: obj_meta.mask_params.get_mask_array(), since I need both the pose estimator and segmenter output how do I get the correct data for each model?

[Fiona.Chen]

According to the sample you refers to. Both of the two models are instance segmentation models while the NvDsObjectMeta supports only one mask array. The suggestion is to change one of the model to the “unknown type” and set the mask array in the customized user meta NvDsUserMeta in the NvDsObjectMeta. Then the two mask arrays are all available.

[md44]

Thank you, it works.