Include feature extractor pipeline

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) NVIDIA GeForce RTX 3090
• DeepStream Version 6.3
• JetPack Version (valid for Jetson only)
• TensorRT Version 8.6.1.6
• NVIDIA GPU Driver Version (valid for GPU only) 535.113.01
• Issue Type( questions, new requirements, bugs) Question
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing) I am using the deepstream-demux-multi-in-multi-out sample app from deepstream_python_apps. My goal is to develop an application that has multiple cameras as inputs and performs a tracking. Not only an individual tracking for each camera, but also a global, in case a same object is in multiple cameras to identify it as unique (same ID in all of them). For that I have added a tracker in the sample application and in the osd_sink_pad_buffer_probe function I have added a feature extractor. This feature extractor extracts the features of each detected object and I store them to compare all of them and check the similarity between them. If the similarity score is above a threshold I consider that both objects are the same and I change the ID of one of them depending the age.

Here comes my question. The app works quite well, however, in terms of fps, since I have added the extractor in the probe function, the performance decreases a lot. Is there anyway to include this feature extractor, is an osnet model, in the pipeline to increase the performance? Or is there any other way to perform multi camera tracking?

I attach the script and the configuration files.

Thank you,

Mikel

deepstream_demux_multi_in_multi_out_custom.txt (24.9 KB)
tracker_config.txt (1.2 KB)
config_infer_primary_yoloV5.txt (1.0 KB)

I know it is the cross cameras tracking case.

Is the osnet model deployed with TensorRT?

We have customization APIs with nvtracker Gst-nvtracker — DeepStream 6.3 Release documentation

We already have solution for multi camera tracking Metropolis - Multi-camera Tracking (nvidia.com)

The model is defined as:
extractor = torchreid.utils.FeatureExtractor(model_name="osnet_x1_0", model_path="./weights/osnet_x1_0_imagenet.pth", device="cuda")
And it is used as follows:
feat = extractor(frame_crop)[0].cpu().numpy()

Is there any example for python or something else that can help me?

I have already seen that demo, but how can I use it? The Launch App doesn’t work and I haven’t found any code or info about how it is developed.

DeepStream is based on TensorRT, please convert your osnet model from pytorch to onnx.

There is no nvtracker low API customization sample but the gst-nvtracker is open source. Please investigate the interface carefully.

For the multi camera tracking demo, please request access from The Multi-Camera Tracking AI Workflow | NVIDIA

Ok, I would try it, thank you!

Perfect, thanks!