Deepstream 6.1, nvtracker plugin - different behaviour on gpu 0 VS gpu 1 on multi-GPU system

Please provide complete information as applicable to your setup.

**• Hardware Platform - GPU (2 x NVIDIA GeForce RTX 3090)
**• DeepStream Version - 6.1
• JetPack Version (valid for Jetson only)
**• TensorRT Version - v8401
**• NVIDIA GPU Driver Version - 515.105.01
**• Issue Type - bug
**• How to reproduce the issue ? - using example deepstream-test2

When running Deepstream6.1 pipeline on a 2 GPU (RTX3090) server, the behavior of nvtracker plugin differs significantly between GPU 0 Vs GPU 1. It looks like on GPU 1 nvtracker fails to re-acquire object track if it was not detected by object detector on more than 2-3 frames (while maximum allowed shadowTrackingAge parameter was set to 15)

The issues is pretty easy to reproduce in the following way :

  • use deepstream-test2 sample from deepstream6.1 library
  • add detected objects printout to osd_sink_pad_buffer_probe function (deepstream_test2_app.c) :
    [line 119] printf(“VID[%u] FID[%d] CID[%d] OID[%lu] BBOX[%.3f %.3f %.3f %.3f]\n”, frame_meta->source_id, frame_meta->frame_num, obj_meta->class_id, obj_meta->object_id, obj_meta->, obj_meta->rect_params.left, obj_meta->rect_params.width, obj_meta->rect_params.height);

After running the example on GPU 0 and GPU 1 (you will need to change gpu-id settings in all configuration files inside the sample folder from “gpu-id : 0” to “gpu-id: 1”) you can notice that :

  • number of different object instances is the same
  • last object id on GPU 1 is ~1.5X higher compared to the results on GPU 0

Basically this means that on GPU0 tracker follows the same object track for longer.
It is also easy to find specific tracks where this happens - for example, in my case, I have found a track which is continuously followed on GPU 0 for ~400 frames, while on GPU 1 it is covered by 4 different object ids. The change in object id always happens when the object was not detected by object detector for more than 2-3 frames (on GPU 0 the same tracker is able to correctly re-id the object even when it was not detected for 10 frames)

Please advice on ways to fix this issue or some sort of a workaround.

Thanks, Simon.

The latest release version of DeepStream is DS6.2. Did you observe the same issue on DS6.2?

I have checked DS6.2 version and I see the same behviour

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

Which tracker config do you use? Do you use NvDCF_perf? Do you observe the same issue with DvSORT or NvDeepSORT?


Can you check the GPU usage with: $ nvidia-smi ? We want to check if any process still running on GPU 0.

Do you observe the same issue with this setting: export CUDA_VISIBLE_DEVICES=1 ?

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.