Query on PeopleNet-Transformer model constantly misses bounding boxes - Calibration Accuracy vs Unified ID Reassociation Stability in MV3DT

Accelerated Computing Intelligent Video Analytics TAO Toolkit
1

Hi Team,
This is in reference to below issue,

I was trying to use Deepstream to track passengers inside lounge.

But we can see our unified IDs keeps on switching when passenger walks along the path.

I thought its calibration issue.

I was already co-ordinating with Kesong, on this issue but then he referred that the issue is with detection.

You can see below comment from him when he tried to run the same videos,

`Person 9 corresponds to the woman with black clothes, and I noticed that PeopleNet-Transformer model constantly misses bounding boxes. This is a detection problem, and I feel it might need some fine-tuning. I would advise to submit topic in TAO Forum for that.`

So kindly can you help me on this.

Most of the details are already present in the above Query.

Please let me know if any other information is needed.

3

For the peoplenet-transfer model, it is trained from D-DETR network. (model card is: GPU-optimized AI, Machine Learning, & HPC Software | NVIDIA NGC | NVIDIA NGC)
You can refer to D-DETR doc and notebook. Deformable DETR - NVIDIA Docs and tao_tutorials/notebooks/tao_launcher_starter_kit/deformable_detr at main · NVIDIA/tao_tutorials · GitHub .

You can run inference against the images.

$ docker run --runtime=nvidia -it --rm --shm-size=96GB nvcr.io/nvidia/tao/tao-toolkit:6.25.10-pyt /bin/bash
Then,

deformable_detr inference  -e $SPECS_DIR/infer.yaml  inference.checkpoint=$RESULTS_DIR/train/dd_model.pth 
results_dir=$RESULTS_DIR/

BTW, you can also use another peoplenet model which is trained from DINO network.(model card is: GPU-optimized AI, Machine Learning, & HPC Software | NVIDIA NGC | NVIDIA NGC). You can refer to DINO doc and notebook. DINO - NVIDIA Docs and tao_tutorials/notebooks/tao_launcher_starter_kit/dino at main · NVIDIA/tao_tutorials · GitHub

4

Hello Morganh,

Can you pls confirm is the issue with detection model?

Because I am still anticipating it is Calibration issue.

I have shared videos with Kesong can you take those videos and run the MV3DT.

So we can also proceed accordingly.

6

Got it. I get the videos now and will run inference.

7

Hi Morganh,

Thanks for the update.

Please run it and let me know what are we doing wrong.

This will help us to correct ourself instead of going in wrong direction and I can also ask specific query to you.

Once you run the MV3DT pls share the output so I can also compare with our output.

8

I use TAO container to check the peoplenet_transformer model.
Step:

  1. Login TAO5.0 container
    $ docker run --runtime=nvidia -it --rm --shm-size=96GB -v /localhome/local-morganh:/localhome/local-morganh nvcr.io/nvidia/tao/tao-toolkit:5.0.0-pyt /bin/bash

  2. Prepare infer.yaml

results_dir: /localhome/local-morganh/peoplenet_transformer/results
inference:
  conf_threshold: 0.2 #0.5
  input_width: 960
  input_height: 544
  color_map:
    person: red
    face: blue
    bag: green
dataset:
  infer_data_sources:
    image_dir:
    - /localhome/local-morganh/video1
    classmap: /localhome/local-morganh/peoplenet_transformer/classmap.txt
  num_classes: 4
  batch_size: 8
  workers: 8
  augmentation:
    fixed_padding: False
model:
  backbone: resnet_50
  num_feature_levels: 2
  return_interm_indices: [1, 2]
  dec_layers: 6
  enc_layers: 6
  num_queries: 300
  with_box_refine: True

And classmap.txt

person
face
bag
  1. Convert mkv file to png file with ffmpeg
  2. Run inference
deformable_detr inference  -e  /localhome/local-morganh/peoplenet_transformer/infer.yaml inference.checkpoint=/localhome/local-morganh/peoplenet_transformer/resnet50_peoplenet_transformer_tao5.0.pth
  1. The woman with black clothes can be detected. Although sometimes the bbox is missing but not “constantly” missing .

You can try to set lower confidence when run deepstream to check if the bbox is found better.

More, the model’s detail info is in GPU-optimized AI, Machine Learning, & HPC Software | NVIDIA NGC | NVIDIA NGC. It is based on deformable-DETR network. The the woman with black clothes may be different with the people in training dataset. So, it is expected to get this result and might need some finetuning with your dataset.

BTW, TAO also provides a v2 version mentioned in GPU-optimized AI, Machine Learning, & HPC Software | NVIDIA NGC | NVIDIA NGC. It is based on DINO network. It can be better than above-mentioned v1 model.

9

Hi Morganh,

Thanks for the update. I will also try to run it.

But do you think the change of unified ids in our video is because of detection because this is just single lady which is having this detection issue.

Other passengers are properly detected still unified id keeps on switching.

Please can you check on this.

10

For the “change of unified ids”, it is a checking item from deepstream side instead of TAO side, I would like to request deeptream moderator @kesong to sync further with you at previous deepstream forum topic (Query on Multi-Camera Calibration Accuracy vs Unified ID Reassociation Stability in MV3DT - #12 by nikhilahire5) . I also observer that the “man with long white clothes” may have random detection issue, you can check if there is also the “change of unified ids” for him.