PeopleNet not detecting bags

Accelerated Computing Intelligent Video Analytics TAO Toolkit
1

Please provide the following information when requesting support.

• Hardware (T4/V100/Xavier/Nano/etc): GTX 1060
• Network Type (Detectnet_v2/Faster_rcnn/Yolo_v4/LPRnet/Mask_rcnn/Classification/etc): PeopleNet
• TLT Version (Please run “tlt info --verbose” and share “docker_tag” here): 8.2.5
• Training spec file(If have, please share here): none
• How to reproduce the issue ? (This is for errors. Please share the command line and the detailed log here.): read on

I tried to use the PeopleNet 2.3 or newer to detect persons, bags and faces. It detects persons and faces just fine but it doesn’t detect a single bag, not even in the showcase in the NGC page. PeopleNet 2.0 based on ResNet18 does detect bags but it’s not as accurate as ResNet34 in the other two classes.

I tried the models pruned_quantized_v2.3 and deployable_quantized_v2.6. I converted them to trt using this command:

./tao-converter resnet34_peoplenet_pruned_int8.etlt -d 3,768,1024 -k tlt_encode -o output_cov/Sigmoid,output_bbox/BiasAdd -i nchw -m 1 -t int8 -e trt_model.trt -s

This is the output of this command:

[INFO] [MemUsageChange] Init CUDA: CPU +193, GPU +0, now: CPU 204, GPU 862 (MiB)
[INFO] [MemUsageSnapshot] Begin constructing builder kernel library: CPU 204 MiB, GPU 862 MiB
[INFO] [MemUsageSnapshot] End constructing builder kernel library: CPU 268 MiB, GPU 862 MiB
[WARNING] TensorRT was linked against cuBLAS/cuBLASLt 11.6.5 but loaded cuBLAS/cuBLASLt 11.6.1
[INFO] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +248, GPU +102, now: CPU 536, GPU 964 (MiB)
[INFO] [MemUsageChange] Init cuDNN: CPU +149, GPU +108, now: CPU 685, GPU 1072 (MiB)
[INFO] Timing cache disabled. Turning it on will improve builder speed.
[INFO] Detected 1 inputs and 2 output network tensors.
[INFO] Total Host Persistent Memory: 10336
[INFO] Total Device Persistent Memory: 0
[INFO] Total Scratch Memory: 0
[INFO] [MemUsageStats] Peak memory usage of TRT CPU/GPU memory allocators: CPU 0 MiB, GPU 410 MiB
[INFO] [BlockAssignment] Algorithm ShiftNTopDown took 35.7075ms to assign 4 blocks to 248 nodes requiring 515899392 bytes.
[INFO] Total Activation Memory: 515899392
[WARNING] TensorRT was linked against cuBLAS/cuBLASLt 11.6.5 but loaded cuBLAS/cuBLASLt 11.6.1
[INFO] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +8, now: CPU 876, GPU 1154 (MiB)
[INFO] [MemUsageChange] Init cuDNN: CPU +0, GPU +8, now: CPU 876, GPU 1162 (MiB)
[WARNING] TensorRT was linked against cuBLAS/cuBLASLt 11.6.5 but loaded cuBLAS/cuBLASLt 11.6.1
[INFO] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +10, now: CPU 875, GPU 1138 (MiB)
[INFO] [MemUsageChange] Init cuDNN: CPU +1, GPU +8, now: CPU 876, GPU 1146 (MiB)
[INFO] [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +492, now: CPU 0, GPU 501 (MiB)
[INFO] Starting Calibration with batch size 8.
[INFO]   Post Processing Calibration data in 6.02e-07 seconds.
[INFO] Calibration completed in 3.1918 seconds.
[INFO] Writing Calibration Cache for calibrator: TRT-8205-EntropyCalibration2
[WARNING] Missing scale and zero-point for tensor output_bbox/bias, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing scale and zero-point for tensor input_1, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing scale and zero-point for tensor conv1/kernel, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing scale and zero-point for tensor conv1/convolution, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing scale and zero-point for tensor conv1/bias, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing scale and zero-point for tensor conv1/BiasAdd, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing scale and zero-point for tensor bn_conv1/moving_variance, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing scale and zero-point for tensor bn_conv1/Reshape_1/shape, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing scale and zero-point for tensor bn_conv1/batchnorm/add/y, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing scale and zero-point for tensor bn_conv1/gamma, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing scale and zero-point for tensor bn_conv1/Reshape_3/shape, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing scale and zero-point for tensor bn_conv1/batchnorm/mul_1, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing scale and zero-point for tensor bn_conv1/beta, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing scale and zero-point for tensor bn_conv1/Reshape_2/shape, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing scale and zero-point for tensor bn_conv1/moving_mean, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing scale and zero-point for tensor bn_conv1/Reshape/shape, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing scale and zero-point for tensor bn_conv1/batchnorm/add_1, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing scale and zero-point for tensor block_1a_conv_1/kernel, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing scale and zero-point for tensor block_1a_conv_1/convolution, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
... (snipped due to size limit)
implementation for any layer consuming or producing given tensor
[WARNING] Missing scale and zero-point for tensor output_cov/BiasAdd, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] Missing scale and zero-point for tensor output_cov/Sigmoid, expect fall back to non-int8 implementation for any layer consuming or producing given tensor
[WARNING] TensorRT was linked against cuBLAS/cuBLASLt 11.6.5 but loaded cuBLAS/cuBLASLt 11.6.1
[INFO] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +10, now: CPU 878, GPU 1138 (MiB)
[INFO] [MemUsageChange] Init cuDNN: CPU +0, GPU +8, now: CPU 878, GPU 1146 (MiB)
[INFO] Local timing cache in use. Profiling results in this builder pass will not be stored.
[INFO] Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output.
[INFO] Detected 1 inputs and 2 output network tensors.
[INFO] Total Host Persistent Memory: 103584
[INFO] Total Device Persistent Memory: 7409664
[INFO] Total Scratch Memory: 0
[INFO] [MemUsageStats] Peak memory usage of TRT CPU/GPU memory allocators: CPU 10 MiB, GPU 1010 MiB
[INFO] [BlockAssignment] Algorithm ShiftNTopDown took 2.21526ms to assign 4 blocks to 67 nodes requiring 38830080 bytes.
[INFO] Total Activation Memory: 38830080
[WARNING] TensorRT was linked against cuBLAS/cuBLASLt 11.6.5 but loaded cuBLAS/cuBLASLt 11.6.1
[INFO] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +8, now: CPU 881, GPU 1152 (MiB)
[INFO] [MemUsageChange] Init cuDNN: CPU +0, GPU +8, now: CPU 881, GPU 1160 (MiB)
[INFO] [MemUsageChange] TensorRT-managed allocation in building engine: CPU +2, GPU +8, now: CPU 2, GPU 8 (MiB)

For inference, I took the example for detectnet_v2 and I just changed the path to the trt file. The example has three classes also, they are named differently but it shouldn’t matter. And I didn’t notice any bag being detected except the case when I tried the ResNet18 model.

I also tried the example code at https://forums.developer.nvidia.com/t/run-peoplenet-with-tensorrt/128000/20. The same thing happens there. Bags are detected only in the ResNet18 model.

3

The "-d 3,768,1024 " is not correct. Based on PeopleNet | NVIDIA NGC, you can set "-d 3,544,960 "

4

Thanks for answer! I didn’t know that the performance may depend that much on the resolution. Is this the only supported resolution?

I used the command:

./tao-converter resnet34_peoplenet_pruned_int8.etlt -d 3,544,960 -k tlt_encode -o output_cov/Sigmoid,output_bbox/BiasAdd -i nchw -m 1 -t int8 -e nodel.trt -s

but there are still no bag detections.
Figure_1

5

The “-d” should be input dimensions. See DetectNet_v2 - NVIDIA Docs.
It should match 960 X 544 X 3 (W x H x C). See
PeopleNet | NVIDIA NGC

For bag, as mentioned in PeopleNet | NVIDIA NGC

#### FACE AND BAG CLASS

Although bag and face class are included in the model, the accuracy of these classes will be much lower than people class. Some re-training will be required on these classes to improve accuracy.

More, can you run with fp32 or fp16 mode and check again?

And how did you test, with deepstream?

6

I tried to evaluate the trainable model directly using the Jupyter notebook for detectnet_v2. I used the command:

!tao detectnet_v2 inference -e $SPECS_DIR/my_inference.txt \
                            -o $USER_EXPERIMENT_DIR/my_data_testing \
                            -i $USER_EXPERIMENT_DIR/my_data \
                            -k $KEY

The folder my_data contains my testing photos. The variables are set in the section 0 in the Jupyter notebook. I ran whole this Jupyter notebook before and all seems to work there. The file my_inference.txt contains:

inferencer_config{
  # defining target class names for the experiment.
  # Note: This must be mentioned in order of the networks classes.
  target_classes: "person"
  target_classes: "bag"
  target_classes: "face"
  # Inference dimensions.
  image_width: 960
  image_height: 544
  # Must match what the model was trained for.
  image_channels: 3
  batch_size: 1
  gpu_index: 0
  # model handler config
  tlt_config{
    model: "/workspace/tao-experiments/detectnet_v2/resnet34_peoplenet.tlt"
  }
}
bbox_handler_config{
  kitti_dump: true
  disable_overlay: false
  overlay_linewidth: 2
  classwise_bbox_handler_config{
    key:"person"
    value: {
      confidence_model: "aggregate_cov"
      output_map: "person"
      bbox_color{
        R: 0
        G: 255
        B: 0
      }
      clustering_config{
        clustering_algorithm: DBSCAN
        coverage_threshold: 0.005
        dbscan_eps: 0.3
        dbscan_min_samples: 0.05
        dbscan_confidence_threshold: 0.9
        minimum_bounding_box_height: 4
      }
    }
  }
  classwise_bbox_handler_config{
    key:"bag"
    value: {
      confidence_model: "aggregate_cov"
      output_map: "bag"
      bbox_color{
        R: 0
        G: 255
        B: 255
      }
      clustering_config{
        clustering_algorithm: DBSCAN
        coverage_threshold: 0.005
        dbscan_eps: 0.3
        dbscan_min_samples: 0.05
        dbscan_confidence_threshold: 0.9
        minimum_bounding_box_height: 4
      }
    }
  }
  classwise_bbox_handler_config{
    key:"face"
    value: {
      confidence_model: "aggregate_cov"
      output_map: "face"
      bbox_color{
        R: 255
        G: 0
        B: 0
      }
      clustering_config{
        clustering_algorithm: DBSCAN
        coverage_threshold: 0.005
        dbscan_eps: 0.3
        dbscan_min_samples: 0.05
        dbscan_confidence_threshold: 0.9
        minimum_bounding_box_height: 4
      }
    }
  }
  classwise_bbox_handler_config{
    key:"default"
    value: {
      confidence_model: "aggregate_cov"
      bbox_color{
        R: 255
        G: 0
        B: 0
      }
      clustering_config{
        clustering_algorithm: DBSCAN
        dbscan_confidence_threshold: 0.9
        coverage_threshold: 0.005
        dbscan_eps: 0.3
        dbscan_min_samples: 0.05
        minimum_bounding_box_height: 4
      }
    }
  }
}

When I use this, all pictures are evaluated. Persons and faces are detected but no bags.

bagz2
bagz2960×544 59.9 KB

There are three thresholds. coverage_threshold, dbscan_min_samples and dbscan_confidence_threshold. If I set them all to 0, there are many detections. However, if I set any of those thresholds to 0.00000001, there are no bag detections at all. Persons and faces are detected normally.

I test it on 8 photos with 11 bags in total. None of them is detected, not even a single false-positive.

EDIT: I set the resolution to 940×544, still the same result

7

To narrow down, could you please use the way mentioned in PeopleNet | NVIDIA NGC ?

## Download Model:

mkdir -p $HOME/peoplenet && \
wget https://api.ngc.nvidia.com/v2/models/nvidia/tao/peoplenet/versions/pruned_v2.1/files/resnet34_peoplenet_pruned.etlt \
-O $HOME/peoplenet/resnet34_peoplenet_pruned.etlt

## Run Application

xhost +
sudo docker run --gpus all -it --rm -v /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY=$DISPLAY -v $HOME:/opt/nvidia/deepstream/deepstream-5.1/samples/models/tlt_pretrained_models \
-w /opt/nvidia/deepstream/deepstream-5.1/samples/configs/tlt_pretrained_models nvcr.io/nvidia/deepstream:5.1-21.02-samples \
deepstream-app -c deepstream_app_source1_peoplenet.txt

More, may I know which version did you use? Could you share the wget link?

1 Like
8

I didn’t experiment with deepstream yet. I will do it now, that will take some time.
For the last example, I used this: wget 'https://api.ngc.nvidia.com/v2/models/nvidia/tao/peoplenet/versions/trainable_v2.6/files/resnet34_peoplenet.tlt'
I was also trying wget 'https://api.ngc.nvidia.com/v2/models/nvidia/tao/peoplenet/versions/deployable_quantized_v2.6/files/resnet34_peoplenet_int8.etlt' and wget 'https://api.ngc.nvidia.com/v2/models/nvidia/tao/peoplenet/versions/pruned_quantized_v2.3/files/resnet34_peoplenet_pruned_int8.etlt'

This one works: wget 'https://api.ngc.nvidia.com/v2/models/nvidia/tao/peoplenet/versions/pruned_v2.0/files/resnet18_peoplenet_pruned.etlt' but it’s ResNet18.

9

I tried the application. I see there two bags and none of them were detected.

Snímek obrazovky pořízený 2022-09-27 10-09-37
Snímek obrazovky pořízený 2022-09-27 10-09-371280×720 84.3 KB

10

Could you please download below resnet34 one? I confirm it can detect bags.
wget ‘https://api.ngc.nvidia.com/v2/models/nvidia/tao/peoplenet/versions/unpruned_v2.0/files/resnet34_peoplenet.tlt’

11

This model does detect bags, thank you!

12

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.