0 MAP using NMS in custom detectnet

[KGerry]

Please provide the following information when requesting support.

• Hardware (V100/Xavier/)
• Network Type (Detectnet_v2)
• TLT Version (latest)
• Training spec file(If have, please share here)
• How to reproduce the issue ? (This is for errors. Please share the command line and the detailed log here.)

I watn to try out NMS as custering config in my detectnet_v2 object detector. Using the default specification files (that include DBSCAN settings) I can get a pretty good result with a MAP 0f 60%, which is pretty ood considering the complexity of the objects.

The documentation states I should use the following values;
nms_iou_threshold float 0.2
nms_confidence_threshold float 0. (0. is this a typo?)

When replacing the DBSCAN lines with the following values:
clustering_algorithm: NMS
nms_iou_threshold: 0.5
nms_confidence_threshold: 0.4

The MAP is 0.x and no detection at all

Can someone tell me what values I should use for NMS? Do I need to put in topK as well?

[Morganh]

Could you try more nms_iou_threshold and nms_confidence_threshold?

[KGerry]

@Morganh I did bump it up, but killed the process after 10 epochs because MAP remained 0.x
I tried with different values, but no change

Here’s the postprocessing that worked good with DBSCAN:

> postprocessing_config {
> target_class_config {
> key: "*******"
> value {      
> clustering_config {
> clustering_algorithm: DBSCAN
>  dbscan_confidence_threshold: 0.9
> coverage_threshold: 0.005
> dbscan_eps: 0.15
> dbscan_min_samples: 0.05
> minimum_bounding_box_height: 10

Here’s the postprocessing that I’m trying now that gives MAP 0.x with NMS:

> postprocessing_config {
>   target_class_config {
>     key: "*******"
>     value {
>       clustering_config {
>         clustering_algorithm: NMS
>         nms_iou_threshold: 0.6
>         nms_confidence_threshold: 0.6
>         minimum_bounding_box_height: 10

Here’s the Eval config

> evaluation_config {
>   validation_period_during_training: 10
>   first_validation_epoch: 10
>   minimum_detection_ground_truth_overlap {
>     key: "*******"
>     value: 0.4
>   }
>   evaluation_box_config {
>     key: "*******"
>     value {
>       minimum_height: 10
>       maximum_height: 9999
>       minimum_width: 10
>       maximum_width: 9999
>     }
>   }
>   average_precision_mode: INTEGRATE
> }

[Morganh]

I recall that you already train a tlt model with a MAP 0f 60%.
Then, you can run “tao evaluate xxx” with this tlt model.
Try to use NMS with different thresholds.

[KGerry]

@Morganh Because most Deepstream apps use NMS clustering in favor of DBSCAN.
Yes, our model had a MAP of 60%, but that was with DBSCAN, but we want to try if NMS gives better results. We tried many many hours, using all different NMS settings.

The documentation states:
nms_iou_threshold float 0.2
nms_confidence_threshold float 0.

which clearly isn’t going to work for our model. Can you suggest some (default) values?
Is our evaluator good, or does it need changes to evaluate NMS?

[Morganh]

@KGerry

killed the process after 10 epochs because MAP remained 0.x

You need not run training. Just need to run evaluation against the tlt model.

DetectNet_v2 - NVIDIA Docs

tao detectnet_v2 evaluate  -e <experiment_spec_with_different_nms_threshold>
                               -m <your_60%_mAP_tlt_model>
                               -k <key>

[KGerry]

So the clustering config (either NMS or DBSCAN) has nothing to do with the training, but is used in evaluating and every model I trained can do inference with either NMS or BDSCAN?

[Morganh]

Yes.