0 MAP using NMS in custom detectnet

Please provide the following information when requesting support.

• Hardware (V100/Xavier/)
• Network Type (Detectnet_v2)
• TLT Version (latest)
• Training spec file(If have, please share here)
• How to reproduce the issue ? (This is for errors. Please share the command line and the detailed log here.)

I watn to try out NMS as custering config in my detectnet_v2 object detector. Using the default specification files (that include DBSCAN settings) I can get a pretty good result with a MAP 0f 60%, which is pretty ood considering the complexity of the objects.

The documentation states I should use the following values;
nms_iou_threshold float 0.2
nms_confidence_threshold float 0. (0. is this a typo?)

When replacing the DBSCAN lines with the following values:
clustering_algorithm: NMS
nms_iou_threshold: 0.5
nms_confidence_threshold: 0.4

The MAP is 0.x and no detection at all

Can someone tell me what values I should use for NMS? Do I need to put in topK as well?

Could you try more nms_iou_threshold and nms_confidence_threshold?

@Morganh I did bump it up, but killed the process after 10 epochs because MAP remained 0.x
I tried with different values, but no change

Here’s the postprocessing that worked good with DBSCAN:

> postprocessing_config {
> target_class_config {
> key: "*******"
> value {      
> clustering_config {
> clustering_algorithm: DBSCAN
>  dbscan_confidence_threshold: 0.9
> coverage_threshold: 0.005
> dbscan_eps: 0.15
> dbscan_min_samples: 0.05
> minimum_bounding_box_height: 10

Here’s the postprocessing that I’m trying now that gives MAP 0.x with NMS:

> postprocessing_config {
>   target_class_config {
>     key: "*******"
>     value {
>       clustering_config {
>         clustering_algorithm: NMS
>         nms_iou_threshold: 0.6
>         nms_confidence_threshold: 0.6
>         minimum_bounding_box_height: 10

Here’s the Eval config

> evaluation_config {
>   validation_period_during_training: 10
>   first_validation_epoch: 10
>   minimum_detection_ground_truth_overlap {
>     key: "*******"
>     value: 0.4
>   }
>   evaluation_box_config {
>     key: "*******"
>     value {
>       minimum_height: 10
>       maximum_height: 9999
>       minimum_width: 10
>       maximum_width: 9999
>     }
>   }
>   average_precision_mode: INTEGRATE
> }

I recall that you already train a tlt model with a MAP 0f 60%.
Then, you can run “tao evaluate xxx” with this tlt model.
Try to use NMS with different thresholds.

@Morganh Because most Deepstream apps use NMS clustering in favor of DBSCAN.
Yes, our model had a MAP of 60%, but that was with DBSCAN, but we want to try if NMS gives better results. We tried many many hours, using all different NMS settings.

The documentation states:
nms_iou_threshold float 0.2
nms_confidence_threshold float 0.

which clearly isn’t going to work for our model. Can you suggest some (default) values?
Is our evaluator good, or does it need changes to evaluate NMS?

You need not run training. Just need to run evaluation against the tlt model.


tao detectnet_v2 evaluate  -e <experiment_spec_with_different_nms_threshold>
                               -m <your_60%_mAP_tlt_model>
                               -k <key>

So the clustering config (either NMS or DBSCAN) has nothing to do with the training, but is used in evaluating and every model I trained can do inference with either NMS or BDSCAN?


This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.