Downgrade of mAP while exporting DetectNet_v2 model in INT8 mode

Hello,

I’m trying to export a model using DetectNet_v2 model in INT8 mode, but when comparing the results making inference in the same test dataset, I'm getting a lower mAP than previous to the export (20% decrease approx.) (the model being exported has been trained using quantization, and similar results were obtained from the first retraining of the model and the second retraining but with quantization).   

The steps I’m following for exporting the model are the following:

  • First, I generate calibration tensor file, using the following command
!tao detectnet_v2 calibration_tensorfile -e $SPECS_DIR/trafficcamnet_retrain_resnet18_kitti.txt \
                                         -m 500 \
                                         -o $USER_EXPERIMENT_DIR/experiment_dir_final/calibration.tensor

After this, I’m exporting the model

!tao detectnet_v2 export \
                  -m $USER_EXPERIMENT_DIR/experiment_dir_retrain/weights/resnet18_detector_pruned_qat.tlt \
                  -o $USER_EXPERIMENT_DIR/experiment_dir_final/resnet18_detector_qat.etlt \
                  -k $KEY  \
                  --cal_data_file $USER_EXPERIMENT_DIR/experiment_dir_final/calibration.tensor \
                  -e $SPECS_DIR/trafficcamnet_retrain_resnet18_kitti.txt \
                  --data_type int8 \
                  --batch_size 64 \
                  --max_batch_size 64\
                  --engine_file $USER_EXPERIMENT_DIR/experiment_dir_final/resnet18_detector_qat.trt.int8 \
                  --cal_cache_file $USER_EXPERIMENT_DIR/experiment_dir_final/calibration_qat.bin \
                  --gen_ds_config \
                  --verbose

Finally, the inference is made with the generated .trt file resnet18_detector_qat.trt.int8. It is worth mentioning that different values have been tested for generating the calibration file. Is there any parameter I could change to recover the majority of the lost precision?

Here is the inference spec file that I’m using:

inferencer_config{
  # Defining input node of the model
  # defining target class names for the experiment.
  # Note: This must be mentioned in order of the networks classes.
  target_classes: "car"
  target_classes: "bicycle"
  target_classes: "motorcycle"
  # Inference dimensions.
  image_width: 1920
  image_height: 1072
  # Must match what the model was trained for.
  image_channels: 3
  batch_size: 16
  gpu_index: 0
  # model handler config
  tensorrt_config{
   parser: ETLT
   etlt_model: "/workspace/tao-experiments/retraining/trafficcamnet/experiment_dir_final/resnet18_detector_qat.etlt"
   backend_data_type: INT8
   trt_engine: "/workspace/tao-experiments/retraining/trafficcamnet/experiment_dir_final/resnet18_detector_qat.trt.int8"
   calibrator_config{
      calibration_cache: "/workspace/tao-experiments/retraining/trafficcamnet/experiment_dir_final/calibration_qat.bin"
      n_batches: 500
   }
  }
}
bbox_handler_config{
  kitti_dump: true
  disable_overlay: false
  overlay_linewidth: 2
  classwise_bbox_handler_config{
    key:"car"
    value: {
      confidence_model: "aggregate_cov"
      output_map: "car"
      bbox_color{
        R: 0
        G: 255
        B: 0
      }
      clustering_config{
        clustering_algorithm: DBSCAN
        coverage_threshold: 0.005
        dbscan_eps: 0.3
        dbscan_min_samples: 0.05
        dbscan_confidence_threshold: 0.9
        minimum_bounding_box_height: 4
      }
    }
  }
classwise_bbox_handler_config{
    key:"bicycle"
    value: {
      confidence_model: "aggregate_cov"
      output_map: "bicycle"
      bbox_color{
        R: 0
        G: 255
        B: 255
      }
      clustering_config{
        clustering_algorithm: DBSCAN
        coverage_threshold: 0.005
        dbscan_eps: 0.3
        dbscan_min_samples: 0.05
        dbscan_confidence_threshold: 0.9
        minimum_bounding_box_height: 4
      }
    }
  }
  classwise_bbox_handler_config{
    key:"motorcycle"
    value: {
      confidence_model: "aggregate_cov"
      output_map: "motorcycle"
      bbox_color{
        R: 255
        G: 0
        B: 0
      clustering_config{
        clustering_algorithm: DBSCAN
        coverage_threshold: 0.005
        dbscan_eps: 0.3
        dbscan_min_samples: 0.05
        dbscan_confidence_threshold: 0.9
        minimum_bounding_box_height: 4
      }
    }
  }
  classwise_bbox_handler_config{
    key:"default"
    value: {
      confidence_model: "aggregate_cov"
      bbox_color{
        R: 255
        G: 0
        B: 0
      }
      clustering_config{
        clustering_algorithm: DBSCAN
        dbscan_confidence_threshold: 0.9
        coverage_threshold: 0.005
        dbscan_eps: 0.3
        dbscan_min_samples: 0.05
        minimum_bounding_box_height: 4
      }
    }
  }
}

Could you please check if you can reproduce with official detectnet_v2 notebook with the public KITTI dataset? Thanks.

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

How much mAP is downgrading?
Can you share more logs?

  • int8 engine without qat
  • int8 engine with qat (currently you are running)

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.