Engine convtered using TAO 4.0.0 has lower performance than 3.21.08

Hello,

I am trying to upgrade my pipeline from TAO 3.21.08 to TAO 4.0.1 to run using TensorRT 8.5.2.2, however when I export and build the engine using the tao-converter for tao 4.0.0_trt_8.5.2.2_aarch64, I get very different results when running inference in Deepstream.

• Hardware - Xavier
• Network Type - YOLOv4
• TLT Version - tao-toolkit-tf:v3.21.08-py3 and tao-toolkit:4.0.1-tf1.15.5
• Training spec file:

random_seed: 42
yolov4_config {
  big_anchor_shape: "[(87.07, 119.20), (119.47, 87.33), (124.67, 123.07)]"
  mid_anchor_shape: "[(78.13, 78.13), (59.73, 105.20), (106.93, 60.80)]"
  small_anchor_shape: "[(36.67, 35.87), (48.00, 66.27), (68.13, 48.53)]"
  box_matching_iou: 0.25
  matching_neutral_box_iou: 0.5
  arch: "cspdarknet"
  nlayers: 19
  arch_conv_blocks: 2
  loss_loc_weight: 1.0
  loss_neg_obj_weights: 1.0
  loss_class_weights: 1.0
  label_smoothing: 0.0
  big_grid_xy_extend: 0.05
  mid_grid_xy_extend: 0.1
  small_grid_xy_extend: 0.2
  freeze_bn: false
  force_relu: false
}
training_config {
  batch_size_per_gpu: 16
  num_epochs: 300
  enable_qat: false
  checkpoint_interval: 5
  learning_rate {
    soft_start_cosine_annealing_schedule {
      min_learning_rate: 1e-07
      max_learning_rate: 0.0001
      soft_start: 0.3
    }
  }
  regularizer {
    type: L1
    weight: 3e-05
  }
  optimizer {
    adam {
      epsilon: 1e-07
      beta1: 0.9
      beta2: 0.999
      amsgrad: false
    }
  }
  use_multiprocessing: true
  visualizer {
    enabled: false
    num_images: 3
  }
}
eval_config {
  average_precision_mode: SAMPLE
  batch_size: 16
  matching_iou_threshold: 0.5
}
nms_config {
  confidence_threshold: 0.001
  clustering_iou_threshold: 0.5
  top_k: 200
  infer_nms_score_bits: 0
  force_on_cpu: false
}
augmentation_config {
  hue: 0.1
  saturation: 1.5
  exposure: 1.5
  vertical_flip: 0.5
  horizontal_flip: 0.5
  jitter: 0.3
  output_width: 512
  output_height: 288
  output_channel: 3
  randomize_input_shape_period: 0
  mosaic_prob: 0.5
  mosaic_min_ratio: 0.2
}
dataset_config {
  ...
}
class_weighting_config {
  ...
}

• How to reproduce the issue ?

  • Train a model using TAO 3.21.08.
  • Export the model using TAO 3.21.08, and convert it to engine format using tao-converter_v3.21.08_trt7.1_aarch64. Create a video with bounding boxes using Deepstream 6.0
  • Export the model using TAO 4.0.1, and convert it to engine format using tao-converter_v4.0.0_trt_8.5.2.2_aarch64. Create a video with bounding boxes using Deepstream 6.2.
  • Observe significant differences in the predicted bounding boxes.

Thanks in advance for any help you can provide.

Kind regards,
Nicholas

According to the repro steps you mentioned, you are training one .tlt model using TAO 3.21.08. Then export to two .etlt models via different TAO docker(3.21.08 and 4.01 ).

Can you train two different models to compare the mAP via tao evaluate firstly?
Step:

  1. Run training using TAO 3.21.08.
    Then still use this TAO 3.21.08 docker to run evaluation against the test dataset.
  2. Run training using TAO 4.0.1.
    Then still use this TAO 4.0.1 docker to run evaluation against the same test dataset.
  3. Compare the mAP result for step 1 and step 2.

Above steps are used to check if there is regression in 4.0.1 docker.
Also, above steps are running via TAO docker only. We need to narrow down via tao docker firstly. So, Deestream docker is not involved yet.

Hi Morganh.

Thanks for the reply.

I followed your instructions and got the following results:

  • TAO4 Model mAP 0.08497
  • TAO3 Model mAP 0.14545

There is a significant mAP drop across the two TAO versions.

Kind regards,
Nicholas

May I know if these results are training against the public KITTI dataset mentioned in the notebook?

Hi Morganh.

They are trained using a custom dataset.

Kind regards,
Nicholas

Hello,

Please find further information in this thread: Performance of TAO 3.22.05 and TAO 4.0.1 is lower than TAO 3.21.08

Best,
Benedetta

Refer to Performance of TAO 3.22.05 and TAO 4.0.1 is lower than TAO 3.21.08 - #8 by benedetta.delfino

Hi Morganh.

Following the discussion on the above thread, I trained two models using the discovered best config. Each model was trained and exported end-to-end with TAO 3.21.08 and TAO 4.0.1 respectively, and then were exported using the matching tao-converter. The resulting models are still different.

Kind regards,
Nicholas

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

The TAO 4.0.1 result is better than TAO3.21.08 ,right?

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.