Hello,
I am trying to upgrade my pipeline from TAO 3.21.08 to TAO 4.0.1 to run using TensorRT 8.5.2.2, however when I export and build the engine using the tao-converter for tao 4.0.0_trt_8.5.2.2_aarch64, I get very different results when running inference in Deepstream.
• Hardware - Xavier
• Network Type - YOLOv4
• TLT Version - tao-toolkit-tf:v3.21.08-py3 and tao-toolkit:4.0.1-tf1.15.5
• Training spec file:
random_seed: 42
yolov4_config {
big_anchor_shape: "[(87.07, 119.20), (119.47, 87.33), (124.67, 123.07)]"
mid_anchor_shape: "[(78.13, 78.13), (59.73, 105.20), (106.93, 60.80)]"
small_anchor_shape: "[(36.67, 35.87), (48.00, 66.27), (68.13, 48.53)]"
box_matching_iou: 0.25
matching_neutral_box_iou: 0.5
arch: "cspdarknet"
nlayers: 19
arch_conv_blocks: 2
loss_loc_weight: 1.0
loss_neg_obj_weights: 1.0
loss_class_weights: 1.0
label_smoothing: 0.0
big_grid_xy_extend: 0.05
mid_grid_xy_extend: 0.1
small_grid_xy_extend: 0.2
freeze_bn: false
force_relu: false
}
training_config {
batch_size_per_gpu: 16
num_epochs: 300
enable_qat: false
checkpoint_interval: 5
learning_rate {
soft_start_cosine_annealing_schedule {
min_learning_rate: 1e-07
max_learning_rate: 0.0001
soft_start: 0.3
}
}
regularizer {
type: L1
weight: 3e-05
}
optimizer {
adam {
epsilon: 1e-07
beta1: 0.9
beta2: 0.999
amsgrad: false
}
}
use_multiprocessing: true
visualizer {
enabled: false
num_images: 3
}
}
eval_config {
average_precision_mode: SAMPLE
batch_size: 16
matching_iou_threshold: 0.5
}
nms_config {
confidence_threshold: 0.001
clustering_iou_threshold: 0.5
top_k: 200
infer_nms_score_bits: 0
force_on_cpu: false
}
augmentation_config {
hue: 0.1
saturation: 1.5
exposure: 1.5
vertical_flip: 0.5
horizontal_flip: 0.5
jitter: 0.3
output_width: 512
output_height: 288
output_channel: 3
randomize_input_shape_period: 0
mosaic_prob: 0.5
mosaic_min_ratio: 0.2
}
dataset_config {
...
}
class_weighting_config {
...
}
• How to reproduce the issue ?
- Train a model using
TAO 3.21.08
. - Export the model using
TAO 3.21.08
, and convert it to engine format usingtao-converter_v3.21.08_trt7.1_aarch64
. Create a video with bounding boxes using Deepstream 6.0 - Export the model using
TAO 4.0.1
, and convert it to engine format usingtao-converter_v4.0.0_trt_8.5.2.2_aarch64
. Create a video with bounding boxes using Deepstream 6.2. - Observe significant differences in the predicted bounding boxes.
Thanks in advance for any help you can provide.
Kind regards,
Nicholas