Please provide the following information when requesting support.
• Hardware (T4/V100/Xavier/Nano/etc) I train on rtx 4090 and I do inference on AGX Xavier
• Network Type (Detectnet_v2/Faster_rcnn/Yolo_v4/LPRnet/Mask_rcnn/Classification/etc) Yolov3
• TLT Version (Please run “tlt info --verbose” and share “docker_tag” here)
dockers:
nvidia/tao/tao-toolkit-tf:
v3.21.11-tf1.15.5-py3:
docker_registry: nvcr.io
tasks:
1. augment
2. bpnet
3. classification
4. dssd
5. emotionnet
6. efficientdet
7. fpenet
8. gazenet
9. gesturenet
10. heartratenet
11. lprnet
12. mask_rcnn
13. multitask_classification
14. retinanet
15. ssd
16. unet
17. yolo_v3
18. yolo_v4
19. yolo_v4_tiny
20. converter
v3.21.11-tf1.15.4-py3:
docker_registry: nvcr.io
tasks:
1. detectnet_v2
2. faster_rcnn
nvidia/tao/tao-toolkit-pyt:
v3.21.11-py3:
docker_registry: nvcr.io
tasks:
1. speech_to_text
2. speech_to_text_citrinet
3. text_classification
4. question_answering
5. token_classification
6. intent_slot_classification
7. punctuation_and_capitalization
8. action_recognition
v3.22.02-py3:
docker_registry: nvcr.io
tasks:
1. spectro_gen
2. vocoder
nvidia/tao/tao-toolkit-lm:
v3.21.08-py3:
docker_registry: nvcr.io
tasks:
1. n_gram
format_version: 2.0
toolkit_version: 3.22.02
published_date: 02/28/2022
• Training spec file(If have, please share here)
random_seed: 42
yolov3_config {
big_anchor_shape: “[(114.94, 60.67), (159.06, 114.59), (297.59, 176.38)]”
mid_anchor_shape: “[(42.99, 31.91), (79.57, 31.75), (56.80, 56.93)]”
small_anchor_shape: “[(15.60, 13.88), (30.25, 20.25), (20.67, 49.63)]”
matching_neutral_box_iou: 0.7
arch: “resnet”
nlayers: 18
arch_conv_blocks: 2
loss_loc_weight: 0.8
loss_neg_obj_weights: 100.0
loss_class_weights: 1.0
freeze_bn: false
#freeze_blocks: 0
force_relu: false
}
training_config {
batch_size_per_gpu: 8
num_epochs: 80
enable_qat: true
checkpoint_interval: 10
learning_rate {
soft_start_annealing_schedule {
min_learning_rate: 1e-6
max_learning_rate: 1e-4
soft_start: 0.1
annealing: 0.5
}
}
regularizer {
type: L1
weight: 3e-9
}
optimizer {
adam {
epsilon: 0.001
beta1: 0.9
beta2: 0.999
amsgrad: false
}
}
pretrain_model_path: “/workspace/tao-experiments/yolov3/pretrained_resnet18/pretrained_object_detection_vresnet18/resnet_18.hdf5”
}
eval_config {
average_precision_mode: SAMPLE
batch_size: 8
matching_iou_threshold: 0.5
}
nms_config {
confidence_threshold: 0.001
clustering_iou_threshold: 0.5
top_k: 200
force_on_cpu: True
}
augmentation_config {
hue: 0.1
saturation: 1.5
exposure:1.5
vertical_flip:0
horizontal_flip: 0.5
jitter: 0.3
output_width: 1280
output_height: 1280
output_channel: 3
randomize_input_shape_period: 0
}
dataset_config {
data_sources: {
tfrecords_path: “/workspace/tao-experiments/data/tfrecords/kitti_trainval/*”
image_directory_path: “/workspace/tao-experiments/try-6”
}
include_difficult_in_training: true
image_extension: “jpg”
target_class_mapping {
key: “cachalot”
value: “cachalot”
}
target_class_mapping {
key: “jet”
value: “jet”
}
target_class_mapping {
key: “rorqual”
value: “rorqual”
}
target_class_mapping {
key: “bateau”
value: “bateau”
}
target_class_mapping {
key: “globicephale”
value: “globicephale”
}
target_class_mapping {
key: “queue_cachalot”
value: “queue_cachalot”
}
validation_fold: 0
}
• How to reproduce the issue ? (This is for errors. Please share the command line and the detailed log here.)
Hello,
I think I have a problem with the generation of my calibration file in TAO. I want to do inference on deepstream in int8 using this yolo v3 model (key = tlt_encode). When I do inference both in fp32 and fp16 everything works well, but when I use int8 it gives pretty bad results.
I’m pretty sure it comes from the calibration file. To generate it I export the model I train using tao-converter and then I use tao-deploy to generate the calibration file, both step on my RTX4090.
Can you tell me if my workflow is correct and if so can you check my calibration file and tell me what’s wrong ?
Thank you for your answer