Tlt3.0 retrain trafficcamnet getting the error when do the evaluation

Accelerated Computing Intelligent Video Analytics TAO Toolkit
22

Hi, Thanks a lot, i ‘ve checked my DataSets’ label name, they are car, bicycle, person, road_sign respectively in this order, i set them in the training config:
image_extension: “jpg”
target_class_mapping {
key: “car”
value: “car”
}
target_class_mapping {
key: “bicycle”
value: “bicycle”
}
target_class_mapping {
key: “person”
value: “person”
}
target_class_mapping {
key: “road_sign”
value: “road_sign”
}
as you can see from above, My datasets’ label show like this:

~…/cat 00630.txt

car 0.0 0 -1 1200.0 427.0 1322.0 468.0 -1 -1 -1 -1 -1 -1 -1
road_sign 0.0 0 -1 188.0 434.0 221.0 576.0 -1 -1 -1 -1 -1 -1 -1
car 0.0 0 -1 1711.0 560.0 1772.0 607.0 -1 -1 -1 -1 -1 -1 -1
bicycle 0.0 0 -1 2025.0 916.0 2133.0 1047.0 -1 -1 -1 -1 -1 -1 -1

they are the *txts which i put in the converting tool. thanks a lot, BY THE WAY, could you please tell me the exact names of trafficCamnet labels? i have checked the introduction there https://ngc.nvidia.com/catalog/models/nvidia:tlt_trafficcamnet

it only indicates here:

Training Data

|Environment|Images|Cars|Persons|Road Signs|Two-Wheelers|

are they Cars\ Persons\ Road Signs\ Two-Wheelers ? in this order? i am not sure, i only find a label.txt from pruned model, it indicates they are exactly same with mine. car person bicycle road_sign, all are lowercase

23

Sorry, I made a mistake. You cannot do this way (Take the names from the output label folder). Because the names are controlled by yourself in the inference spec file.

Please get the class names from the label file released in the deepstream
/opt/nvidia/deepstream/deepstream-5.0/samples/configs/tlt_pretrained_models/labels_trafficnet.txt

24

Please try to modify the target_class_mapping to the class_name of traficcamnet.

target_class_mapping {
key: “car”
value: “Car”
}
target_class_mapping {
key: “bicycle”
value: “Bicycle”
}
target_class_mapping {
key: “person”
value: “Person”
}
target_class_mapping {
key: “road_sign”
value: “Roadsign”
}

25

that is all right, i also checked the labels there ,/opt/nvidia/deepstream/deepstream-5.0/samples/configs/tlt_pretrained_models/labels_trafficnet.txt.

it is same with mine, i remembered , however, no matter what i retrained the model or evaluated the pretrained model, the average precisions are still nan or zero. it is so weird, and made me confused!!!

i show you my training logs here:

Epoch 17/120

Validation cost: 0.000153
Mean average_precision (in %): nan

class name average precision (in %)

bicycle nan
car nan
person nan
road_sign 0

Median Inference Time: 0.007796
INFO:tensorflow:epoch = 17.0, loss = 0.00019100773, step = 221 (31.251 sec)
2021-04-29 10:05:55,462 [INFO] tensorflow: epoch = 17.0, loss = 0.00019100773, step = 221 (31.251 sec)
INFO:tensorflow:global_step/sec: 0.031999
2021-04-29 10:05:55,463 [INFO] tensorflow: global_step/sec: 0.031999
2021-04-29 10:05:55,463 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 17/120: loss: 0.00019 Time taken: 0:00:36.739195 ETA: 1:03:04.137096
INFO:tensorflow:global_step/sec: 2.01125
2021-04-29 10:05:55,960 [INFO] tensorflow: global_step/sec: 2.01125
INFO:tensorflow:global_step/sec: 2.07289
2021-04-29 10:05:56,442 [INFO] tensorflow: global_step/sec: 2.07289
INFO:tensorflow:global_step/sec: 2.2077
2021-04-29 10:05:56,895 [INFO] tensorflow: global_step/sec: 2.2077
2021-04-29 10:05:56,896 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 11.308
INFO:tensorflow:global_step/sec: 1.96092
2021-04-29 10:05:57,405 [INFO] tensorflow: global_step/sec: 1.96092
INFO:tensorflow:global_step/sec: 2.16838
2021-04-29 10:05:57,866 [INFO] tensorflow: global_step/sec: 2.16838
INFO:tensorflow:global_step/sec: 1.9755
2021-04-29 10:05:58,373 [INFO] tensorflow: global_step/sec: 1.9755
INFO:tensorflow:global_step/sec: 2.24644
2021-04-29 10:05:58,818 [INFO] tensorflow: global_step/sec: 2.24644
INFO:tensorflow:global_step/sec: 2.11932
2021-04-29 10:05:59,290 [INFO] tensorflow: global_step/sec: 2.11932
INFO:tensorflow:global_step/sec: 1.97991
2021-04-29 10:05:59,795 [INFO] tensorflow: global_step/sec: 1.97991
INFO:tensorflow:global_step/sec: 2.21341
2021-04-29 10:06:00,246 [INFO] tensorflow: global_step/sec: 2.21341
INFO:tensorflow:global_step/sec: 2.04009
2021-04-29 10:06:00,737 [INFO] tensorflow: global_step/sec: 2.04009
INFO:tensorflow:epoch = 17.923076923076923, loss = 0.0001888557, step = 233 (5.781 sec)
2021-04-29 10:06:01,242 [INFO] tensorflow: epoch = 17.923076923076923, loss = 0.0001888557, step = 233 (5.781 sec)
INFO:tensorflow:global_step/sec: 1.97325
2021-04-29 10:06:01,243 [INFO] tensorflow: global_step/sec: 1.97325
INFO:tensorflow:Saving checkpoints for step-234.
2021-04-29 10:06:01,244 [INFO] tensorflow: Saving checkpoints for step-234.
WARNING:tensorflow:Ignoring: /tmp/tmp6wyrqeu8; No such file or directory
2021-04-29 10:06:01,347 [WARNING] tensorflow: Ignoring: /tmp/tmp6wyrqeu8; No such file or directory
2021-04-29 10:06:03,382 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 2, 0.00s/step
Matching predictions to ground truth, class 1/4.: 100%|█| 1250/1250 [00:00<00:00, 122451.42it/s]

Matching predictions to ground truth, class 2/4.: 100%|█| 10515/10515 [00:00<00:00, 120552.33it/s]
Matching predictions to ground truth, class 3/4.: 100%|█| 24633/24633 [00:00<00:00, 120974.24it/s]
Matching predictions to ground truth, class 4/4.: 100%|█| 9328/9328 [00:00<00:00, 24825.34it/s]
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/evaluation/compute_metrics.py:721: RuntimeWarning: invalid value encountered in true_divide
Epoch 18/120

Validation cost: 0.000153
Mean average_precision (in %): nan

class name average precision (in %)

bicycle nan
car nan
person nan
road_sign 0

Median Inference Time: 0.007637
INFO:tensorflow:epoch = 18.0, loss = 0.00017736945, step = 234 (33.249 sec)
2021-04-29 10:06:34,491 [INFO] tensorflow: epoch = 18.0, loss = 0.00017736945, step = 234 (33.249 sec)
INFO:tensorflow:global_step/sec: 0.0300766
2021-04-29 10:06:34,492 [INFO] tensorflow: global_step/sec: 0.0300766
2021-04-29 10:06:34,493 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 18/120: loss: 0.00018 Time taken: 0:00:39.035802 ETA: 1:06:21.651817
INFO:tensorflow:global_step/sec: 2.18018
2021-04-29 10:06:34,951 [INFO] tensorflow: global_step/sec: 2.18018
INFO:tensorflow:global_step/sec: 2.15134
2021-04-29 10:06:35,415 [INFO] tensorflow: global_step/sec: 2.15134
INFO:tensorflow:global_step/sec: 1.97641
2021-04-29 10:06:35,921 [INFO] tensorflow: global_step/sec: 1.97641
INFO:tensorflow:global_step/sec: 2.0408
2021-04-29 10:06:36,411 [INFO] tensorflow: global_step/sec: 2.0408
INFO:tensorflow:global_step/sec: 2.12092
2021-04-29 10:06:36,883 [INFO] tensorflow: global_step/sec: 2.12092
INFO:tensorflow:global_step/sec: 2.13139
2021-04-29 10:06:37,352 [INFO] tensorflow: global_step/sec: 2.13139
INFO:tensorflow:global_step/sec: 1.99967
2021-04-29 10:06:37,852 [INFO] tensorflow: global_step/sec: 1.99967
INFO:tensorflow:global_step/sec: 2.24289
2021-04-29 10:06:38,298 [INFO] tensorflow: global_step/sec: 2.24289
INFO:tensorflow:global_step/sec: 1.96894
2021-04-29 10:06:38,806 [INFO] tensorflow: global_step/sec: 1.96894
INFO:tensorflow:global_step/sec: 2.09333
2021-04-29 10:06:39,283 [INFO] tensorflow: global_step/sec: 2.09333
INFO:tensorflow:global_step/sec: 2.07369
2021-04-29 10:06:39,766 [INFO] tensorflow: global_step/sec: 2.07369
INFO:tensorflow:epoch = 18.923076923076923, loss = 0.00017428152, step = 246 (5.754 sec)
2021-04-29 10:06:40,245 [INFO] tensorflow: epoch = 18.923076923076923, loss = 0.00017428152, step = 246 (5.754 sec)
INFO:tensorflow:global_step/sec: 2.08401
2021-04-29 10:06:40,246 [INFO] tensorflow: global_step/sec: 2.08401
2021-04-29 10:06:40,251 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 2, 0.00s/step
Matching predictions to ground truth, class 1/4.: 100%|█| 1318/1318 [00:00<00:00, 119363.74it/s]
Matching predictions to ground truth, class 2/4.: 100%|█| 11356/11356 [00:00<00:00, 122404.47it/s]
Matching predictions to ground truth, class 3/4.: 100%|█| 24888/24888 [00:00<00:00, 123658.39it/s]
Matching predictions to ground truth, class 4/4.: 100%|█| 9409/9409 [00:00<00:00, 24905.89it/s]
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/evaluation/compute_metrics.py:721: RuntimeWarning: invalid value encountered in true_divide
Epoch 19/120

Validation cost: 0.000153
Mean average_precision (in %): nan

class name average precision (in %)

bicycle nan
car nan
person nan
road_sign 0

i also show you my training config:
random_seed: 42
dataset_config {
data_sources {
tfrecords_path: “/workspace/tlt-experiments/data/tfrecords/kitti_trainval/*”
image_directory_path: “/workspace/tlt-experiments/data/training”
}
image_extension: “jpg”
target_class_mapping {
key: “car”
value: “car”
}
target_class_mapping {
key: “bicycle”
value: “bicycle”
}
target_class_mapping {
key: “person”
value: “person”
}
target_class_mapping {
key: “person_sitting”
value: “person”
}
target_class_mapping {
key: “van”
value: “car”
}
target_class_mapping {
key: “road_sign”
value: “road_sign”
}

validation_fold: 0
}
augmentation_config {
preprocessing {
output_image_width: 960
output_image_height: 544
crop_right: 960
crop_bottom: 544
min_bbox_width: 1.0
min_bbox_height: 1.0
output_image_channel: 3
}
spatial_augmentation {
hflip_probability: 0.5
zoom_min: 0.5
zoom_max: 4.0
translate_max_x: 8.0
translate_max_y: 8.0
}
color_augmentation {
hue_rotation_max: 25.0
saturation_shift_max: 0.20000000298
contrast_scale_max: 0.10000000149
contrast_center: 0.5
}
}
postprocessing_config {
target_class_config {
key: “car”
value {
clustering_config {
clustering_algorithm: DBSCAN
dbscan_confidence_threshold: 0.9
coverage_threshold: 0.00499999988824
dbscan_eps: 0.20000000298
dbscan_min_samples: 0.0500000007451
minimum_bounding_box_height: 15
}
}
}
target_class_config {
key: “bicycle”
value {
clustering_config {
clustering_algorithm: DBSCAN
dbscan_confidence_threshold: 0.9
coverage_threshold: 0.00499999988824
dbscan_eps: 0.15000000596
dbscan_min_samples: 0.0500000007451
minimum_bounding_box_height: 15
}
}
}
target_class_config {
key: “person”
value {
clustering_config {
clustering_algorithm: DBSCAN
dbscan_confidence_threshold: 0.9
coverage_threshold: 0.00749999983236
dbscan_eps: 0.230000004172
dbscan_min_samples: 0.0500000007451
minimum_bounding_box_height: 15
}
}
}
target_class_config {
key: “road_sign”
value {
clustering_config {
clustering_algorithm: DBSCAN
dbscan_confidence_threshold: 0.9
coverage_threshold: 0.00749999983236
dbscan_eps: 0.230000004172
dbscan_min_samples: 0.0500000007451
minimum_bounding_box_height: 15
}
}
}
}
model_config {
pretrained_model_file: “/workspace/tlt-experiments/detectnet_v2/pretrained_trafficcamnet_Ucit/tlt_trafficcamnet_unpruned_v1.0/resnet18_trafficcamnet.tlt”
num_layers: 18
use_batch_norm: true
objective_set {
bbox {
scale: 35.0
offset: 0.5
}
cov {
}
}
training_precision {
backend_floatx: FLOAT32
}
arch: “resnet”
all_projections:true
}
evaluation_config {
validation_period_during_training: 1
first_validation_epoch: 0
minimum_detection_ground_truth_overlap {
key: “car”
value: 0.699999988079
}
minimum_detection_ground_truth_overlap {
key: “bicycle”
value: 0.5
}
minimum_detection_ground_truth_overlap {
key: “person”
value: 0.5
}
minimum_detection_ground_truth_overlap {
key: “road_sign”
value: 0.5
}
evaluation_box_config {
key: “car”
value {
minimum_height: 20
maximum_height: 9999
minimum_width: 10
maximum_width: 9999
}
}
evaluation_box_config {
key: “bicycle”
value {
minimum_height: 20
maximum_height: 9999
minimum_width: 10
maximum_width: 9999
}
}
evaluation_box_config {
key: “person”
value {
minimum_height: 20
maximum_height: 9999
minimum_width: 10
maximum_width: 9999
}
}
evaluation_box_config {
key: “road_sign”
value {
minimum_height: 20
maximum_height: 9999
minimum_width: 10
maximum_width: 9999
}
}
average_precision_mode: INTEGRATE
}
cost_function_config {
target_classes {
name: “car”
class_weight: 1.0
coverage_foreground_weight: 0.0500000007451
objectives {
name: “cov”
initial_weight: 1.0
weight_target: 1.0
}
objectives {
name: “bbox”
initial_weight: 10.0
weight_target: 10.0
}
}
target_classes {
name: “bicycle”
class_weight: 8.0
coverage_foreground_weight: 0.0500000007451
objectives {
name: “cov”
initial_weight: 1.0
weight_target: 1.0
}
objectives {
name: “bbox”
initial_weight: 10.0
weight_target: 1.0
}
}
target_classes {
name: “person”
class_weight: 4.0
coverage_foreground_weight: 0.0500000007451
objectives {
name: “cov”
initial_weight: 1.0
weight_target: 1.0
}
objectives {
name: “bbox”
initial_weight: 10.0
weight_target: 10.0
}
}
target_classes {
name: “road_sign”
class_weight: 4.0
coverage_foreground_weight: 0.0500000007451
objectives {
name: “cov”
initial_weight: 1.0
weight_target: 1.0
}
objectives {
name: “bbox”
initial_weight: 10.0
weight_target: 10.0
}
}

enable_autoweighting: true
max_objective_weight: 0.999899983406
min_objective_weight: 9.99999974738e-05
}
training_config {
batch_size_per_gpu: 32
num_epochs: 120
learning_rate {
soft_start_annealing_schedule {
min_learning_rate: 10e-15
max_learning_rate: 10e-15
soft_start: 0.10000000149
annealing: 0.699999988079
}
}
regularizer {
type: L1
weight: 3.00000002618e-09
}
optimizer {
adam {
epsilon: 9.99999993923e-09
beta1: 0.899999976158
beta2: 0.999000012875
}
}
cost_scaling {
initial_exponent: 20.0
increment: 0.005
decrement: 1.0
}
checkpoint_interval: 3
}
bbox_rasterizer_config {
target_class_config {
key: “car”
value {
cov_center_x: 0.5
cov_center_y: 0.5
cov_radius_x: 0.40000000596
cov_radius_y: 0.40000000596
bbox_min_radius: 1.0
}
}
target_class_config {
key: “bicycle”
value {
cov_center_x: 0.5
cov_center_y: 0.5
cov_radius_x: 1.0
cov_radius_y: 1.0
bbox_min_radius: 1.0
}
}
target_class_config {
key: “person”
value {
cov_center_x: 0.5
cov_center_y: 0.5
cov_radius_x: 1.0
cov_radius_y: 1.0
bbox_min_radius: 1.0
}
}
target_class_config {
key: “road_sign”
value {
cov_center_x: 0.5
cov_center_y: 0.5
cov_radius_x: 1.0
cov_radius_y: 1.0
bbox_min_radius: 1.0
}
}
deadzone_radius: 0.400000154972
}

26

Hey, what we taking about is to evaluate your dataset with the official released unpruned tlt model. After you set

load_graph: True

min_learning_rate: 10e-10
max_learning_rate: 10e-10

It is just for evaluation(please run tlt detectnet_v2 evaluate). It is not valid for training.

27

Hi, it throws the Assert error:
File “/home/vpraveen/.cache/dazel/_dazel_vpraveen/216c8b41e526c3295d3b802489ac2034/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/evaluation/compute_metrics.py”, line 301, in _check_if_bbox_is_valid
KeyError: ‘Roadsign’
Traceback (most recent call last):

random_seed: 42
dataset_config {
data_sources {
tfrecords_path: “/workspace/tlt-experiments/data/tfrecords/kitti_trainval/*”
image_directory_path: “/workspace/tlt-experiments/data/training”
}
image_extension: “jpg”
target_class_mapping {
key: “car”
value: “Car”
}
target_class_mapping {
key: “bicycle”
value: “Bicycle”
}
target_class_mapping {
key: “person”
value: “Person”
}
target_class_mapping {
key: “road_sign”
value: “Roadsign”
}

validation_fold: 0
}
augmentation_config {
preprocessing {
output_image_width: 960
output_image_height: 544
crop_right: 960
crop_bottom: 544
min_bbox_width: 1.0
min_bbox_height: 1.0
output_image_channel: 3
}
spatial_augmentation {
hflip_probability: 0.5
zoom_min: 0.8
zoom_max: 2.0
translate_max_x: 8.0
translate_max_y: 8.0
}
color_augmentation {
hue_rotation_max: 25.0
saturation_shift_max: 0.20000000298
contrast_scale_max: 0.10000000149
contrast_center: 0.5
}
}
postprocessing_config {
target_class_config {
key: “car”
value {
clustering_config {
clustering_algorithm: DBSCAN
dbscan_confidence_threshold: 0.9
coverage_threshold: 0.00499999988824
dbscan_eps: 0.20000000298
dbscan_min_samples: 0.0500000007451
minimum_bounding_box_height: 20
}
}
}
target_class_config {
key: “bicycle”
value {
clustering_config {
clustering_algorithm: DBSCAN
dbscan_confidence_threshold: 0.9
coverage_threshold: 0.00499999988824
dbscan_eps: 0.15000000596
dbscan_min_samples: 0.0500000007451
minimum_bounding_box_height: 20
}
}
}
target_class_config {
key: “person”
value {
clustering_config {
clustering_algorithm: DBSCAN
dbscan_confidence_threshold: 0.9
coverage_threshold: 0.00749999983236
dbscan_eps: 0.230000004172
dbscan_min_samples: 0.0500000007451
minimum_bounding_box_height: 20
}
}
}
target_class_config {
key: “road_sign”
value {
clustering_config {
clustering_algorithm: DBSCAN
dbscan_confidence_threshold: 0.9
coverage_threshold: 0.00749999983236
dbscan_eps: 0.230000004172
dbscan_min_samples: 0.0500000007451
minimum_bounding_box_height: 20
}
}
}
}
model_config {
pretrained_model_file: “/workspace/tlt-experiments/detectnet_v2/pretrained_trafficcamnet_Ucit/tlt_trafficcamnet_unpruned_v1.0/resnet18_trafficcamnet.tlt”
num_layers: 18
load_graph: True
use_batch_norm: False

activation {
activation_type: “relu”
}
objective_set {
bbox {
scale: 35.0
offset: 0.5
}
cov {
}
}
training_precision {
backend_floatx: FLOAT32
}
arch: “resnet”
}
evaluation_config {
validation_period_during_training: 1
first_validation_epoch: 1
minimum_detection_ground_truth_overlap {
key: “car”
value: 0.699999988079
}
minimum_detection_ground_truth_overlap {
key: “bicycle”
value: 0.5
}
minimum_detection_ground_truth_overlap {
key: “person”
value: 0.5
}
minimum_detection_ground_truth_overlap {
key: “road_sign”
value: 0.5
}
evaluation_box_config {
key: “car”
value {
minimum_height: 20
maximum_height: 9999
minimum_width: 10
maximum_width: 9999
}
}
evaluation_box_config {
key: “bicycle”
value {
minimum_height: 20
maximum_height: 9999
minimum_width: 10
maximum_width: 9999
}
}
evaluation_box_config {
key: “person”
value {
minimum_height: 20
maximum_height: 9999
minimum_width: 10
maximum_width: 9999
}
}
evaluation_box_config {
key: “road_sign”
value {
minimum_height: 20
maximum_height: 9999
minimum_width: 10
maximum_width: 9999
}
}
average_precision_mode: INTEGRATE
}
cost_function_config {
target_classes {
name: “car”
class_weight: 1.0
coverage_foreground_weight: 0.0500000007451
objectives {
name: “cov”
initial_weight: 1.0
weight_target: 1.0
}
objectives {
name: “bbox”
initial_weight: 10.0
weight_target: 10.0
}
}
target_classes {
name: “bicycle”
class_weight: 8.0
coverage_foreground_weight: 0.0500000007451
objectives {
name: “cov”
initial_weight: 1.0
weight_target: 1.0
}
objectives {
name: “bbox”
initial_weight: 10.0
weight_target: 1.0
}
}
target_classes {
name: “person”
class_weight: 4.0
coverage_foreground_weight: 0.0500000007451
objectives {
name: “cov”
initial_weight: 1.0
weight_target: 1.0
}
objectives {
name: “bbox”
initial_weight: 10.0
weight_target: 10.0
}
}
target_classes {
name: “road_sign”
class_weight: 4.0
coverage_foreground_weight: 0.0500000007451
objectives {
name: “cov”
initial_weight: 1.0
weight_target: 1.0
}
objectives {
name: “bbox”
initial_weight: 10.0
weight_target: 10.0
}
}

enable_autoweighting: true
max_objective_weight: 0.999899983406
min_objective_weight: 9.99999974738e-05
}
training_config {
batch_size_per_gpu: 40
num_epochs: 12
learning_rate {
soft_start_annealing_schedule {
min_learning_rate: 10e-10
max_learning_rate: 10e-10
soft_start: 0.10000000149
annealing: 0.699999988079
}
}
regularizer {
type: L1
weight: 3.00000002618e-09
}
optimizer {
adam {
epsilon: 9.99999993923e-09
beta1: 0.899999976158
beta2: 0.999000012875
}
}
cost_scaling {
initial_exponent: 20.0
increment: 0.005
decrement: 1.0
}
checkpoint_interval: 3
}
bbox_rasterizer_config {
target_class_config {
key: “car”
value {
cov_center_x: 0.5
cov_center_y: 0.5
cov_radius_x: 0.40000000596
cov_radius_y: 0.40000000596
bbox_min_radius: 1.0
}
}
target_class_config {
key: “bicycle”
value {
cov_center_x: 0.5
cov_center_y: 0.5
cov_radius_x: 1.0
cov_radius_y: 1.0
bbox_min_radius: 1.0
}
}
target_class_config {
key: “person”
value {
cov_center_x: 0.5
cov_center_y: 0.5
cov_radius_x: 1.0
cov_radius_y: 1.0
bbox_min_radius: 1.0
}
}
target_class_config {
key: “road_sign”
value {
cov_center_x: 0.5
cov_center_y: 0.5
cov_radius_x: 1.0
cov_radius_y: 1.0
bbox_min_radius: 1.0
}
}
deadzone_radius: 0.400000154972
}

28

I also tried to change like

target_class_mapping {
key: “road_sign”
value: “Road_sign”
}

target_class_mapping {
key: “road_sign”
value: “RoadSign”
}

also popped the similar AssertError

29

@Morganh , BY THE WAY, i just want to tried the TLT training procedure step by stey on the basis of jupyter notebook of detectnet_v2…

i only labelled around 400 high quality pics to train, does the unexpected evaluation of model result from too small number of dataSets? i am not sure

30

@Morganh Hi, i keeped target_class_mapping {
key: “road_sign”
value: “road_sign”
}, it is no error for evaluation pretrained model.

Could you plz give me some suggestions to retrain trafficCamnet resnet18?

i don’t intend to modify trafficCamnet too much due to the shortage of my own dataSets(only 400 images), and i only want to enhance a little bit on the performance of person and some bicycle detection on some certain scenarios.

here is my dataSet overall information
2021-04-30 05:21:22,851 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO -
Wrote the following numbers of objects:
b’car’: 1470
b’road_sign’: 422
b’person’: 332
b’bicycle’: 147

i pasted it from coverting log, any suggestions to retrain trafficCamnet? i will appreciate you so much for tuning my SPEC files:

random_seed: 42
dataset_config {
data_sources {
tfrecords_path: “/workspace/tlt-experiments/data/tfrecords/kitti_trainval/*”
image_directory_path: “/workspace/tlt-experiments/data/training”
}
image_extension: “jpg”
target_class_mapping {
key: “car”
value: “Car”
}
target_class_mapping {
key: “bicycle”
value: “Bicycle”
}
target_class_mapping {
key: “person”
value: “Person”
}
target_class_mapping {
key: “person_sitting”
value: “Person”
}
target_class_mapping {
key: “van”
value: “Car”
}
target_class_mapping {
key: “road_sign”
value: “road_sign”
}

validation_fold: 0
}
augmentation_config {
preprocessing {
output_image_width: 960
output_image_height: 544
crop_right: 960
crop_bottom: 544
min_bbox_width: 1.0
min_bbox_height: 1.0
output_image_channel: 3
}
spatial_augmentation {
hflip_probability: 0.5
zoom_min: 1
zoom_max: 1
translate_max_x: 8.0
translate_max_y: 8.0
}
color_augmentation {
hue_rotation_max: 25.0
saturation_shift_max: 0.20000000298
contrast_scale_max: 0.10000000149
contrast_center: 0.5
}
}
postprocessing_config {
target_class_config {
key: “car”
value {
clustering_config {
clustering_algorithm: DBSCAN
dbscan_confidence_threshold: 0.9
coverage_threshold: 0.00499999988824
dbscan_eps: 0.20000000298
dbscan_min_samples: 0.0500000007451
minimum_bounding_box_height: 15
}
}
}
target_class_config {
key: “bicycle”
value {
clustering_config {
clustering_algorithm: DBSCAN
dbscan_confidence_threshold: 0.9
coverage_threshold: 0.00499999988824
dbscan_eps: 0.15000000596
dbscan_min_samples: 0.0500000007451
minimum_bounding_box_height: 15
}
}
}
target_class_config {
key: “person”
value {
clustering_config {
clustering_algorithm: DBSCAN
dbscan_confidence_threshold: 0.9
coverage_threshold: 0.00749999983236
dbscan_eps: 0.230000004172
dbscan_min_samples: 0.0500000007451
minimum_bounding_box_height: 15
}
}
}
target_class_config {
key: “road_sign”
value {
clustering_config {
clustering_algorithm: DBSCAN
dbscan_confidence_threshold: 0.9
coverage_threshold: 0.00749999983236
dbscan_eps: 0.230000004172
dbscan_min_samples: 0.0500000007451
minimum_bounding_box_height: 15
}
}
}
}
model_config {
pretrained_model_file: “/workspace/tlt-experiments/detectnet_v2/pretrained_trafficcamnet_Ucit/tlt_trafficcamnet_unpruned_v1.0/resnet18_trafficcamnet.tlt”
num_layers: 18
use_batch_norm: true
objective_set {
bbox {
scale: 35.0
offset: 0.5
}
cov {
}
}
training_precision {
backend_floatx: FLOAT32
}
arch: “resnet”
all_projections:True
}
evaluation_config {
validation_period_during_training: 5
first_validation_epoch: 1
minimum_detection_ground_truth_overlap {
key: “car”
value: 0.699999988079
}
minimum_detection_ground_truth_overlap {
key: “bicycle”
value: 0.5
}
minimum_detection_ground_truth_overlap {
key: “person”
value: 0.5
}
minimum_detection_ground_truth_overlap {
key: “road_sign”
value: 0.5
}
evaluation_box_config {
key: “car”
value {
minimum_height: 20
maximum_height: 9999
minimum_width: 10
maximum_width: 9999
}
}
evaluation_box_config {
key: “bicycle”
value {
minimum_height: 20
maximum_height: 9999
minimum_width: 10
maximum_width: 9999
}
}
evaluation_box_config {
key: “person”
value {
minimum_height: 20
maximum_height: 9999
minimum_width: 10
maximum_width: 9999
}
}
evaluation_box_config {
key: “road_sign”
value {
minimum_height: 20
maximum_height: 9999
minimum_width: 10
maximum_width: 9999
}
}
average_precision_mode: INTEGRATE
}
cost_function_config {
target_classes {
name: “car”
class_weight: 1.0
coverage_foreground_weight: 0.0500000007451
objectives {
name: “cov”
initial_weight: 1.0
weight_target: 1.0
}
objectives {
name: “bbox”
initial_weight: 10.0
weight_target: 10.0
}
}
target_classes {
name: “bicycle”
class_weight: 8.0
coverage_foreground_weight: 0.0500000007451
objectives {
name: “cov”
initial_weight: 1.0
weight_target: 1.0
}
objectives {
name: “bbox”
initial_weight: 10.0
weight_target: 1.0
}
}
target_classes {
name: “person”
class_weight: 4.0
coverage_foreground_weight: 0.0500000007451
objectives {
name: “cov”
initial_weight: 1.0
weight_target: 1.0
}
objectives {
name: “bbox”
initial_weight: 10.0
weight_target: 10.0
}
}
target_classes {
name: “road_sign”
class_weight: 4.0
coverage_foreground_weight: 0.0500000007451
objectives {
name: “cov”
initial_weight: 1.0
weight_target: 1.0
}
objectives {
name: “bbox”
initial_weight: 10.0
weight_target: 10.0
}
}

enable_autoweighting: true
max_objective_weight: 0.999899983406
min_objective_weight: 9.99999974738e-05
}
training_config {
batch_size_per_gpu: 24
num_epochs: 5
learning_rate {
soft_start_annealing_schedule {
min_learning_rate: 9e-08
max_learning_rate: 0.0000045
soft_start: 0.10000000149
annealing: 0.699999988079
}
}
regularizer {
type: L1
weight: 3.00000002618e-09
}
optimizer {
adam {
epsilon: 9.99999993923e-09
beta1: 0.899999976158
beta2: 0.999000012875
}
}
cost_scaling {
initial_exponent: 20.0
increment: 0.005
decrement: 1.0
}
checkpoint_interval: 5
}
bbox_rasterizer_config {
target_class_config {
key: “car”
value {
cov_center_x: 0.5
cov_center_y: 0.5
cov_radius_x: 0.40000000596
cov_radius_y: 0.40000000596
bbox_min_radius: 1.0
}
}
target_class_config {
key: “bicycle”
value {
cov_center_x: 0.5
cov_center_y: 0.5
cov_radius_x: 1.0
cov_radius_y: 1.0
bbox_min_radius: 1.0
}
}
target_class_config {
key: “person”
value {
cov_center_x: 0.5
cov_center_y: 0.5
cov_radius_x: 1.0
cov_radius_y: 1.0
bbox_min_radius: 1.0
}
}
target_class_config {
key: “road_sign”
value {
cov_center_x: 0.5
cov_center_y: 0.5
cov_radius_x: 1.0
cov_radius_y: 1.0
bbox_min_radius: 1.0
}
}
deadzone_radius: 0.400000154972
}

31

i am wondering that are there any ways to filter some layers not to retrain ? namely froze the chosen layers like this: for example

frozen some layers

for p in self.parameters():
p.requires_grad = False

32

TLT detectnet_v2 network provides freeze_blocks parameters.
See DetectNet_v2 — Transfer Learning Toolkit 3.0 documentation