Morganh:
labels_trafficnet
that is all right, i also checked the labels there ,/opt/nvidia/deepstream/deepstream-5.0/samples/configs/tlt_pretrained_models/labels_trafficnet.txt
.
it is same with mine, i remembered , however, no matter what i retrained the model or evaluated the pretrained model, the average precisions are still nan or zero. it is so weird, and made me confused!!!
i show you my training logs here:
Epoch 17/120
Validation cost: 0.000153
Mean average_precision (in %): nan
class name average precision (in %)
bicycle nan
car nan
person nan
road_sign 0
Median Inference Time: 0.007796
INFO:tensorflow:epoch = 17.0, loss = 0.00019100773, step = 221 (31.251 sec)
2021-04-29 10:05:55,462 [INFO] tensorflow: epoch = 17.0, loss = 0.00019100773, step = 221 (31.251 sec)
INFO:tensorflow:global_step/sec: 0.031999
2021-04-29 10:05:55,463 [INFO] tensorflow: global_step/sec: 0.031999
2021-04-29 10:05:55,463 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 17/120: loss: 0.00019 Time taken: 0:00:36.739195 ETA: 1:03:04.137096
INFO:tensorflow:global_step/sec: 2.01125
2021-04-29 10:05:55,960 [INFO] tensorflow: global_step/sec: 2.01125
INFO:tensorflow:global_step/sec: 2.07289
2021-04-29 10:05:56,442 [INFO] tensorflow: global_step/sec: 2.07289
INFO:tensorflow:global_step/sec: 2.2077
2021-04-29 10:05:56,895 [INFO] tensorflow: global_step/sec: 2.2077
2021-04-29 10:05:56,896 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 11.308
INFO:tensorflow:global_step/sec: 1.96092
2021-04-29 10:05:57,405 [INFO] tensorflow: global_step/sec: 1.96092
INFO:tensorflow:global_step/sec: 2.16838
2021-04-29 10:05:57,866 [INFO] tensorflow: global_step/sec: 2.16838
INFO:tensorflow:global_step/sec: 1.9755
2021-04-29 10:05:58,373 [INFO] tensorflow: global_step/sec: 1.9755
INFO:tensorflow:global_step/sec: 2.24644
2021-04-29 10:05:58,818 [INFO] tensorflow: global_step/sec: 2.24644
INFO:tensorflow:global_step/sec: 2.11932
2021-04-29 10:05:59,290 [INFO] tensorflow: global_step/sec: 2.11932
INFO:tensorflow:global_step/sec: 1.97991
2021-04-29 10:05:59,795 [INFO] tensorflow: global_step/sec: 1.97991
INFO:tensorflow:global_step/sec: 2.21341
2021-04-29 10:06:00,246 [INFO] tensorflow: global_step/sec: 2.21341
INFO:tensorflow:global_step/sec: 2.04009
2021-04-29 10:06:00,737 [INFO] tensorflow: global_step/sec: 2.04009
INFO:tensorflow:epoch = 17.923076923076923, loss = 0.0001888557, step = 233 (5.781 sec)
2021-04-29 10:06:01,242 [INFO] tensorflow: epoch = 17.923076923076923, loss = 0.0001888557, step = 233 (5.781 sec)
INFO:tensorflow:global_step/sec: 1.97325
2021-04-29 10:06:01,243 [INFO] tensorflow: global_step/sec: 1.97325
INFO:tensorflow:Saving checkpoints for step-234.
2021-04-29 10:06:01,244 [INFO] tensorflow: Saving checkpoints for step-234.
WARNING:tensorflow:Ignoring: /tmp/tmp6wyrqeu8; No such file or directory
2021-04-29 10:06:01,347 [WARNING] tensorflow: Ignoring: /tmp/tmp6wyrqeu8; No such file or directory
2021-04-29 10:06:03,382 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 2, 0.00s/step
Matching predictions to ground truth, class 1/4.: 100%|█| 1250/1250 [00:00<00:00, 122451.42it/s]
Matching predictions to ground truth, class 2/4.: 100%|█| 10515/10515 [00:00<00:00, 120552.33it/s]
Matching predictions to ground truth, class 3/4.: 100%|█| 24633/24633 [00:00<00:00, 120974.24it/s]
Matching predictions to ground truth, class 4/4.: 100%|█| 9328/9328 [00:00<00:00, 24825.34it/s]
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/evaluation/compute_metrics.py:721: RuntimeWarning: invalid value encountered in true_divide
Epoch 18/120
Validation cost: 0.000153
Mean average_precision (in %): nan
class name average precision (in %)
bicycle nan
car nan
person nan
road_sign 0
Median Inference Time: 0.007637
INFO:tensorflow:epoch = 18.0, loss = 0.00017736945, step = 234 (33.249 sec)
2021-04-29 10:06:34,491 [INFO] tensorflow: epoch = 18.0, loss = 0.00017736945, step = 234 (33.249 sec)
INFO:tensorflow:global_step/sec: 0.0300766
2021-04-29 10:06:34,492 [INFO] tensorflow: global_step/sec: 0.0300766
2021-04-29 10:06:34,493 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 18/120: loss: 0.00018 Time taken: 0:00:39.035802 ETA: 1:06:21.651817
INFO:tensorflow:global_step/sec: 2.18018
2021-04-29 10:06:34,951 [INFO] tensorflow: global_step/sec: 2.18018
INFO:tensorflow:global_step/sec: 2.15134
2021-04-29 10:06:35,415 [INFO] tensorflow: global_step/sec: 2.15134
INFO:tensorflow:global_step/sec: 1.97641
2021-04-29 10:06:35,921 [INFO] tensorflow: global_step/sec: 1.97641
INFO:tensorflow:global_step/sec: 2.0408
2021-04-29 10:06:36,411 [INFO] tensorflow: global_step/sec: 2.0408
INFO:tensorflow:global_step/sec: 2.12092
2021-04-29 10:06:36,883 [INFO] tensorflow: global_step/sec: 2.12092
INFO:tensorflow:global_step/sec: 2.13139
2021-04-29 10:06:37,352 [INFO] tensorflow: global_step/sec: 2.13139
INFO:tensorflow:global_step/sec: 1.99967
2021-04-29 10:06:37,852 [INFO] tensorflow: global_step/sec: 1.99967
INFO:tensorflow:global_step/sec: 2.24289
2021-04-29 10:06:38,298 [INFO] tensorflow: global_step/sec: 2.24289
INFO:tensorflow:global_step/sec: 1.96894
2021-04-29 10:06:38,806 [INFO] tensorflow: global_step/sec: 1.96894
INFO:tensorflow:global_step/sec: 2.09333
2021-04-29 10:06:39,283 [INFO] tensorflow: global_step/sec: 2.09333
INFO:tensorflow:global_step/sec: 2.07369
2021-04-29 10:06:39,766 [INFO] tensorflow: global_step/sec: 2.07369
INFO:tensorflow:epoch = 18.923076923076923, loss = 0.00017428152, step = 246 (5.754 sec)
2021-04-29 10:06:40,245 [INFO] tensorflow: epoch = 18.923076923076923, loss = 0.00017428152, step = 246 (5.754 sec)
INFO:tensorflow:global_step/sec: 2.08401
2021-04-29 10:06:40,246 [INFO] tensorflow: global_step/sec: 2.08401
2021-04-29 10:06:40,251 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 2, 0.00s/step
Matching predictions to ground truth, class 1/4.: 100%|█| 1318/1318 [00:00<00:00, 119363.74it/s]
Matching predictions to ground truth, class 2/4.: 100%|█| 11356/11356 [00:00<00:00, 122404.47it/s]
Matching predictions to ground truth, class 3/4.: 100%|█| 24888/24888 [00:00<00:00, 123658.39it/s]
Matching predictions to ground truth, class 4/4.: 100%|█| 9409/9409 [00:00<00:00, 24905.89it/s]
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/evaluation/compute_metrics.py:721: RuntimeWarning: invalid value encountered in true_divide
Epoch 19/120
Validation cost: 0.000153
Mean average_precision (in %): nan
class name average precision (in %)
bicycle nan
car nan
person nan
road_sign 0
i also show you my training config:
random_seed: 42
dataset_config {
data_sources {
tfrecords_path: “/workspace/tlt-experiments/data/tfrecords/kitti_trainval/*”
image_directory_path: “/workspace/tlt-experiments/data/training”
}
image_extension: “jpg”
target_class_mapping {
key: “car”
value: “car”
}
target_class_mapping {
key: “bicycle”
value: “bicycle”
}
target_class_mapping {
key: “person”
value: “person”
}
target_class_mapping {
key: “person_sitting”
value: “person”
}
target_class_mapping {
key: “van”
value: “car”
}
target_class_mapping {
key: “road_sign”
value: “road_sign”
}
validation_fold: 0
}
augmentation_config {
preprocessing {
output_image_width: 960
output_image_height: 544
crop_right: 960
crop_bottom: 544
min_bbox_width: 1.0
min_bbox_height: 1.0
output_image_channel: 3
}
spatial_augmentation {
hflip_probability: 0.5
zoom_min: 0.5
zoom_max: 4.0
translate_max_x: 8.0
translate_max_y: 8.0
}
color_augmentation {
hue_rotation_max: 25.0
saturation_shift_max: 0.20000000298
contrast_scale_max: 0.10000000149
contrast_center: 0.5
}
}
postprocessing_config {
target_class_config {
key: “car”
value {
clustering_config {
clustering_algorithm: DBSCAN
dbscan_confidence_threshold: 0.9
coverage_threshold: 0.00499999988824
dbscan_eps: 0.20000000298
dbscan_min_samples: 0.0500000007451
minimum_bounding_box_height: 15
}
}
}
target_class_config {
key: “bicycle”
value {
clustering_config {
clustering_algorithm: DBSCAN
dbscan_confidence_threshold: 0.9
coverage_threshold: 0.00499999988824
dbscan_eps: 0.15000000596
dbscan_min_samples: 0.0500000007451
minimum_bounding_box_height: 15
}
}
}
target_class_config {
key: “person”
value {
clustering_config {
clustering_algorithm: DBSCAN
dbscan_confidence_threshold: 0.9
coverage_threshold: 0.00749999983236
dbscan_eps: 0.230000004172
dbscan_min_samples: 0.0500000007451
minimum_bounding_box_height: 15
}
}
}
target_class_config {
key: “road_sign”
value {
clustering_config {
clustering_algorithm: DBSCAN
dbscan_confidence_threshold: 0.9
coverage_threshold: 0.00749999983236
dbscan_eps: 0.230000004172
dbscan_min_samples: 0.0500000007451
minimum_bounding_box_height: 15
}
}
}
}
model_config {
pretrained_model_file: “/workspace/tlt-experiments/detectnet_v2/pretrained_trafficcamnet_Ucit/tlt_trafficcamnet_unpruned_v1.0/resnet18_trafficcamnet.tlt”
num_layers: 18
use_batch_norm: true
objective_set {
bbox {
scale: 35.0
offset: 0.5
}
cov {
}
}
training_precision {
backend_floatx: FLOAT32
}
arch: “resnet”
all_projections:true
}
evaluation_config {
validation_period_during_training: 1
first_validation_epoch: 0
minimum_detection_ground_truth_overlap {
key: “car”
value: 0.699999988079
}
minimum_detection_ground_truth_overlap {
key: “bicycle”
value: 0.5
}
minimum_detection_ground_truth_overlap {
key: “person”
value: 0.5
}
minimum_detection_ground_truth_overlap {
key: “road_sign”
value: 0.5
}
evaluation_box_config {
key: “car”
value {
minimum_height: 20
maximum_height: 9999
minimum_width: 10
maximum_width: 9999
}
}
evaluation_box_config {
key: “bicycle”
value {
minimum_height: 20
maximum_height: 9999
minimum_width: 10
maximum_width: 9999
}
}
evaluation_box_config {
key: “person”
value {
minimum_height: 20
maximum_height: 9999
minimum_width: 10
maximum_width: 9999
}
}
evaluation_box_config {
key: “road_sign”
value {
minimum_height: 20
maximum_height: 9999
minimum_width: 10
maximum_width: 9999
}
}
average_precision_mode: INTEGRATE
}
cost_function_config {
target_classes {
name: “car”
class_weight: 1.0
coverage_foreground_weight: 0.0500000007451
objectives {
name: “cov”
initial_weight: 1.0
weight_target: 1.0
}
objectives {
name: “bbox”
initial_weight: 10.0
weight_target: 10.0
}
}
target_classes {
name: “bicycle”
class_weight: 8.0
coverage_foreground_weight: 0.0500000007451
objectives {
name: “cov”
initial_weight: 1.0
weight_target: 1.0
}
objectives {
name: “bbox”
initial_weight: 10.0
weight_target: 1.0
}
}
target_classes {
name: “person”
class_weight: 4.0
coverage_foreground_weight: 0.0500000007451
objectives {
name: “cov”
initial_weight: 1.0
weight_target: 1.0
}
objectives {
name: “bbox”
initial_weight: 10.0
weight_target: 10.0
}
}
target_classes {
name: “road_sign”
class_weight: 4.0
coverage_foreground_weight: 0.0500000007451
objectives {
name: “cov”
initial_weight: 1.0
weight_target: 1.0
}
objectives {
name: “bbox”
initial_weight: 10.0
weight_target: 10.0
}
}
enable_autoweighting: true
max_objective_weight: 0.999899983406
min_objective_weight: 9.99999974738e-05
}
training_config {
batch_size_per_gpu: 32
num_epochs: 120
learning_rate {
soft_start_annealing_schedule {
min_learning_rate: 10e-15
max_learning_rate: 10e-15
soft_start: 0.10000000149
annealing: 0.699999988079
}
}
regularizer {
type: L1
weight: 3.00000002618e-09
}
optimizer {
adam {
epsilon: 9.99999993923e-09
beta1: 0.899999976158
beta2: 0.999000012875
}
}
cost_scaling {
initial_exponent: 20.0
increment: 0.005
decrement: 1.0
}
checkpoint_interval: 3
}
bbox_rasterizer_config {
target_class_config {
key: “car”
value {
cov_center_x: 0.5
cov_center_y: 0.5
cov_radius_x: 0.40000000596
cov_radius_y: 0.40000000596
bbox_min_radius: 1.0
}
}
target_class_config {
key: “bicycle”
value {
cov_center_x: 0.5
cov_center_y: 0.5
cov_radius_x: 1.0
cov_radius_y: 1.0
bbox_min_radius: 1.0
}
}
target_class_config {
key: “person”
value {
cov_center_x: 0.5
cov_center_y: 0.5
cov_radius_x: 1.0
cov_radius_y: 1.0
bbox_min_radius: 1.0
}
}
target_class_config {
key: “road_sign”
value {
cov_center_x: 0.5
cov_center_y: 0.5
cov_radius_x: 1.0
cov_radius_y: 1.0
bbox_min_radius: 1.0
}
}
deadzone_radius: 0.400000154972
}