SSD: custom tlt training result in AP:0 for all my classes

martin1 · August 26, 2020, 1:27pm

Hello all,
when using the TLT training with SSD model, i have loaded my own pictures and annotate with kitti format.

my my_ssd_train_resnet18_kitti.txt is:

Summary

random_seed: 42
ssd_config {
aspect_ratios_global: “[1.0, 2.0, 0.5, 3.0, 0.33]”
scales: “[0.05, 0.1, 0.25, 0.4, 0.55, 0.7, 0.85]”
two_boxes_for_ar1: true
clip_boxes: false
loss_loc_weight: 1.0
focal_loss_alpha: 0.25
focal_loss_gamma: 2.0
variances: “[0.1, 0.1, 0.2, 0.2]”
arch: “resnet”
nlayers: 18
freeze_bn: false
freeze_blocks: 0
}
training_config {
batch_size_per_gpu: 8
num_epochs: 80
enable_qat: false
learning_rate {
soft_start_annealing_schedule {
min_learning_rate: 5e-5
max_learning_rate: 2e-2
soft_start: 0.15
annealing: 0.8
}
}
regularizer {
type: L1
weight: 3e-5
}
}
eval_config {
validation_period_during_training: 10
average_precision_mode: SAMPLE
batch_size: 8
matching_iou_threshold: 0.5
}
nms_config {
confidence_threshold: 0.01
clustering_iou_threshold: 0.6
top_k: 200
}
augmentation_config {
preprocessing {
output_image_width: 910
output_image_height: 512
output_image_channel: 3
crop_right: 910
crop_bottom: 512
min_bbox_width: 1.0
min_bbox_height: 1.0
}
spatial_augmentation {
hflip_probability: 0.5
vflip_probability: 0.0
zoom_min: 0.7
zoom_max: 1.8
translate_max_x: 8.0
translate_max_y: 8.0
}
color_augmentation {
hue_rotation_max: 25.0
saturation_shift_max: 0.20000000298
contrast_scale_max: 0.10000000149
contrast_center: 0.5
}
}
dataset_config {
data_sources: {
tfrecords_path: “/workspace/tlt-experiments/data/rackets_v04_tlt_H32_910_512/tfrecords/kitti_trainval/kitti_trainval*”
image_directory_path: “/workspace/tlt-experiments/data/rackets_v04_tlt_H32_910_512/training”
}
image_extension: “jpeg”
target_class_mapping {
key: “BLACK_ARTENGO”
value: “BLACK_ARTENGO”
}
target_class_mapping {
key: “BLUE_ARTENGO”
value: “BLUE_ARTENGO”
}
target_class_mapping {
key: “RED_KUIKMA”
value: “RED_KUIKMA”
}
validation_fold: 0
}

the TLT-training results in AP: 0 for all my 3 classes:

Summary

To run with multigpu, please change --gpus based on the number of available GPUs in your machine.
Using TensorFlow backend.
2020-08-26 11:24:01.506089: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-08-26 11:24:05.412774: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-08-26 11:24:05.437956: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: Quadro M4000 major: 5 minor: 2 memoryClockRate(GHz): 0.7725
pciBusID: 0000:03:00.0
2020-08-26 11:24:05.438042: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-08-26 11:24:05.440499: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-08-26 11:24:05.442646: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-08-26 11:24:05.443244: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-08-26 11:24:05.446116: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-08-26 11:24:05.448268: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-08-26 11:24:05.455031: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-08-26 11:24:05.456931: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2020-08-26 11:24:05.457011: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-08-26 11:24:06.022616: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-08-26 11:24:06.022698: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0
2020-08-26 11:24:06.022712: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N
2020-08-26 11:24:06.024796: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6329 MB memory) → physical GPU (device: 0, name: Quadro M4000, pci bus id: 0000:03:00.0, compute capability: 5.2)
2020-08-26 11:24:06,031 [INFO] /usr/local/lib/python3.6/dist-packages/iva/ssd/utils/spec_loader.pyc: Merging specification from /workspace/examples/ssd/specs/my_ssd_train_resnet18_kitti.txt
2020-08-26 11:24:06,042 [INFO] iva.ssd.scripts.train: Loading pretrained weights. This may take a while…
2020-08-26 11:24:06,236 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Serial augmentation enabled = False
2020-08-26 11:24:06,236 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Pseudo sharding enabled = False
2020-08-26 11:24:06,236 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Max Image Dimensions (all sources): (0, 0)
2020-08-26 11:24:06,236 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: number of cpus: 56, io threads: 112, compute threads: 56, buffered batches: 4
2020-08-26 11:24:06,236 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: total dataset size 288, number of sources: 1, batch size per gpu: 8, steps: 36
2020-08-26 11:24:06,412 [INFO] iva.detectnet_v2.dataloader.default_dataloader: Bounding box coordinates were detected in the input specification! Bboxes will be automatically converted to polygon coordinates.
2020-08-26 11:24:06.465026: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: Quadro M4000 major: 5 minor: 2 memoryClockRate(GHz): 0.7725
pciBusID: 0000:03:00.0
2020-08-26 11:24:06.465084: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-08-26 11:24:06.465133: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-08-26 11:24:06.465177: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-08-26 11:24:06.465219: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-08-26 11:24:06.465262: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-08-26 11:24:06.465305: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-08-26 11:24:06.465348: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-08-26 11:24:06.466566: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2020-08-26 11:24:06,819 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: shuffle: True - shard 0 of 1
2020-08-26 11:24:06,829 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: sampling 1 datasets with weights:
2020-08-26 11:24:06,829 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: source: 0 weight: 1.000000
2020-08-26 11:25:08,536 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Serial augmentation enabled = False
2020-08-26 11:25:08,537 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Pseudo sharding enabled = False
2020-08-26 11:25:08,537 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Max Image Dimensions (all sources): (0, 0)
2020-08-26 11:25:08,537 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: number of cpus: 56, io threads: 112, compute threads: 56, buffered batches: 4
2020-08-26 11:25:08,537 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: total dataset size 46, number of sources: 1, batch size per gpu: 8, steps: 6
2020-08-26 11:25:08,611 [INFO] iva.detectnet_v2.dataloader.default_dataloader: Bounding box coordinates were detected in the input specification! Bboxes will be automatically converted to polygon coordinates.
2020-08-26 11:25:09,012 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: shuffle: False - shard 0 of 1
2020-08-26 11:25:09,021 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: sampling 1 datasets with weights:
2020-08-26 11:25:09,022 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: source: 0 weight: 1.000000

Layer (type) Output Shape Param # Connected to

Input (InputLayer) (8, 3, 512, 910) 0

conv1 (Conv2D) (8, 64, 256, 455) 9408 Input[0][0]

bn_conv1 (BatchNormalization) (8, 64, 256, 455) 256 conv1[0][0]

activation_1 (Activation) (8, 64, 256, 455) 0 bn_conv1[0][0]

block_1a_conv_1 (Conv2D) (8, 64, 128, 228) 36864 activation_1[0][0]

block_1a_bn_1 (BatchNormalizati (8, 64, 128, 228) 256 block_1a_conv_1[0][0]

block_1a_relu_1 (Activation) (8, 64, 128, 228) 0 block_1a_bn_1[0][0]

block_1a_conv_2 (Conv2D) (8, 64, 128, 228) 36864 block_1a_relu_1[0][0]

block_1a_conv_shortcut (Conv2D) (8, 64, 128, 228) 4096 activation_1[0][0]

block_1a_bn_2 (BatchNormalizati (8, 64, 128, 228) 256 block_1a_conv_2[0][0]

block_1a_bn_shortcut (BatchNorm (8, 64, 128, 228) 256 block_1a_conv_shortcut[0][0]

add_1 (Add) (8, 64, 128, 228) 0 block_1a_bn_2[0][0]
block_1a_bn_shortcut[0][0]

block_1a_relu (Activation) (8, 64, 128, 228) 0 add_1[0][0]

block_1b_conv_1 (Conv2D) (8, 64, 128, 228) 36864 block_1a_relu[0][0]

block_1b_bn_1 (BatchNormalizati (8, 64, 128, 228) 256 block_1b_conv_1[0][0]

block_1b_relu_1 (Activation) (8, 64, 128, 228) 0 block_1b_bn_1[0][0]

block_1b_conv_2 (Conv2D) (8, 64, 128, 228) 36864 block_1b_relu_1[0][0]

block_1b_conv_shortcut (Conv2D) (8, 64, 128, 228) 4096 block_1a_relu[0][0]

block_1b_bn_2 (BatchNormalizati (8, 64, 128, 228) 256 block_1b_conv_2[0][0]

block_1b_bn_shortcut (BatchNorm (8, 64, 128, 228) 256 block_1b_conv_shortcut[0][0]

add_2 (Add) (8, 64, 128, 228) 0 block_1b_bn_2[0][0]
block_1b_bn_shortcut[0][0]

block_1b_relu (Activation) (8, 64, 128, 228) 0 add_2[0][0]

block_2a_conv_1 (Conv2D) (8, 128, 64, 114) 73728 block_1b_relu[0][0]

block_2a_bn_1 (BatchNormalizati (8, 128, 64, 114) 512 block_2a_conv_1[0][0]

block_2a_relu_1 (Activation) (8, 128, 64, 114) 0 block_2a_bn_1[0][0]

block_2a_conv_2 (Conv2D) (8, 128, 64, 114) 147456 block_2a_relu_1[0][0]

block_2a_conv_shortcut (Conv2D) (8, 128, 64, 114) 8192 block_1b_relu[0][0]

block_2a_bn_2 (BatchNormalizati (8, 128, 64, 114) 512 block_2a_conv_2[0][0]

block_2a_bn_shortcut (BatchNorm (8, 128, 64, 114) 512 block_2a_conv_shortcut[0][0]

add_3 (Add) (8, 128, 64, 114) 0 block_2a_bn_2[0][0]
block_2a_bn_shortcut[0][0]

block_2a_relu (Activation) (8, 128, 64, 114) 0 add_3[0][0]

block_2b_conv_1 (Conv2D) (8, 128, 64, 114) 147456 block_2a_relu[0][0]

block_2b_bn_1 (BatchNormalizati (8, 128, 64, 114) 512 block_2b_conv_1[0][0]

block_2b_relu_1 (Activation) (8, 128, 64, 114) 0 block_2b_bn_1[0][0]

block_2b_conv_2 (Conv2D) (8, 128, 64, 114) 147456 block_2b_relu_1[0][0]

block_2b_conv_shortcut (Conv2D) (8, 128, 64, 114) 16384 block_2a_relu[0][0]

block_2b_bn_2 (BatchNormalizati (8, 128, 64, 114) 512 block_2b_conv_2[0][0]

block_2b_bn_shortcut (BatchNorm (8, 128, 64, 114) 512 block_2b_conv_shortcut[0][0]

add_4 (Add) (8, 128, 64, 114) 0 block_2b_bn_2[0][0]
block_2b_bn_shortcut[0][0]

block_2b_relu (Activation) (8, 128, 64, 114) 0 add_4[0][0]

block_3a_conv_1 (Conv2D) (8, 256, 32, 57) 294912 block_2b_relu[0][0]

block_3a_bn_1 (BatchNormalizati (8, 256, 32, 57) 1024 block_3a_conv_1[0][0]

block_3a_relu_1 (Activation) (8, 256, 32, 57) 0 block_3a_bn_1[0][0]

block_3a_conv_2 (Conv2D) (8, 256, 32, 57) 589824 block_3a_relu_1[0][0]

block_3a_conv_shortcut (Conv2D) (8, 256, 32, 57) 32768 block_2b_relu[0][0]

block_3a_bn_2 (BatchNormalizati (8, 256, 32, 57) 1024 block_3a_conv_2[0][0]

block_3a_bn_shortcut (BatchNorm (8, 256, 32, 57) 1024 block_3a_conv_shortcut[0][0]

add_5 (Add) (8, 256, 32, 57) 0 block_3a_bn_2[0][0]
block_3a_bn_shortcut[0][0]

block_3a_relu (Activation) (8, 256, 32, 57) 0 add_5[0][0]

block_3b_conv_1 (Conv2D) (8, 256, 32, 57) 589824 block_3a_relu[0][0]

block_3b_bn_1 (BatchNormalizati (8, 256, 32, 57) 1024 block_3b_conv_1[0][0]

block_3b_relu_1 (Activation) (8, 256, 32, 57) 0 block_3b_bn_1[0][0]

block_3b_conv_2 (Conv2D) (8, 256, 32, 57) 589824 block_3b_relu_1[0][0]

block_3b_conv_shortcut (Conv2D) (8, 256, 32, 57) 65536 block_3a_relu[0][0]

block_3b_bn_2 (BatchNormalizati (8, 256, 32, 57) 1024 block_3b_conv_2[0][0]

block_3b_bn_shortcut (BatchNorm (8, 256, 32, 57) 1024 block_3b_conv_shortcut[0][0]

add_6 (Add) (8, 256, 32, 57) 0 block_3b_bn_2[0][0]
block_3b_bn_shortcut[0][0]

block_3b_relu (Activation) (8, 256, 32, 57) 0 add_6[0][0]

block_4a_conv_1 (Conv2D) (8, 512, 32, 57) 1179648 block_3b_relu[0][0]

block_4a_bn_1 (BatchNormalizati (8, 512, 32, 57) 2048 block_4a_conv_1[0][0]

block_4a_relu_1 (Activation) (8, 512, 32, 57) 0 block_4a_bn_1[0][0]

block_4a_conv_2 (Conv2D) (8, 512, 32, 57) 2359296 block_4a_relu_1[0][0]

block_4a_conv_shortcut (Conv2D) (8, 512, 32, 57) 131072 block_3b_relu[0][0]

block_4a_bn_2 (BatchNormalizati (8, 512, 32, 57) 2048 block_4a_conv_2[0][0]

block_4a_bn_shortcut (BatchNorm (8, 512, 32, 57) 2048 block_4a_conv_shortcut[0][0]

add_7 (Add) (8, 512, 32, 57) 0 block_4a_bn_2[0][0]
block_4a_bn_shortcut[0][0]

block_4a_relu (Activation) (8, 512, 32, 57) 0 add_7[0][0]

block_4b_conv_1 (Conv2D) (8, 512, 32, 57) 2359296 block_4a_relu[0][0]

block_4b_bn_1 (BatchNormalizati (8, 512, 32, 57) 2048 block_4b_conv_1[0][0]

block_4b_relu_1 (Activation) (8, 512, 32, 57) 0 block_4b_bn_1[0][0]

block_4b_conv_2 (Conv2D) (8, 512, 32, 57) 2359296 block_4b_relu_1[0][0]

block_4b_conv_shortcut (Conv2D) (8, 512, 32, 57) 262144 block_4a_relu[0][0]

block_4b_bn_2 (BatchNormalizati (8, 512, 32, 57) 2048 block_4b_conv_2[0][0]

block_4b_bn_shortcut (BatchNorm (8, 512, 32, 57) 2048 block_4b_conv_shortcut[0][0]

add_8 (Add) (8, 512, 32, 57) 0 block_4b_bn_2[0][0]
block_4b_bn_shortcut[0][0]

block_4b_relu (Activation) (8, 512, 32, 57) 0 add_8[0][0]

ssd_expand_block_0_conv_0 (Conv (8, 256, 32, 57) 131328 block_4b_relu[0][0]

ssd_expand_block_0_relu_0 (ReLU (8, 256, 32, 57) 0 ssd_expand_block_0_conv_0[0][0]

ssd_expand_block_0_conv_1 (Conv (8, 256, 32, 57) 589824 ssd_expand_block_0_relu_0[0][0]

ssd_expand_block_0_bn_1 (BatchN (8, 256, 32, 57) 1024 ssd_expand_block_0_conv_1[0][0]

ssd_expand_block_0_relu_1 (ReLU (8, 256, 32, 57) 0 ssd_expand_block_0_bn_1[0][0]

ssd_expand_block_1_conv_0 (Conv (8, 128, 32, 57) 32896 ssd_expand_block_0_relu_1[0][0]

ssd_expand_block_1_relu_0 (ReLU (8, 128, 32, 57) 0 ssd_expand_block_1_conv_0[0][0]

ssd_expand_block_1_conv_1 (Conv (8, 256, 16, 29) 294912 ssd_expand_block_1_relu_0[0][0]

ssd_expand_block_1_bn_1 (BatchN (8, 256, 16, 29) 1024 ssd_expand_block_1_conv_1[0][0]

ssd_expand_block_1_relu_1 (ReLU (8, 256, 16, 29) 0 ssd_expand_block_1_bn_1[0][0]

ssd_expand_block_2_conv_0 (Conv (8, 64, 16, 29) 16448 ssd_expand_block_1_relu_1[0][0]

ssd_expand_block_2_relu_0 (ReLU (8, 64, 16, 29) 0 ssd_expand_block_2_conv_0[0][0]

ssd_expand_block_2_conv_1 (Conv (8, 128, 8, 15) 73728 ssd_expand_block_2_relu_0[0][0]

ssd_expand_block_2_bn_1 (BatchN (8, 128, 8, 15) 512 ssd_expand_block_2_conv_1[0][0]

ssd_expand_block_2_relu_1 (ReLU (8, 128, 8, 15) 0 ssd_expand_block_2_bn_1[0][0]

ssd_expand_block_3_conv_0 (Conv (8, 64, 8, 15) 8256 ssd_expand_block_2_relu_1[0][0]

ssd_expand_block_3_relu_0 (ReLU (8, 64, 8, 15) 0 ssd_expand_block_3_conv_0[0][0]

ssd_expand_block_3_conv_1 (Conv (8, 128, 4, 8) 73728 ssd_expand_block_3_relu_0[0][0]

ssd_expand_block_3_bn_1 (BatchN (8, 128, 4, 8) 512 ssd_expand_block_3_conv_1[0][0]

ssd_expand_block_3_relu_1 (ReLU (8, 128, 4, 8) 0 ssd_expand_block_3_bn_1[0][0]

ssd_expand_block_4_conv_0 (Conv (8, 64, 4, 8) 8256 ssd_expand_block_3_relu_1[0][0]

ssd_expand_block_4_relu_0 (ReLU (8, 64, 4, 8) 0 ssd_expand_block_4_conv_0[0][0]

ssd_expand_block_4_conv_1 (Conv (8, 128, 2, 4) 73728 ssd_expand_block_4_relu_0[0][0]

ssd_expand_block_4_bn_1 (BatchN (8, 128, 2, 4) 512 ssd_expand_block_4_conv_1[0][0]

ssd_expand_block_4_relu_1 (ReLU (8, 128, 2, 4) 0 ssd_expand_block_4_bn_1[0][0]

ssd_conf_0 (Conv2D) (8, 18, 64, 114) 20754 block_2b_relu[0][0]

ssd_conf_1 (Conv2D) (8, 18, 32, 57) 41490 ssd_expand_block_0_relu_1[0][0]

ssd_conf_2 (Conv2D) (8, 18, 16, 29) 41490 ssd_expand_block_1_relu_1[0][0]

ssd_conf_3 (Conv2D) (8, 18, 8, 15) 20754 ssd_expand_block_2_relu_1[0][0]

ssd_conf_4 (Conv2D) (8, 18, 4, 8) 20754 ssd_expand_block_3_relu_1[0][0]

ssd_conf_5 (Conv2D) (8, 18, 2, 4) 20754 ssd_expand_block_4_relu_1[0][0]

permute_1 (Permute) (8, 64, 114, 18) 0 ssd_conf_0[0][0]

permute_2 (Permute) (8, 32, 57, 18) 0 ssd_conf_1[0][0]

permute_3 (Permute) (8, 16, 29, 18) 0 ssd_conf_2[0][0]

permute_4 (Permute) (8, 8, 15, 18) 0 ssd_conf_3[0][0]

permute_5 (Permute) (8, 4, 8, 18) 0 ssd_conf_4[0][0]

permute_6 (Permute) (8, 2, 4, 18) 0 ssd_conf_5[0][0]

ssd_loc_0 (Conv2D) (8, 24, 64, 114) 27672 block_2b_relu[0][0]

ssd_loc_1 (Conv2D) (8, 24, 32, 57) 55320 ssd_expand_block_0_relu_1[0][0]

ssd_loc_2 (Conv2D) (8, 24, 16, 29) 55320 ssd_expand_block_1_relu_1[0][0]

ssd_loc_3 (Conv2D) (8, 24, 8, 15) 27672 ssd_expand_block_2_relu_1[0][0]

ssd_loc_4 (Conv2D) (8, 24, 4, 8) 27672 ssd_expand_block_3_relu_1[0][0]

ssd_loc_5 (Conv2D) (8, 24, 2, 4) 27672 ssd_expand_block_4_relu_1[0][0]

conf_reshape_0 (Reshape) (8, 43776, 1, 3) 0 permute_1[0][0]

conf_reshape_1 (Reshape) (8, 10944, 1, 3) 0 permute_2[0][0]

conf_reshape_2 (Reshape) (8, 2784, 1, 3) 0 permute_3[0][0]

conf_reshape_3 (Reshape) (8, 720, 1, 3) 0 permute_4[0][0]

conf_reshape_4 (Reshape) (8, 192, 1, 3) 0 permute_5[0][0]

conf_reshape_5 (Reshape) (8, 48, 1, 3) 0 permute_6[0][0]

permute_7 (Permute) (8, 64, 114, 24) 0 ssd_loc_0[0][0]

permute_8 (Permute) (8, 32, 57, 24) 0 ssd_loc_1[0][0]

permute_9 (Permute) (8, 16, 29, 24) 0 ssd_loc_2[0][0]

permute_10 (Permute) (8, 8, 15, 24) 0 ssd_loc_3[0][0]

permute_11 (Permute) (8, 4, 8, 24) 0 ssd_loc_4[0][0]

permute_12 (Permute) (8, 2, 4, 24) 0 ssd_loc_5[0][0]

ssd_anchor_0 (AnchorBoxes) (8, 7296, 6, 8) 0 ssd_loc_0[0][0]

ssd_anchor_1 (AnchorBoxes) (8, 1824, 6, 8) 0 ssd_loc_1[0][0]

ssd_anchor_2 (AnchorBoxes) (8, 464, 6, 8) 0 ssd_loc_2[0][0]

ssd_anchor_3 (AnchorBoxes) (8, 120, 6, 8) 0 ssd_loc_3[0][0]

ssd_anchor_4 (AnchorBoxes) (8, 32, 6, 8) 0 ssd_loc_4[0][0]

ssd_anchor_5 (AnchorBoxes) (8, 8, 6, 8) 0 ssd_loc_5[0][0]

mbox_conf (Concatenate) (8, 58464, 1, 3) 0 conf_reshape_0[0][0]
conf_reshape_1[0][0]
conf_reshape_2[0][0]
conf_reshape_3[0][0]
conf_reshape_4[0][0]
conf_reshape_5[0][0]

loc_reshape_0 (Reshape) (8, 43776, 1, 4) 0 permute_7[0][0]

loc_reshape_1 (Reshape) (8, 10944, 1, 4) 0 permute_8[0][0]

loc_reshape_2 (Reshape) (8, 2784, 1, 4) 0 permute_9[0][0]

loc_reshape_3 (Reshape) (8, 720, 1, 4) 0 permute_10[0][0]

loc_reshape_4 (Reshape) (8, 192, 1, 4) 0 permute_11[0][0]

loc_reshape_5 (Reshape) (8, 48, 1, 4) 0 permute_12[0][0]

anchor_reshape_0 (Reshape) (8, 43776, 1, 8) 0 ssd_anchor_0[0][0]

anchor_reshape_1 (Reshape) (8, 10944, 1, 8) 0 ssd_anchor_1[0][0]

anchor_reshape_2 (Reshape) (8, 2784, 1, 8) 0 ssd_anchor_2[0][0]

anchor_reshape_3 (Reshape) (8, 720, 1, 8) 0 ssd_anchor_3[0][0]

anchor_reshape_4 (Reshape) (8, 192, 1, 8) 0 ssd_anchor_4[0][0]

anchor_reshape_5 (Reshape) (8, 48, 1, 8) 0 ssd_anchor_5[0][0]

mbox_conf_sigmoid (Activation) (8, 58464, 1, 3) 0 mbox_conf[0][0]

mbox_loc (Concatenate) (8, 58464, 1, 4) 0 loc_reshape_0[0][0]
loc_reshape_1[0][0]
loc_reshape_2[0][0]
loc_reshape_3[0][0]
loc_reshape_4[0][0]
loc_reshape_5[0][0]

mbox_priorbox (Concatenate) (8, 58464, 1, 8) 0 anchor_reshape_0[0][0]
anchor_reshape_1[0][0]
anchor_reshape_2[0][0]
anchor_reshape_3[0][0]
anchor_reshape_4[0][0]
anchor_reshape_5[0][0]

concatenate_1 (Concatenate) (8, 58464, 1, 15) 0 mbox_conf_sigmoid[0][0]
mbox_loc[0][0]
mbox_priorbox[0][0]

ssd_predictions (Reshape) (8, 58464, 15) 0 concatenate_1[0][0]

Total params: 13,236,476
Trainable params: 13,213,628
Non-trainable params: 22,848

2020-08-26 11:25:14,701 [INFO] iva.ssd.scripts.train: Number of images in the training dataset: 288
2020-08-26 11:25:14,702 [INFO] iva.ssd.scripts.train: Number of images in the validation dataset: 46

Epoch 1/80
2020-08-26 11:25:25.225058: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-08-26 11:25:25.801597: I tensorflow/core/kernels/cuda_solvers.cc:159] Creating CudaSolver handles for stream 0x15409ab0
2020-08-26 11:25:25.801837: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-08-26 11:25:26.044771: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-08-26 11:25:26.045975: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-08-26 11:25:27.555419: W tensorflow/core/common_runtime/bfc_allocator.cc:305] Garbage collection: deallocate free memory regions (i.e., allocations) so that we can re-allocate a larger region to avoid OOM due to memory fragmentation. If you see this message frequently, you are running near the threshold of the available device memory and re-allocation may incur great performance overhead. You may try smaller batch sizes to observe the performance impact. Set TF_ENABLE_GPU_GARBAGE_COLLECTION=false if you’d like to disable this feature.
36/36 [==============================] - 43s 1s/step - loss: 17506.8723

Epoch 00001: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_001.tlt
Epoch 2/80
36/36 [==============================] - 28s 764ms/step - loss: 13.3258

Epoch 00002: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_002.tlt
Epoch 3/80
36/36 [==============================] - 27s 763ms/step - loss: 13.3379

Epoch 00003: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_003.tlt
Epoch 4/80
36/36 [==============================] - 27s 762ms/step - loss: 13.3173

Epoch 00004: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_004.tlt
Epoch 5/80
36/36 [==============================] - 27s 761ms/step - loss: 13.2983

Epoch 00005: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_005.tlt
Epoch 6/80
36/36 [==============================] - 28s 765ms/step - loss: 13.2838

Epoch 00006: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_006.tlt
Epoch 7/80
36/36 [==============================] - 28s 764ms/step - loss: 13.2706

Epoch 00007: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_007.tlt
Epoch 8/80
36/36 [==============================] - 27s 751ms/step - loss: 13.2582

Epoch 00008: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_008.tlt
Epoch 9/80
36/36 [==============================] - 27s 752ms/step - loss: 13.2432

Epoch 00009: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_009.tlt
Epoch 10/80
36/36 [==============================] - 27s 752ms/step - loss: 13.2234

Epoch 00010: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_010.tlt
Number of images in the evaluation dataset: 46

Producing predictions: 100%|██████████████████████| 6/6 [00:05<00:00, 1.12it/s]
Start multi-thread per-image matching
Start to calculate AP for each class

BLACK_ARTENGO AP 0.0
BLUE_ARTENGO AP 0.0
RED_KUIKMA AP 0.0
mAP 0.0

Epoch 11/80
36/36 [==============================] - 27s 752ms/step - loss: 13.1938

Epoch 00011: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_011.tlt
Epoch 12/80
36/36 [==============================] - 27s 753ms/step - loss: 13.1467

Epoch 00012: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_012.tlt
Epoch 13/80
36/36 [==============================] - 27s 753ms/step - loss: 13.0748

Epoch 00013: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_013.tlt
Epoch 14/80
36/36 [==============================] - 27s 751ms/step - loss: 12.9918

Epoch 00014: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_014.tlt
Epoch 15/80
36/36 [==============================] - 27s 751ms/step - loss: 12.9093

Epoch 00015: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_015.tlt
Epoch 16/80
36/36 [==============================] - 27s 751ms/step - loss: 12.8276

Epoch 00016: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_016.tlt
Epoch 17/80
36/36 [==============================] - 27s 752ms/step - loss: 12.7467

Epoch 00017: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_017.tlt
Epoch 18/80
36/36 [==============================] - 27s 751ms/step - loss: 12.6667

Epoch 00018: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_018.tlt
Epoch 19/80
36/36 [==============================] - 27s 753ms/step - loss: 12.5875

Epoch 00019: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_019.tlt
Epoch 20/80
36/36 [==============================] - 27s 752ms/step - loss: 12.5092

Epoch 00020: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_020.tlt
Number of images in the evaluation dataset: 46

Producing predictions: 100%|██████████████████████| 6/6 [00:04<00:00, 1.21it/s]
Start multi-thread per-image matching
Start to calculate AP for each class

BLACK_ARTENGO AP 0.0
BLUE_ARTENGO AP 0.0
RED_KUIKMA AP 0.0
mAP 0.0

Epoch 21/80
36/36 [==============================] - 27s 758ms/step - loss: 12.4317

Epoch 00021: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_021.tlt
Epoch 22/80
36/36 [==============================] - 27s 754ms/step - loss: 12.3550

Epoch 00022: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_022.tlt
Epoch 23/80
36/36 [==============================] - 27s 752ms/step - loss: 12.2791

Epoch 00023: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_023.tlt
Epoch 24/80
36/36 [==============================] - 27s 752ms/step - loss: 12.2040

Epoch 00024: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_024.tlt
Epoch 25/80
36/36 [==============================] - 27s 754ms/step - loss: 12.1298

Epoch 00025: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_025.tlt
Epoch 26/80
36/36 [==============================] - 27s 755ms/step - loss: 12.0563

Epoch 00026: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_026.tlt
Epoch 27/80
36/36 [==============================] - 27s 753ms/step - loss: 11.9837

Epoch 00027: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_027.tlt
Epoch 28/80
36/36 [==============================] - 27s 754ms/step - loss: 11.9119

Epoch 00028: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_028.tlt
Epoch 29/80
36/36 [==============================] - 27s 754ms/step - loss: 11.8409

Epoch 00029: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_029.tlt
Epoch 30/80
36/36 [==============================] - 27s 754ms/step - loss: 11.7706

Epoch 00030: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_030.tlt
Number of images in the evaluation dataset: 46

Producing predictions: 100%|██████████████████████| 6/6 [00:04<00:00, 1.25it/s]
Start multi-thread per-image matching
Start to calculate AP for each class

BLACK_ARTENGO AP 0.0
BLUE_ARTENGO AP 0.0
RED_KUIKMA AP 0.0
mAP 0.0

Epoch 31/80
36/36 [==============================] - 27s 757ms/step - loss: 11.7012

Epoch 00031: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_031.tlt
Epoch 32/80
36/36 [==============================] - 27s 754ms/step - loss: 11.6325

Epoch 00032: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_032.tlt
Epoch 33/80
36/36 [==============================] - 27s 754ms/step - loss: 11.5647

Epoch 00033: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_033.tlt
Epoch 34/80
36/36 [==============================] - 27s 754ms/step - loss: 11.4976

Epoch 00034: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_034.tlt
Epoch 35/80
36/36 [==============================] - 27s 752ms/step - loss: 11.4313

Epoch 00035: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_035.tlt
Epoch 36/80
36/36 [==============================] - 27s 754ms/step - loss: 11.3658

Epoch 00036: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_036.tlt
Epoch 37/80
36/36 [==============================] - 27s 753ms/step - loss: 11.3010

Epoch 00037: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_037.tlt
Epoch 38/80
36/36 [==============================] - 27s 754ms/step - loss: 11.2370

Epoch 00038: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_038.tlt
Epoch 39/80
36/36 [==============================] - 27s 754ms/step - loss: 11.1738

Epoch 00039: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_039.tlt
Epoch 40/80
36/36 [==============================] - 27s 754ms/step - loss: 11.1113

Epoch 00040: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_040.tlt
Number of images in the evaluation dataset: 46

Producing predictions: 100%|██████████████████████| 6/6 [00:05<00:00, 1.15it/s]
Start multi-thread per-image matching
Start to calculate AP for each class

BLACK_ARTENGO AP 0.0
BLUE_ARTENGO AP 0.0
RED_KUIKMA AP 0.0
mAP 0.0

Epoch 41/80
36/36 [==============================] - 27s 756ms/step - loss: 11.0496

Epoch 00041: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_041.tlt
Epoch 42/80
36/36 [==============================] - 27s 753ms/step - loss: 10.9886

Epoch 00042: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_042.tlt
Epoch 43/80
36/36 [==============================] - 27s 754ms/step - loss: 10.9284

Epoch 00043: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_043.tlt
Epoch 44/80
36/36 [==============================] - 27s 755ms/step - loss: 10.8689

Epoch 00044: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_044.tlt
Epoch 45/80
36/36 [==============================] - 27s 753ms/step - loss: 10.8102

Epoch 00045: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_045.tlt
Epoch 46/80
36/36 [==============================] - 27s 754ms/step - loss: 10.7521

Epoch 00046: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_046.tlt
Epoch 47/80
36/36 [==============================] - 27s 752ms/step - loss: 10.6948

Epoch 00047: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_047.tlt
Epoch 48/80
36/36 [==============================] - 27s 752ms/step - loss: 10.6382

Epoch 00048: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_048.tlt
Epoch 49/80
36/36 [==============================] - 27s 753ms/step - loss: 10.5824

Epoch 00049: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_049.tlt
Epoch 50/80
36/36 [==============================] - 27s 754ms/step - loss: 10.5272

Epoch 00050: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_050.tlt
Number of images in the evaluation dataset: 46

Producing predictions: 100%|██████████████████████| 6/6 [00:05<00:00, 1.18it/s]
Start multi-thread per-image matching
Start to calculate AP for each class

BLACK_ARTENGO AP 0.0
BLUE_ARTENGO AP 0.0
RED_KUIKMA AP 0.0
mAP 0.0

Epoch 51/80
36/36 [==============================] - 27s 753ms/step - loss: 10.4727

Epoch 00051: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_051.tlt
Epoch 52/80
36/36 [==============================] - 27s 750ms/step - loss: 10.4189

Epoch 00052: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_052.tlt
Epoch 53/80
36/36 [==============================] - 27s 752ms/step - loss: 10.3659

Epoch 00053: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_053.tlt
Epoch 54/80
36/36 [==============================] - 27s 752ms/step - loss: 10.3134

Epoch 00054: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_054.tlt
Epoch 55/80
36/36 [==============================] - 27s 752ms/step - loss: 10.2617

Epoch 00055: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_055.tlt
Epoch 56/80
36/36 [==============================] - 27s 751ms/step - loss: 10.2106

Epoch 00056: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_056.tlt
Epoch 57/80
36/36 [==============================] - 27s 751ms/step - loss: 10.1603

Epoch 00057: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_057.tlt
Epoch 58/80
36/36 [==============================] - 27s 750ms/step - loss: 10.1105

Epoch 00058: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_058.tlt
Epoch 59/80
36/36 [==============================] - 27s 751ms/step - loss: 10.0614

Epoch 00059: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_059.tlt
Epoch 60/80
36/36 [==============================] - 27s 751ms/step - loss: 10.0130

Epoch 00060: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_060.tlt
Number of images in the evaluation dataset: 46

Producing predictions: 100%|██████████████████████| 6/6 [00:05<00:00, 1.16it/s]
Start multi-thread per-image matching
Start to calculate AP for each class

BLACK_ARTENGO AP 0.0
BLUE_ARTENGO AP 0.0
RED_KUIKMA AP 0.0
mAP 0.0

Epoch 61/80
36/36 [==============================] - 27s 751ms/step - loss: 9.9652

Epoch 00061: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_061.tlt
Epoch 62/80
36/36 [==============================] - 27s 753ms/step - loss: 9.9180

Epoch 00062: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_062.tlt
Epoch 63/80
36/36 [==============================] - 27s 751ms/step - loss: 9.8714

Epoch 00063: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_063.tlt
Epoch 64/80

36/36 [==============================] - 27s 751ms/step - loss: 9.8255

Epoch 00064: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_064.tlt
Epoch 65/80
36/36 [==============================] - 27s 750ms/step - loss: 9.7814

Epoch 00065: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_065.tlt
Epoch 66/80
36/36 [==============================] - 27s 751ms/step - loss: 9.7469

Epoch 00066: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_066.tlt
Epoch 67/80
36/36 [==============================] - 27s 751ms/step - loss: 9.7232

Epoch 00067: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_067.tlt
Epoch 68/80
36/36 [==============================] - 27s 752ms/step - loss: 9.7070

Epoch 00068: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_068.tlt
Epoch 69/80
36/36 [==============================] - 27s 750ms/step - loss: 9.6959

Epoch 00069: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_069.tlt
Epoch 70/80
36/36 [==============================] - 27s 751ms/step - loss: 9.6882

Epoch 00070: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_070.tlt
Number of images in the evaluation dataset: 46

Producing predictions: 100%|██████████████████████| 6/6 [00:05<00:00, 1.18it/s]
Start multi-thread per-image matching
Start to calculate AP for each class

BLACK_ARTENGO AP 0.0
BLUE_ARTENGO AP 0.0
RED_KUIKMA AP 0.0
mAP 0.0

Epoch 71/80
36/36 [==============================] - 27s 752ms/step - loss: 9.6830

Epoch 00071: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_071.tlt
Epoch 72/80
36/36 [==============================] - 27s 754ms/step - loss: 9.6794

Epoch 00072: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_072.tlt
Epoch 73/80
36/36 [==============================] - 27s 754ms/step - loss: 9.6770

Epoch 00073: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_073.tlt
Epoch 74/80
36/36 [==============================] - 27s 754ms/step - loss: 9.6753

Epoch 00074: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_074.tlt
Epoch 75/80
36/36 [==============================] - 27s 753ms/step - loss: 9.6741

Epoch 00075: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_075.tlt
Epoch 76/80
36/36 [==============================] - 27s 755ms/step - loss: 9.6733

Epoch 00076: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_076.tlt
Epoch 77/80
36/36 [==============================] - 27s 754ms/step - loss: 9.6728

Epoch 00077: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_077.tlt
Epoch 78/80
36/36 [==============================] - 27s 753ms/step - loss: 9.6724

Epoch 00078: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_078.tlt
Epoch 79/80
36/36 [==============================] - 27s 753ms/step - loss: 9.6721

Epoch 00079: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_079.tlt
Epoch 80/80
36/36 [==============================] - 27s 754ms/step - loss: 9.6719

Epoch 00080: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_080.tlt
Number of images in the evaluation dataset: 46

Producing predictions: 100%|██████████████████████| 6/6 [00:05<00:00, 1.08it/s]
Start multi-thread per-image matching
Start to calculate AP for each class

BLACK_ARTENGO AP 0.0
BLUE_ARTENGO AP 0.0
RED_KUIKMA AP 0.0
mAP 0.0

the training with the example zip files works, but not when i change the settings so to use my own pictures and annotations.

thank you for your help!
br, Martin

beefshepherd · August 26, 2020, 7:03pm

yes same stuff is happening with me, but my precision for a particular class is increasing among three classes but the other two classes remain zero, this has been the same when changing freeze blocks of resnet to 0, 1, 2, 3. Which is bonkers

Morganh · August 27, 2020, 5:57am

@martin1
Please resize training images/labels to meet below requirement offline.
910 is not multiples of 32.

SSD

Input size: C * W * H (where C = 1 or 3, W >= 128, H >= 128, W, H are multiples of 32)
Image format: JPG, JPEG, PNG
Label format: KITTI detection

Note: The tlt-train tool does not support training on images of multiple resolutions, or resizing images during training. All of the images must be resized offline to the final training size and the corresponding bounding boxes must be scaled accordingly.

martin1 · August 27, 2020, 7:56pm

Hello Morganh
I have changed now the pictures to:
width: 1280
height: 640
which should work as multiple of 32… and also annotate all pictures again.
but still i get AP:0 for my classes.

here an example of 1 kitti format file of all 3 classes:

BLUE_ARTENGO 0.0 0 0.0 156 268 310 428 0.0 0.0 0.0 0.0 0.0 0.0 0.0
RED_KUIKMA 0.0 0 0.0 413 162 586 364 0.0 0.0 0.0 0.0 0.0 0.0 0.0
BLACK_ARTENGO 0.0 0 0.0 793 218 1007 388 0.0 0.0 0.0 0.0 0.0 0.0 0.0

the config changes in tfrecords are the path and changes from png to jpeg:

ssd_tfrecords_kitti_trainval.txt

kitti_config {
root_directory_path: “/workspace/tlt-experiments/data/rackets_v05_tlt_H32_1280_640/training”
image_dir_name: “images”
label_dir_name: “labels”
image_extension: “.jpeg”
partition_mode: “random”
num_partitions: 2
val_split: 14
num_shards: 10
}
image_directory_path: “/workspace/tlt-experiments/data/rackets_v05_tlt_H32_1280_640/training”

in training config:

ssd_train_resnet18_kitti.txt

random_seed: 42
ssd_config {
aspect_ratios_global: “[1.0, 2.0, 0.5, 3.0, 1.0/3.0]”
scales: “[0.05, 0.1, 0.25, 0.4, 0.55, 0.7, 0.85]”
two_boxes_for_ar1: true
clip_boxes: false
loss_loc_weight: 0.8
focal_loss_alpha: 0.25
focal_loss_gamma: 2.0
variances: “[0.1, 0.1, 0.2, 0.2]”
arch: “resnet”
nlayers: 18
freeze_bn: false
freeze_blocks: 0
}
training_config {
batch_size_per_gpu: 8
num_epochs: 80
enable_qat: false
learning_rate {
soft_start_annealing_schedule {
min_learning_rate: 5e-5
max_learning_rate: 2e-2
soft_start: 0.15
annealing: 0.8
}
}
regularizer {
type: L1
weight: 3e-5
}
}
eval_config {
validation_period_during_training: 10
average_precision_mode: SAMPLE
batch_size: 8
matching_iou_threshold: 0.5
}
nms_config {
confidence_threshold: 0.01
clustering_iou_threshold: 0.6
top_k: 200
}
augmentation_config {
preprocessing {
output_image_width: 1280
output_image_height: 640
output_image_channel: 3
crop_right: 1280
crop_bottom: 640
min_bbox_width: 1.0
min_bbox_height: 1.0
}
spatial_augmentation {
hflip_probability: 0.5
vflip_probability: 0.0
zoom_min: 0.7
zoom_max: 1.8
translate_max_x: 8.0
translate_max_y: 8.0
}
color_augmentation {
hue_rotation_max: 25.0
saturation_shift_max: 0.20000000298
contrast_scale_max: 0.10000000149
contrast_center: 0.5
}
}
dataset_config {
data_sources: {
tfrecords_path: “/workspace/tlt-experiments/data/rackets_v05_tlt_H32_1280_640/tfrecords/kitti_trainval/kitti_trainval*”
image_directory_path: “/workspace/tlt-experiments/data/rackets_v05_tlt_H32_1280_640/training”
}
image_extension: “jpeg”
target_class_mapping {
key: “BLACK_ARTENGO”
value: “BLACK_ARTENGO”
}
target_class_mapping {
key: “BLUE_ARTENGO”
value: “BLUE_ARTENGO”
}
target_class_mapping {
key: “RED_KUIKMA”
value: “RED_KUIKMA”
}
validation_fold: 0
}

training result still shows AP: 0 by running:

!tlt-train ssd -e $SPECS_DIR/my_ssd_train_resnet18_kitti.txt \
           -r $USER_EXPERIMENT_DIR/experiment_dir_unpruned \
           -k $KEY \
           -m $USER_EXPERIMENT_DIR/pretrained_resnet18/tlt_pretrained_object_detection_vresnet18/resnet_18.hdf5 \
           --gpus 1

Summary

To run with multigpu, please change --gpus based on the number of available GPUs in your machine.
Using TensorFlow backend.
2020-08-27 17:36:28.902156: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-08-27 17:36:33.055142: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-08-27 17:36:33.089104: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: Quadro M4000 major: 5 minor: 2 memoryClockRate(GHz): 0.7725
pciBusID: 0000:03:00.0
2020-08-27 17:36:33.089181: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-08-27 17:36:33.120573: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-08-27 17:36:33.135066: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-08-27 17:36:33.139904: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-08-27 17:36:33.174331: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-08-27 17:36:33.196770: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-08-27 17:36:33.260453: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-08-27 17:36:33.261738: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2020-08-27 17:36:33.262163: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-08-27 17:36:34.650532: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-08-27 17:36:34.650611: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0
2020-08-27 17:36:34.650624: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N
2020-08-27 17:36:34.652679: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6392 MB memory) → physical GPU (device: 0, name: Quadro M4000, pci bus id: 0000:03:00.0, compute capability: 5.2)
2020-08-27 17:36:34,659 [INFO] /usr/local/lib/python3.6/dist-packages/iva/ssd/utils/spec_loader.pyc: Merging specification from /workspace/examples/ssd/specs/my_ssd_train_resnet18_kitti.txt
2020-08-27 17:36:34,665 [INFO] iva.ssd.scripts.train: Loading pretrained weights. This may take a while…
2020-08-27 17:36:34,819 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Serial augmentation enabled = False
2020-08-27 17:36:34,819 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Pseudo sharding enabled = False
2020-08-27 17:36:34,820 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Max Image Dimensions (all sources): (0, 0)
2020-08-27 17:36:34,820 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: number of cpus: 56, io threads: 112, compute threads: 56, buffered batches: 4
2020-08-27 17:36:34,820 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: total dataset size 288, number of sources: 1, batch size per gpu: 8, steps: 36
2020-08-27 17:36:35,002 [INFO] iva.detectnet_v2.dataloader.default_dataloader: Bounding box coordinates were detected in the input specification! Bboxes will be automatically converted to polygon coordinates.
2020-08-27 17:36:35.056342: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: Quadro M4000 major: 5 minor: 2 memoryClockRate(GHz): 0.7725
pciBusID: 0000:03:00.0
2020-08-27 17:36:35.056393: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-08-27 17:36:35.056441: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-08-27 17:36:35.056480: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-08-27 17:36:35.056517: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-08-27 17:36:35.056554: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-08-27 17:36:35.056598: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-08-27 17:36:35.056635: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-08-27 17:36:35.057641: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2020-08-27 17:36:35,398 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: shuffle: True - shard 0 of 1
2020-08-27 17:36:35,407 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: sampling 1 datasets with weights:
2020-08-27 17:36:35,407 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: source: 0 weight: 1.000000
2020-08-27 17:37:35,755 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Serial augmentation enabled = False
2020-08-27 17:37:35,755 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Pseudo sharding enabled = False
2020-08-27 17:37:35,755 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Max Image Dimensions (all sources): (0, 0)
2020-08-27 17:37:35,756 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: number of cpus: 56, io threads: 112, compute threads: 56, buffered batches: 4
2020-08-27 17:37:35,756 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: total dataset size 46, number of sources: 1, batch size per gpu: 8, steps: 6
2020-08-27 17:37:35,822 [INFO] iva.detectnet_v2.dataloader.default_dataloader: Bounding box coordinates were detected in the input specification! Bboxes will be automatically converted to polygon coordinates.
2020-08-27 17:37:36,181 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: shuffle: False - shard 0 of 1
2020-08-27 17:37:36,190 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: sampling 1 datasets with weights:
2020-08-27 17:37:36,190 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: source: 0 weight: 1.000000

Layer (type) Output Shape Param # Connected to

Input (InputLayer) (8, 3, 640, 1280) 0

conv1 (Conv2D) (8, 64, 320, 640) 9408 Input[0][0]

bn_conv1 (BatchNormalization) (8, 64, 320, 640) 256 conv1[0][0]

activation_1 (Activation) (8, 64, 320, 640) 0 bn_conv1[0][0]

block_1a_conv_1 (Conv2D) (8, 64, 160, 320) 36864 activation_1[0][0]

block_1a_bn_1 (BatchNormalizati (8, 64, 160, 320) 256 block_1a_conv_1[0][0]

block_1a_relu_1 (Activation) (8, 64, 160, 320) 0 block_1a_bn_1[0][0]

block_1a_conv_2 (Conv2D) (8, 64, 160, 320) 36864 block_1a_relu_1[0][0]

block_1a_conv_shortcut (Conv2D) (8, 64, 160, 320) 4096 activation_1[0][0]

block_1a_bn_2 (BatchNormalizati (8, 64, 160, 320) 256 block_1a_conv_2[0][0]

block_1a_bn_shortcut (BatchNorm (8, 64, 160, 320) 256 block_1a_conv_shortcut[0][0]

add_1 (Add) (8, 64, 160, 320) 0 block_1a_bn_2[0][0]
block_1a_bn_shortcut[0][0]

block_1a_relu (Activation) (8, 64, 160, 320) 0 add_1[0][0]

block_1b_conv_1 (Conv2D) (8, 64, 160, 320) 36864 block_1a_relu[0][0]

block_1b_bn_1 (BatchNormalizati (8, 64, 160, 320) 256 block_1b_conv_1[0][0]

block_1b_relu_1 (Activation) (8, 64, 160, 320) 0 block_1b_bn_1[0][0]

block_1b_conv_2 (Conv2D) (8, 64, 160, 320) 36864 block_1b_relu_1[0][0]

block_1b_conv_shortcut (Conv2D) (8, 64, 160, 320) 4096 block_1a_relu[0][0]

block_1b_bn_2 (BatchNormalizati (8, 64, 160, 320) 256 block_1b_conv_2[0][0]

block_1b_bn_shortcut (BatchNorm (8, 64, 160, 320) 256 block_1b_conv_shortcut[0][0]

add_2 (Add) (8, 64, 160, 320) 0 block_1b_bn_2[0][0]
block_1b_bn_shortcut[0][0]

block_1b_relu (Activation) (8, 64, 160, 320) 0 add_2[0][0]

block_2a_conv_1 (Conv2D) (8, 128, 80, 160) 73728 block_1b_relu[0][0]

block_2a_bn_1 (BatchNormalizati (8, 128, 80, 160) 512 block_2a_conv_1[0][0]

block_2a_relu_1 (Activation) (8, 128, 80, 160) 0 block_2a_bn_1[0][0]

block_2a_conv_2 (Conv2D) (8, 128, 80, 160) 147456 block_2a_relu_1[0][0]

block_2a_conv_shortcut (Conv2D) (8, 128, 80, 160) 8192 block_1b_relu[0][0]

block_2a_bn_2 (BatchNormalizati (8, 128, 80, 160) 512 block_2a_conv_2[0][0]

block_2a_bn_shortcut (BatchNorm (8, 128, 80, 160) 512 block_2a_conv_shortcut[0][0]

add_3 (Add) (8, 128, 80, 160) 0 block_2a_bn_2[0][0]
block_2a_bn_shortcut[0][0]

block_2a_relu (Activation) (8, 128, 80, 160) 0 add_3[0][0]

block_2b_conv_1 (Conv2D) (8, 128, 80, 160) 147456 block_2a_relu[0][0]

block_2b_bn_1 (BatchNormalizati (8, 128, 80, 160) 512 block_2b_conv_1[0][0]

block_2b_relu_1 (Activation) (8, 128, 80, 160) 0 block_2b_bn_1[0][0]

block_2b_conv_2 (Conv2D) (8, 128, 80, 160) 147456 block_2b_relu_1[0][0]

block_2b_conv_shortcut (Conv2D) (8, 128, 80, 160) 16384 block_2a_relu[0][0]

block_2b_bn_2 (BatchNormalizati (8, 128, 80, 160) 512 block_2b_conv_2[0][0]

block_2b_bn_shortcut (BatchNorm (8, 128, 80, 160) 512 block_2b_conv_shortcut[0][0]

add_4 (Add) (8, 128, 80, 160) 0 block_2b_bn_2[0][0]
block_2b_bn_shortcut[0][0]

block_2b_relu (Activation) (8, 128, 80, 160) 0 add_4[0][0]

block_3a_conv_1 (Conv2D) (8, 256, 40, 80) 294912 block_2b_relu[0][0]

block_3a_bn_1 (BatchNormalizati (8, 256, 40, 80) 1024 block_3a_conv_1[0][0]

block_3a_relu_1 (Activation) (8, 256, 40, 80) 0 block_3a_bn_1[0][0]

block_3a_conv_2 (Conv2D) (8, 256, 40, 80) 589824 block_3a_relu_1[0][0]

block_3a_conv_shortcut (Conv2D) (8, 256, 40, 80) 32768 block_2b_relu[0][0]

block_3a_bn_2 (BatchNormalizati (8, 256, 40, 80) 1024 block_3a_conv_2[0][0]

block_3a_bn_shortcut (BatchNorm (8, 256, 40, 80) 1024 block_3a_conv_shortcut[0][0]

add_5 (Add) (8, 256, 40, 80) 0 block_3a_bn_2[0][0]
block_3a_bn_shortcut[0][0]

block_3a_relu (Activation) (8, 256, 40, 80) 0 add_5[0][0]

block_3b_conv_1 (Conv2D) (8, 256, 40, 80) 589824 block_3a_relu[0][0]

block_3b_bn_1 (BatchNormalizati (8, 256, 40, 80) 1024 block_3b_conv_1[0][0]

block_3b_relu_1 (Activation) (8, 256, 40, 80) 0 block_3b_bn_1[0][0]

block_3b_conv_2 (Conv2D) (8, 256, 40, 80) 589824 block_3b_relu_1[0][0]

block_3b_conv_shortcut (Conv2D) (8, 256, 40, 80) 65536 block_3a_relu[0][0]

block_3b_bn_2 (BatchNormalizati (8, 256, 40, 80) 1024 block_3b_conv_2[0][0]

block_3b_bn_shortcut (BatchNorm (8, 256, 40, 80) 1024 block_3b_conv_shortcut[0][0]

add_6 (Add) (8, 256, 40, 80) 0 block_3b_bn_2[0][0]
block_3b_bn_shortcut[0][0]

block_3b_relu (Activation) (8, 256, 40, 80) 0 add_6[0][0]

block_4a_conv_1 (Conv2D) (8, 512, 40, 80) 1179648 block_3b_relu[0][0]

block_4a_bn_1 (BatchNormalizati (8, 512, 40, 80) 2048 block_4a_conv_1[0][0]

block_4a_relu_1 (Activation) (8, 512, 40, 80) 0 block_4a_bn_1[0][0]

block_4a_conv_2 (Conv2D) (8, 512, 40, 80) 2359296 block_4a_relu_1[0][0]

block_4a_conv_shortcut (Conv2D) (8, 512, 40, 80) 131072 block_3b_relu[0][0]

block_4a_bn_2 (BatchNormalizati (8, 512, 40, 80) 2048 block_4a_conv_2[0][0]

block_4a_bn_shortcut (BatchNorm (8, 512, 40, 80) 2048 block_4a_conv_shortcut[0][0]

add_7 (Add) (8, 512, 40, 80) 0 block_4a_bn_2[0][0]
block_4a_bn_shortcut[0][0]

block_4a_relu (Activation) (8, 512, 40, 80) 0 add_7[0][0]

block_4b_conv_1 (Conv2D) (8, 512, 40, 80) 2359296 block_4a_relu[0][0]

block_4b_bn_1 (BatchNormalizati (8, 512, 40, 80) 2048 block_4b_conv_1[0][0]

block_4b_relu_1 (Activation) (8, 512, 40, 80) 0 block_4b_bn_1[0][0]

block_4b_conv_2 (Conv2D) (8, 512, 40, 80) 2359296 block_4b_relu_1[0][0]

block_4b_conv_shortcut (Conv2D) (8, 512, 40, 80) 262144 block_4a_relu[0][0]

block_4b_bn_2 (BatchNormalizati (8, 512, 40, 80) 2048 block_4b_conv_2[0][0]

block_4b_bn_shortcut (BatchNorm (8, 512, 40, 80) 2048 block_4b_conv_shortcut[0][0]

add_8 (Add) (8, 512, 40, 80) 0 block_4b_bn_2[0][0]
block_4b_bn_shortcut[0][0]

block_4b_relu (Activation) (8, 512, 40, 80) 0 add_8[0][0]

ssd_expand_block_0_conv_0 (Conv (8, 256, 40, 80) 131328 block_4b_relu[0][0]

ssd_expand_block_0_relu_0 (ReLU (8, 256, 40, 80) 0 ssd_expand_block_0_conv_0[0][0]

ssd_expand_block_0_conv_1 (Conv (8, 256, 40, 80) 589824 ssd_expand_block_0_relu_0[0][0]

ssd_expand_block_0_bn_1 (BatchN (8, 256, 40, 80) 1024 ssd_expand_block_0_conv_1[0][0]

ssd_expand_block_0_relu_1 (ReLU (8, 256, 40, 80) 0 ssd_expand_block_0_bn_1[0][0]

ssd_expand_block_1_conv_0 (Conv (8, 128, 40, 80) 32896 ssd_expand_block_0_relu_1[0][0]

ssd_expand_block_1_relu_0 (ReLU (8, 128, 40, 80) 0 ssd_expand_block_1_conv_0[0][0]

ssd_expand_block_1_conv_1 (Conv (8, 256, 20, 40) 294912 ssd_expand_block_1_relu_0[0][0]

ssd_expand_block_1_bn_1 (BatchN (8, 256, 20, 40) 1024 ssd_expand_block_1_conv_1[0][0]

ssd_expand_block_1_relu_1 (ReLU (8, 256, 20, 40) 0 ssd_expand_block_1_bn_1[0][0]

ssd_expand_block_2_conv_0 (Conv (8, 64, 20, 40) 16448 ssd_expand_block_1_relu_1[0][0]

ssd_expand_block_2_relu_0 (ReLU (8, 64, 20, 40) 0 ssd_expand_block_2_conv_0[0][0]

ssd_expand_block_2_conv_1 (Conv (8, 128, 10, 20) 73728 ssd_expand_block_2_relu_0[0][0]

ssd_expand_block_2_bn_1 (BatchN (8, 128, 10, 20) 512 ssd_expand_block_2_conv_1[0][0]

ssd_expand_block_2_relu_1 (ReLU (8, 128, 10, 20) 0 ssd_expand_block_2_bn_1[0][0]

ssd_expand_block_3_conv_0 (Conv (8, 64, 10, 20) 8256 ssd_expand_block_2_relu_1[0][0]

ssd_expand_block_3_relu_0 (ReLU (8, 64, 10, 20) 0 ssd_expand_block_3_conv_0[0][0]

ssd_expand_block_3_conv_1 (Conv (8, 128, 5, 10) 73728 ssd_expand_block_3_relu_0[0][0]

ssd_expand_block_3_bn_1 (BatchN (8, 128, 5, 10) 512 ssd_expand_block_3_conv_1[0][0]

ssd_expand_block_3_relu_1 (ReLU (8, 128, 5, 10) 0 ssd_expand_block_3_bn_1[0][0]

ssd_expand_block_4_conv_0 (Conv (8, 64, 5, 10) 8256 ssd_expand_block_3_relu_1[0][0]

ssd_expand_block_4_relu_0 (ReLU (8, 64, 5, 10) 0 ssd_expand_block_4_conv_0[0][0]

ssd_expand_block_4_conv_1 (Conv (8, 128, 3, 5) 73728 ssd_expand_block_4_relu_0[0][0]

ssd_expand_block_4_bn_1 (BatchN (8, 128, 3, 5) 512 ssd_expand_block_4_conv_1[0][0]

ssd_expand_block_4_relu_1 (ReLU (8, 128, 3, 5) 0 ssd_expand_block_4_bn_1[0][0]

ssd_conf_0 (Conv2D) (8, 18, 80, 160) 20754 block_2b_relu[0][0]

ssd_conf_1 (Conv2D) (8, 18, 40, 80) 41490 ssd_expand_block_0_relu_1[0][0]

ssd_conf_2 (Conv2D) (8, 18, 20, 40) 41490 ssd_expand_block_1_relu_1[0][0]

ssd_conf_3 (Conv2D) (8, 18, 10, 20) 20754 ssd_expand_block_2_relu_1[0][0]

ssd_conf_4 (Conv2D) (8, 18, 5, 10) 20754 ssd_expand_block_3_relu_1[0][0]

ssd_conf_5 (Conv2D) (8, 18, 3, 5) 20754 ssd_expand_block_4_relu_1[0][0]

permute_1 (Permute) (8, 80, 160, 18) 0 ssd_conf_0[0][0]

permute_2 (Permute) (8, 40, 80, 18) 0 ssd_conf_1[0][0]

permute_3 (Permute) (8, 20, 40, 18) 0 ssd_conf_2[0][0]

permute_4 (Permute) (8, 10, 20, 18) 0 ssd_conf_3[0][0]

permute_5 (Permute) (8, 5, 10, 18) 0 ssd_conf_4[0][0]

permute_6 (Permute) (8, 3, 5, 18) 0 ssd_conf_5[0][0]

ssd_loc_0 (Conv2D) (8, 24, 80, 160) 27672 block_2b_relu[0][0]

ssd_loc_1 (Conv2D) (8, 24, 40, 80) 55320 ssd_expand_block_0_relu_1[0][0]

ssd_loc_2 (Conv2D) (8, 24, 20, 40) 55320 ssd_expand_block_1_relu_1[0][0]

ssd_loc_3 (Conv2D) (8, 24, 10, 20) 27672 ssd_expand_block_2_relu_1[0][0]

ssd_loc_4 (Conv2D) (8, 24, 5, 10) 27672 ssd_expand_block_3_relu_1[0][0]

ssd_loc_5 (Conv2D) (8, 24, 3, 5) 27672 ssd_expand_block_4_relu_1[0][0]

conf_reshape_0 (Reshape) (8, 76800, 1, 3) 0 permute_1[0][0]

conf_reshape_1 (Reshape) (8, 19200, 1, 3) 0 permute_2[0][0]

conf_reshape_2 (Reshape) (8, 4800, 1, 3) 0 permute_3[0][0]

conf_reshape_3 (Reshape) (8, 1200, 1, 3) 0 permute_4[0][0]

conf_reshape_4 (Reshape) (8, 300, 1, 3) 0 permute_5[0][0]

conf_reshape_5 (Reshape) (8, 90, 1, 3) 0 permute_6[0][0]

permute_7 (Permute) (8, 80, 160, 24) 0 ssd_loc_0[0][0]

permute_8 (Permute) (8, 40, 80, 24) 0 ssd_loc_1[0][0]

permute_9 (Permute) (8, 20, 40, 24) 0 ssd_loc_2[0][0]

permute_10 (Permute) (8, 10, 20, 24) 0 ssd_loc_3[0][0]

permute_11 (Permute) (8, 5, 10, 24) 0 ssd_loc_4[0][0]

permute_12 (Permute) (8, 3, 5, 24) 0 ssd_loc_5[0][0]

ssd_anchor_0 (AnchorBoxes) (8, 12800, 6, 8) 0 ssd_loc_0[0][0]

ssd_anchor_1 (AnchorBoxes) (8, 3200, 6, 8) 0 ssd_loc_1[0][0]

ssd_anchor_2 (AnchorBoxes) (8, 800, 6, 8) 0 ssd_loc_2[0][0]

ssd_anchor_3 (AnchorBoxes) (8, 200, 6, 8) 0 ssd_loc_3[0][0]

ssd_anchor_4 (AnchorBoxes) (8, 50, 6, 8) 0 ssd_loc_4[0][0]

ssd_anchor_5 (AnchorBoxes) (8, 15, 6, 8) 0 ssd_loc_5[0][0]

mbox_conf (Concatenate) (8, 102390, 1, 3) 0 conf_reshape_0[0][0]
conf_reshape_1[0][0]
conf_reshape_2[0][0]
conf_reshape_3[0][0]
conf_reshape_4[0][0]
conf_reshape_5[0][0]

loc_reshape_0 (Reshape) (8, 76800, 1, 4) 0 permute_7[0][0]

loc_reshape_1 (Reshape) (8, 19200, 1, 4) 0 permute_8[0][0]

loc_reshape_2 (Reshape) (8, 4800, 1, 4) 0 permute_9[0][0]

loc_reshape_3 (Reshape) (8, 1200, 1, 4) 0 permute_10[0][0]

loc_reshape_4 (Reshape) (8, 300, 1, 4) 0 permute_11[0][0]

loc_reshape_5 (Reshape) (8, 90, 1, 4) 0 permute_12[0][0]

anchor_reshape_0 (Reshape) (8, 76800, 1, 8) 0 ssd_anchor_0[0][0]

anchor_reshape_1 (Reshape) (8, 19200, 1, 8) 0 ssd_anchor_1[0][0]

anchor_reshape_2 (Reshape) (8, 4800, 1, 8) 0 ssd_anchor_2[0][0]

anchor_reshape_3 (Reshape) (8, 1200, 1, 8) 0 ssd_anchor_3[0][0]

anchor_reshape_4 (Reshape) (8, 300, 1, 8) 0 ssd_anchor_4[0][0]

anchor_reshape_5 (Reshape) (8, 90, 1, 8) 0 ssd_anchor_5[0][0]

mbox_conf_sigmoid (Activation) (8, 102390, 1, 3) 0 mbox_conf[0][0]

mbox_loc (Concatenate) (8, 102390, 1, 4) 0 loc_reshape_0[0][0]
loc_reshape_1[0][0]
loc_reshape_2[0][0]
loc_reshape_3[0][0]
loc_reshape_4[0][0]
loc_reshape_5[0][0]

mbox_priorbox (Concatenate) (8, 102390, 1, 8) 0 anchor_reshape_0[0][0]
anchor_reshape_1[0][0]
anchor_reshape_2[0][0]
anchor_reshape_3[0][0]
anchor_reshape_4[0][0]
anchor_reshape_5[0][0]

concatenate_1 (Concatenate) (8, 102390, 1, 15) 0 mbox_conf_sigmoid[0][0]
mbox_loc[0][0]
mbox_priorbox[0][0]

ssd_predictions (Reshape) (8, 102390, 15) 0 concatenate_1[0][0]

Total params: 13,236,476
Trainable params: 13,213,628
Non-trainable params: 22,848

2020-08-27 17:37:41,877 [INFO] iva.ssd.scripts.train: Number of images in the training dataset: 288
2020-08-27 17:37:41,877 [INFO] iva.ssd.scripts.train: Number of images in the validation dataset: 46

Epoch 1/80
2020-08-27 17:37:52.940753: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-08-27 17:37:54.104420: I tensorflow/core/kernels/cuda_solvers.cc:159] Creating CudaSolver handles for stream 0x14638440
2020-08-27 17:37:54.104939: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-08-27 17:37:54.617129: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-08-27 17:37:54.643935: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-08-27 17:37:59.114670: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.62GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2020-08-27 17:37:59.167529: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.54GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2020-08-27 17:37:59.405948: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.22GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2020-08-27 17:37:59.458102: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.16GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2020-08-27 17:38:00.163031: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.14GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
36/36 [==============================] - 65s 2s/step - loss: 30916.7681

Epoch 00001: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_001.tlt
Epoch 2/80
36/36 [==============================] - 46s 1s/step - loss: 19.4986

Epoch 00002: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_002.tlt
Epoch 3/80
36/36 [==============================] - 46s 1s/step - loss: 19.5680

Epoch 00003: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_003.tlt
Epoch 4/80
36/36 [==============================] - 46s 1s/step - loss: 19.5587

Epoch 00004: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_004.tlt
Epoch 5/80
36/36 [==============================] - 46s 1s/step - loss: 19.5492

Epoch 00005: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_005.tlt
Epoch 6/80
36/36 [==============================] - 46s 1s/step - loss: 19.5426

Epoch 00006: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_006.tlt
Epoch 7/80
36/36 [==============================] - 46s 1s/step - loss: 19.5319

Epoch 00007: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_007.tlt
Epoch 8/80
36/36 [==============================] - 46s 1s/step - loss: 19.5215

Epoch 00008: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_008.tlt
Epoch 9/80
36/36 [==============================] - 46s 1s/step - loss: 19.5063

Epoch 00009: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_009.tlt
Epoch 10/80
36/36 [==============================] - 45s 1s/step - loss: 19.4872

Epoch 00010: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_010.tlt
Number of images in the evaluation dataset: 46

Producing predictions: 100%|██████████████████████| 6/6 [00:07<00:00, 1.27s/it]
Start multi-thread per-image matching
Start to calculate AP for each class

BLACK_ARTENGO AP 0.0
BLUE_ARTENGO AP 0.0
RED_KUIKMA AP 0.0
mAP 0.0

Epoch 11/80
36/36 [==============================] - 45s 1s/step - loss: 19.4580

Epoch 00011: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_011.tlt
Epoch 12/80
36/36 [==============================] - 45s 1s/step - loss: 19.4107

Epoch 00012: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_012.tlt
Epoch 13/80
36/36 [==============================] - 45s 1s/step - loss: 19.3388

Epoch 00013: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_013.tlt
Epoch 14/80
36/36 [==============================] - 45s 1s/step - loss: 19.2554

Epoch 00014: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_014.tlt
Epoch 15/80
36/36 [==============================] - 46s 1s/step - loss: 19.1727

Epoch 00015: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_015.tlt
Epoch 16/80
36/36 [==============================] - 45s 1s/step - loss: 19.0907

Epoch 00016: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_016.tlt
Epoch 17/80
36/36 [==============================] - 46s 1s/step - loss: 19.0096

Epoch 00017: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_017.tlt
Epoch 18/80
36/36 [==============================] - 45s 1s/step - loss: 18.9293

Epoch 00018: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_018.tlt
Epoch 19/80
36/36 [==============================] - 45s 1s/step - loss: 18.8498

Epoch 00019: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_019.tlt
Epoch 20/80
36/36 [==============================] - 45s 1s/step - loss: 18.7711

Epoch 00020: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_020.tlt
Number of images in the evaluation dataset: 46

Producing predictions: 100%|██████████████████████| 6/6 [00:06<00:00, 1.06s/it]
Start multi-thread per-image matching
Start to calculate AP for each class

BLACK_ARTENGO AP 0.0
BLUE_ARTENGO AP 0.0
RED_KUIKMA AP 0.0
mAP 0.0

Epoch 21/80
36/36 [==============================] - 46s 1s/step - loss: 18.6932

Epoch 00021: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_021.tlt
Epoch 22/80
36/36 [==============================] - 45s 1s/step - loss: 18.6161

Epoch 00022: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_022.tlt
Epoch 23/80
36/36 [==============================] - 45s 1s/step - loss: 18.5397

Epoch 00023: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_023.tlt
Epoch 24/80
36/36 [==============================] - 45s 1s/step - loss: 18.4641

Epoch 00024: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_024.tlt
Epoch 25/80
36/36 [==============================] - 45s 1s/step - loss: 18.3893

Epoch 00025: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_025.tlt
Epoch 26/80
36/36 [==============================] - 45s 1s/step - loss: 18.3152

Epoch 00026: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_026.tlt
Epoch 27/80

36/36 [==============================] - 45s 1s/step - loss: 18.2420

Epoch 00027: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_027.tlt
Epoch 28/80
36/36 [==============================] - 45s 1s/step - loss: 18.1694

Epoch 00028: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_028.tlt
Epoch 29/80
36/36 [==============================] - 45s 1s/step - loss: 18.0977

Epoch 00029: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_029.tlt
Epoch 30/80
36/36 [==============================] - 45s 1s/step - loss: 18.0267

Epoch 00030: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_030.tlt
Number of images in the evaluation dataset: 46

Producing predictions: 100%|██████████████████████| 6/6 [00:06<00:00, 1.08s/it]
Start multi-thread per-image matching
Start to calculate AP for each class

BLACK_ARTENGO AP 0.0
BLUE_ARTENGO AP 0.0
RED_KUIKMA AP 0.0
mAP 0.0

Epoch 31/80
36/36 [==============================] - 45s 1s/step - loss: 17.9565

Epoch 00031: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_031.tlt
Epoch 32/80
36/36 [==============================] - 45s 1s/step - loss: 17.8870

Epoch 00032: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_032.tlt
Epoch 33/80
36/36 [==============================] - 45s 1s/step - loss: 17.8183

Epoch 00033: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_033.tlt
Epoch 34/80
36/36 [==============================] - 45s 1s/step - loss: 17.7503

Epoch 00034: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_034.tlt
Epoch 35/80
36/36 [==============================] - 45s 1s/step - loss: 17.6830

Epoch 00035: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_035.tlt
Epoch 36/80
36/36 [==============================] - 45s 1s/step - loss: 17.6165

Epoch 00036: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_036.tlt
Epoch 37/80
36/36 [==============================] - 45s 1s/step - loss: 17.5508

Epoch 00037: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_037.tlt
Epoch 38/80
36/36 [==============================] - 45s 1s/step - loss: 17.4858

Epoch 00038: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_038.tlt
Epoch 39/80
36/36 [==============================] - 45s 1s/step - loss: 17.4215

Epoch 00039: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_039.tlt
Epoch 40/80
36/36 [==============================] - 45s 1s/step - loss: 17.3579

Epoch 00040: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_040.tlt
Number of images in the evaluation dataset: 46

Producing predictions: 100%|██████████████████████| 6/6 [00:06<00:00, 1.09s/it]
Start multi-thread per-image matching
Start to calculate AP for each class

BLACK_ARTENGO AP 0.0
BLUE_ARTENGO AP 0.0
RED_KUIKMA AP 0.0
mAP 0.0

Epoch 41/80
36/36 [==============================] - 45s 1s/step - loss: 17.2950

Epoch 00041: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_041.tlt
Epoch 42/80
36/36 [==============================] - 45s 1s/step - loss: 17.2328

Epoch 00042: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_042.tlt
Epoch 43/80
36/36 [==============================] - 45s 1s/step - loss: 17.1714

Epoch 00043: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_043.tlt
Epoch 44/80
36/36 [==============================] - 45s 1s/step - loss: 17.1107

Epoch 00044: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_044.tlt
Epoch 45/80
36/36 [==============================] - 45s 1s/step - loss: 17.0506

Epoch 00045: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_045.tlt
Epoch 46/80
36/36 [==============================] - 45s 1s/step - loss: 16.9913

Epoch 00046: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_046.tlt
Epoch 47/80
36/36 [==============================] - 45s 1s/step - loss: 16.9326

Epoch 00047: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_047.tlt
Epoch 48/80
36/36 [==============================] - 45s 1s/step - loss: 16.8747

Epoch 00048: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_048.tlt
Epoch 49/80
36/36 [==============================] - 45s 1s/step - loss: 16.8174

Epoch 00049: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_049.tlt
Epoch 50/80
36/36 [==============================] - 45s 1s/step - loss: 16.7608

Epoch 00050: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_050.tlt
Number of images in the evaluation dataset: 46

Producing predictions: 100%|██████████████████████| 6/6 [00:06<00:00, 1.10s/it]
Start multi-thread per-image matching
Start to calculate AP for each class

BLACK_ARTENGO AP 0.0
BLUE_ARTENGO AP 0.0
RED_KUIKMA AP 0.0
mAP 0.0

Epoch 51/80
36/36 [==============================] - 45s 1s/step - loss: 16.7048

Epoch 00051: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_051.tlt
Epoch 52/80
36/36 [==============================] - 45s 1s/step - loss: 16.6496

Epoch 00052: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_052.tlt
Epoch 53/80
36/36 [==============================] - 45s 1s/step - loss: 16.5950

Epoch 00053: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_053.tlt
Epoch 54/80
36/36 [==============================] - 45s 1s/step - loss: 16.5410

Epoch 00054: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_054.tlt
Epoch 55/80
36/36 [==============================] - 45s 1s/step - loss: 16.4877

Epoch 00055: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_055.tlt
Epoch 56/80
36/36 [==============================] - 45s 1s/step - loss: 16.4351

Epoch 00056: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_056.tlt
Epoch 57/80
36/36 [==============================] - 45s 1s/step - loss: 16.3830

Epoch 00057: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_057.tlt
Epoch 58/80
36/36 [==============================] - 45s 1s/step - loss: 16.3316

Epoch 00058: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_058.tlt
Epoch 59/80
36/36 [==============================] - 45s 1s/step - loss: 16.2809

Epoch 00059: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_059.tlt
Epoch 60/80
36/36 [==============================] - 45s 1s/step - loss: 16.2308

Epoch 00060: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_060.tlt
Number of images in the evaluation dataset: 46

Producing predictions: 100%|██████████████████████| 6/6 [00:06<00:00, 1.14s/it]
Start multi-thread per-image matching
Start to calculate AP for each class

BLACK_ARTENGO AP 0.0
BLUE_ARTENGO AP 0.0
RED_KUIKMA AP 0.0
mAP 0.0

Epoch 61/80

36/36 [==============================] - 45s 1s/step - loss: 16.1813

Epoch 00061: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_061.tlt
Epoch 62/80
36/36 [==============================] - 45s 1s/step - loss: 16.1323

Epoch 00062: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_062.tlt
Epoch 63/80
36/36 [==============================] - 45s 1s/step - loss: 16.0840

Epoch 00063: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_063.tlt
Epoch 64/80
36/36 [==============================] - 45s 1s/step - loss: 16.0364

Epoch 00064: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_064.tlt
Epoch 65/80
36/36 [==============================] - 45s 1s/step - loss: 15.9905

Epoch 00065: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_065.tlt
Epoch 66/80
36/36 [==============================] - 45s 1s/step - loss: 15.9547

Epoch 00066: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_066.tlt
Epoch 67/80
36/36 [==============================] - 45s 1s/step - loss: 15.9300

Epoch 00067: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_067.tlt
Epoch 68/80
36/36 [==============================] - 45s 1s/step - loss: 15.9131

Epoch 00068: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_068.tlt
Epoch 69/80
36/36 [==============================] - 45s 1s/step - loss: 15.9016

Epoch 00069: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_069.tlt
Epoch 70/80
36/36 [==============================] - 45s 1s/step - loss: 15.8936

Epoch 00070: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_070.tlt
Number of images in the evaluation dataset: 46

Producing predictions: 100%|██████████████████████| 6/6 [00:06<00:00, 1.14s/it]
Start multi-thread per-image matching
Start to calculate AP for each class

BLACK_ARTENGO AP 0.0
BLUE_ARTENGO AP 0.0
RED_KUIKMA AP 0.0
mAP 0.0

Epoch 71/80
36/36 [==============================] - 45s 1s/step - loss: 15.8882

Epoch 00071: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_071.tlt
Epoch 72/80
36/36 [==============================] - 45s 1s/step - loss: 15.8844

Epoch 00072: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_072.tlt
Epoch 73/80
36/36 [==============================] - 45s 1s/step - loss: 15.8818

Epoch 00073: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_073.tlt
Epoch 74/80
36/36 [==============================] - 45s 1s/step - loss: 15.8801

Epoch 00074: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_074.tlt
Epoch 75/80
36/36 [==============================] - 45s 1s/step - loss: 15.8789

Epoch 00075: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_075.tlt
Epoch 76/80
36/36 [==============================] - 45s 1s/step - loss: 15.8780

Epoch 00076: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_076.tlt
Epoch 77/80
36/36 [==============================] - 45s 1s/step - loss: 15.8775

Epoch 00077: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_077.tlt
Epoch 78/80
36/36 [==============================] - 45s 1s/step - loss: 15.8771

Epoch 00078: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_078.tlt
Epoch 79/80
36/36 [==============================] - 45s 1s/step - loss: 15.8768

Epoch 00079: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_079.tlt
Epoch 80/80
36/36 [==============================] - 45s 1s/step - loss: 15.8766

Epoch 00080: saving model to /workspace/tlt-experiments/ssd/experiment_dir_unpruned/weights/ssd_resnet18_epoch_080.tlt
Number of images in the evaluation dataset: 46

Producing predictions: 100%|██████████████████████| 6/6 [00:07<00:00, 1.18s/it]
Start multi-thread per-image matching
Start to calculate AP for each class

BLACK_ARTENGO AP 0.0
BLUE_ARTENGO AP 0.0
RED_KUIKMA AP 0.0
mAP 0.0

Morganh · August 28, 2020, 7:51am

In your spec as below, please set to lowercase class name. In your tfrecord, it is already set to lowercase.

target_class_mapping {
key: “BLACK_ARTENGO”
value: “BLACK_ARTENGO”
}
target_class_mapping {
key: “BLUE_ARTENGO”
value: “BLUE_ARTENGO”
}
target_class_mapping {
key: “RED_KUIKMA”
value: “RED_KUIKMA”
}

Reference: Mean average_precision is 0% for all classes using Detectnet_V2 - #2 by Morganh

martin1 · August 28, 2020, 8:23am

Thanks. changing the class names to lower case solved my issue of AP:0

Topic		Replies	Views
TLT training error : Key cost_sums/cyclist-bbox not found in checkpoint TAO Toolkit	6	1193	October 12, 2021
Tlt-train loss is minimal but performances are bad TAO Toolkit	11	517	October 12, 2021
Slow GPU workaround for NHWC error when training TAO Toolkit	7	1220	October 12, 2021
Error training Faster RCNN model TAO Toolkit	17	1554	October 12, 2021
Unable to train SSD-Resnet-34 TAO Toolkit ssd	9	891	October 12, 2021
Errors in Training, 0 or Nan mAP, Low Loss, Tutorial Config TAO Toolkit	35	1780	October 12, 2021
Model retraining warning TAO Toolkit	7	1024	October 12, 2021
tlt-train error when deploy mobilenet_v2 by using DetectNet TAO Toolkit	28	2364	October 12, 2021
TypeError: get_calibrator() got an unexpected keyword argument 'image_mean' TAO Toolkit	9	774	October 12, 2021
DSSD resume error TAO Toolkit	33	1443	March 17, 2022

SSD: custom tlt training result in AP:0 for all my classes

Layer (type) Output Shape Param # Connected to

ssd_predictions (Reshape) (8, 58464, 15) 0 concatenate_1[0][0]

Layer (type) Output Shape Param # Connected to

ssd_predictions (Reshape) (8, 102390, 15) 0 concatenate_1[0][0]

Related topics