SSD Resnet 18 only learns 3 out of 5 classes

Accelerated Computing Intelligent Video Analytics TAO Toolkit
1

Hi Morganh,

I am training an SSD Resnet 18 model with ~7k images in my dataset with 5 classes. However, the model seems to be only learning 2 out of the 5 classes. Wondering if you can help?

More details are below:

  1. TFRecords spec:
    kitti_config {
    root_directory_path: “/workspace/tlt/dataset/package_watch_small/7000”
    image_dir_name: “images”
    label_dir_name: “labels”
    image_extension: “.jpg”
    partition_mode: “random”
    num_partitions: 2
    val_split: 20
    num_shards: 10
    }
    image_directory_path: “/workspace/tlt/dataset/pavkage_watch_small/7000”

  2. TFRecord creation command and log:
    tlt-dataset-convert -d /workspace/tlt/dataset/package_watch_small/7000/specs/ssd_tfrecords_pkg_watch_trainval.txt -o /workspace/tlt/dataset/package_watch_small/7000/tfrecords-test
    Using TensorFlow backend.
    2019-12-17 17:31:35,608 - iva.detectnet_v2.dataio.build_converter - INFO - Instantiating a kitti converter
    2019-12-17 17:31:35,626 - iva.detectnet_v2.dataio.kitti_converter_lib - INFO - Num images in
    Train: 5600 Val: 1400
    2019-12-17 17:31:35,627 - iva.detectnet_v2.dataio.kitti_converter_lib - INFO - Validation data in partition 0. Hence, while choosing the validationset during training choose validation_fold 0.
    2019-12-17 17:31:35,628 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 0
    /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:266: VisibleDeprecationWarning: Reading unicode strings without specifying the encoding argument is deprecated. Set the encoding, use None for the system default.
    2019-12-17 17:31:38,136 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 1
    2019-12-17 17:31:39,951 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 2
    2019-12-17 17:31:41,119 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 3
    2019-12-17 17:31:42,194 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 4
    2019-12-17 17:31:43,175 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 5
    2019-12-17 17:31:43,917 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 6
    2019-12-17 17:31:44,604 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 7
    2019-12-17 17:31:45,449 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 8
    2019-12-17 17:31:46,156 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 9
    2019-12-17 17:31:46,765 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO -
    Wrote the following numbers of objects:
    automobile: 3053
    package: 176
    person: 1469
    animal: 27
    face: 518

2019-12-17 17:31:46,765 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 0
/usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:266: UserWarning: genfromtxt: Empty input file: “/workspace/tlt/dataset/package_watch_small/7000/labels/2019-11-29T18:01:32.772481337+00:00.3.txt”
2019-12-17 17:32:02,975 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 1
2019-12-17 17:32:19,153 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 2
2019-12-17 17:32:34,000 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 3
2019-12-17 17:32:50,915 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 4
2019-12-17 17:33:06,328 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 5
2019-12-17 17:33:21,478 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 6
2019-12-17 17:33:36,833 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 7
/usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:266: UserWarning: genfromtxt: Empty input file: “/workspace/tlt/dataset/package_watch_small/7000/labels/2019-09-13T21:29:31.390381239+00:00.0.txt”
2019-12-17 17:33:52,701 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 8
2019-12-17 17:34:09,132 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 9
/usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:266: UserWarning: genfromtxt: Empty input file: “/workspace/tlt/dataset/package_watch_small/7000/labels/2019-09-14T02:02:07.534862382+00:00.0.txt”
2019-12-17 17:34:25,334 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO -
Wrote the following numbers of objects:
person: 5879
face: 1936
automobile: 12372
animal: 112
package: 675

2019-12-17 17:34:25,334 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Cumulative object statistics
2019-12-17 17:34:25,334 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO -
Wrote the following numbers of objects:
automobile: 15425
package: 851
person: 7348
animal: 139
face: 2454

2019-12-17 17:34:25,334 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Class map.
Label in GT: Label in tfrecords file
automobile: automobile
package: package
face: face
Animal: animal
person: person
For the dataset_config in the experiment_spec, please use labels in the tfrecords file, while writing the classmap.

  1. Training spec file:
    ssd_config {
    aspect_ratios_global: “[1.0, 2.0, 0.5, 3.0, 1.0/3.0]”
    scales: “[0.05, 0.1, 0.25, 0.4, 0.55, 0.7, 0.85]”
    two_boxes_for_ar1: true
    clip_boxes: false
    loss_loc_weight: 0.8
    focal_loss_alpha: 0.25
    focal_loss_gamma: 2.0
    variances: “[0.1, 0.1, 0.2, 0.2]”
    arch: “resnet18”
    freeze_bn: false
    freeze_blocks: 0
    }
    training_config {
    batch_size_per_gpu: 16
    num_epochs: 180
    learning_rate {
    soft_start_annealing_schedule {
    min_learning_rate: 5e-5
    max_learning_rate: 2e-2
    soft_start: 0.1
    annealing: 0.3
    }
    }
    regularizer {
    type: L1
    weight: 3.00000002618e-09
    }
    }
    eval_config {
    validation_period_during_training: 10
    average_precision_mode: SAMPLE
    batch_size: 32
    matching_iou_threshold: 0.5
    }
    nms_config {
    confidence_threshold: 0.01
    clustering_iou_threshold: 0.6
    top_k: 200
    }
    augmentation_config {
    preprocessing {
    output_image_width: 1248
    output_image_height: 384
    output_image_channel: 3
    crop_right: 1248
    crop_bottom: 384
    min_bbox_width: 1.0
    min_bbox_height: 1.0
    }
    spatial_augmentation {
    hflip_probability: 0.5
    vflip_probability: 0.0
    zoom_min: 0.7
    zoom_max: 1.8
    translate_max_x: 8.0
    translate_max_y: 8.0
    }
    color_augmentation {
    hue_rotation_max: 25.0
    saturation_shift_max: 0.20000000298
    contrast_scale_max: 0.10000000149
    contrast_center: 0.5
    }
    }
    dataset_config {
    data_sources: {
    tfrecords_path: “/workspace/tlt/dataset/package_watch_small/7000/tfrecords/package_watch_trainval_7000*”
    image_directory_path: “/workspace/tlt/dataset/package_watch_small/7000”
    }
    image_extension: “jpg”
    target_class_mapping {
    key: “person”
    value: “person”
    }
    target_class_mapping {
    key: “package”
    value: “package”
    }
    target_class_mapping {
    key: “automobile”
    value: “automobile”
    }
    target_class_mapping {
    key: “Animal”
    value: “animal”
    }
    target_class_mapping {
    key: “face”
    value: “face”
    }
    validation_fold: 0
    }
2
  1. Train command and log:
    nohup tlt-train ssd -e /workspace/tlt/dataset/package_watch_small/7000/specs/ssd_train_resnet18_pkg_watch_7000.txt -k -r /workspace/tlt/dataset/package_watch_small/7000/training-test/ -m /workspace/tlt/train-and-test/pretrained_resnet18/tlt_resnet18_ssd_v1/resnet18.hdf5 > training-test/train-log.txt &

Using TensorFlow backend.
2019-12-17 17:38:37.505798: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-12-17 17:38:38.216651: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-12-17 17:38:38.217378: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x6095d20 executing computations on platform CUDA. Devices:
2019-12-17 17:38:38.217425: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): Tesla V100-SXM2-16GB, Compute Capability 7.0
2019-12-17 17:38:38.220220: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2200000000 Hz
2019-12-17 17:38:38.220940: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x60feac0 executing computations on platform Host. Devices:
2019-12-17 17:38:38.220974: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): ,
2019-12-17 17:38:38.221110: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:00:04.0
totalMemory: 15.75GiB freeMemory: 15.44GiB
2019-12-17 17:38:38.221128: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-12-17 17:38:38.222366: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-12-17 17:38:38.222405: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2019-12-17 17:38:38.222418: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2019-12-17 17:38:38.222523: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15024 MB memory) → physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:04.0, compute capability: 7.0)
2019-12-17 17:38:38,225 [INFO] iva.ssd.scripts.train: Loading experiment spec at /workspace/tlt/dataset/package_watch_small/7000/specs/ssd_train_resnet18_pkg_watch_7000.txt.
2019-12-17 17:38:38,226 [INFO] /usr/local/lib/python2.7/dist-packages/iva/ssd/utils/spec_loader.pyc: Merging specification from /workspace/tlt/dataset/package_watch_small/7000/specs/ssd_train_resnet18_pkg_watch_7000.txt
WARNING:tensorflow:From ./detectnet_v2/dataloader/utilities.py:114: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version.
Instructions for updating:
Use eager execution and:
tf.data.TFRecordDataset(path)
2019-12-17 17:38:38,230 [WARNING] tensorflow: From ./detectnet_v2/dataloader/utilities.py:114: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version.
Instructions for updating:
Use eager execution and:
tf.data.TFRecordDataset(path)
WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/python/data/ops/iterator_ops.py:358: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
2019-12-17 17:38:38,287 [WARNING] tensorflow: From /usr/local/lib/python2.7/dist-packages/tensorflow/python/data/ops/iterator_ops.py:358: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
WARNING:tensorflow:From ./ssd/utils/tensor_utils.py:31: setdiff1d (from tensorflow.python.ops.array_ops) is deprecated and will be removed after 2018-11-30.
Instructions for updating:
This op will be removed after the deprecation date. Please switch to tf.sets.difference().
2019-12-17 17:38:49,105 [WARNING] tensorflow: From ./ssd/utils/tensor_utils.py:31: setdiff1d (from tensorflow.python.ops.array_ops) is deprecated and will be removed after 2018-11-30.
Instructions for updating:
This op will be removed after the deprecation date. Please switch to tf.sets.difference().
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains2019-12-17 17:39:06,466 [INFO] iva.ssd.scripts.train: Loading pretrained weights. This may take a while…
WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
2019-12-17 17:39:19,295 [WARNING] tensorflow: From /usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.

Layer (type) Output Shape Param # Connected to

Input (InputLayer) (16, 3, 384, 1248) 0

conv1 (Conv2D) (16, 64, 192, 624) 9472 Input[0][0]

bn_conv1 (BatchNormalization) (16, 64, 192, 624) 256 conv1[0][0]

activation_18 (Activation) (16, 64, 192, 624) 0 bn_conv1[0][0]

block_1a_conv_1 (Conv2D) (16, 64, 96, 312) 36928 activation_18[0][0]

block_1a_bn_1 (BatchNormalizati (16, 64, 96, 312) 256 block_1a_conv_1[0][0]

activation_19 (Activation) (16, 64, 96, 312) 0 block_1a_bn_1[0][0]

block_1a_conv_2 (Conv2D) (16, 64, 96, 312) 36928 activation_19[0][0]

block_1a_conv_shortcut (Conv2D) (16, 64, 96, 312) 4160 activation_18[0][0]

block_1a_bn_2 (BatchNormalizati (16, 64, 96, 312) 256 block_1a_conv_2[0][0]

block_1a_bn_shortcut (BatchNorm (16, 64, 96, 312) 256 block_1a_conv_shortcut[0][0]

add_9 (Add) (16, 64, 96, 312) 0 block_1a_bn_2[0][0]
block_1a_bn_shortcut[0][0]

activation_20 (Activation) (16, 64, 96, 312) 0 add_9[0][0]

block_1b_conv_1 (Conv2D) (16, 64, 96, 312) 36928 activation_20[0][0]

block_1b_bn_1 (BatchNormalizati (16, 64, 96, 312) 256 block_1b_conv_1[0][0]

activation_21 (Activation) (16, 64, 96, 312) 0 block_1b_bn_1[0][0]

block_1b_conv_2 (Conv2D) (16, 64, 96, 312) 36928 activation_21[0][0]

block_1b_conv_shortcut (Conv2D) (16, 64, 96, 312) 4160 activation_20[0][0]

block_1b_bn_2 (BatchNormalizati (16, 64, 96, 312) 256 block_1b_conv_2[0][0]

block_1b_bn_shortcut (BatchNorm (16, 64, 96, 312) 256 block_1b_conv_shortcut[0][0]

add_10 (Add) (16, 64, 96, 312) 0 block_1b_bn_2[0][0]
block_1b_bn_shortcut[0][0]

activation_22 (Activation) (16, 64, 96, 312) 0 add_10[0][0]

block_2a_conv_1 (Conv2D) (16, 128, 48, 156) 73856 activation_22[0][0]

block_2a_bn_1 (BatchNormalizati (16, 128, 48, 156) 512 block_2a_conv_1[0][0]

activation_23 (Activation) (16, 128, 48, 156) 0 block_2a_bn_1[0][0]

block_2a_conv_2 (Conv2D) (16, 128, 48, 156) 147584 activation_23[0][0]

block_2a_conv_shortcut (Conv2D) (16, 128, 48, 156) 8320 activation_22[0][0]

block_2a_bn_2 (BatchNormalizati (16, 128, 48, 156) 512 block_2a_conv_2[0][0]

block_2a_bn_shortcut (BatchNorm (16, 128, 48, 156) 512 block_2a_conv_shortcut[0][0]

add_11 (Add) (16, 128, 48, 156) 0 block_2a_bn_2[0][0]
block_2a_bn_shortcut[0][0]

activation_24 (Activation) (16, 128, 48, 156) 0 add_11[0][0]

block_2b_conv_1 (Conv2D) (16, 128, 48, 156) 147584 activation_24[0][0]

block_2b_bn_1 (BatchNormalizati (16, 128, 48, 156) 512 block_2b_conv_1[0][0]

activation_25 (Activation) (16, 128, 48, 156) 0 block_2b_bn_1[0][0]

block_2b_conv_2 (Conv2D) (16, 128, 48, 156) 147584 activation_25[0][0]

block_2b_conv_shortcut (Conv2D) (16, 128, 48, 156) 16512 activation_24[0][0]

block_2b_bn_2 (BatchNormalizati (16, 128, 48, 156) 512 block_2b_conv_2[0][0]

block_2b_bn_shortcut (BatchNorm (16, 128, 48, 156) 512 block_2b_conv_shortcut[0][0]

add_12 (Add) (16, 128, 48, 156) 0 block_2b_bn_2[0][0]
block_2b_bn_shortcut[0][0]

activation_26 (Activation) (16, 128, 48, 156) 0 add_12[0][0]

block_3a_conv_1 (Conv2D) (16, 256, 24, 78) 295168 activation_26[0][0]

block_3a_bn_1 (BatchNormalizati (16, 256, 24, 78) 1024 block_3a_conv_1[0][0]

activation_27 (Activation) (16, 256, 24, 78) 0 block_3a_bn_1[0][0]

block_3a_conv_2 (Conv2D) (16, 256, 24, 78) 590080 activation_27[0][0]

block_3a_conv_shortcut (Conv2D) (16, 256, 24, 78) 33024 activation_26[0][0]

block_3a_bn_2 (BatchNormalizati (16, 256, 24, 78) 1024 block_3a_conv_2[0][0]

block_3a_bn_shortcut (BatchNorm (16, 256, 24, 78) 1024 block_3a_conv_shortcut[0][0]

add_13 (Add) (16, 256, 24, 78) 0 block_3a_bn_2[0][0]
block_3a_bn_shortcut[0][0]

activation_28 (Activation) (16, 256, 24, 78) 0 add_13[0][0]

block_3b_conv_1 (Conv2D) (16, 256, 24, 78) 590080 activation_28[0][0]

block_3b_bn_1 (BatchNormalizati (16, 256, 24, 78) 1024 block_3b_conv_1[0][0]

activation_29 (Activation) (16, 256, 24, 78) 0 block_3b_bn_1[0][0]

block_3b_conv_2 (Conv2D) (16, 256, 24, 78) 590080 activation_29[0][0]

block_3b_conv_shortcut (Conv2D) (16, 256, 24, 78) 65792 activation_28[0][0]

block_3b_bn_2 (BatchNormalizati (16, 256, 24, 78) 1024 block_3b_conv_2[0][0]

block_3b_bn_shortcut (BatchNorm (16, 256, 24, 78) 1024 block_3b_conv_shortcut[0][0]

add_14 (Add) (16, 256, 24, 78) 0 block_3b_bn_2[0][0]
block_3b_bn_shortcut[0][0]

activation_30 (Activation) (16, 256, 24, 78) 0 add_14[0][0]

block_4a_conv_1 (Conv2D) (16, 512, 24, 78) 1180160 activation_30[0][0]

block_4a_bn_1 (BatchNormalizati (16, 512, 24, 78) 2048 block_4a_conv_1[0][0]

activation_31 (Activation) (16, 512, 24, 78) 0 block_4a_bn_1[0][0]

block_4a_conv_2 (Conv2D) (16, 512, 24, 78) 2359808 activation_31[0][0]

block_4a_conv_shortcut (Conv2D) (16, 512, 24, 78) 131584 activation_30[0][0]

block_4a_bn_2 (BatchNormalizati (16, 512, 24, 78) 2048 block_4a_conv_2[0][0]

block_4a_bn_shortcut (BatchNorm (16, 512, 24, 78) 2048 block_4a_conv_shortcut[0][0]

add_15 (Add) (16, 512, 24, 78) 0 block_4a_bn_2[0][0]
block_4a_bn_shortcut[0][0]

activation_32 (Activation) (16, 512, 24, 78) 0 add_15[0][0]

block_4b_conv_1 (Conv2D) (16, 512, 24, 78) 2359808 activation_32[0][0]

block_4b_bn_1 (BatchNormalizati (16, 512, 24, 78) 2048 block_4b_conv_1[0][0]

activation_33 (Activation) (16, 512, 24, 78) 0 block_4b_bn_1[0][0]

block_4b_conv_2 (Conv2D) (16, 512, 24, 78) 2359808 activation_33[0][0]

block_4b_conv_shortcut (Conv2D) (16, 512, 24, 78) 262656 activation_32[0][0]

block_4b_bn_2 (BatchNormalizati (16, 512, 24, 78) 2048 block_4b_conv_2[0][0]

block_4b_bn_shortcut (BatchNorm (16, 512, 24, 78) 2048 block_4b_conv_shortcut[0][0]

add_16 (Add) (16, 512, 24, 78) 0 block_4b_bn_2[0][0]
block_4b_bn_shortcut[0][0]

activation_34 (Activation) (16, 512, 24, 78) 0 add_16[0][0]

expand_conv1 (Conv2D) (16, 1024, 24, 78) 4719616 activation_34[0][0]

expand1_relu (ReLU) (16, 1024, 24, 78) 0 expand_conv1[0][0]

expand_conv2 (Conv2D) (16, 1024, 24, 78) 1049600 expand1_relu[0][0]

expand2_relu (ReLU) (16, 1024, 24, 78) 0 expand_conv2[0][0]

additional_map0_0 (Conv2D) (16, 256, 24, 78) 262400 expand2_relu[0][0]

additional_map0_0_relu (ReLU) (16, 256, 24, 78) 0 additional_map0_0[0][0]

additional_map0_1 (Conv2D) (16, 512, 12, 39) 1180160 additional_map0_0_relu[0][0]

additional_map0_1_relu (ReLU) (16, 512, 12, 39) 0 additional_map0_1[0][0]

additional_map1_0 (Conv2D) (16, 128, 12, 39) 65664 additional_map0_1_relu[0][0]

additional_map1_0_relu (ReLU) (16, 128, 12, 39) 0 additional_map1_0[0][0]

additional_map1_1 (Conv2D) (16, 256, 6, 20) 295168 additional_map1_0_relu[0][0]

additional_map1_1_relu (ReLU) (16, 256, 6, 20) 0 additional_map1_1[0][0]

additional_map2_0 (Conv2D) (16, 128, 6, 20) 32896 additional_map1_1_relu[0][0]

additional_map2_0_relu (ReLU) (16, 128, 6, 20) 0 additional_map2_0[0][0]

additional_map2_1 (Conv2D) (16, 256, 3, 10) 295168 additional_map2_0_relu[0][0]

additional_map2_1_relu (ReLU) (16, 256, 3, 10) 0 additional_map2_1[0][0]

additional_map3_0 (Conv2D) (16, 128, 3, 10) 32896 additional_map2_1_relu[0][0]

additional_map3_0_relu (ReLU) (16, 128, 3, 10) 0 additional_map3_0[0][0]

additional_map3_1 (Conv2D) (16, 256, 2, 5) 295168 additional_map3_0_relu[0][0]

additional_map3_1_relu (ReLU) (16, 256, 2, 5) 0 additional_map3_1[0][0]

ssd_conf_0 (Conv2D) (16, 30, 48, 156) 34590 activation_26[0][0]

ssd_conf_1 (Conv2D) (16, 30, 24, 78) 138270 activation_34[0][0]

ssd_conf_2 (Conv2D) (16, 30, 12, 39) 138270 additional_map0_1_relu[0][0]

ssd_conf_3 (Conv2D) (16, 30, 6, 20) 69150 additional_map1_1_relu[0][0]

ssd_conf_4 (Conv2D) (16, 30, 3, 10) 69150 additional_map2_1_relu[0][0]

ssd_conf_5 (Conv2D) (16, 30, 2, 5) 69150 additional_map3_1_relu[0][0]

permute_13 (Permute) (16, 48, 156, 30) 0 ssd_conf_0[0][0]

permute_15 (Permute) (16, 24, 78, 30) 0 ssd_conf_1[0][0]

permute_17 (Permute) (16, 12, 39, 30) 0 ssd_conf_2[0][0]

permute_19 (Permute) (16, 6, 20, 30) 0 ssd_conf_3[0][0]

permute_21 (Permute) (16, 3, 10, 30) 0 ssd_conf_4[0][0]

permute_23 (Permute) (16, 2, 5, 30) 0 ssd_conf_5[0][0]

ssd_loc_0 (Conv2D) (16, 24, 48, 156) 27672 activation_26[0][0]

ssd_loc_1 (Conv2D) (16, 24, 24, 78) 110616 activation_34[0][0]

ssd_loc_2 (Conv2D) (16, 24, 12, 39) 110616 additional_map0_1_relu[0][0]

ssd_loc_3 (Conv2D) (16, 24, 6, 20) 55320 additional_map1_1_relu[0][0]

ssd_loc_4 (Conv2D) (16, 24, 3, 10) 55320 additional_map2_1_relu[0][0]

ssd_loc_5 (Conv2D) (16, 24, 2, 5) 55320 additional_map3_1_relu[0][0]

conf_reshape_0 (Reshape) (16, 44928, 1, 5) 0 permute_13[0][0]

conf_reshape_1 (Reshape) (16, 11232, 1, 5) 0 permute_15[0][0]

conf_reshape_2 (Reshape) (16, 2808, 1, 5) 0 permute_17[0][0]

conf_reshape_3 (Reshape) (16, 720, 1, 5) 0 permute_19[0][0]

conf_reshape_4 (Reshape) (16, 180, 1, 5) 0 permute_21[0][0]

conf_reshape_5 (Reshape) (16, 60, 1, 5) 0 permute_23[0][0]

permute_14 (Permute) (16, 48, 156, 24) 0 ssd_loc_0[0][0]

permute_16 (Permute) (16, 24, 78, 24) 0 ssd_loc_1[0][0]

permute_18 (Permute) (16, 12, 39, 24) 0 ssd_loc_2[0][0]

permute_20 (Permute) (16, 6, 20, 24) 0 ssd_loc_3[0][0]

permute_22 (Permute) (16, 3, 10, 24) 0 ssd_loc_4[0][0]

permute_24 (Permute) (16, 2, 5, 24) 0 ssd_loc_5[0][0]

ssd_anchor_0 (AnchorBoxes) (16, 7488, 6, 8) 0 ssd_loc_0[0][0]

ssd_anchor_1 (AnchorBoxes) (16, 1872, 6, 8) 0 ssd_loc_1[0][0]

ssd_anchor_2 (AnchorBoxes) (16, 468, 6, 8) 0 ssd_loc_2[0][0]

ssd_anchor_3 (AnchorBoxes) (16, 120, 6, 8) 0 ssd_loc_3[0][0]

ssd_anchor_4 (AnchorBoxes) (16, 30, 6, 8) 0 ssd_loc_4[0][0]

ssd_anchor_5 (AnchorBoxes) (16, 10, 6, 8) 0 ssd_loc_5[0][0]

mbox_conf (Concatenate) (16, 59928, 1, 5) 0 conf_reshape_0[0][0]
conf_reshape_1[0][0]
conf_reshape_2[0][0]
conf_reshape_3[0][0]
conf_reshape_4[0][0]
conf_reshape_5[0][0]

loc_reshape_0 (Reshape) (16, 44928, 1, 4) 0 permute_14[0][0]

loc_reshape_1 (Reshape) (16, 11232, 1, 4) 0 permute_16[0][0]

loc_reshape_2 (Reshape) (16, 2808, 1, 4) 0 permute_18[0][0]

loc_reshape_3 (Reshape) (16, 720, 1, 4) 0 permute_20[0][0]

loc_reshape_4 (Reshape) (16, 180, 1, 4) 0 permute_22[0][0]

loc_reshape_5 (Reshape) (16, 60, 1, 4) 0 permute_24[0][0]

anchor_reshape_0 (Reshape) (16, 44928, 1, 8) 0 ssd_anchor_0[0][0]

anchor_reshape_1 (Reshape) (16, 11232, 1, 8) 0 ssd_anchor_1[0][0]

anchor_reshape_2 (Reshape) (16, 2808, 1, 8) 0 ssd_anchor_2[0][0]

anchor_reshape_3 (Reshape) (16, 720, 1, 8) 0 ssd_anchor_3[0][0]

anchor_reshape_4 (Reshape) (16, 180, 1, 8) 0 ssd_anchor_4[0][0]

anchor_reshape_5 (Reshape) (16, 60, 1, 8) 0 ssd_anchor_5[0][0]

mbox_conf_sigmoid (Activation) (16, 59928, 1, 5) 0 mbox_conf[0][0]

mbox_loc (Concatenate) (16, 59928, 1, 4) 0 loc_reshape_0[0][0]
loc_reshape_1[0][0]
loc_reshape_2[0][0]
loc_reshape_3[0][0]
loc_reshape_4[0][0]
loc_reshape_5[0][0]

mbox_priorbox (Concatenate) (16, 59928, 1, 8) 0 anchor_reshape_0[0][0]
anchor_reshape_1[0][0]
anchor_reshape_2[0][0]
anchor_reshape_3[0][0]
anchor_reshape_4[0][0]
anchor_reshape_5[0][0]

concatenate_2 (Concatenate) (16, 59928, 1, 17) 0 mbox_conf_sigmoid[0][0]
mbox_loc[0][0]
mbox_priorbox[0][0]2019-12-17 17:43:21,406 [INFO] iva.ssd.scripts.train: Number of images in the training dataset: 5600
2019-12-17 17:43:21,406 [INFO] iva.ssd.scripts.train: Number of images in the validation dataset: 1400
2019-12-17 17:43:43.473642: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_16/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-17 17:43:43.473737: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_16/CornerCoordToCentroids/StridedReplace/strided_slice. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?,1]
2019-12-17 17:43:43.473762: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_16/CornerCoordToCentroids/strided_slice_4. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-17 17:43:43.473784: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_16/CornerCoordToCentroids/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-17 17:43:43.473805: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_16/CornerCoordToCentroids/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-17 17:43:43.473825: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_16/CornerCoordToCentroids/strided_slice_7. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-17 17:43:43.473846: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_16/CornerCoordToCentroids/StridedReplace/strided_slice_1. Error: ValidateStridedSliceOp returned partial shapes [?,0] and [?,0]
2019-12-17 17:43:43.473867: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_16/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,4] and [?,4]
2019-12-17 17:43:43.473934: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_17/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-17 17:43:43.473957: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_17/CornerCoordToCentroids/StridedReplace/strided_slice. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?,1]
2019-12-17 17:43:43.473979: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_17/CornerCoordToCentroids/strided_slice_4. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-17 17:43:43.474015: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_17/CornerCoordToCentroids/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-17 17:43:43.474035: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_17/CornerCoordToCentroids/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-17 17:43:43.474054: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_17/CornerCoordToCentroids/strided_slice_7. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-17 17:43:43.474074: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_17/CornerCoordToCentroids/StridedReplace/strided_slice_1. Error: ValidateStridedSliceOp returned partial shapes [?,0] and [?,0]
2019-12-17 17:43:43.474094: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_17/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,4] and [?,4]
2019-12-17 17:43:43.474122: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_18/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-17 17:43:43.474143: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_18/CornerCoordToCentroids/StridedReplace/strided_slice. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?,1]
2019-12-17 17:43:43.474165: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_18/CornerCoordToCentroids/strided_slice_4. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-17 17:43:43.474185: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_18/CornerCoordToCentroids/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-17 17:43:43.474205: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_18/CornerCoordToCentroids/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-17 17:43:43.474225: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_18/CornerCoordToCentroids/strided_slice_7. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-17 17:43:43.474245: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_18/CornerCoordToCentroids/StridedReplace/strided_slice_1. Error: ValidateStridedSliceOp returned partial shapes [?,0] and [?,0]
2019-12-17 17:43:43.474264: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_18/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,4] and [?,4]
2019-12-17 17:43:43.474295: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_19/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-17 17:43:43.474316: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_19/CornerCoordToCentroids/StridedReplace/strided_slice. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?,1]
2019-12-17 17:43:43.474346: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_19/CornerCoordToCentroids/strided_slice_4. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-17 17:43:43.474367: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_19/CornerCoordToCentroids/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-17 17:43:43.474388: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_19/CornerCoordToCentroids/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-17 17:43:43.474407: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_19/CornerCoordToCentroids/strided_slice_7. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-17 17:43:43.474427: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_19/CornerCoordToCentroids/StridedReplace/strided_slice_1. Error: ValidateStridedSliceOp returned partial shapes [?,0] and [?,0]
2019-12-17 17:43:43.474447: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_19/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,4] and [?,4]
2019-12-17 17:43:43.474473: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_20/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-17 17:43:43.474494: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_20/CornerCoordToCentroids/StridedReplace/strided_slice. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?,1]
2019-12-17 17:43:43.474515: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_20/CornerCoordToCentroids/strided_slice_4. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-17 17:43:43.474535: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_20/CornerCoordToCentroids/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-17 17:43:43.474555: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_20/CornerCoordToCentroids/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-17 17:43:43.474574: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_20/CornerCoordToCentroids/strided_slice_7. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-17 17:43:43.474598: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_20/CornerCoordToCentroids/StridedReplace/strided_slice_1. Error: ValidateStridedSliceOp returned partial shapes [?,0] and [?,0]
2019-12-17 17:43:43.474618: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_20/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,4] and [?,4]
2019-12-17 17:43:43.474645: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_21/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-17 17:43:43.474665: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_21/CornerCoordToCentroids/StridedReplace/strided_slice. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?,1]
2019-12-17 17:43:43.474686: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_21/CornerCoordToCentroids/strided_slice_4. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-17 17:43:43.474706: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_21/CornerCoordToCentroids/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-17 17:43:43.474726: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_21/CornerCoordToCentroids/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-17 17:43:43.474747: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_21/CornerCoordToCentroids/strided_slice_7. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-17 17:43:43.474766: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_21/CornerCoordToCentroids/StridedReplace/strided_slice_1. Error: ValidateStridedSliceOp returned partial shapes [?,0] and [?,0]
2019-12-17 17:43:43.474785: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_21/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,4] and [?,4]
2019-12-17 17:43:43.474813: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_22/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-17 17:43:43.474833: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_22/CornerCoordToCentroids/StridedReplace/strided_slice. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?,1]
2019-12-17 17:43:43.474855: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_22/CornerCoordToCentroids/strided_slice_4. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-17 17:43:43.474876: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_22/CornerCoordToCentroids/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-17 17:43:43.474900: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_22/CornerCoordToCentroids/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-17 17:43:43.474920: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_22/CornerCoordToCentroids/strided_slice_7. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-17 17:43:43.474940: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_22/CornerCoordToCentroids/StridedReplace/strided_slice_1. Error: ValidateStridedSliceOp returned partial shapes [?,0] and [?,0]
2019-12-17 17:43:43.474960: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_22/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,4] and [?,4]
2019-12-17 17:43:43.474987: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_23/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-17 17:43:43.475007: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_23/CornerCoordToCentroids/StridedReplace/strided_slice. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?,1]
2019-12-17 17:43:43.475028: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_23/CornerCoordToCentroids/strided_slice_4. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-17 17:43:43.475048: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_23/CornerCoordToCentroids/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-17 17:43:43.475069: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_23/CornerCoordToCentroids/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-17 17:43:43.475088: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_23/CornerCoordToCentroids/strided_slice_7. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-17 17:43:43.475108: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_23/CornerCoordToCentroids/StridedReplace/strided_slice_1. Error: ValidateStridedSliceOp returned partial shapes [?,0] and [?,0]
2019-12-17 17:43:43.475127: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_23/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,4] and [?,4]
2019-12-17 17:43:43.475154: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_24/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-17 17:43:43.475174: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_24/CornerCoordToCentroids/StridedReplace/strided_slice. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?,1]
2019-12-17 17:43:43.475200: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_24/CornerCoordToCentroids/strided_slice_4. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-17 17:43:43.475222: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_24/CornerCoordToCentroids/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-17 17:43:43.475242: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_24/CornerCoordToCentroids/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-17 17:43:43.475262: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_24/CornerCoordToCentroids/strided_slice_7. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-17 17:43:43.475282: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_24/CornerCoordToCentroids/StridedReplace/strided_slice_1. Error: ValidateStridedSliceOp returned partial shapes [?,0] and [?,0]
2019-12-17 17:43:43.475301: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_24/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,4] and [?,4]
2019-12-17 17:43:43.475328: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_25/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-17 17:43:43.475356: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_25/CornerCoordToCentroids/StridedReplace/strided_slice. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?,1]
2019-12-17 17:43:43.475377: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_25/CornerCoordToCentroids/strided_slice_4. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-17 17:43:43.475398: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_25/CornerCoordToCentroids/strided_slice_5. Error: Valid

3

ssd_predictions (Reshape) (16, 59928, 17) 0 concatenate_2[0][0]

Total params: 20,710,468
Trainable params: 20,689,348
Non-trainable params: 21,120

Epoch 1/180

1/350 […] - ETA: 2:16:18 - loss: 28235.0469
2/350 […] - ETA: 1:09:07 - loss: 14375.8752
3/350 […] - ETA: 46:44 - loss: 9600.8904
4/350 […] - ETA: 35:30 - loss: 7211.3285
5/350 […] - ETA: 28:45 - loss: 5774.2983
6/350 […] - ETA: 24:16 - loss: 4817.0156
7/350 […] - ETA: 21:04 - loss: 4133.2201
8/350 […] - ETA: 18:40 - loss: 3618.9396
9/350 […] - ETA: 16:47 - loss: 3218.1402
10/350 […] - ETA: 15:17 - loss: 2897.8253
11/350 […] - ETA: 14:03 - loss: 2635.8241
12/350 [>…] - ETA: 13:01 - loss: 2417.0847
13/350 [>…] - ETA: 12:09 - loss: 2232.2
. .
..
..

After 10 epochs,

WARNING:tensorflow:From ./ssd/box_coder/output_decoder_layer.py:83: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
2019-12-17 18:05:41,783 [WARNING] tensorflow: From ./ssd/box_coder/output_decoder_layer.py:83: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.

Epoch 00010: saving model to /workspace/tlt/dataset/package_watch_small/7000/training-test/weights/ssd_resnet18_epoch_010.tlt
Number of images in the evaluation dataset: 1400
()
Producing predictions batch-wise: 100%|##########| 44/44 [00:39<00:00, 1.12it/s]
Matching predictions to ground truth, class 1/5.: 100%|##########| 2797/2797 [00:00<00:00, 74143.75it/s]
Matching predictions to ground truth, class 2/5.: 100%|##########| 8492/8492 [00:00<00:00, 65039.96it/s]
Matching predictions to ground truth, class 3/5.: 100%|##########| 40821/40821 [00:01<00:00, 34870.19it/s]
Matching predictions to ground truth, class 4/5.: 100%|##########| 12896/12896 [00:00<00:00, 67760.24it/s]
Matching predictions to ground truth, class 5/5.: 100%|##########| 216594/216594 [00:06<00:00, 35511.55it/s]/usr/local/lib/python2.7/dist-packages/iva/ssd/utils/average_precision_evaluator.py:702: RuntimeWarning: invalid value encountered in true_divide
/usr/local/lib/python2.7/dist-packages/iva/ssd/utils/average_precision_evaluator.py:773: RuntimeWarning: invalid value encountered in greater_equal

Computing precisions and recalls, class 1/5
Computing precisions and recalls, class 2/5
Computing precisions and recalls, class 3/5
Computing precisions and recalls, class 4/5
Computing precisions and recalls, class 5/5
Computing average precision, class 1/5
Computing average precision, class 2/5
Computing average precision, class 3/5
Computing average precision, class 4/5
Computing average precision, class 5/5

Animal AP 0.0
automobile AP 0.0
face AP 0.244
package AP 0.0
person AP 0.352
mAP 0.119

  1. Training by removing the pretrained model -

tlt-train ssd -e /workspace/tlt/dataset/package_watch_small/7000/specs/ssd_train_resnet18_pkg_watch_7000.txt -k -r /workspace/tlt/dataset/package_watch_small/7000/training-from-scratch/

also gives very similar results (only person and face classes have non-zero mAP)

4

Never mind, I just realized I had mismatching image resolutions in augmentation train config and TFRecords. Changing the config to match my TFRecord and label image dimensions as mentioned in Integrating TAO Models into DeepStream — TAO Toolkit 3.22.05 documentation seems to have fixed this issue.

5

Good to know your info.