Model retraining warning

Hi,

I am trying to retrain one of my trained TLT models with additional data. I see this warning:

UserWarning: No training configuration found in save file: the model was not compiled. Compile it manually.

I see that “tlt-train” only stores the weights during training, and I’m using this “.tlt” weights file as the “-m (RESUME_MODEL_WEIGHTS)” argument in “tlt-train”. I do NOT see this warning while using the hdf5 model. Since “compiling” is essentially “preparing for training”, I am wondering if this means my model is being trained from scratch instead?

Thanks,
Anusha

Hi Anusha,
Could you please paste

  1. your retraining spec file
  2. Retrain command line
  3. Retrain full log
    Thanks.

Sure, here you go -

  1. ssd_config {
    aspect_ratios_global: “[1.0, 2.0, 0.5, 3.0, 1.0/3.0]”
    scales: “[0.05, 0.1, 0.25, 0.4, 0.55, 0.7, 0.85]”
    two_boxes_for_ar1: true
    clip_boxes: false
    loss_loc_weight: 0.8
    focal_loss_alpha: 0.25
    focal_loss_gamma: 2.0
    variances: “[0.1, 0.1, 0.2, 0.2]”
    arch: “resnet18”
    freeze_bn: false
    freeze_blocks: 0
    }
    training_config {
    batch_size_per_gpu: 16
    num_epochs: 180
    learning_rate {
    soft_start_annealing_schedule {
    min_learning_rate: 5e-5
    max_learning_rate: 2e-2
    soft_start: 0.1
    annealing: 0.3
    }
    }
    regularizer {
    type: L1
    weight: 3.00000002618e-09
    }
    }
    eval_config {
    validation_period_during_training: 10
    average_precision_mode: SAMPLE
    batch_size: 32
    matching_iou_threshold: 0.5
    }
    nms_config {
    confidence_threshold: 0.01
    clustering_iou_threshold: 0.6
    top_k: 200
    }
    augmentation_config {
    preprocessing {
    output_image_width: 1248
    output_image_height: 384
    output_image_channel: 3
    crop_right: 1248
    crop_bottom: 384
    min_bbox_width: 1.0
    min_bbox_height: 1.0
    }
    spatial_augmentation {
    hflip_probability: 0.5
    vflip_probability: 0.0
    zoom_min: 0.7
    zoom_max: 1.8
    translate_max_x: 8.0
    translate_max_y: 8.0
    }
    color_augmentation {
    hue_rotation_max: 25.0
    saturation_shift_max: 0.20000000298
    contrast_scale_max: 0.10000000149
    contrast_center: 0.5
    }
    }
    dataset_config {
    data_sources: {
    tfrecords_path: “/workspace/tlt/dataset/package_watch_small/7000/tfrecords/package_watch_trainval_7000*”
    image_directory_path: “/workspace/tlt/dataset/package_watch_small/7000”
    }
    image_extension: “jpg”
    target_class_mapping {
    key: “person”
    value: “person”
    }
    target_class_mapping {
    key: “package”
    value: “package”
    }
    target_class_mapping {
    key: “automobile”
    value: “automobile”
    }
    target_class_mapping {
    key: “Animal”
    value: “animal”
    }
    target_class_mapping {
    key: “face”
    value: “face”
    }
    validation_fold: 0
    }

  2. . tlt-train-g1 ssd -e /workspace/tlt/dataset/package_watch_small/7000/specs/ssd_train_resnet18_pkg_watch_7000.txt -r /workspace/tlt/dataset/package_watch_small/7000/training-new -k -m /workspace/tlt/train-and-test/pkg_watch_experiment_dir_unpruned/weights/ssd_resnet18_epoch_180.tlt

  3. Using TensorFlow backend.
    2019-12-14 17:02:10.707119: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
    2019-12-14 17:02:11.404978: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
    2019-12-14 17:02:11.405849: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x55fe6d0 executing computations on platform CUDA. Devices:
    2019-12-14 17:02:11.405897: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): Tesla V100-SXM2-16GB, Compute Capability 7.0
    2019-12-14 17:02:11.408252: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2200000000 Hz
    2019-12-14 17:02:11.408602: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x5667470 executing computations on platform Host. Devices:
    2019-12-14 17:02:11.408633: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): ,
    2019-12-14 17:02:11.408932: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
    name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
    pciBusID: 0000:00:04.0
    totalMemory: 15.75GiB freeMemory: 15.44GiB
    2019-12-14 17:02:11.408961: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
    2019-12-14 17:02:11.409997: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
    2019-12-14 17:02:11.410024: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
    2019-12-14 17:02:11.410032: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
    2019-12-14 17:02:11.410121: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15024 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:04.0, compute capability: 7.0)
    2019-12-14 17:02:11,412 [INFO] iva.ssd.scripts.train: Loading experiment spec at /workspace/tlt/dataset/package_watch_small/7000/specs/ssd_train_resnet18_pkg_watch_7000.txt.
    2019-12-14 17:02:11,413 [INFO] /usr/local/lib/python2.7/dist-packages/iva/ssd/utils/spec_loader.pyc: Merging specification from /workspace/tlt/dataset/package_watch_small/7000/specs/ssd_train_resnet18_pkg_watch_7000.txt
    WARNING:tensorflow:From ./detectnet_v2/dataloader/utilities.py:114: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version.
    Instructions for updating:
    Use eager execution and:
    tf.data.TFRecordDataset(path)
    2019-12-14 17:02:11,417 [WARNING] tensorflow: From ./detectnet_v2/dataloader/utilities.py:114: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version.
    Instructions for updating:
    Use eager execution and:
    tf.data.TFRecordDataset(path)
    WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/python/data/ops/iterator_ops.py:358: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
    Instructions for updating:
    Colocations handled automatically by placer.
    2019-12-14 17:02:11,474 [WARNING] tensorflow: From /usr/local/lib/python2.7/dist-packages/tensorflow/python/data/ops/iterator_ops.py:358: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
    Instructions for updating:
    Colocations handled automatically by placer.
    target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
    target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
    target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
    target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
    target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
    target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
    target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
    target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
    target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
    target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
    target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
    target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
    target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
    target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
    target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
    target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
    WARNING:tensorflow:From ./ssd/utils/tensor_utils.py:31: setdiff1d (from tensorflow.python.ops.array_ops) is deprecated and will be removed after 2018-11-30.
    Instructions for updating:
    This op will be removed after the deprecation date. Please switch to tf.sets.difference().
    2019-12-14 17:02:21,705 [WARNING] tensorflow: From ./ssd/utils/tensor_utils.py:31: setdiff1d (from tensorflow.python.ops.array_ops) is deprecated and will be removed after 2018-11-30.
    Instructions for updating:
    This op will be removed after the deprecation date. Please switch to tf.sets.difference().
    target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
    target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
    target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
    target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
    target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
    target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
    target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
    target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
    target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
    target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
    target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
    target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
    target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
    target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
    target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
    target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
    target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
    target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
    target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
    target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
    target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
    target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
    target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
    target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
    target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
    target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
    target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
    target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
    target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
    target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
    target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
    target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
    2019-12-14 17:02:38,108 [INFO] iva.ssd.scripts.train: Loading pretrained weights. This may take a while…
    /usr/local/lib/python2.7/dist-packages/keras/engine/saving.py:292: UserWarning: No training configuration found in save file: the model was not compiled. Compile it manually.
    warnings.warn('No training configuration found in save file: ’


Layer (type) Output Shape Param # Connected to

Input (InputLayer) (16, 3, 384, 1248) 0


conv1 (Conv2D) (16, 64, 192, 624) 9472 Input[0][0]


bn_conv1 (BatchNormalization) (16, 64, 192, 624) 256 conv1[0][0]


activation_18 (Activation) (16, 64, 192, 624) 0 bn_conv1[0][0]


block_1a_conv_1 (Conv2D) (16, 64, 96, 312) 36928 activation_18[0][0]


block_1a_bn_1 (BatchNormalizati (16, 64, 96, 312) 256 block_1a_conv_1[0][0]


activation_19 (Activation) (16, 64, 96, 312) 0 block_1a_bn_1[0][0]


block_1a_conv_2 (Conv2D) (16, 64, 96, 312) 36928 activation_19[0][0]


block_1a_conv_shortcut (Conv2D) (16, 64, 96, 312) 4160 activation_18[0][0]


block_1a_bn_2 (BatchNormalizati (16, 64, 96, 312) 256 block_1a_conv_2[0][0]


block_1a_bn_shortcut (BatchNorm (16, 64, 96, 312) 256 block_1a_conv_shortcut[0][0]


add_9 (Add) (16, 64, 96, 312) 0 block_1a_bn_2[0][0]
block_1a_bn_shortcut[0][0]


activation_20 (Activation) (16, 64, 96, 312) 0 add_9[0][0]


block_1b_conv_1 (Conv2D) (16, 64, 96, 312) 36928 activation_20[0][0]


block_1b_bn_1 (BatchNormalizati (16, 64, 96, 312) 256 block_1b_conv_1[0][0]


activation_21 (Activation) (16, 64, 96, 312) 0 block_1b_bn_1[0][0]


block_1b_conv_2 (Conv2D) (16, 64, 96, 312) 36928 activation_21[0][0]


block_1b_conv_shortcut (Conv2D) (16, 64, 96, 312) 4160 activation_20[0][0]


block_1b_bn_2 (BatchNormalizati (16, 64, 96, 312) 256 block_1b_conv_2[0][0]


block_1b_bn_shortcut (BatchNorm (16, 64, 96, 312) 256 block_1b_conv_shortcut[0][0]


add_10 (Add) (16, 64, 96, 312) 0 block_1b_bn_2[0][0]
block_1b_bn_shortcut[0][0]


activation_22 (Activation) (16, 64, 96, 312) 0 add_10[0][0]


block_2a_conv_1 (Conv2D) (16, 128, 48, 156) 73856 activation_22[0][0]


block_2a_bn_1 (BatchNormalizati (16, 128, 48, 156) 512 block_2a_conv_1[0][0]


activation_23 (Activation) (16, 128, 48, 156) 0 block_2a_bn_1[0][0]


block_2a_conv_2 (Conv2D) (16, 128, 48, 156) 147584 activation_23[0][0]


block_2a_conv_shortcut (Conv2D) (16, 128, 48, 156) 8320 activation_22[0][0]


block_2a_bn_2 (BatchNormalizati (16, 128, 48, 156) 512 block_2a_conv_2[0][0]


block_2a_bn_shortcut (BatchNorm (16, 128, 48, 156) 512 block_2a_conv_shortcut[0][0]


add_11 (Add) (16, 128, 48, 156) 0 block_2a_bn_2[0][0]
block_2a_bn_shortcut[0][0]


activation_24 (Activation) (16, 128, 48, 156) 0 add_11[0][0]


block_2b_conv_1 (Conv2D) (16, 128, 48, 156) 147584 activation_24[0][0]


block_2b_bn_1 (BatchNormalizati (16, 128, 48, 156) 512 block_2b_conv_1[0][0]


activation_25 (Activation) (16, 128, 48, 156) 0 block_2b_bn_1[0][0]


block_2b_conv_2 (Conv2D) (16, 128, 48, 156) 147584 activation_25[0][0]


block_2b_conv_shortcut (Conv2D) (16, 128, 48, 156) 16512 activation_24[0][0]


block_2b_bn_2 (BatchNormalizati (16, 128, 48, 156) 512 block_2b_conv_2[0][0]


block_2b_bn_shortcut (BatchNorm (16, 128, 48, 156) 512 block_2b_conv_shortcut[0][0]


add_12 (Add) (16, 128, 48, 156) 0 block_2b_bn_2[0][0]
block_2b_bn_shortcut[0][0]


activation_26 (Activation) (16, 128, 48, 156) 0 add_12[0][0]


block_3a_conv_1 (Conv2D) (16, 256, 24, 78) 295168 activation_26[0][0]


block_3a_bn_1 (BatchNormalizati (16, 256, 24, 78) 1024 block_3a_conv_1[0][0]


activation_27 (Activation) (16, 256, 24, 78) 0 block_3a_bn_1[0][0]


block_3a_conv_2 (Conv2D) (16, 256, 24, 78) 590080 activation_27[0][0]


block_3a_conv_shortcut (Conv2D) (16, 256, 24, 78) 33024 activation_26[0][0]


block_3a_bn_2 (BatchNormalizati (16, 256, 24, 78) 1024 block_3a_conv_2[0][0]


block_3a_bn_shortcut (BatchNorm (16, 256, 24, 78) 1024 block_3a_conv_shortcut[0][0]


add_13 (Add) (16, 256, 24, 78) 0 block_3a_bn_2[0][0]
block_3a_bn_shortcut[0][0]


activation_28 (Activation) (16, 256, 24, 78) 0 add_13[0][0]


block_3b_conv_1 (Conv2D) (16, 256, 24, 78) 590080 activation_28[0][0]


block_3b_bn_1 (BatchNormalizati (16, 256, 24, 78) 1024 block_3b_conv_1[0][0]


activation_29 (Activation) (16, 256, 24, 78) 0 block_3b_bn_1[0][0]


block_3b_conv_2 (Conv2D) (16, 256, 24, 78) 590080 activation_29[0][0]


block_3b_conv_shortcut (Conv2D) (16, 256, 24, 78) 65792 activation_28[0][0]


block_3b_bn_2 (BatchNormalizati (16, 256, 24, 78) 1024 block_3b_conv_2[0][0]


block_3b_bn_shortcut (BatchNorm (16, 256, 24, 78) 1024 block_3b_conv_shortcut[0][0]


add_14 (Add) (16, 256, 24, 78) 0 block_3b_bn_2[0][0]
block_3b_bn_shortcut[0][0]


activation_30 (Activation) (16, 256, 24, 78) 0 add_14[0][0]


block_4a_conv_1 (Conv2D) (16, 512, 24, 78) 1180160 activation_30[0][0]


block_4a_bn_1 (BatchNormalizati (16, 512, 24, 78) 2048 block_4a_conv_1[0][0]


activation_31 (Activation) (16, 512, 24, 78) 0 block_4a_bn_1[0][0]


block_4a_conv_2 (Conv2D) (16, 512, 24, 78) 2359808 activation_31[0][0]


block_4a_conv_shortcut (Conv2D) (16, 512, 24, 78) 131584 activation_30[0][0]


block_4a_bn_2 (BatchNormalizati (16, 512, 24, 78) 2048 block_4a_conv_2[0][0]


block_4a_bn_shortcut (BatchNorm (16, 512, 24, 78) 2048 block_4a_conv_shortcut[0][0]


add_15 (Add) (16, 512, 24, 78) 0 block_4a_bn_2[0][0]
block_4a_bn_shortcut[0][0]


activation_32 (Activation) (16, 512, 24, 78) 0 add_15[0][0]


block_4b_conv_1 (Conv2D) (16, 512, 24, 78) 2359808 activation_32[0][0]


block_4b_bn_1 (BatchNormalizati (16, 512, 24, 78) 2048 block_4b_conv_1[0][0]


activation_33 (Activation) (16, 512, 24, 78) 0 block_4b_bn_1[0][0]


block_4b_conv_2 (Conv2D) (16, 512, 24, 78) 2359808 activation_33[0][0]


block_4b_conv_shortcut (Conv2D) (16, 512, 24, 78) 262656 activation_32[0][0]


block_4b_bn_2 (BatchNormalizati (16, 512, 24, 78) 2048 block_4b_conv_2[0][0]


block_4b_bn_shortcut (BatchNorm (16, 512, 24, 78) 2048 block_4b_conv_shortcut[0][0]


add_16 (Add) (16, 512, 24, 78) 0 block_4b_bn_2[0][0]
block_4b_bn_shortcut[0][0]


activation_34 (Activation) (16, 512, 24, 78) 0 add_16[0][0]


expand_conv1 (Conv2D) (16, 1024, 24, 78) 4719616 activation_34[0][0]


expand1_relu (ReLU) (16, 1024, 24, 78) 0 expand_conv1[0][0]


expand_conv2 (Conv2D) (16, 1024, 24, 78) 1049600 expand1_relu[0][0]


expand2_relu (ReLU) (16, 1024, 24, 78) 0 expand_conv2[0][0]


additional_map0_0 (Conv2D) (16, 256, 24, 78) 262400 expand2_relu[0][0]


additional_map0_0_relu (ReLU) (16, 256, 24, 78) 0 additional_map0_0[0][0]


additional_map0_1 (Conv2D) (16, 512, 12, 39) 1180160 additional_map0_0_relu[0][0]


additional_map0_1_relu (ReLU) (16, 512, 12, 39) 0 additional_map0_1[0][0]


additional_map1_0 (Conv2D) (16, 128, 12, 39) 65664 additional_map0_1_relu[0][0]


additional_map1_0_relu (ReLU) (16, 128, 12, 39) 0 additional_map1_0[0][0]


additional_map1_1 (Conv2D) (16, 256, 6, 20) 295168 additional_map1_0_relu[0][0]


additional_map1_1_relu (ReLU) (16, 256, 6, 20) 0 additional_map1_1[0][0]


additional_map2_0 (Conv2D) (16, 128, 6, 20) 32896 additional_map1_1_relu[0][0]


additional_map2_0_relu (ReLU) (16, 128, 6, 20) 0 additional_map2_0[0][0]


additional_map2_1 (Conv2D) (16, 256, 3, 10) 295168 additional_map2_0_relu[0][0]


additional_map2_1_relu (ReLU) (16, 256, 3, 10) 0 additional_map2_1[0][0]


additional_map3_0 (Conv2D) (16, 128, 3, 10) 32896 additional_map2_1_relu[0][0]


additional_map3_0_relu (ReLU) (16, 128, 3, 10) 0 additional_map3_0[0][0]


additional_map3_1 (Conv2D) (16, 256, 2, 5) 295168 additional_map3_0_relu[0][0]


additional_map3_1_relu (ReLU) (16, 256, 2, 5) 0 additional_map3_1[0][0]


ssd_conf_0 (Conv2D) (16, 30, 48, 156) 34590 activation_26[0][0]


ssd_conf_1 (Conv2D) (16, 30, 24, 78) 138270 activation_34[0][0]


ssd_conf_2 (Conv2D) (16, 30, 12, 39) 138270 additional_map0_1_relu[0][0]


ssd_conf_3 (Conv2D) (16, 30, 6, 20) 69150 additional_map1_1_relu[0][0]


ssd_conf_4 (Conv2D) (16, 30, 3, 10) 69150 additional_map2_1_relu[0][0]


ssd_conf_5 (Conv2D) (16, 30, 2, 5) 69150 additional_map3_1_relu[0][0]


permute_13 (Permute) (16, 48, 156, 30) 0 ssd_conf_0[0][0]


permute_15 (Permute) (16, 24, 78, 30) 0 ssd_conf_1[0][0]


permute_17 (Permute) (16, 12, 39, 30) 0 ssd_conf_2[0][0]


permute_19 (Permute) (16, 6, 20, 30) 0 ssd_conf_3[0][0]


permute_21 (Permute) (16, 3, 10, 30) 0 ssd_conf_4[0][0]


permute_23 (Permute) (16, 2, 5, 30) 0 ssd_conf_5[0][0]


ssd_loc_0 (Conv2D) (16, 24, 48, 156) 27672 activation_26[0][0]


ssd_loc_1 (Conv2D) (16, 24, 24, 78) 110616 activation_34[0][0]


ssd_loc_2 (Conv2D) (16, 24, 12, 39) 110616 additional_map0_1_relu[0][0]


ssd_loc_3 (Conv2D) (16, 24, 6, 20) 55320 additional_map1_1_relu[0][0]


ssd_loc_4 (Conv2D) (16, 24, 3, 10) 55320 additional_map2_1_relu[0][0]


ssd_loc_5 (Conv2D) (16, 24, 2, 5) 55320 additional_map3_1_relu[0][0]


conf_reshape_0 (Reshape) (16, 44928, 1, 5) 0 permute_13[0][0]


conf_reshape_1 (Reshape) (16, 11232, 1, 5) 0 permute_15[0][0]


conf_reshape_2 (Reshape) (16, 2808, 1, 5) 0 permute_17[0][0]


conf_reshape_3 (Reshape) (16, 720, 1, 5) 0 permute_19[0][0]


conf_reshape_4 (Reshape) (16, 180, 1, 5) 0 permute_21[0][0]


conf_reshape_5 (Reshape) (16, 60, 1, 5) 0 permute_23[0][0]


permute_14 (Permute) (16, 48, 156, 24) 0 ssd_loc_0[0][0]


permute_16 (Permute) (16, 24, 78, 24) 0 ssd_loc_1[0][0]


permute_18 (Permute) (16, 12, 39, 24) 0 ssd_loc_2[0][0]


permute_20 (Permute) (16, 6, 20, 24) 0 ssd_loc_3[0][0]


permute_22 (Permute) (16, 3, 10, 24) 0 ssd_loc_4[0][0]


permute_24 (Permute) (16, 2, 5, 24) 0 ssd_loc_5[0][0]


ssd_anchor_0 (AnchorBoxes) (16, 7488, 6, 8) 0 ssd_loc_0[0][0]


ssd_anchor_1 (AnchorBoxes) (16, 1872, 6, 8) 0 ssd_loc_1[0][0]


ssd_anchor_2 (AnchorBoxes) (16, 468, 6, 8) 0 ssd_loc_2[0][0]


ssd_anchor_3 (AnchorBoxes) (16, 120, 6, 8) 0 ssd_loc_3[0][0]


ssd_anchor_4 (AnchorBoxes) (16, 30, 6, 8) 0 ssd_loc_4[0][0]


ssd_anchor_5 (AnchorBoxes) (16, 10, 6, 8) 0 ssd_loc_5[0][0]


mbox_conf (Concatenate) (16, 59928, 1, 5) 0 conf_reshape_0[0][0]
conf_reshape_1[0][0]
conf_reshape_2[0][0]
conf_reshape_3[0][0]
conf_reshape_4[0][0]
conf_reshape_5[0][0]


loc_reshape_0 (Reshape) (16, 44928, 1, 4) 0 permute_14[0][0]


loc_reshape_1 (Reshape) (16, 11232, 1, 4) 0 permute_16[0][0]


loc_reshape_2 (Reshape) (16, 2808, 1, 4) 0 permute_18[0][0]


loc_reshape_3 (Reshape) (16, 720, 1, 4) 0 permute_20[0][0]


loc_reshape_4 (Reshape) (16, 180, 1, 4) 0 permute_22[0][0]


loc_reshape_5 (Reshape) (16, 60, 1, 4) 0 permute_24[0][0]


anchor_reshape_0 (Reshape) (16, 44928, 1, 8) 0 ssd_anchor_0[0][0]


anchor_reshape_1 (Reshape) (16, 11232, 1, 8) 0 ssd_anchor_1[0][0]


anchor_reshape_2 (Reshape) (16, 2808, 1, 8) 0 ssd_anchor_2[0][0]


anchor_reshape_3 (Reshape) (16, 720, 1, 8) 0 ssd_anchor_3[0][0]


anchor_reshape_4 (Reshape) (16, 180, 1, 8) 0 ssd_anchor_4[0][0]


anchor_reshape_5 (Reshape) (16, 60, 1, 8) 0 ssd_anchor_5[0][0]


mbox_conf_sigmoid (Activation) (16, 59928, 1, 5) 0 mbox_conf[0][0]


mbox_loc (Concatenate) (16, 59928, 1, 4) 0 loc_reshape_0[0][0]
loc_reshape_1[0][0]
loc_reshape_2[0][0]
loc_reshape_3[0][0]
loc_reshape_4[0][0]
loc_reshape_5[0][0]


mbox_priorbox (Concatenate) (16, 59928, 1, 8) 0 anchor_reshape_0[0][0]
anchor_reshape_1[0][0]
anchor_reshape_2[0][0]
anchor_reshape_3[0][0]
anchor_reshape_4[0][0]
anchor_reshape_5[0][0]


concatenate_2 (Concatenate) (16, 59928, 1, 17) 0 mbox_conf_sigmoid[0][0]
mbox_loc[0][0]
mbox_priorbox[0][0]


ssd_predictions (Reshape) (16, 59928, 17) 0 concatenate_2[0][0]

Total params: 20,710,468
Trainable params: 20,689,348
Non-trainable params: 21,120


2019-12-14 17:13:41,085 [INFO] iva.ssd.scripts.train: Number of images in the training dataset: 5600
2019-12-14 17:13:41,086 [INFO] iva.ssd.scripts.train: Number of images in the validation dataset: 1400
WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
2019-12-14 17:13:41,568 [WARNING] tensorflow: From /usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
Epoch 1/180
2019-12-14 17:14:01.980712: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_16/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-14 17:14:01.980803: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_16/CornerCoordToCentroids/StridedReplace/strided_slice. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?,1]
2019-12-14 17:14:01.980857: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_16/CornerCoordToCentroids/strided_slice_4. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-14 17:14:01.980895: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_16/CornerCoordToCentroids/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-14 17:14:01.980942: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_16/CornerCoordToCentroids/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-14 17:14:01.981005: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_16/CornerCoordToCentroids/strided_slice_7. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-14 17:14:01.981038: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_16/CornerCoordToCentroids/StridedReplace/strided_slice_1. Error: ValidateStridedSliceOp returned partial shapes [?,0] and [?,0]
2019-12-14 17:14:01.981074: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_16/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,4] and [?,4]
2019-12-14 17:14:01.981154: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_17/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-14 17:14:01.981195: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_17/CornerCoordToCentroids/StridedReplace/strided_slice. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?,1]
2019-12-14 17:14:01.981227: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_17/CornerCoordToCentroids/strided_slice_4. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-14 17:14:01.981263: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_17/CornerCoordToCentroids/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-14 17:14:01.981300: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_17/CornerCoordToCentroids/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-14 17:14:01.981352: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_17/CornerCoordToCentroids/strided_slice_7. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-14 17:14:01.981397: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_17/CornerCoordToCentroids/StridedReplace/strided_slice_1. Error: ValidateStridedSliceOp returned partial shapes [?,0] and [?,0]
2019-12-14 17:14:01.981440: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_17/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,4] and [?,4]
2019-12-14 17:14:01.981488: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_18/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-14 17:14:01.981522: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_18/CornerCoordToCentroids/StridedReplace/strided_slice. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?,1]
2019-12-14 17:14:01.981557: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_18/CornerCoordToCentroids/strided_slice_4. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-14 17:14:01.981594: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_18/CornerCoordToCentroids/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-14 17:14:01.981631: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_18/CornerCoordToCentroids/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-14 17:14:01.981668: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_18/CornerCoordToCentroids/strided_slice_7. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-14 17:14:01.981702: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_18/CornerCoordToCentroids/StridedReplace/strided_slice_1. Error: ValidateStridedSliceOp returned partial shapes [?,0] and [?,0]
2019-12-14 17:14:01.981740: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_18/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,4] and [?,4]
2019-12-14 17:14:01.981788: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_19/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-14 17:14:01.981823: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_19/CornerCoordToCentroids/StridedReplace/strided_slice. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?,1]
2019-12-14 17:14:01.981858: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_19/CornerCoordToCentroids/strided_slice_4. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-14 17:14:01.981897: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_19/CornerCoordToCentroids/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-14 17:14:01.981945: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_19/CornerCoordToCentroids/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-14 17:14:01.981985: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_19/CornerCoordToCentroids/strided_slice_7. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-14 17:14:01.982022: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_19/CornerCoordToCentroids/StridedReplace/strided_slice_1. Error: ValidateStridedSliceOp returned partial shapes [?,0] and [?,0]
2019-12-14 17:14:01.982061: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_19/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,4] and [?,4]
2019-12-14 17:14:01.982113: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_20/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-14 17:14:01.982151: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_20/CornerCoordToCentroids/StridedReplace/strided_slice. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?,1]
2019-12-14 17:14:01.982190: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_20/CornerCoordToCentroids/strided_slice_4. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-14 17:14:01.982229: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_20/CornerCoordToCentroids/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-14 17:14:01.982269: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_20/CornerCoordToCentroids/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-14 17:14:01.982307: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_20/CornerCoordToCentroids/strided_slice_7. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-14 17:14:01.982345: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_20/CornerCoordToCentroids/StridedReplace/strided_slice_1. Error: ValidateStridedSliceOp returned partial shapes [?,0] and [?,0]
2019-12-14 17:14:01.982392: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_20/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,4] and [?,4]
2019-12-14 17:14:01.982453: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_21/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-14 17:14:01.982495: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_21/CornerCoordToCentroids/StridedReplace/strided_slice. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?,1]
2019-12-14 17:14:01.982538: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_21/CornerCoordToCentroids/strided_slice_4. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-14 17:14:01.982588: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_21/CornerCoordToCentroids/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-14 17:14:01.982628: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_21/CornerCoordToCentroids/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-14 17:14:01.982667: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_21/CornerCoordToCentroids/strided_slice_7. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-14 17:14:01.982705: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_21/CornerCoordToCentroids/StridedReplace/strided_slice_1. Error: ValidateStridedSliceOp returned partial shapes [?,0] and [?,0]
2019-12-14 17:14:01.982746: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_21/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,4] and [?,4]
2019-12-14 17:14:01.982793: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_22/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-14 17:14:01.982854: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_22/CornerCoordToCentroids/StridedReplace/strided_slice. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?,1]
2019-12-14 17:14:01.982900: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_22/CornerCoordToCentroids/strided_slice_4. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-14 17:14:01.982955: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_22/CornerCoordToCentroids/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-14 17:14:01.982997: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_22/CornerCoordToCentroids/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-14 17:14:01.983032: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_22/CornerCoordToCentroids/strided_slice_7. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-14 17:14:01.983067: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_22/CornerCoordToCentroids/StridedReplace/strided_slice_1. Error: ValidateStridedSliceOp returned partial shapes [?,0] and [?,0]
2019-12-14 17:14:01.983104: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_22/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,4] and [?,4]
2019-12-14 17:14:01.983151: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_23/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-14 17:14:01.983185: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_23/CornerCoordToCentroids/StridedReplace/strided_slice. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?,1]
2019-12-14 17:14:01.983225: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_23/CornerCoordToCentroids/strided_slice_4. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-14 17:14:01.983267: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_23/CornerCoordToCentroids/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-14 17:14:01.983307: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_23/CornerCoordToCentroids/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-14 17:14:01.983344: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_23/CornerCoordToCentroids/strided_slice_7. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-14 17:14:01.983391: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_23/CornerCoordToCentroids/StridedReplace/strided_slice_1. Error: ValidateStridedSliceOp returned partial shapes [?,0] and [?,0]
2019-12-14 17:14:01.983426: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_23/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,4] and [?,4]
2019-12-14 17:14:01.983475: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_24/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-14 17:14:01.983518: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_24/CornerCoordToCentroids/StridedReplace/strided_slice. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?,1]
2019-12-14 17:14:01.983556: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_24/CornerCoordToCentroids/strided_slice_4. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-12-14 17:14:01.983596: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAx

Hi 010akv,
This warning is not harmful.
And during the retraining, your model will not be trained from scratch.
Because I see you set “-m” option in the command line. It is the pre-trained model during the retraining.

-m  /workspace/tlt/train-and-test/pkg_watch_experiment_dir_unpruned/weights/ssd_resnet18_epoch_180.tlt

Refer to https://docs.nvidia.com/metropolis/TLT/tlt-getting-started-guide/index.html#training_ssd_model

Optional arguments:

--gpus num_gpus: Number of GPUs to use and processes to launch for training. The default = 1.
-m, --resume_model_weights: Path to a pre-trained model or model to continue training.
--initial_epoch: Epoch number to resume from.
-h, --help: Show this help message and exit.

Thanks!

Hi
I have already trained the model and performed the tlt-prune operation on it.But when I retrain, I have some problems.
This is my logs:

Using TensorFlow backend.
2019-12-17 07:05:33.756084: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-12-17 07:05:33.869241: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-12-17 07:05:33.869983: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x5770bd0 executing computations on platform CUDA. Devices:
2019-12-17 07:05:33.870031: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): Tesla P4, Compute Capability 6.1
2019-12-17 07:05:33.872881: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2494220000 Hz
2019-12-17 07:05:33.873331: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x57d8a80 executing computations on platform Host. Devices:
2019-12-17 07:05:33.873376: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): ,
2019-12-17 07:05:33.873574: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: Tesla P4 major: 6 minor: 1 memoryClockRate(GHz): 1.1135
pciBusID: 0000:00:07.0
totalMemory: 7.43GiB freeMemory: 5.97GiB
2019-12-17 07:05:33.873610: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-12-17 07:05:33.874461: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-12-17 07:05:33.874496: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2019-12-17 07:05:33.874515: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2019-12-17 07:05:33.874637: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5812 MB memory) -> physical GPU (device: 0, name: Tesla P4, pci bus id: 0000:00:07.0, compute capability: 6.1)
2019-12-17 07:05:33,876 [INFO] iva.detectnet_v2.scripts.train: Loading experiment spec at retrain_hat_person.txt.
2019-12-17 07:05:33,877 [INFO] iva.detectnet_v2.spec_handler.spec_loader: Merging specification from retrain_hat_person.txt
WARNING:tensorflow:From ./detectnet_v2/dataloader/utilities.py:114: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version.
Instructions for updating:
Use eager execution and:
tf.data.TFRecordDataset(path)
2019-12-17 07:05:33,884 [WARNING] tensorflow: From ./detectnet_v2/dataloader/utilities.py:114: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version.
Instructions for updating:
Use eager execution and:
tf.data.TFRecordDataset(path)
WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
2019-12-17 07:05:34,012 [WARNING] tensorflow: From /usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/horovod/tensorflow/init.py:91: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Deprecated in favor of operator or tf.math.divide.
2019-12-17 07:05:34,028 [WARNING] tensorflow: From /usr/local/lib/python2.7/dist-packages/horovod/tensorflow/init.py:91: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Deprecated in favor of operator or tf.math.divide.
/usr/local/lib/python2.7/dist-packages/keras/engine/saving.py:292: UserWarning: No training configuration found in save file: the model was not compiled. Compile it manually.
warnings.warn('No training configuration found in save file: ’


Layer (type) Output Shape Param # Connected to

input_1 (InputLayer) (None, 3, 640, 649) 0


conv1 (Conv2D) (None, 64, 320, 325) 9472 input_1[0][0]


bn_conv1 (BatchNormalization) (None, 64, 320, 325) 256 conv1[0][0]


activation_1 (Activation) (None, 64, 320, 325) 0 bn_conv1[0][0]


block_1a_conv_1 (Conv2D) (None, 64, 160, 163) 36928 activation_1[0][0]


block_1a_bn_1 (BatchNormalizati (None, 64, 160, 163) 256 block_1a_conv_1[0][0]


activation_2 (Activation) (None, 64, 160, 163) 0 block_1a_bn_1[0][0]


block_1a_conv_2 (Conv2D) (None, 64, 160, 163) 36928 activation_2[0][0]


block_1a_conv_shortcut (Conv2D) (None, 64, 160, 163) 4160 activation_1[0][0]


block_1a_bn_2 (BatchNormalizati (None, 64, 160, 163) 256 block_1a_conv_2[0][0]


block_1a_bn_shortcut (BatchNorm (None, 64, 160, 163) 256 block_1a_conv_shortcut[0][0]


add_1 (Add) (None, 64, 160, 163) 0 block_1a_bn_2[0][0]
block_1a_bn_shortcut[0][0]


activation_3 (Activation) (None, 64, 160, 163) 0 add_1[0][0]


block_1b_conv_1 (Conv2D) (None, 64, 160, 163) 36928 activation_3[0][0]


block_1b_bn_1 (BatchNormalizati (None, 64, 160, 163) 256 block_1b_conv_1[0][0]


activation_4 (Activation) (None, 64, 160, 163) 0 block_1b_bn_1[0][0]


block_1b_conv_2 (Conv2D) (None, 64, 160, 163) 36928 activation_4[0][0]


block_1b_bn_2 (BatchNormalizati (None, 64, 160, 163) 256 block_1b_conv_2[0][0]


add_2 (Add) (None, 64, 160, 163) 0 block_1b_bn_2[0][0]
activation_3[0][0]


activation_5 (Activation) (None, 64, 160, 163) 0 add_2[0][0]


block_2a_conv_1 (Conv2D) (None, 128, 80, 82) 73856 activation_5[0][0]


block_2a_bn_1 (BatchNormalizati (None, 128, 80, 82) 512 block_2a_conv_1[0][0]


activation_6 (Activation) (None, 128, 80, 82) 0 block_2a_bn_1[0][0]


block_2a_conv_2 (Conv2D) (None, 128, 80, 82) 147584 activation_6[0][0]


block_2a_conv_shortcut (Conv2D) (None, 128, 80, 82) 8320 activation_5[0][0]


block_2a_bn_2 (BatchNormalizati (None, 128, 80, 82) 512 block_2a_conv_2[0][0]


block_2a_bn_shortcut (BatchNorm (None, 128, 80, 82) 512 block_2a_conv_shortcut[0][0]


add_3 (Add) (None, 128, 80, 82) 0 block_2a_bn_2[0][0]
block_2a_bn_shortcut[0][0]


activation_7 (Activation) (None, 128, 80, 82) 0 add_3[0][0]


block_2b_conv_1 (Conv2D) (None, 128, 80, 82) 147584 activation_7[0][0]


block_2b_bn_1 (BatchNormalizati (None, 128, 80, 82) 512 block_2b_conv_1[0][0]


activation_8 (Activation) (None, 128, 80, 82) 0 block_2b_bn_1[0][0]


block_2b_conv_2 (Conv2D) (None, 128, 80, 82) 147584 activation_8[0][0]


block_2b_bn_2 (BatchNormalizati (None, 128, 80, 82) 512 block_2b_conv_2[0][0]


add_4 (Add) (None, 128, 80, 82) 0 block_2b_bn_2[0][0]
activation_7[0][0]


activation_9 (Activation) (None, 128, 80, 82) 0 add_4[0][0]


block_3a_conv_1 (Conv2D) (None, 184, 40, 41) 212152 activation_9[0][0]


block_3a_bn_1 (BatchNormalizati (None, 184, 40, 41) 736 block_3a_conv_1[0][0]


activation_10 (Activation) (None, 184, 40, 41) 0 block_3a_bn_1[0][0]


block_3a_conv_2 (Conv2D) (None, 176, 40, 41) 291632 activation_10[0][0]


block_3a_conv_shortcut (Conv2D) (None, 176, 40, 41) 22704 activation_9[0][0]


block_3a_bn_2 (BatchNormalizati (None, 176, 40, 41) 704 block_3a_conv_2[0][0]


block_3a_bn_shortcut (BatchNorm (None, 176, 40, 41) 704 block_3a_conv_shortcut[0][0]


add_5 (Add) (None, 176, 40, 41) 0 block_3a_bn_2[0][0]
block_3a_bn_shortcut[0][0]


activation_11 (Activation) (None, 176, 40, 41) 0 add_5[0][0]


block_3b_conv_1 (Conv2D) (None, 168, 40, 41) 266280 activation_11[0][0]


block_3b_bn_1 (BatchNormalizati (None, 168, 40, 41) 672 block_3b_conv_1[0][0]


activation_12 (Activation) (None, 168, 40, 41) 0 block_3b_bn_1[0][0]


block_3b_conv_2 (Conv2D) (None, 176, 40, 41) 266288 activation_12[0][0]


block_3b_bn_2 (BatchNormalizati (None, 176, 40, 41) 704 block_3b_conv_2[0][0]


add_6 (Add) (None, 176, 40, 41) 0 block_3b_bn_2[0][0]
activation_11[0][0]


activation_13 (Activation) (None, 176, 40, 41) 0 add_6[0][0]


block_4a_conv_1 (Conv2D) (None, 136, 40, 41) 215560 activation_13[0][0]


block_4a_bn_1 (BatchNormalizati (None, 136, 40, 41) 544 block_4a_conv_1[0][0]


activation_14 (Activation) (None, 136, 40, 41) 0 block_4a_bn_1[0][0]


block_4a_conv_2 (Conv2D) (None, 216, 40, 41) 264600 activation_14[0][0]


block_4a_conv_shortcut (Conv2D) (None, 216, 40, 41) 38232 activation_13[0][0]


block_4a_bn_2 (BatchNormalizati (None, 216, 40, 41) 864 block_4a_conv_2[0][0]


block_4a_bn_shortcut (BatchNorm (None, 216, 40, 41) 864 block_4a_conv_shortcut[0][0]


add_7 (Add) (None, 216, 40, 41) 0 block_4a_bn_2[0][0]
block_4a_bn_shortcut[0][0]


activation_15 (Activation) (None, 216, 40, 41) 0 add_7[0][0]


block_4b_conv_1 (Conv2D) (None, 88, 40, 41) 171160 activation_15[0][0]


block_4b_bn_1 (BatchNormalizati (None, 88, 40, 41) 352 block_4b_conv_1[0][0]


activation_16 (Activation) (None, 88, 40, 41) 0 block_4b_bn_1[0][0]


block_4b_conv_2 (Conv2D) (None, 216, 40, 41) 171288 activation_16[0][0]


block_4b_bn_2 (BatchNormalizati (None, 216, 40, 41) 864 block_4b_conv_2[0][0]


add_8 (Add) (None, 216, 40, 41) 0 block_4b_bn_2[0][0]
activation_15[0][0]


activation_17 (Activation) (None, 216, 40, 41) 0 add_8[0][0]


output_bbox (Conv2D) (None, 8, 40, 41) 1736 activation_17[0][0]


output_cov (Conv2D) (None, 2, 40, 41) 434 activation_17[0][0]

Total params: 2,619,442
Trainable params: 2,613,890
Non-trainable params: 5,552


target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
2019-12-17 07:05:42,155 [INFO] iva.detectnet_v2.scripts.train: Found 6512 samples in training set
Traceback (most recent call last):
File “/usr/local/bin/tlt-train-g1”, line 10, in
sys.exit(main())
File “./common/magnet_train.py”, line 37, in main
File “</usr/local/lib/python2.7/dist-packages/decorator.pyc:decorator-gen-2>”, line 2, in main
File “./detectnet_v2/utilities/timer.py”, line 46, in wrapped_fn
File “./detectnet_v2/scripts/train.py”, line 632, in main
File “./detectnet_v2/scripts/train.py”, line 556, in run_experiment
File “./detectnet_v2/scripts/train.py”, line 466, in train_gridbox
File “./detectnet_v2/scripts/train.py”, line 320, in build_training_graph
File “./detectnet_v2/model/detectnet_model.py”, line 480, in build_training_graph
File “./detectnet_v2/model/detectnet_model.py”, line 243, in predictions_to_dict
File “./detectnet_v2/objectives/base_objective.py”, line 97, in reshape_output
File “/usr/local/lib/python2.7/dist-packages/keras/engine/base_layer.py”, line 457, in call
output = self.call(inputs, **kwargs)
File “/usr/local/lib/python2.7/dist-packages/keras/layers/core.py”, line 401, in call
return K.reshape(inputs, (K.shape(inputs)[0],) + self.target_shape)
File “/usr/local/lib/python2.7/dist-packages/keras/backend/tensorflow_backend.py”, line 1969, in reshape
return tf.reshape(x, shape)
File “/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_array_ops.py”, line 7179, in reshape
“Reshape”, tensor=tensor, shape=shape, name=name)
File “/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py”, line 788, in _apply_op_helper
op_def=op_def)
File “/usr/local/lib/python2.7/dist-packages/tensorflow/python/util/deprecation.py”, line 507, in new_func
return func(*args, **kwargs)
File “/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py”, line 3300, in create_op
op_def=op_def)
File “/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py”, line 1823, in init
control_input_ops)
File “/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py”, line 1662, in _create_c_op
raise ValueError(str(e))
ValueError: Cannot reshape a tensor with 102400 elements to shape [8,2,4,40,41] (104960 elements) for ‘reshape_1_1/Reshape’ (op: ‘Reshape’) with input shapes: [8,8,40,40], [5] and with input tensors computed as partial shapes: input[1] = [8,2,4,40,41].

Hi 17521754134,
Your problem is not the same as original issue in this topic.
Could you please create a new topic under TLT forum? Thanks.