Unable to train SSD-Resnet-18

Accelerated Computing Intelligent Video Analytics TAO Toolkit
1

Hi,
Here is the error that was thrown from the training command
Using TensorFlow backend.
2019-11-06 12:16:29.791151: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-11-06 12:16:29.936043: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-11-06 12:16:29.937212: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x62df5a0 executing computations on platform CUDA. Devices:
2019-11-06 12:16:29.937235: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): GeForce GTX 1660 Ti, Compute Capability 7.5
2019-11-06 12:16:29.950235: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2904000000 Hz
2019-11-06 12:16:29.950636: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x6347be0 executing computations on platform Host. Devices:
2019-11-06 12:16:29.950653: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): ,
2019-11-06 12:16:29.950739: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: GeForce GTX 1660 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.77
pciBusID: 0000:01:00.0
totalMemory: 5.80GiB freeMemory: 5.36GiB
2019-11-06 12:16:29.950754: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-11-06 12:16:29.952006: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-11-06 12:16:29.952019: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2019-11-06 12:16:29.952025: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2019-11-06 12:16:29.952073: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5189 MB memory) → physical GPU (device: 0, name: GeForce GTX 1660 Ti, pci bus id: 0000:01:00.0, compute capability: 7.5)
2019-11-06 12:16:29,953 [INFO] iva.ssd.scripts.train: Loading experiment spec at /workspace/examples/ssd/specs/ssd_train_resnet18_kitti.txt.
2019-11-06 12:16:29,954 [INFO] /usr/local/lib/python2.7/dist-packages/iva/ssd/utils/spec_loader.pyc: Merging specification from /workspace/examples/ssd/specs/ssd_train_resnet18_kitti.txt
WARNING:tensorflow:From ./detectnet_v2/dataloader/utilities.py:114: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version.
Instructions for updating:
Use eager execution and:
tf.data.TFRecordDataset(path)
2019-11-06 12:16:29,957 [WARNING] tensorflow: From ./detectnet_v2/dataloader/utilities.py:114: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version.
Instructions for updating:
Use eager execution and:
tf.data.TFRecordDataset(path)
WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/python/data/ops/iterator_ops.py:358: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
2019-11-06 12:16:29,999 [WARNING] tensorflow: From /usr/local/lib/python2.7/dist-packages/tensorflow/python/data/ops/iterator_ops.py:358: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
Traceback (most recent call last):
File “/usr/local/bin/tlt-train-g1”, line 10, in
sys.exit(main())
File “./common/magnet_train.py”, line 33, in main
File “./ssd/scripts/train.py”, line 301, in main
File “./ssd/scripts/train.py”, line 167, in run_experiment
File “./ssd/builders/model_builder.py”, line 75, in build
File “./ssd/builders/inputs_builder.py”, line 51, in init
File “./detectnet_v2/dataloader/default_dataloader.py”, line 198, in get_dataset_tensors
File “./detectnet_v2/dataloader/utilities.py”, line 181, in extract_tfrecords_features
StopIteration

Here is the spec file used for training
Spec File:

ssd_config {
aspect_ratios_global: “[1.0, 2.0, 0.5, 3.0, 1.0/3.0]”
scales: “[0.05, 0.1, 0.25, 0.4, 0.55, 0.7, 0.85]”
two_boxes_for_ar1: true
clip_boxes: false
loss_loc_weight: 0.8
focal_loss_alpha: 0.25
focal_loss_gamma: 2.0
variances: “[0.1, 0.1, 0.2, 0.2]”
arch: “resnet18”
freeze_bn: false
freeze_blocks: 0
}
training_config {
batch_size_per_gpu: 16
num_epochs: 180
learning_rate {
soft_start_annealing_schedule {
min_learning_rate: 5e-5
max_learning_rate: 2e-2
soft_start: 0.1
annealing: 0.3
}
}
regularizer {
type: L1
weight: 3.00000002618e-09
}
}
eval_config {
validation_period_during_training: 10
average_precision_mode: SAMPLE
batch_size: 32
matching_iou_threshold: 0.5
}
nms_config {
confidence_threshold: 0.01
clustering_iou_threshold: 0.6
top_k: 200
}
augmentation_config {
preprocessing {
output_image_width: 1248
output_image_height: 384
output_image_channel: 3
crop_right: 1248
crop_bottom: 384
min_bbox_width: 1.0
min_bbox_height: 1.0
}
spatial_augmentation {
hflip_probability: 0.5
vflip_probability: 0.0
zoom_min: 0.7
zoom_max: 1.8
translate_max_x: 8.0
translate_max_y: 8.0
}
color_augmentation {
hue_rotation_max: 25.0
saturation_shift_max: 0.20000000298
contrast_scale_max: 0.10000000149
contrast_center: 0.5
}
}
dataset_config {
data_sources: {
tfrecords_path: “/workspace/tlt-experiments/tfrecords/kitti_trainval*”
image_directory_path: “/workspace/tlt-experiments/data/training”
}
image_extension: “jpg”
target_class_mapping {
key: “Class-1”
value: “Class-1”
}
target_class_mapping {
key: “Class-2”
value: “Class-2”
}
target_class_mapping {
key: “Class-3”
value: “Class-3”
}
validation_fold: 0
}

Thanks in Advance

2

Hi psgr,
Firstly, I want to confirm if you generate the tfrecord successfully now. I saw your topic in https://devtalk.nvidia.com/default/topic/1066039/transfer-learning-toolkit/unable-to-generate-tf-records-/

Next, for SSD, the tlt-train tool does not support training on images of multiple resolutions, or resizing images during training. All of the images must be resized offline to the final training size and the corresponding bounding boxes must be scaled accordingly.
See tlt doc Integrating TAO Models into DeepStream — TAO Toolkit 3.22.05 documentation

The size should match your setting in the training spec.

3

I generated the tf records for a small sample of my own data identified few files where the fields are more than 15 and separated using a python script. I’m working on the other half of the data which is clean and in right format I hope.

Here is the python logic I used for verifying

file_list=
for fname in files:
with open(fname) as f:
lines = [line.rstrip(‘\n’) for line in f]
flag = True
for line in lines:
if(len(line.split())==15):
flag=True
else:
flag = False
break
print(fname)
if(flag==True):
file_list.append(fname)

Speaking of SSD in tlt-train all images are 1920x1080p the bounding boxes generated are of same resolution.
Here is the spec file for training I have commented the places where changes are necessary please confirm them and let me know if I had to change them in any other places

ssd_config {
aspect_ratios_global: “[1.0, 2.0, 0.5, 3.0, 1.0/3.0]”
scales: “[0.05, 0.1, 0.25, 0.4, 0.55, 0.7, 0.85]”
two_boxes_for_ar1: true
clip_boxes: false
loss_loc_weight: 0.8
focal_loss_alpha: 0.25
focal_loss_gamma: 2.0
variances: “[0.1, 0.1, 0.2, 0.2]”
arch: “resnet18”
freeze_bn: false
freeze_blocks: 0
}
training_config {
batch_size_per_gpu: 16
num_epochs: 180
learning_rate {
soft_start_annealing_schedule {
min_learning_rate: 5e-5
max_learning_rate: 2e-2
soft_start: 0.1
annealing: 0.3
}
}
regularizer {
type: L1
weight: 3.00000002618e-09
}
}
eval_config {
validation_period_during_training: 10
average_precision_mode: SAMPLE
batch_size: 32
matching_iou_threshold: 0.5
}
nms_config {
confidence_threshold: 0.01
clustering_iou_threshold: 0.6
top_k: 200
}
augmentation_config {
preprocessing {
output_image_width: 1248 # So I shoud change the value here to 1080
output_image_height: 384 # and the value of height to 1920
output_image_channel: 3
crop_right: 1248 # change value to 1080
crop_bottom: 384 # change value to 1920
min_bbox_width: 1.0
min_bbox_height: 1.0
}
spatial_augmentation {
hflip_probability: 0.5
vflip_probability: 0.0
zoom_min: 0.7
zoom_max: 1.8
translate_max_x: 8.0
translate_max_y: 8.0
}
color_augmentation {
hue_rotation_max: 25.0
saturation_shift_max: 0.20000000298
contrast_scale_max: 0.10000000149
contrast_center: 0.5
}
}
dataset_config {
data_sources: {
tfrecords_path: “/workspace/tlt-experiments/tfrecords/kitti_trainval*”
image_directory_path: “/workspace/tlt-experiments/data/training”
}
image_extension: “jpg”
target_class_mapping {
key: “Class-1”
value: “Class-1”
}
target_class_mapping {
key: “Class-2”
value: “Class-2”
}
target_class_mapping {
key: “Class-3”
value: “Class-3”
}
validation_fold: 0
}

But still I’m facing the same old error after changing the parameters which are commented in the above spec file.

Or Do I actually need to resize my image resolution to 1248x384 and then annotate my data in order to train it

Thanks in Advance

4

Hi psgr,

  1. Please make sure all your label file have KITTI format . Just like below example. It has 15 fields.
car 0.00 0 0.00 587.01 173.33 614.12 200.12 0.00 0.00 0.00 0.00 0.00 0.00 0.00

  1. If you set 1920x1080 in your spec, please resize all your training images to 1920x1080.

  2. And the corresponding bounding boxes must be scaled accordingly. That means your KITTI format label file’s xmin/ymin/xmax/ymax should exactly match the bbox of a 1920x1080 image.

  3. Your modification is reverse.
    output_image_width should change to 1920
    output_image_height should change to 1080
    crop_right should change value to 1920
    crop_bottom should change to 1080

5

Here is a sample out of my kitti annotation

Class-1 0.00 0 0.0 1677.00 587.00 1721.00 657.00 0.0 0.0 0.0 0.0 0.0 0.0 0.0

All my training images are 1980x1080p resolution and still I’m getting the same error

6

I go through the issue you mentioned originally, and think there is something problem in your tfrecords.
Could you run “tlt-dataset-convert” again to generate the tfrecord files?
Please share me the log.

Then,Could you refer to
Tensor reshape error when evaluating a Detectnet_v2 model - TAO Toolkit - NVIDIA Developer Forums or
Tensor reshape error when evaluating a Detectnet_v2 model - TAO Toolkit - NVIDIA Developer Forums
to check your tfrecord files?

7

Here is Log of the TF-records generated.

Using TensorFlow backend.
2019-11-07 09:47:21,053 - iva.detectnet_v2.dataio.build_converter - INFO - Instantiating a kitti converter
2019-11-07 09:47:21,053 - iva.detectnet_v2.dataio.kitti_converter_lib - INFO - Num images in
Train: 5 Val: 0
2019-11-07 09:47:21,053 - iva.detectnet_v2.dataio.kitti_converter_lib - INFO - Validation data in partition 0. Hence, while choosing the validationset during training choose validation_fold 0.
2019-11-07 09:47:21,053 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 0
2019-11-07 09:47:21,053 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 1
2019-11-07 09:47:21,054 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 2
2019-11-07 09:47:21,054 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 3
2019-11-07 09:47:21,054 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 4
2019-11-07 09:47:21,054 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 5
2019-11-07 09:47:21,054 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 6
2019-11-07 09:47:21,054 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 7
2019-11-07 09:47:21,054 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 8
2019-11-07 09:47:21,054 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 9
2019-11-07 09:47:21,054 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO -
Wrote the following numbers of objects:

2019-11-07 09:47:21,054 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 0
2019-11-07 09:47:21,054 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 1
2019-11-07 09:47:21,054 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 2
2019-11-07 09:47:21,054 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 3
2019-11-07 09:47:21,055 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 4
2019-11-07 09:47:21,055 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 5
2019-11-07 09:47:21,055 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 6
2019-11-07 09:47:21,055 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 7
2019-11-07 09:47:21,055 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 8
2019-11-07 09:47:21,055 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 9
/usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:266: VisibleDeprecationWarning: Reading unicode strings without specifying the encoding argument is deprecated. Set the encoding, use None for the system default.
2019-11-07 09:47:21,061 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO -
Wrote the following numbers of objects:
Class-1: 7
Class-2: 1
Class-3: 2

2019-11-07 09:47:21,061 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Cumulative object statistics
2019-11-07 09:47:21,061 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO -
Wrote the following numbers of objects:
Class-1: 7
Class-2: 1
Class-3: 2

2019-11-07 09:47:21,061 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Class map.
Label in GT: Label in tfrecords file
Class-1: Class-1
Class-2: Class-2
Class-3: Class-3
For the dataset_config in the experiment_spec, please use labels in the tfrecords file, while writing the classmap.

2019-11-07 09:47:21,061 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Tfrecords generation complete.

8

From your log, totally 10 images, right?
Could you check your tfrecord file or run tlt-train again?

If possible, could you please send me the 10 images along with their label.txt to me?
I want to cross check on my side.

9

I checked the tfrecords using the script in the link provided :

I used the following command

python3 inspect_tfrecords.py --tfrecords_dir tlt-experiments/tfrecords --expected_width 1920 --expected_height 1080

Here is the output of the script
WARNING:tensorflow:From inspect_tfrecords.py:40: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version.
Instructions for updating:
Use eager execution and:
tf.data.TFRecordDataset(path)
All images found in tlt-experiments/tfrecords/kitti_trainval-fold-001-of-002-shard-00006-of-00010 have expected dimensions
All images found in tlt-experiments/tfrecords/kitti_trainval-fold-001-of-002-shard-00004-of-00010 have expected dimensions
All images found in tlt-experiments/tfrecords/kitti_trainval-fold-000-of-002-shard-00009-of-00010 have expected dimensions
All images found in tlt-experiments/tfrecords/kitti_trainval-fold-000-of-002-shard-00000-of-00010 have expected dimensions
All images found in tlt-experiments/tfrecords/kitti_trainval-fold-000-of-002-shard-00008-of-00010 have expected dimensions
All images found in tlt-experiments/tfrecords/kitti_trainval-fold-001-of-002-shard-00008-of-00010 have expected dimensions
All images found in tlt-experiments/tfrecords/kitti_trainval-fold-000-of-002-shard-00002-of-00010 have expected dimensions
All images found in tlt-experiments/tfrecords/kitti_trainval-fold-001-of-002-shard-00000-of-00010 have expected dimensions
All images found in tlt-experiments/tfrecords/kitti_trainval-fold-001-of-002-shard-00002-of-00010 have expected dimensions
All images found in tlt-experiments/tfrecords/kitti_trainval-fold-000-of-002-shard-00005-of-00010 have expected dimensions
All images found in tlt-experiments/tfrecords/kitti_trainval-fold-000-of-002-shard-00001-of-00010 have expected dimensions
All images found in tlt-experiments/tfrecords/kitti_trainval-fold-001-of-002-shard-00003-of-00010 have expected dimensions
All images found in tlt-experiments/tfrecords/kitti_trainval-fold-000-of-002-shard-00004-of-00010 have expected dimensions
All images found in tlt-experiments/tfrecords/kitti_trainval-fold-001-of-002-shard-00009-of-00010 have expected dimensions
All images found in tlt-experiments/tfrecords/kitti_trainval-fold-000-of-002-shard-00006-of-00010 have expected dimensions
All images found in tlt-experiments/tfrecords/kitti_trainval-fold-001-of-002-shard-00001-of-00010 have expected dimensions
All images found in tlt-experiments/tfrecords/kitti_trainval-fold-001-of-002-shard-00007-of-00010 have expected dimensions
All images found in tlt-experiments/tfrecords/kitti_trainval-fold-001-of-002-shard-00005-of-00010 have expected dimensions
All images found in tlt-experiments/tfrecords/kitti_trainval-fold-000-of-002-shard-00003-of-00010 have expected dimensions
All images found in tlt-experiments/tfrecords/kitti_trainval-fold-000-of-002-shard-00007-of-00010 have expected dimensions

10

Hi,
I fixed all my annotations(with 16 feilds) with a python script and generated my tfrfecords with all my annotations( unlike the above scenario just few ).

I ran my inspect_tfrecords.py script for cross verifying them

WARNING:tensorflow:From inspect_tfrecords.py:40: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version.
Instructions for updating:
Use eager execution and:
tf.data.TFRecordDataset(path)
All images found in tlt-experiments/tfrecords/kitti_trainval-fold-001-of-002-shard-00006-of-00010 have expected dimensions
All images found in tlt-experiments/tfrecords/kitti_trainval-fold-001-of-002-shard-00004-of-00010 have expected dimensions
All images found in tlt-experiments/tfrecords/kitti_trainval-fold-000-of-002-shard-00009-of-00010 have expected dimensions
All images found in tlt-experiments/tfrecords/kitti_trainval-fold-000-of-002-shard-00000-of-00010 have expected dimensions
All images found in tlt-experiments/tfrecords/kitti_trainval-fold-000-of-002-shard-00008-of-00010 have expected dimensions
All images found in tlt-experiments/tfrecords/kitti_trainval-fold-001-of-002-shard-00008-of-00010 have expected dimensions
All images found in tlt-experiments/tfrecords/kitti_trainval-fold-000-of-002-shard-00002-of-00010 have expected dimensions
All images found in tlt-experiments/tfrecords/kitti_trainval-fold-001-of-002-shard-00000-of-00010 have expected dimensions
All images found in tlt-experiments/tfrecords/kitti_trainval-fold-001-of-002-shard-00002-of-00010 have expected dimensions
All images found in tlt-experiments/tfrecords/kitti_trainval-fold-000-of-002-shard-00005-of-00010 have expected dimensions
All images found in tlt-experiments/tfrecords/kitti_trainval-fold-000-of-002-shard-00001-of-00010 have expected dimensions
All images found in tlt-experiments/tfrecords/kitti_trainval-fold-001-of-002-shard-00003-of-00010 have expected dimensions
All images found in tlt-experiments/tfrecords/kitti_trainval-fold-000-of-002-shard-00004-of-00010 have expected dimensions
All images found in tlt-experiments/tfrecords/kitti_trainval-fold-001-of-002-shard-00009-of-00010 have expected dimensions
All images found in tlt-experiments/tfrecords/kitti_trainval-fold-000-of-002-shard-00006-of-00010 have expected dimensions
All images found in tlt-experiments/tfrecords/kitti_trainval-fold-001-of-002-shard-00001-of-00010 have expected dimensions
All images found in tlt-experiments/tfrecords/kitti_trainval-fold-001-of-002-shard-00007-of-00010 have expected dimensions
All images found in tlt-experiments/tfrecords/kitti_trainval-fold-001-of-002-shard-00005-of-00010 have expected dimensions
All images found in tlt-experiments/tfrecords/kitti_trainval-fold-000-of-002-shard-00003-of-00010 have expected dimensions
All images found in tlt-experiments/tfrecords/kitti_trainval-fold-000-of-002-shard-00007-of-00010 have expected dimensions

Now that I have everything in right place I hope tried to my training using the below command

!tlt-train ssd -e $SPECS_DIR/ssd_train_resnet18_kitti.txt
-r $USER_EXPERIMENT_DIR/experiment_dir_unpruned
-k $KEY
-m $USER_EXPERIMENT_DIR/pretrained_resnet18/tlt_resnet18_ssd_v1/resnet18.hdf5

This is the log of the script generated after executing the above command which gave a cudnn error

Using TensorFlow backend.
2019-11-07 16:47:50.392205: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-11-07 16:47:50.488351: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-11-07 16:47:50.488729: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x6035670 executing computations on platform CUDA. Devices:
2019-11-07 16:47:50.488748: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): GeForce GTX 1660 Ti, Compute Capability 7.5
2019-11-07 16:47:50.490405: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2904000000 Hz
2019-11-07 16:47:50.490778: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x609dcb0 executing computations on platform Host. Devices:
2019-11-07 16:47:50.490795: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): ,
2019-11-07 16:47:50.491003: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: GeForce GTX 1660 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.77
pciBusID: 0000:01:00.0
totalMemory: 5.80GiB freeMemory: 5.34GiB
2019-11-07 16:47:50.491022: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-11-07 16:47:50.491446: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-11-07 16:47:50.491459: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2019-11-07 16:47:50.491482: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2019-11-07 16:47:50.491529: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5169 MB memory) → physical GPU (device: 0, name: GeForce GTX 1660 Ti, pci bus id: 0000:01:00.0, compute capability: 7.5)
2019-11-07 16:47:50,493 [INFO] iva.ssd.scripts.train: Loading experiment spec at /workspace/examples/ssd/specs/ssd_train_resnet18_kitti.txt.
2019-11-07 16:47:50,493 [INFO] /usr/local/lib/python2.7/dist-packages/iva/ssd/utils/spec_loader.pyc: Merging specification from /workspace/examples/ssd/specs/ssd_train_resnet18_kitti.txt
WARNING:tensorflow:From ./detectnet_v2/dataloader/utilities.py:114: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version.
Instructions for updating:
Use eager execution and:
tf.data.TFRecordDataset(path)
2019-11-07 16:47:50,496 [WARNING] tensorflow: From ./detectnet_v2/dataloader/utilities.py:114: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version.
Instructions for updating:
Use eager execution and:
tf.data.TFRecordDataset(path)
WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/python/data/ops/iterator_ops.py:358: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
2019-11-07 16:47:50,552 [WARNING] tensorflow: From /usr/local/lib/python2.7/dist-packages/tensorflow/python/data/ops/iterator_ops.py:358: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
WARNING:tensorflow:From ./ssd/utils/tensor_utils.py:31: setdiff1d (from tensorflow.python.ops.array_ops) is deprecated and will be removed after 2018-11-30.
Instructions for updating:
This op will be removed after the deprecation date. Please switch to tf.sets.difference().
2019-11-07 16:47:59,303 [WARNING] tensorflow: From ./ssd/utils/tensor_utils.py:31: setdiff1d (from tensorflow.python.ops.array_ops) is deprecated and will be removed after 2018-11-30.
Instructions for updating:
This op will be removed after the deprecation date. Please switch to tf.sets.difference().
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
2019-11-07 16:48:12,731 [INFO] iva.ssd.scripts.train: Loading pretrained weights. This may take a while…
WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
2019-11-07 16:48:22,904 [WARNING] tensorflow: From /usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.

Layer (type) Output Shape Param # Connected to

Input (InputLayer) (16, 3, 1080, 1920) 0

conv1 (Conv2D) (16, 64, 540, 960) 9472 Input[0][0]

bn_conv1 (BatchNormalization) (16, 64, 540, 960) 256 conv1[0][0]

activation_18 (Activation) (16, 64, 540, 960) 0 bn_conv1[0][0]

block_1a_conv_1 (Conv2D) (16, 64, 270, 480) 36928 activation_18[0][0]

block_1a_bn_1 (BatchNormalizati (16, 64, 270, 480) 256 block_1a_conv_1[0][0]

activation_19 (Activation) (16, 64, 270, 480) 0 block_1a_bn_1[0][0]

block_1a_conv_2 (Conv2D) (16, 64, 270, 480) 36928 activation_19[0][0]

block_1a_conv_shortcut (Conv2D) (16, 64, 270, 480) 4160 activation_18[0][0]

block_1a_bn_2 (BatchNormalizati (16, 64, 270, 480) 256 block_1a_conv_2[0][0]

block_1a_bn_shortcut (BatchNorm (16, 64, 270, 480) 256 block_1a_conv_shortcut[0][0]

add_9 (Add) (16, 64, 270, 480) 0 block_1a_bn_2[0][0]
block_1a_bn_shortcut[0][0]

activation_20 (Activation) (16, 64, 270, 480) 0 add_9[0][0]

block_1b_conv_1 (Conv2D) (16, 64, 270, 480) 36928 activation_20[0][0]

block_1b_bn_1 (BatchNormalizati (16, 64, 270, 480) 256 block_1b_conv_1[0][0]

activation_21 (Activation) (16, 64, 270, 480) 0 block_1b_bn_1[0][0]

block_1b_conv_2 (Conv2D) (16, 64, 270, 480) 36928 activation_21[0][0]

block_1b_conv_shortcut (Conv2D) (16, 64, 270, 480) 4160 activation_20[0][0]

block_1b_bn_2 (BatchNormalizati (16, 64, 270, 480) 256 block_1b_conv_2[0][0]

block_1b_bn_shortcut (BatchNorm (16, 64, 270, 480) 256 block_1b_conv_shortcut[0][0]

add_10 (Add) (16, 64, 270, 480) 0 block_1b_bn_2[0][0]
block_1b_bn_shortcut[0][0]

activation_22 (Activation) (16, 64, 270, 480) 0 add_10[0][0]

block_2a_conv_1 (Conv2D) (16, 128, 135, 240) 73856 activation_22[0][0]

block_2a_bn_1 (BatchNormalizati (16, 128, 135, 240) 512 block_2a_conv_1[0][0]

activation_23 (Activation) (16, 128, 135, 240) 0 block_2a_bn_1[0][0]

block_2a_conv_2 (Conv2D) (16, 128, 135, 240) 147584 activation_23[0][0]

block_2a_conv_shortcut (Conv2D) (16, 128, 135, 240) 8320 activation_22[0][0]

block_2a_bn_2 (BatchNormalizati (16, 128, 135, 240) 512 block_2a_conv_2[0][0]

block_2a_bn_shortcut (BatchNorm (16, 128, 135, 240) 512 block_2a_conv_shortcut[0][0]

add_11 (Add) (16, 128, 135, 240) 0 block_2a_bn_2[0][0]
block_2a_bn_shortcut[0][0]

activation_24 (Activation) (16, 128, 135, 240) 0 add_11[0][0]

block_2b_conv_1 (Conv2D) (16, 128, 135, 240) 147584 activation_24[0][0]

block_2b_bn_1 (BatchNormalizati (16, 128, 135, 240) 512 block_2b_conv_1[0][0]

activation_25 (Activation) (16, 128, 135, 240) 0 block_2b_bn_1[0][0]

block_2b_conv_2 (Conv2D) (16, 128, 135, 240) 147584 activation_25[0][0]

block_2b_conv_shortcut (Conv2D) (16, 128, 135, 240) 16512 activation_24[0][0]

block_2b_bn_2 (BatchNormalizati (16, 128, 135, 240) 512 block_2b_conv_2[0][0]

block_2b_bn_shortcut (BatchNorm (16, 128, 135, 240) 512 block_2b_conv_shortcut[0][0]

add_12 (Add) (16, 128, 135, 240) 0 block_2b_bn_2[0][0]
block_2b_bn_shortcut[0][0]

activation_26 (Activation) (16, 128, 135, 240) 0 add_12[0][0]

block_3a_conv_1 (Conv2D) (16, 256, 68, 120) 295168 activation_26[0][0]

block_3a_bn_1 (BatchNormalizati (16, 256, 68, 120) 1024 block_3a_conv_1[0][0]

activation_27 (Activation) (16, 256, 68, 120) 0 block_3a_bn_1[0][0]

block_3a_conv_2 (Conv2D) (16, 256, 68, 120) 590080 activation_27[0][0]

block_3a_conv_shortcut (Conv2D) (16, 256, 68, 120) 33024 activation_26[0][0]

block_3a_bn_2 (BatchNormalizati (16, 256, 68, 120) 1024 block_3a_conv_2[0][0]

block_3a_bn_shortcut (BatchNorm (16, 256, 68, 120) 1024 block_3a_conv_shortcut[0][0]

add_13 (Add) (16, 256, 68, 120) 0 block_3a_bn_2[0][0]
block_3a_bn_shortcut[0][0]

activation_28 (Activation) (16, 256, 68, 120) 0 add_13[0][0]

block_3b_conv_1 (Conv2D) (16, 256, 68, 120) 590080 activation_28[0][0]

block_3b_bn_1 (BatchNormalizati (16, 256, 68, 120) 1024 block_3b_conv_1[0][0]

activation_29 (Activation) (16, 256, 68, 120) 0 block_3b_bn_1[0][0]

block_3b_conv_2 (Conv2D) (16, 256, 68, 120) 590080 activation_29[0][0]

block_3b_conv_shortcut (Conv2D) (16, 256, 68, 120) 65792 activation_28[0][0]

block_3b_bn_2 (BatchNormalizati (16, 256, 68, 120) 1024 block_3b_conv_2[0][0]

block_3b_bn_shortcut (BatchNorm (16, 256, 68, 120) 1024 block_3b_conv_shortcut[0][0]

add_14 (Add) (16, 256, 68, 120) 0 block_3b_bn_2[0][0]
block_3b_bn_shortcut[0][0]

activation_30 (Activation) (16, 256, 68, 120) 0 add_14[0][0]

block_4a_conv_1 (Conv2D) (16, 512, 68, 120) 1180160 activation_30[0][0]

block_4a_bn_1 (BatchNormalizati (16, 512, 68, 120) 2048 block_4a_conv_1[0][0]

activation_31 (Activation) (16, 512, 68, 120) 0 block_4a_bn_1[0][0]

block_4a_conv_2 (Conv2D) (16, 512, 68, 120) 2359808 activation_31[0][0]

block_4a_conv_shortcut (Conv2D) (16, 512, 68, 120) 131584 activation_30[0][0]

block_4a_bn_2 (BatchNormalizati (16, 512, 68, 120) 2048 block_4a_conv_2[0][0]

block_4a_bn_shortcut (BatchNorm (16, 512, 68, 120) 2048 block_4a_conv_shortcut[0][0]

add_15 (Add) (16, 512, 68, 120) 0 block_4a_bn_2[0][0]
block_4a_bn_shortcut[0][0]

activation_32 (Activation) (16, 512, 68, 120) 0 add_15[0][0]

block_4b_conv_1 (Conv2D) (16, 512, 68, 120) 2359808 activation_32[0][0]

block_4b_bn_1 (BatchNormalizati (16, 512, 68, 120) 2048 block_4b_conv_1[0][0]

activation_33 (Activation) (16, 512, 68, 120) 0 block_4b_bn_1[0][0]

block_4b_conv_2 (Conv2D) (16, 512, 68, 120) 2359808 activation_33[0][0]

block_4b_conv_shortcut (Conv2D) (16, 512, 68, 120) 262656 activation_32[0][0]

block_4b_bn_2 (BatchNormalizati (16, 512, 68, 120) 2048 block_4b_conv_2[0][0]

block_4b_bn_shortcut (BatchNorm (16, 512, 68, 120) 2048 block_4b_conv_shortcut[0][0]

add_16 (Add) (16, 512, 68, 120) 0 block_4b_bn_2[0][0]
block_4b_bn_shortcut[0][0]

activation_34 (Activation) (16, 512, 68, 120) 0 add_16[0][0]

expand_conv1 (Conv2D) (16, 1024, 68, 120) 4719616 activation_34[0][0]

expand1_relu (ReLU) (16, 1024, 68, 120) 0 expand_conv1[0][0]

expand_conv2 (Conv2D) (16, 1024, 68, 120) 1049600 expand1_relu[0][0]

expand2_relu (ReLU) (16, 1024, 68, 120) 0 expand_conv2[0][0]

additional_map0_0 (Conv2D) (16, 256, 68, 120) 262400 expand2_relu[0][0]

additional_map0_0_relu (ReLU) (16, 256, 68, 120) 0 additional_map0_0[0][0]

additional_map0_1 (Conv2D) (16, 512, 34, 60) 1180160 additional_map0_0_relu[0][0]

additional_map0_1_relu (ReLU) (16, 512, 34, 60) 0 additional_map0_1[0][0]

additional_map1_0 (Conv2D) (16, 128, 34, 60) 65664 additional_map0_1_relu[0][0]

additional_map1_0_relu (ReLU) (16, 128, 34, 60) 0 additional_map1_0[0][0]

additional_map1_1 (Conv2D) (16, 256, 17, 30) 295168 additional_map1_0_relu[0][0]

additional_map1_1_relu (ReLU) (16, 256, 17, 30) 0 additional_map1_1[0][0]

additional_map2_0 (Conv2D) (16, 128, 17, 30) 32896 additional_map1_1_relu[0][0]

additional_map2_0_relu (ReLU) (16, 128, 17, 30) 0 additional_map2_0[0][0]

additional_map2_1 (Conv2D) (16, 256, 9, 15) 295168 additional_map2_0_relu[0][0]

additional_map2_1_relu (ReLU) (16, 256, 9, 15) 0 additional_map2_1[0][0]

additional_map3_0 (Conv2D) (16, 128, 9, 15) 32896 additional_map2_1_relu[0][0]

additional_map3_0_relu (ReLU) (16, 128, 9, 15) 0 additional_map3_0[0][0]

additional_map3_1 (Conv2D) (16, 256, 5, 8) 295168 additional_map3_0_relu[0][0]

additional_map3_1_relu (ReLU) (16, 256, 5, 8) 0 additional_map3_1[0][0]

ssd_conf_0 (Conv2D) (16, 18, 135, 240) 20754 activation_26[0][0]

ssd_conf_1 (Conv2D) (16, 18, 68, 120) 82962 activation_34[0][0]

ssd_conf_2 (Conv2D) (16, 18, 34, 60) 82962 additional_map0_1_relu[0][0]

ssd_conf_3 (Conv2D) (16, 18, 17, 30) 41490 additional_map1_1_relu[0][0]

ssd_conf_4 (Conv2D) (16, 18, 9, 15) 41490 additional_map2_1_relu[0][0]

ssd_conf_5 (Conv2D) (16, 18, 5, 8) 41490 additional_map3_1_relu[0][0]

permute_13 (Permute) (16, 135, 240, 18) 0 ssd_conf_0[0][0]

permute_15 (Permute) (16, 68, 120, 18) 0 ssd_conf_1[0][0]

permute_17 (Permute) (16, 34, 60, 18) 0 ssd_conf_2[0][0]

permute_19 (Permute) (16, 17, 30, 18) 0 ssd_conf_3[0][0]

permute_21 (Permute) (16, 9, 15, 18) 0 ssd_conf_4[0][0]

permute_23 (Permute) (16, 5, 8, 18) 0 ssd_conf_5[0][0]

ssd_loc_0 (Conv2D) (16, 24, 135, 240) 27672 activation_26[0][0]

ssd_loc_1 (Conv2D) (16, 24, 68, 120) 110616 activation_34[0][0]

ssd_loc_2 (Conv2D) (16, 24, 34, 60) 110616 additional_map0_1_relu[0][0]

ssd_loc_3 (Conv2D) (16, 24, 17, 30) 55320 additional_map1_1_relu[0][0]

ssd_loc_4 (Conv2D) (16, 24, 9, 15) 55320 additional_map2_1_relu[0][0]

ssd_loc_5 (Conv2D) (16, 24, 5, 8) 55320 additional_map3_1_relu[0][0]

conf_reshape_0 (Reshape) (16, 194400, 1, 3) 0 permute_13[0][0]

conf_reshape_1 (Reshape) (16, 48960, 1, 3) 0 permute_15[0][0]

conf_reshape_2 (Reshape) (16, 12240, 1, 3) 0 permute_17[0][0]

conf_reshape_3 (Reshape) (16, 3060, 1, 3) 0 permute_19[0][0]

conf_reshape_4 (Reshape) (16, 810, 1, 3) 0 permute_21[0][0]

conf_reshape_5 (Reshape) (16, 240, 1, 3) 0 permute_23[0][0]

permute_14 (Permute) (16, 135, 240, 24) 0 ssd_loc_0[0][0]

permute_16 (Permute) (16, 68, 120, 24) 0 ssd_loc_1[0][0]

permute_18 (Permute) (16, 34, 60, 24) 0 ssd_loc_2[0][0]

permute_20 (Permute) (16, 17, 30, 24) 0 ssd_loc_3[0][0]

permute_22 (Permute) (16, 9, 15, 24) 0 ssd_loc_4[0][0]

permute_24 (Permute) (16, 5, 8, 24) 0 ssd_loc_5[0][0]

ssd_anchor_0 (AnchorBoxes) (16, 32400, 6, 8) 0 ssd_loc_0[0][0]

ssd_anchor_1 (AnchorBoxes) (16, 8160, 6, 8) 0 ssd_loc_1[0][0]

ssd_anchor_2 (AnchorBoxes) (16, 2040, 6, 8) 0 ssd_loc_2[0][0]

ssd_anchor_3 (AnchorBoxes) (16, 510, 6, 8) 0 ssd_loc_3[0][0]

ssd_anchor_4 (AnchorBoxes) (16, 135, 6, 8) 0 ssd_loc_4[0][0]

ssd_anchor_5 (AnchorBoxes) (16, 40, 6, 8) 0 ssd_loc_5[0][0]

mbox_conf (Concatenate) (16, 259710, 1, 3) 0 conf_reshape_0[0][0]
conf_reshape_1[0][0]
conf_reshape_2[0][0]
conf_reshape_3[0][0]
conf_reshape_4[0][0]
conf_reshape_5[0][0]

loc_reshape_0 (Reshape) (16, 194400, 1, 4) 0 permute_14[0][0]

loc_reshape_1 (Reshape) (16, 48960, 1, 4) 0 permute_16[0][0]

loc_reshape_2 (Reshape) (16, 12240, 1, 4) 0 permute_18[0][0]

loc_reshape_3 (Reshape) (16, 3060, 1, 4) 0 permute_20[0][0]

loc_reshape_4 (Reshape) (16, 810, 1, 4) 0 permute_22[0][0]

loc_reshape_5 (Reshape) (16, 240, 1, 4) 0 permute_24[0][0]

anchor_reshape_0 (Reshape) (16, 194400, 1, 8) 0 ssd_anchor_0[0][0]

anchor_reshape_1 (Reshape) (16, 48960, 1, 8) 0 ssd_anchor_1[0][0]

anchor_reshape_2 (Reshape) (16, 12240, 1, 8) 0 ssd_anchor_2[0][0]

anchor_reshape_3 (Reshape) (16, 3060, 1, 8) 0 ssd_anchor_3[0][0]

anchor_reshape_4 (Reshape) (16, 810, 1, 8) 0 ssd_anchor_4[0][0]

anchor_reshape_5 (Reshape) (16, 240, 1, 8) 0 ssd_anchor_5[0][0]

mbox_conf_sigmoid (Activation) (16, 259710, 1, 3) 0 mbox_conf[0][0]

mbox_loc (Concatenate) (16, 259710, 1, 4) 0 loc_reshape_0[0][0]
loc_reshape_1[0][0]
loc_reshape_2[0][0]
loc_reshape_3[0][0]
loc_reshape_4[0][0]
loc_reshape_5[0][0]

mbox_priorbox (Concatenate) (16, 259710, 1, 8) 0 anchor_reshape_0[0][0]
anchor_reshape_1[0][0]
anchor_reshape_2[0][0]
anchor_reshape_3[0][0]
anchor_reshape_4[0][0]
anchor_reshape_5[0][0]

concatenate_2 (Concatenate) (16, 259710, 1, 15) 0 mbox_conf_sigmoid[0][0]
mbox_loc[0][0]
mbox_priorbox[0][0]

ssd_predictions (Reshape) (16, 259710, 15) 0 concatenate_2[0][0]

Total params: 20,503,036
Trainable params: 20,481,916
Non-trainable params: 21,120

2019-11-07 16:50:57,605 [INFO] iva.ssd.scripts.train: Number of images in the training dataset: 2566
2019-11-07 16:50:57,605 [INFO] iva.ssd.scripts.train: Number of images in the validation dataset: 417
2019-11-07 16:51:04.345438: W tensorflow/core/framework/allocator.cc:124] Allocation of 18874368 exceeds 10% of system memory.
2019-11-07 16:51:04.363103: W tensorflow/core/framework/allocator.cc:124] Allocation of 18874368 exceeds 10% of system memory.
2019-11-07 16:51:04.375936: W tensorflow/core/framework/allocator.cc:124] Allocation of 18874368 exceeds 10% of system memory.
2019-11-07 16:51:04.456606: W tensorflow/core/framework/allocator.cc:124] Allocation of 18874368 exceeds 10% of system memory.
2019-11-07 16:51:04.485102: W tensorflow/core/framework/allocator.cc:124] Allocation of 18874368 exceeds 10% of system memory.
Epoch 1/180
2019-11-07 16:51:14.587732: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_16/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.587783: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_16/CornerCoordToCentroids/StridedReplace/strided_slice. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?,1]
2019-11-07 16:51:14.587804: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_16/CornerCoordToCentroids/strided_slice_4. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.587823: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_16/CornerCoordToCentroids/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.587841: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_16/CornerCoordToCentroids/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.587859: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_16/CornerCoordToCentroids/strided_slice_7. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.587877: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_16/CornerCoordToCentroids/StridedReplace/strided_slice_1. Error: ValidateStridedSliceOp returned partial shapes [?,0] and [?,0]
2019-11-07 16:51:14.587895: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_16/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,4] and [?,4]
2019-11-07 16:51:14.587943: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_17/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.587963: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_17/CornerCoordToCentroids/StridedReplace/strided_slice. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?,1]
2019-11-07 16:51:14.587981: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_17/CornerCoordToCentroids/strided_slice_4. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.587999: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_17/CornerCoordToCentroids/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.588017: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_17/CornerCoordToCentroids/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.588037: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_17/CornerCoordToCentroids/strided_slice_7. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.588055: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_17/CornerCoordToCentroids/StridedReplace/strided_slice_1. Error: ValidateStridedSliceOp returned partial shapes [?,0] and [?,0]
2019-11-07 16:51:14.588073: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_17/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,4] and [?,4]
2019-11-07 16:51:14.588097: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_18/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.588116: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_18/CornerCoordToCentroids/StridedReplace/strided_slice. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?,1]
2019-11-07 16:51:14.588135: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_18/CornerCoordToCentroids/strided_slice_4. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.588153: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_18/CornerCoordToCentroids/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.588171: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_18/CornerCoordToCentroids/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.588189: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_18/CornerCoordToCentroids/strided_slice_7. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.588207: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_18/CornerCoordToCentroids/StridedReplace/strided_slice_1. Error: ValidateStridedSliceOp returned partial shapes [?,0] and [?,0]
2019-11-07 16:51:14.588225: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_18/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,4] and [?,4]
2019-11-07 16:51:14.588249: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_19/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.588267: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_19/CornerCoordToCentroids/StridedReplace/strided_slice. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?,1]
2019-11-07 16:51:14.588286: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_19/CornerCoordToCentroids/strided_slice_4. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.588304: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_19/CornerCoordToCentroids/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.588322: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_19/CornerCoordToCentroids/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.588340: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_19/CornerCoordToCentroids/strided_slice_7. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.588362: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_19/CornerCoordToCentroids/StridedReplace/strided_slice_1. Error: ValidateStridedSliceOp returned partial shapes [?,0] and [?,0]
2019-11-07 16:51:14.588381: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_19/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,4] and [?,4]
2019-11-07 16:51:14.588404: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_20/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.588423: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_20/CornerCoordToCentroids/StridedReplace/strided_slice. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?,1]
2019-11-07 16:51:14.588442: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_20/CornerCoordToCentroids/strided_slice_4. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.588461: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_20/CornerCoordToCentroids/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.588479: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_20/CornerCoordToCentroids/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.588497: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_20/CornerCoordToCentroids/strided_slice_7. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.588515: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_20/CornerCoordToCentroids/StridedReplace/strided_slice_1. Error: ValidateStridedSliceOp returned partial shapes [?,0] and [?,0]
2019-11-07 16:51:14.588533: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_20/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,4] and [?,4]
2019-11-07 16:51:14.588557: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_21/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.588575: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_21/CornerCoordToCentroids/StridedReplace/strided_slice. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?,1]
2019-11-07 16:51:14.588594: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_21/CornerCoordToCentroids/strided_slice_4. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.588611: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_21/CornerCoordToCentroids/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.588629: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_21/CornerCoordToCentroids/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.588647: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_21/CornerCoordToCentroids/strided_slice_7. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.588664: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_21/CornerCoordToCentroids/StridedReplace/strided_slice_1. Error: ValidateStridedSliceOp returned partial shapes [?,0] and [?,0]
2019-11-07 16:51:14.588681: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_21/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,4] and [?,4]
2019-11-07 16:51:14.588704: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_22/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.588723: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_22/CornerCoordToCentroids/StridedReplace/strided_slice. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?,1]
2019-11-07 16:51:14.588742: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_22/CornerCoordToCentroids/strided_slice_4. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.588760: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_22/CornerCoordToCentroids/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.588777: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_22/CornerCoordToCentroids/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.588794: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_22/CornerCoordToCentroids/strided_slice_7. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.588813: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_22/CornerCoordToCentroids/StridedReplace/strided_slice_1. Error: ValidateStridedSliceOp returned partial shapes [?,0] and [?,0]
2019-11-07 16:51:14.588831: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_22/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,4] and [?,4]
2019-11-07 16:51:14.588854: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_23/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.588873: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_23/CornerCoordToCentroids/StridedReplace/strided_slice. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?,1]
2019-11-07 16:51:14.588892: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_23/CornerCoordToCentroids/strided_slice_4. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.588911: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_23/CornerCoordToCentroids/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.588929: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_23/CornerCoordToCentroids/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:5

11

Hi psgr,
Have you finished the first epoch training? If not, please ignore the error and keep patient to run several epoches.
BTW, it is better for you to paste the log by adding “code block”.

12

Hi,

I there is a character limitation in the forum which has actually chopped my log here is the continuation for the same

2019-11-07 16:51:14.588947: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_23/CornerCoordToCentroids/strided_slice_7. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.588965: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_23/CornerCoordToCentroids/StridedReplace/strided_slice_1. Error: ValidateStridedSliceOp returned partial shapes [?,0] and [?,0]
2019-11-07 16:51:14.588983: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_23/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,4] and [?,4]
2019-11-07 16:51:14.589006: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_24/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.589025: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_24/CornerCoordToCentroids/StridedReplace/strided_slice. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?,1]
2019-11-07 16:51:14.589044: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_24/CornerCoordToCentroids/strided_slice_4. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.589062: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_24/CornerCoordToCentroids/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.589081: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_24/CornerCoordToCentroids/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.589099: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_24/CornerCoordToCentroids/strided_slice_7. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.589117: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_24/CornerCoordToCentroids/StridedReplace/strided_slice_1. Error: ValidateStridedSliceOp returned partial shapes [?,0] and [?,0]
2019-11-07 16:51:14.589135: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_24/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,4] and [?,4]
2019-11-07 16:51:14.589158: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_25/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.589177: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_25/CornerCoordToCentroids/StridedReplace/strided_slice. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?,1]
2019-11-07 16:51:14.589196: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_25/CornerCoordToCentroids/strided_slice_4. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.589214: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_25/CornerCoordToCentroids/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.589233: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_25/CornerCoordToCentroids/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.589251: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_25/CornerCoordToCentroids/strided_slice_7. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.589269: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_25/CornerCoordToCentroids/StridedReplace/strided_slice_1. Error: ValidateStridedSliceOp returned partial shapes [?,0] and [?,0]
2019-11-07 16:51:14.589288: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_25/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,4] and [?,4]
2019-11-07 16:51:14.589311: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_26/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.589330: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_26/CornerCoordToCentroids/StridedReplace/strided_slice. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?,1]
2019-11-07 16:51:14.589349: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_26/CornerCoordToCentroids/strided_slice_4. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.589367: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_26/CornerCoordToCentroids/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.589385: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_26/CornerCoordToCentroids/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.589403: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_26/CornerCoordToCentroids/strided_slice_7. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.589421: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_26/CornerCoordToCentroids/StridedReplace/strided_slice_1. Error: ValidateStridedSliceOp returned partial shapes [?,0] and [?,0]
2019-11-07 16:51:14.589439: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_26/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,4] and [?,4]
2019-11-07 16:51:14.589462: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_27/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.589480: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_27/CornerCoordToCentroids/StridedReplace/strided_slice. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?,1]
2019-11-07 16:51:14.589500: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_27/CornerCoordToCentroids/strided_slice_4. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.589518: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_27/CornerCoordToCentroids/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.589536: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_27/CornerCoordToCentroids/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.589554: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_27/CornerCoordToCentroids/strided_slice_7. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.589572: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_27/CornerCoordToCentroids/StridedReplace/strided_slice_1. Error: ValidateStridedSliceOp returned partial shapes [?,0] and [?,0]
2019-11-07 16:51:14.589591: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_27/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,4] and [?,4]
2019-11-07 16:51:14.589614: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_28/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.589632: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_28/CornerCoordToCentroids/StridedReplace/strided_slice. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?,1]
2019-11-07 16:51:14.589651: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_28/CornerCoordToCentroids/strided_slice_4. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.589669: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_28/CornerCoordToCentroids/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.589688: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_28/CornerCoordToCentroids/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.589705: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_28/CornerCoordToCentroids/strided_slice_7. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.589724: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_28/CornerCoordToCentroids/StridedReplace/strided_slice_1. Error: ValidateStridedSliceOp returned partial shapes [?,0] and [?,0]
2019-11-07 16:51:14.589742: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_28/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,4] and [?,4]
2019-11-07 16:51:14.589765: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_29/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.589783: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_29/CornerCoordToCentroids/StridedReplace/strided_slice. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?,1]
2019-11-07 16:51:14.589801: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_29/CornerCoordToCentroids/strided_slice_4. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.589820: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_29/CornerCoordToCentroids/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.589838: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_29/CornerCoordToCentroids/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.589857: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_29/CornerCoordToCentroids/strided_slice_7. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.589875: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_29/CornerCoordToCentroids/StridedReplace/strided_slice_1. Error: ValidateStridedSliceOp returned partial shapes [?,0] and [?,0]
2019-11-07 16:51:14.589893: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_29/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,4] and [?,4]
2019-11-07 16:51:14.589915: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_30/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.589934: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_30/CornerCoordToCentroids/StridedReplace/strided_slice. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?,1]
2019-11-07 16:51:14.589953: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_30/CornerCoordToCentroids/strided_slice_4. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.589971: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_30/CornerCoordToCentroids/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.589990: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_30/CornerCoordToCentroids/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.590008: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_30/CornerCoordToCentroids/strided_slice_7. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.590026: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_30/CornerCoordToCentroids/StridedReplace/strided_slice_1. Error: ValidateStridedSliceOp returned partial shapes [?,0] and [?,0]
2019-11-07 16:51:14.590044: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_30/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,4] and [?,4]
2019-11-07 16:51:14.590067: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_31/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.590085: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_31/CornerCoordToCentroids/StridedReplace/strided_slice. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?,1]
2019-11-07 16:51:14.590104: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_31/CornerCoordToCentroids/strided_slice_4. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.590122: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_31/CornerCoordToCentroids/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.590140: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_31/CornerCoordToCentroids/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.590158: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_31/CornerCoordToCentroids/strided_slice_7. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:14.590177: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_31/CornerCoordToCentroids/StridedReplace/strided_slice_1. Error: ValidateStridedSliceOp returned partial shapes [?,0] and [?,0]
2019-11-07 16:51:14.590194: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_31/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,4] and [?,4]
2019-11-07 16:51:27.741870: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_16/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.741924: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_16/CornerCoordToCentroids/StridedReplace/strided_slice. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?,1]
2019-11-07 16:51:27.741946: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_16/CornerCoordToCentroids/strided_slice_4. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.741966: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_16/CornerCoordToCentroids/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.741984: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_16/CornerCoordToCentroids/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.742002: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_16/CornerCoordToCentroids/strided_slice_7. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.742021: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_16/CornerCoordToCentroids/StridedReplace/strided_slice_1. Error: ValidateStridedSliceOp returned partial shapes [?,0] and [?,0]
2019-11-07 16:51:27.742040: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_16/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,4] and [?,4]
2019-11-07 16:51:27.742067: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_17/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.742087: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_17/CornerCoordToCentroids/StridedReplace/strided_slice. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?,1]
2019-11-07 16:51:27.742106: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_17/CornerCoordToCentroids/strided_slice_4. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.742125: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_17/CornerCoordToCentroids/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.742143: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_17/CornerCoordToCentroids/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.742162: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_17/CornerCoordToCentroids/strided_slice_7. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.742183: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_17/CornerCoordToCentroids/StridedReplace/strided_slice_1. Error: ValidateStridedSliceOp returned partial shapes [?,0] and [?,0]
2019-11-07 16:51:27.742203: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_17/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,4] and [?,4]
2019-11-07 16:51:27.742227: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_18/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.742247: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_18/CornerCoordToCentroids/StridedReplace/strided_slice. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?,1]
2019-11-07 16:51:27.742267: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_18/CornerCoordToCentroids/strided_slice_4. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.742285: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_18/CornerCoordToCentroids/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.742305: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_18/CornerCoordToCentroids/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.742323: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_18/CornerCoordToCentroids/strided_slice_7. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.742342: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_18/CornerCoordToCentroids/StridedReplace/strided_slice_1. Error: ValidateStridedSliceOp returned partial shapes [?,0] and [?,0]
2019-11-07 16:51:27.742361: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_18/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,4] and [?,4]
2019-11-07 16:51:27.742386: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_19/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.742406: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_19/CornerCoordToCentroids/StridedReplace/strided_slice. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?,1]
2019-11-07 16:51:27.742426: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_19/CornerCoordToCentroids/strided_slice_4. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.742445: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_19/CornerCoordToCentroids/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.742465: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_19/CornerCoordToCentroids/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.742484: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_19/CornerCoordToCentroids/strided_slice_7. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.742503: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_19/CornerCoordToCentroids/StridedReplace/strided_slice_1. Error: ValidateStridedSliceOp returned partial shapes [?,0] and [?,0]
2019-11-07 16:51:27.742522: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_19/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,4] and [?,4]
2019-11-07 16:51:27.742546: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_20/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.742566: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_20/CornerCoordToCentroids/StridedReplace/strided_slice. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?,1]
2019-11-07 16:51:27.742587: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_20/CornerCoordToCentroids/strided_slice_4. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.742606: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_20/CornerCoordToCentroids/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.742625: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_20/CornerCoordToCentroids/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.742644: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_20/CornerCoordToCentroids/strided_slice_7. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.742663: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_20/CornerCoordToCentroids/StridedReplace/strided_slice_1. Error: ValidateStridedSliceOp returned partial shapes [?,0] and [?,0]
2019-11-07 16:51:27.742681: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_20/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,4] and [?,4]
2019-11-07 16:51:27.742705: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_21/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.742725: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_21/CornerCoordToCentroids/StridedReplace/strided_slice. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?,1]
2019-11-07 16:51:27.742745: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_21/CornerCoordToCentroids/strided_slice_4. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.742764: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_21/CornerCoordToCentroids/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.742783: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_21/CornerCoordToCentroids/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.742803: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_21/CornerCoordToCentroids/strided_slice_7. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.742821: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_21/CornerCoordToCentroids/StridedReplace/strided_slice_1. Error: ValidateStridedSliceOp returned partial shapes [?,0] and [?,0]
2019-11-07 16:51:27.742840: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_21/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,4] and [?,4]
2019-11-07 16:51:27.742865: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_22/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.742885: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_22/CornerCoordToCentroids/StridedReplace/strided_slice. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?,1]
2019-11-07 16:51:27.742904: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_22/CornerCoordToCentroids/strided_slice_4. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.742924: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_22/CornerCoordToCentroids/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.742943: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_22/CornerCoordToCentroids/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.742961: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_22/CornerCoordToCentroids/strided_slice_7. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.742981: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_22/CornerCoordToCentroids/StridedReplace/strided_slice_1. Error: ValidateStridedSliceOp returned partial shapes [?,0] and [?,0]
2019-11-07 16:51:27.742999: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_22/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,4] and [?,4]
2019-11-07 16:51:27.743023: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_23/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.743043: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_23/CornerCoordToCentroids/StridedReplace/strided_slice. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?,1]
2019-11-07 16:51:27.743063: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_23/CornerCoordToCentroids/strided_slice_4. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.743082: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_23/CornerCoordToCentroids/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.743101: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_23/CornerCoordToCentroids/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.743119: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_23/CornerCoordToCentroids/strided_slice_7. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.743137: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_23/CornerCoordToCentroids/StridedReplace/strided_slice_1. Error: ValidateStridedSliceOp returned partial shapes [?,0] and [?,0]
2019-11-07 16:51:27.743156: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_23/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,4] and [?,4]
2019-11-07 16:51:27.743180: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_24/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.743198: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_24/CornerCoordToCentroids/StridedReplace/strided_slice. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?,1]
2019-11-07 16:51:27.743219: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_24/CornerCoordToCentroids/strided_slice_4. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.743238: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_24/CornerCoordToCentroids/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.743256: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_24/CornerCoordToCentroids/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.743275: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_24/CornerCoordToCentroids/strided_slice_7. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.743294: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_24/CornerCoordToCentroids/StridedReplace/strided_slice_1. Error: ValidateStridedSliceOp returned partial shapes [?,0] and [?,0]
2019-11-07 16:51:27.743312: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_24/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,4] and [?,4]
2019-11-07 16:51:27.743336: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_25/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.743354: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_25/CornerCoordToCentroids/StridedReplace/strided_slice. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?,1]
2019-11-07 16:51:27.743374: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_25/CornerCoordToCentroids/strided_slice_4. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.743394: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_25/CornerCoordToCentroids/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.743412: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_25/CornerCoordToCentroids/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.743431: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_25/CornerCoordToCentroids/strided_slice_7. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.743450: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_25/CornerCoordToCentroids/StridedReplace/strided_slice_1. Error: ValidateStridedSliceOp returned partial shapes [?,0] and [?,0]
2019-11-07 16:51:27.743469: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_25/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,4] and [?,4]
2019-11-07 16:51:27.743493: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_26/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.743513: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_26/CornerCoordToCentroids/StridedReplace/strided_slice. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?,1]
2019-11-07 16:51:27.743532: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_26/CornerCoordToCentroids/strided_slice_4. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.743551: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_26/CornerCoordToCentroids/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.743571: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_26/CornerCoordToCentroids/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.743588: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_26/CornerCoordToCentroids/strided_slice_7. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.743608: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_26/CornerCoordToCentroids/StridedReplace/strided_slice_1. Error: ValidateStridedSliceOp returned partial shapes [?,0] and [?,0]
2019-11-07 16:51:27.743627: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_26/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,4] and [?,4]
2019-11-07 16:51:27.743650: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_27/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.743670: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_27/CornerCoordToCentroids/StridedReplace/strided_slice. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?,1]
2019-11-07 16:51:27.743689: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_27/CornerCoordToCentroids/strided_slice_4. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.743708: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_27/CornerCoordToCentroids/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.743744: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_27/CornerCoordToCentroids/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.743763: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_27/CornerCoordToCentroids/strided_slice_7. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.743797: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_27/CornerCoordToCentroids/StridedReplace/strided_slice_1. Error: ValidateStridedSliceOp returned partial shapes [?,0] and [?,0]
2019-11-07 16:51:27.743816: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_27/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,4] and [?,4]
2019-11-07 16:51:27.743840: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_28/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.743859: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_28/CornerCoordToCentroids/StridedReplace/strided_slice. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?,1]
2019-11-07 16:51:27.743879: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_28/CornerCoordToCentroids/strided_slice_4. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.743899: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_28/CornerCoordToCentroids/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.743918: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_28/CornerCoordToCentroids/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.743937: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_28/CornerCoordToCentroids/strided_slice_7. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.743955: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_28/CornerCoordToCentroids/StridedReplace/strided_slice_1. Error: ValidateStridedSliceOp returned partial shapes [?,0] and [?,0]
2019-11-07 16:51:27.743975: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_28/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,4] and [?,4]
2019-11-07 16:51:27.743999: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_29/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.744018: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_29/CornerCoordToCentroids/StridedReplace/strided_slice. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?,1]
2019-11-07 16:51:27.744038: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_29/CornerCoordToCentroids/strided_slice_4. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.744057: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_29/CornerCoordToCentroids/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.744077: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_29/CornerCoordToCentroids/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.744095: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_29/CornerCoordToCentroids/strided_slice_7. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.744115: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_29/CornerCoordToCentroids/StridedReplace/strided_slice_1. Error: ValidateStridedSliceOp returned partial shapes [?,0] and [?,0]
2019-11-07 16:51:27.744133: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_29/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,4] and [?,4]
2019-11-07 16:51:27.744157: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_30/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.744177: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_30/CornerCoordToCentroids/StridedReplace/strided_slice. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?,1]
2019-11-07 16:51:27.744196: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_30/CornerCoordToCentroids/strided_slice_4. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.744216: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_30/CornerCoordToCentroids/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.744235: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_30/CornerCoordToCentroids/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.744254: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_30/CornerCoordToCentroids/strided_slice_7. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.744272: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_30/CornerCoordToCentroids/StridedReplace/strided_slice_1. Error: ValidateStridedSliceOp returned partial shapes [?,0] and [?,0]
2019-11-07 16:51:27.744291: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_30/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,4] and [?,4]
2019-11-07 16:51:27.744315: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_31/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.744334: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_31/CornerCoordToCentroids/StridedReplace/strided_slice. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?,1]
2019-11-07 16:51:27.744373: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_31/CornerCoordToCentroids/strided_slice_4. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.744395: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_31/CornerCoordToCentroids/strided_slice_5. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.744427: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_31/CornerCoordToCentroids/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.744446: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_31/CornerCoordToCentroids/strided_slice_7. Error: ValidateStridedSliceOp returned partial shapes [?,1] and [?]
2019-11-07 16:51:27.744465: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_31/CornerCoordToCentroids/StridedReplace/strided_slice_1. Error: ValidateStridedSliceOp returned partial shapes [?,0] and [?,0]
2019-11-07 16:51:27.744484: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node cond_31/strided_slice_6. Error: ValidateStridedSliceOp returned partial shapes [?,4] and [?,4]
2019-11-07 16:51:29.170924: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally
2019-11-07 16:51:29.786102: I tensorflow/core/kernels/cuda_solvers.cc:159] Creating CudaSolver handles for stream 0x60d74b0
2019-11-07 16:51:31.398825: E tensorflow/stream_executor/cuda/cuda_dnn.cc:334] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
2019-11-07 16:51:31.405099: E tensorflow/stream_executor/cuda/cuda_dnn.cc:334] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
Traceback (most recent call last):
File “/usr/local/bin/tlt-train-g1”, line 10, in
sys.exit(main())
File “./common/magnet_train.py”, line 33, in main
File “./ssd/scripts/train.py”, line 301, in main
File “./ssd/scripts/train.py”, line 245, in run_experiment
File “/usr/local/lib/python2.7/dist-packages/keras/engine/training.py”, line 1039, in fit
validation_steps=validation_steps)
File “/usr/local/lib/python2.7/dist-packages/keras/engine/training_arrays.py”, line 154, in fit_loop
outs = f(ins)
File “/usr/local/lib/python2.7/dist-packages/keras/backend/tensorflow_backend.py”, line 2715, in call
return self._call(inputs)
File “/usr/local/lib/python2.7/dist-packages/keras/backend/tensorflow_backend.py”, line 2675, in _call
fetched = self._callable_fn(*array_vals)
File “/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py”, line 1439, in call
run_metadata_ptr)
File “/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/errors_impl.py”, line 528, in exit
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.UnknownError: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[{{node conv1_1/convolution}}]]
[[{{node loss_1/add_46}}]]

So it ended with an CUDNN error. Can you please help with this?

Thanks in Advance

13

Please click " </> " button next time when you “Add Reply” the topic.

14

Sure will click " </> " button while adding logs and codes .

Is this CUDNN error because of OOM error or something from model or tfrecords?

Do I need to edit some batch sizes or some other parameters to make it run ?

15

Please try with lower batch_size_per_gpu. For example, 4 or 8.

16

See https://devtalk.nvidia.com/default/topic/1066050/transfer-learning-toolkit/error-with-cudnn-when-attempting-to-perform-inference-after-training-an-ssd-model-with-tlt/

The solution is

$ export TF_FORCE_GPU_ALLOW_GROWTH=true