Tensorflow Object Detection error while training Faster RCNN

Hi, I encountered the following error while training faster_rcnn_inception_v2_coco_2018_01_28. The training is very slow and stops at the following line: 16064 supervisor.py:1050] Recording summary at step 6507.

I used to be able to training without issues. But starts to have the error recently. Tried reinstall, tensorflow, and tensorflow object detection, but doesn’t seem to help.

Can someone please advise.

Here are the printouts:

D:\object_detection\models\research\object_detection>
D:\object_detection\models\research\object_detection>python train.py --logtostderr --train_dir=training/ --pipeline_config_path=training/faster_rcnn_inception_v2_pets.config
WARNING: Logging before flag parsing goes to stderr.
W0119 07:44:54.407765 16104 lazy_loader.py:50]
The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:

W0119 07:44:54.950470 16104 deprecation_wrapper.py:119] From D:\object_detection\models\research\slim\nets\inception_resnet_v2.py:373: The name tf.GraphKeys is deprecated. Please use tf.compat.v1.GraphKeys instead.

W0119 07:44:55.034149 16104 deprecation_wrapper.py:119] From D:\object_detection\models\research\slim\nets\mobilenet\mobilenet.py:397: The name tf.nn.avg_pool is deprecated. Please use tf.nn.avg_pool2d instead.

W0119 07:44:55.159384 16104 deprecation_wrapper.py:119] From train.py:55: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.

W0119 07:44:55.161942 16104 deprecation_wrapper.py:119] From train.py:55: The name tf.logging.INFO is deprecated. Please use tf.compat.v1.logging.INFO instead.

W0119 07:44:55.164434 16104 deprecation_wrapper.py:119] From train.py:184: The name tf.app.run is deprecated. Please use tf.compat.v1.app.run instead.

W0119 07:44:55.170453 16104 deprecation.py:323] From C:\Users\sikli\AppData\Local\Programs\Python\Python35\lib\site-packages\absl\app.py:251: main (from main) is deprecated and will be removed in a future version.
Instructions for updating:
Use object_detection/model_main.py.
W0119 07:44:55.172410 16104 deprecation_wrapper.py:119] From train.py:90: The name tf.gfile.MakeDirs is deprecated. Please use tf.io.gfile.makedirs instead.

W0119 07:44:55.175020 16104 deprecation_wrapper.py:119] From D:\object_detection\models\research\object_detection\utils\config_util.py:102: The name tf.gfile.GFile is deprecated. Please use tf.io.gfile.GFile instead.

W0119 07:44:55.195727 16104 deprecation_wrapper.py:119] From train.py:95: The name tf.gfile.Copy is deprecated. Please use tf.io.gfile.copy instead.

W0119 07:44:55.244276 16104 deprecation.py:323] From D:\object_detection\models\research\object_detection\legacy\trainer.py:266: create_global_step (from tensorflow.contrib.framework.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.create_global_step
W0119 07:44:55.251557 16104 deprecation_wrapper.py:119] From D:\object_detection\models\research\object_detection\data_decoders\tf_example_decoder.py:182: The name tf.FixedLenFeature is deprecated. Please use tf.io.FixedLenFeature instead.

W0119 07:44:55.251557 16104 deprecation_wrapper.py:119] From D:\object_detection\models\research\object_detection\data_decoders\tf_example_decoder.py:197: The name tf.VarLenFeature is deprecated. Please use tf.io.VarLenFeature instead.

W0119 07:44:55.262580 16104 deprecation_wrapper.py:119] From D:\object_detection\models\research\object_detection\builders\dataset_builder.py:64: The name tf.gfile.Glob is deprecated. Please use tf.io.gfile.glob instead.

W0119 07:44:55.266569 16104 deprecation_wrapper.py:119] From D:\object_detection\models\research\object_detection\builders\dataset_builder.py:71: The name tf.logging.warning is deprecated. Please use tf.compat.v1.logging.warning instead.

W0119 07:44:55.266701 16104 dataset_builder.py:72] num_readers has been reduced to 1 to match input file shards.
W0119 07:44:55.271716 16104 deprecation.py:323] From D:\object_detection\models\research\object_detection\builders\dataset_builder.py:86: parallel_interleave (from tensorflow.contrib.data.python.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.data.experimental.parallel_interleave(...).
W0119 07:44:55.271716 16104 deprecation.py:323] From C:\Users\sikli\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\contrib\data\python\ops\interleave_ops.py:77: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.experimental.AUTOTUNE) instead. If sloppy execution is desired, use tf.data.Options.experimental_determinstic.
W0119 07:44:55.289671 16104 deprecation.py:323] From D:\object_detection\models\research\object_detection\builders\dataset_builder.py:155: DatasetV1.map_with_legacy_function (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.data.Dataset.map() W0119 07:44:55.412343 16104 deprecation.py:323] From D:\object_detection\models\research\object_detection\builders\dataset_builder.py:43: DatasetV1.make_initializable_iterator (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version. Instructions for updating: Use for … in dataset:to iterate over a dataset. If usingtf.estimator, return the Datasetobject directly from your input function. As a last resort, you can usetf.compat.v1.data.make_initializable_iterator(dataset)`.
W0119 07:44:55.419323 16104 deprecation_wrapper.py:119] From D:\object_detection\models\research\object_detection\builders\dataset_builder.py:44: The name tf.add_to_collection is deprecated. Please use tf.compat.v1.add_to_collection instead.

W0119 07:44:55.422315 16104 deprecation_wrapper.py:119] From D:\object_detection\models\research\object_detection\core\preprocessor.py:626: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.

W0119 07:44:55.449247 16104 deprecation.py:323] From D:\object_detection\models\research\object_detection\core\batcher.py:101: batch (from tensorflow.python.training.input) is deprecated and will be removed in a future version.
Instructions for updating:
Queue-based input pipelines have been replaced by tf.data. Use tf.data.Dataset.batch(batch_size) (or padded_batch(...) if dynamic_pad=True).
W0119 07:44:55.453238 16104 deprecation.py:323] From C:\Users\sikli\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\training\input.py:753: QueueRunner.init (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.
Instructions for updating:
To construct input pipelines, use the tf.data module.
W0119 07:44:55.455227 16104 deprecation.py:323] From C:\Users\sikli\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\training\input.py:753: add_queue_runner (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.
Instructions for updating:
To construct input pipelines, use the tf.data module.
W0119 07:44:55.462210 16104 deprecation_wrapper.py:119] From D:\object_detection\models\research\object_detection\core\prefetcher.py:58: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead.

W0119 07:44:55.549862 16104 deprecation_wrapper.py:119] From D:\object_detection\models\research\object_detection\core\preprocessor.py:2412: The name tf.image.resize_images is deprecated. Please use tf.image.resize instead.

W0119 07:44:55.585609 16104 deprecation.py:506] From C:\Users\sikli\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\ops\init_ops.py:1251: calling VarianceScaling.init (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
I0119 07:44:57.120883 16104 regularizers.py:98] Scale of 0 disables regularizer.
I0119 07:44:57.132852 16104 regularizers.py:98] Scale of 0 disables regularizer.
W0119 07:44:57.132852 16104 deprecation_wrapper.py:119] From D:\object_detection\models\research\object_detection\predictors\convolutional_box_predictor.py:150: The name tf.logging.info is deprecated. Please use tf.compat.v1.logging.info instead.

I0119 07:44:57.133893 16104 convolutional_box_predictor.py:151] depth of additional conv before box predictor: 0
W0119 07:44:57.167834 16104 deprecation.py:323] From D:\object_detection\models\research\object_detection\core\box_list_ops.py:174: add_dispatch_support..wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
W0119 07:44:57.849291 16104 deprecation.py:506] From D:\object_detection\models\research\object_detection\utils\spatial_transform_ops.py:418: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version.
Instructions for updating:
box_ind is deprecated, use box_indices instead
W0119 07:44:58.314076 16104 deprecation.py:323] From C:\Users\sikli\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\contrib\layers\python\layers\layers.py:1634: flatten (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.flatten instead.
I0119 07:44:58.465697 16104 regularizers.py:98] Scale of 0 disables regularizer.
I0119 07:44:58.662034 16104 regularizers.py:98] Scale of 0 disables regularizer.
W0119 07:44:58.845309 16104 deprecation_wrapper.py:119] From D:\object_detection\models\research\object_detection\core\losses.py:177: The name tf.losses.huber_loss is deprecated. Please use tf.compat.v1.losses.huber_loss instead.

W0119 07:44:58.848299 16104 deprecation_wrapper.py:119] From D:\object_detection\models\research\object_detection\core\losses.py:183: The name tf.losses.Reduction is deprecated. Please use tf.compat.v1.losses.Reduction instead.

W0119 07:44:58.883206 16104 deprecation.py:323] From D:\object_detection\models\research\object_detection\core\losses.py:350: softmax_cross_entropy_with_logits (from tensorflow.python.ops.nn_ops) is deprecated and will be removed in a future version.
Instructions for updating:

Future major versions of TensorFlow will allow gradients to flow
into the labels input on backprop by default.

See tf.nn.softmax_cross_entropy_with_logits_v2.

W0119 07:44:59.042820 16104 deprecation_wrapper.py:119] From D:\object_detection\models\research\object_detection\legacy\trainer.py:208: The name tf.losses.add_loss is deprecated. Please use tf.compat.v1.losses.add_loss instead.

W0119 07:44:59.043818 16104 deprecation_wrapper.py:119] From D:\object_detection\models\research\object_detection\builders\optimizer_builder.py:95: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.

W0119 07:44:59.050174 16104 deprecation_wrapper.py:119] From D:\object_detection\models\research\object_detection\builders\optimizer_builder.py:58: The name tf.train.MomentumOptimizer is deprecated. Please use tf.compat.v1.train.MomentumOptimizer instead.

C:\Users\sikli\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\ops\gradients_util.py:93: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
"Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
C:\Users\sikli\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\ops\gradients_util.py:93: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
"Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
W0119 07:45:02.398477 16104 deprecation_wrapper.py:119] From D:\object_detection\models\research\object_detection\legacy\trainer.py:353: The name tf.summary.histogram is deprecated. Please use tf.compat.v1.summary.histogram instead.

W0119 07:45:02.634845 16104 deprecation_wrapper.py:119] From D:\object_detection\models\research\object_detection\legacy\trainer.py:355: The name tf.losses.get_losses is deprecated. Please use tf.compat.v1.losses.get_losses instead.

W0119 07:45:02.638110 16104 deprecation_wrapper.py:119] From D:\object_detection\models\research\object_detection\legacy\trainer.py:359: The name tf.losses.get_total_loss is deprecated. Please use tf.compat.v1.losses.get_total_loss instead.

W0119 07:45:02.641516 16104 deprecation_wrapper.py:119] From D:\object_detection\models\research\object_detection\legacy\trainer.py:368: The name tf.summary.merge is deprecated. Please use tf.compat.v1.summary.merge instead.

W0119 07:45:02.648494 16104 deprecation_wrapper.py:119] From D:\object_detection\models\research\object_detection\legacy\trainer.py:376: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

W0119 07:45:02.958678 16104 deprecation.py:323] From D:\object_detection\models\research\object_detection\meta_architectures\faster_rcnn_meta_arch.py:2650: get_or_create_global_step (from tensorflow.contrib.framework.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.get_or_create_global_step
W0119 07:45:02.959645 16104 deprecation_wrapper.py:119] From D:\object_detection\models\research\object_detection\utils\variables_helper.py:139: The name tf.train.NewCheckpointReader is deprecated. Please use tf.compat.v1.train.NewCheckpointReader instead.

W0119 07:45:03.016367 16104 variables_helper.py:157] Variable [Conv/biases/Momentum] is not available in checkpoint
W0119 07:45:03.017611 16104 variables_helper.py:157] Variable [Conv/weights/Momentum] is not available in checkpoint
W0119 07:45:03.019615 16104 variables_helper.py:157] Variable [FirstStageBoxPredictor/BoxEncodingPredictor/biases/Momentum] is not available in checkpoint
W0119 07:45:03.021570 16104 variables_helper.py:157] Variable [FirstStageBoxPredictor/BoxEncodingPredictor/weights/Momentum] is not available in checkpoint
W0119 07:45:03.023555 16104 variables_helper.py:157] Variable [FirstStageBoxPredictor/ClassPredictor/biases/Momentum] is not available in checkpoint
W0119 07:45:03.024514 16104 variables_helper.py:157] Variable [FirstStageBoxPredictor/ClassPredictor/weights/Momentum] is not available in checkpoint
W0119 07:45:03.025552 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Conv2d_1a_7x7/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.026554 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Conv2d_1a_7x7/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.027547 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Conv2d_1a_7x7/depthwise_weights/Momentum] is not available in checkpoint
W0119 07:45:03.031520 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Conv2d_1a_7x7/pointwise_weights/Momentum] is not available in checkpoint
W0119 07:45:03.032528 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Conv2d_2b_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.033521 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Conv2d_2b_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.034533 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Conv2d_2b_1x1/weights/Momentum] is not available in checkpoint
W0119 07:45:03.035532 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Conv2d_2c_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.037478 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Conv2d_2c_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.038510 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Conv2d_2c_3x3/weights/Momentum] is not available in checkpoint
W0119 07:45:03.039474 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_0/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.043494 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_0/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.044461 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_0/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
W0119 07:45:03.045455 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_1/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.046460 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_1/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.047483 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_1/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
W0119 07:45:03.048495 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_1/Conv2d_0b_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.049495 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_1/Conv2d_0b_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.050475 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_1/Conv2d_0b_3x3/weights/Momentum] is not available in checkpoint
W0119 07:45:03.054436 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_2/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.055462 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_2/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.056430 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_2/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
W0119 07:45:03.057461 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_2/Conv2d_0b_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.058454 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_2/Conv2d_0b_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.059458 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_2/Conv2d_0b_3x3/weights/Momentum] is not available in checkpoint
W0119 07:45:03.060448 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_2/Conv2d_0c_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.061448 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_2/Conv2d_0c_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.062446 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_2/Conv2d_0c_3x3/weights/Momentum] is not available in checkpoint
W0119 07:45:03.066401 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_3/Conv2d_0b_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.066401 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_3/Conv2d_0b_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.067399 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_3/Conv2d_0b_1x1/weights/Momentum] is not available in checkpoint
W0119 07:45:03.068425 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_0/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.068425 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_0/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.069422 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_0/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
W0119 07:45:03.070429 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_1/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.071387 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_1/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.071387 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_1/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
W0119 07:45:03.072414 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_1/Conv2d_0b_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.072414 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_1/Conv2d_0b_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.073412 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_1/Conv2d_0b_3x3/weights/Momentum] is not available in checkpoint
W0119 07:45:03.074378 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_2/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.075404 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_2/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.075404 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_2/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
W0119 07:45:03.076371 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_2/Conv2d_0b_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.076371 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_2/Conv2d_0b_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.076371 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_2/Conv2d_0b_3x3/weights/Momentum] is not available in checkpoint
W0119 07:45:03.077369 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_2/Conv2d_0c_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.077369 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_2/Conv2d_0c_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.077369 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_2/Conv2d_0c_3x3/weights/Momentum] is not available in checkpoint
W0119 07:45:03.078366 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_3/Conv2d_0b_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.078366 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_3/Conv2d_0b_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.078366 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_3/Conv2d_0b_1x1/weights/Momentum] is not available in checkpoint
W0119 07:45:03.079364 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4a/Branch_0/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.079364 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4a/Branch_0/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.079364 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4a/Branch_0/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
W0119 07:45:03.079364 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4a/Branch_0/Conv2d_1a_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.080360 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4a/Branch_0/Conv2d_1a_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.080360 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4a/Branch_0/Conv2d_1a_3x3/weights/Momentum] is not available in checkpoint
W0119 07:45:03.081358 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4a/Branch_1/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.081358 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4a/Branch_1/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.081358 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4a/Branch_1/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
W0119 07:45:03.081358 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4a/Branch_1/Conv2d_0b_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.082355 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4a/Branch_1/Conv2d_0b_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.082355 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4a/Branch_1/Conv2d_0b_3x3/weights/Momentum] is not available in checkpoint
W0119 07:45:03.082355 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4a/Branch_1/Conv2d_1a_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.083353 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4a/Branch_1/Conv2d_1a_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.083353 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4a/Branch_1/Conv2d_1a_3x3/weights/Momentum] is not available in checkpoint
W0119 07:45:03.083353 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4b/Branch_0/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.083353 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4b/Branch_0/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.084350 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4b/Branch_0/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
W0119 07:45:03.085956 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4b/Branch_1/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.085956 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4b/Branch_1/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.085956 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4b/Branch_1/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
W0119 07:45:03.085956 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4b/Branch_1/Conv2d_0b_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.086956 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4b/Branch_1/Conv2d_0b_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.086956 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4b/Branch_1/Conv2d_0b_3x3/weights/Momentum] is not available in checkpoint
W0119 07:45:03.086956 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4b/Branch_2/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.087954 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4b/Branch_2/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.087954 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4b/Branch_2/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
W0119 07:45:03.087954 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4b/Branch_2/Conv2d_0b_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.087954 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4b/Branch_2/Conv2d_0b_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.088951 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4b/Branch_2/Conv2d_0b_3x3/weights/Momentum] is not available in checkpoint
W0119 07:45:03.088951 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4b/Branch_2/Conv2d_0c_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.088951 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4b/Branch_2/Conv2d_0c_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.088951 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4b/Branch_2/Conv2d_0c_3x3/weights/Momentum] is not available in checkpoint
W0119 07:45:03.089948 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4b/Branch_3/Conv2d_0b_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.089948 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4b/Branch_3/Conv2d_0b_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.089948 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4b/Branch_3/Conv2d_0b_1x1/weights/Momentum] is not available in checkpoint
W0119 07:45:03.090946 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4c/Branch_0/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.090946 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4c/Branch_0/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.090946 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4c/Branch_0/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
W0119 07:45:03.090946 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4c/Branch_1/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.091943 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4c/Branch_1/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.091943 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4c/Branch_1/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
W0119 07:45:03.091943 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4c/Branch_1/Conv2d_0b_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.091943 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4c/Branch_1/Conv2d_0b_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.092940 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4c/Branch_1/Conv2d_0b_3x3/weights/Momentum] is not available in checkpoint
W0119 07:45:03.092940 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4c/Branch_2/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.092940 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4c/Branch_2/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.092940 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4c/Branch_2/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
W0119 07:45:03.093938 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4c/Branch_2/Conv2d_0b_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.094935 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4c/Branch_2/Conv2d_0b_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.094935 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4c/Branch_2/Conv2d_0b_3x3/weights/Momentum] is not available in checkpoint
W0119 07:45:03.095933 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4c/Branch_2/Conv2d_0c_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.095933 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4c/Branch_2/Conv2d_0c_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.095933 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4c/Branch_2/Conv2d_0c_3x3/weights/Momentum] is not available in checkpoint
W0119 07:45:03.095933 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4c/Branch_3/Conv2d_0b_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.096930 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4c/Branch_3/Conv2d_0b_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.096930 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4c/Branch_3/Conv2d_0b_1x1/weights/Momentum] is not available in checkpoint
W0119 07:45:03.096930 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4d/Branch_0/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.096930 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4d/Branch_0/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.097928 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4d/Branch_0/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
W0119 07:45:03.097928 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4d/Branch_1/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.097928 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4d/Branch_1/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.097928 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4d/Branch_1/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
W0119 07:45:03.098925 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4d/Branch_1/Conv2d_0b_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.098925 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4d/Branch_1/Conv2d_0b_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.098925 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4d/Branch_1/Conv2d_0b_3x3/weights/Momentum] is not available in checkpoint
W0119 07:45:03.098925 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4d/Branch_2/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.099922 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4d/Branch_2/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.099922 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4d/Branch_2/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
W0119 07:45:03.099922 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4d/Branch_2/Conv2d_0b_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.100919 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4d/Branch_2/Conv2d_0b_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.100919 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4d/Branch_2/Conv2d_0b_3x3/weights/Momentum] is not available in checkpoint
W0119 07:45:03.100919 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4d/Branch_2/Conv2d_0c_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.100919 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4d/Branch_2/Conv2d_0c_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.101917 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4d/Branch_2/Conv2d_0c_3x3/weights/Momentum] is not available in checkpoint
W0119 07:45:03.101917 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4d/Branch_3/Conv2d_0b_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.101917 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4d/Branch_3/Conv2d_0b_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.101917 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4d/Branch_3/Conv2d_0b_1x1/weights/Momentum] is not available in checkpoint
W0119 07:45:03.102913 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4e/Branch_0/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.102913 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4e/Branch_0/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.102913 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4e/Branch_0/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
W0119 07:45:03.102913 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4e/Branch_1/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.104908 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4e/Branch_1/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.104908 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4e/Branch_1/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
W0119 07:45:03.105906 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4e/Branch_1/Conv2d_0b_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.105906 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4e/Branch_1/Conv2d_0b_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.105906 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4e/Branch_1/Conv2d_0b_3x3/weights/Momentum] is not available in checkpoint
W0119 07:45:03.105906 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4e/Branch_2/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.106902 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4e/Branch_2/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.107217 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4e/Branch_2/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
W0119 07:45:03.107217 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4e/Branch_2/Conv2d_0b_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.107217 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4e/Branch_2/Conv2d_0b_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.107217 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4e/Branch_2/Conv2d_0b_3x3/weights/Momentum] is not available in checkpoint
W0119 07:45:03.108216 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4e/Branch_2/Conv2d_0c_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.108216 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4e/Branch_2/Conv2d_0c_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.108216 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4e/Branch_2/Conv2d_0c_3x3/weights/Momentum] is not available in checkpoint
W0119 07:45:03.108216 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4e/Branch_3/Conv2d_0b_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.109214 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4e/Branch_3/Conv2d_0b_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.109214 16104 variables_helper.py:157] Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4e/Branch_3/Conv2d_0b_1x1/weights/Momentum] is not available in checkpoint
W0119 07:45:03.109214 16104 variables_helper.py:154] Variable [SecondStageBoxPredictor/BoxEncodingPredictor/biases] is available in checkpoint, but has an incompatible shape with model variable. Checkpoint shape: [[360]], model variable shape: [[4]]. This variable will not be initialized from the checkpoint.
W0119 07:45:03.109214 16104 variables_helper.py:157] Variable [SecondStageBoxPredictor/BoxEncodingPredictor/biases/Momentum] is not available in checkpoint
W0119 07:45:03.110212 16104 variables_helper.py:154] Variable [SecondStageBoxPredictor/BoxEncodingPredictor/weights] is available in checkpoint, but has an incompatible shape with model variable. Checkpoint shape: [[1024, 360]], model variable shape: [[1024, 4]]. This variable will not be initialized from the checkpoint.
W0119 07:45:03.110212 16104 variables_helper.py:157] Variable [SecondStageBoxPredictor/BoxEncodingPredictor/weights/Momentum] is not available in checkpoint
W0119 07:45:03.110212 16104 variables_helper.py:154] Variable [SecondStageBoxPredictor/ClassPredictor/biases] is available in checkpoint, but has an incompatible shape with model variable. Checkpoint shape: [[91]], model variable shape: [[2]]. This variable will not be initialized from the checkpoint.
W0119 07:45:03.111209 16104 variables_helper.py:157] Variable [SecondStageBoxPredictor/ClassPredictor/biases/Momentum] is not available in checkpoint
W0119 07:45:03.111209 16104 variables_helper.py:154] Variable [SecondStageBoxPredictor/ClassPredictor/weights] is available in checkpoint, but has an incompatible shape with model variable. Checkpoint shape: [[1024, 91]], model variable shape: [[1024, 2]]. This variable will not be initialized from the checkpoint.
W0119 07:45:03.111209 16104 variables_helper.py:157] Variable [SecondStageBoxPredictor/ClassPredictor/weights/Momentum] is not available in checkpoint
W0119 07:45:03.111209 16104 variables_helper.py:157] Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5a/Branch_0/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.112227 16104 variables_helper.py:157] Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5a/Branch_0/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.112227 16104 variables_helper.py:157] Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5a/Branch_0/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
W0119 07:45:03.112227 16104 variables_helper.py:157] Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5a/Branch_0/Conv2d_1a_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.112227 16104 variables_helper.py:157] Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5a/Branch_0/Conv2d_1a_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.113240 16104 variables_helper.py:157] Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5a/Branch_0/Conv2d_1a_3x3/weights/Momentum] is not available in checkpoint
W0119 07:45:03.113240 16104 variables_helper.py:157] Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5a/Branch_1/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.113240 16104 variables_helper.py:157] Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5a/Branch_1/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.113240 16104 variables_helper.py:157] Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5a/Branch_1/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
W0119 07:45:03.114201 16104 variables_helper.py:157] Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5a/Branch_1/Conv2d_0b_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.115223 16104 variables_helper.py:157] Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5a/Branch_1/Conv2d_0b_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.115223 16104 variables_helper.py:157] Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5a/Branch_1/Conv2d_0b_3x3/weights/Momentum] is not available in checkpoint
W0119 07:45:03.116195 16104 variables_helper.py:157] Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5a/Branch_1/Conv2d_1a_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.116195 16104 variables_helper.py:157] Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5a/Branch_1/Conv2d_1a_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.116195 16104 variables_helper.py:157] Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5a/Branch_1/Conv2d_1a_3x3/weights/Momentum] is not available in checkpoint
W0119 07:45:03.117192 16104 variables_helper.py:157] Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5b/Branch_0/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.117192 16104 variables_helper.py:157] Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5b/Branch_0/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.117192 16104 variables_helper.py:157] Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5b/Branch_0/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
W0119 07:45:03.117192 16104 variables_helper.py:157] Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5b/Branch_1/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.118190 16104 variables_helper.py:157] Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5b/Branch_1/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.118190 16104 variables_helper.py:157] Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5b/Branch_1/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
W0119 07:45:03.118190 16104 variables_helper.py:157] Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5b/Branch_1/Conv2d_0b_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.118190 16104 variables_helper.py:157] Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5b/Branch_1/Conv2d_0b_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.119188 16104 variables_helper.py:157] Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5b/Branch_1/Conv2d_0b_3x3/weights/Momentum] is not available in checkpoint
W0119 07:45:03.119188 16104 variables_helper.py:157] Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5b/Branch_2/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.119188 16104 variables_helper.py:157] Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5b/Branch_2/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.119188 16104 variables_helper.py:157] Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5b/Branch_2/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
W0119 07:45:03.120185 16104 variables_helper.py:157] Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5b/Branch_2/Conv2d_0b_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.120185 16104 variables_helper.py:157] Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5b/Branch_2/Conv2d_0b_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.120185 16104 variables_helper.py:157] Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5b/Branch_2/Conv2d_0b_3x3/weights/Momentum] is not available in checkpoint
W0119 07:45:03.120185 16104 variables_helper.py:157] Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5b/Branch_2/Conv2d_0c_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.121182 16104 variables_helper.py:157] Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5b/Branch_2/Conv2d_0c_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.121182 16104 variables_helper.py:157] Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5b/Branch_2/Conv2d_0c_3x3/weights/Momentum] is not available in checkpoint
W0119 07:45:03.121182 16104 variables_helper.py:157] Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5b/Branch_3/Conv2d_0b_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.122179 16104 variables_helper.py:157] Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5b/Branch_3/Conv2d_0b_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.122179 16104 variables_helper.py:157] Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5b/Branch_3/Conv2d_0b_1x1/weights/Momentum] is not available in checkpoint
W0119 07:45:03.122179 16104 variables_helper.py:157] Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_0/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.122179 16104 variables_helper.py:157] Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_0/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.123200 16104 variables_helper.py:157] Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_0/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
W0119 07:45:03.123200 16104 variables_helper.py:157] Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_1/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.123200 16104 variables_helper.py:157] Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_1/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.123200 16104 variables_helper.py:157] Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_1/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
W0119 07:45:03.124174 16104 variables_helper.py:157] Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_1/Conv2d_0b_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.125195 16104 variables_helper.py:157] Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_1/Conv2d_0b_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.125195 16104 variables_helper.py:157] Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_1/Conv2d_0b_3x3/weights/Momentum] is not available in checkpoint
W0119 07:45:03.126169 16104 variables_helper.py:157] Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_2/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.126169 16104 variables_helper.py:157] Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_2/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.126169 16104 variables_helper.py:157] Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_2/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
W0119 07:45:03.126169 16104 variables_helper.py:157] Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_2/Conv2d_0b_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.127167 16104 variables_helper.py:157] Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_2/Conv2d_0b_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.127167 16104 variables_helper.py:157] Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_2/Conv2d_0b_3x3/weights/Momentum] is not available in checkpoint
W0119 07:45:03.127167 16104 variables_helper.py:157] Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_2/Conv2d_0c_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.128164 16104 variables_helper.py:157] Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_2/Conv2d_0c_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.128164 16104 variables_helper.py:157] Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_2/Conv2d_0c_3x3/weights/Momentum] is not available in checkpoint
W0119 07:45:03.128164 16104 variables_helper.py:157] Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_3/Conv2d_0b_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
W0119 07:45:03.128164 16104 variables_helper.py:157] Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_3/Conv2d_0b_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
W0119 07:45:03.129166 16104 variables_helper.py:157] Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_3/Conv2d_0b_1x1/weights/Momentum] is not available in checkpoint
W0119 07:45:03.129166 16104 variables_helper.py:157] Variable [global_step] is not available in checkpoint
W0119 07:45:03.567018 16104 deprecation.py:323] From C:\Users\sikli\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\contrib\slim\python\slim\learning.py:742: Supervisor.init (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.MonitoredTrainingSession
2020-01-19 07:45:04.265341: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
2020-01-19 07:45:04.272075: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library nvcuda.dll
2020-01-19 07:45:04.307024: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: GeForce GTX 1060 6GB major: 6 minor: 1 memoryClockRate(GHz): 1.7085
pciBusID: 0000:01:00.0
2020-01-19 07:45:04.312789: I tensorflow/stream_executor/platform/default/dlopen_checker_stub.cc:25] GPU libraries are statically linked, skip dlopen check.
2020-01-19 07:45:04.317353: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2020-01-19 07:45:07.724720: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-01-19 07:45:07.729866: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187] 0
2020-01-19 07:45:07.732434: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0: N
2020-01-19 07:45:07.735962: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4712 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060 6GB, pci bus id: 0000:01:00.0, compute capability: 6.1)
W0119 07:45:07.766932 16104 deprecation.py:323] From C:\Users\sikli\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\training\saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
I0119 07:45:07.771516 16104 saver.py:1280] Restoring parameters from training/model.ckpt-6507
W0119 07:45:09.245152 16104 deprecation.py:323] From C:\Users\sikli\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\training\saver.py:1066: get_checkpoint_mtimes (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file utilities to get mtimes.
I0119 07:45:09.265403 16104 session_manager.py:500] Running local_init_op.
I0119 07:45:09.730169 16104 session_manager.py:502] Done running local_init_op.
I0119 07:45:14.320900 16104 learning.py:754] Starting Session.
I0119 07:45:14.481439 5544 supervisor.py:1117] Saving checkpoint to path training/model.ckpt
I0119 07:45:14.485440 16104 learning.py:768] Starting Queues.
I0119 07:45:16.491874 5008 supervisor.py:1099] global_step/sec: 0
2020-01-19 07:45:25.973480: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:111] Filling up shuffle buffer (this may take a while): 156 of 2048
2020-01-19 07:45:35.907757: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:111] Filling up shuffle buffer (this may take a while): 388 of 2048
2020-01-19 07:45:45.932737: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:111] Filling up shuffle buffer (this may take a while): 570 of 2048
2020-01-19 07:45:56.165027: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:111] Filling up shuffle buffer (this may take a while): 806 of 2048
2020-01-19 07:46:05.964172: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:111] Filling up shuffle buffer (this may take a while): 944 of 2048
2020-01-19 07:46:15.900463: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:111] Filling up shuffle buffer (this may take a while): 1126 of 2048
2020-01-19 07:46:25.911632: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:111] Filling up shuffle buffer (this may take a while): 1309 of 2048
2020-01-19 07:46:35.950018: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:111] Filling up shuffle buffer (this may take a while): 1504 of 2048
2020-01-19 07:46:45.966327: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:111] Filling up shuffle buffer (this may take a while): 1666 of 2048
2020-01-19 07:46:55.893004: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:111] Filling up shuffle buffer (this may take a while): 1847 of 2048
2020-01-19 07:47:05.908988: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:111] Filling up shuffle buffer (this may take a while): 2024 of 2048
2020-01-19 07:47:07.327176: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:162] Shuffle buffer filled.
I0119 07:47:31.706453 5008 supervisor.py:1099] global_step/sec: 0
2020-01-19 07:47:42.540502: W tensorflow/core/common_runtime/bfc_allocator.cc:237] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.83GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
I0119 07:47:43.627555 16064 supervisor.py:1050] Recording summary at step 6507.