2022-11-01 12:40:39,443 [INFO] root: Registry: ['nvcr.io'] 2022-11-01 12:40:39,612 [INFO] tlt.components.instance_handler.local_instance: Running command in container: nvcr.io/nvidia/tao/tao-toolkit-tf:v3.21.11-tf1.15.5-py3 Matplotlib created a temporary config/cache directory at /tmp/matplotlib-k_5rdqgj because the default path (/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing. Using TensorFlow backend. WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them. Using TensorFlow backend. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/hooks/checkpoint_saver_hook.py:21: The name tf.train.CheckpointSaverHook is deprecated. Please use tf.estimator.CheckpointSaverHook instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/hooks/pretrained_restore_hook.py:23: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/hooks/pretrained_restore_hook.py:23: The name tf.logging.WARN is deprecated. Please use tf.compat.v1.logging.WARN instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/scripts/train.py:410: The name tf.logging.INFO is deprecated. Please use tf.compat.v1.logging.INFO instead. Loading experiment spec at /workspace/tao-experiments/specs/unet_train_resnet_6S100.txt. 2022-11-01 10:40:47,277 [INFO] __main__: Loading experiment spec at /workspace/tao-experiments/specs/unet_train_resnet_6S100.txt. 2022-11-01 10:40:47,279 [INFO] iva.unet.spec_handler.spec_loader: Merging specification from /workspace/tao-experiments/specs/unet_train_resnet_6S100.txt 2022-11-01 10:40:47,281 [INFO] root: Initializing the pre-trained weights from /workspace/tao-experiments/pretrained_resnet18/resnet_18.hdf5 WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead. 2022-11-01 10:40:47,284 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead. 2022-11-01 10:40:47,293 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:245: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead. 2022-11-01 10:40:47,303 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:245: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1834: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead. 2022-11-01 10:40:47,311 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1834: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:133: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead. 2022-11-01 10:40:47,315 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:133: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead. WARNING:tensorflow:From /opt/nvidia/third_party/keras/tensorflow_backend.py:187: The name tf.nn.avg_pool is deprecated. Please use tf.nn.avg_pool2d instead. 2022-11-01 10:40:47,954 [WARNING] tensorflow: From /opt/nvidia/third_party/keras/tensorflow_backend.py:187: The name tf.nn.avg_pool is deprecated. Please use tf.nn.avg_pool2d instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead. 2022-11-01 10:40:48,152 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:190: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead. 2022-11-01 10:40:48,152 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:190: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead. 2022-11-01 10:40:48,152 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead. 2022-11-01 10:40:48,383 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:95: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead. 2022-11-01 10:40:48,886 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:95: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead. 2022-11-01 10:40:48,904 [INFO] iva.unet.model.utilities: Label Id 0: Train Id 0 2022-11-01 10:40:48,904 [INFO] iva.unet.model.utilities: Label Id 1: Train Id 1 2022-11-01 10:40:48,907 [INFO] iva.unet.hooks.latest_checkpoint: Getting the latest checkpoint for restoring /workspace/tao-experiments/unpruned/model.step-0.tlt INFO:tensorflow:Using config: {'_model_dir': '/workspace/tao-experiments/unpruned', '_tf_random_seed': None, '_save_summary_steps': 5, '_save_checkpoints_steps': None, '_save_checkpoints_secs': None, '_session_config': intra_op_parallelism_threads: 1 inter_op_parallelism_threads: 38 gpu_options { allow_growth: true visible_device_list: "0" force_gpu_compatible: true } allow_soft_placement: true , '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': None, '_log_step_count_steps': None, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': , '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1} 2022-11-01 10:40:52,573 [INFO] tensorflow: Using config: {'_model_dir': '/workspace/tao-experiments/unpruned', '_tf_random_seed': None, '_save_summary_steps': 5, '_save_checkpoints_steps': None, '_save_checkpoints_secs': None, '_session_config': intra_op_parallelism_threads: 1 inter_op_parallelism_threads: 38 gpu_options { allow_growth: true visible_device_list: "0" force_gpu_compatible: true } allow_soft_placement: true , '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': None, '_log_step_count_steps': None, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': , '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1} Phase train: Total 500 files. 2022-11-01 10:40:52,589 [INFO] iva.unet.model.utilities: The total number of training samples 500 and the batch size per GPU 6 2022-11-01 10:40:52,589 [INFO] iva.unet.model.utilities: Cannot iterate over exactly 500 samples with a batch size of 6; each epoch will therefore take one extra step. 2022-11-01 10:40:52,589 [INFO] iva.unet.model.utilities: Steps per epoch taken: 84 Running for 100 Epochs 2022-11-01 10:40:52,589 [INFO] __main__: Running for 100 Epochs INFO:tensorflow:Create CheckpointSaverHook. 2022-11-01 10:40:52,590 [INFO] tensorflow: Create CheckpointSaverHook. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead. 2022-11-01 10:40:52,928 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead. WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2022-11-01 10:40:52,964 [WARNING] tensorflow: Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code WARNING:tensorflow:Entity . at 0x7fd1ad31bd90> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of . at 0x7fd1ad31bd90>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2022-11-01 10:40:52,983 [WARNING] tensorflow: Entity . at 0x7fd1ad31bd90> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of . at 0x7fd1ad31bd90>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2022-11-01 10:40:52,990 [WARNING] tensorflow: Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2022-11-01 10:40:52,995 [WARNING] tensorflow: Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2022-11-01 10:40:53,000 [WARNING] tensorflow: Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/utils/data_loader.py:451: The name tf.image.resize_images is deprecated. Please use tf.image.resize instead. 2022-11-01 10:40:53,001 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/utils/data_loader.py:451: The name tf.image.resize_images is deprecated. Please use tf.image.resize instead. WARNING:tensorflow:Entity . at 0x7fd1993337b8> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of . at 0x7fd1993337b8>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2022-11-01 10:40:53,010 [WARNING] tensorflow: Entity . at 0x7fd1993337b8> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of . at 0x7fd1993337b8>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code WARNING:tensorflow:Entity . at 0x7fd199333a60> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of . at 0x7fd199333a60>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2022-11-01 10:40:53,016 [WARNING] tensorflow: Entity . at 0x7fd199333a60> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of . at 0x7fd199333a60>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code WARNING:tensorflow:Entity . at 0x7fd199333e18> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of . at 0x7fd199333e18>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2022-11-01 10:40:53,021 [WARNING] tensorflow: Entity . at 0x7fd199333e18> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of . at 0x7fd199333e18>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2022-11-01 10:40:53,026 [WARNING] tensorflow: Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code WARNING:tensorflow:Entity . at 0x7fd1996f16a8> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of . at 0x7fd1996f16a8>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2022-11-01 10:40:53,033 [WARNING] tensorflow: Entity . at 0x7fd1996f16a8> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of . at 0x7fd1996f16a8>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code INFO:tensorflow:Calling model_fn. 2022-11-01 10:40:53,051 [INFO] tensorflow: Calling model_fn. 2022-11-01 10:40:53,051 [INFO] iva.unet.utils.model_fn: {'exec_mode': 'train', 'model_dir': '/workspace/tao-experiments/unpruned', 'resize_padding': False, 'resize_method': 'BILINEAR', 'log_dir': None, 'batch_size': 6, 'learning_rate': 9.999999747378752e-05, 'activation': 'softmax', 'crossvalidation_idx': None, 'max_steps': None, 'regularizer_type': 2, 'weight_decay': 1.9999999494757503e-05, 'log_summary_steps': 10, 'warmup_steps': 0, 'augment': False, 'use_amp': False, 'use_trt': False, 'use_xla': False, 'loss': 'cross_entropy', 'epochs': 100, 'pretrained_weights_file': None, 'lr_scheduler': None, 'unet_model': , 'key': '6S004', 'experiment_spec': random_seed: 42 dataset_config { dataset: "custom" input_image_type: "grayscale" train_images_path: "/workspace/tao-experiments/data/images/train" train_masks_path: "/workspace/tao-experiments/data/masks/train" val_images_path: "/workspace/tao-experiments/data/images/val" val_masks_path: "/workspace/tao-experiments/data/masks/val" test_images_path: "/workspace/tao-experiments/data/images/test" data_class_config { target_classes { name: "foreground" mapping_class: "foreground" } target_classes { name: "background" label_id: 1 mapping_class: "background" } } augmentation_config { spatial_augmentation { hflip_probability: 0.5 vflip_probability: 0.5 crop_and_resize_prob: 0.5 } brightness_augmentation { delta: 0.20000000298023224 } } } model_config { num_layers: 18 training_precision { backend_floatx: FLOAT32 } arch: "resnet" all_projections: true model_input_height: 704 model_input_width: 1280 model_input_channels: 1 } training_config { batch_size: 6 regularizer { type: L2 weight: 1.9999999494757503e-05 } optimizer { adam { epsilon: 9.99999993922529e-09 beta1: 0.8999999761581421 beta2: 0.9990000128746033 } } checkpoint_interval: 1 log_summary_steps: 10 learning_rate: 9.999999747378752e-05 loss: "cross_entropy" epochs: 100 } , 'seed': 42, 'benchmark': False, 'temp_dir': '/tmp/tmpnsaok_d9', 'num_classes': 2, 'num_conf_mat_classes': 2, 'start_step': 0, 'checkpoint_interval': 1, 'model_json': None, 'load_graph': False, 'weights_monitor': False, 'phase': None} __________________________________________________________________________________________________ Layer (type) Output Shape Param # Connected to ================================================================================================== input_1 (InputLayer) (None, 1, 704, 1280) 0 __________________________________________________________________________________________________ conv1 (Conv2D) (None, 64, 352, 640) 3200 input_1[0][0] __________________________________________________________________________________________________ activation_1 (Activation) (None, 64, 352, 640) 0 conv1[0][0] __________________________________________________________________________________________________ block_1a_conv_1 (Conv2D) (None, 64, 176, 320) 36928 activation_1[0][0] __________________________________________________________________________________________________ block_1a_relu_1 (Activation) (None, 64, 176, 320) 0 block_1a_conv_1[0][0] __________________________________________________________________________________________________ block_1a_conv_2 (Conv2D) (None, 64, 176, 320) 36928 block_1a_relu_1[0][0] __________________________________________________________________________________________________ block_1a_conv_shortcut (Conv2D) (None, 64, 176, 320) 4160 activation_1[0][0] __________________________________________________________________________________________________ add_1 (Add) (None, 64, 176, 320) 0 block_1a_conv_2[0][0] block_1a_conv_shortcut[0][0] __________________________________________________________________________________________________ block_1a_relu (Activation) (None, 64, 176, 320) 0 add_1[0][0] __________________________________________________________________________________________________ block_1b_conv_1 (Conv2D) (None, 64, 176, 320) 36928 block_1a_relu[0][0] __________________________________________________________________________________________________ block_1b_relu_1 (Activation) (None, 64, 176, 320) 0 block_1b_conv_1[0][0] __________________________________________________________________________________________________ block_1b_conv_2 (Conv2D) (None, 64, 176, 320) 36928 block_1b_relu_1[0][0] __________________________________________________________________________________________________ block_1b_conv_shortcut (Conv2D) (None, 64, 176, 320) 4160 block_1a_relu[0][0] __________________________________________________________________________________________________ add_2 (Add) (None, 64, 176, 320) 0 block_1b_conv_2[0][0] block_1b_conv_shortcut[0][0] __________________________________________________________________________________________________ block_1b_relu (Activation) (None, 64, 176, 320) 0 add_2[0][0] __________________________________________________________________________________________________ block_2a_conv_1 (Conv2D) (None, 128, 88, 160) 73856 block_1b_relu[0][0] __________________________________________________________________________________________________ block_2a_relu_1 (Activation) (None, 128, 88, 160) 0 block_2a_conv_1[0][0] __________________________________________________________________________________________________ block_2a_conv_2 (Conv2D) (None, 128, 88, 160) 147584 block_2a_relu_1[0][0] __________________________________________________________________________________________________ block_2a_conv_shortcut (Conv2D) (None, 128, 88, 160) 8320 block_1b_relu[0][0] __________________________________________________________________________________________________ add_3 (Add) (None, 128, 88, 160) 0 block_2a_conv_2[0][0] block_2a_conv_shortcut[0][0] __________________________________________________________________________________________________ block_2a_relu (Activation) (None, 128, 88, 160) 0 add_3[0][0] __________________________________________________________________________________________________ block_2b_conv_1 (Conv2D) (None, 128, 88, 160) 147584 block_2a_relu[0][0] __________________________________________________________________________________________________ block_2b_relu_1 (Activation) (None, 128, 88, 160) 0 block_2b_conv_1[0][0] __________________________________________________________________________________________________ block_2b_conv_2 (Conv2D) (None, 128, 88, 160) 147584 block_2b_relu_1[0][0] __________________________________________________________________________________________________ block_2b_conv_shortcut (Conv2D) (None, 128, 88, 160) 16512 block_2a_relu[0][0] __________________________________________________________________________________________________ add_4 (Add) (None, 128, 88, 160) 0 block_2b_conv_2[0][0] block_2b_conv_shortcut[0][0] __________________________________________________________________________________________________ block_2b_relu (Activation) (None, 128, 88, 160) 0 add_4[0][0] __________________________________________________________________________________________________ block_3a_conv_1 (Conv2D) (None, 256, 44, 80) 295168 block_2b_relu[0][0] __________________________________________________________________________________________________ block_3a_relu_1 (Activation) (None, 256, 44, 80) 0 block_3a_conv_1[0][0] __________________________________________________________________________________________________ block_3a_conv_2 (Conv2D) (None, 256, 44, 80) 590080 block_3a_relu_1[0][0] __________________________________________________________________________________________________ block_3a_conv_shortcut (Conv2D) (None, 256, 44, 80) 33024 block_2b_relu[0][0] __________________________________________________________________________________________________ add_5 (Add) (None, 256, 44, 80) 0 block_3a_conv_2[0][0] block_3a_conv_shortcut[0][0] __________________________________________________________________________________________________ block_3a_relu (Activation) (None, 256, 44, 80) 0 add_5[0][0] __________________________________________________________________________________________________ block_3b_conv_1 (Conv2D) (None, 256, 44, 80) 590080 block_3a_relu[0][0] __________________________________________________________________________________________________ block_3b_relu_1 (Activation) (None, 256, 44, 80) 0 block_3b_conv_1[0][0] __________________________________________________________________________________________________ block_3b_conv_2 (Conv2D) (None, 256, 44, 80) 590080 block_3b_relu_1[0][0] __________________________________________________________________________________________________ block_3b_conv_shortcut (Conv2D) (None, 256, 44, 80) 65792 block_3a_relu[0][0] __________________________________________________________________________________________________ add_6 (Add) (None, 256, 44, 80) 0 block_3b_conv_2[0][0] block_3b_conv_shortcut[0][0] __________________________________________________________________________________________________ block_3b_relu (Activation) (None, 256, 44, 80) 0 add_6[0][0] __________________________________________________________________________________________________ block_4a_conv_1 (Conv2D) (None, 512, 44, 80) 1180160 block_3b_relu[0][0] __________________________________________________________________________________________________ block_4a_relu_1 (Activation) (None, 512, 44, 80) 0 block_4a_conv_1[0][0] __________________________________________________________________________________________________ block_4a_conv_2 (Conv2D) (None, 512, 44, 80) 2359808 block_4a_relu_1[0][0] __________________________________________________________________________________________________ block_4a_conv_shortcut (Conv2D) (None, 512, 44, 80) 131584 block_3b_relu[0][0] __________________________________________________________________________________________________ add_7 (Add) (None, 512, 44, 80) 0 block_4a_conv_2[0][0] block_4a_conv_shortcut[0][0] __________________________________________________________________________________________________ block_4a_relu (Activation) (None, 512, 44, 80) 0 add_7[0][0] __________________________________________________________________________________________________ block_4b_conv_1 (Conv2D) (None, 512, 44, 80) 2359808 block_4a_relu[0][0] __________________________________________________________________________________________________ block_4b_relu_1 (Activation) (None, 512, 44, 80) 0 block_4b_conv_1[0][0] __________________________________________________________________________________________________ block_4b_conv_2 (Conv2D) (None, 512, 44, 80) 2359808 block_4b_relu_1[0][0] __________________________________________________________________________________________________ block_4b_conv_shortcut (Conv2D) (None, 512, 44, 80) 262656 block_4a_relu[0][0] __________________________________________________________________________________________________ add_8 (Add) (None, 512, 44, 80) 0 block_4b_conv_2[0][0] block_4b_conv_shortcut[0][0] __________________________________________________________________________________________________ block_4b_relu (Activation) (None, 512, 44, 80) 0 add_8[0][0] __________________________________________________________________________________________________ conv2d_transpose_1 (Conv2DTrans (None, 256, 88, 160) 2097408 block_4b_relu[0][0] __________________________________________________________________________________________________ concatenate_1 (Concatenate) (None, 384, 88, 160) 0 conv2d_transpose_1[0][0] block_2b_relu[0][0] __________________________________________________________________________________________________ activation_2 (Activation) (None, 384, 88, 160) 0 concatenate_1[0][0] __________________________________________________________________________________________________ conv2d_1 (Conv2D) (None, 256, 88, 160) 884992 activation_2[0][0] __________________________________________________________________________________________________ activation_3 (Activation) (None, 256, 88, 160) 0 conv2d_1[0][0] __________________________________________________________________________________________________ conv2d_transpose_2 (Conv2DTrans (None, 128, 176, 320 524416 activation_3[0][0] __________________________________________________________________________________________________ concatenate_2 (Concatenate) (None, 192, 176, 320 0 conv2d_transpose_2[0][0] block_1b_relu[0][0] __________________________________________________________________________________________________ activation_4 (Activation) (None, 192, 176, 320 0 concatenate_2[0][0] __________________________________________________________________________________________________ conv2d_2 (Conv2D) (None, 128, 176, 320 221312 activation_4[0][0] __________________________________________________________________________________________________ activation_5 (Activation) (None, 128, 176, 320 0 conv2d_2[0][0] __________________________________________________________________________________________________ conv2d_transpose_3 (Conv2DTrans (None, 64, 352, 640) 131136 activation_5[0][0] __________________________________________________________________________________________________ concatenate_3 (Concatenate) (None, 128, 352, 640 0 conv2d_transpose_3[0][0] activation_1[0][0] __________________________________________________________________________________________________ activation_6 (Activation) (None, 128, 352, 640 0 concatenate_3[0][0] __________________________________________________________________________________________________ conv2d_3 (Conv2D) (None, 64, 352, 640) 73792 activation_6[0][0] __________________________________________________________________________________________________ activation_7 (Activation) (None, 64, 352, 640) 0 conv2d_3[0][0] __________________________________________________________________________________________________ conv2d_transpose_4 (Conv2DTrans (None, 64, 704, 1280 65600 activation_7[0][0] __________________________________________________________________________________________________ activation_8 (Activation) (None, 64, 704, 1280 0 conv2d_transpose_4[0][0] __________________________________________________________________________________________________ conv2d_4 (Conv2D) (None, 64, 704, 1280 36928 activation_8[0][0] __________________________________________________________________________________________________ activation_9 (Activation) (None, 64, 704, 1280 0 conv2d_4[0][0] __________________________________________________________________________________________________ conv2d_5 (Conv2D) (None, 2, 704, 1280) 1154 activation_9[0][0] ================================================================================================== Total params: 15,555,458 Trainable params: 15,555,458 Non-trainable params: 0 __________________________________________________________________________________________________ WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/utils/model_fn.py:225: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead. 2022-11-01 10:40:53,580 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/utils/model_fn.py:225: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead. INFO:tensorflow:Done calling model_fn. 2022-11-01 10:40:54,633 [INFO] tensorflow: Done calling model_fn. INFO:tensorflow:Graph was finalized. 2022-11-01 10:40:55,570 [INFO] tensorflow: Graph was finalized. INFO:tensorflow:Running local_init_op. 2022-11-01 10:40:56,356 [INFO] tensorflow: Running local_init_op. INFO:tensorflow:Done running local_init_op. 2022-11-01 10:40:56,417 [INFO] tensorflow: Done running local_init_op. [GPU] Restoring pretrained weights from: /tmp/tmpv4xu8pt5/model.ckpt-0 2022-11-01 10:40:56,904 [INFO] iva.unet.hooks.pretrained_restore_hook: Pretrained weights loaded with success... INFO:tensorflow:Saving checkpoints for step-0. 2022-11-01 10:40:58,760 [INFO] tensorflow: Saving checkpoints for step-0. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/hooks/training_hook.py:95: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead. 2022-11-01 10:41:09,860 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/hooks/training_hook.py:95: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead. Epoch: 0/100:, Cur-Step: 0, loss(cross_entropy): 0.75751, Running average loss:0.75751, Time taken: 0:00:00 ETA: 0:00:00 2022-11-01 10:41:39,256 [INFO] __main__: Epoch: 0/100:, Cur-Step: 0, loss(cross_entropy): 0.75751, Running average loss:0.75751, Time taken: 0:00:00 ETA: 0:00:00 Epoch: 0/100:, Cur-Step: 10, loss(cross_entropy): 0.72182, Running average loss:0.74228, Time taken: 0:00:00 ETA: 0:00:00 2022-11-01 10:42:13,048 [INFO] __main__: Epoch: 0/100:, Cur-Step: 10, loss(cross_entropy): 0.72182, Running average loss:0.74228, Time taken: 0:00:00 ETA: 0:00:00 Epoch: 0/100:, Cur-Step: 20, loss(cross_entropy): 0.11583, Running average loss:0.57694, Time taken: 0:00:00 ETA: 0:00:00 2022-11-01 10:42:17,709 [INFO] __main__: Epoch: 0/100:, Cur-Step: 20, loss(cross_entropy): 0.11583, Running average loss:0.57694, Time taken: 0:00:00 ETA: 0:00:00 Epoch: 0/100:, Cur-Step: 30, loss(cross_entropy): 0.07379, Running average loss:0.43292, Time taken: 0:00:00 ETA: 0:00:00 2022-11-01 10:42:22,432 [INFO] __main__: Epoch: 0/100:, Cur-Step: 30, loss(cross_entropy): 0.07379, Running average loss:0.43292, Time taken: 0:00:00 ETA: 0:00:00 Epoch: 0/100:, Cur-Step: 40, loss(cross_entropy): 0.07415, Running average loss:0.34746, Time taken: 0:00:00 ETA: 0:00:00 2022-11-01 10:42:27,178 [INFO] __main__: Epoch: 0/100:, Cur-Step: 40, loss(cross_entropy): 0.07415, Running average loss:0.34746, Time taken: 0:00:00 ETA: 0:00:00 Epoch: 0/100:, Cur-Step: 50, loss(cross_entropy): 0.06368, Running average loss:0.29400, Time taken: 0:00:00 ETA: 0:00:00 2022-11-01 10:42:31,862 [INFO] __main__: Epoch: 0/100:, Cur-Step: 50, loss(cross_entropy): 0.06368, Running average loss:0.29400, Time taken: 0:00:00 ETA: 0:00:00 Epoch: 0/100:, Cur-Step: 60, loss(cross_entropy): 0.06378, Running average loss:0.25668, Time taken: 0:00:00 ETA: 0:00:00 2022-11-01 10:42:36,589 [INFO] __main__: Epoch: 0/100:, Cur-Step: 60, loss(cross_entropy): 0.06378, Running average loss:0.25668, Time taken: 0:00:00 ETA: 0:00:00 Epoch: 0/100:, Cur-Step: 70, loss(cross_entropy): 0.05984, Running average loss:0.22931, Time taken: 0:00:00 ETA: 0:00:00 2022-11-01 10:42:41,301 [INFO] __main__: Epoch: 0/100:, Cur-Step: 70, loss(cross_entropy): 0.05984, Running average loss:0.22931, Time taken: 0:00:00 ETA: 0:00:00 Epoch: 0/100:, Cur-Step: 80, loss(cross_entropy): 0.05694, Running average loss:0.20853, Time taken: 0:00:00 ETA: 0:00:00 2022-11-01 10:42:45,953 [INFO] __main__: Epoch: 0/100:, Cur-Step: 80, loss(cross_entropy): 0.05694, Running average loss:0.20853, Time taken: 0:00:00 ETA: 0:00:00 INFO:tensorflow:Saving checkpoints for step-84. 2022-11-01 10:42:48,064 [INFO] tensorflow: Saving checkpoints for step-84. Epoch: 1/100:, Cur-Step: 90, loss(cross_entropy): 0.05722, Running average loss:0.05878, Time taken: 0:01:55.461992 ETA: 3:10:30.737187 2022-11-01 10:43:02,686 [INFO] __main__: Epoch: 1/100:, Cur-Step: 90, loss(cross_entropy): 0.05722, Running average loss:0.05878, Time taken: 0:01:55.461992 ETA: 3:10:30.737187 Epoch: 1/100:, Cur-Step: 100, loss(cross_entropy): 0.05536, Running average loss:0.05595, Time taken: 0:01:55.461992 ETA: 3:10:30.737187 2022-11-01 10:43:07,447 [INFO] __main__: Epoch: 1/100:, Cur-Step: 100, loss(cross_entropy): 0.05536, Running average loss:0.05595, Time taken: 0:01:55.461992 ETA: 3:10:30.737187 Epoch: 1/100:, Cur-Step: 110, loss(cross_entropy): 0.05312, Running average loss:0.05447, Time taken: 0:01:55.461992 ETA: 3:10:30.737187 2022-11-01 10:43:12,124 [INFO] __main__: Epoch: 1/100:, Cur-Step: 110, loss(cross_entropy): 0.05312, Running average loss:0.05447, Time taken: 0:01:55.461992 ETA: 3:10:30.737187 Epoch: 1/100:, Cur-Step: 120, loss(cross_entropy): 0.05366, Running average loss:0.05334, Time taken: 0:01:55.461992 ETA: 3:10:30.737187 2022-11-01 10:43:16,789 [INFO] __main__: Epoch: 1/100:, Cur-Step: 120, loss(cross_entropy): 0.05366, Running average loss:0.05334, Time taken: 0:01:55.461992 ETA: 3:10:30.737187 Epoch: 1/100:, Cur-Step: 130, loss(cross_entropy): 0.04715, Running average loss:0.05242, Time taken: 0:01:55.461992 ETA: 3:10:30.737187 2022-11-01 10:43:21,465 [INFO] __main__: Epoch: 1/100:, Cur-Step: 130, loss(cross_entropy): 0.04715, Running average loss:0.05242, Time taken: 0:01:55.461992 ETA: 3:10:30.737187 Epoch: 1/100:, Cur-Step: 140, loss(cross_entropy): 0.04713, Running average loss:0.05171, Time taken: 0:01:55.461992 ETA: 3:10:30.737187 2022-11-01 10:43:26,118 [INFO] __main__: Epoch: 1/100:, Cur-Step: 140, loss(cross_entropy): 0.04713, Running average loss:0.05171, Time taken: 0:01:55.461992 ETA: 3:10:30.737187 Epoch: 1/100:, Cur-Step: 150, loss(cross_entropy): 0.04558, Running average loss:0.05083, Time taken: 0:01:55.461992 ETA: 3:10:30.737187 2022-11-01 10:43:30,759 [INFO] __main__: Epoch: 1/100:, Cur-Step: 150, loss(cross_entropy): 0.04558, Running average loss:0.05083, Time taken: 0:01:55.461992 ETA: 3:10:30.737187 Epoch: 1/100:, Cur-Step: 160, loss(cross_entropy): 0.04382, Running average loss:0.05016, Time taken: 0:01:55.461992 ETA: 3:10:30.737187 2022-11-01 10:43:35,400 [INFO] __main__: Epoch: 1/100:, Cur-Step: 160, loss(cross_entropy): 0.04382, Running average loss:0.05016, Time taken: 0:01:55.461992 ETA: 3:10:30.737187 INFO:tensorflow:Saving checkpoints for step-168. 2022-11-01 10:43:38,812 [INFO] tensorflow: Saving checkpoints for step-168. Epoch: 2/100:, Cur-Step: 170, loss(cross_entropy): 0.04961, Running average loss:0.04579, Time taken: 0:00:51.901317 ETA: 1:24:46.329031 2022-11-01 10:43:51,199 [INFO] __main__: Epoch: 2/100:, Cur-Step: 170, loss(cross_entropy): 0.04961, Running average loss:0.04579, Time taken: 0:00:51.901317 ETA: 1:24:46.329031 Epoch: 2/100:, Cur-Step: 180, loss(cross_entropy): 0.04254, Running average loss:0.04378, Time taken: 0:00:51.901317 ETA: 1:24:46.329031 2022-11-01 10:43:55,872 [INFO] __main__: Epoch: 2/100:, Cur-Step: 180, loss(cross_entropy): 0.04254, Running average loss:0.04378, Time taken: 0:00:51.901317 ETA: 1:24:46.329031 Epoch: 2/100:, Cur-Step: 190, loss(cross_entropy): 0.04276, Running average loss:0.04314, Time taken: 0:00:51.901317 ETA: 1:24:46.329031 2022-11-01 10:44:00,577 [INFO] __main__: Epoch: 2/100:, Cur-Step: 190, loss(cross_entropy): 0.04276, Running average loss:0.04314, Time taken: 0:00:51.901317 ETA: 1:24:46.329031 Epoch: 2/100:, Cur-Step: 200, loss(cross_entropy): 0.04091, Running average loss:0.04271, Time taken: 0:00:51.901317 ETA: 1:24:46.329031 2022-11-01 10:44:05,270 [INFO] __main__: Epoch: 2/100:, Cur-Step: 200, loss(cross_entropy): 0.04091, Running average loss:0.04271, Time taken: 0:00:51.901317 ETA: 1:24:46.329031 Epoch: 2/100:, Cur-Step: 210, loss(cross_entropy): 0.04083, Running average loss:0.04233, Time taken: 0:00:51.901317 ETA: 1:24:46.329031 2022-11-01 10:44:09,960 [INFO] __main__: Epoch: 2/100:, Cur-Step: 210, loss(cross_entropy): 0.04083, Running average loss:0.04233, Time taken: 0:00:51.901317 ETA: 1:24:46.329031 Epoch: 2/100:, Cur-Step: 220, loss(cross_entropy): 0.04121, Running average loss:0.04199, Time taken: 0:00:51.901317 ETA: 1:24:46.329031 2022-11-01 10:44:14,631 [INFO] __main__: Epoch: 2/100:, Cur-Step: 220, loss(cross_entropy): 0.04121, Running average loss:0.04199, Time taken: 0:00:51.901317 ETA: 1:24:46.329031 Epoch: 2/100:, Cur-Step: 230, loss(cross_entropy): 0.03954, Running average loss:0.04164, Time taken: 0:00:51.901317 ETA: 1:24:46.329031 2022-11-01 10:44:19,294 [INFO] __main__: Epoch: 2/100:, Cur-Step: 230, loss(cross_entropy): 0.03954, Running average loss:0.04164, Time taken: 0:00:51.901317 ETA: 1:24:46.329031 Epoch: 2/100:, Cur-Step: 240, loss(cross_entropy): 0.04262, Running average loss:0.04141, Time taken: 0:00:51.901317 ETA: 1:24:46.329031 2022-11-01 10:44:23,987 [INFO] __main__: Epoch: 2/100:, Cur-Step: 240, loss(cross_entropy): 0.04262, Running average loss:0.04141, Time taken: 0:00:51.901317 ETA: 1:24:46.329031 Epoch: 2/100:, Cur-Step: 250, loss(cross_entropy): 0.03975, Running average loss:0.04121, Time taken: 0:00:51.901317 ETA: 1:24:46.329031 2022-11-01 10:44:28,845 [INFO] __main__: Epoch: 2/100:, Cur-Step: 250, loss(cross_entropy): 0.03975, Running average loss:0.04121, Time taken: 0:00:51.901317 ETA: 1:24:46.329031 INFO:tensorflow:Saving checkpoints for step-252. 2022-11-01 10:44:29,314 [INFO] tensorflow: Saving checkpoints for step-252. Epoch: 3/100:, Cur-Step: 260, loss(cross_entropy): 0.03734, Running average loss:0.03868, Time taken: 0:00:50.970121 ETA: 1:22:24.101728 2022-11-01 10:44:44,455 [INFO] __main__: Epoch: 3/100:, Cur-Step: 260, loss(cross_entropy): 0.03734, Running average loss:0.03868, Time taken: 0:00:50.970121 ETA: 1:22:24.101728 Epoch: 3/100:, Cur-Step: 270, loss(cross_entropy): 0.03797, Running average loss:0.03875, Time taken: 0:00:50.970121 ETA: 1:22:24.101728 2022-11-01 10:44:49,127 [INFO] __main__: Epoch: 3/100:, Cur-Step: 270, loss(cross_entropy): 0.03797, Running average loss:0.03875, Time taken: 0:00:50.970121 ETA: 1:22:24.101728 Epoch: 3/100:, Cur-Step: 280, loss(cross_entropy): 0.03964, Running average loss:0.03851, Time taken: 0:00:50.970121 ETA: 1:22:24.101728 2022-11-01 10:44:53,799 [INFO] __main__: Epoch: 3/100:, Cur-Step: 280, loss(cross_entropy): 0.03964, Running average loss:0.03851, Time taken: 0:00:50.970121 ETA: 1:22:24.101728 Epoch: 3/100:, Cur-Step: 290, loss(cross_entropy): 0.03510, Running average loss:0.03803, Time taken: 0:00:50.970121 ETA: 1:22:24.101728 2022-11-01 10:44:58,456 [INFO] __main__: Epoch: 3/100:, Cur-Step: 290, loss(cross_entropy): 0.03510, Running average loss:0.03803, Time taken: 0:00:50.970121 ETA: 1:22:24.101728 Epoch: 3/100:, Cur-Step: 300, loss(cross_entropy): 0.03783, Running average loss:0.03780, Time taken: 0:00:50.970121 ETA: 1:22:24.101728 2022-11-01 10:45:03,137 [INFO] __main__: Epoch: 3/100:, Cur-Step: 300, loss(cross_entropy): 0.03783, Running average loss:0.03780, Time taken: 0:00:50.970121 ETA: 1:22:24.101728 Epoch: 3/100:, Cur-Step: 310, loss(cross_entropy): 0.03586, Running average loss:0.03760, Time taken: 0:00:50.970121 ETA: 1:22:24.101728 2022-11-01 10:45:07,946 [INFO] __main__: Epoch: 3/100:, Cur-Step: 310, loss(cross_entropy): 0.03586, Running average loss:0.03760, Time taken: 0:00:50.970121 ETA: 1:22:24.101728 Epoch: 3/100:, Cur-Step: 320, loss(cross_entropy): 0.03641, Running average loss:0.03728, Time taken: 0:00:50.970121 ETA: 1:22:24.101728 2022-11-01 10:45:12,631 [INFO] __main__: Epoch: 3/100:, Cur-Step: 320, loss(cross_entropy): 0.03641, Running average loss:0.03728, Time taken: 0:00:50.970121 ETA: 1:22:24.101728 Epoch: 3/100:, Cur-Step: 330, loss(cross_entropy): 0.03399, Running average loss:0.03700, Time taken: 0:00:50.970121 ETA: 1:22:24.101728 2022-11-01 10:45:17,409 [INFO] __main__: Epoch: 3/100:, Cur-Step: 330, loss(cross_entropy): 0.03399, Running average loss:0.03700, Time taken: 0:00:50.970121 ETA: 1:22:24.101728 INFO:tensorflow:Saving checkpoints for step-336. 2022-11-01 10:45:19,954 [INFO] tensorflow: Saving checkpoints for step-336. Epoch: 4/100:, Cur-Step: 340, loss(cross_entropy): 0.03562, Running average loss:0.03539, Time taken: 0:00:51.094400 ETA: 1:21:45.062393 2022-11-01 10:45:33,889 [INFO] __main__: Epoch: 4/100:, Cur-Step: 340, loss(cross_entropy): 0.03562, Running average loss:0.03539, Time taken: 0:00:51.094400 ETA: 1:21:45.062393 Epoch: 4/100:, Cur-Step: 350, loss(cross_entropy): 0.03489, Running average loss:0.03581, Time taken: 0:00:51.094400 ETA: 1:21:45.062393 2022-11-01 10:45:38,677 [INFO] __main__: Epoch: 4/100:, Cur-Step: 350, loss(cross_entropy): 0.03489, Running average loss:0.03581, Time taken: 0:00:51.094400 ETA: 1:21:45.062393 Epoch: 4/100:, Cur-Step: 360, loss(cross_entropy): 0.03420, Running average loss:0.03523, Time taken: 0:00:51.094400 ETA: 1:21:45.062393 2022-11-01 10:45:43,357 [INFO] __main__: Epoch: 4/100:, Cur-Step: 360, loss(cross_entropy): 0.03420, Running average loss:0.03523, Time taken: 0:00:51.094400 ETA: 1:21:45.062393 Epoch: 4/100:, Cur-Step: 370, loss(cross_entropy): 0.03523, Running average loss:0.03492, Time taken: 0:00:51.094400 ETA: 1:21:45.062393 2022-11-01 10:45:48,033 [INFO] __main__: Epoch: 4/100:, Cur-Step: 370, loss(cross_entropy): 0.03523, Running average loss:0.03492, Time taken: 0:00:51.094400 ETA: 1:21:45.062393 Epoch: 4/100:, Cur-Step: 380, loss(cross_entropy): 0.03317, Running average loss:0.03471, Time taken: 0:00:51.094400 ETA: 1:21:45.062393 2022-11-01 10:45:52,698 [INFO] __main__: Epoch: 4/100:, Cur-Step: 380, loss(cross_entropy): 0.03317, Running average loss:0.03471, Time taken: 0:00:51.094400 ETA: 1:21:45.062393 Epoch: 4/100:, Cur-Step: 390, loss(cross_entropy): 0.03256, Running average loss:0.03447, Time taken: 0:00:51.094400 ETA: 1:21:45.062393 2022-11-01 10:45:57,383 [INFO] __main__: Epoch: 4/100:, Cur-Step: 390, loss(cross_entropy): 0.03256, Running average loss:0.03447, Time taken: 0:00:51.094400 ETA: 1:21:45.062393 Epoch: 4/100:, Cur-Step: 400, loss(cross_entropy): 0.03295, Running average loss:0.03431, Time taken: 0:00:51.094400 ETA: 1:21:45.062393 2022-11-01 10:46:02,076 [INFO] __main__: Epoch: 4/100:, Cur-Step: 400, loss(cross_entropy): 0.03295, Running average loss:0.03431, Time taken: 0:00:51.094400 ETA: 1:21:45.062393 Epoch: 4/100:, Cur-Step: 410, loss(cross_entropy): 0.03480, Running average loss:0.03430, Time taken: 0:00:51.094400 ETA: 1:21:45.062393 2022-11-01 10:46:06,772 [INFO] __main__: Epoch: 4/100:, Cur-Step: 410, loss(cross_entropy): 0.03480, Running average loss:0.03430, Time taken: 0:00:51.094400 ETA: 1:21:45.062393 INFO:tensorflow:Saving checkpoints for step-420. 2022-11-01 10:46:11,134 [INFO] tensorflow: Saving checkpoints for step-420. Epoch: 5/100:, Cur-Step: 420, loss(cross_entropy): 0.03395, Running average loss:0.03395, Time taken: 0:00:51.743382 ETA: 1:21:55.621265 2022-11-01 10:46:22,844 [INFO] __main__: Epoch: 5/100:, Cur-Step: 420, loss(cross_entropy): 0.03395, Running average loss:0.03395, Time taken: 0:00:51.743382 ETA: 1:21:55.621265 Epoch: 5/100:, Cur-Step: 430, loss(cross_entropy): 0.03145, Running average loss:0.03263, Time taken: 0:00:51.743382 ETA: 1:21:55.621265 2022-11-01 10:46:27,521 [INFO] __main__: Epoch: 5/100:, Cur-Step: 430, loss(cross_entropy): 0.03145, Running average loss:0.03263, Time taken: 0:00:51.743382 ETA: 1:21:55.621265 Epoch: 5/100:, Cur-Step: 440, loss(cross_entropy): 0.03086, Running average loss:0.03238, Time taken: 0:00:51.743382 ETA: 1:21:55.621265 2022-11-01 10:46:32,267 [INFO] __main__: Epoch: 5/100:, Cur-Step: 440, loss(cross_entropy): 0.03086, Running average loss:0.03238, Time taken: 0:00:51.743382 ETA: 1:21:55.621265 Epoch: 5/100:, Cur-Step: 450, loss(cross_entropy): 0.03156, Running average loss:0.03234, Time taken: 0:00:51.743382 ETA: 1:21:55.621265 2022-11-01 10:46:36,950 [INFO] __main__: Epoch: 5/100:, Cur-Step: 450, loss(cross_entropy): 0.03156, Running average loss:0.03234, Time taken: 0:00:51.743382 ETA: 1:21:55.621265 Epoch: 5/100:, Cur-Step: 460, loss(cross_entropy): 0.03274, Running average loss:0.03241, Time taken: 0:00:51.743382 ETA: 1:21:55.621265 2022-11-01 10:46:41,622 [INFO] __main__: Epoch: 5/100:, Cur-Step: 460, loss(cross_entropy): 0.03274, Running average loss:0.03241, Time taken: 0:00:51.743382 ETA: 1:21:55.621265 Epoch: 5/100:, Cur-Step: 470, loss(cross_entropy): 0.03239, Running average loss:0.03230, Time taken: 0:00:51.743382 ETA: 1:21:55.621265 2022-11-01 10:46:46,295 [INFO] __main__: Epoch: 5/100:, Cur-Step: 470, loss(cross_entropy): 0.03239, Running average loss:0.03230, Time taken: 0:00:51.743382 ETA: 1:21:55.621265 Epoch: 5/100:, Cur-Step: 480, loss(cross_entropy): 0.03030, Running average loss:0.03219, Time taken: 0:00:51.743382 ETA: 1:21:55.621265 2022-11-01 10:46:50,947 [INFO] __main__: Epoch: 5/100:, Cur-Step: 480, loss(cross_entropy): 0.03030, Running average loss:0.03219, Time taken: 0:00:51.743382 ETA: 1:21:55.621265 Epoch: 5/100:, Cur-Step: 490, loss(cross_entropy): 0.03149, Running average loss:0.03220, Time taken: 0:00:51.743382 ETA: 1:21:55.621265 2022-11-01 10:46:55,620 [INFO] __main__: Epoch: 5/100:, Cur-Step: 490, loss(cross_entropy): 0.03149, Running average loss:0.03220, Time taken: 0:00:51.743382 ETA: 1:21:55.621265 Epoch: 5/100:, Cur-Step: 500, loss(cross_entropy): 0.03193, Running average loss:0.03211, Time taken: 0:00:51.743382 ETA: 1:21:55.621265 2022-11-01 10:47:00,333 [INFO] __main__: Epoch: 5/100:, Cur-Step: 500, loss(cross_entropy): 0.03193, Running average loss:0.03211, Time taken: 0:00:51.743382 ETA: 1:21:55.621265 INFO:tensorflow:Saving checkpoints for step-504. 2022-11-01 10:47:01,759 [INFO] tensorflow: Saving checkpoints for step-504. Epoch: 6/100:, Cur-Step: 510, loss(cross_entropy): 0.03318, Running average loss:0.03116, Time taken: 0:00:51.089455 ETA: 1:20:02.408737 2022-11-01 10:47:16,618 [INFO] __main__: Epoch: 6/100:, Cur-Step: 510, loss(cross_entropy): 0.03318, Running average loss:0.03116, Time taken: 0:00:51.089455 ETA: 1:20:02.408737 Epoch: 6/100:, Cur-Step: 520, loss(cross_entropy): 0.03064, Running average loss:0.03083, Time taken: 0:00:51.089455 ETA: 1:20:02.408737 2022-11-01 10:47:21,270 [INFO] __main__: Epoch: 6/100:, Cur-Step: 520, loss(cross_entropy): 0.03064, Running average loss:0.03083, Time taken: 0:00:51.089455 ETA: 1:20:02.408737 Epoch: 6/100:, Cur-Step: 530, loss(cross_entropy): 0.03142, Running average loss:0.03095, Time taken: 0:00:51.089455 ETA: 1:20:02.408737 2022-11-01 10:47:25,943 [INFO] __main__: Epoch: 6/100:, Cur-Step: 530, loss(cross_entropy): 0.03142, Running average loss:0.03095, Time taken: 0:00:51.089455 ETA: 1:20:02.408737 Epoch: 6/100:, Cur-Step: 540, loss(cross_entropy): 0.03036, Running average loss:0.03078, Time taken: 0:00:51.089455 ETA: 1:20:02.408737 2022-11-01 10:47:30,613 [INFO] __main__: Epoch: 6/100:, Cur-Step: 540, loss(cross_entropy): 0.03036, Running average loss:0.03078, Time taken: 0:00:51.089455 ETA: 1:20:02.408737 Epoch: 6/100:, Cur-Step: 550, loss(cross_entropy): 0.03149, Running average loss:0.03061, Time taken: 0:00:51.089455 ETA: 1:20:02.408737 2022-11-01 10:47:35,265 [INFO] __main__: Epoch: 6/100:, Cur-Step: 550, loss(cross_entropy): 0.03149, Running average loss:0.03061, Time taken: 0:00:51.089455 ETA: 1:20:02.408737 Epoch: 6/100:, Cur-Step: 560, loss(cross_entropy): 0.02960, Running average loss:0.03047, Time taken: 0:00:51.089455 ETA: 1:20:02.408737 2022-11-01 10:47:39,918 [INFO] __main__: Epoch: 6/100:, Cur-Step: 560, loss(cross_entropy): 0.02960, Running average loss:0.03047, Time taken: 0:00:51.089455 ETA: 1:20:02.408737 Epoch: 6/100:, Cur-Step: 570, loss(cross_entropy): 0.02929, Running average loss:0.03052, Time taken: 0:00:51.089455 ETA: 1:20:02.408737 2022-11-01 10:47:44,580 [INFO] __main__: Epoch: 6/100:, Cur-Step: 570, loss(cross_entropy): 0.02929, Running average loss:0.03052, Time taken: 0:00:51.089455 ETA: 1:20:02.408737 Epoch: 6/100:, Cur-Step: 580, loss(cross_entropy): 0.03030, Running average loss:0.03041, Time taken: 0:00:51.089455 ETA: 1:20:02.408737 2022-11-01 10:47:49,200 [INFO] __main__: Epoch: 6/100:, Cur-Step: 580, loss(cross_entropy): 0.03030, Running average loss:0.03041, Time taken: 0:00:51.089455 ETA: 1:20:02.408737 INFO:tensorflow:Saving checkpoints for step-588. 2022-11-01 10:47:52,534 [INFO] tensorflow: Saving checkpoints for step-588. Epoch: 7/100:, Cur-Step: 590, loss(cross_entropy): 0.02954, Running average loss:0.03123, Time taken: 0:00:51.262853 ETA: 1:19:27.445298 2022-11-01 10:48:05,530 [INFO] __main__: Epoch: 7/100:, Cur-Step: 590, loss(cross_entropy): 0.02954, Running average loss:0.03123, Time taken: 0:00:51.262853 ETA: 1:19:27.445298 Epoch: 7/100:, Cur-Step: 600, loss(cross_entropy): 0.02884, Running average loss:0.03016, Time taken: 0:00:51.262853 ETA: 1:19:27.445298 2022-11-01 10:48:10,210 [INFO] __main__: Epoch: 7/100:, Cur-Step: 600, loss(cross_entropy): 0.02884, Running average loss:0.03016, Time taken: 0:00:51.262853 ETA: 1:19:27.445298 Epoch: 7/100:, Cur-Step: 610, loss(cross_entropy): 0.02868, Running average loss:0.02955, Time taken: 0:00:51.262853 ETA: 1:19:27.445298 2022-11-01 10:48:15,150 [INFO] __main__: Epoch: 7/100:, Cur-Step: 610, loss(cross_entropy): 0.02868, Running average loss:0.02955, Time taken: 0:00:51.262853 ETA: 1:19:27.445298 Epoch: 7/100:, Cur-Step: 620, loss(cross_entropy): 0.02939, Running average loss:0.02955, Time taken: 0:00:51.262853 ETA: 1:19:27.445298 2022-11-01 10:48:19,949 [INFO] __main__: Epoch: 7/100:, Cur-Step: 620, loss(cross_entropy): 0.02939, Running average loss:0.02955, Time taken: 0:00:51.262853 ETA: 1:19:27.445298 Epoch: 7/100:, Cur-Step: 630, loss(cross_entropy): 0.02856, Running average loss:0.02931, Time taken: 0:00:51.262853 ETA: 1:19:27.445298 2022-11-01 10:48:24,763 [INFO] __main__: Epoch: 7/100:, Cur-Step: 630, loss(cross_entropy): 0.02856, Running average loss:0.02931, Time taken: 0:00:51.262853 ETA: 1:19:27.445298 Epoch: 7/100:, Cur-Step: 640, loss(cross_entropy): 0.02803, Running average loss:0.02908, Time taken: 0:00:51.262853 ETA: 1:19:27.445298 2022-11-01 10:48:29,452 [INFO] __main__: Epoch: 7/100:, Cur-Step: 640, loss(cross_entropy): 0.02803, Running average loss:0.02908, Time taken: 0:00:51.262853 ETA: 1:19:27.445298 Epoch: 7/100:, Cur-Step: 650, loss(cross_entropy): 0.02810, Running average loss:0.02892, Time taken: 0:00:51.262853 ETA: 1:19:27.445298 2022-11-01 10:48:34,129 [INFO] __main__: Epoch: 7/100:, Cur-Step: 650, loss(cross_entropy): 0.02810, Running average loss:0.02892, Time taken: 0:00:51.262853 ETA: 1:19:27.445298 Epoch: 7/100:, Cur-Step: 660, loss(cross_entropy): 0.02894, Running average loss:0.02889, Time taken: 0:00:51.262853 ETA: 1:19:27.445298 2022-11-01 10:48:38,827 [INFO] __main__: Epoch: 7/100:, Cur-Step: 660, loss(cross_entropy): 0.02894, Running average loss:0.02889, Time taken: 0:00:51.262853 ETA: 1:19:27.445298 Epoch: 7/100:, Cur-Step: 670, loss(cross_entropy): 0.02848, Running average loss:0.02880, Time taken: 0:00:51.262853 ETA: 1:19:27.445298 2022-11-01 10:48:43,702 [INFO] __main__: Epoch: 7/100:, Cur-Step: 670, loss(cross_entropy): 0.02848, Running average loss:0.02880, Time taken: 0:00:51.262853 ETA: 1:19:27.445298 INFO:tensorflow:Saving checkpoints for step-672. 2022-11-01 10:48:44,163 [INFO] tensorflow: Saving checkpoints for step-672. Epoch: 8/100:, Cur-Step: 680, loss(cross_entropy): 0.02733, Running average loss:0.02761, Time taken: 0:00:52.094344 ETA: 1:19:52.679683 2022-11-01 10:48:59,746 [INFO] __main__: Epoch: 8/100:, Cur-Step: 680, loss(cross_entropy): 0.02733, Running average loss:0.02761, Time taken: 0:00:52.094344 ETA: 1:19:52.679683 Epoch: 8/100:, Cur-Step: 690, loss(cross_entropy): 0.02817, Running average loss:0.02778, Time taken: 0:00:52.094344 ETA: 1:19:52.679683 2022-11-01 10:49:04,660 [INFO] __main__: Epoch: 8/100:, Cur-Step: 690, loss(cross_entropy): 0.02817, Running average loss:0.02778, Time taken: 0:00:52.094344 ETA: 1:19:52.679683 Epoch: 8/100:, Cur-Step: 700, loss(cross_entropy): 0.02907, Running average loss:0.02775, Time taken: 0:00:52.094344 ETA: 1:19:52.679683 2022-11-01 10:49:09,389 [INFO] __main__: Epoch: 8/100:, Cur-Step: 700, loss(cross_entropy): 0.02907, Running average loss:0.02775, Time taken: 0:00:52.094344 ETA: 1:19:52.679683 Epoch: 8/100:, Cur-Step: 710, loss(cross_entropy): 0.02750, Running average loss:0.02774, Time taken: 0:00:52.094344 ETA: 1:19:52.679683 2022-11-01 10:49:14,095 [INFO] __main__: Epoch: 8/100:, Cur-Step: 710, loss(cross_entropy): 0.02750, Running average loss:0.02774, Time taken: 0:00:52.094344 ETA: 1:19:52.679683 Epoch: 8/100:, Cur-Step: 720, loss(cross_entropy): 0.02697, Running average loss:0.02773, Time taken: 0:00:52.094344 ETA: 1:19:52.679683 2022-11-01 10:49:18,983 [INFO] __main__: Epoch: 8/100:, Cur-Step: 720, loss(cross_entropy): 0.02697, Running average loss:0.02773, Time taken: 0:00:52.094344 ETA: 1:19:52.679683 Epoch: 8/100:, Cur-Step: 730, loss(cross_entropy): 0.03004, Running average loss:0.02776, Time taken: 0:00:52.094344 ETA: 1:19:52.679683 2022-11-01 10:49:23,647 [INFO] __main__: Epoch: 8/100:, Cur-Step: 730, loss(cross_entropy): 0.03004, Running average loss:0.02776, Time taken: 0:00:52.094344 ETA: 1:19:52.679683 Epoch: 8/100:, Cur-Step: 740, loss(cross_entropy): 0.02780, Running average loss:0.02776, Time taken: 0:00:52.094344 ETA: 1:19:52.679683 2022-11-01 10:49:28,495 [INFO] __main__: Epoch: 8/100:, Cur-Step: 740, loss(cross_entropy): 0.02780, Running average loss:0.02776, Time taken: 0:00:52.094344 ETA: 1:19:52.679683 Epoch: 8/100:, Cur-Step: 750, loss(cross_entropy): 0.02896, Running average loss:0.02764, Time taken: 0:00:52.094344 ETA: 1:19:52.679683 2022-11-01 10:49:33,305 [INFO] __main__: Epoch: 8/100:, Cur-Step: 750, loss(cross_entropy): 0.02896, Running average loss:0.02764, Time taken: 0:00:52.094344 ETA: 1:19:52.679683 INFO:tensorflow:Saving checkpoints for step-756. 2022-11-01 10:49:35,786 [INFO] tensorflow: Saving checkpoints for step-756. Epoch: 9/100:, Cur-Step: 760, loss(cross_entropy): 0.03007, Running average loss:0.02773, Time taken: 0:00:52.079788 ETA: 1:18:59.260684 2022-11-01 10:49:50,298 [INFO] __main__: Epoch: 9/100:, Cur-Step: 760, loss(cross_entropy): 0.03007, Running average loss:0.02773, Time taken: 0:00:52.079788 ETA: 1:18:59.260684 Epoch: 9/100:, Cur-Step: 770, loss(cross_entropy): 0.02724, Running average loss:0.02760, Time taken: 0:00:52.079788 ETA: 1:18:59.260684 2022-11-01 10:49:54,960 [INFO] __main__: Epoch: 9/100:, Cur-Step: 770, loss(cross_entropy): 0.02724, Running average loss:0.02760, Time taken: 0:00:52.079788 ETA: 1:18:59.260684 Epoch: 9/100:, Cur-Step: 780, loss(cross_entropy): 0.02549, Running average loss:0.02712, Time taken: 0:00:52.079788 ETA: 1:18:59.260684 2022-11-01 10:49:59,680 [INFO] __main__: Epoch: 9/100:, Cur-Step: 780, loss(cross_entropy): 0.02549, Running average loss:0.02712, Time taken: 0:00:52.079788 ETA: 1:18:59.260684 Epoch: 9/100:, Cur-Step: 790, loss(cross_entropy): 0.02667, Running average loss:0.02689, Time taken: 0:00:52.079788 ETA: 1:18:59.260684 2022-11-01 10:50:04,369 [INFO] __main__: Epoch: 9/100:, Cur-Step: 790, loss(cross_entropy): 0.02667, Running average loss:0.02689, Time taken: 0:00:52.079788 ETA: 1:18:59.260684 Epoch: 9/100:, Cur-Step: 800, loss(cross_entropy): 0.02762, Running average loss:0.02682, Time taken: 0:00:52.079788 ETA: 1:18:59.260684 2022-11-01 10:50:09,060 [INFO] __main__: Epoch: 9/100:, Cur-Step: 800, loss(cross_entropy): 0.02762, Running average loss:0.02682, Time taken: 0:00:52.079788 ETA: 1:18:59.260684 Epoch: 9/100:, Cur-Step: 810, loss(cross_entropy): 0.02887, Running average loss:0.02676, Time taken: 0:00:52.079788 ETA: 1:18:59.260684 2022-11-01 10:50:13,753 [INFO] __main__: Epoch: 9/100:, Cur-Step: 810, loss(cross_entropy): 0.02887, Running average loss:0.02676, Time taken: 0:00:52.079788 ETA: 1:18:59.260684 Epoch: 9/100:, Cur-Step: 820, loss(cross_entropy): 0.02492, Running average loss:0.02664, Time taken: 0:00:52.079788 ETA: 1:18:59.260684 2022-11-01 10:50:18,659 [INFO] __main__: Epoch: 9/100:, Cur-Step: 820, loss(cross_entropy): 0.02492, Running average loss:0.02664, Time taken: 0:00:52.079788 ETA: 1:18:59.260684 Epoch: 9/100:, Cur-Step: 830, loss(cross_entropy): 0.02749, Running average loss:0.02658, Time taken: 0:00:52.079788 ETA: 1:18:59.260684 2022-11-01 10:50:23,475 [INFO] __main__: Epoch: 9/100:, Cur-Step: 830, loss(cross_entropy): 0.02749, Running average loss:0.02658, Time taken: 0:00:52.079788 ETA: 1:18:59.260684 INFO:tensorflow:Saving checkpoints for step-840. 2022-11-01 10:50:27,828 [INFO] tensorflow: Saving checkpoints for step-840. Epoch: 10/100:, Cur-Step: 840, loss(cross_entropy): 0.02634, Running average loss:0.02634, Time taken: 0:00:52.562998 ETA: 1:18:50.669825 2022-11-01 10:50:38,457 [INFO] __main__: Epoch: 10/100:, Cur-Step: 840, loss(cross_entropy): 0.02634, Running average loss:0.02634, Time taken: 0:00:52.562998 ETA: 1:18:50.669825 Epoch: 10/100:, Cur-Step: 850, loss(cross_entropy): 0.02470, Running average loss:0.02537, Time taken: 0:00:52.562998 ETA: 1:18:50.669825 2022-11-01 10:50:43,148 [INFO] __main__: Epoch: 10/100:, Cur-Step: 850, loss(cross_entropy): 0.02470, Running average loss:0.02537, Time taken: 0:00:52.562998 ETA: 1:18:50.669825 Epoch: 10/100:, Cur-Step: 860, loss(cross_entropy): 0.02519, Running average loss:0.02566, Time taken: 0:00:52.562998 ETA: 1:18:50.669825 2022-11-01 10:50:47,997 [INFO] __main__: Epoch: 10/100:, Cur-Step: 860, loss(cross_entropy): 0.02519, Running average loss:0.02566, Time taken: 0:00:52.562998 ETA: 1:18:50.669825 Epoch: 10/100:, Cur-Step: 870, loss(cross_entropy): 0.02644, Running average loss:0.02574, Time taken: 0:00:52.562998 ETA: 1:18:50.669825 2022-11-01 10:50:52,738 [INFO] __main__: Epoch: 10/100:, Cur-Step: 870, loss(cross_entropy): 0.02644, Running average loss:0.02574, Time taken: 0:00:52.562998 ETA: 1:18:50.669825 Epoch: 10/100:, Cur-Step: 880, loss(cross_entropy): 0.02459, Running average loss:0.02563, Time taken: 0:00:52.562998 ETA: 1:18:50.669825 2022-11-01 10:50:57,650 [INFO] __main__: Epoch: 10/100:, Cur-Step: 880, loss(cross_entropy): 0.02459, Running average loss:0.02563, Time taken: 0:00:52.562998 ETA: 1:18:50.669825 Epoch: 10/100:, Cur-Step: 890, loss(cross_entropy): 0.02481, Running average loss:0.02566, Time taken: 0:00:52.562998 ETA: 1:18:50.669825 2022-11-01 10:51:02,468 [INFO] __main__: Epoch: 10/100:, Cur-Step: 890, loss(cross_entropy): 0.02481, Running average loss:0.02566, Time taken: 0:00:52.562998 ETA: 1:18:50.669825 Epoch: 10/100:, Cur-Step: 900, loss(cross_entropy): 0.02536, Running average loss:0.02551, Time taken: 0:00:52.562998 ETA: 1:18:50.669825 2022-11-01 10:51:07,165 [INFO] __main__: Epoch: 10/100:, Cur-Step: 900, loss(cross_entropy): 0.02536, Running average loss:0.02551, Time taken: 0:00:52.562998 ETA: 1:18:50.669825 Epoch: 10/100:, Cur-Step: 910, loss(cross_entropy): 0.02472, Running average loss:0.02552, Time taken: 0:00:52.562998 ETA: 1:18:50.669825 2022-11-01 10:51:11,856 [INFO] __main__: Epoch: 10/100:, Cur-Step: 910, loss(cross_entropy): 0.02472, Running average loss:0.02552, Time taken: 0:00:52.562998 ETA: 1:18:50.669825 Epoch: 10/100:, Cur-Step: 920, loss(cross_entropy): 0.02449, Running average loss:0.02543, Time taken: 0:00:52.562998 ETA: 1:18:50.669825 2022-11-01 10:51:16,566 [INFO] __main__: Epoch: 10/100:, Cur-Step: 920, loss(cross_entropy): 0.02449, Running average loss:0.02543, Time taken: 0:00:52.562998 ETA: 1:18:50.669825 INFO:tensorflow:Saving checkpoints for step-924. 2022-11-01 10:51:18,003 [INFO] tensorflow: Saving checkpoints for step-924. Epoch: 11/100:, Cur-Step: 930, loss(cross_entropy): 0.02622, Running average loss:0.02477, Time taken: 0:00:50.654305 ETA: 1:15:08.233165 2022-11-01 10:51:32,715 [INFO] __main__: Epoch: 11/100:, Cur-Step: 930, loss(cross_entropy): 0.02622, Running average loss:0.02477, Time taken: 0:00:50.654305 ETA: 1:15:08.233165 Epoch: 11/100:, Cur-Step: 940, loss(cross_entropy): 0.02505, Running average loss:0.02491, Time taken: 0:00:50.654305 ETA: 1:15:08.233165 2022-11-01 10:51:37,541 [INFO] __main__: Epoch: 11/100:, Cur-Step: 940, loss(cross_entropy): 0.02505, Running average loss:0.02491, Time taken: 0:00:50.654305 ETA: 1:15:08.233165 Epoch: 11/100:, Cur-Step: 950, loss(cross_entropy): 0.02496, Running average loss:0.02494, Time taken: 0:00:50.654305 ETA: 1:15:08.233165 2022-11-01 10:51:42,235 [INFO] __main__: Epoch: 11/100:, Cur-Step: 950, loss(cross_entropy): 0.02496, Running average loss:0.02494, Time taken: 0:00:50.654305 ETA: 1:15:08.233165 Epoch: 11/100:, Cur-Step: 960, loss(cross_entropy): 0.02477, Running average loss:0.02503, Time taken: 0:00:50.654305 ETA: 1:15:08.233165 2022-11-01 10:51:46,948 [INFO] __main__: Epoch: 11/100:, Cur-Step: 960, loss(cross_entropy): 0.02477, Running average loss:0.02503, Time taken: 0:00:50.654305 ETA: 1:15:08.233165 Epoch: 11/100:, Cur-Step: 970, loss(cross_entropy): 0.02358, Running average loss:0.02487, Time taken: 0:00:50.654305 ETA: 1:15:08.233165 2022-11-01 10:51:51,714 [INFO] __main__: Epoch: 11/100:, Cur-Step: 970, loss(cross_entropy): 0.02358, Running average loss:0.02487, Time taken: 0:00:50.654305 ETA: 1:15:08.233165 Epoch: 11/100:, Cur-Step: 980, loss(cross_entropy): 0.02575, Running average loss:0.02484, Time taken: 0:00:50.654305 ETA: 1:15:08.233165 2022-11-01 10:51:56,462 [INFO] __main__: Epoch: 11/100:, Cur-Step: 980, loss(cross_entropy): 0.02575, Running average loss:0.02484, Time taken: 0:00:50.654305 ETA: 1:15:08.233165 Epoch: 11/100:, Cur-Step: 990, loss(cross_entropy): 0.02324, Running average loss:0.02478, Time taken: 0:00:50.654305 ETA: 1:15:08.233165 2022-11-01 10:52:01,172 [INFO] __main__: Epoch: 11/100:, Cur-Step: 990, loss(cross_entropy): 0.02324, Running average loss:0.02478, Time taken: 0:00:50.654305 ETA: 1:15:08.233165 Epoch: 11/100:, Cur-Step: 1000, loss(cross_entropy): 0.02345, Running average loss:0.02468, Time taken: 0:00:50.654305 ETA: 1:15:08.233165 2022-11-01 10:52:05,884 [INFO] __main__: Epoch: 11/100:, Cur-Step: 1000, loss(cross_entropy): 0.02345, Running average loss:0.02468, Time taken: 0:00:50.654305 ETA: 1:15:08.233165 INFO:tensorflow:Saving checkpoints for step-1008. 2022-11-01 10:52:09,319 [INFO] tensorflow: Saving checkpoints for step-1008. Epoch: 12/100:, Cur-Step: 1010, loss(cross_entropy): 0.02442, Running average loss:0.02513, Time taken: 0:00:51.795415 ETA: 1:15:57.996534 2022-11-01 10:52:22,196 [INFO] __main__: Epoch: 12/100:, Cur-Step: 1010, loss(cross_entropy): 0.02442, Running average loss:0.02513, Time taken: 0:00:51.795415 ETA: 1:15:57.996534 Epoch: 12/100:, Cur-Step: 1020, loss(cross_entropy): 0.02290, Running average loss:0.02453, Time taken: 0:00:51.795415 ETA: 1:15:57.996534 2022-11-01 10:52:27,258 [INFO] __main__: Epoch: 12/100:, Cur-Step: 1020, loss(cross_entropy): 0.02290, Running average loss:0.02453, Time taken: 0:00:51.795415 ETA: 1:15:57.996534 Epoch: 12/100:, Cur-Step: 1030, loss(cross_entropy): 0.02456, Running average loss:0.02417, Time taken: 0:00:51.795415 ETA: 1:15:57.996534 2022-11-01 10:52:31,949 [INFO] __main__: Epoch: 12/100:, Cur-Step: 1030, loss(cross_entropy): 0.02456, Running average loss:0.02417, Time taken: 0:00:51.795415 ETA: 1:15:57.996534 Epoch: 12/100:, Cur-Step: 1040, loss(cross_entropy): 0.02395, Running average loss:0.02411, Time taken: 0:00:51.795415 ETA: 1:15:57.996534 2022-11-01 10:52:36,643 [INFO] __main__: Epoch: 12/100:, Cur-Step: 1040, loss(cross_entropy): 0.02395, Running average loss:0.02411, Time taken: 0:00:51.795415 ETA: 1:15:57.996534 Epoch: 12/100:, Cur-Step: 1050, loss(cross_entropy): 0.02351, Running average loss:0.02405, Time taken: 0:00:51.795415 ETA: 1:15:57.996534 2022-11-01 10:52:41,357 [INFO] __main__: Epoch: 12/100:, Cur-Step: 1050, loss(cross_entropy): 0.02351, Running average loss:0.02405, Time taken: 0:00:51.795415 ETA: 1:15:57.996534 Epoch: 12/100:, Cur-Step: 1060, loss(cross_entropy): 0.02296, Running average loss:0.02392, Time taken: 0:00:51.795415 ETA: 1:15:57.996534 2022-11-01 10:52:46,062 [INFO] __main__: Epoch: 12/100:, Cur-Step: 1060, loss(cross_entropy): 0.02296, Running average loss:0.02392, Time taken: 0:00:51.795415 ETA: 1:15:57.996534 Epoch: 12/100:, Cur-Step: 1070, loss(cross_entropy): 0.02487, Running average loss:0.02389, Time taken: 0:00:51.795415 ETA: 1:15:57.996534 2022-11-01 10:52:50,863 [INFO] __main__: Epoch: 12/100:, Cur-Step: 1070, loss(cross_entropy): 0.02487, Running average loss:0.02389, Time taken: 0:00:51.795415 ETA: 1:15:57.996534 Epoch: 12/100:, Cur-Step: 1080, loss(cross_entropy): 0.02545, Running average loss:0.02386, Time taken: 0:00:51.795415 ETA: 1:15:57.996534 2022-11-01 10:52:55,528 [INFO] __main__: Epoch: 12/100:, Cur-Step: 1080, loss(cross_entropy): 0.02545, Running average loss:0.02386, Time taken: 0:00:51.795415 ETA: 1:15:57.996534 Epoch: 12/100:, Cur-Step: 1090, loss(cross_entropy): 0.02250, Running average loss:0.02380, Time taken: 0:00:51.795415 ETA: 1:15:57.996534 2022-11-01 10:53:00,348 [INFO] __main__: Epoch: 12/100:, Cur-Step: 1090, loss(cross_entropy): 0.02250, Running average loss:0.02380, Time taken: 0:00:51.795415 ETA: 1:15:57.996534 INFO:tensorflow:Saving checkpoints for step-1092. 2022-11-01 10:53:00,834 [INFO] tensorflow: Saving checkpoints for step-1092. Epoch: 13/100:, Cur-Step: 1100, loss(cross_entropy): 0.02421, Running average loss:0.02345, Time taken: 0:00:52.059219 ETA: 1:15:29.152084 2022-11-01 10:53:16,804 [INFO] __main__: Epoch: 13/100:, Cur-Step: 1100, loss(cross_entropy): 0.02421, Running average loss:0.02345, Time taken: 0:00:52.059219 ETA: 1:15:29.152084 Epoch: 13/100:, Cur-Step: 1110, loss(cross_entropy): 0.02262, Running average loss:0.02320, Time taken: 0:00:52.059219 ETA: 1:15:29.152084 2022-11-01 10:53:21,566 [INFO] __main__: Epoch: 13/100:, Cur-Step: 1110, loss(cross_entropy): 0.02262, Running average loss:0.02320, Time taken: 0:00:52.059219 ETA: 1:15:29.152084 Epoch: 13/100:, Cur-Step: 1120, loss(cross_entropy): 0.02189, Running average loss:0.02304, Time taken: 0:00:52.059219 ETA: 1:15:29.152084 2022-11-01 10:53:26,277 [INFO] __main__: Epoch: 13/100:, Cur-Step: 1120, loss(cross_entropy): 0.02189, Running average loss:0.02304, Time taken: 0:00:52.059219 ETA: 1:15:29.152084 Epoch: 13/100:, Cur-Step: 1130, loss(cross_entropy): 0.02206, Running average loss:0.02297, Time taken: 0:00:52.059219 ETA: 1:15:29.152084 2022-11-01 10:53:30,960 [INFO] __main__: Epoch: 13/100:, Cur-Step: 1130, loss(cross_entropy): 0.02206, Running average loss:0.02297, Time taken: 0:00:52.059219 ETA: 1:15:29.152084 Epoch: 13/100:, Cur-Step: 1140, loss(cross_entropy): 0.02228, Running average loss:0.02301, Time taken: 0:00:52.059219 ETA: 1:15:29.152084 2022-11-01 10:53:35,652 [INFO] __main__: Epoch: 13/100:, Cur-Step: 1140, loss(cross_entropy): 0.02228, Running average loss:0.02301, Time taken: 0:00:52.059219 ETA: 1:15:29.152084 Epoch: 13/100:, Cur-Step: 1150, loss(cross_entropy): 0.02320, Running average loss:0.02302, Time taken: 0:00:52.059219 ETA: 1:15:29.152084 2022-11-01 10:53:40,360 [INFO] __main__: Epoch: 13/100:, Cur-Step: 1150, loss(cross_entropy): 0.02320, Running average loss:0.02302, Time taken: 0:00:52.059219 ETA: 1:15:29.152084 Epoch: 13/100:, Cur-Step: 1160, loss(cross_entropy): 0.02505, Running average loss:0.02304, Time taken: 0:00:52.059219 ETA: 1:15:29.152084 2022-11-01 10:53:45,044 [INFO] __main__: Epoch: 13/100:, Cur-Step: 1160, loss(cross_entropy): 0.02505, Running average loss:0.02304, Time taken: 0:00:52.059219 ETA: 1:15:29.152084 Epoch: 13/100:, Cur-Step: 1170, loss(cross_entropy): 0.02201, Running average loss:0.02301, Time taken: 0:00:52.059219 ETA: 1:15:29.152084 2022-11-01 10:53:49,949 [INFO] __main__: Epoch: 13/100:, Cur-Step: 1170, loss(cross_entropy): 0.02201, Running average loss:0.02301, Time taken: 0:00:52.059219 ETA: 1:15:29.152084 INFO:tensorflow:Saving checkpoints for step-1176. 2022-11-01 10:53:52,380 [INFO] tensorflow: Saving checkpoints for step-1176. Epoch: 14/100:, Cur-Step: 1180, loss(cross_entropy): 0.02324, Running average loss:0.02283, Time taken: 0:00:52.031051 ETA: 1:14:34.670359 2022-11-01 10:54:05,987 [INFO] __main__: Epoch: 14/100:, Cur-Step: 1180, loss(cross_entropy): 0.02324, Running average loss:0.02283, Time taken: 0:00:52.031051 ETA: 1:14:34.670359 Epoch: 14/100:, Cur-Step: 1190, loss(cross_entropy): 0.02115, Running average loss:0.02264, Time taken: 0:00:52.031051 ETA: 1:14:34.670359 2022-11-01 10:54:10,773 [INFO] __main__: Epoch: 14/100:, Cur-Step: 1190, loss(cross_entropy): 0.02115, Running average loss:0.02264, Time taken: 0:00:52.031051 ETA: 1:14:34.670359 Epoch: 14/100:, Cur-Step: 1200, loss(cross_entropy): 0.02475, Running average loss:0.02273, Time taken: 0:00:52.031051 ETA: 1:14:34.670359 2022-11-01 10:54:15,478 [INFO] __main__: Epoch: 14/100:, Cur-Step: 1200, loss(cross_entropy): 0.02475, Running average loss:0.02273, Time taken: 0:00:52.031051 ETA: 1:14:34.670359 Epoch: 14/100:, Cur-Step: 1210, loss(cross_entropy): 0.02132, Running average loss:0.02247, Time taken: 0:00:52.031051 ETA: 1:14:34.670359 2022-11-01 10:54:20,169 [INFO] __main__: Epoch: 14/100:, Cur-Step: 1210, loss(cross_entropy): 0.02132, Running average loss:0.02247, Time taken: 0:00:52.031051 ETA: 1:14:34.670359 Epoch: 14/100:, Cur-Step: 1220, loss(cross_entropy): 0.02158, Running average loss:0.02241, Time taken: 0:00:52.031051 ETA: 1:14:34.670359 2022-11-01 10:54:24,895 [INFO] __main__: Epoch: 14/100:, Cur-Step: 1220, loss(cross_entropy): 0.02158, Running average loss:0.02241, Time taken: 0:00:52.031051 ETA: 1:14:34.670359 Epoch: 14/100:, Cur-Step: 1230, loss(cross_entropy): 0.02101, Running average loss:0.02237, Time taken: 0:00:52.031051 ETA: 1:14:34.670359 2022-11-01 10:54:29,578 [INFO] __main__: Epoch: 14/100:, Cur-Step: 1230, loss(cross_entropy): 0.02101, Running average loss:0.02237, Time taken: 0:00:52.031051 ETA: 1:14:34.670359 Epoch: 14/100:, Cur-Step: 1240, loss(cross_entropy): 0.02222, Running average loss:0.02231, Time taken: 0:00:52.031051 ETA: 1:14:34.670359 2022-11-01 10:54:34,287 [INFO] __main__: Epoch: 14/100:, Cur-Step: 1240, loss(cross_entropy): 0.02222, Running average loss:0.02231, Time taken: 0:00:52.031051 ETA: 1:14:34.670359 Epoch: 14/100:, Cur-Step: 1250, loss(cross_entropy): 0.02136, Running average loss:0.02222, Time taken: 0:00:52.031051 ETA: 1:14:34.670359 2022-11-01 10:54:39,127 [INFO] __main__: Epoch: 14/100:, Cur-Step: 1250, loss(cross_entropy): 0.02136, Running average loss:0.02222, Time taken: 0:00:52.031051 ETA: 1:14:34.670359 INFO:tensorflow:Saving checkpoints for step-1260. 2022-11-01 10:54:43,493 [INFO] tensorflow: Saving checkpoints for step-1260. Epoch: 15/100:, Cur-Step: 1260, loss(cross_entropy): 0.02287, Running average loss:0.02287, Time taken: 0:00:51.589845 ETA: 1:13:05.136861 2022-11-01 10:54:55,459 [INFO] __main__: Epoch: 15/100:, Cur-Step: 1260, loss(cross_entropy): 0.02287, Running average loss:0.02287, Time taken: 0:00:51.589845 ETA: 1:13:05.136861 Epoch: 15/100:, Cur-Step: 1270, loss(cross_entropy): 0.02270, Running average loss:0.02204, Time taken: 0:00:51.589845 ETA: 1:13:05.136861 2022-11-01 10:55:00,385 [INFO] __main__: Epoch: 15/100:, Cur-Step: 1270, loss(cross_entropy): 0.02270, Running average loss:0.02204, Time taken: 0:00:51.589845 ETA: 1:13:05.136861 Epoch: 15/100:, Cur-Step: 1280, loss(cross_entropy): 0.02116, Running average loss:0.02201, Time taken: 0:00:51.589845 ETA: 1:13:05.136861 2022-11-01 10:55:05,046 [INFO] __main__: Epoch: 15/100:, Cur-Step: 1280, loss(cross_entropy): 0.02116, Running average loss:0.02201, Time taken: 0:00:51.589845 ETA: 1:13:05.136861 Epoch: 15/100:, Cur-Step: 1290, loss(cross_entropy): 0.02164, Running average loss:0.02201, Time taken: 0:00:51.589845 ETA: 1:13:05.136861 2022-11-01 10:55:09,703 [INFO] __main__: Epoch: 15/100:, Cur-Step: 1290, loss(cross_entropy): 0.02164, Running average loss:0.02201, Time taken: 0:00:51.589845 ETA: 1:13:05.136861 Epoch: 15/100:, Cur-Step: 1300, loss(cross_entropy): 0.02222, Running average loss:0.02189, Time taken: 0:00:51.589845 ETA: 1:13:05.136861 2022-11-01 10:55:14,366 [INFO] __main__: Epoch: 15/100:, Cur-Step: 1300, loss(cross_entropy): 0.02222, Running average loss:0.02189, Time taken: 0:00:51.589845 ETA: 1:13:05.136861 Epoch: 15/100:, Cur-Step: 1310, loss(cross_entropy): 0.02124, Running average loss:0.02183, Time taken: 0:00:51.589845 ETA: 1:13:05.136861 2022-11-01 10:55:19,085 [INFO] __main__: Epoch: 15/100:, Cur-Step: 1310, loss(cross_entropy): 0.02124, Running average loss:0.02183, Time taken: 0:00:51.589845 ETA: 1:13:05.136861 Epoch: 15/100:, Cur-Step: 1320, loss(cross_entropy): 0.01999, Running average loss:0.02173, Time taken: 0:00:51.589845 ETA: 1:13:05.136861 2022-11-01 10:55:23,736 [INFO] __main__: Epoch: 15/100:, Cur-Step: 1320, loss(cross_entropy): 0.01999, Running average loss:0.02173, Time taken: 0:00:51.589845 ETA: 1:13:05.136861 Epoch: 15/100:, Cur-Step: 1330, loss(cross_entropy): 0.02085, Running average loss:0.02173, Time taken: 0:00:51.589845 ETA: 1:13:05.136861 2022-11-01 10:55:28,354 [INFO] __main__: Epoch: 15/100:, Cur-Step: 1330, loss(cross_entropy): 0.02085, Running average loss:0.02173, Time taken: 0:00:51.589845 ETA: 1:13:05.136861 Epoch: 15/100:, Cur-Step: 1340, loss(cross_entropy): 0.02076, Running average loss:0.02165, Time taken: 0:00:51.589845 ETA: 1:13:05.136861 2022-11-01 10:55:33,320 [INFO] __main__: Epoch: 15/100:, Cur-Step: 1340, loss(cross_entropy): 0.02076, Running average loss:0.02165, Time taken: 0:00:51.589845 ETA: 1:13:05.136861 INFO:tensorflow:Saving checkpoints for step-1344. 2022-11-01 10:55:34,728 [INFO] tensorflow: Saving checkpoints for step-1344. Epoch: 16/100:, Cur-Step: 1350, loss(cross_entropy): 0.02080, Running average loss:0.02161, Time taken: 0:00:51.708752 ETA: 1:12:23.535201 2022-11-01 10:55:50,143 [INFO] __main__: Epoch: 16/100:, Cur-Step: 1350, loss(cross_entropy): 0.02080, Running average loss:0.02161, Time taken: 0:00:51.708752 ETA: 1:12:23.535201 Epoch: 16/100:, Cur-Step: 1360, loss(cross_entropy): 0.02044, Running average loss:0.02141, Time taken: 0:00:51.708752 ETA: 1:12:23.535201 2022-11-01 10:55:55,025 [INFO] __main__: Epoch: 16/100:, Cur-Step: 1360, loss(cross_entropy): 0.02044, Running average loss:0.02141, Time taken: 0:00:51.708752 ETA: 1:12:23.535201 Epoch: 16/100:, Cur-Step: 1370, loss(cross_entropy): 0.02042, Running average loss:0.02127, Time taken: 0:00:51.708752 ETA: 1:12:23.535201 2022-11-01 10:55:59,723 [INFO] __main__: Epoch: 16/100:, Cur-Step: 1370, loss(cross_entropy): 0.02042, Running average loss:0.02127, Time taken: 0:00:51.708752 ETA: 1:12:23.535201 Epoch: 16/100:, Cur-Step: 1380, loss(cross_entropy): 0.02036, Running average loss:0.02114, Time taken: 0:00:51.708752 ETA: 1:12:23.535201 2022-11-01 10:56:04,409 [INFO] __main__: Epoch: 16/100:, Cur-Step: 1380, loss(cross_entropy): 0.02036, Running average loss:0.02114, Time taken: 0:00:51.708752 ETA: 1:12:23.535201 Epoch: 16/100:, Cur-Step: 1390, loss(cross_entropy): 0.02144, Running average loss:0.02106, Time taken: 0:00:51.708752 ETA: 1:12:23.535201 2022-11-01 10:56:09,087 [INFO] __main__: Epoch: 16/100:, Cur-Step: 1390, loss(cross_entropy): 0.02144, Running average loss:0.02106, Time taken: 0:00:51.708752 ETA: 1:12:23.535201 Epoch: 16/100:, Cur-Step: 1400, loss(cross_entropy): 0.02028, Running average loss:0.02104, Time taken: 0:00:51.708752 ETA: 1:12:23.535201 2022-11-01 10:56:13,761 [INFO] __main__: Epoch: 16/100:, Cur-Step: 1400, loss(cross_entropy): 0.02028, Running average loss:0.02104, Time taken: 0:00:51.708752 ETA: 1:12:23.535201 Epoch: 16/100:, Cur-Step: 1410, loss(cross_entropy): 0.02072, Running average loss:0.02100, Time taken: 0:00:51.708752 ETA: 1:12:23.535201 2022-11-01 10:56:18,536 [INFO] __main__: Epoch: 16/100:, Cur-Step: 1410, loss(cross_entropy): 0.02072, Running average loss:0.02100, Time taken: 0:00:51.708752 ETA: 1:12:23.535201 Epoch: 16/100:, Cur-Step: 1420, loss(cross_entropy): 0.02115, Running average loss:0.02095, Time taken: 0:00:51.708752 ETA: 1:12:23.535201 2022-11-01 10:56:23,389 [INFO] __main__: Epoch: 16/100:, Cur-Step: 1420, loss(cross_entropy): 0.02115, Running average loss:0.02095, Time taken: 0:00:51.708752 ETA: 1:12:23.535201 INFO:tensorflow:Saving checkpoints for step-1428. 2022-11-01 10:56:26,675 [INFO] tensorflow: Saving checkpoints for step-1428. Epoch: 17/100:, Cur-Step: 1430, loss(cross_entropy): 0.01964, Running average loss:0.01999, Time taken: 0:00:52.423579 ETA: 1:12:31.157055 2022-11-01 10:56:39,928 [INFO] __main__: Epoch: 17/100:, Cur-Step: 1430, loss(cross_entropy): 0.01964, Running average loss:0.01999, Time taken: 0:00:52.423579 ETA: 1:12:31.157055 Epoch: 17/100:, Cur-Step: 1440, loss(cross_entropy): 0.02538, Running average loss:0.02093, Time taken: 0:00:52.423579 ETA: 1:12:31.157055 2022-11-01 10:56:44,610 [INFO] __main__: Epoch: 17/100:, Cur-Step: 1440, loss(cross_entropy): 0.02538, Running average loss:0.02093, Time taken: 0:00:52.423579 ETA: 1:12:31.157055 Epoch: 17/100:, Cur-Step: 1450, loss(cross_entropy): 0.02132, Running average loss:0.02094, Time taken: 0:00:52.423579 ETA: 1:12:31.157055 2022-11-01 10:56:49,257 [INFO] __main__: Epoch: 17/100:, Cur-Step: 1450, loss(cross_entropy): 0.02132, Running average loss:0.02094, Time taken: 0:00:52.423579 ETA: 1:12:31.157055 Epoch: 17/100:, Cur-Step: 1460, loss(cross_entropy): 0.01961, Running average loss:0.02087, Time taken: 0:00:52.423579 ETA: 1:12:31.157055 2022-11-01 10:56:53,939 [INFO] __main__: Epoch: 17/100:, Cur-Step: 1460, loss(cross_entropy): 0.01961, Running average loss:0.02087, Time taken: 0:00:52.423579 ETA: 1:12:31.157055 Epoch: 17/100:, Cur-Step: 1470, loss(cross_entropy): 0.01976, Running average loss:0.02070, Time taken: 0:00:52.423579 ETA: 1:12:31.157055 2022-11-01 10:56:58,759 [INFO] __main__: Epoch: 17/100:, Cur-Step: 1470, loss(cross_entropy): 0.01976, Running average loss:0.02070, Time taken: 0:00:52.423579 ETA: 1:12:31.157055 Epoch: 17/100:, Cur-Step: 1480, loss(cross_entropy): 0.02083, Running average loss:0.02055, Time taken: 0:00:52.423579 ETA: 1:12:31.157055 2022-11-01 10:57:03,441 [INFO] __main__: Epoch: 17/100:, Cur-Step: 1480, loss(cross_entropy): 0.02083, Running average loss:0.02055, Time taken: 0:00:52.423579 ETA: 1:12:31.157055 Epoch: 17/100:, Cur-Step: 1490, loss(cross_entropy): 0.01979, Running average loss:0.02045, Time taken: 0:00:52.423579 ETA: 1:12:31.157055 2022-11-01 10:57:08,105 [INFO] __main__: Epoch: 17/100:, Cur-Step: 1490, loss(cross_entropy): 0.01979, Running average loss:0.02045, Time taken: 0:00:52.423579 ETA: 1:12:31.157055 Epoch: 17/100:, Cur-Step: 1500, loss(cross_entropy): 0.02050, Running average loss:0.02034, Time taken: 0:00:52.423579 ETA: 1:12:31.157055 2022-11-01 10:57:12,868 [INFO] __main__: Epoch: 17/100:, Cur-Step: 1500, loss(cross_entropy): 0.02050, Running average loss:0.02034, Time taken: 0:00:52.423579 ETA: 1:12:31.157055 Epoch: 17/100:, Cur-Step: 1510, loss(cross_entropy): 0.02085, Running average loss:0.02026, Time taken: 0:00:52.423579 ETA: 1:12:31.157055 2022-11-01 10:57:17,583 [INFO] __main__: Epoch: 17/100:, Cur-Step: 1510, loss(cross_entropy): 0.02085, Running average loss:0.02026, Time taken: 0:00:52.423579 ETA: 1:12:31.157055 INFO:tensorflow:Saving checkpoints for step-1512. 2022-11-01 10:57:18,052 [INFO] tensorflow: Saving checkpoints for step-1512. Epoch: 18/100:, Cur-Step: 1520, loss(cross_entropy): 0.02038, Running average loss:0.01964, Time taken: 0:00:51.842551 ETA: 1:10:51.089142 2022-11-01 10:57:34,115 [INFO] __main__: Epoch: 18/100:, Cur-Step: 1520, loss(cross_entropy): 0.02038, Running average loss:0.01964, Time taken: 0:00:51.842551 ETA: 1:10:51.089142 Epoch: 18/100:, Cur-Step: 1530, loss(cross_entropy): 0.01926, Running average loss:0.01953, Time taken: 0:00:51.842551 ETA: 1:10:51.089142 2022-11-01 10:57:38,910 [INFO] __main__: Epoch: 18/100:, Cur-Step: 1530, loss(cross_entropy): 0.01926, Running average loss:0.01953, Time taken: 0:00:51.842551 ETA: 1:10:51.089142 Epoch: 18/100:, Cur-Step: 1540, loss(cross_entropy): 0.01943, Running average loss:0.01960, Time taken: 0:00:51.842551 ETA: 1:10:51.089142 2022-11-01 10:57:43,599 [INFO] __main__: Epoch: 18/100:, Cur-Step: 1540, loss(cross_entropy): 0.01943, Running average loss:0.01960, Time taken: 0:00:51.842551 ETA: 1:10:51.089142 Epoch: 18/100:, Cur-Step: 1550, loss(cross_entropy): 0.02227, Running average loss:0.01990, Time taken: 0:00:51.842551 ETA: 1:10:51.089142 2022-11-01 10:57:48,292 [INFO] __main__: Epoch: 18/100:, Cur-Step: 1550, loss(cross_entropy): 0.02227, Running average loss:0.01990, Time taken: 0:00:51.842551 ETA: 1:10:51.089142 Epoch: 18/100:, Cur-Step: 1560, loss(cross_entropy): 0.02027, Running average loss:0.01989, Time taken: 0:00:51.842551 ETA: 1:10:51.089142 2022-11-01 10:57:52,975 [INFO] __main__: Epoch: 18/100:, Cur-Step: 1560, loss(cross_entropy): 0.02027, Running average loss:0.01989, Time taken: 0:00:51.842551 ETA: 1:10:51.089142 Epoch: 18/100:, Cur-Step: 1570, loss(cross_entropy): 0.02067, Running average loss:0.01989, Time taken: 0:00:51.842551 ETA: 1:10:51.089142 2022-11-01 10:57:57,753 [INFO] __main__: Epoch: 18/100:, Cur-Step: 1570, loss(cross_entropy): 0.02067, Running average loss:0.01989, Time taken: 0:00:51.842551 ETA: 1:10:51.089142 Epoch: 18/100:, Cur-Step: 1580, loss(cross_entropy): 0.01898, Running average loss:0.01988, Time taken: 0:00:51.842551 ETA: 1:10:51.089142 2022-11-01 10:58:02,425 [INFO] __main__: Epoch: 18/100:, Cur-Step: 1580, loss(cross_entropy): 0.01898, Running average loss:0.01988, Time taken: 0:00:51.842551 ETA: 1:10:51.089142 Epoch: 18/100:, Cur-Step: 1590, loss(cross_entropy): 0.01892, Running average loss:0.01984, Time taken: 0:00:51.842551 ETA: 1:10:51.089142 2022-11-01 10:58:07,456 [INFO] __main__: Epoch: 18/100:, Cur-Step: 1590, loss(cross_entropy): 0.01892, Running average loss:0.01984, Time taken: 0:00:51.842551 ETA: 1:10:51.089142 INFO:tensorflow:Saving checkpoints for step-1596. 2022-11-01 10:58:09,922 [INFO] tensorflow: Saving checkpoints for step-1596. Epoch: 19/100:, Cur-Step: 1600, loss(cross_entropy): 0.01969, Running average loss:0.01914, Time taken: 0:00:52.336950 ETA: 1:10:39.292936 2022-11-01 10:58:23,512 [INFO] __main__: Epoch: 19/100:, Cur-Step: 1600, loss(cross_entropy): 0.01969, Running average loss:0.01914, Time taken: 0:00:52.336950 ETA: 1:10:39.292936 Epoch: 19/100:, Cur-Step: 1610, loss(cross_entropy): 0.02038, Running average loss:0.01942, Time taken: 0:00:52.336950 ETA: 1:10:39.292936 2022-11-01 10:58:28,229 [INFO] __main__: Epoch: 19/100:, Cur-Step: 1610, loss(cross_entropy): 0.02038, Running average loss:0.01942, Time taken: 0:00:52.336950 ETA: 1:10:39.292936 Epoch: 19/100:, Cur-Step: 1620, loss(cross_entropy): 0.01908, Running average loss:0.01939, Time taken: 0:00:52.336950 ETA: 1:10:39.292936 2022-11-01 10:58:32,956 [INFO] __main__: Epoch: 19/100:, Cur-Step: 1620, loss(cross_entropy): 0.01908, Running average loss:0.01939, Time taken: 0:00:52.336950 ETA: 1:10:39.292936 Epoch: 19/100:, Cur-Step: 1630, loss(cross_entropy): 0.01885, Running average loss:0.01927, Time taken: 0:00:52.336950 ETA: 1:10:39.292936 2022-11-01 10:58:37,658 [INFO] __main__: Epoch: 19/100:, Cur-Step: 1630, loss(cross_entropy): 0.01885, Running average loss:0.01927, Time taken: 0:00:52.336950 ETA: 1:10:39.292936 Epoch: 19/100:, Cur-Step: 1640, loss(cross_entropy): 0.01832, Running average loss:0.01924, Time taken: 0:00:52.336950 ETA: 1:10:39.292936 2022-11-01 10:58:42,326 [INFO] __main__: Epoch: 19/100:, Cur-Step: 1640, loss(cross_entropy): 0.01832, Running average loss:0.01924, Time taken: 0:00:52.336950 ETA: 1:10:39.292936 Epoch: 19/100:, Cur-Step: 1650, loss(cross_entropy): 0.01936, Running average loss:0.01919, Time taken: 0:00:52.336950 ETA: 1:10:39.292936 2022-11-01 10:58:47,017 [INFO] __main__: Epoch: 19/100:, Cur-Step: 1650, loss(cross_entropy): 0.01936, Running average loss:0.01919, Time taken: 0:00:52.336950 ETA: 1:10:39.292936 Epoch: 19/100:, Cur-Step: 1660, loss(cross_entropy): 0.01934, Running average loss:0.01913, Time taken: 0:00:52.336950 ETA: 1:10:39.292936 2022-11-01 10:58:51,818 [INFO] __main__: Epoch: 19/100:, Cur-Step: 1660, loss(cross_entropy): 0.01934, Running average loss:0.01913, Time taken: 0:00:52.336950 ETA: 1:10:39.292936 Epoch: 19/100:, Cur-Step: 1670, loss(cross_entropy): 0.01955, Running average loss:0.01909, Time taken: 0:00:52.336950 ETA: 1:10:39.292936 2022-11-01 10:58:56,565 [INFO] __main__: Epoch: 19/100:, Cur-Step: 1670, loss(cross_entropy): 0.01955, Running average loss:0.01909, Time taken: 0:00:52.336950 ETA: 1:10:39.292936 INFO:tensorflow:Saving checkpoints for step-1680. 2022-11-01 10:59:00,795 [INFO] tensorflow: Saving checkpoints for step-1680. Epoch: 20/100:, Cur-Step: 1680, loss(cross_entropy): 0.01852, Running average loss:0.01852, Time taken: 0:00:51.349337 ETA: 1:08:27.946968 2022-11-01 10:59:12,825 [INFO] __main__: Epoch: 20/100:, Cur-Step: 1680, loss(cross_entropy): 0.01852, Running average loss:0.01852, Time taken: 0:00:51.349337 ETA: 1:08:27.946968 Epoch: 20/100:, Cur-Step: 1690, loss(cross_entropy): 0.01830, Running average loss:0.01908, Time taken: 0:00:51.349337 ETA: 1:08:27.946968 2022-11-01 10:59:17,570 [INFO] __main__: Epoch: 20/100:, Cur-Step: 1690, loss(cross_entropy): 0.01830, Running average loss:0.01908, Time taken: 0:00:51.349337 ETA: 1:08:27.946968 Epoch: 20/100:, Cur-Step: 1700, loss(cross_entropy): 0.01947, Running average loss:0.01875, Time taken: 0:00:51.349337 ETA: 1:08:27.946968 2022-11-01 10:59:22,264 [INFO] __main__: Epoch: 20/100:, Cur-Step: 1700, loss(cross_entropy): 0.01947, Running average loss:0.01875, Time taken: 0:00:51.349337 ETA: 1:08:27.946968 Epoch: 20/100:, Cur-Step: 1710, loss(cross_entropy): 0.01755, Running average loss:0.01859, Time taken: 0:00:51.349337 ETA: 1:08:27.946968 2022-11-01 10:59:26,944 [INFO] __main__: Epoch: 20/100:, Cur-Step: 1710, loss(cross_entropy): 0.01755, Running average loss:0.01859, Time taken: 0:00:51.349337 ETA: 1:08:27.946968 Epoch: 20/100:, Cur-Step: 1720, loss(cross_entropy): 0.01807, Running average loss:0.01853, Time taken: 0:00:51.349337 ETA: 1:08:27.946968 2022-11-01 10:59:31,636 [INFO] __main__: Epoch: 20/100:, Cur-Step: 1720, loss(cross_entropy): 0.01807, Running average loss:0.01853, Time taken: 0:00:51.349337 ETA: 1:08:27.946968 Epoch: 20/100:, Cur-Step: 1730, loss(cross_entropy): 0.01881, Running average loss:0.01855, Time taken: 0:00:51.349337 ETA: 1:08:27.946968 2022-11-01 10:59:36,382 [INFO] __main__: Epoch: 20/100:, Cur-Step: 1730, loss(cross_entropy): 0.01881, Running average loss:0.01855, Time taken: 0:00:51.349337 ETA: 1:08:27.946968 Epoch: 20/100:, Cur-Step: 1740, loss(cross_entropy): 0.01746, Running average loss:0.01856, Time taken: 0:00:51.349337 ETA: 1:08:27.946968 2022-11-01 10:59:41,061 [INFO] __main__: Epoch: 20/100:, Cur-Step: 1740, loss(cross_entropy): 0.01746, Running average loss:0.01856, Time taken: 0:00:51.349337 ETA: 1:08:27.946968 Epoch: 20/100:, Cur-Step: 1750, loss(cross_entropy): 0.01745, Running average loss:0.01855, Time taken: 0:00:51.349337 ETA: 1:08:27.946968 2022-11-01 10:59:45,795 [INFO] __main__: Epoch: 20/100:, Cur-Step: 1750, loss(cross_entropy): 0.01745, Running average loss:0.01855, Time taken: 0:00:51.349337 ETA: 1:08:27.946968 Epoch: 20/100:, Cur-Step: 1760, loss(cross_entropy): 0.01695, Running average loss:0.01852, Time taken: 0:00:51.349337 ETA: 1:08:27.946968 2022-11-01 10:59:50,622 [INFO] __main__: Epoch: 20/100:, Cur-Step: 1760, loss(cross_entropy): 0.01695, Running average loss:0.01852, Time taken: 0:00:51.349337 ETA: 1:08:27.946968 INFO:tensorflow:Saving checkpoints for step-1764. 2022-11-01 10:59:52,059 [INFO] tensorflow: Saving checkpoints for step-1764. Epoch: 21/100:, Cur-Step: 1770, loss(cross_entropy): 0.01743, Running average loss:0.01827, Time taken: 0:00:51.723635 ETA: 1:08:06.167199 2022-11-01 11:00:07,395 [INFO] __main__: Epoch: 21/100:, Cur-Step: 1770, loss(cross_entropy): 0.01743, Running average loss:0.01827, Time taken: 0:00:51.723635 ETA: 1:08:06.167199 Epoch: 21/100:, Cur-Step: 1780, loss(cross_entropy): 0.01812, Running average loss:0.01813, Time taken: 0:00:51.723635 ETA: 1:08:06.167199 2022-11-01 11:00:12,148 [INFO] __main__: Epoch: 21/100:, Cur-Step: 1780, loss(cross_entropy): 0.01812, Running average loss:0.01813, Time taken: 0:00:51.723635 ETA: 1:08:06.167199 Epoch: 21/100:, Cur-Step: 1790, loss(cross_entropy): 0.01764, Running average loss:0.01797, Time taken: 0:00:51.723635 ETA: 1:08:06.167199 2022-11-01 11:00:16,827 [INFO] __main__: Epoch: 21/100:, Cur-Step: 1790, loss(cross_entropy): 0.01764, Running average loss:0.01797, Time taken: 0:00:51.723635 ETA: 1:08:06.167199 Epoch: 21/100:, Cur-Step: 1800, loss(cross_entropy): 0.01811, Running average loss:0.01804, Time taken: 0:00:51.723635 ETA: 1:08:06.167199 2022-11-01 11:00:21,504 [INFO] __main__: Epoch: 21/100:, Cur-Step: 1800, loss(cross_entropy): 0.01811, Running average loss:0.01804, Time taken: 0:00:51.723635 ETA: 1:08:06.167199 Epoch: 21/100:, Cur-Step: 1810, loss(cross_entropy): 0.02066, Running average loss:0.01843, Time taken: 0:00:51.723635 ETA: 1:08:06.167199 2022-11-01 11:00:26,192 [INFO] __main__: Epoch: 21/100:, Cur-Step: 1810, loss(cross_entropy): 0.02066, Running average loss:0.01843, Time taken: 0:00:51.723635 ETA: 1:08:06.167199 Epoch: 21/100:, Cur-Step: 1820, loss(cross_entropy): 0.01802, Running average loss:0.01839, Time taken: 0:00:51.723635 ETA: 1:08:06.167199 2022-11-01 11:00:30,861 [INFO] __main__: Epoch: 21/100:, Cur-Step: 1820, loss(cross_entropy): 0.01802, Running average loss:0.01839, Time taken: 0:00:51.723635 ETA: 1:08:06.167199 Epoch: 21/100:, Cur-Step: 1830, loss(cross_entropy): 0.02132, Running average loss:0.01838, Time taken: 0:00:51.723635 ETA: 1:08:06.167199 2022-11-01 11:00:35,515 [INFO] __main__: Epoch: 21/100:, Cur-Step: 1830, loss(cross_entropy): 0.02132, Running average loss:0.01838, Time taken: 0:00:51.723635 ETA: 1:08:06.167199 Epoch: 21/100:, Cur-Step: 1840, loss(cross_entropy): 0.01773, Running average loss:0.01830, Time taken: 0:00:51.723635 ETA: 1:08:06.167199 2022-11-01 11:00:40,294 [INFO] __main__: Epoch: 21/100:, Cur-Step: 1840, loss(cross_entropy): 0.01773, Running average loss:0.01830, Time taken: 0:00:51.723635 ETA: 1:08:06.167199 INFO:tensorflow:Saving checkpoints for step-1848. 2022-11-01 11:00:43,720 [INFO] tensorflow: Saving checkpoints for step-1848. Epoch: 22/100:, Cur-Step: 1850, loss(cross_entropy): 0.01799, Running average loss:0.01741, Time taken: 0:00:52.152073 ETA: 1:07:47.861724 2022-11-01 11:00:56,752 [INFO] __main__: Epoch: 22/100:, Cur-Step: 1850, loss(cross_entropy): 0.01799, Running average loss:0.01741, Time taken: 0:00:52.152073 ETA: 1:07:47.861724 Epoch: 22/100:, Cur-Step: 1860, loss(cross_entropy): 0.01822, Running average loss:0.01754, Time taken: 0:00:52.152073 ETA: 1:07:47.861724 2022-11-01 11:01:01,611 [INFO] __main__: Epoch: 22/100:, Cur-Step: 1860, loss(cross_entropy): 0.01822, Running average loss:0.01754, Time taken: 0:00:52.152073 ETA: 1:07:47.861724 Epoch: 22/100:, Cur-Step: 1870, loss(cross_entropy): 0.01736, Running average loss:0.01764, Time taken: 0:00:52.152073 ETA: 1:07:47.861724 2022-11-01 11:01:06,352 [INFO] __main__: Epoch: 22/100:, Cur-Step: 1870, loss(cross_entropy): 0.01736, Running average loss:0.01764, Time taken: 0:00:52.152073 ETA: 1:07:47.861724 Epoch: 22/100:, Cur-Step: 1880, loss(cross_entropy): 0.01872, Running average loss:0.01771, Time taken: 0:00:52.152073 ETA: 1:07:47.861724 2022-11-01 11:01:11,165 [INFO] __main__: Epoch: 22/100:, Cur-Step: 1880, loss(cross_entropy): 0.01872, Running average loss:0.01771, Time taken: 0:00:52.152073 ETA: 1:07:47.861724 Epoch: 22/100:, Cur-Step: 1890, loss(cross_entropy): 0.01682, Running average loss:0.01771, Time taken: 0:00:52.152073 ETA: 1:07:47.861724 2022-11-01 11:01:15,878 [INFO] __main__: Epoch: 22/100:, Cur-Step: 1890, loss(cross_entropy): 0.01682, Running average loss:0.01771, Time taken: 0:00:52.152073 ETA: 1:07:47.861724 Epoch: 22/100:, Cur-Step: 1900, loss(cross_entropy): 0.01747, Running average loss:0.01775, Time taken: 0:00:52.152073 ETA: 1:07:47.861724 2022-11-01 11:01:20,618 [INFO] __main__: Epoch: 22/100:, Cur-Step: 1900, loss(cross_entropy): 0.01747, Running average loss:0.01775, Time taken: 0:00:52.152073 ETA: 1:07:47.861724 Epoch: 22/100:, Cur-Step: 1910, loss(cross_entropy): 0.01638, Running average loss:0.01770, Time taken: 0:00:52.152073 ETA: 1:07:47.861724 2022-11-01 11:01:25,343 [INFO] __main__: Epoch: 22/100:, Cur-Step: 1910, loss(cross_entropy): 0.01638, Running average loss:0.01770, Time taken: 0:00:52.152073 ETA: 1:07:47.861724 Epoch: 22/100:, Cur-Step: 1920, loss(cross_entropy): 0.01779, Running average loss:0.01765, Time taken: 0:00:52.152073 ETA: 1:07:47.861724 2022-11-01 11:01:30,082 [INFO] __main__: Epoch: 22/100:, Cur-Step: 1920, loss(cross_entropy): 0.01779, Running average loss:0.01765, Time taken: 0:00:52.152073 ETA: 1:07:47.861724 Epoch: 22/100:, Cur-Step: 1930, loss(cross_entropy): 0.01698, Running average loss:0.01758, Time taken: 0:00:52.152073 ETA: 1:07:47.861724 2022-11-01 11:01:34,983 [INFO] __main__: Epoch: 22/100:, Cur-Step: 1930, loss(cross_entropy): 0.01698, Running average loss:0.01758, Time taken: 0:00:52.152073 ETA: 1:07:47.861724 INFO:tensorflow:Saving checkpoints for step-1932. 2022-11-01 11:01:35,455 [INFO] tensorflow: Saving checkpoints for step-1932. Epoch: 23/100:, Cur-Step: 1940, loss(cross_entropy): 0.01707, Running average loss:0.01691, Time taken: 0:00:52.201483 ETA: 1:06:59.514210 2022-11-01 11:01:51,618 [INFO] __main__: Epoch: 23/100:, Cur-Step: 1940, loss(cross_entropy): 0.01707, Running average loss:0.01691, Time taken: 0:00:52.201483 ETA: 1:06:59.514210 Epoch: 23/100:, Cur-Step: 1950, loss(cross_entropy): 0.01777, Running average loss:0.01715, Time taken: 0:00:52.201483 ETA: 1:06:59.514210 2022-11-01 11:01:56,348 [INFO] __main__: Epoch: 23/100:, Cur-Step: 1950, loss(cross_entropy): 0.01777, Running average loss:0.01715, Time taken: 0:00:52.201483 ETA: 1:06:59.514210 Epoch: 23/100:, Cur-Step: 1960, loss(cross_entropy): 0.01702, Running average loss:0.01719, Time taken: 0:00:52.201483 ETA: 1:06:59.514210 2022-11-01 11:02:01,039 [INFO] __main__: Epoch: 23/100:, Cur-Step: 1960, loss(cross_entropy): 0.01702, Running average loss:0.01719, Time taken: 0:00:52.201483 ETA: 1:06:59.514210 Epoch: 23/100:, Cur-Step: 1970, loss(cross_entropy): 0.01716, Running average loss:0.01714, Time taken: 0:00:52.201483 ETA: 1:06:59.514210 2022-11-01 11:02:05,727 [INFO] __main__: Epoch: 23/100:, Cur-Step: 1970, loss(cross_entropy): 0.01716, Running average loss:0.01714, Time taken: 0:00:52.201483 ETA: 1:06:59.514210 Epoch: 23/100:, Cur-Step: 1980, loss(cross_entropy): 0.02272, Running average loss:0.01720, Time taken: 0:00:52.201483 ETA: 1:06:59.514210 2022-11-01 11:02:10,410 [INFO] __main__: Epoch: 23/100:, Cur-Step: 1980, loss(cross_entropy): 0.02272, Running average loss:0.01720, Time taken: 0:00:52.201483 ETA: 1:06:59.514210 Epoch: 23/100:, Cur-Step: 1990, loss(cross_entropy): 0.01689, Running average loss:0.01716, Time taken: 0:00:52.201483 ETA: 1:06:59.514210 2022-11-01 11:02:15,093 [INFO] __main__: Epoch: 23/100:, Cur-Step: 1990, loss(cross_entropy): 0.01689, Running average loss:0.01716, Time taken: 0:00:52.201483 ETA: 1:06:59.514210 Epoch: 23/100:, Cur-Step: 2000, loss(cross_entropy): 0.01648, Running average loss:0.01714, Time taken: 0:00:52.201483 ETA: 1:06:59.514210 2022-11-01 11:02:19,792 [INFO] __main__: Epoch: 23/100:, Cur-Step: 2000, loss(cross_entropy): 0.01648, Running average loss:0.01714, Time taken: 0:00:52.201483 ETA: 1:06:59.514210 Epoch: 23/100:, Cur-Step: 2010, loss(cross_entropy): 0.01716, Running average loss:0.01714, Time taken: 0:00:52.201483 ETA: 1:06:59.514210 2022-11-01 11:02:24,492 [INFO] __main__: Epoch: 23/100:, Cur-Step: 2010, loss(cross_entropy): 0.01716, Running average loss:0.01714, Time taken: 0:00:52.201483 ETA: 1:06:59.514210 INFO:tensorflow:Saving checkpoints for step-2016. 2022-11-01 11:02:26,975 [INFO] tensorflow: Saving checkpoints for step-2016. Epoch: 24/100:, Cur-Step: 2020, loss(cross_entropy): 0.01591, Running average loss:0.01656, Time taken: 0:00:51.990436 ETA: 1:05:51.273160 2022-11-01 11:02:40,892 [INFO] __main__: Epoch: 24/100:, Cur-Step: 2020, loss(cross_entropy): 0.01591, Running average loss:0.01656, Time taken: 0:00:51.990436 ETA: 1:05:51.273160 Epoch: 24/100:, Cur-Step: 2030, loss(cross_entropy): 0.01676, Running average loss:0.01670, Time taken: 0:00:51.990436 ETA: 1:05:51.273160 2022-11-01 11:02:45,561 [INFO] __main__: Epoch: 24/100:, Cur-Step: 2030, loss(cross_entropy): 0.01676, Running average loss:0.01670, Time taken: 0:00:51.990436 ETA: 1:05:51.273160 Epoch: 24/100:, Cur-Step: 2040, loss(cross_entropy): 0.01681, Running average loss:0.01676, Time taken: 0:00:51.990436 ETA: 1:05:51.273160 2022-11-01 11:02:50,253 [INFO] __main__: Epoch: 24/100:, Cur-Step: 2040, loss(cross_entropy): 0.01681, Running average loss:0.01676, Time taken: 0:00:51.990436 ETA: 1:05:51.273160 Epoch: 24/100:, Cur-Step: 2050, loss(cross_entropy): 0.01580, Running average loss:0.01672, Time taken: 0:00:51.990436 ETA: 1:05:51.273160 2022-11-01 11:02:54,932 [INFO] __main__: Epoch: 24/100:, Cur-Step: 2050, loss(cross_entropy): 0.01580, Running average loss:0.01672, Time taken: 0:00:51.990436 ETA: 1:05:51.273160 Epoch: 24/100:, Cur-Step: 2060, loss(cross_entropy): 0.01625, Running average loss:0.01663, Time taken: 0:00:51.990436 ETA: 1:05:51.273160 2022-11-01 11:02:59,658 [INFO] __main__: Epoch: 24/100:, Cur-Step: 2060, loss(cross_entropy): 0.01625, Running average loss:0.01663, Time taken: 0:00:51.990436 ETA: 1:05:51.273160 Epoch: 24/100:, Cur-Step: 2070, loss(cross_entropy): 0.01752, Running average loss:0.01666, Time taken: 0:00:51.990436 ETA: 1:05:51.273160 2022-11-01 11:03:04,356 [INFO] __main__: Epoch: 24/100:, Cur-Step: 2070, loss(cross_entropy): 0.01752, Running average loss:0.01666, Time taken: 0:00:51.990436 ETA: 1:05:51.273160 Epoch: 24/100:, Cur-Step: 2080, loss(cross_entropy): 0.01605, Running average loss:0.01674, Time taken: 0:00:51.990436 ETA: 1:05:51.273160 2022-11-01 11:03:09,019 [INFO] __main__: Epoch: 24/100:, Cur-Step: 2080, loss(cross_entropy): 0.01605, Running average loss:0.01674, Time taken: 0:00:51.990436 ETA: 1:05:51.273160 Epoch: 24/100:, Cur-Step: 2090, loss(cross_entropy): 0.01583, Running average loss:0.01667, Time taken: 0:00:51.990436 ETA: 1:05:51.273160 2022-11-01 11:03:13,851 [INFO] __main__: Epoch: 24/100:, Cur-Step: 2090, loss(cross_entropy): 0.01583, Running average loss:0.01667, Time taken: 0:00:51.990436 ETA: 1:05:51.273160 INFO:tensorflow:Saving checkpoints for step-2100. 2022-11-01 11:03:18,085 [INFO] tensorflow: Saving checkpoints for step-2100. Epoch: 25/100:, Cur-Step: 2100, loss(cross_entropy): 0.01562, Running average loss:0.01562, Time taken: 0:00:51.579361 ETA: 1:04:28.452072 2022-11-01 11:03:30,352 [INFO] __main__: Epoch: 25/100:, Cur-Step: 2100, loss(cross_entropy): 0.01562, Running average loss:0.01562, Time taken: 0:00:51.579361 ETA: 1:04:28.452072 Epoch: 25/100:, Cur-Step: 2110, loss(cross_entropy): 0.01563, Running average loss:0.01587, Time taken: 0:00:51.579361 ETA: 1:04:28.452072 2022-11-01 11:03:35,127 [INFO] __main__: Epoch: 25/100:, Cur-Step: 2110, loss(cross_entropy): 0.01563, Running average loss:0.01587, Time taken: 0:00:51.579361 ETA: 1:04:28.452072 Epoch: 25/100:, Cur-Step: 2120, loss(cross_entropy): 0.01549, Running average loss:0.01614, Time taken: 0:00:51.579361 ETA: 1:04:28.452072 2022-11-01 11:03:39,857 [INFO] __main__: Epoch: 25/100:, Cur-Step: 2120, loss(cross_entropy): 0.01549, Running average loss:0.01614, Time taken: 0:00:51.579361 ETA: 1:04:28.452072 Epoch: 25/100:, Cur-Step: 2130, loss(cross_entropy): 0.01562, Running average loss:0.01609, Time taken: 0:00:51.579361 ETA: 1:04:28.452072 2022-11-01 11:03:44,560 [INFO] __main__: Epoch: 25/100:, Cur-Step: 2130, loss(cross_entropy): 0.01562, Running average loss:0.01609, Time taken: 0:00:51.579361 ETA: 1:04:28.452072 Epoch: 25/100:, Cur-Step: 2140, loss(cross_entropy): 0.01785, Running average loss:0.01615, Time taken: 0:00:51.579361 ETA: 1:04:28.452072 2022-11-01 11:03:49,280 [INFO] __main__: Epoch: 25/100:, Cur-Step: 2140, loss(cross_entropy): 0.01785, Running average loss:0.01615, Time taken: 0:00:51.579361 ETA: 1:04:28.452072 Epoch: 25/100:, Cur-Step: 2150, loss(cross_entropy): 0.01554, Running average loss:0.01624, Time taken: 0:00:51.579361 ETA: 1:04:28.452072 2022-11-01 11:03:53,972 [INFO] __main__: Epoch: 25/100:, Cur-Step: 2150, loss(cross_entropy): 0.01554, Running average loss:0.01624, Time taken: 0:00:51.579361 ETA: 1:04:28.452072 Epoch: 25/100:, Cur-Step: 2160, loss(cross_entropy): 0.01575, Running average loss:0.01621, Time taken: 0:00:51.579361 ETA: 1:04:28.452072 2022-11-01 11:03:58,664 [INFO] __main__: Epoch: 25/100:, Cur-Step: 2160, loss(cross_entropy): 0.01575, Running average loss:0.01621, Time taken: 0:00:51.579361 ETA: 1:04:28.452072 Epoch: 25/100:, Cur-Step: 2170, loss(cross_entropy): 0.01560, Running average loss:0.01622, Time taken: 0:00:51.579361 ETA: 1:04:28.452072 2022-11-01 11:04:03,485 [INFO] __main__: Epoch: 25/100:, Cur-Step: 2170, loss(cross_entropy): 0.01560, Running average loss:0.01622, Time taken: 0:00:51.579361 ETA: 1:04:28.452072 Epoch: 25/100:, Cur-Step: 2180, loss(cross_entropy): 0.01635, Running average loss:0.01616, Time taken: 0:00:51.579361 ETA: 1:04:28.452072 2022-11-01 11:04:08,196 [INFO] __main__: Epoch: 25/100:, Cur-Step: 2180, loss(cross_entropy): 0.01635, Running average loss:0.01616, Time taken: 0:00:51.579361 ETA: 1:04:28.452072 INFO:tensorflow:Saving checkpoints for step-2184. 2022-11-01 11:04:09,620 [INFO] tensorflow: Saving checkpoints for step-2184. Epoch: 26/100:, Cur-Step: 2190, loss(cross_entropy): 0.01532, Running average loss:0.01613, Time taken: 0:00:52.001179 ETA: 1:04:08.087245 2022-11-01 11:04:24,477 [INFO] __main__: Epoch: 26/100:, Cur-Step: 2190, loss(cross_entropy): 0.01532, Running average loss:0.01613, Time taken: 0:00:52.001179 ETA: 1:04:08.087245 Epoch: 26/100:, Cur-Step: 2200, loss(cross_entropy): 0.01584, Running average loss:0.01635, Time taken: 0:00:52.001179 ETA: 1:04:08.087245 2022-11-01 11:04:29,158 [INFO] __main__: Epoch: 26/100:, Cur-Step: 2200, loss(cross_entropy): 0.01584, Running average loss:0.01635, Time taken: 0:00:52.001179 ETA: 1:04:08.087245 Epoch: 26/100:, Cur-Step: 2210, loss(cross_entropy): 0.01564, Running average loss:0.01636, Time taken: 0:00:52.001179 ETA: 1:04:08.087245 2022-11-01 11:04:33,852 [INFO] __main__: Epoch: 26/100:, Cur-Step: 2210, loss(cross_entropy): 0.01564, Running average loss:0.01636, Time taken: 0:00:52.001179 ETA: 1:04:08.087245 Epoch: 26/100:, Cur-Step: 2220, loss(cross_entropy): 0.01593, Running average loss:0.01619, Time taken: 0:00:52.001179 ETA: 1:04:08.087245 2022-11-01 11:04:38,533 [INFO] __main__: Epoch: 26/100:, Cur-Step: 2220, loss(cross_entropy): 0.01593, Running average loss:0.01619, Time taken: 0:00:52.001179 ETA: 1:04:08.087245 Epoch: 26/100:, Cur-Step: 2230, loss(cross_entropy): 0.01536, Running average loss:0.01617, Time taken: 0:00:52.001179 ETA: 1:04:08.087245 2022-11-01 11:04:43,216 [INFO] __main__: Epoch: 26/100:, Cur-Step: 2230, loss(cross_entropy): 0.01536, Running average loss:0.01617, Time taken: 0:00:52.001179 ETA: 1:04:08.087245 Epoch: 26/100:, Cur-Step: 2240, loss(cross_entropy): 0.01556, Running average loss:0.01605, Time taken: 0:00:52.001179 ETA: 1:04:08.087245 2022-11-01 11:04:48,009 [INFO] __main__: Epoch: 26/100:, Cur-Step: 2240, loss(cross_entropy): 0.01556, Running average loss:0.01605, Time taken: 0:00:52.001179 ETA: 1:04:08.087245 Epoch: 26/100:, Cur-Step: 2250, loss(cross_entropy): 0.01981, Running average loss:0.01603, Time taken: 0:00:52.001179 ETA: 1:04:08.087245 2022-11-01 11:04:52,715 [INFO] __main__: Epoch: 26/100:, Cur-Step: 2250, loss(cross_entropy): 0.01981, Running average loss:0.01603, Time taken: 0:00:52.001179 ETA: 1:04:08.087245 Epoch: 26/100:, Cur-Step: 2260, loss(cross_entropy): 0.01514, Running average loss:0.01597, Time taken: 0:00:52.001179 ETA: 1:04:08.087245 2022-11-01 11:04:57,443 [INFO] __main__: Epoch: 26/100:, Cur-Step: 2260, loss(cross_entropy): 0.01514, Running average loss:0.01597, Time taken: 0:00:52.001179 ETA: 1:04:08.087245 INFO:tensorflow:Saving checkpoints for step-2268. 2022-11-01 11:05:00,737 [INFO] tensorflow: Saving checkpoints for step-2268. Epoch: 27/100:, Cur-Step: 2270, loss(cross_entropy): 0.01685, Running average loss:0.01556, Time taken: 0:00:51.599600 ETA: 1:02:46.770788 2022-11-01 11:05:13,244 [INFO] __main__: Epoch: 27/100:, Cur-Step: 2270, loss(cross_entropy): 0.01685, Running average loss:0.01556, Time taken: 0:00:51.599600 ETA: 1:02:46.770788 Epoch: 27/100:, Cur-Step: 2280, loss(cross_entropy): 0.01563, Running average loss:0.01544, Time taken: 0:00:51.599600 ETA: 1:02:46.770788 2022-11-01 11:05:17,900 [INFO] __main__: Epoch: 27/100:, Cur-Step: 2280, loss(cross_entropy): 0.01563, Running average loss:0.01544, Time taken: 0:00:51.599600 ETA: 1:02:46.770788 Epoch: 27/100:, Cur-Step: 2290, loss(cross_entropy): 0.01515, Running average loss:0.01538, Time taken: 0:00:51.599600 ETA: 1:02:46.770788 2022-11-01 11:05:22,596 [INFO] __main__: Epoch: 27/100:, Cur-Step: 2290, loss(cross_entropy): 0.01515, Running average loss:0.01538, Time taken: 0:00:51.599600 ETA: 1:02:46.770788 Epoch: 27/100:, Cur-Step: 2300, loss(cross_entropy): 0.01541, Running average loss:0.01541, Time taken: 0:00:51.599600 ETA: 1:02:46.770788 2022-11-01 11:05:27,414 [INFO] __main__: Epoch: 27/100:, Cur-Step: 2300, loss(cross_entropy): 0.01541, Running average loss:0.01541, Time taken: 0:00:51.599600 ETA: 1:02:46.770788 Epoch: 27/100:, Cur-Step: 2310, loss(cross_entropy): 0.01584, Running average loss:0.01540, Time taken: 0:00:51.599600 ETA: 1:02:46.770788 2022-11-01 11:05:32,086 [INFO] __main__: Epoch: 27/100:, Cur-Step: 2310, loss(cross_entropy): 0.01584, Running average loss:0.01540, Time taken: 0:00:51.599600 ETA: 1:02:46.770788 Epoch: 27/100:, Cur-Step: 2320, loss(cross_entropy): 0.01512, Running average loss:0.01538, Time taken: 0:00:51.599600 ETA: 1:02:46.770788 2022-11-01 11:05:36,803 [INFO] __main__: Epoch: 27/100:, Cur-Step: 2320, loss(cross_entropy): 0.01512, Running average loss:0.01538, Time taken: 0:00:51.599600 ETA: 1:02:46.770788 Epoch: 27/100:, Cur-Step: 2330, loss(cross_entropy): 0.01457, Running average loss:0.01532, Time taken: 0:00:51.599600 ETA: 1:02:46.770788 2022-11-01 11:05:41,484 [INFO] __main__: Epoch: 27/100:, Cur-Step: 2330, loss(cross_entropy): 0.01457, Running average loss:0.01532, Time taken: 0:00:51.599600 ETA: 1:02:46.770788 Epoch: 27/100:, Cur-Step: 2340, loss(cross_entropy): 0.01484, Running average loss:0.01524, Time taken: 0:00:51.599600 ETA: 1:02:46.770788 2022-11-01 11:05:46,373 [INFO] __main__: Epoch: 27/100:, Cur-Step: 2340, loss(cross_entropy): 0.01484, Running average loss:0.01524, Time taken: 0:00:51.599600 ETA: 1:02:46.770788 Epoch: 27/100:, Cur-Step: 2350, loss(cross_entropy): 0.01469, Running average loss:0.01526, Time taken: 0:00:51.599600 ETA: 1:02:46.770788 2022-11-01 11:05:51,262 [INFO] __main__: Epoch: 27/100:, Cur-Step: 2350, loss(cross_entropy): 0.01469, Running average loss:0.01526, Time taken: 0:00:51.599600 ETA: 1:02:46.770788 INFO:tensorflow:Saving checkpoints for step-2352. 2022-11-01 11:05:51,736 [INFO] tensorflow: Saving checkpoints for step-2352. Epoch: 28/100:, Cur-Step: 2360, loss(cross_entropy): 0.01562, Running average loss:0.01479, Time taken: 0:00:51.462977 ETA: 1:01:45.334373 2022-11-01 11:06:07,771 [INFO] __main__: Epoch: 28/100:, Cur-Step: 2360, loss(cross_entropy): 0.01562, Running average loss:0.01479, Time taken: 0:00:51.462977 ETA: 1:01:45.334373 Epoch: 28/100:, Cur-Step: 2370, loss(cross_entropy): 0.01439, Running average loss:0.01493, Time taken: 0:00:51.462977 ETA: 1:01:45.334373 2022-11-01 11:06:12,546 [INFO] __main__: Epoch: 28/100:, Cur-Step: 2370, loss(cross_entropy): 0.01439, Running average loss:0.01493, Time taken: 0:00:51.462977 ETA: 1:01:45.334373 Epoch: 28/100:, Cur-Step: 2380, loss(cross_entropy): 0.01454, Running average loss:0.01492, Time taken: 0:00:51.462977 ETA: 1:01:45.334373 2022-11-01 11:06:17,213 [INFO] __main__: Epoch: 28/100:, Cur-Step: 2380, loss(cross_entropy): 0.01454, Running average loss:0.01492, Time taken: 0:00:51.462977 ETA: 1:01:45.334373 Epoch: 28/100:, Cur-Step: 2390, loss(cross_entropy): 0.01406, Running average loss:0.01494, Time taken: 0:00:51.462977 ETA: 1:01:45.334373 2022-11-01 11:06:22,027 [INFO] __main__: Epoch: 28/100:, Cur-Step: 2390, loss(cross_entropy): 0.01406, Running average loss:0.01494, Time taken: 0:00:51.462977 ETA: 1:01:45.334373 Epoch: 28/100:, Cur-Step: 2400, loss(cross_entropy): 0.01476, Running average loss:0.01488, Time taken: 0:00:51.462977 ETA: 1:01:45.334373 2022-11-01 11:06:26,855 [INFO] __main__: Epoch: 28/100:, Cur-Step: 2400, loss(cross_entropy): 0.01476, Running average loss:0.01488, Time taken: 0:00:51.462977 ETA: 1:01:45.334373 Epoch: 28/100:, Cur-Step: 2410, loss(cross_entropy): 0.01410, Running average loss:0.01489, Time taken: 0:00:51.462977 ETA: 1:01:45.334373 2022-11-01 11:06:31,531 [INFO] __main__: Epoch: 28/100:, Cur-Step: 2410, loss(cross_entropy): 0.01410, Running average loss:0.01489, Time taken: 0:00:51.462977 ETA: 1:01:45.334373 Epoch: 28/100:, Cur-Step: 2420, loss(cross_entropy): 0.01441, Running average loss:0.01491, Time taken: 0:00:51.462977 ETA: 1:01:45.334373 2022-11-01 11:06:36,319 [INFO] __main__: Epoch: 28/100:, Cur-Step: 2420, loss(cross_entropy): 0.01441, Running average loss:0.01491, Time taken: 0:00:51.462977 ETA: 1:01:45.334373 Epoch: 28/100:, Cur-Step: 2430, loss(cross_entropy): 0.01452, Running average loss:0.01488, Time taken: 0:00:51.462977 ETA: 1:01:45.334373 2022-11-01 11:06:40,996 [INFO] __main__: Epoch: 28/100:, Cur-Step: 2430, loss(cross_entropy): 0.01452, Running average loss:0.01488, Time taken: 0:00:51.462977 ETA: 1:01:45.334373 INFO:tensorflow:Saving checkpoints for step-2436. 2022-11-01 11:06:43,358 [INFO] tensorflow: Saving checkpoints for step-2436. Epoch: 29/100:, Cur-Step: 2440, loss(cross_entropy): 0.01522, Running average loss:0.01464, Time taken: 0:00:52.095112 ETA: 1:01:38.752941 2022-11-01 11:06:57,008 [INFO] __main__: Epoch: 29/100:, Cur-Step: 2440, loss(cross_entropy): 0.01522, Running average loss:0.01464, Time taken: 0:00:52.095112 ETA: 1:01:38.752941 Epoch: 29/100:, Cur-Step: 2450, loss(cross_entropy): 0.01538, Running average loss:0.01436, Time taken: 0:00:52.095112 ETA: 1:01:38.752941 2022-11-01 11:07:01,698 [INFO] __main__: Epoch: 29/100:, Cur-Step: 2450, loss(cross_entropy): 0.01538, Running average loss:0.01436, Time taken: 0:00:52.095112 ETA: 1:01:38.752941 Epoch: 29/100:, Cur-Step: 2460, loss(cross_entropy): 0.01535, Running average loss:0.01450, Time taken: 0:00:52.095112 ETA: 1:01:38.752941 2022-11-01 11:07:06,415 [INFO] __main__: Epoch: 29/100:, Cur-Step: 2460, loss(cross_entropy): 0.01535, Running average loss:0.01450, Time taken: 0:00:52.095112 ETA: 1:01:38.752941 Epoch: 29/100:, Cur-Step: 2470, loss(cross_entropy): 0.01496, Running average loss:0.01455, Time taken: 0:00:52.095112 ETA: 1:01:38.752941 2022-11-01 11:07:11,231 [INFO] __main__: Epoch: 29/100:, Cur-Step: 2470, loss(cross_entropy): 0.01496, Running average loss:0.01455, Time taken: 0:00:52.095112 ETA: 1:01:38.752941 Epoch: 29/100:, Cur-Step: 2480, loss(cross_entropy): 0.01456, Running average loss:0.01452, Time taken: 0:00:52.095112 ETA: 1:01:38.752941 2022-11-01 11:07:15,892 [INFO] __main__: Epoch: 29/100:, Cur-Step: 2480, loss(cross_entropy): 0.01456, Running average loss:0.01452, Time taken: 0:00:52.095112 ETA: 1:01:38.752941 Epoch: 29/100:, Cur-Step: 2490, loss(cross_entropy): 0.01437, Running average loss:0.01448, Time taken: 0:00:52.095112 ETA: 1:01:38.752941 2022-11-01 11:07:20,658 [INFO] __main__: Epoch: 29/100:, Cur-Step: 2490, loss(cross_entropy): 0.01437, Running average loss:0.01448, Time taken: 0:00:52.095112 ETA: 1:01:38.752941 Epoch: 29/100:, Cur-Step: 2500, loss(cross_entropy): 0.01695, Running average loss:0.01461, Time taken: 0:00:52.095112 ETA: 1:01:38.752941 2022-11-01 11:07:25,391 [INFO] __main__: Epoch: 29/100:, Cur-Step: 2500, loss(cross_entropy): 0.01695, Running average loss:0.01461, Time taken: 0:00:52.095112 ETA: 1:01:38.752941 Epoch: 29/100:, Cur-Step: 2510, loss(cross_entropy): 0.01370, Running average loss:0.01458, Time taken: 0:00:52.095112 ETA: 1:01:38.752941 2022-11-01 11:07:30,056 [INFO] __main__: Epoch: 29/100:, Cur-Step: 2510, loss(cross_entropy): 0.01370, Running average loss:0.01458, Time taken: 0:00:52.095112 ETA: 1:01:38.752941 INFO:tensorflow:Saving checkpoints for step-2520. 2022-11-01 11:07:34,292 [INFO] tensorflow: Saving checkpoints for step-2520. Epoch: 30/100:, Cur-Step: 2520, loss(cross_entropy): 0.01423, Running average loss:0.01423, Time taken: 0:00:51.400751 ETA: 0:59:58.052545 2022-11-01 11:07:45,907 [INFO] __main__: Epoch: 30/100:, Cur-Step: 2520, loss(cross_entropy): 0.01423, Running average loss:0.01423, Time taken: 0:00:51.400751 ETA: 0:59:58.052545 Epoch: 30/100:, Cur-Step: 2530, loss(cross_entropy): 0.01521, Running average loss:0.01486, Time taken: 0:00:51.400751 ETA: 0:59:58.052545 2022-11-01 11:07:50,592 [INFO] __main__: Epoch: 30/100:, Cur-Step: 2530, loss(cross_entropy): 0.01521, Running average loss:0.01486, Time taken: 0:00:51.400751 ETA: 0:59:58.052545 Epoch: 30/100:, Cur-Step: 2540, loss(cross_entropy): 0.01371, Running average loss:0.01447, Time taken: 0:00:51.400751 ETA: 0:59:58.052545 2022-11-01 11:07:55,262 [INFO] __main__: Epoch: 30/100:, Cur-Step: 2540, loss(cross_entropy): 0.01371, Running average loss:0.01447, Time taken: 0:00:51.400751 ETA: 0:59:58.052545 Epoch: 30/100:, Cur-Step: 2550, loss(cross_entropy): 0.01409, Running average loss:0.01434, Time taken: 0:00:51.400751 ETA: 0:59:58.052545 2022-11-01 11:07:59,928 [INFO] __main__: Epoch: 30/100:, Cur-Step: 2550, loss(cross_entropy): 0.01409, Running average loss:0.01434, Time taken: 0:00:51.400751 ETA: 0:59:58.052545 Epoch: 30/100:, Cur-Step: 2560, loss(cross_entropy): 0.01369, Running average loss:0.01431, Time taken: 0:00:51.400751 ETA: 0:59:58.052545 2022-11-01 11:08:04,750 [INFO] __main__: Epoch: 30/100:, Cur-Step: 2560, loss(cross_entropy): 0.01369, Running average loss:0.01431, Time taken: 0:00:51.400751 ETA: 0:59:58.052545 Epoch: 30/100:, Cur-Step: 2570, loss(cross_entropy): 0.01448, Running average loss:0.01425, Time taken: 0:00:51.400751 ETA: 0:59:58.052545 2022-11-01 11:08:09,429 [INFO] __main__: Epoch: 30/100:, Cur-Step: 2570, loss(cross_entropy): 0.01448, Running average loss:0.01425, Time taken: 0:00:51.400751 ETA: 0:59:58.052545 Epoch: 30/100:, Cur-Step: 2580, loss(cross_entropy): 0.01421, Running average loss:0.01432, Time taken: 0:00:51.400751 ETA: 0:59:58.052545 2022-11-01 11:08:14,095 [INFO] __main__: Epoch: 30/100:, Cur-Step: 2580, loss(cross_entropy): 0.01421, Running average loss:0.01432, Time taken: 0:00:51.400751 ETA: 0:59:58.052545 Epoch: 30/100:, Cur-Step: 2590, loss(cross_entropy): 0.01442, Running average loss:0.01430, Time taken: 0:00:51.400751 ETA: 0:59:58.052545 2022-11-01 11:08:18,923 [INFO] __main__: Epoch: 30/100:, Cur-Step: 2590, loss(cross_entropy): 0.01442, Running average loss:0.01430, Time taken: 0:00:51.400751 ETA: 0:59:58.052545 Epoch: 30/100:, Cur-Step: 2600, loss(cross_entropy): 0.01384, Running average loss:0.01423, Time taken: 0:00:51.400751 ETA: 0:59:58.052545 2022-11-01 11:08:23,814 [INFO] __main__: Epoch: 30/100:, Cur-Step: 2600, loss(cross_entropy): 0.01384, Running average loss:0.01423, Time taken: 0:00:51.400751 ETA: 0:59:58.052545 INFO:tensorflow:Saving checkpoints for step-2604. 2022-11-01 11:08:25,412 [INFO] tensorflow: Saving checkpoints for step-2604. Epoch: 31/100:, Cur-Step: 2610, loss(cross_entropy): 0.01653, Running average loss:0.01443, Time taken: 0:00:51.568223 ETA: 0:59:18.207354 2022-11-01 11:08:44,943 [INFO] __main__: Epoch: 31/100:, Cur-Step: 2610, loss(cross_entropy): 0.01653, Running average loss:0.01443, Time taken: 0:00:51.568223 ETA: 0:59:18.207354 Epoch: 31/100:, Cur-Step: 2620, loss(cross_entropy): 0.01378, Running average loss:0.01421, Time taken: 0:00:51.568223 ETA: 0:59:18.207354 2022-11-01 11:08:49,667 [INFO] __main__: Epoch: 31/100:, Cur-Step: 2620, loss(cross_entropy): 0.01378, Running average loss:0.01421, Time taken: 0:00:51.568223 ETA: 0:59:18.207354 Epoch: 31/100:, Cur-Step: 2630, loss(cross_entropy): 0.01339, Running average loss:0.01397, Time taken: 0:00:51.568223 ETA: 0:59:18.207354 2022-11-01 11:08:54,310 [INFO] __main__: Epoch: 31/100:, Cur-Step: 2630, loss(cross_entropy): 0.01339, Running average loss:0.01397, Time taken: 0:00:51.568223 ETA: 0:59:18.207354 Epoch: 31/100:, Cur-Step: 2640, loss(cross_entropy): 0.01330, Running average loss:0.01387, Time taken: 0:00:51.568223 ETA: 0:59:18.207354 2022-11-01 11:08:59,005 [INFO] __main__: Epoch: 31/100:, Cur-Step: 2640, loss(cross_entropy): 0.01330, Running average loss:0.01387, Time taken: 0:00:51.568223 ETA: 0:59:18.207354 Epoch: 31/100:, Cur-Step: 2650, loss(cross_entropy): 0.01423, Running average loss:0.01383, Time taken: 0:00:51.568223 ETA: 0:59:18.207354 2022-11-01 11:09:03,667 [INFO] __main__: Epoch: 31/100:, Cur-Step: 2650, loss(cross_entropy): 0.01423, Running average loss:0.01383, Time taken: 0:00:51.568223 ETA: 0:59:18.207354 Epoch: 31/100:, Cur-Step: 2660, loss(cross_entropy): 0.01357, Running average loss:0.01385, Time taken: 0:00:51.568223 ETA: 0:59:18.207354 2022-11-01 11:09:08,337 [INFO] __main__: Epoch: 31/100:, Cur-Step: 2660, loss(cross_entropy): 0.01357, Running average loss:0.01385, Time taken: 0:00:51.568223 ETA: 0:59:18.207354 Epoch: 31/100:, Cur-Step: 2670, loss(cross_entropy): 0.01318, Running average loss:0.01383, Time taken: 0:00:51.568223 ETA: 0:59:18.207354 2022-11-01 11:09:13,086 [INFO] __main__: Epoch: 31/100:, Cur-Step: 2670, loss(cross_entropy): 0.01318, Running average loss:0.01383, Time taken: 0:00:51.568223 ETA: 0:59:18.207354 Epoch: 31/100:, Cur-Step: 2680, loss(cross_entropy): 0.01296, Running average loss:0.01380, Time taken: 0:00:51.568223 ETA: 0:59:18.207354 2022-11-01 11:09:17,737 [INFO] __main__: Epoch: 31/100:, Cur-Step: 2680, loss(cross_entropy): 0.01296, Running average loss:0.01380, Time taken: 0:00:51.568223 ETA: 0:59:18.207354 INFO:tensorflow:Saving checkpoints for step-2688. 2022-11-01 11:09:21,029 [INFO] tensorflow: Saving checkpoints for step-2688. Epoch: 32/100:, Cur-Step: 2690, loss(cross_entropy): 0.01262, Running average loss:0.01342, Time taken: 0:00:56.169872 ETA: 1:03:39.551267 2022-11-01 11:09:33,152 [INFO] __main__: Epoch: 32/100:, Cur-Step: 2690, loss(cross_entropy): 0.01262, Running average loss:0.01342, Time taken: 0:00:56.169872 ETA: 1:03:39.551267 Epoch: 32/100:, Cur-Step: 2700, loss(cross_entropy): 0.01280, Running average loss:0.01319, Time taken: 0:00:56.169872 ETA: 1:03:39.551267 2022-11-01 11:09:37,831 [INFO] __main__: Epoch: 32/100:, Cur-Step: 2700, loss(cross_entropy): 0.01280, Running average loss:0.01319, Time taken: 0:00:56.169872 ETA: 1:03:39.551267 Epoch: 32/100:, Cur-Step: 2710, loss(cross_entropy): 0.01308, Running average loss:0.01325, Time taken: 0:00:56.169872 ETA: 1:03:39.551267 2022-11-01 11:09:42,524 [INFO] __main__: Epoch: 32/100:, Cur-Step: 2710, loss(cross_entropy): 0.01308, Running average loss:0.01325, Time taken: 0:00:56.169872 ETA: 1:03:39.551267 Epoch: 32/100:, Cur-Step: 2720, loss(cross_entropy): 0.01328, Running average loss:0.01315, Time taken: 0:00:56.169872 ETA: 1:03:39.551267 2022-11-01 11:09:47,196 [INFO] __main__: Epoch: 32/100:, Cur-Step: 2720, loss(cross_entropy): 0.01328, Running average loss:0.01315, Time taken: 0:00:56.169872 ETA: 1:03:39.551267 Epoch: 32/100:, Cur-Step: 2730, loss(cross_entropy): 0.01363, Running average loss:0.01321, Time taken: 0:00:56.169872 ETA: 1:03:39.551267 2022-11-01 11:09:51,856 [INFO] __main__: Epoch: 32/100:, Cur-Step: 2730, loss(cross_entropy): 0.01363, Running average loss:0.01321, Time taken: 0:00:56.169872 ETA: 1:03:39.551267 Epoch: 32/100:, Cur-Step: 2740, loss(cross_entropy): 0.01349, Running average loss:0.01330, Time taken: 0:00:56.169872 ETA: 1:03:39.551267 2022-11-01 11:09:56,596 [INFO] __main__: Epoch: 32/100:, Cur-Step: 2740, loss(cross_entropy): 0.01349, Running average loss:0.01330, Time taken: 0:00:56.169872 ETA: 1:03:39.551267 Epoch: 32/100:, Cur-Step: 2750, loss(cross_entropy): 0.01323, Running average loss:0.01336, Time taken: 0:00:56.169872 ETA: 1:03:39.551267 2022-11-01 11:10:01,390 [INFO] __main__: Epoch: 32/100:, Cur-Step: 2750, loss(cross_entropy): 0.01323, Running average loss:0.01336, Time taken: 0:00:56.169872 ETA: 1:03:39.551267 Epoch: 32/100:, Cur-Step: 2760, loss(cross_entropy): 0.01419, Running average loss:0.01335, Time taken: 0:00:56.169872 ETA: 1:03:39.551267 2022-11-01 11:10:06,061 [INFO] __main__: Epoch: 32/100:, Cur-Step: 2760, loss(cross_entropy): 0.01419, Running average loss:0.01335, Time taken: 0:00:56.169872 ETA: 1:03:39.551267 Epoch: 32/100:, Cur-Step: 2770, loss(cross_entropy): 0.01341, Running average loss:0.01332, Time taken: 0:00:56.169872 ETA: 1:03:39.551267 2022-11-01 11:10:10,883 [INFO] __main__: Epoch: 32/100:, Cur-Step: 2770, loss(cross_entropy): 0.01341, Running average loss:0.01332, Time taken: 0:00:56.169872 ETA: 1:03:39.551267 INFO:tensorflow:Saving checkpoints for step-2772. 2022-11-01 11:10:11,384 [INFO] tensorflow: Saving checkpoints for step-2772. Epoch: 33/100:, Cur-Step: 2780, loss(cross_entropy): 0.01306, Running average loss:0.01300, Time taken: 0:00:50.816477 ETA: 0:56:44.703979 2022-11-01 11:10:26,278 [INFO] __main__: Epoch: 33/100:, Cur-Step: 2780, loss(cross_entropy): 0.01306, Running average loss:0.01300, Time taken: 0:00:50.816477 ETA: 0:56:44.703979 Epoch: 33/100:, Cur-Step: 2790, loss(cross_entropy): 0.01422, Running average loss:0.01315, Time taken: 0:00:50.816477 ETA: 0:56:44.703979 2022-11-01 11:10:31,022 [INFO] __main__: Epoch: 33/100:, Cur-Step: 2790, loss(cross_entropy): 0.01422, Running average loss:0.01315, Time taken: 0:00:50.816477 ETA: 0:56:44.703979 Epoch: 33/100:, Cur-Step: 2800, loss(cross_entropy): 0.01282, Running average loss:0.01320, Time taken: 0:00:50.816477 ETA: 0:56:44.703979 2022-11-01 11:10:35,680 [INFO] __main__: Epoch: 33/100:, Cur-Step: 2800, loss(cross_entropy): 0.01282, Running average loss:0.01320, Time taken: 0:00:50.816477 ETA: 0:56:44.703979 Epoch: 33/100:, Cur-Step: 2810, loss(cross_entropy): 0.01276, Running average loss:0.01312, Time taken: 0:00:50.816477 ETA: 0:56:44.703979 2022-11-01 11:10:40,366 [INFO] __main__: Epoch: 33/100:, Cur-Step: 2810, loss(cross_entropy): 0.01276, Running average loss:0.01312, Time taken: 0:00:50.816477 ETA: 0:56:44.703979 Epoch: 33/100:, Cur-Step: 2820, loss(cross_entropy): 0.01354, Running average loss:0.01313, Time taken: 0:00:50.816477 ETA: 0:56:44.703979 2022-11-01 11:10:45,151 [INFO] __main__: Epoch: 33/100:, Cur-Step: 2820, loss(cross_entropy): 0.01354, Running average loss:0.01313, Time taken: 0:00:50.816477 ETA: 0:56:44.703979 Epoch: 33/100:, Cur-Step: 2830, loss(cross_entropy): 0.01235, Running average loss:0.01310, Time taken: 0:00:50.816477 ETA: 0:56:44.703979 2022-11-01 11:10:49,820 [INFO] __main__: Epoch: 33/100:, Cur-Step: 2830, loss(cross_entropy): 0.01235, Running average loss:0.01310, Time taken: 0:00:50.816477 ETA: 0:56:44.703979 Epoch: 33/100:, Cur-Step: 2840, loss(cross_entropy): 0.01205, Running average loss:0.01307, Time taken: 0:00:50.816477 ETA: 0:56:44.703979 2022-11-01 11:10:54,563 [INFO] __main__: Epoch: 33/100:, Cur-Step: 2840, loss(cross_entropy): 0.01205, Running average loss:0.01307, Time taken: 0:00:50.816477 ETA: 0:56:44.703979 Epoch: 33/100:, Cur-Step: 2850, loss(cross_entropy): 0.01296, Running average loss:0.01302, Time taken: 0:00:50.816477 ETA: 0:56:44.703979 2022-11-01 11:10:59,246 [INFO] __main__: Epoch: 33/100:, Cur-Step: 2850, loss(cross_entropy): 0.01296, Running average loss:0.01302, Time taken: 0:00:50.816477 ETA: 0:56:44.703979 INFO:tensorflow:Saving checkpoints for step-2856. 2022-11-01 11:11:01,595 [INFO] tensorflow: Saving checkpoints for step-2856. Epoch: 34/100:, Cur-Step: 2860, loss(cross_entropy): 0.01235, Running average loss:0.01269, Time taken: 0:00:50.709965 ETA: 0:55:46.857721 2022-11-01 11:11:14,988 [INFO] __main__: Epoch: 34/100:, Cur-Step: 2860, loss(cross_entropy): 0.01235, Running average loss:0.01269, Time taken: 0:00:50.709965 ETA: 0:55:46.857721 Epoch: 34/100:, Cur-Step: 2870, loss(cross_entropy): 0.01228, Running average loss:0.01277, Time taken: 0:00:50.709965 ETA: 0:55:46.857721 2022-11-01 11:11:19,738 [INFO] __main__: Epoch: 34/100:, Cur-Step: 2870, loss(cross_entropy): 0.01228, Running average loss:0.01277, Time taken: 0:00:50.709965 ETA: 0:55:46.857721 Epoch: 34/100:, Cur-Step: 2880, loss(cross_entropy): 0.01200, Running average loss:0.01270, Time taken: 0:00:50.709965 ETA: 0:55:46.857721 2022-11-01 11:11:24,437 [INFO] __main__: Epoch: 34/100:, Cur-Step: 2880, loss(cross_entropy): 0.01200, Running average loss:0.01270, Time taken: 0:00:50.709965 ETA: 0:55:46.857721 Epoch: 34/100:, Cur-Step: 2890, loss(cross_entropy): 0.01203, Running average loss:0.01266, Time taken: 0:00:50.709965 ETA: 0:55:46.857721 2022-11-01 11:11:29,159 [INFO] __main__: Epoch: 34/100:, Cur-Step: 2890, loss(cross_entropy): 0.01203, Running average loss:0.01266, Time taken: 0:00:50.709965 ETA: 0:55:46.857721 Epoch: 34/100:, Cur-Step: 2900, loss(cross_entropy): 0.01268, Running average loss:0.01272, Time taken: 0:00:50.709965 ETA: 0:55:46.857721 2022-11-01 11:11:33,903 [INFO] __main__: Epoch: 34/100:, Cur-Step: 2900, loss(cross_entropy): 0.01268, Running average loss:0.01272, Time taken: 0:00:50.709965 ETA: 0:55:46.857721 Epoch: 34/100:, Cur-Step: 2910, loss(cross_entropy): 0.01275, Running average loss:0.01272, Time taken: 0:00:50.709965 ETA: 0:55:46.857721 2022-11-01 11:11:38,629 [INFO] __main__: Epoch: 34/100:, Cur-Step: 2910, loss(cross_entropy): 0.01275, Running average loss:0.01272, Time taken: 0:00:50.709965 ETA: 0:55:46.857721 Epoch: 34/100:, Cur-Step: 2920, loss(cross_entropy): 0.01338, Running average loss:0.01275, Time taken: 0:00:50.709965 ETA: 0:55:46.857721 2022-11-01 11:11:43,451 [INFO] __main__: Epoch: 34/100:, Cur-Step: 2920, loss(cross_entropy): 0.01338, Running average loss:0.01275, Time taken: 0:00:50.709965 ETA: 0:55:46.857721 Epoch: 34/100:, Cur-Step: 2930, loss(cross_entropy): 0.01316, Running average loss:0.01271, Time taken: 0:00:50.709965 ETA: 0:55:46.857721 2022-11-01 11:11:48,137 [INFO] __main__: Epoch: 34/100:, Cur-Step: 2930, loss(cross_entropy): 0.01316, Running average loss:0.01271, Time taken: 0:00:50.709965 ETA: 0:55:46.857721 INFO:tensorflow:Saving checkpoints for step-2940. 2022-11-01 11:11:52,346 [INFO] tensorflow: Saving checkpoints for step-2940. Epoch: 35/100:, Cur-Step: 2940, loss(cross_entropy): 0.01288, Running average loss:0.01288, Time taken: 0:00:51.221891 ETA: 0:55:29.422910 2022-11-01 11:12:06,200 [INFO] __main__: Epoch: 35/100:, Cur-Step: 2940, loss(cross_entropy): 0.01288, Running average loss:0.01288, Time taken: 0:00:51.221891 ETA: 0:55:29.422910 Epoch: 35/100:, Cur-Step: 2950, loss(cross_entropy): 0.01379, Running average loss:0.01298, Time taken: 0:00:51.221891 ETA: 0:55:29.422910 2022-11-01 11:12:11,534 [INFO] __main__: Epoch: 35/100:, Cur-Step: 2950, loss(cross_entropy): 0.01379, Running average loss:0.01298, Time taken: 0:00:51.221891 ETA: 0:55:29.422910 Epoch: 35/100:, Cur-Step: 2960, loss(cross_entropy): 0.01163, Running average loss:0.01268, Time taken: 0:00:51.221891 ETA: 0:55:29.422910 2022-11-01 11:12:16,239 [INFO] __main__: Epoch: 35/100:, Cur-Step: 2960, loss(cross_entropy): 0.01163, Running average loss:0.01268, Time taken: 0:00:51.221891 ETA: 0:55:29.422910 Epoch: 35/100:, Cur-Step: 2970, loss(cross_entropy): 0.01201, Running average loss:0.01257, Time taken: 0:00:51.221891 ETA: 0:55:29.422910 2022-11-01 11:12:20,903 [INFO] __main__: Epoch: 35/100:, Cur-Step: 2970, loss(cross_entropy): 0.01201, Running average loss:0.01257, Time taken: 0:00:51.221891 ETA: 0:55:29.422910 Epoch: 35/100:, Cur-Step: 2980, loss(cross_entropy): 0.01213, Running average loss:0.01254, Time taken: 0:00:51.221891 ETA: 0:55:29.422910 2022-11-01 11:12:25,581 [INFO] __main__: Epoch: 35/100:, Cur-Step: 2980, loss(cross_entropy): 0.01213, Running average loss:0.01254, Time taken: 0:00:51.221891 ETA: 0:55:29.422910 Epoch: 35/100:, Cur-Step: 2990, loss(cross_entropy): 0.01164, Running average loss:0.01249, Time taken: 0:00:51.221891 ETA: 0:55:29.422910 2022-11-01 11:12:30,324 [INFO] __main__: Epoch: 35/100:, Cur-Step: 2990, loss(cross_entropy): 0.01164, Running average loss:0.01249, Time taken: 0:00:51.221891 ETA: 0:55:29.422910 Epoch: 35/100:, Cur-Step: 3000, loss(cross_entropy): 0.01186, Running average loss:0.01244, Time taken: 0:00:51.221891 ETA: 0:55:29.422910 2022-11-01 11:12:35,111 [INFO] __main__: Epoch: 35/100:, Cur-Step: 3000, loss(cross_entropy): 0.01186, Running average loss:0.01244, Time taken: 0:00:51.221891 ETA: 0:55:29.422910 Epoch: 35/100:, Cur-Step: 3010, loss(cross_entropy): 0.01220, Running average loss:0.01243, Time taken: 0:00:51.221891 ETA: 0:55:29.422910 2022-11-01 11:12:39,730 [INFO] __main__: Epoch: 35/100:, Cur-Step: 3010, loss(cross_entropy): 0.01220, Running average loss:0.01243, Time taken: 0:00:51.221891 ETA: 0:55:29.422910 Epoch: 35/100:, Cur-Step: 3020, loss(cross_entropy): 0.01280, Running average loss:0.01241, Time taken: 0:00:51.221891 ETA: 0:55:29.422910 2022-11-01 11:12:44,385 [INFO] __main__: Epoch: 35/100:, Cur-Step: 3020, loss(cross_entropy): 0.01280, Running average loss:0.01241, Time taken: 0:00:51.221891 ETA: 0:55:29.422910 INFO:tensorflow:Saving checkpoints for step-3024. 2022-11-01 11:12:45,773 [INFO] tensorflow: Saving checkpoints for step-3024. Epoch: 36/100:, Cur-Step: 3030, loss(cross_entropy): 0.01153, Running average loss:0.01214, Time taken: 0:00:53.891170 ETA: 0:57:29.034882 2022-11-01 11:12:59,964 [INFO] __main__: Epoch: 36/100:, Cur-Step: 3030, loss(cross_entropy): 0.01153, Running average loss:0.01214, Time taken: 0:00:53.891170 ETA: 0:57:29.034882 Epoch: 36/100:, Cur-Step: 3040, loss(cross_entropy): 0.01263, Running average loss:0.01220, Time taken: 0:00:53.891170 ETA: 0:57:29.034882 2022-11-01 11:13:05,345 [INFO] __main__: Epoch: 36/100:, Cur-Step: 3040, loss(cross_entropy): 0.01263, Running average loss:0.01220, Time taken: 0:00:53.891170 ETA: 0:57:29.034882 Epoch: 36/100:, Cur-Step: 3050, loss(cross_entropy): 0.01153, Running average loss:0.01218, Time taken: 0:00:53.891170 ETA: 0:57:29.034882 2022-11-01 11:13:10,337 [INFO] __main__: Epoch: 36/100:, Cur-Step: 3050, loss(cross_entropy): 0.01153, Running average loss:0.01218, Time taken: 0:00:53.891170 ETA: 0:57:29.034882 Epoch: 36/100:, Cur-Step: 3060, loss(cross_entropy): 0.01141, Running average loss:0.01216, Time taken: 0:00:53.891170 ETA: 0:57:29.034882 2022-11-01 11:13:15,057 [INFO] __main__: Epoch: 36/100:, Cur-Step: 3060, loss(cross_entropy): 0.01141, Running average loss:0.01216, Time taken: 0:00:53.891170 ETA: 0:57:29.034882 Epoch: 36/100:, Cur-Step: 3070, loss(cross_entropy): 0.01147, Running average loss:0.01207, Time taken: 0:00:53.891170 ETA: 0:57:29.034882 2022-11-01 11:13:19,719 [INFO] __main__: Epoch: 36/100:, Cur-Step: 3070, loss(cross_entropy): 0.01147, Running average loss:0.01207, Time taken: 0:00:53.891170 ETA: 0:57:29.034882 Epoch: 36/100:, Cur-Step: 3080, loss(cross_entropy): 0.01163, Running average loss:0.01201, Time taken: 0:00:53.891170 ETA: 0:57:29.034882 2022-11-01 11:13:24,355 [INFO] __main__: Epoch: 36/100:, Cur-Step: 3080, loss(cross_entropy): 0.01163, Running average loss:0.01201, Time taken: 0:00:53.891170 ETA: 0:57:29.034882 Epoch: 36/100:, Cur-Step: 3090, loss(cross_entropy): 0.01147, Running average loss:0.01197, Time taken: 0:00:53.891170 ETA: 0:57:29.034882 2022-11-01 11:13:29,056 [INFO] __main__: Epoch: 36/100:, Cur-Step: 3090, loss(cross_entropy): 0.01147, Running average loss:0.01197, Time taken: 0:00:53.891170 ETA: 0:57:29.034882 Epoch: 36/100:, Cur-Step: 3100, loss(cross_entropy): 0.01092, Running average loss:0.01192, Time taken: 0:00:53.891170 ETA: 0:57:29.034882 2022-11-01 11:13:33,864 [INFO] __main__: Epoch: 36/100:, Cur-Step: 3100, loss(cross_entropy): 0.01092, Running average loss:0.01192, Time taken: 0:00:53.891170 ETA: 0:57:29.034882 INFO:tensorflow:Saving checkpoints for step-3108. 2022-11-01 11:13:37,157 [INFO] tensorflow: Saving checkpoints for step-3108. Epoch: 37/100:, Cur-Step: 3110, loss(cross_entropy): 0.01160, Running average loss:0.01120, Time taken: 0:00:51.844479 ETA: 0:54:26.202197 2022-11-01 11:13:49,971 [INFO] __main__: Epoch: 37/100:, Cur-Step: 3110, loss(cross_entropy): 0.01160, Running average loss:0.01120, Time taken: 0:00:51.844479 ETA: 0:54:26.202197 Epoch: 37/100:, Cur-Step: 3120, loss(cross_entropy): 0.01119, Running average loss:0.01173, Time taken: 0:00:51.844479 ETA: 0:54:26.202197 2022-11-01 11:13:55,286 [INFO] __main__: Epoch: 37/100:, Cur-Step: 3120, loss(cross_entropy): 0.01119, Running average loss:0.01173, Time taken: 0:00:51.844479 ETA: 0:54:26.202197 Epoch: 37/100:, Cur-Step: 3130, loss(cross_entropy): 0.01244, Running average loss:0.01175, Time taken: 0:00:51.844479 ETA: 0:54:26.202197 2022-11-01 11:14:00,532 [INFO] __main__: Epoch: 37/100:, Cur-Step: 3130, loss(cross_entropy): 0.01244, Running average loss:0.01175, Time taken: 0:00:51.844479 ETA: 0:54:26.202197 Epoch: 37/100:, Cur-Step: 3140, loss(cross_entropy): 0.01237, Running average loss:0.01187, Time taken: 0:00:51.844479 ETA: 0:54:26.202197 2022-11-01 11:14:05,291 [INFO] __main__: Epoch: 37/100:, Cur-Step: 3140, loss(cross_entropy): 0.01237, Running average loss:0.01187, Time taken: 0:00:51.844479 ETA: 0:54:26.202197 Epoch: 37/100:, Cur-Step: 3150, loss(cross_entropy): 0.01191, Running average loss:0.01184, Time taken: 0:00:51.844479 ETA: 0:54:26.202197 2022-11-01 11:14:09,988 [INFO] __main__: Epoch: 37/100:, Cur-Step: 3150, loss(cross_entropy): 0.01191, Running average loss:0.01184, Time taken: 0:00:51.844479 ETA: 0:54:26.202197 Epoch: 37/100:, Cur-Step: 3160, loss(cross_entropy): 0.01109, Running average loss:0.01177, Time taken: 0:00:51.844479 ETA: 0:54:26.202197 2022-11-01 11:14:14,652 [INFO] __main__: Epoch: 37/100:, Cur-Step: 3160, loss(cross_entropy): 0.01109, Running average loss:0.01177, Time taken: 0:00:51.844479 ETA: 0:54:26.202197 Epoch: 37/100:, Cur-Step: 3170, loss(cross_entropy): 0.01238, Running average loss:0.01181, Time taken: 0:00:51.844479 ETA: 0:54:26.202197 2022-11-01 11:14:19,444 [INFO] __main__: Epoch: 37/100:, Cur-Step: 3170, loss(cross_entropy): 0.01238, Running average loss:0.01181, Time taken: 0:00:51.844479 ETA: 0:54:26.202197 Epoch: 37/100:, Cur-Step: 3180, loss(cross_entropy): 0.01219, Running average loss:0.01178, Time taken: 0:00:51.844479 ETA: 0:54:26.202197 2022-11-01 11:14:24,137 [INFO] __main__: Epoch: 37/100:, Cur-Step: 3180, loss(cross_entropy): 0.01219, Running average loss:0.01178, Time taken: 0:00:51.844479 ETA: 0:54:26.202197 Epoch: 37/100:, Cur-Step: 3190, loss(cross_entropy): 0.01117, Running average loss:0.01171, Time taken: 0:00:51.844479 ETA: 0:54:26.202197 2022-11-01 11:14:28,821 [INFO] __main__: Epoch: 37/100:, Cur-Step: 3190, loss(cross_entropy): 0.01117, Running average loss:0.01171, Time taken: 0:00:51.844479 ETA: 0:54:26.202197 INFO:tensorflow:Saving checkpoints for step-3192. 2022-11-01 11:14:29,287 [INFO] tensorflow: Saving checkpoints for step-3192. Epoch: 38/100:, Cur-Step: 3200, loss(cross_entropy): 0.01144, Running average loss:0.01177, Time taken: 0:00:52.595181 ETA: 0:54:20.901192 2022-11-01 11:14:45,281 [INFO] __main__: Epoch: 38/100:, Cur-Step: 3200, loss(cross_entropy): 0.01144, Running average loss:0.01177, Time taken: 0:00:52.595181 ETA: 0:54:20.901192 Epoch: 38/100:, Cur-Step: 3210, loss(cross_entropy): 0.01224, Running average loss:0.01192, Time taken: 0:00:52.595181 ETA: 0:54:20.901192 2022-11-01 11:14:49,932 [INFO] __main__: Epoch: 38/100:, Cur-Step: 3210, loss(cross_entropy): 0.01224, Running average loss:0.01192, Time taken: 0:00:52.595181 ETA: 0:54:20.901192 Epoch: 38/100:, Cur-Step: 3220, loss(cross_entropy): 0.01122, Running average loss:0.01175, Time taken: 0:00:52.595181 ETA: 0:54:20.901192 2022-11-01 11:14:54,608 [INFO] __main__: Epoch: 38/100:, Cur-Step: 3220, loss(cross_entropy): 0.01122, Running average loss:0.01175, Time taken: 0:00:52.595181 ETA: 0:54:20.901192 Epoch: 38/100:, Cur-Step: 3230, loss(cross_entropy): 0.01122, Running average loss:0.01177, Time taken: 0:00:52.595181 ETA: 0:54:20.901192 2022-11-01 11:14:59,290 [INFO] __main__: Epoch: 38/100:, Cur-Step: 3230, loss(cross_entropy): 0.01122, Running average loss:0.01177, Time taken: 0:00:52.595181 ETA: 0:54:20.901192 Epoch: 38/100:, Cur-Step: 3240, loss(cross_entropy): 0.01139, Running average loss:0.01170, Time taken: 0:00:52.595181 ETA: 0:54:20.901192 2022-11-01 11:15:03,996 [INFO] __main__: Epoch: 38/100:, Cur-Step: 3240, loss(cross_entropy): 0.01139, Running average loss:0.01170, Time taken: 0:00:52.595181 ETA: 0:54:20.901192 Epoch: 38/100:, Cur-Step: 3250, loss(cross_entropy): 0.01047, Running average loss:0.01163, Time taken: 0:00:52.595181 ETA: 0:54:20.901192 2022-11-01 11:15:08,744 [INFO] __main__: Epoch: 38/100:, Cur-Step: 3250, loss(cross_entropy): 0.01047, Running average loss:0.01163, Time taken: 0:00:52.595181 ETA: 0:54:20.901192 Epoch: 38/100:, Cur-Step: 3260, loss(cross_entropy): 0.01139, Running average loss:0.01163, Time taken: 0:00:52.595181 ETA: 0:54:20.901192 2022-11-01 11:15:13,482 [INFO] __main__: Epoch: 38/100:, Cur-Step: 3260, loss(cross_entropy): 0.01139, Running average loss:0.01163, Time taken: 0:00:52.595181 ETA: 0:54:20.901192 Epoch: 38/100:, Cur-Step: 3270, loss(cross_entropy): 0.01111, Running average loss:0.01158, Time taken: 0:00:52.595181 ETA: 0:54:20.901192 2022-11-01 11:15:18,755 [INFO] __main__: Epoch: 38/100:, Cur-Step: 3270, loss(cross_entropy): 0.01111, Running average loss:0.01158, Time taken: 0:00:52.595181 ETA: 0:54:20.901192 INFO:tensorflow:Saving checkpoints for step-3276. 2022-11-01 11:15:21,363 [INFO] tensorflow: Saving checkpoints for step-3276. Epoch: 39/100:, Cur-Step: 3280, loss(cross_entropy): 0.01079, Running average loss:0.01118, Time taken: 0:00:52.536367 ETA: 0:53:24.718383 2022-11-01 11:15:37,086 [INFO] __main__: Epoch: 39/100:, Cur-Step: 3280, loss(cross_entropy): 0.01079, Running average loss:0.01118, Time taken: 0:00:52.536367 ETA: 0:53:24.718383 Epoch: 39/100:, Cur-Step: 3290, loss(cross_entropy): 0.01084, Running average loss:0.01102, Time taken: 0:00:52.536367 ETA: 0:53:24.718383 2022-11-01 11:15:41,960 [INFO] __main__: Epoch: 39/100:, Cur-Step: 3290, loss(cross_entropy): 0.01084, Running average loss:0.01102, Time taken: 0:00:52.536367 ETA: 0:53:24.718383 Epoch: 39/100:, Cur-Step: 3300, loss(cross_entropy): 0.01055, Running average loss:0.01125, Time taken: 0:00:52.536367 ETA: 0:53:24.718383 2022-11-01 11:15:46,699 [INFO] __main__: Epoch: 39/100:, Cur-Step: 3300, loss(cross_entropy): 0.01055, Running average loss:0.01125, Time taken: 0:00:52.536367 ETA: 0:53:24.718383 Epoch: 39/100:, Cur-Step: 3310, loss(cross_entropy): 0.01222, Running average loss:0.01129, Time taken: 0:00:52.536367 ETA: 0:53:24.718383 2022-11-01 11:15:51,440 [INFO] __main__: Epoch: 39/100:, Cur-Step: 3310, loss(cross_entropy): 0.01222, Running average loss:0.01129, Time taken: 0:00:52.536367 ETA: 0:53:24.718383 Epoch: 39/100:, Cur-Step: 3320, loss(cross_entropy): 0.01034, Running average loss:0.01120, Time taken: 0:00:52.536367 ETA: 0:53:24.718383 2022-11-01 11:15:56,173 [INFO] __main__: Epoch: 39/100:, Cur-Step: 3320, loss(cross_entropy): 0.01034, Running average loss:0.01120, Time taken: 0:00:52.536367 ETA: 0:53:24.718383 Epoch: 39/100:, Cur-Step: 3330, loss(cross_entropy): 0.01175, Running average loss:0.01120, Time taken: 0:00:52.536367 ETA: 0:53:24.718383 2022-11-01 11:16:00,845 [INFO] __main__: Epoch: 39/100:, Cur-Step: 3330, loss(cross_entropy): 0.01175, Running average loss:0.01120, Time taken: 0:00:52.536367 ETA: 0:53:24.718383 Epoch: 39/100:, Cur-Step: 3340, loss(cross_entropy): 0.01018, Running average loss:0.01114, Time taken: 0:00:52.536367 ETA: 0:53:24.718383 2022-11-01 11:16:05,608 [INFO] __main__: Epoch: 39/100:, Cur-Step: 3340, loss(cross_entropy): 0.01018, Running average loss:0.01114, Time taken: 0:00:52.536367 ETA: 0:53:24.718383 Epoch: 39/100:, Cur-Step: 3350, loss(cross_entropy): 0.01158, Running average loss:0.01112, Time taken: 0:00:52.536367 ETA: 0:53:24.718383 2022-11-01 11:16:10,531 [INFO] __main__: Epoch: 39/100:, Cur-Step: 3350, loss(cross_entropy): 0.01158, Running average loss:0.01112, Time taken: 0:00:52.536367 ETA: 0:53:24.718383 INFO:tensorflow:Saving checkpoints for step-3360. 2022-11-01 11:16:14,789 [INFO] tensorflow: Saving checkpoints for step-3360. Epoch: 40/100:, Cur-Step: 3360, loss(cross_entropy): 0.01135, Running average loss:0.01135, Time taken: 0:00:53.988822 ETA: 0:53:59.329348 2022-11-01 11:16:26,634 [INFO] __main__: Epoch: 40/100:, Cur-Step: 3360, loss(cross_entropy): 0.01135, Running average loss:0.01135, Time taken: 0:00:53.988822 ETA: 0:53:59.329348 Epoch: 40/100:, Cur-Step: 3370, loss(cross_entropy): 0.01056, Running average loss:0.01119, Time taken: 0:00:53.988822 ETA: 0:53:59.329348 2022-11-01 11:16:31,506 [INFO] __main__: Epoch: 40/100:, Cur-Step: 3370, loss(cross_entropy): 0.01056, Running average loss:0.01119, Time taken: 0:00:53.988822 ETA: 0:53:59.329348 Epoch: 40/100:, Cur-Step: 3380, loss(cross_entropy): 0.01028, Running average loss:0.01110, Time taken: 0:00:53.988822 ETA: 0:53:59.329348 2022-11-01 11:16:36,283 [INFO] __main__: Epoch: 40/100:, Cur-Step: 3380, loss(cross_entropy): 0.01028, Running average loss:0.01110, Time taken: 0:00:53.988822 ETA: 0:53:59.329348 Epoch: 40/100:, Cur-Step: 3390, loss(cross_entropy): 0.01119, Running average loss:0.01112, Time taken: 0:00:53.988822 ETA: 0:53:59.329348 2022-11-01 11:16:40,942 [INFO] __main__: Epoch: 40/100:, Cur-Step: 3390, loss(cross_entropy): 0.01119, Running average loss:0.01112, Time taken: 0:00:53.988822 ETA: 0:53:59.329348 Epoch: 40/100:, Cur-Step: 3400, loss(cross_entropy): 0.01079, Running average loss:0.01107, Time taken: 0:00:53.988822 ETA: 0:53:59.329348 2022-11-01 11:16:45,679 [INFO] __main__: Epoch: 40/100:, Cur-Step: 3400, loss(cross_entropy): 0.01079, Running average loss:0.01107, Time taken: 0:00:53.988822 ETA: 0:53:59.329348 Epoch: 40/100:, Cur-Step: 3410, loss(cross_entropy): 0.01081, Running average loss:0.01107, Time taken: 0:00:53.988822 ETA: 0:53:59.329348 2022-11-01 11:16:50,366 [INFO] __main__: Epoch: 40/100:, Cur-Step: 3410, loss(cross_entropy): 0.01081, Running average loss:0.01107, Time taken: 0:00:53.988822 ETA: 0:53:59.329348 Epoch: 40/100:, Cur-Step: 3420, loss(cross_entropy): 0.01138, Running average loss:0.01104, Time taken: 0:00:53.988822 ETA: 0:53:59.329348 2022-11-01 11:16:55,148 [INFO] __main__: Epoch: 40/100:, Cur-Step: 3420, loss(cross_entropy): 0.01138, Running average loss:0.01104, Time taken: 0:00:53.988822 ETA: 0:53:59.329348 Epoch: 40/100:, Cur-Step: 3430, loss(cross_entropy): 0.00981, Running average loss:0.01099, Time taken: 0:00:53.988822 ETA: 0:53:59.329348 2022-11-01 11:16:59,866 [INFO] __main__: Epoch: 40/100:, Cur-Step: 3430, loss(cross_entropy): 0.00981, Running average loss:0.01099, Time taken: 0:00:53.988822 ETA: 0:53:59.329348 Epoch: 40/100:, Cur-Step: 3440, loss(cross_entropy): 0.01068, Running average loss:0.01097, Time taken: 0:00:53.988822 ETA: 0:53:59.329348 2022-11-01 11:17:04,732 [INFO] __main__: Epoch: 40/100:, Cur-Step: 3440, loss(cross_entropy): 0.01068, Running average loss:0.01097, Time taken: 0:00:53.988822 ETA: 0:53:59.329348 INFO:tensorflow:Saving checkpoints for step-3444. 2022-11-01 11:17:06,155 [INFO] tensorflow: Saving checkpoints for step-3444. Epoch: 41/100:, Cur-Step: 3450, loss(cross_entropy): 0.01023, Running average loss:0.01058, Time taken: 0:00:51.846478 ETA: 0:50:58.942201 2022-11-01 11:17:20,544 [INFO] __main__: Epoch: 41/100:, Cur-Step: 3450, loss(cross_entropy): 0.01023, Running average loss:0.01058, Time taken: 0:00:51.846478 ETA: 0:50:58.942201 Epoch: 41/100:, Cur-Step: 3460, loss(cross_entropy): 0.01041, Running average loss:0.01073, Time taken: 0:00:51.846478 ETA: 0:50:58.942201 2022-11-01 11:17:25,237 [INFO] __main__: Epoch: 41/100:, Cur-Step: 3460, loss(cross_entropy): 0.01041, Running average loss:0.01073, Time taken: 0:00:51.846478 ETA: 0:50:58.942201 Epoch: 41/100:, Cur-Step: 3470, loss(cross_entropy): 0.01114, Running average loss:0.01069, Time taken: 0:00:51.846478 ETA: 0:50:58.942201 2022-11-01 11:17:29,930 [INFO] __main__: Epoch: 41/100:, Cur-Step: 3470, loss(cross_entropy): 0.01114, Running average loss:0.01069, Time taken: 0:00:51.846478 ETA: 0:50:58.942201 Epoch: 41/100:, Cur-Step: 3480, loss(cross_entropy): 0.01075, Running average loss:0.01069, Time taken: 0:00:51.846478 ETA: 0:50:58.942201 2022-11-01 11:17:34,617 [INFO] __main__: Epoch: 41/100:, Cur-Step: 3480, loss(cross_entropy): 0.01075, Running average loss:0.01069, Time taken: 0:00:51.846478 ETA: 0:50:58.942201 Epoch: 41/100:, Cur-Step: 3490, loss(cross_entropy): 0.01147, Running average loss:0.01063, Time taken: 0:00:51.846478 ETA: 0:50:58.942201 2022-11-01 11:17:39,324 [INFO] __main__: Epoch: 41/100:, Cur-Step: 3490, loss(cross_entropy): 0.01147, Running average loss:0.01063, Time taken: 0:00:51.846478 ETA: 0:50:58.942201 Epoch: 41/100:, Cur-Step: 3500, loss(cross_entropy): 0.01080, Running average loss:0.01058, Time taken: 0:00:51.846478 ETA: 0:50:58.942201 2022-11-01 11:17:44,092 [INFO] __main__: Epoch: 41/100:, Cur-Step: 3500, loss(cross_entropy): 0.01080, Running average loss:0.01058, Time taken: 0:00:51.846478 ETA: 0:50:58.942201 Epoch: 41/100:, Cur-Step: 3510, loss(cross_entropy): 0.01018, Running average loss:0.01054, Time taken: 0:00:51.846478 ETA: 0:50:58.942201 2022-11-01 11:17:48,895 [INFO] __main__: Epoch: 41/100:, Cur-Step: 3510, loss(cross_entropy): 0.01018, Running average loss:0.01054, Time taken: 0:00:51.846478 ETA: 0:50:58.942201 Epoch: 41/100:, Cur-Step: 3520, loss(cross_entropy): 0.01031, Running average loss:0.01056, Time taken: 0:00:51.846478 ETA: 0:50:58.942201 2022-11-01 11:17:53,606 [INFO] __main__: Epoch: 41/100:, Cur-Step: 3520, loss(cross_entropy): 0.01031, Running average loss:0.01056, Time taken: 0:00:51.846478 ETA: 0:50:58.942201 INFO:tensorflow:Saving checkpoints for step-3528. 2022-11-01 11:17:56,925 [INFO] tensorflow: Saving checkpoints for step-3528. Epoch: 42/100:, Cur-Step: 3530, loss(cross_entropy): 0.01080, Running average loss:0.01117, Time taken: 0:00:51.254387 ETA: 0:49:32.754468 2022-11-01 11:18:09,217 [INFO] __main__: Epoch: 42/100:, Cur-Step: 3530, loss(cross_entropy): 0.01080, Running average loss:0.01117, Time taken: 0:00:51.254387 ETA: 0:49:32.754468 Epoch: 42/100:, Cur-Step: 3540, loss(cross_entropy): 0.00970, Running average loss:0.01059, Time taken: 0:00:51.254387 ETA: 0:49:32.754468 2022-11-01 11:18:13,943 [INFO] __main__: Epoch: 42/100:, Cur-Step: 3540, loss(cross_entropy): 0.00970, Running average loss:0.01059, Time taken: 0:00:51.254387 ETA: 0:49:32.754468 Epoch: 42/100:, Cur-Step: 3550, loss(cross_entropy): 0.01050, Running average loss:0.01068, Time taken: 0:00:51.254387 ETA: 0:49:32.754468 2022-11-01 11:18:18,638 [INFO] __main__: Epoch: 42/100:, Cur-Step: 3550, loss(cross_entropy): 0.01050, Running average loss:0.01068, Time taken: 0:00:51.254387 ETA: 0:49:32.754468 Epoch: 42/100:, Cur-Step: 3560, loss(cross_entropy): 0.01064, Running average loss:0.01064, Time taken: 0:00:51.254387 ETA: 0:49:32.754468 2022-11-01 11:18:23,303 [INFO] __main__: Epoch: 42/100:, Cur-Step: 3560, loss(cross_entropy): 0.01064, Running average loss:0.01064, Time taken: 0:00:51.254387 ETA: 0:49:32.754468 Epoch: 42/100:, Cur-Step: 3570, loss(cross_entropy): 0.01047, Running average loss:0.01061, Time taken: 0:00:51.254387 ETA: 0:49:32.754468 2022-11-01 11:18:27,986 [INFO] __main__: Epoch: 42/100:, Cur-Step: 3570, loss(cross_entropy): 0.01047, Running average loss:0.01061, Time taken: 0:00:51.254387 ETA: 0:49:32.754468 Epoch: 42/100:, Cur-Step: 3580, loss(cross_entropy): 0.00972, Running average loss:0.01050, Time taken: 0:00:51.254387 ETA: 0:49:32.754468 2022-11-01 11:18:32,642 [INFO] __main__: Epoch: 42/100:, Cur-Step: 3580, loss(cross_entropy): 0.00972, Running average loss:0.01050, Time taken: 0:00:51.254387 ETA: 0:49:32.754468 Epoch: 42/100:, Cur-Step: 3590, loss(cross_entropy): 0.01020, Running average loss:0.01047, Time taken: 0:00:51.254387 ETA: 0:49:32.754468 2022-11-01 11:18:37,298 [INFO] __main__: Epoch: 42/100:, Cur-Step: 3590, loss(cross_entropy): 0.01020, Running average loss:0.01047, Time taken: 0:00:51.254387 ETA: 0:49:32.754468 Epoch: 42/100:, Cur-Step: 3600, loss(cross_entropy): 0.01041, Running average loss:0.01048, Time taken: 0:00:51.254387 ETA: 0:49:32.754468 2022-11-01 11:18:42,001 [INFO] __main__: Epoch: 42/100:, Cur-Step: 3600, loss(cross_entropy): 0.01041, Running average loss:0.01048, Time taken: 0:00:51.254387 ETA: 0:49:32.754468 Epoch: 42/100:, Cur-Step: 3610, loss(cross_entropy): 0.00959, Running average loss:0.01041, Time taken: 0:00:51.254387 ETA: 0:49:32.754468 2022-11-01 11:18:46,916 [INFO] __main__: Epoch: 42/100:, Cur-Step: 3610, loss(cross_entropy): 0.00959, Running average loss:0.01041, Time taken: 0:00:51.254387 ETA: 0:49:32.754468 INFO:tensorflow:Saving checkpoints for step-3612. 2022-11-01 11:18:47,383 [INFO] tensorflow: Saving checkpoints for step-3612. Epoch: 43/100:, Cur-Step: 3620, loss(cross_entropy): 0.00967, Running average loss:0.00989, Time taken: 0:00:50.927217 ETA: 0:48:22.851369 2022-11-01 11:19:02,807 [INFO] __main__: Epoch: 43/100:, Cur-Step: 3620, loss(cross_entropy): 0.00967, Running average loss:0.00989, Time taken: 0:00:50.927217 ETA: 0:48:22.851369 Epoch: 43/100:, Cur-Step: 3630, loss(cross_entropy): 0.01140, Running average loss:0.00999, Time taken: 0:00:50.927217 ETA: 0:48:22.851369 2022-11-01 11:19:07,529 [INFO] __main__: Epoch: 43/100:, Cur-Step: 3630, loss(cross_entropy): 0.01140, Running average loss:0.00999, Time taken: 0:00:50.927217 ETA: 0:48:22.851369 Epoch: 43/100:, Cur-Step: 3640, loss(cross_entropy): 0.01017, Running average loss:0.00996, Time taken: 0:00:50.927217 ETA: 0:48:22.851369 2022-11-01 11:19:12,214 [INFO] __main__: Epoch: 43/100:, Cur-Step: 3640, loss(cross_entropy): 0.01017, Running average loss:0.00996, Time taken: 0:00:50.927217 ETA: 0:48:22.851369 Epoch: 43/100:, Cur-Step: 3650, loss(cross_entropy): 0.00963, Running average loss:0.00997, Time taken: 0:00:50.927217 ETA: 0:48:22.851369 2022-11-01 11:19:16,914 [INFO] __main__: Epoch: 43/100:, Cur-Step: 3650, loss(cross_entropy): 0.00963, Running average loss:0.00997, Time taken: 0:00:50.927217 ETA: 0:48:22.851369 Epoch: 43/100:, Cur-Step: 3660, loss(cross_entropy): 0.00970, Running average loss:0.00997, Time taken: 0:00:50.927217 ETA: 0:48:22.851369 2022-11-01 11:19:21,624 [INFO] __main__: Epoch: 43/100:, Cur-Step: 3660, loss(cross_entropy): 0.00970, Running average loss:0.00997, Time taken: 0:00:50.927217 ETA: 0:48:22.851369 Epoch: 43/100:, Cur-Step: 3670, loss(cross_entropy): 0.01040, Running average loss:0.01001, Time taken: 0:00:50.927217 ETA: 0:48:22.851369 2022-11-01 11:19:26,588 [INFO] __main__: Epoch: 43/100:, Cur-Step: 3670, loss(cross_entropy): 0.01040, Running average loss:0.01001, Time taken: 0:00:50.927217 ETA: 0:48:22.851369 Epoch: 43/100:, Cur-Step: 3680, loss(cross_entropy): 0.00977, Running average loss:0.01003, Time taken: 0:00:50.927217 ETA: 0:48:22.851369 2022-11-01 11:19:31,425 [INFO] __main__: Epoch: 43/100:, Cur-Step: 3680, loss(cross_entropy): 0.00977, Running average loss:0.01003, Time taken: 0:00:50.927217 ETA: 0:48:22.851369 Epoch: 43/100:, Cur-Step: 3690, loss(cross_entropy): 0.00970, Running average loss:0.01003, Time taken: 0:00:50.927217 ETA: 0:48:22.851369 2022-11-01 11:19:36,181 [INFO] __main__: Epoch: 43/100:, Cur-Step: 3690, loss(cross_entropy): 0.00970, Running average loss:0.01003, Time taken: 0:00:50.927217 ETA: 0:48:22.851369 INFO:tensorflow:Saving checkpoints for step-3696. 2022-11-01 11:19:38,543 [INFO] tensorflow: Saving checkpoints for step-3696. Epoch: 44/100:, Cur-Step: 3700, loss(cross_entropy): 0.00882, Running average loss:0.00988, Time taken: 0:00:51.625292 ETA: 0:48:11.016369 2022-11-01 11:19:52,179 [INFO] __main__: Epoch: 44/100:, Cur-Step: 3700, loss(cross_entropy): 0.00882, Running average loss:0.00988, Time taken: 0:00:51.625292 ETA: 0:48:11.016369 Epoch: 44/100:, Cur-Step: 3710, loss(cross_entropy): 0.00971, Running average loss:0.00971, Time taken: 0:00:51.625292 ETA: 0:48:11.016369 2022-11-01 11:19:57,081 [INFO] __main__: Epoch: 44/100:, Cur-Step: 3710, loss(cross_entropy): 0.00971, Running average loss:0.00971, Time taken: 0:00:51.625292 ETA: 0:48:11.016369 Epoch: 44/100:, Cur-Step: 3720, loss(cross_entropy): 0.01017, Running average loss:0.00988, Time taken: 0:00:51.625292 ETA: 0:48:11.016369 2022-11-01 11:20:01,853 [INFO] __main__: Epoch: 44/100:, Cur-Step: 3720, loss(cross_entropy): 0.01017, Running average loss:0.00988, Time taken: 0:00:51.625292 ETA: 0:48:11.016369 Epoch: 44/100:, Cur-Step: 3730, loss(cross_entropy): 0.00947, Running average loss:0.00992, Time taken: 0:00:51.625292 ETA: 0:48:11.016369 2022-11-01 11:20:06,629 [INFO] __main__: Epoch: 44/100:, Cur-Step: 3730, loss(cross_entropy): 0.00947, Running average loss:0.00992, Time taken: 0:00:51.625292 ETA: 0:48:11.016369 Epoch: 44/100:, Cur-Step: 3740, loss(cross_entropy): 0.00933, Running average loss:0.00989, Time taken: 0:00:51.625292 ETA: 0:48:11.016369 2022-11-01 11:20:11,294 [INFO] __main__: Epoch: 44/100:, Cur-Step: 3740, loss(cross_entropy): 0.00933, Running average loss:0.00989, Time taken: 0:00:51.625292 ETA: 0:48:11.016369 Epoch: 44/100:, Cur-Step: 3750, loss(cross_entropy): 0.00999, Running average loss:0.00986, Time taken: 0:00:51.625292 ETA: 0:48:11.016369 2022-11-01 11:20:15,967 [INFO] __main__: Epoch: 44/100:, Cur-Step: 3750, loss(cross_entropy): 0.00999, Running average loss:0.00986, Time taken: 0:00:51.625292 ETA: 0:48:11.016369 Epoch: 44/100:, Cur-Step: 3760, loss(cross_entropy): 0.00938, Running average loss:0.00989, Time taken: 0:00:51.625292 ETA: 0:48:11.016369 2022-11-01 11:20:20,662 [INFO] __main__: Epoch: 44/100:, Cur-Step: 3760, loss(cross_entropy): 0.00938, Running average loss:0.00989, Time taken: 0:00:51.625292 ETA: 0:48:11.016369 Epoch: 44/100:, Cur-Step: 3770, loss(cross_entropy): 0.01085, Running average loss:0.00988, Time taken: 0:00:51.625292 ETA: 0:48:11.016369 2022-11-01 11:20:25,581 [INFO] __main__: Epoch: 44/100:, Cur-Step: 3770, loss(cross_entropy): 0.01085, Running average loss:0.00988, Time taken: 0:00:51.625292 ETA: 0:48:11.016369 INFO:tensorflow:Saving checkpoints for step-3780. 2022-11-01 11:20:29,779 [INFO] tensorflow: Saving checkpoints for step-3780. Epoch: 45/100:, Cur-Step: 3780, loss(cross_entropy): 0.00906, Running average loss:0.00906, Time taken: 0:00:51.703506 ETA: 0:47:23.692803 2022-11-01 11:20:41,473 [INFO] __main__: Epoch: 45/100:, Cur-Step: 3780, loss(cross_entropy): 0.00906, Running average loss:0.00906, Time taken: 0:00:51.703506 ETA: 0:47:23.692803 Epoch: 45/100:, Cur-Step: 3790, loss(cross_entropy): 0.00953, Running average loss:0.00951, Time taken: 0:00:51.703506 ETA: 0:47:23.692803 2022-11-01 11:20:46,547 [INFO] __main__: Epoch: 45/100:, Cur-Step: 3790, loss(cross_entropy): 0.00953, Running average loss:0.00951, Time taken: 0:00:51.703506 ETA: 0:47:23.692803 Epoch: 45/100:, Cur-Step: 3800, loss(cross_entropy): 0.00906, Running average loss:0.00955, Time taken: 0:00:51.703506 ETA: 0:47:23.692803 2022-11-01 11:20:51,255 [INFO] __main__: Epoch: 45/100:, Cur-Step: 3800, loss(cross_entropy): 0.00906, Running average loss:0.00955, Time taken: 0:00:51.703506 ETA: 0:47:23.692803 Epoch: 45/100:, Cur-Step: 3810, loss(cross_entropy): 0.01001, Running average loss:0.00959, Time taken: 0:00:51.703506 ETA: 0:47:23.692803 2022-11-01 11:20:55,889 [INFO] __main__: Epoch: 45/100:, Cur-Step: 3810, loss(cross_entropy): 0.01001, Running average loss:0.00959, Time taken: 0:00:51.703506 ETA: 0:47:23.692803 Epoch: 45/100:, Cur-Step: 3820, loss(cross_entropy): 0.00980, Running average loss:0.00958, Time taken: 0:00:51.703506 ETA: 0:47:23.692803 2022-11-01 11:21:00,592 [INFO] __main__: Epoch: 45/100:, Cur-Step: 3820, loss(cross_entropy): 0.00980, Running average loss:0.00958, Time taken: 0:00:51.703506 ETA: 0:47:23.692803 Epoch: 45/100:, Cur-Step: 3830, loss(cross_entropy): 0.00938, Running average loss:0.00955, Time taken: 0:00:51.703506 ETA: 0:47:23.692803 2022-11-01 11:21:05,300 [INFO] __main__: Epoch: 45/100:, Cur-Step: 3830, loss(cross_entropy): 0.00938, Running average loss:0.00955, Time taken: 0:00:51.703506 ETA: 0:47:23.692803 Epoch: 45/100:, Cur-Step: 3840, loss(cross_entropy): 0.00937, Running average loss:0.00954, Time taken: 0:00:51.703506 ETA: 0:47:23.692803 2022-11-01 11:21:10,104 [INFO] __main__: Epoch: 45/100:, Cur-Step: 3840, loss(cross_entropy): 0.00937, Running average loss:0.00954, Time taken: 0:00:51.703506 ETA: 0:47:23.692803 Epoch: 45/100:, Cur-Step: 3850, loss(cross_entropy): 0.00920, Running average loss:0.00949, Time taken: 0:00:51.703506 ETA: 0:47:23.692803 2022-11-01 11:21:14,808 [INFO] __main__: Epoch: 45/100:, Cur-Step: 3850, loss(cross_entropy): 0.00920, Running average loss:0.00949, Time taken: 0:00:51.703506 ETA: 0:47:23.692803 Epoch: 45/100:, Cur-Step: 3860, loss(cross_entropy): 0.00859, Running average loss:0.00946, Time taken: 0:00:51.703506 ETA: 0:47:23.692803 2022-11-01 11:21:19,722 [INFO] __main__: Epoch: 45/100:, Cur-Step: 3860, loss(cross_entropy): 0.00859, Running average loss:0.00946, Time taken: 0:00:51.703506 ETA: 0:47:23.692803 INFO:tensorflow:Saving checkpoints for step-3864. 2022-11-01 11:21:21,192 [INFO] tensorflow: Saving checkpoints for step-3864. Epoch: 46/100:, Cur-Step: 3870, loss(cross_entropy): 0.01020, Running average loss:0.00954, Time taken: 0:00:51.878923 ETA: 0:46:41.461852 2022-11-01 11:21:35,872 [INFO] __main__: Epoch: 46/100:, Cur-Step: 3870, loss(cross_entropy): 0.01020, Running average loss:0.00954, Time taken: 0:00:51.878923 ETA: 0:46:41.461852 Epoch: 46/100:, Cur-Step: 3880, loss(cross_entropy): 0.00892, Running average loss:0.00928, Time taken: 0:00:51.878923 ETA: 0:46:41.461852 2022-11-01 11:21:40,563 [INFO] __main__: Epoch: 46/100:, Cur-Step: 3880, loss(cross_entropy): 0.00892, Running average loss:0.00928, Time taken: 0:00:51.878923 ETA: 0:46:41.461852 Epoch: 46/100:, Cur-Step: 3890, loss(cross_entropy): 0.00992, Running average loss:0.00938, Time taken: 0:00:51.878923 ETA: 0:46:41.461852 2022-11-01 11:21:45,364 [INFO] __main__: Epoch: 46/100:, Cur-Step: 3890, loss(cross_entropy): 0.00992, Running average loss:0.00938, Time taken: 0:00:51.878923 ETA: 0:46:41.461852 Epoch: 46/100:, Cur-Step: 3900, loss(cross_entropy): 0.00903, Running average loss:0.00945, Time taken: 0:00:51.878923 ETA: 0:46:41.461852 2022-11-01 11:21:50,085 [INFO] __main__: Epoch: 46/100:, Cur-Step: 3900, loss(cross_entropy): 0.00903, Running average loss:0.00945, Time taken: 0:00:51.878923 ETA: 0:46:41.461852 Epoch: 46/100:, Cur-Step: 3910, loss(cross_entropy): 0.00956, Running average loss:0.00953, Time taken: 0:00:51.878923 ETA: 0:46:41.461852 2022-11-01 11:21:54,813 [INFO] __main__: Epoch: 46/100:, Cur-Step: 3910, loss(cross_entropy): 0.00956, Running average loss:0.00953, Time taken: 0:00:51.878923 ETA: 0:46:41.461852 Epoch: 46/100:, Cur-Step: 3920, loss(cross_entropy): 0.00960, Running average loss:0.00960, Time taken: 0:00:51.878923 ETA: 0:46:41.461852 2022-11-01 11:21:59,593 [INFO] __main__: Epoch: 46/100:, Cur-Step: 3920, loss(cross_entropy): 0.00960, Running average loss:0.00960, Time taken: 0:00:51.878923 ETA: 0:46:41.461852 Epoch: 46/100:, Cur-Step: 3930, loss(cross_entropy): 0.00873, Running average loss:0.00961, Time taken: 0:00:51.878923 ETA: 0:46:41.461852 2022-11-01 11:22:04,278 [INFO] __main__: Epoch: 46/100:, Cur-Step: 3930, loss(cross_entropy): 0.00873, Running average loss:0.00961, Time taken: 0:00:51.878923 ETA: 0:46:41.461852 Epoch: 46/100:, Cur-Step: 3940, loss(cross_entropy): 0.00916, Running average loss:0.00960, Time taken: 0:00:51.878923 ETA: 0:46:41.461852 2022-11-01 11:22:09,168 [INFO] __main__: Epoch: 46/100:, Cur-Step: 3940, loss(cross_entropy): 0.00916, Running average loss:0.00960, Time taken: 0:00:51.878923 ETA: 0:46:41.461852 INFO:tensorflow:Saving checkpoints for step-3948. 2022-11-01 11:22:12,465 [INFO] tensorflow: Saving checkpoints for step-3948. Epoch: 47/100:, Cur-Step: 3950, loss(cross_entropy): 0.00978, Running average loss:0.00941, Time taken: 0:00:51.771886 ETA: 0:45:43.909951 2022-11-01 11:22:25,387 [INFO] __main__: Epoch: 47/100:, Cur-Step: 3950, loss(cross_entropy): 0.00978, Running average loss:0.00941, Time taken: 0:00:51.771886 ETA: 0:45:43.909951 Epoch: 47/100:, Cur-Step: 3960, loss(cross_entropy): 0.00904, Running average loss:0.00949, Time taken: 0:00:51.771886 ETA: 0:45:43.909951 2022-11-01 11:22:30,245 [INFO] __main__: Epoch: 47/100:, Cur-Step: 3960, loss(cross_entropy): 0.00904, Running average loss:0.00949, Time taken: 0:00:51.771886 ETA: 0:45:43.909951 Epoch: 47/100:, Cur-Step: 3970, loss(cross_entropy): 0.00919, Running average loss:0.00951, Time taken: 0:00:51.771886 ETA: 0:45:43.909951 2022-11-01 11:22:35,168 [INFO] __main__: Epoch: 47/100:, Cur-Step: 3970, loss(cross_entropy): 0.00919, Running average loss:0.00951, Time taken: 0:00:51.771886 ETA: 0:45:43.909951 Epoch: 47/100:, Cur-Step: 3980, loss(cross_entropy): 0.00973, Running average loss:0.00946, Time taken: 0:00:51.771886 ETA: 0:45:43.909951 2022-11-01 11:22:39,912 [INFO] __main__: Epoch: 47/100:, Cur-Step: 3980, loss(cross_entropy): 0.00973, Running average loss:0.00946, Time taken: 0:00:51.771886 ETA: 0:45:43.909951 Epoch: 47/100:, Cur-Step: 3990, loss(cross_entropy): 0.00943, Running average loss:0.00939, Time taken: 0:00:51.771886 ETA: 0:45:43.909951 2022-11-01 11:22:44,721 [INFO] __main__: Epoch: 47/100:, Cur-Step: 3990, loss(cross_entropy): 0.00943, Running average loss:0.00939, Time taken: 0:00:51.771886 ETA: 0:45:43.909951 Epoch: 47/100:, Cur-Step: 4000, loss(cross_entropy): 0.00889, Running average loss:0.00942, Time taken: 0:00:51.771886 ETA: 0:45:43.909951 2022-11-01 11:22:49,986 [INFO] __main__: Epoch: 47/100:, Cur-Step: 4000, loss(cross_entropy): 0.00889, Running average loss:0.00942, Time taken: 0:00:51.771886 ETA: 0:45:43.909951 Epoch: 47/100:, Cur-Step: 4010, loss(cross_entropy): 0.00891, Running average loss:0.00941, Time taken: 0:00:51.771886 ETA: 0:45:43.909951 2022-11-01 11:22:54,757 [INFO] __main__: Epoch: 47/100:, Cur-Step: 4010, loss(cross_entropy): 0.00891, Running average loss:0.00941, Time taken: 0:00:51.771886 ETA: 0:45:43.909951 Epoch: 47/100:, Cur-Step: 4020, loss(cross_entropy): 0.00843, Running average loss:0.00936, Time taken: 0:00:51.771886 ETA: 0:45:43.909951 2022-11-01 11:22:59,564 [INFO] __main__: Epoch: 47/100:, Cur-Step: 4020, loss(cross_entropy): 0.00843, Running average loss:0.00936, Time taken: 0:00:51.771886 ETA: 0:45:43.909951 Epoch: 47/100:, Cur-Step: 4030, loss(cross_entropy): 0.00821, Running average loss:0.00931, Time taken: 0:00:51.771886 ETA: 0:45:43.909951 2022-11-01 11:23:04,394 [INFO] __main__: Epoch: 47/100:, Cur-Step: 4030, loss(cross_entropy): 0.00821, Running average loss:0.00931, Time taken: 0:00:51.771886 ETA: 0:45:43.909951 INFO:tensorflow:Saving checkpoints for step-4032. 2022-11-01 11:23:04,862 [INFO] tensorflow: Saving checkpoints for step-4032. Epoch: 48/100:, Cur-Step: 4040, loss(cross_entropy): 0.00852, Running average loss:0.00910, Time taken: 0:00:52.862641 ETA: 0:45:48.857312 2022-11-01 11:23:20,334 [INFO] __main__: Epoch: 48/100:, Cur-Step: 4040, loss(cross_entropy): 0.00852, Running average loss:0.00910, Time taken: 0:00:52.862641 ETA: 0:45:48.857312 Epoch: 48/100:, Cur-Step: 4050, loss(cross_entropy): 0.00909, Running average loss:0.00907, Time taken: 0:00:52.862641 ETA: 0:45:48.857312 2022-11-01 11:23:25,006 [INFO] __main__: Epoch: 48/100:, Cur-Step: 4050, loss(cross_entropy): 0.00909, Running average loss:0.00907, Time taken: 0:00:52.862641 ETA: 0:45:48.857312 Epoch: 48/100:, Cur-Step: 4060, loss(cross_entropy): 0.00867, Running average loss:0.00913, Time taken: 0:00:52.862641 ETA: 0:45:48.857312 2022-11-01 11:23:29,686 [INFO] __main__: Epoch: 48/100:, Cur-Step: 4060, loss(cross_entropy): 0.00867, Running average loss:0.00913, Time taken: 0:00:52.862641 ETA: 0:45:48.857312 Epoch: 48/100:, Cur-Step: 4070, loss(cross_entropy): 0.00802, Running average loss:0.00905, Time taken: 0:00:52.862641 ETA: 0:45:48.857312 2022-11-01 11:23:34,398 [INFO] __main__: Epoch: 48/100:, Cur-Step: 4070, loss(cross_entropy): 0.00802, Running average loss:0.00905, Time taken: 0:00:52.862641 ETA: 0:45:48.857312 Epoch: 48/100:, Cur-Step: 4080, loss(cross_entropy): 0.00893, Running average loss:0.00903, Time taken: 0:00:52.862641 ETA: 0:45:48.857312 2022-11-01 11:23:39,082 [INFO] __main__: Epoch: 48/100:, Cur-Step: 4080, loss(cross_entropy): 0.00893, Running average loss:0.00903, Time taken: 0:00:52.862641 ETA: 0:45:48.857312 Epoch: 48/100:, Cur-Step: 4090, loss(cross_entropy): 0.00908, Running average loss:0.00898, Time taken: 0:00:52.862641 ETA: 0:45:48.857312 2022-11-01 11:23:43,795 [INFO] __main__: Epoch: 48/100:, Cur-Step: 4090, loss(cross_entropy): 0.00908, Running average loss:0.00898, Time taken: 0:00:52.862641 ETA: 0:45:48.857312 Epoch: 48/100:, Cur-Step: 4100, loss(cross_entropy): 0.00901, Running average loss:0.00896, Time taken: 0:00:52.862641 ETA: 0:45:48.857312 2022-11-01 11:23:48,614 [INFO] __main__: Epoch: 48/100:, Cur-Step: 4100, loss(cross_entropy): 0.00901, Running average loss:0.00896, Time taken: 0:00:52.862641 ETA: 0:45:48.857312 Epoch: 48/100:, Cur-Step: 4110, loss(cross_entropy): 0.00838, Running average loss:0.00894, Time taken: 0:00:52.862641 ETA: 0:45:48.857312 2022-11-01 11:23:53,312 [INFO] __main__: Epoch: 48/100:, Cur-Step: 4110, loss(cross_entropy): 0.00838, Running average loss:0.00894, Time taken: 0:00:52.862641 ETA: 0:45:48.857312 INFO:tensorflow:Saving checkpoints for step-4116. 2022-11-01 11:23:55,651 [INFO] tensorflow: Saving checkpoints for step-4116. Epoch: 49/100:, Cur-Step: 4120, loss(cross_entropy): 0.00907, Running average loss:0.00864, Time taken: 0:00:51.255422 ETA: 0:43:34.026516 2022-11-01 11:24:09,513 [INFO] __main__: Epoch: 49/100:, Cur-Step: 4120, loss(cross_entropy): 0.00907, Running average loss:0.00864, Time taken: 0:00:51.255422 ETA: 0:43:34.026516 Epoch: 49/100:, Cur-Step: 4130, loss(cross_entropy): 0.00877, Running average loss:0.00879, Time taken: 0:00:51.255422 ETA: 0:43:34.026516 2022-11-01 11:24:14,183 [INFO] __main__: Epoch: 49/100:, Cur-Step: 4130, loss(cross_entropy): 0.00877, Running average loss:0.00879, Time taken: 0:00:51.255422 ETA: 0:43:34.026516 Epoch: 49/100:, Cur-Step: 4140, loss(cross_entropy): 0.00806, Running average loss:0.00873, Time taken: 0:00:51.255422 ETA: 0:43:34.026516 2022-11-01 11:24:18,893 [INFO] __main__: Epoch: 49/100:, Cur-Step: 4140, loss(cross_entropy): 0.00806, Running average loss:0.00873, Time taken: 0:00:51.255422 ETA: 0:43:34.026516 Epoch: 49/100:, Cur-Step: 4150, loss(cross_entropy): 0.00911, Running average loss:0.00879, Time taken: 0:00:51.255422 ETA: 0:43:34.026516 2022-11-01 11:24:23,608 [INFO] __main__: Epoch: 49/100:, Cur-Step: 4150, loss(cross_entropy): 0.00911, Running average loss:0.00879, Time taken: 0:00:51.255422 ETA: 0:43:34.026516 Epoch: 49/100:, Cur-Step: 4160, loss(cross_entropy): 0.00897, Running average loss:0.00878, Time taken: 0:00:51.255422 ETA: 0:43:34.026516 2022-11-01 11:24:28,331 [INFO] __main__: Epoch: 49/100:, Cur-Step: 4160, loss(cross_entropy): 0.00897, Running average loss:0.00878, Time taken: 0:00:51.255422 ETA: 0:43:34.026516 Epoch: 49/100:, Cur-Step: 4170, loss(cross_entropy): 0.00835, Running average loss:0.00881, Time taken: 0:00:51.255422 ETA: 0:43:34.026516 2022-11-01 11:24:33,148 [INFO] __main__: Epoch: 49/100:, Cur-Step: 4170, loss(cross_entropy): 0.00835, Running average loss:0.00881, Time taken: 0:00:51.255422 ETA: 0:43:34.026516 Epoch: 49/100:, Cur-Step: 4180, loss(cross_entropy): 0.00767, Running average loss:0.00879, Time taken: 0:00:51.255422 ETA: 0:43:34.026516 2022-11-01 11:24:38,012 [INFO] __main__: Epoch: 49/100:, Cur-Step: 4180, loss(cross_entropy): 0.00767, Running average loss:0.00879, Time taken: 0:00:51.255422 ETA: 0:43:34.026516 Epoch: 49/100:, Cur-Step: 4190, loss(cross_entropy): 0.00843, Running average loss:0.00875, Time taken: 0:00:51.255422 ETA: 0:43:34.026516 2022-11-01 11:24:42,724 [INFO] __main__: Epoch: 49/100:, Cur-Step: 4190, loss(cross_entropy): 0.00843, Running average loss:0.00875, Time taken: 0:00:51.255422 ETA: 0:43:34.026516 INFO:tensorflow:Saving checkpoints for step-4200. 2022-11-01 11:24:46,960 [INFO] tensorflow: Saving checkpoints for step-4200. Epoch: 50/100:, Cur-Step: 4200, loss(cross_entropy): 0.00889, Running average loss:0.00889, Time taken: 0:00:51.775280 ETA: 0:43:08.764000 2022-11-01 11:24:59,092 [INFO] __main__: Epoch: 50/100:, Cur-Step: 4200, loss(cross_entropy): 0.00889, Running average loss:0.00889, Time taken: 0:00:51.775280 ETA: 0:43:08.764000 Epoch: 50/100:, Cur-Step: 4210, loss(cross_entropy): 0.00921, Running average loss:0.00869, Time taken: 0:00:51.775280 ETA: 0:43:08.764000 2022-11-01 11:25:03,995 [INFO] __main__: Epoch: 50/100:, Cur-Step: 4210, loss(cross_entropy): 0.00921, Running average loss:0.00869, Time taken: 0:00:51.775280 ETA: 0:43:08.764000 Epoch: 50/100:, Cur-Step: 4220, loss(cross_entropy): 0.00815, Running average loss:0.00865, Time taken: 0:00:51.775280 ETA: 0:43:08.764000 2022-11-01 11:25:08,707 [INFO] __main__: Epoch: 50/100:, Cur-Step: 4220, loss(cross_entropy): 0.00815, Running average loss:0.00865, Time taken: 0:00:51.775280 ETA: 0:43:08.764000 Epoch: 50/100:, Cur-Step: 4230, loss(cross_entropy): 0.00952, Running average loss:0.00876, Time taken: 0:00:51.775280 ETA: 0:43:08.764000 2022-11-01 11:25:13,485 [INFO] __main__: Epoch: 50/100:, Cur-Step: 4230, loss(cross_entropy): 0.00952, Running average loss:0.00876, Time taken: 0:00:51.775280 ETA: 0:43:08.764000 Epoch: 50/100:, Cur-Step: 4240, loss(cross_entropy): 0.00966, Running average loss:0.00878, Time taken: 0:00:51.775280 ETA: 0:43:08.764000 2022-11-01 11:25:18,170 [INFO] __main__: Epoch: 50/100:, Cur-Step: 4240, loss(cross_entropy): 0.00966, Running average loss:0.00878, Time taken: 0:00:51.775280 ETA: 0:43:08.764000 Epoch: 50/100:, Cur-Step: 4250, loss(cross_entropy): 0.00846, Running average loss:0.00875, Time taken: 0:00:51.775280 ETA: 0:43:08.764000 2022-11-01 11:25:22,923 [INFO] __main__: Epoch: 50/100:, Cur-Step: 4250, loss(cross_entropy): 0.00846, Running average loss:0.00875, Time taken: 0:00:51.775280 ETA: 0:43:08.764000 Epoch: 50/100:, Cur-Step: 4260, loss(cross_entropy): 0.00807, Running average loss:0.00874, Time taken: 0:00:51.775280 ETA: 0:43:08.764000 2022-11-01 11:25:27,815 [INFO] __main__: Epoch: 50/100:, Cur-Step: 4260, loss(cross_entropy): 0.00807, Running average loss:0.00874, Time taken: 0:00:51.775280 ETA: 0:43:08.764000 Epoch: 50/100:, Cur-Step: 4270, loss(cross_entropy): 0.00894, Running average loss:0.00876, Time taken: 0:00:51.775280 ETA: 0:43:08.764000 2022-11-01 11:25:32,527 [INFO] __main__: Epoch: 50/100:, Cur-Step: 4270, loss(cross_entropy): 0.00894, Running average loss:0.00876, Time taken: 0:00:51.775280 ETA: 0:43:08.764000 Epoch: 50/100:, Cur-Step: 4280, loss(cross_entropy): 0.00899, Running average loss:0.00881, Time taken: 0:00:51.775280 ETA: 0:43:08.764000 2022-11-01 11:25:37,203 [INFO] __main__: Epoch: 50/100:, Cur-Step: 4280, loss(cross_entropy): 0.00899, Running average loss:0.00881, Time taken: 0:00:51.775280 ETA: 0:43:08.764000 INFO:tensorflow:Saving checkpoints for step-4284. 2022-11-01 11:25:38,610 [INFO] tensorflow: Saving checkpoints for step-4284. Epoch: 51/100:, Cur-Step: 4290, loss(cross_entropy): 0.00819, Running average loss:0.00825, Time taken: 0:00:52.115544 ETA: 0:42:33.661672 2022-11-01 11:25:53,763 [INFO] __main__: Epoch: 51/100:, Cur-Step: 4290, loss(cross_entropy): 0.00819, Running average loss:0.00825, Time taken: 0:00:52.115544 ETA: 0:42:33.661672 Epoch: 51/100:, Cur-Step: 4300, loss(cross_entropy): 0.00782, Running average loss:0.00865, Time taken: 0:00:52.115544 ETA: 0:42:33.661672 2022-11-01 11:25:58,434 [INFO] __main__: Epoch: 51/100:, Cur-Step: 4300, loss(cross_entropy): 0.00782, Running average loss:0.00865, Time taken: 0:00:52.115544 ETA: 0:42:33.661672 Epoch: 51/100:, Cur-Step: 4310, loss(cross_entropy): 0.00819, Running average loss:0.00863, Time taken: 0:00:52.115544 ETA: 0:42:33.661672 2022-11-01 11:26:03,179 [INFO] __main__: Epoch: 51/100:, Cur-Step: 4310, loss(cross_entropy): 0.00819, Running average loss:0.00863, Time taken: 0:00:52.115544 ETA: 0:42:33.661672 Epoch: 51/100:, Cur-Step: 4320, loss(cross_entropy): 0.00847, Running average loss:0.00860, Time taken: 0:00:52.115544 ETA: 0:42:33.661672 2022-11-01 11:26:07,882 [INFO] __main__: Epoch: 51/100:, Cur-Step: 4320, loss(cross_entropy): 0.00847, Running average loss:0.00860, Time taken: 0:00:52.115544 ETA: 0:42:33.661672 Epoch: 51/100:, Cur-Step: 4330, loss(cross_entropy): 0.00822, Running average loss:0.00855, Time taken: 0:00:52.115544 ETA: 0:42:33.661672 2022-11-01 11:26:12,631 [INFO] __main__: Epoch: 51/100:, Cur-Step: 4330, loss(cross_entropy): 0.00822, Running average loss:0.00855, Time taken: 0:00:52.115544 ETA: 0:42:33.661672 Epoch: 51/100:, Cur-Step: 4340, loss(cross_entropy): 0.00773, Running average loss:0.00853, Time taken: 0:00:52.115544 ETA: 0:42:33.661672 2022-11-01 11:26:17,517 [INFO] __main__: Epoch: 51/100:, Cur-Step: 4340, loss(cross_entropy): 0.00773, Running average loss:0.00853, Time taken: 0:00:52.115544 ETA: 0:42:33.661672 Epoch: 51/100:, Cur-Step: 4350, loss(cross_entropy): 0.00831, Running average loss:0.00854, Time taken: 0:00:52.115544 ETA: 0:42:33.661672 2022-11-01 11:26:22,385 [INFO] __main__: Epoch: 51/100:, Cur-Step: 4350, loss(cross_entropy): 0.00831, Running average loss:0.00854, Time taken: 0:00:52.115544 ETA: 0:42:33.661672 Epoch: 51/100:, Cur-Step: 4360, loss(cross_entropy): 0.00772, Running average loss:0.00847, Time taken: 0:00:52.115544 ETA: 0:42:33.661672 2022-11-01 11:26:27,087 [INFO] __main__: Epoch: 51/100:, Cur-Step: 4360, loss(cross_entropy): 0.00772, Running average loss:0.00847, Time taken: 0:00:52.115544 ETA: 0:42:33.661672 INFO:tensorflow:Saving checkpoints for step-4368. 2022-11-01 11:26:30,390 [INFO] tensorflow: Saving checkpoints for step-4368. Epoch: 52/100:, Cur-Step: 4370, loss(cross_entropy): 0.00771, Running average loss:0.00800, Time taken: 0:00:52.261278 ETA: 0:41:48.541363 2022-11-01 11:26:43,139 [INFO] __main__: Epoch: 52/100:, Cur-Step: 4370, loss(cross_entropy): 0.00771, Running average loss:0.00800, Time taken: 0:00:52.261278 ETA: 0:41:48.541363 Epoch: 52/100:, Cur-Step: 4380, loss(cross_entropy): 0.00865, Running average loss:0.00800, Time taken: 0:00:52.261278 ETA: 0:41:48.541363 2022-11-01 11:26:47,945 [INFO] __main__: Epoch: 52/100:, Cur-Step: 4380, loss(cross_entropy): 0.00865, Running average loss:0.00800, Time taken: 0:00:52.261278 ETA: 0:41:48.541363 Epoch: 52/100:, Cur-Step: 4390, loss(cross_entropy): 0.00865, Running average loss:0.00800, Time taken: 0:00:52.261278 ETA: 0:41:48.541363 2022-11-01 11:26:52,596 [INFO] __main__: Epoch: 52/100:, Cur-Step: 4390, loss(cross_entropy): 0.00865, Running average loss:0.00800, Time taken: 0:00:52.261278 ETA: 0:41:48.541363 Epoch: 52/100:, Cur-Step: 4400, loss(cross_entropy): 0.00797, Running average loss:0.00806, Time taken: 0:00:52.261278 ETA: 0:41:48.541363 2022-11-01 11:26:57,244 [INFO] __main__: Epoch: 52/100:, Cur-Step: 4400, loss(cross_entropy): 0.00797, Running average loss:0.00806, Time taken: 0:00:52.261278 ETA: 0:41:48.541363 Epoch: 52/100:, Cur-Step: 4410, loss(cross_entropy): 0.00891, Running average loss:0.00808, Time taken: 0:00:52.261278 ETA: 0:41:48.541363 2022-11-01 11:27:01,936 [INFO] __main__: Epoch: 52/100:, Cur-Step: 4410, loss(cross_entropy): 0.00891, Running average loss:0.00808, Time taken: 0:00:52.261278 ETA: 0:41:48.541363 Epoch: 52/100:, Cur-Step: 4420, loss(cross_entropy): 0.00895, Running average loss:0.00811, Time taken: 0:00:52.261278 ETA: 0:41:48.541363 2022-11-01 11:27:06,724 [INFO] __main__: Epoch: 52/100:, Cur-Step: 4420, loss(cross_entropy): 0.00895, Running average loss:0.00811, Time taken: 0:00:52.261278 ETA: 0:41:48.541363 Epoch: 52/100:, Cur-Step: 4430, loss(cross_entropy): 0.00797, Running average loss:0.00807, Time taken: 0:00:52.261278 ETA: 0:41:48.541363 2022-11-01 11:27:11,714 [INFO] __main__: Epoch: 52/100:, Cur-Step: 4430, loss(cross_entropy): 0.00797, Running average loss:0.00807, Time taken: 0:00:52.261278 ETA: 0:41:48.541363 Epoch: 52/100:, Cur-Step: 4440, loss(cross_entropy): 0.00828, Running average loss:0.00808, Time taken: 0:00:52.261278 ETA: 0:41:48.541363 2022-11-01 11:27:16,475 [INFO] __main__: Epoch: 52/100:, Cur-Step: 4440, loss(cross_entropy): 0.00828, Running average loss:0.00808, Time taken: 0:00:52.261278 ETA: 0:41:48.541363 Epoch: 52/100:, Cur-Step: 4450, loss(cross_entropy): 0.00798, Running average loss:0.00809, Time taken: 0:00:52.261278 ETA: 0:41:48.541363 2022-11-01 11:27:21,124 [INFO] __main__: Epoch: 52/100:, Cur-Step: 4450, loss(cross_entropy): 0.00798, Running average loss:0.00809, Time taken: 0:00:52.261278 ETA: 0:41:48.541363 INFO:tensorflow:Saving checkpoints for step-4452. 2022-11-01 11:27:21,588 [INFO] tensorflow: Saving checkpoints for step-4452. Epoch: 53/100:, Cur-Step: 4460, loss(cross_entropy): 0.00869, Running average loss:0.00781, Time taken: 0:00:51.665132 ETA: 0:40:28.261217 2022-11-01 11:27:36,818 [INFO] __main__: Epoch: 53/100:, Cur-Step: 4460, loss(cross_entropy): 0.00869, Running average loss:0.00781, Time taken: 0:00:51.665132 ETA: 0:40:28.261217 Epoch: 53/100:, Cur-Step: 4470, loss(cross_entropy): 0.00879, Running average loss:0.00812, Time taken: 0:00:51.665132 ETA: 0:40:28.261217 2022-11-01 11:27:41,490 [INFO] __main__: Epoch: 53/100:, Cur-Step: 4470, loss(cross_entropy): 0.00879, Running average loss:0.00812, Time taken: 0:00:51.665132 ETA: 0:40:28.261217 Epoch: 53/100:, Cur-Step: 4480, loss(cross_entropy): 0.00820, Running average loss:0.00811, Time taken: 0:00:51.665132 ETA: 0:40:28.261217 2022-11-01 11:27:46,316 [INFO] __main__: Epoch: 53/100:, Cur-Step: 4480, loss(cross_entropy): 0.00820, Running average loss:0.00811, Time taken: 0:00:51.665132 ETA: 0:40:28.261217 Epoch: 53/100:, Cur-Step: 4490, loss(cross_entropy): 0.00829, Running average loss:0.00806, Time taken: 0:00:51.665132 ETA: 0:40:28.261217 2022-11-01 11:27:51,013 [INFO] __main__: Epoch: 53/100:, Cur-Step: 4490, loss(cross_entropy): 0.00829, Running average loss:0.00806, Time taken: 0:00:51.665132 ETA: 0:40:28.261217 Epoch: 53/100:, Cur-Step: 4500, loss(cross_entropy): 0.00742, Running average loss:0.00800, Time taken: 0:00:51.665132 ETA: 0:40:28.261217 2022-11-01 11:27:55,803 [INFO] __main__: Epoch: 53/100:, Cur-Step: 4500, loss(cross_entropy): 0.00742, Running average loss:0.00800, Time taken: 0:00:51.665132 ETA: 0:40:28.261217 Epoch: 53/100:, Cur-Step: 4510, loss(cross_entropy): 0.00779, Running average loss:0.00797, Time taken: 0:00:51.665132 ETA: 0:40:28.261217 2022-11-01 11:28:00,591 [INFO] __main__: Epoch: 53/100:, Cur-Step: 4510, loss(cross_entropy): 0.00779, Running average loss:0.00797, Time taken: 0:00:51.665132 ETA: 0:40:28.261217 Epoch: 53/100:, Cur-Step: 4520, loss(cross_entropy): 0.00822, Running average loss:0.00798, Time taken: 0:00:51.665132 ETA: 0:40:28.261217 2022-11-01 11:28:05,549 [INFO] __main__: Epoch: 53/100:, Cur-Step: 4520, loss(cross_entropy): 0.00822, Running average loss:0.00798, Time taken: 0:00:51.665132 ETA: 0:40:28.261217 Epoch: 53/100:, Cur-Step: 4530, loss(cross_entropy): 0.00748, Running average loss:0.00794, Time taken: 0:00:51.665132 ETA: 0:40:28.261217 2022-11-01 11:28:10,227 [INFO] __main__: Epoch: 53/100:, Cur-Step: 4530, loss(cross_entropy): 0.00748, Running average loss:0.00794, Time taken: 0:00:51.665132 ETA: 0:40:28.261217 INFO:tensorflow:Saving checkpoints for step-4536. 2022-11-01 11:28:12,576 [INFO] tensorflow: Saving checkpoints for step-4536. Epoch: 54/100:, Cur-Step: 4540, loss(cross_entropy): 0.00759, Running average loss:0.00775, Time taken: 0:00:51.449859 ETA: 0:39:26.693510 2022-11-01 11:28:27,117 [INFO] __main__: Epoch: 54/100:, Cur-Step: 4540, loss(cross_entropy): 0.00759, Running average loss:0.00775, Time taken: 0:00:51.449859 ETA: 0:39:26.693510 Epoch: 54/100:, Cur-Step: 4550, loss(cross_entropy): 0.00742, Running average loss:0.00768, Time taken: 0:00:51.449859 ETA: 0:39:26.693510 2022-11-01 11:28:31,789 [INFO] __main__: Epoch: 54/100:, Cur-Step: 4550, loss(cross_entropy): 0.00742, Running average loss:0.00768, Time taken: 0:00:51.449859 ETA: 0:39:26.693510 Epoch: 54/100:, Cur-Step: 4560, loss(cross_entropy): 0.00754, Running average loss:0.00767, Time taken: 0:00:51.449859 ETA: 0:39:26.693510 2022-11-01 11:28:36,480 [INFO] __main__: Epoch: 54/100:, Cur-Step: 4560, loss(cross_entropy): 0.00754, Running average loss:0.00767, Time taken: 0:00:51.449859 ETA: 0:39:26.693510 Epoch: 54/100:, Cur-Step: 4570, loss(cross_entropy): 0.00765, Running average loss:0.00765, Time taken: 0:00:51.449859 ETA: 0:39:26.693510 2022-11-01 11:28:41,256 [INFO] __main__: Epoch: 54/100:, Cur-Step: 4570, loss(cross_entropy): 0.00765, Running average loss:0.00765, Time taken: 0:00:51.449859 ETA: 0:39:26.693510 Epoch: 54/100:, Cur-Step: 4580, loss(cross_entropy): 0.00738, Running average loss:0.00769, Time taken: 0:00:51.449859 ETA: 0:39:26.693510 2022-11-01 11:28:45,981 [INFO] __main__: Epoch: 54/100:, Cur-Step: 4580, loss(cross_entropy): 0.00738, Running average loss:0.00769, Time taken: 0:00:51.449859 ETA: 0:39:26.693510 Epoch: 54/100:, Cur-Step: 4590, loss(cross_entropy): 0.00762, Running average loss:0.00766, Time taken: 0:00:51.449859 ETA: 0:39:26.693510 2022-11-01 11:28:50,726 [INFO] __main__: Epoch: 54/100:, Cur-Step: 4590, loss(cross_entropy): 0.00762, Running average loss:0.00766, Time taken: 0:00:51.449859 ETA: 0:39:26.693510 Epoch: 54/100:, Cur-Step: 4600, loss(cross_entropy): 0.00763, Running average loss:0.00766, Time taken: 0:00:51.449859 ETA: 0:39:26.693510 2022-11-01 11:28:55,484 [INFO] __main__: Epoch: 54/100:, Cur-Step: 4600, loss(cross_entropy): 0.00763, Running average loss:0.00766, Time taken: 0:00:51.449859 ETA: 0:39:26.693510 Epoch: 54/100:, Cur-Step: 4610, loss(cross_entropy): 0.00726, Running average loss:0.00765, Time taken: 0:00:51.449859 ETA: 0:39:26.693510 2022-11-01 11:29:00,302 [INFO] __main__: Epoch: 54/100:, Cur-Step: 4610, loss(cross_entropy): 0.00726, Running average loss:0.00765, Time taken: 0:00:51.449859 ETA: 0:39:26.693510 INFO:tensorflow:Saving checkpoints for step-4620. 2022-11-01 11:29:04,534 [INFO] tensorflow: Saving checkpoints for step-4620. Epoch: 55/100:, Cur-Step: 4620, loss(cross_entropy): 0.00768, Running average loss:0.00768, Time taken: 0:00:52.421159 ETA: 0:39:18.952135 2022-11-01 11:29:16,910 [INFO] __main__: Epoch: 55/100:, Cur-Step: 4620, loss(cross_entropy): 0.00768, Running average loss:0.00768, Time taken: 0:00:52.421159 ETA: 0:39:18.952135 Epoch: 55/100:, Cur-Step: 4630, loss(cross_entropy): 0.00749, Running average loss:0.00748, Time taken: 0:00:52.421159 ETA: 0:39:18.952135 2022-11-01 11:29:21,896 [INFO] __main__: Epoch: 55/100:, Cur-Step: 4630, loss(cross_entropy): 0.00749, Running average loss:0.00748, Time taken: 0:00:52.421159 ETA: 0:39:18.952135 Epoch: 55/100:, Cur-Step: 4640, loss(cross_entropy): 0.00760, Running average loss:0.00746, Time taken: 0:00:52.421159 ETA: 0:39:18.952135 2022-11-01 11:29:26,636 [INFO] __main__: Epoch: 55/100:, Cur-Step: 4640, loss(cross_entropy): 0.00760, Running average loss:0.00746, Time taken: 0:00:52.421159 ETA: 0:39:18.952135 Epoch: 55/100:, Cur-Step: 4650, loss(cross_entropy): 0.00839, Running average loss:0.00751, Time taken: 0:00:52.421159 ETA: 0:39:18.952135 2022-11-01 11:29:31,373 [INFO] __main__: Epoch: 55/100:, Cur-Step: 4650, loss(cross_entropy): 0.00839, Running average loss:0.00751, Time taken: 0:00:52.421159 ETA: 0:39:18.952135 Epoch: 55/100:, Cur-Step: 4660, loss(cross_entropy): 0.00741, Running average loss:0.00751, Time taken: 0:00:52.421159 ETA: 0:39:18.952135 2022-11-01 11:29:36,086 [INFO] __main__: Epoch: 55/100:, Cur-Step: 4660, loss(cross_entropy): 0.00741, Running average loss:0.00751, Time taken: 0:00:52.421159 ETA: 0:39:18.952135 Epoch: 55/100:, Cur-Step: 4670, loss(cross_entropy): 0.00716, Running average loss:0.00748, Time taken: 0:00:52.421159 ETA: 0:39:18.952135 2022-11-01 11:29:40,899 [INFO] __main__: Epoch: 55/100:, Cur-Step: 4670, loss(cross_entropy): 0.00716, Running average loss:0.00748, Time taken: 0:00:52.421159 ETA: 0:39:18.952135 Epoch: 55/100:, Cur-Step: 4680, loss(cross_entropy): 0.00750, Running average loss:0.00745, Time taken: 0:00:52.421159 ETA: 0:39:18.952135 2022-11-01 11:29:45,591 [INFO] __main__: Epoch: 55/100:, Cur-Step: 4680, loss(cross_entropy): 0.00750, Running average loss:0.00745, Time taken: 0:00:52.421159 ETA: 0:39:18.952135 Epoch: 55/100:, Cur-Step: 4690, loss(cross_entropy): 0.00712, Running average loss:0.00743, Time taken: 0:00:52.421159 ETA: 0:39:18.952135 2022-11-01 11:29:50,306 [INFO] __main__: Epoch: 55/100:, Cur-Step: 4690, loss(cross_entropy): 0.00712, Running average loss:0.00743, Time taken: 0:00:52.421159 ETA: 0:39:18.952135 Epoch: 55/100:, Cur-Step: 4700, loss(cross_entropy): 0.00755, Running average loss:0.00747, Time taken: 0:00:52.421159 ETA: 0:39:18.952135 2022-11-01 11:29:55,018 [INFO] __main__: Epoch: 55/100:, Cur-Step: 4700, loss(cross_entropy): 0.00755, Running average loss:0.00747, Time taken: 0:00:52.421159 ETA: 0:39:18.952135 INFO:tensorflow:Saving checkpoints for step-4704. 2022-11-01 11:29:56,569 [INFO] tensorflow: Saving checkpoints for step-4704. Traceback (most recent call last): File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/scripts/train.py", line 424, in File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/scripts/train.py", line 418, in main File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/scripts/train.py", line 319, in run_experiment File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/scripts/train.py", line 234, in train_unet File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/scripts/train.py", line 105, in run_training_loop File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 370, in train loss = self._train_model(input_fn, hooks, saving_listeners) File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1161, in _train_model return self._train_model_default(input_fn, hooks, saving_listeners) File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1195, in _train_model_default saving_listeners) File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1494, in _train_with_estimator_spec _, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss]) File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/monitored_session.py", line 754, in run run_metadata=run_metadata) File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/monitored_session.py", line 1259, in run run_metadata=run_metadata) File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/monitored_session.py", line 1360, in run raise six.reraise(*original_exc_info) File "/usr/local/lib/python3.6/dist-packages/six.py", line 696, in reraise raise value File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/monitored_session.py", line 1345, in run return self._sess.run(*args, **kwargs) File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/monitored_session.py", line 1426, in run run_metadata=run_metadata)) File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/basic_session_run_hooks.py", line 594, in after_run if self._save(run_context.session, global_step): File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/hooks/checkpoint_saver_hook.py", line 75, in _save File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/hooks/checkpoint_saver_hook.py", line 90, in _save_encrypted_checkpoint File "/usr/lib/python3.6/tempfile.py", line 509, in mkdtemp _os.mkdir(file, 0o700) OSError: [Errno 28] No space left on device: '/tmp/tmp0ld14dko' 2022-11-01 13:30:00,985 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.