2022-07-31 11:53:34,764 [INFO] root: Registry: ['nvcr.io'] 2022-07-31 11:53:34,907 [INFO] tlt.components.instance_handler.local_instance: Running command in container: nvcr.io/nvidia/tao/tao-toolkit-tf:v3.21.11-tf1.15.5-py3 Matplotlib created a temporary config/cache directory at /tmp/matplotlib-lkeuuwd1 because the default path (/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing. Using TensorFlow backend. WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them. Using TensorFlow backend. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/hooks/checkpoint_saver_hook.py:21: The name tf.train.CheckpointSaverHook is deprecated. Please use tf.estimator.CheckpointSaverHook instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/hooks/pretrained_restore_hook.py:23: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/hooks/pretrained_restore_hook.py:23: The name tf.logging.WARN is deprecated. Please use tf.compat.v1.logging.WARN instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/scripts/train.py:410: The name tf.logging.INFO is deprecated. Please use tf.compat.v1.logging.INFO instead. Loading experiment spec at /workspace/tao-experiments/specs/unet_retrain_vgg_6S1100.txt. 2022-07-31 15:53:42,325 [INFO] __main__: Loading experiment spec at /workspace/tao-experiments/specs/unet_retrain_vgg_6S1100.txt. 2022-07-31 15:53:42,327 [INFO] iva.unet.spec_handler.spec_loader: Merging specification from /workspace/tao-experiments/specs/unet_retrain_vgg_6S1100.txt 2022-07-31 15:53:42,330 [INFO] root: Initializing the pre-trained weights from /workspace/tao-experiments/pruned/model_pruned.tlt 2022-07-31 15:53:42,330 [INFO] iva.unet.model.utilities: Loading weights from /workspace/tao-experiments/pruned/model_pruned.tlt WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead. 2022-07-31 15:53:42,861 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead. 2022-07-31 15:53:42,868 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:245: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead. 2022-07-31 15:53:42,883 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:245: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead. WARNING:tensorflow:From /opt/nvidia/third_party/keras/tensorflow_backend.py:183: The name tf.nn.max_pool is deprecated. Please use tf.nn.max_pool2d instead. 2022-07-31 15:53:42,898 [WARNING] tensorflow: From /opt/nvidia/third_party/keras/tensorflow_backend.py:183: The name tf.nn.max_pool is deprecated. Please use tf.nn.max_pool2d instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead. 2022-07-31 15:53:43,283 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:190: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead. 2022-07-31 15:53:43,283 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:190: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead. 2022-07-31 15:53:43,283 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead. 2022-07-31 15:53:43,352 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:95: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead. 2022-07-31 15:53:43,678 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:95: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:98: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead. 2022-07-31 15:53:43,678 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:98: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead. 2022-07-31 15:53:43,689 [INFO] iva.unet.model.utilities: Label Id 0: Train Id 0 2022-07-31 15:53:43,689 [INFO] iva.unet.model.utilities: Label Id 1: Train Id 1 2022-07-31 15:53:43,689 [INFO] iva.unet.model.utilities: Label Id 2: Train Id 2 2022-07-31 15:53:43,689 [INFO] iva.unet.model.utilities: Label Id 3: Train Id 3 2022-07-31 15:53:43,689 [INFO] iva.unet.model.utilities: Label Id 4: Train Id 4 2022-07-31 15:53:43,689 [INFO] iva.unet.model.utilities: Label Id 5: Train Id 5 2022-07-31 15:53:43,691 [INFO] iva.unet.hooks.latest_checkpoint: Getting the latest checkpoint for restoring /workspace/tao-experiments/retrain/model.step-20000.tlt INFO:tensorflow:Using config: {'_model_dir': '/workspace/tao-experiments/retrain', '_tf_random_seed': None, '_save_summary_steps': 5, '_save_checkpoints_steps': None, '_save_checkpoints_secs': None, '_session_config': intra_op_parallelism_threads: 1 inter_op_parallelism_threads: 38 gpu_options { allow_growth: true visible_device_list: "0" force_gpu_compatible: true } allow_soft_placement: true , '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': None, '_log_step_count_steps': None, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': , '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1} 2022-07-31 15:53:47,373 [INFO] tensorflow: Using config: {'_model_dir': '/workspace/tao-experiments/retrain', '_tf_random_seed': None, '_save_summary_steps': 5, '_save_checkpoints_steps': None, '_save_checkpoints_secs': None, '_session_config': intra_op_parallelism_threads: 1 inter_op_parallelism_threads: 38 gpu_options { allow_growth: true visible_device_list: "0" force_gpu_compatible: true } allow_soft_placement: true , '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': None, '_log_step_count_steps': None, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': , '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1} Phase train: Total 78 files. 2022-07-31 15:53:47,403 [INFO] iva.unet.model.utilities: The total number of training samples 78 and the batch size per GPU 4 2022-07-31 15:53:47,403 [INFO] iva.unet.model.utilities: Cannot iterate over exactly 78 samples with a batch size of 4; each epoch will therefore take one extra step. 2022-07-31 15:53:47,403 [INFO] iva.unet.model.utilities: Steps per epoch taken: 20 Running for 1100 Epochs 2022-07-31 15:53:47,403 [INFO] __main__: Running for 1100 Epochs INFO:tensorflow:Create CheckpointSaverHook. 2022-07-31 15:53:47,404 [INFO] tensorflow: Create CheckpointSaverHook. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead. 2022-07-31 15:53:47,847 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead. WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2022-07-31 15:53:47,897 [WARNING] tensorflow: Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code WARNING:tensorflow:Entity . at 0x7f8bac1c70d0> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of . at 0x7f8bac1c70d0>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2022-07-31 15:53:47,913 [WARNING] tensorflow: Entity . at 0x7f8bac1c70d0> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of . at 0x7f8bac1c70d0>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code WARNING:tensorflow: The TensorFlow contrib module will not be included in TensorFlow 2.0. For more information, please see: * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md * https://github.com/tensorflow/addons * https://github.com/tensorflow/io (for I/O related ops) If you depend on functionality not listed there, please file an issue. 2022-07-31 15:53:47,915 [WARNING] tensorflow: The TensorFlow contrib module will not be included in TensorFlow 2.0. For more information, please see: * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md * https://github.com/tensorflow/addons * https://github.com/tensorflow/io (for I/O related ops) If you depend on functionality not listed there, please file an issue. /opt/nvidia/third_party/keras/tensorflow_backend.py:356: UserWarning: Creating resources inside a function passed to Dataset.map() is not supported. Create each resource outside the function, and capture it inside the function to use it. self, _map_func_set_random_wrapper, num_parallel_calls=num_parallel_calls WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2022-07-31 15:53:47,924 [WARNING] tensorflow: Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2022-07-31 15:53:47,931 [WARNING] tensorflow: Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2022-07-31 15:53:47,938 [WARNING] tensorflow: Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/utils/data_loader.py:451: The name tf.image.resize_images is deprecated. Please use tf.image.resize instead. 2022-07-31 15:53:47,938 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/utils/data_loader.py:451: The name tf.image.resize_images is deprecated. Please use tf.image.resize instead. WARNING:tensorflow:Entity . at 0x7f8b9a8ee620> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of . at 0x7f8b9a8ee620>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2022-07-31 15:53:47,948 [WARNING] tensorflow: Entity . at 0x7f8b9a8ee620> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of . at 0x7f8b9a8ee620>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code WARNING:tensorflow:Entity . at 0x7f8b9a8ee8c8> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of . at 0x7f8b9a8ee8c8>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2022-07-31 15:53:47,955 [WARNING] tensorflow: Entity . at 0x7f8b9a8ee8c8> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of . at 0x7f8b9a8ee8c8>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code WARNING:tensorflow:Entity . at 0x7f8b9a8eec80> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of . at 0x7f8b9a8eec80>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2022-07-31 15:53:47,961 [WARNING] tensorflow: Entity . at 0x7f8b9a8eec80> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of . at 0x7f8b9a8eec80>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2022-07-31 15:53:47,967 [WARNING] tensorflow: Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code WARNING:tensorflow:Entity . at 0x7f8b9d090048> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of . at 0x7f8b9d090048>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2022-07-31 15:53:47,974 [WARNING] tensorflow: Entity . at 0x7f8b9d090048> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of . at 0x7f8b9d090048>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code INFO:tensorflow:Calling model_fn. 2022-07-31 15:53:47,990 [INFO] tensorflow: Calling model_fn. 2022-07-31 15:53:47,990 [INFO] iva.unet.utils.model_fn: {'exec_mode': 'train', 'model_dir': '/workspace/tao-experiments/retrain', 'resize_padding': False, 'resize_method': 'BILINEAR', 'log_dir': None, 'batch_size': 4, 'learning_rate': 9.999999747378752e-05, 'activation': 'softmax', 'crossvalidation_idx': None, 'max_steps': None, 'regularizer_type': 2, 'weight_decay': 1.9999999949504854e-06, 'log_summary_steps': 10, 'warmup_steps': 0, 'augment': False, 'use_amp': False, 'use_trt': False, 'use_xla': False, 'loss': 'cross_entropy', 'epochs': 1100, 'pretrained_weights_file': None, 'lr_scheduler': None, 'unet_model': , 'key': 'nvidia_tlt', 'experiment_spec': random_seed: 42 dataset_config { dataset: "custom" input_image_type: "color" train_images_path: "/workspace/tao-experiments/data/images/train" train_masks_path: "/workspace/tao-experiments/data/masks/train" val_images_path: "/workspace/tao-experiments/data/images/val" val_masks_path: "/workspace/tao-experiments/data/masks/val" test_images_path: "/workspace/tao-experiments/data/images/test" data_class_config { target_classes { name: "Background" mapping_class: "Background" } target_classes { name: "Joint" label_id: 1 mapping_class: "Joint" } target_classes { name: "Robot" label_id: 2 mapping_class: "Robot" } target_classes { name: "End" label_id: 3 mapping_class: "End" } target_classes { name: "Stem" label_id: 4 mapping_class: "Stem" } target_classes { name: "Leaf" label_id: 5 mapping_class: "Leaf" } } } model_config { num_layers: 16 training_precision { backend_floatx: FLOAT32 } arch: "vgg" all_projections: true model_input_height: 704 model_input_width: 1280 model_input_channels: 3 } training_config { batch_size: 4 regularizer { type: L2 weight: 1.9999999949504854e-06 } optimizer { adam { epsilon: 9.99999993922529e-09 beta1: 0.8999999761581421 beta2: 0.9990000128746033 } } checkpoint_interval: 100 log_summary_steps: 10 learning_rate: 9.999999747378752e-05 loss: "cross_entropy" epochs: 1100 } , 'seed': 42, 'benchmark': False, 'temp_dir': '/tmp/tmpgc67ajgp', 'num_classes': 6, 'num_conf_mat_classes': 6, 'start_step': 20000, 'checkpoint_interval': 100, 'model_json': '/tmp/tmp1bxghvr0/model.json', 'load_graph': False, 'weights_monitor': False, 'phase': None} __________________________________________________________________________________________________ Layer (type) Output Shape Param # Connected to ================================================================================================== input_1 (InputLayer) (None, 3, 704, 1280) 0 __________________________________________________________________________________________________ block_1a_conv_1 (Conv2D) (None, 64, 704, 1280 1792 input_1[0][0] __________________________________________________________________________________________________ block_1a_relu (Activation) (None, 64, 704, 1280 0 block_1a_conv_1[0][0] __________________________________________________________________________________________________ block_1b_conv_1 (Conv2D) (None, 64, 704, 1280 36928 block_1a_relu[0][0] __________________________________________________________________________________________________ block_1b_relu (Activation) (None, 64, 704, 1280 0 block_1b_conv_1[0][0] __________________________________________________________________________________________________ block1_pool (MaxPooling2D) (None, 64, 352, 640) 0 block_1b_relu[0][0] __________________________________________________________________________________________________ block_2a_conv_1 (Conv2D) (None, 128, 352, 640 73856 block1_pool[0][0] __________________________________________________________________________________________________ block_2a_relu (Activation) (None, 128, 352, 640 0 block_2a_conv_1[0][0] __________________________________________________________________________________________________ block_2b_conv_1 (Conv2D) (None, 128, 352, 640 147584 block_2a_relu[0][0] __________________________________________________________________________________________________ block_2b_relu (Activation) (None, 128, 352, 640 0 block_2b_conv_1[0][0] __________________________________________________________________________________________________ block2_pool (MaxPooling2D) (None, 128, 176, 320 0 block_2b_relu[0][0] __________________________________________________________________________________________________ block_3a_conv_1 (Conv2D) (None, 256, 176, 320 295168 block2_pool[0][0] __________________________________________________________________________________________________ block_3a_relu (Activation) (None, 256, 176, 320 0 block_3a_conv_1[0][0] __________________________________________________________________________________________________ block_3b_conv_1 (Conv2D) (None, 256, 176, 320 590080 block_3a_relu[0][0] __________________________________________________________________________________________________ block_3b_relu (Activation) (None, 256, 176, 320 0 block_3b_conv_1[0][0] __________________________________________________________________________________________________ block_3c_conv_1 (Conv2D) (None, 256, 176, 320 590080 block_3b_relu[0][0] __________________________________________________________________________________________________ block_3c_relu (Activation) (None, 256, 176, 320 0 block_3c_conv_1[0][0] __________________________________________________________________________________________________ block3_pool (MaxPooling2D) (None, 256, 88, 160) 0 block_3c_relu[0][0] __________________________________________________________________________________________________ block_4a_conv_1 (Conv2D) (None, 512, 88, 160) 1180160 block3_pool[0][0] __________________________________________________________________________________________________ block_4a_relu (Activation) (None, 512, 88, 160) 0 block_4a_conv_1[0][0] __________________________________________________________________________________________________ block_4b_conv_1 (Conv2D) (None, 512, 88, 160) 2359808 block_4a_relu[0][0] __________________________________________________________________________________________________ block_4b_relu (Activation) (None, 512, 88, 160) 0 block_4b_conv_1[0][0] __________________________________________________________________________________________________ block_4c_conv_1 (Conv2D) (None, 512, 88, 160) 2359808 block_4b_relu[0][0] __________________________________________________________________________________________________ block_4c_relu (Activation) (None, 512, 88, 160) 0 block_4c_conv_1[0][0] __________________________________________________________________________________________________ block4_pool (MaxPooling2D) (None, 512, 44, 80) 0 block_4c_relu[0][0] __________________________________________________________________________________________________ block_5a_conv_1 (Conv2D) (None, 512, 44, 80) 2359808 block4_pool[0][0] __________________________________________________________________________________________________ block_5a_relu (Activation) (None, 512, 44, 80) 0 block_5a_conv_1[0][0] __________________________________________________________________________________________________ block_5b_conv_1 (Conv2D) (None, 512, 44, 80) 2359808 block_5a_relu[0][0] __________________________________________________________________________________________________ block_5b_relu (Activation) (None, 512, 44, 80) 0 block_5b_conv_1[0][0] __________________________________________________________________________________________________ block_5c_conv_1 (Conv2D) (None, 512, 44, 80) 2359808 block_5b_relu[0][0] __________________________________________________________________________________________________ block_5c_relu (Activation) (None, 512, 44, 80) 0 block_5c_conv_1[0][0] __________________________________________________________________________________________________ max_pooling2d_1 (MaxPooling2D) (None, 512, 22, 40) 0 block_5c_relu[0][0] __________________________________________________________________________________________________ conv2d_transpose_1 (Conv2DTrans (None, 512, 44, 80) 4194816 max_pooling2d_1[0][0] __________________________________________________________________________________________________ concatenate_1 (Concatenate) (None, 1024, 44, 80) 0 conv2d_transpose_1[0][0] block4_pool[0][0] __________________________________________________________________________________________________ conv2d_1 (Conv2D) (None, 512, 44, 80) 4719104 concatenate_1[0][0] __________________________________________________________________________________________________ conv2d_2 (Conv2D) (None, 512, 44, 80) 2359808 conv2d_1[0][0] __________________________________________________________________________________________________ conv2d_transpose_2 (Conv2DTrans (None, 256, 88, 160) 2097408 conv2d_2[0][0] __________________________________________________________________________________________________ concatenate_2 (Concatenate) (None, 512, 88, 160) 0 conv2d_transpose_2[0][0] block3_pool[0][0] __________________________________________________________________________________________________ conv2d_3 (Conv2D) (None, 256, 88, 160) 1179904 concatenate_2[0][0] __________________________________________________________________________________________________ conv2d_4 (Conv2D) (None, 256, 88, 160) 590080 conv2d_3[0][0] __________________________________________________________________________________________________ conv2d_transpose_3 (Conv2DTrans (None, 128, 176, 320 524416 conv2d_4[0][0] __________________________________________________________________________________________________ concatenate_3 (Concatenate) (None, 256, 176, 320 0 conv2d_transpose_3[0][0] block2_pool[0][0] __________________________________________________________________________________________________ conv2d_5 (Conv2D) (None, 128, 176, 320 295040 concatenate_3[0][0] __________________________________________________________________________________________________ conv2d_6 (Conv2D) (None, 128, 176, 320 147584 conv2d_5[0][0] __________________________________________________________________________________________________ conv2d_transpose_4 (Conv2DTrans (None, 64, 352, 640) 131136 conv2d_6[0][0] __________________________________________________________________________________________________ concatenate_4 (Concatenate) (None, 128, 352, 640 0 conv2d_transpose_4[0][0] block1_pool[0][0] __________________________________________________________________________________________________ conv2d_7 (Conv2D) (None, 64, 352, 640) 73792 concatenate_4[0][0] __________________________________________________________________________________________________ conv2d_8 (Conv2D) (None, 64, 352, 640) 36928 conv2d_7[0][0] __________________________________________________________________________________________________ conv2d_transpose_5 (Conv2DTrans (None, 64, 704, 1280 65600 conv2d_8[0][0] __________________________________________________________________________________________________ concatenate_5 (Concatenate) (None, 128, 704, 128 0 conv2d_transpose_5[0][0] block_1a_relu[0][0] __________________________________________________________________________________________________ conv2d_9 (Conv2D) (None, 64, 704, 1280 73792 concatenate_5[0][0] __________________________________________________________________________________________________ conv2d_10 (Conv2D) (None, 64, 704, 1280 36928 conv2d_9[0][0] __________________________________________________________________________________________________ conv2d_11 (Conv2D) (None, 6, 704, 1280) 390 conv2d_10[0][0] ================================================================================================== Total params: 31,241,414 Trainable params: 31,241,414 Non-trainable params: 0 __________________________________________________________________________________________________ WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/utils/model_fn.py:225: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead. 2022-07-31 15:53:48,419 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/utils/model_fn.py:225: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead. INFO:tensorflow:Done calling model_fn. 2022-07-31 15:53:49,502 [INFO] tensorflow: Done calling model_fn. INFO:tensorflow:Graph was finalized. 2022-07-31 15:53:50,396 [INFO] tensorflow: Graph was finalized. INFO:tensorflow:Running local_init_op. 2022-07-31 15:53:51,289 [INFO] tensorflow: Running local_init_op. INFO:tensorflow:Done running local_init_op. 2022-07-31 15:53:51,351 [INFO] tensorflow: Done running local_init_op. [GPU] Restoring pretrained weights from: /tmp/tmp1bxghvr0/model.ckpt-20000 2022-07-31 15:53:51,749 [INFO] iva.unet.hooks.pretrained_restore_hook: Pretrained weights loaded with success... INFO:tensorflow:Saving checkpoints for step-20000. 2022-07-31 15:53:53,748 [INFO] tensorflow: Saving checkpoints for step-20000. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/hooks/training_hook.py:95: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead. 2022-07-31 15:54:16,793 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/hooks/training_hook.py:95: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead. Epoch: 1000/1100:, Cur-Step: 20000, loss(cross_entropy): 0.00249, Running average loss:0.00249, Time taken: 0:00:00 ETA: 0:00:00 2022-07-31 15:55:22,446 [INFO] __main__: Epoch: 1000/1100:, Cur-Step: 20000, loss(cross_entropy): 0.00249, Running average loss:0.00249, Time taken: 0:00:00 ETA: 0:00:00 Epoch: 1000/1100:, Cur-Step: 20010, loss(cross_entropy): 0.00383, Running average loss:0.00435, Time taken: 0:00:00 ETA: 0:00:00 2022-07-31 15:58:27,246 [INFO] __main__: Epoch: 1000/1100:, Cur-Step: 20010, loss(cross_entropy): 0.00383, Running average loss:0.00435, Time taken: 0:00:00 ETA: 0:00:00 Epoch: 1001/1100:, Cur-Step: 20020, loss(cross_entropy): 0.00307, Running average loss:0.00307, Time taken: 0:04:46.386097 ETA: 7:52:32.223575 2022-07-31 15:58:34,565 [INFO] __main__: Epoch: 1001/1100:, Cur-Step: 20020, loss(cross_entropy): 0.00307, Running average loss:0.00307, Time taken: 0:04:46.386097 ETA: 7:52:32.223575 Epoch: 1001/1100:, Cur-Step: 20030, loss(cross_entropy): 0.00308, Running average loss:0.00306, Time taken: 0:04:46.386097 ETA: 7:52:32.223575 2022-07-31 15:58:41,849 [INFO] __main__: Epoch: 1001/1100:, Cur-Step: 20030, loss(cross_entropy): 0.00308, Running average loss:0.00306, Time taken: 0:04:46.386097 ETA: 7:52:32.223575 Epoch: 1002/1100:, Cur-Step: 20040, loss(cross_entropy): 0.00261, Running average loss:0.00261, Time taken: 0:00:15.675306 ETA: 0:25:36.179996 2022-07-31 15:58:49,316 [INFO] __main__: Epoch: 1002/1100:, Cur-Step: 20040, loss(cross_entropy): 0.00261, Running average loss:0.00261, Time taken: 0:00:15.675306 ETA: 0:25:36.179996 Epoch: 1002/1100:, Cur-Step: 20050, loss(cross_entropy): 0.00297, Running average loss:0.00267, Time taken: 0:00:15.675306 ETA: 0:25:36.179996 2022-07-31 15:58:56,618 [INFO] __main__: Epoch: 1002/1100:, Cur-Step: 20050, loss(cross_entropy): 0.00297, Running average loss:0.00267, Time taken: 0:00:15.675306 ETA: 0:25:36.179996 Epoch: 1003/1100:, Cur-Step: 20060, loss(cross_entropy): 0.00241, Running average loss:0.00241, Time taken: 0:00:15.435985 ETA: 0:24:57.290530 2022-07-31 15:59:04,056 [INFO] __main__: Epoch: 1003/1100:, Cur-Step: 20060, loss(cross_entropy): 0.00241, Running average loss:0.00241, Time taken: 0:00:15.435985 ETA: 0:24:57.290530 Epoch: 1003/1100:, Cur-Step: 20070, loss(cross_entropy): 0.00232, Running average loss:0.00239, Time taken: 0:00:15.435985 ETA: 0:24:57.290530 2022-07-31 15:59:11,324 [INFO] __main__: Epoch: 1003/1100:, Cur-Step: 20070, loss(cross_entropy): 0.00232, Running average loss:0.00239, Time taken: 0:00:15.435985 ETA: 0:24:57.290530 Epoch: 1004/1100:, Cur-Step: 20080, loss(cross_entropy): 0.00235, Running average loss:0.00235, Time taken: 0:00:15.395547 ETA: 0:24:37.972504 2022-07-31 15:59:18,687 [INFO] __main__: Epoch: 1004/1100:, Cur-Step: 20080, loss(cross_entropy): 0.00235, Running average loss:0.00235, Time taken: 0:00:15.395547 ETA: 0:24:37.972504 Epoch: 1004/1100:, Cur-Step: 20090, loss(cross_entropy): 0.00227, Running average loss:0.00230, Time taken: 0:00:15.395547 ETA: 0:24:37.972504 2022-07-31 15:59:26,105 [INFO] __main__: Epoch: 1004/1100:, Cur-Step: 20090, loss(cross_entropy): 0.00227, Running average loss:0.00230, Time taken: 0:00:15.395547 ETA: 0:24:37.972504 Epoch: 1005/1100:, Cur-Step: 20100, loss(cross_entropy): 0.00235, Running average loss:0.00235, Time taken: 0:00:15.569257 ETA: 0:24:39.079394 2022-07-31 15:59:33,574 [INFO] __main__: Epoch: 1005/1100:, Cur-Step: 20100, loss(cross_entropy): 0.00235, Running average loss:0.00235, Time taken: 0:00:15.569257 ETA: 0:24:39.079394 Epoch: 1005/1100:, Cur-Step: 20110, loss(cross_entropy): 0.00229, Running average loss:0.00227, Time taken: 0:00:15.569257 ETA: 0:24:39.079394 2022-07-31 15:59:40,765 [INFO] __main__: Epoch: 1005/1100:, Cur-Step: 20110, loss(cross_entropy): 0.00229, Running average loss:0.00227, Time taken: 0:00:15.569257 ETA: 0:24:39.079394 Epoch: 1006/1100:, Cur-Step: 20120, loss(cross_entropy): 0.00237, Running average loss:0.00237, Time taken: 0:00:15.401328 ETA: 0:24:07.724863 2022-07-31 15:59:48,271 [INFO] __main__: Epoch: 1006/1100:, Cur-Step: 20120, loss(cross_entropy): 0.00237, Running average loss:0.00237, Time taken: 0:00:15.401328 ETA: 0:24:07.724863 Epoch: 1006/1100:, Cur-Step: 20130, loss(cross_entropy): 0.00239, Running average loss:0.00233, Time taken: 0:00:15.401328 ETA: 0:24:07.724863 2022-07-31 15:59:55,508 [INFO] __main__: Epoch: 1006/1100:, Cur-Step: 20130, loss(cross_entropy): 0.00239, Running average loss:0.00233, Time taken: 0:00:15.401328 ETA: 0:24:07.724863 Epoch: 1007/1100:, Cur-Step: 20140, loss(cross_entropy): 0.00229, Running average loss:0.00229, Time taken: 0:00:15.374355 ETA: 0:23:49.814978 2022-07-31 16:00:02,858 [INFO] __main__: Epoch: 1007/1100:, Cur-Step: 20140, loss(cross_entropy): 0.00229, Running average loss:0.00229, Time taken: 0:00:15.374355 ETA: 0:23:49.814978 Epoch: 1007/1100:, Cur-Step: 20150, loss(cross_entropy): 0.00253, Running average loss:0.00241, Time taken: 0:00:15.374355 ETA: 0:23:49.814978 2022-07-31 16:00:10,156 [INFO] __main__: Epoch: 1007/1100:, Cur-Step: 20150, loss(cross_entropy): 0.00253, Running average loss:0.00241, Time taken: 0:00:15.374355 ETA: 0:23:49.814978 Epoch: 1008/1100:, Cur-Step: 20160, loss(cross_entropy): 0.00249, Running average loss:0.00249, Time taken: 0:00:15.473591 ETA: 0:23:43.570336 2022-07-31 16:00:17,614 [INFO] __main__: Epoch: 1008/1100:, Cur-Step: 20160, loss(cross_entropy): 0.00249, Running average loss:0.00249, Time taken: 0:00:15.473591 ETA: 0:23:43.570336 Epoch: 1008/1100:, Cur-Step: 20170, loss(cross_entropy): 0.00269, Running average loss:0.00254, Time taken: 0:00:15.473591 ETA: 0:23:43.570336 2022-07-31 16:00:24,898 [INFO] __main__: Epoch: 1008/1100:, Cur-Step: 20170, loss(cross_entropy): 0.00269, Running average loss:0.00254, Time taken: 0:00:15.473591 ETA: 0:23:43.570336 Epoch: 1009/1100:, Cur-Step: 20180, loss(cross_entropy): 0.00242, Running average loss:0.00242, Time taken: 0:00:15.408648 ETA: 0:23:22.186926 2022-07-31 16:00:32,276 [INFO] __main__: Epoch: 1009/1100:, Cur-Step: 20180, loss(cross_entropy): 0.00242, Running average loss:0.00242, Time taken: 0:00:15.408648 ETA: 0:23:22.186926 Epoch: 1009/1100:, Cur-Step: 20190, loss(cross_entropy): 0.00251, Running average loss:0.00265, Time taken: 0:00:15.408648 ETA: 0:23:22.186926 2022-07-31 16:00:39,559 [INFO] __main__: Epoch: 1009/1100:, Cur-Step: 20190, loss(cross_entropy): 0.00251, Running average loss:0.00265, Time taken: 0:00:15.408648 ETA: 0:23:22.186926 Epoch: 1010/1100:, Cur-Step: 20200, loss(cross_entropy): 0.00253, Running average loss:0.00253, Time taken: 0:00:15.432939 ETA: 0:23:08.964493 2022-07-31 16:00:46,992 [INFO] __main__: Epoch: 1010/1100:, Cur-Step: 20200, loss(cross_entropy): 0.00253, Running average loss:0.00253, Time taken: 0:00:15.432939 ETA: 0:23:08.964493 Epoch: 1010/1100:, Cur-Step: 20210, loss(cross_entropy): 0.00287, Running average loss:0.00268, Time taken: 0:00:15.432939 ETA: 0:23:08.964493 2022-07-31 16:00:54,300 [INFO] __main__: Epoch: 1010/1100:, Cur-Step: 20210, loss(cross_entropy): 0.00287, Running average loss:0.00268, Time taken: 0:00:15.432939 ETA: 0:23:08.964493 Epoch: 1011/1100:, Cur-Step: 20220, loss(cross_entropy): 0.00284, Running average loss:0.00284, Time taken: 0:00:15.292622 ETA: 0:22:41.043387 2022-07-31 16:01:01,550 [INFO] __main__: Epoch: 1011/1100:, Cur-Step: 20220, loss(cross_entropy): 0.00284, Running average loss:0.00284, Time taken: 0:00:15.292622 ETA: 0:22:41.043387 Epoch: 1011/1100:, Cur-Step: 20230, loss(cross_entropy): 0.00265, Running average loss:0.00276, Time taken: 0:00:15.292622 ETA: 0:22:41.043387 2022-07-31 16:01:08,718 [INFO] __main__: Epoch: 1011/1100:, Cur-Step: 20230, loss(cross_entropy): 0.00265, Running average loss:0.00276, Time taken: 0:00:15.292622 ETA: 0:22:41.043387 Epoch: 1012/1100:, Cur-Step: 20240, loss(cross_entropy): 0.00282, Running average loss:0.00282, Time taken: 0:00:15.244866 ETA: 0:22:21.548178 2022-07-31 16:01:16,061 [INFO] __main__: Epoch: 1012/1100:, Cur-Step: 20240, loss(cross_entropy): 0.00282, Running average loss:0.00282, Time taken: 0:00:15.244866 ETA: 0:22:21.548178 Epoch: 1012/1100:, Cur-Step: 20250, loss(cross_entropy): 0.00281, Running average loss:0.00272, Time taken: 0:00:15.244866 ETA: 0:22:21.548178 2022-07-31 16:01:23,342 [INFO] __main__: Epoch: 1012/1100:, Cur-Step: 20250, loss(cross_entropy): 0.00281, Running average loss:0.00272, Time taken: 0:00:15.244866 ETA: 0:22:21.548178 Epoch: 1013/1100:, Cur-Step: 20260, loss(cross_entropy): 0.00256, Running average loss:0.00256, Time taken: 0:00:15.380019 ETA: 0:22:18.061690 2022-07-31 16:01:30,651 [INFO] __main__: Epoch: 1013/1100:, Cur-Step: 20260, loss(cross_entropy): 0.00256, Running average loss:0.00256, Time taken: 0:00:15.380019 ETA: 0:22:18.061690 Epoch: 1013/1100:, Cur-Step: 20270, loss(cross_entropy): 0.00266, Running average loss:0.00274, Time taken: 0:00:15.380019 ETA: 0:22:18.061690 2022-07-31 16:01:37,839 [INFO] __main__: Epoch: 1013/1100:, Cur-Step: 20270, loss(cross_entropy): 0.00266, Running average loss:0.00274, Time taken: 0:00:15.380019 ETA: 0:22:18.061690 Epoch: 1014/1100:, Cur-Step: 20280, loss(cross_entropy): 0.00261, Running average loss:0.00261, Time taken: 0:00:15.214819 ETA: 0:21:48.474471 2022-07-31 16:01:45,171 [INFO] __main__: Epoch: 1014/1100:, Cur-Step: 20280, loss(cross_entropy): 0.00261, Running average loss:0.00261, Time taken: 0:00:15.214819 ETA: 0:21:48.474471 Epoch: 1014/1100:, Cur-Step: 20290, loss(cross_entropy): 0.00237, Running average loss:0.00264, Time taken: 0:00:15.214819 ETA: 0:21:48.474471 2022-07-31 16:01:52,558 [INFO] __main__: Epoch: 1014/1100:, Cur-Step: 20290, loss(cross_entropy): 0.00237, Running average loss:0.00264, Time taken: 0:00:15.214819 ETA: 0:21:48.474471 Epoch: 1015/1100:, Cur-Step: 20300, loss(cross_entropy): 0.00259, Running average loss:0.00259, Time taken: 0:00:15.353643 ETA: 0:21:45.059690 2022-07-31 16:01:59,813 [INFO] __main__: Epoch: 1015/1100:, Cur-Step: 20300, loss(cross_entropy): 0.00259, Running average loss:0.00259, Time taken: 0:00:15.353643 ETA: 0:21:45.059690 Epoch: 1015/1100:, Cur-Step: 20310, loss(cross_entropy): 0.00245, Running average loss:0.00257, Time taken: 0:00:15.353643 ETA: 0:21:45.059690 2022-07-31 16:02:07,082 [INFO] __main__: Epoch: 1015/1100:, Cur-Step: 20310, loss(cross_entropy): 0.00245, Running average loss:0.00257, Time taken: 0:00:15.353643 ETA: 0:21:45.059690 Epoch: 1016/1100:, Cur-Step: 20320, loss(cross_entropy): 0.00271, Running average loss:0.00271, Time taken: 0:00:15.202587 ETA: 0:21:17.017279 2022-07-31 16:02:14,285 [INFO] __main__: Epoch: 1016/1100:, Cur-Step: 20320, loss(cross_entropy): 0.00271, Running average loss:0.00271, Time taken: 0:00:15.202587 ETA: 0:21:17.017279 Epoch: 1016/1100:, Cur-Step: 20330, loss(cross_entropy): 0.00263, Running average loss:0.00257, Time taken: 0:00:15.202587 ETA: 0:21:17.017279 2022-07-31 16:02:21,736 [INFO] __main__: Epoch: 1016/1100:, Cur-Step: 20330, loss(cross_entropy): 0.00263, Running average loss:0.00257, Time taken: 0:00:15.202587 ETA: 0:21:17.017279 Epoch: 1017/1100:, Cur-Step: 20340, loss(cross_entropy): 0.00285, Running average loss:0.00285, Time taken: 0:00:15.379076 ETA: 0:21:16.463289 2022-07-31 16:02:28,986 [INFO] __main__: Epoch: 1017/1100:, Cur-Step: 20340, loss(cross_entropy): 0.00285, Running average loss:0.00285, Time taken: 0:00:15.379076 ETA: 0:21:16.463289 Epoch: 1017/1100:, Cur-Step: 20350, loss(cross_entropy): 0.00276, Running average loss:0.00270, Time taken: 0:00:15.379076 ETA: 0:21:16.463289 2022-07-31 16:02:36,242 [INFO] __main__: Epoch: 1017/1100:, Cur-Step: 20350, loss(cross_entropy): 0.00276, Running average loss:0.00270, Time taken: 0:00:15.379076 ETA: 0:21:16.463289 Epoch: 1018/1100:, Cur-Step: 20360, loss(cross_entropy): 0.00269, Running average loss:0.00269, Time taken: 0:00:15.194410 ETA: 0:20:45.941627 2022-07-31 16:02:43,462 [INFO] __main__: Epoch: 1018/1100:, Cur-Step: 20360, loss(cross_entropy): 0.00269, Running average loss:0.00269, Time taken: 0:00:15.194410 ETA: 0:20:45.941627 Epoch: 1018/1100:, Cur-Step: 20370, loss(cross_entropy): 0.00259, Running average loss:0.00275, Time taken: 0:00:15.194410 ETA: 0:20:45.941627 2022-07-31 16:02:50,890 [INFO] __main__: Epoch: 1018/1100:, Cur-Step: 20370, loss(cross_entropy): 0.00259, Running average loss:0.00275, Time taken: 0:00:15.194410 ETA: 0:20:45.941627 Epoch: 1019/1100:, Cur-Step: 20380, loss(cross_entropy): 0.00268, Running average loss:0.00268, Time taken: 0:00:15.420049 ETA: 0:20:49.023984 2022-07-31 16:02:58,090 [INFO] __main__: Epoch: 1019/1100:, Cur-Step: 20380, loss(cross_entropy): 0.00268, Running average loss:0.00268, Time taken: 0:00:15.420049 ETA: 0:20:49.023984 Epoch: 1019/1100:, Cur-Step: 20390, loss(cross_entropy): 0.00264, Running average loss:0.00273, Time taken: 0:00:15.420049 ETA: 0:20:49.023984 2022-07-31 16:03:05,362 [INFO] __main__: Epoch: 1019/1100:, Cur-Step: 20390, loss(cross_entropy): 0.00264, Running average loss:0.00273, Time taken: 0:00:15.420049 ETA: 0:20:49.023984 Epoch: 1020/1100:, Cur-Step: 20400, loss(cross_entropy): 0.00267, Running average loss:0.00267, Time taken: 0:00:15.187896 ETA: 0:20:15.031719 2022-07-31 16:03:12,573 [INFO] __main__: Epoch: 1020/1100:, Cur-Step: 20400, loss(cross_entropy): 0.00267, Running average loss:0.00267, Time taken: 0:00:15.187896 ETA: 0:20:15.031719 Epoch: 1020/1100:, Cur-Step: 20410, loss(cross_entropy): 0.00251, Running average loss:0.00290, Time taken: 0:00:15.187896 ETA: 0:20:15.031719 2022-07-31 16:03:19,929 [INFO] __main__: Epoch: 1020/1100:, Cur-Step: 20410, loss(cross_entropy): 0.00251, Running average loss:0.00290, Time taken: 0:00:15.187896 ETA: 0:20:15.031719 Epoch: 1021/1100:, Cur-Step: 20420, loss(cross_entropy): 0.00278, Running average loss:0.00278, Time taken: 0:00:15.316114 ETA: 0:20:09.972983 2022-07-31 16:03:27,164 [INFO] __main__: Epoch: 1021/1100:, Cur-Step: 20420, loss(cross_entropy): 0.00278, Running average loss:0.00278, Time taken: 0:00:15.316114 ETA: 0:20:09.972983 Epoch: 1021/1100:, Cur-Step: 20430, loss(cross_entropy): 0.00314, Running average loss:0.00298, Time taken: 0:00:15.316114 ETA: 0:20:09.972983 2022-07-31 16:03:34,480 [INFO] __main__: Epoch: 1021/1100:, Cur-Step: 20430, loss(cross_entropy): 0.00314, Running average loss:0.00298, Time taken: 0:00:15.316114 ETA: 0:20:09.972983 Epoch: 1022/1100:, Cur-Step: 20440, loss(cross_entropy): 0.00327, Running average loss:0.00327, Time taken: 0:00:15.240322 ETA: 0:19:48.745088 2022-07-31 16:03:41,701 [INFO] __main__: Epoch: 1022/1100:, Cur-Step: 20440, loss(cross_entropy): 0.00327, Running average loss:0.00327, Time taken: 0:00:15.240322 ETA: 0:19:48.745088 Epoch: 1022/1100:, Cur-Step: 20450, loss(cross_entropy): 0.00290, Running average loss:0.00284, Time taken: 0:00:15.240322 ETA: 0:19:48.745088 2022-07-31 16:03:49,153 [INFO] __main__: Epoch: 1022/1100:, Cur-Step: 20450, loss(cross_entropy): 0.00290, Running average loss:0.00284, Time taken: 0:00:15.240322 ETA: 0:19:48.745088 Epoch: 1023/1100:, Cur-Step: 20460, loss(cross_entropy): 0.00280, Running average loss:0.00280, Time taken: 0:00:15.493762 ETA: 0:19:53.019675 2022-07-31 16:03:56,521 [INFO] __main__: Epoch: 1023/1100:, Cur-Step: 20460, loss(cross_entropy): 0.00280, Running average loss:0.00280, Time taken: 0:00:15.493762 ETA: 0:19:53.019675 Epoch: 1023/1100:, Cur-Step: 20470, loss(cross_entropy): 0.00256, Running average loss:0.00269, Time taken: 0:00:15.493762 ETA: 0:19:53.019675 2022-07-31 16:04:03,770 [INFO] __main__: Epoch: 1023/1100:, Cur-Step: 20470, loss(cross_entropy): 0.00256, Running average loss:0.00269, Time taken: 0:00:15.493762 ETA: 0:19:53.019675 Epoch: 1024/1100:, Cur-Step: 20480, loss(cross_entropy): 0.00247, Running average loss:0.00247, Time taken: 0:00:15.192507 ETA: 0:19:14.630534 2022-07-31 16:04:10,953 [INFO] __main__: Epoch: 1024/1100:, Cur-Step: 20480, loss(cross_entropy): 0.00247, Running average loss:0.00247, Time taken: 0:00:15.192507 ETA: 0:19:14.630534 Epoch: 1024/1100:, Cur-Step: 20490, loss(cross_entropy): 0.00243, Running average loss:0.00250, Time taken: 0:00:15.192507 ETA: 0:19:14.630534 2022-07-31 16:04:18,430 [INFO] __main__: Epoch: 1024/1100:, Cur-Step: 20490, loss(cross_entropy): 0.00243, Running average loss:0.00250, Time taken: 0:00:15.192507 ETA: 0:19:14.630534 Epoch: 1025/1100:, Cur-Step: 20500, loss(cross_entropy): 0.00290, Running average loss:0.00290, Time taken: 0:00:15.579433 ETA: 0:19:28.457490 2022-07-31 16:04:25,784 [INFO] __main__: Epoch: 1025/1100:, Cur-Step: 20500, loss(cross_entropy): 0.00290, Running average loss:0.00290, Time taken: 0:00:15.579433 ETA: 0:19:28.457490 Epoch: 1025/1100:, Cur-Step: 20510, loss(cross_entropy): 0.00244, Running average loss:0.00258, Time taken: 0:00:15.579433 ETA: 0:19:28.457490 2022-07-31 16:04:33,192 [INFO] __main__: Epoch: 1025/1100:, Cur-Step: 20510, loss(cross_entropy): 0.00244, Running average loss:0.00258, Time taken: 0:00:15.579433 ETA: 0:19:28.457490 Epoch: 1026/1100:, Cur-Step: 20520, loss(cross_entropy): 0.00261, Running average loss:0.00261, Time taken: 0:00:15.361898 ETA: 0:18:56.780483 2022-07-31 16:04:40,486 [INFO] __main__: Epoch: 1026/1100:, Cur-Step: 20520, loss(cross_entropy): 0.00261, Running average loss:0.00261, Time taken: 0:00:15.361898 ETA: 0:18:56.780483 Epoch: 1026/1100:, Cur-Step: 20530, loss(cross_entropy): 0.00268, Running average loss:0.00255, Time taken: 0:00:15.361898 ETA: 0:18:56.780483 2022-07-31 16:04:47,942 [INFO] __main__: Epoch: 1026/1100:, Cur-Step: 20530, loss(cross_entropy): 0.00268, Running average loss:0.00255, Time taken: 0:00:15.361898 ETA: 0:18:56.780483 Epoch: 1027/1100:, Cur-Step: 20540, loss(cross_entropy): 0.00273, Running average loss:0.00273, Time taken: 0:00:15.631758 ETA: 0:19:01.118350 2022-07-31 16:04:55,382 [INFO] __main__: Epoch: 1027/1100:, Cur-Step: 20540, loss(cross_entropy): 0.00273, Running average loss:0.00273, Time taken: 0:00:15.631758 ETA: 0:19:01.118350 Epoch: 1027/1100:, Cur-Step: 20550, loss(cross_entropy): 0.00290, Running average loss:0.00271, Time taken: 0:00:15.631758 ETA: 0:19:01.118350 2022-07-31 16:05:02,711 [INFO] __main__: Epoch: 1027/1100:, Cur-Step: 20550, loss(cross_entropy): 0.00290, Running average loss:0.00271, Time taken: 0:00:15.631758 ETA: 0:19:01.118350 Epoch: 1028/1100:, Cur-Step: 20560, loss(cross_entropy): 0.00323, Running average loss:0.00323, Time taken: 0:00:15.276570 ETA: 0:18:19.913029 2022-07-31 16:05:09,955 [INFO] __main__: Epoch: 1028/1100:, Cur-Step: 20560, loss(cross_entropy): 0.00323, Running average loss:0.00323, Time taken: 0:00:15.276570 ETA: 0:18:19.913029 Epoch: 1028/1100:, Cur-Step: 20570, loss(cross_entropy): 0.00288, Running average loss:0.00276, Time taken: 0:00:15.276570 ETA: 0:18:19.913029 2022-07-31 16:05:17,312 [INFO] __main__: Epoch: 1028/1100:, Cur-Step: 20570, loss(cross_entropy): 0.00288, Running average loss:0.00276, Time taken: 0:00:15.276570 ETA: 0:18:19.913029 Epoch: 1029/1100:, Cur-Step: 20580, loss(cross_entropy): 0.00313, Running average loss:0.00313, Time taken: 0:00:15.401511 ETA: 0:18:13.507295 2022-07-31 16:05:24,633 [INFO] __main__: Epoch: 1029/1100:, Cur-Step: 20580, loss(cross_entropy): 0.00313, Running average loss:0.00313, Time taken: 0:00:15.401511 ETA: 0:18:13.507295 Epoch: 1029/1100:, Cur-Step: 20590, loss(cross_entropy): 0.00330, Running average loss:0.00343, Time taken: 0:00:15.401511 ETA: 0:18:13.507295 2022-07-31 16:05:31,911 [INFO] __main__: Epoch: 1029/1100:, Cur-Step: 20590, loss(cross_entropy): 0.00330, Running average loss:0.00343, Time taken: 0:00:15.401511 ETA: 0:18:13.507295 Epoch: 1030/1100:, Cur-Step: 20600, loss(cross_entropy): 0.00466, Running average loss:0.00466, Time taken: 0:00:15.253258 ETA: 0:17:47.728043 2022-07-31 16:05:39,108 [INFO] __main__: Epoch: 1030/1100:, Cur-Step: 20600, loss(cross_entropy): 0.00466, Running average loss:0.00466, Time taken: 0:00:15.253258 ETA: 0:17:47.728043 Epoch: 1030/1100:, Cur-Step: 20610, loss(cross_entropy): 0.00375, Running average loss:0.00398, Time taken: 0:00:15.253258 ETA: 0:17:47.728043 2022-07-31 16:05:46,355 [INFO] __main__: Epoch: 1030/1100:, Cur-Step: 20610, loss(cross_entropy): 0.00375, Running average loss:0.00398, Time taken: 0:00:15.253258 ETA: 0:17:47.728043 Epoch: 1031/1100:, Cur-Step: 20620, loss(cross_entropy): 0.00402, Running average loss:0.00402, Time taken: 0:00:15.333652 ETA: 0:17:38.022006 2022-07-31 16:05:53,740 [INFO] __main__: Epoch: 1031/1100:, Cur-Step: 20620, loss(cross_entropy): 0.00402, Running average loss:0.00402, Time taken: 0:00:15.333652 ETA: 0:17:38.022006 Epoch: 1031/1100:, Cur-Step: 20630, loss(cross_entropy): 0.00306, Running average loss:0.00337, Time taken: 0:00:15.333652 ETA: 0:17:38.022006 2022-07-31 16:06:00,968 [INFO] __main__: Epoch: 1031/1100:, Cur-Step: 20630, loss(cross_entropy): 0.00306, Running average loss:0.00337, Time taken: 0:00:15.333652 ETA: 0:17:38.022006 Epoch: 1032/1100:, Cur-Step: 20640, loss(cross_entropy): 0.00305, Running average loss:0.00305, Time taken: 0:00:15.179921 ETA: 0:17:12.234654 2022-07-31 16:06:08,222 [INFO] __main__: Epoch: 1032/1100:, Cur-Step: 20640, loss(cross_entropy): 0.00305, Running average loss:0.00305, Time taken: 0:00:15.179921 ETA: 0:17:12.234654 Epoch: 1032/1100:, Cur-Step: 20650, loss(cross_entropy): 0.00316, Running average loss:0.00307, Time taken: 0:00:15.179921 ETA: 0:17:12.234654 2022-07-31 16:06:15,484 [INFO] __main__: Epoch: 1032/1100:, Cur-Step: 20650, loss(cross_entropy): 0.00316, Running average loss:0.00307, Time taken: 0:00:15.179921 ETA: 0:17:12.234654 Epoch: 1033/1100:, Cur-Step: 20660, loss(cross_entropy): 0.00286, Running average loss:0.00286, Time taken: 0:00:15.253378 ETA: 0:17:01.976336 2022-07-31 16:06:22,772 [INFO] __main__: Epoch: 1033/1100:, Cur-Step: 20660, loss(cross_entropy): 0.00286, Running average loss:0.00286, Time taken: 0:00:15.253378 ETA: 0:17:01.976336 Epoch: 1033/1100:, Cur-Step: 20670, loss(cross_entropy): 0.00255, Running average loss:0.00268, Time taken: 0:00:15.253378 ETA: 0:17:01.976336 2022-07-31 16:06:30,163 [INFO] __main__: Epoch: 1033/1100:, Cur-Step: 20670, loss(cross_entropy): 0.00255, Running average loss:0.00268, Time taken: 0:00:15.253378 ETA: 0:17:01.976336 Epoch: 1034/1100:, Cur-Step: 20680, loss(cross_entropy): 0.00261, Running average loss:0.00261, Time taken: 0:00:15.425831 ETA: 0:16:58.104836 2022-07-31 16:06:37,555 [INFO] __main__: Epoch: 1034/1100:, Cur-Step: 20680, loss(cross_entropy): 0.00261, Running average loss:0.00261, Time taken: 0:00:15.425831 ETA: 0:16:58.104836 Epoch: 1034/1100:, Cur-Step: 20690, loss(cross_entropy): 0.00241, Running average loss:0.00247, Time taken: 0:00:15.425831 ETA: 0:16:58.104836 2022-07-31 16:06:44,737 [INFO] __main__: Epoch: 1034/1100:, Cur-Step: 20690, loss(cross_entropy): 0.00241, Running average loss:0.00247, Time taken: 0:00:15.425831 ETA: 0:16:58.104836 Epoch: 1035/1100:, Cur-Step: 20700, loss(cross_entropy): 0.00232, Running average loss:0.00232, Time taken: 0:00:15.259528 ETA: 0:16:31.869315 2022-07-31 16:06:52,042 [INFO] __main__: Epoch: 1035/1100:, Cur-Step: 20700, loss(cross_entropy): 0.00232, Running average loss:0.00232, Time taken: 0:00:15.259528 ETA: 0:16:31.869315 Epoch: 1035/1100:, Cur-Step: 20710, loss(cross_entropy): 0.00228, Running average loss:0.00239, Time taken: 0:00:15.259528 ETA: 0:16:31.869315 2022-07-31 16:06:59,322 [INFO] __main__: Epoch: 1035/1100:, Cur-Step: 20710, loss(cross_entropy): 0.00228, Running average loss:0.00239, Time taken: 0:00:15.259528 ETA: 0:16:31.869315 Epoch: 1036/1100:, Cur-Step: 20720, loss(cross_entropy): 0.00229, Running average loss:0.00229, Time taken: 0:00:15.342098 ETA: 0:16:21.894287 2022-07-31 16:07:06,548 [INFO] __main__: Epoch: 1036/1100:, Cur-Step: 20720, loss(cross_entropy): 0.00229, Running average loss:0.00229, Time taken: 0:00:15.342098 ETA: 0:16:21.894287 Epoch: 1036/1100:, Cur-Step: 20730, loss(cross_entropy): 0.00227, Running average loss:0.00234, Time taken: 0:00:15.342098 ETA: 0:16:21.894287 2022-07-31 16:07:13,710 [INFO] __main__: Epoch: 1036/1100:, Cur-Step: 20730, loss(cross_entropy): 0.00227, Running average loss:0.00234, Time taken: 0:00:15.342098 ETA: 0:16:21.894287 Epoch: 1037/1100:, Cur-Step: 20740, loss(cross_entropy): 0.00231, Running average loss:0.00231, Time taken: 0:00:15.228203 ETA: 0:15:59.376763 2022-07-31 16:07:21,022 [INFO] __main__: Epoch: 1037/1100:, Cur-Step: 20740, loss(cross_entropy): 0.00231, Running average loss:0.00231, Time taken: 0:00:15.228203 ETA: 0:15:59.376763 Epoch: 1037/1100:, Cur-Step: 20750, loss(cross_entropy): 0.00231, Running average loss:0.00234, Time taken: 0:00:15.228203 ETA: 0:15:59.376763 2022-07-31 16:07:28,255 [INFO] __main__: Epoch: 1037/1100:, Cur-Step: 20750, loss(cross_entropy): 0.00231, Running average loss:0.00234, Time taken: 0:00:15.228203 ETA: 0:15:59.376763 Epoch: 1038/1100:, Cur-Step: 20760, loss(cross_entropy): 0.00238, Running average loss:0.00238, Time taken: 0:00:15.265332 ETA: 0:15:46.450553 2022-07-31 16:07:35,482 [INFO] __main__: Epoch: 1038/1100:, Cur-Step: 20760, loss(cross_entropy): 0.00238, Running average loss:0.00238, Time taken: 0:00:15.265332 ETA: 0:15:46.450553 Epoch: 1038/1100:, Cur-Step: 20770, loss(cross_entropy): 0.00236, Running average loss:0.00237, Time taken: 0:00:15.265332 ETA: 0:15:46.450553 2022-07-31 16:07:42,835 [INFO] __main__: Epoch: 1038/1100:, Cur-Step: 20770, loss(cross_entropy): 0.00236, Running average loss:0.00237, Time taken: 0:00:15.265332 ETA: 0:15:46.450553 Epoch: 1039/1100:, Cur-Step: 20780, loss(cross_entropy): 0.00243, Running average loss:0.00243, Time taken: 0:00:15.339843 ETA: 0:15:35.730411 2022-07-31 16:07:50,108 [INFO] __main__: Epoch: 1039/1100:, Cur-Step: 20780, loss(cross_entropy): 0.00243, Running average loss:0.00243, Time taken: 0:00:15.339843 ETA: 0:15:35.730411 Epoch: 1039/1100:, Cur-Step: 20790, loss(cross_entropy): 0.00235, Running average loss:0.00240, Time taken: 0:00:15.339843 ETA: 0:15:35.730411 2022-07-31 16:07:57,488 [INFO] __main__: Epoch: 1039/1100:, Cur-Step: 20790, loss(cross_entropy): 0.00235, Running average loss:0.00240, Time taken: 0:00:15.339843 ETA: 0:15:35.730411 Epoch: 1040/1100:, Cur-Step: 20800, loss(cross_entropy): 0.00254, Running average loss:0.00254, Time taken: 0:00:15.496322 ETA: 0:15:29.779329 2022-07-31 16:08:04,890 [INFO] __main__: Epoch: 1040/1100:, Cur-Step: 20800, loss(cross_entropy): 0.00254, Running average loss:0.00254, Time taken: 0:00:15.496322 ETA: 0:15:29.779329 Epoch: 1040/1100:, Cur-Step: 20810, loss(cross_entropy): 0.00246, Running average loss:0.00245, Time taken: 0:00:15.496322 ETA: 0:15:29.779329 2022-07-31 16:08:12,097 [INFO] __main__: Epoch: 1040/1100:, Cur-Step: 20810, loss(cross_entropy): 0.00246, Running average loss:0.00245, Time taken: 0:00:15.496322 ETA: 0:15:29.779329 Epoch: 1041/1100:, Cur-Step: 20820, loss(cross_entropy): 0.00245, Running average loss:0.00245, Time taken: 0:00:15.237987 ETA: 0:14:59.041221 2022-07-31 16:08:19,378 [INFO] __main__: Epoch: 1041/1100:, Cur-Step: 20820, loss(cross_entropy): 0.00245, Running average loss:0.00245, Time taken: 0:00:15.237987 ETA: 0:14:59.041221 Epoch: 1041/1100:, Cur-Step: 20830, loss(cross_entropy): 0.00237, Running average loss:0.00255, Time taken: 0:00:15.237987 ETA: 0:14:59.041221 2022-07-31 16:08:26,747 [INFO] __main__: Epoch: 1041/1100:, Cur-Step: 20830, loss(cross_entropy): 0.00237, Running average loss:0.00255, Time taken: 0:00:15.237987 ETA: 0:14:59.041221 Epoch: 1042/1100:, Cur-Step: 20840, loss(cross_entropy): 0.00233, Running average loss:0.00233, Time taken: 0:00:15.322342 ETA: 0:14:48.695817 2022-07-31 16:08:34,015 [INFO] __main__: Epoch: 1042/1100:, Cur-Step: 20840, loss(cross_entropy): 0.00233, Running average loss:0.00233, Time taken: 0:00:15.322342 ETA: 0:14:48.695817 Epoch: 1042/1100:, Cur-Step: 20850, loss(cross_entropy): 0.00239, Running average loss:0.00248, Time taken: 0:00:15.322342 ETA: 0:14:48.695817 2022-07-31 16:08:41,283 [INFO] __main__: Epoch: 1042/1100:, Cur-Step: 20850, loss(cross_entropy): 0.00239, Running average loss:0.00248, Time taken: 0:00:15.322342 ETA: 0:14:48.695817 Epoch: 1043/1100:, Cur-Step: 20860, loss(cross_entropy): 0.00229, Running average loss:0.00229, Time taken: 0:00:15.285629 ETA: 0:14:31.280869 2022-07-31 16:08:48,556 [INFO] __main__: Epoch: 1043/1100:, Cur-Step: 20860, loss(cross_entropy): 0.00229, Running average loss:0.00229, Time taken: 0:00:15.285629 ETA: 0:14:31.280869 Epoch: 1043/1100:, Cur-Step: 20870, loss(cross_entropy): 0.00232, Running average loss:0.00243, Time taken: 0:00:15.285629 ETA: 0:14:31.280869 2022-07-31 16:08:55,825 [INFO] __main__: Epoch: 1043/1100:, Cur-Step: 20870, loss(cross_entropy): 0.00232, Running average loss:0.00243, Time taken: 0:00:15.285629 ETA: 0:14:31.280869 Epoch: 1044/1100:, Cur-Step: 20880, loss(cross_entropy): 0.00240, Running average loss:0.00240, Time taken: 0:00:15.333117 ETA: 0:14:18.654552 2022-07-31 16:09:03,156 [INFO] __main__: Epoch: 1044/1100:, Cur-Step: 20880, loss(cross_entropy): 0.00240, Running average loss:0.00240, Time taken: 0:00:15.333117 ETA: 0:14:18.654552 Epoch: 1044/1100:, Cur-Step: 20890, loss(cross_entropy): 0.00236, Running average loss:0.00243, Time taken: 0:00:15.333117 ETA: 0:14:18.654552 2022-07-31 16:09:10,450 [INFO] __main__: Epoch: 1044/1100:, Cur-Step: 20890, loss(cross_entropy): 0.00236, Running average loss:0.00243, Time taken: 0:00:15.333117 ETA: 0:14:18.654552 Epoch: 1045/1100:, Cur-Step: 20900, loss(cross_entropy): 0.00226, Running average loss:0.00226, Time taken: 0:00:15.264098 ETA: 0:13:59.525399 2022-07-31 16:09:17,702 [INFO] __main__: Epoch: 1045/1100:, Cur-Step: 20900, loss(cross_entropy): 0.00226, Running average loss:0.00226, Time taken: 0:00:15.264098 ETA: 0:13:59.525399 Epoch: 1045/1100:, Cur-Step: 20910, loss(cross_entropy): 0.00257, Running average loss:0.00256, Time taken: 0:00:15.264098 ETA: 0:13:59.525399 2022-07-31 16:09:24,948 [INFO] __main__: Epoch: 1045/1100:, Cur-Step: 20910, loss(cross_entropy): 0.00257, Running average loss:0.00256, Time taken: 0:00:15.264098 ETA: 0:13:59.525399 Epoch: 1046/1100:, Cur-Step: 20920, loss(cross_entropy): 0.00255, Running average loss:0.00255, Time taken: 0:00:15.295828 ETA: 0:13:45.974718 2022-07-31 16:09:32,330 [INFO] __main__: Epoch: 1046/1100:, Cur-Step: 20920, loss(cross_entropy): 0.00255, Running average loss:0.00255, Time taken: 0:00:15.295828 ETA: 0:13:45.974718 Epoch: 1046/1100:, Cur-Step: 20930, loss(cross_entropy): 0.00240, Running average loss:0.00269, Time taken: 0:00:15.295828 ETA: 0:13:45.974718 2022-07-31 16:09:39,486 [INFO] __main__: Epoch: 1046/1100:, Cur-Step: 20930, loss(cross_entropy): 0.00240, Running average loss:0.00269, Time taken: 0:00:15.295828 ETA: 0:13:45.974718 Epoch: 1047/1100:, Cur-Step: 20940, loss(cross_entropy): 0.00266, Running average loss:0.00266, Time taken: 0:00:15.139558 ETA: 0:13:22.396565 2022-07-31 16:09:46,735 [INFO] __main__: Epoch: 1047/1100:, Cur-Step: 20940, loss(cross_entropy): 0.00266, Running average loss:0.00266, Time taken: 0:00:15.139558 ETA: 0:13:22.396565 Epoch: 1047/1100:, Cur-Step: 20950, loss(cross_entropy): 0.00269, Running average loss:0.00270, Time taken: 0:00:15.139558 ETA: 0:13:22.396565 2022-07-31 16:09:53,985 [INFO] __main__: Epoch: 1047/1100:, Cur-Step: 20950, loss(cross_entropy): 0.00269, Running average loss:0.00270, Time taken: 0:00:15.139558 ETA: 0:13:22.396565 Epoch: 1048/1100:, Cur-Step: 20960, loss(cross_entropy): 0.00297, Running average loss:0.00297, Time taken: 0:00:15.313565 ETA: 0:13:16.305393 2022-07-31 16:10:01,315 [INFO] __main__: Epoch: 1048/1100:, Cur-Step: 20960, loss(cross_entropy): 0.00297, Running average loss:0.00297, Time taken: 0:00:15.313565 ETA: 0:13:16.305393 Epoch: 1048/1100:, Cur-Step: 20970, loss(cross_entropy): 0.00272, Running average loss:0.00278, Time taken: 0:00:15.313565 ETA: 0:13:16.305393 2022-07-31 16:10:08,508 [INFO] __main__: Epoch: 1048/1100:, Cur-Step: 20970, loss(cross_entropy): 0.00272, Running average loss:0.00278, Time taken: 0:00:15.313565 ETA: 0:13:16.305393 Epoch: 1049/1100:, Cur-Step: 20980, loss(cross_entropy): 0.00285, Running average loss:0.00285, Time taken: 0:00:15.250754 ETA: 0:12:57.788460 2022-07-31 16:10:15,811 [INFO] __main__: Epoch: 1049/1100:, Cur-Step: 20980, loss(cross_entropy): 0.00285, Running average loss:0.00285, Time taken: 0:00:15.250754 ETA: 0:12:57.788460 Epoch: 1049/1100:, Cur-Step: 20990, loss(cross_entropy): 0.00304, Running average loss:0.00275, Time taken: 0:00:15.250754 ETA: 0:12:57.788460 2022-07-31 16:10:23,105 [INFO] __main__: Epoch: 1049/1100:, Cur-Step: 20990, loss(cross_entropy): 0.00304, Running average loss:0.00275, Time taken: 0:00:15.250754 ETA: 0:12:57.788460 Epoch: 1050/1100:, Cur-Step: 21000, loss(cross_entropy): 0.00283, Running average loss:0.00283, Time taken: 0:00:15.352745 ETA: 0:12:47.637253 2022-07-31 16:10:30,428 [INFO] __main__: Epoch: 1050/1100:, Cur-Step: 21000, loss(cross_entropy): 0.00283, Running average loss:0.00283, Time taken: 0:00:15.352745 ETA: 0:12:47.637253 Epoch: 1050/1100:, Cur-Step: 21010, loss(cross_entropy): 0.00268, Running average loss:0.00267, Time taken: 0:00:15.352745 ETA: 0:12:47.637253 2022-07-31 16:10:37,602 [INFO] __main__: Epoch: 1050/1100:, Cur-Step: 21010, loss(cross_entropy): 0.00268, Running average loss:0.00267, Time taken: 0:00:15.352745 ETA: 0:12:47.637253 Epoch: 1051/1100:, Cur-Step: 21020, loss(cross_entropy): 0.00261, Running average loss:0.00261, Time taken: 0:00:15.033939 ETA: 0:12:16.663017 2022-07-31 16:10:44,796 [INFO] __main__: Epoch: 1051/1100:, Cur-Step: 21020, loss(cross_entropy): 0.00261, Running average loss:0.00261, Time taken: 0:00:15.033939 ETA: 0:12:16.663017 Epoch: 1051/1100:, Cur-Step: 21030, loss(cross_entropy): 0.00262, Running average loss:0.00252, Time taken: 0:00:15.033939 ETA: 0:12:16.663017 2022-07-31 16:10:51,995 [INFO] __main__: Epoch: 1051/1100:, Cur-Step: 21030, loss(cross_entropy): 0.00262, Running average loss:0.00252, Time taken: 0:00:15.033939 ETA: 0:12:16.663017 Epoch: 1052/1100:, Cur-Step: 21040, loss(cross_entropy): 0.00240, Running average loss:0.00240, Time taken: 0:00:15.300800 ETA: 0:12:14.438381 2022-07-31 16:10:59,376 [INFO] __main__: Epoch: 1052/1100:, Cur-Step: 21040, loss(cross_entropy): 0.00240, Running average loss:0.00240, Time taken: 0:00:15.300800 ETA: 0:12:14.438381 Epoch: 1052/1100:, Cur-Step: 21050, loss(cross_entropy): 0.00240, Running average loss:0.00243, Time taken: 0:00:15.300800 ETA: 0:12:14.438381 2022-07-31 16:11:06,585 [INFO] __main__: Epoch: 1052/1100:, Cur-Step: 21050, loss(cross_entropy): 0.00240, Running average loss:0.00243, Time taken: 0:00:15.300800 ETA: 0:12:14.438381 Epoch: 1053/1100:, Cur-Step: 21060, loss(cross_entropy): 0.00236, Running average loss:0.00236, Time taken: 0:00:15.157244 ETA: 0:11:52.390466 2022-07-31 16:11:13,843 [INFO] __main__: Epoch: 1053/1100:, Cur-Step: 21060, loss(cross_entropy): 0.00236, Running average loss:0.00236, Time taken: 0:00:15.157244 ETA: 0:11:52.390466 Epoch: 1053/1100:, Cur-Step: 21070, loss(cross_entropy): 0.00250, Running average loss:0.00241, Time taken: 0:00:15.157244 ETA: 0:11:52.390466 2022-07-31 16:11:21,028 [INFO] __main__: Epoch: 1053/1100:, Cur-Step: 21070, loss(cross_entropy): 0.00250, Running average loss:0.00241, Time taken: 0:00:15.157244 ETA: 0:11:52.390466 Epoch: 1054/1100:, Cur-Step: 21080, loss(cross_entropy): 0.00243, Running average loss:0.00243, Time taken: 0:00:15.269440 ETA: 0:11:42.394248 2022-07-31 16:11:28,439 [INFO] __main__: Epoch: 1054/1100:, Cur-Step: 21080, loss(cross_entropy): 0.00243, Running average loss:0.00243, Time taken: 0:00:15.269440 ETA: 0:11:42.394248 Epoch: 1054/1100:, Cur-Step: 21090, loss(cross_entropy): 0.00243, Running average loss:0.00241, Time taken: 0:00:15.269440 ETA: 0:11:42.394248 2022-07-31 16:11:35,669 [INFO] __main__: Epoch: 1054/1100:, Cur-Step: 21090, loss(cross_entropy): 0.00243, Running average loss:0.00241, Time taken: 0:00:15.269440 ETA: 0:11:42.394248 Epoch: 1055/1100:, Cur-Step: 21100, loss(cross_entropy): 0.00243, Running average loss:0.00243, Time taken: 0:00:15.193675 ETA: 0:11:23.715377 2022-07-31 16:11:42,868 [INFO] __main__: Epoch: 1055/1100:, Cur-Step: 21100, loss(cross_entropy): 0.00243, Running average loss:0.00243, Time taken: 0:00:15.193675 ETA: 0:11:23.715377 Epoch: 1055/1100:, Cur-Step: 21110, loss(cross_entropy): 0.00249, Running average loss:0.00239, Time taken: 0:00:15.193675 ETA: 0:11:23.715377 2022-07-31 16:11:50,108 [INFO] __main__: Epoch: 1055/1100:, Cur-Step: 21110, loss(cross_entropy): 0.00249, Running average loss:0.00239, Time taken: 0:00:15.193675 ETA: 0:11:23.715377 Epoch: 1056/1100:, Cur-Step: 21120, loss(cross_entropy): 0.00249, Running average loss:0.00249, Time taken: 0:00:15.315157 ETA: 0:11:13.866905 2022-07-31 16:11:57,451 [INFO] __main__: Epoch: 1056/1100:, Cur-Step: 21120, loss(cross_entropy): 0.00249, Running average loss:0.00249, Time taken: 0:00:15.315157 ETA: 0:11:13.866905 Epoch: 1056/1100:, Cur-Step: 21130, loss(cross_entropy): 0.00230, Running average loss:0.00242, Time taken: 0:00:15.315157 ETA: 0:11:13.866905 2022-07-31 16:12:04,692 [INFO] __main__: Epoch: 1056/1100:, Cur-Step: 21130, loss(cross_entropy): 0.00230, Running average loss:0.00242, Time taken: 0:00:15.315157 ETA: 0:11:13.866905 Epoch: 1057/1100:, Cur-Step: 21140, loss(cross_entropy): 0.00219, Running average loss:0.00219, Time taken: 0:00:15.148960 ETA: 0:10:51.405264 2022-07-31 16:12:11,859 [INFO] __main__: Epoch: 1057/1100:, Cur-Step: 21140, loss(cross_entropy): 0.00219, Running average loss:0.00219, Time taken: 0:00:15.148960 ETA: 0:10:51.405264 Epoch: 1057/1100:, Cur-Step: 21150, loss(cross_entropy): 0.00246, Running average loss:0.00241, Time taken: 0:00:15.148960 ETA: 0:10:51.405264 2022-07-31 16:12:19,164 [INFO] __main__: Epoch: 1057/1100:, Cur-Step: 21150, loss(cross_entropy): 0.00246, Running average loss:0.00241, Time taken: 0:00:15.148960 ETA: 0:10:51.405264 Epoch: 1058/1100:, Cur-Step: 21160, loss(cross_entropy): 0.00244, Running average loss:0.00244, Time taken: 0:00:15.430224 ETA: 0:10:48.069426 2022-07-31 16:12:26,610 [INFO] __main__: Epoch: 1058/1100:, Cur-Step: 21160, loss(cross_entropy): 0.00244, Running average loss:0.00244, Time taken: 0:00:15.430224 ETA: 0:10:48.069426 Epoch: 1058/1100:, Cur-Step: 21170, loss(cross_entropy): 0.00269, Running average loss:0.00245, Time taken: 0:00:15.430224 ETA: 0:10:48.069426 2022-07-31 16:12:33,976 [INFO] __main__: Epoch: 1058/1100:, Cur-Step: 21170, loss(cross_entropy): 0.00269, Running average loss:0.00245, Time taken: 0:00:15.430224 ETA: 0:10:48.069426 Epoch: 1059/1100:, Cur-Step: 21180, loss(cross_entropy): 0.00246, Running average loss:0.00246, Time taken: 0:00:15.287887 ETA: 0:10:26.803361 2022-07-31 16:12:41,171 [INFO] __main__: Epoch: 1059/1100:, Cur-Step: 21180, loss(cross_entropy): 0.00246, Running average loss:0.00246, Time taken: 0:00:15.287887 ETA: 0:10:26.803361 Epoch: 1059/1100:, Cur-Step: 21190, loss(cross_entropy): 0.00233, Running average loss:0.00250, Time taken: 0:00:15.287887 ETA: 0:10:26.803361 2022-07-31 16:12:48,455 [INFO] __main__: Epoch: 1059/1100:, Cur-Step: 21190, loss(cross_entropy): 0.00233, Running average loss:0.00250, Time taken: 0:00:15.287887 ETA: 0:10:26.803361 Epoch: 1060/1100:, Cur-Step: 21200, loss(cross_entropy): 0.00265, Running average loss:0.00265, Time taken: 0:00:15.284758 ETA: 0:10:11.390305 2022-07-31 16:12:55,755 [INFO] __main__: Epoch: 1060/1100:, Cur-Step: 21200, loss(cross_entropy): 0.00265, Running average loss:0.00265, Time taken: 0:00:15.284758 ETA: 0:10:11.390305 Epoch: 1060/1100:, Cur-Step: 21210, loss(cross_entropy): 0.00263, Running average loss:0.00263, Time taken: 0:00:15.284758 ETA: 0:10:11.390305 2022-07-31 16:13:03,108 [INFO] __main__: Epoch: 1060/1100:, Cur-Step: 21210, loss(cross_entropy): 0.00263, Running average loss:0.00263, Time taken: 0:00:15.284758 ETA: 0:10:11.390305 Epoch: 1061/1100:, Cur-Step: 21220, loss(cross_entropy): 0.00269, Running average loss:0.00269, Time taken: 0:00:15.313073 ETA: 0:09:57.209835 2022-07-31 16:13:10,275 [INFO] __main__: Epoch: 1061/1100:, Cur-Step: 21220, loss(cross_entropy): 0.00269, Running average loss:0.00269, Time taken: 0:00:15.313073 ETA: 0:09:57.209835 Epoch: 1061/1100:, Cur-Step: 21230, loss(cross_entropy): 0.00241, Running average loss:0.00256, Time taken: 0:00:15.313073 ETA: 0:09:57.209835 2022-07-31 16:13:17,552 [INFO] __main__: Epoch: 1061/1100:, Cur-Step: 21230, loss(cross_entropy): 0.00241, Running average loss:0.00256, Time taken: 0:00:15.313073 ETA: 0:09:57.209835 Epoch: 1062/1100:, Cur-Step: 21240, loss(cross_entropy): 0.00259, Running average loss:0.00259, Time taken: 0:00:15.302746 ETA: 0:09:41.504332 2022-07-31 16:13:24,873 [INFO] __main__: Epoch: 1062/1100:, Cur-Step: 21240, loss(cross_entropy): 0.00259, Running average loss:0.00259, Time taken: 0:00:15.302746 ETA: 0:09:41.504332 Epoch: 1062/1100:, Cur-Step: 21250, loss(cross_entropy): 0.00248, Running average loss:0.00257, Time taken: 0:00:15.302746 ETA: 0:09:41.504332 2022-07-31 16:13:32,224 [INFO] __main__: Epoch: 1062/1100:, Cur-Step: 21250, loss(cross_entropy): 0.00248, Running average loss:0.00257, Time taken: 0:00:15.302746 ETA: 0:09:41.504332 Epoch: 1063/1100:, Cur-Step: 21260, loss(cross_entropy): 0.00260, Running average loss:0.00260, Time taken: 0:00:15.362201 ETA: 0:09:28.401436 2022-07-31 16:13:39,487 [INFO] __main__: Epoch: 1063/1100:, Cur-Step: 21260, loss(cross_entropy): 0.00260, Running average loss:0.00260, Time taken: 0:00:15.362201 ETA: 0:09:28.401436 Epoch: 1063/1100:, Cur-Step: 21270, loss(cross_entropy): 0.00248, Running average loss:0.00254, Time taken: 0:00:15.362201 ETA: 0:09:28.401436 2022-07-31 16:13:46,900 [INFO] __main__: Epoch: 1063/1100:, Cur-Step: 21270, loss(cross_entropy): 0.00248, Running average loss:0.00254, Time taken: 0:00:15.362201 ETA: 0:09:28.401436 Epoch: 1064/1100:, Cur-Step: 21280, loss(cross_entropy): 0.00263, Running average loss:0.00263, Time taken: 0:00:15.409705 ETA: 0:09:14.749386 2022-07-31 16:13:54,161 [INFO] __main__: Epoch: 1064/1100:, Cur-Step: 21280, loss(cross_entropy): 0.00263, Running average loss:0.00263, Time taken: 0:00:15.409705 ETA: 0:09:14.749386 Epoch: 1064/1100:, Cur-Step: 21290, loss(cross_entropy): 0.00262, Running average loss:0.00254, Time taken: 0:00:15.409705 ETA: 0:09:14.749386 2022-07-31 16:14:01,670 [INFO] __main__: Epoch: 1064/1100:, Cur-Step: 21290, loss(cross_entropy): 0.00262, Running average loss:0.00254, Time taken: 0:00:15.409705 ETA: 0:09:14.749386 Epoch: 1065/1100:, Cur-Step: 21300, loss(cross_entropy): 0.00243, Running average loss:0.00243, Time taken: 0:00:15.544751 ETA: 0:09:04.066291 2022-07-31 16:14:09,019 [INFO] __main__: Epoch: 1065/1100:, Cur-Step: 21300, loss(cross_entropy): 0.00243, Running average loss:0.00243, Time taken: 0:00:15.544751 ETA: 0:09:04.066291 Epoch: 1065/1100:, Cur-Step: 21310, loss(cross_entropy): 0.00240, Running average loss:0.00246, Time taken: 0:00:15.544751 ETA: 0:09:04.066291 2022-07-31 16:14:16,286 [INFO] __main__: Epoch: 1065/1100:, Cur-Step: 21310, loss(cross_entropy): 0.00240, Running average loss:0.00246, Time taken: 0:00:15.544751 ETA: 0:09:04.066291 Epoch: 1066/1100:, Cur-Step: 21320, loss(cross_entropy): 0.00258, Running average loss:0.00258, Time taken: 0:00:15.441269 ETA: 0:08:45.003160 2022-07-31 16:14:23,702 [INFO] __main__: Epoch: 1066/1100:, Cur-Step: 21320, loss(cross_entropy): 0.00258, Running average loss:0.00258, Time taken: 0:00:15.441269 ETA: 0:08:45.003160 Epoch: 1066/1100:, Cur-Step: 21330, loss(cross_entropy): 0.00250, Running average loss:0.00243, Time taken: 0:00:15.441269 ETA: 0:08:45.003160 2022-07-31 16:14:31,229 [INFO] __main__: Epoch: 1066/1100:, Cur-Step: 21330, loss(cross_entropy): 0.00250, Running average loss:0.00243, Time taken: 0:00:15.441269 ETA: 0:08:45.003160 Epoch: 1067/1100:, Cur-Step: 21340, loss(cross_entropy): 0.00251, Running average loss:0.00251, Time taken: 0:00:15.482343 ETA: 0:08:30.917333 2022-07-31 16:14:38,397 [INFO] __main__: Epoch: 1067/1100:, Cur-Step: 21340, loss(cross_entropy): 0.00251, Running average loss:0.00251, Time taken: 0:00:15.482343 ETA: 0:08:30.917333 Epoch: 1067/1100:, Cur-Step: 21350, loss(cross_entropy): 0.00236, Running average loss:0.00239, Time taken: 0:00:15.482343 ETA: 0:08:30.917333 2022-07-31 16:14:45,739 [INFO] __main__: Epoch: 1067/1100:, Cur-Step: 21350, loss(cross_entropy): 0.00236, Running average loss:0.00239, Time taken: 0:00:15.482343 ETA: 0:08:30.917333 Epoch: 1068/1100:, Cur-Step: 21360, loss(cross_entropy): 0.00252, Running average loss:0.00252, Time taken: 0:00:15.275632 ETA: 0:08:08.820236 2022-07-31 16:14:52,977 [INFO] __main__: Epoch: 1068/1100:, Cur-Step: 21360, loss(cross_entropy): 0.00252, Running average loss:0.00252, Time taken: 0:00:15.275632 ETA: 0:08:08.820236 Epoch: 1068/1100:, Cur-Step: 21370, loss(cross_entropy): 0.00239, Running average loss:0.00240, Time taken: 0:00:15.275632 ETA: 0:08:08.820236 2022-07-31 16:15:00,501 [INFO] __main__: Epoch: 1068/1100:, Cur-Step: 21370, loss(cross_entropy): 0.00239, Running average loss:0.00240, Time taken: 0:00:15.275632 ETA: 0:08:08.820236 Epoch: 1069/1100:, Cur-Step: 21380, loss(cross_entropy): 0.00236, Running average loss:0.00236, Time taken: 0:00:15.507321 ETA: 0:08:00.726940 2022-07-31 16:15:07,759 [INFO] __main__: Epoch: 1069/1100:, Cur-Step: 21380, loss(cross_entropy): 0.00236, Running average loss:0.00236, Time taken: 0:00:15.507321 ETA: 0:08:00.726940 Epoch: 1069/1100:, Cur-Step: 21390, loss(cross_entropy): 0.00279, Running average loss:0.00247, Time taken: 0:00:15.507321 ETA: 0:08:00.726940 2022-07-31 16:15:15,029 [INFO] __main__: Epoch: 1069/1100:, Cur-Step: 21390, loss(cross_entropy): 0.00279, Running average loss:0.00247, Time taken: 0:00:15.507321 ETA: 0:08:00.726940 Epoch: 1070/1100:, Cur-Step: 21400, loss(cross_entropy): 0.00256, Running average loss:0.00256, Time taken: 0:00:15.172700 ETA: 0:07:35.180991 2022-07-31 16:15:22,242 [INFO] __main__: Epoch: 1070/1100:, Cur-Step: 21400, loss(cross_entropy): 0.00256, Running average loss:0.00256, Time taken: 0:00:15.172700 ETA: 0:07:35.180991 Epoch: 1070/1100:, Cur-Step: 21410, loss(cross_entropy): 0.00243, Running average loss:0.00248, Time taken: 0:00:15.172700 ETA: 0:07:35.180991 2022-07-31 16:15:29,614 [INFO] __main__: Epoch: 1070/1100:, Cur-Step: 21410, loss(cross_entropy): 0.00243, Running average loss:0.00248, Time taken: 0:00:15.172700 ETA: 0:07:35.180991 Epoch: 1071/1100:, Cur-Step: 21420, loss(cross_entropy): 0.00248, Running average loss:0.00248, Time taken: 0:00:15.321369 ETA: 0:07:24.319699 2022-07-31 16:15:36,873 [INFO] __main__: Epoch: 1071/1100:, Cur-Step: 21420, loss(cross_entropy): 0.00248, Running average loss:0.00248, Time taken: 0:00:15.321369 ETA: 0:07:24.319699 Epoch: 1071/1100:, Cur-Step: 21430, loss(cross_entropy): 0.00253, Running average loss:0.00259, Time taken: 0:00:15.321369 ETA: 0:07:24.319699 2022-07-31 16:15:44,103 [INFO] __main__: Epoch: 1071/1100:, Cur-Step: 21430, loss(cross_entropy): 0.00253, Running average loss:0.00259, Time taken: 0:00:15.321369 ETA: 0:07:24.319699 Epoch: 1072/1100:, Cur-Step: 21440, loss(cross_entropy): 0.00288, Running average loss:0.00288, Time taken: 0:00:15.103293 ETA: 0:07:02.892196 2022-07-31 16:15:51,247 [INFO] __main__: Epoch: 1072/1100:, Cur-Step: 21440, loss(cross_entropy): 0.00288, Running average loss:0.00288, Time taken: 0:00:15.103293 ETA: 0:07:02.892196 Epoch: 1072/1100:, Cur-Step: 21450, loss(cross_entropy): 0.00275, Running average loss:0.00279, Time taken: 0:00:15.103293 ETA: 0:07:02.892196 2022-07-31 16:15:58,735 [INFO] __main__: Epoch: 1072/1100:, Cur-Step: 21450, loss(cross_entropy): 0.00275, Running average loss:0.00279, Time taken: 0:00:15.103293 ETA: 0:07:02.892196 Epoch: 1073/1100:, Cur-Step: 21460, loss(cross_entropy): 0.00287, Running average loss:0.00287, Time taken: 0:00:15.597297 ETA: 0:07:01.127024 2022-07-31 16:16:06,149 [INFO] __main__: Epoch: 1073/1100:, Cur-Step: 21460, loss(cross_entropy): 0.00287, Running average loss:0.00287, Time taken: 0:00:15.597297 ETA: 0:07:01.127024 Epoch: 1073/1100:, Cur-Step: 21470, loss(cross_entropy): 0.00282, Running average loss:0.00283, Time taken: 0:00:15.597297 ETA: 0:07:01.127024 2022-07-31 16:16:13,413 [INFO] __main__: Epoch: 1073/1100:, Cur-Step: 21470, loss(cross_entropy): 0.00282, Running average loss:0.00283, Time taken: 0:00:15.597297 ETA: 0:07:01.127024 Epoch: 1074/1100:, Cur-Step: 21480, loss(cross_entropy): 0.00280, Running average loss:0.00280, Time taken: 0:00:15.258764 ETA: 0:06:36.727871 2022-07-31 16:16:20,689 [INFO] __main__: Epoch: 1074/1100:, Cur-Step: 21480, loss(cross_entropy): 0.00280, Running average loss:0.00280, Time taken: 0:00:15.258764 ETA: 0:06:36.727871 Epoch: 1074/1100:, Cur-Step: 21490, loss(cross_entropy): 0.00255, Running average loss:0.00265, Time taken: 0:00:15.258764 ETA: 0:06:36.727871 2022-07-31 16:16:28,037 [INFO] __main__: Epoch: 1074/1100:, Cur-Step: 21490, loss(cross_entropy): 0.00255, Running average loss:0.00265, Time taken: 0:00:15.258764 ETA: 0:06:36.727871 Epoch: 1075/1100:, Cur-Step: 21500, loss(cross_entropy): 0.00252, Running average loss:0.00252, Time taken: 0:00:15.459996 ETA: 0:06:26.499912 2022-07-31 16:16:35,427 [INFO] __main__: Epoch: 1075/1100:, Cur-Step: 21500, loss(cross_entropy): 0.00252, Running average loss:0.00252, Time taken: 0:00:15.459996 ETA: 0:06:26.499912 Epoch: 1075/1100:, Cur-Step: 21510, loss(cross_entropy): 0.00248, Running average loss:0.00251, Time taken: 0:00:15.459996 ETA: 0:06:26.499912 2022-07-31 16:16:42,657 [INFO] __main__: Epoch: 1075/1100:, Cur-Step: 21510, loss(cross_entropy): 0.00248, Running average loss:0.00251, Time taken: 0:00:15.459996 ETA: 0:06:26.499912 Epoch: 1076/1100:, Cur-Step: 21520, loss(cross_entropy): 0.00242, Running average loss:0.00242, Time taken: 0:00:15.315961 ETA: 0:06:07.583067 2022-07-31 16:16:49,998 [INFO] __main__: Epoch: 1076/1100:, Cur-Step: 21520, loss(cross_entropy): 0.00242, Running average loss:0.00242, Time taken: 0:00:15.315961 ETA: 0:06:07.583067 Epoch: 1076/1100:, Cur-Step: 21530, loss(cross_entropy): 0.00261, Running average loss:0.00250, Time taken: 0:00:15.315961 ETA: 0:06:07.583067 2022-07-31 16:16:57,327 [INFO] __main__: Epoch: 1076/1100:, Cur-Step: 21530, loss(cross_entropy): 0.00261, Running average loss:0.00250, Time taken: 0:00:15.315961 ETA: 0:06:07.583067 Epoch: 1077/1100:, Cur-Step: 21540, loss(cross_entropy): 0.00223, Running average loss:0.00223, Time taken: 0:00:15.554081 ETA: 0:05:57.743873 2022-07-31 16:17:04,786 [INFO] __main__: Epoch: 1077/1100:, Cur-Step: 21540, loss(cross_entropy): 0.00223, Running average loss:0.00223, Time taken: 0:00:15.554081 ETA: 0:05:57.743873 Epoch: 1077/1100:, Cur-Step: 21550, loss(cross_entropy): 0.00268, Running average loss:0.00240, Time taken: 0:00:15.554081 ETA: 0:05:57.743873 2022-07-31 16:17:12,070 [INFO] __main__: Epoch: 1077/1100:, Cur-Step: 21550, loss(cross_entropy): 0.00268, Running average loss:0.00240, Time taken: 0:00:15.554081 ETA: 0:05:57.743873 Epoch: 1078/1100:, Cur-Step: 21560, loss(cross_entropy): 0.00228, Running average loss:0.00228, Time taken: 0:00:15.360638 ETA: 0:05:37.934034 2022-07-31 16:17:19,391 [INFO] __main__: Epoch: 1078/1100:, Cur-Step: 21560, loss(cross_entropy): 0.00228, Running average loss:0.00228, Time taken: 0:00:15.360638 ETA: 0:05:37.934034 Epoch: 1078/1100:, Cur-Step: 21570, loss(cross_entropy): 0.00237, Running average loss:0.00237, Time taken: 0:00:15.360638 ETA: 0:05:37.934034 2022-07-31 16:17:26,673 [INFO] __main__: Epoch: 1078/1100:, Cur-Step: 21570, loss(cross_entropy): 0.00237, Running average loss:0.00237, Time taken: 0:00:15.360638 ETA: 0:05:37.934034 Epoch: 1079/1100:, Cur-Step: 21580, loss(cross_entropy): 0.00232, Running average loss:0.00232, Time taken: 0:00:15.483155 ETA: 0:05:25.146255 2022-07-31 16:17:34,073 [INFO] __main__: Epoch: 1079/1100:, Cur-Step: 21580, loss(cross_entropy): 0.00232, Running average loss:0.00232, Time taken: 0:00:15.483155 ETA: 0:05:25.146255 Epoch: 1079/1100:, Cur-Step: 21590, loss(cross_entropy): 0.00235, Running average loss:0.00240, Time taken: 0:00:15.483155 ETA: 0:05:25.146255 2022-07-31 16:17:41,530 [INFO] __main__: Epoch: 1079/1100:, Cur-Step: 21590, loss(cross_entropy): 0.00235, Running average loss:0.00240, Time taken: 0:00:15.483155 ETA: 0:05:25.146255 Epoch: 1080/1100:, Cur-Step: 21600, loss(cross_entropy): 0.00233, Running average loss:0.00233, Time taken: 0:00:15.466826 ETA: 0:05:09.336514 2022-07-31 16:17:48,836 [INFO] __main__: Epoch: 1080/1100:, Cur-Step: 21600, loss(cross_entropy): 0.00233, Running average loss:0.00233, Time taken: 0:00:15.466826 ETA: 0:05:09.336514 Epoch: 1080/1100:, Cur-Step: 21610, loss(cross_entropy): 0.00230, Running average loss:0.00239, Time taken: 0:00:15.466826 ETA: 0:05:09.336514 2022-07-31 16:17:56,052 [INFO] __main__: Epoch: 1080/1100:, Cur-Step: 21610, loss(cross_entropy): 0.00230, Running average loss:0.00239, Time taken: 0:00:15.466826 ETA: 0:05:09.336514 Epoch: 1081/1100:, Cur-Step: 21620, loss(cross_entropy): 0.00239, Running average loss:0.00239, Time taken: 0:00:15.285491 ETA: 0:04:50.424338 2022-07-31 16:18:03,437 [INFO] __main__: Epoch: 1081/1100:, Cur-Step: 21620, loss(cross_entropy): 0.00239, Running average loss:0.00239, Time taken: 0:00:15.285491 ETA: 0:04:50.424338 Epoch: 1081/1100:, Cur-Step: 21630, loss(cross_entropy): 0.00247, Running average loss:0.00244, Time taken: 0:00:15.285491 ETA: 0:04:50.424338 2022-07-31 16:18:10,641 [INFO] __main__: Epoch: 1081/1100:, Cur-Step: 21630, loss(cross_entropy): 0.00247, Running average loss:0.00244, Time taken: 0:00:15.285491 ETA: 0:04:50.424338 Epoch: 1082/1100:, Cur-Step: 21640, loss(cross_entropy): 0.00239, Running average loss:0.00239, Time taken: 0:00:15.664133 ETA: 0:04:41.954391 2022-07-31 16:18:18,412 [INFO] __main__: Epoch: 1082/1100:, Cur-Step: 21640, loss(cross_entropy): 0.00239, Running average loss:0.00239, Time taken: 0:00:15.664133 ETA: 0:04:41.954391 Epoch: 1082/1100:, Cur-Step: 21650, loss(cross_entropy): 0.00275, Running average loss:0.00255, Time taken: 0:00:15.664133 ETA: 0:04:41.954391 2022-07-31 16:18:25,999 [INFO] __main__: Epoch: 1082/1100:, Cur-Step: 21650, loss(cross_entropy): 0.00275, Running average loss:0.00255, Time taken: 0:00:15.664133 ETA: 0:04:41.954391 Epoch: 1083/1100:, Cur-Step: 21660, loss(cross_entropy): 0.00253, Running average loss:0.00253, Time taken: 0:00:16.046412 ETA: 0:04:32.789008 2022-07-31 16:18:33,601 [INFO] __main__: Epoch: 1083/1100:, Cur-Step: 21660, loss(cross_entropy): 0.00253, Running average loss:0.00253, Time taken: 0:00:16.046412 ETA: 0:04:32.789008 Epoch: 1083/1100:, Cur-Step: 21670, loss(cross_entropy): 0.00312, Running average loss:0.00259, Time taken: 0:00:16.046412 ETA: 0:04:32.789008 2022-07-31 16:18:40,990 [INFO] __main__: Epoch: 1083/1100:, Cur-Step: 21670, loss(cross_entropy): 0.00312, Running average loss:0.00259, Time taken: 0:00:16.046412 ETA: 0:04:32.789008 Epoch: 1084/1100:, Cur-Step: 21680, loss(cross_entropy): 0.00292, Running average loss:0.00292, Time taken: 0:00:15.622888 ETA: 0:04:09.966202 2022-07-31 16:18:48,440 [INFO] __main__: Epoch: 1084/1100:, Cur-Step: 21680, loss(cross_entropy): 0.00292, Running average loss:0.00292, Time taken: 0:00:15.622888 ETA: 0:04:09.966202 Epoch: 1084/1100:, Cur-Step: 21690, loss(cross_entropy): 0.00238, Running average loss:0.00266, Time taken: 0:00:15.622888 ETA: 0:04:09.966202 2022-07-31 16:18:55,740 [INFO] __main__: Epoch: 1084/1100:, Cur-Step: 21690, loss(cross_entropy): 0.00238, Running average loss:0.00266, Time taken: 0:00:15.622888 ETA: 0:04:09.966202 Epoch: 1085/1100:, Cur-Step: 21700, loss(cross_entropy): 0.00271, Running average loss:0.00271, Time taken: 0:00:15.467168 ETA: 0:03:52.007518 2022-07-31 16:19:03,199 [INFO] __main__: Epoch: 1085/1100:, Cur-Step: 21700, loss(cross_entropy): 0.00271, Running average loss:0.00271, Time taken: 0:00:15.467168 ETA: 0:03:52.007518 Epoch: 1085/1100:, Cur-Step: 21710, loss(cross_entropy): 0.00283, Running average loss:0.00273, Time taken: 0:00:15.467168 ETA: 0:03:52.007518 2022-07-31 16:19:10,467 [INFO] __main__: Epoch: 1085/1100:, Cur-Step: 21710, loss(cross_entropy): 0.00283, Running average loss:0.00273, Time taken: 0:00:15.467168 ETA: 0:03:52.007518 Epoch: 1086/1100:, Cur-Step: 21720, loss(cross_entropy): 0.00296, Running average loss:0.00296, Time taken: 0:00:15.315397 ETA: 0:03:34.415552 2022-07-31 16:19:17,800 [INFO] __main__: Epoch: 1086/1100:, Cur-Step: 21720, loss(cross_entropy): 0.00296, Running average loss:0.00296, Time taken: 0:00:15.315397 ETA: 0:03:34.415552 Epoch: 1086/1100:, Cur-Step: 21730, loss(cross_entropy): 0.00265, Running average loss:0.00279, Time taken: 0:00:15.315397 ETA: 0:03:34.415552 2022-07-31 16:19:25,033 [INFO] __main__: Epoch: 1086/1100:, Cur-Step: 21730, loss(cross_entropy): 0.00265, Running average loss:0.00279, Time taken: 0:00:15.315397 ETA: 0:03:34.415552 Epoch: 1087/1100:, Cur-Step: 21740, loss(cross_entropy): 0.00267, Running average loss:0.00267, Time taken: 0:00:15.403550 ETA: 0:03:20.246146 2022-07-31 16:19:32,428 [INFO] __main__: Epoch: 1087/1100:, Cur-Step: 21740, loss(cross_entropy): 0.00267, Running average loss:0.00267, Time taken: 0:00:15.403550 ETA: 0:03:20.246146 Epoch: 1087/1100:, Cur-Step: 21750, loss(cross_entropy): 0.00283, Running average loss:0.00269, Time taken: 0:00:15.403550 ETA: 0:03:20.246146 2022-07-31 16:19:39,806 [INFO] __main__: Epoch: 1087/1100:, Cur-Step: 21750, loss(cross_entropy): 0.00283, Running average loss:0.00269, Time taken: 0:00:15.403550 ETA: 0:03:20.246146 Epoch: 1088/1100:, Cur-Step: 21760, loss(cross_entropy): 0.00261, Running average loss:0.00261, Time taken: 0:00:15.623229 ETA: 0:03:07.478751 2022-07-31 16:19:47,339 [INFO] __main__: Epoch: 1088/1100:, Cur-Step: 21760, loss(cross_entropy): 0.00261, Running average loss:0.00261, Time taken: 0:00:15.623229 ETA: 0:03:07.478751 Epoch: 1088/1100:, Cur-Step: 21770, loss(cross_entropy): 0.00268, Running average loss:0.00257, Time taken: 0:00:15.623229 ETA: 0:03:07.478751 2022-07-31 16:19:54,840 [INFO] __main__: Epoch: 1088/1100:, Cur-Step: 21770, loss(cross_entropy): 0.00268, Running average loss:0.00257, Time taken: 0:00:15.623229 ETA: 0:03:07.478751 Epoch: 1089/1100:, Cur-Step: 21780, loss(cross_entropy): 0.00261, Running average loss:0.00261, Time taken: 0:00:15.883363 ETA: 0:02:54.716990 2022-07-31 16:20:02,483 [INFO] __main__: Epoch: 1089/1100:, Cur-Step: 21780, loss(cross_entropy): 0.00261, Running average loss:0.00261, Time taken: 0:00:15.883363 ETA: 0:02:54.716990 Epoch: 1089/1100:, Cur-Step: 21790, loss(cross_entropy): 0.00229, Running average loss:0.00244, Time taken: 0:00:15.883363 ETA: 0:02:54.716990 2022-07-31 16:20:09,805 [INFO] __main__: Epoch: 1089/1100:, Cur-Step: 21790, loss(cross_entropy): 0.00229, Running average loss:0.00244, Time taken: 0:00:15.883363 ETA: 0:02:54.716990 Epoch: 1090/1100:, Cur-Step: 21800, loss(cross_entropy): 0.00242, Running average loss:0.00242, Time taken: 0:00:15.482575 ETA: 0:02:34.825752 2022-07-31 16:20:17,205 [INFO] __main__: Epoch: 1090/1100:, Cur-Step: 21800, loss(cross_entropy): 0.00242, Running average loss:0.00242, Time taken: 0:00:15.482575 ETA: 0:02:34.825752 Epoch: 1090/1100:, Cur-Step: 21810, loss(cross_entropy): 0.00228, Running average loss:0.00234, Time taken: 0:00:15.482575 ETA: 0:02:34.825752 2022-07-31 16:20:24,454 [INFO] __main__: Epoch: 1090/1100:, Cur-Step: 21810, loss(cross_entropy): 0.00228, Running average loss:0.00234, Time taken: 0:00:15.482575 ETA: 0:02:34.825752 Epoch: 1091/1100:, Cur-Step: 21820, loss(cross_entropy): 0.00233, Running average loss:0.00233, Time taken: 0:00:15.483497 ETA: 0:02:19.351470 2022-07-31 16:20:31,913 [INFO] __main__: Epoch: 1091/1100:, Cur-Step: 21820, loss(cross_entropy): 0.00233, Running average loss:0.00233, Time taken: 0:00:15.483497 ETA: 0:02:19.351470 Epoch: 1091/1100:, Cur-Step: 21830, loss(cross_entropy): 0.00231, Running average loss:0.00229, Time taken: 0:00:15.483497 ETA: 0:02:19.351470 2022-07-31 16:20:39,178 [INFO] __main__: Epoch: 1091/1100:, Cur-Step: 21830, loss(cross_entropy): 0.00231, Running average loss:0.00229, Time taken: 0:00:15.483497 ETA: 0:02:19.351470 Epoch: 1092/1100:, Cur-Step: 21840, loss(cross_entropy): 0.00237, Running average loss:0.00237, Time taken: 0:00:15.415489 ETA: 0:02:03.323912 2022-07-31 16:20:46,560 [INFO] __main__: Epoch: 1092/1100:, Cur-Step: 21840, loss(cross_entropy): 0.00237, Running average loss:0.00237, Time taken: 0:00:15.415489 ETA: 0:02:03.323912 Epoch: 1092/1100:, Cur-Step: 21850, loss(cross_entropy): 0.00232, Running average loss:0.00230, Time taken: 0:00:15.415489 ETA: 0:02:03.323912 2022-07-31 16:20:53,927 [INFO] __main__: Epoch: 1092/1100:, Cur-Step: 21850, loss(cross_entropy): 0.00232, Running average loss:0.00230, Time taken: 0:00:15.415489 ETA: 0:02:03.323912 Epoch: 1093/1100:, Cur-Step: 21860, loss(cross_entropy): 0.00253, Running average loss:0.00253, Time taken: 0:00:15.512495 ETA: 0:01:48.587462 2022-07-31 16:21:01,381 [INFO] __main__: Epoch: 1093/1100:, Cur-Step: 21860, loss(cross_entropy): 0.00253, Running average loss:0.00253, Time taken: 0:00:15.512495 ETA: 0:01:48.587462 Epoch: 1093/1100:, Cur-Step: 21870, loss(cross_entropy): 0.00231, Running average loss:0.00241, Time taken: 0:00:15.512495 ETA: 0:01:48.587462 2022-07-31 16:21:08,806 [INFO] __main__: Epoch: 1093/1100:, Cur-Step: 21870, loss(cross_entropy): 0.00231, Running average loss:0.00241, Time taken: 0:00:15.512495 ETA: 0:01:48.587462 Epoch: 1094/1100:, Cur-Step: 21880, loss(cross_entropy): 0.00226, Running average loss:0.00226, Time taken: 0:00:15.491325 ETA: 0:01:32.947951 2022-07-31 16:21:16,160 [INFO] __main__: Epoch: 1094/1100:, Cur-Step: 21880, loss(cross_entropy): 0.00226, Running average loss:0.00226, Time taken: 0:00:15.491325 ETA: 0:01:32.947951 Epoch: 1094/1100:, Cur-Step: 21890, loss(cross_entropy): 0.00248, Running average loss:0.00241, Time taken: 0:00:15.491325 ETA: 0:01:32.947951 2022-07-31 16:21:23,492 [INFO] __main__: Epoch: 1094/1100:, Cur-Step: 21890, loss(cross_entropy): 0.00248, Running average loss:0.00241, Time taken: 0:00:15.491325 ETA: 0:01:32.947951 Epoch: 1095/1100:, Cur-Step: 21900, loss(cross_entropy): 0.00242, Running average loss:0.00242, Time taken: 0:00:15.372395 ETA: 0:01:16.861973 2022-07-31 16:21:30,777 [INFO] __main__: Epoch: 1095/1100:, Cur-Step: 21900, loss(cross_entropy): 0.00242, Running average loss:0.00242, Time taken: 0:00:15.372395 ETA: 0:01:16.861973 Epoch: 1095/1100:, Cur-Step: 21910, loss(cross_entropy): 0.00248, Running average loss:0.00244, Time taken: 0:00:15.372395 ETA: 0:01:16.861973 2022-07-31 16:21:38,240 [INFO] __main__: Epoch: 1095/1100:, Cur-Step: 21910, loss(cross_entropy): 0.00248, Running average loss:0.00244, Time taken: 0:00:15.372395 ETA: 0:01:16.861973 Epoch: 1096/1100:, Cur-Step: 21920, loss(cross_entropy): 0.00230, Running average loss:0.00230, Time taken: 0:00:15.555461 ETA: 0:01:02.221845 2022-07-31 16:21:45,562 [INFO] __main__: Epoch: 1096/1100:, Cur-Step: 21920, loss(cross_entropy): 0.00230, Running average loss:0.00230, Time taken: 0:00:15.555461 ETA: 0:01:02.221845 Epoch: 1096/1100:, Cur-Step: 21930, loss(cross_entropy): 0.00258, Running average loss:0.00248, Time taken: 0:00:15.555461 ETA: 0:01:02.221845 2022-07-31 16:21:52,830 [INFO] __main__: Epoch: 1096/1100:, Cur-Step: 21930, loss(cross_entropy): 0.00258, Running average loss:0.00248, Time taken: 0:00:15.555461 ETA: 0:01:02.221845 Epoch: 1097/1100:, Cur-Step: 21940, loss(cross_entropy): 0.00235, Running average loss:0.00235, Time taken: 0:00:15.408650 ETA: 0:00:46.225949 2022-07-31 16:22:00,253 [INFO] __main__: Epoch: 1097/1100:, Cur-Step: 21940, loss(cross_entropy): 0.00235, Running average loss:0.00235, Time taken: 0:00:15.408650 ETA: 0:00:46.225949 Epoch: 1097/1100:, Cur-Step: 21950, loss(cross_entropy): 0.00246, Running average loss:0.00244, Time taken: 0:00:15.408650 ETA: 0:00:46.225949 2022-07-31 16:22:07,764 [INFO] __main__: Epoch: 1097/1100:, Cur-Step: 21950, loss(cross_entropy): 0.00246, Running average loss:0.00244, Time taken: 0:00:15.408650 ETA: 0:00:46.225949 Epoch: 1098/1100:, Cur-Step: 21960, loss(cross_entropy): 0.00251, Running average loss:0.00251, Time taken: 0:00:15.661350 ETA: 0:00:31.322699 2022-07-31 16:22:15,094 [INFO] __main__: Epoch: 1098/1100:, Cur-Step: 21960, loss(cross_entropy): 0.00251, Running average loss:0.00251, Time taken: 0:00:15.661350 ETA: 0:00:31.322699 Epoch: 1098/1100:, Cur-Step: 21970, loss(cross_entropy): 0.00237, Running average loss:0.00244, Time taken: 0:00:15.661350 ETA: 0:00:31.322699 2022-07-31 16:22:22,462 [INFO] __main__: Epoch: 1098/1100:, Cur-Step: 21970, loss(cross_entropy): 0.00237, Running average loss:0.00244, Time taken: 0:00:15.661350 ETA: 0:00:31.322699 Epoch: 1099/1100:, Cur-Step: 21980, loss(cross_entropy): 0.00256, Running average loss:0.00256, Time taken: 0:00:15.317851 ETA: 0:00:15.317851 2022-07-31 16:22:29,691 [INFO] __main__: Epoch: 1099/1100:, Cur-Step: 21980, loss(cross_entropy): 0.00256, Running average loss:0.00256, Time taken: 0:00:15.317851 ETA: 0:00:15.317851 Epoch: 1099/1100:, Cur-Step: 21990, loss(cross_entropy): 0.00245, Running average loss:0.00251, Time taken: 0:00:15.317851 ETA: 0:00:15.317851 2022-07-31 16:22:37,152 [INFO] __main__: Epoch: 1099/1100:, Cur-Step: 21990, loss(cross_entropy): 0.00245, Running average loss:0.00251, Time taken: 0:00:15.317851 ETA: 0:00:15.317851 INFO:tensorflow:Saving checkpoints for step-22000. 2022-07-31 16:22:43,672 [INFO] tensorflow: Saving checkpoints for step-22000. Throughput Avg: 5.467 img/s Latency Avg: 820.595 ms Latency 90%: 925.598 ms Latency 95%: 945.704 ms Latency 99%: 985.025 ms DLL 2022-07-31 16:23:06.858652 - () throughput_train:5.4672583129103405 latency_train:820.5950745586874 elapsed_time:1763.168061 INFO:tensorflow:Loss for final step: 0.0027139364. 2022-07-31 16:23:06,966 [INFO] tensorflow: Loss for final step: 0.0027139364. Saving the final step model to /workspace/tao-experiments/retrain/weights/model_retrained.tlt 2022-07-31 16:23:06,968 [INFO] __main__: Saving the final step model to /workspace/tao-experiments/retrain/weights/model_retrained.tlt 2022-07-31 12:23:16,592 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.