2022-07-28 15:48:20,386 [INFO] root: Registry: ['nvcr.io'] 2022-07-28 15:48:20,537 [INFO] tlt.components.instance_handler.local_instance: Running command in container: nvcr.io/nvidia/tao/tao-toolkit-tf:v3.21.11-tf1.15.5-py3 Matplotlib created a temporary config/cache directory at /tmp/matplotlib-7srfqfej because the default path (/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing. Using TensorFlow backend. WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them. Using TensorFlow backend. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/hooks/checkpoint_saver_hook.py:21: The name tf.train.CheckpointSaverHook is deprecated. Please use tf.estimator.CheckpointSaverHook instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/hooks/pretrained_restore_hook.py:23: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/hooks/pretrained_restore_hook.py:23: The name tf.logging.WARN is deprecated. Please use tf.compat.v1.logging.WARN instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/scripts/train.py:410: The name tf.logging.INFO is deprecated. Please use tf.compat.v1.logging.INFO instead. Loading experiment spec at /workspace/tao-experiments/specs/unet_train_vgg_6S900.txt. 2022-07-28 19:48:28,105 [INFO] __main__: Loading experiment spec at /workspace/tao-experiments/specs/unet_train_vgg_6S900.txt. 2022-07-28 19:48:28,109 [INFO] iva.unet.spec_handler.spec_loader: Merging specification from /workspace/tao-experiments/specs/unet_train_vgg_6S900.txt 2022-07-28 19:48:28,115 [INFO] root: Initializing the pre-trained weights from /workspace/tao-experiments/pretrained_vgg16/vgg_16.hdf5 WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead. 2022-07-28 19:48:28,120 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead. 2022-07-28 19:48:28,130 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:245: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead. 2022-07-28 19:48:28,142 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:245: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1834: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead. 2022-07-28 19:48:28,150 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1834: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:133: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead. 2022-07-28 19:48:28,155 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:133: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead. WARNING:tensorflow:From /opt/nvidia/third_party/keras/tensorflow_backend.py:183: The name tf.nn.max_pool is deprecated. Please use tf.nn.max_pool2d instead. 2022-07-28 19:48:28,200 [WARNING] tensorflow: From /opt/nvidia/third_party/keras/tensorflow_backend.py:183: The name tf.nn.max_pool is deprecated. Please use tf.nn.max_pool2d instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead. 2022-07-28 19:48:29,034 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:190: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead. 2022-07-28 19:48:29,034 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:190: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead. 2022-07-28 19:48:29,034 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead. 2022-07-28 19:48:29,117 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:95: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead. 2022-07-28 19:48:30,200 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:95: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead. 2022-07-28 19:48:30,211 [INFO] iva.unet.model.utilities: Label Id 0: Train Id 0 2022-07-28 19:48:30,211 [INFO] iva.unet.model.utilities: Label Id 1: Train Id 1 2022-07-28 19:48:30,211 [INFO] iva.unet.model.utilities: Label Id 2: Train Id 2 2022-07-28 19:48:30,211 [INFO] iva.unet.model.utilities: Label Id 3: Train Id 3 2022-07-28 19:48:30,212 [INFO] iva.unet.model.utilities: Label Id 4: Train Id 4 2022-07-28 19:48:30,212 [INFO] iva.unet.model.utilities: Label Id 5: Train Id 5 2022-07-28 19:48:30,214 [INFO] iva.unet.hooks.latest_checkpoint: Getting the latest checkpoint for restoring /workspace/tao-experiments/unpruned/model.step-16000.tlt INFO:tensorflow:Using config: {'_model_dir': '/workspace/tao-experiments/unpruned', '_tf_random_seed': None, '_save_summary_steps': 5, '_save_checkpoints_steps': None, '_save_checkpoints_secs': None, '_session_config': intra_op_parallelism_threads: 1 inter_op_parallelism_threads: 38 gpu_options { allow_growth: true visible_device_list: "0" force_gpu_compatible: true } allow_soft_placement: true , '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': None, '_log_step_count_steps': None, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': , '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1} 2022-07-28 19:48:33,987 [INFO] tensorflow: Using config: {'_model_dir': '/workspace/tao-experiments/unpruned', '_tf_random_seed': None, '_save_summary_steps': 5, '_save_checkpoints_steps': None, '_save_checkpoints_secs': None, '_session_config': intra_op_parallelism_threads: 1 inter_op_parallelism_threads: 38 gpu_options { allow_growth: true visible_device_list: "0" force_gpu_compatible: true } allow_soft_placement: true , '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': None, '_log_step_count_steps': None, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': , '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1} Phase train: Total 78 files. 2022-07-28 19:48:34,016 [INFO] iva.unet.model.utilities: The total number of training samples 78 and the batch size per GPU 4 2022-07-28 19:48:34,016 [INFO] iva.unet.model.utilities: Cannot iterate over exactly 78 samples with a batch size of 4; each epoch will therefore take one extra step. 2022-07-28 19:48:34,016 [INFO] iva.unet.model.utilities: Steps per epoch taken: 20 Running for 1000 Epochs 2022-07-28 19:48:34,016 [INFO] __main__: Running for 1000 Epochs INFO:tensorflow:Create CheckpointSaverHook. 2022-07-28 19:48:34,016 [INFO] tensorflow: Create CheckpointSaverHook. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead. 2022-07-28 19:48:34,474 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead. WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2022-07-28 19:48:34,525 [WARNING] tensorflow: Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code WARNING:tensorflow:Entity . at 0x7fdf718fa8c8> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of . at 0x7fdf718fa8c8>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2022-07-28 19:48:34,541 [WARNING] tensorflow: Entity . at 0x7fdf718fa8c8> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of . at 0x7fdf718fa8c8>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code WARNING:tensorflow: The TensorFlow contrib module will not be included in TensorFlow 2.0. For more information, please see: * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md * https://github.com/tensorflow/addons * https://github.com/tensorflow/io (for I/O related ops) If you depend on functionality not listed there, please file an issue. 2022-07-28 19:48:34,543 [WARNING] tensorflow: The TensorFlow contrib module will not be included in TensorFlow 2.0. For more information, please see: * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md * https://github.com/tensorflow/addons * https://github.com/tensorflow/io (for I/O related ops) If you depend on functionality not listed there, please file an issue. /opt/nvidia/third_party/keras/tensorflow_backend.py:356: UserWarning: Creating resources inside a function passed to Dataset.map() is not supported. Create each resource outside the function, and capture it inside the function to use it. self, _map_func_set_random_wrapper, num_parallel_calls=num_parallel_calls WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2022-07-28 19:48:34,554 [WARNING] tensorflow: Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2022-07-28 19:48:34,562 [WARNING] tensorflow: Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2022-07-28 19:48:34,570 [WARNING] tensorflow: Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/utils/data_loader.py:451: The name tf.image.resize_images is deprecated. Please use tf.image.resize instead. 2022-07-28 19:48:34,570 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/utils/data_loader.py:451: The name tf.image.resize_images is deprecated. Please use tf.image.resize instead. WARNING:tensorflow:Entity . at 0x7fdf5b05b620> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of . at 0x7fdf5b05b620>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2022-07-28 19:48:34,581 [WARNING] tensorflow: Entity . at 0x7fdf5b05b620> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of . at 0x7fdf5b05b620>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code WARNING:tensorflow:Entity . at 0x7fdf5b05b8c8> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of . at 0x7fdf5b05b8c8>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2022-07-28 19:48:34,588 [WARNING] tensorflow: Entity . at 0x7fdf5b05b8c8> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of . at 0x7fdf5b05b8c8>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code WARNING:tensorflow:Entity . at 0x7fdf5b05bc80> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of . at 0x7fdf5b05bc80>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2022-07-28 19:48:34,594 [WARNING] tensorflow: Entity . at 0x7fdf5b05bc80> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of . at 0x7fdf5b05bc80>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2022-07-28 19:48:34,600 [WARNING] tensorflow: Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code WARNING:tensorflow:Entity . at 0x7fdf5b1cb048> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of . at 0x7fdf5b1cb048>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2022-07-28 19:48:34,608 [WARNING] tensorflow: Entity . at 0x7fdf5b1cb048> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of . at 0x7fdf5b1cb048>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code INFO:tensorflow:Calling model_fn. 2022-07-28 19:48:34,625 [INFO] tensorflow: Calling model_fn. 2022-07-28 19:48:34,625 [INFO] iva.unet.utils.model_fn: {'exec_mode': 'train', 'model_dir': '/workspace/tao-experiments/unpruned', 'resize_padding': False, 'resize_method': 'BILINEAR', 'log_dir': None, 'batch_size': 4, 'learning_rate': 9.999999747378752e-05, 'activation': 'softmax', 'crossvalidation_idx': None, 'max_steps': None, 'regularizer_type': 2, 'weight_decay': 1.9999999949504854e-06, 'log_summary_steps': 10, 'warmup_steps': 0, 'augment': False, 'use_amp': False, 'use_trt': False, 'use_xla': False, 'loss': 'cross_entropy', 'epochs': 1000, 'pretrained_weights_file': None, 'lr_scheduler': None, 'unet_model': , 'key': 'nvidia_tlt', 'experiment_spec': random_seed: 42 dataset_config { dataset: "custom" input_image_type: "color" train_images_path: "/workspace/tao-experiments/data/images/train" train_masks_path: "/workspace/tao-experiments/data/masks/train" val_images_path: "/workspace/tao-experiments/data/images/val" val_masks_path: "/workspace/tao-experiments/data/masks/val" test_images_path: "/workspace/tao-experiments/data/images/test" data_class_config { target_classes { name: "Background" mapping_class: "Background" } target_classes { name: "Joint" label_id: 1 mapping_class: "Joint" } target_classes { name: "Robot" label_id: 2 mapping_class: "Robot" } target_classes { name: "End" label_id: 3 mapping_class: "End" } target_classes { name: "Stem" label_id: 4 mapping_class: "Stem" } target_classes { name: "Leaf" label_id: 5 mapping_class: "Leaf" } } } model_config { num_layers: 16 training_precision { backend_floatx: FLOAT32 } arch: "vgg" all_projections: true model_input_height: 704 model_input_width: 1280 model_input_channels: 3 } training_config { batch_size: 4 regularizer { type: L2 weight: 1.9999999949504854e-06 } optimizer { adam { epsilon: 9.99999993922529e-09 beta1: 0.8999999761581421 beta2: 0.9990000128746033 } } checkpoint_interval: 100 log_summary_steps: 10 learning_rate: 9.999999747378752e-05 loss: "cross_entropy" epochs: 1000 } , 'seed': 42, 'benchmark': False, 'temp_dir': '/tmp/tmpjxf6u4j0', 'num_classes': 6, 'num_conf_mat_classes': 6, 'start_step': 16000, 'checkpoint_interval': 100, 'model_json': None, 'load_graph': False, 'weights_monitor': False, 'phase': None} __________________________________________________________________________________________________ Layer (type) Output Shape Param # Connected to ================================================================================================== input_1 (InputLayer) (None, 3, 704, 1280) 0 __________________________________________________________________________________________________ block_1a_conv_1 (Conv2D) (None, 64, 704, 1280 1792 input_1[0][0] __________________________________________________________________________________________________ block_1a_relu (Activation) (None, 64, 704, 1280 0 block_1a_conv_1[0][0] __________________________________________________________________________________________________ block_1b_conv_1 (Conv2D) (None, 64, 704, 1280 36928 block_1a_relu[0][0] __________________________________________________________________________________________________ block_1b_relu (Activation) (None, 64, 704, 1280 0 block_1b_conv_1[0][0] __________________________________________________________________________________________________ block1_pool (MaxPooling2D) (None, 64, 352, 640) 0 block_1b_relu[0][0] __________________________________________________________________________________________________ block_2a_conv_1 (Conv2D) (None, 128, 352, 640 73856 block1_pool[0][0] __________________________________________________________________________________________________ block_2a_relu (Activation) (None, 128, 352, 640 0 block_2a_conv_1[0][0] __________________________________________________________________________________________________ block_2b_conv_1 (Conv2D) (None, 128, 352, 640 147584 block_2a_relu[0][0] __________________________________________________________________________________________________ block_2b_relu (Activation) (None, 128, 352, 640 0 block_2b_conv_1[0][0] __________________________________________________________________________________________________ block2_pool (MaxPooling2D) (None, 128, 176, 320 0 block_2b_relu[0][0] __________________________________________________________________________________________________ block_3a_conv_1 (Conv2D) (None, 256, 176, 320 295168 block2_pool[0][0] __________________________________________________________________________________________________ block_3a_relu (Activation) (None, 256, 176, 320 0 block_3a_conv_1[0][0] __________________________________________________________________________________________________ block_3b_conv_1 (Conv2D) (None, 256, 176, 320 590080 block_3a_relu[0][0] __________________________________________________________________________________________________ block_3b_relu (Activation) (None, 256, 176, 320 0 block_3b_conv_1[0][0] __________________________________________________________________________________________________ block_3c_conv_1 (Conv2D) (None, 256, 176, 320 590080 block_3b_relu[0][0] __________________________________________________________________________________________________ block_3c_relu (Activation) (None, 256, 176, 320 0 block_3c_conv_1[0][0] __________________________________________________________________________________________________ block3_pool (MaxPooling2D) (None, 256, 88, 160) 0 block_3c_relu[0][0] __________________________________________________________________________________________________ block_4a_conv_1 (Conv2D) (None, 512, 88, 160) 1180160 block3_pool[0][0] __________________________________________________________________________________________________ block_4a_relu (Activation) (None, 512, 88, 160) 0 block_4a_conv_1[0][0] __________________________________________________________________________________________________ block_4b_conv_1 (Conv2D) (None, 512, 88, 160) 2359808 block_4a_relu[0][0] __________________________________________________________________________________________________ block_4b_relu (Activation) (None, 512, 88, 160) 0 block_4b_conv_1[0][0] __________________________________________________________________________________________________ block_4c_conv_1 (Conv2D) (None, 512, 88, 160) 2359808 block_4b_relu[0][0] __________________________________________________________________________________________________ block_4c_relu (Activation) (None, 512, 88, 160) 0 block_4c_conv_1[0][0] __________________________________________________________________________________________________ block4_pool (MaxPooling2D) (None, 512, 44, 80) 0 block_4c_relu[0][0] __________________________________________________________________________________________________ block_5a_conv_1 (Conv2D) (None, 512, 44, 80) 2359808 block4_pool[0][0] __________________________________________________________________________________________________ block_5a_relu (Activation) (None, 512, 44, 80) 0 block_5a_conv_1[0][0] __________________________________________________________________________________________________ block_5b_conv_1 (Conv2D) (None, 512, 44, 80) 2359808 block_5a_relu[0][0] __________________________________________________________________________________________________ block_5b_relu (Activation) (None, 512, 44, 80) 0 block_5b_conv_1[0][0] __________________________________________________________________________________________________ block_5c_conv_1 (Conv2D) (None, 512, 44, 80) 2359808 block_5b_relu[0][0] __________________________________________________________________________________________________ block_5c_relu (Activation) (None, 512, 44, 80) 0 block_5c_conv_1[0][0] __________________________________________________________________________________________________ max_pooling2d_1 (MaxPooling2D) (None, 512, 22, 40) 0 block_5c_relu[0][0] __________________________________________________________________________________________________ conv2d_transpose_1 (Conv2DTrans (None, 512, 44, 80) 4194816 max_pooling2d_1[0][0] __________________________________________________________________________________________________ concatenate_1 (Concatenate) (None, 1024, 44, 80) 0 conv2d_transpose_1[0][0] block4_pool[0][0] __________________________________________________________________________________________________ conv2d_1 (Conv2D) (None, 512, 44, 80) 4719104 concatenate_1[0][0] __________________________________________________________________________________________________ conv2d_2 (Conv2D) (None, 512, 44, 80) 2359808 conv2d_1[0][0] __________________________________________________________________________________________________ conv2d_transpose_2 (Conv2DTrans (None, 256, 88, 160) 2097408 conv2d_2[0][0] __________________________________________________________________________________________________ concatenate_2 (Concatenate) (None, 512, 88, 160) 0 conv2d_transpose_2[0][0] block3_pool[0][0] __________________________________________________________________________________________________ conv2d_3 (Conv2D) (None, 256, 88, 160) 1179904 concatenate_2[0][0] __________________________________________________________________________________________________ conv2d_4 (Conv2D) (None, 256, 88, 160) 590080 conv2d_3[0][0] __________________________________________________________________________________________________ conv2d_transpose_3 (Conv2DTrans (None, 128, 176, 320 524416 conv2d_4[0][0] __________________________________________________________________________________________________ concatenate_3 (Concatenate) (None, 256, 176, 320 0 conv2d_transpose_3[0][0] block2_pool[0][0] __________________________________________________________________________________________________ conv2d_5 (Conv2D) (None, 128, 176, 320 295040 concatenate_3[0][0] __________________________________________________________________________________________________ conv2d_6 (Conv2D) (None, 128, 176, 320 147584 conv2d_5[0][0] __________________________________________________________________________________________________ conv2d_transpose_4 (Conv2DTrans (None, 64, 352, 640) 131136 conv2d_6[0][0] __________________________________________________________________________________________________ concatenate_4 (Concatenate) (None, 128, 352, 640 0 conv2d_transpose_4[0][0] block1_pool[0][0] __________________________________________________________________________________________________ conv2d_7 (Conv2D) (None, 64, 352, 640) 73792 concatenate_4[0][0] __________________________________________________________________________________________________ conv2d_8 (Conv2D) (None, 64, 352, 640) 36928 conv2d_7[0][0] __________________________________________________________________________________________________ conv2d_transpose_5 (Conv2DTrans (None, 64, 704, 1280 65600 conv2d_8[0][0] __________________________________________________________________________________________________ concatenate_5 (Concatenate) (None, 128, 704, 128 0 conv2d_transpose_5[0][0] block_1a_relu[0][0] __________________________________________________________________________________________________ conv2d_9 (Conv2D) (None, 64, 704, 1280 73792 concatenate_5[0][0] __________________________________________________________________________________________________ conv2d_10 (Conv2D) (None, 64, 704, 1280 36928 conv2d_9[0][0] __________________________________________________________________________________________________ conv2d_11 (Conv2D) (None, 6, 704, 1280) 390 conv2d_10[0][0] ================================================================================================== Total params: 31,241,414 Trainable params: 31,241,414 Non-trainable params: 0 __________________________________________________________________________________________________ WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/utils/model_fn.py:225: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead. 2022-07-28 19:48:35,056 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/utils/model_fn.py:225: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead. INFO:tensorflow:Done calling model_fn. 2022-07-28 19:48:36,188 [INFO] tensorflow: Done calling model_fn. INFO:tensorflow:Graph was finalized. 2022-07-28 19:48:37,128 [INFO] tensorflow: Graph was finalized. INFO:tensorflow:Running local_init_op. 2022-07-28 19:48:38,055 [INFO] tensorflow: Running local_init_op. INFO:tensorflow:Done running local_init_op. 2022-07-28 19:48:38,112 [INFO] tensorflow: Done running local_init_op. [GPU] Restoring pretrained weights from: /tmp/tmpw1uc8nys/model.ckpt-16000 2022-07-28 19:48:38,521 [INFO] iva.unet.hooks.pretrained_restore_hook: Pretrained weights loaded with success... INFO:tensorflow:Saving checkpoints for step-16000. 2022-07-28 19:48:40,519 [INFO] tensorflow: Saving checkpoints for step-16000. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/hooks/training_hook.py:95: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead. 2022-07-28 19:49:02,746 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/hooks/training_hook.py:95: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead. Epoch: 800/1000:, Cur-Step: 16000, loss(cross_entropy): 0.00304, Running average loss:0.00304, Time taken: 0:00:00 ETA: 0:00:00 2022-07-28 19:49:59,089 [INFO] __main__: Epoch: 800/1000:, Cur-Step: 16000, loss(cross_entropy): 0.00304, Running average loss:0.00304, Time taken: 0:00:00 ETA: 0:00:00 Epoch: 800/1000:, Cur-Step: 16010, loss(cross_entropy): 0.00574, Running average loss:0.00589, Time taken: 0:00:00 ETA: 0:00:00 2022-07-28 19:51:05,350 [INFO] __main__: Epoch: 800/1000:, Cur-Step: 16010, loss(cross_entropy): 0.00574, Running average loss:0.00589, Time taken: 0:00:00 ETA: 0:00:00 Epoch: 801/1000:, Cur-Step: 16020, loss(cross_entropy): 0.00354, Running average loss:0.00354, Time taken: 0:02:37.743047 ETA: 8:43:10.866353 2022-07-28 19:51:12,462 [INFO] __main__: Epoch: 801/1000:, Cur-Step: 16020, loss(cross_entropy): 0.00354, Running average loss:0.00354, Time taken: 0:02:37.743047 ETA: 8:43:10.866353 Epoch: 801/1000:, Cur-Step: 16030, loss(cross_entropy): 0.00363, Running average loss:0.00365, Time taken: 0:02:37.743047 ETA: 8:43:10.866353 2022-07-28 19:51:19,374 [INFO] __main__: Epoch: 801/1000:, Cur-Step: 16030, loss(cross_entropy): 0.00363, Running average loss:0.00365, Time taken: 0:02:37.743047 ETA: 8:43:10.866353 Epoch: 802/1000:, Cur-Step: 16040, loss(cross_entropy): 0.00316, Running average loss:0.00316, Time taken: 0:00:14.641017 ETA: 0:48:18.921405 2022-07-28 19:51:26,305 [INFO] __main__: Epoch: 802/1000:, Cur-Step: 16040, loss(cross_entropy): 0.00316, Running average loss:0.00316, Time taken: 0:00:14.641017 ETA: 0:48:18.921405 Epoch: 802/1000:, Cur-Step: 16050, loss(cross_entropy): 0.00343, Running average loss:0.00317, Time taken: 0:00:14.641017 ETA: 0:48:18.921405 2022-07-28 19:51:33,181 [INFO] __main__: Epoch: 802/1000:, Cur-Step: 16050, loss(cross_entropy): 0.00343, Running average loss:0.00317, Time taken: 0:00:14.641017 ETA: 0:48:18.921405 Epoch: 803/1000:, Cur-Step: 16060, loss(cross_entropy): 0.00294, Running average loss:0.00294, Time taken: 0:00:14.529094 ETA: 0:47:42.231420 2022-07-28 19:51:40,175 [INFO] __main__: Epoch: 803/1000:, Cur-Step: 16060, loss(cross_entropy): 0.00294, Running average loss:0.00294, Time taken: 0:00:14.529094 ETA: 0:47:42.231420 Epoch: 803/1000:, Cur-Step: 16070, loss(cross_entropy): 0.00284, Running average loss:0.00290, Time taken: 0:00:14.529094 ETA: 0:47:42.231420 2022-07-28 19:51:47,220 [INFO] __main__: Epoch: 803/1000:, Cur-Step: 16070, loss(cross_entropy): 0.00284, Running average loss:0.00290, Time taken: 0:00:14.529094 ETA: 0:47:42.231420 Epoch: 804/1000:, Cur-Step: 16080, loss(cross_entropy): 0.00281, Running average loss:0.00281, Time taken: 0:00:14.693653 ETA: 0:47:59.955962 2022-07-28 19:51:54,178 [INFO] __main__: Epoch: 804/1000:, Cur-Step: 16080, loss(cross_entropy): 0.00281, Running average loss:0.00281, Time taken: 0:00:14.693653 ETA: 0:47:59.955962 Epoch: 804/1000:, Cur-Step: 16090, loss(cross_entropy): 0.00271, Running average loss:0.00277, Time taken: 0:00:14.693653 ETA: 0:47:59.955962 2022-07-28 19:52:01,078 [INFO] __main__: Epoch: 804/1000:, Cur-Step: 16090, loss(cross_entropy): 0.00271, Running average loss:0.00277, Time taken: 0:00:14.693653 ETA: 0:47:59.955962 Epoch: 805/1000:, Cur-Step: 16100, loss(cross_entropy): 0.00280, Running average loss:0.00280, Time taken: 0:00:14.504676 ETA: 0:47:08.411887 2022-07-28 19:52:08,012 [INFO] __main__: Epoch: 805/1000:, Cur-Step: 16100, loss(cross_entropy): 0.00280, Running average loss:0.00280, Time taken: 0:00:14.504676 ETA: 0:47:08.411887 Epoch: 805/1000:, Cur-Step: 16110, loss(cross_entropy): 0.00276, Running average loss:0.00272, Time taken: 0:00:14.504676 ETA: 0:47:08.411887 2022-07-28 19:52:15,052 [INFO] __main__: Epoch: 805/1000:, Cur-Step: 16110, loss(cross_entropy): 0.00276, Running average loss:0.00272, Time taken: 0:00:14.504676 ETA: 0:47:08.411887 Epoch: 806/1000:, Cur-Step: 16120, loss(cross_entropy): 0.00271, Running average loss:0.00271, Time taken: 0:00:14.705590 ETA: 0:47:32.884369 2022-07-28 19:52:22,043 [INFO] __main__: Epoch: 806/1000:, Cur-Step: 16120, loss(cross_entropy): 0.00271, Running average loss:0.00271, Time taken: 0:00:14.705590 ETA: 0:47:32.884369 Epoch: 806/1000:, Cur-Step: 16130, loss(cross_entropy): 0.00271, Running average loss:0.00271, Time taken: 0:00:14.705590 ETA: 0:47:32.884369 2022-07-28 19:52:28,941 [INFO] __main__: Epoch: 806/1000:, Cur-Step: 16130, loss(cross_entropy): 0.00271, Running average loss:0.00271, Time taken: 0:00:14.705590 ETA: 0:47:32.884369 Epoch: 807/1000:, Cur-Step: 16140, loss(cross_entropy): 0.00269, Running average loss:0.00269, Time taken: 0:00:14.554867 ETA: 0:46:49.089337 2022-07-28 19:52:35,919 [INFO] __main__: Epoch: 807/1000:, Cur-Step: 16140, loss(cross_entropy): 0.00269, Running average loss:0.00269, Time taken: 0:00:14.554867 ETA: 0:46:49.089337 Epoch: 807/1000:, Cur-Step: 16150, loss(cross_entropy): 0.00288, Running average loss:0.00277, Time taken: 0:00:14.554867 ETA: 0:46:49.089337 2022-07-28 19:52:42,890 [INFO] __main__: Epoch: 807/1000:, Cur-Step: 16150, loss(cross_entropy): 0.00288, Running average loss:0.00277, Time taken: 0:00:14.554867 ETA: 0:46:49.089337 Epoch: 808/1000:, Cur-Step: 16160, loss(cross_entropy): 0.00284, Running average loss:0.00284, Time taken: 0:00:14.662591 ETA: 0:46:55.217377 2022-07-28 19:52:49,910 [INFO] __main__: Epoch: 808/1000:, Cur-Step: 16160, loss(cross_entropy): 0.00284, Running average loss:0.00284, Time taken: 0:00:14.662591 ETA: 0:46:55.217377 Epoch: 808/1000:, Cur-Step: 16170, loss(cross_entropy): 0.00309, Running average loss:0.00289, Time taken: 0:00:14.662591 ETA: 0:46:55.217377 2022-07-28 19:52:56,774 [INFO] __main__: Epoch: 808/1000:, Cur-Step: 16170, loss(cross_entropy): 0.00309, Running average loss:0.00289, Time taken: 0:00:14.662591 ETA: 0:46:55.217377 Epoch: 809/1000:, Cur-Step: 16180, loss(cross_entropy): 0.00275, Running average loss:0.00275, Time taken: 0:00:14.515699 ETA: 0:46:12.498446 2022-07-28 19:53:03,723 [INFO] __main__: Epoch: 809/1000:, Cur-Step: 16180, loss(cross_entropy): 0.00275, Running average loss:0.00275, Time taken: 0:00:14.515699 ETA: 0:46:12.498446 Epoch: 809/1000:, Cur-Step: 16190, loss(cross_entropy): 0.00297, Running average loss:0.00304, Time taken: 0:00:14.515699 ETA: 0:46:12.498446 2022-07-28 19:53:10,601 [INFO] __main__: Epoch: 809/1000:, Cur-Step: 16190, loss(cross_entropy): 0.00297, Running average loss:0.00304, Time taken: 0:00:14.515699 ETA: 0:46:12.498446 Epoch: 810/1000:, Cur-Step: 16200, loss(cross_entropy): 0.00299, Running average loss:0.00299, Time taken: 0:00:14.606686 ETA: 0:46:15.270317 2022-07-28 19:53:17,662 [INFO] __main__: Epoch: 810/1000:, Cur-Step: 16200, loss(cross_entropy): 0.00299, Running average loss:0.00299, Time taken: 0:00:14.606686 ETA: 0:46:15.270317 Epoch: 810/1000:, Cur-Step: 16210, loss(cross_entropy): 0.00327, Running average loss:0.00310, Time taken: 0:00:14.606686 ETA: 0:46:15.270317 2022-07-28 19:53:24,550 [INFO] __main__: Epoch: 810/1000:, Cur-Step: 16210, loss(cross_entropy): 0.00327, Running average loss:0.00310, Time taken: 0:00:14.606686 ETA: 0:46:15.270317 Epoch: 811/1000:, Cur-Step: 16220, loss(cross_entropy): 0.00320, Running average loss:0.00320, Time taken: 0:00:14.546875 ETA: 0:45:49.359330 2022-07-28 19:53:31,501 [INFO] __main__: Epoch: 811/1000:, Cur-Step: 16220, loss(cross_entropy): 0.00320, Running average loss:0.00320, Time taken: 0:00:14.546875 ETA: 0:45:49.359330 Epoch: 811/1000:, Cur-Step: 16230, loss(cross_entropy): 0.00299, Running average loss:0.00314, Time taken: 0:00:14.546875 ETA: 0:45:49.359330 2022-07-28 19:53:38,423 [INFO] __main__: Epoch: 811/1000:, Cur-Step: 16230, loss(cross_entropy): 0.00299, Running average loss:0.00314, Time taken: 0:00:14.546875 ETA: 0:45:49.359330 Epoch: 812/1000:, Cur-Step: 16240, loss(cross_entropy): 0.00327, Running average loss:0.00327, Time taken: 0:00:14.647547 ETA: 0:45:53.738837 2022-07-28 19:53:45,502 [INFO] __main__: Epoch: 812/1000:, Cur-Step: 16240, loss(cross_entropy): 0.00327, Running average loss:0.00327, Time taken: 0:00:14.647547 ETA: 0:45:53.738837 Epoch: 812/1000:, Cur-Step: 16250, loss(cross_entropy): 0.00310, Running average loss:0.00312, Time taken: 0:00:14.647547 ETA: 0:45:53.738837 2022-07-28 19:53:52,370 [INFO] __main__: Epoch: 812/1000:, Cur-Step: 16250, loss(cross_entropy): 0.00310, Running average loss:0.00312, Time taken: 0:00:14.647547 ETA: 0:45:53.738837 Epoch: 813/1000:, Cur-Step: 16260, loss(cross_entropy): 0.00289, Running average loss:0.00289, Time taken: 0:00:14.595948 ETA: 0:45:29.442362 2022-07-28 19:53:59,376 [INFO] __main__: Epoch: 813/1000:, Cur-Step: 16260, loss(cross_entropy): 0.00289, Running average loss:0.00289, Time taken: 0:00:14.595948 ETA: 0:45:29.442362 Epoch: 813/1000:, Cur-Step: 16270, loss(cross_entropy): 0.00320, Running average loss:0.00321, Time taken: 0:00:14.595948 ETA: 0:45:29.442362 2022-07-28 19:54:06,250 [INFO] __main__: Epoch: 813/1000:, Cur-Step: 16270, loss(cross_entropy): 0.00320, Running average loss:0.00321, Time taken: 0:00:14.595948 ETA: 0:45:29.442362 Epoch: 814/1000:, Cur-Step: 16280, loss(cross_entropy): 0.00318, Running average loss:0.00318, Time taken: 0:00:14.614715 ETA: 0:45:18.337008 2022-07-28 19:54:13,321 [INFO] __main__: Epoch: 814/1000:, Cur-Step: 16280, loss(cross_entropy): 0.00318, Running average loss:0.00318, Time taken: 0:00:14.614715 ETA: 0:45:18.337008 Epoch: 814/1000:, Cur-Step: 16290, loss(cross_entropy): 0.00296, Running average loss:0.00321, Time taken: 0:00:14.614715 ETA: 0:45:18.337008 2022-07-28 19:54:20,415 [INFO] __main__: Epoch: 814/1000:, Cur-Step: 16290, loss(cross_entropy): 0.00296, Running average loss:0.00321, Time taken: 0:00:14.614715 ETA: 0:45:18.337008 Epoch: 815/1000:, Cur-Step: 16300, loss(cross_entropy): 0.00326, Running average loss:0.00326, Time taken: 0:00:14.743806 ETA: 0:45:27.604133 2022-07-28 19:54:27,321 [INFO] __main__: Epoch: 815/1000:, Cur-Step: 16300, loss(cross_entropy): 0.00326, Running average loss:0.00326, Time taken: 0:00:14.743806 ETA: 0:45:27.604133 Epoch: 815/1000:, Cur-Step: 16310, loss(cross_entropy): 0.00317, Running average loss:0.00322, Time taken: 0:00:14.743806 ETA: 0:45:27.604133 2022-07-28 19:54:34,265 [INFO] __main__: Epoch: 815/1000:, Cur-Step: 16310, loss(cross_entropy): 0.00317, Running average loss:0.00322, Time taken: 0:00:14.743806 ETA: 0:45:27.604133 Epoch: 816/1000:, Cur-Step: 16320, loss(cross_entropy): 0.00329, Running average loss:0.00329, Time taken: 0:00:14.568172 ETA: 0:44:40.543556 2022-07-28 19:54:41,198 [INFO] __main__: Epoch: 816/1000:, Cur-Step: 16320, loss(cross_entropy): 0.00329, Running average loss:0.00329, Time taken: 0:00:14.568172 ETA: 0:44:40.543556 Epoch: 816/1000:, Cur-Step: 16330, loss(cross_entropy): 0.00325, Running average loss:0.00321, Time taken: 0:00:14.568172 ETA: 0:44:40.543556 2022-07-28 19:54:48,308 [INFO] __main__: Epoch: 816/1000:, Cur-Step: 16330, loss(cross_entropy): 0.00325, Running average loss:0.00321, Time taken: 0:00:14.568172 ETA: 0:44:40.543556 Epoch: 817/1000:, Cur-Step: 16340, loss(cross_entropy): 0.00352, Running average loss:0.00352, Time taken: 0:00:14.706351 ETA: 0:44:51.262284 2022-07-28 19:54:55,184 [INFO] __main__: Epoch: 817/1000:, Cur-Step: 16340, loss(cross_entropy): 0.00352, Running average loss:0.00352, Time taken: 0:00:14.706351 ETA: 0:44:51.262284 Epoch: 817/1000:, Cur-Step: 16350, loss(cross_entropy): 0.00367, Running average loss:0.00335, Time taken: 0:00:14.706351 ETA: 0:44:51.262284 2022-07-28 19:55:02,194 [INFO] __main__: Epoch: 817/1000:, Cur-Step: 16350, loss(cross_entropy): 0.00367, Running average loss:0.00335, Time taken: 0:00:14.706351 ETA: 0:44:51.262284 Epoch: 818/1000:, Cur-Step: 16360, loss(cross_entropy): 0.00345, Running average loss:0.00345, Time taken: 0:00:14.589420 ETA: 0:44:15.274368 2022-07-28 19:55:09,093 [INFO] __main__: Epoch: 818/1000:, Cur-Step: 16360, loss(cross_entropy): 0.00345, Running average loss:0.00345, Time taken: 0:00:14.589420 ETA: 0:44:15.274368 Epoch: 818/1000:, Cur-Step: 16370, loss(cross_entropy): 0.00310, Running average loss:0.00331, Time taken: 0:00:14.589420 ETA: 0:44:15.274368 2022-07-28 19:55:16,130 [INFO] __main__: Epoch: 818/1000:, Cur-Step: 16370, loss(cross_entropy): 0.00310, Running average loss:0.00331, Time taken: 0:00:14.589420 ETA: 0:44:15.274368 Epoch: 819/1000:, Cur-Step: 16380, loss(cross_entropy): 0.00315, Running average loss:0.00315, Time taken: 0:00:14.694039 ETA: 0:44:19.621035 2022-07-28 19:55:23,112 [INFO] __main__: Epoch: 819/1000:, Cur-Step: 16380, loss(cross_entropy): 0.00315, Running average loss:0.00315, Time taken: 0:00:14.694039 ETA: 0:44:19.621035 Epoch: 819/1000:, Cur-Step: 16390, loss(cross_entropy): 0.00320, Running average loss:0.00318, Time taken: 0:00:14.694039 ETA: 0:44:19.621035 2022-07-28 19:55:30,028 [INFO] __main__: Epoch: 819/1000:, Cur-Step: 16390, loss(cross_entropy): 0.00320, Running average loss:0.00318, Time taken: 0:00:14.694039 ETA: 0:44:19.621035 Epoch: 820/1000:, Cur-Step: 16400, loss(cross_entropy): 0.00300, Running average loss:0.00300, Time taken: 0:00:14.488355 ETA: 0:43:27.903929 2022-07-28 19:55:36,889 [INFO] __main__: Epoch: 820/1000:, Cur-Step: 16400, loss(cross_entropy): 0.00300, Running average loss:0.00300, Time taken: 0:00:14.488355 ETA: 0:43:27.903929 Epoch: 820/1000:, Cur-Step: 16410, loss(cross_entropy): 0.00275, Running average loss:0.00312, Time taken: 0:00:14.488355 ETA: 0:43:27.903929 2022-07-28 19:55:43,900 [INFO] __main__: Epoch: 820/1000:, Cur-Step: 16410, loss(cross_entropy): 0.00275, Running average loss:0.00312, Time taken: 0:00:14.488355 ETA: 0:43:27.903929 Epoch: 821/1000:, Cur-Step: 16420, loss(cross_entropy): 0.00295, Running average loss:0.00295, Time taken: 0:00:14.689557 ETA: 0:43:49.430674 2022-07-28 19:55:50,909 [INFO] __main__: Epoch: 821/1000:, Cur-Step: 16420, loss(cross_entropy): 0.00295, Running average loss:0.00295, Time taken: 0:00:14.689557 ETA: 0:43:49.430674 Epoch: 821/1000:, Cur-Step: 16430, loss(cross_entropy): 0.00334, Running average loss:0.00314, Time taken: 0:00:14.689557 ETA: 0:43:49.430674 2022-07-28 19:55:57,907 [INFO] __main__: Epoch: 821/1000:, Cur-Step: 16430, loss(cross_entropy): 0.00334, Running average loss:0.00314, Time taken: 0:00:14.689557 ETA: 0:43:49.430674 Epoch: 822/1000:, Cur-Step: 16440, loss(cross_entropy): 0.00335, Running average loss:0.00335, Time taken: 0:00:14.632361 ETA: 0:43:24.560331 2022-07-28 19:56:04,831 [INFO] __main__: Epoch: 822/1000:, Cur-Step: 16440, loss(cross_entropy): 0.00335, Running average loss:0.00335, Time taken: 0:00:14.632361 ETA: 0:43:24.560331 Epoch: 822/1000:, Cur-Step: 16450, loss(cross_entropy): 0.00318, Running average loss:0.00304, Time taken: 0:00:14.632361 ETA: 0:43:24.560331 2022-07-28 19:56:11,844 [INFO] __main__: Epoch: 822/1000:, Cur-Step: 16450, loss(cross_entropy): 0.00318, Running average loss:0.00304, Time taken: 0:00:14.632361 ETA: 0:43:24.560331 Epoch: 823/1000:, Cur-Step: 16460, loss(cross_entropy): 0.00306, Running average loss:0.00306, Time taken: 0:00:14.752341 ETA: 0:43:31.164320 2022-07-28 19:56:18,896 [INFO] __main__: Epoch: 823/1000:, Cur-Step: 16460, loss(cross_entropy): 0.00306, Running average loss:0.00306, Time taken: 0:00:14.752341 ETA: 0:43:31.164320 Epoch: 823/1000:, Cur-Step: 16470, loss(cross_entropy): 0.00322, Running average loss:0.00314, Time taken: 0:00:14.752341 ETA: 0:43:31.164320 2022-07-28 19:56:25,866 [INFO] __main__: Epoch: 823/1000:, Cur-Step: 16470, loss(cross_entropy): 0.00322, Running average loss:0.00314, Time taken: 0:00:14.752341 ETA: 0:43:31.164320 Epoch: 824/1000:, Cur-Step: 16480, loss(cross_entropy): 0.00309, Running average loss:0.00309, Time taken: 0:00:14.616751 ETA: 0:42:52.548252 2022-07-28 19:56:32,769 [INFO] __main__: Epoch: 824/1000:, Cur-Step: 16480, loss(cross_entropy): 0.00309, Running average loss:0.00309, Time taken: 0:00:14.616751 ETA: 0:42:52.548252 Epoch: 824/1000:, Cur-Step: 16490, loss(cross_entropy): 0.00312, Running average loss:0.00310, Time taken: 0:00:14.616751 ETA: 0:42:52.548252 2022-07-28 19:56:39,790 [INFO] __main__: Epoch: 824/1000:, Cur-Step: 16490, loss(cross_entropy): 0.00312, Running average loss:0.00310, Time taken: 0:00:14.616751 ETA: 0:42:52.548252 Epoch: 825/1000:, Cur-Step: 16500, loss(cross_entropy): 0.00354, Running average loss:0.00354, Time taken: 0:00:14.716757 ETA: 0:42:55.432444 2022-07-28 19:56:46,794 [INFO] __main__: Epoch: 825/1000:, Cur-Step: 16500, loss(cross_entropy): 0.00354, Running average loss:0.00354, Time taken: 0:00:14.716757 ETA: 0:42:55.432444 Epoch: 825/1000:, Cur-Step: 16510, loss(cross_entropy): 0.00316, Running average loss:0.00321, Time taken: 0:00:14.716757 ETA: 0:42:55.432444 2022-07-28 19:56:53,873 [INFO] __main__: Epoch: 825/1000:, Cur-Step: 16510, loss(cross_entropy): 0.00316, Running average loss:0.00321, Time taken: 0:00:14.716757 ETA: 0:42:55.432444 Epoch: 826/1000:, Cur-Step: 16520, loss(cross_entropy): 0.00311, Running average loss:0.00311, Time taken: 0:00:14.721231 ETA: 0:42:41.494191 2022-07-28 19:57:00,786 [INFO] __main__: Epoch: 826/1000:, Cur-Step: 16520, loss(cross_entropy): 0.00311, Running average loss:0.00311, Time taken: 0:00:14.721231 ETA: 0:42:41.494191 Epoch: 826/1000:, Cur-Step: 16530, loss(cross_entropy): 0.00308, Running average loss:0.00300, Time taken: 0:00:14.721231 ETA: 0:42:41.494191 2022-07-28 19:57:07,738 [INFO] __main__: Epoch: 826/1000:, Cur-Step: 16530, loss(cross_entropy): 0.00308, Running average loss:0.00300, Time taken: 0:00:14.721231 ETA: 0:42:41.494191 Epoch: 827/1000:, Cur-Step: 16540, loss(cross_entropy): 0.00300, Running average loss:0.00300, Time taken: 0:00:14.541908 ETA: 0:41:55.750047 2022-07-28 19:57:14,651 [INFO] __main__: Epoch: 827/1000:, Cur-Step: 16540, loss(cross_entropy): 0.00300, Running average loss:0.00300, Time taken: 0:00:14.541908 ETA: 0:41:55.750047 Epoch: 827/1000:, Cur-Step: 16550, loss(cross_entropy): 0.00298, Running average loss:0.00296, Time taken: 0:00:14.541908 ETA: 0:41:55.750047 2022-07-28 19:57:21,706 [INFO] __main__: Epoch: 827/1000:, Cur-Step: 16550, loss(cross_entropy): 0.00298, Running average loss:0.00296, Time taken: 0:00:14.541908 ETA: 0:41:55.750047 Epoch: 828/1000:, Cur-Step: 16560, loss(cross_entropy): 0.00317, Running average loss:0.00317, Time taken: 0:00:14.672802 ETA: 0:42:03.722021 2022-07-28 19:57:28,603 [INFO] __main__: Epoch: 828/1000:, Cur-Step: 16560, loss(cross_entropy): 0.00317, Running average loss:0.00317, Time taken: 0:00:14.672802 ETA: 0:42:03.722021 Epoch: 828/1000:, Cur-Step: 16570, loss(cross_entropy): 0.00281, Running average loss:0.00283, Time taken: 0:00:14.672802 ETA: 0:42:03.722021 2022-07-28 19:57:35,608 [INFO] __main__: Epoch: 828/1000:, Cur-Step: 16570, loss(cross_entropy): 0.00281, Running average loss:0.00283, Time taken: 0:00:14.672802 ETA: 0:42:03.722021 Epoch: 829/1000:, Cur-Step: 16580, loss(cross_entropy): 0.00270, Running average loss:0.00270, Time taken: 0:00:14.596274 ETA: 0:41:35.962837 2022-07-28 19:57:42,489 [INFO] __main__: Epoch: 829/1000:, Cur-Step: 16580, loss(cross_entropy): 0.00270, Running average loss:0.00270, Time taken: 0:00:14.596274 ETA: 0:41:35.962837 Epoch: 829/1000:, Cur-Step: 16590, loss(cross_entropy): 0.00287, Running average loss:0.00290, Time taken: 0:00:14.596274 ETA: 0:41:35.962837 2022-07-28 19:57:49,548 [INFO] __main__: Epoch: 829/1000:, Cur-Step: 16590, loss(cross_entropy): 0.00287, Running average loss:0.00290, Time taken: 0:00:14.596274 ETA: 0:41:35.962837 Epoch: 830/1000:, Cur-Step: 16600, loss(cross_entropy): 0.00320, Running average loss:0.00320, Time taken: 0:00:14.692173 ETA: 0:41:37.669330 2022-07-28 19:57:56,482 [INFO] __main__: Epoch: 830/1000:, Cur-Step: 16600, loss(cross_entropy): 0.00320, Running average loss:0.00320, Time taken: 0:00:14.692173 ETA: 0:41:37.669330 Epoch: 830/1000:, Cur-Step: 16610, loss(cross_entropy): 0.00282, Running average loss:0.00289, Time taken: 0:00:14.692173 ETA: 0:41:37.669330 2022-07-28 19:58:03,469 [INFO] __main__: Epoch: 830/1000:, Cur-Step: 16610, loss(cross_entropy): 0.00282, Running average loss:0.00289, Time taken: 0:00:14.692173 ETA: 0:41:37.669330 Epoch: 831/1000:, Cur-Step: 16620, loss(cross_entropy): 0.00301, Running average loss:0.00301, Time taken: 0:00:14.576582 ETA: 0:41:03.442431 2022-07-28 19:58:10,355 [INFO] __main__: Epoch: 831/1000:, Cur-Step: 16620, loss(cross_entropy): 0.00301, Running average loss:0.00301, Time taken: 0:00:14.576582 ETA: 0:41:03.442431 Epoch: 831/1000:, Cur-Step: 16630, loss(cross_entropy): 0.00283, Running average loss:0.00284, Time taken: 0:00:14.576582 ETA: 0:41:03.442431 2022-07-28 19:58:17,360 [INFO] __main__: Epoch: 831/1000:, Cur-Step: 16630, loss(cross_entropy): 0.00283, Running average loss:0.00284, Time taken: 0:00:14.576582 ETA: 0:41:03.442431 Epoch: 832/1000:, Cur-Step: 16640, loss(cross_entropy): 0.00288, Running average loss:0.00288, Time taken: 0:00:14.728616 ETA: 0:41:14.407568 2022-07-28 19:58:24,396 [INFO] __main__: Epoch: 832/1000:, Cur-Step: 16640, loss(cross_entropy): 0.00288, Running average loss:0.00288, Time taken: 0:00:14.728616 ETA: 0:41:14.407568 Epoch: 832/1000:, Cur-Step: 16650, loss(cross_entropy): 0.00327, Running average loss:0.00299, Time taken: 0:00:14.728616 ETA: 0:41:14.407568 2022-07-28 19:58:31,400 [INFO] __main__: Epoch: 832/1000:, Cur-Step: 16650, loss(cross_entropy): 0.00327, Running average loss:0.00299, Time taken: 0:00:14.728616 ETA: 0:41:14.407568 Epoch: 833/1000:, Cur-Step: 16660, loss(cross_entropy): 0.00324, Running average loss:0.00324, Time taken: 0:00:14.663897 ETA: 0:40:48.870805 2022-07-28 19:58:38,336 [INFO] __main__: Epoch: 833/1000:, Cur-Step: 16660, loss(cross_entropy): 0.00324, Running average loss:0.00324, Time taken: 0:00:14.663897 ETA: 0:40:48.870805 Epoch: 833/1000:, Cur-Step: 16670, loss(cross_entropy): 0.00317, Running average loss:0.00317, Time taken: 0:00:14.663897 ETA: 0:40:48.870805 2022-07-28 19:58:45,354 [INFO] __main__: Epoch: 833/1000:, Cur-Step: 16670, loss(cross_entropy): 0.00317, Running average loss:0.00317, Time taken: 0:00:14.663897 ETA: 0:40:48.870805 Epoch: 834/1000:, Cur-Step: 16680, loss(cross_entropy): 0.00330, Running average loss:0.00330, Time taken: 0:00:14.737475 ETA: 0:40:46.420797 2022-07-28 19:58:52,429 [INFO] __main__: Epoch: 834/1000:, Cur-Step: 16680, loss(cross_entropy): 0.00330, Running average loss:0.00330, Time taken: 0:00:14.737475 ETA: 0:40:46.420797 Epoch: 834/1000:, Cur-Step: 16690, loss(cross_entropy): 0.00322, Running average loss:0.00320, Time taken: 0:00:14.737475 ETA: 0:40:46.420797 2022-07-28 19:58:59,370 [INFO] __main__: Epoch: 834/1000:, Cur-Step: 16690, loss(cross_entropy): 0.00322, Running average loss:0.00320, Time taken: 0:00:14.737475 ETA: 0:40:46.420797 Epoch: 835/1000:, Cur-Step: 16700, loss(cross_entropy): 0.00303, Running average loss:0.00303, Time taken: 0:00:14.568667 ETA: 0:40:03.830044 2022-07-28 19:59:06,300 [INFO] __main__: Epoch: 835/1000:, Cur-Step: 16700, loss(cross_entropy): 0.00303, Running average loss:0.00303, Time taken: 0:00:14.568667 ETA: 0:40:03.830044 Epoch: 835/1000:, Cur-Step: 16710, loss(cross_entropy): 0.00297, Running average loss:0.00316, Time taken: 0:00:14.568667 ETA: 0:40:03.830044 2022-07-28 19:59:13,264 [INFO] __main__: Epoch: 835/1000:, Cur-Step: 16710, loss(cross_entropy): 0.00297, Running average loss:0.00316, Time taken: 0:00:14.568667 ETA: 0:40:03.830044 Epoch: 836/1000:, Cur-Step: 16720, loss(cross_entropy): 0.00313, Running average loss:0.00313, Time taken: 0:00:14.715269 ETA: 0:40:13.304131 2022-07-28 19:59:20,327 [INFO] __main__: Epoch: 836/1000:, Cur-Step: 16720, loss(cross_entropy): 0.00313, Running average loss:0.00313, Time taken: 0:00:14.715269 ETA: 0:40:13.304131 Epoch: 836/1000:, Cur-Step: 16730, loss(cross_entropy): 0.00296, Running average loss:0.00305, Time taken: 0:00:14.715269 ETA: 0:40:13.304131 2022-07-28 19:59:27,299 [INFO] __main__: Epoch: 836/1000:, Cur-Step: 16730, loss(cross_entropy): 0.00296, Running average loss:0.00305, Time taken: 0:00:14.715269 ETA: 0:40:13.304131 Epoch: 837/1000:, Cur-Step: 16740, loss(cross_entropy): 0.00311, Running average loss:0.00311, Time taken: 0:00:14.708926 ETA: 0:39:57.554971 2022-07-28 19:59:34,302 [INFO] __main__: Epoch: 837/1000:, Cur-Step: 16740, loss(cross_entropy): 0.00311, Running average loss:0.00311, Time taken: 0:00:14.708926 ETA: 0:39:57.554971 Epoch: 837/1000:, Cur-Step: 16750, loss(cross_entropy): 0.00284, Running average loss:0.00299, Time taken: 0:00:14.708926 ETA: 0:39:57.554971 2022-07-28 19:59:41,217 [INFO] __main__: Epoch: 837/1000:, Cur-Step: 16750, loss(cross_entropy): 0.00284, Running average loss:0.00299, Time taken: 0:00:14.708926 ETA: 0:39:57.554971 Epoch: 838/1000:, Cur-Step: 16760, loss(cross_entropy): 0.00306, Running average loss:0.00306, Time taken: 0:00:14.687861 ETA: 0:39:39.433554 2022-07-28 19:59:48,229 [INFO] __main__: Epoch: 838/1000:, Cur-Step: 16760, loss(cross_entropy): 0.00306, Running average loss:0.00306, Time taken: 0:00:14.687861 ETA: 0:39:39.433554 Epoch: 838/1000:, Cur-Step: 16770, loss(cross_entropy): 0.00311, Running average loss:0.00310, Time taken: 0:00:14.687861 ETA: 0:39:39.433554 2022-07-28 19:59:55,284 [INFO] __main__: Epoch: 838/1000:, Cur-Step: 16770, loss(cross_entropy): 0.00311, Running average loss:0.00310, Time taken: 0:00:14.687861 ETA: 0:39:39.433554 Epoch: 839/1000:, Cur-Step: 16780, loss(cross_entropy): 0.00316, Running average loss:0.00316, Time taken: 0:00:14.749898 ETA: 0:39:34.733533 2022-07-28 20:00:02,228 [INFO] __main__: Epoch: 839/1000:, Cur-Step: 16780, loss(cross_entropy): 0.00316, Running average loss:0.00316, Time taken: 0:00:14.749898 ETA: 0:39:34.733533 Epoch: 839/1000:, Cur-Step: 16790, loss(cross_entropy): 0.00304, Running average loss:0.00312, Time taken: 0:00:14.749898 ETA: 0:39:34.733533 2022-07-28 20:00:09,142 [INFO] __main__: Epoch: 839/1000:, Cur-Step: 16790, loss(cross_entropy): 0.00304, Running average loss:0.00312, Time taken: 0:00:14.749898 ETA: 0:39:34.733533 Epoch: 840/1000:, Cur-Step: 16800, loss(cross_entropy): 0.00320, Running average loss:0.00320, Time taken: 0:00:14.590901 ETA: 0:38:54.544144 2022-07-28 20:00:16,122 [INFO] __main__: Epoch: 840/1000:, Cur-Step: 16800, loss(cross_entropy): 0.00320, Running average loss:0.00320, Time taken: 0:00:14.590901 ETA: 0:38:54.544144 Epoch: 840/1000:, Cur-Step: 16810, loss(cross_entropy): 0.00295, Running average loss:0.00296, Time taken: 0:00:14.590901 ETA: 0:38:54.544144 2022-07-28 20:00:23,131 [INFO] __main__: Epoch: 840/1000:, Cur-Step: 16810, loss(cross_entropy): 0.00295, Running average loss:0.00296, Time taken: 0:00:14.590901 ETA: 0:38:54.544144 Epoch: 841/1000:, Cur-Step: 16820, loss(cross_entropy): 0.00282, Running average loss:0.00282, Time taken: 0:00:14.690030 ETA: 0:38:55.714710 2022-07-28 20:00:30,095 [INFO] __main__: Epoch: 841/1000:, Cur-Step: 16820, loss(cross_entropy): 0.00282, Running average loss:0.00282, Time taken: 0:00:14.690030 ETA: 0:38:55.714710 Epoch: 841/1000:, Cur-Step: 16830, loss(cross_entropy): 0.00269, Running average loss:0.00292, Time taken: 0:00:14.690030 ETA: 0:38:55.714710 2022-07-28 20:00:36,962 [INFO] __main__: Epoch: 841/1000:, Cur-Step: 16830, loss(cross_entropy): 0.00269, Running average loss:0.00292, Time taken: 0:00:14.690030 ETA: 0:38:55.714710 Epoch: 842/1000:, Cur-Step: 16840, loss(cross_entropy): 0.00277, Running average loss:0.00277, Time taken: 0:00:14.513684 ETA: 0:38:13.162115 2022-07-28 20:00:43,952 [INFO] __main__: Epoch: 842/1000:, Cur-Step: 16840, loss(cross_entropy): 0.00277, Running average loss:0.00277, Time taken: 0:00:14.513684 ETA: 0:38:13.162115 Epoch: 842/1000:, Cur-Step: 16850, loss(cross_entropy): 0.00269, Running average loss:0.00285, Time taken: 0:00:14.513684 ETA: 0:38:13.162115 2022-07-28 20:00:50,919 [INFO] __main__: Epoch: 842/1000:, Cur-Step: 16850, loss(cross_entropy): 0.00269, Running average loss:0.00285, Time taken: 0:00:14.513684 ETA: 0:38:13.162115 Epoch: 843/1000:, Cur-Step: 16860, loss(cross_entropy): 0.00273, Running average loss:0.00273, Time taken: 0:00:14.672525 ETA: 0:38:23.586489 2022-07-28 20:00:57,935 [INFO] __main__: Epoch: 843/1000:, Cur-Step: 16860, loss(cross_entropy): 0.00273, Running average loss:0.00273, Time taken: 0:00:14.672525 ETA: 0:38:23.586489 Epoch: 843/1000:, Cur-Step: 16870, loss(cross_entropy): 0.00277, Running average loss:0.00286, Time taken: 0:00:14.672525 ETA: 0:38:23.586489 2022-07-28 20:01:04,860 [INFO] __main__: Epoch: 843/1000:, Cur-Step: 16870, loss(cross_entropy): 0.00277, Running average loss:0.00286, Time taken: 0:00:14.672525 ETA: 0:38:23.586489 Epoch: 844/1000:, Cur-Step: 16880, loss(cross_entropy): 0.00294, Running average loss:0.00294, Time taken: 0:00:14.548151 ETA: 0:37:49.511521 2022-07-28 20:01:11,822 [INFO] __main__: Epoch: 844/1000:, Cur-Step: 16880, loss(cross_entropy): 0.00294, Running average loss:0.00294, Time taken: 0:00:14.548151 ETA: 0:37:49.511521 Epoch: 844/1000:, Cur-Step: 16890, loss(cross_entropy): 0.00273, Running average loss:0.00293, Time taken: 0:00:14.548151 ETA: 0:37:49.511521 2022-07-28 20:01:18,765 [INFO] __main__: Epoch: 844/1000:, Cur-Step: 16890, loss(cross_entropy): 0.00273, Running average loss:0.00293, Time taken: 0:00:14.548151 ETA: 0:37:49.511521 Epoch: 845/1000:, Cur-Step: 16900, loss(cross_entropy): 0.00268, Running average loss:0.00268, Time taken: 0:00:14.711403 ETA: 0:38:00.267485 2022-07-28 20:01:25,891 [INFO] __main__: Epoch: 845/1000:, Cur-Step: 16900, loss(cross_entropy): 0.00268, Running average loss:0.00268, Time taken: 0:00:14.711403 ETA: 0:38:00.267485 Epoch: 845/1000:, Cur-Step: 16910, loss(cross_entropy): 0.00298, Running average loss:0.00305, Time taken: 0:00:14.711403 ETA: 0:38:00.267485 2022-07-28 20:01:32,797 [INFO] __main__: Epoch: 845/1000:, Cur-Step: 16910, loss(cross_entropy): 0.00298, Running average loss:0.00305, Time taken: 0:00:14.711403 ETA: 0:38:00.267485 Epoch: 846/1000:, Cur-Step: 16920, loss(cross_entropy): 0.00305, Running average loss:0.00305, Time taken: 0:00:14.592082 ETA: 0:37:27.180632 2022-07-28 20:01:39,754 [INFO] __main__: Epoch: 846/1000:, Cur-Step: 16920, loss(cross_entropy): 0.00305, Running average loss:0.00305, Time taken: 0:00:14.592082 ETA: 0:37:27.180632 Epoch: 846/1000:, Cur-Step: 16930, loss(cross_entropy): 0.00277, Running average loss:0.00319, Time taken: 0:00:14.592082 ETA: 0:37:27.180632 2022-07-28 20:01:46,648 [INFO] __main__: Epoch: 846/1000:, Cur-Step: 16930, loss(cross_entropy): 0.00277, Running average loss:0.00319, Time taken: 0:00:14.592082 ETA: 0:37:27.180632 Epoch: 847/1000:, Cur-Step: 16940, loss(cross_entropy): 0.00314, Running average loss:0.00314, Time taken: 0:00:14.636727 ETA: 0:37:19.419282 2022-07-28 20:01:53,752 [INFO] __main__: Epoch: 847/1000:, Cur-Step: 16940, loss(cross_entropy): 0.00314, Running average loss:0.00314, Time taken: 0:00:14.636727 ETA: 0:37:19.419282 Epoch: 847/1000:, Cur-Step: 16950, loss(cross_entropy): 0.00303, Running average loss:0.00311, Time taken: 0:00:14.636727 ETA: 0:37:19.419282 2022-07-28 20:02:00,726 [INFO] __main__: Epoch: 847/1000:, Cur-Step: 16950, loss(cross_entropy): 0.00303, Running average loss:0.00311, Time taken: 0:00:14.636727 ETA: 0:37:19.419282 Epoch: 848/1000:, Cur-Step: 16960, loss(cross_entropy): 0.00323, Running average loss:0.00323, Time taken: 0:00:14.735806 ETA: 0:37:19.842474 2022-07-28 20:02:07,764 [INFO] __main__: Epoch: 848/1000:, Cur-Step: 16960, loss(cross_entropy): 0.00323, Running average loss:0.00323, Time taken: 0:00:14.735806 ETA: 0:37:19.842474 Epoch: 848/1000:, Cur-Step: 16970, loss(cross_entropy): 0.00314, Running average loss:0.00313, Time taken: 0:00:14.735806 ETA: 0:37:19.842474 2022-07-28 20:02:14,629 [INFO] __main__: Epoch: 848/1000:, Cur-Step: 16970, loss(cross_entropy): 0.00314, Running average loss:0.00313, Time taken: 0:00:14.735806 ETA: 0:37:19.842474 Epoch: 849/1000:, Cur-Step: 16980, loss(cross_entropy): 0.00316, Running average loss:0.00316, Time taken: 0:00:14.602803 ETA: 0:36:45.023252 2022-07-28 20:02:21,683 [INFO] __main__: Epoch: 849/1000:, Cur-Step: 16980, loss(cross_entropy): 0.00316, Running average loss:0.00316, Time taken: 0:00:14.602803 ETA: 0:36:45.023252 Epoch: 849/1000:, Cur-Step: 16990, loss(cross_entropy): 0.00357, Running average loss:0.00322, Time taken: 0:00:14.602803 ETA: 0:36:45.023252 2022-07-28 20:02:28,736 [INFO] __main__: Epoch: 849/1000:, Cur-Step: 16990, loss(cross_entropy): 0.00357, Running average loss:0.00322, Time taken: 0:00:14.602803 ETA: 0:36:45.023252 Epoch: 850/1000:, Cur-Step: 17000, loss(cross_entropy): 0.00324, Running average loss:0.00324, Time taken: 0:00:14.703039 ETA: 0:36:45.455804 2022-07-28 20:02:35,666 [INFO] __main__: Epoch: 850/1000:, Cur-Step: 17000, loss(cross_entropy): 0.00324, Running average loss:0.00324, Time taken: 0:00:14.703039 ETA: 0:36:45.455804 Epoch: 850/1000:, Cur-Step: 17010, loss(cross_entropy): 0.00319, Running average loss:0.00314, Time taken: 0:00:14.703039 ETA: 0:36:45.455804 2022-07-28 20:02:42,508 [INFO] __main__: Epoch: 850/1000:, Cur-Step: 17010, loss(cross_entropy): 0.00319, Running average loss:0.00314, Time taken: 0:00:14.703039 ETA: 0:36:45.455804 Epoch: 851/1000:, Cur-Step: 17020, loss(cross_entropy): 0.00301, Running average loss:0.00301, Time taken: 0:00:14.530965 ETA: 0:36:05.113798 2022-07-28 20:02:49,544 [INFO] __main__: Epoch: 851/1000:, Cur-Step: 17020, loss(cross_entropy): 0.00301, Running average loss:0.00301, Time taken: 0:00:14.530965 ETA: 0:36:05.113798 Epoch: 851/1000:, Cur-Step: 17030, loss(cross_entropy): 0.00333, Running average loss:0.00306, Time taken: 0:00:14.530965 ETA: 0:36:05.113798 2022-07-28 20:02:56,601 [INFO] __main__: Epoch: 851/1000:, Cur-Step: 17030, loss(cross_entropy): 0.00333, Running average loss:0.00306, Time taken: 0:00:14.530965 ETA: 0:36:05.113798 Epoch: 852/1000:, Cur-Step: 17040, loss(cross_entropy): 0.00305, Running average loss:0.00305, Time taken: 0:00:14.730176 ETA: 0:36:20.066114 2022-07-28 20:03:03,567 [INFO] __main__: Epoch: 852/1000:, Cur-Step: 17040, loss(cross_entropy): 0.00305, Running average loss:0.00305, Time taken: 0:00:14.730176 ETA: 0:36:20.066114 Epoch: 852/1000:, Cur-Step: 17050, loss(cross_entropy): 0.00298, Running average loss:0.00300, Time taken: 0:00:14.730176 ETA: 0:36:20.066114 2022-07-28 20:03:10,433 [INFO] __main__: Epoch: 852/1000:, Cur-Step: 17050, loss(cross_entropy): 0.00298, Running average loss:0.00300, Time taken: 0:00:14.730176 ETA: 0:36:20.066114 Epoch: 853/1000:, Cur-Step: 17060, loss(cross_entropy): 0.00302, Running average loss:0.00302, Time taken: 0:00:14.489944 ETA: 0:35:30.021835 2022-07-28 20:03:17,389 [INFO] __main__: Epoch: 853/1000:, Cur-Step: 17060, loss(cross_entropy): 0.00302, Running average loss:0.00302, Time taken: 0:00:14.489944 ETA: 0:35:30.021835 Epoch: 853/1000:, Cur-Step: 17070, loss(cross_entropy): 0.00313, Running average loss:0.00302, Time taken: 0:00:14.489944 ETA: 0:35:30.021835 2022-07-28 20:03:24,371 [INFO] __main__: Epoch: 853/1000:, Cur-Step: 17070, loss(cross_entropy): 0.00313, Running average loss:0.00302, Time taken: 0:00:14.489944 ETA: 0:35:30.021835 Epoch: 854/1000:, Cur-Step: 17080, loss(cross_entropy): 0.00311, Running average loss:0.00311, Time taken: 0:00:14.657920 ETA: 0:35:40.056268 2022-07-28 20:03:31,355 [INFO] __main__: Epoch: 854/1000:, Cur-Step: 17080, loss(cross_entropy): 0.00311, Running average loss:0.00311, Time taken: 0:00:14.657920 ETA: 0:35:40.056268 Epoch: 854/1000:, Cur-Step: 17090, loss(cross_entropy): 0.00294, Running average loss:0.00308, Time taken: 0:00:14.657920 ETA: 0:35:40.056268 2022-07-28 20:03:38,315 [INFO] __main__: Epoch: 854/1000:, Cur-Step: 17090, loss(cross_entropy): 0.00294, Running average loss:0.00308, Time taken: 0:00:14.657920 ETA: 0:35:40.056268 Epoch: 855/1000:, Cur-Step: 17100, loss(cross_entropy): 0.00304, Running average loss:0.00304, Time taken: 0:00:14.577964 ETA: 0:35:13.804755 2022-07-28 20:03:45,273 [INFO] __main__: Epoch: 855/1000:, Cur-Step: 17100, loss(cross_entropy): 0.00304, Running average loss:0.00304, Time taken: 0:00:14.577964 ETA: 0:35:13.804755 Epoch: 855/1000:, Cur-Step: 17110, loss(cross_entropy): 0.00341, Running average loss:0.00331, Time taken: 0:00:14.577964 ETA: 0:35:13.804755 2022-07-28 20:03:52,206 [INFO] __main__: Epoch: 855/1000:, Cur-Step: 17110, loss(cross_entropy): 0.00341, Running average loss:0.00331, Time taken: 0:00:14.577964 ETA: 0:35:13.804755 Epoch: 856/1000:, Cur-Step: 17120, loss(cross_entropy): 0.00325, Running average loss:0.00325, Time taken: 0:00:14.647599 ETA: 0:35:09.254219 2022-07-28 20:03:59,207 [INFO] __main__: Epoch: 856/1000:, Cur-Step: 17120, loss(cross_entropy): 0.00325, Running average loss:0.00325, Time taken: 0:00:14.647599 ETA: 0:35:09.254219 Epoch: 856/1000:, Cur-Step: 17130, loss(cross_entropy): 0.00300, Running average loss:0.00325, Time taken: 0:00:14.647599 ETA: 0:35:09.254219 2022-07-28 20:04:06,156 [INFO] __main__: Epoch: 856/1000:, Cur-Step: 17130, loss(cross_entropy): 0.00300, Running average loss:0.00325, Time taken: 0:00:14.647599 ETA: 0:35:09.254219 Epoch: 857/1000:, Cur-Step: 17140, loss(cross_entropy): 0.00290, Running average loss:0.00290, Time taken: 0:00:14.554557 ETA: 0:34:41.301595 2022-07-28 20:04:13,069 [INFO] __main__: Epoch: 857/1000:, Cur-Step: 17140, loss(cross_entropy): 0.00290, Running average loss:0.00290, Time taken: 0:00:14.554557 ETA: 0:34:41.301595 Epoch: 857/1000:, Cur-Step: 17150, loss(cross_entropy): 0.00309, Running average loss:0.00302, Time taken: 0:00:14.554557 ETA: 0:34:41.301595 2022-07-28 20:04:20,026 [INFO] __main__: Epoch: 857/1000:, Cur-Step: 17150, loss(cross_entropy): 0.00309, Running average loss:0.00302, Time taken: 0:00:14.554557 ETA: 0:34:41.301595 Epoch: 858/1000:, Cur-Step: 17160, loss(cross_entropy): 0.00283, Running average loss:0.00283, Time taken: 0:00:14.650820 ETA: 0:34:40.416442 2022-07-28 20:04:27,058 [INFO] __main__: Epoch: 858/1000:, Cur-Step: 17160, loss(cross_entropy): 0.00283, Running average loss:0.00283, Time taken: 0:00:14.650820 ETA: 0:34:40.416442 Epoch: 858/1000:, Cur-Step: 17170, loss(cross_entropy): 0.00302, Running average loss:0.00287, Time taken: 0:00:14.650820 ETA: 0:34:40.416442 2022-07-28 20:04:34,052 [INFO] __main__: Epoch: 858/1000:, Cur-Step: 17170, loss(cross_entropy): 0.00302, Running average loss:0.00287, Time taken: 0:00:14.650820 ETA: 0:34:40.416442 Epoch: 859/1000:, Cur-Step: 17180, loss(cross_entropy): 0.00277, Running average loss:0.00277, Time taken: 0:00:14.603385 ETA: 0:34:19.077315 2022-07-28 20:04:40,955 [INFO] __main__: Epoch: 859/1000:, Cur-Step: 17180, loss(cross_entropy): 0.00277, Running average loss:0.00277, Time taken: 0:00:14.603385 ETA: 0:34:19.077315 Epoch: 859/1000:, Cur-Step: 17190, loss(cross_entropy): 0.00263, Running average loss:0.00278, Time taken: 0:00:14.603385 ETA: 0:34:19.077315 2022-07-28 20:04:47,873 [INFO] __main__: Epoch: 859/1000:, Cur-Step: 17190, loss(cross_entropy): 0.00263, Running average loss:0.00278, Time taken: 0:00:14.603385 ETA: 0:34:19.077315 Epoch: 860/1000:, Cur-Step: 17200, loss(cross_entropy): 0.00288, Running average loss:0.00288, Time taken: 0:00:14.511669 ETA: 0:33:51.633682 2022-07-28 20:04:54,791 [INFO] __main__: Epoch: 860/1000:, Cur-Step: 17200, loss(cross_entropy): 0.00288, Running average loss:0.00288, Time taken: 0:00:14.511669 ETA: 0:33:51.633682 Epoch: 860/1000:, Cur-Step: 17210, loss(cross_entropy): 0.00275, Running average loss:0.00278, Time taken: 0:00:14.511669 ETA: 0:33:51.633682 2022-07-28 20:05:01,839 [INFO] __main__: Epoch: 860/1000:, Cur-Step: 17210, loss(cross_entropy): 0.00275, Running average loss:0.00278, Time taken: 0:00:14.511669 ETA: 0:33:51.633682 Epoch: 861/1000:, Cur-Step: 17220, loss(cross_entropy): 0.00272, Running average loss:0.00272, Time taken: 0:00:14.585586 ETA: 0:33:47.396497 2022-07-28 20:05:08,707 [INFO] __main__: Epoch: 861/1000:, Cur-Step: 17220, loss(cross_entropy): 0.00272, Running average loss:0.00272, Time taken: 0:00:14.585586 ETA: 0:33:47.396497 Epoch: 861/1000:, Cur-Step: 17230, loss(cross_entropy): 0.00265, Running average loss:0.00275, Time taken: 0:00:14.585586 ETA: 0:33:47.396497 2022-07-28 20:05:15,683 [INFO] __main__: Epoch: 861/1000:, Cur-Step: 17230, loss(cross_entropy): 0.00265, Running average loss:0.00275, Time taken: 0:00:14.585586 ETA: 0:33:47.396497 Epoch: 862/1000:, Cur-Step: 17240, loss(cross_entropy): 0.00278, Running average loss:0.00278, Time taken: 0:00:14.527359 ETA: 0:33:24.775609 2022-07-28 20:05:22,537 [INFO] __main__: Epoch: 862/1000:, Cur-Step: 17240, loss(cross_entropy): 0.00278, Running average loss:0.00278, Time taken: 0:00:14.527359 ETA: 0:33:24.775609 Epoch: 862/1000:, Cur-Step: 17250, loss(cross_entropy): 0.00279, Running average loss:0.00283, Time taken: 0:00:14.527359 ETA: 0:33:24.775609 2022-07-28 20:05:29,660 [INFO] __main__: Epoch: 862/1000:, Cur-Step: 17250, loss(cross_entropy): 0.00279, Running average loss:0.00283, Time taken: 0:00:14.527359 ETA: 0:33:24.775609 Epoch: 863/1000:, Cur-Step: 17260, loss(cross_entropy): 0.00289, Running average loss:0.00289, Time taken: 0:00:14.642297 ETA: 0:33:25.994628 2022-07-28 20:05:36,540 [INFO] __main__: Epoch: 863/1000:, Cur-Step: 17260, loss(cross_entropy): 0.00289, Running average loss:0.00289, Time taken: 0:00:14.642297 ETA: 0:33:25.994628 Epoch: 863/1000:, Cur-Step: 17270, loss(cross_entropy): 0.00289, Running average loss:0.00300, Time taken: 0:00:14.642297 ETA: 0:33:25.994628 2022-07-28 20:05:43,497 [INFO] __main__: Epoch: 863/1000:, Cur-Step: 17270, loss(cross_entropy): 0.00289, Running average loss:0.00300, Time taken: 0:00:14.642297 ETA: 0:33:25.994628 Epoch: 864/1000:, Cur-Step: 17280, loss(cross_entropy): 0.00335, Running average loss:0.00335, Time taken: 0:00:14.496608 ETA: 0:32:51.538658 2022-07-28 20:05:50,342 [INFO] __main__: Epoch: 864/1000:, Cur-Step: 17280, loss(cross_entropy): 0.00335, Running average loss:0.00335, Time taken: 0:00:14.496608 ETA: 0:32:51.538658 Epoch: 864/1000:, Cur-Step: 17290, loss(cross_entropy): 0.00310, Running average loss:0.00308, Time taken: 0:00:14.496608 ETA: 0:32:51.538658 2022-07-28 20:05:57,454 [INFO] __main__: Epoch: 864/1000:, Cur-Step: 17290, loss(cross_entropy): 0.00310, Running average loss:0.00308, Time taken: 0:00:14.496608 ETA: 0:32:51.538658 Epoch: 865/1000:, Cur-Step: 17300, loss(cross_entropy): 0.00288, Running average loss:0.00288, Time taken: 0:00:14.731272 ETA: 0:33:08.721718 2022-07-28 20:06:04,419 [INFO] __main__: Epoch: 865/1000:, Cur-Step: 17300, loss(cross_entropy): 0.00288, Running average loss:0.00288, Time taken: 0:00:14.731272 ETA: 0:33:08.721718 Epoch: 865/1000:, Cur-Step: 17310, loss(cross_entropy): 0.00286, Running average loss:0.00295, Time taken: 0:00:14.731272 ETA: 0:33:08.721718 2022-07-28 20:06:11,363 [INFO] __main__: Epoch: 865/1000:, Cur-Step: 17310, loss(cross_entropy): 0.00286, Running average loss:0.00295, Time taken: 0:00:14.731272 ETA: 0:33:08.721718 Epoch: 866/1000:, Cur-Step: 17320, loss(cross_entropy): 0.00308, Running average loss:0.00308, Time taken: 0:00:14.503802 ETA: 0:32:23.509444 2022-07-28 20:06:18,235 [INFO] __main__: Epoch: 866/1000:, Cur-Step: 17320, loss(cross_entropy): 0.00308, Running average loss:0.00308, Time taken: 0:00:14.503802 ETA: 0:32:23.509444 Epoch: 866/1000:, Cur-Step: 17330, loss(cross_entropy): 0.00301, Running average loss:0.00292, Time taken: 0:00:14.503802 ETA: 0:32:23.509444 2022-07-28 20:06:25,270 [INFO] __main__: Epoch: 866/1000:, Cur-Step: 17330, loss(cross_entropy): 0.00301, Running average loss:0.00292, Time taken: 0:00:14.503802 ETA: 0:32:23.509444 Epoch: 867/1000:, Cur-Step: 17340, loss(cross_entropy): 0.00292, Running average loss:0.00292, Time taken: 0:00:14.774929 ETA: 0:32:45.065563 2022-07-28 20:06:32,323 [INFO] __main__: Epoch: 867/1000:, Cur-Step: 17340, loss(cross_entropy): 0.00292, Running average loss:0.00292, Time taken: 0:00:14.774929 ETA: 0:32:45.065563 Epoch: 867/1000:, Cur-Step: 17350, loss(cross_entropy): 0.00273, Running average loss:0.00281, Time taken: 0:00:14.774929 ETA: 0:32:45.065563 2022-07-28 20:06:39,286 [INFO] __main__: Epoch: 867/1000:, Cur-Step: 17350, loss(cross_entropy): 0.00273, Running average loss:0.00281, Time taken: 0:00:14.774929 ETA: 0:32:45.065563 Epoch: 868/1000:, Cur-Step: 17360, loss(cross_entropy): 0.00298, Running average loss:0.00298, Time taken: 0:00:14.505397 ETA: 0:31:54.712415 2022-07-28 20:06:46,160 [INFO] __main__: Epoch: 868/1000:, Cur-Step: 17360, loss(cross_entropy): 0.00298, Running average loss:0.00298, Time taken: 0:00:14.505397 ETA: 0:31:54.712415 Epoch: 868/1000:, Cur-Step: 17370, loss(cross_entropy): 0.00280, Running average loss:0.00279, Time taken: 0:00:14.505397 ETA: 0:31:54.712415 2022-07-28 20:06:53,097 [INFO] __main__: Epoch: 868/1000:, Cur-Step: 17370, loss(cross_entropy): 0.00280, Running average loss:0.00279, Time taken: 0:00:14.505397 ETA: 0:31:54.712415 Epoch: 869/1000:, Cur-Step: 17380, loss(cross_entropy): 0.00276, Running average loss:0.00276, Time taken: 0:00:14.638229 ETA: 0:31:57.607985 2022-07-28 20:07:00,126 [INFO] __main__: Epoch: 869/1000:, Cur-Step: 17380, loss(cross_entropy): 0.00276, Running average loss:0.00276, Time taken: 0:00:14.638229 ETA: 0:31:57.607985 Epoch: 869/1000:, Cur-Step: 17390, loss(cross_entropy): 0.00310, Running average loss:0.00285, Time taken: 0:00:14.638229 ETA: 0:31:57.607985 2022-07-28 20:07:07,129 [INFO] __main__: Epoch: 869/1000:, Cur-Step: 17390, loss(cross_entropy): 0.00310, Running average loss:0.00285, Time taken: 0:00:14.638229 ETA: 0:31:57.607985 Epoch: 870/1000:, Cur-Step: 17400, loss(cross_entropy): 0.00305, Running average loss:0.00305, Time taken: 0:00:14.572517 ETA: 0:31:34.427168 2022-07-28 20:07:13,985 [INFO] __main__: Epoch: 870/1000:, Cur-Step: 17400, loss(cross_entropy): 0.00305, Running average loss:0.00305, Time taken: 0:00:14.572517 ETA: 0:31:34.427168 Epoch: 870/1000:, Cur-Step: 17410, loss(cross_entropy): 0.00285, Running average loss:0.00285, Time taken: 0:00:14.572517 ETA: 0:31:34.427168 2022-07-28 20:07:20,892 [INFO] __main__: Epoch: 870/1000:, Cur-Step: 17410, loss(cross_entropy): 0.00285, Running average loss:0.00285, Time taken: 0:00:14.572517 ETA: 0:31:34.427168 Epoch: 871/1000:, Cur-Step: 17420, loss(cross_entropy): 0.00280, Running average loss:0.00280, Time taken: 0:00:14.533712 ETA: 0:31:14.848898 2022-07-28 20:07:27,816 [INFO] __main__: Epoch: 871/1000:, Cur-Step: 17420, loss(cross_entropy): 0.00280, Running average loss:0.00280, Time taken: 0:00:14.533712 ETA: 0:31:14.848898 Epoch: 871/1000:, Cur-Step: 17430, loss(cross_entropy): 0.00272, Running average loss:0.00289, Time taken: 0:00:14.533712 ETA: 0:31:14.848898 2022-07-28 20:07:34,940 [INFO] __main__: Epoch: 871/1000:, Cur-Step: 17430, loss(cross_entropy): 0.00272, Running average loss:0.00289, Time taken: 0:00:14.533712 ETA: 0:31:14.848898 Epoch: 872/1000:, Cur-Step: 17440, loss(cross_entropy): 0.00323, Running average loss:0.00323, Time taken: 0:00:14.725253 ETA: 0:31:24.832397 2022-07-28 20:07:41,825 [INFO] __main__: Epoch: 872/1000:, Cur-Step: 17440, loss(cross_entropy): 0.00323, Running average loss:0.00323, Time taken: 0:00:14.725253 ETA: 0:31:24.832397 Epoch: 872/1000:, Cur-Step: 17450, loss(cross_entropy): 0.00291, Running average loss:0.00300, Time taken: 0:00:14.725253 ETA: 0:31:24.832397 2022-07-28 20:07:48,811 [INFO] __main__: Epoch: 872/1000:, Cur-Step: 17450, loss(cross_entropy): 0.00291, Running average loss:0.00300, Time taken: 0:00:14.725253 ETA: 0:31:24.832397 Epoch: 873/1000:, Cur-Step: 17460, loss(cross_entropy): 0.00301, Running average loss:0.00301, Time taken: 0:00:14.548465 ETA: 0:30:47.655057 2022-07-28 20:07:55,721 [INFO] __main__: Epoch: 873/1000:, Cur-Step: 17460, loss(cross_entropy): 0.00301, Running average loss:0.00301, Time taken: 0:00:14.548465 ETA: 0:30:47.655057 Epoch: 873/1000:, Cur-Step: 17470, loss(cross_entropy): 0.00312, Running average loss:0.00307, Time taken: 0:00:14.548465 ETA: 0:30:47.655057 2022-07-28 20:08:02,831 [INFO] __main__: Epoch: 873/1000:, Cur-Step: 17470, loss(cross_entropy): 0.00312, Running average loss:0.00307, Time taken: 0:00:14.548465 ETA: 0:30:47.655057 Epoch: 874/1000:, Cur-Step: 17480, loss(cross_entropy): 0.00298, Running average loss:0.00298, Time taken: 0:00:14.746196 ETA: 0:30:58.020640 2022-07-28 20:08:09,776 [INFO] __main__: Epoch: 874/1000:, Cur-Step: 17480, loss(cross_entropy): 0.00298, Running average loss:0.00298, Time taken: 0:00:14.746196 ETA: 0:30:58.020640 Epoch: 874/1000:, Cur-Step: 17490, loss(cross_entropy): 0.00286, Running average loss:0.00302, Time taken: 0:00:14.746196 ETA: 0:30:58.020640 2022-07-28 20:08:16,744 [INFO] __main__: Epoch: 874/1000:, Cur-Step: 17490, loss(cross_entropy): 0.00286, Running average loss:0.00302, Time taken: 0:00:14.746196 ETA: 0:30:58.020640 Epoch: 875/1000:, Cur-Step: 17500, loss(cross_entropy): 0.00310, Running average loss:0.00310, Time taken: 0:00:14.662514 ETA: 0:30:32.814276 2022-07-28 20:08:23,677 [INFO] __main__: Epoch: 875/1000:, Cur-Step: 17500, loss(cross_entropy): 0.00310, Running average loss:0.00310, Time taken: 0:00:14.662514 ETA: 0:30:32.814276 Epoch: 875/1000:, Cur-Step: 17510, loss(cross_entropy): 0.00311, Running average loss:0.00305, Time taken: 0:00:14.662514 ETA: 0:30:32.814276 2022-07-28 20:08:30,673 [INFO] __main__: Epoch: 875/1000:, Cur-Step: 17510, loss(cross_entropy): 0.00311, Running average loss:0.00305, Time taken: 0:00:14.662514 ETA: 0:30:32.814276 Epoch: 876/1000:, Cur-Step: 17520, loss(cross_entropy): 0.00362, Running average loss:0.00362, Time taken: 0:00:14.718924 ETA: 0:30:25.146582 2022-07-28 20:08:37,692 [INFO] __main__: Epoch: 876/1000:, Cur-Step: 17520, loss(cross_entropy): 0.00362, Running average loss:0.00362, Time taken: 0:00:14.718924 ETA: 0:30:25.146582 Epoch: 876/1000:, Cur-Step: 17530, loss(cross_entropy): 0.00333, Running average loss:0.00335, Time taken: 0:00:14.718924 ETA: 0:30:25.146582 2022-07-28 20:08:44,570 [INFO] __main__: Epoch: 876/1000:, Cur-Step: 17530, loss(cross_entropy): 0.00333, Running average loss:0.00335, Time taken: 0:00:14.718924 ETA: 0:30:25.146582 Epoch: 877/1000:, Cur-Step: 17540, loss(cross_entropy): 0.00304, Running average loss:0.00304, Time taken: 0:00:14.602493 ETA: 0:29:56.106616 2022-07-28 20:08:51,509 [INFO] __main__: Epoch: 877/1000:, Cur-Step: 17540, loss(cross_entropy): 0.00304, Running average loss:0.00304, Time taken: 0:00:14.602493 ETA: 0:29:56.106616 Epoch: 877/1000:, Cur-Step: 17550, loss(cross_entropy): 0.00337, Running average loss:0.00319, Time taken: 0:00:14.602493 ETA: 0:29:56.106616 2022-07-28 20:08:58,404 [INFO] __main__: Epoch: 877/1000:, Cur-Step: 17550, loss(cross_entropy): 0.00337, Running average loss:0.00319, Time taken: 0:00:14.602493 ETA: 0:29:56.106616 Epoch: 878/1000:, Cur-Step: 17560, loss(cross_entropy): 0.00323, Running average loss:0.00323, Time taken: 0:00:14.642767 ETA: 0:29:46.417626 2022-07-28 20:09:05,429 [INFO] __main__: Epoch: 878/1000:, Cur-Step: 17560, loss(cross_entropy): 0.00323, Running average loss:0.00323, Time taken: 0:00:14.642767 ETA: 0:29:46.417626 Epoch: 878/1000:, Cur-Step: 17570, loss(cross_entropy): 0.00319, Running average loss:0.00316, Time taken: 0:00:14.642767 ETA: 0:29:46.417626 2022-07-28 20:09:12,303 [INFO] __main__: Epoch: 878/1000:, Cur-Step: 17570, loss(cross_entropy): 0.00319, Running average loss:0.00316, Time taken: 0:00:14.642767 ETA: 0:29:46.417626 Epoch: 879/1000:, Cur-Step: 17580, loss(cross_entropy): 0.00295, Running average loss:0.00295, Time taken: 0:00:14.562891 ETA: 0:29:22.109841 2022-07-28 20:09:19,244 [INFO] __main__: Epoch: 879/1000:, Cur-Step: 17580, loss(cross_entropy): 0.00295, Running average loss:0.00295, Time taken: 0:00:14.562891 ETA: 0:29:22.109841 Epoch: 879/1000:, Cur-Step: 17590, loss(cross_entropy): 0.00281, Running average loss:0.00292, Time taken: 0:00:14.562891 ETA: 0:29:22.109841 2022-07-28 20:09:26,114 [INFO] __main__: Epoch: 879/1000:, Cur-Step: 17590, loss(cross_entropy): 0.00281, Running average loss:0.00292, Time taken: 0:00:14.562891 ETA: 0:29:22.109841 Epoch: 880/1000:, Cur-Step: 17600, loss(cross_entropy): 0.00282, Running average loss:0.00282, Time taken: 0:00:14.648686 ETA: 0:29:17.842283 2022-07-28 20:09:33,219 [INFO] __main__: Epoch: 880/1000:, Cur-Step: 17600, loss(cross_entropy): 0.00282, Running average loss:0.00282, Time taken: 0:00:14.648686 ETA: 0:29:17.842283 Epoch: 880/1000:, Cur-Step: 17610, loss(cross_entropy): 0.00275, Running average loss:0.00283, Time taken: 0:00:14.648686 ETA: 0:29:17.842283 2022-07-28 20:09:40,107 [INFO] __main__: Epoch: 880/1000:, Cur-Step: 17610, loss(cross_entropy): 0.00275, Running average loss:0.00283, Time taken: 0:00:14.648686 ETA: 0:29:17.842283 Epoch: 881/1000:, Cur-Step: 17620, loss(cross_entropy): 0.00271, Running average loss:0.00271, Time taken: 0:00:14.533691 ETA: 0:28:49.509249 2022-07-28 20:09:47,030 [INFO] __main__: Epoch: 881/1000:, Cur-Step: 17620, loss(cross_entropy): 0.00271, Running average loss:0.00271, Time taken: 0:00:14.533691 ETA: 0:28:49.509249 Epoch: 881/1000:, Cur-Step: 17630, loss(cross_entropy): 0.00268, Running average loss:0.00277, Time taken: 0:00:14.533691 ETA: 0:28:49.509249 2022-07-28 20:09:53,878 [INFO] __main__: Epoch: 881/1000:, Cur-Step: 17630, loss(cross_entropy): 0.00268, Running average loss:0.00277, Time taken: 0:00:14.533691 ETA: 0:28:49.509249 Epoch: 882/1000:, Cur-Step: 17640, loss(cross_entropy): 0.00265, Running average loss:0.00265, Time taken: 0:00:14.570676 ETA: 0:28:39.339750 2022-07-28 20:10:00,928 [INFO] __main__: Epoch: 882/1000:, Cur-Step: 17640, loss(cross_entropy): 0.00265, Running average loss:0.00265, Time taken: 0:00:14.570676 ETA: 0:28:39.339750 Epoch: 882/1000:, Cur-Step: 17650, loss(cross_entropy): 0.00277, Running average loss:0.00273, Time taken: 0:00:14.570676 ETA: 0:28:39.339750 2022-07-28 20:10:07,893 [INFO] __main__: Epoch: 882/1000:, Cur-Step: 17650, loss(cross_entropy): 0.00277, Running average loss:0.00273, Time taken: 0:00:14.570676 ETA: 0:28:39.339750 Epoch: 883/1000:, Cur-Step: 17660, loss(cross_entropy): 0.00275, Running average loss:0.00275, Time taken: 0:00:14.609765 ETA: 0:28:29.342511 2022-07-28 20:10:14,826 [INFO] __main__: Epoch: 883/1000:, Cur-Step: 17660, loss(cross_entropy): 0.00275, Running average loss:0.00275, Time taken: 0:00:14.609765 ETA: 0:28:29.342511 Epoch: 883/1000:, Cur-Step: 17670, loss(cross_entropy): 0.00314, Running average loss:0.00271, Time taken: 0:00:14.609765 ETA: 0:28:29.342511 2022-07-28 20:10:21,698 [INFO] __main__: Epoch: 883/1000:, Cur-Step: 17670, loss(cross_entropy): 0.00314, Running average loss:0.00271, Time taken: 0:00:14.609765 ETA: 0:28:29.342511 Epoch: 884/1000:, Cur-Step: 17680, loss(cross_entropy): 0.00292, Running average loss:0.00292, Time taken: 0:00:14.533408 ETA: 0:28:05.875292 2022-07-28 20:10:28,681 [INFO] __main__: Epoch: 884/1000:, Cur-Step: 17680, loss(cross_entropy): 0.00292, Running average loss:0.00292, Time taken: 0:00:14.533408 ETA: 0:28:05.875292 Epoch: 884/1000:, Cur-Step: 17690, loss(cross_entropy): 0.00258, Running average loss:0.00281, Time taken: 0:00:14.533408 ETA: 0:28:05.875292 2022-07-28 20:10:35,765 [INFO] __main__: Epoch: 884/1000:, Cur-Step: 17690, loss(cross_entropy): 0.00258, Running average loss:0.00281, Time taken: 0:00:14.533408 ETA: 0:28:05.875292 Epoch: 885/1000:, Cur-Step: 17700, loss(cross_entropy): 0.00269, Running average loss:0.00269, Time taken: 0:00:14.711633 ETA: 0:28:11.837764 2022-07-28 20:10:42,713 [INFO] __main__: Epoch: 885/1000:, Cur-Step: 17700, loss(cross_entropy): 0.00269, Running average loss:0.00269, Time taken: 0:00:14.711633 ETA: 0:28:11.837764 Epoch: 885/1000:, Cur-Step: 17710, loss(cross_entropy): 0.00294, Running average loss:0.00290, Time taken: 0:00:14.711633 ETA: 0:28:11.837764 2022-07-28 20:10:49,678 [INFO] __main__: Epoch: 885/1000:, Cur-Step: 17710, loss(cross_entropy): 0.00294, Running average loss:0.00290, Time taken: 0:00:14.711633 ETA: 0:28:11.837764 Epoch: 886/1000:, Cur-Step: 17720, loss(cross_entropy): 0.00321, Running average loss:0.00321, Time taken: 0:00:14.638906 ETA: 0:27:48.835284 2022-07-28 20:10:56,673 [INFO] __main__: Epoch: 886/1000:, Cur-Step: 17720, loss(cross_entropy): 0.00321, Running average loss:0.00321, Time taken: 0:00:14.638906 ETA: 0:27:48.835284 Epoch: 886/1000:, Cur-Step: 17730, loss(cross_entropy): 0.00315, Running average loss:0.00312, Time taken: 0:00:14.638906 ETA: 0:27:48.835284 2022-07-28 20:11:03,673 [INFO] __main__: Epoch: 886/1000:, Cur-Step: 17730, loss(cross_entropy): 0.00315, Running average loss:0.00312, Time taken: 0:00:14.638906 ETA: 0:27:48.835284 Epoch: 887/1000:, Cur-Step: 17740, loss(cross_entropy): 0.00316, Running average loss:0.00316, Time taken: 0:00:14.616493 ETA: 0:27:31.663761 2022-07-28 20:11:10,629 [INFO] __main__: Epoch: 887/1000:, Cur-Step: 17740, loss(cross_entropy): 0.00316, Running average loss:0.00316, Time taken: 0:00:14.616493 ETA: 0:27:31.663761 Epoch: 887/1000:, Cur-Step: 17750, loss(cross_entropy): 0.00336, Running average loss:0.00311, Time taken: 0:00:14.616493 ETA: 0:27:31.663761 2022-07-28 20:11:17,520 [INFO] __main__: Epoch: 887/1000:, Cur-Step: 17750, loss(cross_entropy): 0.00336, Running average loss:0.00311, Time taken: 0:00:14.616493 ETA: 0:27:31.663761 Epoch: 888/1000:, Cur-Step: 17760, loss(cross_entropy): 0.00297, Running average loss:0.00297, Time taken: 0:00:14.564936 ETA: 0:27:11.272823 2022-07-28 20:11:24,510 [INFO] __main__: Epoch: 888/1000:, Cur-Step: 17760, loss(cross_entropy): 0.00297, Running average loss:0.00297, Time taken: 0:00:14.564936 ETA: 0:27:11.272823 Epoch: 888/1000:, Cur-Step: 17770, loss(cross_entropy): 0.00355, Running average loss:0.00307, Time taken: 0:00:14.564936 ETA: 0:27:11.272823 2022-07-28 20:11:31,456 [INFO] __main__: Epoch: 888/1000:, Cur-Step: 17770, loss(cross_entropy): 0.00355, Running average loss:0.00307, Time taken: 0:00:14.564936 ETA: 0:27:11.272823 Epoch: 889/1000:, Cur-Step: 17780, loss(cross_entropy): 0.00339, Running average loss:0.00339, Time taken: 0:00:14.745144 ETA: 0:27:16.710945 2022-07-28 20:11:38,592 [INFO] __main__: Epoch: 889/1000:, Cur-Step: 17780, loss(cross_entropy): 0.00339, Running average loss:0.00339, Time taken: 0:00:14.745144 ETA: 0:27:16.710945 Epoch: 889/1000:, Cur-Step: 17790, loss(cross_entropy): 0.00280, Running average loss:0.00304, Time taken: 0:00:14.745144 ETA: 0:27:16.710945 2022-07-28 20:11:45,489 [INFO] __main__: Epoch: 889/1000:, Cur-Step: 17790, loss(cross_entropy): 0.00280, Running average loss:0.00304, Time taken: 0:00:14.745144 ETA: 0:27:16.710945 Epoch: 890/1000:, Cur-Step: 17800, loss(cross_entropy): 0.00294, Running average loss:0.00294, Time taken: 0:00:14.583412 ETA: 0:26:44.175339 2022-07-28 20:11:52,452 [INFO] __main__: Epoch: 890/1000:, Cur-Step: 17800, loss(cross_entropy): 0.00294, Running average loss:0.00294, Time taken: 0:00:14.583412 ETA: 0:26:44.175339 Epoch: 890/1000:, Cur-Step: 17810, loss(cross_entropy): 0.00286, Running average loss:0.00289, Time taken: 0:00:14.583412 ETA: 0:26:44.175339 2022-07-28 20:11:59,302 [INFO] __main__: Epoch: 890/1000:, Cur-Step: 17810, loss(cross_entropy): 0.00286, Running average loss:0.00289, Time taken: 0:00:14.583412 ETA: 0:26:44.175339 Epoch: 891/1000:, Cur-Step: 17820, loss(cross_entropy): 0.00291, Running average loss:0.00291, Time taken: 0:00:14.596180 ETA: 0:26:30.983616 2022-07-28 20:12:06,400 [INFO] __main__: Epoch: 891/1000:, Cur-Step: 17820, loss(cross_entropy): 0.00291, Running average loss:0.00291, Time taken: 0:00:14.596180 ETA: 0:26:30.983616 Epoch: 891/1000:, Cur-Step: 17830, loss(cross_entropy): 0.00277, Running average loss:0.00277, Time taken: 0:00:14.596180 ETA: 0:26:30.983616 2022-07-28 20:12:13,282 [INFO] __main__: Epoch: 891/1000:, Cur-Step: 17830, loss(cross_entropy): 0.00277, Running average loss:0.00277, Time taken: 0:00:14.596180 ETA: 0:26:30.983616 Epoch: 892/1000:, Cur-Step: 17840, loss(cross_entropy): 0.00276, Running average loss:0.00276, Time taken: 0:00:14.588441 ETA: 0:26:15.551642 2022-07-28 20:12:20,251 [INFO] __main__: Epoch: 892/1000:, Cur-Step: 17840, loss(cross_entropy): 0.00276, Running average loss:0.00276, Time taken: 0:00:14.588441 ETA: 0:26:15.551642 Epoch: 892/1000:, Cur-Step: 17850, loss(cross_entropy): 0.00261, Running average loss:0.00266, Time taken: 0:00:14.588441 ETA: 0:26:15.551642 2022-07-28 20:12:27,191 [INFO] __main__: Epoch: 892/1000:, Cur-Step: 17850, loss(cross_entropy): 0.00261, Running average loss:0.00266, Time taken: 0:00:14.588441 ETA: 0:26:15.551642 Epoch: 893/1000:, Cur-Step: 17860, loss(cross_entropy): 0.00269, Running average loss:0.00269, Time taken: 0:00:14.690996 ETA: 0:26:11.936616 2022-07-28 20:12:34,296 [INFO] __main__: Epoch: 893/1000:, Cur-Step: 17860, loss(cross_entropy): 0.00269, Running average loss:0.00269, Time taken: 0:00:14.690996 ETA: 0:26:11.936616 Epoch: 893/1000:, Cur-Step: 17870, loss(cross_entropy): 0.00260, Running average loss:0.00266, Time taken: 0:00:14.690996 ETA: 0:26:11.936616 2022-07-28 20:12:41,292 [INFO] __main__: Epoch: 893/1000:, Cur-Step: 17870, loss(cross_entropy): 0.00260, Running average loss:0.00266, Time taken: 0:00:14.690996 ETA: 0:26:11.936616 Epoch: 894/1000:, Cur-Step: 17880, loss(cross_entropy): 0.00255, Running average loss:0.00255, Time taken: 0:00:14.808143 ETA: 0:26:09.663173 2022-07-28 20:12:48,354 [INFO] __main__: Epoch: 894/1000:, Cur-Step: 17880, loss(cross_entropy): 0.00255, Running average loss:0.00255, Time taken: 0:00:14.808143 ETA: 0:26:09.663173 Epoch: 894/1000:, Cur-Step: 17890, loss(cross_entropy): 0.00264, Running average loss:0.00262, Time taken: 0:00:14.808143 ETA: 0:26:09.663173 2022-07-28 20:12:55,482 [INFO] __main__: Epoch: 894/1000:, Cur-Step: 17890, loss(cross_entropy): 0.00264, Running average loss:0.00262, Time taken: 0:00:14.808143 ETA: 0:26:09.663173 Epoch: 895/1000:, Cur-Step: 17900, loss(cross_entropy): 0.00272, Running average loss:0.00272, Time taken: 0:00:14.887097 ETA: 0:26:03.145198 2022-07-28 20:13:02,628 [INFO] __main__: Epoch: 895/1000:, Cur-Step: 17900, loss(cross_entropy): 0.00272, Running average loss:0.00272, Time taken: 0:00:14.887097 ETA: 0:26:03.145198 Epoch: 895/1000:, Cur-Step: 17910, loss(cross_entropy): 0.00271, Running average loss:0.00265, Time taken: 0:00:14.887097 ETA: 0:26:03.145198 2022-07-28 20:13:09,786 [INFO] __main__: Epoch: 895/1000:, Cur-Step: 17910, loss(cross_entropy): 0.00271, Running average loss:0.00265, Time taken: 0:00:14.887097 ETA: 0:26:03.145198 Epoch: 896/1000:, Cur-Step: 17920, loss(cross_entropy): 0.00258, Running average loss:0.00258, Time taken: 0:00:14.927914 ETA: 0:25:52.503096 2022-07-28 20:13:16,796 [INFO] __main__: Epoch: 896/1000:, Cur-Step: 17920, loss(cross_entropy): 0.00258, Running average loss:0.00258, Time taken: 0:00:14.927914 ETA: 0:25:52.503096 Epoch: 896/1000:, Cur-Step: 17930, loss(cross_entropy): 0.00290, Running average loss:0.00271, Time taken: 0:00:14.927914 ETA: 0:25:52.503096 2022-07-28 20:13:23,877 [INFO] __main__: Epoch: 896/1000:, Cur-Step: 17930, loss(cross_entropy): 0.00290, Running average loss:0.00271, Time taken: 0:00:14.927914 ETA: 0:25:52.503096 Epoch: 897/1000:, Cur-Step: 17940, loss(cross_entropy): 0.00275, Running average loss:0.00275, Time taken: 0:00:14.698348 ETA: 0:25:13.929849 2022-07-28 20:13:30,823 [INFO] __main__: Epoch: 897/1000:, Cur-Step: 17940, loss(cross_entropy): 0.00275, Running average loss:0.00275, Time taken: 0:00:14.698348 ETA: 0:25:13.929849 Epoch: 897/1000:, Cur-Step: 17950, loss(cross_entropy): 0.00277, Running average loss:0.00276, Time taken: 0:00:14.698348 ETA: 0:25:13.929849 2022-07-28 20:13:38,001 [INFO] __main__: Epoch: 897/1000:, Cur-Step: 17950, loss(cross_entropy): 0.00277, Running average loss:0.00276, Time taken: 0:00:14.698348 ETA: 0:25:13.929849 Epoch: 898/1000:, Cur-Step: 17960, loss(cross_entropy): 0.00276, Running average loss:0.00276, Time taken: 0:00:14.828454 ETA: 0:25:12.502334 2022-07-28 20:13:45,002 [INFO] __main__: Epoch: 898/1000:, Cur-Step: 17960, loss(cross_entropy): 0.00276, Running average loss:0.00276, Time taken: 0:00:14.828454 ETA: 0:25:12.502334 Epoch: 898/1000:, Cur-Step: 17970, loss(cross_entropy): 0.00272, Running average loss:0.00280, Time taken: 0:00:14.828454 ETA: 0:25:12.502334 2022-07-28 20:13:52,090 [INFO] __main__: Epoch: 898/1000:, Cur-Step: 17970, loss(cross_entropy): 0.00272, Running average loss:0.00280, Time taken: 0:00:14.828454 ETA: 0:25:12.502334 Epoch: 899/1000:, Cur-Step: 17980, loss(cross_entropy): 0.00309, Running average loss:0.00309, Time taken: 0:00:14.703632 ETA: 0:24:45.066796 2022-07-28 20:13:58,983 [INFO] __main__: Epoch: 899/1000:, Cur-Step: 17980, loss(cross_entropy): 0.00309, Running average loss:0.00309, Time taken: 0:00:14.703632 ETA: 0:24:45.066796 Epoch: 899/1000:, Cur-Step: 17990, loss(cross_entropy): 0.00294, Running average loss:0.00302, Time taken: 0:00:14.703632 ETA: 0:24:45.066796 2022-07-28 20:14:06,071 [INFO] __main__: Epoch: 899/1000:, Cur-Step: 17990, loss(cross_entropy): 0.00294, Running average loss:0.00302, Time taken: 0:00:14.703632 ETA: 0:24:45.066796 INFO:tensorflow:Saving checkpoints for step-18000. 2022-07-28 20:14:12,425 [INFO] tensorflow: Saving checkpoints for step-18000. Epoch: 900/1000:, Cur-Step: 18000, loss(cross_entropy): 0.00322, Running average loss:0.00322, Time taken: 0:00:14.839088 ETA: 0:24:43.908820 2022-07-28 20:14:35,482 [INFO] __main__: Epoch: 900/1000:, Cur-Step: 18000, loss(cross_entropy): 0.00322, Running average loss:0.00322, Time taken: 0:00:14.839088 ETA: 0:24:43.908820 Epoch: 900/1000:, Cur-Step: 18010, loss(cross_entropy): 0.00344, Running average loss:0.00346, Time taken: 0:00:14.839088 ETA: 0:24:43.908820 2022-07-28 20:14:42,606 [INFO] __main__: Epoch: 900/1000:, Cur-Step: 18010, loss(cross_entropy): 0.00344, Running average loss:0.00346, Time taken: 0:00:14.839088 ETA: 0:24:43.908820 Epoch: 901/1000:, Cur-Step: 18020, loss(cross_entropy): 0.00338, Running average loss:0.00338, Time taken: 0:00:37.243757 ETA: 1:01:27.131968 2022-07-28 20:14:49,632 [INFO] __main__: Epoch: 901/1000:, Cur-Step: 18020, loss(cross_entropy): 0.00338, Running average loss:0.00338, Time taken: 0:00:37.243757 ETA: 1:01:27.131968 Epoch: 901/1000:, Cur-Step: 18030, loss(cross_entropy): 0.00293, Running average loss:0.00324, Time taken: 0:00:37.243757 ETA: 1:01:27.131968 2022-07-28 20:14:56,777 [INFO] __main__: Epoch: 901/1000:, Cur-Step: 18030, loss(cross_entropy): 0.00293, Running average loss:0.00324, Time taken: 0:00:37.243757 ETA: 1:01:27.131968 Epoch: 902/1000:, Cur-Step: 18040, loss(cross_entropy): 0.00301, Running average loss:0.00301, Time taken: 0:00:14.865602 ETA: 0:24:16.829021 2022-07-28 20:15:03,850 [INFO] __main__: Epoch: 902/1000:, Cur-Step: 18040, loss(cross_entropy): 0.00301, Running average loss:0.00301, Time taken: 0:00:14.865602 ETA: 0:24:16.829021 Epoch: 902/1000:, Cur-Step: 18050, loss(cross_entropy): 0.00328, Running average loss:0.00298, Time taken: 0:00:14.865602 ETA: 0:24:16.829021 2022-07-28 20:15:11,195 [INFO] __main__: Epoch: 902/1000:, Cur-Step: 18050, loss(cross_entropy): 0.00328, Running average loss:0.00298, Time taken: 0:00:14.865602 ETA: 0:24:16.829021 Epoch: 903/1000:, Cur-Step: 18060, loss(cross_entropy): 0.00363, Running average loss:0.00363, Time taken: 0:00:15.208045 ETA: 0:24:35.180366 2022-07-28 20:15:18,353 [INFO] __main__: Epoch: 903/1000:, Cur-Step: 18060, loss(cross_entropy): 0.00363, Running average loss:0.00363, Time taken: 0:00:15.208045 ETA: 0:24:35.180366 Epoch: 903/1000:, Cur-Step: 18070, loss(cross_entropy): 0.00288, Running average loss:0.00299, Time taken: 0:00:15.208045 ETA: 0:24:35.180366 2022-07-28 20:15:25,662 [INFO] __main__: Epoch: 903/1000:, Cur-Step: 18070, loss(cross_entropy): 0.00288, Running average loss:0.00299, Time taken: 0:00:15.208045 ETA: 0:24:35.180366 Epoch: 904/1000:, Cur-Step: 18080, loss(cross_entropy): 0.00278, Running average loss:0.00278, Time taken: 0:00:15.222272 ETA: 0:24:21.338104 2022-07-28 20:15:32,886 [INFO] __main__: Epoch: 904/1000:, Cur-Step: 18080, loss(cross_entropy): 0.00278, Running average loss:0.00278, Time taken: 0:00:15.222272 ETA: 0:24:21.338104 Epoch: 904/1000:, Cur-Step: 18090, loss(cross_entropy): 0.00292, Running average loss:0.00281, Time taken: 0:00:15.222272 ETA: 0:24:21.338104 2022-07-28 20:15:40,326 [INFO] __main__: Epoch: 904/1000:, Cur-Step: 18090, loss(cross_entropy): 0.00292, Running average loss:0.00281, Time taken: 0:00:15.222272 ETA: 0:24:21.338104 Epoch: 905/1000:, Cur-Step: 18100, loss(cross_entropy): 0.00263, Running average loss:0.00263, Time taken: 0:00:15.426914 ETA: 0:24:25.556828 2022-07-28 20:15:47,606 [INFO] __main__: Epoch: 905/1000:, Cur-Step: 18100, loss(cross_entropy): 0.00263, Running average loss:0.00263, Time taken: 0:00:15.426914 ETA: 0:24:25.556828 Epoch: 905/1000:, Cur-Step: 18110, loss(cross_entropy): 0.00257, Running average loss:0.00274, Time taken: 0:00:15.426914 ETA: 0:24:25.556828 2022-07-28 20:15:54,880 [INFO] __main__: Epoch: 905/1000:, Cur-Step: 18110, loss(cross_entropy): 0.00257, Running average loss:0.00274, Time taken: 0:00:15.426914 ETA: 0:24:25.556828 Epoch: 906/1000:, Cur-Step: 18120, loss(cross_entropy): 0.00264, Running average loss:0.00264, Time taken: 0:00:15.256907 ETA: 0:23:54.149302 2022-07-28 20:16:02,118 [INFO] __main__: Epoch: 906/1000:, Cur-Step: 18120, loss(cross_entropy): 0.00264, Running average loss:0.00264, Time taken: 0:00:15.256907 ETA: 0:23:54.149302 Epoch: 906/1000:, Cur-Step: 18130, loss(cross_entropy): 0.00278, Running average loss:0.00270, Time taken: 0:00:15.256907 ETA: 0:23:54.149302 2022-07-28 20:16:09,458 [INFO] __main__: Epoch: 906/1000:, Cur-Step: 18130, loss(cross_entropy): 0.00278, Running average loss:0.00270, Time taken: 0:00:15.256907 ETA: 0:23:54.149302 Epoch: 907/1000:, Cur-Step: 18140, loss(cross_entropy): 0.00262, Running average loss:0.00262, Time taken: 0:00:15.300066 ETA: 0:23:42.906115 2022-07-28 20:16:16,673 [INFO] __main__: Epoch: 907/1000:, Cur-Step: 18140, loss(cross_entropy): 0.00262, Running average loss:0.00262, Time taken: 0:00:15.300066 ETA: 0:23:42.906115 Epoch: 907/1000:, Cur-Step: 18150, loss(cross_entropy): 0.00259, Running average loss:0.00260, Time taken: 0:00:15.300066 ETA: 0:23:42.906115 2022-07-28 20:16:23,884 [INFO] __main__: Epoch: 907/1000:, Cur-Step: 18150, loss(cross_entropy): 0.00259, Running average loss:0.00260, Time taken: 0:00:15.300066 ETA: 0:23:42.906115 Epoch: 908/1000:, Cur-Step: 18160, loss(cross_entropy): 0.00265, Running average loss:0.00265, Time taken: 0:00:15.171770 ETA: 0:23:15.802805 2022-07-28 20:16:31,119 [INFO] __main__: Epoch: 908/1000:, Cur-Step: 18160, loss(cross_entropy): 0.00265, Running average loss:0.00265, Time taken: 0:00:15.171770 ETA: 0:23:15.802805 Epoch: 908/1000:, Cur-Step: 18170, loss(cross_entropy): 0.00287, Running average loss:0.00261, Time taken: 0:00:15.171770 ETA: 0:23:15.802805 2022-07-28 20:16:38,421 [INFO] __main__: Epoch: 908/1000:, Cur-Step: 18170, loss(cross_entropy): 0.00287, Running average loss:0.00261, Time taken: 0:00:15.171770 ETA: 0:23:15.802805 Epoch: 909/1000:, Cur-Step: 18180, loss(cross_entropy): 0.00260, Running average loss:0.00260, Time taken: 0:00:15.272774 ETA: 0:23:09.822476 2022-07-28 20:16:45,665 [INFO] __main__: Epoch: 909/1000:, Cur-Step: 18180, loss(cross_entropy): 0.00260, Running average loss:0.00260, Time taken: 0:00:15.272774 ETA: 0:23:09.822476 Epoch: 909/1000:, Cur-Step: 18190, loss(cross_entropy): 0.00250, Running average loss:0.00258, Time taken: 0:00:15.272774 ETA: 0:23:09.822476 2022-07-28 20:16:52,851 [INFO] __main__: Epoch: 909/1000:, Cur-Step: 18190, loss(cross_entropy): 0.00250, Running average loss:0.00258, Time taken: 0:00:15.272774 ETA: 0:23:09.822476 Epoch: 910/1000:, Cur-Step: 18200, loss(cross_entropy): 0.00249, Running average loss:0.00249, Time taken: 0:00:15.067367 ETA: 0:22:36.063058 2022-07-28 20:17:00,046 [INFO] __main__: Epoch: 910/1000:, Cur-Step: 18200, loss(cross_entropy): 0.00249, Running average loss:0.00249, Time taken: 0:00:15.067367 ETA: 0:22:36.063058 Epoch: 910/1000:, Cur-Step: 18210, loss(cross_entropy): 0.00270, Running average loss:0.00258, Time taken: 0:00:15.067367 ETA: 0:22:36.063058 2022-07-28 20:17:07,385 [INFO] __main__: Epoch: 910/1000:, Cur-Step: 18210, loss(cross_entropy): 0.00270, Running average loss:0.00258, Time taken: 0:00:15.067367 ETA: 0:22:36.063058 Epoch: 911/1000:, Cur-Step: 18220, loss(cross_entropy): 0.00266, Running average loss:0.00266, Time taken: 0:00:15.299202 ETA: 0:22:41.628975 2022-07-28 20:17:14,647 [INFO] __main__: Epoch: 911/1000:, Cur-Step: 18220, loss(cross_entropy): 0.00266, Running average loss:0.00266, Time taken: 0:00:15.299202 ETA: 0:22:41.628975 Epoch: 911/1000:, Cur-Step: 18230, loss(cross_entropy): 0.00263, Running average loss:0.00275, Time taken: 0:00:15.299202 ETA: 0:22:41.628975 2022-07-28 20:17:21,853 [INFO] __main__: Epoch: 911/1000:, Cur-Step: 18230, loss(cross_entropy): 0.00263, Running average loss:0.00275, Time taken: 0:00:15.299202 ETA: 0:22:41.628975 Epoch: 912/1000:, Cur-Step: 18240, loss(cross_entropy): 0.00274, Running average loss:0.00274, Time taken: 0:00:15.108086 ETA: 0:22:09.511599 2022-07-28 20:17:29,079 [INFO] __main__: Epoch: 912/1000:, Cur-Step: 18240, loss(cross_entropy): 0.00274, Running average loss:0.00274, Time taken: 0:00:15.108086 ETA: 0:22:09.511599 Epoch: 912/1000:, Cur-Step: 18250, loss(cross_entropy): 0.00270, Running average loss:0.00274, Time taken: 0:00:15.108086 ETA: 0:22:09.511599 2022-07-28 20:17:36,329 [INFO] __main__: Epoch: 912/1000:, Cur-Step: 18250, loss(cross_entropy): 0.00270, Running average loss:0.00274, Time taken: 0:00:15.108086 ETA: 0:22:09.511599 Epoch: 913/1000:, Cur-Step: 18260, loss(cross_entropy): 0.00270, Running average loss:0.00270, Time taken: 0:00:15.311352 ETA: 0:22:12.087646 2022-07-28 20:17:43,652 [INFO] __main__: Epoch: 913/1000:, Cur-Step: 18260, loss(cross_entropy): 0.00270, Running average loss:0.00270, Time taken: 0:00:15.311352 ETA: 0:22:12.087646 Epoch: 913/1000:, Cur-Step: 18270, loss(cross_entropy): 0.00290, Running average loss:0.00282, Time taken: 0:00:15.311352 ETA: 0:22:12.087646 2022-07-28 20:17:50,953 [INFO] __main__: Epoch: 913/1000:, Cur-Step: 18270, loss(cross_entropy): 0.00290, Running average loss:0.00282, Time taken: 0:00:15.311352 ETA: 0:22:12.087646 Epoch: 914/1000:, Cur-Step: 18280, loss(cross_entropy): 0.00279, Running average loss:0.00279, Time taken: 0:00:15.305539 ETA: 0:21:56.276324 2022-07-28 20:17:58,194 [INFO] __main__: Epoch: 914/1000:, Cur-Step: 18280, loss(cross_entropy): 0.00279, Running average loss:0.00279, Time taken: 0:00:15.305539 ETA: 0:21:56.276324 Epoch: 914/1000:, Cur-Step: 18290, loss(cross_entropy): 0.00280, Running average loss:0.00275, Time taken: 0:00:15.305539 ETA: 0:21:56.276324 2022-07-28 20:18:05,395 [INFO] __main__: Epoch: 914/1000:, Cur-Step: 18290, loss(cross_entropy): 0.00280, Running average loss:0.00275, Time taken: 0:00:15.305539 ETA: 0:21:56.276324 Epoch: 915/1000:, Cur-Step: 18300, loss(cross_entropy): 0.00275, Running average loss:0.00275, Time taken: 0:00:15.280018 ETA: 0:21:38.801538 2022-07-28 20:18:12,764 [INFO] __main__: Epoch: 915/1000:, Cur-Step: 18300, loss(cross_entropy): 0.00275, Running average loss:0.00275, Time taken: 0:00:15.280018 ETA: 0:21:38.801538 Epoch: 915/1000:, Cur-Step: 18310, loss(cross_entropy): 0.00273, Running average loss:0.00285, Time taken: 0:00:15.280018 ETA: 0:21:38.801538 2022-07-28 20:18:19,630 [INFO] __main__: Epoch: 915/1000:, Cur-Step: 18310, loss(cross_entropy): 0.00273, Running average loss:0.00285, Time taken: 0:00:15.280018 ETA: 0:21:38.801538 Epoch: 916/1000:, Cur-Step: 18320, loss(cross_entropy): 0.00279, Running average loss:0.00279, Time taken: 0:00:14.659331 ETA: 0:20:31.383811 2022-07-28 20:18:26,567 [INFO] __main__: Epoch: 916/1000:, Cur-Step: 18320, loss(cross_entropy): 0.00279, Running average loss:0.00279, Time taken: 0:00:14.659331 ETA: 0:20:31.383811 Epoch: 916/1000:, Cur-Step: 18330, loss(cross_entropy): 0.00310, Running average loss:0.00296, Time taken: 0:00:14.659331 ETA: 0:20:31.383811 2022-07-28 20:18:33,396 [INFO] __main__: Epoch: 916/1000:, Cur-Step: 18330, loss(cross_entropy): 0.00310, Running average loss:0.00296, Time taken: 0:00:14.659331 ETA: 0:20:31.383811 Epoch: 917/1000:, Cur-Step: 18340, loss(cross_entropy): 0.00316, Running average loss:0.00316, Time taken: 0:00:14.489187 ETA: 0:20:02.602541 2022-07-28 20:18:40,331 [INFO] __main__: Epoch: 917/1000:, Cur-Step: 18340, loss(cross_entropy): 0.00316, Running average loss:0.00316, Time taken: 0:00:14.489187 ETA: 0:20:02.602541 Epoch: 917/1000:, Cur-Step: 18350, loss(cross_entropy): 0.00314, Running average loss:0.00329, Time taken: 0:00:14.489187 ETA: 0:20:02.602541 2022-07-28 20:18:47,263 [INFO] __main__: Epoch: 917/1000:, Cur-Step: 18350, loss(cross_entropy): 0.00314, Running average loss:0.00329, Time taken: 0:00:14.489187 ETA: 0:20:02.602541 Epoch: 918/1000:, Cur-Step: 18360, loss(cross_entropy): 0.00309, Running average loss:0.00309, Time taken: 0:00:14.648108 ETA: 0:20:01.144876 2022-07-28 20:18:54,250 [INFO] __main__: Epoch: 918/1000:, Cur-Step: 18360, loss(cross_entropy): 0.00309, Running average loss:0.00309, Time taken: 0:00:14.648108 ETA: 0:20:01.144876 Epoch: 918/1000:, Cur-Step: 18370, loss(cross_entropy): 0.00338, Running average loss:0.00316, Time taken: 0:00:14.648108 ETA: 0:20:01.144876 2022-07-28 20:19:01,131 [INFO] __main__: Epoch: 918/1000:, Cur-Step: 18370, loss(cross_entropy): 0.00338, Running average loss:0.00316, Time taken: 0:00:14.648108 ETA: 0:20:01.144876 Epoch: 919/1000:, Cur-Step: 18380, loss(cross_entropy): 0.00309, Running average loss:0.00309, Time taken: 0:00:14.480385 ETA: 0:19:32.911210 2022-07-28 20:19:08,031 [INFO] __main__: Epoch: 919/1000:, Cur-Step: 18380, loss(cross_entropy): 0.00309, Running average loss:0.00309, Time taken: 0:00:14.480385 ETA: 0:19:32.911210 Epoch: 919/1000:, Cur-Step: 18390, loss(cross_entropy): 0.00325, Running average loss:0.00313, Time taken: 0:00:14.480385 ETA: 0:19:32.911210 2022-07-28 20:19:15,053 [INFO] __main__: Epoch: 919/1000:, Cur-Step: 18390, loss(cross_entropy): 0.00325, Running average loss:0.00313, Time taken: 0:00:14.480385 ETA: 0:19:32.911210 Epoch: 920/1000:, Cur-Step: 18400, loss(cross_entropy): 0.00337, Running average loss:0.00337, Time taken: 0:00:14.633216 ETA: 0:19:30.657272 2022-07-28 20:19:21,949 [INFO] __main__: Epoch: 920/1000:, Cur-Step: 18400, loss(cross_entropy): 0.00337, Running average loss:0.00337, Time taken: 0:00:14.633216 ETA: 0:19:30.657272 Epoch: 920/1000:, Cur-Step: 18410, loss(cross_entropy): 0.00324, Running average loss:0.00311, Time taken: 0:00:14.633216 ETA: 0:19:30.657272 2022-07-28 20:19:28,846 [INFO] __main__: Epoch: 920/1000:, Cur-Step: 18410, loss(cross_entropy): 0.00324, Running average loss:0.00311, Time taken: 0:00:14.633216 ETA: 0:19:30.657272 Epoch: 921/1000:, Cur-Step: 18420, loss(cross_entropy): 0.00274, Running average loss:0.00274, Time taken: 0:00:14.530025 ETA: 0:19:07.872013 2022-07-28 20:19:35,777 [INFO] __main__: Epoch: 921/1000:, Cur-Step: 18420, loss(cross_entropy): 0.00274, Running average loss:0.00274, Time taken: 0:00:14.530025 ETA: 0:19:07.872013 Epoch: 921/1000:, Cur-Step: 18430, loss(cross_entropy): 0.00271, Running average loss:0.00271, Time taken: 0:00:14.530025 ETA: 0:19:07.872013 2022-07-28 20:19:42,767 [INFO] __main__: Epoch: 921/1000:, Cur-Step: 18430, loss(cross_entropy): 0.00271, Running average loss:0.00271, Time taken: 0:00:14.530025 ETA: 0:19:07.872013 Epoch: 922/1000:, Cur-Step: 18440, loss(cross_entropy): 0.00258, Running average loss:0.00258, Time taken: 0:00:14.683798 ETA: 0:19:05.336213 2022-07-28 20:19:49,759 [INFO] __main__: Epoch: 922/1000:, Cur-Step: 18440, loss(cross_entropy): 0.00258, Running average loss:0.00258, Time taken: 0:00:14.683798 ETA: 0:19:05.336213 Epoch: 922/1000:, Cur-Step: 18450, loss(cross_entropy): 0.00251, Running average loss:0.00259, Time taken: 0:00:14.683798 ETA: 0:19:05.336213 2022-07-28 20:19:56,616 [INFO] __main__: Epoch: 922/1000:, Cur-Step: 18450, loss(cross_entropy): 0.00251, Running average loss:0.00259, Time taken: 0:00:14.683798 ETA: 0:19:05.336213 Epoch: 923/1000:, Cur-Step: 18460, loss(cross_entropy): 0.00263, Running average loss:0.00263, Time taken: 0:00:14.446104 ETA: 0:18:32.350030 2022-07-28 20:20:03,516 [INFO] __main__: Epoch: 923/1000:, Cur-Step: 18460, loss(cross_entropy): 0.00263, Running average loss:0.00263, Time taken: 0:00:14.446104 ETA: 0:18:32.350030 Epoch: 923/1000:, Cur-Step: 18470, loss(cross_entropy): 0.00255, Running average loss:0.00256, Time taken: 0:00:14.446104 ETA: 0:18:32.350030 2022-07-28 20:20:10,404 [INFO] __main__: Epoch: 923/1000:, Cur-Step: 18470, loss(cross_entropy): 0.00255, Running average loss:0.00256, Time taken: 0:00:14.446104 ETA: 0:18:32.350030 Epoch: 924/1000:, Cur-Step: 18480, loss(cross_entropy): 0.00255, Running average loss:0.00255, Time taken: 0:00:14.574469 ETA: 0:18:27.659615 2022-07-28 20:20:17,398 [INFO] __main__: Epoch: 924/1000:, Cur-Step: 18480, loss(cross_entropy): 0.00255, Running average loss:0.00255, Time taken: 0:00:14.574469 ETA: 0:18:27.659615 Epoch: 924/1000:, Cur-Step: 18490, loss(cross_entropy): 0.00257, Running average loss:0.00259, Time taken: 0:00:14.574469 ETA: 0:18:27.659615 2022-07-28 20:20:24,226 [INFO] __main__: Epoch: 924/1000:, Cur-Step: 18490, loss(cross_entropy): 0.00257, Running average loss:0.00259, Time taken: 0:00:14.574469 ETA: 0:18:27.659615 Epoch: 925/1000:, Cur-Step: 18500, loss(cross_entropy): 0.00258, Running average loss:0.00258, Time taken: 0:00:14.408164 ETA: 0:18:00.612302 2022-07-28 20:20:31,118 [INFO] __main__: Epoch: 925/1000:, Cur-Step: 18500, loss(cross_entropy): 0.00258, Running average loss:0.00258, Time taken: 0:00:14.408164 ETA: 0:18:00.612302 Epoch: 925/1000:, Cur-Step: 18510, loss(cross_entropy): 0.00258, Running average loss:0.00262, Time taken: 0:00:14.408164 ETA: 0:18:00.612302 2022-07-28 20:20:37,964 [INFO] __main__: Epoch: 925/1000:, Cur-Step: 18510, loss(cross_entropy): 0.00258, Running average loss:0.00262, Time taken: 0:00:14.408164 ETA: 0:18:00.612302 Epoch: 926/1000:, Cur-Step: 18520, loss(cross_entropy): 0.00260, Running average loss:0.00260, Time taken: 0:00:14.488378 ETA: 0:17:52.139940 2022-07-28 20:20:44,959 [INFO] __main__: Epoch: 926/1000:, Cur-Step: 18520, loss(cross_entropy): 0.00260, Running average loss:0.00260, Time taken: 0:00:14.488378 ETA: 0:17:52.139940 Epoch: 926/1000:, Cur-Step: 18530, loss(cross_entropy): 0.00250, Running average loss:0.00266, Time taken: 0:00:14.488378 ETA: 0:17:52.139940 2022-07-28 20:20:51,797 [INFO] __main__: Epoch: 926/1000:, Cur-Step: 18530, loss(cross_entropy): 0.00250, Running average loss:0.00266, Time taken: 0:00:14.488378 ETA: 0:17:52.139940 Epoch: 927/1000:, Cur-Step: 18540, loss(cross_entropy): 0.00261, Running average loss:0.00261, Time taken: 0:00:14.459294 ETA: 0:17:35.528468 2022-07-28 20:20:58,703 [INFO] __main__: Epoch: 927/1000:, Cur-Step: 18540, loss(cross_entropy): 0.00261, Running average loss:0.00261, Time taken: 0:00:14.459294 ETA: 0:17:35.528468 Epoch: 927/1000:, Cur-Step: 18550, loss(cross_entropy): 0.00268, Running average loss:0.00265, Time taken: 0:00:14.459294 ETA: 0:17:35.528468 2022-07-28 20:21:05,530 [INFO] __main__: Epoch: 927/1000:, Cur-Step: 18550, loss(cross_entropy): 0.00268, Running average loss:0.00265, Time taken: 0:00:14.459294 ETA: 0:17:35.528468 Epoch: 928/1000:, Cur-Step: 18560, loss(cross_entropy): 0.00258, Running average loss:0.00258, Time taken: 0:00:14.513371 ETA: 0:17:24.962677 2022-07-28 20:21:12,561 [INFO] __main__: Epoch: 928/1000:, Cur-Step: 18560, loss(cross_entropy): 0.00258, Running average loss:0.00258, Time taken: 0:00:14.513371 ETA: 0:17:24.962677 Epoch: 928/1000:, Cur-Step: 18570, loss(cross_entropy): 0.00262, Running average loss:0.00265, Time taken: 0:00:14.513371 ETA: 0:17:24.962677 2022-07-28 20:21:19,482 [INFO] __main__: Epoch: 928/1000:, Cur-Step: 18570, loss(cross_entropy): 0.00262, Running average loss:0.00265, Time taken: 0:00:14.513371 ETA: 0:17:24.962677 Epoch: 929/1000:, Cur-Step: 18580, loss(cross_entropy): 0.00263, Running average loss:0.00263, Time taken: 0:00:14.520037 ETA: 0:17:10.922656 2022-07-28 20:21:26,350 [INFO] __main__: Epoch: 929/1000:, Cur-Step: 18580, loss(cross_entropy): 0.00263, Running average loss:0.00263, Time taken: 0:00:14.520037 ETA: 0:17:10.922656 Epoch: 929/1000:, Cur-Step: 18590, loss(cross_entropy): 0.00265, Running average loss:0.00263, Time taken: 0:00:14.520037 ETA: 0:17:10.922656 2022-07-28 20:21:33,124 [INFO] __main__: Epoch: 929/1000:, Cur-Step: 18590, loss(cross_entropy): 0.00265, Running average loss:0.00263, Time taken: 0:00:14.520037 ETA: 0:17:10.922656 Epoch: 930/1000:, Cur-Step: 18600, loss(cross_entropy): 0.00265, Running average loss:0.00265, Time taken: 0:00:14.314851 ETA: 0:16:42.039590 2022-07-28 20:21:40,020 [INFO] __main__: Epoch: 930/1000:, Cur-Step: 18600, loss(cross_entropy): 0.00265, Running average loss:0.00265, Time taken: 0:00:14.314851 ETA: 0:16:42.039590 Epoch: 930/1000:, Cur-Step: 18610, loss(cross_entropy): 0.00274, Running average loss:0.00264, Time taken: 0:00:14.314851 ETA: 0:16:42.039590 2022-07-28 20:21:47,039 [INFO] __main__: Epoch: 930/1000:, Cur-Step: 18610, loss(cross_entropy): 0.00274, Running average loss:0.00264, Time taken: 0:00:14.314851 ETA: 0:16:42.039590 Epoch: 931/1000:, Cur-Step: 18620, loss(cross_entropy): 0.00284, Running average loss:0.00284, Time taken: 0:00:14.604812 ETA: 0:16:47.732054 2022-07-28 20:21:53,935 [INFO] __main__: Epoch: 931/1000:, Cur-Step: 18620, loss(cross_entropy): 0.00284, Running average loss:0.00284, Time taken: 0:00:14.604812 ETA: 0:16:47.732054 Epoch: 931/1000:, Cur-Step: 18630, loss(cross_entropy): 0.00264, Running average loss:0.00280, Time taken: 0:00:14.604812 ETA: 0:16:47.732054 2022-07-28 20:22:00,827 [INFO] __main__: Epoch: 931/1000:, Cur-Step: 18630, loss(cross_entropy): 0.00264, Running average loss:0.00280, Time taken: 0:00:14.604812 ETA: 0:16:47.732054 Epoch: 932/1000:, Cur-Step: 18640, loss(cross_entropy): 0.00305, Running average loss:0.00305, Time taken: 0:00:14.457481 ETA: 0:16:23.108685 2022-07-28 20:22:07,728 [INFO] __main__: Epoch: 932/1000:, Cur-Step: 18640, loss(cross_entropy): 0.00305, Running average loss:0.00305, Time taken: 0:00:14.457481 ETA: 0:16:23.108685 Epoch: 932/1000:, Cur-Step: 18650, loss(cross_entropy): 0.00335, Running average loss:0.00307, Time taken: 0:00:14.457481 ETA: 0:16:23.108685 2022-07-28 20:22:14,763 [INFO] __main__: Epoch: 932/1000:, Cur-Step: 18650, loss(cross_entropy): 0.00335, Running average loss:0.00307, Time taken: 0:00:14.457481 ETA: 0:16:23.108685 Epoch: 933/1000:, Cur-Step: 18660, loss(cross_entropy): 0.00325, Running average loss:0.00325, Time taken: 0:00:14.618132 ETA: 0:16:19.414868 2022-07-28 20:22:21,687 [INFO] __main__: Epoch: 933/1000:, Cur-Step: 18660, loss(cross_entropy): 0.00325, Running average loss:0.00325, Time taken: 0:00:14.618132 ETA: 0:16:19.414868 Epoch: 933/1000:, Cur-Step: 18670, loss(cross_entropy): 0.00338, Running average loss:0.00340, Time taken: 0:00:14.618132 ETA: 0:16:19.414868 2022-07-28 20:22:28,625 [INFO] __main__: Epoch: 933/1000:, Cur-Step: 18670, loss(cross_entropy): 0.00338, Running average loss:0.00340, Time taken: 0:00:14.618132 ETA: 0:16:19.414868 Epoch: 934/1000:, Cur-Step: 18680, loss(cross_entropy): 0.00330, Running average loss:0.00330, Time taken: 0:00:14.458153 ETA: 0:15:54.238083 2022-07-28 20:22:35,474 [INFO] __main__: Epoch: 934/1000:, Cur-Step: 18680, loss(cross_entropy): 0.00330, Running average loss:0.00330, Time taken: 0:00:14.458153 ETA: 0:15:54.238083 Epoch: 934/1000:, Cur-Step: 18690, loss(cross_entropy): 0.00338, Running average loss:0.00335, Time taken: 0:00:14.458153 ETA: 0:15:54.238083 2022-07-28 20:22:42,441 [INFO] __main__: Epoch: 934/1000:, Cur-Step: 18690, loss(cross_entropy): 0.00338, Running average loss:0.00335, Time taken: 0:00:14.458153 ETA: 0:15:54.238083 Epoch: 935/1000:, Cur-Step: 18700, loss(cross_entropy): 0.00326, Running average loss:0.00326, Time taken: 0:00:14.630055 ETA: 0:15:50.953556 2022-07-28 20:22:49,440 [INFO] __main__: Epoch: 935/1000:, Cur-Step: 18700, loss(cross_entropy): 0.00326, Running average loss:0.00326, Time taken: 0:00:14.630055 ETA: 0:15:50.953556 Epoch: 935/1000:, Cur-Step: 18710, loss(cross_entropy): 0.00291, Running average loss:0.00300, Time taken: 0:00:14.630055 ETA: 0:15:50.953556 2022-07-28 20:22:56,353 [INFO] __main__: Epoch: 935/1000:, Cur-Step: 18710, loss(cross_entropy): 0.00291, Running average loss:0.00300, Time taken: 0:00:14.630055 ETA: 0:15:50.953556 Epoch: 936/1000:, Cur-Step: 18720, loss(cross_entropy): 0.00292, Running average loss:0.00292, Time taken: 0:00:14.387506 ETA: 0:15:20.800354 2022-07-28 20:23:03,167 [INFO] __main__: Epoch: 936/1000:, Cur-Step: 18720, loss(cross_entropy): 0.00292, Running average loss:0.00292, Time taken: 0:00:14.387506 ETA: 0:15:20.800354 Epoch: 936/1000:, Cur-Step: 18730, loss(cross_entropy): 0.00271, Running average loss:0.00283, Time taken: 0:00:14.387506 ETA: 0:15:20.800354 2022-07-28 20:23:10,103 [INFO] __main__: Epoch: 936/1000:, Cur-Step: 18730, loss(cross_entropy): 0.00271, Running average loss:0.00283, Time taken: 0:00:14.387506 ETA: 0:15:20.800354 Epoch: 937/1000:, Cur-Step: 18740, loss(cross_entropy): 0.00256, Running average loss:0.00256, Time taken: 0:00:14.586622 ETA: 0:15:18.957186 2022-07-28 20:23:17,099 [INFO] __main__: Epoch: 937/1000:, Cur-Step: 18740, loss(cross_entropy): 0.00256, Running average loss:0.00256, Time taken: 0:00:14.586622 ETA: 0:15:18.957186 Epoch: 937/1000:, Cur-Step: 18750, loss(cross_entropy): 0.00241, Running average loss:0.00258, Time taken: 0:00:14.586622 ETA: 0:15:18.957186 2022-07-28 20:23:24,023 [INFO] __main__: Epoch: 937/1000:, Cur-Step: 18750, loss(cross_entropy): 0.00241, Running average loss:0.00258, Time taken: 0:00:14.586622 ETA: 0:15:18.957186 Epoch: 938/1000:, Cur-Step: 18760, loss(cross_entropy): 0.00246, Running average loss:0.00246, Time taken: 0:00:14.427027 ETA: 0:14:54.475688 2022-07-28 20:23:30,838 [INFO] __main__: Epoch: 938/1000:, Cur-Step: 18760, loss(cross_entropy): 0.00246, Running average loss:0.00246, Time taken: 0:00:14.427027 ETA: 0:14:54.475688 Epoch: 938/1000:, Cur-Step: 18770, loss(cross_entropy): 0.00264, Running average loss:0.00252, Time taken: 0:00:14.427027 ETA: 0:14:54.475688 2022-07-28 20:23:37,784 [INFO] __main__: Epoch: 938/1000:, Cur-Step: 18770, loss(cross_entropy): 0.00264, Running average loss:0.00252, Time taken: 0:00:14.427027 ETA: 0:14:54.475688 Epoch: 939/1000:, Cur-Step: 18780, loss(cross_entropy): 0.00243, Running average loss:0.00243, Time taken: 0:00:14.539281 ETA: 0:14:46.896149 2022-07-28 20:23:44,699 [INFO] __main__: Epoch: 939/1000:, Cur-Step: 18780, loss(cross_entropy): 0.00243, Running average loss:0.00243, Time taken: 0:00:14.539281 ETA: 0:14:46.896149 Epoch: 939/1000:, Cur-Step: 18790, loss(cross_entropy): 0.00245, Running average loss:0.00247, Time taken: 0:00:14.539281 ETA: 0:14:46.896149 2022-07-28 20:23:51,717 [INFO] __main__: Epoch: 939/1000:, Cur-Step: 18790, loss(cross_entropy): 0.00245, Running average loss:0.00247, Time taken: 0:00:14.539281 ETA: 0:14:46.896149 Epoch: 940/1000:, Cur-Step: 18800, loss(cross_entropy): 0.00250, Running average loss:0.00250, Time taken: 0:00:14.577180 ETA: 0:14:34.630795 2022-07-28 20:23:58,592 [INFO] __main__: Epoch: 940/1000:, Cur-Step: 18800, loss(cross_entropy): 0.00250, Running average loss:0.00250, Time taken: 0:00:14.577180 ETA: 0:14:34.630795 Epoch: 940/1000:, Cur-Step: 18810, loss(cross_entropy): 0.00254, Running average loss:0.00243, Time taken: 0:00:14.577180 ETA: 0:14:34.630795 2022-07-28 20:24:05,515 [INFO] __main__: Epoch: 940/1000:, Cur-Step: 18810, loss(cross_entropy): 0.00254, Running average loss:0.00243, Time taken: 0:00:14.577180 ETA: 0:14:34.630795 Epoch: 941/1000:, Cur-Step: 18820, loss(cross_entropy): 0.00231, Running average loss:0.00231, Time taken: 0:00:14.481537 ETA: 0:14:14.410661 2022-07-28 20:24:12,347 [INFO] __main__: Epoch: 941/1000:, Cur-Step: 18820, loss(cross_entropy): 0.00231, Running average loss:0.00231, Time taken: 0:00:14.481537 ETA: 0:14:14.410661 Epoch: 941/1000:, Cur-Step: 18830, loss(cross_entropy): 0.00246, Running average loss:0.00254, Time taken: 0:00:14.481537 ETA: 0:14:14.410661 2022-07-28 20:24:19,497 [INFO] __main__: Epoch: 941/1000:, Cur-Step: 18830, loss(cross_entropy): 0.00246, Running average loss:0.00254, Time taken: 0:00:14.481537 ETA: 0:14:14.410661 Epoch: 942/1000:, Cur-Step: 18840, loss(cross_entropy): 0.00265, Running average loss:0.00265, Time taken: 0:00:14.694122 ETA: 0:14:12.259053 2022-07-28 20:24:26,394 [INFO] __main__: Epoch: 942/1000:, Cur-Step: 18840, loss(cross_entropy): 0.00265, Running average loss:0.00265, Time taken: 0:00:14.694122 ETA: 0:14:12.259053 Epoch: 942/1000:, Cur-Step: 18850, loss(cross_entropy): 0.00267, Running average loss:0.00255, Time taken: 0:00:14.694122 ETA: 0:14:12.259053 2022-07-28 20:24:33,335 [INFO] __main__: Epoch: 942/1000:, Cur-Step: 18850, loss(cross_entropy): 0.00267, Running average loss:0.00255, Time taken: 0:00:14.694122 ETA: 0:14:12.259053 Epoch: 943/1000:, Cur-Step: 18860, loss(cross_entropy): 0.00264, Running average loss:0.00264, Time taken: 0:00:14.503084 ETA: 0:13:46.675771 2022-07-28 20:24:40,166 [INFO] __main__: Epoch: 943/1000:, Cur-Step: 18860, loss(cross_entropy): 0.00264, Running average loss:0.00264, Time taken: 0:00:14.503084 ETA: 0:13:46.675771 Epoch: 943/1000:, Cur-Step: 18870, loss(cross_entropy): 0.00246, Running average loss:0.00261, Time taken: 0:00:14.503084 ETA: 0:13:46.675771 2022-07-28 20:24:47,160 [INFO] __main__: Epoch: 943/1000:, Cur-Step: 18870, loss(cross_entropy): 0.00246, Running average loss:0.00261, Time taken: 0:00:14.503084 ETA: 0:13:46.675771 Epoch: 944/1000:, Cur-Step: 18880, loss(cross_entropy): 0.00322, Running average loss:0.00322, Time taken: 0:00:14.623143 ETA: 0:13:38.896032 2022-07-28 20:24:54,140 [INFO] __main__: Epoch: 944/1000:, Cur-Step: 18880, loss(cross_entropy): 0.00322, Running average loss:0.00322, Time taken: 0:00:14.623143 ETA: 0:13:38.896032 Epoch: 944/1000:, Cur-Step: 18890, loss(cross_entropy): 0.00267, Running average loss:0.00275, Time taken: 0:00:14.623143 ETA: 0:13:38.896032 2022-07-28 20:25:01,084 [INFO] __main__: Epoch: 944/1000:, Cur-Step: 18890, loss(cross_entropy): 0.00267, Running average loss:0.00275, Time taken: 0:00:14.623143 ETA: 0:13:38.896032 Epoch: 945/1000:, Cur-Step: 18900, loss(cross_entropy): 0.00246, Running average loss:0.00246, Time taken: 0:00:14.503717 ETA: 0:13:17.704432 2022-07-28 20:25:07,941 [INFO] __main__: Epoch: 945/1000:, Cur-Step: 18900, loss(cross_entropy): 0.00246, Running average loss:0.00246, Time taken: 0:00:14.503717 ETA: 0:13:17.704432 Epoch: 945/1000:, Cur-Step: 18910, loss(cross_entropy): 0.00277, Running average loss:0.00271, Time taken: 0:00:14.503717 ETA: 0:13:17.704432 2022-07-28 20:25:14,892 [INFO] __main__: Epoch: 945/1000:, Cur-Step: 18910, loss(cross_entropy): 0.00277, Running average loss:0.00271, Time taken: 0:00:14.503717 ETA: 0:13:17.704432 Epoch: 946/1000:, Cur-Step: 18920, loss(cross_entropy): 0.00270, Running average loss:0.00270, Time taken: 0:00:14.630143 ETA: 0:13:10.027744 2022-07-28 20:25:21,892 [INFO] __main__: Epoch: 946/1000:, Cur-Step: 18920, loss(cross_entropy): 0.00270, Running average loss:0.00270, Time taken: 0:00:14.630143 ETA: 0:13:10.027744 Epoch: 946/1000:, Cur-Step: 18930, loss(cross_entropy): 0.00282, Running average loss:0.00281, Time taken: 0:00:14.630143 ETA: 0:13:10.027744 2022-07-28 20:25:28,825 [INFO] __main__: Epoch: 946/1000:, Cur-Step: 18930, loss(cross_entropy): 0.00282, Running average loss:0.00281, Time taken: 0:00:14.630143 ETA: 0:13:10.027744 Epoch: 947/1000:, Cur-Step: 18940, loss(cross_entropy): 0.00265, Running average loss:0.00265, Time taken: 0:00:14.449018 ETA: 0:12:45.797954 2022-07-28 20:25:35,659 [INFO] __main__: Epoch: 947/1000:, Cur-Step: 18940, loss(cross_entropy): 0.00265, Running average loss:0.00265, Time taken: 0:00:14.449018 ETA: 0:12:45.797954 Epoch: 947/1000:, Cur-Step: 18950, loss(cross_entropy): 0.00293, Running average loss:0.00281, Time taken: 0:00:14.449018 ETA: 0:12:45.797954 2022-07-28 20:25:42,575 [INFO] __main__: Epoch: 947/1000:, Cur-Step: 18950, loss(cross_entropy): 0.00293, Running average loss:0.00281, Time taken: 0:00:14.449018 ETA: 0:12:45.797954 Epoch: 948/1000:, Cur-Step: 18960, loss(cross_entropy): 0.00281, Running average loss:0.00281, Time taken: 0:00:14.571587 ETA: 0:12:37.722504 2022-07-28 20:25:49,554 [INFO] __main__: Epoch: 948/1000:, Cur-Step: 18960, loss(cross_entropy): 0.00281, Running average loss:0.00281, Time taken: 0:00:14.571587 ETA: 0:12:37.722504 Epoch: 948/1000:, Cur-Step: 18970, loss(cross_entropy): 0.00253, Running average loss:0.00273, Time taken: 0:00:14.571587 ETA: 0:12:37.722504 2022-07-28 20:25:56,555 [INFO] __main__: Epoch: 948/1000:, Cur-Step: 18970, loss(cross_entropy): 0.00253, Running average loss:0.00273, Time taken: 0:00:14.571587 ETA: 0:12:37.722504 Epoch: 949/1000:, Cur-Step: 18980, loss(cross_entropy): 0.00277, Running average loss:0.00277, Time taken: 0:00:14.621003 ETA: 0:12:25.671173 2022-07-28 20:26:03,468 [INFO] __main__: Epoch: 949/1000:, Cur-Step: 18980, loss(cross_entropy): 0.00277, Running average loss:0.00277, Time taken: 0:00:14.621003 ETA: 0:12:25.671173 Epoch: 949/1000:, Cur-Step: 18990, loss(cross_entropy): 0.00251, Running average loss:0.00267, Time taken: 0:00:14.621003 ETA: 0:12:25.671173 2022-07-28 20:26:10,500 [INFO] __main__: Epoch: 949/1000:, Cur-Step: 18990, loss(cross_entropy): 0.00251, Running average loss:0.00267, Time taken: 0:00:14.621003 ETA: 0:12:25.671173 Epoch: 950/1000:, Cur-Step: 19000, loss(cross_entropy): 0.00260, Running average loss:0.00260, Time taken: 0:00:14.635941 ETA: 0:12:11.797040 2022-07-28 20:26:17,434 [INFO] __main__: Epoch: 950/1000:, Cur-Step: 19000, loss(cross_entropy): 0.00260, Running average loss:0.00260, Time taken: 0:00:14.635941 ETA: 0:12:11.797040 Epoch: 950/1000:, Cur-Step: 19010, loss(cross_entropy): 0.00260, Running average loss:0.00260, Time taken: 0:00:14.635941 ETA: 0:12:11.797040 2022-07-28 20:26:24,581 [INFO] __main__: Epoch: 950/1000:, Cur-Step: 19010, loss(cross_entropy): 0.00260, Running average loss:0.00260, Time taken: 0:00:14.635941 ETA: 0:12:11.797040 Epoch: 951/1000:, Cur-Step: 19020, loss(cross_entropy): 0.00240, Running average loss:0.00240, Time taken: 0:00:15.004514 ETA: 0:12:15.221173 2022-07-28 20:26:31,761 [INFO] __main__: Epoch: 951/1000:, Cur-Step: 19020, loss(cross_entropy): 0.00240, Running average loss:0.00240, Time taken: 0:00:15.004514 ETA: 0:12:15.221173 Epoch: 951/1000:, Cur-Step: 19030, loss(cross_entropy): 0.00239, Running average loss:0.00253, Time taken: 0:00:15.004514 ETA: 0:12:15.221173 2022-07-28 20:26:38,722 [INFO] __main__: Epoch: 951/1000:, Cur-Step: 19030, loss(cross_entropy): 0.00239, Running average loss:0.00253, Time taken: 0:00:15.004514 ETA: 0:12:15.221173 Epoch: 952/1000:, Cur-Step: 19040, loss(cross_entropy): 0.00243, Running average loss:0.00243, Time taken: 0:00:14.627105 ETA: 0:11:42.101017 2022-07-28 20:26:45,662 [INFO] __main__: Epoch: 952/1000:, Cur-Step: 19040, loss(cross_entropy): 0.00243, Running average loss:0.00243, Time taken: 0:00:14.627105 ETA: 0:11:42.101017 Epoch: 952/1000:, Cur-Step: 19050, loss(cross_entropy): 0.00260, Running average loss:0.00256, Time taken: 0:00:14.627105 ETA: 0:11:42.101017 2022-07-28 20:26:52,780 [INFO] __main__: Epoch: 952/1000:, Cur-Step: 19050, loss(cross_entropy): 0.00260, Running average loss:0.00256, Time taken: 0:00:14.627105 ETA: 0:11:42.101017 Epoch: 953/1000:, Cur-Step: 19060, loss(cross_entropy): 0.00277, Running average loss:0.00277, Time taken: 0:00:14.783698 ETA: 0:11:34.833787 2022-07-28 20:26:59,745 [INFO] __main__: Epoch: 953/1000:, Cur-Step: 19060, loss(cross_entropy): 0.00277, Running average loss:0.00277, Time taken: 0:00:14.783698 ETA: 0:11:34.833787 Epoch: 953/1000:, Cur-Step: 19070, loss(cross_entropy): 0.00263, Running average loss:0.00260, Time taken: 0:00:14.783698 ETA: 0:11:34.833787 2022-07-28 20:27:06,686 [INFO] __main__: Epoch: 953/1000:, Cur-Step: 19070, loss(cross_entropy): 0.00263, Running average loss:0.00260, Time taken: 0:00:14.783698 ETA: 0:11:34.833787 Epoch: 954/1000:, Cur-Step: 19080, loss(cross_entropy): 0.00261, Running average loss:0.00261, Time taken: 0:00:14.852038 ETA: 0:11:23.193755 2022-07-28 20:27:13,862 [INFO] __main__: Epoch: 954/1000:, Cur-Step: 19080, loss(cross_entropy): 0.00261, Running average loss:0.00261, Time taken: 0:00:14.852038 ETA: 0:11:23.193755 Epoch: 954/1000:, Cur-Step: 19090, loss(cross_entropy): 0.00264, Running average loss:0.00262, Time taken: 0:00:14.852038 ETA: 0:11:23.193755 2022-07-28 20:27:21,104 [INFO] __main__: Epoch: 954/1000:, Cur-Step: 19090, loss(cross_entropy): 0.00264, Running average loss:0.00262, Time taken: 0:00:14.852038 ETA: 0:11:23.193755 Epoch: 955/1000:, Cur-Step: 19100, loss(cross_entropy): 0.00274, Running average loss:0.00274, Time taken: 0:00:15.245760 ETA: 0:11:26.059220 2022-07-28 20:27:28,360 [INFO] __main__: Epoch: 955/1000:, Cur-Step: 19100, loss(cross_entropy): 0.00274, Running average loss:0.00274, Time taken: 0:00:15.245760 ETA: 0:11:26.059220 Epoch: 955/1000:, Cur-Step: 19110, loss(cross_entropy): 0.00260, Running average loss:0.00268, Time taken: 0:00:15.245760 ETA: 0:11:26.059220 2022-07-28 20:27:35,451 [INFO] __main__: Epoch: 955/1000:, Cur-Step: 19110, loss(cross_entropy): 0.00260, Running average loss:0.00268, Time taken: 0:00:15.245760 ETA: 0:11:26.059220 Epoch: 956/1000:, Cur-Step: 19120, loss(cross_entropy): 0.00260, Running average loss:0.00260, Time taken: 0:00:15.069354 ETA: 0:11:03.051579 2022-07-28 20:27:42,678 [INFO] __main__: Epoch: 956/1000:, Cur-Step: 19120, loss(cross_entropy): 0.00260, Running average loss:0.00260, Time taken: 0:00:15.069354 ETA: 0:11:03.051579 Epoch: 956/1000:, Cur-Step: 19130, loss(cross_entropy): 0.00265, Running average loss:0.00269, Time taken: 0:00:15.069354 ETA: 0:11:03.051579 2022-07-28 20:27:49,782 [INFO] __main__: Epoch: 956/1000:, Cur-Step: 19130, loss(cross_entropy): 0.00265, Running average loss:0.00269, Time taken: 0:00:15.069354 ETA: 0:11:03.051579 Epoch: 957/1000:, Cur-Step: 19140, loss(cross_entropy): 0.00303, Running average loss:0.00303, Time taken: 0:00:15.124912 ETA: 0:10:50.371217 2022-07-28 20:27:56,986 [INFO] __main__: Epoch: 957/1000:, Cur-Step: 19140, loss(cross_entropy): 0.00303, Running average loss:0.00303, Time taken: 0:00:15.124912 ETA: 0:10:50.371217 Epoch: 957/1000:, Cur-Step: 19150, loss(cross_entropy): 0.00242, Running average loss:0.00273, Time taken: 0:00:15.124912 ETA: 0:10:50.371217 2022-07-28 20:28:04,067 [INFO] __main__: Epoch: 957/1000:, Cur-Step: 19150, loss(cross_entropy): 0.00242, Running average loss:0.00273, Time taken: 0:00:15.124912 ETA: 0:10:50.371217 Epoch: 958/1000:, Cur-Step: 19160, loss(cross_entropy): 0.00238, Running average loss:0.00238, Time taken: 0:00:14.904597 ETA: 0:10:25.993066 2022-07-28 20:28:11,111 [INFO] __main__: Epoch: 958/1000:, Cur-Step: 19160, loss(cross_entropy): 0.00238, Running average loss:0.00238, Time taken: 0:00:14.904597 ETA: 0:10:25.993066 Epoch: 958/1000:, Cur-Step: 19170, loss(cross_entropy): 0.00257, Running average loss:0.00262, Time taken: 0:00:14.904597 ETA: 0:10:25.993066 2022-07-28 20:28:18,039 [INFO] __main__: Epoch: 958/1000:, Cur-Step: 19170, loss(cross_entropy): 0.00257, Running average loss:0.00262, Time taken: 0:00:14.904597 ETA: 0:10:25.993066 Epoch: 959/1000:, Cur-Step: 19180, loss(cross_entropy): 0.00248, Running average loss:0.00248, Time taken: 0:00:14.657270 ETA: 0:10:00.948058 2022-07-28 20:28:25,106 [INFO] __main__: Epoch: 959/1000:, Cur-Step: 19180, loss(cross_entropy): 0.00248, Running average loss:0.00248, Time taken: 0:00:14.657270 ETA: 0:10:00.948058 Epoch: 959/1000:, Cur-Step: 19190, loss(cross_entropy): 0.00273, Running average loss:0.00269, Time taken: 0:00:14.657270 ETA: 0:10:00.948058 2022-07-28 20:28:32,464 [INFO] __main__: Epoch: 959/1000:, Cur-Step: 19190, loss(cross_entropy): 0.00273, Running average loss:0.00269, Time taken: 0:00:14.657270 ETA: 0:10:00.948058 Epoch: 960/1000:, Cur-Step: 19200, loss(cross_entropy): 0.00257, Running average loss:0.00257, Time taken: 0:00:15.265586 ETA: 0:10:10.623455 2022-07-28 20:28:39,725 [INFO] __main__: Epoch: 960/1000:, Cur-Step: 19200, loss(cross_entropy): 0.00257, Running average loss:0.00257, Time taken: 0:00:15.265586 ETA: 0:10:10.623455 Epoch: 960/1000:, Cur-Step: 19210, loss(cross_entropy): 0.00256, Running average loss:0.00257, Time taken: 0:00:15.265586 ETA: 0:10:10.623455 2022-07-28 20:28:46,845 [INFO] __main__: Epoch: 960/1000:, Cur-Step: 19210, loss(cross_entropy): 0.00256, Running average loss:0.00257, Time taken: 0:00:15.265586 ETA: 0:10:10.623455 Epoch: 961/1000:, Cur-Step: 19220, loss(cross_entropy): 0.00258, Running average loss:0.00258, Time taken: 0:00:15.150985 ETA: 0:09:50.888424 2022-07-28 20:28:54,164 [INFO] __main__: Epoch: 961/1000:, Cur-Step: 19220, loss(cross_entropy): 0.00258, Running average loss:0.00258, Time taken: 0:00:15.150985 ETA: 0:09:50.888424 Epoch: 961/1000:, Cur-Step: 19230, loss(cross_entropy): 0.00270, Running average loss:0.00260, Time taken: 0:00:15.150985 ETA: 0:09:50.888424 2022-07-28 20:29:01,367 [INFO] __main__: Epoch: 961/1000:, Cur-Step: 19230, loss(cross_entropy): 0.00270, Running average loss:0.00260, Time taken: 0:00:15.150985 ETA: 0:09:50.888424 Epoch: 962/1000:, Cur-Step: 19240, loss(cross_entropy): 0.00237, Running average loss:0.00237, Time taken: 0:00:15.070605 ETA: 0:09:32.682992 2022-07-28 20:29:08,548 [INFO] __main__: Epoch: 962/1000:, Cur-Step: 19240, loss(cross_entropy): 0.00237, Running average loss:0.00237, Time taken: 0:00:15.070605 ETA: 0:09:32.682992 Epoch: 962/1000:, Cur-Step: 19250, loss(cross_entropy): 0.00260, Running average loss:0.00256, Time taken: 0:00:15.070605 ETA: 0:09:32.682992 2022-07-28 20:29:15,457 [INFO] __main__: Epoch: 962/1000:, Cur-Step: 19250, loss(cross_entropy): 0.00260, Running average loss:0.00256, Time taken: 0:00:15.070605 ETA: 0:09:32.682992 Epoch: 963/1000:, Cur-Step: 19260, loss(cross_entropy): 0.00240, Running average loss:0.00240, Time taken: 0:00:14.713845 ETA: 0:09:04.412248 2022-07-28 20:29:22,538 [INFO] __main__: Epoch: 963/1000:, Cur-Step: 19260, loss(cross_entropy): 0.00240, Running average loss:0.00240, Time taken: 0:00:14.713845 ETA: 0:09:04.412248 Epoch: 963/1000:, Cur-Step: 19270, loss(cross_entropy): 0.00253, Running average loss:0.00251, Time taken: 0:00:14.713845 ETA: 0:09:04.412248 2022-07-28 20:29:29,439 [INFO] __main__: Epoch: 963/1000:, Cur-Step: 19270, loss(cross_entropy): 0.00253, Running average loss:0.00251, Time taken: 0:00:14.713845 ETA: 0:09:04.412248 Epoch: 964/1000:, Cur-Step: 19280, loss(cross_entropy): 0.00254, Running average loss:0.00254, Time taken: 0:00:14.590404 ETA: 0:08:45.254537 2022-07-28 20:29:36,410 [INFO] __main__: Epoch: 964/1000:, Cur-Step: 19280, loss(cross_entropy): 0.00254, Running average loss:0.00254, Time taken: 0:00:14.590404 ETA: 0:08:45.254537 Epoch: 964/1000:, Cur-Step: 19290, loss(cross_entropy): 0.00255, Running average loss:0.00253, Time taken: 0:00:14.590404 ETA: 0:08:45.254537 2022-07-28 20:29:43,441 [INFO] __main__: Epoch: 964/1000:, Cur-Step: 19290, loss(cross_entropy): 0.00255, Running average loss:0.00253, Time taken: 0:00:14.590404 ETA: 0:08:45.254537 Epoch: 965/1000:, Cur-Step: 19300, loss(cross_entropy): 0.00252, Running average loss:0.00252, Time taken: 0:00:14.915582 ETA: 0:08:42.045385 2022-07-28 20:29:50,650 [INFO] __main__: Epoch: 965/1000:, Cur-Step: 19300, loss(cross_entropy): 0.00252, Running average loss:0.00252, Time taken: 0:00:14.915582 ETA: 0:08:42.045385 Epoch: 965/1000:, Cur-Step: 19310, loss(cross_entropy): 0.00247, Running average loss:0.00256, Time taken: 0:00:14.915582 ETA: 0:08:42.045385 2022-07-28 20:29:57,761 [INFO] __main__: Epoch: 965/1000:, Cur-Step: 19310, loss(cross_entropy): 0.00247, Running average loss:0.00256, Time taken: 0:00:14.915582 ETA: 0:08:42.045385 Epoch: 966/1000:, Cur-Step: 19320, loss(cross_entropy): 0.00279, Running average loss:0.00279, Time taken: 0:00:15.020226 ETA: 0:08:30.687684 2022-07-28 20:30:04,952 [INFO] __main__: Epoch: 966/1000:, Cur-Step: 19320, loss(cross_entropy): 0.00279, Running average loss:0.00279, Time taken: 0:00:15.020226 ETA: 0:08:30.687684 Epoch: 966/1000:, Cur-Step: 19330, loss(cross_entropy): 0.00292, Running average loss:0.00270, Time taken: 0:00:15.020226 ETA: 0:08:30.687684 2022-07-28 20:30:11,865 [INFO] __main__: Epoch: 966/1000:, Cur-Step: 19330, loss(cross_entropy): 0.00292, Running average loss:0.00270, Time taken: 0:00:15.020226 ETA: 0:08:30.687684 Epoch: 967/1000:, Cur-Step: 19340, loss(cross_entropy): 0.00408, Running average loss:0.00408, Time taken: 0:00:14.789504 ETA: 0:08:08.053626 2022-07-28 20:30:19,050 [INFO] __main__: Epoch: 967/1000:, Cur-Step: 19340, loss(cross_entropy): 0.00408, Running average loss:0.00408, Time taken: 0:00:14.789504 ETA: 0:08:08.053626 Epoch: 967/1000:, Cur-Step: 19350, loss(cross_entropy): 0.00270, Running average loss:0.00330, Time taken: 0:00:14.789504 ETA: 0:08:08.053626 2022-07-28 20:30:26,206 [INFO] __main__: Epoch: 967/1000:, Cur-Step: 19350, loss(cross_entropy): 0.00270, Running average loss:0.00330, Time taken: 0:00:14.789504 ETA: 0:08:08.053626 Epoch: 968/1000:, Cur-Step: 19360, loss(cross_entropy): 0.00348, Running average loss:0.00348, Time taken: 0:00:14.794236 ETA: 0:07:53.415543 2022-07-28 20:30:33,153 [INFO] __main__: Epoch: 968/1000:, Cur-Step: 19360, loss(cross_entropy): 0.00348, Running average loss:0.00348, Time taken: 0:00:14.794236 ETA: 0:07:53.415543 Epoch: 968/1000:, Cur-Step: 19370, loss(cross_entropy): 0.00308, Running average loss:0.00313, Time taken: 0:00:14.794236 ETA: 0:07:53.415543 2022-07-28 20:30:40,094 [INFO] __main__: Epoch: 968/1000:, Cur-Step: 19370, loss(cross_entropy): 0.00308, Running average loss:0.00313, Time taken: 0:00:14.794236 ETA: 0:07:53.415543 Epoch: 969/1000:, Cur-Step: 19380, loss(cross_entropy): 0.00304, Running average loss:0.00304, Time taken: 0:00:14.769917 ETA: 0:07:37.867427 2022-07-28 20:30:47,273 [INFO] __main__: Epoch: 969/1000:, Cur-Step: 19380, loss(cross_entropy): 0.00304, Running average loss:0.00304, Time taken: 0:00:14.769917 ETA: 0:07:37.867427 Epoch: 969/1000:, Cur-Step: 19390, loss(cross_entropy): 0.00271, Running average loss:0.00281, Time taken: 0:00:14.769917 ETA: 0:07:37.867427 2022-07-28 20:30:54,447 [INFO] __main__: Epoch: 969/1000:, Cur-Step: 19390, loss(cross_entropy): 0.00271, Running average loss:0.00281, Time taken: 0:00:14.769917 ETA: 0:07:37.867427 Epoch: 970/1000:, Cur-Step: 19400, loss(cross_entropy): 0.00259, Running average loss:0.00259, Time taken: 0:00:15.028527 ETA: 0:07:30.855803 2022-07-28 20:31:01,555 [INFO] __main__: Epoch: 970/1000:, Cur-Step: 19400, loss(cross_entropy): 0.00259, Running average loss:0.00259, Time taken: 0:00:15.028527 ETA: 0:07:30.855803 Epoch: 970/1000:, Cur-Step: 19410, loss(cross_entropy): 0.00258, Running average loss:0.00265, Time taken: 0:00:15.028527 ETA: 0:07:30.855803 2022-07-28 20:31:08,653 [INFO] __main__: Epoch: 970/1000:, Cur-Step: 19410, loss(cross_entropy): 0.00258, Running average loss:0.00265, Time taken: 0:00:15.028527 ETA: 0:07:30.855803 Epoch: 971/1000:, Cur-Step: 19420, loss(cross_entropy): 0.00248, Running average loss:0.00248, Time taken: 0:00:14.896127 ETA: 0:07:11.987669 2022-07-28 20:31:15,692 [INFO] __main__: Epoch: 971/1000:, Cur-Step: 19420, loss(cross_entropy): 0.00248, Running average loss:0.00248, Time taken: 0:00:14.896127 ETA: 0:07:11.987669 Epoch: 971/1000:, Cur-Step: 19430, loss(cross_entropy): 0.00268, Running average loss:0.00257, Time taken: 0:00:14.896127 ETA: 0:07:11.987669 2022-07-28 20:31:22,946 [INFO] __main__: Epoch: 971/1000:, Cur-Step: 19430, loss(cross_entropy): 0.00268, Running average loss:0.00257, Time taken: 0:00:14.896127 ETA: 0:07:11.987669 Epoch: 972/1000:, Cur-Step: 19440, loss(cross_entropy): 0.00247, Running average loss:0.00247, Time taken: 0:00:15.153296 ETA: 0:07:04.292295 2022-07-28 20:31:30,196 [INFO] __main__: Epoch: 972/1000:, Cur-Step: 19440, loss(cross_entropy): 0.00247, Running average loss:0.00247, Time taken: 0:00:15.153296 ETA: 0:07:04.292295 Epoch: 972/1000:, Cur-Step: 19450, loss(cross_entropy): 0.00249, Running average loss:0.00252, Time taken: 0:00:15.153296 ETA: 0:07:04.292295 2022-07-28 20:31:37,405 [INFO] __main__: Epoch: 972/1000:, Cur-Step: 19450, loss(cross_entropy): 0.00249, Running average loss:0.00252, Time taken: 0:00:15.153296 ETA: 0:07:04.292295 Epoch: 973/1000:, Cur-Step: 19460, loss(cross_entropy): 0.00234, Running average loss:0.00234, Time taken: 0:00:15.251776 ETA: 0:06:51.797958 2022-07-28 20:31:44,757 [INFO] __main__: Epoch: 973/1000:, Cur-Step: 19460, loss(cross_entropy): 0.00234, Running average loss:0.00234, Time taken: 0:00:15.251776 ETA: 0:06:51.797958 Epoch: 973/1000:, Cur-Step: 19470, loss(cross_entropy): 0.00240, Running average loss:0.00249, Time taken: 0:00:15.251776 ETA: 0:06:51.797958 2022-07-28 20:31:52,087 [INFO] __main__: Epoch: 973/1000:, Cur-Step: 19470, loss(cross_entropy): 0.00240, Running average loss:0.00249, Time taken: 0:00:15.251776 ETA: 0:06:51.797958 Epoch: 974/1000:, Cur-Step: 19480, loss(cross_entropy): 0.00238, Running average loss:0.00238, Time taken: 0:00:15.391659 ETA: 0:06:40.183141 2022-07-28 20:31:59,357 [INFO] __main__: Epoch: 974/1000:, Cur-Step: 19480, loss(cross_entropy): 0.00238, Running average loss:0.00238, Time taken: 0:00:15.391659 ETA: 0:06:40.183141 Epoch: 974/1000:, Cur-Step: 19490, loss(cross_entropy): 0.00256, Running average loss:0.00248, Time taken: 0:00:15.391659 ETA: 0:06:40.183141 2022-07-28 20:32:06,445 [INFO] __main__: Epoch: 974/1000:, Cur-Step: 19490, loss(cross_entropy): 0.00256, Running average loss:0.00248, Time taken: 0:00:15.391659 ETA: 0:06:40.183141 Epoch: 975/1000:, Cur-Step: 19500, loss(cross_entropy): 0.00244, Running average loss:0.00244, Time taken: 0:00:14.992605 ETA: 0:06:14.815136 2022-07-28 20:32:13,581 [INFO] __main__: Epoch: 975/1000:, Cur-Step: 19500, loss(cross_entropy): 0.00244, Running average loss:0.00244, Time taken: 0:00:14.992605 ETA: 0:06:14.815136 Epoch: 975/1000:, Cur-Step: 19510, loss(cross_entropy): 0.00241, Running average loss:0.00250, Time taken: 0:00:14.992605 ETA: 0:06:14.815136 2022-07-28 20:32:20,814 [INFO] __main__: Epoch: 975/1000:, Cur-Step: 19510, loss(cross_entropy): 0.00241, Running average loss:0.00250, Time taken: 0:00:14.992605 ETA: 0:06:14.815136 Epoch: 976/1000:, Cur-Step: 19520, loss(cross_entropy): 0.00272, Running average loss:0.00272, Time taken: 0:00:15.125497 ETA: 0:06:03.011925 2022-07-28 20:32:28,037 [INFO] __main__: Epoch: 976/1000:, Cur-Step: 19520, loss(cross_entropy): 0.00272, Running average loss:0.00272, Time taken: 0:00:15.125497 ETA: 0:06:03.011925 Epoch: 976/1000:, Cur-Step: 19530, loss(cross_entropy): 0.00264, Running average loss:0.00270, Time taken: 0:00:15.125497 ETA: 0:06:03.011925 2022-07-28 20:32:35,178 [INFO] __main__: Epoch: 976/1000:, Cur-Step: 19530, loss(cross_entropy): 0.00264, Running average loss:0.00270, Time taken: 0:00:15.125497 ETA: 0:06:03.011925 Epoch: 977/1000:, Cur-Step: 19540, loss(cross_entropy): 0.00292, Running average loss:0.00292, Time taken: 0:00:14.917306 ETA: 0:05:43.098037 2022-07-28 20:32:42,229 [INFO] __main__: Epoch: 977/1000:, Cur-Step: 19540, loss(cross_entropy): 0.00292, Running average loss:0.00292, Time taken: 0:00:14.917306 ETA: 0:05:43.098037 Epoch: 977/1000:, Cur-Step: 19550, loss(cross_entropy): 0.00278, Running average loss:0.00297, Time taken: 0:00:14.917306 ETA: 0:05:43.098037 2022-07-28 20:32:49,336 [INFO] __main__: Epoch: 977/1000:, Cur-Step: 19550, loss(cross_entropy): 0.00278, Running average loss:0.00297, Time taken: 0:00:14.917306 ETA: 0:05:43.098037 Epoch: 978/1000:, Cur-Step: 19560, loss(cross_entropy): 0.00315, Running average loss:0.00315, Time taken: 0:00:14.976432 ETA: 0:05:29.481511 2022-07-28 20:32:56,569 [INFO] __main__: Epoch: 978/1000:, Cur-Step: 19560, loss(cross_entropy): 0.00315, Running average loss:0.00315, Time taken: 0:00:14.976432 ETA: 0:05:29.481511 Epoch: 978/1000:, Cur-Step: 19570, loss(cross_entropy): 0.00299, Running average loss:0.00316, Time taken: 0:00:14.976432 ETA: 0:05:29.481511 2022-07-28 20:33:03,638 [INFO] __main__: Epoch: 978/1000:, Cur-Step: 19570, loss(cross_entropy): 0.00299, Running average loss:0.00316, Time taken: 0:00:14.976432 ETA: 0:05:29.481511 Epoch: 979/1000:, Cur-Step: 19580, loss(cross_entropy): 0.00309, Running average loss:0.00309, Time taken: 0:00:14.966698 ETA: 0:05:14.300667 2022-07-28 20:33:10,826 [INFO] __main__: Epoch: 979/1000:, Cur-Step: 19580, loss(cross_entropy): 0.00309, Running average loss:0.00309, Time taken: 0:00:14.966698 ETA: 0:05:14.300667 Epoch: 979/1000:, Cur-Step: 19590, loss(cross_entropy): 0.00367, Running average loss:0.00317, Time taken: 0:00:14.966698 ETA: 0:05:14.300667 2022-07-28 20:33:18,114 [INFO] __main__: Epoch: 979/1000:, Cur-Step: 19590, loss(cross_entropy): 0.00367, Running average loss:0.00317, Time taken: 0:00:14.966698 ETA: 0:05:14.300667 Epoch: 980/1000:, Cur-Step: 19600, loss(cross_entropy): 0.00277, Running average loss:0.00277, Time taken: 0:00:15.262718 ETA: 0:05:05.254359 2022-07-28 20:33:25,395 [INFO] __main__: Epoch: 980/1000:, Cur-Step: 19600, loss(cross_entropy): 0.00277, Running average loss:0.00277, Time taken: 0:00:15.262718 ETA: 0:05:05.254359 Epoch: 980/1000:, Cur-Step: 19610, loss(cross_entropy): 0.00272, Running average loss:0.00279, Time taken: 0:00:15.262718 ETA: 0:05:05.254359 2022-07-28 20:33:32,583 [INFO] __main__: Epoch: 980/1000:, Cur-Step: 19610, loss(cross_entropy): 0.00272, Running average loss:0.00279, Time taken: 0:00:15.262718 ETA: 0:05:05.254359 Epoch: 981/1000:, Cur-Step: 19620, loss(cross_entropy): 0.00261, Running average loss:0.00261, Time taken: 0:00:15.201308 ETA: 0:04:48.824861 2022-07-28 20:33:39,828 [INFO] __main__: Epoch: 981/1000:, Cur-Step: 19620, loss(cross_entropy): 0.00261, Running average loss:0.00261, Time taken: 0:00:15.201308 ETA: 0:04:48.824861 Epoch: 981/1000:, Cur-Step: 19630, loss(cross_entropy): 0.00250, Running average loss:0.00258, Time taken: 0:00:15.201308 ETA: 0:04:48.824861 2022-07-28 20:33:47,116 [INFO] __main__: Epoch: 981/1000:, Cur-Step: 19630, loss(cross_entropy): 0.00250, Running average loss:0.00258, Time taken: 0:00:15.201308 ETA: 0:04:48.824861 Epoch: 982/1000:, Cur-Step: 19640, loss(cross_entropy): 0.00252, Running average loss:0.00252, Time taken: 0:00:15.103649 ETA: 0:04:31.865676 2022-07-28 20:33:54,181 [INFO] __main__: Epoch: 982/1000:, Cur-Step: 19640, loss(cross_entropy): 0.00252, Running average loss:0.00252, Time taken: 0:00:15.103649 ETA: 0:04:31.865676 Epoch: 982/1000:, Cur-Step: 19650, loss(cross_entropy): 0.00247, Running average loss:0.00243, Time taken: 0:00:15.103649 ETA: 0:04:31.865676 2022-07-28 20:34:01,345 [INFO] __main__: Epoch: 982/1000:, Cur-Step: 19650, loss(cross_entropy): 0.00247, Running average loss:0.00243, Time taken: 0:00:15.103649 ETA: 0:04:31.865676 Epoch: 983/1000:, Cur-Step: 19660, loss(cross_entropy): 0.00230, Running average loss:0.00230, Time taken: 0:00:14.890328 ETA: 0:04:13.135575 2022-07-28 20:34:08,385 [INFO] __main__: Epoch: 983/1000:, Cur-Step: 19660, loss(cross_entropy): 0.00230, Running average loss:0.00230, Time taken: 0:00:14.890328 ETA: 0:04:13.135575 Epoch: 983/1000:, Cur-Step: 19670, loss(cross_entropy): 0.00240, Running average loss:0.00236, Time taken: 0:00:14.890328 ETA: 0:04:13.135575 2022-07-28 20:34:15,559 [INFO] __main__: Epoch: 983/1000:, Cur-Step: 19670, loss(cross_entropy): 0.00240, Running average loss:0.00236, Time taken: 0:00:14.890328 ETA: 0:04:13.135575 Epoch: 984/1000:, Cur-Step: 19680, loss(cross_entropy): 0.00248, Running average loss:0.00248, Time taken: 0:00:14.977632 ETA: 0:03:59.642113 2022-07-28 20:34:22,646 [INFO] __main__: Epoch: 984/1000:, Cur-Step: 19680, loss(cross_entropy): 0.00248, Running average loss:0.00248, Time taken: 0:00:14.977632 ETA: 0:03:59.642113 Epoch: 984/1000:, Cur-Step: 19690, loss(cross_entropy): 0.00247, Running average loss:0.00236, Time taken: 0:00:14.977632 ETA: 0:03:59.642113 2022-07-28 20:34:29,978 [INFO] __main__: Epoch: 984/1000:, Cur-Step: 19690, loss(cross_entropy): 0.00247, Running average loss:0.00236, Time taken: 0:00:14.977632 ETA: 0:03:59.642113 Epoch: 985/1000:, Cur-Step: 19700, loss(cross_entropy): 0.00230, Running average loss:0.00230, Time taken: 0:00:15.113051 ETA: 0:03:46.695764 2022-07-28 20:34:37,050 [INFO] __main__: Epoch: 985/1000:, Cur-Step: 19700, loss(cross_entropy): 0.00230, Running average loss:0.00230, Time taken: 0:00:15.113051 ETA: 0:03:46.695764 Epoch: 985/1000:, Cur-Step: 19710, loss(cross_entropy): 0.00249, Running average loss:0.00235, Time taken: 0:00:15.113051 ETA: 0:03:46.695764 2022-07-28 20:34:44,118 [INFO] __main__: Epoch: 985/1000:, Cur-Step: 19710, loss(cross_entropy): 0.00249, Running average loss:0.00235, Time taken: 0:00:15.113051 ETA: 0:03:46.695764 Epoch: 986/1000:, Cur-Step: 19720, loss(cross_entropy): 0.00232, Running average loss:0.00232, Time taken: 0:00:14.741565 ETA: 0:03:26.381907 2022-07-28 20:34:51,113 [INFO] __main__: Epoch: 986/1000:, Cur-Step: 19720, loss(cross_entropy): 0.00232, Running average loss:0.00232, Time taken: 0:00:14.741565 ETA: 0:03:26.381907 Epoch: 986/1000:, Cur-Step: 19730, loss(cross_entropy): 0.00254, Running average loss:0.00241, Time taken: 0:00:14.741565 ETA: 0:03:26.381907 2022-07-28 20:34:58,416 [INFO] __main__: Epoch: 986/1000:, Cur-Step: 19730, loss(cross_entropy): 0.00254, Running average loss:0.00241, Time taken: 0:00:14.741565 ETA: 0:03:26.381907 Epoch: 987/1000:, Cur-Step: 19740, loss(cross_entropy): 0.00268, Running average loss:0.00268, Time taken: 0:00:15.011565 ETA: 0:03:15.150342 2022-07-28 20:35:05,437 [INFO] __main__: Epoch: 987/1000:, Cur-Step: 19740, loss(cross_entropy): 0.00268, Running average loss:0.00268, Time taken: 0:00:15.011565 ETA: 0:03:15.150342 Epoch: 987/1000:, Cur-Step: 19750, loss(cross_entropy): 0.00266, Running average loss:0.00255, Time taken: 0:00:15.011565 ETA: 0:03:15.150342 2022-07-28 20:35:12,552 [INFO] __main__: Epoch: 987/1000:, Cur-Step: 19750, loss(cross_entropy): 0.00266, Running average loss:0.00255, Time taken: 0:00:15.011565 ETA: 0:03:15.150342 Epoch: 988/1000:, Cur-Step: 19760, loss(cross_entropy): 0.00256, Running average loss:0.00256, Time taken: 0:00:14.713608 ETA: 0:02:56.563299 2022-07-28 20:35:19,468 [INFO] __main__: Epoch: 988/1000:, Cur-Step: 19760, loss(cross_entropy): 0.00256, Running average loss:0.00256, Time taken: 0:00:14.713608 ETA: 0:02:56.563299 Epoch: 988/1000:, Cur-Step: 19770, loss(cross_entropy): 0.00238, Running average loss:0.00248, Time taken: 0:00:14.713608 ETA: 0:02:56.563299 2022-07-28 20:35:26,551 [INFO] __main__: Epoch: 988/1000:, Cur-Step: 19770, loss(cross_entropy): 0.00238, Running average loss:0.00248, Time taken: 0:00:14.713608 ETA: 0:02:56.563299 Epoch: 989/1000:, Cur-Step: 19780, loss(cross_entropy): 0.00259, Running average loss:0.00259, Time taken: 0:00:14.736200 ETA: 0:02:42.098201 2022-07-28 20:35:33,539 [INFO] __main__: Epoch: 989/1000:, Cur-Step: 19780, loss(cross_entropy): 0.00259, Running average loss:0.00259, Time taken: 0:00:14.736200 ETA: 0:02:42.098201 Epoch: 989/1000:, Cur-Step: 19790, loss(cross_entropy): 0.00250, Running average loss:0.00250, Time taken: 0:00:14.736200 ETA: 0:02:42.098201 2022-07-28 20:35:40,395 [INFO] __main__: Epoch: 989/1000:, Cur-Step: 19790, loss(cross_entropy): 0.00250, Running average loss:0.00250, Time taken: 0:00:14.736200 ETA: 0:02:42.098201 Epoch: 990/1000:, Cur-Step: 19800, loss(cross_entropy): 0.00242, Running average loss:0.00242, Time taken: 0:00:14.375534 ETA: 0:02:23.755341 2022-07-28 20:35:47,300 [INFO] __main__: Epoch: 990/1000:, Cur-Step: 19800, loss(cross_entropy): 0.00242, Running average loss:0.00242, Time taken: 0:00:14.375534 ETA: 0:02:23.755341 Epoch: 990/1000:, Cur-Step: 19810, loss(cross_entropy): 0.00257, Running average loss:0.00253, Time taken: 0:00:14.375534 ETA: 0:02:23.755341 2022-07-28 20:35:54,225 [INFO] __main__: Epoch: 990/1000:, Cur-Step: 19810, loss(cross_entropy): 0.00257, Running average loss:0.00253, Time taken: 0:00:14.375534 ETA: 0:02:23.755341 Epoch: 991/1000:, Cur-Step: 19820, loss(cross_entropy): 0.00262, Running average loss:0.00262, Time taken: 0:00:14.627045 ETA: 0:02:11.643404 2022-07-28 20:36:01,269 [INFO] __main__: Epoch: 991/1000:, Cur-Step: 19820, loss(cross_entropy): 0.00262, Running average loss:0.00262, Time taken: 0:00:14.627045 ETA: 0:02:11.643404 Epoch: 991/1000:, Cur-Step: 19830, loss(cross_entropy): 0.00291, Running average loss:0.00261, Time taken: 0:00:14.627045 ETA: 0:02:11.643404 2022-07-28 20:36:08,185 [INFO] __main__: Epoch: 991/1000:, Cur-Step: 19830, loss(cross_entropy): 0.00291, Running average loss:0.00261, Time taken: 0:00:14.627045 ETA: 0:02:11.643404 Epoch: 992/1000:, Cur-Step: 19840, loss(cross_entropy): 0.00268, Running average loss:0.00268, Time taken: 0:00:14.570726 ETA: 0:01:56.565805 2022-07-28 20:36:15,127 [INFO] __main__: Epoch: 992/1000:, Cur-Step: 19840, loss(cross_entropy): 0.00268, Running average loss:0.00268, Time taken: 0:00:14.570726 ETA: 0:01:56.565805 Epoch: 992/1000:, Cur-Step: 19850, loss(cross_entropy): 0.00293, Running average loss:0.00277, Time taken: 0:00:14.570726 ETA: 0:01:56.565805 2022-07-28 20:36:21,998 [INFO] __main__: Epoch: 992/1000:, Cur-Step: 19850, loss(cross_entropy): 0.00293, Running average loss:0.00277, Time taken: 0:00:14.570726 ETA: 0:01:56.565805 Epoch: 993/1000:, Cur-Step: 19860, loss(cross_entropy): 0.00285, Running average loss:0.00285, Time taken: 0:00:14.621414 ETA: 0:01:42.349896 2022-07-28 20:36:29,026 [INFO] __main__: Epoch: 993/1000:, Cur-Step: 19860, loss(cross_entropy): 0.00285, Running average loss:0.00285, Time taken: 0:00:14.621414 ETA: 0:01:42.349896 Epoch: 993/1000:, Cur-Step: 19870, loss(cross_entropy): 0.00323, Running average loss:0.00326, Time taken: 0:00:14.621414 ETA: 0:01:42.349896 2022-07-28 20:36:35,900 [INFO] __main__: Epoch: 993/1000:, Cur-Step: 19870, loss(cross_entropy): 0.00323, Running average loss:0.00326, Time taken: 0:00:14.621414 ETA: 0:01:42.349896 Epoch: 994/1000:, Cur-Step: 19880, loss(cross_entropy): 0.00345, Running average loss:0.00345, Time taken: 0:00:14.537515 ETA: 0:01:27.225088 2022-07-28 20:36:42,810 [INFO] __main__: Epoch: 994/1000:, Cur-Step: 19880, loss(cross_entropy): 0.00345, Running average loss:0.00345, Time taken: 0:00:14.537515 ETA: 0:01:27.225088 Epoch: 994/1000:, Cur-Step: 19890, loss(cross_entropy): 0.00364, Running average loss:0.00360, Time taken: 0:00:14.537515 ETA: 0:01:27.225088 2022-07-28 20:36:49,622 [INFO] __main__: Epoch: 994/1000:, Cur-Step: 19890, loss(cross_entropy): 0.00364, Running average loss:0.00360, Time taken: 0:00:14.537515 ETA: 0:01:27.225088 Epoch: 995/1000:, Cur-Step: 19900, loss(cross_entropy): 0.00392, Running average loss:0.00392, Time taken: 0:00:14.414642 ETA: 0:01:12.073209 2022-07-28 20:36:56,531 [INFO] __main__: Epoch: 995/1000:, Cur-Step: 19900, loss(cross_entropy): 0.00392, Running average loss:0.00392, Time taken: 0:00:14.414642 ETA: 0:01:12.073209 Epoch: 995/1000:, Cur-Step: 19910, loss(cross_entropy): 0.00331, Running average loss:0.00344, Time taken: 0:00:14.414642 ETA: 0:01:12.073209 2022-07-28 20:37:03,443 [INFO] __main__: Epoch: 995/1000:, Cur-Step: 19910, loss(cross_entropy): 0.00331, Running average loss:0.00344, Time taken: 0:00:14.414642 ETA: 0:01:12.073209 Epoch: 996/1000:, Cur-Step: 19920, loss(cross_entropy): 0.00289, Running average loss:0.00289, Time taken: 0:00:14.551091 ETA: 0:00:58.204363 2022-07-28 20:37:10,368 [INFO] __main__: Epoch: 996/1000:, Cur-Step: 19920, loss(cross_entropy): 0.00289, Running average loss:0.00289, Time taken: 0:00:14.551091 ETA: 0:00:58.204363 Epoch: 996/1000:, Cur-Step: 19930, loss(cross_entropy): 0.00275, Running average loss:0.00296, Time taken: 0:00:14.551091 ETA: 0:00:58.204363 2022-07-28 20:37:17,179 [INFO] __main__: Epoch: 996/1000:, Cur-Step: 19930, loss(cross_entropy): 0.00275, Running average loss:0.00296, Time taken: 0:00:14.551091 ETA: 0:00:58.204363 Epoch: 997/1000:, Cur-Step: 19940, loss(cross_entropy): 0.00275, Running average loss:0.00275, Time taken: 0:00:14.464030 ETA: 0:00:43.392091 2022-07-28 20:37:24,136 [INFO] __main__: Epoch: 997/1000:, Cur-Step: 19940, loss(cross_entropy): 0.00275, Running average loss:0.00275, Time taken: 0:00:14.464030 ETA: 0:00:43.392091 Epoch: 997/1000:, Cur-Step: 19950, loss(cross_entropy): 0.00267, Running average loss:0.00282, Time taken: 0:00:14.464030 ETA: 0:00:43.392091 2022-07-28 20:37:31,176 [INFO] __main__: Epoch: 997/1000:, Cur-Step: 19950, loss(cross_entropy): 0.00267, Running average loss:0.00282, Time taken: 0:00:14.464030 ETA: 0:00:43.392091 Epoch: 998/1000:, Cur-Step: 19960, loss(cross_entropy): 0.00258, Running average loss:0.00258, Time taken: 0:00:14.741427 ETA: 0:00:29.482855 2022-07-28 20:37:38,171 [INFO] __main__: Epoch: 998/1000:, Cur-Step: 19960, loss(cross_entropy): 0.00258, Running average loss:0.00258, Time taken: 0:00:14.741427 ETA: 0:00:29.482855 Epoch: 998/1000:, Cur-Step: 19970, loss(cross_entropy): 0.00265, Running average loss:0.00269, Time taken: 0:00:14.741427 ETA: 0:00:29.482855 2022-07-28 20:37:45,021 [INFO] __main__: Epoch: 998/1000:, Cur-Step: 19970, loss(cross_entropy): 0.00265, Running average loss:0.00269, Time taken: 0:00:14.741427 ETA: 0:00:29.482855 Epoch: 999/1000:, Cur-Step: 19980, loss(cross_entropy): 0.00268, Running average loss:0.00268, Time taken: 0:00:14.558043 ETA: 0:00:14.558043 2022-07-28 20:37:52,035 [INFO] __main__: Epoch: 999/1000:, Cur-Step: 19980, loss(cross_entropy): 0.00268, Running average loss:0.00268, Time taken: 0:00:14.558043 ETA: 0:00:14.558043 Epoch: 999/1000:, Cur-Step: 19990, loss(cross_entropy): 0.00266, Running average loss:0.00253, Time taken: 0:00:14.558043 ETA: 0:00:14.558043 2022-07-28 20:37:59,040 [INFO] __main__: Epoch: 999/1000:, Cur-Step: 19990, loss(cross_entropy): 0.00266, Running average loss:0.00253, Time taken: 0:00:14.558043 ETA: 0:00:14.558043 INFO:tensorflow:Saving checkpoints for step-20000. 2022-07-28 20:38:05,416 [INFO] tensorflow: Saving checkpoints for step-20000. Throughput Avg: 5.711 img/s Latency Avg: 715.768 ms Latency 90%: 735.631 ms Latency 95%: 739.435 ms Latency 99%: 746.873 ms DLL 2022-07-28 20:38:28.039675 - () throughput_train:5.710793299407804 latency_train:715.7683600244238 elapsed_time:2997.826479 INFO:tensorflow:Loss for final step: 0.002450631. 2022-07-28 20:38:28,126 [INFO] tensorflow: Loss for final step: 0.002450631. Saving the final step model to /workspace/tao-experiments/unpruned/weights/model.tlt 2022-07-28 20:38:28,130 [INFO] __main__: Saving the final step model to /workspace/tao-experiments/unpruned/weights/model.tlt 2022-07-28 16:38:37,695 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.