2021-10-19 02:51:27,983 [INFO] root: Registry: ['nvcr.io'] 2021-10-19 02:51:28,130 [WARNING] tlt.components.docker_handler.docker_handler: Docker will run the commands as root. If you would like to retain your local host permissions, please add the "user":"UID:GID" in the DockerOptions portion of the "/home/paperspace/.tao_mounts.json" file. You can obtain your users UID and GID by using the "id -u" and "id -g" commands on the terminal. Using TensorFlow backend. WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them. Using TensorFlow backend. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:43: The name tf.train.SessionRunHook is deprecated. Please use tf.estimator.SessionRunHook instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/checkpoint_saver_hook.py:25: The name tf.train.CheckpointSaverHook is deprecated. Please use tf.estimator.CheckpointSaverHook instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/scripts/train.py:68: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/scripts/train.py:68: The name tf.logging.INFO is deprecated. Please use tf.compat.v1.logging.INFO instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod/tensorflow/__init__.py:117: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead. 2021-10-19 06:51:36,251 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/horovod/tensorflow/__init__.py:117: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod/tensorflow/__init__.py:143: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead. 2021-10-19 06:51:36,252 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/horovod/tensorflow/__init__.py:143: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead. 2021-10-19 06:51:36,882 [INFO] iva.common.logging.logging: Log file already exists at /workspace/tao-experiments/detectnet_v2/experiment_dir_unpruned/status.json 2021-10-19 06:51:36,883 [INFO] __main__: Loading experiment spec at /workspace/tao-experiments/detectnet_v2/specs/detectnet_v2_train_trafficcamnet.txt. 2021-10-19 06:51:36,885 [INFO] iva.detectnet_v2.spec_handler.spec_loader: Merging specification from /workspace/tao-experiments/detectnet_v2/specs/detectnet_v2_train_trafficcamnet.txt 2021-10-19 06:51:37,556 [INFO] __main__: Cannot iterate over exactly 1543 samples with a batch size of 4; each epoch will therefore take one extra step. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:107: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead. 2021-10-19 06:51:37,560 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:107: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:110: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead. 2021-10-19 06:51:37,561 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:110: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:113: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead. 2021-10-19 06:51:37,564 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:113: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead. 2021-10-19 06:51:37,771 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead. 2021-10-19 06:51:37,772 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1834: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead. 2021-10-19 06:51:37,802 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1834: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead. 2021-10-19 06:51:40,225 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead. 2021-10-19 06:51:40,226 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead. 2021-10-19 06:51:40,723 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead. /usr/local/lib/python3.6/dist-packages/keras/engine/saving.py:292: UserWarning: No training configuration found in save file: the model was *not* compiled. Compile it manually. warnings.warn('No training configuration found in save file: ' 2021-10-19 06:51:55,129 [INFO] iva.detectnet_v2.objectives.bbox_objective: Default L1 loss function will be used. __________________________________________________________________________________________________ Layer (type) Output Shape Param # Connected to ================================================================================================== input_1 (InputLayer) (None, 3, 544, 960) 0 __________________________________________________________________________________________________ conv1 (Conv2D) (None, 64, 272, 480) 9472 input_1[0][0] __________________________________________________________________________________________________ bn_conv1 (BatchNormalization) (None, 64, 272, 480) 256 conv1[0][0] __________________________________________________________________________________________________ activation_1 (Activation) (None, 64, 272, 480) 0 bn_conv1[0][0] __________________________________________________________________________________________________ block_1a_conv_1 (Conv2D) (None, 64, 136, 240) 36928 activation_1[0][0] __________________________________________________________________________________________________ block_1a_bn_1 (BatchNormalizati (None, 64, 136, 240) 256 block_1a_conv_1[0][0] __________________________________________________________________________________________________ block_1a_relu_1 (Activation) (None, 64, 136, 240) 0 block_1a_bn_1[0][0] __________________________________________________________________________________________________ block_1a_conv_2 (Conv2D) (None, 64, 136, 240) 36928 block_1a_relu_1[0][0] __________________________________________________________________________________________________ block_1a_conv_shortcut (Conv2D) (None, 64, 136, 240) 4160 activation_1[0][0] __________________________________________________________________________________________________ block_1a_bn_2 (BatchNormalizati (None, 64, 136, 240) 256 block_1a_conv_2[0][0] __________________________________________________________________________________________________ block_1a_bn_shortcut (BatchNorm (None, 64, 136, 240) 256 block_1a_conv_shortcut[0][0] __________________________________________________________________________________________________ add_1 (Add) (None, 64, 136, 240) 0 block_1a_bn_2[0][0] block_1a_bn_shortcut[0][0] __________________________________________________________________________________________________ block_1a_relu (Activation) (None, 64, 136, 240) 0 add_1[0][0] __________________________________________________________________________________________________ block_1b_conv_1 (Conv2D) (None, 64, 136, 240) 36928 block_1a_relu[0][0] __________________________________________________________________________________________________ block_1b_bn_1 (BatchNormalizati (None, 64, 136, 240) 256 block_1b_conv_1[0][0] __________________________________________________________________________________________________ block_1b_relu_1 (Activation) (None, 64, 136, 240) 0 block_1b_bn_1[0][0] __________________________________________________________________________________________________ block_1b_conv_2 (Conv2D) (None, 64, 136, 240) 36928 block_1b_relu_1[0][0] __________________________________________________________________________________________________ block_1b_conv_shortcut (Conv2D) (None, 64, 136, 240) 4160 block_1a_relu[0][0] __________________________________________________________________________________________________ block_1b_bn_2 (BatchNormalizati (None, 64, 136, 240) 256 block_1b_conv_2[0][0] __________________________________________________________________________________________________ block_1b_bn_shortcut (BatchNorm (None, 64, 136, 240) 256 block_1b_conv_shortcut[0][0] __________________________________________________________________________________________________ add_2 (Add) (None, 64, 136, 240) 0 block_1b_bn_2[0][0] block_1b_bn_shortcut[0][0] __________________________________________________________________________________________________ block_1b_relu (Activation) (None, 64, 136, 240) 0 add_2[0][0] __________________________________________________________________________________________________ block_2a_conv_1 (Conv2D) (None, 128, 68, 120) 73856 block_1b_relu[0][0] __________________________________________________________________________________________________ block_2a_bn_1 (BatchNormalizati (None, 128, 68, 120) 512 block_2a_conv_1[0][0] __________________________________________________________________________________________________ block_2a_relu_1 (Activation) (None, 128, 68, 120) 0 block_2a_bn_1[0][0] __________________________________________________________________________________________________ block_2a_conv_2 (Conv2D) (None, 128, 68, 120) 147584 block_2a_relu_1[0][0] __________________________________________________________________________________________________ block_2a_conv_shortcut (Conv2D) (None, 128, 68, 120) 8320 block_1b_relu[0][0] __________________________________________________________________________________________________ block_2a_bn_2 (BatchNormalizati (None, 128, 68, 120) 512 block_2a_conv_2[0][0] __________________________________________________________________________________________________ block_2a_bn_shortcut (BatchNorm (None, 128, 68, 120) 512 block_2a_conv_shortcut[0][0] __________________________________________________________________________________________________ add_3 (Add) (None, 128, 68, 120) 0 block_2a_bn_2[0][0] block_2a_bn_shortcut[0][0] __________________________________________________________________________________________________ block_2a_relu (Activation) (None, 128, 68, 120) 0 add_3[0][0] __________________________________________________________________________________________________ block_2b_conv_1 (Conv2D) (None, 128, 68, 120) 147584 block_2a_relu[0][0] __________________________________________________________________________________________________ block_2b_bn_1 (BatchNormalizati (None, 128, 68, 120) 512 block_2b_conv_1[0][0] __________________________________________________________________________________________________ block_2b_relu_1 (Activation) (None, 128, 68, 120) 0 block_2b_bn_1[0][0] __________________________________________________________________________________________________ block_2b_conv_2 (Conv2D) (None, 128, 68, 120) 147584 block_2b_relu_1[0][0] __________________________________________________________________________________________________ block_2b_conv_shortcut (Conv2D) (None, 128, 68, 120) 16512 block_2a_relu[0][0] __________________________________________________________________________________________________ block_2b_bn_2 (BatchNormalizati (None, 128, 68, 120) 512 block_2b_conv_2[0][0] __________________________________________________________________________________________________ block_2b_bn_shortcut (BatchNorm (None, 128, 68, 120) 512 block_2b_conv_shortcut[0][0] __________________________________________________________________________________________________ add_4 (Add) (None, 128, 68, 120) 0 block_2b_bn_2[0][0] block_2b_bn_shortcut[0][0] __________________________________________________________________________________________________ block_2b_relu (Activation) (None, 128, 68, 120) 0 add_4[0][0] __________________________________________________________________________________________________ block_3a_conv_1 (Conv2D) (None, 256, 34, 60) 295168 block_2b_relu[0][0] __________________________________________________________________________________________________ block_3a_bn_1 (BatchNormalizati (None, 256, 34, 60) 1024 block_3a_conv_1[0][0] __________________________________________________________________________________________________ block_3a_relu_1 (Activation) (None, 256, 34, 60) 0 block_3a_bn_1[0][0] __________________________________________________________________________________________________ block_3a_conv_2 (Conv2D) (None, 256, 34, 60) 590080 block_3a_relu_1[0][0] __________________________________________________________________________________________________ block_3a_conv_shortcut (Conv2D) (None, 256, 34, 60) 33024 block_2b_relu[0][0] __________________________________________________________________________________________________ block_3a_bn_2 (BatchNormalizati (None, 256, 34, 60) 1024 block_3a_conv_2[0][0] __________________________________________________________________________________________________ block_3a_bn_shortcut (BatchNorm (None, 256, 34, 60) 1024 block_3a_conv_shortcut[0][0] __________________________________________________________________________________________________ add_5 (Add) (None, 256, 34, 60) 0 block_3a_bn_2[0][0] block_3a_bn_shortcut[0][0] __________________________________________________________________________________________________ block_3a_relu (Activation) (None, 256, 34, 60) 0 add_5[0][0] __________________________________________________________________________________________________ block_3b_conv_1 (Conv2D) (None, 256, 34, 60) 590080 block_3a_relu[0][0] __________________________________________________________________________________________________ block_3b_bn_1 (BatchNormalizati (None, 256, 34, 60) 1024 block_3b_conv_1[0][0] __________________________________________________________________________________________________ block_3b_relu_1 (Activation) (None, 256, 34, 60) 0 block_3b_bn_1[0][0] __________________________________________________________________________________________________ block_3b_conv_2 (Conv2D) (None, 256, 34, 60) 590080 block_3b_relu_1[0][0] __________________________________________________________________________________________________ block_3b_conv_shortcut (Conv2D) (None, 256, 34, 60) 65792 block_3a_relu[0][0] __________________________________________________________________________________________________ block_3b_bn_2 (BatchNormalizati (None, 256, 34, 60) 1024 block_3b_conv_2[0][0] __________________________________________________________________________________________________ block_3b_bn_shortcut (BatchNorm (None, 256, 34, 60) 1024 block_3b_conv_shortcut[0][0] __________________________________________________________________________________________________ add_6 (Add) (None, 256, 34, 60) 0 block_3b_bn_2[0][0] block_3b_bn_shortcut[0][0] __________________________________________________________________________________________________ block_3b_relu (Activation) (None, 256, 34, 60) 0 add_6[0][0] __________________________________________________________________________________________________ block_4a_conv_1 (Conv2D) (None, 512, 34, 60) 1180160 block_3b_relu[0][0] __________________________________________________________________________________________________ block_4a_bn_1 (BatchNormalizati (None, 512, 34, 60) 2048 block_4a_conv_1[0][0] __________________________________________________________________________________________________ block_4a_relu_1 (Activation) (None, 512, 34, 60) 0 block_4a_bn_1[0][0] __________________________________________________________________________________________________ block_4a_conv_2 (Conv2D) (None, 512, 34, 60) 2359808 block_4a_relu_1[0][0] __________________________________________________________________________________________________ block_4a_conv_shortcut (Conv2D) (None, 512, 34, 60) 131584 block_3b_relu[0][0] __________________________________________________________________________________________________ block_4a_bn_2 (BatchNormalizati (None, 512, 34, 60) 2048 block_4a_conv_2[0][0] __________________________________________________________________________________________________ block_4a_bn_shortcut (BatchNorm (None, 512, 34, 60) 2048 block_4a_conv_shortcut[0][0] __________________________________________________________________________________________________ add_7 (Add) (None, 512, 34, 60) 0 block_4a_bn_2[0][0] block_4a_bn_shortcut[0][0] __________________________________________________________________________________________________ block_4a_relu (Activation) (None, 512, 34, 60) 0 add_7[0][0] __________________________________________________________________________________________________ block_4b_conv_1 (Conv2D) (None, 512, 34, 60) 2359808 block_4a_relu[0][0] __________________________________________________________________________________________________ block_4b_bn_1 (BatchNormalizati (None, 512, 34, 60) 2048 block_4b_conv_1[0][0] __________________________________________________________________________________________________ block_4b_relu_1 (Activation) (None, 512, 34, 60) 0 block_4b_bn_1[0][0] __________________________________________________________________________________________________ block_4b_conv_2 (Conv2D) (None, 512, 34, 60) 2359808 block_4b_relu_1[0][0] __________________________________________________________________________________________________ block_4b_conv_shortcut (Conv2D) (None, 512, 34, 60) 262656 block_4a_relu[0][0] __________________________________________________________________________________________________ block_4b_bn_2 (BatchNormalizati (None, 512, 34, 60) 2048 block_4b_conv_2[0][0] __________________________________________________________________________________________________ block_4b_bn_shortcut (BatchNorm (None, 512, 34, 60) 2048 block_4b_conv_shortcut[0][0] __________________________________________________________________________________________________ add_8 (Add) (None, 512, 34, 60) 0 block_4b_bn_2[0][0] block_4b_bn_shortcut[0][0] __________________________________________________________________________________________________ block_4b_relu (Activation) (None, 512, 34, 60) 0 add_8[0][0] __________________________________________________________________________________________________ output_bbox (Conv2D) (None, 28, 34, 60) 14364 block_4b_relu[0][0] __________________________________________________________________________________________________ output_cov (Conv2D) (None, 7, 34, 60) 3591 block_4b_relu[0][0] ================================================================================================== Total params: 11,566,243 Trainable params: 11,554,595 Non-trainable params: 11,648 __________________________________________________________________________________________________ 2021-10-19 06:51:55,182 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Serial augmentation enabled = False 2021-10-19 06:51:55,182 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Pseudo sharding enabled = False 2021-10-19 06:51:55,182 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Max Image Dimensions (all sources): (0, 0) 2021-10-19 06:51:55,182 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: number of cpus: 8, io threads: 16, compute threads: 8, buffered batches: 4 2021-10-19 06:51:55,183 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: total dataset size 1543, number of sources: 1, batch size per gpu: 4, steps: 386 WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead. 2021-10-19 06:51:55,236 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead. WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2021-10-19 06:51:55,289 [WARNING] tensorflow: Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2021-10-19 06:51:55,315 [INFO] iva.detectnet_v2.dataloader.default_dataloader: Bounding box coordinates were detected in the input specification! Bboxes will be automatically converted to polygon coordinates. 2021-10-19 06:51:55,644 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: shuffle: True - shard 0 of 1 2021-10-19 06:51:55,654 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: sampling 1 datasets with weights: 2021-10-19 06:51:55,654 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: source: 0 weight: 1.000000 WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2021-10-19 06:51:55,673 [WARNING] tensorflow: Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2021-10-19 06:51:56,289 [INFO] __main__: Found 1543 samples in training set WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/rasterizers/bbox_rasterizer.py:347: The name tf.bincount is deprecated. Please use tf.math.bincount instead. 2021-10-19 06:51:56,423 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/rasterizers/bbox_rasterizer.py:347: The name tf.bincount is deprecated. Please use tf.math.bincount instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/training_proto_utilities.py:89: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead. 2021-10-19 06:51:56,626 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/training_proto_utilities.py:89: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/training_proto_utilities.py:36: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead. 2021-10-19 06:51:56,649 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/training_proto_utilities.py:36: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_functions.py:17: The name tf.log is deprecated. Please use tf.math.log instead. 2021-10-19 06:51:56,888 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_functions.py:17: The name tf.log is deprecated. Please use tf.math.log instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:235: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead. 2021-10-19 06:51:57,022 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:235: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/model/detectnet_model.py:587: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead. 2021-10-19 06:51:57,051 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/model/detectnet_model.py:587: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead. 2021-10-19 06:51:59,778 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Serial augmentation enabled = False 2021-10-19 06:51:59,778 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Pseudo sharding enabled = False 2021-10-19 06:51:59,779 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Max Image Dimensions (all sources): (0, 0) 2021-10-19 06:51:59,779 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: number of cpus: 8, io threads: 16, compute threads: 8, buffered batches: 4 2021-10-19 06:51:59,779 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: total dataset size 251, number of sources: 1, batch size per gpu: 4, steps: 63 WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2021-10-19 06:51:59,795 [WARNING] tensorflow: Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2021-10-19 06:51:59,822 [INFO] iva.detectnet_v2.dataloader.default_dataloader: Bounding box coordinates were detected in the input specification! Bboxes will be automatically converted to polygon coordinates. 2021-10-19 06:52:00,109 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: shuffle: False - shard 0 of 1 2021-10-19 06:52:00,115 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: sampling 1 datasets with weights: 2021-10-19 06:52:00,115 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: source: 0 weight: 1.000000 WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2021-10-19 06:52:00,133 [WARNING] tensorflow: Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2021-10-19 06:52:00,569 [INFO] __main__: Found 251 samples in validation set WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/validation_hook.py:40: The name tf.summary.FileWriterCache is deprecated. Please use tf.compat.v1.summary.FileWriterCache instead. 2021-10-19 06:52:01,493 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/validation_hook.py:40: The name tf.summary.FileWriterCache is deprecated. Please use tf.compat.v1.summary.FileWriterCache instead. 2021-10-19 06:52:03,401 [INFO] __main__: Checkpoint interval: 10 WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/scripts/train.py:108: The name tf.train.Scaffold is deprecated. Please use tf.compat.v1.train.Scaffold instead. 2021-10-19 06:52:03,402 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/scripts/train.py:108: The name tf.train.Scaffold is deprecated. Please use tf.compat.v1.train.Scaffold instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:14: The name tf.local_variables_initializer is deprecated. Please use tf.compat.v1.local_variables_initializer instead. 2021-10-19 06:52:03,402 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:14: The name tf.local_variables_initializer is deprecated. Please use tf.compat.v1.local_variables_initializer instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:15: The name tf.tables_initializer is deprecated. Please use tf.compat.v1.tables_initializer instead. 2021-10-19 06:52:03,403 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:15: The name tf.tables_initializer is deprecated. Please use tf.compat.v1.tables_initializer instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:16: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead. 2021-10-19 06:52:03,404 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:16: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/utils.py:59: The name tf.train.LoggingTensorHook is deprecated. Please use tf.estimator.LoggingTensorHook instead. 2021-10-19 06:52:03,407 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/utils.py:59: The name tf.train.LoggingTensorHook is deprecated. Please use tf.estimator.LoggingTensorHook instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/utils.py:60: The name tf.train.StopAtStepHook is deprecated. Please use tf.estimator.StopAtStepHook instead. 2021-10-19 06:52:03,407 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/utils.py:60: The name tf.train.StopAtStepHook is deprecated. Please use tf.estimator.StopAtStepHook instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/utils.py:73: The name tf.train.StepCounterHook is deprecated. Please use tf.estimator.StepCounterHook instead. 2021-10-19 06:52:03,407 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/utils.py:73: The name tf.train.StepCounterHook is deprecated. Please use tf.estimator.StepCounterHook instead. INFO:tensorflow:Create CheckpointSaverHook. 2021-10-19 06:52:03,408 [INFO] tensorflow: Create CheckpointSaverHook. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/utils.py:99: The name tf.train.SummarySaverHook is deprecated. Please use tf.estimator.SummarySaverHook instead. 2021-10-19 06:52:03,408 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/utils.py:99: The name tf.train.SummarySaverHook is deprecated. Please use tf.estimator.SummarySaverHook instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/utilities.py:140: The name tf.train.SingularMonitoredSession is deprecated. Please use tf.compat.v1.train.SingularMonitoredSession instead. 2021-10-19 06:52:03,409 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/utilities.py:140: The name tf.train.SingularMonitoredSession is deprecated. Please use tf.compat.v1.train.SingularMonitoredSession instead. INFO:tensorflow:Graph was finalized. 2021-10-19 06:52:04,607 [INFO] tensorflow: Graph was finalized. INFO:tensorflow:Running local_init_op. 2021-10-19 06:52:07,388 [INFO] tensorflow: Running local_init_op. INFO:tensorflow:Done running local_init_op. 2021-10-19 06:52:08,376 [INFO] tensorflow: Done running local_init_op. INFO:tensorflow:Saving checkpoints for step-0. 2021-10-19 06:52:17,663 [INFO] tensorflow: Saving checkpoints for step-0. INFO:tensorflow:epoch = 0.0, learning_rate = 4.9999994e-06, loss = 0.05273571, step = 0 2021-10-19 06:53:03,933 [INFO] tensorflow: epoch = 0.0, learning_rate = 4.9999994e-06, loss = 0.05273571, step = 0 2021-10-19 06:53:03,936 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 0/120: loss: 0.05274 learning rate: 0.00000 Time taken: 0:00:00 ETA: 0:00:00 2021-10-19 06:53:03,936 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 0.334 INFO:tensorflow:epoch = 0.02072538860103627, learning_rate = 5.039926e-06, loss = 0.051589854, step = 8 (5.321 sec) 2021-10-19 06:53:09,255 [INFO] tensorflow: epoch = 0.02072538860103627, learning_rate = 5.039926e-06, loss = 0.051589854, step = 8 (5.321 sec) 2021-10-19 06:53:12,534 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 4.864 INFO:tensorflow:epoch = 0.08808290155440415, learning_rate = 5.171903e-06, loss = 0.049986076, step = 34 (5.287 sec) 2021-10-19 06:53:14,541 [INFO] tensorflow: epoch = 0.08808290155440415, learning_rate = 5.171903e-06, loss = 0.049986076, step = 34 (5.287 sec) INFO:tensorflow:global_step/sec: 3.33459 2021-10-19 06:53:15,331 [INFO] tensorflow: global_step/sec: 3.33459 2021-10-19 06:53:17,611 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.699 INFO:tensorflow:epoch = 0.15544041450777202, learning_rate = 5.307336e-06, loss = 0.048010126, step = 60 (5.271 sec) 2021-10-19 06:53:19,812 [INFO] tensorflow: epoch = 0.15544041450777202, learning_rate = 5.307336e-06, loss = 0.048010126, step = 60 (5.271 sec) 2021-10-19 06:53:22,547 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.262 INFO:tensorflow:global_step/sec: 4.99425 2021-10-19 06:53:22,940 [INFO] tensorflow: global_step/sec: 4.99425 INFO:tensorflow:epoch = 0.22538860103626943, learning_rate = 5.451736e-06, loss = 0.046450987, step = 87 (5.304 sec) 2021-10-19 06:53:25,116 [INFO] tensorflow: epoch = 0.22538860103626943, learning_rate = 5.451736e-06, loss = 0.046450987, step = 87 (5.304 sec) 2021-10-19 06:53:27,547 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.002 INFO:tensorflow:epoch = 0.2927461139896373, learning_rate = 5.594497e-06, loss = 0.04489236, step = 113 (5.252 sec) 2021-10-19 06:53:30,368 [INFO] tensorflow: epoch = 0.2927461139896373, learning_rate = 5.594497e-06, loss = 0.04489236, step = 113 (5.252 sec) INFO:tensorflow:global_step/sec: 4.98236 2021-10-19 06:53:30,567 [INFO] tensorflow: global_step/sec: 4.98236 2021-10-19 06:53:32,521 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.106 INFO:tensorflow:epoch = 0.3626943005181347, learning_rate = 5.7467046e-06, loss = 0.042737048, step = 140 (5.273 sec) 2021-10-19 06:53:35,641 [INFO] tensorflow: epoch = 0.3626943005181347, learning_rate = 5.7467046e-06, loss = 0.042737048, step = 140 (5.273 sec) 2021-10-19 06:53:37,419 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.415 INFO:tensorflow:global_step/sec: 5.11132 2021-10-19 06:53:38,001 [INFO] tensorflow: global_step/sec: 5.11132 INFO:tensorflow:epoch = 0.43264248704663216, learning_rate = 5.903058e-06, loss = 0.041425988, step = 167 (5.280 sec) 2021-10-19 06:53:40,921 [INFO] tensorflow: epoch = 0.43264248704663216, learning_rate = 5.903058e-06, loss = 0.041425988, step = 167 (5.280 sec) 2021-10-19 06:53:42,279 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.579 INFO:tensorflow:global_step/sec: 5.14249 2021-10-19 06:53:45,390 [INFO] tensorflow: global_step/sec: 5.14249 INFO:tensorflow:epoch = 0.5025906735751295, learning_rate = 6.0636603e-06, loss = 0.039969925, step = 194 (5.296 sec) 2021-10-19 06:53:46,216 [INFO] tensorflow: epoch = 0.5025906735751295, learning_rate = 6.0636603e-06, loss = 0.039969925, step = 194 (5.296 sec) 2021-10-19 06:53:47,196 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.341 INFO:tensorflow:epoch = 0.572538860103627, learning_rate = 6.228637e-06, loss = 0.03822692, step = 221 (5.265 sec) 2021-10-19 06:53:51,481 [INFO] tensorflow: epoch = 0.572538860103627, learning_rate = 6.228637e-06, loss = 0.03822692, step = 221 (5.265 sec) 2021-10-19 06:53:52,063 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.549 INFO:tensorflow:global_step/sec: 5.09899 2021-10-19 06:53:52,843 [INFO] tensorflow: global_step/sec: 5.09899 INFO:tensorflow:epoch = 0.6424870466321244, learning_rate = 6.398104e-06, loss = 0.036993742, step = 248 (5.308 sec) 2021-10-19 06:53:56,789 [INFO] tensorflow: epoch = 0.6424870466321244, learning_rate = 6.398104e-06, loss = 0.036993742, step = 248 (5.308 sec) 2021-10-19 06:53:56,986 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.312 INFO:tensorflow:global_step/sec: 5.06485 2021-10-19 06:54:00,346 [INFO] tensorflow: global_step/sec: 5.06485 2021-10-19 06:54:01,898 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.360 INFO:tensorflow:epoch = 0.7124352331606217, learning_rate = 6.5721747e-06, loss = 0.035692748, step = 275 (5.323 sec) 2021-10-19 06:54:02,112 [INFO] tensorflow: epoch = 0.7124352331606217, learning_rate = 6.5721747e-06, loss = 0.035692748, step = 275 (5.323 sec) 2021-10-19 06:54:06,782 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.480 INFO:tensorflow:epoch = 0.7823834196891192, learning_rate = 6.750987e-06, loss = 0.034320153, step = 302 (5.249 sec) 2021-10-19 06:54:07,361 [INFO] tensorflow: epoch = 0.7823834196891192, learning_rate = 6.750987e-06, loss = 0.034320153, step = 302 (5.249 sec) INFO:tensorflow:global_step/sec: 5.12083 2021-10-19 06:54:07,766 [INFO] tensorflow: global_step/sec: 5.12083 2021-10-19 06:54:11,660 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.500 INFO:tensorflow:epoch = 0.8523316062176166, learning_rate = 6.934659e-06, loss = 0.033337835, step = 329 (5.295 sec) 2021-10-19 06:54:12,655 [INFO] tensorflow: epoch = 0.8523316062176166, learning_rate = 6.934659e-06, loss = 0.033337835, step = 329 (5.295 sec) INFO:tensorflow:global_step/sec: 5.08879 2021-10-19 06:54:15,234 [INFO] tensorflow: global_step/sec: 5.08879 2021-10-19 06:54:16,631 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.121 INFO:tensorflow:epoch = 0.922279792746114, learning_rate = 7.123334e-06, loss = 0.03216769, step = 356 (5.362 sec) 2021-10-19 06:54:18,017 [INFO] tensorflow: epoch = 0.922279792746114, learning_rate = 7.123334e-06, loss = 0.03216769, step = 356 (5.362 sec) 2021-10-19 06:54:21,581 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.202 INFO:tensorflow:global_step/sec: 5.0151 2021-10-19 06:54:22,811 [INFO] tensorflow: global_step/sec: 5.0151 INFO:tensorflow:epoch = 0.9896373056994819, learning_rate = 7.309868e-06, loss = 0.031315926, step = 382 (5.197 sec) 2021-10-19 06:54:23,214 [INFO] tensorflow: epoch = 0.9896373056994819, learning_rate = 7.309868e-06, loss = 0.031315926, step = 382 (5.197 sec) 269e1e9e15a4:46:66 [0] NCCL INFO Bootstrap : Using [0]lo:127.0.0.1<0> [1]eth0:172.17.0.2<0> 269e1e9e15a4:46:66 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation 269e1e9e15a4:46:66 [0] NCCL INFO NET/IB : No device found. 269e1e9e15a4:46:66 [0] NCCL INFO NET/Socket : Using [0]lo:127.0.0.1<0> [1]eth0:172.17.0.2<0> 269e1e9e15a4:46:66 [0] NCCL INFO Using network Socket NCCL version 2.7.8+cuda11.1 269e1e9e15a4:46:66 [0] NCCL INFO Channel 00/32 : 0 269e1e9e15a4:46:66 [0] NCCL INFO Channel 01/32 : 0 269e1e9e15a4:46:66 [0] NCCL INFO Channel 02/32 : 0 269e1e9e15a4:46:66 [0] NCCL INFO Channel 03/32 : 0 269e1e9e15a4:46:66 [0] NCCL INFO Channel 04/32 : 0 269e1e9e15a4:46:66 [0] NCCL INFO Channel 05/32 : 0 269e1e9e15a4:46:66 [0] NCCL INFO Channel 06/32 : 0 269e1e9e15a4:46:66 [0] NCCL INFO Channel 07/32 : 0 269e1e9e15a4:46:66 [0] NCCL INFO Channel 08/32 : 0 269e1e9e15a4:46:66 [0] NCCL INFO Channel 09/32 : 0 269e1e9e15a4:46:66 [0] NCCL INFO Channel 10/32 : 0 269e1e9e15a4:46:66 [0] NCCL INFO Channel 11/32 : 0 269e1e9e15a4:46:66 [0] NCCL INFO Channel 12/32 : 0 269e1e9e15a4:46:66 [0] NCCL INFO Channel 13/32 : 0 269e1e9e15a4:46:66 [0] NCCL INFO Channel 14/32 : 0 269e1e9e15a4:46:66 [0] NCCL INFO Channel 15/32 : 0 269e1e9e15a4:46:66 [0] NCCL INFO Channel 16/32 : 0 269e1e9e15a4:46:66 [0] NCCL INFO Channel 17/32 : 0 269e1e9e15a4:46:66 [0] NCCL INFO Channel 18/32 : 0 269e1e9e15a4:46:66 [0] NCCL INFO Channel 19/32 : 0 269e1e9e15a4:46:66 [0] NCCL INFO Channel 20/32 : 0 269e1e9e15a4:46:66 [0] NCCL INFO Channel 21/32 : 0 269e1e9e15a4:46:66 [0] NCCL INFO Channel 22/32 : 0 269e1e9e15a4:46:66 [0] NCCL INFO Channel 23/32 : 0 269e1e9e15a4:46:66 [0] NCCL INFO Channel 24/32 : 0 269e1e9e15a4:46:66 [0] NCCL INFO Channel 25/32 : 0 269e1e9e15a4:46:66 [0] NCCL INFO Channel 26/32 : 0 269e1e9e15a4:46:66 [0] NCCL INFO Channel 27/32 : 0 269e1e9e15a4:46:66 [0] NCCL INFO Channel 28/32 : 0 269e1e9e15a4:46:66 [0] NCCL INFO Channel 29/32 : 0 269e1e9e15a4:46:66 [0] NCCL INFO Channel 30/32 : 0 269e1e9e15a4:46:66 [0] NCCL INFO Channel 31/32 : 0 269e1e9e15a4:46:66 [0] NCCL INFO Trees [0] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [1] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [2] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [3] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [4] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [5] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [6] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [7] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [8] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [9] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [10] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [11] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [12] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [13] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [14] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [15] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [16] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [17] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [18] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [19] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [20] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [21] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [22] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [23] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [24] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [25] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [26] -1/-1/-1->0->-1|-1->0->-1/- 269e1e9e15a4:46:66 [0] NCCL INFO 32 coll channels, 32 p2p channels, 32 p2p channels per peer 269e1e9e15a4:46:66 [0] NCCL INFO comm 0x7f2aac385170 rank 0 nranks 1 cudaDev 0 busId 50 - Init COMPLETE 2021-10-19 06:54:24,730 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 1/120: loss: 0.16971 learning rate: 0.00001 Time taken: 0:01:32.538130 ETA: 3:03:32.037475 2021-10-19 06:54:27,276 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 17.562 INFO:tensorflow:epoch = 1.049222797927461, learning_rate = 7.478943e-06, loss = 0.1547409, step = 405 (5.239 sec) 2021-10-19 06:54:28,453 [INFO] tensorflow: epoch = 1.049222797927461, learning_rate = 7.478943e-06, loss = 0.1547409, step = 405 (5.239 sec) INFO:tensorflow:global_step/sec: 4.61755 2021-10-19 06:54:31,040 [INFO] tensorflow: global_step/sec: 4.61755 2021-10-19 06:54:32,231 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.185 INFO:tensorflow:epoch = 1.1191709844559585, learning_rate = 7.682427e-06, loss = 0.1405969, step = 432 (5.392 sec) 2021-10-19 06:54:33,846 [INFO] tensorflow: epoch = 1.1191709844559585, learning_rate = 7.682427e-06, loss = 0.1405969, step = 432 (5.392 sec) 2021-10-19 06:54:37,230 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.004 INFO:tensorflow:global_step/sec: 5.01968 2021-10-19 06:54:38,611 [INFO] tensorflow: global_step/sec: 5.01968 INFO:tensorflow:epoch = 1.189119170984456, learning_rate = 7.891446e-06, loss = 0.13145886, step = 459 (5.354 sec) 2021-10-19 06:54:39,199 [INFO] tensorflow: epoch = 1.189119170984456, learning_rate = 7.891446e-06, loss = 0.13145886, step = 459 (5.354 sec) 2021-10-19 06:54:42,171 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.240 INFO:tensorflow:epoch = 1.2590673575129534, learning_rate = 8.106146e-06, loss = 0.12489269, step = 486 (5.311 sec) 2021-10-19 06:54:44,510 [INFO] tensorflow: epoch = 1.2590673575129534, learning_rate = 8.106146e-06, loss = 0.12489269, step = 486 (5.311 sec) INFO:tensorflow:global_step/sec: 5.07081 2021-10-19 06:54:46,104 [INFO] tensorflow: global_step/sec: 5.07081 2021-10-19 06:54:47,098 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.298 INFO:tensorflow:epoch = 1.3290155440414508, learning_rate = 8.326695e-06, loss = 0.11936016, step = 513 (5.372 sec) 2021-10-19 06:54:49,882 [INFO] tensorflow: epoch = 1.3290155440414508, learning_rate = 8.326695e-06, loss = 0.11936016, step = 513 (5.372 sec) 2021-10-19 06:54:52,082 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.069 INFO:tensorflow:global_step/sec: 4.99443 2021-10-19 06:54:53,713 [INFO] tensorflow: global_step/sec: 4.99443 INFO:tensorflow:epoch = 1.3963730569948187, learning_rate = 8.54474e-06, loss = 0.11529837, step = 539 (5.242 sec) 2021-10-19 06:54:55,124 [INFO] tensorflow: epoch = 1.3963730569948187, learning_rate = 8.54474e-06, loss = 0.11529837, step = 539 (5.242 sec) 2021-10-19 06:54:57,095 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.948 INFO:tensorflow:epoch = 1.4663212435233162, learning_rate = 8.777214e-06, loss = 0.11199291, step = 566 (5.331 sec) 2021-10-19 06:55:00,455 [INFO] tensorflow: epoch = 1.4663212435233162, learning_rate = 8.777214e-06, loss = 0.11199291, step = 566 (5.331 sec) INFO:tensorflow:global_step/sec: 5.04324 2021-10-19 06:55:01,248 [INFO] tensorflow: global_step/sec: 5.04324 2021-10-19 06:55:02,053 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.173 INFO:tensorflow:epoch = 1.5362694300518136, learning_rate = 9.01602e-06, loss = 0.10846, step = 593 (5.379 sec) 2021-10-19 06:55:05,834 [INFO] tensorflow: epoch = 1.5362694300518136, learning_rate = 9.01602e-06, loss = 0.10846, step = 593 (5.379 sec) 2021-10-19 06:55:07,040 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.052 INFO:tensorflow:global_step/sec: 5.01286 2021-10-19 06:55:08,828 [INFO] tensorflow: global_step/sec: 5.01286 INFO:tensorflow:epoch = 1.606217616580311, learning_rate = 9.261315e-06, loss = 0.10542947, step = 620 (5.359 sec) 2021-10-19 06:55:11,194 [INFO] tensorflow: epoch = 1.606217616580311, learning_rate = 9.261315e-06, loss = 0.10542947, step = 620 (5.359 sec) 2021-10-19 06:55:12,001 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.160 INFO:tensorflow:epoch = 1.673575129533679, learning_rate = 9.503844e-06, loss = 0.10227715, step = 646 (5.249 sec) 2021-10-19 06:55:16,443 [INFO] tensorflow: epoch = 1.673575129533679, learning_rate = 9.503844e-06, loss = 0.10227715, step = 646 (5.249 sec) INFO:tensorflow:global_step/sec: 4.98948 2021-10-19 06:55:16,444 [INFO] tensorflow: global_step/sec: 4.98948 2021-10-19 06:55:17,038 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.856 INFO:tensorflow:epoch = 1.7409326424870466, learning_rate = 9.752715e-06, loss = 0.09947312, step = 672 (5.213 sec) 2021-10-19 06:55:21,656 [INFO] tensorflow: epoch = 1.7409326424870466, learning_rate = 9.752715e-06, loss = 0.09947312, step = 672 (5.213 sec) 2021-10-19 06:55:22,051 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.947 INFO:tensorflow:global_step/sec: 4.99032 2021-10-19 06:55:24,059 [INFO] tensorflow: global_step/sec: 4.99032 INFO:tensorflow:epoch = 1.810880829015544, learning_rate = 1.0018054e-05, loss = 0.0970317, step = 699 (5.368 sec) 2021-10-19 06:55:27,024 [INFO] tensorflow: epoch = 1.810880829015544, learning_rate = 1.0018054e-05, loss = 0.0970317, step = 699 (5.368 sec) 2021-10-19 06:55:27,024 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.111 INFO:tensorflow:global_step/sec: 5.01924 2021-10-19 06:55:31,630 [INFO] tensorflow: global_step/sec: 5.01924 2021-10-19 06:55:32,018 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.026 INFO:tensorflow:epoch = 1.8808290155440415, learning_rate = 1.02906215e-05, loss = 0.09458302, step = 726 (5.376 sec) 2021-10-19 06:55:32,400 [INFO] tensorflow: epoch = 1.8808290155440415, learning_rate = 1.02906215e-05, loss = 0.09458302, step = 726 (5.376 sec) 2021-10-19 06:55:37,029 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.960 INFO:tensorflow:epoch = 1.9481865284974094, learning_rate = 1.0560095e-05, loss = 0.09228908, step = 752 (5.262 sec) 2021-10-19 06:55:37,663 [INFO] tensorflow: epoch = 1.9481865284974094, learning_rate = 1.0560095e-05, loss = 0.09228908, step = 752 (5.262 sec) INFO:tensorflow:global_step/sec: 4.98455 2021-10-19 06:55:39,253 [INFO] tensorflow: global_step/sec: 4.98455 2021-10-19 06:55:41,618 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 2/120: loss: 0.09088 learning rate: 0.00001 Time taken: 0:01:16.908342 ETA: 2:31:15.184399 2021-10-19 06:55:42,045 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.937 INFO:tensorflow:epoch = 2.018134715025907, learning_rate = 1.0847399e-05, loss = 0.09023421, step = 779 (5.392 sec) 2021-10-19 06:55:43,054 [INFO] tensorflow: epoch = 2.018134715025907, learning_rate = 1.0847399e-05, loss = 0.09023421, step = 779 (5.392 sec) INFO:tensorflow:global_step/sec: 4.99761 2021-10-19 06:55:46,857 [INFO] tensorflow: global_step/sec: 4.99761 2021-10-19 06:55:47,053 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.968 INFO:tensorflow:epoch = 2.088082901554404, learning_rate = 1.114253e-05, loss = 0.08776415, step = 806 (5.398 sec) 2021-10-19 06:55:48,453 [INFO] tensorflow: epoch = 2.088082901554404, learning_rate = 1.114253e-05, loss = 0.08776415, step = 806 (5.398 sec) 2021-10-19 06:55:52,069 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.940 INFO:tensorflow:epoch = 2.155440414507772, learning_rate = 1.1434312e-05, loss = 0.085671805, step = 832 (5.226 sec) 2021-10-19 06:55:53,678 [INFO] tensorflow: epoch = 2.155440414507772, learning_rate = 1.1434312e-05, loss = 0.085671805, step = 832 (5.226 sec) INFO:tensorflow:global_step/sec: 5.00321 2021-10-19 06:55:54,452 [INFO] tensorflow: global_step/sec: 5.00321 2021-10-19 06:55:56,997 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.296 INFO:tensorflow:epoch = 2.2253886010362693, learning_rate = 1.1745411e-05, loss = 0.084041774, step = 859 (5.291 sec) 2021-10-19 06:55:58,969 [INFO] tensorflow: epoch = 2.2253886010362693, learning_rate = 1.1745411e-05, loss = 0.084041774, step = 859 (5.291 sec) INFO:tensorflow:global_step/sec: 5.09233 2021-10-19 06:56:01,914 [INFO] tensorflow: global_step/sec: 5.09233 2021-10-19 06:56:01,916 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.329 INFO:tensorflow:epoch = 2.295336787564767, learning_rate = 1.2064965e-05, loss = 0.081546985, step = 886 (5.363 sec) 2021-10-19 06:56:04,332 [INFO] tensorflow: epoch = 2.295336787564767, learning_rate = 1.2064965e-05, loss = 0.081546985, step = 886 (5.363 sec) 2021-10-19 06:56:06,890 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.105 INFO:tensorflow:global_step/sec: 5.06621 2021-10-19 06:56:09,415 [INFO] tensorflow: global_step/sec: 5.06621 INFO:tensorflow:epoch = 2.365284974093264, learning_rate = 1.2393222e-05, loss = 0.079302326, step = 913 (5.279 sec) 2021-10-19 06:56:09,611 [INFO] tensorflow: epoch = 2.365284974093264, learning_rate = 1.2393222e-05, loss = 0.079302326, step = 913 (5.279 sec) 2021-10-19 06:56:11,785 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.431 INFO:tensorflow:epoch = 2.435233160621762, learning_rate = 1.2730399e-05, loss = 0.07706892, step = 940 (5.345 sec) 2021-10-19 06:56:14,956 [INFO] tensorflow: epoch = 2.435233160621762, learning_rate = 1.2730399e-05, loss = 0.07706892, step = 940 (5.345 sec) 2021-10-19 06:56:16,760 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.103 INFO:tensorflow:global_step/sec: 5.02656 2021-10-19 06:56:16,975 [INFO] tensorflow: global_step/sec: 5.02656 INFO:tensorflow:epoch = 2.5025906735751295, learning_rate = 1.3063762e-05, loss = 0.07505472, step = 966 (5.278 sec) 2021-10-19 06:56:20,233 [INFO] tensorflow: epoch = 2.5025906735751295, learning_rate = 1.3063762e-05, loss = 0.07505472, step = 966 (5.278 sec) 2021-10-19 06:56:21,870 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.571 INFO:tensorflow:global_step/sec: 4.9527 2021-10-19 06:56:24,647 [INFO] tensorflow: global_step/sec: 4.9527 INFO:tensorflow:epoch = 2.5699481865284977, learning_rate = 1.3405855e-05, loss = 0.073761486, step = 992 (5.202 sec) 2021-10-19 06:56:25,436 [INFO] tensorflow: epoch = 2.5699481865284977, learning_rate = 1.3405855e-05, loss = 0.073761486, step = 992 (5.202 sec) 2021-10-19 06:56:26,825 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.188 INFO:tensorflow:epoch = 2.639896373056995, learning_rate = 1.3770596e-05, loss = 0.07117513, step = 1019 (5.343 sec) 2021-10-19 06:56:30,779 [INFO] tensorflow: epoch = 2.639896373056995, learning_rate = 1.3770596e-05, loss = 0.07117513, step = 1019 (5.343 sec) 2021-10-19 06:56:31,744 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.330 INFO:tensorflow:global_step/sec: 5.06111 2021-10-19 06:56:32,156 [INFO] tensorflow: global_step/sec: 5.06111 INFO:tensorflow:epoch = 2.7098445595854925, learning_rate = 1.4145246e-05, loss = 0.06934603, step = 1046 (5.364 sec) 2021-10-19 06:56:36,143 [INFO] tensorflow: epoch = 2.7098445595854925, learning_rate = 1.4145246e-05, loss = 0.06934603, step = 1046 (5.364 sec) 2021-10-19 06:56:36,726 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.076 INFO:tensorflow:global_step/sec: 5.04407 2021-10-19 06:56:39,689 [INFO] tensorflow: global_step/sec: 5.04407 INFO:tensorflow:epoch = 2.7797927461139897, learning_rate = 1.4530104e-05, loss = 0.06743916, step = 1073 (5.358 sec) 2021-10-19 06:56:41,501 [INFO] tensorflow: epoch = 2.7797927461139897, learning_rate = 1.4530104e-05, loss = 0.06743916, step = 1073 (5.358 sec) 2021-10-19 06:56:41,694 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.129 INFO:tensorflow:epoch = 2.8471502590673574, learning_rate = 1.4910594e-05, loss = 0.065491885, step = 1099 (5.245 sec) 2021-10-19 06:56:46,746 [INFO] tensorflow: epoch = 2.8471502590673574, learning_rate = 1.4910594e-05, loss = 0.065491885, step = 1099 (5.245 sec) 2021-10-19 06:56:46,746 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.795 INFO:tensorflow:global_step/sec: 4.97257 2021-10-19 06:56:47,331 [INFO] tensorflow: global_step/sec: 4.97257 2021-10-19 06:56:51,735 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.047 INFO:tensorflow:epoch = 2.917098445595855, learning_rate = 1.5316276e-05, loss = 0.06381582, step = 1126 (5.418 sec) 2021-10-19 06:56:52,164 [INFO] tensorflow: epoch = 2.917098445595855, learning_rate = 1.5316276e-05, loss = 0.06381582, step = 1126 (5.418 sec) INFO:tensorflow:global_step/sec: 4.99825 2021-10-19 06:56:54,934 [INFO] tensorflow: global_step/sec: 4.99825 2021-10-19 06:56:56,768 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.872 INFO:tensorflow:epoch = 2.9844559585492227, learning_rate = 1.5717353e-05, loss = 0.062154073, step = 1152 (5.198 sec) 2021-10-19 06:56:57,362 [INFO] tensorflow: epoch = 2.9844559585492227, learning_rate = 1.5717353e-05, loss = 0.062154073, step = 1152 (5.198 sec) 2021-10-19 06:56:58,615 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 3/120: loss: 0.06180 learning rate: 0.00002 Time taken: 0:01:16.980294 ETA: 2:30:06.694397 2021-10-19 06:57:01,809 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.839 INFO:tensorflow:epoch = 3.051813471502591, learning_rate = 1.6128932e-05, loss = 0.060345598, step = 1178 (5.311 sec) 2021-10-19 06:57:02,673 [INFO] tensorflow: epoch = 3.051813471502591, learning_rate = 1.6128932e-05, loss = 0.060345598, step = 1178 (5.311 sec) INFO:tensorflow:global_step/sec: 4.90941 2021-10-19 06:57:02,674 [INFO] tensorflow: global_step/sec: 4.90941 2021-10-19 06:57:06,944 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.476 INFO:tensorflow:epoch = 3.1191709844559585, learning_rate = 1.655129e-05, loss = 0.05884333, step = 1204 (5.248 sec) 2021-10-19 06:57:07,921 [INFO] tensorflow: epoch = 3.1191709844559585, learning_rate = 1.655129e-05, loss = 0.05884333, step = 1204 (5.248 sec) INFO:tensorflow:global_step/sec: 4.88886 2021-10-19 06:57:10,447 [INFO] tensorflow: global_step/sec: 4.88886 2021-10-19 06:57:12,098 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.403 INFO:tensorflow:epoch = 3.1865284974093266, learning_rate = 1.6984708e-05, loss = 0.057229508, step = 1230 (5.393 sec) 2021-10-19 06:57:13,314 [INFO] tensorflow: epoch = 3.1865284974093266, learning_rate = 1.6984708e-05, loss = 0.057229508, step = 1230 (5.393 sec) 2021-10-19 06:57:17,126 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.888 INFO:tensorflow:global_step/sec: 4.94541 2021-10-19 06:57:18,131 [INFO] tensorflow: global_step/sec: 4.94541 INFO:tensorflow:epoch = 3.2538860103626943, learning_rate = 1.7429475e-05, loss = 0.055639364, step = 1256 (5.234 sec) 2021-10-19 06:57:18,549 [INFO] tensorflow: epoch = 3.2538860103626943, learning_rate = 1.7429475e-05, loss = 0.055639364, step = 1256 (5.234 sec) 2021-10-19 06:57:22,319 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.262 INFO:tensorflow:epoch = 3.318652849740933, learning_rate = 1.7868106e-05, loss = 0.05422652, step = 1281 (5.259 sec) 2021-10-19 06:57:23,808 [INFO] tensorflow: epoch = 3.318652849740933, learning_rate = 1.7868106e-05, loss = 0.05422652, step = 1281 (5.259 sec) INFO:tensorflow:global_step/sec: 4.77364 2021-10-19 06:57:26,091 [INFO] tensorflow: global_step/sec: 4.77364 2021-10-19 06:57:27,596 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.952 INFO:tensorflow:epoch = 3.383419689119171, learning_rate = 1.8317794e-05, loss = 0.052687313, step = 1306 (5.266 sec) 2021-10-19 06:57:29,074 [INFO] tensorflow: epoch = 3.383419689119171, learning_rate = 1.8317794e-05, loss = 0.052687313, step = 1306 (5.266 sec) 2021-10-19 06:57:32,709 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.558 INFO:tensorflow:global_step/sec: 4.86467 2021-10-19 06:57:33,903 [INFO] tensorflow: global_step/sec: 4.86467 INFO:tensorflow:epoch = 3.450777202072539, learning_rate = 1.879747e-05, loss = 0.051112484, step = 1332 (5.235 sec) 2021-10-19 06:57:34,309 [INFO] tensorflow: epoch = 3.450777202072539, learning_rate = 1.879747e-05, loss = 0.051112484, step = 1332 (5.235 sec) 2021-10-19 06:57:37,794 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.666 INFO:tensorflow:epoch = 3.518134715025907, learning_rate = 1.9289706e-05, loss = 0.049741387, step = 1358 (5.355 sec) 2021-10-19 06:57:39,664 [INFO] tensorflow: epoch = 3.518134715025907, learning_rate = 1.9289706e-05, loss = 0.049741387, step = 1358 (5.355 sec) INFO:tensorflow:global_step/sec: 4.89647 2021-10-19 06:57:41,663 [INFO] tensorflow: global_step/sec: 4.89647 2021-10-19 06:57:42,911 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.544 INFO:tensorflow:epoch = 3.585492227979275, learning_rate = 1.9794834e-05, loss = 0.048217226, step = 1384 (5.311 sec) 2021-10-19 06:57:44,975 [INFO] tensorflow: epoch = 3.585492227979275, learning_rate = 1.9794834e-05, loss = 0.048217226, step = 1384 (5.311 sec) 2021-10-19 06:57:47,971 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.767 INFO:tensorflow:global_step/sec: 4.92137 2021-10-19 06:57:49,385 [INFO] tensorflow: global_step/sec: 4.92137 INFO:tensorflow:epoch = 3.6528497409326426, learning_rate = 2.0313188e-05, loss = 0.046960805, step = 1410 (5.208 sec) 2021-10-19 06:57:50,183 [INFO] tensorflow: epoch = 3.6528497409326426, learning_rate = 2.0313188e-05, loss = 0.046960805, step = 1410 (5.208 sec) 2021-10-19 06:57:52,982 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.958 INFO:tensorflow:epoch = 3.7227979274611402, learning_rate = 2.086586e-05, loss = 0.045442328, step = 1437 (5.347 sec) 2021-10-19 06:57:55,530 [INFO] tensorflow: epoch = 3.7227979274611402, learning_rate = 2.086586e-05, loss = 0.045442328, step = 1437 (5.347 sec) INFO:tensorflow:global_step/sec: 5.05861 2021-10-19 06:57:56,897 [INFO] tensorflow: global_step/sec: 5.05861 2021-10-19 06:57:57,907 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.304 INFO:tensorflow:epoch = 3.7927461139896375, learning_rate = 2.143357e-05, loss = 0.04400702, step = 1464 (5.353 sec) 2021-10-19 06:58:00,883 [INFO] tensorflow: epoch = 3.7927461139896375, learning_rate = 2.143357e-05, loss = 0.04400702, step = 1464 (5.353 sec) 2021-10-19 06:58:02,852 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.223 INFO:tensorflow:global_step/sec: 4.98275 2021-10-19 06:58:04,523 [INFO] tensorflow: global_step/sec: 4.98275 INFO:tensorflow:epoch = 3.860103626943005, learning_rate = 2.1994838e-05, loss = 0.042781617, step = 1490 (5.212 sec) 2021-10-19 06:58:06,095 [INFO] tensorflow: epoch = 3.860103626943005, learning_rate = 2.1994838e-05, loss = 0.042781617, step = 1490 (5.212 sec) 2021-10-19 06:58:07,858 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.979 INFO:tensorflow:epoch = 3.930051813471503, learning_rate = 2.2593242e-05, loss = 0.041465003, step = 1517 (5.387 sec) 2021-10-19 06:58:11,482 [INFO] tensorflow: epoch = 3.930051813471503, learning_rate = 2.2593242e-05, loss = 0.041465003, step = 1517 (5.387 sec) INFO:tensorflow:global_step/sec: 5.0289 2021-10-19 06:58:12,079 [INFO] tensorflow: global_step/sec: 5.0289 2021-10-19 06:58:12,882 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.908 INFO:tensorflow:epoch = 4.0, learning_rate = 2.320795e-05, loss = 0.039924562, step = 1544 (5.307 sec) 2021-10-19 06:58:16,789 [INFO] tensorflow: epoch = 4.0, learning_rate = 2.320795e-05, loss = 0.039924562, step = 1544 (5.307 sec) 2021-10-19 06:58:16,790 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 4/120: loss: 0.03992 learning rate: 0.00002 Time taken: 0:01:18.184980 ETA: 2:31:09.457643 2021-10-19 06:58:17,756 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.517 INFO:tensorflow:global_step/sec: 5.07027 2021-10-19 06:58:19,574 [INFO] tensorflow: global_step/sec: 5.07027 INFO:tensorflow:epoch = 4.069948186528498, learning_rate = 2.3839357e-05, loss = 0.038698208, step = 1571 (5.402 sec) 2021-10-19 06:58:22,191 [INFO] tensorflow: epoch = 4.069948186528498, learning_rate = 2.3839357e-05, loss = 0.038698208, step = 1571 (5.402 sec) 2021-10-19 06:58:22,779 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.911 INFO:tensorflow:global_step/sec: 5.00482 2021-10-19 06:58:27,167 [INFO] tensorflow: global_step/sec: 5.00482 INFO:tensorflow:epoch = 4.139896373056995, learning_rate = 2.448797e-05, loss = 0.037318908, step = 1598 (5.387 sec) 2021-10-19 06:58:27,578 [INFO] tensorflow: epoch = 4.139896373056995, learning_rate = 2.448797e-05, loss = 0.037318908, step = 1598 (5.387 sec) 2021-10-19 06:58:27,769 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.040 INFO:tensorflow:epoch = 4.2072538860103625, learning_rate = 2.512922e-05, loss = 0.03612631, step = 1624 (5.218 sec) 2021-10-19 06:58:32,796 [INFO] tensorflow: epoch = 4.2072538860103625, learning_rate = 2.512922e-05, loss = 0.03612631, step = 1624 (5.218 sec) 2021-10-19 06:58:32,797 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.891 INFO:tensorflow:global_step/sec: 4.99743 2021-10-19 06:58:34,770 [INFO] tensorflow: global_step/sec: 4.99743 2021-10-19 06:58:37,741 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.227 INFO:tensorflow:epoch = 4.27720207253886, learning_rate = 2.58129e-05, loss = 0.034910344, step = 1651 (5.329 sec) 2021-10-19 06:58:38,126 [INFO] tensorflow: epoch = 4.27720207253886, learning_rate = 2.58129e-05, loss = 0.034910344, step = 1651 (5.329 sec) INFO:tensorflow:global_step/sec: 5.05663 2021-10-19 06:58:42,285 [INFO] tensorflow: global_step/sec: 5.05663 2021-10-19 06:58:42,671 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.290 INFO:tensorflow:epoch = 4.347150259067358, learning_rate = 2.6515207e-05, loss = 0.033787776, step = 1678 (5.319 sec) 2021-10-19 06:58:43,445 [INFO] tensorflow: epoch = 4.347150259067358, learning_rate = 2.6515207e-05, loss = 0.033787776, step = 1678 (5.319 sec) 2021-10-19 06:58:47,679 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.967 INFO:tensorflow:epoch = 4.414507772020726, learning_rate = 2.7209542e-05, loss = 0.032628387, step = 1704 (5.208 sec) 2021-10-19 06:58:48,653 [INFO] tensorflow: epoch = 4.414507772020726, learning_rate = 2.7209542e-05, loss = 0.032628387, step = 1704 (5.208 sec) INFO:tensorflow:global_step/sec: 5.04009 2021-10-19 06:58:49,825 [INFO] tensorflow: global_step/sec: 5.04009 2021-10-19 06:58:52,699 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.924 INFO:tensorflow:epoch = 4.481865284974093, learning_rate = 2.792206e-05, loss = 0.031525224, step = 1730 (5.221 sec) 2021-10-19 06:58:53,874 [INFO] tensorflow: epoch = 4.481865284974093, learning_rate = 2.792206e-05, loss = 0.031525224, step = 1730 (5.221 sec) INFO:tensorflow:global_step/sec: 4.96824 2021-10-19 06:58:57,474 [INFO] tensorflow: global_step/sec: 4.96824 2021-10-19 06:58:57,673 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.107 INFO:tensorflow:epoch = 4.551813471502591, learning_rate = 2.8681752e-05, loss = 0.030504895, step = 1757 (5.390 sec) 2021-10-19 06:58:59,264 [INFO] tensorflow: epoch = 4.551813471502591, learning_rate = 2.8681752e-05, loss = 0.030504895, step = 1757 (5.390 sec) 2021-10-19 06:59:02,642 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.127 INFO:tensorflow:epoch = 4.6217616580310885, learning_rate = 2.9462084e-05, loss = 0.02933241, step = 1784 (5.354 sec) 2021-10-19 06:59:04,618 [INFO] tensorflow: epoch = 4.6217616580310885, learning_rate = 2.9462084e-05, loss = 0.02933241, step = 1784 (5.354 sec) INFO:tensorflow:global_step/sec: 5.03876 2021-10-19 06:59:05,015 [INFO] tensorflow: global_step/sec: 5.03876 2021-10-19 06:59:07,600 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.171 INFO:tensorflow:epoch = 4.691709844559585, learning_rate = 3.0263676e-05, loss = 0.02830446, step = 1811 (5.375 sec) 2021-10-19 06:59:09,993 [INFO] tensorflow: epoch = 4.691709844559585, learning_rate = 3.0263676e-05, loss = 0.02830446, step = 1811 (5.375 sec) INFO:tensorflow:global_step/sec: 5.01289 2021-10-19 06:59:12,596 [INFO] tensorflow: global_step/sec: 5.01289 2021-10-19 06:59:12,597 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.014 INFO:tensorflow:epoch = 4.761658031088083, learning_rate = 3.108705e-05, loss = 0.027272003, step = 1838 (5.355 sec) 2021-10-19 06:59:15,349 [INFO] tensorflow: epoch = 4.761658031088083, learning_rate = 3.108705e-05, loss = 0.027272003, step = 1838 (5.355 sec) 2021-10-19 06:59:17,519 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.319 INFO:tensorflow:global_step/sec: 5.03694 2021-10-19 06:59:20,140 [INFO] tensorflow: global_step/sec: 5.03694 INFO:tensorflow:epoch = 4.831606217616581, learning_rate = 3.193285e-05, loss = 0.026265508, step = 1865 (5.391 sec) 2021-10-19 06:59:20,740 [INFO] tensorflow: epoch = 4.831606217616581, learning_rate = 3.193285e-05, loss = 0.026265508, step = 1865 (5.391 sec) 2021-10-19 06:59:22,513 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.024 INFO:tensorflow:epoch = 4.898963730569949, learning_rate = 3.2769054e-05, loss = 0.02539639, step = 1891 (5.226 sec) 2021-10-19 06:59:25,966 [INFO] tensorflow: epoch = 4.898963730569949, learning_rate = 3.2769054e-05, loss = 0.02539639, step = 1891 (5.226 sec) 2021-10-19 06:59:27,568 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.786 INFO:tensorflow:global_step/sec: 4.98666 2021-10-19 06:59:27,760 [INFO] tensorflow: global_step/sec: 4.98666 INFO:tensorflow:epoch = 4.966321243523316, learning_rate = 3.3627155e-05, loss = 0.024417646, step = 1917 (5.220 sec) 2021-10-19 06:59:31,185 [INFO] tensorflow: epoch = 4.966321243523316, learning_rate = 3.3627155e-05, loss = 0.024417646, step = 1917 (5.220 sec) 2021-10-19 06:59:32,625 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.777 2021-10-19 06:59:33,827 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 5/120: loss: 0.02392 learning rate: 0.00003 Time taken: 0:01:17.030944 ETA: 2:27:38.558518 INFO:tensorflow:global_step/sec: 4.97264 2021-10-19 06:59:35,402 [INFO] tensorflow: global_step/sec: 4.97264 INFO:tensorflow:epoch = 5.036269430051814, learning_rate = 3.4542067e-05, loss = 0.023508586, step = 1944 (5.389 sec) 2021-10-19 06:59:36,574 [INFO] tensorflow: epoch = 5.036269430051814, learning_rate = 3.4542067e-05, loss = 0.023508586, step = 1944 (5.389 sec) 2021-10-19 06:59:37,588 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.150 INFO:tensorflow:epoch = 5.106217616580311, learning_rate = 3.548184e-05, loss = 0.022569124, step = 1971 (5.296 sec) 2021-10-19 06:59:41,870 [INFO] tensorflow: epoch = 5.106217616580311, learning_rate = 3.548184e-05, loss = 0.022569124, step = 1971 (5.296 sec) 2021-10-19 06:59:42,487 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.418 INFO:tensorflow:global_step/sec: 5.04967 2021-10-19 06:59:42,927 [INFO] tensorflow: global_step/sec: 5.04967 INFO:tensorflow:epoch = 5.173575129533679, learning_rate = 3.641098e-05, loss = 0.021747159, step = 1997 (5.266 sec) 2021-10-19 06:59:47,137 [INFO] tensorflow: epoch = 5.173575129533679, learning_rate = 3.641098e-05, loss = 0.021747159, step = 1997 (5.266 sec) 2021-10-19 06:59:47,542 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.782 INFO:tensorflow:global_step/sec: 5.02412 2021-10-19 06:59:50,491 [INFO] tensorflow: global_step/sec: 5.02412 INFO:tensorflow:epoch = 5.243523316062176, learning_rate = 3.7401634e-05, loss = 0.020962918, step = 2024 (5.367 sec) 2021-10-19 06:59:52,504 [INFO] tensorflow: epoch = 5.243523316062176, learning_rate = 3.7401634e-05, loss = 0.020962918, step = 2024 (5.367 sec) 2021-10-19 06:59:52,505 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.154 2021-10-19 06:59:57,480 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.101 INFO:tensorflow:epoch = 5.313471502590674, learning_rate = 3.841924e-05, loss = 0.020130122, step = 2051 (5.380 sec) 2021-10-19 06:59:57,884 [INFO] tensorflow: epoch = 5.313471502590674, learning_rate = 3.841924e-05, loss = 0.020130122, step = 2051 (5.380 sec) INFO:tensorflow:global_step/sec: 5.00533 2021-10-19 06:59:58,083 [INFO] tensorflow: global_step/sec: 5.00533 2021-10-19 07:00:02,456 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.099 INFO:tensorflow:epoch = 5.3834196891191715, learning_rate = 3.94645e-05, loss = 0.019389708, step = 2078 (5.386 sec) 2021-10-19 07:00:03,270 [INFO] tensorflow: epoch = 5.3834196891191715, learning_rate = 3.94645e-05, loss = 0.019389708, step = 2078 (5.386 sec) INFO:tensorflow:global_step/sec: 5.01407 2021-10-19 07:00:05,661 [INFO] tensorflow: global_step/sec: 5.01407 2021-10-19 07:00:07,443 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.052 INFO:tensorflow:epoch = 5.453367875647668, learning_rate = 4.053819e-05, loss = 0.018520817, step = 2105 (5.378 sec) 2021-10-19 07:00:08,648 [INFO] tensorflow: epoch = 5.453367875647668, learning_rate = 4.053819e-05, loss = 0.018520817, step = 2105 (5.378 sec) 2021-10-19 07:00:12,395 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.197 INFO:tensorflow:global_step/sec: 5.06464 2021-10-19 07:00:13,164 [INFO] tensorflow: global_step/sec: 5.06464 INFO:tensorflow:epoch = 5.523316062176166, learning_rate = 4.164114e-05, loss = 0.017780636, step = 2132 (5.317 sec) 2021-10-19 07:00:13,965 [INFO] tensorflow: epoch = 5.523316062176166, learning_rate = 4.164114e-05, loss = 0.017780636, step = 2132 (5.317 sec) 2021-10-19 07:00:17,310 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.351 INFO:tensorflow:epoch = 5.5932642487046635, learning_rate = 4.2774092e-05, loss = 0.017069062, step = 2159 (5.328 sec) 2021-10-19 07:00:19,293 [INFO] tensorflow: epoch = 5.5932642487046635, learning_rate = 4.2774092e-05, loss = 0.017069062, step = 2159 (5.328 sec) INFO:tensorflow:global_step/sec: 5.0684 2021-10-19 07:00:20,662 [INFO] tensorflow: global_step/sec: 5.0684 2021-10-19 07:00:22,278 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.129 INFO:tensorflow:epoch = 5.660621761658032, learning_rate = 4.3894193e-05, loss = 0.016424252, step = 2185 (5.236 sec) 2021-10-19 07:00:24,529 [INFO] tensorflow: epoch = 5.660621761658032, learning_rate = 4.3894193e-05, loss = 0.016424252, step = 2185 (5.236 sec) 2021-10-19 07:00:27,266 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.049 INFO:tensorflow:global_step/sec: 5.01031 2021-10-19 07:00:28,246 [INFO] tensorflow: global_step/sec: 5.01031 INFO:tensorflow:epoch = 5.730569948186528, learning_rate = 4.50884e-05, loss = 0.015731791, step = 2212 (5.364 sec) 2021-10-19 07:00:29,894 [INFO] tensorflow: epoch = 5.730569948186528, learning_rate = 4.50884e-05, loss = 0.015731791, step = 2212 (5.364 sec) 2021-10-19 07:00:32,401 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.475 INFO:tensorflow:epoch = 5.7979274611398965, learning_rate = 4.62691e-05, loss = 0.015101284, step = 2238 (5.276 sec) 2021-10-19 07:00:35,169 [INFO] tensorflow: epoch = 5.7979274611398965, learning_rate = 4.62691e-05, loss = 0.015101284, step = 2238 (5.276 sec) INFO:tensorflow:global_step/sec: 4.9271 2021-10-19 07:00:35,958 [INFO] tensorflow: global_step/sec: 4.9271 2021-10-19 07:00:37,380 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.086 INFO:tensorflow:epoch = 5.867875647668394, learning_rate = 4.7527974e-05, loss = 0.014490562, step = 2265 (5.368 sec) 2021-10-19 07:00:40,537 [INFO] tensorflow: epoch = 5.867875647668394, learning_rate = 4.7527974e-05, loss = 0.014490562, step = 2265 (5.368 sec) 2021-10-19 07:00:42,395 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.942 INFO:tensorflow:global_step/sec: 4.99292 2021-10-19 07:00:43,569 [INFO] tensorflow: global_step/sec: 4.99292 INFO:tensorflow:epoch = 5.935233160621762, learning_rate = 4.8772556e-05, loss = 0.013897242, step = 2291 (5.252 sec) 2021-10-19 07:00:45,790 [INFO] tensorflow: epoch = 5.935233160621762, learning_rate = 4.8772556e-05, loss = 0.013897242, step = 2291 (5.252 sec) 2021-10-19 07:00:47,379 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.068 2021-10-19 07:00:50,807 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 6/120: loss: 0.01335 learning rate: 0.00005 Time taken: 0:01:16.983776 ETA: 2:26:16.150447 INFO:tensorflow:epoch = 6.0025906735751295, learning_rate = 5.004973e-05, loss = 0.01327842, step = 2317 (5.210 sec) 2021-10-19 07:00:50,999 [INFO] tensorflow: epoch = 6.0025906735751295, learning_rate = 5.004973e-05, loss = 0.01327842, step = 2317 (5.210 sec) INFO:tensorflow:global_step/sec: 4.98173 2021-10-19 07:00:51,197 [INFO] tensorflow: global_step/sec: 4.98173 2021-10-19 07:00:52,405 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.896 INFO:tensorflow:epoch = 6.069948186528498, learning_rate = 5.1360348e-05, loss = 0.0127774505, step = 2343 (5.201 sec) 2021-10-19 07:00:56,201 [INFO] tensorflow: epoch = 6.069948186528498, learning_rate = 5.1360348e-05, loss = 0.0127774505, step = 2343 (5.201 sec) 2021-10-19 07:00:57,389 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.068 INFO:tensorflow:global_step/sec: 5.0258 2021-10-19 07:00:58,758 [INFO] tensorflow: global_step/sec: 5.0258 INFO:tensorflow:epoch = 6.139896373056995, learning_rate = 5.275774e-05, loss = 0.012257849, step = 2370 (5.358 sec) 2021-10-19 07:01:01,558 [INFO] tensorflow: epoch = 6.139896373056995, learning_rate = 5.275774e-05, loss = 0.012257849, step = 2370 (5.358 sec) 2021-10-19 07:01:02,336 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.215 INFO:tensorflow:global_step/sec: 5.04016 2021-10-19 07:01:06,298 [INFO] tensorflow: global_step/sec: 5.04016 INFO:tensorflow:epoch = 6.209844559585492, learning_rate = 5.4193097e-05, loss = 0.011708287, step = 2397 (5.333 sec) 2021-10-19 07:01:06,891 [INFO] tensorflow: epoch = 6.209844559585492, learning_rate = 5.4193097e-05, loss = 0.011708287, step = 2397 (5.333 sec) 2021-10-19 07:01:07,275 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.251 INFO:tensorflow:epoch = 6.27720207253886, learning_rate = 5.5612214e-05, loss = 0.011171993, step = 2423 (5.233 sec) 2021-10-19 07:01:12,124 [INFO] tensorflow: epoch = 6.27720207253886, learning_rate = 5.5612214e-05, loss = 0.011171993, step = 2423 (5.233 sec) 2021-10-19 07:01:12,321 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.817 INFO:tensorflow:global_step/sec: 5.00372 2021-10-19 07:01:13,892 [INFO] tensorflow: global_step/sec: 5.00372 2021-10-19 07:01:17,291 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.124 INFO:tensorflow:epoch = 6.347150259067358, learning_rate = 5.712529e-05, loss = 0.010688008, step = 2450 (5.362 sec) 2021-10-19 07:01:17,486 [INFO] tensorflow: epoch = 6.347150259067358, learning_rate = 5.712529e-05, loss = 0.010688008, step = 2450 (5.362 sec) INFO:tensorflow:global_step/sec: 5.01152 2021-10-19 07:01:21,474 [INFO] tensorflow: global_step/sec: 5.01152 2021-10-19 07:01:22,274 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.068 INFO:tensorflow:epoch = 6.4170984455958555, learning_rate = 5.8679532e-05, loss = 0.010189653, step = 2477 (5.378 sec) 2021-10-19 07:01:22,864 [INFO] tensorflow: epoch = 6.4170984455958555, learning_rate = 5.8679532e-05, loss = 0.010189653, step = 2477 (5.378 sec) 2021-10-19 07:01:27,231 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.175 INFO:tensorflow:epoch = 6.487046632124352, learning_rate = 6.0275997e-05, loss = 0.009768186, step = 2504 (5.386 sec) 2021-10-19 07:01:28,250 [INFO] tensorflow: epoch = 6.487046632124352, learning_rate = 6.0275997e-05, loss = 0.009768186, step = 2504 (5.386 sec) INFO:tensorflow:global_step/sec: 5.02605 2021-10-19 07:01:29,035 [INFO] tensorflow: global_step/sec: 5.02605 2021-10-19 07:01:32,205 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.109 INFO:tensorflow:epoch = 6.55699481865285, learning_rate = 6.1915904e-05, loss = 0.0093254, step = 2531 (5.342 sec) 2021-10-19 07:01:33,592 [INFO] tensorflow: epoch = 6.55699481865285, learning_rate = 6.1915904e-05, loss = 0.0093254, step = 2531 (5.342 sec) INFO:tensorflow:global_step/sec: 5.04754 2021-10-19 07:01:36,563 [INFO] tensorflow: global_step/sec: 5.04754 2021-10-19 07:01:37,146 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.240 INFO:tensorflow:epoch = 6.626943005181348, learning_rate = 6.3600484e-05, loss = 0.0089109745, step = 2558 (5.312 sec) 2021-10-19 07:01:38,904 [INFO] tensorflow: epoch = 6.626943005181348, learning_rate = 6.3600484e-05, loss = 0.0089109745, step = 2558 (5.312 sec) 2021-10-19 07:01:42,099 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.189 INFO:tensorflow:global_step/sec: 5.049 2021-10-19 07:01:44,090 [INFO] tensorflow: global_step/sec: 5.049 INFO:tensorflow:epoch = 6.696891191709844, learning_rate = 6.5330896e-05, loss = 0.008506472, step = 2585 (5.384 sec) 2021-10-19 07:01:44,288 [INFO] tensorflow: epoch = 6.696891191709844, learning_rate = 6.5330896e-05, loss = 0.008506472, step = 2585 (5.384 sec) 2021-10-19 07:01:47,091 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.037 INFO:tensorflow:epoch = 6.766839378238342, learning_rate = 6.71084e-05, loss = 0.008140364, step = 2612 (5.338 sec) 2021-10-19 07:01:49,625 [INFO] tensorflow: epoch = 6.766839378238342, learning_rate = 6.71084e-05, loss = 0.008140364, step = 2612 (5.338 sec) INFO:tensorflow:global_step/sec: 5.05262 2021-10-19 07:01:51,611 [INFO] tensorflow: global_step/sec: 5.05262 2021-10-19 07:01:52,027 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.260 INFO:tensorflow:epoch = 6.83678756476684, learning_rate = 6.893418e-05, loss = 0.007705061, step = 2639 (5.362 sec) 2021-10-19 07:01:54,987 [INFO] tensorflow: epoch = 6.83678756476684, learning_rate = 6.893418e-05, loss = 0.007705061, step = 2639 (5.362 sec) 2021-10-19 07:01:57,006 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.085 INFO:tensorflow:global_step/sec: 5.0255 2021-10-19 07:01:59,172 [INFO] tensorflow: global_step/sec: 5.0255 INFO:tensorflow:epoch = 6.906735751295337, learning_rate = 7.080965e-05, loss = 0.007372619, step = 2666 (5.358 sec) 2021-10-19 07:02:00,345 [INFO] tensorflow: epoch = 6.906735751295337, learning_rate = 7.080965e-05, loss = 0.007372619, step = 2666 (5.358 sec) 2021-10-19 07:02:01,939 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.273 INFO:tensorflow:epoch = 6.976683937823834, learning_rate = 7.2736206e-05, loss = 0.0070545594, step = 2693 (5.402 sec) 2021-10-19 07:02:05,747 [INFO] tensorflow: epoch = 6.976683937823834, learning_rate = 7.2736206e-05, loss = 0.0070545594, step = 2693 (5.402 sec) INFO:tensorflow:global_step/sec: 5.00375 2021-10-19 07:02:06,766 [INFO] tensorflow: global_step/sec: 5.00375 2021-10-19 07:02:06,970 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.881 2021-10-19 07:02:07,556 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 7/120: loss: 0.00699 learning rate: 0.00007 Time taken: 0:01:16.750989 ETA: 2:24:32.861779 INFO:tensorflow:epoch = 7.044041450777202, learning_rate = 7.4640906e-05, loss = 0.0067612454, step = 2719 (5.230 sec) 2021-10-19 07:02:10,977 [INFO] tensorflow: epoch = 7.044041450777202, learning_rate = 7.4640906e-05, loss = 0.0067612454, step = 2719 (5.230 sec) 2021-10-19 07:02:11,957 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.052 INFO:tensorflow:global_step/sec: 5.01459 2021-10-19 07:02:14,344 [INFO] tensorflow: global_step/sec: 5.01459 INFO:tensorflow:epoch = 7.1139896373057, learning_rate = 7.66717e-05, loss = 0.006414545, step = 2746 (5.337 sec) 2021-10-19 07:02:16,313 [INFO] tensorflow: epoch = 7.1139896373057, learning_rate = 7.66717e-05, loss = 0.006414545, step = 2746 (5.337 sec) 2021-10-19 07:02:16,904 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.217 INFO:tensorflow:epoch = 7.181347150259068, learning_rate = 7.8679455e-05, loss = 0.0061101564, step = 2772 (5.235 sec) 2021-10-19 07:02:21,548 [INFO] tensorflow: epoch = 7.181347150259068, learning_rate = 7.8679455e-05, loss = 0.0061101564, step = 2772 (5.235 sec) INFO:tensorflow:global_step/sec: 4.99312 2021-10-19 07:02:21,955 [INFO] tensorflow: global_step/sec: 4.99312 2021-10-19 07:02:21,956 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.795 INFO:tensorflow:epoch = 7.251295336787565, learning_rate = 8.082005e-05, loss = 0.00587021, step = 2799 (5.381 sec) 2021-10-19 07:02:26,929 [INFO] tensorflow: epoch = 7.251295336787565, learning_rate = 8.082005e-05, loss = 0.00587021, step = 2799 (5.381 sec) 2021-10-19 07:02:26,929 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.109 INFO:tensorflow:global_step/sec: 5.00539 2021-10-19 07:02:29,547 [INFO] tensorflow: global_step/sec: 5.00539 2021-10-19 07:02:31,959 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.886 INFO:tensorflow:epoch = 7.318652849740933, learning_rate = 8.293643e-05, loss = 0.005589033, step = 2825 (5.239 sec) 2021-10-19 07:02:32,168 [INFO] tensorflow: epoch = 7.318652849740933, learning_rate = 8.293643e-05, loss = 0.005589033, step = 2825 (5.239 sec) 2021-10-19 07:02:36,941 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.072 INFO:tensorflow:global_step/sec: 5.00692 2021-10-19 07:02:37,136 [INFO] tensorflow: global_step/sec: 5.00692 INFO:tensorflow:epoch = 7.3886010362694305, learning_rate = 8.519293e-05, loss = 0.005293322, step = 2852 (5.357 sec) 2021-10-19 07:02:37,525 [INFO] tensorflow: epoch = 7.3886010362694305, learning_rate = 8.519293e-05, loss = 0.005293322, step = 2852 (5.357 sec) 2021-10-19 07:02:41,917 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.099 INFO:tensorflow:epoch = 7.458549222797927, learning_rate = 8.751082e-05, loss = 0.0050536036, step = 2879 (5.419 sec) 2021-10-19 07:02:42,944 [INFO] tensorflow: epoch = 7.458549222797927, learning_rate = 8.751082e-05, loss = 0.0050536036, step = 2879 (5.419 sec) INFO:tensorflow:global_step/sec: 5.03242 2021-10-19 07:02:44,687 [INFO] tensorflow: global_step/sec: 5.03242 2021-10-19 07:02:46,847 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.287 INFO:tensorflow:epoch = 7.528497409326425, learning_rate = 8.989169e-05, loss = 0.004821262, step = 2906 (5.285 sec) 2021-10-19 07:02:48,229 [INFO] tensorflow: epoch = 7.528497409326425, learning_rate = 8.989169e-05, loss = 0.004821262, step = 2906 (5.285 sec) 2021-10-19 07:02:51,764 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.339 INFO:tensorflow:global_step/sec: 5.0814 2021-10-19 07:02:52,165 [INFO] tensorflow: global_step/sec: 5.0814 INFO:tensorflow:epoch = 7.598445595854923, learning_rate = 9.2337425e-05, loss = 0.004589538, step = 2933 (5.295 sec) 2021-10-19 07:02:53,524 [INFO] tensorflow: epoch = 7.598445595854923, learning_rate = 9.2337425e-05, loss = 0.004589538, step = 2933 (5.295 sec) 2021-10-19 07:02:56,713 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.207 INFO:tensorflow:epoch = 7.66839378238342, learning_rate = 9.4849616e-05, loss = 0.0043871314, step = 2960 (5.335 sec) 2021-10-19 07:02:58,860 [INFO] tensorflow: epoch = 7.66839378238342, learning_rate = 9.4849616e-05, loss = 0.0043871314, step = 2960 (5.335 sec) INFO:tensorflow:global_step/sec: 5.07066 2021-10-19 07:02:59,659 [INFO] tensorflow: global_step/sec: 5.07066 2021-10-19 07:03:01,656 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.234 INFO:tensorflow:epoch = 7.738341968911917, learning_rate = 9.743024e-05, loss = 0.004187956, step = 2987 (5.349 sec) 2021-10-19 07:03:04,209 [INFO] tensorflow: epoch = 7.738341968911917, learning_rate = 9.743024e-05, loss = 0.004187956, step = 2987 (5.349 sec) 2021-10-19 07:03:06,646 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.041 INFO:tensorflow:global_step/sec: 5.00878 2021-10-19 07:03:07,246 [INFO] tensorflow: global_step/sec: 5.00878 INFO:tensorflow:epoch = 7.805699481865285, learning_rate = 9.998158e-05, loss = 0.0039614984, step = 3013 (5.191 sec) 2021-10-19 07:03:09,400 [INFO] tensorflow: epoch = 7.805699481865285, learning_rate = 9.998158e-05, loss = 0.0039614984, step = 3013 (5.191 sec) 2021-10-19 07:03:11,617 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.121 INFO:tensorflow:epoch = 7.875647668393783, learning_rate = 0.00010270175, loss = 0.0038192968, step = 3040 (5.368 sec) 2021-10-19 07:03:14,768 [INFO] tensorflow: epoch = 7.875647668393783, learning_rate = 0.00010270175, loss = 0.0038192968, step = 3040 (5.368 sec) INFO:tensorflow:global_step/sec: 5.05102 2021-10-19 07:03:14,769 [INFO] tensorflow: global_step/sec: 5.05102 2021-10-19 07:03:16,614 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.013 INFO:tensorflow:epoch = 7.943005181347151, learning_rate = 0.000105391126, loss = 0.0036421996, step = 3066 (5.225 sec) 2021-10-19 07:03:19,993 [INFO] tensorflow: epoch = 7.943005181347151, learning_rate = 0.000105391126, loss = 0.0036421996, step = 3066 (5.225 sec) 2021-10-19 07:03:21,554 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.245 INFO:tensorflow:global_step/sec: 4.9969 2021-10-19 07:03:22,374 [INFO] tensorflow: global_step/sec: 4.9969 2021-10-19 07:03:24,374 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 8/120: loss: 0.00348 learning rate: 0.00011 Time taken: 0:01:16.815183 ETA: 2:23:23.300541 INFO:tensorflow:epoch = 8.012953367875648, learning_rate = 0.00010825855, loss = 0.0034324771, step = 3093 (5.364 sec) 2021-10-19 07:03:25,357 [INFO] tensorflow: epoch = 8.012953367875648, learning_rate = 0.00010825855, loss = 0.0034324771, step = 3093 (5.364 sec) 2021-10-19 07:03:26,540 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.059 INFO:tensorflow:global_step/sec: 5.03833 2021-10-19 07:03:29,916 [INFO] tensorflow: global_step/sec: 5.03833 INFO:tensorflow:epoch = 8.082901554404145, learning_rate = 0.00011120391, loss = 0.003319085, step = 3120 (5.394 sec) 2021-10-19 07:03:30,751 [INFO] tensorflow: epoch = 8.082901554404145, learning_rate = 0.00011120391, loss = 0.003319085, step = 3120 (5.394 sec) 2021-10-19 07:03:31,533 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.029 INFO:tensorflow:epoch = 8.152849740932643, learning_rate = 0.0001142295, loss = 0.003157224, step = 3147 (5.326 sec) 2021-10-19 07:03:36,077 [INFO] tensorflow: epoch = 8.152849740932643, learning_rate = 0.0001142295, loss = 0.003157224, step = 3147 (5.326 sec) 2021-10-19 07:03:36,497 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.146 INFO:tensorflow:global_step/sec: 5.0314 2021-10-19 07:03:37,469 [INFO] tensorflow: global_step/sec: 5.0314 INFO:tensorflow:epoch = 8.22279792746114, learning_rate = 0.0001173374, loss = 0.0029856875, step = 3174 (5.358 sec) 2021-10-19 07:03:41,435 [INFO] tensorflow: epoch = 8.22279792746114, learning_rate = 0.0001173374, loss = 0.0029856875, step = 3174 (5.358 sec) 2021-10-19 07:03:41,436 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.252 INFO:tensorflow:global_step/sec: 5.05015 2021-10-19 07:03:44,993 [INFO] tensorflow: global_step/sec: 5.05015 2021-10-19 07:03:46,380 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.226 INFO:tensorflow:epoch = 8.292746113989638, learning_rate = 0.00012052975, loss = 0.00288443, step = 3201 (5.329 sec) 2021-10-19 07:03:46,764 [INFO] tensorflow: epoch = 8.292746113989638, learning_rate = 0.00012052975, loss = 0.00288443, step = 3201 (5.329 sec) 2021-10-19 07:03:51,530 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.420 INFO:tensorflow:epoch = 8.360103626943005, learning_rate = 0.00012368597, loss = 0.0027171923, step = 3227 (5.347 sec) 2021-10-19 07:03:52,111 [INFO] tensorflow: epoch = 8.360103626943005, learning_rate = 0.00012368597, loss = 0.0027171923, step = 3227 (5.347 sec) INFO:tensorflow:global_step/sec: 4.93363 2021-10-19 07:03:52,696 [INFO] tensorflow: global_step/sec: 4.93363 2021-10-19 07:03:56,432 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.403 INFO:tensorflow:epoch = 8.430051813471502, learning_rate = 0.00012705117, loss = 0.002583369, step = 3254 (5.319 sec) 2021-10-19 07:03:57,431 [INFO] tensorflow: epoch = 8.430051813471502, learning_rate = 0.00012705117, loss = 0.002583369, step = 3254 (5.319 sec) INFO:tensorflow:global_step/sec: 5.05841 2021-10-19 07:04:00,208 [INFO] tensorflow: global_step/sec: 5.05841 2021-10-19 07:04:01,408 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.096 INFO:tensorflow:epoch = 8.5, learning_rate = 0.0001305078, loss = 0.0024513288, step = 3281 (5.363 sec) 2021-10-19 07:04:02,794 [INFO] tensorflow: epoch = 8.5, learning_rate = 0.0001305078, loss = 0.0024513288, step = 3281 (5.363 sec) 2021-10-19 07:04:06,389 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.081 INFO:tensorflow:global_step/sec: 5.00658 2021-10-19 07:04:07,798 [INFO] tensorflow: global_step/sec: 5.00658 INFO:tensorflow:epoch = 8.567357512953368, learning_rate = 0.00013392534, loss = 0.0024175725, step = 3307 (5.210 sec) 2021-10-19 07:04:08,003 [INFO] tensorflow: epoch = 8.567357512953368, learning_rate = 0.00013392534, loss = 0.0024175725, step = 3307 (5.210 sec) 2021-10-19 07:04:11,310 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.320 INFO:tensorflow:epoch = 8.637305699481866, learning_rate = 0.00013756912, loss = 0.0022397137, step = 3334 (5.238 sec) 2021-10-19 07:04:13,242 [INFO] tensorflow: epoch = 8.637305699481866, learning_rate = 0.00013756912, loss = 0.0022397137, step = 3334 (5.238 sec) INFO:tensorflow:global_step/sec: 5.10747 2021-10-19 07:04:15,238 [INFO] tensorflow: global_step/sec: 5.10747 2021-10-19 07:04:16,204 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.438 INFO:tensorflow:epoch = 8.704663212435234, learning_rate = 0.00014117155, loss = 0.0021339445, step = 3360 (5.231 sec) 2021-10-19 07:04:18,473 [INFO] tensorflow: epoch = 8.704663212435234, learning_rate = 0.00014117155, loss = 0.0021339445, step = 3360 (5.231 sec) 2021-10-19 07:04:21,336 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.485 INFO:tensorflow:global_step/sec: 4.94917 2021-10-19 07:04:22,916 [INFO] tensorflow: global_step/sec: 4.94917 INFO:tensorflow:epoch = 8.772020725388602, learning_rate = 0.00014486832, loss = 0.002046443, step = 3386 (5.273 sec) 2021-10-19 07:04:23,747 [INFO] tensorflow: epoch = 8.772020725388602, learning_rate = 0.00014486832, loss = 0.002046443, step = 3386 (5.273 sec) 2021-10-19 07:04:26,326 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.045 INFO:tensorflow:epoch = 8.83937823834197, learning_rate = 0.00014866187, loss = 0.0019401677, step = 3412 (5.234 sec) 2021-10-19 07:04:28,981 [INFO] tensorflow: epoch = 8.83937823834197, learning_rate = 0.00014866187, loss = 0.0019401677, step = 3412 (5.234 sec) INFO:tensorflow:global_step/sec: 4.97492 2021-10-19 07:04:30,554 [INFO] tensorflow: global_step/sec: 4.97492 2021-10-19 07:04:31,334 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.969 INFO:tensorflow:epoch = 8.909326424870466, learning_rate = 0.00015270646, loss = 0.0018425812, step = 3439 (5.372 sec) 2021-10-19 07:04:34,352 [INFO] tensorflow: epoch = 8.909326424870466, learning_rate = 0.00015270646, loss = 0.0018425812, step = 3439 (5.372 sec) 2021-10-19 07:04:36,353 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.928 INFO:tensorflow:global_step/sec: 5.0276 2021-10-19 07:04:38,113 [INFO] tensorflow: global_step/sec: 5.0276 INFO:tensorflow:epoch = 8.979274611398964, learning_rate = 0.00015686124, loss = 0.0017597482, step = 3466 (5.323 sec) 2021-10-19 07:04:39,675 [INFO] tensorflow: epoch = 8.979274611398964, learning_rate = 0.00015686124, loss = 0.0017597482, step = 3466 (5.323 sec) 2021-10-19 07:04:41,280 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 9/120: loss: 0.00174 learning rate: 0.00016 Time taken: 0:01:16.909204 ETA: 2:22:16.921618 2021-10-19 07:04:41,280 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.296 INFO:tensorflow:epoch = 9.049222797927461, learning_rate = 0.00016112905, loss = 0.0016758384, step = 3493 (5.357 sec) 2021-10-19 07:04:45,032 [INFO] tensorflow: epoch = 9.049222797927461, learning_rate = 0.00016112905, loss = 0.0016758384, step = 3493 (5.357 sec) INFO:tensorflow:global_step/sec: 5.0433 2021-10-19 07:04:45,647 [INFO] tensorflow: global_step/sec: 5.0433 2021-10-19 07:04:46,251 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.122 INFO:tensorflow:epoch = 9.119170984455959, learning_rate = 0.00016551281, loss = 0.0016279179, step = 3520 (5.376 sec) 2021-10-19 07:04:50,408 [INFO] tensorflow: epoch = 9.119170984455959, learning_rate = 0.00016551281, loss = 0.0016279179, step = 3520 (5.376 sec) 2021-10-19 07:04:51,220 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.126 INFO:tensorflow:global_step/sec: 5.03195 2021-10-19 07:04:53,199 [INFO] tensorflow: global_step/sec: 5.03195 INFO:tensorflow:epoch = 9.186528497409327, learning_rate = 0.00016984699, loss = 0.0015138502, step = 3546 (5.194 sec) 2021-10-19 07:04:55,602 [INFO] tensorflow: epoch = 9.186528497409327, learning_rate = 0.00016984699, loss = 0.0015138502, step = 3546 (5.194 sec) 2021-10-19 07:04:56,193 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.112 INFO:tensorflow:global_step/sec: 5.04347 2021-10-19 07:05:00,734 [INFO] tensorflow: global_step/sec: 5.04347 INFO:tensorflow:epoch = 9.256476683937825, learning_rate = 0.00017446812, loss = 0.001507592, step = 3573 (5.326 sec) 2021-10-19 07:05:00,928 [INFO] tensorflow: epoch = 9.256476683937825, learning_rate = 0.00017446812, loss = 0.001507592, step = 3573 (5.326 sec) 2021-10-19 07:05:01,138 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.225 2021-10-19 07:05:06,121 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.069 INFO:tensorflow:epoch = 9.326424870466322, learning_rate = 0.00017921499, loss = 0.0013854624, step = 3600 (5.395 sec) 2021-10-19 07:05:06,323 [INFO] tensorflow: epoch = 9.326424870466322, learning_rate = 0.00017921499, loss = 0.0013854624, step = 3600 (5.395 sec) INFO:tensorflow:global_step/sec: 5.02365 2021-10-19 07:05:08,298 [INFO] tensorflow: global_step/sec: 5.02365 2021-10-19 07:05:11,075 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.188 INFO:tensorflow:epoch = 9.396373056994818, learning_rate = 0.00018409081, loss = 0.0013712565, step = 3627 (5.390 sec) 2021-10-19 07:05:11,713 [INFO] tensorflow: epoch = 9.396373056994818, learning_rate = 0.00018409081, loss = 0.0013712565, step = 3627 (5.390 sec) INFO:tensorflow:global_step/sec: 4.95824 2021-10-19 07:05:15,962 [INFO] tensorflow: global_step/sec: 4.95824 2021-10-19 07:05:16,158 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.674 INFO:tensorflow:epoch = 9.463730569948186, learning_rate = 0.00018891146, loss = 0.0013053565, step = 3653 (5.232 sec) 2021-10-19 07:05:16,945 [INFO] tensorflow: epoch = 9.463730569948186, learning_rate = 0.00018891146, loss = 0.0013053565, step = 3653 (5.232 sec) 2021-10-19 07:05:21,087 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.290 INFO:tensorflow:epoch = 9.533678756476684, learning_rate = 0.00019405129, loss = 0.0012217377, step = 3680 (5.315 sec) 2021-10-19 07:05:22,260 [INFO] tensorflow: epoch = 9.533678756476684, learning_rate = 0.00019405129, loss = 0.0012217377, step = 3680 (5.315 sec) INFO:tensorflow:global_step/sec: 5.0673 2021-10-19 07:05:23,461 [INFO] tensorflow: global_step/sec: 5.0673 2021-10-19 07:05:26,005 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.338 INFO:tensorflow:epoch = 9.603626943005182, learning_rate = 0.00019933094, loss = 0.0011603164, step = 3707 (5.374 sec) 2021-10-19 07:05:27,634 [INFO] tensorflow: epoch = 9.603626943005182, learning_rate = 0.00019933094, loss = 0.0011603164, step = 3707 (5.374 sec) INFO:tensorflow:global_step/sec: 5.0432 2021-10-19 07:05:30,996 [INFO] tensorflow: global_step/sec: 5.0432 2021-10-19 07:05:30,996 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.035 INFO:tensorflow:epoch = 9.67357512953368, learning_rate = 0.00020475408, loss = 0.0010990256, step = 3734 (5.310 sec) 2021-10-19 07:05:32,944 [INFO] tensorflow: epoch = 9.67357512953368, learning_rate = 0.00020475408, loss = 0.0010990256, step = 3734 (5.310 sec) 2021-10-19 07:05:35,946 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.205 INFO:tensorflow:epoch = 9.743523316062177, learning_rate = 0.00021032493, loss = 0.0010559915, step = 3761 (5.300 sec) 2021-10-19 07:05:38,244 [INFO] tensorflow: epoch = 9.743523316062177, learning_rate = 0.00021032493, loss = 0.0010559915, step = 3761 (5.300 sec) INFO:tensorflow:global_step/sec: 5.10652 2021-10-19 07:05:38,437 [INFO] tensorflow: global_step/sec: 5.10652 2021-10-19 07:05:40,759 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.779 INFO:tensorflow:epoch = 9.813471502590675, learning_rate = 0.00021604715, loss = 0.000985771, step = 3788 (5.302 sec) 2021-10-19 07:05:43,546 [INFO] tensorflow: epoch = 9.813471502590675, learning_rate = 0.00021604715, loss = 0.000985771, step = 3788 (5.302 sec) 2021-10-19 07:05:45,825 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.743 INFO:tensorflow:global_step/sec: 5.00509 2021-10-19 07:05:46,029 [INFO] tensorflow: global_step/sec: 5.00509 INFO:tensorflow:epoch = 9.880829015544041, learning_rate = 0.00022170464, loss = 0.0009390662, step = 3814 (5.366 sec) 2021-10-19 07:05:48,913 [INFO] tensorflow: epoch = 9.880829015544041, learning_rate = 0.00022170464, loss = 0.0009390662, step = 3814 (5.366 sec) 2021-10-19 07:05:50,934 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.574 INFO:tensorflow:global_step/sec: 4.95091 2021-10-19 07:05:53,705 [INFO] tensorflow: global_step/sec: 4.95091 INFO:tensorflow:epoch = 9.950777202072539, learning_rate = 0.00022773667, loss = 0.0009032046, step = 3841 (5.389 sec) 2021-10-19 07:05:54,302 [INFO] tensorflow: epoch = 9.950777202072539, learning_rate = 0.00022773667, loss = 0.0009032046, step = 3841 (5.389 sec) 2021-10-19 07:05:55,901 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.134 INFO:tensorflow:Saving checkpoints for step-3860. 2021-10-19 07:05:57,903 [INFO] tensorflow: Saving checkpoints for step-3860. INFO:tensorflow:epoch = 10.0, learning_rate = 0.00023207937, loss = 0.00083821645, step = 3860 (8.593 sec) 2021-10-19 07:06:02,894 [INFO] tensorflow: epoch = 10.0, learning_rate = 0.00023207937, loss = 0.00083821645, step = 3860 (8.593 sec) 2021-10-19 07:06:02,895 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 10/120: loss: 0.00084 learning rate: 0.00023 Time taken: 0:01:21.594275 ETA: 2:29:35.370250 2021-10-19 07:06:05,758 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 10.146 INFO:tensorflow:global_step/sec: 3.04912 2021-10-19 07:06:06,167 [INFO] tensorflow: global_step/sec: 3.04912 INFO:tensorflow:epoch = 10.067357512953368, learning_rate = 0.00023815667, loss = 0.0008534422, step = 3886 (5.300 sec) 2021-10-19 07:06:08,195 [INFO] tensorflow: epoch = 10.067357512953368, learning_rate = 0.00023815667, loss = 0.0008534422, step = 3886 (5.300 sec) 2021-10-19 07:06:10,829 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.720 INFO:tensorflow:epoch = 10.134715025906736, learning_rate = 0.00024439313, loss = 0.0008250105, step = 3912 (5.320 sec) 2021-10-19 07:06:13,515 [INFO] tensorflow: epoch = 10.134715025906736, learning_rate = 0.00024439313, loss = 0.0008250105, step = 3912 (5.320 sec) INFO:tensorflow:global_step/sec: 4.88942 2021-10-19 07:06:13,939 [INFO] tensorflow: global_step/sec: 4.88942 2021-10-19 07:06:16,023 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.258 INFO:tensorflow:epoch = 10.199481865284975, learning_rate = 0.0002505438, loss = 0.0007736633, step = 3937 (5.264 sec) 2021-10-19 07:06:18,778 [INFO] tensorflow: epoch = 10.199481865284975, learning_rate = 0.0002505438, loss = 0.0007736633, step = 3937 (5.264 sec) 2021-10-19 07:06:21,223 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.231 INFO:tensorflow:global_step/sec: 4.77487 2021-10-19 07:06:21,898 [INFO] tensorflow: global_step/sec: 4.77487 INFO:tensorflow:epoch = 10.266839378238343, learning_rate = 0.00025710464, loss = 0.00071485527, step = 3963 (5.377 sec) 2021-10-19 07:06:24,156 [INFO] tensorflow: epoch = 10.266839378238343, learning_rate = 0.00025710464, loss = 0.00071485527, step = 3963 (5.377 sec) 2021-10-19 07:06:26,411 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.280 INFO:tensorflow:epoch = 10.334196891191711, learning_rate = 0.00026383725, loss = 0.00071961107, step = 3989 (5.336 sec) 2021-10-19 07:06:29,491 [INFO] tensorflow: epoch = 10.334196891191711, learning_rate = 0.00026383725, loss = 0.00071961107, step = 3989 (5.336 sec) INFO:tensorflow:global_step/sec: 4.8624 2021-10-19 07:06:29,713 [INFO] tensorflow: global_step/sec: 4.8624 2021-10-19 07:06:31,537 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.508 INFO:tensorflow:epoch = 10.398963730569948, learning_rate = 0.00027047726, loss = 0.00066971814, step = 4014 (5.209 sec) 2021-10-19 07:06:34,700 [INFO] tensorflow: epoch = 10.398963730569948, learning_rate = 0.00027047726, loss = 0.00066971814, step = 4014 (5.209 sec) 2021-10-19 07:06:36,779 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.077 INFO:tensorflow:global_step/sec: 4.8174 2021-10-19 07:06:37,601 [INFO] tensorflow: global_step/sec: 4.8174 INFO:tensorflow:epoch = 10.463730569948186, learning_rate = 0.00027728413, loss = 0.0006452283, step = 4039 (5.238 sec) 2021-10-19 07:06:39,938 [INFO] tensorflow: epoch = 10.463730569948186, learning_rate = 0.00027728413, loss = 0.0006452283, step = 4039 (5.238 sec) 2021-10-19 07:06:42,034 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.034 INFO:tensorflow:epoch = 10.531088082901555, learning_rate = 0.00028454544, loss = 0.000636391, step = 4065 (5.390 sec) 2021-10-19 07:06:45,328 [INFO] tensorflow: epoch = 10.531088082901555, learning_rate = 0.00028454544, loss = 0.000636391, step = 4065 (5.390 sec) INFO:tensorflow:global_step/sec: 4.78512 2021-10-19 07:06:45,542 [INFO] tensorflow: global_step/sec: 4.78512 2021-10-19 07:06:47,249 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.178 INFO:tensorflow:epoch = 10.598445595854923, learning_rate = 0.00029199637, loss = 0.0005764494, step = 4091 (5.352 sec) 2021-10-19 07:06:50,680 [INFO] tensorflow: epoch = 10.598445595854923, learning_rate = 0.00029199637, loss = 0.0005764494, step = 4091 (5.352 sec) 2021-10-19 07:06:52,363 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.556 INFO:tensorflow:global_step/sec: 4.85406 2021-10-19 07:06:53,371 [INFO] tensorflow: global_step/sec: 4.85406 INFO:tensorflow:epoch = 10.66580310880829, learning_rate = 0.00029964297, loss = 0.00055668456, step = 4117 (5.267 sec) 2021-10-19 07:06:55,947 [INFO] tensorflow: epoch = 10.66580310880829, learning_rate = 0.00029964297, loss = 0.00055668456, step = 4117 (5.267 sec) 2021-10-19 07:06:57,362 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.006 INFO:tensorflow:global_step/sec: 5.01485 2021-10-19 07:07:00,948 [INFO] tensorflow: global_step/sec: 5.01485 INFO:tensorflow:epoch = 10.733160621761659, learning_rate = 0.00030748924, loss = 0.0005433838, step = 4143 (5.225 sec) 2021-10-19 07:07:01,172 [INFO] tensorflow: epoch = 10.733160621761659, learning_rate = 0.00030748924, loss = 0.0005433838, step = 4143 (5.225 sec) 2021-10-19 07:07:02,409 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.816 INFO:tensorflow:epoch = 10.800518134715027, learning_rate = 0.00031554155, loss = 0.00050851976, step = 4169 (5.425 sec) 2021-10-19 07:07:06,597 [INFO] tensorflow: epoch = 10.800518134715027, learning_rate = 0.00031554155, loss = 0.00050851976, step = 4169 (5.425 sec) 2021-10-19 07:07:07,605 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.245 INFO:tensorflow:global_step/sec: 4.83939 2021-10-19 07:07:08,800 [INFO] tensorflow: global_step/sec: 4.83939 INFO:tensorflow:epoch = 10.867875647668393, learning_rate = 0.00032380412, loss = 0.00045737202, step = 4195 (5.211 sec) 2021-10-19 07:07:11,808 [INFO] tensorflow: epoch = 10.867875647668393, learning_rate = 0.00032380412, loss = 0.00045737202, step = 4195 (5.211 sec) 2021-10-19 07:07:12,599 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.026 INFO:tensorflow:global_step/sec: 5.01024 2021-10-19 07:07:16,385 [INFO] tensorflow: global_step/sec: 5.01024 INFO:tensorflow:epoch = 10.937823834196891, learning_rate = 0.00033261406, loss = 0.00048746864, step = 4222 (5.407 sec) 2021-10-19 07:07:17,215 [INFO] tensorflow: epoch = 10.937823834196891, learning_rate = 0.00033261406, loss = 0.00048746864, step = 4222 (5.407 sec) 2021-10-19 07:07:17,607 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.968 2021-10-19 07:07:21,991 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 11/120: loss: 0.00043 learning rate: 0.00034 Time taken: 0:01:19.094245 ETA: 2:23:41.272700 INFO:tensorflow:epoch = 11.007772020725389, learning_rate = 0.00034166352, loss = 0.00042695302, step = 4249 (5.366 sec) 2021-10-19 07:07:22,581 [INFO] tensorflow: epoch = 11.007772020725389, learning_rate = 0.00034166352, loss = 0.00042695302, step = 4249 (5.366 sec) 2021-10-19 07:07:22,582 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.105 INFO:tensorflow:global_step/sec: 4.97265 2021-10-19 07:07:24,027 [INFO] tensorflow: global_step/sec: 4.97265 2021-10-19 07:07:27,610 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.889 INFO:tensorflow:epoch = 11.075129533678757, learning_rate = 0.0003506106, loss = 0.00041938067, step = 4275 (5.227 sec) 2021-10-19 07:07:27,808 [INFO] tensorflow: epoch = 11.075129533678757, learning_rate = 0.0003506106, loss = 0.00041938067, step = 4275 (5.227 sec) INFO:tensorflow:global_step/sec: 5.01066 2021-10-19 07:07:31,610 [INFO] tensorflow: global_step/sec: 5.01066 2021-10-19 07:07:32,614 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.987 INFO:tensorflow:epoch = 11.142487046632125, learning_rate = 0.00035979165, loss = 0.00039808394, step = 4301 (5.220 sec) 2021-10-19 07:07:33,028 [INFO] tensorflow: epoch = 11.142487046632125, learning_rate = 0.00035979165, loss = 0.00039808394, step = 4301 (5.220 sec) 2021-10-19 07:07:37,679 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.744 INFO:tensorflow:epoch = 11.209844559585493, learning_rate = 0.0003692134, loss = 0.0003931818, step = 4327 (5.258 sec) 2021-10-19 07:07:38,286 [INFO] tensorflow: epoch = 11.209844559585493, learning_rate = 0.0003692134, loss = 0.0003931818, step = 4327 (5.258 sec) INFO:tensorflow:global_step/sec: 4.96644 2021-10-19 07:07:39,262 [INFO] tensorflow: global_step/sec: 4.96644 2021-10-19 07:07:42,670 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.036 INFO:tensorflow:epoch = 11.27979274611399, learning_rate = 0.00037925862, loss = 0.00039819637, step = 4354 (5.392 sec) 2021-10-19 07:07:43,677 [INFO] tensorflow: epoch = 11.27979274611399, learning_rate = 0.00037925862, loss = 0.00039819637, step = 4354 (5.392 sec) INFO:tensorflow:global_step/sec: 4.97974 2021-10-19 07:07:46,893 [INFO] tensorflow: global_step/sec: 4.97974 2021-10-19 07:07:47,722 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.798 INFO:tensorflow:epoch = 11.347150259067359, learning_rate = 0.00038919, loss = 0.00037924037, step = 4380 (5.263 sec) 2021-10-19 07:07:48,940 [INFO] tensorflow: epoch = 11.347150259067359, learning_rate = 0.00038919, loss = 0.00037924037, step = 4380 (5.263 sec) 2021-10-19 07:07:52,757 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.864 INFO:tensorflow:epoch = 11.414507772020725, learning_rate = 0.00039938148, loss = 0.00035751014, step = 4406 (5.218 sec) 2021-10-19 07:07:54,158 [INFO] tensorflow: epoch = 11.414507772020725, learning_rate = 0.00039938148, loss = 0.00035751014, step = 4406 (5.218 sec) INFO:tensorflow:global_step/sec: 4.9652 2021-10-19 07:07:54,546 [INFO] tensorflow: global_step/sec: 4.9652 2021-10-19 07:07:57,800 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.829 INFO:tensorflow:epoch = 11.481865284974093, learning_rate = 0.00040983982, loss = 0.00033366733, step = 4432 (5.281 sec) 2021-10-19 07:07:59,439 [INFO] tensorflow: epoch = 11.481865284974093, learning_rate = 0.00040983982, loss = 0.00033366733, step = 4432 (5.281 sec) INFO:tensorflow:global_step/sec: 4.90432 2021-10-19 07:08:02,294 [INFO] tensorflow: global_step/sec: 4.90432 2021-10-19 07:08:02,909 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.575 INFO:tensorflow:epoch = 11.549222797927461, learning_rate = 0.000420572, loss = 0.00032001606, step = 4458 (5.295 sec) 2021-10-19 07:08:04,734 [INFO] tensorflow: epoch = 11.549222797927461, learning_rate = 0.000420572, loss = 0.00032001606, step = 4458 (5.295 sec) 2021-10-19 07:08:07,974 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.744 INFO:tensorflow:epoch = 11.61658031088083, learning_rate = 0.00043158524, loss = 0.00030278042, step = 4484 (5.301 sec) 2021-10-19 07:08:10,035 [INFO] tensorflow: epoch = 11.61658031088083, learning_rate = 0.00043158524, loss = 0.00030278042, step = 4484 (5.301 sec) INFO:tensorflow:global_step/sec: 4.90833 2021-10-19 07:08:10,036 [INFO] tensorflow: global_step/sec: 4.90833 2021-10-19 07:08:13,024 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.805 INFO:tensorflow:epoch = 11.686528497409327, learning_rate = 0.00044332747, loss = 0.00030039754, step = 4511 (5.410 sec) 2021-10-19 07:08:15,445 [INFO] tensorflow: epoch = 11.686528497409327, learning_rate = 0.00044332747, loss = 0.00030039754, step = 4511 (5.410 sec) INFO:tensorflow:global_step/sec: 4.9888 2021-10-19 07:08:17,653 [INFO] tensorflow: global_step/sec: 4.9888 2021-10-19 07:08:18,056 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.874 INFO:tensorflow:epoch = 11.753886010362695, learning_rate = 0.00045493682, loss = 0.00026732168, step = 4537 (5.260 sec) 2021-10-19 07:08:20,705 [INFO] tensorflow: epoch = 11.753886010362695, learning_rate = 0.00045493682, loss = 0.00026732168, step = 4537 (5.260 sec) 2021-10-19 07:08:23,109 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.791 INFO:tensorflow:global_step/sec: 4.92664 2021-10-19 07:08:25,366 [INFO] tensorflow: global_step/sec: 4.92664 INFO:tensorflow:epoch = 11.821243523316063, learning_rate = 0.0004668497, loss = 0.00028095095, step = 4563 (5.281 sec) 2021-10-19 07:08:25,986 [INFO] tensorflow: epoch = 11.821243523316063, learning_rate = 0.0004668497, loss = 0.00028095095, step = 4563 (5.281 sec) 2021-10-19 07:08:28,182 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.715 INFO:tensorflow:epoch = 11.89119170984456, learning_rate = 0.00047955156, loss = 0.000282644, step = 4590 (5.395 sec) 2021-10-19 07:08:31,381 [INFO] tensorflow: epoch = 11.89119170984456, learning_rate = 0.00047955156, loss = 0.000282644, step = 4590 (5.395 sec) INFO:tensorflow:global_step/sec: 4.9636 2021-10-19 07:08:33,022 [INFO] tensorflow: global_step/sec: 4.9636 2021-10-19 07:08:33,229 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.815 INFO:tensorflow:epoch = 11.958549222797927, learning_rate = 0.0004921092, loss = 0.0002624146, step = 4616 (5.382 sec) 2021-10-19 07:08:36,762 [INFO] tensorflow: epoch = 11.958549222797927, learning_rate = 0.0004921092, loss = 0.0002624146, step = 4616 (5.382 sec) 2021-10-19 07:08:38,341 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.564 2021-10-19 07:08:39,919 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 12/120: loss: 0.00024 learning rate: 0.00050 Time taken: 0:01:17.941867 ETA: 2:20:17.721648 INFO:tensorflow:global_step/sec: 4.93684 2021-10-19 07:08:40,719 [INFO] tensorflow: global_step/sec: 4.93684 INFO:tensorflow:epoch = 12.028497409326425, learning_rate = 0.00049999997, loss = 0.00024806728, step = 4643 (5.335 sec) 2021-10-19 07:08:42,098 [INFO] tensorflow: epoch = 12.028497409326425, learning_rate = 0.00049999997, loss = 0.00024806728, step = 4643 (5.335 sec) 2021-10-19 07:08:43,276 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.264 INFO:tensorflow:epoch = 12.098445595854923, learning_rate = 0.00049999997, loss = 0.0002706613, step = 4670 (5.339 sec) 2021-10-19 07:08:47,437 [INFO] tensorflow: epoch = 12.098445595854923, learning_rate = 0.00049999997, loss = 0.0002706613, step = 4670 (5.339 sec) INFO:tensorflow:global_step/sec: 5.05667 2021-10-19 07:08:48,234 [INFO] tensorflow: global_step/sec: 5.05667 2021-10-19 07:08:48,236 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.166 INFO:tensorflow:epoch = 12.16839378238342, learning_rate = 0.00049999997, loss = 0.00020965928, step = 4697 (5.379 sec) 2021-10-19 07:08:52,816 [INFO] tensorflow: epoch = 12.16839378238342, learning_rate = 0.00049999997, loss = 0.00020965928, step = 4697 (5.379 sec) 2021-10-19 07:08:53,211 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.103 INFO:tensorflow:global_step/sec: 4.96737 2021-10-19 07:08:55,884 [INFO] tensorflow: global_step/sec: 4.96737 INFO:tensorflow:epoch = 12.235751295336788, learning_rate = 0.00049999997, loss = 0.0002496301, step = 4723 (5.268 sec) 2021-10-19 07:08:58,084 [INFO] tensorflow: epoch = 12.235751295336788, learning_rate = 0.00049999997, loss = 0.0002496301, step = 4723 (5.268 sec) 2021-10-19 07:08:58,278 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.738 2021-10-19 07:09:03,250 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.112 INFO:tensorflow:epoch = 12.305699481865286, learning_rate = 0.00049999997, loss = 0.00020846752, step = 4750 (5.372 sec) 2021-10-19 07:09:03,456 [INFO] tensorflow: epoch = 12.305699481865286, learning_rate = 0.00049999997, loss = 0.00020846752, step = 4750 (5.372 sec) INFO:tensorflow:global_step/sec: 5.01778 2021-10-19 07:09:03,457 [INFO] tensorflow: global_step/sec: 5.01778 2021-10-19 07:09:08,179 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.290 INFO:tensorflow:epoch = 12.375647668393784, learning_rate = 0.00049999997, loss = 0.00020733892, step = 4777 (5.314 sec) 2021-10-19 07:09:08,770 [INFO] tensorflow: epoch = 12.375647668393784, learning_rate = 0.00049999997, loss = 0.00020733892, step = 4777 (5.314 sec) INFO:tensorflow:global_step/sec: 5.00597 2021-10-19 07:09:11,048 [INFO] tensorflow: global_step/sec: 5.00597 2021-10-19 07:09:13,284 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.590 INFO:tensorflow:epoch = 12.44300518134715, learning_rate = 0.00049999997, loss = 0.00021972288, step = 4803 (5.339 sec) 2021-10-19 07:09:14,109 [INFO] tensorflow: epoch = 12.44300518134715, learning_rate = 0.00049999997, loss = 0.00021972288, step = 4803 (5.339 sec) 2021-10-19 07:09:18,409 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.515 INFO:tensorflow:global_step/sec: 4.90275 2021-10-19 07:09:18,799 [INFO] tensorflow: global_step/sec: 4.90275 INFO:tensorflow:epoch = 12.510362694300518, learning_rate = 0.00049999997, loss = 0.00020024493, step = 4829 (5.268 sec) 2021-10-19 07:09:19,377 [INFO] tensorflow: epoch = 12.510362694300518, learning_rate = 0.00049999997, loss = 0.00020024493, step = 4829 (5.268 sec) 2021-10-19 07:09:23,366 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.175 INFO:tensorflow:epoch = 12.580310880829016, learning_rate = 0.00049999997, loss = 0.00017835021, step = 4856 (5.370 sec) 2021-10-19 07:09:24,747 [INFO] tensorflow: epoch = 12.580310880829016, learning_rate = 0.00049999997, loss = 0.00017835021, step = 4856 (5.370 sec) INFO:tensorflow:global_step/sec: 5.06077 2021-10-19 07:09:26,308 [INFO] tensorflow: global_step/sec: 5.06077 2021-10-19 07:09:28,322 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.178 INFO:tensorflow:epoch = 12.650259067357513, learning_rate = 0.00049999997, loss = 0.00017947267, step = 4883 (5.355 sec) 2021-10-19 07:09:30,102 [INFO] tensorflow: epoch = 12.650259067357513, learning_rate = 0.00049999997, loss = 0.00017947267, step = 4883 (5.355 sec) 2021-10-19 07:09:33,279 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.177 INFO:tensorflow:global_step/sec: 5.02968 2021-10-19 07:09:33,863 [INFO] tensorflow: global_step/sec: 5.02968 INFO:tensorflow:epoch = 12.720207253886011, learning_rate = 0.00049999997, loss = 0.00017189143, step = 4910 (5.360 sec) 2021-10-19 07:09:35,462 [INFO] tensorflow: epoch = 12.720207253886011, learning_rate = 0.00049999997, loss = 0.00017189143, step = 4910 (5.360 sec) 2021-10-19 07:09:38,290 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.959 INFO:tensorflow:epoch = 12.78756476683938, learning_rate = 0.00049999997, loss = 0.00015444457, step = 4936 (5.203 sec) 2021-10-19 07:09:40,665 [INFO] tensorflow: epoch = 12.78756476683938, learning_rate = 0.00049999997, loss = 0.00015444457, step = 4936 (5.203 sec) INFO:tensorflow:global_step/sec: 5.0169 2021-10-19 07:09:41,437 [INFO] tensorflow: global_step/sec: 5.0169 2021-10-19 07:09:43,187 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.421 INFO:tensorflow:epoch = 12.857512953367877, learning_rate = 0.00049999997, loss = 0.00016153774, step = 4963 (5.313 sec) 2021-10-19 07:09:45,978 [INFO] tensorflow: epoch = 12.857512953367877, learning_rate = 0.00049999997, loss = 0.00016153774, step = 4963 (5.313 sec) 2021-10-19 07:09:48,149 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.155 INFO:tensorflow:global_step/sec: 5.06719 2021-10-19 07:09:48,936 [INFO] tensorflow: global_step/sec: 5.06719 INFO:tensorflow:epoch = 12.927461139896373, learning_rate = 0.00049999997, loss = 0.00016743092, step = 4990 (5.330 sec) 2021-10-19 07:09:51,309 [INFO] tensorflow: epoch = 12.927461139896373, learning_rate = 0.00049999997, loss = 0.00016743092, step = 4990 (5.330 sec) 2021-10-19 07:09:53,135 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.059 INFO:tensorflow:global_step/sec: 5.02612 2021-10-19 07:09:56,497 [INFO] tensorflow: global_step/sec: 5.02612 INFO:tensorflow:epoch = 12.99740932642487, learning_rate = 0.00049999997, loss = 0.00017529647, step = 5017 (5.381 sec) 2021-10-19 07:09:56,690 [INFO] tensorflow: epoch = 12.99740932642487, learning_rate = 0.00049999997, loss = 0.00017529647, step = 5017 (5.381 sec) 2021-10-19 07:09:56,891 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 13/120: loss: 0.00018 learning rate: 0.00050 Time taken: 0:01:16.966406 ETA: 2:17:15.405479 2021-10-19 07:09:58,069 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.269 INFO:tensorflow:epoch = 13.067357512953368, learning_rate = 0.00049999997, loss = 0.0001415632, step = 5044 (5.384 sec) 2021-10-19 07:10:02,074 [INFO] tensorflow: epoch = 13.067357512953368, learning_rate = 0.00049999997, loss = 0.0001415632, step = 5044 (5.384 sec) 2021-10-19 07:10:03,061 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.030 INFO:tensorflow:global_step/sec: 5.02599 2021-10-19 07:10:04,058 [INFO] tensorflow: global_step/sec: 5.02599 INFO:tensorflow:epoch = 13.134715025906736, learning_rate = 0.00049999997, loss = 0.00013346228, step = 5070 (5.204 sec) 2021-10-19 07:10:07,278 [INFO] tensorflow: epoch = 13.134715025906736, learning_rate = 0.00049999997, loss = 0.00013346228, step = 5070 (5.204 sec) 2021-10-19 07:10:08,073 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.957 INFO:tensorflow:global_step/sec: 5.05212 2021-10-19 07:10:11,579 [INFO] tensorflow: global_step/sec: 5.05212 INFO:tensorflow:epoch = 13.204663212435234, learning_rate = 0.00049999997, loss = 0.00014388328, step = 5097 (5.308 sec) 2021-10-19 07:10:12,587 [INFO] tensorflow: epoch = 13.204663212435234, learning_rate = 0.00049999997, loss = 0.00014388328, step = 5097 (5.308 sec) 2021-10-19 07:10:12,984 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.364 INFO:tensorflow:epoch = 13.272020725388602, learning_rate = 0.00049999997, loss = 0.0001534031, step = 5123 (5.226 sec) 2021-10-19 07:10:17,813 [INFO] tensorflow: epoch = 13.272020725388602, learning_rate = 0.00049999997, loss = 0.0001534031, step = 5123 (5.226 sec) 2021-10-19 07:10:18,017 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.869 INFO:tensorflow:global_step/sec: 4.99126 2021-10-19 07:10:19,192 [INFO] tensorflow: global_step/sec: 4.99126 2021-10-19 07:10:22,973 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.181 INFO:tensorflow:epoch = 13.3419689119171, learning_rate = 0.00049999997, loss = 0.00017426115, step = 5150 (5.372 sec) 2021-10-19 07:10:23,185 [INFO] tensorflow: epoch = 13.3419689119171, learning_rate = 0.00049999997, loss = 0.00017426115, step = 5150 (5.372 sec) INFO:tensorflow:global_step/sec: 5.03648 2021-10-19 07:10:26,737 [INFO] tensorflow: global_step/sec: 5.03648 2021-10-19 07:10:28,017 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.825 INFO:tensorflow:epoch = 13.409326424870466, learning_rate = 0.00049999997, loss = 0.00014977256, step = 5176 (5.230 sec) 2021-10-19 07:10:28,415 [INFO] tensorflow: epoch = 13.409326424870466, learning_rate = 0.00049999997, loss = 0.00014977256, step = 5176 (5.230 sec) 2021-10-19 07:10:33,160 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.446 INFO:tensorflow:epoch = 13.476683937823834, learning_rate = 0.00049999997, loss = 0.0001195707, step = 5202 (5.378 sec) 2021-10-19 07:10:33,793 [INFO] tensorflow: epoch = 13.476683937823834, learning_rate = 0.00049999997, loss = 0.0001195707, step = 5202 (5.378 sec) INFO:tensorflow:global_step/sec: 4.81622 2021-10-19 07:10:34,627 [INFO] tensorflow: global_step/sec: 4.81622 2021-10-19 07:10:38,296 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.472 INFO:tensorflow:epoch = 13.544041450777202, learning_rate = 0.00049999997, loss = 0.000119268196, step = 5228 (5.320 sec) 2021-10-19 07:10:39,113 [INFO] tensorflow: epoch = 13.544041450777202, learning_rate = 0.00049999997, loss = 0.000119268196, step = 5228 (5.320 sec) INFO:tensorflow:global_step/sec: 4.88864 2021-10-19 07:10:42,401 [INFO] tensorflow: global_step/sec: 4.88864 2021-10-19 07:10:43,399 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.600 INFO:tensorflow:epoch = 13.61139896373057, learning_rate = 0.00049999997, loss = 0.00012542644, step = 5254 (5.317 sec) 2021-10-19 07:10:44,431 [INFO] tensorflow: epoch = 13.61139896373057, learning_rate = 0.00049999997, loss = 0.00012542644, step = 5254 (5.317 sec) 2021-10-19 07:10:48,593 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.255 INFO:tensorflow:epoch = 13.676165803108809, learning_rate = 0.00049999997, loss = 0.0001248169, step = 5279 (5.221 sec) 2021-10-19 07:10:49,652 [INFO] tensorflow: epoch = 13.676165803108809, learning_rate = 0.00049999997, loss = 0.0001248169, step = 5279 (5.221 sec) INFO:tensorflow:global_step/sec: 4.82647 2021-10-19 07:10:50,274 [INFO] tensorflow: global_step/sec: 4.82647 2021-10-19 07:10:53,755 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.373 INFO:tensorflow:epoch = 13.743523316062177, learning_rate = 0.00049999997, loss = 0.00013549191, step = 5305 (5.363 sec) 2021-10-19 07:10:55,014 [INFO] tensorflow: epoch = 13.743523316062177, learning_rate = 0.00049999997, loss = 0.00013549191, step = 5305 (5.363 sec) INFO:tensorflow:global_step/sec: 4.84024 2021-10-19 07:10:58,125 [INFO] tensorflow: global_step/sec: 4.84024 2021-10-19 07:10:58,945 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.269 INFO:tensorflow:epoch = 13.810880829015545, learning_rate = 0.00049999997, loss = 0.0001391504, step = 5331 (5.332 sec) 2021-10-19 07:11:00,346 [INFO] tensorflow: epoch = 13.810880829015545, learning_rate = 0.00049999997, loss = 0.0001391504, step = 5331 (5.332 sec) 2021-10-19 07:11:03,981 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.858 INFO:tensorflow:epoch = 13.878238341968913, learning_rate = 0.00049999997, loss = 0.000119081706, step = 5357 (5.213 sec) 2021-10-19 07:11:05,560 [INFO] tensorflow: epoch = 13.878238341968913, learning_rate = 0.00049999997, loss = 0.000119081706, step = 5357 (5.213 sec) INFO:tensorflow:global_step/sec: 4.97591 2021-10-19 07:11:05,761 [INFO] tensorflow: global_step/sec: 4.97591 2021-10-19 07:11:08,887 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.387 INFO:tensorflow:epoch = 13.94818652849741, learning_rate = 0.00049999997, loss = 0.00011194327, step = 5384 (5.386 sec) 2021-10-19 07:11:10,946 [INFO] tensorflow: epoch = 13.94818652849741, learning_rate = 0.00049999997, loss = 0.00011194327, step = 5384 (5.386 sec) INFO:tensorflow:global_step/sec: 4.92179 2021-10-19 07:11:13,482 [INFO] tensorflow: global_step/sec: 4.92179 2021-10-19 07:11:14,100 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.184 2021-10-19 07:11:15,171 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 14/120: loss: 0.00012 learning rate: 0.00050 Time taken: 0:01:18.263528 ETA: 2:18:15.934005 INFO:tensorflow:epoch = 14.012953367875648, learning_rate = 0.00049999997, loss = 0.00014636786, step = 5409 (5.254 sec) 2021-10-19 07:11:16,200 [INFO] tensorflow: epoch = 14.012953367875648, learning_rate = 0.00049999997, loss = 0.00014636786, step = 5409 (5.254 sec) 2021-10-19 07:11:19,254 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.403 INFO:tensorflow:global_step/sec: 4.83929 2021-10-19 07:11:21,335 [INFO] tensorflow: global_step/sec: 4.83929 INFO:tensorflow:epoch = 14.080310880829016, learning_rate = 0.00049999997, loss = 0.00011032258, step = 5435 (5.344 sec) 2021-10-19 07:11:21,544 [INFO] tensorflow: epoch = 14.080310880829016, learning_rate = 0.00049999997, loss = 0.00011032258, step = 5435 (5.344 sec) 2021-10-19 07:11:24,339 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.668 INFO:tensorflow:epoch = 14.150259067357513, learning_rate = 0.00049999997, loss = 0.00010240976, step = 5462 (5.368 sec) 2021-10-19 07:11:26,912 [INFO] tensorflow: epoch = 14.150259067357513, learning_rate = 0.00049999997, loss = 0.00010240976, step = 5462 (5.368 sec) INFO:tensorflow:global_step/sec: 5.02354 2021-10-19 07:11:28,899 [INFO] tensorflow: global_step/sec: 5.02354 2021-10-19 07:11:29,299 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.163 INFO:tensorflow:epoch = 14.217616580310882, learning_rate = 0.00049999997, loss = 0.00011181923, step = 5488 (5.384 sec) 2021-10-19 07:11:32,295 [INFO] tensorflow: epoch = 14.217616580310882, learning_rate = 0.00049999997, loss = 0.00011181923, step = 5488 (5.384 sec) 2021-10-19 07:11:34,510 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.191 INFO:tensorflow:global_step/sec: 4.879 2021-10-19 07:11:36,687 [INFO] tensorflow: global_step/sec: 4.879 INFO:tensorflow:epoch = 14.28497409326425, learning_rate = 0.00049999997, loss = 0.00013103479, step = 5514 (5.209 sec) 2021-10-19 07:11:37,504 [INFO] tensorflow: epoch = 14.28497409326425, learning_rate = 0.00049999997, loss = 0.00013103479, step = 5514 (5.209 sec) 2021-10-19 07:11:39,534 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.908 INFO:tensorflow:epoch = 14.352331606217618, learning_rate = 0.00049999997, loss = 0.00011272935, step = 5540 (5.324 sec) 2021-10-19 07:11:42,828 [INFO] tensorflow: epoch = 14.352331606217618, learning_rate = 0.00049999997, loss = 0.00011272935, step = 5540 (5.324 sec) INFO:tensorflow:global_step/sec: 4.82172 2021-10-19 07:11:44,568 [INFO] tensorflow: global_step/sec: 4.82172 2021-10-19 07:11:44,782 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.065 INFO:tensorflow:epoch = 14.419689119170984, learning_rate = 0.00049999997, loss = 0.000106705964, step = 5566 (5.311 sec) 2021-10-19 07:11:48,139 [INFO] tensorflow: epoch = 14.419689119170984, learning_rate = 0.00049999997, loss = 0.000106705964, step = 5566 (5.311 sec) 2021-10-19 07:11:49,727 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.226 INFO:tensorflow:global_step/sec: 4.96346 2021-10-19 07:11:52,224 [INFO] tensorflow: global_step/sec: 4.96346 INFO:tensorflow:epoch = 14.487046632124352, learning_rate = 0.00049999997, loss = 9.347133e-05, step = 5592 (5.378 sec) 2021-10-19 07:11:53,517 [INFO] tensorflow: epoch = 14.487046632124352, learning_rate = 0.00049999997, loss = 9.347133e-05, step = 5592 (5.378 sec) 2021-10-19 07:11:54,964 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.098 INFO:tensorflow:epoch = 14.55440414507772, learning_rate = 0.00049999997, loss = 0.00010512964, step = 5618 (5.277 sec) 2021-10-19 07:11:58,794 [INFO] tensorflow: epoch = 14.55440414507772, learning_rate = 0.00049999997, loss = 0.00010512964, step = 5618 (5.277 sec) INFO:tensorflow:global_step/sec: 4.82787 2021-10-19 07:12:00,095 [INFO] tensorflow: global_step/sec: 4.82787 2021-10-19 07:12:00,097 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.484 INFO:tensorflow:epoch = 14.619170984455959, learning_rate = 0.00049999997, loss = 8.952267e-05, step = 5643 (5.303 sec) 2021-10-19 07:12:04,097 [INFO] tensorflow: epoch = 14.619170984455959, learning_rate = 0.00049999997, loss = 8.952267e-05, step = 5643 (5.303 sec) 2021-10-19 07:12:05,393 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.884 INFO:tensorflow:global_step/sec: 4.75875 2021-10-19 07:12:08,081 [INFO] tensorflow: global_step/sec: 4.75875 INFO:tensorflow:epoch = 14.686528497409327, learning_rate = 0.00049999997, loss = 8.537149e-05, step = 5669 (5.424 sec) 2021-10-19 07:12:09,520 [INFO] tensorflow: epoch = 14.686528497409327, learning_rate = 0.00049999997, loss = 8.537149e-05, step = 5669 (5.424 sec) 2021-10-19 07:12:10,577 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.292 INFO:tensorflow:epoch = 14.753886010362695, learning_rate = 0.00049999997, loss = 8.051534e-05, step = 5695 (5.307 sec) 2021-10-19 07:12:14,828 [INFO] tensorflow: epoch = 14.753886010362695, learning_rate = 0.00049999997, loss = 8.051534e-05, step = 5695 (5.307 sec) 2021-10-19 07:12:15,639 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.759 INFO:tensorflow:global_step/sec: 4.89447 2021-10-19 07:12:15,845 [INFO] tensorflow: global_step/sec: 4.89447 INFO:tensorflow:epoch = 14.821243523316063, learning_rate = 0.00049999997, loss = 9.165757e-05, step = 5721 (5.322 sec) 2021-10-19 07:12:20,150 [INFO] tensorflow: epoch = 14.821243523316063, learning_rate = 0.00049999997, loss = 9.165757e-05, step = 5721 (5.322 sec) 2021-10-19 07:12:20,836 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.243 INFO:tensorflow:global_step/sec: 4.77636 2021-10-19 07:12:23,800 [INFO] tensorflow: global_step/sec: 4.77636 INFO:tensorflow:epoch = 14.886010362694302, learning_rate = 0.00049999997, loss = 0.000103051796, step = 5746 (5.277 sec) 2021-10-19 07:12:25,427 [INFO] tensorflow: epoch = 14.886010362694302, learning_rate = 0.00049999997, loss = 0.000103051796, step = 5746 (5.277 sec) 2021-10-19 07:12:26,038 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.227 INFO:tensorflow:epoch = 14.953367875647668, learning_rate = 0.00049999997, loss = 7.690147e-05, step = 5772 (5.362 sec) 2021-10-19 07:12:30,789 [INFO] tensorflow: epoch = 14.953367875647668, learning_rate = 0.00049999997, loss = 7.690147e-05, step = 5772 (5.362 sec) 2021-10-19 07:12:31,265 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.133 INFO:tensorflow:global_step/sec: 4.81035 2021-10-19 07:12:31,700 [INFO] tensorflow: global_step/sec: 4.81035 2021-10-19 07:12:34,610 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 15/120: loss: 0.00008 learning rate: 0.00050 Time taken: 0:01:19.454887 ETA: 2:19:02.763151 INFO:tensorflow:epoch = 15.018134715025907, learning_rate = 0.00049999997, loss = 7.9251215e-05, step = 5797 (5.205 sec) 2021-10-19 07:12:35,994 [INFO] tensorflow: epoch = 15.018134715025907, learning_rate = 0.00049999997, loss = 7.9251215e-05, step = 5797 (5.205 sec) 2021-10-19 07:12:36,421 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.396 INFO:tensorflow:global_step/sec: 4.8155 2021-10-19 07:12:39,591 [INFO] tensorflow: global_step/sec: 4.8155 INFO:tensorflow:epoch = 15.082901554404145, learning_rate = 0.00049999997, loss = 7.882522e-05, step = 5822 (5.325 sec) 2021-10-19 07:12:41,320 [INFO] tensorflow: epoch = 15.082901554404145, learning_rate = 0.00049999997, loss = 7.882522e-05, step = 5822 (5.325 sec) 2021-10-19 07:12:41,726 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.851 INFO:tensorflow:epoch = 15.150259067357513, learning_rate = 0.00049999997, loss = 7.174941e-05, step = 5848 (5.281 sec) 2021-10-19 07:12:46,601 [INFO] tensorflow: epoch = 15.150259067357513, learning_rate = 0.00049999997, loss = 7.174941e-05, step = 5848 (5.281 sec) 2021-10-19 07:12:46,799 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.717 INFO:tensorflow:global_step/sec: 4.84552 2021-10-19 07:12:47,433 [INFO] tensorflow: global_step/sec: 4.84552 2021-10-19 07:12:51,781 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.074 INFO:tensorflow:epoch = 15.220207253886011, learning_rate = 0.00049999997, loss = 9.821234e-05, step = 5875 (5.376 sec) 2021-10-19 07:12:51,977 [INFO] tensorflow: epoch = 15.220207253886011, learning_rate = 0.00049999997, loss = 9.821234e-05, step = 5875 (5.376 sec) INFO:tensorflow:global_step/sec: 5.03872 2021-10-19 07:12:54,975 [INFO] tensorflow: global_step/sec: 5.03872 2021-10-19 07:12:56,772 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.037 INFO:tensorflow:epoch = 15.290155440414509, learning_rate = 0.00049999997, loss = 0.00012542943, step = 5902 (5.404 sec) 2021-10-19 07:12:57,380 [INFO] tensorflow: epoch = 15.290155440414509, learning_rate = 0.00049999997, loss = 0.00012542943, step = 5902 (5.404 sec) 2021-10-19 07:13:01,797 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.903 INFO:tensorflow:epoch = 15.357512953367877, learning_rate = 0.00049999997, loss = 9.135129e-05, step = 5928 (5.207 sec) 2021-10-19 07:13:02,587 [INFO] tensorflow: epoch = 15.357512953367877, learning_rate = 0.00049999997, loss = 9.135129e-05, step = 5928 (5.207 sec) INFO:tensorflow:global_step/sec: 4.99137 2021-10-19 07:13:02,588 [INFO] tensorflow: global_step/sec: 4.99137 2021-10-19 07:13:06,768 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.118 INFO:tensorflow:epoch = 15.427461139896373, learning_rate = 0.00049999997, loss = 6.591003e-05, step = 5955 (5.381 sec) 2021-10-19 07:13:07,968 [INFO] tensorflow: epoch = 15.427461139896373, learning_rate = 0.00049999997, loss = 6.591003e-05, step = 5955 (5.381 sec) INFO:tensorflow:global_step/sec: 4.97204 2021-10-19 07:13:10,231 [INFO] tensorflow: global_step/sec: 4.97204 2021-10-19 07:13:11,826 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.772 INFO:tensorflow:epoch = 15.494818652849741, learning_rate = 0.00049999997, loss = 6.761087e-05, step = 5981 (5.255 sec) 2021-10-19 07:13:13,223 [INFO] tensorflow: epoch = 15.494818652849741, learning_rate = 0.00049999997, loss = 6.761087e-05, step = 5981 (5.255 sec) 2021-10-19 07:13:16,809 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.068 INFO:tensorflow:global_step/sec: 5.02762 2021-10-19 07:13:17,789 [INFO] tensorflow: global_step/sec: 5.02762 INFO:tensorflow:epoch = 15.564766839378239, learning_rate = 0.00049999997, loss = 5.6632198e-05, step = 6008 (5.352 sec) 2021-10-19 07:13:18,575 [INFO] tensorflow: epoch = 15.564766839378239, learning_rate = 0.00049999997, loss = 5.6632198e-05, step = 6008 (5.352 sec) 2021-10-19 07:13:21,809 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.001 INFO:tensorflow:epoch = 15.632124352331607, learning_rate = 0.00049999997, loss = 7.463238e-05, step = 6034 (5.218 sec) 2021-10-19 07:13:23,793 [INFO] tensorflow: epoch = 15.632124352331607, learning_rate = 0.00049999997, loss = 7.463238e-05, step = 6034 (5.218 sec) INFO:tensorflow:global_step/sec: 5.02091 2021-10-19 07:13:25,358 [INFO] tensorflow: global_step/sec: 5.02091 2021-10-19 07:13:26,772 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.150 INFO:tensorflow:epoch = 15.702072538860104, learning_rate = 0.00049999997, loss = 8.621083e-05, step = 6061 (5.362 sec) 2021-10-19 07:13:29,155 [INFO] tensorflow: epoch = 15.702072538860104, learning_rate = 0.00049999997, loss = 8.621083e-05, step = 6061 (5.362 sec) 2021-10-19 07:13:31,787 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.942 INFO:tensorflow:global_step/sec: 4.97633 2021-10-19 07:13:32,994 [INFO] tensorflow: global_step/sec: 4.97633 INFO:tensorflow:epoch = 15.769430051813472, learning_rate = 0.00049999997, loss = 7.342028e-05, step = 6087 (5.267 sec) 2021-10-19 07:13:34,421 [INFO] tensorflow: epoch = 15.769430051813472, learning_rate = 0.00049999997, loss = 7.342028e-05, step = 6087 (5.267 sec) 2021-10-19 07:13:36,849 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.759 INFO:tensorflow:epoch = 15.83678756476684, learning_rate = 0.00049999997, loss = 6.8231966e-05, step = 6113 (5.204 sec) 2021-10-19 07:13:39,625 [INFO] tensorflow: epoch = 15.83678756476684, learning_rate = 0.00049999997, loss = 6.8231966e-05, step = 6113 (5.204 sec) INFO:tensorflow:global_step/sec: 4.99159 2021-10-19 07:13:40,607 [INFO] tensorflow: global_step/sec: 4.99159 2021-10-19 07:13:41,826 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.093 INFO:tensorflow:epoch = 15.904145077720209, learning_rate = 0.00049999997, loss = 0.000103091326, step = 6139 (5.288 sec) 2021-10-19 07:13:44,913 [INFO] tensorflow: epoch = 15.904145077720209, learning_rate = 0.00049999997, loss = 0.000103091326, step = 6139 (5.288 sec) 2021-10-19 07:13:46,915 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.653 INFO:tensorflow:global_step/sec: 4.90759 2021-10-19 07:13:48,350 [INFO] tensorflow: global_step/sec: 4.90759 INFO:tensorflow:epoch = 15.971502590673575, learning_rate = 0.00049999997, loss = 5.2697556e-05, step = 6165 (5.234 sec) 2021-10-19 07:13:50,147 [INFO] tensorflow: epoch = 15.971502590673575, learning_rate = 0.00049999997, loss = 5.2697556e-05, step = 6165 (5.234 sec) 2021-10-19 07:13:51,916 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.997 2021-10-19 07:13:52,321 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 16/120: loss: 0.00012 learning rate: 0.00050 Time taken: 0:01:17.702147 ETA: 2:14:41.023289 INFO:tensorflow:epoch = 16.041450777202073, learning_rate = 0.00049999997, loss = 6.942077e-05, step = 6192 (5.357 sec) 2021-10-19 07:13:55,504 [INFO] tensorflow: epoch = 16.041450777202073, learning_rate = 0.00049999997, loss = 6.942077e-05, step = 6192 (5.357 sec) INFO:tensorflow:global_step/sec: 5.03312 2021-10-19 07:13:55,900 [INFO] tensorflow: global_step/sec: 5.03312 2021-10-19 07:13:56,896 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.080 INFO:tensorflow:epoch = 16.111398963730572, learning_rate = 0.00049999997, loss = 6.5859596e-05, step = 6219 (5.388 sec) 2021-10-19 07:14:00,892 [INFO] tensorflow: epoch = 16.111398963730572, learning_rate = 0.00049999997, loss = 6.5859596e-05, step = 6219 (5.388 sec) 2021-10-19 07:14:01,877 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.077 INFO:tensorflow:global_step/sec: 5.02045 2021-10-19 07:14:03,469 [INFO] tensorflow: global_step/sec: 5.02045 INFO:tensorflow:epoch = 16.178756476683937, learning_rate = 0.00049999997, loss = 5.80166e-05, step = 6245 (5.205 sec) 2021-10-19 07:14:06,097 [INFO] tensorflow: epoch = 16.178756476683937, learning_rate = 0.00049999997, loss = 5.80166e-05, step = 6245 (5.205 sec) 2021-10-19 07:14:06,901 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.909 INFO:tensorflow:global_step/sec: 5.01687 2021-10-19 07:14:11,043 [INFO] tensorflow: global_step/sec: 5.01687 INFO:tensorflow:epoch = 16.248704663212436, learning_rate = 0.00049999997, loss = 5.6335182e-05, step = 6272 (5.349 sec) 2021-10-19 07:14:11,445 [INFO] tensorflow: epoch = 16.248704663212436, learning_rate = 0.00049999997, loss = 5.6335182e-05, step = 6272 (5.349 sec) 2021-10-19 07:14:11,868 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.134 INFO:tensorflow:epoch = 16.316062176165804, learning_rate = 0.00049999997, loss = 7.374033e-05, step = 6298 (5.246 sec) 2021-10-19 07:14:16,692 [INFO] tensorflow: epoch = 16.316062176165804, learning_rate = 0.00049999997, loss = 7.374033e-05, step = 6298 (5.246 sec) 2021-10-19 07:14:16,894 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.896 INFO:tensorflow:global_step/sec: 4.9767 2021-10-19 07:14:18,679 [INFO] tensorflow: global_step/sec: 4.9767 INFO:tensorflow:epoch = 16.383419689119172, learning_rate = 0.00049999997, loss = 6.296757e-05, step = 6324 (5.251 sec) 2021-10-19 07:14:21,943 [INFO] tensorflow: epoch = 16.383419689119172, learning_rate = 0.00049999997, loss = 6.296757e-05, step = 6324 (5.251 sec) 2021-10-19 07:14:21,943 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.808 INFO:tensorflow:global_step/sec: 4.99135 2021-10-19 07:14:26,292 [INFO] tensorflow: global_step/sec: 4.99135 2021-10-19 07:14:26,905 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.155 INFO:tensorflow:epoch = 16.45336787564767, learning_rate = 0.00049999997, loss = 6.320995e-05, step = 6351 (5.379 sec) 2021-10-19 07:14:27,322 [INFO] tensorflow: epoch = 16.45336787564767, learning_rate = 0.00049999997, loss = 6.320995e-05, step = 6351 (5.379 sec) 2021-10-19 07:14:31,957 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.795 INFO:tensorflow:epoch = 16.520725388601036, learning_rate = 0.00049999997, loss = 8.827185e-05, step = 6377 (5.264 sec) 2021-10-19 07:14:32,585 [INFO] tensorflow: epoch = 16.520725388601036, learning_rate = 0.00049999997, loss = 8.827185e-05, step = 6377 (5.264 sec) INFO:tensorflow:global_step/sec: 4.90174 2021-10-19 07:14:34,044 [INFO] tensorflow: global_step/sec: 4.90174 2021-10-19 07:14:37,067 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.569 INFO:tensorflow:epoch = 16.588082901554404, learning_rate = 0.00049999997, loss = 5.76256e-05, step = 6403 (5.289 sec) 2021-10-19 07:14:37,874 [INFO] tensorflow: epoch = 16.588082901554404, learning_rate = 0.00049999997, loss = 5.76256e-05, step = 6403 (5.289 sec) INFO:tensorflow:global_step/sec: 5.0061 2021-10-19 07:14:41,635 [INFO] tensorflow: global_step/sec: 5.0061 2021-10-19 07:14:42,038 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.120 INFO:tensorflow:epoch = 16.658031088082904, learning_rate = 0.00049999997, loss = 4.0719053e-05, step = 6430 (5.344 sec) 2021-10-19 07:14:43,219 [INFO] tensorflow: epoch = 16.658031088082904, learning_rate = 0.00049999997, loss = 4.0719053e-05, step = 6430 (5.344 sec) 2021-10-19 07:14:47,069 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.876 INFO:tensorflow:epoch = 16.72538860103627, learning_rate = 0.00049999997, loss = 4.9772207e-05, step = 6456 (5.290 sec) 2021-10-19 07:14:48,509 [INFO] tensorflow: epoch = 16.72538860103627, learning_rate = 0.00049999997, loss = 4.9772207e-05, step = 6456 (5.290 sec) INFO:tensorflow:global_step/sec: 4.94035 2021-10-19 07:14:49,327 [INFO] tensorflow: global_step/sec: 4.94035 2021-10-19 07:14:52,091 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.914 INFO:tensorflow:epoch = 16.795336787564768, learning_rate = 0.00049999997, loss = 4.978728e-05, step = 6483 (5.386 sec) 2021-10-19 07:14:53,894 [INFO] tensorflow: epoch = 16.795336787564768, learning_rate = 0.00049999997, loss = 4.978728e-05, step = 6483 (5.386 sec) INFO:tensorflow:global_step/sec: 5.05417 2021-10-19 07:14:56,845 [INFO] tensorflow: global_step/sec: 5.05417 2021-10-19 07:14:57,049 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.174 INFO:tensorflow:epoch = 16.865284974093264, learning_rate = 0.00049999997, loss = 5.1437666e-05, step = 6510 (5.381 sec) 2021-10-19 07:14:59,275 [INFO] tensorflow: epoch = 16.865284974093264, learning_rate = 0.00049999997, loss = 5.1437666e-05, step = 6510 (5.381 sec) 2021-10-19 07:15:02,082 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.871 INFO:tensorflow:epoch = 16.932642487046632, learning_rate = 0.00049999997, loss = 5.0183902e-05, step = 6536 (5.216 sec) 2021-10-19 07:15:04,490 [INFO] tensorflow: epoch = 16.932642487046632, learning_rate = 0.00049999997, loss = 5.0183902e-05, step = 6536 (5.216 sec) INFO:tensorflow:global_step/sec: 4.96967 2021-10-19 07:15:04,492 [INFO] tensorflow: global_step/sec: 4.96967 2021-10-19 07:15:07,096 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.945 INFO:tensorflow:epoch = 17.0, learning_rate = 0.00049999997, loss = 5.9248065e-05, step = 6562 (5.295 sec) 2021-10-19 07:15:09,786 [INFO] tensorflow: epoch = 17.0, learning_rate = 0.00049999997, loss = 5.9248065e-05, step = 6562 (5.295 sec) 2021-10-19 07:15:09,786 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 17/120: loss: 0.00006 learning rate: 0.00050 Time taken: 0:01:17.462772 ETA: 2:12:58.665480 INFO:tensorflow:global_step/sec: 4.92362 2021-10-19 07:15:12,209 [INFO] tensorflow: global_step/sec: 4.92362 2021-10-19 07:15:12,211 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.554 INFO:tensorflow:epoch = 17.0699481865285, learning_rate = 0.00049999997, loss = 6.612395e-05, step = 6589 (5.391 sec) 2021-10-19 07:15:15,176 [INFO] tensorflow: epoch = 17.0699481865285, learning_rate = 0.00049999997, loss = 6.612395e-05, step = 6589 (5.391 sec) 2021-10-19 07:15:17,209 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.009 INFO:tensorflow:global_step/sec: 5.00696 2021-10-19 07:15:19,799 [INFO] tensorflow: global_step/sec: 5.00696 INFO:tensorflow:epoch = 17.137305699481868, learning_rate = 0.00049999997, loss = 5.636763e-05, step = 6615 (5.235 sec) 2021-10-19 07:15:20,412 [INFO] tensorflow: epoch = 17.137305699481868, learning_rate = 0.00049999997, loss = 5.636763e-05, step = 6615 (5.235 sec) 2021-10-19 07:15:22,209 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.999 INFO:tensorflow:epoch = 17.207253886010363, learning_rate = 0.00049999997, loss = 5.8546793e-05, step = 6642 (5.379 sec) 2021-10-19 07:15:25,791 [INFO] tensorflow: epoch = 17.207253886010363, learning_rate = 0.00049999997, loss = 5.8546793e-05, step = 6642 (5.379 sec) 2021-10-19 07:15:27,184 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.103 INFO:tensorflow:global_step/sec: 5.0046 2021-10-19 07:15:27,392 [INFO] tensorflow: global_step/sec: 5.0046 INFO:tensorflow:epoch = 17.27461139896373, learning_rate = 0.00049999997, loss = 5.9866994e-05, step = 6668 (5.203 sec) 2021-10-19 07:15:30,994 [INFO] tensorflow: epoch = 17.27461139896373, learning_rate = 0.00049999997, loss = 5.9866994e-05, step = 6668 (5.203 sec) 2021-10-19 07:15:32,171 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.056 INFO:tensorflow:global_step/sec: 5.00537 2021-10-19 07:15:34,984 [INFO] tensorflow: global_step/sec: 5.00537 INFO:tensorflow:epoch = 17.3419689119171, learning_rate = 0.00049999997, loss = 3.5117744e-05, step = 6694 (5.201 sec) 2021-10-19 07:15:36,195 [INFO] tensorflow: epoch = 17.3419689119171, learning_rate = 0.00049999997, loss = 3.5117744e-05, step = 6694 (5.201 sec) 2021-10-19 07:15:37,200 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.887 INFO:tensorflow:epoch = 17.411917098445596, learning_rate = 0.00049999997, loss = 5.6976372e-05, step = 6721 (5.365 sec) 2021-10-19 07:15:41,560 [INFO] tensorflow: epoch = 17.411917098445596, learning_rate = 0.00049999997, loss = 5.6976372e-05, step = 6721 (5.365 sec) 2021-10-19 07:15:42,196 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.014 INFO:tensorflow:global_step/sec: 4.98878 2021-10-19 07:15:42,601 [INFO] tensorflow: global_step/sec: 4.98878 INFO:tensorflow:epoch = 17.479274611398964, learning_rate = 0.00049999997, loss = 5.323951e-05, step = 6747 (5.253 sec) 2021-10-19 07:15:46,814 [INFO] tensorflow: epoch = 17.479274611398964, learning_rate = 0.00049999997, loss = 5.323951e-05, step = 6747 (5.253 sec) 2021-10-19 07:15:47,229 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.872 INFO:tensorflow:global_step/sec: 4.95213 2021-10-19 07:15:50,274 [INFO] tensorflow: global_step/sec: 4.95213 INFO:tensorflow:epoch = 17.54663212435233, learning_rate = 0.00049999997, loss = 6.37633e-05, step = 6773 (5.267 sec) 2021-10-19 07:15:52,080 [INFO] tensorflow: epoch = 17.54663212435233, learning_rate = 0.00049999997, loss = 6.37633e-05, step = 6773 (5.267 sec) 2021-10-19 07:15:52,302 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.712 2021-10-19 07:15:57,235 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.274 INFO:tensorflow:epoch = 17.61658031088083, learning_rate = 0.00049999997, loss = 3.834135e-05, step = 6800 (5.359 sec) 2021-10-19 07:15:57,439 [INFO] tensorflow: epoch = 17.61658031088083, learning_rate = 0.00049999997, loss = 3.834135e-05, step = 6800 (5.359 sec) INFO:tensorflow:global_step/sec: 5.02028 2021-10-19 07:15:57,844 [INFO] tensorflow: global_step/sec: 5.02028 2021-10-19 07:16:02,244 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.966 INFO:tensorflow:epoch = 17.6839378238342, learning_rate = 0.00049999997, loss = 4.5225002e-05, step = 6826 (5.202 sec) 2021-10-19 07:16:02,641 [INFO] tensorflow: epoch = 17.6839378238342, learning_rate = 0.00049999997, loss = 4.5225002e-05, step = 6826 (5.202 sec) INFO:tensorflow:global_step/sec: 4.98995 2021-10-19 07:16:05,459 [INFO] tensorflow: global_step/sec: 4.98995 2021-10-19 07:16:07,239 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.022 INFO:tensorflow:epoch = 17.751295336787564, learning_rate = 0.00049999997, loss = 4.398468e-05, step = 6852 (5.222 sec) 2021-10-19 07:16:07,862 [INFO] tensorflow: epoch = 17.751295336787564, learning_rate = 0.00049999997, loss = 4.398468e-05, step = 6852 (5.222 sec) 2021-10-19 07:16:12,218 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.087 INFO:tensorflow:global_step/sec: 5.02625 2021-10-19 07:16:13,019 [INFO] tensorflow: global_step/sec: 5.02625 INFO:tensorflow:epoch = 17.821243523316063, learning_rate = 0.00049999997, loss = 6.179405e-05, step = 6879 (5.360 sec) 2021-10-19 07:16:13,222 [INFO] tensorflow: epoch = 17.821243523316063, learning_rate = 0.00049999997, loss = 6.179405e-05, step = 6879 (5.360 sec) 2021-10-19 07:16:17,134 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.345 INFO:tensorflow:epoch = 17.89119170984456, learning_rate = 0.00049999997, loss = 5.7779893e-05, step = 6906 (5.288 sec) 2021-10-19 07:16:18,510 [INFO] tensorflow: epoch = 17.89119170984456, learning_rate = 0.00049999997, loss = 5.7779893e-05, step = 6906 (5.288 sec) INFO:tensorflow:global_step/sec: 5.09074 2021-10-19 07:16:20,484 [INFO] tensorflow: global_step/sec: 5.09074 2021-10-19 07:16:22,110 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.099 INFO:tensorflow:epoch = 17.96113989637306, learning_rate = 0.00049999997, loss = 4.1977422e-05, step = 6933 (5.383 sec) 2021-10-19 07:16:23,894 [INFO] tensorflow: epoch = 17.96113989637306, learning_rate = 0.00049999997, loss = 4.1977422e-05, step = 6933 (5.383 sec) 2021-10-19 07:16:26,916 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 18/120: loss: 0.00006 learning rate: 0.00050 Time taken: 0:01:17.134470 ETA: 2:11:07.715987 2021-10-19 07:16:27,116 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.978 INFO:tensorflow:global_step/sec: 4.99595 2021-10-19 07:16:28,090 [INFO] tensorflow: global_step/sec: 4.99595 INFO:tensorflow:epoch = 18.031088082901555, learning_rate = 0.00049999997, loss = 4.5121415e-05, step = 6960 (5.395 sec) 2021-10-19 07:16:29,289 [INFO] tensorflow: epoch = 18.031088082901555, learning_rate = 0.00049999997, loss = 4.5121415e-05, step = 6960 (5.395 sec) 2021-10-19 07:16:32,085 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.126 INFO:tensorflow:epoch = 18.101036269430054, learning_rate = 0.00049999997, loss = 3.7433972e-05, step = 6987 (5.382 sec) 2021-10-19 07:16:34,671 [INFO] tensorflow: epoch = 18.101036269430054, learning_rate = 0.00049999997, loss = 3.7433972e-05, step = 6987 (5.382 sec) INFO:tensorflow:global_step/sec: 5.02886 2021-10-19 07:16:35,646 [INFO] tensorflow: global_step/sec: 5.02886 2021-10-19 07:16:37,031 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.217 INFO:tensorflow:epoch = 18.17098445595855, learning_rate = 0.00049999997, loss = 6.76813e-05, step = 7014 (5.396 sec) 2021-10-19 07:16:40,066 [INFO] tensorflow: epoch = 18.17098445595855, learning_rate = 0.00049999997, loss = 6.76813e-05, step = 7014 (5.396 sec) 2021-10-19 07:16:42,108 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.697 INFO:tensorflow:global_step/sec: 4.93204 2021-10-19 07:16:43,351 [INFO] tensorflow: global_step/sec: 4.93204 INFO:tensorflow:epoch = 18.238341968911918, learning_rate = 0.00049999997, loss = 4.9302274e-05, step = 7040 (5.272 sec) 2021-10-19 07:16:45,338 [INFO] tensorflow: epoch = 18.238341968911918, learning_rate = 0.00049999997, loss = 4.9302274e-05, step = 7040 (5.272 sec) 2021-10-19 07:16:47,132 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.910 INFO:tensorflow:epoch = 18.308290155440414, learning_rate = 0.00049999997, loss = 6.055971e-05, step = 7067 (5.363 sec) 2021-10-19 07:16:50,702 [INFO] tensorflow: epoch = 18.308290155440414, learning_rate = 0.00049999997, loss = 6.055971e-05, step = 7067 (5.363 sec) INFO:tensorflow:global_step/sec: 5.02664 2021-10-19 07:16:50,911 [INFO] tensorflow: global_step/sec: 5.02664 2021-10-19 07:16:52,098 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.138 INFO:tensorflow:epoch = 18.378238341968913, learning_rate = 0.00049999997, loss = 4.6428995e-05, step = 7094 (5.325 sec) 2021-10-19 07:16:56,027 [INFO] tensorflow: epoch = 18.378238341968913, learning_rate = 0.00049999997, loss = 4.6428995e-05, step = 7094 (5.325 sec) 2021-10-19 07:16:57,023 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.307 INFO:tensorflow:global_step/sec: 5.05312 2021-10-19 07:16:58,431 [INFO] tensorflow: global_step/sec: 5.05312 INFO:tensorflow:epoch = 18.44559585492228, learning_rate = 0.00049999997, loss = 5.3307038e-05, step = 7120 (5.205 sec) 2021-10-19 07:17:01,231 [INFO] tensorflow: epoch = 18.44559585492228, learning_rate = 0.00049999997, loss = 5.3307038e-05, step = 7120 (5.205 sec) 2021-10-19 07:17:02,021 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.010 INFO:tensorflow:global_step/sec: 5.0559 2021-10-19 07:17:05,947 [INFO] tensorflow: global_step/sec: 5.0559 INFO:tensorflow:epoch = 18.515544041450777, learning_rate = 0.00049999997, loss = 3.8540813e-05, step = 7147 (5.343 sec) 2021-10-19 07:17:06,575 [INFO] tensorflow: epoch = 18.515544041450777, learning_rate = 0.00049999997, loss = 3.8540813e-05, step = 7147 (5.343 sec) 2021-10-19 07:17:06,994 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.107 INFO:tensorflow:epoch = 18.582901554404145, learning_rate = 0.00049999997, loss = 6.954506e-05, step = 7173 (5.240 sec) 2021-10-19 07:17:11,814 [INFO] tensorflow: epoch = 18.582901554404145, learning_rate = 0.00049999997, loss = 6.954506e-05, step = 7173 (5.240 sec) 2021-10-19 07:17:12,029 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.863 INFO:tensorflow:global_step/sec: 4.958 2021-10-19 07:17:13,611 [INFO] tensorflow: global_step/sec: 4.958 2021-10-19 07:17:16,928 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.415 INFO:tensorflow:epoch = 18.652849740932645, learning_rate = 0.00049999997, loss = 4.5217916e-05, step = 7200 (5.320 sec) 2021-10-19 07:17:17,134 [INFO] tensorflow: epoch = 18.652849740932645, learning_rate = 0.00049999997, loss = 4.5217916e-05, step = 7200 (5.320 sec) INFO:tensorflow:global_step/sec: 5.11046 2021-10-19 07:17:21,047 [INFO] tensorflow: global_step/sec: 5.11046 2021-10-19 07:17:21,872 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.228 INFO:tensorflow:epoch = 18.72279792746114, learning_rate = 0.00049999997, loss = 3.0159488e-05, step = 7227 (5.358 sec) 2021-10-19 07:17:22,492 [INFO] tensorflow: epoch = 18.72279792746114, learning_rate = 0.00049999997, loss = 3.0159488e-05, step = 7227 (5.358 sec) 2021-10-19 07:17:26,891 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.928 INFO:tensorflow:epoch = 18.792746113989637, learning_rate = 0.00049999997, loss = 4.2722295e-05, step = 7254 (5.379 sec) 2021-10-19 07:17:27,871 [INFO] tensorflow: epoch = 18.792746113989637, learning_rate = 0.00049999997, loss = 4.2722295e-05, step = 7254 (5.379 sec) INFO:tensorflow:global_step/sec: 4.97592 2021-10-19 07:17:28,684 [INFO] tensorflow: global_step/sec: 4.97592 2021-10-19 07:17:31,861 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.122 INFO:tensorflow:epoch = 18.860103626943005, learning_rate = 0.00049999997, loss = 4.0027473e-05, step = 7280 (5.193 sec) 2021-10-19 07:17:33,064 [INFO] tensorflow: epoch = 18.860103626943005, learning_rate = 0.00049999997, loss = 4.0027473e-05, step = 7280 (5.193 sec) INFO:tensorflow:global_step/sec: 5.0072 2021-10-19 07:17:36,273 [INFO] tensorflow: global_step/sec: 5.0072 2021-10-19 07:17:36,900 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.844 INFO:tensorflow:epoch = 18.927461139896373, learning_rate = 0.00049999997, loss = 4.5502456e-05, step = 7306 (5.243 sec) 2021-10-19 07:17:38,307 [INFO] tensorflow: epoch = 18.927461139896373, learning_rate = 0.00049999997, loss = 4.5502456e-05, step = 7306 (5.243 sec) 2021-10-19 07:17:41,939 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.851 INFO:tensorflow:epoch = 18.99481865284974, learning_rate = 0.00049999997, loss = 3.1642143e-05, step = 7332 (5.267 sec) 2021-10-19 07:17:43,574 [INFO] tensorflow: epoch = 18.99481865284974, learning_rate = 0.00049999997, loss = 3.1642143e-05, step = 7332 (5.267 sec) INFO:tensorflow:global_step/sec: 4.92096 2021-10-19 07:17:43,995 [INFO] tensorflow: global_step/sec: 4.92096 2021-10-19 07:17:43,996 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 19/120: loss: 0.00004 learning rate: 0.00050 Time taken: 0:01:17.075088 ETA: 2:09:44.583890 2021-10-19 07:17:47,011 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.718 INFO:tensorflow:epoch = 19.06217616580311, learning_rate = 0.00049999997, loss = 3.9385326e-05, step = 7358 (5.266 sec) 2021-10-19 07:17:48,840 [INFO] tensorflow: epoch = 19.06217616580311, learning_rate = 0.00049999997, loss = 3.9385326e-05, step = 7358 (5.266 sec) INFO:tensorflow:global_step/sec: 4.99688 2021-10-19 07:17:51,600 [INFO] tensorflow: global_step/sec: 4.99688 2021-10-19 07:17:51,994 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.071 INFO:tensorflow:epoch = 19.13212435233161, learning_rate = 0.00049999997, loss = 4.7244343e-05, step = 7385 (5.364 sec) 2021-10-19 07:17:54,204 [INFO] tensorflow: epoch = 19.13212435233161, learning_rate = 0.00049999997, loss = 4.7244343e-05, step = 7385 (5.364 sec) 2021-10-19 07:17:56,954 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.160 INFO:tensorflow:global_step/sec: 5.05224 2021-10-19 07:17:59,121 [INFO] tensorflow: global_step/sec: 5.05224 INFO:tensorflow:epoch = 19.202072538860104, learning_rate = 0.00049999997, loss = 4.0486942e-05, step = 7412 (5.316 sec) 2021-10-19 07:17:59,520 [INFO] tensorflow: epoch = 19.202072538860104, learning_rate = 0.00049999997, loss = 4.0486942e-05, step = 7412 (5.316 sec) 2021-10-19 07:18:01,861 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.383 INFO:tensorflow:epoch = 19.2720207253886, learning_rate = 0.00049999997, loss = 4.624436e-05, step = 7439 (5.353 sec) 2021-10-19 07:18:04,873 [INFO] tensorflow: epoch = 19.2720207253886, learning_rate = 0.00049999997, loss = 4.624436e-05, step = 7439 (5.353 sec) INFO:tensorflow:global_step/sec: 5.05833 2021-10-19 07:18:06,633 [INFO] tensorflow: global_step/sec: 5.05833 2021-10-19 07:18:06,841 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.083 INFO:tensorflow:epoch = 19.33937823834197, learning_rate = 0.00049999997, loss = 3.4181932e-05, step = 7465 (5.243 sec) 2021-10-19 07:18:10,116 [INFO] tensorflow: epoch = 19.33937823834197, learning_rate = 0.00049999997, loss = 3.4181932e-05, step = 7465 (5.243 sec) 2021-10-19 07:18:11,881 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.840 INFO:tensorflow:global_step/sec: 4.99781 2021-10-19 07:18:14,237 [INFO] tensorflow: global_step/sec: 4.99781 INFO:tensorflow:epoch = 19.409326424870468, learning_rate = 0.00049999997, loss = 2.8451057e-05, step = 7492 (5.328 sec) 2021-10-19 07:18:15,445 [INFO] tensorflow: epoch = 19.409326424870468, learning_rate = 0.00049999997, loss = 2.8451057e-05, step = 7492 (5.328 sec) 2021-10-19 07:18:16,816 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.267 INFO:tensorflow:epoch = 19.479274611398964, learning_rate = 0.00049999997, loss = 5.542268e-05, step = 7519 (5.323 sec) 2021-10-19 07:18:20,768 [INFO] tensorflow: epoch = 19.479274611398964, learning_rate = 0.00049999997, loss = 5.542268e-05, step = 7519 (5.323 sec) INFO:tensorflow:global_step/sec: 5.04256 2021-10-19 07:18:21,773 [INFO] tensorflow: global_step/sec: 5.04256 2021-10-19 07:18:21,774 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.172 INFO:tensorflow:epoch = 19.54663212435233, learning_rate = 0.00049999997, loss = 2.7798184e-05, step = 7545 (5.258 sec) 2021-10-19 07:18:26,026 [INFO] tensorflow: epoch = 19.54663212435233, learning_rate = 0.00049999997, loss = 2.7798184e-05, step = 7545 (5.258 sec) 2021-10-19 07:18:26,817 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.828 INFO:tensorflow:global_step/sec: 4.97271 2021-10-19 07:18:29,414 [INFO] tensorflow: global_step/sec: 4.97271 INFO:tensorflow:epoch = 19.61658031088083, learning_rate = 0.00049999997, loss = 4.129744e-05, step = 7572 (5.388 sec) 2021-10-19 07:18:31,414 [INFO] tensorflow: epoch = 19.61658031088083, learning_rate = 0.00049999997, loss = 4.129744e-05, step = 7572 (5.388 sec) 2021-10-19 07:18:31,829 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.957 INFO:tensorflow:epoch = 19.686528497409327, learning_rate = 0.00049999997, loss = 5.796604e-05, step = 7599 (5.366 sec) 2021-10-19 07:18:36,780 [INFO] tensorflow: epoch = 19.686528497409327, learning_rate = 0.00049999997, loss = 5.796604e-05, step = 7599 (5.366 sec) 2021-10-19 07:18:36,780 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.198 INFO:tensorflow:global_step/sec: 5.01683 2021-10-19 07:18:36,989 [INFO] tensorflow: global_step/sec: 5.01683 2021-10-19 07:18:41,765 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.062 INFO:tensorflow:epoch = 19.756476683937823, learning_rate = 0.00049999997, loss = 7.813958e-05, step = 7626 (5.392 sec) 2021-10-19 07:18:42,171 [INFO] tensorflow: epoch = 19.756476683937823, learning_rate = 0.00049999997, loss = 7.813958e-05, step = 7626 (5.392 sec) INFO:tensorflow:global_step/sec: 5.04163 2021-10-19 07:18:44,526 [INFO] tensorflow: global_step/sec: 5.04163 2021-10-19 07:18:46,717 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.195 INFO:tensorflow:epoch = 19.826424870466322, learning_rate = 0.00049999997, loss = 3.3814777e-05, step = 7653 (5.370 sec) 2021-10-19 07:18:47,541 [INFO] tensorflow: epoch = 19.826424870466322, learning_rate = 0.00049999997, loss = 3.3814777e-05, step = 7653 (5.370 sec) 2021-10-19 07:18:51,762 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.825 INFO:tensorflow:global_step/sec: 4.97507 2021-10-19 07:18:52,164 [INFO] tensorflow: global_step/sec: 4.97507 INFO:tensorflow:epoch = 19.89378238341969, learning_rate = 0.00049999997, loss = 8.5249456e-05, step = 7679 (5.241 sec) 2021-10-19 07:18:52,782 [INFO] tensorflow: epoch = 19.89378238341969, learning_rate = 0.00049999997, loss = 8.5249456e-05, step = 7679 (5.241 sec) 2021-10-19 07:18:56,800 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.850 INFO:tensorflow:epoch = 19.96113989637306, learning_rate = 0.00049999997, loss = 3.2911877e-05, step = 7705 (5.275 sec) 2021-10-19 07:18:58,057 [INFO] tensorflow: epoch = 19.96113989637306, learning_rate = 0.00049999997, loss = 3.2911877e-05, step = 7705 (5.275 sec) INFO:tensorflow:global_step/sec: 4.95226 2021-10-19 07:18:59,837 [INFO] tensorflow: global_step/sec: 4.95226 INFO:tensorflow:Saving checkpoints for step-7720. 2021-10-19 07:19:00,884 [INFO] tensorflow: Saving checkpoints for step-7720. INFO:tensorflow:epoch = 20.0, learning_rate = 0.00049999997, loss = 4.4912016e-05, step = 7720 (7.668 sec) 2021-10-19 07:19:05,725 [INFO] tensorflow: epoch = 20.0, learning_rate = 0.00049999997, loss = 4.4912016e-05, step = 7720 (7.668 sec) 2021-10-19 07:19:05,725 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 20/120: loss: 0.00004 learning rate: 0.00050 Time taken: 0:01:21.718894 ETA: 2:16:11.889424 2021-10-19 07:19:06,509 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 10.300 INFO:tensorflow:epoch = 20.0699481865285, learning_rate = 0.00049999997, loss = 3.651619e-05, step = 7747 (5.376 sec) 2021-10-19 07:19:11,101 [INFO] tensorflow: epoch = 20.0699481865285, learning_rate = 0.00049999997, loss = 3.651619e-05, step = 7747 (5.376 sec) 2021-10-19 07:19:11,522 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.949 INFO:tensorflow:global_step/sec: 3.09107 2021-10-19 07:19:12,131 [INFO] tensorflow: global_step/sec: 3.09107 INFO:tensorflow:epoch = 20.137305699481868, learning_rate = 0.00049999997, loss = 2.609567e-05, step = 7773 (5.238 sec) 2021-10-19 07:19:16,339 [INFO] tensorflow: epoch = 20.137305699481868, learning_rate = 0.00049999997, loss = 2.609567e-05, step = 7773 (5.238 sec) 2021-10-19 07:19:16,547 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.904 INFO:tensorflow:global_step/sec: 4.94243 2021-10-19 07:19:19,819 [INFO] tensorflow: global_step/sec: 4.94243 INFO:tensorflow:epoch = 20.204663212435236, learning_rate = 0.00049999997, loss = 7.062517e-05, step = 7799 (5.281 sec) 2021-10-19 07:19:21,620 [INFO] tensorflow: epoch = 20.204663212435236, learning_rate = 0.00049999997, loss = 7.062517e-05, step = 7799 (5.281 sec) 2021-10-19 07:19:21,621 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.712 2021-10-19 07:19:26,677 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.777 INFO:tensorflow:epoch = 20.2720207253886, learning_rate = 0.00049999997, loss = 3.7883125e-05, step = 7825 (5.299 sec) 2021-10-19 07:19:26,919 [INFO] tensorflow: epoch = 20.2720207253886, learning_rate = 0.00049999997, loss = 3.7883125e-05, step = 7825 (5.299 sec) INFO:tensorflow:global_step/sec: 4.92567 2021-10-19 07:19:27,534 [INFO] tensorflow: global_step/sec: 4.92567 2021-10-19 07:19:31,768 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.644 INFO:tensorflow:epoch = 20.33937823834197, learning_rate = 0.00049999997, loss = 6.282084e-05, step = 7851 (5.255 sec) 2021-10-19 07:19:32,174 [INFO] tensorflow: epoch = 20.33937823834197, learning_rate = 0.00049999997, loss = 6.282084e-05, step = 7851 (5.255 sec) INFO:tensorflow:global_step/sec: 4.95593 2021-10-19 07:19:35,202 [INFO] tensorflow: global_step/sec: 4.95593 2021-10-19 07:19:36,775 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.974 INFO:tensorflow:epoch = 20.406735751295336, learning_rate = 0.00049999997, loss = 6.009402e-05, step = 7877 (5.194 sec) 2021-10-19 07:19:37,368 [INFO] tensorflow: epoch = 20.406735751295336, learning_rate = 0.00049999997, loss = 6.009402e-05, step = 7877 (5.194 sec) 2021-10-19 07:19:41,740 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.146 INFO:tensorflow:epoch = 20.476683937823836, learning_rate = 0.00049999997, loss = 4.1459447e-05, step = 7904 (5.383 sec) 2021-10-19 07:19:42,751 [INFO] tensorflow: epoch = 20.476683937823836, learning_rate = 0.00049999997, loss = 4.1459447e-05, step = 7904 (5.383 sec) INFO:tensorflow:global_step/sec: 5.03267 2021-10-19 07:19:42,752 [INFO] tensorflow: global_step/sec: 5.03267 2021-10-19 07:19:46,880 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.454 INFO:tensorflow:epoch = 20.544041450777204, learning_rate = 0.00049999997, loss = 3.8286453e-05, step = 7930 (5.339 sec) 2021-10-19 07:19:48,090 [INFO] tensorflow: epoch = 20.544041450777204, learning_rate = 0.00049999997, loss = 3.8286453e-05, step = 7930 (5.339 sec) INFO:tensorflow:global_step/sec: 4.8878 2021-10-19 07:19:50,527 [INFO] tensorflow: global_step/sec: 4.8878 2021-10-19 07:19:51,910 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.884 INFO:tensorflow:epoch = 20.611398963730572, learning_rate = 0.00049999997, loss = 3.949675e-05, step = 7956 (5.230 sec) 2021-10-19 07:19:53,321 [INFO] tensorflow: epoch = 20.611398963730572, learning_rate = 0.00049999997, loss = 3.949675e-05, step = 7956 (5.230 sec) 2021-10-19 07:19:56,904 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.025 INFO:tensorflow:global_step/sec: 5.02975 2021-10-19 07:19:58,082 [INFO] tensorflow: global_step/sec: 5.02975 INFO:tensorflow:epoch = 20.681347150259068, learning_rate = 0.00049999997, loss = 3.6242676e-05, step = 7983 (5.362 sec) 2021-10-19 07:19:58,682 [INFO] tensorflow: epoch = 20.681347150259068, learning_rate = 0.00049999997, loss = 3.6242676e-05, step = 7983 (5.362 sec) 2021-10-19 07:20:01,941 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.856 INFO:tensorflow:epoch = 20.748704663212436, learning_rate = 0.00049999997, loss = 3.3613214e-05, step = 8009 (5.214 sec) 2021-10-19 07:20:03,897 [INFO] tensorflow: epoch = 20.748704663212436, learning_rate = 0.00049999997, loss = 3.3613214e-05, step = 8009 (5.214 sec) INFO:tensorflow:global_step/sec: 4.98743 2021-10-19 07:20:05,701 [INFO] tensorflow: global_step/sec: 4.98743 2021-10-19 07:20:06,875 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.268 INFO:tensorflow:epoch = 20.818652849740932, learning_rate = 0.00049999997, loss = 2.902042e-05, step = 8036 (5.377 sec) 2021-10-19 07:20:09,273 [INFO] tensorflow: epoch = 20.818652849740932, learning_rate = 0.00049999997, loss = 2.902042e-05, step = 8036 (5.377 sec) 2021-10-19 07:20:11,859 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.068 INFO:tensorflow:global_step/sec: 5.02455 2021-10-19 07:20:13,264 [INFO] tensorflow: global_step/sec: 5.02455 INFO:tensorflow:epoch = 20.88860103626943, learning_rate = 0.00049999997, loss = 4.1955325e-05, step = 8063 (5.390 sec) 2021-10-19 07:20:14,664 [INFO] tensorflow: epoch = 20.88860103626943, learning_rate = 0.00049999997, loss = 4.1955325e-05, step = 8063 (5.390 sec) 2021-10-19 07:20:16,861 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.993 INFO:tensorflow:epoch = 20.9559585492228, learning_rate = 0.00049999997, loss = 3.67762e-05, step = 8089 (5.217 sec) 2021-10-19 07:20:19,881 [INFO] tensorflow: epoch = 20.9559585492228, learning_rate = 0.00049999997, loss = 3.67762e-05, step = 8089 (5.217 sec) INFO:tensorflow:global_step/sec: 4.99014 2021-10-19 07:20:20,879 [INFO] tensorflow: global_step/sec: 4.99014 2021-10-19 07:20:21,866 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.985 2021-10-19 07:20:23,301 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 21/120: loss: 0.00003 learning rate: 0.00050 Time taken: 0:01:17.563142 ETA: 2:07:58.751088 INFO:tensorflow:epoch = 21.025906735751295, learning_rate = 0.00049999997, loss = 4.8325488e-05, step = 8116 (5.409 sec) 2021-10-19 07:20:25,290 [INFO] tensorflow: epoch = 21.025906735751295, learning_rate = 0.00049999997, loss = 4.8325488e-05, step = 8116 (5.409 sec) 2021-10-19 07:20:26,842 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.098 INFO:tensorflow:global_step/sec: 5.02057 2021-10-19 07:20:28,448 [INFO] tensorflow: global_step/sec: 5.02057 INFO:tensorflow:epoch = 21.095854922279795, learning_rate = 0.00049999997, loss = 3.329489e-05, step = 8143 (5.355 sec) 2021-10-19 07:20:30,645 [INFO] tensorflow: epoch = 21.095854922279795, learning_rate = 0.00049999997, loss = 3.329489e-05, step = 8143 (5.355 sec) 2021-10-19 07:20:31,860 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.931 INFO:tensorflow:epoch = 21.163212435233163, learning_rate = 0.00049999997, loss = 4.063043e-05, step = 8169 (5.216 sec) 2021-10-19 07:20:35,861 [INFO] tensorflow: epoch = 21.163212435233163, learning_rate = 0.00049999997, loss = 4.063043e-05, step = 8169 (5.216 sec) INFO:tensorflow:global_step/sec: 4.98417 2021-10-19 07:20:36,072 [INFO] tensorflow: global_step/sec: 4.98417 2021-10-19 07:20:36,874 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.946 INFO:tensorflow:epoch = 21.230569948186528, learning_rate = 0.00049999997, loss = 3.4579636e-05, step = 8195 (5.231 sec) 2021-10-19 07:20:41,092 [INFO] tensorflow: epoch = 21.230569948186528, learning_rate = 0.00049999997, loss = 3.4579636e-05, step = 8195 (5.231 sec) 2021-10-19 07:20:41,927 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.790 INFO:tensorflow:global_step/sec: 4.98005 2021-10-19 07:20:43,702 [INFO] tensorflow: global_step/sec: 4.98005 INFO:tensorflow:epoch = 21.300518134715027, learning_rate = 0.00049999997, loss = 2.7534836e-05, step = 8222 (5.393 sec) 2021-10-19 07:20:46,485 [INFO] tensorflow: epoch = 21.300518134715027, learning_rate = 0.00049999997, loss = 2.7534836e-05, step = 8222 (5.393 sec) 2021-10-19 07:20:46,890 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.149 INFO:tensorflow:global_step/sec: 5.0707 2021-10-19 07:20:51,196 [INFO] tensorflow: global_step/sec: 5.0707 INFO:tensorflow:epoch = 21.370466321243523, learning_rate = 0.00049999997, loss = 3.8090096e-05, step = 8249 (5.339 sec) 2021-10-19 07:20:51,824 [INFO] tensorflow: epoch = 21.370466321243523, learning_rate = 0.00049999997, loss = 3.8090096e-05, step = 8249 (5.339 sec) 2021-10-19 07:20:51,824 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.271 2021-10-19 07:20:56,824 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.999 INFO:tensorflow:epoch = 21.43782383419689, learning_rate = 0.00049999997, loss = 4.901697e-05, step = 8275 (5.199 sec) 2021-10-19 07:20:57,022 [INFO] tensorflow: epoch = 21.43782383419689, learning_rate = 0.00049999997, loss = 4.901697e-05, step = 8275 (5.199 sec) INFO:tensorflow:global_step/sec: 4.98851 2021-10-19 07:20:58,814 [INFO] tensorflow: global_step/sec: 4.98851 2021-10-19 07:21:01,815 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.039 INFO:tensorflow:epoch = 21.50777202072539, learning_rate = 0.00049999997, loss = 3.8420112e-05, step = 8302 (5.383 sec) 2021-10-19 07:21:02,406 [INFO] tensorflow: epoch = 21.50777202072539, learning_rate = 0.00049999997, loss = 3.8420112e-05, step = 8302 (5.383 sec) INFO:tensorflow:global_step/sec: 4.99381 2021-10-19 07:21:06,423 [INFO] tensorflow: global_step/sec: 4.99381 2021-10-19 07:21:06,824 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.967 INFO:tensorflow:epoch = 21.57512953367876, learning_rate = 0.00049999997, loss = 4.4795015e-05, step = 8328 (5.258 sec) 2021-10-19 07:21:07,664 [INFO] tensorflow: epoch = 21.57512953367876, learning_rate = 0.00049999997, loss = 4.4795015e-05, step = 8328 (5.258 sec) 2021-10-19 07:21:11,854 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.886 INFO:tensorflow:epoch = 21.645077720207254, learning_rate = 0.00049999997, loss = 2.5371863e-05, step = 8355 (5.386 sec) 2021-10-19 07:21:13,050 [INFO] tensorflow: epoch = 21.645077720207254, learning_rate = 0.00049999997, loss = 2.5371863e-05, step = 8355 (5.386 sec) INFO:tensorflow:global_step/sec: 4.96766 2021-10-19 07:21:14,073 [INFO] tensorflow: global_step/sec: 4.96766 2021-10-19 07:21:16,866 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.954 INFO:tensorflow:epoch = 21.712435233160623, learning_rate = 0.00049999997, loss = 3.7575017e-05, step = 8381 (5.235 sec) 2021-10-19 07:21:18,285 [INFO] tensorflow: epoch = 21.712435233160623, learning_rate = 0.00049999997, loss = 3.7575017e-05, step = 8381 (5.235 sec) INFO:tensorflow:global_step/sec: 5.02649 2021-10-19 07:21:21,633 [INFO] tensorflow: global_step/sec: 5.02649 2021-10-19 07:21:21,859 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.032 INFO:tensorflow:epoch = 21.78238341968912, learning_rate = 0.00049999997, loss = 3.8767248e-05, step = 8408 (5.376 sec) 2021-10-19 07:21:23,662 [INFO] tensorflow: epoch = 21.78238341968912, learning_rate = 0.00049999997, loss = 3.8767248e-05, step = 8408 (5.376 sec) 2021-10-19 07:21:26,802 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.231 INFO:tensorflow:epoch = 21.852331606217618, learning_rate = 0.00049999997, loss = 2.5951056e-05, step = 8435 (5.336 sec) 2021-10-19 07:21:28,997 [INFO] tensorflow: epoch = 21.852331606217618, learning_rate = 0.00049999997, loss = 2.5951056e-05, step = 8435 (5.336 sec) INFO:tensorflow:global_step/sec: 5.01156 2021-10-19 07:21:29,215 [INFO] tensorflow: global_step/sec: 5.01156 2021-10-19 07:21:31,833 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.876 INFO:tensorflow:epoch = 21.919689119170986, learning_rate = 0.00049999997, loss = 7.732877e-05, step = 8461 (5.293 sec) 2021-10-19 07:21:34,291 [INFO] tensorflow: epoch = 21.919689119170986, learning_rate = 0.00049999997, loss = 7.732877e-05, step = 8461 (5.293 sec) INFO:tensorflow:global_step/sec: 4.92988 2021-10-19 07:21:36,923 [INFO] tensorflow: global_step/sec: 4.92988 2021-10-19 07:21:36,925 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.642 INFO:tensorflow:epoch = 21.987046632124354, learning_rate = 0.00049999997, loss = 2.4655246e-05, step = 8487 (5.236 sec) 2021-10-19 07:21:39,527 [INFO] tensorflow: epoch = 21.987046632124354, learning_rate = 0.00049999997, loss = 2.4655246e-05, step = 8487 (5.236 sec) 2021-10-19 07:21:40,527 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 22/120: loss: 0.00003 learning rate: 0.00050 Time taken: 0:01:17.245734 ETA: 2:06:10.081953 2021-10-19 07:21:41,950 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.900 INFO:tensorflow:global_step/sec: 5.00494 2021-10-19 07:21:44,516 [INFO] tensorflow: global_step/sec: 5.00494 INFO:tensorflow:epoch = 22.05699481865285, learning_rate = 0.00049999997, loss = 3.3977543e-05, step = 8514 (5.387 sec) 2021-10-19 07:21:44,913 [INFO] tensorflow: epoch = 22.05699481865285, learning_rate = 0.00049999997, loss = 3.3977543e-05, step = 8514 (5.387 sec) 2021-10-19 07:21:46,915 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.143 INFO:tensorflow:epoch = 22.124352331606218, learning_rate = 0.00049999997, loss = 2.4587565e-05, step = 8540 (5.223 sec) 2021-10-19 07:21:50,136 [INFO] tensorflow: epoch = 22.124352331606218, learning_rate = 0.00049999997, loss = 2.4587565e-05, step = 8540 (5.223 sec) 2021-10-19 07:21:51,915 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.003 INFO:tensorflow:global_step/sec: 5.00002 2021-10-19 07:21:52,116 [INFO] tensorflow: global_step/sec: 5.00002 INFO:tensorflow:epoch = 22.194300518134717, learning_rate = 0.00049999997, loss = 3.6598467e-05, step = 8567 (5.400 sec) 2021-10-19 07:21:55,537 [INFO] tensorflow: epoch = 22.194300518134717, learning_rate = 0.00049999997, loss = 3.6598467e-05, step = 8567 (5.400 sec) 2021-10-19 07:21:56,916 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.994 INFO:tensorflow:global_step/sec: 5.00754 2021-10-19 07:21:59,704 [INFO] tensorflow: global_step/sec: 5.00754 INFO:tensorflow:epoch = 22.264248704663213, learning_rate = 0.00049999997, loss = 5.821592e-05, step = 8594 (5.378 sec) 2021-10-19 07:22:00,915 [INFO] tensorflow: epoch = 22.264248704663213, learning_rate = 0.00049999997, loss = 5.821592e-05, step = 8594 (5.378 sec) 2021-10-19 07:22:01,917 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.001 INFO:tensorflow:epoch = 22.33160621761658, learning_rate = 0.00049999997, loss = 2.4153036e-05, step = 8620 (5.207 sec) 2021-10-19 07:22:06,121 [INFO] tensorflow: epoch = 22.33160621761658, learning_rate = 0.00049999997, loss = 2.4153036e-05, step = 8620 (5.207 sec) 2021-10-19 07:22:06,925 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.969 INFO:tensorflow:global_step/sec: 4.97587 2021-10-19 07:22:07,341 [INFO] tensorflow: global_step/sec: 4.97587 INFO:tensorflow:epoch = 22.39896373056995, learning_rate = 0.00049999997, loss = 3.410202e-05, step = 8646 (5.233 sec) 2021-10-19 07:22:11,354 [INFO] tensorflow: epoch = 22.39896373056995, learning_rate = 0.00049999997, loss = 3.410202e-05, step = 8646 (5.233 sec) 2021-10-19 07:22:12,009 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.670 INFO:tensorflow:global_step/sec: 4.97509 2021-10-19 07:22:14,979 [INFO] tensorflow: global_step/sec: 4.97509 INFO:tensorflow:epoch = 22.466321243523318, learning_rate = 0.00049999997, loss = 5.6374538e-05, step = 8672 (5.229 sec) 2021-10-19 07:22:16,583 [INFO] tensorflow: epoch = 22.466321243523318, learning_rate = 0.00049999997, loss = 5.6374538e-05, step = 8672 (5.229 sec) 2021-10-19 07:22:17,007 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.007 INFO:tensorflow:epoch = 22.536269430051814, learning_rate = 0.00049999997, loss = 3.9704762e-05, step = 8699 (5.393 sec) 2021-10-19 07:22:21,976 [INFO] tensorflow: epoch = 22.536269430051814, learning_rate = 0.00049999997, loss = 3.9704762e-05, step = 8699 (5.393 sec) 2021-10-19 07:22:21,977 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.125 INFO:tensorflow:global_step/sec: 5.01502 2021-10-19 07:22:22,556 [INFO] tensorflow: global_step/sec: 5.01502 2021-10-19 07:22:26,897 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.325 INFO:tensorflow:epoch = 22.606217616580313, learning_rate = 0.00049999997, loss = 0.00010676526, step = 8726 (5.330 sec) 2021-10-19 07:22:27,306 [INFO] tensorflow: epoch = 22.606217616580313, learning_rate = 0.00049999997, loss = 0.00010676526, step = 8726 (5.330 sec) INFO:tensorflow:global_step/sec: 5.01739 2021-10-19 07:22:30,130 [INFO] tensorflow: global_step/sec: 5.01739 2021-10-19 07:22:31,918 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.919 INFO:tensorflow:epoch = 22.67357512953368, learning_rate = 0.00049999997, loss = 4.1545907e-05, step = 8752 (5.226 sec) 2021-10-19 07:22:32,532 [INFO] tensorflow: epoch = 22.67357512953368, learning_rate = 0.00049999997, loss = 4.1545907e-05, step = 8752 (5.226 sec) 2021-10-19 07:22:36,937 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.924 INFO:tensorflow:epoch = 22.740932642487046, learning_rate = 0.00049999997, loss = 3.2807944e-05, step = 8778 (5.222 sec) 2021-10-19 07:22:37,754 [INFO] tensorflow: epoch = 22.740932642487046, learning_rate = 0.00049999997, loss = 3.2807944e-05, step = 8778 (5.222 sec) INFO:tensorflow:global_step/sec: 4.98308 2021-10-19 07:22:37,756 [INFO] tensorflow: global_step/sec: 4.98308 2021-10-19 07:22:41,924 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.055 INFO:tensorflow:epoch = 22.810880829015545, learning_rate = 0.00049999997, loss = 2.6844948e-05, step = 8805 (5.399 sec) 2021-10-19 07:22:43,154 [INFO] tensorflow: epoch = 22.810880829015545, learning_rate = 0.00049999997, loss = 2.6844948e-05, step = 8805 (5.399 sec) INFO:tensorflow:global_step/sec: 4.95621 2021-10-19 07:22:45,423 [INFO] tensorflow: global_step/sec: 4.95621 2021-10-19 07:22:47,033 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.574 INFO:tensorflow:epoch = 22.878238341968913, learning_rate = 0.00049999997, loss = 6.241856e-05, step = 8831 (5.291 sec) 2021-10-19 07:22:48,445 [INFO] tensorflow: epoch = 22.878238341968913, learning_rate = 0.00049999997, loss = 6.241856e-05, step = 8831 (5.291 sec) 2021-10-19 07:22:52,077 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.827 INFO:tensorflow:global_step/sec: 4.9704 2021-10-19 07:22:53,068 [INFO] tensorflow: global_step/sec: 4.9704 INFO:tensorflow:epoch = 22.94559585492228, learning_rate = 0.00049999997, loss = 4.885396e-05, step = 8857 (5.220 sec) 2021-10-19 07:22:53,665 [INFO] tensorflow: epoch = 22.94559585492228, learning_rate = 0.00049999997, loss = 4.885396e-05, step = 8857 (5.220 sec) 2021-10-19 07:22:57,113 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.860 2021-10-19 07:22:57,986 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 23/120: loss: 0.00004 learning rate: 0.00050 Time taken: 0:01:17.427653 ETA: 2:05:10.482371 INFO:tensorflow:epoch = 23.01295336787565, learning_rate = 0.00049999997, loss = 3.44998e-05, step = 8883 (5.321 sec) 2021-10-19 07:22:58,986 [INFO] tensorflow: epoch = 23.01295336787565, learning_rate = 0.00049999997, loss = 3.44998e-05, step = 8883 (5.321 sec) INFO:tensorflow:global_step/sec: 4.92192 2021-10-19 07:23:00,789 [INFO] tensorflow: global_step/sec: 4.92192 2021-10-19 07:23:02,179 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.741 INFO:tensorflow:epoch = 23.082901554404145, learning_rate = 0.00049999997, loss = 2.7907774e-05, step = 8910 (5.377 sec) 2021-10-19 07:23:04,363 [INFO] tensorflow: epoch = 23.082901554404145, learning_rate = 0.00049999997, loss = 2.7907774e-05, step = 8910 (5.377 sec) 2021-10-19 07:23:07,098 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.330 INFO:tensorflow:global_step/sec: 5.06743 2021-10-19 07:23:08,288 [INFO] tensorflow: global_step/sec: 5.06743 INFO:tensorflow:epoch = 23.152849740932645, learning_rate = 0.00049999997, loss = 3.2717948e-05, step = 8937 (5.426 sec) 2021-10-19 07:23:09,790 [INFO] tensorflow: epoch = 23.152849740932645, learning_rate = 0.00049999997, loss = 3.2717948e-05, step = 8937 (5.426 sec) 2021-10-19 07:23:12,203 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.590 INFO:tensorflow:epoch = 23.22279792746114, learning_rate = 0.00049999997, loss = 3.6706122e-05, step = 8964 (5.368 sec) 2021-10-19 07:23:15,157 [INFO] tensorflow: epoch = 23.22279792746114, learning_rate = 0.00049999997, loss = 3.6706122e-05, step = 8964 (5.368 sec) INFO:tensorflow:global_step/sec: 4.95498 2021-10-19 07:23:15,957 [INFO] tensorflow: global_step/sec: 4.95498 2021-10-19 07:23:17,187 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.067 INFO:tensorflow:epoch = 23.29015544041451, learning_rate = 0.00049999997, loss = 2.1408421e-05, step = 8990 (5.214 sec) 2021-10-19 07:23:20,371 [INFO] tensorflow: epoch = 23.29015544041451, learning_rate = 0.00049999997, loss = 2.1408421e-05, step = 8990 (5.214 sec) 2021-10-19 07:23:22,149 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.154 INFO:tensorflow:global_step/sec: 5.02256 2021-10-19 07:23:23,523 [INFO] tensorflow: global_step/sec: 5.02256 INFO:tensorflow:epoch = 23.357512953367877, learning_rate = 0.00049999997, loss = 3.38494e-05, step = 9016 (5.233 sec) 2021-10-19 07:23:25,604 [INFO] tensorflow: epoch = 23.357512953367877, learning_rate = 0.00049999997, loss = 3.38494e-05, step = 9016 (5.233 sec) 2021-10-19 07:23:27,184 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.864 INFO:tensorflow:epoch = 23.424870466321245, learning_rate = 0.00049999997, loss = 2.6061352e-05, step = 9042 (5.285 sec) 2021-10-19 07:23:30,889 [INFO] tensorflow: epoch = 23.424870466321245, learning_rate = 0.00049999997, loss = 2.6061352e-05, step = 9042 (5.285 sec) INFO:tensorflow:global_step/sec: 4.88562 2021-10-19 07:23:31,301 [INFO] tensorflow: global_step/sec: 4.88562 2021-10-19 07:23:32,323 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.460 INFO:tensorflow:epoch = 23.49481865284974, learning_rate = 0.00049999997, loss = 4.4355274e-05, step = 9069 (5.364 sec) 2021-10-19 07:23:36,254 [INFO] tensorflow: epoch = 23.49481865284974, learning_rate = 0.00049999997, loss = 4.4355274e-05, step = 9069 (5.364 sec) 2021-10-19 07:23:37,235 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.358 INFO:tensorflow:global_step/sec: 5.06621 2021-10-19 07:23:38,801 [INFO] tensorflow: global_step/sec: 5.06621 INFO:tensorflow:epoch = 23.56476683937824, learning_rate = 0.00049999997, loss = 4.8382197e-05, step = 9096 (5.314 sec) 2021-10-19 07:23:41,568 [INFO] tensorflow: epoch = 23.56476683937824, learning_rate = 0.00049999997, loss = 4.8382197e-05, step = 9096 (5.314 sec) 2021-10-19 07:23:42,179 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.230 INFO:tensorflow:global_step/sec: 5.0063 2021-10-19 07:23:46,392 [INFO] tensorflow: global_step/sec: 5.0063 INFO:tensorflow:epoch = 23.63212435233161, learning_rate = 0.00049999997, loss = 3.149532e-05, step = 9122 (5.222 sec) 2021-10-19 07:23:46,789 [INFO] tensorflow: epoch = 23.63212435233161, learning_rate = 0.00049999997, loss = 3.149532e-05, step = 9122 (5.222 sec) 2021-10-19 07:23:47,192 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.949 INFO:tensorflow:epoch = 23.699481865284977, learning_rate = 0.00049999997, loss = 3.1283766e-05, step = 9148 (5.251 sec) 2021-10-19 07:23:52,041 [INFO] tensorflow: epoch = 23.699481865284977, learning_rate = 0.00049999997, loss = 3.1283766e-05, step = 9148 (5.251 sec) 2021-10-19 07:23:52,246 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.787 INFO:tensorflow:global_step/sec: 4.92826 2021-10-19 07:23:54,102 [INFO] tensorflow: global_step/sec: 4.92826 INFO:tensorflow:epoch = 23.76683937823834, learning_rate = 0.00049999997, loss = 2.4475044e-05, step = 9174 (5.267 sec) 2021-10-19 07:23:57,307 [INFO] tensorflow: epoch = 23.76683937823834, learning_rate = 0.00049999997, loss = 2.4475044e-05, step = 9174 (5.267 sec) 2021-10-19 07:23:57,308 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.760 INFO:tensorflow:global_step/sec: 5.04719 2021-10-19 07:24:01,631 [INFO] tensorflow: global_step/sec: 5.04719 2021-10-19 07:24:02,233 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.304 INFO:tensorflow:epoch = 23.83678756476684, learning_rate = 0.00049999997, loss = 4.4979643e-05, step = 9201 (5.318 sec) 2021-10-19 07:24:02,626 [INFO] tensorflow: epoch = 23.83678756476684, learning_rate = 0.00049999997, loss = 4.4979643e-05, step = 9201 (5.318 sec) 2021-10-19 07:24:07,243 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.961 INFO:tensorflow:epoch = 23.90414507772021, learning_rate = 0.00049999997, loss = 3.7636393e-05, step = 9227 (5.217 sec) 2021-10-19 07:24:07,842 [INFO] tensorflow: epoch = 23.90414507772021, learning_rate = 0.00049999997, loss = 3.7636393e-05, step = 9227 (5.217 sec) INFO:tensorflow:global_step/sec: 5.01718 2021-10-19 07:24:09,205 [INFO] tensorflow: global_step/sec: 5.01718 2021-10-19 07:24:12,207 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.148 INFO:tensorflow:epoch = 23.974093264248705, learning_rate = 0.00049999997, loss = 3.4221244e-05, step = 9254 (5.394 sec) 2021-10-19 07:24:13,236 [INFO] tensorflow: epoch = 23.974093264248705, learning_rate = 0.00049999997, loss = 3.4221244e-05, step = 9254 (5.394 sec) 2021-10-19 07:24:15,285 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 24/120: loss: 0.00004 learning rate: 0.00050 Time taken: 0:01:17.309165 ETA: 2:03:41.679886 INFO:tensorflow:global_step/sec: 4.94085 2021-10-19 07:24:16,896 [INFO] tensorflow: global_step/sec: 4.94085 2021-10-19 07:24:17,286 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.691 INFO:tensorflow:epoch = 24.041450777202073, learning_rate = 0.00049999997, loss = 3.8489634e-05, step = 9280 (5.250 sec) 2021-10-19 07:24:18,486 [INFO] tensorflow: epoch = 24.041450777202073, learning_rate = 0.00049999997, loss = 3.8489634e-05, step = 9280 (5.250 sec) 2021-10-19 07:24:22,311 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.903 INFO:tensorflow:epoch = 24.10880829015544, learning_rate = 0.00049999997, loss = 6.298398e-05, step = 9306 (5.236 sec) 2021-10-19 07:24:23,722 [INFO] tensorflow: epoch = 24.10880829015544, learning_rate = 0.00049999997, loss = 6.298398e-05, step = 9306 (5.236 sec) INFO:tensorflow:global_step/sec: 4.98713 2021-10-19 07:24:24,516 [INFO] tensorflow: global_step/sec: 4.98713 2021-10-19 07:24:27,354 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.832 INFO:tensorflow:epoch = 24.17616580310881, learning_rate = 0.00049999997, loss = 3.148648e-05, step = 9332 (5.207 sec) 2021-10-19 07:24:28,928 [INFO] tensorflow: epoch = 24.17616580310881, learning_rate = 0.00049999997, loss = 3.148648e-05, step = 9332 (5.207 sec) INFO:tensorflow:global_step/sec: 4.9895 2021-10-19 07:24:32,132 [INFO] tensorflow: global_step/sec: 4.9895 2021-10-19 07:24:32,336 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.074 INFO:tensorflow:epoch = 24.24611398963731, learning_rate = 0.00049999997, loss = 3.057938e-05, step = 9359 (5.377 sec) 2021-10-19 07:24:34,305 [INFO] tensorflow: epoch = 24.24611398963731, learning_rate = 0.00049999997, loss = 3.057938e-05, step = 9359 (5.377 sec) 2021-10-19 07:24:37,224 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.459 INFO:tensorflow:epoch = 24.316062176165804, learning_rate = 0.00049999997, loss = 3.727968e-05, step = 9386 (5.279 sec) 2021-10-19 07:24:39,584 [INFO] tensorflow: epoch = 24.316062176165804, learning_rate = 0.00049999997, loss = 3.727968e-05, step = 9386 (5.279 sec) INFO:tensorflow:global_step/sec: 5.098 2021-10-19 07:24:39,586 [INFO] tensorflow: global_step/sec: 5.098 2021-10-19 07:24:42,184 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.162 INFO:tensorflow:epoch = 24.3860103626943, learning_rate = 0.00049999997, loss = 2.4729867e-05, step = 9413 (5.378 sec) 2021-10-19 07:24:44,962 [INFO] tensorflow: epoch = 24.3860103626943, learning_rate = 0.00049999997, loss = 2.4729867e-05, step = 9413 (5.378 sec) INFO:tensorflow:global_step/sec: 4.98196 2021-10-19 07:24:47,213 [INFO] tensorflow: global_step/sec: 4.98196 2021-10-19 07:24:47,214 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.883 INFO:tensorflow:epoch = 24.45336787564767, learning_rate = 0.00049999997, loss = 4.2494816e-05, step = 9439 (5.261 sec) 2021-10-19 07:24:50,223 [INFO] tensorflow: epoch = 24.45336787564767, learning_rate = 0.00049999997, loss = 4.2494816e-05, step = 9439 (5.261 sec) 2021-10-19 07:24:52,219 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.983 INFO:tensorflow:global_step/sec: 4.97936 2021-10-19 07:24:54,845 [INFO] tensorflow: global_step/sec: 4.97936 INFO:tensorflow:epoch = 24.520725388601036, learning_rate = 0.00049999997, loss = 3.9576516e-05, step = 9465 (5.234 sec) 2021-10-19 07:24:55,457 [INFO] tensorflow: epoch = 24.520725388601036, learning_rate = 0.00049999997, loss = 3.9576516e-05, step = 9465 (5.234 sec) 2021-10-19 07:24:57,219 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.001 INFO:tensorflow:epoch = 24.590673575129536, learning_rate = 0.00049999997, loss = 5.3811404e-05, step = 9492 (5.317 sec) 2021-10-19 07:25:00,775 [INFO] tensorflow: epoch = 24.590673575129536, learning_rate = 0.00049999997, loss = 5.3811404e-05, step = 9492 (5.317 sec) 2021-10-19 07:25:02,165 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.222 INFO:tensorflow:global_step/sec: 5.04924 2021-10-19 07:25:02,371 [INFO] tensorflow: global_step/sec: 5.04924 INFO:tensorflow:epoch = 24.658031088082904, learning_rate = 0.00049999997, loss = 4.395403e-05, step = 9518 (5.228 sec) 2021-10-19 07:25:06,003 [INFO] tensorflow: epoch = 24.658031088082904, learning_rate = 0.00049999997, loss = 4.395403e-05, step = 9518 (5.228 sec) 2021-10-19 07:25:07,217 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.793 INFO:tensorflow:global_step/sec: 4.95399 2021-10-19 07:25:10,041 [INFO] tensorflow: global_step/sec: 4.95399 INFO:tensorflow:epoch = 24.725388601036272, learning_rate = 0.00049999997, loss = 5.4253134e-05, step = 9544 (5.215 sec) 2021-10-19 07:25:11,218 [INFO] tensorflow: epoch = 24.725388601036272, learning_rate = 0.00049999997, loss = 5.4253134e-05, step = 9544 (5.215 sec) 2021-10-19 07:25:12,241 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.908 INFO:tensorflow:epoch = 24.792746113989637, learning_rate = 0.00049999997, loss = 4.7267076e-05, step = 9570 (5.287 sec) 2021-10-19 07:25:16,505 [INFO] tensorflow: epoch = 24.792746113989637, learning_rate = 0.00049999997, loss = 4.7267076e-05, step = 9570 (5.287 sec) 2021-10-19 07:25:17,287 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.818 INFO:tensorflow:global_step/sec: 4.96337 2021-10-19 07:25:17,697 [INFO] tensorflow: global_step/sec: 4.96337 INFO:tensorflow:epoch = 24.862694300518136, learning_rate = 0.00049999997, loss = 4.7387286e-05, step = 9597 (5.346 sec) 2021-10-19 07:25:21,851 [INFO] tensorflow: epoch = 24.862694300518136, learning_rate = 0.00049999997, loss = 4.7387286e-05, step = 9597 (5.346 sec) 2021-10-19 07:25:22,264 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.094 INFO:tensorflow:global_step/sec: 5.05076 2021-10-19 07:25:25,221 [INFO] tensorflow: global_step/sec: 5.05076 INFO:tensorflow:epoch = 24.932642487046632, learning_rate = 0.00049999997, loss = 2.6867656e-05, step = 9624 (5.382 sec) 2021-10-19 07:25:27,233 [INFO] tensorflow: epoch = 24.932642487046632, learning_rate = 0.00049999997, loss = 2.6867656e-05, step = 9624 (5.382 sec) 2021-10-19 07:25:27,233 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.126 2021-10-19 07:25:32,206 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.113 2021-10-19 07:25:32,424 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 25/120: loss: 0.00002 learning rate: 0.00050 Time taken: 0:01:17.130968 ETA: 2:02:07.441992 INFO:tensorflow:epoch = 25.00259067357513, learning_rate = 0.00049999997, loss = 3.2888653e-05, step = 9651 (5.396 sec) 2021-10-19 07:25:32,629 [INFO] tensorflow: epoch = 25.00259067357513, learning_rate = 0.00049999997, loss = 3.2888653e-05, step = 9651 (5.396 sec) INFO:tensorflow:global_step/sec: 4.97835 2021-10-19 07:25:32,854 [INFO] tensorflow: global_step/sec: 4.97835 2021-10-19 07:25:37,285 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.696 INFO:tensorflow:epoch = 25.0699481865285, learning_rate = 0.00049999997, loss = 3.2181073e-05, step = 9677 (5.257 sec) 2021-10-19 07:25:37,886 [INFO] tensorflow: epoch = 25.0699481865285, learning_rate = 0.00049999997, loss = 3.2181073e-05, step = 9677 (5.257 sec) INFO:tensorflow:global_step/sec: 4.96524 2021-10-19 07:25:40,507 [INFO] tensorflow: global_step/sec: 4.96524 2021-10-19 07:25:42,272 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.055 INFO:tensorflow:epoch = 25.139896373056995, learning_rate = 0.00049999997, loss = 4.4437715e-05, step = 9704 (5.368 sec) 2021-10-19 07:25:43,254 [INFO] tensorflow: epoch = 25.139896373056995, learning_rate = 0.00049999997, loss = 4.4437715e-05, step = 9704 (5.368 sec) 2021-10-19 07:25:47,188 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.344 INFO:tensorflow:global_step/sec: 5.06663 2021-10-19 07:25:48,007 [INFO] tensorflow: global_step/sec: 5.06663 INFO:tensorflow:epoch = 25.209844559585495, learning_rate = 0.00049999997, loss = 2.6010592e-05, step = 9731 (5.407 sec) 2021-10-19 07:25:48,661 [INFO] tensorflow: epoch = 25.209844559585495, learning_rate = 0.00049999997, loss = 2.6010592e-05, step = 9731 (5.407 sec) 2021-10-19 07:25:52,309 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.528 INFO:tensorflow:epoch = 25.27720207253886, learning_rate = 0.00049999997, loss = 2.9763414e-05, step = 9757 (5.257 sec) 2021-10-19 07:25:53,917 [INFO] tensorflow: epoch = 25.27720207253886, learning_rate = 0.00049999997, loss = 2.9763414e-05, step = 9757 (5.257 sec) INFO:tensorflow:global_step/sec: 4.91333 2021-10-19 07:25:55,741 [INFO] tensorflow: global_step/sec: 4.91333 2021-10-19 07:25:57,305 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.019 INFO:tensorflow:epoch = 25.344559585492227, learning_rate = 0.00049999997, loss = 3.8873426e-05, step = 9783 (5.228 sec) 2021-10-19 07:25:59,146 [INFO] tensorflow: epoch = 25.344559585492227, learning_rate = 0.00049999997, loss = 3.8873426e-05, step = 9783 (5.228 sec) 2021-10-19 07:26:02,315 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.961 INFO:tensorflow:global_step/sec: 5.02032 2021-10-19 07:26:03,311 [INFO] tensorflow: global_step/sec: 5.02032 INFO:tensorflow:epoch = 25.414507772020727, learning_rate = 0.00049999997, loss = 9.7770564e-05, step = 9810 (5.356 sec) 2021-10-19 07:26:04,502 [INFO] tensorflow: epoch = 25.414507772020727, learning_rate = 0.00049999997, loss = 9.7770564e-05, step = 9810 (5.356 sec) 2021-10-19 07:26:07,289 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.106 INFO:tensorflow:epoch = 25.484455958549223, learning_rate = 0.00049999997, loss = 3.852336e-05, step = 9837 (5.375 sec) 2021-10-19 07:26:09,876 [INFO] tensorflow: epoch = 25.484455958549223, learning_rate = 0.00049999997, loss = 3.852336e-05, step = 9837 (5.375 sec) INFO:tensorflow:global_step/sec: 5.02141 2021-10-19 07:26:10,878 [INFO] tensorflow: global_step/sec: 5.02141 2021-10-19 07:26:12,277 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.051 INFO:tensorflow:epoch = 25.554404145077722, learning_rate = 0.00049999997, loss = 2.4296418e-05, step = 9864 (5.357 sec) 2021-10-19 07:26:15,233 [INFO] tensorflow: epoch = 25.554404145077722, learning_rate = 0.00049999997, loss = 2.4296418e-05, step = 9864 (5.357 sec) 2021-10-19 07:26:17,261 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.065 INFO:tensorflow:global_step/sec: 5.018 2021-10-19 07:26:18,451 [INFO] tensorflow: global_step/sec: 5.018 INFO:tensorflow:epoch = 25.624352331606218, learning_rate = 0.00049999997, loss = 2.867466e-05, step = 9891 (5.400 sec) 2021-10-19 07:26:20,633 [INFO] tensorflow: epoch = 25.624352331606218, learning_rate = 0.00049999997, loss = 2.867466e-05, step = 9891 (5.400 sec) 2021-10-19 07:26:22,238 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.096 INFO:tensorflow:epoch = 25.694300518134717, learning_rate = 0.00049999997, loss = 3.0764917e-05, step = 9918 (5.326 sec) 2021-10-19 07:26:25,959 [INFO] tensorflow: epoch = 25.694300518134717, learning_rate = 0.00049999997, loss = 3.0764917e-05, step = 9918 (5.326 sec) INFO:tensorflow:global_step/sec: 5.06011 2021-10-19 07:26:25,961 [INFO] tensorflow: global_step/sec: 5.06011 2021-10-19 07:26:27,150 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.357 INFO:tensorflow:epoch = 25.764248704663213, learning_rate = 0.00049999997, loss = 3.466654e-05, step = 9945 (5.331 sec) 2021-10-19 07:26:31,290 [INFO] tensorflow: epoch = 25.764248704663213, learning_rate = 0.00049999997, loss = 3.466654e-05, step = 9945 (5.331 sec) 2021-10-19 07:26:32,066 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.346 INFO:tensorflow:global_step/sec: 5.06692 2021-10-19 07:26:33,460 [INFO] tensorflow: global_step/sec: 5.06692 INFO:tensorflow:epoch = 25.83419689119171, learning_rate = 0.00049999997, loss = 2.7623708e-05, step = 9972 (5.347 sec) 2021-10-19 07:26:36,637 [INFO] tensorflow: epoch = 25.83419689119171, learning_rate = 0.00049999997, loss = 2.7623708e-05, step = 9972 (5.347 sec) 2021-10-19 07:26:37,064 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.009 INFO:tensorflow:global_step/sec: 5.02433 2021-10-19 07:26:41,023 [INFO] tensorflow: global_step/sec: 5.02433 INFO:tensorflow:epoch = 25.901554404145077, learning_rate = 0.00049999997, loss = 3.371464e-05, step = 9998 (5.194 sec) 2021-10-19 07:26:41,831 [INFO] tensorflow: epoch = 25.901554404145077, learning_rate = 0.00049999997, loss = 3.371464e-05, step = 9998 (5.194 sec) 2021-10-19 07:26:42,033 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.129 INFO:tensorflow:epoch = 25.968911917098445, learning_rate = 0.00049999997, loss = 4.924862e-05, step = 10024 (5.215 sec) 2021-10-19 07:26:47,047 [INFO] tensorflow: epoch = 25.968911917098445, learning_rate = 0.00049999997, loss = 4.924862e-05, step = 10024 (5.215 sec) 2021-10-19 07:26:47,047 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.944 INFO:tensorflow:global_step/sec: 4.97865 2021-10-19 07:26:48,656 [INFO] tensorflow: global_step/sec: 4.97865 2021-10-19 07:26:49,460 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 26/120: loss: 0.00003 learning rate: 0.00050 Time taken: 0:01:17.053433 ETA: 2:00:43.022719 2021-10-19 07:26:52,033 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.058 INFO:tensorflow:epoch = 26.038860103626945, learning_rate = 0.00049999997, loss = 3.4784236e-05, step = 10051 (5.381 sec) 2021-10-19 07:26:52,427 [INFO] tensorflow: epoch = 26.038860103626945, learning_rate = 0.00049999997, loss = 3.4784236e-05, step = 10051 (5.381 sec) INFO:tensorflow:global_step/sec: 5.03936 2021-10-19 07:26:56,197 [INFO] tensorflow: global_step/sec: 5.03936 2021-10-19 07:26:56,993 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.161 INFO:tensorflow:epoch = 26.10880829015544, learning_rate = 0.00049999997, loss = 3.9594765e-05, step = 10078 (5.385 sec) 2021-10-19 07:26:57,812 [INFO] tensorflow: epoch = 26.10880829015544, learning_rate = 0.00049999997, loss = 3.9594765e-05, step = 10078 (5.385 sec) 2021-10-19 07:27:01,986 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.028 INFO:tensorflow:epoch = 26.17875647668394, learning_rate = 0.00049999997, loss = 2.2017779e-05, step = 10105 (5.337 sec) 2021-10-19 07:27:03,149 [INFO] tensorflow: epoch = 26.17875647668394, learning_rate = 0.00049999997, loss = 2.2017779e-05, step = 10105 (5.337 sec) INFO:tensorflow:global_step/sec: 5.01002 2021-10-19 07:27:03,781 [INFO] tensorflow: global_step/sec: 5.01002 2021-10-19 07:27:06,925 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.251 INFO:tensorflow:epoch = 26.248704663212436, learning_rate = 0.00049999997, loss = 5.837055e-05, step = 10132 (5.349 sec) 2021-10-19 07:27:08,498 [INFO] tensorflow: epoch = 26.248704663212436, learning_rate = 0.00049999997, loss = 5.837055e-05, step = 10132 (5.349 sec) INFO:tensorflow:global_step/sec: 5.13206 2021-10-19 07:27:11,186 [INFO] tensorflow: global_step/sec: 5.13206 2021-10-19 07:27:11,784 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.582 INFO:tensorflow:epoch = 26.318652849740932, learning_rate = 0.00049999997, loss = 5.3925494e-05, step = 10159 (5.266 sec) 2021-10-19 07:27:13,764 [INFO] tensorflow: epoch = 26.318652849740932, learning_rate = 0.00049999997, loss = 5.3925494e-05, step = 10159 (5.266 sec) 2021-10-19 07:27:16,741 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.174 INFO:tensorflow:global_step/sec: 5.05391 2021-10-19 07:27:18,705 [INFO] tensorflow: global_step/sec: 5.05391 INFO:tensorflow:epoch = 26.38860103626943, learning_rate = 0.00049999997, loss = 3.174339e-05, step = 10186 (5.335 sec) 2021-10-19 07:27:19,099 [INFO] tensorflow: epoch = 26.38860103626943, learning_rate = 0.00049999997, loss = 3.174339e-05, step = 10186 (5.335 sec) 2021-10-19 07:27:21,711 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.122 INFO:tensorflow:epoch = 26.458549222797927, learning_rate = 0.00049999997, loss = 3.6435897e-05, step = 10213 (5.391 sec) 2021-10-19 07:27:24,490 [INFO] tensorflow: epoch = 26.458549222797927, learning_rate = 0.00049999997, loss = 3.6435897e-05, step = 10213 (5.391 sec) INFO:tensorflow:global_step/sec: 4.97959 2021-10-19 07:27:26,336 [INFO] tensorflow: global_step/sec: 4.97959 2021-10-19 07:27:26,750 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.849 INFO:tensorflow:epoch = 26.525906735751295, learning_rate = 0.00049999997, loss = 3.2912187e-05, step = 10239 (5.260 sec) 2021-10-19 07:27:29,751 [INFO] tensorflow: epoch = 26.525906735751295, learning_rate = 0.00049999997, loss = 3.2912187e-05, step = 10239 (5.260 sec) 2021-10-19 07:27:31,741 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.039 INFO:tensorflow:global_step/sec: 4.99961 2021-10-19 07:27:33,937 [INFO] tensorflow: global_step/sec: 4.99961 INFO:tensorflow:epoch = 26.595854922279795, learning_rate = 0.00049999997, loss = 4.837074e-05, step = 10266 (5.377 sec) 2021-10-19 07:27:35,128 [INFO] tensorflow: epoch = 26.595854922279795, learning_rate = 0.00049999997, loss = 4.837074e-05, step = 10266 (5.377 sec) 2021-10-19 07:27:36,743 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.993 INFO:tensorflow:epoch = 26.66580310880829, learning_rate = 0.00049999997, loss = 5.2150775e-05, step = 10293 (5.384 sec) 2021-10-19 07:27:40,512 [INFO] tensorflow: epoch = 26.66580310880829, learning_rate = 0.00049999997, loss = 5.2150775e-05, step = 10293 (5.384 sec) INFO:tensorflow:global_step/sec: 5.03612 2021-10-19 07:27:41,482 [INFO] tensorflow: global_step/sec: 5.03612 2021-10-19 07:27:41,690 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.217 INFO:tensorflow:epoch = 26.73575129533679, learning_rate = 0.00049999997, loss = 4.720436e-05, step = 10320 (5.337 sec) 2021-10-19 07:27:45,849 [INFO] tensorflow: epoch = 26.73575129533679, learning_rate = 0.00049999997, loss = 4.720436e-05, step = 10320 (5.337 sec) 2021-10-19 07:27:46,665 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.098 INFO:tensorflow:global_step/sec: 5.0111 2021-10-19 07:27:49,065 [INFO] tensorflow: global_step/sec: 5.0111 INFO:tensorflow:epoch = 26.803108808290155, learning_rate = 0.00049999997, loss = 3.5017154e-05, step = 10346 (5.259 sec) 2021-10-19 07:27:51,108 [INFO] tensorflow: epoch = 26.803108808290155, learning_rate = 0.00049999997, loss = 3.5017154e-05, step = 10346 (5.259 sec) 2021-10-19 07:27:51,696 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.879 INFO:tensorflow:epoch = 26.870466321243523, learning_rate = 0.00049999997, loss = 2.1368513e-05, step = 10372 (5.238 sec) 2021-10-19 07:27:56,345 [INFO] tensorflow: epoch = 26.870466321243523, learning_rate = 0.00049999997, loss = 2.1368513e-05, step = 10372 (5.238 sec) INFO:tensorflow:global_step/sec: 4.9404 2021-10-19 07:27:56,757 [INFO] tensorflow: global_step/sec: 4.9404 2021-10-19 07:27:56,758 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.758 INFO:tensorflow:epoch = 26.93782383419689, learning_rate = 0.00049999997, loss = 2.8787006e-05, step = 10398 (5.211 sec) 2021-10-19 07:28:01,557 [INFO] tensorflow: epoch = 26.93782383419689, learning_rate = 0.00049999997, loss = 2.8787006e-05, step = 10398 (5.211 sec) 2021-10-19 07:28:01,756 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.007 INFO:tensorflow:global_step/sec: 4.94584 2021-10-19 07:28:04,440 [INFO] tensorflow: global_step/sec: 4.94584 2021-10-19 07:28:06,440 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 27/120: loss: 0.00004 learning rate: 0.00050 Time taken: 0:01:16.955400 ETA: 1:59:16.852199 INFO:tensorflow:epoch = 27.00518134715026, learning_rate = 0.00049999997, loss = 4.7445967e-05, step = 10424 (5.277 sec) 2021-10-19 07:28:06,834 [INFO] tensorflow: epoch = 27.00518134715026, learning_rate = 0.00049999997, loss = 4.7445967e-05, step = 10424 (5.277 sec) 2021-10-19 07:28:06,834 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.697 2021-10-19 07:28:11,829 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.020 INFO:tensorflow:global_step/sec: 5.00577 2021-10-19 07:28:12,031 [INFO] tensorflow: global_step/sec: 5.00577 INFO:tensorflow:epoch = 27.07512953367876, learning_rate = 0.00049999997, loss = 3.551079e-05, step = 10451 (5.403 sec) 2021-10-19 07:28:12,237 [INFO] tensorflow: epoch = 27.07512953367876, learning_rate = 0.00049999997, loss = 3.551079e-05, step = 10451 (5.403 sec) 2021-10-19 07:28:16,809 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.084 INFO:tensorflow:epoch = 27.145077720207254, learning_rate = 0.00049999997, loss = 2.7818838e-05, step = 10478 (5.404 sec) 2021-10-19 07:28:17,640 [INFO] tensorflow: epoch = 27.145077720207254, learning_rate = 0.00049999997, loss = 2.7818838e-05, step = 10478 (5.404 sec) INFO:tensorflow:global_step/sec: 4.98749 2021-10-19 07:28:19,650 [INFO] tensorflow: global_step/sec: 4.98749 2021-10-19 07:28:21,892 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.673 INFO:tensorflow:epoch = 27.212435233160623, learning_rate = 0.00049999997, loss = 3.5038374e-05, step = 10504 (5.260 sec) 2021-10-19 07:28:22,901 [INFO] tensorflow: epoch = 27.212435233160623, learning_rate = 0.00049999997, loss = 3.5038374e-05, step = 10504 (5.260 sec) 2021-10-19 07:28:26,835 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.233 INFO:tensorflow:global_step/sec: 5.00357 2021-10-19 07:28:27,245 [INFO] tensorflow: global_step/sec: 5.00357 INFO:tensorflow:epoch = 27.282383419689122, learning_rate = 0.00049999997, loss = 3.1662657e-05, step = 10531 (5.342 sec) 2021-10-19 07:28:28,242 [INFO] tensorflow: epoch = 27.282383419689122, learning_rate = 0.00049999997, loss = 3.1662657e-05, step = 10531 (5.342 sec) 2021-10-19 07:28:31,838 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.989 INFO:tensorflow:epoch = 27.352331606217618, learning_rate = 0.00049999997, loss = 4.753624e-05, step = 10558 (5.394 sec) 2021-10-19 07:28:33,636 [INFO] tensorflow: epoch = 27.352331606217618, learning_rate = 0.00049999997, loss = 4.753624e-05, step = 10558 (5.394 sec) INFO:tensorflow:global_step/sec: 5.00495 2021-10-19 07:28:34,837 [INFO] tensorflow: global_step/sec: 5.00495 2021-10-19 07:28:36,809 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.117 INFO:tensorflow:epoch = 27.422279792746114, learning_rate = 0.00049999997, loss = 4.0427294e-05, step = 10585 (5.343 sec) 2021-10-19 07:28:38,979 [INFO] tensorflow: epoch = 27.422279792746114, learning_rate = 0.00049999997, loss = 4.0427294e-05, step = 10585 (5.343 sec) 2021-10-19 07:28:41,803 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.027 INFO:tensorflow:global_step/sec: 5.00092 2021-10-19 07:28:42,436 [INFO] tensorflow: global_step/sec: 5.00092 INFO:tensorflow:epoch = 27.489637305699482, learning_rate = 0.00049999997, loss = 2.1061729e-05, step = 10611 (5.267 sec) 2021-10-19 07:28:44,246 [INFO] tensorflow: epoch = 27.489637305699482, learning_rate = 0.00049999997, loss = 2.1061729e-05, step = 10611 (5.267 sec) 2021-10-19 07:28:46,832 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.889 INFO:tensorflow:epoch = 27.55958549222798, learning_rate = 0.00049999997, loss = 3.2382784e-05, step = 10638 (5.408 sec) 2021-10-19 07:28:49,654 [INFO] tensorflow: epoch = 27.55958549222798, learning_rate = 0.00049999997, loss = 3.2382784e-05, step = 10638 (5.408 sec) INFO:tensorflow:global_step/sec: 4.97274 2021-10-19 07:28:50,078 [INFO] tensorflow: global_step/sec: 4.97274 2021-10-19 07:28:51,915 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.676 INFO:tensorflow:epoch = 27.629533678756477, learning_rate = 0.00049999997, loss = 4.009393e-05, step = 10665 (5.390 sec) 2021-10-19 07:28:55,045 [INFO] tensorflow: epoch = 27.629533678756477, learning_rate = 0.00049999997, loss = 4.009393e-05, step = 10665 (5.390 sec) 2021-10-19 07:28:56,814 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.414 INFO:tensorflow:global_step/sec: 5.03111 2021-10-19 07:28:57,631 [INFO] tensorflow: global_step/sec: 5.03111 INFO:tensorflow:epoch = 27.699481865284977, learning_rate = 0.00049999997, loss = 3.5787598e-05, step = 10692 (5.382 sec) 2021-10-19 07:29:00,426 [INFO] tensorflow: epoch = 27.699481865284977, learning_rate = 0.00049999997, loss = 3.5787598e-05, step = 10692 (5.382 sec) 2021-10-19 07:29:01,822 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.970 INFO:tensorflow:global_step/sec: 4.9946 2021-10-19 07:29:05,239 [INFO] tensorflow: global_step/sec: 4.9946 INFO:tensorflow:epoch = 27.766839378238345, learning_rate = 0.00049999997, loss = 3.177786e-05, step = 10718 (5.236 sec) 2021-10-19 07:29:05,662 [INFO] tensorflow: epoch = 27.766839378238345, learning_rate = 0.00049999997, loss = 3.177786e-05, step = 10718 (5.236 sec) 2021-10-19 07:29:06,839 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.937 INFO:tensorflow:epoch = 27.83678756476684, learning_rate = 0.00049999997, loss = 2.6936232e-05, step = 10745 (5.317 sec) 2021-10-19 07:29:10,979 [INFO] tensorflow: epoch = 27.83678756476684, learning_rate = 0.00049999997, loss = 2.6936232e-05, step = 10745 (5.317 sec) 2021-10-19 07:29:11,780 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.239 INFO:tensorflow:global_step/sec: 5.04224 2021-10-19 07:29:12,775 [INFO] tensorflow: global_step/sec: 5.04224 INFO:tensorflow:epoch = 27.906735751295336, learning_rate = 0.00049999997, loss = 2.7387658e-05, step = 10772 (5.387 sec) 2021-10-19 07:29:16,367 [INFO] tensorflow: epoch = 27.906735751295336, learning_rate = 0.00049999997, loss = 2.7387658e-05, step = 10772 (5.387 sec) 2021-10-19 07:29:16,770 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.044 INFO:tensorflow:global_step/sec: 4.96985 2021-10-19 07:29:20,421 [INFO] tensorflow: global_step/sec: 4.96985 INFO:tensorflow:epoch = 27.974093264248705, learning_rate = 0.00049999997, loss = 2.3520988e-05, step = 10798 (5.266 sec) 2021-10-19 07:29:21,632 [INFO] tensorflow: epoch = 27.974093264248705, learning_rate = 0.00049999997, loss = 2.3520988e-05, step = 10798 (5.266 sec) 2021-10-19 07:29:21,842 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.716 2021-10-19 07:29:23,601 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 28/120: loss: 0.00004 learning rate: 0.00050 Time taken: 0:01:17.184543 ETA: 1:58:20.977946 2021-10-19 07:29:26,743 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.409 INFO:tensorflow:epoch = 28.044041450777204, learning_rate = 0.00049999997, loss = 3.0012676e-05, step = 10825 (5.319 sec) 2021-10-19 07:29:26,951 [INFO] tensorflow: epoch = 28.044041450777204, learning_rate = 0.00049999997, loss = 3.0012676e-05, step = 10825 (5.319 sec) INFO:tensorflow:global_step/sec: 5.06585 2021-10-19 07:29:27,923 [INFO] tensorflow: global_step/sec: 5.06585 2021-10-19 07:29:31,652 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.373 INFO:tensorflow:epoch = 28.1139896373057, learning_rate = 0.00049999997, loss = 5.788778e-05, step = 10852 (5.312 sec) 2021-10-19 07:29:32,262 [INFO] tensorflow: epoch = 28.1139896373057, learning_rate = 0.00049999997, loss = 5.788778e-05, step = 10852 (5.312 sec) INFO:tensorflow:global_step/sec: 5.04375 2021-10-19 07:29:35,457 [INFO] tensorflow: global_step/sec: 5.04375 2021-10-19 07:29:36,624 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.114 INFO:tensorflow:epoch = 28.1839378238342, learning_rate = 0.00049999997, loss = 2.0112702e-05, step = 10879 (5.410 sec) 2021-10-19 07:29:37,673 [INFO] tensorflow: epoch = 28.1839378238342, learning_rate = 0.00049999997, loss = 2.0112702e-05, step = 10879 (5.410 sec) 2021-10-19 07:29:41,679 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.787 INFO:tensorflow:epoch = 28.251295336787567, learning_rate = 0.00049999997, loss = 3.0273997e-05, step = 10905 (5.256 sec) 2021-10-19 07:29:42,929 [INFO] tensorflow: epoch = 28.251295336787567, learning_rate = 0.00049999997, loss = 3.0273997e-05, step = 10905 (5.256 sec) INFO:tensorflow:global_step/sec: 4.94557 2021-10-19 07:29:43,140 [INFO] tensorflow: global_step/sec: 4.94557 2021-10-19 07:29:46,809 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.496 INFO:tensorflow:epoch = 28.318652849740932, learning_rate = 0.00049999997, loss = 2.524189e-05, step = 10931 (5.271 sec) 2021-10-19 07:29:48,200 [INFO] tensorflow: epoch = 28.318652849740932, learning_rate = 0.00049999997, loss = 2.524189e-05, step = 10931 (5.271 sec) INFO:tensorflow:global_step/sec: 4.97636 2021-10-19 07:29:50,776 [INFO] tensorflow: global_step/sec: 4.97636 2021-10-19 07:29:51,793 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.063 INFO:tensorflow:epoch = 28.3860103626943, learning_rate = 0.00049999997, loss = 3.2149685e-05, step = 10957 (5.216 sec) 2021-10-19 07:29:53,416 [INFO] tensorflow: epoch = 28.3860103626943, learning_rate = 0.00049999997, loss = 3.2149685e-05, step = 10957 (5.216 sec) 2021-10-19 07:29:56,807 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.948 INFO:tensorflow:global_step/sec: 4.99336 2021-10-19 07:29:58,387 [INFO] tensorflow: global_step/sec: 4.99336 INFO:tensorflow:epoch = 28.4559585492228, learning_rate = 0.00049999997, loss = 3.100384e-05, step = 10984 (5.372 sec) 2021-10-19 07:29:58,788 [INFO] tensorflow: epoch = 28.4559585492228, learning_rate = 0.00049999997, loss = 3.100384e-05, step = 10984 (5.372 sec) 2021-10-19 07:30:01,773 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.141 INFO:tensorflow:epoch = 28.525906735751295, learning_rate = 0.00049999997, loss = 2.1057098e-05, step = 11011 (5.360 sec) 2021-10-19 07:30:04,148 [INFO] tensorflow: epoch = 28.525906735751295, learning_rate = 0.00049999997, loss = 2.1057098e-05, step = 11011 (5.360 sec) INFO:tensorflow:global_step/sec: 5.01807 2021-10-19 07:30:05,959 [INFO] tensorflow: global_step/sec: 5.01807 2021-10-19 07:30:06,768 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.021 INFO:tensorflow:epoch = 28.595854922279795, learning_rate = 0.00049999997, loss = 3.28196e-05, step = 11038 (5.393 sec) 2021-10-19 07:30:09,541 [INFO] tensorflow: epoch = 28.595854922279795, learning_rate = 0.00049999997, loss = 3.28196e-05, step = 11038 (5.393 sec) 2021-10-19 07:30:11,712 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.228 INFO:tensorflow:global_step/sec: 5.00958 2021-10-19 07:30:13,545 [INFO] tensorflow: global_step/sec: 5.00958 INFO:tensorflow:epoch = 28.663212435233163, learning_rate = 0.00049999997, loss = 5.3629265e-05, step = 11064 (5.223 sec) 2021-10-19 07:30:14,765 [INFO] tensorflow: epoch = 28.663212435233163, learning_rate = 0.00049999997, loss = 5.3629265e-05, step = 11064 (5.223 sec) 2021-10-19 07:30:16,815 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.596 INFO:tensorflow:epoch = 28.73316062176166, learning_rate = 0.00049999997, loss = 2.9656276e-05, step = 11091 (5.353 sec) 2021-10-19 07:30:20,118 [INFO] tensorflow: epoch = 28.73316062176166, learning_rate = 0.00049999997, loss = 2.9656276e-05, step = 11091 (5.353 sec) INFO:tensorflow:global_step/sec: 4.99478 2021-10-19 07:30:21,153 [INFO] tensorflow: global_step/sec: 4.99478 2021-10-19 07:30:21,784 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.129 INFO:tensorflow:epoch = 28.800518134715027, learning_rate = 0.00049999997, loss = 4.7765447e-05, step = 11117 (5.249 sec) 2021-10-19 07:30:25,366 [INFO] tensorflow: epoch = 28.800518134715027, learning_rate = 0.00049999997, loss = 4.7765447e-05, step = 11117 (5.249 sec) 2021-10-19 07:30:26,766 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.072 INFO:tensorflow:global_step/sec: 4.99634 2021-10-19 07:30:28,758 [INFO] tensorflow: global_step/sec: 4.99634 INFO:tensorflow:epoch = 28.870466321243523, learning_rate = 0.00049999997, loss = 7.008973e-05, step = 11144 (5.374 sec) 2021-10-19 07:30:30,741 [INFO] tensorflow: epoch = 28.870466321243523, learning_rate = 0.00049999997, loss = 7.008973e-05, step = 11144 (5.374 sec) 2021-10-19 07:30:31,762 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.017 INFO:tensorflow:epoch = 28.940414507772022, learning_rate = 0.00049999997, loss = 3.5097655e-05, step = 11171 (5.363 sec) 2021-10-19 07:30:36,104 [INFO] tensorflow: epoch = 28.940414507772022, learning_rate = 0.00049999997, loss = 3.5097655e-05, step = 11171 (5.363 sec) INFO:tensorflow:global_step/sec: 5.0252 2021-10-19 07:30:36,320 [INFO] tensorflow: global_step/sec: 5.0252 2021-10-19 07:30:36,760 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.013 2021-10-19 07:30:40,668 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 29/120: loss: 0.00003 learning rate: 0.00050 Time taken: 0:01:17.063781 ETA: 1:56:52.804030 INFO:tensorflow:epoch = 29.010362694300518, learning_rate = 0.00049999997, loss = 4.1869676e-05, step = 11198 (5.348 sec) 2021-10-19 07:30:41,452 [INFO] tensorflow: epoch = 29.010362694300518, learning_rate = 0.00049999997, loss = 4.1869676e-05, step = 11198 (5.348 sec) 2021-10-19 07:30:41,657 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.419 INFO:tensorflow:global_step/sec: 5.07149 2021-10-19 07:30:43,813 [INFO] tensorflow: global_step/sec: 5.07149 2021-10-19 07:30:46,605 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.215 INFO:tensorflow:epoch = 29.080310880829018, learning_rate = 0.00049999997, loss = 5.247187e-05, step = 11225 (5.350 sec) 2021-10-19 07:30:46,802 [INFO] tensorflow: epoch = 29.080310880829018, learning_rate = 0.00049999997, loss = 5.247187e-05, step = 11225 (5.350 sec) INFO:tensorflow:global_step/sec: 5.0088 2021-10-19 07:30:51,400 [INFO] tensorflow: global_step/sec: 5.0088 2021-10-19 07:30:51,609 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.985 INFO:tensorflow:epoch = 29.147668393782386, learning_rate = 0.00049999997, loss = 3.0228595e-05, step = 11251 (5.226 sec) 2021-10-19 07:30:52,028 [INFO] tensorflow: epoch = 29.147668393782386, learning_rate = 0.00049999997, loss = 3.0228595e-05, step = 11251 (5.226 sec) 2021-10-19 07:30:56,647 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.852 INFO:tensorflow:epoch = 29.215025906735754, learning_rate = 0.00049999997, loss = 2.5115423e-05, step = 11277 (5.231 sec) 2021-10-19 07:30:57,258 [INFO] tensorflow: epoch = 29.215025906735754, learning_rate = 0.00049999997, loss = 2.5115423e-05, step = 11277 (5.231 sec) INFO:tensorflow:global_step/sec: 4.94605 2021-10-19 07:30:59,082 [INFO] tensorflow: global_step/sec: 4.94605 2021-10-19 07:31:01,685 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.848 INFO:tensorflow:epoch = 29.282383419689122, learning_rate = 0.00049999997, loss = 3.279827e-05, step = 11303 (5.236 sec) 2021-10-19 07:31:02,495 [INFO] tensorflow: epoch = 29.282383419689122, learning_rate = 0.00049999997, loss = 3.279827e-05, step = 11303 (5.236 sec) INFO:tensorflow:global_step/sec: 5.04721 2021-10-19 07:31:06,611 [INFO] tensorflow: global_step/sec: 5.04721 2021-10-19 07:31:06,613 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.294 INFO:tensorflow:epoch = 29.352331606217618, learning_rate = 0.00049999997, loss = 3.873593e-05, step = 11330 (5.296 sec) 2021-10-19 07:31:07,791 [INFO] tensorflow: epoch = 29.352331606217618, learning_rate = 0.00049999997, loss = 3.873593e-05, step = 11330 (5.296 sec) 2021-10-19 07:31:11,556 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.231 INFO:tensorflow:epoch = 29.422279792746114, learning_rate = 0.00049999997, loss = 4.1592764e-05, step = 11357 (5.342 sec) 2021-10-19 07:31:13,133 [INFO] tensorflow: epoch = 29.422279792746114, learning_rate = 0.00049999997, loss = 4.1592764e-05, step = 11357 (5.342 sec) INFO:tensorflow:global_step/sec: 5.04327 2021-10-19 07:31:14,146 [INFO] tensorflow: global_step/sec: 5.04327 2021-10-19 07:31:16,590 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.867 INFO:tensorflow:epoch = 29.489637305699482, learning_rate = 0.00049999997, loss = 2.980324e-05, step = 11383 (5.271 sec) 2021-10-19 07:31:18,404 [INFO] tensorflow: epoch = 29.489637305699482, learning_rate = 0.00049999997, loss = 2.980324e-05, step = 11383 (5.271 sec) 2021-10-19 07:31:21,612 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.916 INFO:tensorflow:global_step/sec: 4.95783 2021-10-19 07:31:21,811 [INFO] tensorflow: global_step/sec: 4.95783 INFO:tensorflow:epoch = 29.55699481865285, learning_rate = 0.00049999997, loss = 4.25464e-05, step = 11409 (5.292 sec) 2021-10-19 07:31:23,695 [INFO] tensorflow: epoch = 29.55699481865285, learning_rate = 0.00049999997, loss = 4.25464e-05, step = 11409 (5.292 sec) 2021-10-19 07:31:26,651 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.846 INFO:tensorflow:epoch = 29.62694300518135, learning_rate = 0.00049999997, loss = 3.2618755e-05, step = 11436 (5.334 sec) 2021-10-19 07:31:29,029 [INFO] tensorflow: epoch = 29.62694300518135, learning_rate = 0.00049999997, loss = 3.2618755e-05, step = 11436 (5.334 sec) INFO:tensorflow:global_step/sec: 4.98503 2021-10-19 07:31:29,434 [INFO] tensorflow: global_step/sec: 4.98503 2021-10-19 07:31:31,639 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.048 INFO:tensorflow:epoch = 29.694300518134717, learning_rate = 0.00049999997, loss = 2.3852663e-05, step = 11462 (5.233 sec) 2021-10-19 07:31:34,262 [INFO] tensorflow: epoch = 29.694300518134717, learning_rate = 0.00049999997, loss = 2.3852663e-05, step = 11462 (5.233 sec) 2021-10-19 07:31:36,678 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.849 INFO:tensorflow:global_step/sec: 4.97061 2021-10-19 07:31:37,079 [INFO] tensorflow: global_step/sec: 4.97061 INFO:tensorflow:epoch = 29.764248704663213, learning_rate = 0.00049999997, loss = 4.31448e-05, step = 11489 (5.368 sec) 2021-10-19 07:31:39,630 [INFO] tensorflow: epoch = 29.764248704663213, learning_rate = 0.00049999997, loss = 4.31448e-05, step = 11489 (5.368 sec) 2021-10-19 07:31:41,580 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.403 INFO:tensorflow:global_step/sec: 5.09112 2021-10-19 07:31:44,542 [INFO] tensorflow: global_step/sec: 5.09112 INFO:tensorflow:epoch = 29.83419689119171, learning_rate = 0.00049999997, loss = 2.7745966e-05, step = 11516 (5.330 sec) 2021-10-19 07:31:44,959 [INFO] tensorflow: epoch = 29.83419689119171, learning_rate = 0.00049999997, loss = 2.7745966e-05, step = 11516 (5.330 sec) 2021-10-19 07:31:46,518 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.251 INFO:tensorflow:epoch = 29.90414507772021, learning_rate = 0.00049999997, loss = 5.1926243e-05, step = 11543 (5.301 sec) 2021-10-19 07:31:50,260 [INFO] tensorflow: epoch = 29.90414507772021, learning_rate = 0.00049999997, loss = 5.1926243e-05, step = 11543 (5.301 sec) 2021-10-19 07:31:51,458 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.245 INFO:tensorflow:global_step/sec: 5.05589 2021-10-19 07:31:52,059 [INFO] tensorflow: global_step/sec: 5.05589 INFO:tensorflow:epoch = 29.974093264248705, learning_rate = 0.00049999997, loss = 3.7904883e-05, step = 11570 (5.344 sec) 2021-10-19 07:31:55,604 [INFO] tensorflow: epoch = 29.974093264248705, learning_rate = 0.00049999997, loss = 3.7904883e-05, step = 11570 (5.344 sec) 2021-10-19 07:31:56,410 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.195 INFO:tensorflow:Saving checkpoints for step-11580. 2021-10-19 07:31:57,400 [INFO] tensorflow: Saving checkpoints for step-11580. 2021-10-19 07:32:02,043 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 62, 0.00s/step 2021-10-19 07:32:04,205 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 62, 0.22s/step 2021-10-19 07:32:05,117 [INFO] iva.detectnet_v2.evaluation.evaluation: step 20 / 62, 0.09s/step 2021-10-19 07:32:06,067 [INFO] iva.detectnet_v2.evaluation.evaluation: step 30 / 62, 0.09s/step 2021-10-19 07:32:07,003 [INFO] iva.detectnet_v2.evaluation.evaluation: step 40 / 62, 0.09s/step 2021-10-19 07:32:07,928 [INFO] iva.detectnet_v2.evaluation.evaluation: step 50 / 62, 0.09s/step 2021-10-19 07:32:08,866 [INFO] iva.detectnet_v2.evaluation.evaluation: step 60 / 62, 0.09s/step Matching predictions to ground truth, class 1/7.: 100%|█| 581/581 [00:00<00:00, 16578.62it/s] Matching predictions to ground truth, class 2/7.: 100%|█| 58/58 [00:00<00:00, 11883.04it/s] Matching predictions to ground truth, class 7/7.: 100%|█| 13/13 [00:00<00:00, 14653.57it/s] Epoch 30/120 ========================= Validation cost: 0.000042 Mean average_precision (in %): 6.4477 class name average precision (in %) ------------- -------------------------- auto_rickshaw 0 bicycle 22.3224 bus 5 car 17.8113 person 0 road_sign 0 truck 0 Median Inference Time: 0.018002 INFO:tensorflow:epoch = 30.0, learning_rate = 0.00049999997, loss = 3.1431053e-05, step = 11580 (13.742 sec) 2021-10-19 07:32:09,346 [INFO] tensorflow: epoch = 30.0, learning_rate = 0.00049999997, loss = 3.1431053e-05, step = 11580 (13.742 sec) 2021-10-19 07:32:09,346 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 30/120: loss: 0.00003 learning rate: 0.00050 Time taken: 0:01:28.650460 ETA: 2:12:58.541422 INFO:tensorflow:global_step/sec: 1.97132 2021-10-19 07:32:11,335 [INFO] tensorflow: global_step/sec: 1.97132 2021-10-19 07:32:13,175 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 5.965 INFO:tensorflow:epoch = 30.067357512953368, learning_rate = 0.00049999997, loss = 1.9083509e-05, step = 11606 (5.232 sec) 2021-10-19 07:32:14,578 [INFO] tensorflow: epoch = 30.067357512953368, learning_rate = 0.00049999997, loss = 1.9083509e-05, step = 11606 (5.232 sec) 2021-10-19 07:32:18,106 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.284 INFO:tensorflow:global_step/sec: 4.97816 2021-10-19 07:32:18,968 [INFO] tensorflow: global_step/sec: 4.97816 INFO:tensorflow:epoch = 30.134715025906736, learning_rate = 0.00049999997, loss = 4.543939e-05, step = 11632 (5.198 sec) 2021-10-19 07:32:19,776 [INFO] tensorflow: epoch = 30.134715025906736, learning_rate = 0.00049999997, loss = 4.543939e-05, step = 11632 (5.198 sec) 2021-10-19 07:32:23,148 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.832 INFO:tensorflow:epoch = 30.204663212435236, learning_rate = 0.00049999997, loss = 5.10956e-05, step = 11659 (5.364 sec) 2021-10-19 07:32:25,140 [INFO] tensorflow: epoch = 30.204663212435236, learning_rate = 0.00049999997, loss = 5.10956e-05, step = 11659 (5.364 sec) INFO:tensorflow:global_step/sec: 5.01091 2021-10-19 07:32:26,552 [INFO] tensorflow: global_step/sec: 5.01091 2021-10-19 07:32:28,120 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.115 INFO:tensorflow:epoch = 30.27461139896373, learning_rate = 0.00049999997, loss = 2.9056706e-05, step = 11686 (5.315 sec) 2021-10-19 07:32:30,454 [INFO] tensorflow: epoch = 30.27461139896373, learning_rate = 0.00049999997, loss = 2.9056706e-05, step = 11686 (5.315 sec) 2021-10-19 07:32:33,051 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.281 INFO:tensorflow:global_step/sec: 5.08311 2021-10-19 07:32:34,028 [INFO] tensorflow: global_step/sec: 5.08311 INFO:tensorflow:epoch = 30.3419689119171, learning_rate = 0.00049999997, loss = 2.5713867e-05, step = 11712 (5.195 sec) 2021-10-19 07:32:35,649 [INFO] tensorflow: epoch = 30.3419689119171, learning_rate = 0.00049999997, loss = 2.5713867e-05, step = 11712 (5.195 sec) 2021-10-19 07:32:38,010 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.168 INFO:tensorflow:epoch = 30.411917098445596, learning_rate = 0.00049999997, loss = 2.7243193e-05, step = 11739 (5.411 sec) 2021-10-19 07:32:41,060 [INFO] tensorflow: epoch = 30.411917098445596, learning_rate = 0.00049999997, loss = 2.7243193e-05, step = 11739 (5.411 sec) INFO:tensorflow:global_step/sec: 4.98959 2021-10-19 07:32:41,643 [INFO] tensorflow: global_step/sec: 4.98959 2021-10-19 07:32:43,065 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.785 INFO:tensorflow:epoch = 30.481865284974095, learning_rate = 0.00049999997, loss = 3.2741053e-05, step = 11766 (5.414 sec) 2021-10-19 07:32:46,474 [INFO] tensorflow: epoch = 30.481865284974095, learning_rate = 0.00049999997, loss = 3.2741053e-05, step = 11766 (5.414 sec) 2021-10-19 07:32:48,049 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.066 INFO:tensorflow:global_step/sec: 4.97454 2021-10-19 07:32:49,282 [INFO] tensorflow: global_step/sec: 4.97454 INFO:tensorflow:epoch = 30.55181347150259, learning_rate = 0.00049999997, loss = 2.4780167e-05, step = 11793 (5.375 sec) 2021-10-19 07:32:51,850 [INFO] tensorflow: epoch = 30.55181347150259, learning_rate = 0.00049999997, loss = 2.4780167e-05, step = 11793 (5.375 sec) 2021-10-19 07:32:53,048 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.007 INFO:tensorflow:global_step/sec: 5.03315 2021-10-19 07:32:56,832 [INFO] tensorflow: global_step/sec: 5.03315 INFO:tensorflow:epoch = 30.62176165803109, learning_rate = 0.00049999997, loss = 3.281493e-05, step = 11820 (5.383 sec) 2021-10-19 07:32:57,233 [INFO] tensorflow: epoch = 30.62176165803109, learning_rate = 0.00049999997, loss = 3.281493e-05, step = 11820 (5.383 sec) 2021-10-19 07:32:58,043 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.024 INFO:tensorflow:epoch = 30.691709844559586, learning_rate = 0.00049999997, loss = 3.7369304e-05, step = 11847 (5.361 sec) 2021-10-19 07:33:02,594 [INFO] tensorflow: epoch = 30.691709844559586, learning_rate = 0.00049999997, loss = 3.7369304e-05, step = 11847 (5.361 sec) 2021-10-19 07:33:02,988 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.222 INFO:tensorflow:global_step/sec: 5.01584 2021-10-19 07:33:04,408 [INFO] tensorflow: global_step/sec: 5.01584 INFO:tensorflow:epoch = 30.761658031088086, learning_rate = 0.00049999997, loss = 2.8615283e-05, step = 11874 (5.356 sec) 2021-10-19 07:33:07,950 [INFO] tensorflow: epoch = 30.761658031088086, learning_rate = 0.00049999997, loss = 2.8615283e-05, step = 11874 (5.356 sec) 2021-10-19 07:33:07,950 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.156 INFO:tensorflow:global_step/sec: 5.05307 2021-10-19 07:33:11,928 [INFO] tensorflow: global_step/sec: 5.05307 2021-10-19 07:33:12,897 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.215 INFO:tensorflow:epoch = 30.83160621761658, learning_rate = 0.00049999997, loss = 3.2327487e-05, step = 11901 (5.351 sec) 2021-10-19 07:33:13,301 [INFO] tensorflow: epoch = 30.83160621761658, learning_rate = 0.00049999997, loss = 3.2327487e-05, step = 11901 (5.351 sec) 2021-10-19 07:33:17,834 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.257 INFO:tensorflow:epoch = 30.901554404145077, learning_rate = 0.00049999997, loss = 1.9981973e-05, step = 11928 (5.345 sec) 2021-10-19 07:33:18,646 [INFO] tensorflow: epoch = 30.901554404145077, learning_rate = 0.00049999997, loss = 1.9981973e-05, step = 11928 (5.345 sec) INFO:tensorflow:global_step/sec: 5.04782 2021-10-19 07:33:19,456 [INFO] tensorflow: global_step/sec: 5.04782 2021-10-19 07:33:22,785 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.203 INFO:tensorflow:epoch = 30.971502590673577, learning_rate = 0.00049999997, loss = 4.1771982e-05, step = 11955 (5.332 sec) 2021-10-19 07:33:23,978 [INFO] tensorflow: epoch = 30.971502590673577, learning_rate = 0.00049999997, loss = 4.1771982e-05, step = 11955 (5.332 sec) 2021-10-19 07:33:26,175 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 31/120: loss: 0.00002 learning rate: 0.00050 Time taken: 0:01:16.855088 ETA: 1:54:00.102810 INFO:tensorflow:global_step/sec: 5.04517 2021-10-19 07:33:26,988 [INFO] tensorflow: global_step/sec: 5.04517 2021-10-19 07:33:27,802 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.934 INFO:tensorflow:epoch = 31.038860103626945, learning_rate = 0.00049999997, loss = 6.7534565e-05, step = 11981 (5.230 sec) 2021-10-19 07:33:29,208 [INFO] tensorflow: epoch = 31.038860103626945, learning_rate = 0.00049999997, loss = 6.7534565e-05, step = 11981 (5.230 sec) 2021-10-19 07:33:32,786 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.067 INFO:tensorflow:epoch = 31.10880829015544, learning_rate = 0.00049999997, loss = 2.7232069e-05, step = 12008 (5.372 sec) 2021-10-19 07:33:34,580 [INFO] tensorflow: epoch = 31.10880829015544, learning_rate = 0.00049999997, loss = 2.7232069e-05, step = 12008 (5.372 sec) INFO:tensorflow:global_step/sec: 5.00465 2021-10-19 07:33:34,581 [INFO] tensorflow: global_step/sec: 5.00465 2021-10-19 07:33:37,811 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.900 INFO:tensorflow:epoch = 31.17616580310881, learning_rate = 0.00049999997, loss = 3.2111482e-05, step = 12034 (5.267 sec) 2021-10-19 07:33:39,847 [INFO] tensorflow: epoch = 31.17616580310881, learning_rate = 0.00049999997, loss = 3.2111482e-05, step = 12034 (5.267 sec) INFO:tensorflow:global_step/sec: 4.95757 2021-10-19 07:33:42,246 [INFO] tensorflow: global_step/sec: 4.95757 2021-10-19 07:33:42,853 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.836 INFO:tensorflow:epoch = 31.24611398963731, learning_rate = 0.00049999997, loss = 2.5352743e-05, step = 12061 (5.384 sec) 2021-10-19 07:33:45,231 [INFO] tensorflow: epoch = 31.24611398963731, learning_rate = 0.00049999997, loss = 2.5352743e-05, step = 12061 (5.384 sec) 2021-10-19 07:33:47,863 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.965 INFO:tensorflow:global_step/sec: 5.01326 2021-10-19 07:33:49,826 [INFO] tensorflow: global_step/sec: 5.01326 INFO:tensorflow:epoch = 31.313471502590676, learning_rate = 0.00049999997, loss = 3.564841e-05, step = 12087 (5.203 sec) 2021-10-19 07:33:50,433 [INFO] tensorflow: epoch = 31.313471502590676, learning_rate = 0.00049999997, loss = 3.564841e-05, step = 12087 (5.203 sec) 2021-10-19 07:33:52,782 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.331 INFO:tensorflow:epoch = 31.383419689119172, learning_rate = 0.00049999997, loss = 4.348492e-05, step = 12114 (5.314 sec) 2021-10-19 07:33:55,747 [INFO] tensorflow: epoch = 31.383419689119172, learning_rate = 0.00049999997, loss = 4.348492e-05, step = 12114 (5.314 sec) INFO:tensorflow:global_step/sec: 5.03323 2021-10-19 07:33:57,376 [INFO] tensorflow: global_step/sec: 5.03323 2021-10-19 07:33:57,791 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.965 INFO:tensorflow:epoch = 31.45336787564767, learning_rate = 0.00049999997, loss = 4.1105402e-05, step = 12141 (5.373 sec) 2021-10-19 07:34:01,120 [INFO] tensorflow: epoch = 31.45336787564767, learning_rate = 0.00049999997, loss = 4.1105402e-05, step = 12141 (5.373 sec) 2021-10-19 07:34:02,728 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.257 INFO:tensorflow:global_step/sec: 5.05044 2021-10-19 07:34:04,900 [INFO] tensorflow: global_step/sec: 5.05044 INFO:tensorflow:epoch = 31.523316062176168, learning_rate = 0.00049999997, loss = 5.330833e-05, step = 12168 (5.342 sec) 2021-10-19 07:34:06,462 [INFO] tensorflow: epoch = 31.523316062176168, learning_rate = 0.00049999997, loss = 5.330833e-05, step = 12168 (5.342 sec) 2021-10-19 07:34:07,670 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.237 INFO:tensorflow:epoch = 31.593264248704664, learning_rate = 0.00049999997, loss = 2.6911175e-05, step = 12195 (5.365 sec) 2021-10-19 07:34:11,827 [INFO] tensorflow: epoch = 31.593264248704664, learning_rate = 0.00049999997, loss = 2.6911175e-05, step = 12195 (5.365 sec) INFO:tensorflow:global_step/sec: 5.02814 2021-10-19 07:34:12,458 [INFO] tensorflow: global_step/sec: 5.02814 2021-10-19 07:34:12,656 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.059 INFO:tensorflow:epoch = 31.66062176165803, learning_rate = 0.00049999997, loss = 4.4231194e-05, step = 12221 (5.272 sec) 2021-10-19 07:34:17,100 [INFO] tensorflow: epoch = 31.66062176165803, learning_rate = 0.00049999997, loss = 4.4231194e-05, step = 12221 (5.272 sec) 2021-10-19 07:34:17,691 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.863 INFO:tensorflow:global_step/sec: 4.97032 2021-10-19 07:34:20,103 [INFO] tensorflow: global_step/sec: 4.97032 INFO:tensorflow:epoch = 31.73056994818653, learning_rate = 0.00049999997, loss = 3.123312e-05, step = 12248 (5.368 sec) 2021-10-19 07:34:22,468 [INFO] tensorflow: epoch = 31.73056994818653, learning_rate = 0.00049999997, loss = 3.123312e-05, step = 12248 (5.368 sec) 2021-10-19 07:34:22,672 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.076 INFO:tensorflow:global_step/sec: 5.01944 2021-10-19 07:34:27,674 [INFO] tensorflow: global_step/sec: 5.01944 2021-10-19 07:34:27,674 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.993 INFO:tensorflow:epoch = 31.800518134715027, learning_rate = 0.00049999997, loss = 3.3317734e-05, step = 12275 (5.412 sec) 2021-10-19 07:34:27,879 [INFO] tensorflow: epoch = 31.800518134715027, learning_rate = 0.00049999997, loss = 3.3317734e-05, step = 12275 (5.412 sec) 2021-10-19 07:34:32,600 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.302 INFO:tensorflow:epoch = 31.870466321243523, learning_rate = 0.00049999997, loss = 3.8516497e-05, step = 12302 (5.338 sec) 2021-10-19 07:34:33,217 [INFO] tensorflow: epoch = 31.870466321243523, learning_rate = 0.00049999997, loss = 3.8516497e-05, step = 12302 (5.338 sec) INFO:tensorflow:global_step/sec: 4.99121 2021-10-19 07:34:35,287 [INFO] tensorflow: global_step/sec: 4.99121 2021-10-19 07:34:37,686 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.663 INFO:tensorflow:epoch = 31.93782383419689, learning_rate = 0.00049999997, loss = 6.219022e-05, step = 12328 (5.267 sec) 2021-10-19 07:34:38,484 [INFO] tensorflow: epoch = 31.93782383419689, learning_rate = 0.00049999997, loss = 6.219022e-05, step = 12328 (5.267 sec) 2021-10-19 07:34:42,623 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.258 INFO:tensorflow:global_step/sec: 5.03923 2021-10-19 07:34:42,828 [INFO] tensorflow: global_step/sec: 5.03923 2021-10-19 07:34:43,231 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 32/120: loss: 0.00004 learning rate: 0.00050 Time taken: 0:01:17.057602 ETA: 1:53:01.068949 INFO:tensorflow:epoch = 32.00777202072539, learning_rate = 0.00049999997, loss = 4.9299906e-05, step = 12355 (5.366 sec) 2021-10-19 07:34:43,850 [INFO] tensorflow: epoch = 32.00777202072539, learning_rate = 0.00049999997, loss = 4.9299906e-05, step = 12355 (5.366 sec) 2021-10-19 07:34:47,608 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.063 INFO:tensorflow:epoch = 32.075129533678755, learning_rate = 0.00049999997, loss = 4.792807e-05, step = 12381 (5.204 sec) 2021-10-19 07:34:49,054 [INFO] tensorflow: epoch = 32.075129533678755, learning_rate = 0.00049999997, loss = 4.792807e-05, step = 12381 (5.204 sec) INFO:tensorflow:global_step/sec: 4.96903 2021-10-19 07:34:50,475 [INFO] tensorflow: global_step/sec: 4.96903 2021-10-19 07:34:52,623 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.943 INFO:tensorflow:epoch = 32.145077720207254, learning_rate = 0.00049999997, loss = 3.598228e-05, step = 12408 (5.325 sec) 2021-10-19 07:34:54,379 [INFO] tensorflow: epoch = 32.145077720207254, learning_rate = 0.00049999997, loss = 3.598228e-05, step = 12408 (5.325 sec) 2021-10-19 07:34:57,547 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.310 INFO:tensorflow:global_step/sec: 5.06807 2021-10-19 07:34:57,973 [INFO] tensorflow: global_step/sec: 5.06807 INFO:tensorflow:epoch = 32.215025906735754, learning_rate = 0.00049999997, loss = 4.6058103e-05, step = 12435 (5.384 sec) 2021-10-19 07:34:59,763 [INFO] tensorflow: epoch = 32.215025906735754, learning_rate = 0.00049999997, loss = 4.6058103e-05, step = 12435 (5.384 sec) 2021-10-19 07:35:02,606 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.767 INFO:tensorflow:epoch = 32.28497409326425, learning_rate = 0.00049999997, loss = 4.3833323e-05, step = 12462 (5.397 sec) 2021-10-19 07:35:05,160 [INFO] tensorflow: epoch = 32.28497409326425, learning_rate = 0.00049999997, loss = 4.3833323e-05, step = 12462 (5.397 sec) INFO:tensorflow:global_step/sec: 5.00953 2021-10-19 07:35:05,559 [INFO] tensorflow: global_step/sec: 5.00953 2021-10-19 07:35:07,595 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.046 INFO:tensorflow:epoch = 32.35233160621762, learning_rate = 0.00049999997, loss = 1.7266102e-05, step = 12488 (5.253 sec) 2021-10-19 07:35:10,413 [INFO] tensorflow: epoch = 32.35233160621762, learning_rate = 0.00049999997, loss = 1.7266102e-05, step = 12488 (5.253 sec) 2021-10-19 07:35:12,564 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.125 INFO:tensorflow:global_step/sec: 5.00699 2021-10-19 07:35:13,148 [INFO] tensorflow: global_step/sec: 5.00699 INFO:tensorflow:epoch = 32.42227979274612, learning_rate = 0.00049999997, loss = 4.656187e-05, step = 12515 (5.338 sec) 2021-10-19 07:35:15,751 [INFO] tensorflow: epoch = 32.42227979274612, learning_rate = 0.00049999997, loss = 4.656187e-05, step = 12515 (5.338 sec) 2021-10-19 07:35:17,564 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.003 INFO:tensorflow:global_step/sec: 4.95764 2021-10-19 07:35:20,813 [INFO] tensorflow: global_step/sec: 4.95764 INFO:tensorflow:epoch = 32.48963730569948, learning_rate = 0.00049999997, loss = 5.620509e-05, step = 12541 (5.269 sec) 2021-10-19 07:35:21,020 [INFO] tensorflow: epoch = 32.48963730569948, learning_rate = 0.00049999997, loss = 5.620509e-05, step = 12541 (5.269 sec) 2021-10-19 07:35:22,609 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.822 INFO:tensorflow:epoch = 32.55699481865285, learning_rate = 0.00049999997, loss = 4.011814e-05, step = 12567 (5.219 sec) 2021-10-19 07:35:26,239 [INFO] tensorflow: epoch = 32.55699481865285, learning_rate = 0.00049999997, loss = 4.011814e-05, step = 12567 (5.219 sec) 2021-10-19 07:35:27,621 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.957 INFO:tensorflow:global_step/sec: 4.99392 2021-10-19 07:35:28,422 [INFO] tensorflow: global_step/sec: 4.99392 INFO:tensorflow:epoch = 32.626943005181346, learning_rate = 0.00049999997, loss = 3.28724e-05, step = 12594 (5.389 sec) 2021-10-19 07:35:31,629 [INFO] tensorflow: epoch = 32.626943005181346, learning_rate = 0.00049999997, loss = 3.28724e-05, step = 12594 (5.389 sec) 2021-10-19 07:35:32,626 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.981 INFO:tensorflow:global_step/sec: 5.00215 2021-10-19 07:35:36,019 [INFO] tensorflow: global_step/sec: 5.00215 INFO:tensorflow:epoch = 32.69430051813472, learning_rate = 0.00049999997, loss = 2.4828476e-05, step = 12620 (5.230 sec) 2021-10-19 07:35:36,859 [INFO] tensorflow: epoch = 32.69430051813472, learning_rate = 0.00049999997, loss = 2.4828476e-05, step = 12620 (5.230 sec) 2021-10-19 07:35:37,661 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.863 INFO:tensorflow:epoch = 32.76424870466322, learning_rate = 0.00049999997, loss = 2.0931062e-05, step = 12647 (5.405 sec) 2021-10-19 07:35:42,264 [INFO] tensorflow: epoch = 32.76424870466322, learning_rate = 0.00049999997, loss = 2.0931062e-05, step = 12647 (5.405 sec) 2021-10-19 07:35:42,694 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.871 INFO:tensorflow:global_step/sec: 4.96291 2021-10-19 07:35:43,676 [INFO] tensorflow: global_step/sec: 4.96291 INFO:tensorflow:epoch = 32.83160621761658, learning_rate = 0.00049999997, loss = 2.3438493e-05, step = 12673 (5.232 sec) 2021-10-19 07:35:47,496 [INFO] tensorflow: epoch = 32.83160621761658, learning_rate = 0.00049999997, loss = 2.3438493e-05, step = 12673 (5.232 sec) 2021-10-19 07:35:47,690 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.016 INFO:tensorflow:global_step/sec: 4.90637 2021-10-19 07:35:51,421 [INFO] tensorflow: global_step/sec: 4.90637 INFO:tensorflow:epoch = 32.898963730569946, learning_rate = 0.00049999997, loss = 4.663322e-05, step = 12699 (5.328 sec) 2021-10-19 07:35:52,824 [INFO] tensorflow: epoch = 32.898963730569946, learning_rate = 0.00049999997, loss = 4.663322e-05, step = 12699 (5.328 sec) 2021-10-19 07:35:52,825 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.479 2021-10-19 07:35:57,846 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.923 INFO:tensorflow:epoch = 32.96632124352332, learning_rate = 0.00049999997, loss = 3.3869088e-05, step = 12725 (5.226 sec) 2021-10-19 07:35:58,050 [INFO] tensorflow: epoch = 32.96632124352332, learning_rate = 0.00049999997, loss = 3.3869088e-05, step = 12725 (5.226 sec) INFO:tensorflow:global_step/sec: 4.98269 2021-10-19 07:35:59,047 [INFO] tensorflow: global_step/sec: 4.98269 2021-10-19 07:36:00,672 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 33/120: loss: 0.00004 learning rate: 0.00050 Time taken: 0:01:17.420508 ETA: 1:52:15.584188 2021-10-19 07:36:02,838 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.031 INFO:tensorflow:epoch = 33.03626943005182, learning_rate = 0.00049999997, loss = 3.7335347e-05, step = 12752 (5.389 sec) 2021-10-19 07:36:03,439 [INFO] tensorflow: epoch = 33.03626943005182, learning_rate = 0.00049999997, loss = 3.7335347e-05, step = 12752 (5.389 sec) INFO:tensorflow:global_step/sec: 5.02456 2021-10-19 07:36:06,610 [INFO] tensorflow: global_step/sec: 5.02456 2021-10-19 07:36:07,833 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.025 INFO:tensorflow:epoch = 33.10621761658031, learning_rate = 0.00049999997, loss = 3.2706994e-05, step = 12779 (5.416 sec) 2021-10-19 07:36:08,856 [INFO] tensorflow: epoch = 33.10621761658031, learning_rate = 0.00049999997, loss = 3.2706994e-05, step = 12779 (5.416 sec) 2021-10-19 07:36:12,837 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.987 INFO:tensorflow:epoch = 33.17357512953368, learning_rate = 0.00049999997, loss = 3.943374e-05, step = 12805 (5.211 sec) 2021-10-19 07:36:14,067 [INFO] tensorflow: epoch = 33.17357512953368, learning_rate = 0.00049999997, loss = 3.943374e-05, step = 12805 (5.211 sec) INFO:tensorflow:global_step/sec: 4.95437 2021-10-19 07:36:14,280 [INFO] tensorflow: global_step/sec: 4.95437 2021-10-19 07:36:17,836 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.004 INFO:tensorflow:epoch = 33.24352331606218, learning_rate = 0.00049999997, loss = 4.3450826e-05, step = 12832 (5.369 sec) 2021-10-19 07:36:19,436 [INFO] tensorflow: epoch = 33.24352331606218, learning_rate = 0.00049999997, loss = 4.3450826e-05, step = 12832 (5.369 sec) INFO:tensorflow:global_step/sec: 5.04125 2021-10-19 07:36:21,818 [INFO] tensorflow: global_step/sec: 5.04125 2021-10-19 07:36:22,820 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.070 INFO:tensorflow:epoch = 33.310880829015545, learning_rate = 0.00049999997, loss = 3.9042505e-05, step = 12858 (5.259 sec) 2021-10-19 07:36:24,695 [INFO] tensorflow: epoch = 33.310880829015545, learning_rate = 0.00049999997, loss = 3.9042505e-05, step = 12858 (5.259 sec) 2021-10-19 07:36:27,817 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.012 INFO:tensorflow:global_step/sec: 5.01462 2021-10-19 07:36:29,396 [INFO] tensorflow: global_step/sec: 5.01462 INFO:tensorflow:epoch = 33.380829015544045, learning_rate = 0.00049999997, loss = 4.9650334e-05, step = 12885 (5.285 sec) 2021-10-19 07:36:29,980 [INFO] tensorflow: epoch = 33.380829015544045, learning_rate = 0.00049999997, loss = 4.9650334e-05, step = 12885 (5.285 sec) 2021-10-19 07:36:32,760 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.233 INFO:tensorflow:epoch = 33.44818652849741, learning_rate = 0.00049999997, loss = 3.1356794e-05, step = 12911 (5.225 sec) 2021-10-19 07:36:35,206 [INFO] tensorflow: epoch = 33.44818652849741, learning_rate = 0.00049999997, loss = 3.1356794e-05, step = 12911 (5.225 sec) INFO:tensorflow:global_step/sec: 5.0028 2021-10-19 07:36:36,991 [INFO] tensorflow: global_step/sec: 5.0028 2021-10-19 07:36:37,794 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.868 INFO:tensorflow:epoch = 33.51554404145078, learning_rate = 0.00049999997, loss = 4.5755965e-05, step = 12937 (5.226 sec) 2021-10-19 07:36:40,432 [INFO] tensorflow: epoch = 33.51554404145078, learning_rate = 0.00049999997, loss = 4.5755965e-05, step = 12937 (5.226 sec) 2021-10-19 07:36:42,829 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.862 INFO:tensorflow:global_step/sec: 4.97492 2021-10-19 07:36:44,630 [INFO] tensorflow: global_step/sec: 4.97492 INFO:tensorflow:epoch = 33.582901554404145, learning_rate = 0.00049999997, loss = 4.1871746e-05, step = 12963 (5.214 sec) 2021-10-19 07:36:45,646 [INFO] tensorflow: epoch = 33.582901554404145, learning_rate = 0.00049999997, loss = 4.1871746e-05, step = 12963 (5.214 sec) 2021-10-19 07:36:47,833 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.986 INFO:tensorflow:epoch = 33.65025906735752, learning_rate = 0.00049999997, loss = 3.7089318e-05, step = 12989 (5.218 sec) 2021-10-19 07:36:50,864 [INFO] tensorflow: epoch = 33.65025906735752, learning_rate = 0.00049999997, loss = 3.7089318e-05, step = 12989 (5.218 sec) INFO:tensorflow:global_step/sec: 4.98272 2021-10-19 07:36:52,256 [INFO] tensorflow: global_step/sec: 4.98272 2021-10-19 07:36:52,862 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.884 INFO:tensorflow:epoch = 33.71761658031088, learning_rate = 0.00049999997, loss = 3.091526e-05, step = 13015 (5.301 sec) 2021-10-19 07:36:56,165 [INFO] tensorflow: epoch = 33.71761658031088, learning_rate = 0.00049999997, loss = 3.091526e-05, step = 13015 (5.301 sec) 2021-10-19 07:36:57,940 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.694 INFO:tensorflow:global_step/sec: 4.94231 2021-10-19 07:36:59,945 [INFO] tensorflow: global_step/sec: 4.94231 INFO:tensorflow:epoch = 33.78497409326425, learning_rate = 0.00049999997, loss = 4.3598506e-05, step = 13041 (5.195 sec) 2021-10-19 07:37:01,360 [INFO] tensorflow: epoch = 33.78497409326425, learning_rate = 0.00049999997, loss = 4.3598506e-05, step = 13041 (5.195 sec) 2021-10-19 07:37:02,947 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.977 INFO:tensorflow:epoch = 33.854922279792746, learning_rate = 0.00049999997, loss = 5.3524014e-05, step = 13068 (5.345 sec) 2021-10-19 07:37:06,706 [INFO] tensorflow: epoch = 33.854922279792746, learning_rate = 0.00049999997, loss = 5.3524014e-05, step = 13068 (5.345 sec) INFO:tensorflow:global_step/sec: 5.02797 2021-10-19 07:37:07,503 [INFO] tensorflow: global_step/sec: 5.02797 2021-10-19 07:37:07,913 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.139 INFO:tensorflow:epoch = 33.924870466321245, learning_rate = 0.00049999997, loss = 3.1809155e-05, step = 13095 (5.367 sec) 2021-10-19 07:37:12,072 [INFO] tensorflow: epoch = 33.924870466321245, learning_rate = 0.00049999997, loss = 3.1809155e-05, step = 13095 (5.367 sec) 2021-10-19 07:37:12,877 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.148 INFO:tensorflow:global_step/sec: 5.0214 2021-10-19 07:37:15,070 [INFO] tensorflow: global_step/sec: 5.0214 INFO:tensorflow:epoch = 33.994818652849744, learning_rate = 0.00049999997, loss = 3.0433166e-05, step = 13122 (5.385 sec) 2021-10-19 07:37:17,457 [INFO] tensorflow: epoch = 33.994818652849744, learning_rate = 0.00049999997, loss = 3.0433166e-05, step = 13122 (5.385 sec) 2021-10-19 07:37:17,874 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 34/120: loss: 0.00006 learning rate: 0.00050 Time taken: 0:01:17.215139 ETA: 1:50:40.501967 2021-10-19 07:37:17,874 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.012 INFO:tensorflow:global_step/sec: 5.03226 2021-10-19 07:37:22,621 [INFO] tensorflow: global_step/sec: 5.03226 INFO:tensorflow:epoch = 34.06476683937824, learning_rate = 0.00049999997, loss = 5.1064642e-05, step = 13149 (5.378 sec) 2021-10-19 07:37:22,836 [INFO] tensorflow: epoch = 34.06476683937824, learning_rate = 0.00049999997, loss = 5.1064642e-05, step = 13149 (5.378 sec) 2021-10-19 07:37:22,836 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.155 2021-10-19 07:37:27,807 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.119 INFO:tensorflow:epoch = 34.134715025906736, learning_rate = 0.00049999997, loss = 3.4345638e-05, step = 13176 (5.373 sec) 2021-10-19 07:37:28,209 [INFO] tensorflow: epoch = 34.134715025906736, learning_rate = 0.00049999997, loss = 3.4345638e-05, step = 13176 (5.373 sec) INFO:tensorflow:global_step/sec: 4.99873 2021-10-19 07:37:30,223 [INFO] tensorflow: global_step/sec: 4.99873 2021-10-19 07:37:32,835 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.889 INFO:tensorflow:epoch = 34.20207253886011, learning_rate = 0.00049999997, loss = 4.140984e-05, step = 13202 (5.251 sec) 2021-10-19 07:37:33,459 [INFO] tensorflow: epoch = 34.20207253886011, learning_rate = 0.00049999997, loss = 4.140984e-05, step = 13202 (5.251 sec) INFO:tensorflow:global_step/sec: 4.94112 2021-10-19 07:37:37,914 [INFO] tensorflow: global_step/sec: 4.94112 2021-10-19 07:37:37,915 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.688 INFO:tensorflow:epoch = 34.26943005181347, learning_rate = 0.00049999997, loss = 2.9161394e-05, step = 13228 (5.258 sec) 2021-10-19 07:37:38,717 [INFO] tensorflow: epoch = 34.26943005181347, learning_rate = 0.00049999997, loss = 2.9161394e-05, step = 13228 (5.258 sec) 2021-10-19 07:37:42,803 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.460 INFO:tensorflow:epoch = 34.33937823834197, learning_rate = 0.00049999997, loss = 3.8419246e-05, step = 13255 (5.281 sec) 2021-10-19 07:37:43,998 [INFO] tensorflow: epoch = 34.33937823834197, learning_rate = 0.00049999997, loss = 3.8419246e-05, step = 13255 (5.281 sec) INFO:tensorflow:global_step/sec: 5.03774 2021-10-19 07:37:45,457 [INFO] tensorflow: global_step/sec: 5.03774 2021-10-19 07:37:47,828 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.902 INFO:tensorflow:epoch = 34.40673575129534, learning_rate = 0.00049999997, loss = 2.3553775e-05, step = 13281 (5.240 sec) 2021-10-19 07:37:49,238 [INFO] tensorflow: epoch = 34.40673575129534, learning_rate = 0.00049999997, loss = 2.3553775e-05, step = 13281 (5.240 sec) 2021-10-19 07:37:52,854 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.900 INFO:tensorflow:global_step/sec: 4.98514 2021-10-19 07:37:53,080 [INFO] tensorflow: global_step/sec: 4.98514 INFO:tensorflow:epoch = 34.47409326424871, learning_rate = 0.00049999997, loss = 5.0987375e-05, step = 13307 (5.289 sec) 2021-10-19 07:37:54,528 [INFO] tensorflow: epoch = 34.47409326424871, learning_rate = 0.00049999997, loss = 5.0987375e-05, step = 13307 (5.289 sec) 2021-10-19 07:37:57,919 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.746 INFO:tensorflow:epoch = 34.5440414507772, learning_rate = 0.00049999997, loss = 2.5127027e-05, step = 13334 (5.357 sec) 2021-10-19 07:37:59,885 [INFO] tensorflow: epoch = 34.5440414507772, learning_rate = 0.00049999997, loss = 2.5127027e-05, step = 13334 (5.357 sec) INFO:tensorflow:global_step/sec: 4.98851 2021-10-19 07:38:00,697 [INFO] tensorflow: global_step/sec: 4.98851 2021-10-19 07:38:02,952 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.868 INFO:tensorflow:epoch = 34.61139896373057, learning_rate = 0.00049999997, loss = 3.042716e-05, step = 13360 (5.282 sec) 2021-10-19 07:38:05,167 [INFO] tensorflow: epoch = 34.61139896373057, learning_rate = 0.00049999997, loss = 3.042716e-05, step = 13360 (5.282 sec) 2021-10-19 07:38:07,961 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.967 INFO:tensorflow:global_step/sec: 4.96184 2021-10-19 07:38:08,356 [INFO] tensorflow: global_step/sec: 4.96184 INFO:tensorflow:epoch = 34.68134715025907, learning_rate = 0.00049999997, loss = 4.161532e-05, step = 13387 (5.332 sec) 2021-10-19 07:38:10,499 [INFO] tensorflow: epoch = 34.68134715025907, learning_rate = 0.00049999997, loss = 4.161532e-05, step = 13387 (5.332 sec) 2021-10-19 07:38:12,821 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.579 INFO:tensorflow:epoch = 34.751295336787564, learning_rate = 0.00049999997, loss = 3.328584e-05, step = 13414 (5.266 sec) 2021-10-19 07:38:15,765 [INFO] tensorflow: epoch = 34.751295336787564, learning_rate = 0.00049999997, loss = 3.328584e-05, step = 13414 (5.266 sec) INFO:tensorflow:global_step/sec: 5.12739 2021-10-19 07:38:15,767 [INFO] tensorflow: global_step/sec: 5.12739 2021-10-19 07:38:17,713 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.440 INFO:tensorflow:epoch = 34.818652849740936, learning_rate = 0.00049999997, loss = 4.091727e-05, step = 13440 (5.217 sec) 2021-10-19 07:38:20,982 [INFO] tensorflow: epoch = 34.818652849740936, learning_rate = 0.00049999997, loss = 4.091727e-05, step = 13440 (5.217 sec) 2021-10-19 07:38:22,795 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.680 INFO:tensorflow:global_step/sec: 4.96924 2021-10-19 07:38:23,414 [INFO] tensorflow: global_step/sec: 4.96924 INFO:tensorflow:epoch = 34.8860103626943, learning_rate = 0.00049999997, loss = 1.8100105e-05, step = 13466 (5.322 sec) 2021-10-19 07:38:26,305 [INFO] tensorflow: epoch = 34.8860103626943, learning_rate = 0.00049999997, loss = 1.8100105e-05, step = 13466 (5.322 sec) 2021-10-19 07:38:27,887 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.643 INFO:tensorflow:global_step/sec: 4.91219 2021-10-19 07:38:31,150 [INFO] tensorflow: global_step/sec: 4.91219 INFO:tensorflow:epoch = 34.95336787564767, learning_rate = 0.00049999997, loss = 1.724211e-05, step = 13492 (5.266 sec) 2021-10-19 07:38:31,570 [INFO] tensorflow: epoch = 34.95336787564767, learning_rate = 0.00049999997, loss = 1.724211e-05, step = 13492 (5.266 sec) 2021-10-19 07:38:33,002 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.554 2021-10-19 07:38:35,218 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 35/120: loss: 0.00003 learning rate: 0.00050 Time taken: 0:01:17.327876 ETA: 1:49:32.869468 INFO:tensorflow:epoch = 35.020725388601036, learning_rate = 0.00049999997, loss = 4.650526e-05, step = 13518 (5.295 sec) 2021-10-19 07:38:36,865 [INFO] tensorflow: epoch = 35.020725388601036, learning_rate = 0.00049999997, loss = 4.650526e-05, step = 13518 (5.295 sec) 2021-10-19 07:38:38,073 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.722 INFO:tensorflow:global_step/sec: 4.91637 2021-10-19 07:38:38,879 [INFO] tensorflow: global_step/sec: 4.91637 INFO:tensorflow:epoch = 35.08808290155441, learning_rate = 0.00049999997, loss = 3.430067e-05, step = 13544 (5.243 sec) 2021-10-19 07:38:42,108 [INFO] tensorflow: epoch = 35.08808290155441, learning_rate = 0.00049999997, loss = 3.430067e-05, step = 13544 (5.243 sec) 2021-10-19 07:38:43,170 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.620 INFO:tensorflow:global_step/sec: 4.96552 2021-10-19 07:38:46,532 [INFO] tensorflow: global_step/sec: 4.96552 INFO:tensorflow:epoch = 35.15544041450777, learning_rate = 0.00049999997, loss = 2.2453703e-05, step = 13570 (5.215 sec) 2021-10-19 07:38:47,322 [INFO] tensorflow: epoch = 35.15544041450777, learning_rate = 0.00049999997, loss = 2.2453703e-05, step = 13570 (5.215 sec) 2021-10-19 07:38:48,160 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.041 INFO:tensorflow:epoch = 35.222797927461144, learning_rate = 0.00049999997, loss = 4.023357e-05, step = 13596 (5.238 sec) 2021-10-19 07:38:52,561 [INFO] tensorflow: epoch = 35.222797927461144, learning_rate = 0.00049999997, loss = 4.023357e-05, step = 13596 (5.238 sec) 2021-10-19 07:38:53,170 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.961 INFO:tensorflow:global_step/sec: 4.96806 2021-10-19 07:38:54,181 [INFO] tensorflow: global_step/sec: 4.96806 INFO:tensorflow:epoch = 35.29274611398964, learning_rate = 0.00049999997, loss = 3.2653086e-05, step = 13623 (5.402 sec) 2021-10-19 07:38:57,963 [INFO] tensorflow: epoch = 35.29274611398964, learning_rate = 0.00049999997, loss = 3.2653086e-05, step = 13623 (5.402 sec) 2021-10-19 07:38:58,194 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.908 INFO:tensorflow:global_step/sec: 5.00942 2021-10-19 07:39:01,766 [INFO] tensorflow: global_step/sec: 5.00942 INFO:tensorflow:epoch = 35.36010362694301, learning_rate = 0.00049999997, loss = 3.675367e-05, step = 13649 (5.248 sec) 2021-10-19 07:39:03,211 [INFO] tensorflow: epoch = 35.36010362694301, learning_rate = 0.00049999997, loss = 3.675367e-05, step = 13649 (5.248 sec) 2021-10-19 07:39:03,211 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.933 2021-10-19 07:39:08,237 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.900 INFO:tensorflow:epoch = 35.42746113989637, learning_rate = 0.00049999997, loss = 2.4886494e-05, step = 13675 (5.231 sec) 2021-10-19 07:39:08,442 [INFO] tensorflow: epoch = 35.42746113989637, learning_rate = 0.00049999997, loss = 2.4886494e-05, step = 13675 (5.231 sec) INFO:tensorflow:global_step/sec: 4.92798 2021-10-19 07:39:09,477 [INFO] tensorflow: global_step/sec: 4.92798 2021-10-19 07:39:13,207 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.120 INFO:tensorflow:epoch = 35.49740932642487, learning_rate = 0.00049999997, loss = 1.7515453e-05, step = 13702 (5.368 sec) 2021-10-19 07:39:13,810 [INFO] tensorflow: epoch = 35.49740932642487, learning_rate = 0.00049999997, loss = 1.7515453e-05, step = 13702 (5.368 sec) INFO:tensorflow:global_step/sec: 5.0578 2021-10-19 07:39:16,991 [INFO] tensorflow: global_step/sec: 5.0578 2021-10-19 07:39:18,162 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.188 INFO:tensorflow:epoch = 35.56735751295337, learning_rate = 0.00049999997, loss = 3.1105905e-05, step = 13729 (5.340 sec) 2021-10-19 07:39:19,151 [INFO] tensorflow: epoch = 35.56735751295337, learning_rate = 0.00049999997, loss = 3.1105905e-05, step = 13729 (5.340 sec) 2021-10-19 07:39:23,168 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.976 INFO:tensorflow:epoch = 35.634715025906736, learning_rate = 0.00049999997, loss = 3.0394378e-05, step = 13755 (5.238 sec) 2021-10-19 07:39:24,388 [INFO] tensorflow: epoch = 35.634715025906736, learning_rate = 0.00049999997, loss = 3.0394378e-05, step = 13755 (5.238 sec) INFO:tensorflow:global_step/sec: 5.00227 2021-10-19 07:39:24,587 [INFO] tensorflow: global_step/sec: 5.00227 2021-10-19 07:39:28,207 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.850 INFO:tensorflow:epoch = 35.70207253886011, learning_rate = 0.00049999997, loss = 3.5196386e-05, step = 13781 (5.216 sec) 2021-10-19 07:39:29,605 [INFO] tensorflow: epoch = 35.70207253886011, learning_rate = 0.00049999997, loss = 3.5196386e-05, step = 13781 (5.216 sec) INFO:tensorflow:global_step/sec: 5.00619 2021-10-19 07:39:32,178 [INFO] tensorflow: global_step/sec: 5.00619 2021-10-19 07:39:33,182 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.102 INFO:tensorflow:epoch = 35.7720207253886, learning_rate = 0.00049999997, loss = 3.4322366e-05, step = 13808 (5.381 sec) 2021-10-19 07:39:34,985 [INFO] tensorflow: epoch = 35.7720207253886, learning_rate = 0.00049999997, loss = 3.4322366e-05, step = 13808 (5.381 sec) 2021-10-19 07:39:38,176 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.025 INFO:tensorflow:global_step/sec: 4.99654 2021-10-19 07:39:39,783 [INFO] tensorflow: global_step/sec: 4.99654 INFO:tensorflow:epoch = 35.83937823834197, learning_rate = 0.00049999997, loss = 2.8986247e-05, step = 13834 (5.233 sec) 2021-10-19 07:39:40,219 [INFO] tensorflow: epoch = 35.83937823834197, learning_rate = 0.00049999997, loss = 2.8986247e-05, step = 13834 (5.233 sec) 2021-10-19 07:39:43,221 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.824 INFO:tensorflow:epoch = 35.909326424870464, learning_rate = 0.00049999997, loss = 2.0796031e-05, step = 13861 (5.378 sec) 2021-10-19 07:39:45,597 [INFO] tensorflow: epoch = 35.909326424870464, learning_rate = 0.00049999997, loss = 2.0796031e-05, step = 13861 (5.378 sec) INFO:tensorflow:global_step/sec: 4.98383 2021-10-19 07:39:47,408 [INFO] tensorflow: global_step/sec: 4.98383 2021-10-19 07:39:48,187 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.137 INFO:tensorflow:epoch = 35.976683937823836, learning_rate = 0.00049999997, loss = 1.8091238e-05, step = 13887 (5.212 sec) 2021-10-19 07:39:50,809 [INFO] tensorflow: epoch = 35.976683937823836, learning_rate = 0.00049999997, loss = 1.8091238e-05, step = 13887 (5.212 sec) 2021-10-19 07:39:52,656 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 36/120: loss: 0.00005 learning rate: 0.00050 Time taken: 0:01:17.450254 ETA: 1:48:25.821353 2021-10-19 07:39:53,247 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.767 INFO:tensorflow:global_step/sec: 4.98676 2021-10-19 07:39:55,028 [INFO] tensorflow: global_step/sec: 4.98676 INFO:tensorflow:epoch = 36.0440414507772, learning_rate = 0.00049999997, loss = 6.0312515e-05, step = 13913 (5.203 sec) 2021-10-19 07:39:56,013 [INFO] tensorflow: epoch = 36.0440414507772, learning_rate = 0.00049999997, loss = 6.0312515e-05, step = 13913 (5.203 sec) 2021-10-19 07:39:58,221 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.107 INFO:tensorflow:epoch = 36.11139896373057, learning_rate = 0.00049999997, loss = 3.5406414e-05, step = 13939 (5.267 sec) 2021-10-19 07:40:01,280 [INFO] tensorflow: epoch = 36.11139896373057, learning_rate = 0.00049999997, loss = 3.5406414e-05, step = 13939 (5.267 sec) INFO:tensorflow:global_step/sec: 4.95325 2021-10-19 07:40:02,700 [INFO] tensorflow: global_step/sec: 4.95325 2021-10-19 07:40:03,278 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.777 INFO:tensorflow:epoch = 36.18134715025907, learning_rate = 0.00049999997, loss = 2.2870303e-05, step = 13966 (5.362 sec) 2021-10-19 07:40:06,642 [INFO] tensorflow: epoch = 36.18134715025907, learning_rate = 0.00049999997, loss = 2.2870303e-05, step = 13966 (5.362 sec) 2021-10-19 07:40:08,306 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.888 INFO:tensorflow:global_step/sec: 5.00631 2021-10-19 07:40:10,290 [INFO] tensorflow: global_step/sec: 5.00631 INFO:tensorflow:epoch = 36.248704663212436, learning_rate = 0.00049999997, loss = 4.5537003e-05, step = 13992 (5.299 sec) 2021-10-19 07:40:11,941 [INFO] tensorflow: epoch = 36.248704663212436, learning_rate = 0.00049999997, loss = 4.5537003e-05, step = 13992 (5.299 sec) 2021-10-19 07:40:13,354 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.811 INFO:tensorflow:epoch = 36.318652849740936, learning_rate = 0.00049999997, loss = 3.0909036e-05, step = 14019 (5.374 sec) 2021-10-19 07:40:17,315 [INFO] tensorflow: epoch = 36.318652849740936, learning_rate = 0.00049999997, loss = 3.0909036e-05, step = 14019 (5.374 sec) INFO:tensorflow:global_step/sec: 4.99641 2021-10-19 07:40:17,895 [INFO] tensorflow: global_step/sec: 4.99641 2021-10-19 07:40:18,288 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.272 INFO:tensorflow:epoch = 36.38860103626943, learning_rate = 0.00049999997, loss = 2.6363146e-05, step = 14046 (5.351 sec) 2021-10-19 07:40:22,665 [INFO] tensorflow: epoch = 36.38860103626943, learning_rate = 0.00049999997, loss = 2.6363146e-05, step = 14046 (5.351 sec) 2021-10-19 07:40:23,303 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.944 INFO:tensorflow:global_step/sec: 5.02295 2021-10-19 07:40:25,461 [INFO] tensorflow: global_step/sec: 5.02295 INFO:tensorflow:epoch = 36.45854922279793, learning_rate = 0.00049999997, loss = 2.2454311e-05, step = 14073 (5.397 sec) 2021-10-19 07:40:28,062 [INFO] tensorflow: epoch = 36.45854922279793, learning_rate = 0.00049999997, loss = 2.2454311e-05, step = 14073 (5.397 sec) 2021-10-19 07:40:28,296 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.026 INFO:tensorflow:global_step/sec: 4.96071 2021-10-19 07:40:33,121 [INFO] tensorflow: global_step/sec: 4.96071 INFO:tensorflow:epoch = 36.5259067357513, learning_rate = 0.00049999997, loss = 2.2699018e-05, step = 14099 (5.255 sec) 2021-10-19 07:40:33,317 [INFO] tensorflow: epoch = 36.5259067357513, learning_rate = 0.00049999997, loss = 2.2699018e-05, step = 14099 (5.255 sec) 2021-10-19 07:40:33,318 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.917 2021-10-19 07:40:38,373 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.782 INFO:tensorflow:epoch = 36.59326424870466, learning_rate = 0.00049999997, loss = 2.2677861e-05, step = 14125 (5.255 sec) 2021-10-19 07:40:38,572 [INFO] tensorflow: epoch = 36.59326424870466, learning_rate = 0.00049999997, loss = 2.2677861e-05, step = 14125 (5.255 sec) INFO:tensorflow:global_step/sec: 4.97411 2021-10-19 07:40:40,760 [INFO] tensorflow: global_step/sec: 4.97411 2021-10-19 07:40:43,403 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.885 INFO:tensorflow:epoch = 36.660621761658035, learning_rate = 0.00049999997, loss = 3.5694553e-05, step = 14151 (5.228 sec) 2021-10-19 07:40:43,800 [INFO] tensorflow: epoch = 36.660621761658035, learning_rate = 0.00049999997, loss = 3.5694553e-05, step = 14151 (5.228 sec) INFO:tensorflow:global_step/sec: 4.99034 2021-10-19 07:40:48,375 [INFO] tensorflow: global_step/sec: 4.99034 2021-10-19 07:40:48,376 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.109 INFO:tensorflow:epoch = 36.73056994818653, learning_rate = 0.00049999997, loss = 4.7518835e-05, step = 14178 (5.368 sec) 2021-10-19 07:40:49,169 [INFO] tensorflow: epoch = 36.73056994818653, learning_rate = 0.00049999997, loss = 4.7518835e-05, step = 14178 (5.368 sec) 2021-10-19 07:40:53,300 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.311 INFO:tensorflow:epoch = 36.80051813471503, learning_rate = 0.00049999997, loss = 2.9938787e-05, step = 14205 (5.322 sec) 2021-10-19 07:40:54,490 [INFO] tensorflow: epoch = 36.80051813471503, learning_rate = 0.00049999997, loss = 2.9938787e-05, step = 14205 (5.322 sec) INFO:tensorflow:global_step/sec: 5.05709 2021-10-19 07:40:55,889 [INFO] tensorflow: global_step/sec: 5.05709 2021-10-19 07:40:58,295 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.022 INFO:tensorflow:epoch = 36.870466321243526, learning_rate = 0.00049999997, loss = 3.4827957e-05, step = 14232 (5.373 sec) 2021-10-19 07:40:59,864 [INFO] tensorflow: epoch = 36.870466321243526, learning_rate = 0.00049999997, loss = 3.4827957e-05, step = 14232 (5.373 sec) 2021-10-19 07:41:03,350 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.787 INFO:tensorflow:global_step/sec: 4.95822 2021-10-19 07:41:03,553 [INFO] tensorflow: global_step/sec: 4.95822 INFO:tensorflow:epoch = 36.93782383419689, learning_rate = 0.00049999997, loss = 3.336893e-05, step = 14258 (5.311 sec) 2021-10-19 07:41:05,175 [INFO] tensorflow: epoch = 36.93782383419689, learning_rate = 0.00049999997, loss = 3.336893e-05, step = 14258 (5.311 sec) 2021-10-19 07:41:08,300 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.205 2021-10-19 07:41:09,889 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 37/120: loss: 0.00003 learning rate: 0.00050 Time taken: 0:01:17.219040 ETA: 1:46:49.180293 INFO:tensorflow:epoch = 37.00777202072539, learning_rate = 0.00049999997, loss = 2.5121855e-05, step = 14285 (5.316 sec) 2021-10-19 07:41:10,490 [INFO] tensorflow: epoch = 37.00777202072539, learning_rate = 0.00049999997, loss = 2.5121855e-05, step = 14285 (5.316 sec) INFO:tensorflow:global_step/sec: 5.02899 2021-10-19 07:41:11,109 [INFO] tensorflow: global_step/sec: 5.02899 2021-10-19 07:41:13,273 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.108 INFO:tensorflow:epoch = 37.07772020725389, learning_rate = 0.00049999997, loss = 3.310726e-05, step = 14312 (5.353 sec) 2021-10-19 07:41:15,843 [INFO] tensorflow: epoch = 37.07772020725389, learning_rate = 0.00049999997, loss = 3.310726e-05, step = 14312 (5.353 sec) 2021-10-19 07:41:18,176 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.398 INFO:tensorflow:global_step/sec: 5.0963 2021-10-19 07:41:18,566 [INFO] tensorflow: global_step/sec: 5.0963 INFO:tensorflow:epoch = 37.14766839378238, learning_rate = 0.00049999997, loss = 1.9169029e-05, step = 14339 (5.279 sec) 2021-10-19 07:41:21,123 [INFO] tensorflow: epoch = 37.14766839378238, learning_rate = 0.00049999997, loss = 1.9169029e-05, step = 14339 (5.279 sec) 2021-10-19 07:41:23,080 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.390 INFO:tensorflow:global_step/sec: 5.05531 2021-10-19 07:41:26,083 [INFO] tensorflow: global_step/sec: 5.05531 INFO:tensorflow:epoch = 37.21761658031088, learning_rate = 0.00049999997, loss = 1.623296e-05, step = 14366 (5.376 sec) 2021-10-19 07:41:26,499 [INFO] tensorflow: epoch = 37.21761658031088, learning_rate = 0.00049999997, loss = 1.623296e-05, step = 14366 (5.376 sec) 2021-10-19 07:41:28,112 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.878 INFO:tensorflow:epoch = 37.28756476683938, learning_rate = 0.00049999997, loss = 4.3278276e-05, step = 14393 (5.372 sec) 2021-10-19 07:41:31,871 [INFO] tensorflow: epoch = 37.28756476683938, learning_rate = 0.00049999997, loss = 4.3278276e-05, step = 14393 (5.372 sec) 2021-10-19 07:41:33,109 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.013 INFO:tensorflow:global_step/sec: 4.95781 2021-10-19 07:41:33,747 [INFO] tensorflow: global_step/sec: 4.95781 INFO:tensorflow:epoch = 37.354922279792746, learning_rate = 0.00049999997, loss = 3.848884e-05, step = 14419 (5.272 sec) 2021-10-19 07:41:37,143 [INFO] tensorflow: epoch = 37.354922279792746, learning_rate = 0.00049999997, loss = 3.848884e-05, step = 14419 (5.272 sec) 2021-10-19 07:41:38,117 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.970 INFO:tensorflow:global_step/sec: 5.04079 2021-10-19 07:41:41,286 [INFO] tensorflow: global_step/sec: 5.04079 INFO:tensorflow:epoch = 37.424870466321245, learning_rate = 0.00049999997, loss = 2.2625694e-05, step = 14446 (5.346 sec) 2021-10-19 07:41:42,489 [INFO] tensorflow: epoch = 37.424870466321245, learning_rate = 0.00049999997, loss = 2.2625694e-05, step = 14446 (5.346 sec) 2021-10-19 07:41:43,118 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.999 INFO:tensorflow:epoch = 37.49222797927461, learning_rate = 0.00049999997, loss = 4.6614194e-05, step = 14472 (5.254 sec) 2021-10-19 07:41:47,743 [INFO] tensorflow: epoch = 37.49222797927461, learning_rate = 0.00049999997, loss = 4.6614194e-05, step = 14472 (5.254 sec) 2021-10-19 07:41:48,147 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.886 INFO:tensorflow:global_step/sec: 4.96758 2021-10-19 07:41:48,935 [INFO] tensorflow: global_step/sec: 4.96758 INFO:tensorflow:epoch = 37.55958549222798, learning_rate = 0.00049999997, loss = 2.7183332e-05, step = 14498 (5.240 sec) 2021-10-19 07:41:52,984 [INFO] tensorflow: epoch = 37.55958549222798, learning_rate = 0.00049999997, loss = 2.7183332e-05, step = 14498 (5.240 sec) 2021-10-19 07:41:53,189 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.835 INFO:tensorflow:global_step/sec: 4.95313 2021-10-19 07:41:56,607 [INFO] tensorflow: global_step/sec: 4.95313 INFO:tensorflow:epoch = 37.626943005181346, learning_rate = 0.00049999997, loss = 1.9262065e-05, step = 14524 (5.204 sec) 2021-10-19 07:41:58,188 [INFO] tensorflow: epoch = 37.626943005181346, learning_rate = 0.00049999997, loss = 1.9262065e-05, step = 14524 (5.204 sec) 2021-10-19 07:41:58,188 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.004 2021-10-19 07:42:03,110 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.320 INFO:tensorflow:epoch = 37.696891191709845, learning_rate = 0.00049999997, loss = 3.0404775e-05, step = 14551 (5.320 sec) 2021-10-19 07:42:03,508 [INFO] tensorflow: epoch = 37.696891191709845, learning_rate = 0.00049999997, loss = 3.0404775e-05, step = 14551 (5.320 sec) INFO:tensorflow:global_step/sec: 5.06915 2021-10-19 07:42:04,104 [INFO] tensorflow: global_step/sec: 5.06915 2021-10-19 07:42:08,072 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.153 INFO:tensorflow:epoch = 37.76424870466322, learning_rate = 0.00049999997, loss = 3.161363e-05, step = 14577 (5.225 sec) 2021-10-19 07:42:08,732 [INFO] tensorflow: epoch = 37.76424870466322, learning_rate = 0.00049999997, loss = 3.161363e-05, step = 14577 (5.225 sec) INFO:tensorflow:global_step/sec: 4.9957 2021-10-19 07:42:11,710 [INFO] tensorflow: global_step/sec: 4.9957 2021-10-19 07:42:13,106 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.870 INFO:tensorflow:epoch = 37.83419689119171, learning_rate = 0.00049999997, loss = 2.739276e-05, step = 14604 (5.379 sec) 2021-10-19 07:42:14,111 [INFO] tensorflow: epoch = 37.83419689119171, learning_rate = 0.00049999997, loss = 2.739276e-05, step = 14604 (5.379 sec) 2021-10-19 07:42:18,068 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.155 INFO:tensorflow:global_step/sec: 5.04042 2021-10-19 07:42:19,249 [INFO] tensorflow: global_step/sec: 5.04042 INFO:tensorflow:epoch = 37.90414507772021, learning_rate = 0.00049999997, loss = 4.164758e-05, step = 14631 (5.365 sec) 2021-10-19 07:42:19,476 [INFO] tensorflow: epoch = 37.90414507772021, learning_rate = 0.00049999997, loss = 4.164758e-05, step = 14631 (5.365 sec) 2021-10-19 07:42:23,092 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.905 INFO:tensorflow:epoch = 37.97150259067357, learning_rate = 0.00049999997, loss = 3.6438316e-05, step = 14657 (5.200 sec) 2021-10-19 07:42:24,676 [INFO] tensorflow: epoch = 37.97150259067357, learning_rate = 0.00049999997, loss = 3.6438316e-05, step = 14657 (5.200 sec) INFO:tensorflow:global_step/sec: 4.99424 2021-10-19 07:42:26,858 [INFO] tensorflow: global_step/sec: 4.99424 2021-10-19 07:42:26,860 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 38/120: loss: 0.00003 learning rate: 0.00050 Time taken: 0:01:16.966238 ETA: 1:45:11.231518 2021-10-19 07:42:28,032 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.243 INFO:tensorflow:epoch = 38.04145077720207, learning_rate = 0.00049999997, loss = 1.7091339e-05, step = 14684 (5.374 sec) 2021-10-19 07:42:30,050 [INFO] tensorflow: epoch = 38.04145077720207, learning_rate = 0.00049999997, loss = 1.7091339e-05, step = 14684 (5.374 sec) 2021-10-19 07:42:32,995 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.153 INFO:tensorflow:global_step/sec: 5.04542 2021-10-19 07:42:34,390 [INFO] tensorflow: global_step/sec: 5.04542 INFO:tensorflow:epoch = 38.11139896373057, learning_rate = 0.00049999997, loss = 4.7399484e-05, step = 14711 (5.358 sec) 2021-10-19 07:42:35,408 [INFO] tensorflow: epoch = 38.11139896373057, learning_rate = 0.00049999997, loss = 4.7399484e-05, step = 14711 (5.358 sec) 2021-10-19 07:42:37,953 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.170 INFO:tensorflow:epoch = 38.18134715025907, learning_rate = 0.00049999997, loss = 4.5016845e-05, step = 14738 (5.292 sec) 2021-10-19 07:42:40,701 [INFO] tensorflow: epoch = 38.18134715025907, learning_rate = 0.00049999997, loss = 4.5016845e-05, step = 14738 (5.292 sec) INFO:tensorflow:global_step/sec: 5.08124 2021-10-19 07:42:41,868 [INFO] tensorflow: global_step/sec: 5.08124 2021-10-19 07:42:42,855 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.405 INFO:tensorflow:epoch = 38.251295336787564, learning_rate = 0.00049999997, loss = 3.6155725e-05, step = 14765 (5.318 sec) 2021-10-19 07:42:46,019 [INFO] tensorflow: epoch = 38.251295336787564, learning_rate = 0.00049999997, loss = 3.6155725e-05, step = 14765 (5.318 sec) 2021-10-19 07:42:47,811 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.177 INFO:tensorflow:global_step/sec: 5.06302 2021-10-19 07:42:49,374 [INFO] tensorflow: global_step/sec: 5.06302 INFO:tensorflow:epoch = 38.32124352331606, learning_rate = 0.00049999997, loss = 3.6820362e-05, step = 14792 (5.301 sec) 2021-10-19 07:42:51,320 [INFO] tensorflow: epoch = 38.32124352331606, learning_rate = 0.00049999997, loss = 3.6820362e-05, step = 14792 (5.301 sec) 2021-10-19 07:42:52,737 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.305 INFO:tensorflow:epoch = 38.39119170984456, learning_rate = 0.00049999997, loss = 4.9123217e-05, step = 14819 (5.376 sec) 2021-10-19 07:42:56,696 [INFO] tensorflow: epoch = 38.39119170984456, learning_rate = 0.00049999997, loss = 4.9123217e-05, step = 14819 (5.376 sec) INFO:tensorflow:global_step/sec: 5.04709 2021-10-19 07:42:56,903 [INFO] tensorflow: global_step/sec: 5.04709 2021-10-19 07:42:57,689 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.194 INFO:tensorflow:epoch = 38.45854922279793, learning_rate = 0.00049999997, loss = 2.060831e-05, step = 14845 (5.272 sec) 2021-10-19 07:43:01,968 [INFO] tensorflow: epoch = 38.45854922279793, learning_rate = 0.00049999997, loss = 2.060831e-05, step = 14845 (5.272 sec) 2021-10-19 07:43:02,756 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.737 INFO:tensorflow:global_step/sec: 4.97942 2021-10-19 07:43:04,534 [INFO] tensorflow: global_step/sec: 4.97942 INFO:tensorflow:epoch = 38.5259067357513, learning_rate = 0.00049999997, loss = 2.6013395e-05, step = 14871 (5.234 sec) 2021-10-19 07:43:07,202 [INFO] tensorflow: epoch = 38.5259067357513, learning_rate = 0.00049999997, loss = 2.6013395e-05, step = 14871 (5.234 sec) 2021-10-19 07:43:07,788 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.875 INFO:tensorflow:global_step/sec: 4.9714 2021-10-19 07:43:12,178 [INFO] tensorflow: global_step/sec: 4.9714 INFO:tensorflow:epoch = 38.59585492227979, learning_rate = 0.00049999997, loss = 3.9830444e-05, step = 14898 (5.379 sec) 2021-10-19 07:43:12,580 [INFO] tensorflow: epoch = 38.59585492227979, learning_rate = 0.00049999997, loss = 3.9830444e-05, step = 14898 (5.379 sec) 2021-10-19 07:43:12,794 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.977 2021-10-19 07:43:17,768 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.107 INFO:tensorflow:epoch = 38.66580310880829, learning_rate = 0.00049999997, loss = 3.6271515e-05, step = 14925 (5.384 sec) 2021-10-19 07:43:17,965 [INFO] tensorflow: epoch = 38.66580310880829, learning_rate = 0.00049999997, loss = 3.6271515e-05, step = 14925 (5.384 sec) INFO:tensorflow:global_step/sec: 5.02705 2021-10-19 07:43:19,737 [INFO] tensorflow: global_step/sec: 5.02705 2021-10-19 07:43:22,637 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.541 INFO:tensorflow:epoch = 38.73575129533679, learning_rate = 0.00049999997, loss = 5.5925986e-05, step = 14952 (5.265 sec) 2021-10-19 07:43:23,229 [INFO] tensorflow: epoch = 38.73575129533679, learning_rate = 0.00049999997, loss = 5.5925986e-05, step = 14952 (5.265 sec) INFO:tensorflow:global_step/sec: 5.07282 2021-10-19 07:43:27,228 [INFO] tensorflow: global_step/sec: 5.07282 2021-10-19 07:43:27,625 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.048 INFO:tensorflow:epoch = 38.803108808290155, learning_rate = 0.00049999997, loss = 1.7217917e-05, step = 14978 (5.219 sec) 2021-10-19 07:43:28,449 [INFO] tensorflow: epoch = 38.803108808290155, learning_rate = 0.00049999997, loss = 1.7217917e-05, step = 14978 (5.219 sec) 2021-10-19 07:43:32,727 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.605 INFO:tensorflow:epoch = 38.870466321243526, learning_rate = 0.00049999997, loss = 1.9201547e-05, step = 15004 (5.294 sec) 2021-10-19 07:43:33,742 [INFO] tensorflow: epoch = 38.870466321243526, learning_rate = 0.00049999997, loss = 1.9201547e-05, step = 15004 (5.294 sec) INFO:tensorflow:global_step/sec: 4.91926 2021-10-19 07:43:34,952 [INFO] tensorflow: global_step/sec: 4.91926 2021-10-19 07:43:37,798 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.722 INFO:tensorflow:epoch = 38.93782383419689, learning_rate = 0.00049999997, loss = 5.266599e-05, step = 15030 (5.233 sec) 2021-10-19 07:43:38,975 [INFO] tensorflow: epoch = 38.93782383419689, learning_rate = 0.00049999997, loss = 5.266599e-05, step = 15030 (5.233 sec) INFO:tensorflow:global_step/sec: 4.96035 2021-10-19 07:43:42,613 [INFO] tensorflow: global_step/sec: 4.96035 2021-10-19 07:43:42,838 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.841 2021-10-19 07:43:43,858 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 39/120: loss: 0.00003 learning rate: 0.00050 Time taken: 0:01:17.000403 ETA: 1:43:57.032656 INFO:tensorflow:epoch = 39.00518134715026, learning_rate = 0.00049999997, loss = 4.6592875e-05, step = 15056 (5.318 sec) 2021-10-19 07:43:44,293 [INFO] tensorflow: epoch = 39.00518134715026, learning_rate = 0.00049999997, loss = 4.6592875e-05, step = 15056 (5.318 sec) 2021-10-19 07:43:47,887 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.807 INFO:tensorflow:epoch = 39.075129533678755, learning_rate = 0.00049999997, loss = 3.7278514e-05, step = 15083 (5.340 sec) 2021-10-19 07:43:49,633 [INFO] tensorflow: epoch = 39.075129533678755, learning_rate = 0.00049999997, loss = 3.7278514e-05, step = 15083 (5.340 sec) INFO:tensorflow:global_step/sec: 4.98283 2021-10-19 07:43:50,239 [INFO] tensorflow: global_step/sec: 4.98283 2021-10-19 07:43:52,785 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.420 INFO:tensorflow:epoch = 39.145077720207254, learning_rate = 0.00049999997, loss = 3.73344e-05, step = 15110 (5.376 sec) 2021-10-19 07:43:55,008 [INFO] tensorflow: epoch = 39.145077720207254, learning_rate = 0.00049999997, loss = 3.73344e-05, step = 15110 (5.376 sec) INFO:tensorflow:global_step/sec: 4.9853 2021-10-19 07:43:57,862 [INFO] tensorflow: global_step/sec: 4.9853 2021-10-19 07:43:57,863 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.693 INFO:tensorflow:epoch = 39.212435233160626, learning_rate = 0.00049999997, loss = 3.799092e-05, step = 15136 (5.270 sec) 2021-10-19 07:44:00,278 [INFO] tensorflow: epoch = 39.212435233160626, learning_rate = 0.00049999997, loss = 3.799092e-05, step = 15136 (5.270 sec) 2021-10-19 07:44:02,852 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.048 INFO:tensorflow:epoch = 39.27979274611399, learning_rate = 0.00049999997, loss = 2.134826e-05, step = 15162 (5.217 sec) 2021-10-19 07:44:05,496 [INFO] tensorflow: epoch = 39.27979274611399, learning_rate = 0.00049999997, loss = 2.134826e-05, step = 15162 (5.217 sec) INFO:tensorflow:global_step/sec: 4.97673 2021-10-19 07:44:05,497 [INFO] tensorflow: global_step/sec: 4.97673 2021-10-19 07:44:07,902 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.801 INFO:tensorflow:epoch = 39.34974093264249, learning_rate = 0.00049999997, loss = 2.058862e-05, step = 15189 (5.359 sec) 2021-10-19 07:44:10,854 [INFO] tensorflow: epoch = 39.34974093264249, learning_rate = 0.00049999997, loss = 2.058862e-05, step = 15189 (5.359 sec) 2021-10-19 07:44:12,833 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.283 INFO:tensorflow:global_step/sec: 5.02867 2021-10-19 07:44:13,054 [INFO] tensorflow: global_step/sec: 5.02867 INFO:tensorflow:epoch = 39.41968911917098, learning_rate = 0.00049999997, loss = 3.2370175e-05, step = 15216 (5.345 sec) 2021-10-19 07:44:16,199 [INFO] tensorflow: epoch = 39.41968911917098, learning_rate = 0.00049999997, loss = 3.2370175e-05, step = 15216 (5.345 sec) 2021-10-19 07:44:17,771 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.255 INFO:tensorflow:global_step/sec: 5.05668 2021-10-19 07:44:20,569 [INFO] tensorflow: global_step/sec: 5.05668 INFO:tensorflow:epoch = 39.48963730569948, learning_rate = 0.00049999997, loss = 1.6021491e-05, step = 15243 (5.397 sec) 2021-10-19 07:44:21,596 [INFO] tensorflow: epoch = 39.48963730569948, learning_rate = 0.00049999997, loss = 1.6021491e-05, step = 15243 (5.397 sec) 2021-10-19 07:44:22,795 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.905 INFO:tensorflow:epoch = 39.55699481865285, learning_rate = 0.00049999997, loss = 2.7209015e-05, step = 15269 (5.207 sec) 2021-10-19 07:44:26,803 [INFO] tensorflow: epoch = 39.55699481865285, learning_rate = 0.00049999997, loss = 2.7209015e-05, step = 15269 (5.207 sec) 2021-10-19 07:44:27,818 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.910 INFO:tensorflow:global_step/sec: 4.96743 2021-10-19 07:44:28,219 [INFO] tensorflow: global_step/sec: 4.96743 INFO:tensorflow:epoch = 39.62435233160622, learning_rate = 0.00049999997, loss = 1.9745681e-05, step = 15295 (5.238 sec) 2021-10-19 07:44:32,041 [INFO] tensorflow: epoch = 39.62435233160622, learning_rate = 0.00049999997, loss = 1.9745681e-05, step = 15295 (5.238 sec) 2021-10-19 07:44:32,833 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.944 INFO:tensorflow:global_step/sec: 4.99627 2021-10-19 07:44:35,824 [INFO] tensorflow: global_step/sec: 4.99627 INFO:tensorflow:epoch = 39.69430051813472, learning_rate = 0.00049999997, loss = 2.0919004e-05, step = 15322 (5.417 sec) 2021-10-19 07:44:37,459 [INFO] tensorflow: epoch = 39.69430051813472, learning_rate = 0.00049999997, loss = 2.0919004e-05, step = 15322 (5.417 sec) 2021-10-19 07:44:37,852 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.923 INFO:tensorflow:epoch = 39.76424870466322, learning_rate = 0.00049999997, loss = 2.4966295e-05, step = 15349 (5.298 sec) 2021-10-19 07:44:42,757 [INFO] tensorflow: epoch = 39.76424870466322, learning_rate = 0.00049999997, loss = 2.4966295e-05, step = 15349 (5.298 sec) 2021-10-19 07:44:42,757 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.390 INFO:tensorflow:global_step/sec: 5.04497 2021-10-19 07:44:43,357 [INFO] tensorflow: global_step/sec: 5.04497 2021-10-19 07:44:47,747 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.040 INFO:tensorflow:epoch = 39.83419689119171, learning_rate = 0.00049999997, loss = 2.9664561e-05, step = 15376 (5.387 sec) 2021-10-19 07:44:48,144 [INFO] tensorflow: epoch = 39.83419689119171, learning_rate = 0.00049999997, loss = 2.9664561e-05, step = 15376 (5.387 sec) INFO:tensorflow:global_step/sec: 5.03171 2021-10-19 07:44:50,909 [INFO] tensorflow: global_step/sec: 5.03171 2021-10-19 07:44:52,688 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.240 INFO:tensorflow:epoch = 39.90414507772021, learning_rate = 0.00049999997, loss = 2.3316083e-05, step = 15403 (5.374 sec) 2021-10-19 07:44:53,518 [INFO] tensorflow: epoch = 39.90414507772021, learning_rate = 0.00049999997, loss = 2.3316083e-05, step = 15403 (5.374 sec) 2021-10-19 07:44:57,642 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.188 INFO:tensorflow:global_step/sec: 5.01966 2021-10-19 07:44:58,479 [INFO] tensorflow: global_step/sec: 5.01966 INFO:tensorflow:epoch = 39.97409326424871, learning_rate = 0.00049999997, loss = 3.340726e-05, step = 15430 (5.391 sec) 2021-10-19 07:44:58,909 [INFO] tensorflow: epoch = 39.97409326424871, learning_rate = 0.00049999997, loss = 3.340726e-05, step = 15430 (5.391 sec) INFO:tensorflow:Saving checkpoints for step-15440. 2021-10-19 07:45:00,708 [INFO] tensorflow: Saving checkpoints for step-15440. 2021-10-19 07:45:05,352 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 62, 0.00s/step 2021-10-19 07:45:06,217 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 62, 0.09s/step 2021-10-19 07:45:07,132 [INFO] iva.detectnet_v2.evaluation.evaluation: step 20 / 62, 0.09s/step 2021-10-19 07:45:08,013 [INFO] iva.detectnet_v2.evaluation.evaluation: step 30 / 62, 0.09s/step 2021-10-19 07:45:08,957 [INFO] iva.detectnet_v2.evaluation.evaluation: step 40 / 62, 0.09s/step 2021-10-19 07:45:09,864 [INFO] iva.detectnet_v2.evaluation.evaluation: step 50 / 62, 0.09s/step 2021-10-19 07:45:10,830 [INFO] iva.detectnet_v2.evaluation.evaluation: step 60 / 62, 0.10s/step Matching predictions to ground truth, class 1/7.: 100%|█| 325/325 [00:00<00:00, 13900.04it/s] Matching predictions to ground truth, class 2/7.: 100%|█| 223/223 [00:00<00:00, 15425.32it/s] Matching predictions to ground truth, class 7/7.: 100%|█| 19/19 [00:00<00:00, 18294.71it/s] Epoch 40/120 ========================= Validation cost: 0.000025 Mean average_precision (in %): 14.8508 class name average precision (in %) ------------- -------------------------- auto_rickshaw 0 bicycle 57.2439 bus 12.5 car 34.2115 person 0 road_sign 0 truck 0 Median Inference Time: 0.019540 INFO:tensorflow:epoch = 40.0, learning_rate = 0.00049999997, loss = 2.6390211e-05, step = 15440 (12.401 sec) 2021-10-19 07:45:11,309 [INFO] tensorflow: epoch = 40.0, learning_rate = 0.00049999997, loss = 2.6390211e-05, step = 15440 (12.401 sec) 2021-10-19 07:45:11,310 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 40/120: loss: 0.00003 learning rate: 0.00050 Time taken: 0:01:27.430343 ETA: 1:56:34.427433 2021-10-19 07:45:13,049 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 6.491 INFO:tensorflow:global_step/sec: 2.1139 2021-10-19 07:45:16,455 [INFO] tensorflow: global_step/sec: 2.1139 INFO:tensorflow:epoch = 40.0699481865285, learning_rate = 0.00049999997, loss = 2.0082562e-05, step = 15467 (5.385 sec) 2021-10-19 07:45:16,694 [INFO] tensorflow: epoch = 40.0699481865285, learning_rate = 0.00049999997, loss = 2.0082562e-05, step = 15467 (5.385 sec) 2021-10-19 07:45:18,124 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.707 INFO:tensorflow:epoch = 40.137305699481864, learning_rate = 0.00049999997, loss = 1.6651868e-05, step = 15493 (5.277 sec) 2021-10-19 07:45:21,971 [INFO] tensorflow: epoch = 40.137305699481864, learning_rate = 0.00049999997, loss = 1.6651868e-05, step = 15493 (5.277 sec) 2021-10-19 07:45:23,185 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.758 INFO:tensorflow:global_step/sec: 4.92377 2021-10-19 07:45:24,173 [INFO] tensorflow: global_step/sec: 4.92377 INFO:tensorflow:epoch = 40.204663212435236, learning_rate = 0.00049999997, loss = 1.8559413e-05, step = 15519 (5.220 sec) 2021-10-19 07:45:27,190 [INFO] tensorflow: epoch = 40.204663212435236, learning_rate = 0.00049999997, loss = 1.8559413e-05, step = 15519 (5.220 sec) 2021-10-19 07:45:28,153 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.131 INFO:tensorflow:global_step/sec: 4.99422 2021-10-19 07:45:31,782 [INFO] tensorflow: global_step/sec: 4.99422 INFO:tensorflow:epoch = 40.274611398963735, learning_rate = 0.00049999997, loss = 3.0136001e-05, step = 15546 (5.385 sec) 2021-10-19 07:45:32,575 [INFO] tensorflow: epoch = 40.274611398963735, learning_rate = 0.00049999997, loss = 3.0136001e-05, step = 15546 (5.385 sec) 2021-10-19 07:45:33,172 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.927 INFO:tensorflow:epoch = 40.3419689119171, learning_rate = 0.00049999997, loss = 1.5735965e-05, step = 15572 (5.214 sec) 2021-10-19 07:45:37,789 [INFO] tensorflow: epoch = 40.3419689119171, learning_rate = 0.00049999997, loss = 1.5735965e-05, step = 15572 (5.214 sec) 2021-10-19 07:45:38,190 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.929 INFO:tensorflow:global_step/sec: 5.00762 2021-10-19 07:45:39,370 [INFO] tensorflow: global_step/sec: 5.00762 INFO:tensorflow:epoch = 40.4119170984456, learning_rate = 0.00049999997, loss = 2.7172407e-05, step = 15599 (5.352 sec) 2021-10-19 07:45:43,141 [INFO] tensorflow: epoch = 40.4119170984456, learning_rate = 0.00049999997, loss = 2.7172407e-05, step = 15599 (5.352 sec) 2021-10-19 07:45:43,141 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.200 INFO:tensorflow:global_step/sec: 5.04025 2021-10-19 07:45:46,909 [INFO] tensorflow: global_step/sec: 5.04025 2021-10-19 07:45:48,093 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.197 INFO:tensorflow:epoch = 40.48186528497409, learning_rate = 0.00049999997, loss = 3.7105885e-05, step = 15626 (5.347 sec) 2021-10-19 07:45:48,488 [INFO] tensorflow: epoch = 40.48186528497409, learning_rate = 0.00049999997, loss = 3.7105885e-05, step = 15626 (5.347 sec) 2021-10-19 07:45:53,069 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.097 INFO:tensorflow:epoch = 40.55181347150259, learning_rate = 0.00049999997, loss = 3.503741e-05, step = 15653 (5.373 sec) 2021-10-19 07:45:53,861 [INFO] tensorflow: epoch = 40.55181347150259, learning_rate = 0.00049999997, loss = 3.503741e-05, step = 15653 (5.373 sec) INFO:tensorflow:global_step/sec: 5.04427 2021-10-19 07:45:54,443 [INFO] tensorflow: global_step/sec: 5.04427 2021-10-19 07:45:57,999 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.286 INFO:tensorflow:epoch = 40.62176165803109, learning_rate = 0.00049999997, loss = 4.8373557e-05, step = 15680 (5.362 sec) 2021-10-19 07:45:59,223 [INFO] tensorflow: epoch = 40.62176165803109, learning_rate = 0.00049999997, loss = 4.8373557e-05, step = 15680 (5.362 sec) INFO:tensorflow:global_step/sec: 5.01391 2021-10-19 07:46:02,022 [INFO] tensorflow: global_step/sec: 5.01391 2021-10-19 07:46:03,031 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.874 INFO:tensorflow:epoch = 40.689119170984455, learning_rate = 0.00049999997, loss = 6.086425e-05, step = 15706 (5.224 sec) 2021-10-19 07:46:04,447 [INFO] tensorflow: epoch = 40.689119170984455, learning_rate = 0.00049999997, loss = 6.086425e-05, step = 15706 (5.224 sec) 2021-10-19 07:46:08,053 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.914 INFO:tensorflow:epoch = 40.75647668393783, learning_rate = 0.00049999997, loss = 2.9177556e-05, step = 15732 (5.194 sec) 2021-10-19 07:46:09,641 [INFO] tensorflow: epoch = 40.75647668393783, learning_rate = 0.00049999997, loss = 2.9177556e-05, step = 15732 (5.194 sec) INFO:tensorflow:global_step/sec: 4.98636 2021-10-19 07:46:09,642 [INFO] tensorflow: global_step/sec: 4.98636 2021-10-19 07:46:12,964 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.365 INFO:tensorflow:epoch = 40.826424870466326, learning_rate = 0.00049999997, loss = 2.744271e-05, step = 15759 (5.337 sec) 2021-10-19 07:46:14,978 [INFO] tensorflow: epoch = 40.826424870466326, learning_rate = 0.00049999997, loss = 2.744271e-05, step = 15759 (5.337 sec) INFO:tensorflow:global_step/sec: 5.03953 2021-10-19 07:46:17,183 [INFO] tensorflow: global_step/sec: 5.03953 2021-10-19 07:46:18,011 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.817 INFO:tensorflow:epoch = 40.89637305699482, learning_rate = 0.00049999997, loss = 2.0401454e-05, step = 15786 (5.383 sec) 2021-10-19 07:46:20,361 [INFO] tensorflow: epoch = 40.89637305699482, learning_rate = 0.00049999997, loss = 2.0401454e-05, step = 15786 (5.383 sec) 2021-10-19 07:46:22,920 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.374 INFO:tensorflow:global_step/sec: 5.02996 2021-10-19 07:46:24,738 [INFO] tensorflow: global_step/sec: 5.02996 INFO:tensorflow:epoch = 40.96632124352332, learning_rate = 0.00049999997, loss = 3.635335e-05, step = 15813 (5.367 sec) 2021-10-19 07:46:25,728 [INFO] tensorflow: epoch = 40.96632124352332, learning_rate = 0.00049999997, loss = 3.635335e-05, step = 15813 (5.367 sec) 2021-10-19 07:46:27,977 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.774 2021-10-19 07:46:28,390 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 41/120: loss: 0.00004 learning rate: 0.00050 Time taken: 0:01:17.096684 ETA: 1:41:30.638034 INFO:tensorflow:epoch = 41.03367875647668, learning_rate = 0.00049999997, loss = 4.823583e-05, step = 15839 (5.273 sec) 2021-10-19 07:46:31,001 [INFO] tensorflow: epoch = 41.03367875647668, learning_rate = 0.00049999997, loss = 4.823583e-05, step = 15839 (5.273 sec) INFO:tensorflow:global_step/sec: 4.94519 2021-10-19 07:46:32,422 [INFO] tensorflow: global_step/sec: 4.94519 2021-10-19 07:46:33,024 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.816 INFO:tensorflow:epoch = 41.10362694300518, learning_rate = 0.00049999997, loss = 2.0728721e-05, step = 15866 (5.394 sec) 2021-10-19 07:46:36,395 [INFO] tensorflow: epoch = 41.10362694300518, learning_rate = 0.00049999997, loss = 2.0728721e-05, step = 15866 (5.394 sec) 2021-10-19 07:46:38,013 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.046 INFO:tensorflow:global_step/sec: 5.00263 2021-10-19 07:46:40,018 [INFO] tensorflow: global_step/sec: 5.00263 INFO:tensorflow:epoch = 41.17098445595855, learning_rate = 0.00049999997, loss = 2.3015453e-05, step = 15892 (5.214 sec) 2021-10-19 07:46:41,609 [INFO] tensorflow: epoch = 41.17098445595855, learning_rate = 0.00049999997, loss = 2.3015453e-05, step = 15892 (5.214 sec) 2021-10-19 07:46:43,023 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.962 INFO:tensorflow:epoch = 41.23834196891192, learning_rate = 0.00049999997, loss = 1.4814591e-05, step = 15918 (5.307 sec) 2021-10-19 07:46:46,915 [INFO] tensorflow: epoch = 41.23834196891192, learning_rate = 0.00049999997, loss = 1.4814591e-05, step = 15918 (5.307 sec) INFO:tensorflow:global_step/sec: 4.93466 2021-10-19 07:46:47,718 [INFO] tensorflow: global_step/sec: 4.93466 2021-10-19 07:46:48,131 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.580 INFO:tensorflow:epoch = 41.30569948186529, learning_rate = 0.00049999997, loss = 4.2493924e-05, step = 15944 (5.266 sec) 2021-10-19 07:46:52,182 [INFO] tensorflow: epoch = 41.30569948186529, learning_rate = 0.00049999997, loss = 4.2493924e-05, step = 15944 (5.266 sec) 2021-10-19 07:46:53,171 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.842 INFO:tensorflow:global_step/sec: 4.94604 2021-10-19 07:46:55,401 [INFO] tensorflow: global_step/sec: 4.94604 INFO:tensorflow:epoch = 41.373056994818654, learning_rate = 0.00049999997, loss = 6.037445e-05, step = 15970 (5.301 sec) 2021-10-19 07:46:57,483 [INFO] tensorflow: epoch = 41.373056994818654, learning_rate = 0.00049999997, loss = 6.037445e-05, step = 15970 (5.301 sec) 2021-10-19 07:46:58,290 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.538 INFO:tensorflow:epoch = 41.443005181347154, learning_rate = 0.00049999997, loss = 4.0533952e-05, step = 15997 (5.354 sec) 2021-10-19 07:47:02,837 [INFO] tensorflow: epoch = 41.443005181347154, learning_rate = 0.00049999997, loss = 4.0533952e-05, step = 15997 (5.354 sec) INFO:tensorflow:global_step/sec: 4.97563 2021-10-19 07:47:03,039 [INFO] tensorflow: global_step/sec: 4.97563 2021-10-19 07:47:03,258 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.129 INFO:tensorflow:epoch = 41.51036269430052, learning_rate = 0.00049999997, loss = 1.620262e-05, step = 16023 (5.227 sec) 2021-10-19 07:47:08,064 [INFO] tensorflow: epoch = 41.51036269430052, learning_rate = 0.00049999997, loss = 1.620262e-05, step = 16023 (5.227 sec) 2021-10-19 07:47:08,266 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.971 INFO:tensorflow:global_step/sec: 4.98273 2021-10-19 07:47:10,665 [INFO] tensorflow: global_step/sec: 4.98273 2021-10-19 07:47:13,251 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.062 INFO:tensorflow:epoch = 41.58031088082902, learning_rate = 0.00049999997, loss = 4.0913554e-05, step = 16050 (5.393 sec) 2021-10-19 07:47:13,457 [INFO] tensorflow: epoch = 41.58031088082902, learning_rate = 0.00049999997, loss = 4.0913554e-05, step = 16050 (5.393 sec) INFO:tensorflow:global_step/sec: 4.9485 2021-10-19 07:47:18,344 [INFO] tensorflow: global_step/sec: 4.9485 2021-10-19 07:47:18,345 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.632 INFO:tensorflow:epoch = 41.64766839378238, learning_rate = 0.00049999997, loss = 2.879373e-05, step = 16076 (5.284 sec) 2021-10-19 07:47:18,740 [INFO] tensorflow: epoch = 41.64766839378238, learning_rate = 0.00049999997, loss = 2.879373e-05, step = 16076 (5.284 sec) 2021-10-19 07:47:23,262 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.341 INFO:tensorflow:epoch = 41.71761658031088, learning_rate = 0.00049999997, loss = 3.4393495e-05, step = 16103 (5.317 sec) 2021-10-19 07:47:24,057 [INFO] tensorflow: epoch = 41.71761658031088, learning_rate = 0.00049999997, loss = 3.4393495e-05, step = 16103 (5.317 sec) INFO:tensorflow:global_step/sec: 5.07876 2021-10-19 07:47:25,826 [INFO] tensorflow: global_step/sec: 5.07876 2021-10-19 07:47:28,215 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.191 INFO:tensorflow:epoch = 41.78756476683938, learning_rate = 0.00049999997, loss = 2.787644e-05, step = 16130 (5.355 sec) 2021-10-19 07:47:29,412 [INFO] tensorflow: epoch = 41.78756476683938, learning_rate = 0.00049999997, loss = 2.787644e-05, step = 16130 (5.355 sec) 2021-10-19 07:47:33,291 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.701 INFO:tensorflow:global_step/sec: 4.95215 2021-10-19 07:47:33,500 [INFO] tensorflow: global_step/sec: 4.95215 INFO:tensorflow:epoch = 41.854922279792746, learning_rate = 0.00049999997, loss = 3.072584e-05, step = 16156 (5.282 sec) 2021-10-19 07:47:34,694 [INFO] tensorflow: epoch = 41.854922279792746, learning_rate = 0.00049999997, loss = 3.072584e-05, step = 16156 (5.282 sec) 2021-10-19 07:47:38,224 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.274 INFO:tensorflow:epoch = 41.924870466321245, learning_rate = 0.00049999997, loss = 2.8070495e-05, step = 16183 (5.292 sec) 2021-10-19 07:47:39,986 [INFO] tensorflow: epoch = 41.924870466321245, learning_rate = 0.00049999997, loss = 2.8070495e-05, step = 16183 (5.292 sec) INFO:tensorflow:global_step/sec: 5.09412 2021-10-19 07:47:40,959 [INFO] tensorflow: global_step/sec: 5.09412 2021-10-19 07:47:43,145 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.324 INFO:tensorflow:epoch = 41.994818652849744, learning_rate = 0.00049999997, loss = 3.3734756e-05, step = 16210 (5.302 sec) 2021-10-19 07:47:45,288 [INFO] tensorflow: epoch = 41.994818652849744, learning_rate = 0.00049999997, loss = 3.3734756e-05, step = 16210 (5.302 sec) 2021-10-19 07:47:45,694 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 42/120: loss: 0.00003 learning rate: 0.00050 Time taken: 0:01:17.313671 ETA: 1:40:30.466328 2021-10-19 07:47:48,061 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.344 INFO:tensorflow:global_step/sec: 5.05761 2021-10-19 07:47:48,473 [INFO] tensorflow: global_step/sec: 5.05761 INFO:tensorflow:epoch = 42.06217616580311, learning_rate = 0.00049999997, loss = 3.9573744e-05, step = 16236 (5.271 sec) 2021-10-19 07:47:50,559 [INFO] tensorflow: epoch = 42.06217616580311, learning_rate = 0.00049999997, loss = 3.9573744e-05, step = 16236 (5.271 sec) 2021-10-19 07:47:53,171 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.569 INFO:tensorflow:epoch = 42.13212435233161, learning_rate = 0.00049999997, loss = 1.6810503e-05, step = 16263 (5.354 sec) 2021-10-19 07:47:55,912 [INFO] tensorflow: epoch = 42.13212435233161, learning_rate = 0.00049999997, loss = 1.6810503e-05, step = 16263 (5.354 sec) INFO:tensorflow:global_step/sec: 4.97447 2021-10-19 07:47:56,112 [INFO] tensorflow: global_step/sec: 4.97447 2021-10-19 07:47:58,119 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.213 INFO:tensorflow:epoch = 42.19948186528497, learning_rate = 0.00049999997, loss = 1.7372025e-05, step = 16289 (5.309 sec) 2021-10-19 07:48:01,221 [INFO] tensorflow: epoch = 42.19948186528497, learning_rate = 0.00049999997, loss = 1.7372025e-05, step = 16289 (5.309 sec) 2021-10-19 07:48:03,213 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.633 INFO:tensorflow:global_step/sec: 4.9325 2021-10-19 07:48:03,816 [INFO] tensorflow: global_step/sec: 4.9325 INFO:tensorflow:epoch = 42.26943005181347, learning_rate = 0.00049999997, loss = 1.6221577e-05, step = 16316 (5.396 sec) 2021-10-19 07:48:06,617 [INFO] tensorflow: epoch = 42.26943005181347, learning_rate = 0.00049999997, loss = 1.6221577e-05, step = 16316 (5.396 sec) 2021-10-19 07:48:08,228 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.939 INFO:tensorflow:global_step/sec: 5.00621 2021-10-19 07:48:11,406 [INFO] tensorflow: global_step/sec: 5.00621 INFO:tensorflow:epoch = 42.336787564766844, learning_rate = 0.00049999997, loss = 1.4730292e-05, step = 16342 (5.212 sec) 2021-10-19 07:48:11,829 [INFO] tensorflow: epoch = 42.336787564766844, learning_rate = 0.00049999997, loss = 1.4730292e-05, step = 16342 (5.212 sec) 2021-10-19 07:48:13,225 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.017 INFO:tensorflow:epoch = 42.40414507772021, learning_rate = 0.00049999997, loss = 2.9217623e-05, step = 16368 (5.227 sec) 2021-10-19 07:48:17,056 [INFO] tensorflow: epoch = 42.40414507772021, learning_rate = 0.00049999997, loss = 2.9217623e-05, step = 16368 (5.227 sec) 2021-10-19 07:48:18,238 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.949 INFO:tensorflow:global_step/sec: 4.94594 2021-10-19 07:48:19,089 [INFO] tensorflow: global_step/sec: 4.94594 INFO:tensorflow:epoch = 42.47150259067357, learning_rate = 0.00049999997, loss = 4.5799716e-05, step = 16394 (5.236 sec) 2021-10-19 07:48:22,292 [INFO] tensorflow: epoch = 42.47150259067357, learning_rate = 0.00049999997, loss = 4.5799716e-05, step = 16394 (5.236 sec) 2021-10-19 07:48:23,283 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.822 INFO:tensorflow:global_step/sec: 5.05324 2021-10-19 07:48:26,609 [INFO] tensorflow: global_step/sec: 5.05324 INFO:tensorflow:epoch = 42.54145077720207, learning_rate = 0.00049999997, loss = 3.162577e-05, step = 16421 (5.341 sec) 2021-10-19 07:48:27,634 [INFO] tensorflow: epoch = 42.54145077720207, learning_rate = 0.00049999997, loss = 3.162577e-05, step = 16421 (5.341 sec) 2021-10-19 07:48:28,220 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.259 INFO:tensorflow:epoch = 42.61139896373057, learning_rate = 0.00049999997, loss = 2.2557952e-05, step = 16448 (5.366 sec) 2021-10-19 07:48:32,999 [INFO] tensorflow: epoch = 42.61139896373057, learning_rate = 0.00049999997, loss = 2.2557952e-05, step = 16448 (5.366 sec) 2021-10-19 07:48:33,218 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.010 INFO:tensorflow:global_step/sec: 4.98527 2021-10-19 07:48:34,232 [INFO] tensorflow: global_step/sec: 4.98527 INFO:tensorflow:epoch = 42.67875647668394, learning_rate = 0.00049999997, loss = 5.2745523e-05, step = 16474 (5.218 sec) 2021-10-19 07:48:38,217 [INFO] tensorflow: epoch = 42.67875647668394, learning_rate = 0.00049999997, loss = 5.2745523e-05, step = 16474 (5.218 sec) 2021-10-19 07:48:38,217 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.003 INFO:tensorflow:global_step/sec: 5.00267 2021-10-19 07:48:41,828 [INFO] tensorflow: global_step/sec: 5.00267 2021-10-19 07:48:43,199 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.074 INFO:tensorflow:epoch = 42.748704663212436, learning_rate = 0.00049999997, loss = 3.564202e-05, step = 16501 (5.380 sec) 2021-10-19 07:48:43,597 [INFO] tensorflow: epoch = 42.748704663212436, learning_rate = 0.00049999997, loss = 3.564202e-05, step = 16501 (5.380 sec) 2021-10-19 07:48:48,192 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.031 INFO:tensorflow:epoch = 42.818652849740936, learning_rate = 0.00049999997, loss = 2.5477326e-05, step = 16528 (5.396 sec) 2021-10-19 07:48:48,993 [INFO] tensorflow: epoch = 42.818652849740936, learning_rate = 0.00049999997, loss = 2.5477326e-05, step = 16528 (5.396 sec) INFO:tensorflow:global_step/sec: 5.00984 2021-10-19 07:48:49,413 [INFO] tensorflow: global_step/sec: 5.00984 2021-10-19 07:48:53,149 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.177 INFO:tensorflow:epoch = 42.888601036269435, learning_rate = 0.00049999997, loss = 6.0078673e-05, step = 16555 (5.351 sec) 2021-10-19 07:48:54,344 [INFO] tensorflow: epoch = 42.888601036269435, learning_rate = 0.00049999997, loss = 6.0078673e-05, step = 16555 (5.351 sec) INFO:tensorflow:global_step/sec: 5.04575 2021-10-19 07:48:56,944 [INFO] tensorflow: global_step/sec: 5.04575 2021-10-19 07:48:58,169 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.919 INFO:tensorflow:epoch = 42.9559585492228, learning_rate = 0.00049999997, loss = 4.058475e-05, step = 16581 (5.245 sec) 2021-10-19 07:48:59,589 [INFO] tensorflow: epoch = 42.9559585492228, learning_rate = 0.00049999997, loss = 4.058475e-05, step = 16581 (5.245 sec) 2021-10-19 07:49:02,999 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 43/120: loss: 0.00004 learning rate: 0.00050 Time taken: 0:01:17.301296 ETA: 1:39:12.199755 2021-10-19 07:49:03,203 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.870 INFO:tensorflow:global_step/sec: 4.96003 2021-10-19 07:49:04,605 [INFO] tensorflow: global_step/sec: 4.96003 INFO:tensorflow:epoch = 43.023316062176164, learning_rate = 0.00049999997, loss = 2.2313461e-05, step = 16607 (5.246 sec) 2021-10-19 07:49:04,834 [INFO] tensorflow: epoch = 43.023316062176164, learning_rate = 0.00049999997, loss = 2.2313461e-05, step = 16607 (5.246 sec) 2021-10-19 07:49:08,282 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.692 INFO:tensorflow:epoch = 43.090673575129536, learning_rate = 0.00049999997, loss = 2.8315255e-05, step = 16633 (5.236 sec) 2021-10-19 07:49:10,070 [INFO] tensorflow: epoch = 43.090673575129536, learning_rate = 0.00049999997, loss = 2.8315255e-05, step = 16633 (5.236 sec) INFO:tensorflow:global_step/sec: 4.96789 2021-10-19 07:49:12,254 [INFO] tensorflow: global_step/sec: 4.96789 2021-10-19 07:49:13,224 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.235 INFO:tensorflow:epoch = 43.160621761658035, learning_rate = 0.00049999997, loss = 4.0105217e-05, step = 16660 (5.319 sec) 2021-10-19 07:49:15,390 [INFO] tensorflow: epoch = 43.160621761658035, learning_rate = 0.00049999997, loss = 4.0105217e-05, step = 16660 (5.319 sec) 2021-10-19 07:49:18,229 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.981 INFO:tensorflow:global_step/sec: 5.00313 2021-10-19 07:49:19,849 [INFO] tensorflow: global_step/sec: 5.00313 INFO:tensorflow:epoch = 43.2279792746114, learning_rate = 0.00049999997, loss = 1.7668503e-05, step = 16686 (5.257 sec) 2021-10-19 07:49:20,646 [INFO] tensorflow: epoch = 43.2279792746114, learning_rate = 0.00049999997, loss = 1.7668503e-05, step = 16686 (5.257 sec) 2021-10-19 07:49:23,238 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.967 INFO:tensorflow:epoch = 43.2979274611399, learning_rate = 0.00049999997, loss = 3.829419e-05, step = 16713 (5.357 sec) 2021-10-19 07:49:26,004 [INFO] tensorflow: epoch = 43.2979274611399, learning_rate = 0.00049999997, loss = 3.829419e-05, step = 16713 (5.357 sec) INFO:tensorflow:global_step/sec: 5.02443 2021-10-19 07:49:27,412 [INFO] tensorflow: global_step/sec: 5.02443 2021-10-19 07:49:28,220 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.075 INFO:tensorflow:epoch = 43.365284974093264, learning_rate = 0.00049999997, loss = 3.0876217e-05, step = 16739 (5.287 sec) 2021-10-19 07:49:31,291 [INFO] tensorflow: epoch = 43.365284974093264, learning_rate = 0.00049999997, loss = 3.0876217e-05, step = 16739 (5.287 sec) 2021-10-19 07:49:33,307 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.659 INFO:tensorflow:global_step/sec: 4.87003 2021-10-19 07:49:35,215 [INFO] tensorflow: global_step/sec: 4.87003 INFO:tensorflow:epoch = 43.432642487046635, learning_rate = 0.00049999997, loss = 5.3125954e-05, step = 16765 (5.326 sec) 2021-10-19 07:49:36,617 [INFO] tensorflow: epoch = 43.432642487046635, learning_rate = 0.00049999997, loss = 5.3125954e-05, step = 16765 (5.326 sec) 2021-10-19 07:49:38,408 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.606 INFO:tensorflow:epoch = 43.5, learning_rate = 0.00049999997, loss = 3.3437555e-05, step = 16791 (5.267 sec) 2021-10-19 07:49:41,884 [INFO] tensorflow: epoch = 43.5, learning_rate = 0.00049999997, loss = 3.3437555e-05, step = 16791 (5.267 sec) INFO:tensorflow:global_step/sec: 4.97037 2021-10-19 07:49:42,860 [INFO] tensorflow: global_step/sec: 4.97037 2021-10-19 07:49:43,486 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.693 INFO:tensorflow:epoch = 43.56735751295337, learning_rate = 0.00049999997, loss = 3.7398502e-05, step = 16817 (5.280 sec) 2021-10-19 07:49:47,164 [INFO] tensorflow: epoch = 43.56735751295337, learning_rate = 0.00049999997, loss = 3.7398502e-05, step = 16817 (5.280 sec) 2021-10-19 07:49:48,591 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.589 INFO:tensorflow:global_step/sec: 4.89195 2021-10-19 07:49:50,628 [INFO] tensorflow: global_step/sec: 4.89195 INFO:tensorflow:epoch = 43.634715025906736, learning_rate = 0.00049999997, loss = 2.0256413e-05, step = 16843 (5.243 sec) 2021-10-19 07:49:52,406 [INFO] tensorflow: epoch = 43.634715025906736, learning_rate = 0.00049999997, loss = 2.0256413e-05, step = 16843 (5.243 sec) 2021-10-19 07:49:53,601 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.965 INFO:tensorflow:epoch = 43.704663212435236, learning_rate = 0.00049999997, loss = 1.6164944e-05, step = 16870 (5.342 sec) 2021-10-19 07:49:57,749 [INFO] tensorflow: epoch = 43.704663212435236, learning_rate = 0.00049999997, loss = 1.6164944e-05, step = 16870 (5.342 sec) INFO:tensorflow:global_step/sec: 5.05438 2021-10-19 07:49:58,147 [INFO] tensorflow: global_step/sec: 5.05438 2021-10-19 07:49:58,574 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.110 INFO:tensorflow:epoch = 43.774611398963735, learning_rate = 0.00049999997, loss = 2.4074974e-05, step = 16897 (5.311 sec) 2021-10-19 07:50:03,060 [INFO] tensorflow: epoch = 43.774611398963735, learning_rate = 0.00049999997, loss = 2.4074974e-05, step = 16897 (5.311 sec) 2021-10-19 07:50:03,497 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.314 INFO:tensorflow:global_step/sec: 5.03307 2021-10-19 07:50:05,697 [INFO] tensorflow: global_step/sec: 5.03307 INFO:tensorflow:epoch = 43.8419689119171, learning_rate = 0.00049999997, loss = 2.1271751e-05, step = 16923 (5.191 sec) 2021-10-19 07:50:08,251 [INFO] tensorflow: epoch = 43.8419689119171, learning_rate = 0.00049999997, loss = 2.1271751e-05, step = 16923 (5.191 sec) 2021-10-19 07:50:08,457 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.164 INFO:tensorflow:global_step/sec: 5.02353 2021-10-19 07:50:13,261 [INFO] tensorflow: global_step/sec: 5.02353 INFO:tensorflow:epoch = 43.90932642487047, learning_rate = 0.00049999997, loss = 3.2494725e-05, step = 16949 (5.218 sec) 2021-10-19 07:50:13,470 [INFO] tensorflow: epoch = 43.90932642487047, learning_rate = 0.00049999997, loss = 3.2494725e-05, step = 16949 (5.218 sec) 2021-10-19 07:50:13,470 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.948 2021-10-19 07:50:18,436 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.140 INFO:tensorflow:epoch = 43.979274611398964, learning_rate = 0.00049999997, loss = 2.8720231e-05, step = 16976 (5.393 sec) 2021-10-19 07:50:18,862 [INFO] tensorflow: epoch = 43.979274611398964, learning_rate = 0.00049999997, loss = 2.8720231e-05, step = 16976 (5.393 sec) 2021-10-19 07:50:20,448 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 44/120: loss: 0.00004 learning rate: 0.00050 Time taken: 0:01:17.447784 ETA: 1:38:06.031580 INFO:tensorflow:global_step/sec: 4.98809 2021-10-19 07:50:20,879 [INFO] tensorflow: global_step/sec: 4.98809 2021-10-19 07:50:23,425 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.044 INFO:tensorflow:epoch = 44.04922279792746, learning_rate = 0.00049999997, loss = 2.240044e-05, step = 17003 (5.345 sec) 2021-10-19 07:50:24,207 [INFO] tensorflow: epoch = 44.04922279792746, learning_rate = 0.00049999997, loss = 2.240044e-05, step = 17003 (5.345 sec) INFO:tensorflow:global_step/sec: 5.03884 2021-10-19 07:50:28,421 [INFO] tensorflow: global_step/sec: 5.03884 2021-10-19 07:50:28,422 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.015 INFO:tensorflow:epoch = 44.11658031088083, learning_rate = 0.00049999997, loss = 3.3140735e-05, step = 17029 (5.211 sec) 2021-10-19 07:50:29,418 [INFO] tensorflow: epoch = 44.11658031088083, learning_rate = 0.00049999997, loss = 3.3140735e-05, step = 17029 (5.211 sec) 2021-10-19 07:50:33,360 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.251 INFO:tensorflow:epoch = 44.18652849740933, learning_rate = 0.00049999997, loss = 3.3227454e-05, step = 17056 (5.345 sec) 2021-10-19 07:50:34,764 [INFO] tensorflow: epoch = 44.18652849740933, learning_rate = 0.00049999997, loss = 3.3227454e-05, step = 17056 (5.345 sec) INFO:tensorflow:global_step/sec: 5.02911 2021-10-19 07:50:35,977 [INFO] tensorflow: global_step/sec: 5.02911 2021-10-19 07:50:38,382 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.915 INFO:tensorflow:epoch = 44.25647668393783, learning_rate = 0.00049999997, loss = 2.0456644e-05, step = 17083 (5.371 sec) 2021-10-19 07:50:40,134 [INFO] tensorflow: epoch = 44.25647668393783, learning_rate = 0.00049999997, loss = 2.0456644e-05, step = 17083 (5.371 sec) 2021-10-19 07:50:43,303 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.324 INFO:tensorflow:global_step/sec: 5.04879 2021-10-19 07:50:43,503 [INFO] tensorflow: global_step/sec: 5.04879 INFO:tensorflow:epoch = 44.326424870466326, learning_rate = 0.00049999997, loss = 1.472246e-05, step = 17110 (5.348 sec) 2021-10-19 07:50:45,482 [INFO] tensorflow: epoch = 44.326424870466326, learning_rate = 0.00049999997, loss = 1.472246e-05, step = 17110 (5.348 sec) 2021-10-19 07:50:48,296 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.029 INFO:tensorflow:epoch = 44.39637305699482, learning_rate = 0.00049999997, loss = 2.7558977e-05, step = 17137 (5.376 sec) 2021-10-19 07:50:50,858 [INFO] tensorflow: epoch = 44.39637305699482, learning_rate = 0.00049999997, loss = 2.7558977e-05, step = 17137 (5.376 sec) INFO:tensorflow:global_step/sec: 5.02711 2021-10-19 07:50:51,062 [INFO] tensorflow: global_step/sec: 5.02711 2021-10-19 07:50:53,277 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.080 INFO:tensorflow:epoch = 44.46632124352332, learning_rate = 0.00049999997, loss = 2.5462608e-05, step = 17164 (5.394 sec) 2021-10-19 07:50:56,253 [INFO] tensorflow: epoch = 44.46632124352332, learning_rate = 0.00049999997, loss = 2.5462608e-05, step = 17164 (5.394 sec) 2021-10-19 07:50:58,259 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.071 INFO:tensorflow:global_step/sec: 5.0002 2021-10-19 07:50:58,662 [INFO] tensorflow: global_step/sec: 5.0002 INFO:tensorflow:epoch = 44.53367875647668, learning_rate = 0.00049999997, loss = 2.1994025e-05, step = 17190 (5.243 sec) 2021-10-19 07:51:01,495 [INFO] tensorflow: epoch = 44.53367875647668, learning_rate = 0.00049999997, loss = 2.1994025e-05, step = 17190 (5.243 sec) 2021-10-19 07:51:03,304 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.826 INFO:tensorflow:global_step/sec: 5.00622 2021-10-19 07:51:06,252 [INFO] tensorflow: global_step/sec: 5.00622 INFO:tensorflow:epoch = 44.60362694300518, learning_rate = 0.00049999997, loss = 3.078831e-05, step = 17217 (5.345 sec) 2021-10-19 07:51:06,840 [INFO] tensorflow: epoch = 44.60362694300518, learning_rate = 0.00049999997, loss = 3.078831e-05, step = 17217 (5.345 sec) 2021-10-19 07:51:08,205 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.403 INFO:tensorflow:epoch = 44.67357512953368, learning_rate = 0.00049999997, loss = 2.7743441e-05, step = 17244 (5.379 sec) 2021-10-19 07:51:12,220 [INFO] tensorflow: epoch = 44.67357512953368, learning_rate = 0.00049999997, loss = 2.7743441e-05, step = 17244 (5.379 sec) 2021-10-19 07:51:13,200 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.024 INFO:tensorflow:global_step/sec: 5.02982 2021-10-19 07:51:13,807 [INFO] tensorflow: global_step/sec: 5.02982 INFO:tensorflow:epoch = 44.740932642487046, learning_rate = 0.00049999997, loss = 1.9526318e-05, step = 17270 (5.226 sec) 2021-10-19 07:51:17,446 [INFO] tensorflow: epoch = 44.740932642487046, learning_rate = 0.00049999997, loss = 1.9526318e-05, step = 17270 (5.226 sec) 2021-10-19 07:51:18,246 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.823 INFO:tensorflow:global_step/sec: 5.00849 2021-10-19 07:51:21,395 [INFO] tensorflow: global_step/sec: 5.00849 INFO:tensorflow:epoch = 44.810880829015545, learning_rate = 0.00049999997, loss = 2.9034847e-05, step = 17297 (5.377 sec) 2021-10-19 07:51:22,823 [INFO] tensorflow: epoch = 44.810880829015545, learning_rate = 0.00049999997, loss = 2.9034847e-05, step = 17297 (5.377 sec) 2021-10-19 07:51:23,210 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.147 INFO:tensorflow:epoch = 44.880829015544045, learning_rate = 0.00049999997, loss = 2.8157549e-05, step = 17324 (5.378 sec) 2021-10-19 07:51:28,201 [INFO] tensorflow: epoch = 44.880829015544045, learning_rate = 0.00049999997, loss = 2.8157549e-05, step = 17324 (5.378 sec) 2021-10-19 07:51:28,202 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.034 INFO:tensorflow:global_step/sec: 5.01365 2021-10-19 07:51:28,974 [INFO] tensorflow: global_step/sec: 5.01365 2021-10-19 07:51:33,183 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.078 INFO:tensorflow:epoch = 44.950777202072544, learning_rate = 0.00049999997, loss = 4.7787722e-05, step = 17351 (5.382 sec) 2021-10-19 07:51:33,583 [INFO] tensorflow: epoch = 44.950777202072544, learning_rate = 0.00049999997, loss = 4.7787722e-05, step = 17351 (5.382 sec) INFO:tensorflow:global_step/sec: 5.01632 2021-10-19 07:51:36,549 [INFO] tensorflow: global_step/sec: 5.01632 2021-10-19 07:51:37,378 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 45/120: loss: 0.00003 learning rate: 0.00050 Time taken: 0:01:16.911332 ETA: 1:36:08.349910 2021-10-19 07:51:38,211 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.887 INFO:tensorflow:epoch = 45.01813471502591, learning_rate = 0.00049999997, loss = 2.5668345e-05, step = 17377 (5.240 sec) 2021-10-19 07:51:38,823 [INFO] tensorflow: epoch = 45.01813471502591, learning_rate = 0.00049999997, loss = 2.5668345e-05, step = 17377 (5.240 sec) 2021-10-19 07:51:43,208 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.016 INFO:tensorflow:epoch = 45.08808290155441, learning_rate = 0.00049999997, loss = 4.7711423e-05, step = 17404 (5.366 sec) 2021-10-19 07:51:44,189 [INFO] tensorflow: epoch = 45.08808290155441, learning_rate = 0.00049999997, loss = 4.7711423e-05, step = 17404 (5.366 sec) INFO:tensorflow:global_step/sec: 4.97305 2021-10-19 07:51:44,190 [INFO] tensorflow: global_step/sec: 4.97305 2021-10-19 07:51:48,257 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.809 INFO:tensorflow:epoch = 45.15544041450777, learning_rate = 0.00049999997, loss = 3.471326e-05, step = 17430 (5.272 sec) 2021-10-19 07:51:49,461 [INFO] tensorflow: epoch = 45.15544041450777, learning_rate = 0.00049999997, loss = 3.471326e-05, step = 17430 (5.272 sec) INFO:tensorflow:global_step/sec: 4.97012 2021-10-19 07:51:51,836 [INFO] tensorflow: global_step/sec: 4.97012 2021-10-19 07:51:53,220 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.147 INFO:tensorflow:epoch = 45.22538860103627, learning_rate = 0.00049999997, loss = 4.082566e-05, step = 17457 (5.351 sec) 2021-10-19 07:51:54,812 [INFO] tensorflow: epoch = 45.22538860103627, learning_rate = 0.00049999997, loss = 4.082566e-05, step = 17457 (5.351 sec) 2021-10-19 07:51:58,171 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.202 INFO:tensorflow:global_step/sec: 5.05118 2021-10-19 07:51:59,359 [INFO] tensorflow: global_step/sec: 5.05118 INFO:tensorflow:epoch = 45.29533678756477, learning_rate = 0.00049999997, loss = 2.0029718e-05, step = 17484 (5.354 sec) 2021-10-19 07:52:00,166 [INFO] tensorflow: epoch = 45.29533678756477, learning_rate = 0.00049999997, loss = 2.0029718e-05, step = 17484 (5.354 sec) 2021-10-19 07:52:03,107 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.259 INFO:tensorflow:epoch = 45.365284974093264, learning_rate = 0.00049999997, loss = 1.935436e-05, step = 17511 (5.323 sec) 2021-10-19 07:52:05,490 [INFO] tensorflow: epoch = 45.365284974093264, learning_rate = 0.00049999997, loss = 1.935436e-05, step = 17511 (5.323 sec) INFO:tensorflow:global_step/sec: 5.05117 2021-10-19 07:52:06,882 [INFO] tensorflow: global_step/sec: 5.05117 2021-10-19 07:52:08,087 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.085 INFO:tensorflow:epoch = 45.432642487046635, learning_rate = 0.00049999997, loss = 3.249535e-05, step = 17537 (5.216 sec) 2021-10-19 07:52:10,706 [INFO] tensorflow: epoch = 45.432642487046635, learning_rate = 0.00049999997, loss = 3.249535e-05, step = 17537 (5.216 sec) 2021-10-19 07:52:13,080 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.028 INFO:tensorflow:global_step/sec: 4.98967 2021-10-19 07:52:14,498 [INFO] tensorflow: global_step/sec: 4.98967 INFO:tensorflow:epoch = 45.5, learning_rate = 0.00049999997, loss = 4.816662e-05, step = 17563 (5.195 sec) 2021-10-19 07:52:15,901 [INFO] tensorflow: epoch = 45.5, learning_rate = 0.00049999997, loss = 4.816662e-05, step = 17563 (5.195 sec) 2021-10-19 07:52:18,101 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.918 INFO:tensorflow:epoch = 45.5699481865285, learning_rate = 0.00049999997, loss = 2.2370647e-05, step = 17590 (5.344 sec) 2021-10-19 07:52:21,245 [INFO] tensorflow: epoch = 45.5699481865285, learning_rate = 0.00049999997, loss = 2.2370647e-05, step = 17590 (5.344 sec) INFO:tensorflow:global_step/sec: 5.03021 2021-10-19 07:52:22,052 [INFO] tensorflow: global_step/sec: 5.03021 2021-10-19 07:52:23,043 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.238 INFO:tensorflow:epoch = 45.639896373057, learning_rate = 0.00049999997, loss = 5.6676607e-05, step = 17617 (5.379 sec) 2021-10-19 07:52:26,624 [INFO] tensorflow: epoch = 45.639896373057, learning_rate = 0.00049999997, loss = 5.6676607e-05, step = 17617 (5.379 sec) 2021-10-19 07:52:28,042 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.005 INFO:tensorflow:global_step/sec: 5.01478 2021-10-19 07:52:29,630 [INFO] tensorflow: global_step/sec: 5.01478 INFO:tensorflow:epoch = 45.70725388601036, learning_rate = 0.00049999997, loss = 7.952756e-05, step = 17643 (5.260 sec) 2021-10-19 07:52:31,884 [INFO] tensorflow: epoch = 45.70725388601036, learning_rate = 0.00049999997, loss = 7.952756e-05, step = 17643 (5.260 sec) 2021-10-19 07:52:33,082 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.844 INFO:tensorflow:epoch = 45.774611398963735, learning_rate = 0.00049999997, loss = 3.008903e-05, step = 17669 (5.248 sec) 2021-10-19 07:52:37,131 [INFO] tensorflow: epoch = 45.774611398963735, learning_rate = 0.00049999997, loss = 3.008903e-05, step = 17669 (5.248 sec) INFO:tensorflow:global_step/sec: 4.93289 2021-10-19 07:52:37,333 [INFO] tensorflow: global_step/sec: 4.93289 2021-10-19 07:52:38,138 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.780 INFO:tensorflow:epoch = 45.8419689119171, learning_rate = 0.00049999997, loss = 4.4647233e-05, step = 17695 (5.294 sec) 2021-10-19 07:52:42,426 [INFO] tensorflow: epoch = 45.8419689119171, learning_rate = 0.00049999997, loss = 4.4647233e-05, step = 17695 (5.294 sec) 2021-10-19 07:52:43,247 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.572 INFO:tensorflow:global_step/sec: 4.93849 2021-10-19 07:52:45,028 [INFO] tensorflow: global_step/sec: 4.93849 INFO:tensorflow:epoch = 45.9119170984456, learning_rate = 0.00049999997, loss = 3.1304095e-05, step = 17722 (5.331 sec) 2021-10-19 07:52:47,757 [INFO] tensorflow: epoch = 45.9119170984456, learning_rate = 0.00049999997, loss = 3.1304095e-05, step = 17722 (5.331 sec) 2021-10-19 07:52:48,147 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.413 INFO:tensorflow:global_step/sec: 5.00219 2021-10-19 07:52:52,624 [INFO] tensorflow: global_step/sec: 5.00219 INFO:tensorflow:epoch = 45.979274611398964, learning_rate = 0.00049999997, loss = 3.668692e-05, step = 17748 (5.257 sec) 2021-10-19 07:52:53,014 [INFO] tensorflow: epoch = 45.979274611398964, learning_rate = 0.00049999997, loss = 3.668692e-05, step = 17748 (5.257 sec) 2021-10-19 07:52:53,212 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.742 2021-10-19 07:52:54,674 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 46/120: loss: 0.00001 learning rate: 0.00050 Time taken: 0:01:17.304801 ETA: 1:35:20.555238 INFO:tensorflow:epoch = 46.046632124352335, learning_rate = 0.00049999997, loss = 4.498659e-05, step = 17774 (5.257 sec) 2021-10-19 07:52:58,271 [INFO] tensorflow: epoch = 46.046632124352335, learning_rate = 0.00049999997, loss = 4.498659e-05, step = 17774 (5.257 sec) 2021-10-19 07:52:58,271 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.770 INFO:tensorflow:global_step/sec: 5.00529 2021-10-19 07:53:00,216 [INFO] tensorflow: global_step/sec: 5.00529 2021-10-19 07:53:03,211 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.245 INFO:tensorflow:epoch = 46.11658031088083, learning_rate = 0.00049999997, loss = 1.5781132e-05, step = 17801 (5.338 sec) 2021-10-19 07:53:03,609 [INFO] tensorflow: epoch = 46.11658031088083, learning_rate = 0.00049999997, loss = 1.5781132e-05, step = 17801 (5.338 sec) INFO:tensorflow:global_step/sec: 5.01419 2021-10-19 07:53:07,795 [INFO] tensorflow: global_step/sec: 5.01419 2021-10-19 07:53:08,202 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.035 INFO:tensorflow:epoch = 46.1839378238342, learning_rate = 0.00049999997, loss = 2.7276716e-05, step = 17827 (5.217 sec) 2021-10-19 07:53:08,826 [INFO] tensorflow: epoch = 46.1839378238342, learning_rate = 0.00049999997, loss = 2.7276716e-05, step = 17827 (5.217 sec) 2021-10-19 07:53:13,230 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.890 INFO:tensorflow:epoch = 46.2538860103627, learning_rate = 0.00049999997, loss = 2.8230701e-05, step = 17854 (5.388 sec) 2021-10-19 07:53:14,214 [INFO] tensorflow: epoch = 46.2538860103627, learning_rate = 0.00049999997, loss = 2.8230701e-05, step = 17854 (5.388 sec) INFO:tensorflow:global_step/sec: 4.99741 2021-10-19 07:53:15,399 [INFO] tensorflow: global_step/sec: 4.99741 2021-10-19 07:53:18,178 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.214 INFO:tensorflow:epoch = 46.32383419689119, learning_rate = 0.00049999997, loss = 1.5171311e-05, step = 17881 (5.330 sec) 2021-10-19 07:53:19,543 [INFO] tensorflow: epoch = 46.32383419689119, learning_rate = 0.00049999997, loss = 1.5171311e-05, step = 17881 (5.330 sec) INFO:tensorflow:global_step/sec: 5.06417 2021-10-19 07:53:22,903 [INFO] tensorflow: global_step/sec: 5.06417 2021-10-19 07:53:23,102 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.311 INFO:tensorflow:epoch = 46.39378238341969, learning_rate = 0.00049999997, loss = 3.0590098e-05, step = 17908 (5.417 sec) 2021-10-19 07:53:24,960 [INFO] tensorflow: epoch = 46.39378238341969, learning_rate = 0.00049999997, loss = 3.0590098e-05, step = 17908 (5.417 sec) 2021-10-19 07:53:28,229 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.505 INFO:tensorflow:epoch = 46.46113989637306, learning_rate = 0.00049999997, loss = 4.6217676e-05, step = 17934 (5.243 sec) 2021-10-19 07:53:30,203 [INFO] tensorflow: epoch = 46.46113989637306, learning_rate = 0.00049999997, loss = 4.6217676e-05, step = 17934 (5.243 sec) INFO:tensorflow:global_step/sec: 4.93731 2021-10-19 07:53:30,599 [INFO] tensorflow: global_step/sec: 4.93731 2021-10-19 07:53:33,235 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.980 INFO:tensorflow:epoch = 46.52849740932643, learning_rate = 0.00049999997, loss = 1.9251112e-05, step = 17960 (5.300 sec) 2021-10-19 07:53:35,503 [INFO] tensorflow: epoch = 46.52849740932643, learning_rate = 0.00049999997, loss = 1.9251112e-05, step = 17960 (5.300 sec) INFO:tensorflow:global_step/sec: 4.90502 2021-10-19 07:53:38,346 [INFO] tensorflow: global_step/sec: 4.90502 2021-10-19 07:53:38,348 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.559 INFO:tensorflow:epoch = 46.59585492227979, learning_rate = 0.00049999997, loss = 2.3926732e-05, step = 17986 (5.252 sec) 2021-10-19 07:53:40,755 [INFO] tensorflow: epoch = 46.59585492227979, learning_rate = 0.00049999997, loss = 2.3926732e-05, step = 17986 (5.252 sec) 2021-10-19 07:53:43,368 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.920 INFO:tensorflow:epoch = 46.66321243523316, learning_rate = 0.00049999997, loss = 2.926967e-05, step = 18012 (5.210 sec) 2021-10-19 07:53:45,965 [INFO] tensorflow: epoch = 46.66321243523316, learning_rate = 0.00049999997, loss = 2.926967e-05, step = 18012 (5.210 sec) INFO:tensorflow:global_step/sec: 4.9871 2021-10-19 07:53:45,966 [INFO] tensorflow: global_step/sec: 4.9871 2021-10-19 07:53:48,349 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.079 INFO:tensorflow:epoch = 46.73316062176166, learning_rate = 0.00049999997, loss = 4.4954355e-05, step = 18039 (5.359 sec) 2021-10-19 07:53:51,323 [INFO] tensorflow: epoch = 46.73316062176166, learning_rate = 0.00049999997, loss = 4.4954355e-05, step = 18039 (5.359 sec) 2021-10-19 07:53:53,314 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.142 INFO:tensorflow:global_step/sec: 5.03629 2021-10-19 07:53:53,511 [INFO] tensorflow: global_step/sec: 5.03629 INFO:tensorflow:epoch = 46.803108808290155, learning_rate = 0.00049999997, loss = 3.064258e-05, step = 18066 (5.385 sec) 2021-10-19 07:53:56,708 [INFO] tensorflow: epoch = 46.803108808290155, learning_rate = 0.00049999997, loss = 3.064258e-05, step = 18066 (5.385 sec) 2021-10-19 07:53:58,279 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.143 INFO:tensorflow:global_step/sec: 5.03092 2021-10-19 07:54:01,064 [INFO] tensorflow: global_step/sec: 5.03092 INFO:tensorflow:epoch = 46.873056994818654, learning_rate = 0.00049999997, loss = 3.123283e-05, step = 18093 (5.335 sec) 2021-10-19 07:54:02,043 [INFO] tensorflow: epoch = 46.873056994818654, learning_rate = 0.00049999997, loss = 3.123283e-05, step = 18093 (5.335 sec) 2021-10-19 07:54:03,256 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.094 INFO:tensorflow:epoch = 46.943005181347154, learning_rate = 0.00049999997, loss = 3.5259807e-05, step = 18120 (5.386 sec) 2021-10-19 07:54:07,429 [INFO] tensorflow: epoch = 46.943005181347154, learning_rate = 0.00049999997, loss = 3.5259807e-05, step = 18120 (5.386 sec) 2021-10-19 07:54:08,236 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.083 INFO:tensorflow:global_step/sec: 5.02474 2021-10-19 07:54:08,627 [INFO] tensorflow: global_step/sec: 5.02474 2021-10-19 07:54:11,917 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 47/120: loss: 0.00002 learning rate: 0.00050 Time taken: 0:01:17.236030 ETA: 1:33:58.230163 INFO:tensorflow:epoch = 47.01036269430052, learning_rate = 0.00049999997, loss = 2.6799364e-05, step = 18146 (5.305 sec) 2021-10-19 07:54:12,733 [INFO] tensorflow: epoch = 47.01036269430052, learning_rate = 0.00049999997, loss = 2.6799364e-05, step = 18146 (5.305 sec) 2021-10-19 07:54:13,361 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.512 INFO:tensorflow:global_step/sec: 4.93133 2021-10-19 07:54:16,333 [INFO] tensorflow: global_step/sec: 4.93133 INFO:tensorflow:epoch = 47.08031088082902, learning_rate = 0.00049999997, loss = 1.870816e-05, step = 18173 (5.403 sec) 2021-10-19 07:54:18,136 [INFO] tensorflow: epoch = 47.08031088082902, learning_rate = 0.00049999997, loss = 1.870816e-05, step = 18173 (5.403 sec) 2021-10-19 07:54:18,335 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.106 INFO:tensorflow:epoch = 47.14766839378238, learning_rate = 0.00049999997, loss = 2.9434426e-05, step = 18199 (5.236 sec) 2021-10-19 07:54:23,372 [INFO] tensorflow: epoch = 47.14766839378238, learning_rate = 0.00049999997, loss = 2.9434426e-05, step = 18199 (5.236 sec) 2021-10-19 07:54:23,372 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.855 INFO:tensorflow:global_step/sec: 4.97283 2021-10-19 07:54:23,974 [INFO] tensorflow: global_step/sec: 4.97283 2021-10-19 07:54:28,374 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.995 INFO:tensorflow:epoch = 47.215025906735754, learning_rate = 0.00049999997, loss = 4.2633787e-05, step = 18225 (5.217 sec) 2021-10-19 07:54:28,589 [INFO] tensorflow: epoch = 47.215025906735754, learning_rate = 0.00049999997, loss = 4.2633787e-05, step = 18225 (5.217 sec) INFO:tensorflow:global_step/sec: 5.02849 2021-10-19 07:54:31,531 [INFO] tensorflow: global_step/sec: 5.02849 2021-10-19 07:54:33,328 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.188 INFO:tensorflow:epoch = 47.28497409326425, learning_rate = 0.00049999997, loss = 4.2627504e-05, step = 18252 (5.326 sec) 2021-10-19 07:54:33,916 [INFO] tensorflow: epoch = 47.28497409326425, learning_rate = 0.00049999997, loss = 4.2627504e-05, step = 18252 (5.326 sec) 2021-10-19 07:54:38,316 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.048 INFO:tensorflow:global_step/sec: 5.0136 2021-10-19 07:54:39,111 [INFO] tensorflow: global_step/sec: 5.0136 INFO:tensorflow:epoch = 47.354922279792746, learning_rate = 0.00049999997, loss = 1.6320493e-05, step = 18279 (5.396 sec) 2021-10-19 07:54:39,311 [INFO] tensorflow: epoch = 47.354922279792746, learning_rate = 0.00049999997, loss = 1.6320493e-05, step = 18279 (5.396 sec) 2021-10-19 07:54:43,250 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.270 INFO:tensorflow:epoch = 47.424870466321245, learning_rate = 0.00049999997, loss = 2.0422038e-05, step = 18306 (5.337 sec) 2021-10-19 07:54:44,649 [INFO] tensorflow: epoch = 47.424870466321245, learning_rate = 0.00049999997, loss = 2.0422038e-05, step = 18306 (5.337 sec) INFO:tensorflow:global_step/sec: 5.05314 2021-10-19 07:54:46,631 [INFO] tensorflow: global_step/sec: 5.05314 2021-10-19 07:54:48,197 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.216 INFO:tensorflow:epoch = 47.494818652849744, learning_rate = 0.00049999997, loss = 3.472303e-05, step = 18333 (5.336 sec) 2021-10-19 07:54:49,984 [INFO] tensorflow: epoch = 47.494818652849744, learning_rate = 0.00049999997, loss = 3.472303e-05, step = 18333 (5.336 sec) 2021-10-19 07:54:53,195 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.010 INFO:tensorflow:global_step/sec: 4.98977 2021-10-19 07:54:54,246 [INFO] tensorflow: global_step/sec: 4.98977 INFO:tensorflow:epoch = 47.56217616580311, learning_rate = 0.00049999997, loss = 3.2169766e-05, step = 18359 (5.304 sec) 2021-10-19 07:54:55,288 [INFO] tensorflow: epoch = 47.56217616580311, learning_rate = 0.00049999997, loss = 3.2169766e-05, step = 18359 (5.304 sec) 2021-10-19 07:54:58,270 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.706 INFO:tensorflow:epoch = 47.63212435233161, learning_rate = 0.00049999997, loss = 4.3955402e-05, step = 18386 (5.354 sec) 2021-10-19 07:55:00,642 [INFO] tensorflow: epoch = 47.63212435233161, learning_rate = 0.00049999997, loss = 4.3955402e-05, step = 18386 (5.354 sec) INFO:tensorflow:global_step/sec: 5.01143 2021-10-19 07:55:01,829 [INFO] tensorflow: global_step/sec: 5.01143 2021-10-19 07:55:03,212 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.234 INFO:tensorflow:epoch = 47.70207253886011, learning_rate = 0.00049999997, loss = 2.351345e-05, step = 18413 (5.337 sec) 2021-10-19 07:55:05,979 [INFO] tensorflow: epoch = 47.70207253886011, learning_rate = 0.00049999997, loss = 2.351345e-05, step = 18413 (5.337 sec) 2021-10-19 07:55:08,172 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.164 INFO:tensorflow:global_step/sec: 5.03441 2021-10-19 07:55:09,377 [INFO] tensorflow: global_step/sec: 5.03441 INFO:tensorflow:epoch = 47.7720207253886, learning_rate = 0.00049999997, loss = 1.6437241e-05, step = 18440 (5.377 sec) 2021-10-19 07:55:11,356 [INFO] tensorflow: epoch = 47.7720207253886, learning_rate = 0.00049999997, loss = 1.6437241e-05, step = 18440 (5.377 sec) 2021-10-19 07:55:13,130 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.170 INFO:tensorflow:epoch = 47.8419689119171, learning_rate = 0.00049999997, loss = 3.2349162e-05, step = 18467 (5.336 sec) 2021-10-19 07:55:16,692 [INFO] tensorflow: epoch = 47.8419689119171, learning_rate = 0.00049999997, loss = 3.2349162e-05, step = 18467 (5.336 sec) INFO:tensorflow:global_step/sec: 5.05737 2021-10-19 07:55:16,891 [INFO] tensorflow: global_step/sec: 5.05737 2021-10-19 07:55:18,084 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.188 INFO:tensorflow:epoch = 47.9119170984456, learning_rate = 0.00049999997, loss = 2.1802087e-05, step = 18494 (5.353 sec) 2021-10-19 07:55:22,044 [INFO] tensorflow: epoch = 47.9119170984456, learning_rate = 0.00049999997, loss = 2.1802087e-05, step = 18494 (5.353 sec) 2021-10-19 07:55:23,026 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.236 INFO:tensorflow:global_step/sec: 5.01563 2021-10-19 07:55:24,467 [INFO] tensorflow: global_step/sec: 5.01563 INFO:tensorflow:epoch = 47.979274611398964, learning_rate = 0.00049999997, loss = 2.1367243e-05, step = 18520 (5.206 sec) 2021-10-19 07:55:27,251 [INFO] tensorflow: epoch = 47.979274611398964, learning_rate = 0.00049999997, loss = 2.1367243e-05, step = 18520 (5.206 sec) 2021-10-19 07:55:28,026 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.004 2021-10-19 07:55:28,840 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 48/120: loss: 0.00002 learning rate: 0.00050 Time taken: 0:01:16.931491 ETA: 1:32:19.067379 INFO:tensorflow:global_step/sec: 5 2021-10-19 07:55:32,067 [INFO] tensorflow: global_step/sec: 5 INFO:tensorflow:epoch = 48.046632124352335, learning_rate = 0.00049999997, loss = 2.6786493e-05, step = 18546 (5.215 sec) 2021-10-19 07:55:32,465 [INFO] tensorflow: epoch = 48.046632124352335, learning_rate = 0.00049999997, loss = 2.6786493e-05, step = 18546 (5.215 sec) 2021-10-19 07:55:33,068 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.833 INFO:tensorflow:epoch = 48.1139896373057, learning_rate = 0.00049999997, loss = 1.7608942e-05, step = 18572 (5.195 sec) 2021-10-19 07:55:37,661 [INFO] tensorflow: epoch = 48.1139896373057, learning_rate = 0.00049999997, loss = 1.7608942e-05, step = 18572 (5.195 sec) 2021-10-19 07:55:38,067 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.007 INFO:tensorflow:global_step/sec: 4.99893 2021-10-19 07:55:39,669 [INFO] tensorflow: global_step/sec: 4.99893 INFO:tensorflow:epoch = 48.1839378238342, learning_rate = 0.00049999997, loss = 2.036803e-05, step = 18599 (5.372 sec) 2021-10-19 07:55:43,032 [INFO] tensorflow: epoch = 48.1839378238342, learning_rate = 0.00049999997, loss = 2.036803e-05, step = 18599 (5.372 sec) 2021-10-19 07:55:43,033 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.139 INFO:tensorflow:global_step/sec: 5.05818 2021-10-19 07:55:47,181 [INFO] tensorflow: global_step/sec: 5.05818 2021-10-19 07:55:47,979 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.219 INFO:tensorflow:epoch = 48.2538860103627, learning_rate = 0.00049999997, loss = 3.2526357e-05, step = 18626 (5.359 sec) 2021-10-19 07:55:48,391 [INFO] tensorflow: epoch = 48.2538860103627, learning_rate = 0.00049999997, loss = 3.2526357e-05, step = 18626 (5.359 sec) 2021-10-19 07:55:52,965 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.058 INFO:tensorflow:epoch = 48.32383419689119, learning_rate = 0.00049999997, loss = 4.0477327e-05, step = 18653 (5.374 sec) 2021-10-19 07:55:53,765 [INFO] tensorflow: epoch = 48.32383419689119, learning_rate = 0.00049999997, loss = 4.0477327e-05, step = 18653 (5.374 sec) INFO:tensorflow:global_step/sec: 4.99156 2021-10-19 07:55:54,794 [INFO] tensorflow: global_step/sec: 4.99156 2021-10-19 07:55:58,005 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.840 INFO:tensorflow:epoch = 48.39119170984456, learning_rate = 0.00049999997, loss = 1.5755551e-05, step = 18679 (5.260 sec) 2021-10-19 07:55:59,025 [INFO] tensorflow: epoch = 48.39119170984456, learning_rate = 0.00049999997, loss = 1.5755551e-05, step = 18679 (5.260 sec) INFO:tensorflow:global_step/sec: 5.0357 2021-10-19 07:56:02,340 [INFO] tensorflow: global_step/sec: 5.0357 2021-10-19 07:56:02,938 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.274 INFO:tensorflow:epoch = 48.46113989637306, learning_rate = 0.00049999997, loss = 2.7239432e-05, step = 18706 (5.333 sec) 2021-10-19 07:56:04,358 [INFO] tensorflow: epoch = 48.46113989637306, learning_rate = 0.00049999997, loss = 2.7239432e-05, step = 18706 (5.333 sec) 2021-10-19 07:56:07,947 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.967 INFO:tensorflow:epoch = 48.531088082901555, learning_rate = 0.00049999997, loss = 2.6640473e-05, step = 18733 (5.385 sec) 2021-10-19 07:56:09,743 [INFO] tensorflow: epoch = 48.531088082901555, learning_rate = 0.00049999997, loss = 2.6640473e-05, step = 18733 (5.385 sec) INFO:tensorflow:global_step/sec: 4.97583 2021-10-19 07:56:09,977 [INFO] tensorflow: global_step/sec: 4.97583 2021-10-19 07:56:12,949 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.993 INFO:tensorflow:epoch = 48.598445595854926, learning_rate = 0.00049999997, loss = 2.0277028e-05, step = 18759 (5.248 sec) 2021-10-19 07:56:14,991 [INFO] tensorflow: epoch = 48.598445595854926, learning_rate = 0.00049999997, loss = 2.0277028e-05, step = 18759 (5.248 sec) INFO:tensorflow:global_step/sec: 5.00574 2021-10-19 07:56:17,569 [INFO] tensorflow: global_step/sec: 5.00574 2021-10-19 07:56:17,963 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.947 INFO:tensorflow:epoch = 48.66839378238342, learning_rate = 0.00049999997, loss = 5.9893377e-05, step = 18786 (5.321 sec) 2021-10-19 07:56:20,312 [INFO] tensorflow: epoch = 48.66839378238342, learning_rate = 0.00049999997, loss = 5.9893377e-05, step = 18786 (5.321 sec) 2021-10-19 07:56:22,904 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.239 INFO:tensorflow:global_step/sec: 5.05281 2021-10-19 07:56:25,089 [INFO] tensorflow: global_step/sec: 5.05281 INFO:tensorflow:epoch = 48.73834196891192, learning_rate = 0.00049999997, loss = 2.7508388e-05, step = 18813 (5.394 sec) 2021-10-19 07:56:25,706 [INFO] tensorflow: epoch = 48.73834196891192, learning_rate = 0.00049999997, loss = 2.7508388e-05, step = 18813 (5.394 sec) 2021-10-19 07:56:27,976 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.720 INFO:tensorflow:epoch = 48.80569948186529, learning_rate = 0.00049999997, loss = 1.9256127e-05, step = 18839 (5.279 sec) 2021-10-19 07:56:30,985 [INFO] tensorflow: epoch = 48.80569948186529, learning_rate = 0.00049999997, loss = 1.9256127e-05, step = 18839 (5.279 sec) INFO:tensorflow:global_step/sec: 4.90935 2021-10-19 07:56:32,829 [INFO] tensorflow: global_step/sec: 4.90935 2021-10-19 07:56:33,027 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.797 INFO:tensorflow:epoch = 48.873056994818654, learning_rate = 0.00049999997, loss = 2.3207574e-05, step = 18865 (5.317 sec) 2021-10-19 07:56:36,302 [INFO] tensorflow: epoch = 48.873056994818654, learning_rate = 0.00049999997, loss = 2.3207574e-05, step = 18865 (5.317 sec) 2021-10-19 07:56:38,105 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.694 INFO:tensorflow:global_step/sec: 4.9636 2021-10-19 07:56:40,485 [INFO] tensorflow: global_step/sec: 4.9636 INFO:tensorflow:epoch = 48.943005181347154, learning_rate = 0.00049999997, loss = 2.816076e-05, step = 18892 (5.371 sec) 2021-10-19 07:56:41,673 [INFO] tensorflow: epoch = 48.943005181347154, learning_rate = 0.00049999997, loss = 2.816076e-05, step = 18892 (5.371 sec) 2021-10-19 07:56:43,113 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.973 2021-10-19 07:56:46,125 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 49/120: loss: 0.00004 learning rate: 0.00050 Time taken: 0:01:17.283577 ETA: 1:31:27.133998 INFO:tensorflow:epoch = 49.01036269430052, learning_rate = 0.00049999997, loss = 1.3620685e-05, step = 18918 (5.250 sec) 2021-10-19 07:56:46,923 [INFO] tensorflow: epoch = 49.01036269430052, learning_rate = 0.00049999997, loss = 1.3620685e-05, step = 18918 (5.250 sec) INFO:tensorflow:global_step/sec: 4.95552 2021-10-19 07:56:48,153 [INFO] tensorflow: global_step/sec: 4.95552 2021-10-19 07:56:48,155 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.833 INFO:tensorflow:epoch = 49.07772020725389, learning_rate = 0.00049999997, loss = 2.9357969e-05, step = 18944 (5.228 sec) 2021-10-19 07:56:52,151 [INFO] tensorflow: epoch = 49.07772020725389, learning_rate = 0.00049999997, loss = 2.9357969e-05, step = 18944 (5.228 sec) 2021-10-19 07:56:53,144 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.047 INFO:tensorflow:global_step/sec: 4.99022 2021-10-19 07:56:55,768 [INFO] tensorflow: global_step/sec: 4.99022 INFO:tensorflow:epoch = 49.145077720207254, learning_rate = 0.00049999997, loss = 1.7759538e-05, step = 18970 (5.234 sec) 2021-10-19 07:56:57,385 [INFO] tensorflow: epoch = 49.145077720207254, learning_rate = 0.00049999997, loss = 1.7759538e-05, step = 18970 (5.234 sec) 2021-10-19 07:56:58,180 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.860 INFO:tensorflow:epoch = 49.215025906735754, learning_rate = 0.00049999997, loss = 2.0284773e-05, step = 18997 (5.359 sec) 2021-10-19 07:57:02,744 [INFO] tensorflow: epoch = 49.215025906735754, learning_rate = 0.00049999997, loss = 2.0284773e-05, step = 18997 (5.359 sec) 2021-10-19 07:57:03,156 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.095 INFO:tensorflow:global_step/sec: 4.99399 2021-10-19 07:57:03,377 [INFO] tensorflow: global_step/sec: 4.99399 INFO:tensorflow:epoch = 49.28238341968912, learning_rate = 0.00049999997, loss = 2.6183312e-05, step = 19023 (5.311 sec) 2021-10-19 07:57:08,055 [INFO] tensorflow: epoch = 49.28238341968912, learning_rate = 0.00049999997, loss = 2.6183312e-05, step = 19023 (5.311 sec) 2021-10-19 07:57:08,253 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.622 INFO:tensorflow:global_step/sec: 4.94874 2021-10-19 07:57:11,056 [INFO] tensorflow: global_step/sec: 4.94874 2021-10-19 07:57:13,252 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.005 INFO:tensorflow:epoch = 49.35233160621762, learning_rate = 0.00049999997, loss = 2.3317696e-05, step = 19050 (5.425 sec) 2021-10-19 07:57:13,480 [INFO] tensorflow: epoch = 49.35233160621762, learning_rate = 0.00049999997, loss = 2.3317696e-05, step = 19050 (5.425 sec) 2021-10-19 07:57:18,221 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.126 INFO:tensorflow:global_step/sec: 5.01488 2021-10-19 07:57:18,634 [INFO] tensorflow: global_step/sec: 5.01488 INFO:tensorflow:epoch = 49.42227979274612, learning_rate = 0.00049999997, loss = 1.7440314e-05, step = 19077 (5.350 sec) 2021-10-19 07:57:18,829 [INFO] tensorflow: epoch = 49.42227979274612, learning_rate = 0.00049999997, loss = 1.7440314e-05, step = 19077 (5.350 sec) 2021-10-19 07:57:23,243 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.916 INFO:tensorflow:epoch = 49.49222797927462, learning_rate = 0.00049999997, loss = 2.0236008e-05, step = 19104 (5.400 sec) 2021-10-19 07:57:24,229 [INFO] tensorflow: epoch = 49.49222797927462, learning_rate = 0.00049999997, loss = 2.0236008e-05, step = 19104 (5.400 sec) INFO:tensorflow:global_step/sec: 5.03836 2021-10-19 07:57:26,176 [INFO] tensorflow: global_step/sec: 5.03836 2021-10-19 07:57:28,138 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.430 INFO:tensorflow:epoch = 49.56217616580311, learning_rate = 0.00049999997, loss = 5.10138e-05, step = 19131 (5.338 sec) 2021-10-19 07:57:29,568 [INFO] tensorflow: epoch = 49.56217616580311, learning_rate = 0.00049999997, loss = 5.10138e-05, step = 19131 (5.338 sec) 2021-10-19 07:57:33,105 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.134 INFO:tensorflow:global_step/sec: 5.05814 2021-10-19 07:57:33,688 [INFO] tensorflow: global_step/sec: 5.05814 INFO:tensorflow:epoch = 49.63212435233161, learning_rate = 0.00049999997, loss = 3.213526e-05, step = 19158 (5.321 sec) 2021-10-19 07:57:34,888 [INFO] tensorflow: epoch = 49.63212435233161, learning_rate = 0.00049999997, loss = 3.213526e-05, step = 19158 (5.321 sec) 2021-10-19 07:57:38,215 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.571 INFO:tensorflow:epoch = 49.69948186528497, learning_rate = 0.00049999997, loss = 2.3621815e-05, step = 19184 (5.331 sec) 2021-10-19 07:57:40,219 [INFO] tensorflow: epoch = 49.69948186528497, learning_rate = 0.00049999997, loss = 2.3621815e-05, step = 19184 (5.331 sec) INFO:tensorflow:global_step/sec: 4.91472 2021-10-19 07:57:41,420 [INFO] tensorflow: global_step/sec: 4.91472 2021-10-19 07:57:43,206 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.037 INFO:tensorflow:epoch = 49.766839378238345, learning_rate = 0.00049999997, loss = 2.5264731e-05, step = 19210 (5.196 sec) 2021-10-19 07:57:45,415 [INFO] tensorflow: epoch = 49.766839378238345, learning_rate = 0.00049999997, loss = 2.5264731e-05, step = 19210 (5.196 sec) 2021-10-19 07:57:48,297 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.644 INFO:tensorflow:global_step/sec: 4.92665 2021-10-19 07:57:49,133 [INFO] tensorflow: global_step/sec: 4.92665 INFO:tensorflow:epoch = 49.83419689119171, learning_rate = 0.00049999997, loss = 2.861828e-05, step = 19236 (5.301 sec) 2021-10-19 07:57:50,716 [INFO] tensorflow: epoch = 49.83419689119171, learning_rate = 0.00049999997, loss = 2.861828e-05, step = 19236 (5.301 sec) 2021-10-19 07:57:53,336 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.850 INFO:tensorflow:epoch = 49.90155440414508, learning_rate = 0.00049999997, loss = 2.796686e-05, step = 19262 (5.210 sec) 2021-10-19 07:57:55,926 [INFO] tensorflow: epoch = 49.90155440414508, learning_rate = 0.00049999997, loss = 2.796686e-05, step = 19262 (5.210 sec) INFO:tensorflow:global_step/sec: 5.00449 2021-10-19 07:57:56,726 [INFO] tensorflow: global_step/sec: 5.00449 2021-10-19 07:57:58,292 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.175 INFO:tensorflow:epoch = 49.97150259067358, learning_rate = 0.00049999997, loss = 3.6817895e-05, step = 19289 (5.349 sec) 2021-10-19 07:58:01,274 [INFO] tensorflow: epoch = 49.97150259067358, learning_rate = 0.00049999997, loss = 3.6817895e-05, step = 19289 (5.349 sec) INFO:tensorflow:Saving checkpoints for step-19300. 2021-10-19 07:58:03,281 [INFO] tensorflow: Saving checkpoints for step-19300. WARNING:tensorflow:Ignoring: /tmp/tmph6m8_ecq; No such file or directory 2021-10-19 07:58:03,511 [WARNING] tensorflow: Ignoring: /tmp/tmph6m8_ecq; No such file or directory 2021-10-19 07:58:08,139 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 62, 0.00s/step 2021-10-19 07:58:09,077 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 62, 0.09s/step 2021-10-19 07:58:10,030 [INFO] iva.detectnet_v2.evaluation.evaluation: step 20 / 62, 0.10s/step 2021-10-19 07:58:11,009 [INFO] iva.detectnet_v2.evaluation.evaluation: step 30 / 62, 0.10s/step 2021-10-19 07:58:11,970 [INFO] iva.detectnet_v2.evaluation.evaluation: step 40 / 62, 0.10s/step 2021-10-19 07:58:12,867 [INFO] iva.detectnet_v2.evaluation.evaluation: step 50 / 62, 0.09s/step 2021-10-19 07:58:13,716 [INFO] iva.detectnet_v2.evaluation.evaluation: step 60 / 62, 0.08s/step Matching predictions to ground truth, class 1/7.: 100%|█| 485/485 [00:00<00:00, 13417.48it/s] Matching predictions to ground truth, class 2/7.: 100%|█| 175/175 [00:00<00:00, 11257.55it/s] Matching predictions to ground truth, class 7/7.: 100%|█| 42/42 [00:00<00:00, 17677.95it/s] Epoch 50/120 ========================= Validation cost: 0.000025 Mean average_precision (in %): 16.4494 class name average precision (in %) ------------- -------------------------- auto_rickshaw 0 bicycle 53.1927 bus 11.9298 car 50.0236 person 0 road_sign 0 truck 0 Median Inference Time: 0.019887 2021-10-19 07:58:13,972 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 6.378 INFO:tensorflow:epoch = 50.0, learning_rate = 0.00049999997, loss = 2.4285315e-05, step = 19300 (12.915 sec) 2021-10-19 07:58:14,189 [INFO] tensorflow: epoch = 50.0, learning_rate = 0.00049999997, loss = 2.4285315e-05, step = 19300 (12.915 sec) 2021-10-19 07:58:14,190 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 50/120: loss: 0.00002 learning rate: 0.00050 Time taken: 0:01:28.037678 ETA: 1:42:42.637460 INFO:tensorflow:global_step/sec: 2.07788 2021-10-19 07:58:15,014 [INFO] tensorflow: global_step/sec: 2.07788 2021-10-19 07:58:19,021 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.813 INFO:tensorflow:epoch = 50.06735751295337, learning_rate = 0.00049999997, loss = 1.455927e-05, step = 19326 (5.238 sec) 2021-10-19 07:58:19,427 [INFO] tensorflow: epoch = 50.06735751295337, learning_rate = 0.00049999997, loss = 1.455927e-05, step = 19326 (5.238 sec) INFO:tensorflow:global_step/sec: 4.94917 2021-10-19 07:58:22,692 [INFO] tensorflow: global_step/sec: 4.94917 2021-10-19 07:58:24,150 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.500 INFO:tensorflow:epoch = 50.134715025906736, learning_rate = 0.00049999997, loss = 3.7454636e-05, step = 19352 (5.326 sec) 2021-10-19 07:58:24,753 [INFO] tensorflow: epoch = 50.134715025906736, learning_rate = 0.00049999997, loss = 3.7454636e-05, step = 19352 (5.326 sec) 2021-10-19 07:58:29,122 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.115 INFO:tensorflow:epoch = 50.204663212435236, learning_rate = 0.00049999997, loss = 3.3872115e-05, step = 19379 (5.390 sec) 2021-10-19 07:58:30,143 [INFO] tensorflow: epoch = 50.204663212435236, learning_rate = 0.00049999997, loss = 3.3872115e-05, step = 19379 (5.390 sec) INFO:tensorflow:global_step/sec: 4.96685 2021-10-19 07:58:30,343 [INFO] tensorflow: global_step/sec: 4.96685 2021-10-19 07:58:34,261 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.460 INFO:tensorflow:epoch = 50.2720207253886, learning_rate = 0.00049999997, loss = 3.0394851e-05, step = 19405 (5.361 sec) 2021-10-19 07:58:35,504 [INFO] tensorflow: epoch = 50.2720207253886, learning_rate = 0.00049999997, loss = 3.0394851e-05, step = 19405 (5.361 sec) INFO:tensorflow:global_step/sec: 4.91203 2021-10-19 07:58:38,079 [INFO] tensorflow: global_step/sec: 4.91203 2021-10-19 07:58:39,313 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.797 INFO:tensorflow:epoch = 50.33937823834197, learning_rate = 0.00049999997, loss = 1.8259534e-05, step = 19431 (5.195 sec) 2021-10-19 07:58:40,699 [INFO] tensorflow: epoch = 50.33937823834197, learning_rate = 0.00049999997, loss = 1.8259534e-05, step = 19431 (5.195 sec) 2021-10-19 07:58:44,303 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.041 INFO:tensorflow:global_step/sec: 4.96103 2021-10-19 07:58:45,739 [INFO] tensorflow: global_step/sec: 4.96103 INFO:tensorflow:epoch = 50.40673575129534, learning_rate = 0.00049999997, loss = 2.5710437e-05, step = 19457 (5.235 sec) 2021-10-19 07:58:45,934 [INFO] tensorflow: epoch = 50.40673575129534, learning_rate = 0.00049999997, loss = 2.5710437e-05, step = 19457 (5.235 sec) 2021-10-19 07:58:49,268 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.143 INFO:tensorflow:epoch = 50.476683937823836, learning_rate = 0.00049999997, loss = 1.3724641e-05, step = 19484 (5.357 sec) 2021-10-19 07:58:51,291 [INFO] tensorflow: epoch = 50.476683937823836, learning_rate = 0.00049999997, loss = 1.3724641e-05, step = 19484 (5.357 sec) INFO:tensorflow:global_step/sec: 5.00841 2021-10-19 07:58:53,326 [INFO] tensorflow: global_step/sec: 5.00841 2021-10-19 07:58:54,333 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.745 INFO:tensorflow:epoch = 50.5440414507772, learning_rate = 0.00049999997, loss = 2.5283702e-05, step = 19510 (5.219 sec) 2021-10-19 07:58:56,510 [INFO] tensorflow: epoch = 50.5440414507772, learning_rate = 0.00049999997, loss = 2.5283702e-05, step = 19510 (5.219 sec) 2021-10-19 07:58:59,284 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.201 INFO:tensorflow:global_step/sec: 5.04359 2021-10-19 07:59:00,861 [INFO] tensorflow: global_step/sec: 5.04359 INFO:tensorflow:epoch = 50.6139896373057, learning_rate = 0.00049999997, loss = 1.6755495e-05, step = 19537 (5.385 sec) 2021-10-19 07:59:01,895 [INFO] tensorflow: epoch = 50.6139896373057, learning_rate = 0.00049999997, loss = 1.6755495e-05, step = 19537 (5.385 sec) 2021-10-19 07:59:04,340 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.779 INFO:tensorflow:epoch = 50.68134715025907, learning_rate = 0.00049999997, loss = 2.8961906e-05, step = 19563 (5.225 sec) 2021-10-19 07:59:07,120 [INFO] tensorflow: epoch = 50.68134715025907, learning_rate = 0.00049999997, loss = 2.8961906e-05, step = 19563 (5.225 sec) INFO:tensorflow:global_step/sec: 4.94104 2021-10-19 07:59:08,551 [INFO] tensorflow: global_step/sec: 4.94104 2021-10-19 07:59:09,352 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.953 INFO:tensorflow:epoch = 50.751295336787564, learning_rate = 0.00049999997, loss = 2.375988e-05, step = 19590 (5.383 sec) 2021-10-19 07:59:12,503 [INFO] tensorflow: epoch = 50.751295336787564, learning_rate = 0.00049999997, loss = 2.375988e-05, step = 19590 (5.383 sec) 2021-10-19 07:59:14,304 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.197 INFO:tensorflow:global_step/sec: 5.06865 2021-10-19 07:59:16,048 [INFO] tensorflow: global_step/sec: 5.06865 INFO:tensorflow:epoch = 50.82124352331606, learning_rate = 0.00049999997, loss = 3.710219e-05, step = 19617 (5.328 sec) 2021-10-19 07:59:17,830 [INFO] tensorflow: epoch = 50.82124352331606, learning_rate = 0.00049999997, loss = 3.710219e-05, step = 19617 (5.328 sec) 2021-10-19 07:59:19,228 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.313 INFO:tensorflow:epoch = 50.888601036269435, learning_rate = 0.00049999997, loss = 1.2408965e-05, step = 19643 (5.219 sec) 2021-10-19 07:59:23,049 [INFO] tensorflow: epoch = 50.888601036269435, learning_rate = 0.00049999997, loss = 1.2408965e-05, step = 19643 (5.219 sec) INFO:tensorflow:global_step/sec: 5.01071 2021-10-19 07:59:23,632 [INFO] tensorflow: global_step/sec: 5.01071 2021-10-19 07:59:24,230 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.994 INFO:tensorflow:epoch = 50.95854922279793, learning_rate = 0.00049999997, loss = 3.3781413e-05, step = 19670 (5.326 sec) 2021-10-19 07:59:28,375 [INFO] tensorflow: epoch = 50.95854922279793, learning_rate = 0.00049999997, loss = 3.3781413e-05, step = 19670 (5.326 sec) 2021-10-19 07:59:29,213 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.070 INFO:tensorflow:global_step/sec: 5.01863 2021-10-19 07:59:31,204 [INFO] tensorflow: global_step/sec: 5.01863 2021-10-19 07:59:31,604 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 51/120: loss: 0.00001 learning rate: 0.00050 Time taken: 0:01:17.430937 ETA: 1:29:02.734624 INFO:tensorflow:epoch = 51.0259067357513, learning_rate = 0.00049999997, loss = 2.9391678e-05, step = 19696 (5.227 sec) 2021-10-19 07:59:33,602 [INFO] tensorflow: epoch = 51.0259067357513, learning_rate = 0.00049999997, loss = 2.9391678e-05, step = 19696 (5.227 sec) 2021-10-19 07:59:34,183 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.121 INFO:tensorflow:global_step/sec: 5.07427 2021-10-19 07:59:38,693 [INFO] tensorflow: global_step/sec: 5.07427 INFO:tensorflow:epoch = 51.09585492227979, learning_rate = 0.00049999997, loss = 1.2810176e-05, step = 19723 (5.301 sec) 2021-10-19 07:59:38,904 [INFO] tensorflow: epoch = 51.09585492227979, learning_rate = 0.00049999997, loss = 1.2810176e-05, step = 19723 (5.301 sec) 2021-10-19 07:59:39,116 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.273 2021-10-19 07:59:44,117 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.998 INFO:tensorflow:epoch = 51.16580310880829, learning_rate = 0.00049999997, loss = 3.568408e-05, step = 19750 (5.443 sec) 2021-10-19 07:59:44,347 [INFO] tensorflow: epoch = 51.16580310880829, learning_rate = 0.00049999997, loss = 3.568408e-05, step = 19750 (5.443 sec) INFO:tensorflow:global_step/sec: 4.97672 2021-10-19 07:59:46,328 [INFO] tensorflow: global_step/sec: 4.97672 2021-10-19 07:59:49,158 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.840 INFO:tensorflow:epoch = 51.23316062176166, learning_rate = 0.00049999997, loss = 1.7659833e-05, step = 19776 (5.240 sec) 2021-10-19 07:59:49,587 [INFO] tensorflow: epoch = 51.23316062176166, learning_rate = 0.00049999997, loss = 1.7659833e-05, step = 19776 (5.240 sec) INFO:tensorflow:global_step/sec: 4.9863 2021-10-19 07:59:53,949 [INFO] tensorflow: global_step/sec: 4.9863 2021-10-19 07:59:54,164 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.978 INFO:tensorflow:epoch = 51.303108808290155, learning_rate = 0.00049999997, loss = 2.4425835e-05, step = 19803 (5.385 sec) 2021-10-19 07:59:54,972 [INFO] tensorflow: epoch = 51.303108808290155, learning_rate = 0.00049999997, loss = 2.4425835e-05, step = 19803 (5.385 sec) 2021-10-19 07:59:59,194 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.882 INFO:tensorflow:epoch = 51.370466321243526, learning_rate = 0.00049999997, loss = 2.151385e-05, step = 19829 (5.207 sec) 2021-10-19 08:00:00,179 [INFO] tensorflow: epoch = 51.370466321243526, learning_rate = 0.00049999997, loss = 2.151385e-05, step = 19829 (5.207 sec) INFO:tensorflow:global_step/sec: 4.96236 2021-10-19 08:00:01,607 [INFO] tensorflow: global_step/sec: 4.96236 2021-10-19 08:00:04,308 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.554 INFO:tensorflow:epoch = 51.43782383419689, learning_rate = 0.00049999997, loss = 1.1886414e-05, step = 19855 (5.306 sec) 2021-10-19 08:00:05,485 [INFO] tensorflow: epoch = 51.43782383419689, learning_rate = 0.00049999997, loss = 1.1886414e-05, step = 19855 (5.306 sec) INFO:tensorflow:global_step/sec: 4.96173 2021-10-19 08:00:09,265 [INFO] tensorflow: global_step/sec: 4.96173 2021-10-19 08:00:09,267 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.169 INFO:tensorflow:epoch = 51.50518134715026, learning_rate = 0.00049999997, loss = 1.944122e-05, step = 19881 (5.205 sec) 2021-10-19 08:00:10,690 [INFO] tensorflow: epoch = 51.50518134715026, learning_rate = 0.00049999997, loss = 1.944122e-05, step = 19881 (5.205 sec) 2021-10-19 08:00:14,373 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.588 INFO:tensorflow:epoch = 51.57253886010363, learning_rate = 0.00049999997, loss = 1.4819143e-05, step = 19907 (5.267 sec) 2021-10-19 08:00:15,957 [INFO] tensorflow: epoch = 51.57253886010363, learning_rate = 0.00049999997, loss = 1.4819143e-05, step = 19907 (5.267 sec) INFO:tensorflow:global_step/sec: 4.95634 2021-10-19 08:00:16,932 [INFO] tensorflow: global_step/sec: 4.95634 2021-10-19 08:00:19,375 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.994 INFO:tensorflow:epoch = 51.639896373057, learning_rate = 0.00049999997, loss = 2.7523765e-05, step = 19933 (5.228 sec) 2021-10-19 08:00:21,185 [INFO] tensorflow: epoch = 51.639896373057, learning_rate = 0.00049999997, loss = 2.7523765e-05, step = 19933 (5.228 sec) 2021-10-19 08:00:24,372 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.013 INFO:tensorflow:global_step/sec: 4.97515 2021-10-19 08:00:24,570 [INFO] tensorflow: global_step/sec: 4.97515 INFO:tensorflow:epoch = 51.70984455958549, learning_rate = 0.00049999997, loss = 2.1355754e-05, step = 19960 (5.383 sec) 2021-10-19 08:00:26,568 [INFO] tensorflow: epoch = 51.70984455958549, learning_rate = 0.00049999997, loss = 2.1355754e-05, step = 19960 (5.383 sec) 2021-10-19 08:00:29,368 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.019 INFO:tensorflow:epoch = 51.77979274611399, learning_rate = 0.00049999997, loss = 4.052072e-05, step = 19987 (5.396 sec) 2021-10-19 08:00:31,964 [INFO] tensorflow: epoch = 51.77979274611399, learning_rate = 0.00049999997, loss = 4.052072e-05, step = 19987 (5.396 sec) INFO:tensorflow:global_step/sec: 5.00497 2021-10-19 08:00:32,163 [INFO] tensorflow: global_step/sec: 5.00497 2021-10-19 08:00:34,368 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.002 INFO:tensorflow:epoch = 51.84974093264249, learning_rate = 0.00049999997, loss = 4.6145255e-05, step = 20014 (5.336 sec) 2021-10-19 08:00:37,299 [INFO] tensorflow: epoch = 51.84974093264249, learning_rate = 0.00049999997, loss = 4.6145255e-05, step = 20014 (5.336 sec) 2021-10-19 08:00:39,287 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.333 INFO:tensorflow:global_step/sec: 5.04768 2021-10-19 08:00:39,691 [INFO] tensorflow: global_step/sec: 5.04768 INFO:tensorflow:epoch = 51.91968911917099, learning_rate = 0.00049999997, loss = 3.0257903e-05, step = 20041 (5.366 sec) 2021-10-19 08:00:42,665 [INFO] tensorflow: epoch = 51.91968911917099, learning_rate = 0.00049999997, loss = 3.0257903e-05, step = 20041 (5.366 sec) 2021-10-19 08:00:44,223 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.260 INFO:tensorflow:global_step/sec: 5.04676 2021-10-19 08:00:47,220 [INFO] tensorflow: global_step/sec: 5.04676 INFO:tensorflow:epoch = 51.98963730569948, learning_rate = 0.00049999997, loss = 1.846757e-05, step = 20068 (5.366 sec) 2021-10-19 08:00:48,031 [INFO] tensorflow: epoch = 51.98963730569948, learning_rate = 0.00049999997, loss = 1.846757e-05, step = 20068 (5.366 sec) 2021-10-19 08:00:48,865 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 52/120: loss: 0.00002 learning rate: 0.00050 Time taken: 0:01:17.228403 ETA: 1:27:31.531410 2021-10-19 08:00:49,260 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.856 INFO:tensorflow:epoch = 52.05699481865285, learning_rate = 0.00049999997, loss = 1.29378905e-05, step = 20094 (5.285 sec) 2021-10-19 08:00:53,316 [INFO] tensorflow: epoch = 52.05699481865285, learning_rate = 0.00049999997, loss = 1.29378905e-05, step = 20094 (5.285 sec) 2021-10-19 08:00:54,312 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.795 INFO:tensorflow:global_step/sec: 4.94546 2021-10-19 08:00:54,904 [INFO] tensorflow: global_step/sec: 4.94546 INFO:tensorflow:epoch = 52.12435233160622, learning_rate = 0.00049999997, loss = 2.3095527e-05, step = 20120 (5.312 sec) 2021-10-19 08:00:58,627 [INFO] tensorflow: epoch = 52.12435233160622, learning_rate = 0.00049999997, loss = 2.3095527e-05, step = 20120 (5.312 sec) 2021-10-19 08:00:59,471 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.387 INFO:tensorflow:global_step/sec: 4.88591 2021-10-19 08:01:02,682 [INFO] tensorflow: global_step/sec: 4.88591 INFO:tensorflow:epoch = 52.19170984455959, learning_rate = 0.00049999997, loss = 1.7894754e-05, step = 20146 (5.264 sec) 2021-10-19 08:01:03,891 [INFO] tensorflow: epoch = 52.19170984455959, learning_rate = 0.00049999997, loss = 1.7894754e-05, step = 20146 (5.264 sec) 2021-10-19 08:01:04,535 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.749 INFO:tensorflow:epoch = 52.259067357512954, learning_rate = 0.00049999997, loss = 2.2615572e-05, step = 20172 (5.310 sec) 2021-10-19 08:01:09,201 [INFO] tensorflow: epoch = 52.259067357512954, learning_rate = 0.00049999997, loss = 2.2615572e-05, step = 20172 (5.310 sec) 2021-10-19 08:01:09,598 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.753 INFO:tensorflow:global_step/sec: 4.91844 2021-10-19 08:01:10,408 [INFO] tensorflow: global_step/sec: 4.91844 INFO:tensorflow:epoch = 52.326424870466326, learning_rate = 0.00049999997, loss = 2.3561095e-05, step = 20198 (5.210 sec) 2021-10-19 08:01:14,411 [INFO] tensorflow: epoch = 52.326424870466326, learning_rate = 0.00049999997, loss = 2.3561095e-05, step = 20198 (5.210 sec) 2021-10-19 08:01:14,614 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.936 INFO:tensorflow:global_step/sec: 5.02807 2021-10-19 08:01:17,965 [INFO] tensorflow: global_step/sec: 5.02807 2021-10-19 08:01:19,604 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.045 INFO:tensorflow:epoch = 52.39637305699482, learning_rate = 0.00049999997, loss = 1.2795895e-05, step = 20225 (5.390 sec) 2021-10-19 08:01:19,801 [INFO] tensorflow: epoch = 52.39637305699482, learning_rate = 0.00049999997, loss = 1.2795895e-05, step = 20225 (5.390 sec) 2021-10-19 08:01:24,532 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.294 INFO:tensorflow:epoch = 52.46632124352332, learning_rate = 0.00049999997, loss = 2.4611736e-05, step = 20252 (5.345 sec) 2021-10-19 08:01:25,146 [INFO] tensorflow: epoch = 52.46632124352332, learning_rate = 0.00049999997, loss = 2.4611736e-05, step = 20252 (5.345 sec) INFO:tensorflow:global_step/sec: 5.01487 2021-10-19 08:01:25,543 [INFO] tensorflow: global_step/sec: 5.01487 2021-10-19 08:01:29,503 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.117 INFO:tensorflow:epoch = 52.53626943005182, learning_rate = 0.00049999997, loss = 1.8842551e-05, step = 20279 (5.341 sec) 2021-10-19 08:01:30,487 [INFO] tensorflow: epoch = 52.53626943005182, learning_rate = 0.00049999997, loss = 1.8842551e-05, step = 20279 (5.341 sec) INFO:tensorflow:global_step/sec: 5.02921 2021-10-19 08:01:33,099 [INFO] tensorflow: global_step/sec: 5.02921 2021-10-19 08:01:34,465 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.156 INFO:tensorflow:epoch = 52.60621761658031, learning_rate = 0.00049999997, loss = 1.23105365e-05, step = 20306 (5.392 sec) 2021-10-19 08:01:35,879 [INFO] tensorflow: epoch = 52.60621761658031, learning_rate = 0.00049999997, loss = 1.23105365e-05, step = 20306 (5.392 sec) 2021-10-19 08:01:39,462 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.016 INFO:tensorflow:global_step/sec: 5.03435 2021-10-19 08:01:40,647 [INFO] tensorflow: global_step/sec: 5.03435 INFO:tensorflow:epoch = 52.67616580310881, learning_rate = 0.00049999997, loss = 1.29367945e-05, step = 20333 (5.385 sec) 2021-10-19 08:01:41,264 [INFO] tensorflow: epoch = 52.67616580310881, learning_rate = 0.00049999997, loss = 1.29367945e-05, step = 20333 (5.385 sec) 2021-10-19 08:01:44,443 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.079 INFO:tensorflow:epoch = 52.74611398963731, learning_rate = 0.00049999997, loss = 1.8273957e-05, step = 20360 (5.359 sec) 2021-10-19 08:01:46,623 [INFO] tensorflow: epoch = 52.74611398963731, learning_rate = 0.00049999997, loss = 1.8273957e-05, step = 20360 (5.359 sec) INFO:tensorflow:global_step/sec: 5.0066 2021-10-19 08:01:48,237 [INFO] tensorflow: global_step/sec: 5.0066 2021-10-19 08:01:49,428 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.061 INFO:tensorflow:epoch = 52.81347150259067, learning_rate = 0.00049999997, loss = 1.4330604e-05, step = 20386 (5.215 sec) 2021-10-19 08:01:51,838 [INFO] tensorflow: epoch = 52.81347150259067, learning_rate = 0.00049999997, loss = 1.4330604e-05, step = 20386 (5.215 sec) 2021-10-19 08:01:54,454 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.902 INFO:tensorflow:global_step/sec: 4.97565 2021-10-19 08:01:55,874 [INFO] tensorflow: global_step/sec: 4.97565 INFO:tensorflow:epoch = 52.880829015544045, learning_rate = 0.00049999997, loss = 2.454684e-05, step = 20412 (5.213 sec) 2021-10-19 08:01:57,050 [INFO] tensorflow: epoch = 52.880829015544045, learning_rate = 0.00049999997, loss = 2.454684e-05, step = 20412 (5.213 sec) 2021-10-19 08:01:59,426 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.112 INFO:tensorflow:epoch = 52.950777202072544, learning_rate = 0.00049999997, loss = 3.035642e-05, step = 20439 (5.336 sec) 2021-10-19 08:02:02,387 [INFO] tensorflow: epoch = 52.950777202072544, learning_rate = 0.00049999997, loss = 3.035642e-05, step = 20439 (5.336 sec) INFO:tensorflow:global_step/sec: 5.04398 2021-10-19 08:02:03,408 [INFO] tensorflow: global_step/sec: 5.04398 2021-10-19 08:02:04,381 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.183 2021-10-19 08:02:06,195 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 53/120: loss: 0.00002 learning rate: 0.00050 Time taken: 0:01:17.359281 ETA: 1:26:23.071847 INFO:tensorflow:epoch = 53.01813471502591, learning_rate = 0.00049999997, loss = 1.2000509e-05, step = 20465 (5.231 sec) 2021-10-19 08:02:07,618 [INFO] tensorflow: epoch = 53.01813471502591, learning_rate = 0.00049999997, loss = 1.2000509e-05, step = 20465 (5.231 sec) 2021-10-19 08:02:09,404 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.910 INFO:tensorflow:global_step/sec: 4.98985 2021-10-19 08:02:11,023 [INFO] tensorflow: global_step/sec: 4.98985 INFO:tensorflow:epoch = 53.08549222797927, learning_rate = 0.00049999997, loss = 4.1611223e-05, step = 20491 (5.196 sec) 2021-10-19 08:02:12,814 [INFO] tensorflow: epoch = 53.08549222797927, learning_rate = 0.00049999997, loss = 4.1611223e-05, step = 20491 (5.196 sec) 2021-10-19 08:02:14,493 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.652 INFO:tensorflow:epoch = 53.152849740932645, learning_rate = 0.00049999997, loss = 1.992162e-05, step = 20517 (5.227 sec) 2021-10-19 08:02:18,041 [INFO] tensorflow: epoch = 53.152849740932645, learning_rate = 0.00049999997, loss = 1.992162e-05, step = 20517 (5.227 sec) INFO:tensorflow:global_step/sec: 4.98527 2021-10-19 08:02:18,646 [INFO] tensorflow: global_step/sec: 4.98527 2021-10-19 08:02:19,454 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.161 INFO:tensorflow:epoch = 53.22020725388601, learning_rate = 0.00049999997, loss = 3.0866588e-05, step = 20543 (5.208 sec) 2021-10-19 08:02:23,250 [INFO] tensorflow: epoch = 53.22020725388601, learning_rate = 0.00049999997, loss = 3.0866588e-05, step = 20543 (5.208 sec) 2021-10-19 08:02:24,440 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.057 INFO:tensorflow:global_step/sec: 5.0009 2021-10-19 08:02:26,244 [INFO] tensorflow: global_step/sec: 5.0009 INFO:tensorflow:epoch = 53.29015544041451, learning_rate = 0.00049999997, loss = 4.7393296e-05, step = 20570 (5.387 sec) 2021-10-19 08:02:28,637 [INFO] tensorflow: epoch = 53.29015544041451, learning_rate = 0.00049999997, loss = 4.7393296e-05, step = 20570 (5.387 sec) 2021-10-19 08:02:29,479 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.847 INFO:tensorflow:epoch = 53.35751295336788, learning_rate = 0.00049999997, loss = 2.3677467e-05, step = 20596 (5.211 sec) 2021-10-19 08:02:33,848 [INFO] tensorflow: epoch = 53.35751295336788, learning_rate = 0.00049999997, loss = 2.3677467e-05, step = 20596 (5.211 sec) INFO:tensorflow:global_step/sec: 4.99694 2021-10-19 08:02:33,849 [INFO] tensorflow: global_step/sec: 4.99694 2021-10-19 08:02:34,465 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.059 INFO:tensorflow:epoch = 53.424870466321245, learning_rate = 0.00049999997, loss = 1.1693429e-05, step = 20622 (5.219 sec) 2021-10-19 08:02:39,066 [INFO] tensorflow: epoch = 53.424870466321245, learning_rate = 0.00049999997, loss = 1.1693429e-05, step = 20622 (5.219 sec) 2021-10-19 08:02:39,485 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.920 INFO:tensorflow:global_step/sec: 4.99661 2021-10-19 08:02:41,454 [INFO] tensorflow: global_step/sec: 4.99661 INFO:tensorflow:epoch = 53.49222797927462, learning_rate = 0.00049999997, loss = 1.589406e-05, step = 20648 (5.199 sec) 2021-10-19 08:02:44,265 [INFO] tensorflow: epoch = 53.49222797927462, learning_rate = 0.00049999997, loss = 1.589406e-05, step = 20648 (5.199 sec) 2021-10-19 08:02:44,460 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.102 INFO:tensorflow:global_step/sec: 4.98449 2021-10-19 08:02:49,078 [INFO] tensorflow: global_step/sec: 4.98449 INFO:tensorflow:epoch = 53.55958549222798, learning_rate = 0.00049999997, loss = 3.2879318e-05, step = 20674 (5.258 sec) 2021-10-19 08:02:49,523 [INFO] tensorflow: epoch = 53.55958549222798, learning_rate = 0.00049999997, loss = 3.2879318e-05, step = 20674 (5.258 sec) 2021-10-19 08:02:49,523 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.752 2021-10-19 08:02:54,493 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.123 INFO:tensorflow:epoch = 53.62953367875648, learning_rate = 0.00049999997, loss = 4.3176064e-05, step = 20701 (5.370 sec) 2021-10-19 08:02:54,893 [INFO] tensorflow: epoch = 53.62953367875648, learning_rate = 0.00049999997, loss = 4.3176064e-05, step = 20701 (5.370 sec) INFO:tensorflow:global_step/sec: 4.9889 2021-10-19 08:02:56,695 [INFO] tensorflow: global_step/sec: 4.9889 2021-10-19 08:02:59,457 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.149 INFO:tensorflow:epoch = 53.69948186528497, learning_rate = 0.00049999997, loss = 3.5479767e-05, step = 20728 (5.361 sec) 2021-10-19 08:03:00,254 [INFO] tensorflow: epoch = 53.69948186528497, learning_rate = 0.00049999997, loss = 3.5479767e-05, step = 20728 (5.361 sec) INFO:tensorflow:global_step/sec: 5.06196 2021-10-19 08:03:04,202 [INFO] tensorflow: global_step/sec: 5.06196 2021-10-19 08:03:04,406 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.207 INFO:tensorflow:epoch = 53.76943005181347, learning_rate = 0.00049999997, loss = 2.2387187e-05, step = 20755 (5.337 sec) 2021-10-19 08:03:05,591 [INFO] tensorflow: epoch = 53.76943005181347, learning_rate = 0.00049999997, loss = 2.2387187e-05, step = 20755 (5.337 sec) 2021-10-19 08:03:09,370 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.148 INFO:tensorflow:epoch = 53.83937823834197, learning_rate = 0.00049999997, loss = 3.464366e-05, step = 20782 (5.383 sec) 2021-10-19 08:03:10,973 [INFO] tensorflow: epoch = 53.83937823834197, learning_rate = 0.00049999997, loss = 3.464366e-05, step = 20782 (5.383 sec) INFO:tensorflow:global_step/sec: 5.02823 2021-10-19 08:03:11,759 [INFO] tensorflow: global_step/sec: 5.02823 2021-10-19 08:03:14,399 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.885 INFO:tensorflow:epoch = 53.90932642487047, learning_rate = 0.00049999997, loss = 1.2536269e-05, step = 20809 (5.366 sec) 2021-10-19 08:03:16,339 [INFO] tensorflow: epoch = 53.90932642487047, learning_rate = 0.00049999997, loss = 1.2536269e-05, step = 20809 (5.366 sec) INFO:tensorflow:global_step/sec: 5.02657 2021-10-19 08:03:19,319 [INFO] tensorflow: global_step/sec: 5.02657 2021-10-19 08:03:19,321 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.320 INFO:tensorflow:epoch = 53.979274611398964, learning_rate = 0.00049999997, loss = 3.783119e-05, step = 20836 (5.332 sec) 2021-10-19 08:03:21,672 [INFO] tensorflow: epoch = 53.979274611398964, learning_rate = 0.00049999997, loss = 3.783119e-05, step = 20836 (5.332 sec) 2021-10-19 08:03:23,330 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 54/120: loss: 0.00002 learning rate: 0.00050 Time taken: 0:01:17.128387 ETA: 1:24:50.473556 2021-10-19 08:03:24,324 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.989 INFO:tensorflow:epoch = 54.046632124352335, learning_rate = 0.00049999997, loss = 3.4820187e-05, step = 20862 (5.226 sec) 2021-10-19 08:03:26,898 [INFO] tensorflow: epoch = 54.046632124352335, learning_rate = 0.00049999997, loss = 3.4820187e-05, step = 20862 (5.226 sec) INFO:tensorflow:global_step/sec: 5.01211 2021-10-19 08:03:26,901 [INFO] tensorflow: global_step/sec: 5.01211 2021-10-19 08:03:29,383 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.768 INFO:tensorflow:epoch = 54.1139896373057, learning_rate = 0.00049999997, loss = 2.471423e-05, step = 20888 (5.208 sec) 2021-10-19 08:03:32,106 [INFO] tensorflow: epoch = 54.1139896373057, learning_rate = 0.00049999997, loss = 2.471423e-05, step = 20888 (5.208 sec) 2021-10-19 08:03:34,276 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.444 INFO:tensorflow:global_step/sec: 5.01145 2021-10-19 08:03:34,483 [INFO] tensorflow: global_step/sec: 5.01145 INFO:tensorflow:epoch = 54.1839378238342, learning_rate = 0.00049999997, loss = 2.2119717e-05, step = 20915 (5.392 sec) 2021-10-19 08:03:37,499 [INFO] tensorflow: epoch = 54.1839378238342, learning_rate = 0.00049999997, loss = 2.2119717e-05, step = 20915 (5.392 sec) 2021-10-19 08:03:39,291 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.941 INFO:tensorflow:global_step/sec: 5.00435 2021-10-19 08:03:42,076 [INFO] tensorflow: global_step/sec: 5.00435 INFO:tensorflow:epoch = 54.2538860103627, learning_rate = 0.00049999997, loss = 3.772146e-05, step = 20942 (5.356 sec) 2021-10-19 08:03:42,854 [INFO] tensorflow: epoch = 54.2538860103627, learning_rate = 0.00049999997, loss = 3.772146e-05, step = 20942 (5.356 sec) 2021-10-19 08:03:44,219 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.295 INFO:tensorflow:epoch = 54.32383419689119, learning_rate = 0.00049999997, loss = 1.6137783e-05, step = 20969 (5.309 sec) 2021-10-19 08:03:48,163 [INFO] tensorflow: epoch = 54.32383419689119, learning_rate = 0.00049999997, loss = 1.6137783e-05, step = 20969 (5.309 sec) 2021-10-19 08:03:49,168 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.205 INFO:tensorflow:global_step/sec: 5.07466 2021-10-19 08:03:49,565 [INFO] tensorflow: global_step/sec: 5.07466 INFO:tensorflow:epoch = 54.39119170984456, learning_rate = 0.00049999997, loss = 2.2970748e-05, step = 20995 (5.232 sec) 2021-10-19 08:03:53,395 [INFO] tensorflow: epoch = 54.39119170984456, learning_rate = 0.00049999997, loss = 2.2970748e-05, step = 20995 (5.232 sec) 2021-10-19 08:03:54,212 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.827 INFO:tensorflow:global_step/sec: 5.00222 2021-10-19 08:03:57,161 [INFO] tensorflow: global_step/sec: 5.00222 INFO:tensorflow:epoch = 54.46113989637306, learning_rate = 0.00049999997, loss = 2.125204e-05, step = 21022 (5.375 sec) 2021-10-19 08:03:58,770 [INFO] tensorflow: epoch = 54.46113989637306, learning_rate = 0.00049999997, loss = 2.125204e-05, step = 21022 (5.375 sec) 2021-10-19 08:03:59,185 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.112 INFO:tensorflow:epoch = 54.531088082901555, learning_rate = 0.00049999997, loss = 2.6592472e-05, step = 21049 (5.341 sec) 2021-10-19 08:04:04,111 [INFO] tensorflow: epoch = 54.531088082901555, learning_rate = 0.00049999997, loss = 2.6592472e-05, step = 21049 (5.341 sec) 2021-10-19 08:04:04,111 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.301 INFO:tensorflow:global_step/sec: 5.02004 2021-10-19 08:04:04,731 [INFO] tensorflow: global_step/sec: 5.02004 2021-10-19 08:04:09,123 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.953 INFO:tensorflow:epoch = 54.598445595854926, learning_rate = 0.00049999997, loss = 1.5929007e-05, step = 21075 (5.208 sec) 2021-10-19 08:04:09,318 [INFO] tensorflow: epoch = 54.598445595854926, learning_rate = 0.00049999997, loss = 1.5929007e-05, step = 21075 (5.208 sec) INFO:tensorflow:global_step/sec: 5.05144 2021-10-19 08:04:12,254 [INFO] tensorflow: global_step/sec: 5.05144 2021-10-19 08:04:13,990 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.548 INFO:tensorflow:epoch = 54.66839378238342, learning_rate = 0.00049999997, loss = 2.162009e-05, step = 21102 (5.261 sec) 2021-10-19 08:04:14,580 [INFO] tensorflow: epoch = 54.66839378238342, learning_rate = 0.00049999997, loss = 2.162009e-05, step = 21102 (5.261 sec) 2021-10-19 08:04:18,871 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.491 INFO:tensorflow:global_step/sec: 5.12738 2021-10-19 08:04:19,665 [INFO] tensorflow: global_step/sec: 5.12738 INFO:tensorflow:epoch = 54.73834196891192, learning_rate = 0.00049999997, loss = 4.9038812e-05, step = 21129 (5.288 sec) 2021-10-19 08:04:19,868 [INFO] tensorflow: epoch = 54.73834196891192, learning_rate = 0.00049999997, loss = 4.9038812e-05, step = 21129 (5.288 sec) 2021-10-19 08:04:23,907 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.860 INFO:tensorflow:epoch = 54.80569948186529, learning_rate = 0.00049999997, loss = 1.9380775e-05, step = 21155 (5.218 sec) 2021-10-19 08:04:25,086 [INFO] tensorflow: epoch = 54.80569948186529, learning_rate = 0.00049999997, loss = 1.9380775e-05, step = 21155 (5.218 sec) INFO:tensorflow:global_step/sec: 4.94951 2021-10-19 08:04:27,342 [INFO] tensorflow: global_step/sec: 4.94951 2021-10-19 08:04:28,913 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.977 INFO:tensorflow:epoch = 54.873056994818654, learning_rate = 0.00049999997, loss = 2.3532762e-05, step = 21181 (5.219 sec) 2021-10-19 08:04:30,305 [INFO] tensorflow: epoch = 54.873056994818654, learning_rate = 0.00049999997, loss = 2.3532762e-05, step = 21181 (5.219 sec) 2021-10-19 08:04:33,884 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.121 INFO:tensorflow:global_step/sec: 5.05356 2021-10-19 08:04:34,862 [INFO] tensorflow: global_step/sec: 5.05356 INFO:tensorflow:epoch = 54.943005181347154, learning_rate = 0.00049999997, loss = 2.7068443e-05, step = 21208 (5.386 sec) 2021-10-19 08:04:35,691 [INFO] tensorflow: epoch = 54.943005181347154, learning_rate = 0.00049999997, loss = 2.7068443e-05, step = 21208 (5.386 sec) 2021-10-19 08:04:38,926 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.836 2021-10-19 08:04:40,138 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 55/120: loss: 0.00002 learning rate: 0.00050 Time taken: 0:01:16.813781 ETA: 1:23:12.895736 INFO:tensorflow:epoch = 55.01036269430052, learning_rate = 0.00049999997, loss = 2.1469135e-05, step = 21234 (5.229 sec) 2021-10-19 08:04:40,920 [INFO] tensorflow: epoch = 55.01036269430052, learning_rate = 0.00049999997, loss = 2.1469135e-05, step = 21234 (5.229 sec) INFO:tensorflow:global_step/sec: 4.98043 2021-10-19 08:04:42,492 [INFO] tensorflow: global_step/sec: 4.98043 2021-10-19 08:04:43,861 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.264 INFO:tensorflow:epoch = 55.08031088082902, learning_rate = 0.00049999997, loss = 2.888044e-05, step = 21261 (5.301 sec) 2021-10-19 08:04:46,221 [INFO] tensorflow: epoch = 55.08031088082902, learning_rate = 0.00049999997, loss = 2.888044e-05, step = 21261 (5.301 sec) 2021-10-19 08:04:48,839 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.092 INFO:tensorflow:global_step/sec: 5.04901 2021-10-19 08:04:50,018 [INFO] tensorflow: global_step/sec: 5.04901 INFO:tensorflow:epoch = 55.15025906735752, learning_rate = 0.00049999997, loss = 1.1230542e-05, step = 21288 (5.381 sec) 2021-10-19 08:04:51,603 [INFO] tensorflow: epoch = 55.15025906735752, learning_rate = 0.00049999997, loss = 1.1230542e-05, step = 21288 (5.381 sec) 2021-10-19 08:04:53,824 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.061 INFO:tensorflow:epoch = 55.22020725388601, learning_rate = 0.00049999997, loss = 2.3525285e-05, step = 21315 (5.358 sec) 2021-10-19 08:04:56,960 [INFO] tensorflow: epoch = 55.22020725388601, learning_rate = 0.00049999997, loss = 2.3525285e-05, step = 21315 (5.358 sec) INFO:tensorflow:global_step/sec: 5.04452 2021-10-19 08:04:57,551 [INFO] tensorflow: global_step/sec: 5.04452 2021-10-19 08:04:58,733 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.371 INFO:tensorflow:epoch = 55.29015544041451, learning_rate = 0.00049999997, loss = 3.5296052e-05, step = 21342 (5.351 sec) 2021-10-19 08:05:02,312 [INFO] tensorflow: epoch = 55.29015544041451, learning_rate = 0.00049999997, loss = 3.5296052e-05, step = 21342 (5.351 sec) 2021-10-19 08:05:03,706 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.112 INFO:tensorflow:global_step/sec: 5.0067 2021-10-19 08:05:05,141 [INFO] tensorflow: global_step/sec: 5.0067 INFO:tensorflow:epoch = 55.36010362694301, learning_rate = 0.00049999997, loss = 2.3136547e-05, step = 21369 (5.348 sec) 2021-10-19 08:05:07,660 [INFO] tensorflow: epoch = 55.36010362694301, learning_rate = 0.00049999997, loss = 2.3136547e-05, step = 21369 (5.348 sec) 2021-10-19 08:05:08,678 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.115 INFO:tensorflow:global_step/sec: 5.05091 2021-10-19 08:05:12,664 [INFO] tensorflow: global_step/sec: 5.05091 INFO:tensorflow:epoch = 55.42746113989637, learning_rate = 0.00049999997, loss = 1.707247e-05, step = 21395 (5.205 sec) 2021-10-19 08:05:12,865 [INFO] tensorflow: epoch = 55.42746113989637, learning_rate = 0.00049999997, loss = 1.707247e-05, step = 21395 (5.205 sec) 2021-10-19 08:05:13,677 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.009 INFO:tensorflow:epoch = 55.494818652849744, learning_rate = 0.00049999997, loss = 2.2805874e-05, step = 21421 (5.258 sec) 2021-10-19 08:05:18,124 [INFO] tensorflow: epoch = 55.494818652849744, learning_rate = 0.00049999997, loss = 2.2805874e-05, step = 21421 (5.258 sec) 2021-10-19 08:05:18,713 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.857 INFO:tensorflow:global_step/sec: 4.97941 2021-10-19 08:05:20,295 [INFO] tensorflow: global_step/sec: 4.97941 INFO:tensorflow:epoch = 55.564766839378244, learning_rate = 0.00049999997, loss = 2.2943714e-05, step = 21448 (5.320 sec) 2021-10-19 08:05:23,443 [INFO] tensorflow: epoch = 55.564766839378244, learning_rate = 0.00049999997, loss = 2.2943714e-05, step = 21448 (5.320 sec) 2021-10-19 08:05:23,641 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.295 INFO:tensorflow:global_step/sec: 5.08504 2021-10-19 08:05:27,768 [INFO] tensorflow: global_step/sec: 5.08504 2021-10-19 08:05:28,567 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.304 INFO:tensorflow:epoch = 55.634715025906736, learning_rate = 0.00049999997, loss = 3.433552e-05, step = 21475 (5.326 sec) 2021-10-19 08:05:28,769 [INFO] tensorflow: epoch = 55.634715025906736, learning_rate = 0.00049999997, loss = 3.433552e-05, step = 21475 (5.326 sec) 2021-10-19 08:05:33,587 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.919 INFO:tensorflow:epoch = 55.70207253886011, learning_rate = 0.00049999997, loss = 2.4418883e-05, step = 21501 (5.218 sec) 2021-10-19 08:05:33,987 [INFO] tensorflow: epoch = 55.70207253886011, learning_rate = 0.00049999997, loss = 2.4418883e-05, step = 21501 (5.218 sec) INFO:tensorflow:global_step/sec: 4.97807 2021-10-19 08:05:35,402 [INFO] tensorflow: global_step/sec: 4.97807 2021-10-19 08:05:38,605 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.933 INFO:tensorflow:epoch = 55.76943005181347, learning_rate = 0.00049999997, loss = 2.4082996e-05, step = 21527 (5.204 sec) 2021-10-19 08:05:39,192 [INFO] tensorflow: epoch = 55.76943005181347, learning_rate = 0.00049999997, loss = 2.4082996e-05, step = 21527 (5.204 sec) INFO:tensorflow:global_step/sec: 5.03199 2021-10-19 08:05:42,953 [INFO] tensorflow: global_step/sec: 5.03199 2021-10-19 08:05:43,595 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.040 INFO:tensorflow:epoch = 55.836787564766844, learning_rate = 0.00049999997, loss = 2.2888544e-05, step = 21553 (5.210 sec) 2021-10-19 08:05:44,402 [INFO] tensorflow: epoch = 55.836787564766844, learning_rate = 0.00049999997, loss = 2.2888544e-05, step = 21553 (5.210 sec) 2021-10-19 08:05:48,516 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.321 INFO:tensorflow:epoch = 55.90673575129534, learning_rate = 0.00049999997, loss = 5.369031e-05, step = 21580 (5.298 sec) 2021-10-19 08:05:49,700 [INFO] tensorflow: epoch = 55.90673575129534, learning_rate = 0.00049999997, loss = 5.369031e-05, step = 21580 (5.298 sec) INFO:tensorflow:global_step/sec: 5.04345 2021-10-19 08:05:50,488 [INFO] tensorflow: global_step/sec: 5.04345 2021-10-19 08:05:53,516 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.004 INFO:tensorflow:epoch = 55.97409326424871, learning_rate = 0.00049999997, loss = 2.9009641e-05, step = 21606 (5.211 sec) 2021-10-19 08:05:54,910 [INFO] tensorflow: epoch = 55.97409326424871, learning_rate = 0.00049999997, loss = 2.9009641e-05, step = 21606 (5.211 sec) 2021-10-19 08:05:56,893 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 56/120: loss: 0.00001 learning rate: 0.00050 Time taken: 0:01:16.752998 ETA: 1:21:52.191895 INFO:tensorflow:global_step/sec: 4.97739 2021-10-19 08:05:58,122 [INFO] tensorflow: global_step/sec: 4.97739 2021-10-19 08:05:58,550 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.866 INFO:tensorflow:epoch = 56.04145077720207, learning_rate = 0.00049999997, loss = 2.6243244e-05, step = 21632 (5.251 sec) 2021-10-19 08:06:00,162 [INFO] tensorflow: epoch = 56.04145077720207, learning_rate = 0.00049999997, loss = 2.6243244e-05, step = 21632 (5.251 sec) 2021-10-19 08:06:03,550 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.001 INFO:tensorflow:epoch = 56.11139896373057, learning_rate = 0.00049999997, loss = 1.2419014e-05, step = 21659 (5.338 sec) 2021-10-19 08:06:05,500 [INFO] tensorflow: epoch = 56.11139896373057, learning_rate = 0.00049999997, loss = 1.2419014e-05, step = 21659 (5.338 sec) INFO:tensorflow:global_step/sec: 5.00936 2021-10-19 08:06:05,708 [INFO] tensorflow: global_step/sec: 5.00936 2021-10-19 08:06:08,557 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.976 INFO:tensorflow:epoch = 56.18134715025907, learning_rate = 0.00049999997, loss = 3.9306644e-05, step = 21686 (5.377 sec) 2021-10-19 08:06:10,877 [INFO] tensorflow: epoch = 56.18134715025907, learning_rate = 0.00049999997, loss = 3.9306644e-05, step = 21686 (5.377 sec) INFO:tensorflow:global_step/sec: 5.01824 2021-10-19 08:06:13,281 [INFO] tensorflow: global_step/sec: 5.01824 2021-10-19 08:06:13,495 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.251 INFO:tensorflow:epoch = 56.251295336787564, learning_rate = 0.00049999997, loss = 4.7493435e-05, step = 21713 (5.421 sec) 2021-10-19 08:06:16,299 [INFO] tensorflow: epoch = 56.251295336787564, learning_rate = 0.00049999997, loss = 4.7493435e-05, step = 21713 (5.421 sec) 2021-10-19 08:06:18,478 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.069 INFO:tensorflow:global_step/sec: 5.00961 2021-10-19 08:06:20,866 [INFO] tensorflow: global_step/sec: 5.00961 INFO:tensorflow:epoch = 56.32124352331606, learning_rate = 0.00049999997, loss = 4.0101095e-05, step = 21740 (5.349 sec) 2021-10-19 08:06:21,647 [INFO] tensorflow: epoch = 56.32124352331606, learning_rate = 0.00049999997, loss = 4.0101095e-05, step = 21740 (5.349 sec) 2021-10-19 08:06:23,439 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.159 INFO:tensorflow:epoch = 56.39119170984456, learning_rate = 0.00049999997, loss = 2.2931275e-05, step = 21767 (5.344 sec) 2021-10-19 08:06:26,991 [INFO] tensorflow: epoch = 56.39119170984456, learning_rate = 0.00049999997, loss = 2.2931275e-05, step = 21767 (5.344 sec) INFO:tensorflow:global_step/sec: 5.01104 2021-10-19 08:06:28,449 [INFO] tensorflow: global_step/sec: 5.01104 2021-10-19 08:06:28,451 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.955 INFO:tensorflow:epoch = 56.45854922279793, learning_rate = 0.00049999997, loss = 1.4841315e-05, step = 21793 (5.239 sec) 2021-10-19 08:06:32,230 [INFO] tensorflow: epoch = 56.45854922279793, learning_rate = 0.00049999997, loss = 1.4841315e-05, step = 21793 (5.239 sec) 2021-10-19 08:06:33,403 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.195 INFO:tensorflow:global_step/sec: 4.98409 2021-10-19 08:06:36,074 [INFO] tensorflow: global_step/sec: 4.98409 INFO:tensorflow:epoch = 56.5259067357513, learning_rate = 0.00049999997, loss = 2.6811496e-05, step = 21819 (5.268 sec) 2021-10-19 08:06:37,498 [INFO] tensorflow: epoch = 56.5259067357513, learning_rate = 0.00049999997, loss = 2.6811496e-05, step = 21819 (5.268 sec) 2021-10-19 08:06:38,515 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.561 INFO:tensorflow:epoch = 56.59326424870466, learning_rate = 0.00049999997, loss = 1.7694581e-05, step = 21845 (5.199 sec) 2021-10-19 08:06:42,697 [INFO] tensorflow: epoch = 56.59326424870466, learning_rate = 0.00049999997, loss = 1.7694581e-05, step = 21845 (5.199 sec) 2021-10-19 08:06:43,487 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.115 INFO:tensorflow:global_step/sec: 4.99018 2021-10-19 08:06:43,689 [INFO] tensorflow: global_step/sec: 4.99018 INFO:tensorflow:epoch = 56.66321243523316, learning_rate = 0.00049999997, loss = 4.6643952e-05, step = 21872 (5.366 sec) 2021-10-19 08:06:48,064 [INFO] tensorflow: epoch = 56.66321243523316, learning_rate = 0.00049999997, loss = 4.6643952e-05, step = 21872 (5.366 sec) 2021-10-19 08:06:48,491 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.986 INFO:tensorflow:global_step/sec: 4.97168 2021-10-19 08:06:51,332 [INFO] tensorflow: global_step/sec: 4.97168 INFO:tensorflow:epoch = 56.73056994818653, learning_rate = 0.00049999997, loss = 2.849987e-05, step = 21898 (5.278 sec) 2021-10-19 08:06:53,342 [INFO] tensorflow: epoch = 56.73056994818653, learning_rate = 0.00049999997, loss = 2.849987e-05, step = 21898 (5.278 sec) 2021-10-19 08:06:53,549 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.772 INFO:tensorflow:epoch = 56.7979274611399, learning_rate = 0.00049999997, loss = 1.8633615e-05, step = 21924 (5.215 sec) 2021-10-19 08:06:58,557 [INFO] tensorflow: epoch = 56.7979274611399, learning_rate = 0.00049999997, loss = 1.8633615e-05, step = 21924 (5.215 sec) 2021-10-19 08:06:58,558 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.966 INFO:tensorflow:global_step/sec: 4.9882 2021-10-19 08:06:58,950 [INFO] tensorflow: global_step/sec: 4.9882 2021-10-19 08:07:03,448 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.450 INFO:tensorflow:epoch = 56.8678756476684, learning_rate = 0.00049999997, loss = 2.015984e-05, step = 21951 (5.280 sec) 2021-10-19 08:07:03,838 [INFO] tensorflow: epoch = 56.8678756476684, learning_rate = 0.00049999997, loss = 2.015984e-05, step = 21951 (5.280 sec) INFO:tensorflow:global_step/sec: 5.12298 2021-10-19 08:07:06,368 [INFO] tensorflow: global_step/sec: 5.12298 2021-10-19 08:07:08,380 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.276 INFO:tensorflow:epoch = 56.93782383419689, learning_rate = 0.00049999997, loss = 2.3086272e-05, step = 21978 (5.371 sec) 2021-10-19 08:07:09,208 [INFO] tensorflow: epoch = 56.93782383419689, learning_rate = 0.00049999997, loss = 2.3086272e-05, step = 21978 (5.371 sec) 2021-10-19 08:07:13,413 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.873 INFO:tensorflow:global_step/sec: 4.96121 2021-10-19 08:07:14,027 [INFO] tensorflow: global_step/sec: 4.96121 2021-10-19 08:07:14,028 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 57/120: loss: 0.00002 learning rate: 0.00050 Time taken: 0:01:17.131135 ETA: 1:20:59.261519 INFO:tensorflow:epoch = 57.00518134715026, learning_rate = 0.00049999997, loss = 1.1007593e-05, step = 22004 (5.221 sec) 2021-10-19 08:07:14,430 [INFO] tensorflow: epoch = 57.00518134715026, learning_rate = 0.00049999997, loss = 1.1007593e-05, step = 22004 (5.221 sec) 2021-10-19 08:07:18,478 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.743 INFO:tensorflow:epoch = 57.07253886010363, learning_rate = 0.00049999997, loss = 1.908406e-05, step = 22030 (5.271 sec) 2021-10-19 08:07:19,701 [INFO] tensorflow: epoch = 57.07253886010363, learning_rate = 0.00049999997, loss = 1.908406e-05, step = 22030 (5.271 sec) INFO:tensorflow:global_step/sec: 4.97935 2021-10-19 08:07:21,658 [INFO] tensorflow: global_step/sec: 4.97935 2021-10-19 08:07:23,412 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.272 INFO:tensorflow:epoch = 57.14248704663213, learning_rate = 0.00049999997, loss = 3.9555132e-05, step = 22057 (5.259 sec) 2021-10-19 08:07:24,960 [INFO] tensorflow: epoch = 57.14248704663213, learning_rate = 0.00049999997, loss = 3.9555132e-05, step = 22057 (5.259 sec) 2021-10-19 08:07:28,316 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.393 INFO:tensorflow:global_step/sec: 5.10542 2021-10-19 08:07:29,101 [INFO] tensorflow: global_step/sec: 5.10542 INFO:tensorflow:epoch = 57.212435233160626, learning_rate = 0.00049999997, loss = 2.3519482e-05, step = 22084 (5.333 sec) 2021-10-19 08:07:30,293 [INFO] tensorflow: epoch = 57.212435233160626, learning_rate = 0.00049999997, loss = 2.3519482e-05, step = 22084 (5.333 sec) 2021-10-19 08:07:33,224 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.376 INFO:tensorflow:epoch = 57.28238341968912, learning_rate = 0.00049999997, loss = 1.9404813e-05, step = 22111 (5.321 sec) 2021-10-19 08:07:35,614 [INFO] tensorflow: epoch = 57.28238341968912, learning_rate = 0.00049999997, loss = 1.9404813e-05, step = 22111 (5.321 sec) INFO:tensorflow:global_step/sec: 5.06518 2021-10-19 08:07:36,604 [INFO] tensorflow: global_step/sec: 5.06518 2021-10-19 08:07:38,199 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.102 INFO:tensorflow:epoch = 57.35233160621762, learning_rate = 0.00049999997, loss = 3.1287225e-05, step = 22138 (5.356 sec) 2021-10-19 08:07:40,970 [INFO] tensorflow: epoch = 57.35233160621762, learning_rate = 0.00049999997, loss = 3.1287225e-05, step = 22138 (5.356 sec) 2021-10-19 08:07:43,176 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.094 INFO:tensorflow:global_step/sec: 5.01813 2021-10-19 08:07:44,176 [INFO] tensorflow: global_step/sec: 5.01813 INFO:tensorflow:epoch = 57.41968911917099, learning_rate = 0.00049999997, loss = 1.8043158e-05, step = 22164 (5.202 sec) 2021-10-19 08:07:46,172 [INFO] tensorflow: epoch = 57.41968911917099, learning_rate = 0.00049999997, loss = 1.8043158e-05, step = 22164 (5.202 sec) 2021-10-19 08:07:48,187 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.959 INFO:tensorflow:epoch = 57.48963730569948, learning_rate = 0.00049999997, loss = 2.0716678e-05, step = 22191 (5.361 sec) 2021-10-19 08:07:51,534 [INFO] tensorflow: epoch = 57.48963730569948, learning_rate = 0.00049999997, loss = 2.0716678e-05, step = 22191 (5.361 sec) INFO:tensorflow:global_step/sec: 5.02783 2021-10-19 08:07:51,734 [INFO] tensorflow: global_step/sec: 5.02783 2021-10-19 08:07:53,108 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.323 INFO:tensorflow:epoch = 57.55958549222798, learning_rate = 0.00049999997, loss = 1.8179508e-05, step = 22218 (5.337 sec) 2021-10-19 08:07:56,871 [INFO] tensorflow: epoch = 57.55958549222798, learning_rate = 0.00049999997, loss = 1.8179508e-05, step = 22218 (5.337 sec) 2021-10-19 08:07:58,054 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.217 INFO:tensorflow:global_step/sec: 5.05735 2021-10-19 08:07:59,248 [INFO] tensorflow: global_step/sec: 5.05735 INFO:tensorflow:epoch = 57.62953367875648, learning_rate = 0.00049999997, loss = 3.3641023e-05, step = 22245 (5.330 sec) 2021-10-19 08:08:02,201 [INFO] tensorflow: epoch = 57.62953367875648, learning_rate = 0.00049999997, loss = 3.3641023e-05, step = 22245 (5.330 sec) 2021-10-19 08:08:03,034 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.084 INFO:tensorflow:global_step/sec: 5.02263 2021-10-19 08:08:06,814 [INFO] tensorflow: global_step/sec: 5.02263 INFO:tensorflow:epoch = 57.696891191709845, learning_rate = 0.00049999997, loss = 4.667323e-05, step = 22271 (5.215 sec) 2021-10-19 08:08:07,416 [INFO] tensorflow: epoch = 57.696891191709845, learning_rate = 0.00049999997, loss = 4.667323e-05, step = 22271 (5.215 sec) 2021-10-19 08:08:08,035 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.995 INFO:tensorflow:epoch = 57.76424870466322, learning_rate = 0.00049999997, loss = 3.9412786e-05, step = 22297 (5.223 sec) 2021-10-19 08:08:12,639 [INFO] tensorflow: epoch = 57.76424870466322, learning_rate = 0.00049999997, loss = 3.9412786e-05, step = 22297 (5.223 sec) 2021-10-19 08:08:13,045 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.965 INFO:tensorflow:global_step/sec: 4.99716 2021-10-19 08:08:14,418 [INFO] tensorflow: global_step/sec: 4.99716 INFO:tensorflow:epoch = 57.83160621761658, learning_rate = 0.00049999997, loss = 1.9853658e-05, step = 22323 (5.279 sec) 2021-10-19 08:08:17,918 [INFO] tensorflow: epoch = 57.83160621761658, learning_rate = 0.00049999997, loss = 1.9853658e-05, step = 22323 (5.279 sec) 2021-10-19 08:08:18,129 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.671 INFO:tensorflow:global_step/sec: 4.94705 2021-10-19 08:08:22,099 [INFO] tensorflow: global_step/sec: 4.94705 2021-10-19 08:08:23,104 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.102 INFO:tensorflow:epoch = 57.90155440414508, learning_rate = 0.00049999997, loss = 3.1430576e-05, step = 22350 (5.382 sec) 2021-10-19 08:08:23,300 [INFO] tensorflow: epoch = 57.90155440414508, learning_rate = 0.00049999997, loss = 3.1430576e-05, step = 22350 (5.382 sec) 2021-10-19 08:08:28,106 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.993 INFO:tensorflow:epoch = 57.97150259067358, learning_rate = 0.00049999997, loss = 1.7386068e-05, step = 22377 (5.428 sec) 2021-10-19 08:08:28,728 [INFO] tensorflow: epoch = 57.97150259067358, learning_rate = 0.00049999997, loss = 1.7386068e-05, step = 22377 (5.428 sec) INFO:tensorflow:global_step/sec: 4.97435 2021-10-19 08:08:29,738 [INFO] tensorflow: global_step/sec: 4.97435 2021-10-19 08:08:30,945 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 58/120: loss: 0.00003 learning rate: 0.00050 Time taken: 0:01:16.913635 ETA: 1:19:28.645356 2021-10-19 08:08:33,095 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.047 INFO:tensorflow:epoch = 58.04145077720207, learning_rate = 0.00049999997, loss = 1.28853e-05, step = 22404 (5.356 sec) 2021-10-19 08:08:34,084 [INFO] tensorflow: epoch = 58.04145077720207, learning_rate = 0.00049999997, loss = 1.28853e-05, step = 22404 (5.356 sec) INFO:tensorflow:global_step/sec: 5.0482 2021-10-19 08:08:37,266 [INFO] tensorflow: global_step/sec: 5.0482 2021-10-19 08:08:38,071 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.099 INFO:tensorflow:epoch = 58.11139896373057, learning_rate = 0.00049999997, loss = 3.4125096e-05, step = 22431 (5.372 sec) 2021-10-19 08:08:39,456 [INFO] tensorflow: epoch = 58.11139896373057, learning_rate = 0.00049999997, loss = 3.4125096e-05, step = 22431 (5.372 sec) 2021-10-19 08:08:43,101 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.880 INFO:tensorflow:epoch = 58.17875647668394, learning_rate = 0.00049999997, loss = 1.514543e-05, step = 22457 (5.232 sec) 2021-10-19 08:08:44,688 [INFO] tensorflow: epoch = 58.17875647668394, learning_rate = 0.00049999997, loss = 1.514543e-05, step = 22457 (5.232 sec) INFO:tensorflow:global_step/sec: 4.98256 2021-10-19 08:08:44,893 [INFO] tensorflow: global_step/sec: 4.98256 2021-10-19 08:08:48,071 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.123 INFO:tensorflow:epoch = 58.248704663212436, learning_rate = 0.00049999997, loss = 1.6238433e-05, step = 22484 (5.367 sec) 2021-10-19 08:08:50,056 [INFO] tensorflow: epoch = 58.248704663212436, learning_rate = 0.00049999997, loss = 1.6238433e-05, step = 22484 (5.367 sec) INFO:tensorflow:global_step/sec: 5.02601 2021-10-19 08:08:52,453 [INFO] tensorflow: global_step/sec: 5.02601 2021-10-19 08:08:53,042 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.119 INFO:tensorflow:epoch = 58.31606217616581, learning_rate = 0.00049999997, loss = 3.2965443e-05, step = 22510 (5.215 sec) 2021-10-19 08:08:55,270 [INFO] tensorflow: epoch = 58.31606217616581, learning_rate = 0.00049999997, loss = 3.2965443e-05, step = 22510 (5.215 sec) 2021-10-19 08:08:58,073 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.878 INFO:tensorflow:global_step/sec: 4.99814 2021-10-19 08:09:00,056 [INFO] tensorflow: global_step/sec: 4.99814 INFO:tensorflow:epoch = 58.3860103626943, learning_rate = 0.00049999997, loss = 1.719301e-05, step = 22537 (5.372 sec) 2021-10-19 08:09:00,642 [INFO] tensorflow: epoch = 58.3860103626943, learning_rate = 0.00049999997, loss = 1.719301e-05, step = 22537 (5.372 sec) 2021-10-19 08:09:03,039 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.139 INFO:tensorflow:epoch = 58.45336787564767, learning_rate = 0.00049999997, loss = 1.662552e-05, step = 22563 (5.195 sec) 2021-10-19 08:09:05,837 [INFO] tensorflow: epoch = 58.45336787564767, learning_rate = 0.00049999997, loss = 1.662552e-05, step = 22563 (5.195 sec) INFO:tensorflow:global_step/sec: 5.02004 2021-10-19 08:09:07,626 [INFO] tensorflow: global_step/sec: 5.02004 2021-10-19 08:09:08,018 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.086 INFO:tensorflow:epoch = 58.520725388601036, learning_rate = 0.00049999997, loss = 3.2225427e-05, step = 22589 (5.206 sec) 2021-10-19 08:09:11,043 [INFO] tensorflow: epoch = 58.520725388601036, learning_rate = 0.00049999997, loss = 3.2225427e-05, step = 22589 (5.206 sec) 2021-10-19 08:09:13,026 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.968 INFO:tensorflow:global_step/sec: 5.01972 2021-10-19 08:09:15,196 [INFO] tensorflow: global_step/sec: 5.01972 INFO:tensorflow:epoch = 58.590673575129536, learning_rate = 0.00049999997, loss = 2.8247348e-05, step = 22616 (5.382 sec) 2021-10-19 08:09:16,424 [INFO] tensorflow: epoch = 58.590673575129536, learning_rate = 0.00049999997, loss = 2.8247348e-05, step = 22616 (5.382 sec) 2021-10-19 08:09:18,043 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.934 INFO:tensorflow:epoch = 58.660621761658035, learning_rate = 0.00049999997, loss = 2.7456903e-05, step = 22643 (5.355 sec) 2021-10-19 08:09:21,780 [INFO] tensorflow: epoch = 58.660621761658035, learning_rate = 0.00049999997, loss = 2.7456903e-05, step = 22643 (5.355 sec) INFO:tensorflow:global_step/sec: 5.01056 2021-10-19 08:09:22,780 [INFO] tensorflow: global_step/sec: 5.01056 2021-10-19 08:09:22,977 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.269 INFO:tensorflow:epoch = 58.73056994818653, learning_rate = 0.00049999997, loss = 1.6109418e-05, step = 22670 (5.341 sec) 2021-10-19 08:09:27,121 [INFO] tensorflow: epoch = 58.73056994818653, learning_rate = 0.00049999997, loss = 1.6109418e-05, step = 22670 (5.341 sec) 2021-10-19 08:09:27,921 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.230 INFO:tensorflow:global_step/sec: 5.01182 2021-10-19 08:09:30,362 [INFO] tensorflow: global_step/sec: 5.01182 INFO:tensorflow:epoch = 58.7979274611399, learning_rate = 0.00049999997, loss = 3.0784027e-05, step = 22696 (5.272 sec) 2021-10-19 08:09:32,393 [INFO] tensorflow: epoch = 58.7979274611399, learning_rate = 0.00049999997, loss = 3.0784027e-05, step = 22696 (5.272 sec) 2021-10-19 08:09:33,017 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.623 INFO:tensorflow:epoch = 58.865284974093264, learning_rate = 0.00049999997, loss = 1.9703388e-05, step = 22722 (5.315 sec) 2021-10-19 08:09:37,708 [INFO] tensorflow: epoch = 58.865284974093264, learning_rate = 0.00049999997, loss = 1.9703388e-05, step = 22722 (5.315 sec) INFO:tensorflow:global_step/sec: 4.91145 2021-10-19 08:09:38,099 [INFO] tensorflow: global_step/sec: 4.91145 2021-10-19 08:09:38,101 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.674 INFO:tensorflow:epoch = 58.93523316062176, learning_rate = 0.00049999997, loss = 2.8427196e-05, step = 22749 (5.374 sec) 2021-10-19 08:09:43,082 [INFO] tensorflow: epoch = 58.93523316062176, learning_rate = 0.00049999997, loss = 2.8427196e-05, step = 22749 (5.374 sec) 2021-10-19 08:09:43,083 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.074 INFO:tensorflow:global_step/sec: 5.03068 2021-10-19 08:09:45,653 [INFO] tensorflow: global_step/sec: 5.03068 2021-10-19 08:09:48,094 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 59/120: loss: 0.00003 learning rate: 0.00050 Time taken: 0:01:17.148816 ETA: 1:18:26.077768 2021-10-19 08:09:48,094 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.955 INFO:tensorflow:epoch = 59.002590673575135, learning_rate = 0.00049999997, loss = 3.4809156e-05, step = 22775 (5.226 sec) 2021-10-19 08:09:48,309 [INFO] tensorflow: epoch = 59.002590673575135, learning_rate = 0.00049999997, loss = 3.4809156e-05, step = 22775 (5.226 sec) 2021-10-19 08:09:53,057 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.151 INFO:tensorflow:global_step/sec: 4.99864 2021-10-19 08:09:53,255 [INFO] tensorflow: global_step/sec: 4.99864 INFO:tensorflow:epoch = 59.07253886010363, learning_rate = 0.00049999997, loss = 3.143553e-05, step = 22802 (5.347 sec) 2021-10-19 08:09:53,656 [INFO] tensorflow: epoch = 59.07253886010363, learning_rate = 0.00049999997, loss = 3.143553e-05, step = 22802 (5.347 sec) 2021-10-19 08:09:58,035 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.089 INFO:tensorflow:epoch = 59.14248704663213, learning_rate = 0.00049999997, loss = 1.9904533e-05, step = 22829 (5.400 sec) 2021-10-19 08:09:59,056 [INFO] tensorflow: epoch = 59.14248704663213, learning_rate = 0.00049999997, loss = 1.9904533e-05, step = 22829 (5.400 sec) INFO:tensorflow:global_step/sec: 5.01109 2021-10-19 08:10:00,838 [INFO] tensorflow: global_step/sec: 5.01109 2021-10-19 08:10:03,052 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.936 INFO:tensorflow:epoch = 59.212435233160626, learning_rate = 0.00049999997, loss = 1.7142847e-05, step = 22856 (5.377 sec) 2021-10-19 08:10:04,432 [INFO] tensorflow: epoch = 59.212435233160626, learning_rate = 0.00049999997, loss = 1.7142847e-05, step = 22856 (5.377 sec) 2021-10-19 08:10:07,987 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.267 INFO:tensorflow:global_step/sec: 5.02696 2021-10-19 08:10:08,397 [INFO] tensorflow: global_step/sec: 5.02696 INFO:tensorflow:epoch = 59.28238341968912, learning_rate = 0.00049999997, loss = 2.0935335e-05, step = 22883 (5.336 sec) 2021-10-19 08:10:09,768 [INFO] tensorflow: epoch = 59.28238341968912, learning_rate = 0.00049999997, loss = 2.0935335e-05, step = 22883 (5.336 sec) 2021-10-19 08:10:13,012 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.901 INFO:tensorflow:epoch = 59.34974093264249, learning_rate = 0.00049999997, loss = 1.631817e-05, step = 22909 (5.300 sec) 2021-10-19 08:10:15,068 [INFO] tensorflow: epoch = 59.34974093264249, learning_rate = 0.00049999997, loss = 1.631817e-05, step = 22909 (5.300 sec) INFO:tensorflow:global_step/sec: 4.95947 2021-10-19 08:10:16,059 [INFO] tensorflow: global_step/sec: 4.95947 2021-10-19 08:10:18,033 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.917 INFO:tensorflow:epoch = 59.41968911917099, learning_rate = 0.00049999997, loss = 1.9903391e-05, step = 22936 (5.388 sec) 2021-10-19 08:10:20,456 [INFO] tensorflow: epoch = 59.41968911917099, learning_rate = 0.00049999997, loss = 1.9903391e-05, step = 22936 (5.388 sec) 2021-10-19 08:10:23,088 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.785 INFO:tensorflow:global_step/sec: 4.97928 2021-10-19 08:10:23,691 [INFO] tensorflow: global_step/sec: 4.97928 INFO:tensorflow:epoch = 59.487046632124354, learning_rate = 0.00049999997, loss = 2.6244783e-05, step = 22962 (5.229 sec) 2021-10-19 08:10:25,684 [INFO] tensorflow: epoch = 59.487046632124354, learning_rate = 0.00049999997, loss = 2.6244783e-05, step = 22962 (5.229 sec) 2021-10-19 08:10:28,019 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.283 INFO:tensorflow:epoch = 59.55699481865285, learning_rate = 0.00049999997, loss = 1.5193328e-05, step = 22989 (5.399 sec) 2021-10-19 08:10:31,083 [INFO] tensorflow: epoch = 59.55699481865285, learning_rate = 0.00049999997, loss = 1.5193328e-05, step = 22989 (5.399 sec) INFO:tensorflow:global_step/sec: 5.00266 2021-10-19 08:10:31,287 [INFO] tensorflow: global_step/sec: 5.00266 2021-10-19 08:10:33,040 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.917 INFO:tensorflow:epoch = 59.626943005181346, learning_rate = 0.00049999997, loss = 2.2039709e-05, step = 23016 (5.324 sec) 2021-10-19 08:10:36,407 [INFO] tensorflow: epoch = 59.626943005181346, learning_rate = 0.00049999997, loss = 2.2039709e-05, step = 23016 (5.324 sec) 2021-10-19 08:10:37,973 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.272 INFO:tensorflow:global_step/sec: 5.09211 2021-10-19 08:10:38,749 [INFO] tensorflow: global_step/sec: 5.09211 INFO:tensorflow:epoch = 59.696891191709845, learning_rate = 0.00049999997, loss = 1.895604e-05, step = 23043 (5.316 sec) 2021-10-19 08:10:41,723 [INFO] tensorflow: epoch = 59.696891191709845, learning_rate = 0.00049999997, loss = 1.895604e-05, step = 23043 (5.316 sec) 2021-10-19 08:10:42,904 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.284 INFO:tensorflow:global_step/sec: 5.05088 2021-10-19 08:10:46,273 [INFO] tensorflow: global_step/sec: 5.05088 INFO:tensorflow:epoch = 59.766839378238345, learning_rate = 0.00049999997, loss = 1.7877874e-05, step = 23070 (5.342 sec) 2021-10-19 08:10:47,065 [INFO] tensorflow: epoch = 59.766839378238345, learning_rate = 0.00049999997, loss = 1.7877874e-05, step = 23070 (5.342 sec) 2021-10-19 08:10:47,844 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.245 INFO:tensorflow:epoch = 59.836787564766844, learning_rate = 0.00049999997, loss = 1.7747845e-05, step = 23097 (5.272 sec) 2021-10-19 08:10:52,337 [INFO] tensorflow: epoch = 59.836787564766844, learning_rate = 0.00049999997, loss = 1.7747845e-05, step = 23097 (5.272 sec) 2021-10-19 08:10:52,754 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.368 INFO:tensorflow:global_step/sec: 5.08143 2021-10-19 08:10:53,751 [INFO] tensorflow: global_step/sec: 5.08143 INFO:tensorflow:epoch = 59.90414507772021, learning_rate = 0.00049999997, loss = 4.23598e-05, step = 23123 (5.206 sec) 2021-10-19 08:10:57,542 [INFO] tensorflow: epoch = 59.90414507772021, learning_rate = 0.00049999997, loss = 4.23598e-05, step = 23123 (5.206 sec) 2021-10-19 08:10:57,750 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.017 INFO:tensorflow:global_step/sec: 5.04703 2021-10-19 08:11:01,280 [INFO] tensorflow: global_step/sec: 5.04703 2021-10-19 08:11:02,645 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.432 INFO:tensorflow:epoch = 59.97409326424871, learning_rate = 0.00049999997, loss = 3.2450407e-05, step = 23150 (5.313 sec) 2021-10-19 08:11:02,856 [INFO] tensorflow: epoch = 59.97409326424871, learning_rate = 0.00049999997, loss = 3.2450407e-05, step = 23150 (5.313 sec) INFO:tensorflow:Saving checkpoints for step-23160. 2021-10-19 08:11:04,635 [INFO] tensorflow: Saving checkpoints for step-23160. WARNING:tensorflow:Ignoring: /tmp/tmprkpgq8ua; No such file or directory 2021-10-19 08:11:04,849 [WARNING] tensorflow: Ignoring: /tmp/tmprkpgq8ua; No such file or directory 2021-10-19 08:11:09,672 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 62, 0.00s/step 2021-10-19 08:11:10,777 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 62, 0.11s/step 2021-10-19 08:11:11,857 [INFO] iva.detectnet_v2.evaluation.evaluation: step 20 / 62, 0.11s/step 2021-10-19 08:11:12,981 [INFO] iva.detectnet_v2.evaluation.evaluation: step 30 / 62, 0.11s/step 2021-10-19 08:11:14,033 [INFO] iva.detectnet_v2.evaluation.evaluation: step 40 / 62, 0.11s/step 2021-10-19 08:11:15,095 [INFO] iva.detectnet_v2.evaluation.evaluation: step 50 / 62, 0.11s/step 2021-10-19 08:11:16,146 [INFO] iva.detectnet_v2.evaluation.evaluation: step 60 / 62, 0.11s/step Matching predictions to ground truth, class 1/7.: 100%|█| 2187/2187 [00:00<00:00, 15786.73it/s] Matching predictions to ground truth, class 2/7.: 100%|█| 443/443 [00:00<00:00, 16266.81it/s] Matching predictions to ground truth, class 7/7.: 100%|█| 240/240 [00:00<00:00, 36251.55it/s] Epoch 60/120 ========================= Validation cost: 0.000018 Mean average_precision (in %): 25.4521 class name average precision (in %) ------------- -------------------------- auto_rickshaw 0 bicycle 77.0737 bus 38.3074 car 62.7835 person 0 road_sign 0 truck 0 Median Inference Time: 0.019869 INFO:tensorflow:epoch = 60.0, learning_rate = 0.00049999997, loss = 3.4473043e-05, step = 23160 (13.920 sec) 2021-10-19 08:11:16,776 [INFO] tensorflow: epoch = 60.0, learning_rate = 0.00049999997, loss = 3.4473043e-05, step = 23160 (13.920 sec) 2021-10-19 08:11:16,777 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 60/120: loss: 0.00003 learning rate: 0.00050 Time taken: 0:01:28.673001 ETA: 1:28:40.380063 2021-10-19 08:11:19,547 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 5.917 INFO:tensorflow:global_step/sec: 1.95259 2021-10-19 08:11:20,741 [INFO] tensorflow: global_step/sec: 1.95259 INFO:tensorflow:epoch = 60.0699481865285, learning_rate = 0.00049999997, loss = 1.0846551e-05, step = 23187 (5.336 sec) 2021-10-19 08:11:22,112 [INFO] tensorflow: epoch = 60.0699481865285, learning_rate = 0.00049999997, loss = 1.0846551e-05, step = 23187 (5.336 sec) 2021-10-19 08:11:24,522 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.103 INFO:tensorflow:epoch = 60.137305699481864, learning_rate = 0.00049999997, loss = 2.2530807e-05, step = 23213 (5.285 sec) 2021-10-19 08:11:27,396 [INFO] tensorflow: epoch = 60.137305699481864, learning_rate = 0.00049999997, loss = 2.2530807e-05, step = 23213 (5.285 sec) INFO:tensorflow:global_step/sec: 4.96463 2021-10-19 08:11:28,396 [INFO] tensorflow: global_step/sec: 4.96463 2021-10-19 08:11:29,598 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.704 INFO:tensorflow:epoch = 60.20725388601036, learning_rate = 0.00049999997, loss = 3.5277546e-05, step = 23240 (5.362 sec) 2021-10-19 08:11:32,758 [INFO] tensorflow: epoch = 60.20725388601036, learning_rate = 0.00049999997, loss = 3.5277546e-05, step = 23240 (5.362 sec) 2021-10-19 08:11:34,517 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.329 INFO:tensorflow:global_step/sec: 5.06015 2021-10-19 08:11:35,905 [INFO] tensorflow: global_step/sec: 5.06015 INFO:tensorflow:epoch = 60.27720207253886, learning_rate = 0.00049999997, loss = 3.1098436e-05, step = 23267 (5.292 sec) 2021-10-19 08:11:38,051 [INFO] tensorflow: epoch = 60.27720207253886, learning_rate = 0.00049999997, loss = 3.1098436e-05, step = 23267 (5.292 sec) 2021-10-19 08:11:39,420 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.398 INFO:tensorflow:epoch = 60.34715025906736, learning_rate = 0.00049999997, loss = 4.300939e-05, step = 23294 (5.378 sec) 2021-10-19 08:11:43,428 [INFO] tensorflow: epoch = 60.34715025906736, learning_rate = 0.00049999997, loss = 4.300939e-05, step = 23294 (5.378 sec) INFO:tensorflow:global_step/sec: 5.04993 2021-10-19 08:11:43,430 [INFO] tensorflow: global_step/sec: 5.04993 2021-10-19 08:11:44,427 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.974 INFO:tensorflow:epoch = 60.417098445595855, learning_rate = 0.00049999997, loss = 2.009685e-05, step = 23321 (5.369 sec) 2021-10-19 08:11:48,797 [INFO] tensorflow: epoch = 60.417098445595855, learning_rate = 0.00049999997, loss = 2.009685e-05, step = 23321 (5.369 sec) 2021-10-19 08:11:49,399 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.114 INFO:tensorflow:global_step/sec: 5.01539 2021-10-19 08:11:51,007 [INFO] tensorflow: global_step/sec: 5.01539 INFO:tensorflow:epoch = 60.484455958549226, learning_rate = 0.00049999997, loss = 1.774016e-05, step = 23347 (5.279 sec) 2021-10-19 08:11:54,076 [INFO] tensorflow: epoch = 60.484455958549226, learning_rate = 0.00049999997, loss = 1.774016e-05, step = 23347 (5.279 sec) 2021-10-19 08:11:54,482 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.675 INFO:tensorflow:global_step/sec: 4.93585 2021-10-19 08:11:58,706 [INFO] tensorflow: global_step/sec: 4.93585 INFO:tensorflow:epoch = 60.55181347150259, learning_rate = 0.00049999997, loss = 2.536906e-05, step = 23373 (5.231 sec) 2021-10-19 08:11:59,307 [INFO] tensorflow: epoch = 60.55181347150259, learning_rate = 0.00049999997, loss = 2.536906e-05, step = 23373 (5.231 sec) 2021-10-19 08:11:59,528 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.820 2021-10-19 08:12:04,496 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.131 INFO:tensorflow:epoch = 60.62176165803109, learning_rate = 0.00049999997, loss = 2.3886794e-05, step = 23400 (5.382 sec) 2021-10-19 08:12:04,690 [INFO] tensorflow: epoch = 60.62176165803109, learning_rate = 0.00049999997, loss = 2.3886794e-05, step = 23400 (5.382 sec) INFO:tensorflow:global_step/sec: 5.0074 2021-10-19 08:12:06,294 [INFO] tensorflow: global_step/sec: 5.0074 2021-10-19 08:12:09,509 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.948 INFO:tensorflow:epoch = 60.689119170984455, learning_rate = 0.00049999997, loss = 1.6105758e-05, step = 23426 (5.210 sec) 2021-10-19 08:12:09,900 [INFO] tensorflow: epoch = 60.689119170984455, learning_rate = 0.00049999997, loss = 1.6105758e-05, step = 23426 (5.210 sec) INFO:tensorflow:global_step/sec: 5.01227 2021-10-19 08:12:13,876 [INFO] tensorflow: global_step/sec: 5.01227 2021-10-19 08:12:14,455 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.222 INFO:tensorflow:epoch = 60.759067357512954, learning_rate = 0.00049999997, loss = 2.4383196e-05, step = 23453 (5.337 sec) 2021-10-19 08:12:15,237 [INFO] tensorflow: epoch = 60.759067357512954, learning_rate = 0.00049999997, loss = 2.4383196e-05, step = 23453 (5.337 sec) 2021-10-19 08:12:19,413 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.173 INFO:tensorflow:epoch = 60.829015544041454, learning_rate = 0.00049999997, loss = 3.318529e-05, step = 23480 (5.416 sec) 2021-10-19 08:12:20,653 [INFO] tensorflow: epoch = 60.829015544041454, learning_rate = 0.00049999997, loss = 3.318529e-05, step = 23480 (5.416 sec) INFO:tensorflow:global_step/sec: 4.98834 2021-10-19 08:12:21,494 [INFO] tensorflow: global_step/sec: 4.98834 2021-10-19 08:12:24,524 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.568 INFO:tensorflow:epoch = 60.89637305699482, learning_rate = 0.00049999997, loss = 2.7433929e-05, step = 23506 (5.234 sec) 2021-10-19 08:12:25,888 [INFO] tensorflow: epoch = 60.89637305699482, learning_rate = 0.00049999997, loss = 2.7433929e-05, step = 23506 (5.234 sec) INFO:tensorflow:global_step/sec: 5.0047 2021-10-19 08:12:29,086 [INFO] tensorflow: global_step/sec: 5.0047 2021-10-19 08:12:29,507 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.069 INFO:tensorflow:epoch = 60.96373056994819, learning_rate = 0.00049999997, loss = 3.926634e-05, step = 23532 (5.274 sec) 2021-10-19 08:12:31,162 [INFO] tensorflow: epoch = 60.96373056994819, learning_rate = 0.00049999997, loss = 3.926634e-05, step = 23532 (5.274 sec) 2021-10-19 08:12:33,954 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 61/120: loss: 0.00004 learning rate: 0.00050 Time taken: 0:01:17.172979 ETA: 1:15:53.205754 2021-10-19 08:12:34,558 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.798 INFO:tensorflow:epoch = 61.031088082901555, learning_rate = 0.00049999997, loss = 5.2451043e-05, step = 23558 (5.235 sec) 2021-10-19 08:12:36,396 [INFO] tensorflow: epoch = 61.031088082901555, learning_rate = 0.00049999997, loss = 5.2451043e-05, step = 23558 (5.235 sec) INFO:tensorflow:global_step/sec: 4.91656 2021-10-19 08:12:36,815 [INFO] tensorflow: global_step/sec: 4.91656 2021-10-19 08:12:39,636 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.695 INFO:tensorflow:epoch = 61.098445595854926, learning_rate = 0.00049999997, loss = 3.9884733e-05, step = 23584 (5.228 sec) 2021-10-19 08:12:41,624 [INFO] tensorflow: epoch = 61.098445595854926, learning_rate = 0.00049999997, loss = 3.9884733e-05, step = 23584 (5.228 sec) INFO:tensorflow:global_step/sec: 4.95978 2021-10-19 08:12:44,477 [INFO] tensorflow: global_step/sec: 4.95978 2021-10-19 08:12:44,696 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.764 INFO:tensorflow:epoch = 61.16580310880829, learning_rate = 0.00049999997, loss = 1.2465696e-05, step = 23610 (5.250 sec) 2021-10-19 08:12:46,874 [INFO] tensorflow: epoch = 61.16580310880829, learning_rate = 0.00049999997, loss = 1.2465696e-05, step = 23610 (5.250 sec) 2021-10-19 08:12:49,645 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.208 INFO:tensorflow:global_step/sec: 5.01259 2021-10-19 08:12:52,058 [INFO] tensorflow: global_step/sec: 5.01259 INFO:tensorflow:epoch = 61.23575129533679, learning_rate = 0.00049999997, loss = 2.3353123e-05, step = 23637 (5.387 sec) 2021-10-19 08:12:52,261 [INFO] tensorflow: epoch = 61.23575129533679, learning_rate = 0.00049999997, loss = 2.3353123e-05, step = 23637 (5.387 sec) 2021-10-19 08:12:54,653 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.971 INFO:tensorflow:epoch = 61.30569948186529, learning_rate = 0.00049999997, loss = 1.4863709e-05, step = 23664 (5.404 sec) 2021-10-19 08:12:57,665 [INFO] tensorflow: epoch = 61.30569948186529, learning_rate = 0.00049999997, loss = 1.4863709e-05, step = 23664 (5.404 sec) INFO:tensorflow:global_step/sec: 4.98999 2021-10-19 08:12:59,673 [INFO] tensorflow: global_step/sec: 4.98999 2021-10-19 08:12:59,674 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.917 INFO:tensorflow:epoch = 61.373056994818654, learning_rate = 0.00049999997, loss = 1.9824442e-05, step = 23690 (5.207 sec) 2021-10-19 08:13:02,872 [INFO] tensorflow: epoch = 61.373056994818654, learning_rate = 0.00049999997, loss = 1.9824442e-05, step = 23690 (5.207 sec) 2021-10-19 08:13:04,636 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.154 INFO:tensorflow:global_step/sec: 5.00551 2021-10-19 08:13:07,265 [INFO] tensorflow: global_step/sec: 5.00551 INFO:tensorflow:epoch = 61.443005181347154, learning_rate = 0.00049999997, loss = 2.641489e-05, step = 23717 (5.373 sec) 2021-10-19 08:13:08,245 [INFO] tensorflow: epoch = 61.443005181347154, learning_rate = 0.00049999997, loss = 2.641489e-05, step = 23717 (5.373 sec) 2021-10-19 08:13:09,685 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.811 INFO:tensorflow:epoch = 61.51295336787565, learning_rate = 0.00049999997, loss = 3.8231003e-05, step = 23744 (5.377 sec) 2021-10-19 08:13:13,622 [INFO] tensorflow: epoch = 61.51295336787565, learning_rate = 0.00049999997, loss = 3.8231003e-05, step = 23744 (5.377 sec) 2021-10-19 08:13:14,682 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.013 INFO:tensorflow:global_step/sec: 4.9867 2021-10-19 08:13:14,885 [INFO] tensorflow: global_step/sec: 4.9867 INFO:tensorflow:epoch = 61.58031088082902, learning_rate = 0.00049999997, loss = 2.3007087e-05, step = 23770 (5.221 sec) 2021-10-19 08:13:18,843 [INFO] tensorflow: epoch = 61.58031088082902, learning_rate = 0.00049999997, loss = 2.3007087e-05, step = 23770 (5.221 sec) 2021-10-19 08:13:19,648 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.137 INFO:tensorflow:global_step/sec: 5.07424 2021-10-19 08:13:22,374 [INFO] tensorflow: global_step/sec: 5.07424 INFO:tensorflow:epoch = 61.65025906735752, learning_rate = 0.00049999997, loss = 1.9015126e-05, step = 23797 (5.321 sec) 2021-10-19 08:13:24,164 [INFO] tensorflow: epoch = 61.65025906735752, learning_rate = 0.00049999997, loss = 1.9015126e-05, step = 23797 (5.321 sec) 2021-10-19 08:13:24,593 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.224 INFO:tensorflow:epoch = 61.71761658031088, learning_rate = 0.00049999997, loss = 3.139308e-05, step = 23823 (5.221 sec) 2021-10-19 08:13:29,384 [INFO] tensorflow: epoch = 61.71761658031088, learning_rate = 0.00049999997, loss = 3.139308e-05, step = 23823 (5.221 sec) 2021-10-19 08:13:29,599 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.980 INFO:tensorflow:global_step/sec: 4.98155 2021-10-19 08:13:30,002 [INFO] tensorflow: global_step/sec: 4.98155 2021-10-19 08:13:34,594 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.020 INFO:tensorflow:epoch = 61.78756476683938, learning_rate = 0.00049999997, loss = 3.1020583e-05, step = 23850 (5.435 sec) 2021-10-19 08:13:34,819 [INFO] tensorflow: epoch = 61.78756476683938, learning_rate = 0.00049999997, loss = 3.1020583e-05, step = 23850 (5.435 sec) INFO:tensorflow:global_step/sec: 5.02971 2021-10-19 08:13:37,557 [INFO] tensorflow: global_step/sec: 5.02971 2021-10-19 08:13:39,555 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.160 INFO:tensorflow:epoch = 61.85751295336788, learning_rate = 0.00049999997, loss = 1.2130629e-05, step = 23877 (5.333 sec) 2021-10-19 08:13:40,152 [INFO] tensorflow: epoch = 61.85751295336788, learning_rate = 0.00049999997, loss = 1.2130629e-05, step = 23877 (5.333 sec) 2021-10-19 08:13:44,504 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.209 INFO:tensorflow:global_step/sec: 5.04103 2021-10-19 08:13:45,095 [INFO] tensorflow: global_step/sec: 5.04103 INFO:tensorflow:epoch = 61.92746113989637, learning_rate = 0.00049999997, loss = 1.51364075e-05, step = 23904 (5.336 sec) 2021-10-19 08:13:45,488 [INFO] tensorflow: epoch = 61.92746113989637, learning_rate = 0.00049999997, loss = 1.51364075e-05, step = 23904 (5.336 sec) 2021-10-19 08:13:49,484 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.079 INFO:tensorflow:epoch = 61.99740932642487, learning_rate = 0.00049999997, loss = 1.3792913e-05, step = 23931 (5.364 sec) 2021-10-19 08:13:50,852 [INFO] tensorflow: epoch = 61.99740932642487, learning_rate = 0.00049999997, loss = 1.3792913e-05, step = 23931 (5.364 sec) 2021-10-19 08:13:51,065 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 62/120: loss: 0.00001 learning rate: 0.00050 Time taken: 0:01:17.107276 ETA: 1:14:32.222020 INFO:tensorflow:global_step/sec: 5.06129 2021-10-19 08:13:52,603 [INFO] tensorflow: global_step/sec: 5.06129 2021-10-19 08:13:54,430 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.223 INFO:tensorflow:epoch = 62.064766839378244, learning_rate = 0.00049999997, loss = 1.6244097e-05, step = 23957 (5.193 sec) 2021-10-19 08:13:56,045 [INFO] tensorflow: epoch = 62.064766839378244, learning_rate = 0.00049999997, loss = 1.6244097e-05, step = 23957 (5.193 sec) 2021-10-19 08:13:59,536 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.585 INFO:tensorflow:global_step/sec: 4.9103 2021-10-19 08:14:00,342 [INFO] tensorflow: global_step/sec: 4.9103 INFO:tensorflow:epoch = 62.13212435233161, learning_rate = 0.00049999997, loss = 2.0394795e-05, step = 23983 (5.291 sec) 2021-10-19 08:14:01,336 [INFO] tensorflow: epoch = 62.13212435233161, learning_rate = 0.00049999997, loss = 2.0394795e-05, step = 23983 (5.291 sec) 2021-10-19 08:14:04,483 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.217 INFO:tensorflow:epoch = 62.20207253886011, learning_rate = 0.00049999997, loss = 3.609866e-05, step = 24010 (5.293 sec) 2021-10-19 08:14:06,629 [INFO] tensorflow: epoch = 62.20207253886011, learning_rate = 0.00049999997, loss = 3.609866e-05, step = 24010 (5.293 sec) INFO:tensorflow:global_step/sec: 5.08512 2021-10-19 08:14:07,815 [INFO] tensorflow: global_step/sec: 5.08512 2021-10-19 08:14:09,365 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.486 INFO:tensorflow:epoch = 62.2720207253886, learning_rate = 0.00049999997, loss = 4.4874843e-05, step = 24037 (5.369 sec) 2021-10-19 08:14:11,998 [INFO] tensorflow: epoch = 62.2720207253886, learning_rate = 0.00049999997, loss = 4.4874843e-05, step = 24037 (5.369 sec) 2021-10-19 08:14:14,350 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.060 INFO:tensorflow:global_step/sec: 5.07154 2021-10-19 08:14:15,308 [INFO] tensorflow: global_step/sec: 5.07154 INFO:tensorflow:epoch = 62.3419689119171, learning_rate = 0.00049999997, loss = 4.7968853e-05, step = 24064 (5.267 sec) 2021-10-19 08:14:17,265 [INFO] tensorflow: epoch = 62.3419689119171, learning_rate = 0.00049999997, loss = 4.7968853e-05, step = 24064 (5.267 sec) 2021-10-19 08:14:19,259 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.374 INFO:tensorflow:epoch = 62.40932642487047, learning_rate = 0.00049999997, loss = 1.3840125e-05, step = 24090 (5.268 sec) 2021-10-19 08:14:22,533 [INFO] tensorflow: epoch = 62.40932642487047, learning_rate = 0.00049999997, loss = 1.3840125e-05, step = 24090 (5.268 sec) INFO:tensorflow:global_step/sec: 4.96852 2021-10-19 08:14:22,956 [INFO] tensorflow: global_step/sec: 4.96852 2021-10-19 08:14:24,318 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.771 INFO:tensorflow:epoch = 62.479274611398964, learning_rate = 0.00049999997, loss = 3.6961792e-05, step = 24117 (5.310 sec) 2021-10-19 08:14:27,843 [INFO] tensorflow: epoch = 62.479274611398964, learning_rate = 0.00049999997, loss = 3.6961792e-05, step = 24117 (5.310 sec) 2021-10-19 08:14:29,213 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.428 INFO:tensorflow:global_step/sec: 5.09846 2021-10-19 08:14:30,409 [INFO] tensorflow: global_step/sec: 5.09846 INFO:tensorflow:epoch = 62.54922279792746, learning_rate = 0.00049999997, loss = 2.358701e-05, step = 24144 (5.356 sec) 2021-10-19 08:14:33,199 [INFO] tensorflow: epoch = 62.54922279792746, learning_rate = 0.00049999997, loss = 2.358701e-05, step = 24144 (5.356 sec) 2021-10-19 08:14:34,188 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.102 INFO:tensorflow:global_step/sec: 4.98823 2021-10-19 08:14:38,027 [INFO] tensorflow: global_step/sec: 4.98823 INFO:tensorflow:epoch = 62.616580310880835, learning_rate = 0.00049999997, loss = 2.5185056e-05, step = 24170 (5.230 sec) 2021-10-19 08:14:38,429 [INFO] tensorflow: epoch = 62.616580310880835, learning_rate = 0.00049999997, loss = 2.5185056e-05, step = 24170 (5.230 sec) 2021-10-19 08:14:39,206 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.932 INFO:tensorflow:epoch = 62.6839378238342, learning_rate = 0.00049999997, loss = 2.1441832e-05, step = 24196 (5.196 sec) 2021-10-19 08:14:43,625 [INFO] tensorflow: epoch = 62.6839378238342, learning_rate = 0.00049999997, loss = 2.1441832e-05, step = 24196 (5.196 sec) 2021-10-19 08:14:44,237 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.878 INFO:tensorflow:global_step/sec: 5.00216 2021-10-19 08:14:45,624 [INFO] tensorflow: global_step/sec: 5.00216 INFO:tensorflow:epoch = 62.751295336787564, learning_rate = 0.00049999997, loss = 1.0202172e-05, step = 24222 (5.213 sec) 2021-10-19 08:14:48,838 [INFO] tensorflow: epoch = 62.751295336787564, learning_rate = 0.00049999997, loss = 1.0202172e-05, step = 24222 (5.213 sec) 2021-10-19 08:14:49,231 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.025 INFO:tensorflow:global_step/sec: 4.9543 2021-10-19 08:14:53,294 [INFO] tensorflow: global_step/sec: 4.9543 INFO:tensorflow:epoch = 62.818652849740936, learning_rate = 0.00049999997, loss = 3.7145728e-05, step = 24248 (5.282 sec) 2021-10-19 08:14:54,120 [INFO] tensorflow: epoch = 62.818652849740936, learning_rate = 0.00049999997, loss = 3.7145728e-05, step = 24248 (5.282 sec) 2021-10-19 08:14:54,317 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.665 INFO:tensorflow:epoch = 62.8860103626943, learning_rate = 0.00049999997, loss = 5.2184376e-05, step = 24274 (5.204 sec) 2021-10-19 08:14:59,324 [INFO] tensorflow: epoch = 62.8860103626943, learning_rate = 0.00049999997, loss = 5.2184376e-05, step = 24274 (5.204 sec) 2021-10-19 08:14:59,324 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.973 INFO:tensorflow:global_step/sec: 4.97211 2021-10-19 08:15:00,936 [INFO] tensorflow: global_step/sec: 4.97211 2021-10-19 08:15:04,277 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.193 INFO:tensorflow:epoch = 62.9559585492228, learning_rate = 0.00049999997, loss = 2.9934246e-05, step = 24301 (5.342 sec) 2021-10-19 08:15:04,665 [INFO] tensorflow: epoch = 62.9559585492228, learning_rate = 0.00049999997, loss = 2.9934246e-05, step = 24301 (5.342 sec) 2021-10-19 08:15:08,082 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 63/120: loss: 0.00002 learning rate: 0.00050 Time taken: 0:01:17.010521 ETA: 1:13:09.599720 INFO:tensorflow:global_step/sec: 5.01658 2021-10-19 08:15:08,511 [INFO] tensorflow: global_step/sec: 5.01658 2021-10-19 08:15:09,367 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.646 INFO:tensorflow:epoch = 63.02331606217617, learning_rate = 0.00049999997, loss = 2.1113592e-05, step = 24327 (5.301 sec) 2021-10-19 08:15:09,967 [INFO] tensorflow: epoch = 63.02331606217617, learning_rate = 0.00049999997, loss = 2.1113592e-05, step = 24327 (5.301 sec) 2021-10-19 08:15:14,309 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.238 INFO:tensorflow:epoch = 63.09326424870466, learning_rate = 0.00049999997, loss = 2.581895e-05, step = 24354 (5.322 sec) 2021-10-19 08:15:15,289 [INFO] tensorflow: epoch = 63.09326424870466, learning_rate = 0.00049999997, loss = 2.581895e-05, step = 24354 (5.322 sec) INFO:tensorflow:global_step/sec: 5.01545 2021-10-19 08:15:16,088 [INFO] tensorflow: global_step/sec: 5.01545 2021-10-19 08:15:19,265 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.181 INFO:tensorflow:epoch = 63.16321243523316, learning_rate = 0.00049999997, loss = 3.0380434e-05, step = 24381 (5.356 sec) 2021-10-19 08:15:20,645 [INFO] tensorflow: epoch = 63.16321243523316, learning_rate = 0.00049999997, loss = 3.0380434e-05, step = 24381 (5.356 sec) INFO:tensorflow:global_step/sec: 5.00655 2021-10-19 08:15:23,678 [INFO] tensorflow: global_step/sec: 5.00655 2021-10-19 08:15:24,274 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.964 INFO:tensorflow:epoch = 63.23056994818653, learning_rate = 0.00049999997, loss = 4.0790786e-05, step = 24407 (5.235 sec) 2021-10-19 08:15:25,881 [INFO] tensorflow: epoch = 63.23056994818653, learning_rate = 0.00049999997, loss = 4.0790786e-05, step = 24407 (5.235 sec) 2021-10-19 08:15:29,264 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.041 INFO:tensorflow:epoch = 63.30051813471503, learning_rate = 0.00049999997, loss = 2.3814142e-05, step = 24434 (5.389 sec) 2021-10-19 08:15:31,269 [INFO] tensorflow: epoch = 63.30051813471503, learning_rate = 0.00049999997, loss = 2.3814142e-05, step = 24434 (5.389 sec) INFO:tensorflow:global_step/sec: 5.00436 2021-10-19 08:15:31,271 [INFO] tensorflow: global_step/sec: 5.00436 2021-10-19 08:15:34,222 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.174 INFO:tensorflow:epoch = 63.370466321243526, learning_rate = 0.00049999997, loss = 2.274635e-05, step = 24461 (5.348 sec) 2021-10-19 08:15:36,617 [INFO] tensorflow: epoch = 63.370466321243526, learning_rate = 0.00049999997, loss = 2.274635e-05, step = 24461 (5.348 sec) INFO:tensorflow:global_step/sec: 5.05337 2021-10-19 08:15:38,791 [INFO] tensorflow: global_step/sec: 5.05337 2021-10-19 08:15:39,221 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.004 INFO:tensorflow:epoch = 63.440414507772026, learning_rate = 0.00049999997, loss = 1.917622e-05, step = 24488 (5.391 sec) 2021-10-19 08:15:42,008 [INFO] tensorflow: epoch = 63.440414507772026, learning_rate = 0.00049999997, loss = 1.917622e-05, step = 24488 (5.391 sec) 2021-10-19 08:15:44,179 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.174 INFO:tensorflow:global_step/sec: 4.99973 2021-10-19 08:15:46,391 [INFO] tensorflow: global_step/sec: 4.99973 INFO:tensorflow:epoch = 63.51036269430052, learning_rate = 0.00049999997, loss = 1.1285864e-05, step = 24515 (5.375 sec) 2021-10-19 08:15:47,384 [INFO] tensorflow: epoch = 63.51036269430052, learning_rate = 0.00049999997, loss = 1.1285864e-05, step = 24515 (5.375 sec) 2021-10-19 08:15:49,180 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.996 INFO:tensorflow:epoch = 63.58031088082902, learning_rate = 0.00049999997, loss = 3.797005e-05, step = 24542 (5.341 sec) 2021-10-19 08:15:52,725 [INFO] tensorflow: epoch = 63.58031088082902, learning_rate = 0.00049999997, loss = 3.797005e-05, step = 24542 (5.341 sec) INFO:tensorflow:global_step/sec: 5.04692 2021-10-19 08:15:53,921 [INFO] tensorflow: global_step/sec: 5.04692 2021-10-19 08:15:54,131 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.201 INFO:tensorflow:epoch = 63.65025906735752, learning_rate = 0.00049999997, loss = 1.758139e-05, step = 24569 (5.310 sec) 2021-10-19 08:15:58,035 [INFO] tensorflow: epoch = 63.65025906735752, learning_rate = 0.00049999997, loss = 1.758139e-05, step = 24569 (5.310 sec) 2021-10-19 08:15:59,050 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.332 INFO:tensorflow:global_step/sec: 5.05359 2021-10-19 08:16:01,440 [INFO] tensorflow: global_step/sec: 5.05359 INFO:tensorflow:epoch = 63.71761658031088, learning_rate = 0.00049999997, loss = 1.803254e-05, step = 24595 (5.270 sec) 2021-10-19 08:16:03,305 [INFO] tensorflow: epoch = 63.71761658031088, learning_rate = 0.00049999997, loss = 1.803254e-05, step = 24595 (5.270 sec) 2021-10-19 08:16:04,096 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.818 INFO:tensorflow:epoch = 63.78756476683938, learning_rate = 0.00049999997, loss = 1.7026334e-05, step = 24622 (5.309 sec) 2021-10-19 08:16:08,614 [INFO] tensorflow: epoch = 63.78756476683938, learning_rate = 0.00049999997, loss = 1.7026334e-05, step = 24622 (5.309 sec) INFO:tensorflow:global_step/sec: 5.01961 2021-10-19 08:16:09,011 [INFO] tensorflow: global_step/sec: 5.01961 2021-10-19 08:16:09,012 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.344 INFO:tensorflow:epoch = 63.85751295336788, learning_rate = 0.00049999997, loss = 2.3939514e-05, step = 24649 (5.347 sec) 2021-10-19 08:16:13,961 [INFO] tensorflow: epoch = 63.85751295336788, learning_rate = 0.00049999997, loss = 2.3939514e-05, step = 24649 (5.347 sec) 2021-10-19 08:16:13,962 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.203 INFO:tensorflow:global_step/sec: 5.02072 2021-10-19 08:16:16,579 [INFO] tensorflow: global_step/sec: 5.02072 2021-10-19 08:16:18,916 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.189 INFO:tensorflow:epoch = 63.92746113989637, learning_rate = 0.00049999997, loss = 9.834241e-06, step = 24676 (5.359 sec) 2021-10-19 08:16:19,321 [INFO] tensorflow: epoch = 63.92746113989637, learning_rate = 0.00049999997, loss = 9.834241e-06, step = 24676 (5.359 sec) 2021-10-19 08:16:23,879 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.147 INFO:tensorflow:global_step/sec: 5.06184 2021-10-19 08:16:24,086 [INFO] tensorflow: global_step/sec: 5.06184 INFO:tensorflow:epoch = 63.99740932642487, learning_rate = 0.00049999997, loss = 2.370018e-05, step = 24703 (5.401 sec) 2021-10-19 08:16:24,722 [INFO] tensorflow: epoch = 63.99740932642487, learning_rate = 0.00049999997, loss = 2.370018e-05, step = 24703 (5.401 sec) 2021-10-19 08:16:24,935 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 64/120: loss: 0.00003 learning rate: 0.00050 Time taken: 0:01:16.859167 ETA: 1:11:44.113358 2021-10-19 08:16:28,979 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.610 INFO:tensorflow:epoch = 64.06476683937824, learning_rate = 0.00049999997, loss = 1.9528186e-05, step = 24729 (5.278 sec) 2021-10-19 08:16:29,999 [INFO] tensorflow: epoch = 64.06476683937824, learning_rate = 0.00049999997, loss = 1.9528186e-05, step = 24729 (5.278 sec) INFO:tensorflow:global_step/sec: 4.91264 2021-10-19 08:16:31,821 [INFO] tensorflow: global_step/sec: 4.91264 2021-10-19 08:16:33,982 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.991 INFO:tensorflow:epoch = 64.13471502590674, learning_rate = 0.00049999997, loss = 4.333337e-05, step = 24756 (5.383 sec) 2021-10-19 08:16:35,382 [INFO] tensorflow: epoch = 64.13471502590674, learning_rate = 0.00049999997, loss = 4.333337e-05, step = 24756 (5.383 sec) 2021-10-19 08:16:39,008 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.899 INFO:tensorflow:global_step/sec: 4.99997 2021-10-19 08:16:39,421 [INFO] tensorflow: global_step/sec: 4.99997 INFO:tensorflow:epoch = 64.20207253886011, learning_rate = 0.00049999997, loss = 1.5114689e-05, step = 24782 (5.248 sec) 2021-10-19 08:16:40,630 [INFO] tensorflow: epoch = 64.20207253886011, learning_rate = 0.00049999997, loss = 1.5114689e-05, step = 24782 (5.248 sec) 2021-10-19 08:16:44,072 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.749 INFO:tensorflow:epoch = 64.26943005181347, learning_rate = 0.00049999997, loss = 1.801065e-05, step = 24808 (5.249 sec) 2021-10-19 08:16:45,879 [INFO] tensorflow: epoch = 64.26943005181347, learning_rate = 0.00049999997, loss = 1.801065e-05, step = 24808 (5.249 sec) INFO:tensorflow:global_step/sec: 4.95352 2021-10-19 08:16:47,093 [INFO] tensorflow: global_step/sec: 4.95352 2021-10-19 08:16:49,083 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.956 INFO:tensorflow:epoch = 64.33937823834196, learning_rate = 0.00049999997, loss = 1.7708646e-05, step = 24835 (5.403 sec) 2021-10-19 08:16:51,282 [INFO] tensorflow: epoch = 64.33937823834196, learning_rate = 0.00049999997, loss = 1.7708646e-05, step = 24835 (5.403 sec) 2021-10-19 08:16:54,006 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.318 INFO:tensorflow:global_step/sec: 5.06189 2021-10-19 08:16:54,600 [INFO] tensorflow: global_step/sec: 5.06189 INFO:tensorflow:epoch = 64.40932642487047, learning_rate = 0.00049999997, loss = 3.4335506e-05, step = 24862 (5.280 sec) 2021-10-19 08:16:56,562 [INFO] tensorflow: epoch = 64.40932642487047, learning_rate = 0.00049999997, loss = 3.4335506e-05, step = 24862 (5.280 sec) 2021-10-19 08:16:59,036 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.879 INFO:tensorflow:epoch = 64.47668393782384, learning_rate = 0.00049999997, loss = 1.49745e-05, step = 24888 (5.258 sec) 2021-10-19 08:17:01,820 [INFO] tensorflow: epoch = 64.47668393782384, learning_rate = 0.00049999997, loss = 1.49745e-05, step = 24888 (5.258 sec) INFO:tensorflow:global_step/sec: 4.99221 2021-10-19 08:17:02,212 [INFO] tensorflow: global_step/sec: 4.99221 2021-10-19 08:17:04,080 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.830 INFO:tensorflow:epoch = 64.5440414507772, learning_rate = 0.00049999997, loss = 1.2226223e-05, step = 24914 (5.227 sec) 2021-10-19 08:17:07,047 [INFO] tensorflow: epoch = 64.5440414507772, learning_rate = 0.00049999997, loss = 1.2226223e-05, step = 24914 (5.227 sec) 2021-10-19 08:17:09,073 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.026 INFO:tensorflow:global_step/sec: 4.95425 2021-10-19 08:17:09,882 [INFO] tensorflow: global_step/sec: 4.95425 INFO:tensorflow:epoch = 64.61139896373058, learning_rate = 0.00049999997, loss = 1.6400676e-05, step = 24940 (5.276 sec) 2021-10-19 08:17:12,323 [INFO] tensorflow: epoch = 64.61139896373058, learning_rate = 0.00049999997, loss = 1.6400676e-05, step = 24940 (5.276 sec) 2021-10-19 08:17:14,131 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.774 INFO:tensorflow:global_step/sec: 4.98012 2021-10-19 08:17:17,512 [INFO] tensorflow: global_step/sec: 4.98012 INFO:tensorflow:epoch = 64.68134715025907, learning_rate = 0.00049999997, loss = 3.137968e-05, step = 24967 (5.393 sec) 2021-10-19 08:17:17,716 [INFO] tensorflow: epoch = 64.68134715025907, learning_rate = 0.00049999997, loss = 3.137968e-05, step = 24967 (5.393 sec) 2021-10-19 08:17:19,086 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.184 INFO:tensorflow:epoch = 64.75129533678756, learning_rate = 0.00049999997, loss = 1.795434e-05, step = 24994 (5.336 sec) 2021-10-19 08:17:23,051 [INFO] tensorflow: epoch = 64.75129533678756, learning_rate = 0.00049999997, loss = 1.795434e-05, step = 24994 (5.336 sec) 2021-10-19 08:17:24,019 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.276 INFO:tensorflow:global_step/sec: 5.05893 2021-10-19 08:17:25,024 [INFO] tensorflow: global_step/sec: 5.05893 INFO:tensorflow:epoch = 64.82124352331607, learning_rate = 0.00049999997, loss = 1.638717e-05, step = 25021 (5.344 sec) 2021-10-19 08:17:28,396 [INFO] tensorflow: epoch = 64.82124352331607, learning_rate = 0.00049999997, loss = 1.638717e-05, step = 25021 (5.344 sec) 2021-10-19 08:17:28,993 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.107 INFO:tensorflow:global_step/sec: 5.08704 2021-10-19 08:17:32,494 [INFO] tensorflow: global_step/sec: 5.08704 INFO:tensorflow:epoch = 64.89119170984456, learning_rate = 0.00049999997, loss = 3.656347e-05, step = 25048 (5.276 sec) 2021-10-19 08:17:33,672 [INFO] tensorflow: epoch = 64.89119170984456, learning_rate = 0.00049999997, loss = 3.656347e-05, step = 25048 (5.276 sec) 2021-10-19 08:17:33,892 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.413 2021-10-19 08:17:38,849 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.174 INFO:tensorflow:epoch = 64.96113989637306, learning_rate = 0.00049999997, loss = 2.5881924e-05, step = 25075 (5.371 sec) 2021-10-19 08:17:39,043 [INFO] tensorflow: epoch = 64.96113989637306, learning_rate = 0.00049999997, loss = 2.5881924e-05, step = 25075 (5.371 sec) INFO:tensorflow:global_step/sec: 5.03702 2021-10-19 08:17:40,038 [INFO] tensorflow: global_step/sec: 5.03702 2021-10-19 08:17:42,021 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 65/120: loss: 0.00002 learning rate: 0.00050 Time taken: 0:01:17.093412 ETA: 1:10:40.137682 2021-10-19 08:17:43,784 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.264 INFO:tensorflow:epoch = 65.03108808290156, learning_rate = 0.00049999997, loss = 1.7420021e-05, step = 25102 (5.334 sec) 2021-10-19 08:17:44,377 [INFO] tensorflow: epoch = 65.03108808290156, learning_rate = 0.00049999997, loss = 1.7420021e-05, step = 25102 (5.334 sec) INFO:tensorflow:global_step/sec: 5.03666 2021-10-19 08:17:47,583 [INFO] tensorflow: global_step/sec: 5.03666 2021-10-19 08:17:48,804 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.924 INFO:tensorflow:epoch = 65.09844559585493, learning_rate = 0.00049999997, loss = 2.0556137e-05, step = 25128 (5.223 sec) 2021-10-19 08:17:49,600 [INFO] tensorflow: epoch = 65.09844559585493, learning_rate = 0.00049999997, loss = 2.0556137e-05, step = 25128 (5.223 sec) 2021-10-19 08:17:53,737 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.274 INFO:tensorflow:epoch = 65.16839378238342, learning_rate = 0.00049999997, loss = 2.0337195e-05, step = 25155 (5.306 sec) 2021-10-19 08:17:54,905 [INFO] tensorflow: epoch = 65.16839378238342, learning_rate = 0.00049999997, loss = 2.0337195e-05, step = 25155 (5.306 sec) INFO:tensorflow:global_step/sec: 5.05248 2021-10-19 08:17:55,104 [INFO] tensorflow: global_step/sec: 5.05248 2021-10-19 08:17:58,687 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.203 INFO:tensorflow:epoch = 65.23575129533678, learning_rate = 0.00049999997, loss = 2.3082997e-05, step = 25181 (5.273 sec) 2021-10-19 08:18:00,179 [INFO] tensorflow: epoch = 65.23575129533678, learning_rate = 0.00049999997, loss = 2.3082997e-05, step = 25181 (5.273 sec) INFO:tensorflow:global_step/sec: 4.97306 2021-10-19 08:18:02,745 [INFO] tensorflow: global_step/sec: 4.97306 2021-10-19 08:18:03,732 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.821 INFO:tensorflow:epoch = 65.30569948186529, learning_rate = 0.00049999997, loss = 1.4395789e-05, step = 25208 (5.341 sec) 2021-10-19 08:18:05,519 [INFO] tensorflow: epoch = 65.30569948186529, learning_rate = 0.00049999997, loss = 1.4395789e-05, step = 25208 (5.341 sec) 2021-10-19 08:18:08,748 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.942 INFO:tensorflow:global_step/sec: 5.01709 2021-10-19 08:18:10,319 [INFO] tensorflow: global_step/sec: 5.01709 INFO:tensorflow:epoch = 65.37305699481865, learning_rate = 0.00049999997, loss = 3.4844285e-05, step = 25234 (5.194 sec) 2021-10-19 08:18:10,713 [INFO] tensorflow: epoch = 65.37305699481865, learning_rate = 0.00049999997, loss = 3.4844285e-05, step = 25234 (5.194 sec) 2021-10-19 08:18:13,728 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.082 INFO:tensorflow:epoch = 65.44300518134715, learning_rate = 0.00049999997, loss = 1.4672931e-05, step = 25261 (5.388 sec) 2021-10-19 08:18:16,101 [INFO] tensorflow: epoch = 65.44300518134715, learning_rate = 0.00049999997, loss = 1.4672931e-05, step = 25261 (5.388 sec) INFO:tensorflow:global_step/sec: 5.02115 2021-10-19 08:18:17,887 [INFO] tensorflow: global_step/sec: 5.02115 2021-10-19 08:18:18,664 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.260 INFO:tensorflow:epoch = 65.51295336787565, learning_rate = 0.00049999997, loss = 1.1874944e-05, step = 25288 (5.335 sec) 2021-10-19 08:18:21,436 [INFO] tensorflow: epoch = 65.51295336787565, learning_rate = 0.00049999997, loss = 1.1874944e-05, step = 25288 (5.335 sec) 2021-10-19 08:18:23,660 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.018 INFO:tensorflow:global_step/sec: 5.02632 2021-10-19 08:18:25,447 [INFO] tensorflow: global_step/sec: 5.02632 INFO:tensorflow:epoch = 65.58290155440415, learning_rate = 0.00049999997, loss = 1.1880244e-05, step = 25315 (5.378 sec) 2021-10-19 08:18:26,814 [INFO] tensorflow: epoch = 65.58290155440415, learning_rate = 0.00049999997, loss = 1.1880244e-05, step = 25315 (5.378 sec) 2021-10-19 08:18:28,602 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.240 INFO:tensorflow:epoch = 65.65284974093265, learning_rate = 0.00049999997, loss = 1.5811678e-05, step = 25342 (5.347 sec) 2021-10-19 08:18:32,161 [INFO] tensorflow: epoch = 65.65284974093265, learning_rate = 0.00049999997, loss = 1.5811678e-05, step = 25342 (5.347 sec) INFO:tensorflow:global_step/sec: 5.05347 2021-10-19 08:18:32,967 [INFO] tensorflow: global_step/sec: 5.05347 2021-10-19 08:18:33,550 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.209 INFO:tensorflow:epoch = 65.72279792746114, learning_rate = 0.00049999997, loss = 2.4963283e-05, step = 25369 (5.389 sec) 2021-10-19 08:18:37,550 [INFO] tensorflow: epoch = 65.72279792746114, learning_rate = 0.00049999997, loss = 2.4963283e-05, step = 25369 (5.389 sec) 2021-10-19 08:18:38,547 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.015 INFO:tensorflow:global_step/sec: 5.02458 2021-10-19 08:18:40,529 [INFO] tensorflow: global_step/sec: 5.02458 INFO:tensorflow:epoch = 65.79274611398964, learning_rate = 0.00049999997, loss = 9.687064e-06, step = 25396 (5.378 sec) 2021-10-19 08:18:42,928 [INFO] tensorflow: epoch = 65.79274611398964, learning_rate = 0.00049999997, loss = 9.687064e-06, step = 25396 (5.378 sec) 2021-10-19 08:18:43,570 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.912 INFO:tensorflow:epoch = 65.860103626943, learning_rate = 0.00049999997, loss = 2.9996756e-05, step = 25422 (5.270 sec) 2021-10-19 08:18:48,199 [INFO] tensorflow: epoch = 65.860103626943, learning_rate = 0.00049999997, loss = 2.9996756e-05, step = 25422 (5.270 sec) INFO:tensorflow:global_step/sec: 4.95366 2021-10-19 08:18:48,200 [INFO] tensorflow: global_step/sec: 4.95366 2021-10-19 08:18:48,602 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.873 INFO:tensorflow:epoch = 65.93005181347151, learning_rate = 0.00049999997, loss = 3.4812518e-05, step = 25449 (5.392 sec) 2021-10-19 08:18:53,591 [INFO] tensorflow: epoch = 65.93005181347151, learning_rate = 0.00049999997, loss = 3.4812518e-05, step = 25449 (5.392 sec) 2021-10-19 08:18:53,591 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.044 INFO:tensorflow:global_step/sec: 4.99748 2021-10-19 08:18:55,804 [INFO] tensorflow: global_step/sec: 4.99748 2021-10-19 08:18:58,552 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.161 INFO:tensorflow:epoch = 66.0, learning_rate = 0.00049999997, loss = 3.4248427e-05, step = 25476 (5.371 sec) 2021-10-19 08:18:58,962 [INFO] tensorflow: epoch = 66.0, learning_rate = 0.00049999997, loss = 3.4248427e-05, step = 25476 (5.371 sec) 2021-10-19 08:18:58,963 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 66/120: loss: 0.00003 learning rate: 0.00050 Time taken: 0:01:16.941514 ETA: 1:09:14.841757 INFO:tensorflow:global_step/sec: 5.05359 2021-10-19 08:19:03,324 [INFO] tensorflow: global_step/sec: 5.05359 2021-10-19 08:19:03,521 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.128 INFO:tensorflow:epoch = 66.0699481865285, learning_rate = 0.00049999997, loss = 2.6249898e-05, step = 25503 (5.393 sec) 2021-10-19 08:19:04,355 [INFO] tensorflow: epoch = 66.0699481865285, learning_rate = 0.00049999997, loss = 2.6249898e-05, step = 25503 (5.393 sec) 2021-10-19 08:19:08,550 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.884 INFO:tensorflow:epoch = 66.13730569948187, learning_rate = 0.00049999997, loss = 2.8313225e-05, step = 25529 (5.193 sec) 2021-10-19 08:19:09,548 [INFO] tensorflow: epoch = 66.13730569948187, learning_rate = 0.00049999997, loss = 2.8313225e-05, step = 25529 (5.193 sec) INFO:tensorflow:global_step/sec: 5.00477 2021-10-19 08:19:10,916 [INFO] tensorflow: global_step/sec: 5.00477 2021-10-19 08:19:13,561 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.958 INFO:tensorflow:epoch = 66.20466321243524, learning_rate = 0.00049999997, loss = 1.7801623e-05, step = 25555 (5.221 sec) 2021-10-19 08:19:14,769 [INFO] tensorflow: epoch = 66.20466321243524, learning_rate = 0.00049999997, loss = 1.7801623e-05, step = 25555 (5.221 sec) INFO:tensorflow:global_step/sec: 4.97095 2021-10-19 08:19:18,561 [INFO] tensorflow: global_step/sec: 4.97095 2021-10-19 08:19:18,562 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.996 INFO:tensorflow:epoch = 66.2720207253886, learning_rate = 0.00049999997, loss = 3.3253433e-05, step = 25581 (5.196 sec) 2021-10-19 08:19:19,966 [INFO] tensorflow: epoch = 66.2720207253886, learning_rate = 0.00049999997, loss = 3.3253433e-05, step = 25581 (5.196 sec) 2021-10-19 08:19:23,598 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.862 INFO:tensorflow:epoch = 66.33937823834196, learning_rate = 0.00049999997, loss = 1.6420983e-05, step = 25607 (5.258 sec) 2021-10-19 08:19:25,224 [INFO] tensorflow: epoch = 66.33937823834196, learning_rate = 0.00049999997, loss = 1.6420983e-05, step = 25607 (5.258 sec) INFO:tensorflow:global_step/sec: 4.95595 2021-10-19 08:19:26,228 [INFO] tensorflow: global_step/sec: 4.95595 2021-10-19 08:19:28,644 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.817 INFO:tensorflow:epoch = 66.40673575129534, learning_rate = 0.00049999997, loss = 2.0906811e-05, step = 25633 (5.236 sec) 2021-10-19 08:19:30,459 [INFO] tensorflow: epoch = 66.40673575129534, learning_rate = 0.00049999997, loss = 2.0906811e-05, step = 25633 (5.236 sec) 2021-10-19 08:19:33,604 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.163 INFO:tensorflow:global_step/sec: 5.01634 2021-10-19 08:19:33,804 [INFO] tensorflow: global_step/sec: 5.01634 INFO:tensorflow:epoch = 66.47668393782384, learning_rate = 0.00049999997, loss = 3.4761986e-05, step = 25660 (5.361 sec) 2021-10-19 08:19:35,820 [INFO] tensorflow: epoch = 66.47668393782384, learning_rate = 0.00049999997, loss = 3.4761986e-05, step = 25660 (5.361 sec) 2021-10-19 08:19:38,590 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.058 INFO:tensorflow:epoch = 66.54663212435233, learning_rate = 0.00049999997, loss = 1.222404e-05, step = 25687 (5.327 sec) 2021-10-19 08:19:41,147 [INFO] tensorflow: epoch = 66.54663212435233, learning_rate = 0.00049999997, loss = 1.222404e-05, step = 25687 (5.327 sec) INFO:tensorflow:global_step/sec: 5.03861 2021-10-19 08:19:41,345 [INFO] tensorflow: global_step/sec: 5.03861 2021-10-19 08:19:43,517 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.297 INFO:tensorflow:epoch = 66.61658031088083, learning_rate = 0.00049999997, loss = 1.3479832e-05, step = 25714 (5.338 sec) 2021-10-19 08:19:46,484 [INFO] tensorflow: epoch = 66.61658031088083, learning_rate = 0.00049999997, loss = 1.3479832e-05, step = 25714 (5.338 sec) 2021-10-19 08:19:48,512 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.024 INFO:tensorflow:global_step/sec: 5.02919 2021-10-19 08:19:48,901 [INFO] tensorflow: global_step/sec: 5.02919 INFO:tensorflow:epoch = 66.6839378238342, learning_rate = 0.00049999997, loss = 3.636063e-05, step = 25740 (5.239 sec) 2021-10-19 08:19:51,724 [INFO] tensorflow: epoch = 66.6839378238342, learning_rate = 0.00049999997, loss = 3.636063e-05, step = 25740 (5.239 sec) 2021-10-19 08:19:53,511 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.005 INFO:tensorflow:global_step/sec: 5.00919 2021-10-19 08:19:56,487 [INFO] tensorflow: global_step/sec: 5.00919 INFO:tensorflow:epoch = 66.75388601036269, learning_rate = 0.00049999997, loss = 2.7973703e-05, step = 25767 (5.368 sec) 2021-10-19 08:19:57,092 [INFO] tensorflow: epoch = 66.75388601036269, learning_rate = 0.00049999997, loss = 2.7973703e-05, step = 25767 (5.368 sec) 2021-10-19 08:19:58,515 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.987 INFO:tensorflow:epoch = 66.8238341968912, learning_rate = 0.00049999997, loss = 4.0948085e-05, step = 25794 (5.370 sec) 2021-10-19 08:20:02,462 [INFO] tensorflow: epoch = 66.8238341968912, learning_rate = 0.00049999997, loss = 4.0948085e-05, step = 25794 (5.370 sec) 2021-10-19 08:20:03,492 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.094 INFO:tensorflow:global_step/sec: 4.98286 2021-10-19 08:20:04,114 [INFO] tensorflow: global_step/sec: 4.98286 INFO:tensorflow:epoch = 66.89378238341969, learning_rate = 0.00049999997, loss = 1.7244094e-05, step = 25821 (5.393 sec) 2021-10-19 08:20:07,855 [INFO] tensorflow: epoch = 66.89378238341969, learning_rate = 0.00049999997, loss = 1.7244094e-05, step = 25821 (5.393 sec) 2021-10-19 08:20:08,481 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.044 INFO:tensorflow:global_step/sec: 5.02095 2021-10-19 08:20:11,682 [INFO] tensorflow: global_step/sec: 5.02095 INFO:tensorflow:epoch = 66.96113989637306, learning_rate = 0.00049999997, loss = 3.305534e-05, step = 25847 (5.227 sec) 2021-10-19 08:20:13,082 [INFO] tensorflow: epoch = 66.96113989637306, learning_rate = 0.00049999997, loss = 3.305534e-05, step = 25847 (5.227 sec) 2021-10-19 08:20:13,511 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.886 2021-10-19 08:20:16,101 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 67/120: loss: 0.00003 learning rate: 0.00050 Time taken: 0:01:17.139440 ETA: 1:08:08.390311 INFO:tensorflow:epoch = 67.02849740932643, learning_rate = 0.00049999997, loss = 1.0965777e-05, step = 25873 (5.254 sec) 2021-10-19 08:20:18,336 [INFO] tensorflow: epoch = 67.02849740932643, learning_rate = 0.00049999997, loss = 1.0965777e-05, step = 25873 (5.254 sec) 2021-10-19 08:20:18,549 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.851 INFO:tensorflow:global_step/sec: 4.95004 2021-10-19 08:20:19,358 [INFO] tensorflow: global_step/sec: 4.95004 INFO:tensorflow:epoch = 67.0958549222798, learning_rate = 0.00049999997, loss = 2.6341766e-05, step = 25899 (5.221 sec) 2021-10-19 08:20:23,557 [INFO] tensorflow: epoch = 67.0958549222798, learning_rate = 0.00049999997, loss = 2.6341766e-05, step = 25899 (5.221 sec) 2021-10-19 08:20:23,558 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.966 INFO:tensorflow:global_step/sec: 4.9926 2021-10-19 08:20:26,970 [INFO] tensorflow: global_step/sec: 4.9926 2021-10-19 08:20:28,596 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.850 INFO:tensorflow:epoch = 67.16321243523316, learning_rate = 0.00049999997, loss = 2.7356873e-05, step = 25925 (5.243 sec) 2021-10-19 08:20:28,800 [INFO] tensorflow: epoch = 67.16321243523316, learning_rate = 0.00049999997, loss = 2.7356873e-05, step = 25925 (5.243 sec) 2021-10-19 08:20:33,591 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.020 INFO:tensorflow:epoch = 67.23316062176166, learning_rate = 0.00049999997, loss = 1.9154533e-05, step = 25952 (5.398 sec) 2021-10-19 08:20:34,198 [INFO] tensorflow: epoch = 67.23316062176166, learning_rate = 0.00049999997, loss = 1.9154533e-05, step = 25952 (5.398 sec) INFO:tensorflow:global_step/sec: 4.97651 2021-10-19 08:20:34,606 [INFO] tensorflow: global_step/sec: 4.97651 2021-10-19 08:20:38,559 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.133 INFO:tensorflow:epoch = 67.30310880829016, learning_rate = 0.00049999997, loss = 4.857153e-05, step = 25979 (5.340 sec) 2021-10-19 08:20:39,538 [INFO] tensorflow: epoch = 67.30310880829016, learning_rate = 0.00049999997, loss = 4.857153e-05, step = 25979 (5.340 sec) INFO:tensorflow:global_step/sec: 5.0883 2021-10-19 08:20:42,074 [INFO] tensorflow: global_step/sec: 5.0883 2021-10-19 08:20:43,447 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.459 INFO:tensorflow:epoch = 67.37305699481865, learning_rate = 0.00049999997, loss = 1.8465444e-05, step = 26006 (5.280 sec) 2021-10-19 08:20:44,818 [INFO] tensorflow: epoch = 67.37305699481865, learning_rate = 0.00049999997, loss = 1.8465444e-05, step = 26006 (5.280 sec) 2021-10-19 08:20:48,501 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.791 INFO:tensorflow:global_step/sec: 4.99382 2021-10-19 08:20:49,683 [INFO] tensorflow: global_step/sec: 4.99382 INFO:tensorflow:epoch = 67.44041450777202, learning_rate = 0.00049999997, loss = 2.6001275e-05, step = 26032 (5.280 sec) 2021-10-19 08:20:50,098 [INFO] tensorflow: epoch = 67.44041450777202, learning_rate = 0.00049999997, loss = 2.6001275e-05, step = 26032 (5.280 sec) 2021-10-19 08:20:53,470 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.124 INFO:tensorflow:epoch = 67.51036269430053, learning_rate = 0.00049999997, loss = 1.0492915e-05, step = 26059 (5.366 sec) 2021-10-19 08:20:55,464 [INFO] tensorflow: epoch = 67.51036269430053, learning_rate = 0.00049999997, loss = 1.0492915e-05, step = 26059 (5.366 sec) INFO:tensorflow:global_step/sec: 5.0136 2021-10-19 08:20:57,263 [INFO] tensorflow: global_step/sec: 5.0136 2021-10-19 08:20:58,460 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.044 INFO:tensorflow:epoch = 67.57772020725389, learning_rate = 0.00049999997, loss = 4.1678875e-05, step = 26085 (5.205 sec) 2021-10-19 08:21:00,669 [INFO] tensorflow: epoch = 67.57772020725389, learning_rate = 0.00049999997, loss = 4.1678875e-05, step = 26085 (5.205 sec) 2021-10-19 08:21:03,464 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.987 INFO:tensorflow:global_step/sec: 5.00897 2021-10-19 08:21:04,849 [INFO] tensorflow: global_step/sec: 5.00897 INFO:tensorflow:epoch = 67.64766839378238, learning_rate = 0.00049999997, loss = 1.905064e-05, step = 26112 (5.378 sec) 2021-10-19 08:21:06,048 [INFO] tensorflow: epoch = 67.64766839378238, learning_rate = 0.00049999997, loss = 1.905064e-05, step = 26112 (5.378 sec) 2021-10-19 08:21:08,425 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.156 INFO:tensorflow:epoch = 67.71761658031089, learning_rate = 0.00049999997, loss = 2.0036488e-05, step = 26139 (5.333 sec) 2021-10-19 08:21:11,380 [INFO] tensorflow: epoch = 67.71761658031089, learning_rate = 0.00049999997, loss = 2.0036488e-05, step = 26139 (5.333 sec) INFO:tensorflow:global_step/sec: 5.06495 2021-10-19 08:21:12,351 [INFO] tensorflow: global_step/sec: 5.06495 2021-10-19 08:21:13,331 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.388 INFO:tensorflow:epoch = 67.78756476683938, learning_rate = 0.00049999997, loss = 1.0965783e-05, step = 26166 (5.274 sec) 2021-10-19 08:21:16,655 [INFO] tensorflow: epoch = 67.78756476683938, learning_rate = 0.00049999997, loss = 1.0965783e-05, step = 26166 (5.274 sec) 2021-10-19 08:21:18,219 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.458 INFO:tensorflow:global_step/sec: 5.08916 2021-10-19 08:21:19,818 [INFO] tensorflow: global_step/sec: 5.08916 INFO:tensorflow:epoch = 67.85751295336787, learning_rate = 0.00049999997, loss = 2.219803e-05, step = 26193 (5.336 sec) 2021-10-19 08:21:21,991 [INFO] tensorflow: epoch = 67.85751295336787, learning_rate = 0.00049999997, loss = 2.219803e-05, step = 26193 (5.336 sec) 2021-10-19 08:21:23,178 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.167 INFO:tensorflow:epoch = 67.92746113989638, learning_rate = 0.00049999997, loss = 3.58455e-05, step = 26220 (5.384 sec) 2021-10-19 08:21:27,375 [INFO] tensorflow: epoch = 67.92746113989638, learning_rate = 0.00049999997, loss = 3.58455e-05, step = 26220 (5.384 sec) INFO:tensorflow:global_step/sec: 5.02761 2021-10-19 08:21:27,377 [INFO] tensorflow: global_step/sec: 5.02761 2021-10-19 08:21:28,166 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.050 INFO:tensorflow:epoch = 67.99740932642487, learning_rate = 0.00049999997, loss = 3.8670998e-05, step = 26247 (5.379 sec) 2021-10-19 08:21:32,754 [INFO] tensorflow: epoch = 67.99740932642487, learning_rate = 0.00049999997, loss = 3.8670998e-05, step = 26247 (5.379 sec) 2021-10-19 08:21:32,960 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 68/120: loss: 0.00001 learning rate: 0.00050 Time taken: 0:01:16.850451 ETA: 1:06:36.223476 2021-10-19 08:21:33,164 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.011 INFO:tensorflow:global_step/sec: 5.02714 2021-10-19 08:21:34,936 [INFO] tensorflow: global_step/sec: 5.02714 INFO:tensorflow:epoch = 68.06735751295336, learning_rate = 0.00049999997, loss = 1.2029073e-05, step = 26274 (5.352 sec) 2021-10-19 08:21:38,106 [INFO] tensorflow: epoch = 68.06735751295336, learning_rate = 0.00049999997, loss = 1.2029073e-05, step = 26274 (5.352 sec) 2021-10-19 08:21:38,107 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.234 INFO:tensorflow:global_step/sec: 5.03428 2021-10-19 08:21:42,484 [INFO] tensorflow: global_step/sec: 5.03428 2021-10-19 08:21:43,084 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.094 INFO:tensorflow:epoch = 68.13730569948187, learning_rate = 0.00049999997, loss = 3.2605432e-05, step = 26301 (5.387 sec) 2021-10-19 08:21:43,493 [INFO] tensorflow: epoch = 68.13730569948187, learning_rate = 0.00049999997, loss = 3.2605432e-05, step = 26301 (5.387 sec) 2021-10-19 08:21:48,034 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.203 INFO:tensorflow:epoch = 68.20725388601036, learning_rate = 0.00049999997, loss = 2.2517654e-05, step = 26328 (5.347 sec) 2021-10-19 08:21:48,840 [INFO] tensorflow: epoch = 68.20725388601036, learning_rate = 0.00049999997, loss = 2.2517654e-05, step = 26328 (5.347 sec) INFO:tensorflow:global_step/sec: 5.03532 2021-10-19 08:21:50,031 [INFO] tensorflow: global_step/sec: 5.03532 2021-10-19 08:21:53,088 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.787 INFO:tensorflow:epoch = 68.27461139896373, learning_rate = 0.00049999997, loss = 3.2103104e-05, step = 26354 (5.257 sec) 2021-10-19 08:21:54,097 [INFO] tensorflow: epoch = 68.27461139896373, learning_rate = 0.00049999997, loss = 3.2103104e-05, step = 26354 (5.257 sec) INFO:tensorflow:global_step/sec: 4.95916 2021-10-19 08:21:57,693 [INFO] tensorflow: global_step/sec: 4.95916 2021-10-19 08:21:58,109 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.919 INFO:tensorflow:epoch = 68.34455958549223, learning_rate = 0.00049999997, loss = 2.4625777e-05, step = 26381 (5.399 sec) 2021-10-19 08:21:59,496 [INFO] tensorflow: epoch = 68.34455958549223, learning_rate = 0.00049999997, loss = 2.4625777e-05, step = 26381 (5.399 sec) 2021-10-19 08:22:03,026 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.337 INFO:tensorflow:epoch = 68.41450777202073, learning_rate = 0.00049999997, loss = 3.7548278e-05, step = 26408 (5.297 sec) 2021-10-19 08:22:04,793 [INFO] tensorflow: epoch = 68.41450777202073, learning_rate = 0.00049999997, loss = 3.7548278e-05, step = 26408 (5.297 sec) INFO:tensorflow:global_step/sec: 5.07354 2021-10-19 08:22:05,183 [INFO] tensorflow: global_step/sec: 5.07354 2021-10-19 08:22:07,927 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.408 INFO:tensorflow:epoch = 68.48445595854922, learning_rate = 0.00049999997, loss = 1.534309e-05, step = 26435 (5.342 sec) 2021-10-19 08:22:10,135 [INFO] tensorflow: epoch = 68.48445595854922, learning_rate = 0.00049999997, loss = 1.534309e-05, step = 26435 (5.342 sec) INFO:tensorflow:global_step/sec: 5.01736 2021-10-19 08:22:12,757 [INFO] tensorflow: global_step/sec: 5.01736 2021-10-19 08:22:12,955 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.890 INFO:tensorflow:epoch = 68.55440414507773, learning_rate = 0.00049999997, loss = 1.4011178e-05, step = 26462 (5.407 sec) 2021-10-19 08:22:15,543 [INFO] tensorflow: epoch = 68.55440414507773, learning_rate = 0.00049999997, loss = 1.4011178e-05, step = 26462 (5.407 sec) 2021-10-19 08:22:17,951 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.015 INFO:tensorflow:global_step/sec: 5.02641 2021-10-19 08:22:20,317 [INFO] tensorflow: global_step/sec: 5.02641 INFO:tensorflow:epoch = 68.62176165803109, learning_rate = 0.00049999997, loss = 1.5463755e-05, step = 26488 (5.208 sec) 2021-10-19 08:22:20,751 [INFO] tensorflow: epoch = 68.62176165803109, learning_rate = 0.00049999997, loss = 1.5463755e-05, step = 26488 (5.208 sec) 2021-10-19 08:22:22,890 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.249 INFO:tensorflow:epoch = 68.69170984455958, learning_rate = 0.00049999997, loss = 2.3518736e-05, step = 26515 (5.320 sec) 2021-10-19 08:22:26,070 [INFO] tensorflow: epoch = 68.69170984455958, learning_rate = 0.00049999997, loss = 2.3518736e-05, step = 26515 (5.320 sec) INFO:tensorflow:global_step/sec: 5.04991 2021-10-19 08:22:27,842 [INFO] tensorflow: global_step/sec: 5.04991 2021-10-19 08:22:27,843 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.192 INFO:tensorflow:epoch = 68.76165803108809, learning_rate = 0.00049999997, loss = 1.0997758e-05, step = 26542 (5.323 sec) 2021-10-19 08:22:31,394 [INFO] tensorflow: epoch = 68.76165803108809, learning_rate = 0.00049999997, loss = 1.0997758e-05, step = 26542 (5.323 sec) 2021-10-19 08:22:32,817 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.107 INFO:tensorflow:global_step/sec: 5.05451 2021-10-19 08:22:35,360 [INFO] tensorflow: global_step/sec: 5.05451 INFO:tensorflow:epoch = 68.83160621761658, learning_rate = 0.00049999997, loss = 5.407572e-05, step = 26569 (5.347 sec) 2021-10-19 08:22:36,741 [INFO] tensorflow: epoch = 68.83160621761658, learning_rate = 0.00049999997, loss = 5.407572e-05, step = 26569 (5.347 sec) 2021-10-19 08:22:37,753 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.261 INFO:tensorflow:epoch = 68.90155440414507, learning_rate = 0.00049999997, loss = 1.2523192e-05, step = 26596 (5.338 sec) 2021-10-19 08:22:42,079 [INFO] tensorflow: epoch = 68.90155440414507, learning_rate = 0.00049999997, loss = 1.2523192e-05, step = 26596 (5.338 sec) 2021-10-19 08:22:42,669 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.345 INFO:tensorflow:global_step/sec: 5.05107 2021-10-19 08:22:42,883 [INFO] tensorflow: global_step/sec: 5.05107 INFO:tensorflow:epoch = 68.97150259067358, learning_rate = 0.00049999997, loss = 1.5723106e-05, step = 26623 (5.354 sec) 2021-10-19 08:22:47,433 [INFO] tensorflow: epoch = 68.97150259067358, learning_rate = 0.00049999997, loss = 1.5723106e-05, step = 26623 (5.354 sec) 2021-10-19 08:22:47,631 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.154 2021-10-19 08:22:49,625 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 69/120: loss: 0.00002 learning rate: 0.00050 Time taken: 0:01:16.644696 ETA: 1:05:08.879520 INFO:tensorflow:global_step/sec: 5.04398 2021-10-19 08:22:50,416 [INFO] tensorflow: global_step/sec: 5.04398 2021-10-19 08:22:52,619 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.049 INFO:tensorflow:epoch = 69.04145077720207, learning_rate = 0.00049999997, loss = 2.103419e-05, step = 26650 (5.382 sec) 2021-10-19 08:22:52,815 [INFO] tensorflow: epoch = 69.04145077720207, learning_rate = 0.00049999997, loss = 2.103419e-05, step = 26650 (5.382 sec) 2021-10-19 08:22:57,531 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.360 INFO:tensorflow:global_step/sec: 5.06081 2021-10-19 08:22:57,925 [INFO] tensorflow: global_step/sec: 5.06081 INFO:tensorflow:epoch = 69.11139896373058, learning_rate = 0.00049999997, loss = 1.8144496e-05, step = 26677 (5.313 sec) 2021-10-19 08:22:58,128 [INFO] tensorflow: epoch = 69.11139896373058, learning_rate = 0.00049999997, loss = 1.8144496e-05, step = 26677 (5.313 sec) 2021-10-19 08:23:02,444 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.354 INFO:tensorflow:epoch = 69.18134715025907, learning_rate = 0.00049999997, loss = 3.749507e-05, step = 26704 (5.330 sec) 2021-10-19 08:23:03,458 [INFO] tensorflow: epoch = 69.18134715025907, learning_rate = 0.00049999997, loss = 3.749507e-05, step = 26704 (5.330 sec) INFO:tensorflow:global_step/sec: 5.03049 2021-10-19 08:23:05,479 [INFO] tensorflow: global_step/sec: 5.03049 2021-10-19 08:23:07,532 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.657 INFO:tensorflow:epoch = 69.24870466321244, learning_rate = 0.00049999997, loss = 1.3593795e-05, step = 26730 (5.265 sec) 2021-10-19 08:23:08,723 [INFO] tensorflow: epoch = 69.24870466321244, learning_rate = 0.00049999997, loss = 1.3593795e-05, step = 26730 (5.265 sec) 2021-10-19 08:23:12,473 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.243 INFO:tensorflow:global_step/sec: 5.01421 2021-10-19 08:23:13,058 [INFO] tensorflow: global_step/sec: 5.01421 INFO:tensorflow:epoch = 69.31865284974093, learning_rate = 0.00049999997, loss = 2.951771e-05, step = 26757 (5.331 sec) 2021-10-19 08:23:14,054 [INFO] tensorflow: epoch = 69.31865284974093, learning_rate = 0.00049999997, loss = 2.951771e-05, step = 26757 (5.331 sec) 2021-10-19 08:23:17,391 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.335 INFO:tensorflow:epoch = 69.38860103626943, learning_rate = 0.00049999997, loss = 1.6569858e-05, step = 26784 (5.353 sec) 2021-10-19 08:23:19,407 [INFO] tensorflow: epoch = 69.38860103626943, learning_rate = 0.00049999997, loss = 1.6569858e-05, step = 26784 (5.353 sec) INFO:tensorflow:global_step/sec: 5.03683 2021-10-19 08:23:20,602 [INFO] tensorflow: global_step/sec: 5.03683 2021-10-19 08:23:22,385 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.022 INFO:tensorflow:epoch = 69.45854922279793, learning_rate = 0.00049999997, loss = 2.5337788e-05, step = 26811 (5.377 sec) 2021-10-19 08:23:24,784 [INFO] tensorflow: epoch = 69.45854922279793, learning_rate = 0.00049999997, loss = 2.5337788e-05, step = 26811 (5.377 sec) 2021-10-19 08:23:27,414 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.887 INFO:tensorflow:global_step/sec: 5.00155 2021-10-19 08:23:28,200 [INFO] tensorflow: global_step/sec: 5.00155 INFO:tensorflow:epoch = 69.52590673575129, learning_rate = 0.00049999997, loss = 3.5478108e-05, step = 26837 (5.269 sec) 2021-10-19 08:23:30,053 [INFO] tensorflow: epoch = 69.52590673575129, learning_rate = 0.00049999997, loss = 3.5478108e-05, step = 26837 (5.269 sec) 2021-10-19 08:23:32,447 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.872 INFO:tensorflow:epoch = 69.5958549222798, learning_rate = 0.00049999997, loss = 1.9402183e-05, step = 26864 (5.380 sec) 2021-10-19 08:23:35,433 [INFO] tensorflow: epoch = 69.5958549222798, learning_rate = 0.00049999997, loss = 1.9402183e-05, step = 26864 (5.380 sec) INFO:tensorflow:global_step/sec: 4.95685 2021-10-19 08:23:35,866 [INFO] tensorflow: global_step/sec: 4.95685 2021-10-19 08:23:37,496 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.807 INFO:tensorflow:epoch = 69.66321243523316, learning_rate = 0.00049999997, loss = 1.9648705e-05, step = 26890 (5.204 sec) 2021-10-19 08:23:40,637 [INFO] tensorflow: epoch = 69.66321243523316, learning_rate = 0.00049999997, loss = 1.9648705e-05, step = 26890 (5.204 sec) 2021-10-19 08:23:42,487 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.038 INFO:tensorflow:global_step/sec: 4.98712 2021-10-19 08:23:43,485 [INFO] tensorflow: global_step/sec: 4.98712 INFO:tensorflow:epoch = 69.73056994818653, learning_rate = 0.00049999997, loss = 3.807696e-05, step = 26916 (5.208 sec) 2021-10-19 08:23:45,845 [INFO] tensorflow: epoch = 69.73056994818653, learning_rate = 0.00049999997, loss = 3.807696e-05, step = 26916 (5.208 sec) 2021-10-19 08:23:47,468 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.077 INFO:tensorflow:epoch = 69.79792746113989, learning_rate = 0.00049999997, loss = 1.2865394e-05, step = 26942 (5.234 sec) 2021-10-19 08:23:51,079 [INFO] tensorflow: epoch = 69.79792746113989, learning_rate = 0.00049999997, loss = 1.2865394e-05, step = 26942 (5.234 sec) INFO:tensorflow:global_step/sec: 5.00337 2021-10-19 08:23:51,080 [INFO] tensorflow: global_step/sec: 5.00337 2021-10-19 08:23:52,459 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.038 INFO:tensorflow:epoch = 69.8678756476684, learning_rate = 0.00049999997, loss = 9.476884e-06, step = 26969 (5.399 sec) 2021-10-19 08:23:56,479 [INFO] tensorflow: epoch = 69.8678756476684, learning_rate = 0.00049999997, loss = 9.476884e-06, step = 26969 (5.399 sec) 2021-10-19 08:23:57,498 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.847 INFO:tensorflow:global_step/sec: 4.95994 2021-10-19 08:23:58,742 [INFO] tensorflow: global_step/sec: 4.95994 INFO:tensorflow:epoch = 69.93523316062176, learning_rate = 0.00049999997, loss = 1.2445003e-05, step = 26995 (5.230 sec) 2021-10-19 08:24:01,709 [INFO] tensorflow: epoch = 69.93523316062176, learning_rate = 0.00049999997, loss = 1.2445003e-05, step = 26995 (5.230 sec) 2021-10-19 08:24:02,487 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.045 INFO:tensorflow:global_step/sec: 5.02349 2021-10-19 08:24:06,306 [INFO] tensorflow: global_step/sec: 5.02349 INFO:tensorflow:Saving checkpoints for step-27020. 2021-10-19 08:24:06,538 [INFO] tensorflow: Saving checkpoints for step-27020. WARNING:tensorflow:Ignoring: /tmp/tmp2aaedw94; No such file or directory 2021-10-19 08:24:06,751 [WARNING] tensorflow: Ignoring: /tmp/tmp2aaedw94; No such file or directory 2021-10-19 08:24:11,388 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 62, 0.00s/step 2021-10-19 08:24:12,393 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 62, 0.10s/step 2021-10-19 08:24:13,379 [INFO] iva.detectnet_v2.evaluation.evaluation: step 20 / 62, 0.10s/step 2021-10-19 08:24:14,407 [INFO] iva.detectnet_v2.evaluation.evaluation: step 30 / 62, 0.10s/step 2021-10-19 08:24:15,378 [INFO] iva.detectnet_v2.evaluation.evaluation: step 40 / 62, 0.10s/step 2021-10-19 08:24:16,362 [INFO] iva.detectnet_v2.evaluation.evaluation: step 50 / 62, 0.10s/step 2021-10-19 08:24:17,380 [INFO] iva.detectnet_v2.evaluation.evaluation: step 60 / 62, 0.10s/step Matching predictions to ground truth, class 1/7.: 100%|█| 341/341 [00:00<00:00, 14641.23it/s] Matching predictions to ground truth, class 2/7.: 100%|█| 167/167 [00:00<00:00, 11827.51it/s] Matching predictions to ground truth, class 6/7.: 100%|█| 2/2 [00:00<00:00, 8004.40it/s] Matching predictions to ground truth, class 7/7.: 100%|█| 459/459 [00:00<00:00, 34498.44it/s] Epoch 70/120 ========================= Validation cost: 0.000018 Mean average_precision (in %): 26.2257 class name average precision (in %) ------------- -------------------------- auto_rickshaw 0 bicycle 72.7951 bus 37.2045 car 73.5806 person 0 road_sign 0 truck 0 Median Inference Time: 0.020007 INFO:tensorflow:epoch = 70.0, learning_rate = 0.00049999997, loss = 8.206941e-06, step = 27020 (16.174 sec) 2021-10-19 08:24:17,883 [INFO] tensorflow: epoch = 70.0, learning_rate = 0.00049999997, loss = 8.206941e-06, step = 27020 (16.174 sec) 2021-10-19 08:24:17,884 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 70/120: loss: 0.00001 learning rate: 0.00050 Time taken: 0:01:28.267757 ETA: 1:13:33.387859 2021-10-19 08:24:18,667 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 6.181 INFO:tensorflow:epoch = 70.06735751295336, learning_rate = 0.00049999997, loss = 1.4900759e-05, step = 27046 (5.210 sec) 2021-10-19 08:24:23,093 [INFO] tensorflow: epoch = 70.06735751295336, learning_rate = 0.00049999997, loss = 1.4900759e-05, step = 27046 (5.210 sec) 2021-10-19 08:24:23,695 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.892 INFO:tensorflow:global_step/sec: 2.02097 2021-10-19 08:24:25,109 [INFO] tensorflow: global_step/sec: 2.02097 INFO:tensorflow:epoch = 70.13730569948187, learning_rate = 0.00049999997, loss = 2.1764277e-05, step = 27073 (5.380 sec) 2021-10-19 08:24:28,473 [INFO] tensorflow: epoch = 70.13730569948187, learning_rate = 0.00049999997, loss = 2.1764277e-05, step = 27073 (5.380 sec) 2021-10-19 08:24:28,668 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.108 INFO:tensorflow:global_step/sec: 5.00375 2021-10-19 08:24:32,703 [INFO] tensorflow: global_step/sec: 5.00375 INFO:tensorflow:epoch = 70.20466321243524, learning_rate = 0.00049999997, loss = 2.0041902e-05, step = 27099 (5.245 sec) 2021-10-19 08:24:33,718 [INFO] tensorflow: epoch = 70.20466321243524, learning_rate = 0.00049999997, loss = 2.0041902e-05, step = 27099 (5.245 sec) 2021-10-19 08:24:33,719 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.803 2021-10-19 08:24:38,651 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.277 INFO:tensorflow:epoch = 70.27461139896373, learning_rate = 0.00049999997, loss = 1.9827614e-05, step = 27126 (5.326 sec) 2021-10-19 08:24:39,044 [INFO] tensorflow: epoch = 70.27461139896373, learning_rate = 0.00049999997, loss = 1.9827614e-05, step = 27126 (5.326 sec) INFO:tensorflow:global_step/sec: 5.0339 2021-10-19 08:24:40,252 [INFO] tensorflow: global_step/sec: 5.0339 2021-10-19 08:24:43,592 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.240 INFO:tensorflow:epoch = 70.34455958549223, learning_rate = 0.00049999997, loss = 4.8171743e-05, step = 27153 (5.364 sec) 2021-10-19 08:24:44,408 [INFO] tensorflow: epoch = 70.34455958549223, learning_rate = 0.00049999997, loss = 4.8171743e-05, step = 27153 (5.364 sec) INFO:tensorflow:global_step/sec: 4.98956 2021-10-19 08:24:47,868 [INFO] tensorflow: global_step/sec: 4.98956 2021-10-19 08:24:48,699 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.582 INFO:tensorflow:epoch = 70.4119170984456, learning_rate = 0.00049999997, loss = 1.7389531e-05, step = 27179 (5.262 sec) 2021-10-19 08:24:49,669 [INFO] tensorflow: epoch = 70.4119170984456, learning_rate = 0.00049999997, loss = 1.7389531e-05, step = 27179 (5.262 sec) 2021-10-19 08:24:53,615 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.343 INFO:tensorflow:epoch = 70.48186528497409, learning_rate = 0.00049999997, loss = 1.7978713e-05, step = 27206 (5.388 sec) 2021-10-19 08:24:55,057 [INFO] tensorflow: epoch = 70.48186528497409, learning_rate = 0.00049999997, loss = 1.7978713e-05, step = 27206 (5.388 sec) INFO:tensorflow:global_step/sec: 5.00922 2021-10-19 08:24:55,454 [INFO] tensorflow: global_step/sec: 5.00922 2021-10-19 08:24:58,706 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.643 INFO:tensorflow:epoch = 70.54922279792747, learning_rate = 0.00049999997, loss = 1.5722391e-05, step = 27232 (5.226 sec) 2021-10-19 08:25:00,283 [INFO] tensorflow: epoch = 70.54922279792747, learning_rate = 0.00049999997, loss = 1.5722391e-05, step = 27232 (5.226 sec) INFO:tensorflow:global_step/sec: 4.98669 2021-10-19 08:25:03,074 [INFO] tensorflow: global_step/sec: 4.98669 2021-10-19 08:25:03,680 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.106 INFO:tensorflow:epoch = 70.61917098445596, learning_rate = 0.00049999997, loss = 3.4064175e-05, step = 27259 (5.371 sec) 2021-10-19 08:25:05,654 [INFO] tensorflow: epoch = 70.61917098445596, learning_rate = 0.00049999997, loss = 3.4064175e-05, step = 27259 (5.371 sec) 2021-10-19 08:25:08,655 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.103 INFO:tensorflow:global_step/sec: 5.01598 2021-10-19 08:25:10,650 [INFO] tensorflow: global_step/sec: 5.01598 INFO:tensorflow:epoch = 70.68911917098445, learning_rate = 0.00049999997, loss = 2.6426947e-05, step = 27286 (5.396 sec) 2021-10-19 08:25:11,050 [INFO] tensorflow: epoch = 70.68911917098445, learning_rate = 0.00049999997, loss = 2.6426947e-05, step = 27286 (5.396 sec) 2021-10-19 08:25:13,648 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.028 INFO:tensorflow:epoch = 70.75906735751296, learning_rate = 0.00049999997, loss = 1.3111061e-05, step = 27313 (5.360 sec) 2021-10-19 08:25:16,410 [INFO] tensorflow: epoch = 70.75906735751296, learning_rate = 0.00049999997, loss = 1.3111061e-05, step = 27313 (5.360 sec) INFO:tensorflow:global_step/sec: 5.04977 2021-10-19 08:25:18,175 [INFO] tensorflow: global_step/sec: 5.04977 2021-10-19 08:25:18,559 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.364 INFO:tensorflow:epoch = 70.82901554404145, learning_rate = 0.00049999997, loss = 1.4952385e-05, step = 27340 (5.285 sec) 2021-10-19 08:25:21,696 [INFO] tensorflow: epoch = 70.82901554404145, learning_rate = 0.00049999997, loss = 1.4952385e-05, step = 27340 (5.285 sec) 2021-10-19 08:25:23,474 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.349 INFO:tensorflow:global_step/sec: 5.10482 2021-10-19 08:25:25,619 [INFO] tensorflow: global_step/sec: 5.10482 INFO:tensorflow:epoch = 70.89896373056995, learning_rate = 0.00049999997, loss = 3.1358224e-05, step = 27367 (5.339 sec) 2021-10-19 08:25:27,035 [INFO] tensorflow: epoch = 70.89896373056995, learning_rate = 0.00049999997, loss = 3.1358224e-05, step = 27367 (5.339 sec) 2021-10-19 08:25:28,475 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.998 INFO:tensorflow:epoch = 70.96632124352332, learning_rate = 0.00049999997, loss = 2.2439268e-05, step = 27393 (5.257 sec) 2021-10-19 08:25:32,292 [INFO] tensorflow: epoch = 70.96632124352332, learning_rate = 0.00049999997, loss = 2.2439268e-05, step = 27393 (5.257 sec) INFO:tensorflow:global_step/sec: 4.95393 2021-10-19 08:25:33,290 [INFO] tensorflow: global_step/sec: 4.95393 2021-10-19 08:25:33,494 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.926 2021-10-19 08:25:34,878 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 71/120: loss: 0.00004 learning rate: 0.00050 Time taken: 0:01:16.998226 ETA: 1:02:52.913094 INFO:tensorflow:epoch = 71.03626943005182, learning_rate = 0.00049999997, loss = 2.4099738e-05, step = 27420 (5.382 sec) 2021-10-19 08:25:37,674 [INFO] tensorflow: epoch = 71.03626943005182, learning_rate = 0.00049999997, loss = 2.4099738e-05, step = 27420 (5.382 sec) 2021-10-19 08:25:38,445 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.198 INFO:tensorflow:global_step/sec: 5.06924 2021-10-19 08:25:40,786 [INFO] tensorflow: global_step/sec: 5.06924 INFO:tensorflow:epoch = 71.10621761658031, learning_rate = 0.00049999997, loss = 1.940606e-05, step = 27447 (5.314 sec) 2021-10-19 08:25:42,988 [INFO] tensorflow: epoch = 71.10621761658031, learning_rate = 0.00049999997, loss = 1.940606e-05, step = 27447 (5.314 sec) 2021-10-19 08:25:43,389 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.229 INFO:tensorflow:epoch = 71.17616580310882, learning_rate = 0.00049999997, loss = 1.1641439e-05, step = 27474 (5.360 sec) 2021-10-19 08:25:48,348 [INFO] tensorflow: epoch = 71.17616580310882, learning_rate = 0.00049999997, loss = 1.1641439e-05, step = 27474 (5.360 sec) INFO:tensorflow:global_step/sec: 5.02442 2021-10-19 08:25:48,349 [INFO] tensorflow: global_step/sec: 5.02442 2021-10-19 08:25:48,350 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.157 2021-10-19 08:25:53,293 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.233 INFO:tensorflow:epoch = 71.24611398963731, learning_rate = 0.00049999997, loss = 3.8780054e-05, step = 27501 (5.359 sec) 2021-10-19 08:25:53,707 [INFO] tensorflow: epoch = 71.24611398963731, learning_rate = 0.00049999997, loss = 3.8780054e-05, step = 27501 (5.359 sec) INFO:tensorflow:global_step/sec: 5.05315 2021-10-19 08:25:55,869 [INFO] tensorflow: global_step/sec: 5.05315 2021-10-19 08:25:58,238 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.223 INFO:tensorflow:epoch = 71.3160621761658, learning_rate = 0.00049999997, loss = 3.395632e-05, step = 27528 (5.396 sec) 2021-10-19 08:25:59,102 [INFO] tensorflow: epoch = 71.3160621761658, learning_rate = 0.00049999997, loss = 3.395632e-05, step = 27528 (5.396 sec) 2021-10-19 08:26:03,306 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.737 INFO:tensorflow:global_step/sec: 4.96779 2021-10-19 08:26:03,518 [INFO] tensorflow: global_step/sec: 4.96779 INFO:tensorflow:epoch = 71.38341968911918, learning_rate = 0.00049999997, loss = 2.772416e-05, step = 27554 (5.232 sec) 2021-10-19 08:26:04,334 [INFO] tensorflow: epoch = 71.38341968911918, learning_rate = 0.00049999997, loss = 2.772416e-05, step = 27554 (5.232 sec) 2021-10-19 08:26:08,375 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.730 INFO:tensorflow:epoch = 71.45077720207254, learning_rate = 0.00049999997, loss = 2.8798608e-05, step = 27580 (5.224 sec) 2021-10-19 08:26:09,559 [INFO] tensorflow: epoch = 71.45077720207254, learning_rate = 0.00049999997, loss = 2.8798608e-05, step = 27580 (5.224 sec) INFO:tensorflow:global_step/sec: 5.00546 2021-10-19 08:26:11,110 [INFO] tensorflow: global_step/sec: 5.00546 2021-10-19 08:26:13,306 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.285 INFO:tensorflow:epoch = 71.52072538860104, learning_rate = 0.00049999997, loss = 1.8180403e-05, step = 27607 (5.320 sec) 2021-10-19 08:26:14,879 [INFO] tensorflow: epoch = 71.52072538860104, learning_rate = 0.00049999997, loss = 1.8180403e-05, step = 27607 (5.320 sec) 2021-10-19 08:26:18,217 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.365 INFO:tensorflow:global_step/sec: 5.07154 2021-10-19 08:26:18,603 [INFO] tensorflow: global_step/sec: 5.07154 INFO:tensorflow:epoch = 71.59067357512954, learning_rate = 0.00049999997, loss = 3.352362e-05, step = 27634 (5.292 sec) 2021-10-19 08:26:20,170 [INFO] tensorflow: epoch = 71.59067357512954, learning_rate = 0.00049999997, loss = 3.352362e-05, step = 27634 (5.292 sec) 2021-10-19 08:26:23,155 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.252 INFO:tensorflow:epoch = 71.66062176165804, learning_rate = 0.00049999997, loss = 2.2134263e-05, step = 27661 (5.361 sec) 2021-10-19 08:26:25,532 [INFO] tensorflow: epoch = 71.66062176165804, learning_rate = 0.00049999997, loss = 2.2134263e-05, step = 27661 (5.361 sec) INFO:tensorflow:global_step/sec: 5.05421 2021-10-19 08:26:26,121 [INFO] tensorflow: global_step/sec: 5.05421 2021-10-19 08:26:28,146 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.039 INFO:tensorflow:epoch = 71.7279792746114, learning_rate = 0.00049999997, loss = 4.4034343e-05, step = 27687 (5.214 sec) 2021-10-19 08:26:30,746 [INFO] tensorflow: epoch = 71.7279792746114, learning_rate = 0.00049999997, loss = 4.4034343e-05, step = 27687 (5.214 sec) 2021-10-19 08:26:33,123 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.095 INFO:tensorflow:global_step/sec: 4.97758 2021-10-19 08:26:33,756 [INFO] tensorflow: global_step/sec: 4.97758 INFO:tensorflow:epoch = 71.79533678756476, learning_rate = 0.00049999997, loss = 1.4121381e-05, step = 27713 (5.237 sec) 2021-10-19 08:26:35,982 [INFO] tensorflow: epoch = 71.79533678756476, learning_rate = 0.00049999997, loss = 1.4121381e-05, step = 27713 (5.237 sec) 2021-10-19 08:26:38,232 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.574 INFO:tensorflow:epoch = 71.86269430051814, learning_rate = 0.00049999997, loss = 1.9217154e-05, step = 27739 (5.198 sec) 2021-10-19 08:26:41,180 [INFO] tensorflow: epoch = 71.86269430051814, learning_rate = 0.00049999997, loss = 1.9217154e-05, step = 27739 (5.198 sec) INFO:tensorflow:global_step/sec: 4.97877 2021-10-19 08:26:41,388 [INFO] tensorflow: global_step/sec: 4.97877 2021-10-19 08:26:43,166 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.271 INFO:tensorflow:epoch = 71.93264248704664, learning_rate = 0.00049999997, loss = 2.5906356e-05, step = 27766 (5.362 sec) 2021-10-19 08:26:46,542 [INFO] tensorflow: epoch = 71.93264248704664, learning_rate = 0.00049999997, loss = 2.5906356e-05, step = 27766 (5.362 sec) 2021-10-19 08:26:48,203 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.854 INFO:tensorflow:global_step/sec: 5.00276 2021-10-19 08:26:48,984 [INFO] tensorflow: global_step/sec: 5.00276 INFO:tensorflow:epoch = 72.0, learning_rate = 0.00049999997, loss = 3.522967e-05, step = 27792 (5.223 sec) 2021-10-19 08:26:51,765 [INFO] tensorflow: epoch = 72.0, learning_rate = 0.00049999997, loss = 3.522967e-05, step = 27792 (5.223 sec) 2021-10-19 08:26:51,767 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 72/120: loss: 0.00004 learning rate: 0.00050 Time taken: 0:01:16.885622 ETA: 1:01:30.509846 2021-10-19 08:26:53,202 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.007 INFO:tensorflow:global_step/sec: 5.04033 2021-10-19 08:26:56,523 [INFO] tensorflow: global_step/sec: 5.04033 INFO:tensorflow:epoch = 72.0699481865285, learning_rate = 0.00049999997, loss = 8.815022e-06, step = 27819 (5.390 sec) 2021-10-19 08:26:57,156 [INFO] tensorflow: epoch = 72.0699481865285, learning_rate = 0.00049999997, loss = 8.815022e-06, step = 27819 (5.390 sec) 2021-10-19 08:26:58,145 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.231 INFO:tensorflow:epoch = 72.139896373057, learning_rate = 0.00049999997, loss = 3.5430436e-05, step = 27846 (5.336 sec) 2021-10-19 08:27:02,492 [INFO] tensorflow: epoch = 72.139896373057, learning_rate = 0.00049999997, loss = 3.5430436e-05, step = 27846 (5.336 sec) 2021-10-19 08:27:03,099 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.189 INFO:tensorflow:global_step/sec: 4.97931 2021-10-19 08:27:04,155 [INFO] tensorflow: global_step/sec: 4.97931 INFO:tensorflow:epoch = 72.20725388601036, learning_rate = 0.00049999997, loss = 1.5496e-05, step = 27872 (5.208 sec) 2021-10-19 08:27:07,699 [INFO] tensorflow: epoch = 72.20725388601036, learning_rate = 0.00049999997, loss = 1.5496e-05, step = 27872 (5.208 sec) 2021-10-19 08:27:08,087 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.050 INFO:tensorflow:global_step/sec: 4.97179 2021-10-19 08:27:11,798 [INFO] tensorflow: global_step/sec: 4.97179 INFO:tensorflow:epoch = 72.27461139896373, learning_rate = 0.00049999997, loss = 2.89421e-05, step = 27898 (5.306 sec) 2021-10-19 08:27:13,005 [INFO] tensorflow: epoch = 72.27461139896373, learning_rate = 0.00049999997, loss = 2.89421e-05, step = 27898 (5.306 sec) 2021-10-19 08:27:13,219 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.487 INFO:tensorflow:epoch = 72.3419689119171, learning_rate = 0.00049999997, loss = 1.6391157e-05, step = 27924 (5.213 sec) 2021-10-19 08:27:18,218 [INFO] tensorflow: epoch = 72.3419689119171, learning_rate = 0.00049999997, loss = 1.6391157e-05, step = 27924 (5.213 sec) 2021-10-19 08:27:18,219 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.004 INFO:tensorflow:global_step/sec: 4.98015 2021-10-19 08:27:19,428 [INFO] tensorflow: global_step/sec: 4.98015 2021-10-19 08:27:23,151 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.275 INFO:tensorflow:epoch = 72.4119170984456, learning_rate = 0.00049999997, loss = 1.4340936e-05, step = 27951 (5.357 sec) 2021-10-19 08:27:23,575 [INFO] tensorflow: epoch = 72.4119170984456, learning_rate = 0.00049999997, loss = 1.4340936e-05, step = 27951 (5.357 sec) INFO:tensorflow:global_step/sec: 5.04184 2021-10-19 08:27:26,965 [INFO] tensorflow: global_step/sec: 5.04184 2021-10-19 08:27:28,155 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.986 INFO:tensorflow:epoch = 72.48186528497409, learning_rate = 0.00049999997, loss = 2.525376e-05, step = 27978 (5.350 sec) 2021-10-19 08:27:28,925 [INFO] tensorflow: epoch = 72.48186528497409, learning_rate = 0.00049999997, loss = 2.525376e-05, step = 27978 (5.350 sec) 2021-10-19 08:27:33,104 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.209 INFO:tensorflow:epoch = 72.5518134715026, learning_rate = 0.00049999997, loss = 1.1056531e-05, step = 28005 (5.412 sec) 2021-10-19 08:27:34,337 [INFO] tensorflow: epoch = 72.5518134715026, learning_rate = 0.00049999997, loss = 1.1056531e-05, step = 28005 (5.412 sec) INFO:tensorflow:global_step/sec: 5.00684 2021-10-19 08:27:34,555 [INFO] tensorflow: global_step/sec: 5.00684 2021-10-19 08:27:38,225 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.529 INFO:tensorflow:epoch = 72.61917098445596, learning_rate = 0.00049999997, loss = 2.3380431e-05, step = 28031 (5.295 sec) 2021-10-19 08:27:39,632 [INFO] tensorflow: epoch = 72.61917098445596, learning_rate = 0.00049999997, loss = 2.3380431e-05, step = 28031 (5.295 sec) INFO:tensorflow:global_step/sec: 4.99174 2021-10-19 08:27:42,167 [INFO] tensorflow: global_step/sec: 4.99174 2021-10-19 08:27:43,217 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.034 INFO:tensorflow:epoch = 72.68911917098445, learning_rate = 0.00049999997, loss = 2.8627146e-05, step = 28058 (5.346 sec) 2021-10-19 08:27:44,978 [INFO] tensorflow: epoch = 72.68911917098445, learning_rate = 0.00049999997, loss = 2.8627146e-05, step = 28058 (5.346 sec) 2021-10-19 08:27:48,146 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.290 INFO:tensorflow:global_step/sec: 4.9991 2021-10-19 08:27:49,769 [INFO] tensorflow: global_step/sec: 4.9991 INFO:tensorflow:epoch = 72.75906735751296, learning_rate = 0.00049999997, loss = 1.248625e-05, step = 28085 (5.384 sec) 2021-10-19 08:27:50,363 [INFO] tensorflow: epoch = 72.75906735751296, learning_rate = 0.00049999997, loss = 1.248625e-05, step = 28085 (5.384 sec) 2021-10-19 08:27:53,182 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.857 INFO:tensorflow:epoch = 72.82642487046633, learning_rate = 0.00049999997, loss = 1.797598e-05, step = 28111 (5.233 sec) 2021-10-19 08:27:55,595 [INFO] tensorflow: epoch = 72.82642487046633, learning_rate = 0.00049999997, loss = 1.797598e-05, step = 28111 (5.233 sec) INFO:tensorflow:global_step/sec: 4.96685 2021-10-19 08:27:57,419 [INFO] tensorflow: global_step/sec: 4.96685 2021-10-19 08:27:58,251 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.730 INFO:tensorflow:epoch = 72.89378238341969, learning_rate = 0.00049999997, loss = 2.453571e-05, step = 28137 (5.280 sec) 2021-10-19 08:28:00,875 [INFO] tensorflow: epoch = 72.89378238341969, learning_rate = 0.00049999997, loss = 2.453571e-05, step = 28137 (5.280 sec) 2021-10-19 08:28:03,222 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.120 INFO:tensorflow:global_step/sec: 4.99134 2021-10-19 08:28:05,032 [INFO] tensorflow: global_step/sec: 4.99134 INFO:tensorflow:epoch = 72.96373056994818, learning_rate = 0.00049999997, loss = 6.152405e-05, step = 28164 (5.348 sec) 2021-10-19 08:28:06,224 [INFO] tensorflow: epoch = 72.96373056994818, learning_rate = 0.00049999997, loss = 6.152405e-05, step = 28164 (5.348 sec) 2021-10-19 08:28:08,193 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.120 2021-10-19 08:28:09,053 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 73/120: loss: 0.00002 learning rate: 0.00050 Time taken: 0:01:17.254262 ETA: 1:00:30.950301 INFO:tensorflow:epoch = 73.03108808290156, learning_rate = 0.00049999997, loss = 2.5335255e-05, step = 28190 (5.203 sec) 2021-10-19 08:28:11,426 [INFO] tensorflow: epoch = 73.03108808290156, learning_rate = 0.00049999997, loss = 2.5335255e-05, step = 28190 (5.203 sec) INFO:tensorflow:global_step/sec: 4.97728 2021-10-19 08:28:12,667 [INFO] tensorflow: global_step/sec: 4.97728 2021-10-19 08:28:13,272 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.692 INFO:tensorflow:epoch = 73.10103626943005, learning_rate = 0.00049999997, loss = 1.3448169e-05, step = 28217 (5.436 sec) 2021-10-19 08:28:16,862 [INFO] tensorflow: epoch = 73.10103626943005, learning_rate = 0.00049999997, loss = 1.3448169e-05, step = 28217 (5.436 sec) 2021-10-19 08:28:18,278 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.976 INFO:tensorflow:global_step/sec: 5.01658 2021-10-19 08:28:20,242 [INFO] tensorflow: global_step/sec: 5.01658 INFO:tensorflow:epoch = 73.16839378238342, learning_rate = 0.00049999997, loss = 1.9213108e-05, step = 28243 (5.227 sec) 2021-10-19 08:28:22,089 [INFO] tensorflow: epoch = 73.16839378238342, learning_rate = 0.00049999997, loss = 1.9213108e-05, step = 28243 (5.227 sec) 2021-10-19 08:28:23,289 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.960 INFO:tensorflow:epoch = 73.23834196891193, learning_rate = 0.00049999997, loss = 3.189406e-05, step = 28270 (5.337 sec) 2021-10-19 08:28:27,426 [INFO] tensorflow: epoch = 73.23834196891193, learning_rate = 0.00049999997, loss = 3.189406e-05, step = 28270 (5.337 sec) INFO:tensorflow:global_step/sec: 5.01601 2021-10-19 08:28:27,818 [INFO] tensorflow: global_step/sec: 5.01601 2021-10-19 08:28:28,223 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.269 INFO:tensorflow:epoch = 73.30829015544042, learning_rate = 0.00049999997, loss = 1.1506437e-05, step = 28297 (5.361 sec) 2021-10-19 08:28:32,787 [INFO] tensorflow: epoch = 73.30829015544042, learning_rate = 0.00049999997, loss = 1.1506437e-05, step = 28297 (5.361 sec) 2021-10-19 08:28:33,191 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.130 INFO:tensorflow:global_step/sec: 5.02536 2021-10-19 08:28:35,379 [INFO] tensorflow: global_step/sec: 5.02536 INFO:tensorflow:epoch = 73.37823834196891, learning_rate = 0.00049999997, loss = 1.4965002e-05, step = 28324 (5.377 sec) 2021-10-19 08:28:38,164 [INFO] tensorflow: epoch = 73.37823834196891, learning_rate = 0.00049999997, loss = 1.4965002e-05, step = 28324 (5.377 sec) 2021-10-19 08:28:38,164 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.109 INFO:tensorflow:global_step/sec: 4.97375 2021-10-19 08:28:43,019 [INFO] tensorflow: global_step/sec: 4.97375 2021-10-19 08:28:43,219 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.787 INFO:tensorflow:epoch = 73.44559585492229, learning_rate = 0.00049999997, loss = 1.1164031e-05, step = 28350 (5.256 sec) 2021-10-19 08:28:43,420 [INFO] tensorflow: epoch = 73.44559585492229, learning_rate = 0.00049999997, loss = 1.1164031e-05, step = 28350 (5.256 sec) 2021-10-19 08:28:48,198 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.084 INFO:tensorflow:epoch = 73.51554404145078, learning_rate = 0.00049999997, loss = 2.3851533e-05, step = 28377 (5.355 sec) 2021-10-19 08:28:48,775 [INFO] tensorflow: epoch = 73.51554404145078, learning_rate = 0.00049999997, loss = 2.3851533e-05, step = 28377 (5.355 sec) INFO:tensorflow:global_step/sec: 5.03068 2021-10-19 08:28:50,573 [INFO] tensorflow: global_step/sec: 5.03068 2021-10-19 08:28:53,206 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.973 INFO:tensorflow:epoch = 73.58290155440415, learning_rate = 0.00049999997, loss = 3.332962e-05, step = 28403 (5.230 sec) 2021-10-19 08:28:54,005 [INFO] tensorflow: epoch = 73.58290155440415, learning_rate = 0.00049999997, loss = 3.332962e-05, step = 28403 (5.230 sec) INFO:tensorflow:global_step/sec: 4.98597 2021-10-19 08:28:58,195 [INFO] tensorflow: global_step/sec: 4.98597 2021-10-19 08:28:58,196 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.038 INFO:tensorflow:epoch = 73.65284974093265, learning_rate = 0.00049999997, loss = 2.3482951e-05, step = 28430 (5.373 sec) 2021-10-19 08:28:59,378 [INFO] tensorflow: epoch = 73.65284974093265, learning_rate = 0.00049999997, loss = 2.3482951e-05, step = 28430 (5.373 sec) 2021-10-19 08:29:03,233 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.856 INFO:tensorflow:epoch = 73.72020725388602, learning_rate = 0.00049999997, loss = 1.2248915e-05, step = 28456 (5.250 sec) 2021-10-19 08:29:04,629 [INFO] tensorflow: epoch = 73.72020725388602, learning_rate = 0.00049999997, loss = 1.2248915e-05, step = 28456 (5.250 sec) INFO:tensorflow:global_step/sec: 4.96472 2021-10-19 08:29:05,849 [INFO] tensorflow: global_step/sec: 4.96472 2021-10-19 08:29:08,199 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.138 INFO:tensorflow:epoch = 73.79015544041451, learning_rate = 0.00049999997, loss = 2.5032552e-05, step = 28483 (5.326 sec) 2021-10-19 08:29:09,955 [INFO] tensorflow: epoch = 73.79015544041451, learning_rate = 0.00049999997, loss = 2.5032552e-05, step = 28483 (5.326 sec) 2021-10-19 08:29:13,135 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.262 INFO:tensorflow:global_step/sec: 5.07148 2021-10-19 08:29:13,341 [INFO] tensorflow: global_step/sec: 5.07148 INFO:tensorflow:epoch = 73.85751295336787, learning_rate = 0.00049999997, loss = 2.7207836e-05, step = 28509 (5.223 sec) 2021-10-19 08:29:15,178 [INFO] tensorflow: epoch = 73.85751295336787, learning_rate = 0.00049999997, loss = 2.7207836e-05, step = 28509 (5.223 sec) 2021-10-19 08:29:18,175 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.844 INFO:tensorflow:epoch = 73.92487046632125, learning_rate = 0.00049999997, loss = 3.873939e-05, step = 28535 (5.206 sec) 2021-10-19 08:29:20,384 [INFO] tensorflow: epoch = 73.92487046632125, learning_rate = 0.00049999997, loss = 3.873939e-05, step = 28535 (5.206 sec) INFO:tensorflow:global_step/sec: 4.97489 2021-10-19 08:29:20,980 [INFO] tensorflow: global_step/sec: 4.97489 2021-10-19 08:29:23,182 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.972 INFO:tensorflow:epoch = 73.99222797927462, learning_rate = 0.00049999997, loss = 1.8310015e-05, step = 28561 (5.222 sec) 2021-10-19 08:29:25,606 [INFO] tensorflow: epoch = 73.99222797927462, learning_rate = 0.00049999997, loss = 1.8310015e-05, step = 28561 (5.222 sec) 2021-10-19 08:29:26,218 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 74/120: loss: 0.00003 learning rate: 0.00050 Time taken: 0:01:17.196350 ETA: 0:59:11.032094 2021-10-19 08:29:28,208 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.899 INFO:tensorflow:global_step/sec: 4.9902 2021-10-19 08:29:28,595 [INFO] tensorflow: global_step/sec: 4.9902 INFO:tensorflow:epoch = 74.06217616580311, learning_rate = 0.00049999997, loss = 3.120648e-05, step = 28588 (5.356 sec) 2021-10-19 08:29:30,962 [INFO] tensorflow: epoch = 74.06217616580311, learning_rate = 0.00049999997, loss = 3.120648e-05, step = 28588 (5.356 sec) 2021-10-19 08:29:33,190 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.071 INFO:tensorflow:epoch = 74.12953367875647, learning_rate = 0.00049999997, loss = 4.5055524e-05, step = 28614 (5.212 sec) 2021-10-19 08:29:36,174 [INFO] tensorflow: epoch = 74.12953367875647, learning_rate = 0.00049999997, loss = 4.5055524e-05, step = 28614 (5.212 sec) INFO:tensorflow:global_step/sec: 5.01268 2021-10-19 08:29:36,175 [INFO] tensorflow: global_step/sec: 5.01268 2021-10-19 08:29:38,172 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.074 INFO:tensorflow:epoch = 74.19948186528498, learning_rate = 0.00049999997, loss = 2.2015898e-05, step = 28641 (5.352 sec) 2021-10-19 08:29:41,526 [INFO] tensorflow: epoch = 74.19948186528498, learning_rate = 0.00049999997, loss = 2.2015898e-05, step = 28641 (5.352 sec) 2021-10-19 08:29:43,152 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.084 INFO:tensorflow:global_step/sec: 4.98359 2021-10-19 08:29:43,800 [INFO] tensorflow: global_step/sec: 4.98359 INFO:tensorflow:epoch = 74.26683937823834, learning_rate = 0.00049999997, loss = 2.3623208e-05, step = 28667 (5.292 sec) 2021-10-19 08:29:46,818 [INFO] tensorflow: epoch = 74.26683937823834, learning_rate = 0.00049999997, loss = 2.3623208e-05, step = 28667 (5.292 sec) 2021-10-19 08:29:48,226 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.710 INFO:tensorflow:global_step/sec: 4.98378 2021-10-19 08:29:51,425 [INFO] tensorflow: global_step/sec: 4.98378 INFO:tensorflow:epoch = 74.33419689119171, learning_rate = 0.00049999997, loss = 1.0353386e-05, step = 28693 (5.207 sec) 2021-10-19 08:29:52,025 [INFO] tensorflow: epoch = 74.33419689119171, learning_rate = 0.00049999997, loss = 1.0353386e-05, step = 28693 (5.207 sec) 2021-10-19 08:29:53,219 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.026 INFO:tensorflow:epoch = 74.40414507772022, learning_rate = 0.00049999997, loss = 2.9471505e-05, step = 28720 (5.376 sec) 2021-10-19 08:29:57,401 [INFO] tensorflow: epoch = 74.40414507772022, learning_rate = 0.00049999997, loss = 2.9471505e-05, step = 28720 (5.376 sec) 2021-10-19 08:29:58,240 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.920 INFO:tensorflow:global_step/sec: 5.00136 2021-10-19 08:29:59,023 [INFO] tensorflow: global_step/sec: 5.00136 INFO:tensorflow:epoch = 74.47150259067358, learning_rate = 0.00049999997, loss = 3.1211326e-05, step = 28746 (5.358 sec) 2021-10-19 08:30:02,759 [INFO] tensorflow: epoch = 74.47150259067358, learning_rate = 0.00049999997, loss = 3.1211326e-05, step = 28746 (5.358 sec) 2021-10-19 08:30:03,359 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.536 INFO:tensorflow:global_step/sec: 4.92165 2021-10-19 08:30:06,744 [INFO] tensorflow: global_step/sec: 4.92165 INFO:tensorflow:epoch = 74.53886010362694, learning_rate = 0.00049999997, loss = 2.3887871e-05, step = 28772 (5.215 sec) 2021-10-19 08:30:07,974 [INFO] tensorflow: epoch = 74.53886010362694, learning_rate = 0.00049999997, loss = 2.3887871e-05, step = 28772 (5.215 sec) 2021-10-19 08:30:08,402 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.830 INFO:tensorflow:epoch = 74.60621761658031, learning_rate = 0.00049999997, loss = 2.3935456e-05, step = 28798 (5.237 sec) 2021-10-19 08:30:13,211 [INFO] tensorflow: epoch = 74.60621761658031, learning_rate = 0.00049999997, loss = 2.3935456e-05, step = 28798 (5.237 sec) 2021-10-19 08:30:13,409 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.973 INFO:tensorflow:global_step/sec: 4.95476 2021-10-19 08:30:14,414 [INFO] tensorflow: global_step/sec: 4.95476 2021-10-19 08:30:18,344 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.265 INFO:tensorflow:epoch = 74.67616580310882, learning_rate = 0.00049999997, loss = 2.428939e-05, step = 28825 (5.327 sec) 2021-10-19 08:30:18,538 [INFO] tensorflow: epoch = 74.67616580310882, learning_rate = 0.00049999997, loss = 2.428939e-05, step = 28825 (5.327 sec) INFO:tensorflow:global_step/sec: 5.05761 2021-10-19 08:30:21,927 [INFO] tensorflow: global_step/sec: 5.05761 2021-10-19 08:30:23,311 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.134 INFO:tensorflow:epoch = 74.74611398963731, learning_rate = 0.00049999997, loss = 1.7375314e-05, step = 28852 (5.360 sec) 2021-10-19 08:30:23,899 [INFO] tensorflow: epoch = 74.74611398963731, learning_rate = 0.00049999997, loss = 1.7375314e-05, step = 28852 (5.360 sec) 2021-10-19 08:30:28,319 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.973 INFO:tensorflow:epoch = 74.81347150259067, learning_rate = 0.00049999997, loss = 3.747717e-05, step = 28878 (5.223 sec) 2021-10-19 08:30:29,122 [INFO] tensorflow: epoch = 74.81347150259067, learning_rate = 0.00049999997, loss = 3.747717e-05, step = 28878 (5.223 sec) INFO:tensorflow:global_step/sec: 4.99987 2021-10-19 08:30:29,527 [INFO] tensorflow: global_step/sec: 4.99987 2021-10-19 08:30:33,320 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.995 INFO:tensorflow:epoch = 74.88341968911918, learning_rate = 0.00049999997, loss = 2.0254232e-05, step = 28905 (5.388 sec) 2021-10-19 08:30:34,511 [INFO] tensorflow: epoch = 74.88341968911918, learning_rate = 0.00049999997, loss = 2.0254232e-05, step = 28905 (5.388 sec) INFO:tensorflow:global_step/sec: 4.99182 2021-10-19 08:30:37,140 [INFO] tensorflow: global_step/sec: 4.99182 2021-10-19 08:30:38,361 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.840 INFO:tensorflow:epoch = 74.95077720207254, learning_rate = 0.00049999997, loss = 2.1950369e-05, step = 28931 (5.241 sec) 2021-10-19 08:30:39,751 [INFO] tensorflow: epoch = 74.95077720207254, learning_rate = 0.00049999997, loss = 2.1950369e-05, step = 28931 (5.241 sec) 2021-10-19 08:30:43,375 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.946 2021-10-19 08:30:43,572 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 75/120: loss: 0.00001 learning rate: 0.00050 Time taken: 0:01:17.351495 ETA: 0:58:00.817255 INFO:tensorflow:global_step/sec: 4.99765 2021-10-19 08:30:44,743 [INFO] tensorflow: global_step/sec: 4.99765 INFO:tensorflow:epoch = 75.02072538860104, learning_rate = 0.00049999997, loss = 1.2803744e-05, step = 28958 (5.380 sec) 2021-10-19 08:30:45,131 [INFO] tensorflow: epoch = 75.02072538860104, learning_rate = 0.00049999997, loss = 1.2803744e-05, step = 28958 (5.380 sec) 2021-10-19 08:30:48,302 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.307 INFO:tensorflow:epoch = 75.09067357512954, learning_rate = 0.00049999997, loss = 2.011422e-05, step = 28985 (5.305 sec) 2021-10-19 08:30:50,436 [INFO] tensorflow: epoch = 75.09067357512954, learning_rate = 0.00049999997, loss = 2.011422e-05, step = 28985 (5.305 sec) INFO:tensorflow:global_step/sec: 5.06309 2021-10-19 08:30:52,249 [INFO] tensorflow: global_step/sec: 5.06309 2021-10-19 08:30:53,241 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.247 INFO:tensorflow:epoch = 75.15803108808291, learning_rate = 0.00049999997, loss = 2.0389762e-05, step = 29011 (5.206 sec) 2021-10-19 08:30:55,642 [INFO] tensorflow: epoch = 75.15803108808291, learning_rate = 0.00049999997, loss = 2.0389762e-05, step = 29011 (5.206 sec) 2021-10-19 08:30:58,192 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.201 INFO:tensorflow:global_step/sec: 5.05732 2021-10-19 08:30:59,762 [INFO] tensorflow: global_step/sec: 5.05732 INFO:tensorflow:epoch = 75.2279792746114, learning_rate = 0.00049999997, loss = 1.9675019e-05, step = 29038 (5.321 sec) 2021-10-19 08:31:00,964 [INFO] tensorflow: epoch = 75.2279792746114, learning_rate = 0.00049999997, loss = 1.9675019e-05, step = 29038 (5.321 sec) 2021-10-19 08:31:03,150 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.171 INFO:tensorflow:epoch = 75.29792746113989, learning_rate = 0.00049999997, loss = 9.089222e-06, step = 29065 (5.377 sec) 2021-10-19 08:31:06,340 [INFO] tensorflow: epoch = 75.29792746113989, learning_rate = 0.00049999997, loss = 9.089222e-06, step = 29065 (5.377 sec) INFO:tensorflow:global_step/sec: 5.02604 2021-10-19 08:31:07,323 [INFO] tensorflow: global_step/sec: 5.02604 2021-10-19 08:31:08,177 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.893 INFO:tensorflow:epoch = 75.36528497409327, learning_rate = 0.00049999997, loss = 1.929251e-05, step = 29091 (5.220 sec) 2021-10-19 08:31:11,561 [INFO] tensorflow: epoch = 75.36528497409327, learning_rate = 0.00049999997, loss = 1.929251e-05, step = 29091 (5.220 sec) 2021-10-19 08:31:13,190 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.952 INFO:tensorflow:global_step/sec: 4.94764 2021-10-19 08:31:15,003 [INFO] tensorflow: global_step/sec: 4.94764 INFO:tensorflow:epoch = 75.43264248704664, learning_rate = 0.00049999997, loss = 2.5302914e-05, step = 29117 (5.238 sec) 2021-10-19 08:31:16,799 [INFO] tensorflow: epoch = 75.43264248704664, learning_rate = 0.00049999997, loss = 2.5302914e-05, step = 29117 (5.238 sec) 2021-10-19 08:31:18,213 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.907 INFO:tensorflow:epoch = 75.50259067357513, learning_rate = 0.00049999997, loss = 2.0202999e-05, step = 29144 (5.373 sec) 2021-10-19 08:31:22,173 [INFO] tensorflow: epoch = 75.50259067357513, learning_rate = 0.00049999997, loss = 2.0202999e-05, step = 29144 (5.373 sec) INFO:tensorflow:global_step/sec: 5.02108 2021-10-19 08:31:22,572 [INFO] tensorflow: global_step/sec: 5.02108 2021-10-19 08:31:23,191 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.093 INFO:tensorflow:epoch = 75.57253886010363, learning_rate = 0.00049999997, loss = 2.0982658e-05, step = 29171 (5.353 sec) 2021-10-19 08:31:27,525 [INFO] tensorflow: epoch = 75.57253886010363, learning_rate = 0.00049999997, loss = 2.0982658e-05, step = 29171 (5.353 sec) 2021-10-19 08:31:28,118 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.296 INFO:tensorflow:global_step/sec: 5.03617 2021-10-19 08:31:30,117 [INFO] tensorflow: global_step/sec: 5.03617 INFO:tensorflow:epoch = 75.64248704663213, learning_rate = 0.00049999997, loss = 1.5358055e-05, step = 29198 (5.397 sec) 2021-10-19 08:31:32,922 [INFO] tensorflow: epoch = 75.64248704663213, learning_rate = 0.00049999997, loss = 1.5358055e-05, step = 29198 (5.397 sec) 2021-10-19 08:31:33,121 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.989 INFO:tensorflow:global_step/sec: 5.04047 2021-10-19 08:31:37,656 [INFO] tensorflow: global_step/sec: 5.04047 2021-10-19 08:31:38,054 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.277 INFO:tensorflow:epoch = 75.71243523316062, learning_rate = 0.00049999997, loss = 2.1774007e-05, step = 29225 (5.354 sec) 2021-10-19 08:31:38,276 [INFO] tensorflow: epoch = 75.71243523316062, learning_rate = 0.00049999997, loss = 2.1774007e-05, step = 29225 (5.354 sec) 2021-10-19 08:31:43,014 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.162 INFO:tensorflow:epoch = 75.78238341968913, learning_rate = 0.00049999997, loss = 2.7009406e-05, step = 29252 (5.351 sec) 2021-10-19 08:31:43,627 [INFO] tensorflow: epoch = 75.78238341968913, learning_rate = 0.00049999997, loss = 2.7009406e-05, step = 29252 (5.351 sec) INFO:tensorflow:global_step/sec: 5.01876 2021-10-19 08:31:45,228 [INFO] tensorflow: global_step/sec: 5.01876 2021-10-19 08:31:47,961 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.215 INFO:tensorflow:epoch = 75.85233160621762, learning_rate = 0.00049999997, loss = 2.2727552e-05, step = 29279 (5.310 sec) 2021-10-19 08:31:48,937 [INFO] tensorflow: epoch = 75.85233160621762, learning_rate = 0.00049999997, loss = 2.2727552e-05, step = 29279 (5.310 sec) INFO:tensorflow:global_step/sec: 5.04227 2021-10-19 08:31:52,764 [INFO] tensorflow: global_step/sec: 5.04227 2021-10-19 08:31:52,991 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.884 INFO:tensorflow:epoch = 75.91968911917098, learning_rate = 0.00049999997, loss = 1.28123365e-05, step = 29305 (5.280 sec) 2021-10-19 08:31:54,217 [INFO] tensorflow: epoch = 75.91968911917098, learning_rate = 0.00049999997, loss = 1.28123365e-05, step = 29305 (5.280 sec) 2021-10-19 08:31:57,969 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.087 INFO:tensorflow:epoch = 75.98963730569949, learning_rate = 0.00049999997, loss = 4.113362e-05, step = 29332 (5.343 sec) 2021-10-19 08:31:59,561 [INFO] tensorflow: epoch = 75.98963730569949, learning_rate = 0.00049999997, loss = 4.113362e-05, step = 29332 (5.343 sec) INFO:tensorflow:global_step/sec: 4.99098 2021-10-19 08:32:00,378 [INFO] tensorflow: global_step/sec: 4.99098 2021-10-19 08:32:00,379 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 76/120: loss: 0.00003 learning rate: 0.00050 Time taken: 0:01:16.792633 ETA: 0:56:18.875844 2021-10-19 08:32:02,953 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.067 INFO:tensorflow:epoch = 76.05958549222798, learning_rate = 0.00049999997, loss = 2.1217116e-05, step = 29359 (5.332 sec) 2021-10-19 08:32:04,893 [INFO] tensorflow: epoch = 76.05958549222798, learning_rate = 0.00049999997, loss = 2.1217116e-05, step = 29359 (5.332 sec) INFO:tensorflow:global_step/sec: 5.06885 2021-10-19 08:32:07,874 [INFO] tensorflow: global_step/sec: 5.06885 2021-10-19 08:32:07,876 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.316 INFO:tensorflow:epoch = 76.12953367875647, learning_rate = 0.00049999997, loss = 1.5914462e-05, step = 29386 (5.382 sec) 2021-10-19 08:32:10,274 [INFO] tensorflow: epoch = 76.12953367875647, learning_rate = 0.00049999997, loss = 1.5914462e-05, step = 29386 (5.382 sec) 2021-10-19 08:32:12,835 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.164 INFO:tensorflow:global_step/sec: 5.05462 2021-10-19 08:32:15,392 [INFO] tensorflow: global_step/sec: 5.05462 INFO:tensorflow:epoch = 76.19948186528498, learning_rate = 0.00049999997, loss = 9.074674e-06, step = 29413 (5.332 sec) 2021-10-19 08:32:15,606 [INFO] tensorflow: epoch = 76.19948186528498, learning_rate = 0.00049999997, loss = 9.074674e-06, step = 29413 (5.332 sec) 2021-10-19 08:32:17,793 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.173 INFO:tensorflow:epoch = 76.26943005181347, learning_rate = 0.00049999997, loss = 1.8478926e-05, step = 29440 (5.329 sec) 2021-10-19 08:32:20,935 [INFO] tensorflow: epoch = 76.26943005181347, learning_rate = 0.00049999997, loss = 1.8478926e-05, step = 29440 (5.329 sec) 2021-10-19 08:32:22,735 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.238 INFO:tensorflow:global_step/sec: 5.03548 2021-10-19 08:32:22,939 [INFO] tensorflow: global_step/sec: 5.03548 INFO:tensorflow:epoch = 76.33937823834196, learning_rate = 0.00049999997, loss = 1.6505142e-05, step = 29467 (5.372 sec) 2021-10-19 08:32:26,307 [INFO] tensorflow: epoch = 76.33937823834196, learning_rate = 0.00049999997, loss = 1.6505142e-05, step = 29467 (5.372 sec) 2021-10-19 08:32:27,697 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.156 INFO:tensorflow:global_step/sec: 5.04707 2021-10-19 08:32:30,468 [INFO] tensorflow: global_step/sec: 5.04707 INFO:tensorflow:epoch = 76.40932642487047, learning_rate = 0.00049999997, loss = 1.854185e-05, step = 29494 (5.380 sec) 2021-10-19 08:32:31,687 [INFO] tensorflow: epoch = 76.40932642487047, learning_rate = 0.00049999997, loss = 1.854185e-05, step = 29494 (5.380 sec) 2021-10-19 08:32:32,678 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.078 INFO:tensorflow:epoch = 76.47927461139896, learning_rate = 0.00049999997, loss = 5.288219e-05, step = 29521 (5.387 sec) 2021-10-19 08:32:37,074 [INFO] tensorflow: epoch = 76.47927461139896, learning_rate = 0.00049999997, loss = 5.288219e-05, step = 29521 (5.387 sec) 2021-10-19 08:32:37,692 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.946 INFO:tensorflow:global_step/sec: 4.98935 2021-10-19 08:32:38,084 [INFO] tensorflow: global_step/sec: 4.98935 INFO:tensorflow:epoch = 76.54663212435233, learning_rate = 0.00049999997, loss = 2.0061601e-05, step = 29547 (5.223 sec) 2021-10-19 08:32:42,297 [INFO] tensorflow: epoch = 76.54663212435233, learning_rate = 0.00049999997, loss = 2.0061601e-05, step = 29547 (5.223 sec) 2021-10-19 08:32:42,716 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.907 INFO:tensorflow:global_step/sec: 4.93373 2021-10-19 08:32:45,786 [INFO] tensorflow: global_step/sec: 4.93373 INFO:tensorflow:epoch = 76.6139896373057, learning_rate = 0.00049999997, loss = 2.126432e-05, step = 29573 (5.250 sec) 2021-10-19 08:32:47,547 [INFO] tensorflow: epoch = 76.6139896373057, learning_rate = 0.00049999997, loss = 2.126432e-05, step = 29573 (5.250 sec) 2021-10-19 08:32:47,772 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.781 2021-10-19 08:32:52,743 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.115 INFO:tensorflow:epoch = 76.6839378238342, learning_rate = 0.00049999997, loss = 1.910766e-05, step = 29600 (5.411 sec) 2021-10-19 08:32:52,959 [INFO] tensorflow: epoch = 76.6839378238342, learning_rate = 0.00049999997, loss = 1.910766e-05, step = 29600 (5.411 sec) INFO:tensorflow:global_step/sec: 5.02277 2021-10-19 08:32:53,352 [INFO] tensorflow: global_step/sec: 5.02277 2021-10-19 08:32:57,711 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.130 INFO:tensorflow:epoch = 76.75388601036269, learning_rate = 0.00049999997, loss = 2.5486588e-05, step = 29627 (5.345 sec) 2021-10-19 08:32:58,304 [INFO] tensorflow: epoch = 76.75388601036269, learning_rate = 0.00049999997, loss = 2.5486588e-05, step = 29627 (5.345 sec) INFO:tensorflow:global_step/sec: 5.00933 2021-10-19 08:33:00,937 [INFO] tensorflow: global_step/sec: 5.00933 2021-10-19 08:33:02,760 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.809 INFO:tensorflow:epoch = 76.82124352331607, learning_rate = 0.00049999997, loss = 1.6894975e-05, step = 29653 (5.274 sec) 2021-10-19 08:33:03,577 [INFO] tensorflow: epoch = 76.82124352331607, learning_rate = 0.00049999997, loss = 1.6894975e-05, step = 29653 (5.274 sec) 2021-10-19 08:33:07,768 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.971 INFO:tensorflow:global_step/sec: 4.99079 2021-10-19 08:33:08,551 [INFO] tensorflow: global_step/sec: 4.99079 INFO:tensorflow:epoch = 76.89119170984456, learning_rate = 0.00049999997, loss = 4.192686e-05, step = 29680 (5.376 sec) 2021-10-19 08:33:08,953 [INFO] tensorflow: epoch = 76.89119170984456, learning_rate = 0.00049999997, loss = 4.192686e-05, step = 29680 (5.376 sec) 2021-10-19 08:33:12,683 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.348 INFO:tensorflow:epoch = 76.96113989637306, learning_rate = 0.00049999997, loss = 9.191786e-06, step = 29707 (5.374 sec) 2021-10-19 08:33:14,327 [INFO] tensorflow: epoch = 76.96113989637306, learning_rate = 0.00049999997, loss = 9.191786e-06, step = 29707 (5.374 sec) INFO:tensorflow:global_step/sec: 5.02216 2021-10-19 08:33:16,118 [INFO] tensorflow: global_step/sec: 5.02216 2021-10-19 08:33:17,292 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 77/120: loss: 0.00003 learning rate: 0.00050 Time taken: 0:01:16.911878 ETA: 0:55:07.210748 2021-10-19 08:33:17,682 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.005 INFO:tensorflow:epoch = 77.03108808290156, learning_rate = 0.00049999997, loss = 1.5123709e-05, step = 29734 (5.383 sec) 2021-10-19 08:33:19,709 [INFO] tensorflow: epoch = 77.03108808290156, learning_rate = 0.00049999997, loss = 1.5123709e-05, step = 29734 (5.383 sec) 2021-10-19 08:33:22,691 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.967 INFO:tensorflow:global_step/sec: 5.01459 2021-10-19 08:33:23,696 [INFO] tensorflow: global_step/sec: 5.01459 INFO:tensorflow:epoch = 77.10103626943005, learning_rate = 0.00049999997, loss = 2.0861142e-05, step = 29761 (5.369 sec) 2021-10-19 08:33:25,078 [INFO] tensorflow: epoch = 77.10103626943005, learning_rate = 0.00049999997, loss = 2.0861142e-05, step = 29761 (5.369 sec) 2021-10-19 08:33:27,695 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.983 INFO:tensorflow:epoch = 77.17098445595855, learning_rate = 0.00049999997, loss = 2.6903954e-05, step = 29788 (5.362 sec) 2021-10-19 08:33:30,440 [INFO] tensorflow: epoch = 77.17098445595855, learning_rate = 0.00049999997, loss = 2.6903954e-05, step = 29788 (5.362 sec) INFO:tensorflow:global_step/sec: 5.03623 2021-10-19 08:33:31,241 [INFO] tensorflow: global_step/sec: 5.03623 2021-10-19 08:33:32,637 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.238 INFO:tensorflow:epoch = 77.23834196891193, learning_rate = 0.00049999997, loss = 2.2676017e-05, step = 29814 (5.238 sec) 2021-10-19 08:33:35,678 [INFO] tensorflow: epoch = 77.23834196891193, learning_rate = 0.00049999997, loss = 2.2676017e-05, step = 29814 (5.238 sec) 2021-10-19 08:33:37,676 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.848 INFO:tensorflow:global_step/sec: 4.99338 2021-10-19 08:33:38,851 [INFO] tensorflow: global_step/sec: 4.99338 INFO:tensorflow:epoch = 77.30829015544042, learning_rate = 0.00049999997, loss = 3.140763e-05, step = 29841 (5.373 sec) 2021-10-19 08:33:41,051 [INFO] tensorflow: epoch = 77.30829015544042, learning_rate = 0.00049999997, loss = 3.140763e-05, step = 29841 (5.373 sec) 2021-10-19 08:33:42,661 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.063 INFO:tensorflow:epoch = 77.37823834196891, learning_rate = 0.00049999997, loss = 1.6920989e-05, step = 29868 (5.388 sec) 2021-10-19 08:33:46,439 [INFO] tensorflow: epoch = 77.37823834196891, learning_rate = 0.00049999997, loss = 1.6920989e-05, step = 29868 (5.388 sec) INFO:tensorflow:global_step/sec: 5.00704 2021-10-19 08:33:46,441 [INFO] tensorflow: global_step/sec: 5.00704 2021-10-19 08:33:47,647 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.058 INFO:tensorflow:epoch = 77.44818652849742, learning_rate = 0.00049999997, loss = 1.3417973e-05, step = 29895 (5.357 sec) 2021-10-19 08:33:51,796 [INFO] tensorflow: epoch = 77.44818652849742, learning_rate = 0.00049999997, loss = 1.3417973e-05, step = 29895 (5.357 sec) 2021-10-19 08:33:52,599 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.193 INFO:tensorflow:global_step/sec: 5.03407 2021-10-19 08:33:53,989 [INFO] tensorflow: global_step/sec: 5.03407 INFO:tensorflow:epoch = 77.51813471502591, learning_rate = 0.00049999997, loss = 2.615582e-05, step = 29922 (5.369 sec) 2021-10-19 08:33:57,165 [INFO] tensorflow: epoch = 77.51813471502591, learning_rate = 0.00049999997, loss = 2.615582e-05, step = 29922 (5.369 sec) 2021-10-19 08:33:57,554 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.184 INFO:tensorflow:global_step/sec: 5.01178 2021-10-19 08:34:01,571 [INFO] tensorflow: global_step/sec: 5.01178 INFO:tensorflow:epoch = 77.58549222797927, learning_rate = 0.00049999997, loss = 1.1997166e-05, step = 29948 (5.198 sec) 2021-10-19 08:34:02,364 [INFO] tensorflow: epoch = 77.58549222797927, learning_rate = 0.00049999997, loss = 1.1997166e-05, step = 29948 (5.198 sec) 2021-10-19 08:34:02,582 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.893 2021-10-19 08:34:07,560 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.089 INFO:tensorflow:epoch = 77.65544041450778, learning_rate = 0.00049999997, loss = 3.0516328e-05, step = 29975 (5.395 sec) 2021-10-19 08:34:07,759 [INFO] tensorflow: epoch = 77.65544041450778, learning_rate = 0.00049999997, loss = 3.0516328e-05, step = 29975 (5.395 sec) INFO:tensorflow:global_step/sec: 5.00223 2021-10-19 08:34:09,168 [INFO] tensorflow: global_step/sec: 5.00223 2021-10-19 08:34:12,619 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.768 INFO:tensorflow:epoch = 77.72279792746114, learning_rate = 0.00049999997, loss = 1.5012945e-05, step = 30001 (5.269 sec) 2021-10-19 08:34:13,028 [INFO] tensorflow: epoch = 77.72279792746114, learning_rate = 0.00049999997, loss = 1.5012945e-05, step = 30001 (5.269 sec) INFO:tensorflow:global_step/sec: 5.00309 2021-10-19 08:34:16,763 [INFO] tensorflow: global_step/sec: 5.00309 2021-10-19 08:34:17,565 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.220 INFO:tensorflow:epoch = 77.79274611398964, learning_rate = 0.00049999997, loss = 2.4563287e-05, step = 30028 (5.339 sec) 2021-10-19 08:34:18,366 [INFO] tensorflow: epoch = 77.79274611398964, learning_rate = 0.00049999997, loss = 2.4563287e-05, step = 30028 (5.339 sec) 2021-10-19 08:34:22,517 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.196 INFO:tensorflow:epoch = 77.86269430051814, learning_rate = 0.00049999997, loss = 8.929424e-06, step = 30055 (5.392 sec) 2021-10-19 08:34:23,759 [INFO] tensorflow: epoch = 77.86269430051814, learning_rate = 0.00049999997, loss = 8.929424e-06, step = 30055 (5.392 sec) INFO:tensorflow:global_step/sec: 5.00902 2021-10-19 08:34:24,349 [INFO] tensorflow: global_step/sec: 5.00902 2021-10-19 08:34:27,542 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.901 INFO:tensorflow:epoch = 77.93005181347151, learning_rate = 0.00049999997, loss = 1.4027153e-05, step = 30081 (5.203 sec) 2021-10-19 08:34:28,962 [INFO] tensorflow: epoch = 77.93005181347151, learning_rate = 0.00049999997, loss = 1.4027153e-05, step = 30081 (5.203 sec) INFO:tensorflow:global_step/sec: 5.01977 2021-10-19 08:34:31,920 [INFO] tensorflow: global_step/sec: 5.01977 2021-10-19 08:34:32,527 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.061 INFO:tensorflow:epoch = 78.0, learning_rate = 0.00049999997, loss = 1.7000919e-05, step = 30108 (5.386 sec) 2021-10-19 08:34:34,349 [INFO] tensorflow: epoch = 78.0, learning_rate = 0.00049999997, loss = 1.7000919e-05, step = 30108 (5.386 sec) 2021-10-19 08:34:34,350 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 78/120: loss: 0.00002 learning rate: 0.00050 Time taken: 0:01:17.070668 ETA: 0:53:56.968045 2021-10-19 08:34:37,520 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.030 INFO:tensorflow:epoch = 78.06735751295336, learning_rate = 0.00049999997, loss = 3.663895e-05, step = 30134 (5.221 sec) 2021-10-19 08:34:39,570 [INFO] tensorflow: epoch = 78.06735751295336, learning_rate = 0.00049999997, loss = 3.663895e-05, step = 30134 (5.221 sec) INFO:tensorflow:global_step/sec: 4.96561 2021-10-19 08:34:39,572 [INFO] tensorflow: global_step/sec: 4.96561 2021-10-19 08:34:42,586 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.740 INFO:tensorflow:epoch = 78.13730569948187, learning_rate = 0.00049999997, loss = 1.755092e-05, step = 30161 (5.375 sec) 2021-10-19 08:34:44,945 [INFO] tensorflow: epoch = 78.13730569948187, learning_rate = 0.00049999997, loss = 1.755092e-05, step = 30161 (5.375 sec) INFO:tensorflow:global_step/sec: 5.02502 2021-10-19 08:34:47,134 [INFO] tensorflow: global_step/sec: 5.02502 2021-10-19 08:34:47,544 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.174 INFO:tensorflow:epoch = 78.20725388601036, learning_rate = 0.00049999997, loss = 1.7562756e-05, step = 30188 (5.373 sec) 2021-10-19 08:34:50,318 [INFO] tensorflow: epoch = 78.20725388601036, learning_rate = 0.00049999997, loss = 1.7562756e-05, step = 30188 (5.373 sec) 2021-10-19 08:34:52,528 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.066 INFO:tensorflow:global_step/sec: 5.00064 2021-10-19 08:34:54,733 [INFO] tensorflow: global_step/sec: 5.00064 INFO:tensorflow:epoch = 78.27461139896373, learning_rate = 0.00049999997, loss = 3.349786e-05, step = 30214 (5.245 sec) 2021-10-19 08:34:55,563 [INFO] tensorflow: epoch = 78.27461139896373, learning_rate = 0.00049999997, loss = 3.349786e-05, step = 30214 (5.245 sec) 2021-10-19 08:34:57,609 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.682 INFO:tensorflow:epoch = 78.3419689119171, learning_rate = 0.00049999997, loss = 1.8227616e-05, step = 30240 (5.239 sec) 2021-10-19 08:35:00,802 [INFO] tensorflow: epoch = 78.3419689119171, learning_rate = 0.00049999997, loss = 1.8227616e-05, step = 30240 (5.239 sec) INFO:tensorflow:global_step/sec: 4.94289 2021-10-19 08:35:02,421 [INFO] tensorflow: global_step/sec: 4.94289 2021-10-19 08:35:02,615 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.979 INFO:tensorflow:epoch = 78.40932642487047, learning_rate = 0.00049999997, loss = 1.5181465e-05, step = 30266 (5.256 sec) 2021-10-19 08:35:06,059 [INFO] tensorflow: epoch = 78.40932642487047, learning_rate = 0.00049999997, loss = 1.5181465e-05, step = 30266 (5.256 sec) 2021-10-19 08:35:07,642 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.893 INFO:tensorflow:global_step/sec: 5.00227 2021-10-19 08:35:10,018 [INFO] tensorflow: global_step/sec: 5.00227 INFO:tensorflow:epoch = 78.47927461139896, learning_rate = 0.00049999997, loss = 2.963943e-05, step = 30293 (5.333 sec) 2021-10-19 08:35:11,392 [INFO] tensorflow: epoch = 78.47927461139896, learning_rate = 0.00049999997, loss = 2.963943e-05, step = 30293 (5.333 sec) 2021-10-19 08:35:12,562 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.328 INFO:tensorflow:epoch = 78.54922279792747, learning_rate = 0.00049999997, loss = 3.1763757e-05, step = 30320 (5.318 sec) 2021-10-19 08:35:16,710 [INFO] tensorflow: epoch = 78.54922279792747, learning_rate = 0.00049999997, loss = 3.1763757e-05, step = 30320 (5.318 sec) INFO:tensorflow:global_step/sec: 5.07686 2021-10-19 08:35:17,503 [INFO] tensorflow: global_step/sec: 5.07686 2021-10-19 08:35:17,504 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.238 INFO:tensorflow:epoch = 78.61917098445596, learning_rate = 0.00049999997, loss = 1.4357632e-05, step = 30347 (5.332 sec) 2021-10-19 08:35:22,042 [INFO] tensorflow: epoch = 78.61917098445596, learning_rate = 0.00049999997, loss = 1.4357632e-05, step = 30347 (5.332 sec) 2021-10-19 08:35:22,443 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.249 INFO:tensorflow:global_step/sec: 5.02496 2021-10-19 08:35:25,065 [INFO] tensorflow: global_step/sec: 5.02496 INFO:tensorflow:epoch = 78.68652849740933, learning_rate = 0.00049999997, loss = 1.5503014e-05, step = 30373 (5.214 sec) 2021-10-19 08:35:27,256 [INFO] tensorflow: epoch = 78.68652849740933, learning_rate = 0.00049999997, loss = 1.5503014e-05, step = 30373 (5.214 sec) 2021-10-19 08:35:27,460 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.934 INFO:tensorflow:epoch = 78.75388601036269, learning_rate = 0.00049999997, loss = 2.6194437e-05, step = 30399 (5.225 sec) 2021-10-19 08:35:32,481 [INFO] tensorflow: epoch = 78.75388601036269, learning_rate = 0.00049999997, loss = 2.6194437e-05, step = 30399 (5.225 sec) 2021-10-19 08:35:32,482 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.916 INFO:tensorflow:global_step/sec: 4.97484 2021-10-19 08:35:32,703 [INFO] tensorflow: global_step/sec: 4.97484 2021-10-19 08:35:37,440 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.169 INFO:tensorflow:epoch = 78.8238341968912, learning_rate = 0.00049999997, loss = 2.8671238e-05, step = 30426 (5.343 sec) 2021-10-19 08:35:37,824 [INFO] tensorflow: epoch = 78.8238341968912, learning_rate = 0.00049999997, loss = 2.8671238e-05, step = 30426 (5.343 sec) INFO:tensorflow:global_step/sec: 5.08326 2021-10-19 08:35:40,179 [INFO] tensorflow: global_step/sec: 5.08326 2021-10-19 08:35:42,411 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.120 INFO:tensorflow:epoch = 78.89378238341969, learning_rate = 0.00049999997, loss = 4.5292312e-05, step = 30453 (5.372 sec) 2021-10-19 08:35:43,197 [INFO] tensorflow: epoch = 78.89378238341969, learning_rate = 0.00049999997, loss = 4.5292312e-05, step = 30453 (5.372 sec) 2021-10-19 08:35:47,422 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.960 INFO:tensorflow:global_step/sec: 4.96633 2021-10-19 08:35:47,830 [INFO] tensorflow: global_step/sec: 4.96633 INFO:tensorflow:epoch = 78.96113989637306, learning_rate = 0.00049999997, loss = 2.3076958e-05, step = 30479 (5.229 sec) 2021-10-19 08:35:48,426 [INFO] tensorflow: epoch = 78.96113989637306, learning_rate = 0.00049999997, loss = 2.3076958e-05, step = 30479 (5.229 sec) 2021-10-19 08:35:51,436 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 79/120: loss: 0.00002 learning rate: 0.00050 Time taken: 0:01:17.081246 ETA: 0:52:40.331101 2021-10-19 08:35:52,445 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.910 INFO:tensorflow:epoch = 79.02849740932643, learning_rate = 0.00049999997, loss = 1.3569179e-05, step = 30505 (5.225 sec) 2021-10-19 08:35:53,651 [INFO] tensorflow: epoch = 79.02849740932643, learning_rate = 0.00049999997, loss = 1.3569179e-05, step = 30505 (5.225 sec) INFO:tensorflow:global_step/sec: 4.99982 2021-10-19 08:35:55,431 [INFO] tensorflow: global_step/sec: 4.99982 2021-10-19 08:35:57,469 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.915 INFO:tensorflow:epoch = 79.0958549222798, learning_rate = 0.00049999997, loss = 2.0389303e-05, step = 30531 (5.239 sec) 2021-10-19 08:35:58,889 [INFO] tensorflow: epoch = 79.0958549222798, learning_rate = 0.00049999997, loss = 2.0389303e-05, step = 30531 (5.239 sec) 2021-10-19 08:36:02,546 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.696 INFO:tensorflow:global_step/sec: 4.9087 2021-10-19 08:36:03,172 [INFO] tensorflow: global_step/sec: 4.9087 INFO:tensorflow:epoch = 79.16321243523316, learning_rate = 0.00049999997, loss = 2.1230968e-05, step = 30557 (5.287 sec) 2021-10-19 08:36:04,176 [INFO] tensorflow: epoch = 79.16321243523316, learning_rate = 0.00049999997, loss = 2.1230968e-05, step = 30557 (5.287 sec) 2021-10-19 08:36:07,504 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.173 INFO:tensorflow:epoch = 79.23316062176166, learning_rate = 0.00049999997, loss = 3.8661026e-05, step = 30584 (5.322 sec) 2021-10-19 08:36:09,498 [INFO] tensorflow: epoch = 79.23316062176166, learning_rate = 0.00049999997, loss = 3.8661026e-05, step = 30584 (5.322 sec) INFO:tensorflow:global_step/sec: 5.054 2021-10-19 08:36:10,691 [INFO] tensorflow: global_step/sec: 5.054 2021-10-19 08:36:12,468 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.149 INFO:tensorflow:epoch = 79.30310880829016, learning_rate = 0.00049999997, loss = 5.3270862e-05, step = 30611 (5.345 sec) 2021-10-19 08:36:14,843 [INFO] tensorflow: epoch = 79.30310880829016, learning_rate = 0.00049999997, loss = 5.3270862e-05, step = 30611 (5.345 sec) 2021-10-19 08:36:17,425 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.173 INFO:tensorflow:global_step/sec: 5.04526 2021-10-19 08:36:18,223 [INFO] tensorflow: global_step/sec: 5.04526 INFO:tensorflow:epoch = 79.37305699481865, learning_rate = 0.00049999997, loss = 1.316528e-05, step = 30638 (5.333 sec) 2021-10-19 08:36:20,176 [INFO] tensorflow: epoch = 79.37305699481865, learning_rate = 0.00049999997, loss = 1.316528e-05, step = 30638 (5.333 sec) 2021-10-19 08:36:22,437 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.958 INFO:tensorflow:epoch = 79.44041450777202, learning_rate = 0.00049999997, loss = 4.4048982e-05, step = 30664 (5.330 sec) 2021-10-19 08:36:25,506 [INFO] tensorflow: epoch = 79.44041450777202, learning_rate = 0.00049999997, loss = 4.4048982e-05, step = 30664 (5.330 sec) INFO:tensorflow:global_step/sec: 4.92542 2021-10-19 08:36:25,938 [INFO] tensorflow: global_step/sec: 4.92542 2021-10-19 08:36:27,513 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.703 INFO:tensorflow:epoch = 79.5077720207254, learning_rate = 0.00049999997, loss = 1.3307263e-05, step = 30690 (5.235 sec) 2021-10-19 08:36:30,741 [INFO] tensorflow: epoch = 79.5077720207254, learning_rate = 0.00049999997, loss = 1.3307263e-05, step = 30690 (5.235 sec) 2021-10-19 08:36:32,544 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.878 INFO:tensorflow:global_step/sec: 4.9984 2021-10-19 08:36:33,540 [INFO] tensorflow: global_step/sec: 4.9984 INFO:tensorflow:epoch = 79.57772020725389, learning_rate = 0.00049999997, loss = 2.5216817e-05, step = 30717 (5.389 sec) 2021-10-19 08:36:36,130 [INFO] tensorflow: epoch = 79.57772020725389, learning_rate = 0.00049999997, loss = 2.5216817e-05, step = 30717 (5.389 sec) 2021-10-19 08:36:37,549 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.983 INFO:tensorflow:global_step/sec: 5.00574 2021-10-19 08:36:41,131 [INFO] tensorflow: global_step/sec: 5.00574 INFO:tensorflow:epoch = 79.64507772020725, learning_rate = 0.00049999997, loss = 1.4432171e-05, step = 30743 (5.205 sec) 2021-10-19 08:36:41,335 [INFO] tensorflow: epoch = 79.64507772020725, learning_rate = 0.00049999997, loss = 1.4432171e-05, step = 30743 (5.205 sec) 2021-10-19 08:36:42,524 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.101 INFO:tensorflow:epoch = 79.71502590673576, learning_rate = 0.00049999997, loss = 1.550758e-05, step = 30770 (5.369 sec) 2021-10-19 08:36:46,704 [INFO] tensorflow: epoch = 79.71502590673576, learning_rate = 0.00049999997, loss = 1.550758e-05, step = 30770 (5.369 sec) 2021-10-19 08:36:47,512 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.050 INFO:tensorflow:global_step/sec: 5.02822 2021-10-19 08:36:48,689 [INFO] tensorflow: global_step/sec: 5.02822 INFO:tensorflow:epoch = 79.78497409326425, learning_rate = 0.00049999997, loss = 1.5031908e-05, step = 30797 (5.361 sec) 2021-10-19 08:36:52,065 [INFO] tensorflow: epoch = 79.78497409326425, learning_rate = 0.00049999997, loss = 1.5031908e-05, step = 30797 (5.361 sec) 2021-10-19 08:36:52,450 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.251 INFO:tensorflow:global_step/sec: 5.04888 2021-10-19 08:36:56,215 [INFO] tensorflow: global_step/sec: 5.04888 INFO:tensorflow:epoch = 79.85492227979275, learning_rate = 0.00049999997, loss = 2.1815367e-05, step = 30824 (5.313 sec) 2021-10-19 08:36:57,378 [INFO] tensorflow: epoch = 79.85492227979275, learning_rate = 0.00049999997, loss = 2.1815367e-05, step = 30824 (5.313 sec) 2021-10-19 08:36:57,378 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.295 2021-10-19 08:37:02,345 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.136 INFO:tensorflow:epoch = 79.92487046632125, learning_rate = 0.00049999997, loss = 1.7083514e-05, step = 30851 (5.362 sec) 2021-10-19 08:37:02,740 [INFO] tensorflow: epoch = 79.92487046632125, learning_rate = 0.00049999997, loss = 1.7083514e-05, step = 30851 (5.362 sec) INFO:tensorflow:global_step/sec: 5.04656 2021-10-19 08:37:03,745 [INFO] tensorflow: global_step/sec: 5.04656 2021-10-19 08:37:07,322 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.092 INFO:tensorflow:epoch = 79.99222797927462, learning_rate = 0.00049999997, loss = 2.3828874e-05, step = 30877 (5.218 sec) 2021-10-19 08:37:07,958 [INFO] tensorflow: epoch = 79.99222797927462, learning_rate = 0.00049999997, loss = 2.3828874e-05, step = 30877 (5.218 sec) INFO:tensorflow:Saving checkpoints for step-30880. 2021-10-19 08:37:08,346 [INFO] tensorflow: Saving checkpoints for step-30880. WARNING:tensorflow:Ignoring: /tmp/tmpfj5fd2nw; No such file or directory 2021-10-19 08:37:08,570 [WARNING] tensorflow: Ignoring: /tmp/tmpfj5fd2nw; No such file or directory 2021-10-19 08:37:13,420 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 62, 0.00s/step 2021-10-19 08:37:14,355 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 62, 0.09s/step 2021-10-19 08:37:15,279 [INFO] iva.detectnet_v2.evaluation.evaluation: step 20 / 62, 0.09s/step 2021-10-19 08:37:16,244 [INFO] iva.detectnet_v2.evaluation.evaluation: step 30 / 62, 0.10s/step 2021-10-19 08:37:17,189 [INFO] iva.detectnet_v2.evaluation.evaluation: step 40 / 62, 0.09s/step 2021-10-19 08:37:18,175 [INFO] iva.detectnet_v2.evaluation.evaluation: step 50 / 62, 0.10s/step 2021-10-19 08:37:19,099 [INFO] iva.detectnet_v2.evaluation.evaluation: step 60 / 62, 0.09s/step Matching predictions to ground truth, class 1/7.: 100%|█| 406/406 [00:00<00:00, 13721.79it/s] Matching predictions to ground truth, class 2/7.: 100%|█| 254/254 [00:00<00:00, 13655.92it/s] Matching predictions to ground truth, class 7/7.: 100%|█| 169/169 [00:00<00:00, 26233.80it/s] Epoch 80/120 ========================= Validation cost: 0.000015 Mean average_precision (in %): 26.7361 class name average precision (in %) ------------- -------------------------- auto_rickshaw 0 bicycle 82.1526 bus 32.6471 car 72.3528 person 0 road_sign 0 truck 0 Median Inference Time: 0.019818 INFO:tensorflow:epoch = 80.0, learning_rate = 0.00049999997, loss = 1.1474294e-05, step = 30880 (11.631 sec) 2021-10-19 08:37:19,589 [INFO] tensorflow: epoch = 80.0, learning_rate = 0.00049999997, loss = 1.1474294e-05, step = 30880 (11.631 sec) 2021-10-19 08:37:19,589 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 80/120: loss: 0.00001 learning rate: 0.00050 Time taken: 0:01:28.150312 ETA: 0:58:46.012478 INFO:tensorflow:global_step/sec: 2.03627 2021-10-19 08:37:22,407 [INFO] tensorflow: global_step/sec: 2.03627 2021-10-19 08:37:23,373 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 6.230 INFO:tensorflow:epoch = 80.06735751295336, learning_rate = 0.00049999997, loss = 1.1354424e-05, step = 30906 (5.221 sec) 2021-10-19 08:37:24,809 [INFO] tensorflow: epoch = 80.06735751295336, learning_rate = 0.00049999997, loss = 1.1354424e-05, step = 30906 (5.221 sec) 2021-10-19 08:37:28,399 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.899 INFO:tensorflow:global_step/sec: 5.04092 2021-10-19 08:37:29,945 [INFO] tensorflow: global_step/sec: 5.04092 INFO:tensorflow:epoch = 80.13730569948187, learning_rate = 0.00049999997, loss = 1.9023872e-05, step = 30933 (5.333 sec) 2021-10-19 08:37:30,142 [INFO] tensorflow: epoch = 80.13730569948187, learning_rate = 0.00049999997, loss = 1.9023872e-05, step = 30933 (5.333 sec) 2021-10-19 08:37:33,293 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.437 INFO:tensorflow:epoch = 80.20725388601036, learning_rate = 0.00049999997, loss = 1.9192245e-05, step = 30960 (5.328 sec) 2021-10-19 08:37:35,470 [INFO] tensorflow: epoch = 80.20725388601036, learning_rate = 0.00049999997, loss = 1.9192245e-05, step = 30960 (5.328 sec) INFO:tensorflow:global_step/sec: 5.05485 2021-10-19 08:37:37,463 [INFO] tensorflow: global_step/sec: 5.05485 2021-10-19 08:37:38,258 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.139 INFO:tensorflow:epoch = 80.27720207253887, learning_rate = 0.00049999997, loss = 2.3844193e-05, step = 30987 (5.322 sec) 2021-10-19 08:37:40,792 [INFO] tensorflow: epoch = 80.27720207253887, learning_rate = 0.00049999997, loss = 2.3844193e-05, step = 30987 (5.322 sec) 2021-10-19 08:37:43,162 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.396 INFO:tensorflow:global_step/sec: 5.08285 2021-10-19 08:37:44,939 [INFO] tensorflow: global_step/sec: 5.08285 INFO:tensorflow:epoch = 80.34715025906736, learning_rate = 0.00049999997, loss = 4.9309823e-05, step = 31014 (5.347 sec) 2021-10-19 08:37:46,140 [INFO] tensorflow: epoch = 80.34715025906736, learning_rate = 0.00049999997, loss = 4.9309823e-05, step = 31014 (5.347 sec) 2021-10-19 08:37:48,138 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.095 INFO:tensorflow:epoch = 80.41450777202073, learning_rate = 0.00049999997, loss = 2.5670568e-05, step = 31040 (5.205 sec) 2021-10-19 08:37:51,345 [INFO] tensorflow: epoch = 80.41450777202073, learning_rate = 0.00049999997, loss = 2.5670568e-05, step = 31040 (5.205 sec) INFO:tensorflow:global_step/sec: 5.00504 2021-10-19 08:37:52,531 [INFO] tensorflow: global_step/sec: 5.00504 2021-10-19 08:37:53,115 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.095 INFO:tensorflow:epoch = 80.48445595854922, learning_rate = 0.00049999997, loss = 3.8076563e-05, step = 31067 (5.342 sec) 2021-10-19 08:37:56,686 [INFO] tensorflow: epoch = 80.48445595854922, learning_rate = 0.00049999997, loss = 3.8076563e-05, step = 31067 (5.342 sec) 2021-10-19 08:37:58,068 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.194 INFO:tensorflow:global_step/sec: 5.04183 2021-10-19 08:38:00,068 [INFO] tensorflow: global_step/sec: 5.04183 INFO:tensorflow:epoch = 80.55440414507773, learning_rate = 0.00049999997, loss = 3.1991516e-05, step = 31094 (5.326 sec) 2021-10-19 08:38:02,013 [INFO] tensorflow: epoch = 80.55440414507773, learning_rate = 0.00049999997, loss = 3.1991516e-05, step = 31094 (5.326 sec) 2021-10-19 08:38:02,993 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.304 INFO:tensorflow:epoch = 80.62435233160622, learning_rate = 0.00049999997, loss = 3.3241045e-05, step = 31121 (5.355 sec) 2021-10-19 08:38:07,368 [INFO] tensorflow: epoch = 80.62435233160622, learning_rate = 0.00049999997, loss = 3.3241045e-05, step = 31121 (5.355 sec) INFO:tensorflow:global_step/sec: 5.06502 2021-10-19 08:38:07,570 [INFO] tensorflow: global_step/sec: 5.06502 2021-10-19 08:38:07,985 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.033 INFO:tensorflow:epoch = 80.69430051813472, learning_rate = 0.00049999997, loss = 8.0826885e-06, step = 31148 (5.347 sec) 2021-10-19 08:38:12,714 [INFO] tensorflow: epoch = 80.69430051813472, learning_rate = 0.00049999997, loss = 8.0826885e-06, step = 31148 (5.347 sec) 2021-10-19 08:38:12,907 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.318 INFO:tensorflow:global_step/sec: 5.04758 2021-10-19 08:38:15,099 [INFO] tensorflow: global_step/sec: 5.04758 2021-10-19 08:38:17,880 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.112 INFO:tensorflow:epoch = 80.76424870466322, learning_rate = 0.00049999997, loss = 4.362942e-05, step = 31175 (5.359 sec) 2021-10-19 08:38:18,073 [INFO] tensorflow: epoch = 80.76424870466322, learning_rate = 0.00049999997, loss = 4.362942e-05, step = 31175 (5.359 sec) INFO:tensorflow:global_step/sec: 4.98968 2021-10-19 08:38:22,714 [INFO] tensorflow: global_step/sec: 4.98968 2021-10-19 08:38:22,919 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.846 INFO:tensorflow:epoch = 80.83160621761658, learning_rate = 0.00049999997, loss = 1.0486696e-05, step = 31201 (5.260 sec) 2021-10-19 08:38:23,333 [INFO] tensorflow: epoch = 80.83160621761658, learning_rate = 0.00049999997, loss = 1.0486696e-05, step = 31201 (5.260 sec) 2021-10-19 08:38:27,989 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.723 INFO:tensorflow:epoch = 80.89896373056995, learning_rate = 0.00049999997, loss = 8.586645e-06, step = 31227 (5.288 sec) 2021-10-19 08:38:28,621 [INFO] tensorflow: epoch = 80.89896373056995, learning_rate = 0.00049999997, loss = 8.586645e-06, step = 31227 (5.288 sec) INFO:tensorflow:global_step/sec: 4.92922 2021-10-19 08:38:30,424 [INFO] tensorflow: global_step/sec: 4.92922 2021-10-19 08:38:32,954 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.143 INFO:tensorflow:epoch = 80.96891191709845, learning_rate = 0.00049999997, loss = 2.1264448e-05, step = 31254 (5.329 sec) 2021-10-19 08:38:33,950 [INFO] tensorflow: epoch = 80.96891191709845, learning_rate = 0.00049999997, loss = 2.1264448e-05, step = 31254 (5.329 sec) 2021-10-19 08:38:36,391 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 81/120: loss: 0.00002 learning rate: 0.00050 Time taken: 0:01:16.801500 ETA: 0:49:55.258503 INFO:tensorflow:global_step/sec: 5.04343 2021-10-19 08:38:37,958 [INFO] tensorflow: global_step/sec: 5.04343 2021-10-19 08:38:37,959 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.983 INFO:tensorflow:epoch = 81.03886010362694, learning_rate = 0.00049999997, loss = 1.1693864e-05, step = 31281 (5.379 sec) 2021-10-19 08:38:39,329 [INFO] tensorflow: epoch = 81.03886010362694, learning_rate = 0.00049999997, loss = 1.1693864e-05, step = 31281 (5.379 sec) 2021-10-19 08:38:42,877 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.336 INFO:tensorflow:epoch = 81.10880829015544, learning_rate = 0.00049999997, loss = 1.9283485e-05, step = 31308 (5.309 sec) 2021-10-19 08:38:44,638 [INFO] tensorflow: epoch = 81.10880829015544, learning_rate = 0.00049999997, loss = 1.9283485e-05, step = 31308 (5.309 sec) INFO:tensorflow:global_step/sec: 5.07467 2021-10-19 08:38:45,446 [INFO] tensorflow: global_step/sec: 5.07467 2021-10-19 08:38:47,838 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.159 INFO:tensorflow:epoch = 81.17616580310882, learning_rate = 0.00049999997, loss = 2.5725101e-05, step = 31334 (5.195 sec) 2021-10-19 08:38:49,833 [INFO] tensorflow: epoch = 81.17616580310882, learning_rate = 0.00049999997, loss = 2.5725101e-05, step = 31334 (5.195 sec) 2021-10-19 08:38:52,764 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.304 INFO:tensorflow:global_step/sec: 5.04462 2021-10-19 08:38:52,979 [INFO] tensorflow: global_step/sec: 5.04462 INFO:tensorflow:epoch = 81.24611398963731, learning_rate = 0.00049999997, loss = 9.017756e-06, step = 31361 (5.297 sec) 2021-10-19 08:38:55,131 [INFO] tensorflow: epoch = 81.24611398963731, learning_rate = 0.00049999997, loss = 9.017756e-06, step = 31361 (5.297 sec) 2021-10-19 08:38:57,701 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.257 INFO:tensorflow:epoch = 81.3160621761658, learning_rate = 0.00049999997, loss = 2.1548465e-05, step = 31388 (5.346 sec) 2021-10-19 08:39:00,476 [INFO] tensorflow: epoch = 81.3160621761658, learning_rate = 0.00049999997, loss = 2.1548465e-05, step = 31388 (5.346 sec) INFO:tensorflow:global_step/sec: 5.06782 2021-10-19 08:39:00,477 [INFO] tensorflow: global_step/sec: 5.06782 2021-10-19 08:39:02,681 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.084 INFO:tensorflow:epoch = 81.38601036269431, learning_rate = 0.00049999997, loss = 2.7598337e-05, step = 31415 (5.322 sec) 2021-10-19 08:39:05,798 [INFO] tensorflow: epoch = 81.38601036269431, learning_rate = 0.00049999997, loss = 2.7598337e-05, step = 31415 (5.322 sec) 2021-10-19 08:39:07,589 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.377 INFO:tensorflow:global_step/sec: 5.05754 2021-10-19 08:39:07,991 [INFO] tensorflow: global_step/sec: 5.05754 INFO:tensorflow:epoch = 81.45336787564767, learning_rate = 0.00049999997, loss = 1.3084002e-05, step = 31441 (5.225 sec) 2021-10-19 08:39:11,023 [INFO] tensorflow: epoch = 81.45336787564767, learning_rate = 0.00049999997, loss = 1.3084002e-05, step = 31441 (5.225 sec) 2021-10-19 08:39:12,621 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.875 INFO:tensorflow:global_step/sec: 5.00551 2021-10-19 08:39:15,583 [INFO] tensorflow: global_step/sec: 5.00551 INFO:tensorflow:epoch = 81.52331606217616, learning_rate = 0.00049999997, loss = 9.742768e-06, step = 31468 (5.348 sec) 2021-10-19 08:39:16,371 [INFO] tensorflow: epoch = 81.52331606217616, learning_rate = 0.00049999997, loss = 9.742768e-06, step = 31468 (5.348 sec) 2021-10-19 08:39:17,561 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.245 INFO:tensorflow:epoch = 81.59067357512954, learning_rate = 0.00049999997, loss = 2.76319e-05, step = 31494 (5.194 sec) 2021-10-19 08:39:21,566 [INFO] tensorflow: epoch = 81.59067357512954, learning_rate = 0.00049999997, loss = 2.76319e-05, step = 31494 (5.194 sec) 2021-10-19 08:39:22,547 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.056 INFO:tensorflow:global_step/sec: 5.01804 2021-10-19 08:39:23,155 [INFO] tensorflow: global_step/sec: 5.01804 INFO:tensorflow:epoch = 81.66062176165804, learning_rate = 0.00049999997, loss = 1.9626124e-05, step = 31521 (5.380 sec) 2021-10-19 08:39:26,946 [INFO] tensorflow: epoch = 81.66062176165804, learning_rate = 0.00049999997, loss = 1.9626124e-05, step = 31521 (5.380 sec) 2021-10-19 08:39:27,530 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.069 INFO:tensorflow:global_step/sec: 5.03018 2021-10-19 08:39:30,710 [INFO] tensorflow: global_step/sec: 5.03018 INFO:tensorflow:epoch = 81.7279792746114, learning_rate = 0.00049999997, loss = 1.8650355e-05, step = 31547 (5.191 sec) 2021-10-19 08:39:32,137 [INFO] tensorflow: epoch = 81.7279792746114, learning_rate = 0.00049999997, loss = 1.8650355e-05, step = 31547 (5.191 sec) 2021-10-19 08:39:32,582 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.798 INFO:tensorflow:epoch = 81.79533678756476, learning_rate = 0.00049999997, loss = 2.609095e-05, step = 31573 (5.215 sec) 2021-10-19 08:39:37,352 [INFO] tensorflow: epoch = 81.79533678756476, learning_rate = 0.00049999997, loss = 2.609095e-05, step = 31573 (5.215 sec) 2021-10-19 08:39:37,562 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.081 INFO:tensorflow:global_step/sec: 4.9752 2021-10-19 08:39:38,348 [INFO] tensorflow: global_step/sec: 4.9752 2021-10-19 08:39:42,564 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.994 INFO:tensorflow:epoch = 81.86528497409327, learning_rate = 0.00049999997, loss = 2.6484167e-05, step = 31600 (5.408 sec) 2021-10-19 08:39:42,760 [INFO] tensorflow: epoch = 81.86528497409327, learning_rate = 0.00049999997, loss = 2.6484167e-05, step = 31600 (5.408 sec) INFO:tensorflow:global_step/sec: 4.9848 2021-10-19 08:39:45,971 [INFO] tensorflow: global_step/sec: 4.9848 2021-10-19 08:39:47,546 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.072 INFO:tensorflow:epoch = 81.93523316062176, learning_rate = 0.00049999997, loss = 1.491538e-05, step = 31627 (5.397 sec) 2021-10-19 08:39:48,156 [INFO] tensorflow: epoch = 81.93523316062176, learning_rate = 0.00049999997, loss = 1.491538e-05, step = 31627 (5.397 sec) 2021-10-19 08:39:52,529 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.072 2021-10-19 08:39:53,131 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 82/120: loss: 0.00002 learning rate: 0.00050 Time taken: 0:01:16.739040 ETA: 0:48:36.083516 INFO:tensorflow:epoch = 82.00518134715026, learning_rate = 0.00049999997, loss = 2.3185778e-05, step = 31654 (5.367 sec) 2021-10-19 08:39:53,523 [INFO] tensorflow: epoch = 82.00518134715026, learning_rate = 0.00049999997, loss = 2.3185778e-05, step = 31654 (5.367 sec) INFO:tensorflow:global_step/sec: 5.0305 2021-10-19 08:39:53,525 [INFO] tensorflow: global_step/sec: 5.0305 2021-10-19 08:39:57,503 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.105 INFO:tensorflow:epoch = 82.07512953367876, learning_rate = 0.00049999997, loss = 2.7050935e-05, step = 31681 (5.398 sec) 2021-10-19 08:39:58,921 [INFO] tensorflow: epoch = 82.07512953367876, learning_rate = 0.00049999997, loss = 2.7050935e-05, step = 31681 (5.398 sec) INFO:tensorflow:global_step/sec: 5.01283 2021-10-19 08:40:01,105 [INFO] tensorflow: global_step/sec: 5.01283 2021-10-19 08:40:02,502 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.005 INFO:tensorflow:epoch = 82.14507772020725, learning_rate = 0.00049999997, loss = 2.1203597e-05, step = 31708 (5.359 sec) 2021-10-19 08:40:04,280 [INFO] tensorflow: epoch = 82.14507772020725, learning_rate = 0.00049999997, loss = 2.1203597e-05, step = 31708 (5.359 sec) 2021-10-19 08:40:07,446 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.230 INFO:tensorflow:global_step/sec: 5.00515 2021-10-19 08:40:08,697 [INFO] tensorflow: global_step/sec: 5.00515 INFO:tensorflow:epoch = 82.21243523316062, learning_rate = 0.00049999997, loss = 1.4725687e-05, step = 31734 (5.199 sec) 2021-10-19 08:40:09,479 [INFO] tensorflow: epoch = 82.21243523316062, learning_rate = 0.00049999997, loss = 1.4725687e-05, step = 31734 (5.199 sec) 2021-10-19 08:40:12,440 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.026 INFO:tensorflow:epoch = 82.28238341968913, learning_rate = 0.00049999997, loss = 1.2380275e-05, step = 31761 (5.325 sec) 2021-10-19 08:40:14,805 [INFO] tensorflow: epoch = 82.28238341968913, learning_rate = 0.00049999997, loss = 1.2380275e-05, step = 31761 (5.325 sec) INFO:tensorflow:global_step/sec: 5.06332 2021-10-19 08:40:16,202 [INFO] tensorflow: global_step/sec: 5.06332 2021-10-19 08:40:17,362 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.320 INFO:tensorflow:epoch = 82.35233160621762, learning_rate = 0.00049999997, loss = 8.866864e-06, step = 31788 (5.387 sec) 2021-10-19 08:40:20,192 [INFO] tensorflow: epoch = 82.35233160621762, learning_rate = 0.00049999997, loss = 8.866864e-06, step = 31788 (5.387 sec) 2021-10-19 08:40:22,415 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.791 INFO:tensorflow:global_step/sec: 5.00645 2021-10-19 08:40:23,793 [INFO] tensorflow: global_step/sec: 5.00645 INFO:tensorflow:epoch = 82.42227979274611, learning_rate = 0.00049999997, loss = 3.1593398e-05, step = 31815 (5.391 sec) 2021-10-19 08:40:25,583 [INFO] tensorflow: epoch = 82.42227979274611, learning_rate = 0.00049999997, loss = 3.1593398e-05, step = 31815 (5.391 sec) 2021-10-19 08:40:27,370 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.182 INFO:tensorflow:epoch = 82.49222797927462, learning_rate = 0.00049999997, loss = 2.6475996e-05, step = 31842 (5.349 sec) 2021-10-19 08:40:30,932 [INFO] tensorflow: epoch = 82.49222797927462, learning_rate = 0.00049999997, loss = 2.6475996e-05, step = 31842 (5.349 sec) INFO:tensorflow:global_step/sec: 5.04785 2021-10-19 08:40:31,320 [INFO] tensorflow: global_step/sec: 5.04785 2021-10-19 08:40:32,324 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.190 INFO:tensorflow:epoch = 82.56217616580311, learning_rate = 0.00049999997, loss = 2.4441262e-05, step = 31869 (5.398 sec) 2021-10-19 08:40:36,330 [INFO] tensorflow: epoch = 82.56217616580311, learning_rate = 0.00049999997, loss = 2.4441262e-05, step = 31869 (5.398 sec) 2021-10-19 08:40:37,317 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.031 INFO:tensorflow:global_step/sec: 5.01052 2021-10-19 08:40:38,905 [INFO] tensorflow: global_step/sec: 5.01052 INFO:tensorflow:epoch = 82.63212435233162, learning_rate = 0.00049999997, loss = 7.838929e-06, step = 31896 (5.379 sec) 2021-10-19 08:40:41,709 [INFO] tensorflow: epoch = 82.63212435233162, learning_rate = 0.00049999997, loss = 7.838929e-06, step = 31896 (5.379 sec) 2021-10-19 08:40:42,302 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.061 INFO:tensorflow:global_step/sec: 4.9941 2021-10-19 08:40:46,513 [INFO] tensorflow: global_step/sec: 4.9941 INFO:tensorflow:epoch = 82.69948186528498, learning_rate = 0.00049999997, loss = 7.658991e-06, step = 31922 (5.199 sec) 2021-10-19 08:40:46,908 [INFO] tensorflow: epoch = 82.69948186528498, learning_rate = 0.00049999997, loss = 7.658991e-06, step = 31922 (5.199 sec) 2021-10-19 08:40:47,334 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.872 INFO:tensorflow:epoch = 82.76683937823834, learning_rate = 0.00049999997, loss = 3.044551e-05, step = 31948 (5.222 sec) 2021-10-19 08:40:52,130 [INFO] tensorflow: epoch = 82.76683937823834, learning_rate = 0.00049999997, loss = 3.044551e-05, step = 31948 (5.222 sec) 2021-10-19 08:40:52,325 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.040 INFO:tensorflow:global_step/sec: 5.00682 2021-10-19 08:40:54,103 [INFO] tensorflow: global_step/sec: 5.00682 INFO:tensorflow:epoch = 82.83419689119171, learning_rate = 0.00049999997, loss = 4.605824e-05, step = 31974 (5.207 sec) 2021-10-19 08:40:57,337 [INFO] tensorflow: epoch = 82.83419689119171, learning_rate = 0.00049999997, loss = 4.605824e-05, step = 31974 (5.207 sec) 2021-10-19 08:40:57,338 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.949 INFO:tensorflow:global_step/sec: 4.92972 2021-10-19 08:41:01,811 [INFO] tensorflow: global_step/sec: 4.92972 2021-10-19 08:41:02,403 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.745 INFO:tensorflow:epoch = 82.90155440414507, learning_rate = 0.00049999997, loss = 1.8905772e-05, step = 32000 (5.267 sec) 2021-10-19 08:41:02,604 [INFO] tensorflow: epoch = 82.90155440414507, learning_rate = 0.00049999997, loss = 1.8905772e-05, step = 32000 (5.267 sec) 2021-10-19 08:41:07,370 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.135 INFO:tensorflow:epoch = 82.97150259067358, learning_rate = 0.00049999997, loss = 2.7416363e-05, step = 32027 (5.350 sec) 2021-10-19 08:41:07,954 [INFO] tensorflow: epoch = 82.97150259067358, learning_rate = 0.00049999997, loss = 2.7416363e-05, step = 32027 (5.350 sec) INFO:tensorflow:global_step/sec: 5.02364 2021-10-19 08:41:09,376 [INFO] tensorflow: global_step/sec: 5.02364 2021-10-19 08:41:10,175 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 83/120: loss: 0.00003 learning rate: 0.00050 Time taken: 0:01:17.043188 ETA: 0:47:30.597968 2021-10-19 08:41:12,369 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.005 INFO:tensorflow:epoch = 83.03886010362694, learning_rate = 0.00049999997, loss = 2.2931836e-05, step = 32053 (5.227 sec) 2021-10-19 08:41:13,181 [INFO] tensorflow: epoch = 83.03886010362694, learning_rate = 0.00049999997, loss = 2.2931836e-05, step = 32053 (5.227 sec) INFO:tensorflow:global_step/sec: 5.06199 2021-10-19 08:41:16,883 [INFO] tensorflow: global_step/sec: 5.06199 2021-10-19 08:41:17,280 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.367 INFO:tensorflow:epoch = 83.10880829015544, learning_rate = 0.00049999997, loss = 1.445386e-05, step = 32080 (5.290 sec) 2021-10-19 08:41:18,471 [INFO] tensorflow: epoch = 83.10880829015544, learning_rate = 0.00049999997, loss = 1.445386e-05, step = 32080 (5.290 sec) 2021-10-19 08:41:22,288 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.967 INFO:tensorflow:epoch = 83.17616580310882, learning_rate = 0.00049999997, loss = 7.744158e-06, step = 32106 (5.225 sec) 2021-10-19 08:41:23,696 [INFO] tensorflow: epoch = 83.17616580310882, learning_rate = 0.00049999997, loss = 7.744158e-06, step = 32106 (5.225 sec) INFO:tensorflow:global_step/sec: 5.00704 2021-10-19 08:41:24,472 [INFO] tensorflow: global_step/sec: 5.00704 2021-10-19 08:41:27,247 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.169 INFO:tensorflow:epoch = 83.24611398963731, learning_rate = 0.00049999997, loss = 4.124321e-05, step = 32133 (5.375 sec) 2021-10-19 08:41:29,071 [INFO] tensorflow: epoch = 83.24611398963731, learning_rate = 0.00049999997, loss = 4.124321e-05, step = 32133 (5.375 sec) INFO:tensorflow:global_step/sec: 4.9829 2021-10-19 08:41:32,098 [INFO] tensorflow: global_step/sec: 4.9829 2021-10-19 08:41:32,293 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.817 INFO:tensorflow:epoch = 83.31347150259067, learning_rate = 0.00049999997, loss = 1.4868617e-05, step = 32159 (5.267 sec) 2021-10-19 08:41:34,339 [INFO] tensorflow: epoch = 83.31347150259067, learning_rate = 0.00049999997, loss = 1.4868617e-05, step = 32159 (5.267 sec) 2021-10-19 08:41:37,345 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.797 INFO:tensorflow:epoch = 83.38341968911918, learning_rate = 0.00049999997, loss = 3.890882e-05, step = 32186 (5.351 sec) 2021-10-19 08:41:39,689 [INFO] tensorflow: epoch = 83.38341968911918, learning_rate = 0.00049999997, loss = 3.890882e-05, step = 32186 (5.351 sec) INFO:tensorflow:global_step/sec: 5.00435 2021-10-19 08:41:39,691 [INFO] tensorflow: global_step/sec: 5.00435 2021-10-19 08:41:42,267 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.316 INFO:tensorflow:epoch = 83.45336787564767, learning_rate = 0.00049999997, loss = 2.8136052e-05, step = 32213 (5.334 sec) 2021-10-19 08:41:45,023 [INFO] tensorflow: epoch = 83.45336787564767, learning_rate = 0.00049999997, loss = 2.8136052e-05, step = 32213 (5.334 sec) INFO:tensorflow:global_step/sec: 5.07729 2021-10-19 08:41:47,176 [INFO] tensorflow: global_step/sec: 5.07729 2021-10-19 08:41:47,177 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.368 INFO:tensorflow:epoch = 83.52331606217616, learning_rate = 0.00049999997, loss = 3.4376568e-05, step = 32240 (5.306 sec) 2021-10-19 08:41:50,329 [INFO] tensorflow: epoch = 83.52331606217616, learning_rate = 0.00049999997, loss = 3.4376568e-05, step = 32240 (5.306 sec) 2021-10-19 08:41:52,102 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.310 INFO:tensorflow:global_step/sec: 5.02655 2021-10-19 08:41:54,736 [INFO] tensorflow: global_step/sec: 5.02655 INFO:tensorflow:epoch = 83.59067357512954, learning_rate = 0.00049999997, loss = 2.1361568e-05, step = 32266 (5.208 sec) 2021-10-19 08:41:55,537 [INFO] tensorflow: epoch = 83.59067357512954, learning_rate = 0.00049999997, loss = 2.1361568e-05, step = 32266 (5.208 sec) 2021-10-19 08:41:57,140 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.848 INFO:tensorflow:epoch = 83.66062176165804, learning_rate = 0.00049999997, loss = 1.8083196e-05, step = 32293 (5.347 sec) 2021-10-19 08:42:00,884 [INFO] tensorflow: epoch = 83.66062176165804, learning_rate = 0.00049999997, loss = 1.8083196e-05, step = 32293 (5.347 sec) 2021-10-19 08:42:02,105 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.144 INFO:tensorflow:global_step/sec: 5.02282 2021-10-19 08:42:02,301 [INFO] tensorflow: global_step/sec: 5.02282 INFO:tensorflow:epoch = 83.73056994818653, learning_rate = 0.00049999997, loss = 2.5778798e-05, step = 32320 (5.420 sec) 2021-10-19 08:42:06,303 [INFO] tensorflow: epoch = 83.73056994818653, learning_rate = 0.00049999997, loss = 2.5778798e-05, step = 32320 (5.420 sec) 2021-10-19 08:42:07,094 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.044 INFO:tensorflow:global_step/sec: 5.03171 2021-10-19 08:42:09,853 [INFO] tensorflow: global_step/sec: 5.03171 INFO:tensorflow:epoch = 83.80051813471503, learning_rate = 0.00049999997, loss = 2.131399e-05, step = 32347 (5.368 sec) 2021-10-19 08:42:11,672 [INFO] tensorflow: epoch = 83.80051813471503, learning_rate = 0.00049999997, loss = 2.131399e-05, step = 32347 (5.368 sec) 2021-10-19 08:42:12,069 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.105 INFO:tensorflow:epoch = 83.87046632124353, learning_rate = 0.00049999997, loss = 2.1168962e-05, step = 32374 (5.337 sec) 2021-10-19 08:42:17,009 [INFO] tensorflow: epoch = 83.87046632124353, learning_rate = 0.00049999997, loss = 2.1168962e-05, step = 32374 (5.337 sec) 2021-10-19 08:42:17,009 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.242 INFO:tensorflow:global_step/sec: 5.02491 2021-10-19 08:42:17,415 [INFO] tensorflow: global_step/sec: 5.02491 2021-10-19 08:42:21,991 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.074 INFO:tensorflow:epoch = 83.94041450777202, learning_rate = 0.00049999997, loss = 1.5795673e-05, step = 32401 (5.394 sec) 2021-10-19 08:42:22,403 [INFO] tensorflow: epoch = 83.94041450777202, learning_rate = 0.00049999997, loss = 1.5795673e-05, step = 32401 (5.394 sec) INFO:tensorflow:global_step/sec: 5.00117 2021-10-19 08:42:25,014 [INFO] tensorflow: global_step/sec: 5.00117 2021-10-19 08:42:27,064 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 84/120: loss: 0.00003 learning rate: 0.00050 Time taken: 0:01:16.859070 ETA: 0:46:06.926531 2021-10-19 08:42:27,064 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.715 INFO:tensorflow:epoch = 84.0077720207254, learning_rate = 0.0004995029, loss = 2.1026686e-05, step = 32427 (5.260 sec) 2021-10-19 08:42:27,663 [INFO] tensorflow: epoch = 84.0077720207254, learning_rate = 0.0004995029, loss = 2.1026686e-05, step = 32427 (5.260 sec) 2021-10-19 08:42:32,067 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.991 INFO:tensorflow:global_step/sec: 4.97763 2021-10-19 08:42:32,648 [INFO] tensorflow: global_step/sec: 4.97763 INFO:tensorflow:epoch = 84.07772020725389, learning_rate = 0.00049505354, loss = 2.375271e-05, step = 32454 (5.397 sec) 2021-10-19 08:42:33,060 [INFO] tensorflow: epoch = 84.07772020725389, learning_rate = 0.00049505354, loss = 2.375271e-05, step = 32454 (5.397 sec) 2021-10-19 08:42:37,006 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.247 INFO:tensorflow:epoch = 84.14766839378238, learning_rate = 0.00049064343, loss = 3.078113e-05, step = 32481 (5.339 sec) 2021-10-19 08:42:38,400 [INFO] tensorflow: epoch = 84.14766839378238, learning_rate = 0.00049064343, loss = 3.078113e-05, step = 32481 (5.339 sec) INFO:tensorflow:global_step/sec: 5.02325 2021-10-19 08:42:40,213 [INFO] tensorflow: global_step/sec: 5.02325 2021-10-19 08:42:42,002 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.017 INFO:tensorflow:epoch = 84.21761658031089, learning_rate = 0.00048627303, loss = 5.5352753e-05, step = 32508 (5.385 sec) 2021-10-19 08:42:43,785 [INFO] tensorflow: epoch = 84.21761658031089, learning_rate = 0.00048627303, loss = 5.5352753e-05, step = 32508 (5.385 sec) 2021-10-19 08:42:46,880 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.501 INFO:tensorflow:global_step/sec: 5.10676 2021-10-19 08:42:47,654 [INFO] tensorflow: global_step/sec: 5.10676 INFO:tensorflow:epoch = 84.28756476683938, learning_rate = 0.0004819411, loss = 1.9361358e-05, step = 32535 (5.285 sec) 2021-10-19 08:42:49,070 [INFO] tensorflow: epoch = 84.28756476683938, learning_rate = 0.0004819411, loss = 1.9361358e-05, step = 32535 (5.285 sec) 2021-10-19 08:42:51,826 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.222 INFO:tensorflow:epoch = 84.35751295336787, learning_rate = 0.00047764822, loss = 2.2577939e-05, step = 32562 (5.336 sec) 2021-10-19 08:42:54,406 [INFO] tensorflow: epoch = 84.35751295336787, learning_rate = 0.00047764822, loss = 2.2577939e-05, step = 32562 (5.336 sec) INFO:tensorflow:global_step/sec: 5.01051 2021-10-19 08:42:55,238 [INFO] tensorflow: global_step/sec: 5.01051 2021-10-19 08:42:56,799 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.111 INFO:tensorflow:epoch = 84.42746113989638, learning_rate = 0.00047339316, loss = 1.2027985e-05, step = 32589 (5.344 sec) 2021-10-19 08:42:59,750 [INFO] tensorflow: epoch = 84.42746113989638, learning_rate = 0.00047339316, loss = 1.2027985e-05, step = 32589 (5.344 sec) 2021-10-19 08:43:01,747 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.211 INFO:tensorflow:global_step/sec: 5.06171 2021-10-19 08:43:02,745 [INFO] tensorflow: global_step/sec: 5.06171 INFO:tensorflow:epoch = 84.49481865284974, learning_rate = 0.00046933172, loss = 1.5993828e-05, step = 32615 (5.233 sec) 2021-10-19 08:43:04,983 [INFO] tensorflow: epoch = 84.49481865284974, learning_rate = 0.00046933172, loss = 1.5993828e-05, step = 32615 (5.233 sec) 2021-10-19 08:43:06,843 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.627 INFO:tensorflow:epoch = 84.56217616580311, learning_rate = 0.00046530535, loss = 7.02891e-06, step = 32641 (5.237 sec) 2021-10-19 08:43:10,220 [INFO] tensorflow: epoch = 84.56217616580311, learning_rate = 0.00046530535, loss = 7.02891e-06, step = 32641 (5.237 sec) INFO:tensorflow:global_step/sec: 4.95314 2021-10-19 08:43:10,417 [INFO] tensorflow: global_step/sec: 4.95314 2021-10-19 08:43:11,831 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.051 INFO:tensorflow:epoch = 84.63212435233162, learning_rate = 0.0004611602, loss = 1.4767982e-05, step = 32668 (5.375 sec) 2021-10-19 08:43:15,595 [INFO] tensorflow: epoch = 84.63212435233162, learning_rate = 0.0004611602, loss = 1.4767982e-05, step = 32668 (5.375 sec) 2021-10-19 08:43:16,788 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.176 INFO:tensorflow:global_step/sec: 5.03586 2021-10-19 08:43:17,963 [INFO] tensorflow: global_step/sec: 5.03586 INFO:tensorflow:epoch = 84.70207253886011, learning_rate = 0.00045705223, loss = 1.790409e-05, step = 32695 (5.360 sec) 2021-10-19 08:43:20,955 [INFO] tensorflow: epoch = 84.70207253886011, learning_rate = 0.00045705223, loss = 1.790409e-05, step = 32695 (5.360 sec) 2021-10-19 08:43:21,766 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.090 INFO:tensorflow:global_step/sec: 5.03353 2021-10-19 08:43:25,512 [INFO] tensorflow: global_step/sec: 5.03353 INFO:tensorflow:epoch = 84.7720207253886, learning_rate = 0.0004529806, loss = 1.1206927e-05, step = 32722 (5.394 sec) 2021-10-19 08:43:26,349 [INFO] tensorflow: epoch = 84.7720207253886, learning_rate = 0.0004529806, loss = 1.1206927e-05, step = 32722 (5.394 sec) 2021-10-19 08:43:26,746 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.080 INFO:tensorflow:epoch = 84.8419689119171, learning_rate = 0.00044894568, loss = 2.619748e-05, step = 32749 (5.303 sec) 2021-10-19 08:43:31,652 [INFO] tensorflow: epoch = 84.8419689119171, learning_rate = 0.00044894568, loss = 2.619748e-05, step = 32749 (5.303 sec) 2021-10-19 08:43:31,653 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.384 INFO:tensorflow:global_step/sec: 5.04485 2021-10-19 08:43:33,045 [INFO] tensorflow: global_step/sec: 5.04485 2021-10-19 08:43:36,580 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.295 INFO:tensorflow:epoch = 84.9119170984456, learning_rate = 0.00044494632, loss = 3.5216006e-05, step = 32776 (5.336 sec) 2021-10-19 08:43:36,988 [INFO] tensorflow: epoch = 84.9119170984456, learning_rate = 0.00044494632, loss = 3.5216006e-05, step = 32776 (5.336 sec) INFO:tensorflow:global_step/sec: 5.04702 2021-10-19 08:43:40,574 [INFO] tensorflow: global_step/sec: 5.04702 2021-10-19 08:43:41,550 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.124 INFO:tensorflow:epoch = 84.98186528497409, learning_rate = 0.00044098299, loss = 1.5875585e-05, step = 32803 (5.368 sec) 2021-10-19 08:43:42,356 [INFO] tensorflow: epoch = 84.98186528497409, learning_rate = 0.00044098299, loss = 1.5875585e-05, step = 32803 (5.368 sec) 2021-10-19 08:43:43,736 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 85/120: loss: 0.00002 learning rate: 0.00044 Time taken: 0:01:16.698758 ETA: 0:44:44.456534 2021-10-19 08:43:46,484 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.269 INFO:tensorflow:epoch = 85.0518134715026, learning_rate = 0.0004370545, loss = 3.8203285e-05, step = 32830 (5.350 sec) 2021-10-19 08:43:47,706 [INFO] tensorflow: epoch = 85.0518134715026, learning_rate = 0.0004370545, loss = 3.8203285e-05, step = 32830 (5.350 sec) INFO:tensorflow:global_step/sec: 5.04041 2021-10-19 08:43:48,113 [INFO] tensorflow: global_step/sec: 5.04041 2021-10-19 08:43:51,472 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.051 INFO:tensorflow:epoch = 85.12176165803109, learning_rate = 0.00043316145, loss = 1.8835655e-05, step = 32857 (5.389 sec) 2021-10-19 08:43:53,095 [INFO] tensorflow: epoch = 85.12176165803109, learning_rate = 0.00043316145, loss = 1.8835655e-05, step = 32857 (5.389 sec) INFO:tensorflow:global_step/sec: 5.01716 2021-10-19 08:43:55,687 [INFO] tensorflow: global_step/sec: 5.01716 2021-10-19 08:43:56,467 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.021 INFO:tensorflow:epoch = 85.19170984455958, learning_rate = 0.00042930307, loss = 2.0191757e-05, step = 32884 (5.360 sec) 2021-10-19 08:43:58,455 [INFO] tensorflow: epoch = 85.19170984455958, learning_rate = 0.00042930307, loss = 2.0191757e-05, step = 32884 (5.360 sec) 2021-10-19 08:44:01,440 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.111 INFO:tensorflow:global_step/sec: 5.00799 2021-10-19 08:44:03,275 [INFO] tensorflow: global_step/sec: 5.00799 INFO:tensorflow:epoch = 85.25906735751296, learning_rate = 0.0004256195, loss = 3.8407903e-05, step = 32910 (5.230 sec) 2021-10-19 08:44:03,686 [INFO] tensorflow: epoch = 85.25906735751296, learning_rate = 0.0004256195, loss = 3.8407903e-05, step = 32910 (5.230 sec) 2021-10-19 08:44:06,433 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.029 INFO:tensorflow:epoch = 85.32901554404145, learning_rate = 0.00042182833, loss = 1.2123668e-05, step = 32937 (5.317 sec) 2021-10-19 08:44:09,003 [INFO] tensorflow: epoch = 85.32901554404145, learning_rate = 0.00042182833, loss = 1.2123668e-05, step = 32937 (5.317 sec) INFO:tensorflow:global_step/sec: 5.04542 2021-10-19 08:44:10,806 [INFO] tensorflow: global_step/sec: 5.04542 2021-10-19 08:44:11,403 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.124 INFO:tensorflow:epoch = 85.39896373056995, learning_rate = 0.00041807047, loss = 1.6533177e-05, step = 32964 (5.378 sec) 2021-10-19 08:44:14,381 [INFO] tensorflow: epoch = 85.39896373056995, learning_rate = 0.00041807047, loss = 1.6533177e-05, step = 32964 (5.378 sec) 2021-10-19 08:44:16,372 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.124 INFO:tensorflow:global_step/sec: 5.00342 2021-10-19 08:44:18,401 [INFO] tensorflow: global_step/sec: 5.00342 INFO:tensorflow:epoch = 85.46632124352332, learning_rate = 0.0004144839, loss = 1.337018e-05, step = 32990 (5.241 sec) 2021-10-19 08:44:19,622 [INFO] tensorflow: epoch = 85.46632124352332, learning_rate = 0.0004144839, loss = 1.337018e-05, step = 32990 (5.241 sec) 2021-10-19 08:44:21,389 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.936 INFO:tensorflow:epoch = 85.53626943005182, learning_rate = 0.00041079166, loss = 3.5780286e-05, step = 33017 (5.385 sec) 2021-10-19 08:44:25,008 [INFO] tensorflow: epoch = 85.53626943005182, learning_rate = 0.00041079166, loss = 3.5780286e-05, step = 33017 (5.385 sec) INFO:tensorflow:global_step/sec: 4.99335 2021-10-19 08:44:26,011 [INFO] tensorflow: global_step/sec: 4.99335 2021-10-19 08:44:26,411 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.912 INFO:tensorflow:epoch = 85.60362694300518, learning_rate = 0.00040726713, loss = 1.0573807e-05, step = 33043 (5.225 sec) 2021-10-19 08:44:30,233 [INFO] tensorflow: epoch = 85.60362694300518, learning_rate = 0.00040726713, loss = 1.0573807e-05, step = 33043 (5.225 sec) 2021-10-19 08:44:31,406 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.024 INFO:tensorflow:global_step/sec: 5.02186 2021-10-19 08:44:33,578 [INFO] tensorflow: global_step/sec: 5.02186 INFO:tensorflow:epoch = 85.67098445595855, learning_rate = 0.00040377319, loss = 1.2158526e-05, step = 33069 (5.199 sec) 2021-10-19 08:44:35,432 [INFO] tensorflow: epoch = 85.67098445595855, learning_rate = 0.00040377319, loss = 1.2158526e-05, step = 33069 (5.199 sec) 2021-10-19 08:44:36,478 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.718 INFO:tensorflow:epoch = 85.73834196891193, learning_rate = 0.00040030904, loss = 1.034007e-05, step = 33095 (5.272 sec) 2021-10-19 08:44:40,705 [INFO] tensorflow: epoch = 85.73834196891193, learning_rate = 0.00040030904, loss = 1.034007e-05, step = 33095 (5.272 sec) INFO:tensorflow:global_step/sec: 4.91497 2021-10-19 08:44:41,310 [INFO] tensorflow: global_step/sec: 4.91497 2021-10-19 08:44:41,524 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.818 INFO:tensorflow:epoch = 85.80569948186529, learning_rate = 0.00039687462, loss = 8.860559e-06, step = 33121 (5.294 sec) 2021-10-19 08:44:45,999 [INFO] tensorflow: epoch = 85.80569948186529, learning_rate = 0.00039687462, loss = 8.860559e-06, step = 33121 (5.294 sec) 2021-10-19 08:44:46,596 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.716 INFO:tensorflow:global_step/sec: 4.95009 2021-10-19 08:44:48,986 [INFO] tensorflow: global_step/sec: 4.95009 INFO:tensorflow:epoch = 85.87564766839378, learning_rate = 0.0003933391, loss = 1.5967034e-05, step = 33148 (5.333 sec) 2021-10-19 08:44:51,332 [INFO] tensorflow: epoch = 85.87564766839378, learning_rate = 0.0003933391, loss = 1.5967034e-05, step = 33148 (5.333 sec) 2021-10-19 08:44:51,529 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.276 INFO:tensorflow:global_step/sec: 5.04727 2021-10-19 08:44:56,515 [INFO] tensorflow: global_step/sec: 5.04727 2021-10-19 08:44:56,516 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.051 INFO:tensorflow:epoch = 85.94559585492229, learning_rate = 0.00038983545, loss = 1.6046173e-05, step = 33175 (5.386 sec) 2021-10-19 08:44:56,718 [INFO] tensorflow: epoch = 85.94559585492229, learning_rate = 0.00038983545, loss = 1.6046173e-05, step = 33175 (5.386 sec) 2021-10-19 08:45:00,880 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 86/120: loss: 0.00002 learning rate: 0.00039 Time taken: 0:01:17.141183 ETA: 0:43:42.800235 2021-10-19 08:45:01,494 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.092 INFO:tensorflow:epoch = 86.01554404145078, learning_rate = 0.00038636263, loss = 2.2692207e-05, step = 33202 (5.367 sec) 2021-10-19 08:45:02,085 [INFO] tensorflow: epoch = 86.01554404145078, learning_rate = 0.00038636263, loss = 2.2692207e-05, step = 33202 (5.367 sec) INFO:tensorflow:global_step/sec: 5.01212 2021-10-19 08:45:04,097 [INFO] tensorflow: global_step/sec: 5.01212 2021-10-19 08:45:06,573 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.688 INFO:tensorflow:epoch = 86.08290155440415, learning_rate = 0.00038304785, loss = 3.1984782e-05, step = 33228 (5.325 sec) 2021-10-19 08:45:07,409 [INFO] tensorflow: epoch = 86.08290155440415, learning_rate = 0.00038304785, loss = 3.1984782e-05, step = 33228 (5.325 sec) 2021-10-19 08:45:11,598 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.903 INFO:tensorflow:global_step/sec: 4.93372 2021-10-19 08:45:11,799 [INFO] tensorflow: global_step/sec: 4.93372 INFO:tensorflow:epoch = 86.15284974093265, learning_rate = 0.00037963554, loss = 3.0987117e-05, step = 33255 (5.390 sec) 2021-10-19 08:45:12,800 [INFO] tensorflow: epoch = 86.15284974093265, learning_rate = 0.00037963554, loss = 3.0987117e-05, step = 33255 (5.390 sec) 2021-10-19 08:45:16,555 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.175 INFO:tensorflow:epoch = 86.22279792746114, learning_rate = 0.0003762539, loss = 1.664117e-05, step = 33282 (5.359 sec) 2021-10-19 08:45:18,159 [INFO] tensorflow: epoch = 86.22279792746114, learning_rate = 0.0003762539, loss = 1.664117e-05, step = 33282 (5.359 sec) INFO:tensorflow:global_step/sec: 5.0232 2021-10-19 08:45:19,364 [INFO] tensorflow: global_step/sec: 5.0232 2021-10-19 08:45:21,557 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.995 INFO:tensorflow:epoch = 86.29274611398964, learning_rate = 0.0003729021, loss = 3.594481e-05, step = 33309 (5.373 sec) 2021-10-19 08:45:23,532 [INFO] tensorflow: epoch = 86.29274611398964, learning_rate = 0.0003729021, loss = 3.594481e-05, step = 33309 (5.373 sec) 2021-10-19 08:45:26,475 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.336 INFO:tensorflow:global_step/sec: 5.06725 2021-10-19 08:45:26,863 [INFO] tensorflow: global_step/sec: 5.06725 INFO:tensorflow:epoch = 86.36269430051814, learning_rate = 0.0003695805, loss = 2.7221162e-05, step = 33336 (5.323 sec) 2021-10-19 08:45:28,855 [INFO] tensorflow: epoch = 86.36269430051814, learning_rate = 0.0003695805, loss = 2.7221162e-05, step = 33336 (5.323 sec) 2021-10-19 08:45:31,425 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.200 INFO:tensorflow:epoch = 86.43264248704664, learning_rate = 0.00036628847, loss = 2.6193078e-05, step = 33363 (5.315 sec) 2021-10-19 08:45:34,170 [INFO] tensorflow: epoch = 86.43264248704664, learning_rate = 0.00036628847, loss = 2.6193078e-05, step = 33363 (5.315 sec) INFO:tensorflow:global_step/sec: 5.06019 2021-10-19 08:45:34,373 [INFO] tensorflow: global_step/sec: 5.06019 2021-10-19 08:45:36,366 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.242 INFO:tensorflow:epoch = 86.50259067357513, learning_rate = 0.0003630254, loss = 3.3917328e-05, step = 33390 (5.372 sec) 2021-10-19 08:45:39,542 [INFO] tensorflow: epoch = 86.50259067357513, learning_rate = 0.0003630254, loss = 3.3917328e-05, step = 33390 (5.372 sec) 2021-10-19 08:45:41,354 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.051 INFO:tensorflow:global_step/sec: 5.01174 2021-10-19 08:45:41,955 [INFO] tensorflow: global_step/sec: 5.01174 INFO:tensorflow:epoch = 86.57253886010363, learning_rate = 0.00035979165, loss = 1.9486357e-05, step = 33417 (5.375 sec) 2021-10-19 08:45:44,917 [INFO] tensorflow: epoch = 86.57253886010363, learning_rate = 0.00035979165, loss = 1.9486357e-05, step = 33417 (5.375 sec) 2021-10-19 08:45:46,287 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.271 INFO:tensorflow:global_step/sec: 5.05332 2021-10-19 08:45:49,475 [INFO] tensorflow: global_step/sec: 5.05332 INFO:tensorflow:epoch = 86.64248704663213, learning_rate = 0.00035658645, loss = 7.549877e-06, step = 33444 (5.386 sec) 2021-10-19 08:45:50,303 [INFO] tensorflow: epoch = 86.64248704663213, learning_rate = 0.00035658645, loss = 7.549877e-06, step = 33444 (5.386 sec) 2021-10-19 08:45:51,286 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.009 INFO:tensorflow:epoch = 86.71243523316062, learning_rate = 0.00035341035, loss = 1.4125835e-05, step = 33471 (5.330 sec) 2021-10-19 08:45:55,633 [INFO] tensorflow: epoch = 86.71243523316062, learning_rate = 0.00035341035, loss = 1.4125835e-05, step = 33471 (5.330 sec) 2021-10-19 08:45:56,236 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.204 INFO:tensorflow:global_step/sec: 4.9865 2021-10-19 08:45:57,095 [INFO] tensorflow: global_step/sec: 4.9865 INFO:tensorflow:epoch = 86.779792746114, learning_rate = 0.00035037796, loss = 2.0470323e-05, step = 33497 (5.331 sec) 2021-10-19 08:46:00,963 [INFO] tensorflow: epoch = 86.779792746114, learning_rate = 0.00035037796, loss = 2.0470323e-05, step = 33497 (5.331 sec) 2021-10-19 08:46:01,371 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.474 INFO:tensorflow:global_step/sec: 4.96729 2021-10-19 08:46:04,745 [INFO] tensorflow: global_step/sec: 4.96729 INFO:tensorflow:epoch = 86.84974093264249, learning_rate = 0.00034725698, loss = 3.5858353e-05, step = 33524 (5.371 sec) 2021-10-19 08:46:06,334 [INFO] tensorflow: epoch = 86.84974093264249, learning_rate = 0.00034725698, loss = 3.5858353e-05, step = 33524 (5.371 sec) 2021-10-19 08:46:06,334 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.150 2021-10-19 08:46:11,372 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.850 INFO:tensorflow:epoch = 86.91709844559585, learning_rate = 0.0003442777, loss = 1.3469658e-05, step = 33550 (5.248 sec) 2021-10-19 08:46:11,582 [INFO] tensorflow: epoch = 86.91709844559585, learning_rate = 0.0003442777, loss = 1.3469658e-05, step = 33550 (5.248 sec) INFO:tensorflow:global_step/sec: 4.96903 2021-10-19 08:46:12,393 [INFO] tensorflow: global_step/sec: 4.96903 2021-10-19 08:46:16,349 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.097 INFO:tensorflow:epoch = 86.98704663212436, learning_rate = 0.00034121072, loss = 1.8174713e-05, step = 33577 (5.355 sec) 2021-10-19 08:46:16,937 [INFO] tensorflow: epoch = 86.98704663212436, learning_rate = 0.00034121072, loss = 1.8174713e-05, step = 33577 (5.355 sec) 2021-10-19 08:46:17,966 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 87/120: loss: 0.00001 learning rate: 0.00034 Time taken: 0:01:17.064117 ETA: 0:42:23.115852 INFO:tensorflow:global_step/sec: 5.00055 2021-10-19 08:46:19,992 [INFO] tensorflow: global_step/sec: 5.00055 2021-10-19 08:46:21,377 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.888 INFO:tensorflow:epoch = 87.05440414507773, learning_rate = 0.00033828348, loss = 2.0347292e-05, step = 33603 (5.280 sec) 2021-10-19 08:46:22,216 [INFO] tensorflow: epoch = 87.05440414507773, learning_rate = 0.00033828348, loss = 2.0347292e-05, step = 33603 (5.280 sec) 2021-10-19 08:46:26,495 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.542 INFO:tensorflow:epoch = 87.12176165803109, learning_rate = 0.00033538137, loss = 1.2373595e-05, step = 33629 (5.313 sec) 2021-10-19 08:46:27,529 [INFO] tensorflow: epoch = 87.12176165803109, learning_rate = 0.00033538137, loss = 1.2373595e-05, step = 33629 (5.313 sec) INFO:tensorflow:global_step/sec: 4.91259 2021-10-19 08:46:27,727 [INFO] tensorflow: global_step/sec: 4.91259 2021-10-19 08:46:31,573 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.692 INFO:tensorflow:epoch = 87.18911917098445, learning_rate = 0.00033250367, loss = 8.642177e-06, step = 33655 (5.287 sec) 2021-10-19 08:46:32,817 [INFO] tensorflow: epoch = 87.18911917098445, learning_rate = 0.00033250367, loss = 8.642177e-06, step = 33655 (5.287 sec) INFO:tensorflow:global_step/sec: 4.95968 2021-10-19 08:46:35,389 [INFO] tensorflow: global_step/sec: 4.95968 2021-10-19 08:46:36,566 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.032 INFO:tensorflow:epoch = 87.25906735751296, learning_rate = 0.00032954192, loss = 2.2826109e-05, step = 33682 (5.299 sec) 2021-10-19 08:46:38,116 [INFO] tensorflow: epoch = 87.25906735751296, learning_rate = 0.00032954192, loss = 2.2826109e-05, step = 33682 (5.299 sec) 2021-10-19 08:46:41,457 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.446 INFO:tensorflow:global_step/sec: 5.08495 2021-10-19 08:46:42,862 [INFO] tensorflow: global_step/sec: 5.08495 INFO:tensorflow:epoch = 87.32901554404145, learning_rate = 0.00032660653, loss = 7.535982e-06, step = 33709 (5.335 sec) 2021-10-19 08:46:43,451 [INFO] tensorflow: epoch = 87.32901554404145, learning_rate = 0.00032660653, loss = 7.535982e-06, step = 33709 (5.335 sec) 2021-10-19 08:46:46,425 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.131 INFO:tensorflow:epoch = 87.39637305699482, learning_rate = 0.00032380412, loss = 6.4849773e-06, step = 33735 (5.212 sec) 2021-10-19 08:46:48,663 [INFO] tensorflow: epoch = 87.39637305699482, learning_rate = 0.00032380412, loss = 6.4849773e-06, step = 33735 (5.212 sec) INFO:tensorflow:global_step/sec: 4.98444 2021-10-19 08:46:50,486 [INFO] tensorflow: global_step/sec: 4.98444 2021-10-19 08:46:51,462 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.854 INFO:tensorflow:epoch = 87.46373056994818, learning_rate = 0.00032102637, loss = 1.7106751e-05, step = 33761 (5.204 sec) 2021-10-19 08:46:53,867 [INFO] tensorflow: epoch = 87.46373056994818, learning_rate = 0.00032102637, loss = 1.7106751e-05, step = 33761 (5.204 sec) 2021-10-19 08:46:56,513 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.800 INFO:tensorflow:global_step/sec: 5.00059 2021-10-19 08:46:58,085 [INFO] tensorflow: global_step/sec: 5.00059 INFO:tensorflow:epoch = 87.53367875647669, learning_rate = 0.00031816654, loss = 2.2467113e-05, step = 33788 (5.404 sec) 2021-10-19 08:46:59,271 [INFO] tensorflow: epoch = 87.53367875647669, learning_rate = 0.00031816654, loss = 2.2467113e-05, step = 33788 (5.404 sec) 2021-10-19 08:47:01,506 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.029 INFO:tensorflow:epoch = 87.60103626943005, learning_rate = 0.00031543686, loss = 1.5287973e-05, step = 33814 (5.213 sec) 2021-10-19 08:47:04,483 [INFO] tensorflow: epoch = 87.60103626943005, learning_rate = 0.00031543686, loss = 1.5287973e-05, step = 33814 (5.213 sec) INFO:tensorflow:global_step/sec: 4.99357 2021-10-19 08:47:05,694 [INFO] tensorflow: global_step/sec: 4.99357 2021-10-19 08:47:06,481 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.103 INFO:tensorflow:epoch = 87.67098445595855, learning_rate = 0.00031262712, loss = 1.3553556e-05, step = 33841 (5.362 sec) 2021-10-19 08:47:09,846 [INFO] tensorflow: epoch = 87.67098445595855, learning_rate = 0.00031262712, loss = 1.3553556e-05, step = 33841 (5.362 sec) 2021-10-19 08:47:11,434 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.193 INFO:tensorflow:global_step/sec: 5.03853 2021-10-19 08:47:13,236 [INFO] tensorflow: global_step/sec: 5.03853 INFO:tensorflow:epoch = 87.74093264248705, learning_rate = 0.0003098418, loss = 1.1206958e-05, step = 33868 (5.361 sec) 2021-10-19 08:47:15,206 [INFO] tensorflow: epoch = 87.74093264248705, learning_rate = 0.0003098418, loss = 1.1206958e-05, step = 33868 (5.361 sec) 2021-10-19 08:47:16,417 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.068 INFO:tensorflow:epoch = 87.81088082901555, learning_rate = 0.0003070819, loss = 1.3426186e-05, step = 33895 (5.300 sec) 2021-10-19 08:47:20,506 [INFO] tensorflow: epoch = 87.81088082901555, learning_rate = 0.0003070819, loss = 1.3426186e-05, step = 33895 (5.300 sec) INFO:tensorflow:global_step/sec: 5.08657 2021-10-19 08:47:20,707 [INFO] tensorflow: global_step/sec: 5.08657 2021-10-19 08:47:21,320 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.398 INFO:tensorflow:epoch = 87.87823834196891, learning_rate = 0.0003044476, loss = 1.9675845e-05, step = 33921 (5.193 sec) 2021-10-19 08:47:25,698 [INFO] tensorflow: epoch = 87.87823834196891, learning_rate = 0.0003044476, loss = 1.9675845e-05, step = 33921 (5.193 sec) 2021-10-19 08:47:26,282 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.155 INFO:tensorflow:global_step/sec: 5.04178 2021-10-19 08:47:28,244 [INFO] tensorflow: global_step/sec: 5.04178 INFO:tensorflow:epoch = 87.94818652849742, learning_rate = 0.00030173518, loss = 2.597132e-05, step = 33948 (5.317 sec) 2021-10-19 08:47:31,015 [INFO] tensorflow: epoch = 87.94818652849742, learning_rate = 0.00030173518, loss = 2.597132e-05, step = 33948 (5.317 sec) 2021-10-19 08:47:31,218 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.262 2021-10-19 08:47:35,104 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 88/120: loss: 0.00002 learning rate: 0.00030 Time taken: 0:01:17.129594 ETA: 0:41:08.147011 INFO:tensorflow:global_step/sec: 4.95371 2021-10-19 08:47:35,915 [INFO] tensorflow: global_step/sec: 4.95371 INFO:tensorflow:epoch = 88.01554404145078, learning_rate = 0.00029914675, loss = 1.1499727e-05, step = 33974 (5.291 sec) 2021-10-19 08:47:36,306 [INFO] tensorflow: epoch = 88.01554404145078, learning_rate = 0.00029914675, loss = 1.1499727e-05, step = 33974 (5.291 sec) 2021-10-19 08:47:36,306 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.654 2021-10-19 08:47:41,262 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.180 INFO:tensorflow:epoch = 88.08549222797927, learning_rate = 0.00029648154, loss = 1.9305175e-05, step = 34001 (5.361 sec) 2021-10-19 08:47:41,667 [INFO] tensorflow: epoch = 88.08549222797927, learning_rate = 0.00029648154, loss = 1.9305175e-05, step = 34001 (5.361 sec) INFO:tensorflow:global_step/sec: 5.05585 2021-10-19 08:47:43,431 [INFO] tensorflow: global_step/sec: 5.05585 2021-10-19 08:47:46,204 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.239 INFO:tensorflow:epoch = 88.15544041450778, learning_rate = 0.00029384063, loss = 1.7337245e-05, step = 34028 (5.331 sec) 2021-10-19 08:47:46,998 [INFO] tensorflow: epoch = 88.15544041450778, learning_rate = 0.00029384063, loss = 1.7337245e-05, step = 34028 (5.331 sec) INFO:tensorflow:global_step/sec: 4.97488 2021-10-19 08:47:51,069 [INFO] tensorflow: global_step/sec: 4.97488 2021-10-19 08:47:51,268 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.749 INFO:tensorflow:epoch = 88.22279792746114, learning_rate = 0.00029131994, loss = 1.2946033e-05, step = 34054 (5.237 sec) 2021-10-19 08:47:52,235 [INFO] tensorflow: epoch = 88.22279792746114, learning_rate = 0.00029131994, loss = 1.2946033e-05, step = 34054 (5.237 sec) 2021-10-19 08:47:56,232 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.144 INFO:tensorflow:epoch = 88.29015544041451, learning_rate = 0.0002888203, loss = 2.1550339e-05, step = 34080 (5.253 sec) 2021-10-19 08:47:57,489 [INFO] tensorflow: epoch = 88.29015544041451, learning_rate = 0.0002888203, loss = 2.1550339e-05, step = 34080 (5.253 sec) INFO:tensorflow:global_step/sec: 5.00141 2021-10-19 08:47:58,667 [INFO] tensorflow: global_step/sec: 5.00141 2021-10-19 08:48:01,202 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.122 INFO:tensorflow:epoch = 88.36010362694302, learning_rate = 0.0002862476, loss = 1.8746188e-05, step = 34107 (5.292 sec) 2021-10-19 08:48:02,780 [INFO] tensorflow: epoch = 88.36010362694302, learning_rate = 0.0002862476, loss = 1.8746188e-05, step = 34107 (5.292 sec) INFO:tensorflow:global_step/sec: 5.08358 2021-10-19 08:48:06,142 [INFO] tensorflow: global_step/sec: 5.08358 2021-10-19 08:48:06,144 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.239 INFO:tensorflow:epoch = 88.43005181347151, learning_rate = 0.00028369762, loss = 1.2379379e-05, step = 34134 (5.392 sec) 2021-10-19 08:48:08,172 [INFO] tensorflow: epoch = 88.43005181347151, learning_rate = 0.00028369762, loss = 1.2379379e-05, step = 34134 (5.392 sec) 2021-10-19 08:48:11,132 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.048 INFO:tensorflow:epoch = 88.5, learning_rate = 0.0002811706, loss = 1.1820593e-05, step = 34161 (5.360 sec) 2021-10-19 08:48:13,532 [INFO] tensorflow: epoch = 88.5, learning_rate = 0.0002811706, loss = 1.1820593e-05, step = 34161 (5.360 sec) INFO:tensorflow:global_step/sec: 5.00768 2021-10-19 08:48:13,731 [INFO] tensorflow: global_step/sec: 5.00768 2021-10-19 08:48:16,144 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.957 INFO:tensorflow:epoch = 88.5699481865285, learning_rate = 0.0002786658, loss = 1.5112079e-05, step = 34188 (5.344 sec) 2021-10-19 08:48:18,876 [INFO] tensorflow: epoch = 88.5699481865285, learning_rate = 0.0002786658, loss = 1.5112079e-05, step = 34188 (5.344 sec) 2021-10-19 08:48:21,046 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.400 INFO:tensorflow:global_step/sec: 5.05983 2021-10-19 08:48:21,241 [INFO] tensorflow: global_step/sec: 5.05983 INFO:tensorflow:epoch = 88.63730569948187, learning_rate = 0.000276275, loss = 2.4073295e-05, step = 34214 (5.216 sec) 2021-10-19 08:48:24,091 [INFO] tensorflow: epoch = 88.63730569948187, learning_rate = 0.000276275, loss = 2.4073295e-05, step = 34214 (5.216 sec) 2021-10-19 08:48:26,055 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.967 INFO:tensorflow:global_step/sec: 5.02452 2021-10-19 08:48:28,804 [INFO] tensorflow: global_step/sec: 5.02452 INFO:tensorflow:epoch = 88.70725388601036, learning_rate = 0.0002738141, loss = 1.3559777e-05, step = 34241 (5.309 sec) 2021-10-19 08:48:29,400 [INFO] tensorflow: epoch = 88.70725388601036, learning_rate = 0.0002738141, loss = 1.3559777e-05, step = 34241 (5.309 sec) 2021-10-19 08:48:31,025 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.121 INFO:tensorflow:epoch = 88.77461139896373, learning_rate = 0.00027146493, loss = 1.6999318e-05, step = 34267 (5.193 sec) 2021-10-19 08:48:34,593 [INFO] tensorflow: epoch = 88.77461139896373, learning_rate = 0.00027146493, loss = 1.6999318e-05, step = 34267 (5.193 sec) 2021-10-19 08:48:35,999 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.108 INFO:tensorflow:global_step/sec: 5.00374 2021-10-19 08:48:36,398 [INFO] tensorflow: global_step/sec: 5.00374 INFO:tensorflow:epoch = 88.84455958549223, learning_rate = 0.0002690466, loss = 7.5355515e-06, step = 34294 (5.359 sec) 2021-10-19 08:48:39,952 [INFO] tensorflow: epoch = 88.84455958549223, learning_rate = 0.0002690466, loss = 7.5355515e-06, step = 34294 (5.359 sec) 2021-10-19 08:48:40,924 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.305 INFO:tensorflow:global_step/sec: 5.06565 2021-10-19 08:48:43,900 [INFO] tensorflow: global_step/sec: 5.06565 INFO:tensorflow:epoch = 88.91450777202073, learning_rate = 0.0002666501, loss = 1.5641319e-05, step = 34321 (5.353 sec) 2021-10-19 08:48:45,305 [INFO] tensorflow: epoch = 88.91450777202073, learning_rate = 0.0002666501, loss = 1.5641319e-05, step = 34321 (5.353 sec) 2021-10-19 08:48:45,914 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.040 INFO:tensorflow:epoch = 88.98445595854922, learning_rate = 0.00026427468, loss = 1.3930451e-05, step = 34348 (5.320 sec) 2021-10-19 08:48:50,624 [INFO] tensorflow: epoch = 88.98445595854922, learning_rate = 0.00026427468, loss = 1.3930451e-05, step = 34348 (5.320 sec) 2021-10-19 08:48:50,827 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.355 INFO:tensorflow:global_step/sec: 5.03971 2021-10-19 08:48:51,440 [INFO] tensorflow: global_step/sec: 5.03971 2021-10-19 08:48:51,854 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 89/120: loss: 0.00001 learning rate: 0.00026 Time taken: 0:01:16.769515 ETA: 0:39:39.854951 2021-10-19 08:48:55,799 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.116 INFO:tensorflow:epoch = 89.05440414507773, learning_rate = 0.00026192065, loss = 1.7792678e-05, step = 34375 (5.379 sec) 2021-10-19 08:48:56,003 [INFO] tensorflow: epoch = 89.05440414507773, learning_rate = 0.00026192065, loss = 1.7792678e-05, step = 34375 (5.379 sec) INFO:tensorflow:global_step/sec: 5.03381 2021-10-19 08:48:58,989 [INFO] tensorflow: global_step/sec: 5.03381 2021-10-19 08:49:00,800 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.996 INFO:tensorflow:epoch = 89.12176165803109, learning_rate = 0.00025967328, loss = 1.8359395e-05, step = 34401 (5.217 sec) 2021-10-19 08:49:01,220 [INFO] tensorflow: epoch = 89.12176165803109, learning_rate = 0.00025967328, loss = 1.8359395e-05, step = 34401 (5.217 sec) 2021-10-19 08:49:05,804 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.986 INFO:tensorflow:epoch = 89.19170984455958, learning_rate = 0.00025736025, loss = 1.8842613e-05, step = 34428 (5.388 sec) 2021-10-19 08:49:06,608 [INFO] tensorflow: epoch = 89.19170984455958, learning_rate = 0.00025736025, loss = 1.8842613e-05, step = 34428 (5.388 sec) INFO:tensorflow:global_step/sec: 4.98669 2021-10-19 08:49:06,609 [INFO] tensorflow: global_step/sec: 4.98669 2021-10-19 08:49:10,727 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.315 INFO:tensorflow:epoch = 89.26165803108809, learning_rate = 0.00025506783, loss = 1.1838943e-05, step = 34455 (5.350 sec) 2021-10-19 08:49:11,958 [INFO] tensorflow: epoch = 89.26165803108809, learning_rate = 0.00025506783, loss = 1.1838943e-05, step = 34455 (5.350 sec) INFO:tensorflow:global_step/sec: 5.00703 2021-10-19 08:49:14,198 [INFO] tensorflow: global_step/sec: 5.00703 2021-10-19 08:49:15,787 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.766 INFO:tensorflow:epoch = 89.32901554404145, learning_rate = 0.00025287922, loss = 1.4407233e-05, step = 34481 (5.228 sec) 2021-10-19 08:49:17,186 [INFO] tensorflow: epoch = 89.32901554404145, learning_rate = 0.00025287922, loss = 1.4407233e-05, step = 34481 (5.228 sec) 2021-10-19 08:49:20,781 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.023 INFO:tensorflow:global_step/sec: 4.98344 2021-10-19 08:49:21,823 [INFO] tensorflow: global_step/sec: 4.98344 INFO:tensorflow:epoch = 89.39637305699482, learning_rate = 0.0002507099, loss = 1.5146848e-05, step = 34507 (5.274 sec) 2021-10-19 08:49:22,460 [INFO] tensorflow: epoch = 89.39637305699482, learning_rate = 0.0002507099, loss = 1.5146848e-05, step = 34507 (5.274 sec) 2021-10-19 08:49:25,852 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.724 INFO:tensorflow:epoch = 89.46373056994818, learning_rate = 0.00024855896, loss = 7.490462e-06, step = 34533 (5.194 sec) 2021-10-19 08:49:27,654 [INFO] tensorflow: epoch = 89.46373056994818, learning_rate = 0.00024855896, loss = 7.490462e-06, step = 34533 (5.194 sec) INFO:tensorflow:global_step/sec: 4.98076 2021-10-19 08:49:29,453 [INFO] tensorflow: global_step/sec: 4.98076 2021-10-19 08:49:30,841 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.046 INFO:tensorflow:epoch = 89.53367875647669, learning_rate = 0.00024634472, loss = 2.388767e-05, step = 34560 (5.355 sec) 2021-10-19 08:49:33,009 [INFO] tensorflow: epoch = 89.53367875647669, learning_rate = 0.00024634472, loss = 2.388767e-05, step = 34560 (5.355 sec) 2021-10-19 08:49:35,762 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.323 INFO:tensorflow:global_step/sec: 5.05222 2021-10-19 08:49:36,974 [INFO] tensorflow: global_step/sec: 5.05222 INFO:tensorflow:epoch = 89.60103626943005, learning_rate = 0.00024423123, loss = 3.8178077e-05, step = 34586 (5.230 sec) 2021-10-19 08:49:38,238 [INFO] tensorflow: epoch = 89.60103626943005, learning_rate = 0.00024423123, loss = 3.8178077e-05, step = 34586 (5.230 sec) 2021-10-19 08:49:40,813 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.801 INFO:tensorflow:epoch = 89.67098445595855, learning_rate = 0.00024205571, loss = 1.8479288e-05, step = 34613 (5.310 sec) 2021-10-19 08:49:43,548 [INFO] tensorflow: epoch = 89.67098445595855, learning_rate = 0.00024205571, loss = 1.8479288e-05, step = 34613 (5.310 sec) INFO:tensorflow:global_step/sec: 5.03385 2021-10-19 08:49:44,523 [INFO] tensorflow: global_step/sec: 5.03385 2021-10-19 08:49:45,715 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.403 INFO:tensorflow:epoch = 89.74093264248705, learning_rate = 0.0002398994, loss = 2.6098176e-05, step = 34640 (5.329 sec) 2021-10-19 08:49:48,877 [INFO] tensorflow: epoch = 89.74093264248705, learning_rate = 0.0002398994, loss = 2.6098176e-05, step = 34640 (5.329 sec) 2021-10-19 08:49:50,668 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.193 INFO:tensorflow:global_step/sec: 5.03893 2021-10-19 08:49:52,064 [INFO] tensorflow: global_step/sec: 5.03893 INFO:tensorflow:epoch = 89.81088082901555, learning_rate = 0.0002377625, loss = 6.5118315e-06, step = 34667 (5.370 sec) 2021-10-19 08:49:54,247 [INFO] tensorflow: epoch = 89.81088082901555, learning_rate = 0.0002377625, loss = 6.5118315e-06, step = 34667 (5.370 sec) 2021-10-19 08:49:55,688 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.923 INFO:tensorflow:epoch = 89.87823834196891, learning_rate = 0.0002357224, loss = 1.3837493e-05, step = 34693 (5.279 sec) 2021-10-19 08:49:59,526 [INFO] tensorflow: epoch = 89.87823834196891, learning_rate = 0.0002357224, loss = 1.3837493e-05, step = 34693 (5.279 sec) INFO:tensorflow:global_step/sec: 4.95802 2021-10-19 08:49:59,729 [INFO] tensorflow: global_step/sec: 4.95802 2021-10-19 08:50:00,750 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.757 INFO:tensorflow:epoch = 89.94559585492229, learning_rate = 0.00023370003, loss = 2.7857099e-05, step = 34719 (5.222 sec) 2021-10-19 08:50:04,747 [INFO] tensorflow: epoch = 89.94559585492229, learning_rate = 0.00023370003, loss = 2.7857099e-05, step = 34719 (5.222 sec) 2021-10-19 08:50:05,727 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.095 INFO:tensorflow:global_step/sec: 5.02015 2021-10-19 08:50:07,298 [INFO] tensorflow: global_step/sec: 5.02015 INFO:tensorflow:Saving checkpoints for step-34740. 2021-10-19 08:50:08,713 [INFO] tensorflow: Saving checkpoints for step-34740. WARNING:tensorflow:Ignoring: /tmp/tmpxtrtlkgb; No such file or directory 2021-10-19 08:50:08,920 [WARNING] tensorflow: Ignoring: /tmp/tmpxtrtlkgb; No such file or directory 2021-10-19 08:50:13,302 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 62, 0.00s/step 2021-10-19 08:50:14,292 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 62, 0.10s/step 2021-10-19 08:50:15,257 [INFO] iva.detectnet_v2.evaluation.evaluation: step 20 / 62, 0.10s/step 2021-10-19 08:50:16,251 [INFO] iva.detectnet_v2.evaluation.evaluation: step 30 / 62, 0.10s/step 2021-10-19 08:50:17,227 [INFO] iva.detectnet_v2.evaluation.evaluation: step 40 / 62, 0.10s/step 2021-10-19 08:50:18,212 [INFO] iva.detectnet_v2.evaluation.evaluation: step 50 / 62, 0.10s/step 2021-10-19 08:50:19,170 [INFO] iva.detectnet_v2.evaluation.evaluation: step 60 / 62, 0.10s/step Matching predictions to ground truth, class 1/7.: 100%|█| 428/428 [00:00<00:00, 14769.16it/s] Matching predictions to ground truth, class 2/7.: 100%|█| 257/257 [00:00<00:00, 10468.96it/s] Matching predictions to ground truth, class 7/7.: 100%|█| 365/365 [00:00<00:00, 29279.19it/s] Epoch 90/120 ========================= Validation cost: 0.000011 Mean average_precision (in %): 31.2811 class name average precision (in %) ------------- -------------------------- auto_rickshaw 0 bicycle 93.4561 bus 42.6182 car 82.8937 person 0 road_sign 0 truck 0 Median Inference Time: 0.018755 INFO:tensorflow:epoch = 90.0, learning_rate = 0.00023207937, loss = 1.5131992e-05, step = 34740 (14.943 sec) 2021-10-19 08:50:19,691 [INFO] tensorflow: epoch = 90.0, learning_rate = 0.00023207937, loss = 1.5131992e-05, step = 34740 (14.943 sec) 2021-10-19 08:50:19,692 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 90/120: loss: 0.00002 learning rate: 0.00023 Time taken: 0:01:27.830044 ETA: 0:43:54.901307 2021-10-19 08:50:21,479 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 6.349 INFO:tensorflow:epoch = 90.06735751295336, learning_rate = 0.00023008826, loss = 1.8921874e-05, step = 34766 (5.209 sec) 2021-10-19 08:50:24,900 [INFO] tensorflow: epoch = 90.06735751295336, learning_rate = 0.00023008826, loss = 1.8921874e-05, step = 34766 (5.209 sec) INFO:tensorflow:global_step/sec: 2.06171 2021-10-19 08:50:25,730 [INFO] tensorflow: global_step/sec: 2.06171 2021-10-19 08:50:26,520 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.841 INFO:tensorflow:epoch = 90.13471502590674, learning_rate = 0.00022811424, loss = 1.797706e-05, step = 34792 (5.293 sec) 2021-10-19 08:50:30,192 [INFO] tensorflow: epoch = 90.13471502590674, learning_rate = 0.00022811424, loss = 1.797706e-05, step = 34792 (5.293 sec) 2021-10-19 08:50:31,592 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.716 INFO:tensorflow:global_step/sec: 4.95642 2021-10-19 08:50:33,396 [INFO] tensorflow: global_step/sec: 4.95642 INFO:tensorflow:epoch = 90.20466321243524, learning_rate = 0.00022608231, loss = 8.3971245e-06, step = 34819 (5.345 sec) 2021-10-19 08:50:35,537 [INFO] tensorflow: epoch = 90.20466321243524, learning_rate = 0.00022608231, loss = 8.3971245e-06, step = 34819 (5.345 sec) 2021-10-19 08:50:36,508 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.344 INFO:tensorflow:epoch = 90.27461139896373, learning_rate = 0.00022406806, loss = 2.7687744e-05, step = 34846 (5.305 sec) 2021-10-19 08:50:40,842 [INFO] tensorflow: epoch = 90.27461139896373, learning_rate = 0.00022406806, loss = 2.7687744e-05, step = 34846 (5.305 sec) INFO:tensorflow:global_step/sec: 5.10285 2021-10-19 08:50:40,843 [INFO] tensorflow: global_step/sec: 5.10285 2021-10-19 08:50:41,486 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.091 INFO:tensorflow:epoch = 90.3419689119171, learning_rate = 0.00022214591, loss = 1.5119091e-05, step = 34872 (5.202 sec) 2021-10-19 08:50:46,043 [INFO] tensorflow: epoch = 90.3419689119171, learning_rate = 0.00022214591, loss = 1.5119091e-05, step = 34872 (5.202 sec) 2021-10-19 08:50:46,435 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.210 INFO:tensorflow:global_step/sec: 5.01726 2021-10-19 08:50:48,417 [INFO] tensorflow: global_step/sec: 5.01726 INFO:tensorflow:epoch = 90.4119170984456, learning_rate = 0.00022016715, loss = 2.0793856e-05, step = 34899 (5.374 sec) 2021-10-19 08:50:51,417 [INFO] tensorflow: epoch = 90.4119170984456, learning_rate = 0.00022016715, loss = 2.0793856e-05, step = 34899 (5.374 sec) 2021-10-19 08:50:51,417 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.073 INFO:tensorflow:global_step/sec: 4.9534 2021-10-19 08:50:56,089 [INFO] tensorflow: global_step/sec: 4.9534 2021-10-19 08:50:56,498 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.684 INFO:tensorflow:epoch = 90.47927461139896, learning_rate = 0.00021827804, loss = 1.9999701e-05, step = 34925 (5.277 sec) 2021-10-19 08:50:56,694 [INFO] tensorflow: epoch = 90.47927461139896, learning_rate = 0.00021827804, loss = 1.9999701e-05, step = 34925 (5.277 sec) 2021-10-19 08:51:01,450 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.199 INFO:tensorflow:epoch = 90.54922279792747, learning_rate = 0.00021633373, loss = 1.9643856e-05, step = 34952 (5.341 sec) 2021-10-19 08:51:02,035 [INFO] tensorflow: epoch = 90.54922279792747, learning_rate = 0.00021633373, loss = 1.9643856e-05, step = 34952 (5.341 sec) INFO:tensorflow:global_step/sec: 5.02232 2021-10-19 08:51:03,655 [INFO] tensorflow: global_step/sec: 5.02232 2021-10-19 08:51:06,450 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.000 INFO:tensorflow:epoch = 90.61658031088083, learning_rate = 0.00021447772, loss = 1.4043222e-05, step = 34978 (5.241 sec) 2021-10-19 08:51:07,277 [INFO] tensorflow: epoch = 90.61658031088083, learning_rate = 0.00021447772, loss = 1.4043222e-05, step = 34978 (5.241 sec) INFO:tensorflow:global_step/sec: 4.99178 2021-10-19 08:51:11,267 [INFO] tensorflow: global_step/sec: 4.99178 2021-10-19 08:51:11,475 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.902 INFO:tensorflow:epoch = 90.6839378238342, learning_rate = 0.00021263742, loss = 2.3233339e-05, step = 35004 (5.225 sec) 2021-10-19 08:51:12,501 [INFO] tensorflow: epoch = 90.6839378238342, learning_rate = 0.00021263742, loss = 2.3233339e-05, step = 35004 (5.225 sec) 2021-10-19 08:51:16,521 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.821 INFO:tensorflow:epoch = 90.75129533678756, learning_rate = 0.0002108131, loss = 2.1317263e-05, step = 35030 (5.198 sec) 2021-10-19 08:51:17,700 [INFO] tensorflow: epoch = 90.75129533678756, learning_rate = 0.0002108131, loss = 2.1317263e-05, step = 35030 (5.198 sec) INFO:tensorflow:global_step/sec: 4.98848 2021-10-19 08:51:18,885 [INFO] tensorflow: global_step/sec: 4.98848 2021-10-19 08:51:21,481 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.166 INFO:tensorflow:epoch = 90.82124352331607, learning_rate = 0.0002089353, loss = 3.1611016e-05, step = 35057 (5.380 sec) 2021-10-19 08:51:23,080 [INFO] tensorflow: epoch = 90.82124352331607, learning_rate = 0.0002089353, loss = 3.1611016e-05, step = 35057 (5.380 sec) INFO:tensorflow:global_step/sec: 4.96281 2021-10-19 08:51:26,542 [INFO] tensorflow: global_step/sec: 4.96281 2021-10-19 08:51:26,543 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.754 INFO:tensorflow:epoch = 90.88860103626943, learning_rate = 0.00020714274, loss = 1.0458672e-05, step = 35083 (5.265 sec) 2021-10-19 08:51:28,344 [INFO] tensorflow: epoch = 90.88860103626943, learning_rate = 0.00020714274, loss = 1.0458672e-05, step = 35083 (5.265 sec) 2021-10-19 08:51:31,558 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.941 INFO:tensorflow:epoch = 90.95854922279793, learning_rate = 0.00020529745, loss = 1.9941635e-05, step = 35110 (5.416 sec) 2021-10-19 08:51:33,760 [INFO] tensorflow: epoch = 90.95854922279793, learning_rate = 0.00020529745, loss = 1.9941635e-05, step = 35110 (5.416 sec) INFO:tensorflow:global_step/sec: 4.99278 2021-10-19 08:51:34,153 [INFO] tensorflow: global_step/sec: 4.99278 2021-10-19 08:51:36,518 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.164 2021-10-19 08:51:36,935 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 91/120: loss: 0.00001 learning rate: 0.00020 Time taken: 0:01:17.241083 ETA: 0:37:19.991397 INFO:tensorflow:epoch = 91.02590673575129, learning_rate = 0.00020353628, loss = 8.368426e-06, step = 35136 (5.204 sec) 2021-10-19 08:51:38,965 [INFO] tensorflow: epoch = 91.02590673575129, learning_rate = 0.00020353628, loss = 8.368426e-06, step = 35136 (5.204 sec) 2021-10-19 08:51:41,529 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.961 INFO:tensorflow:global_step/sec: 5.01252 2021-10-19 08:51:41,734 [INFO] tensorflow: global_step/sec: 5.01252 INFO:tensorflow:epoch = 91.0958549222798, learning_rate = 0.00020172312, loss = 2.127402e-05, step = 35163 (5.332 sec) 2021-10-19 08:51:44,297 [INFO] tensorflow: epoch = 91.0958549222798, learning_rate = 0.00020172312, loss = 2.127402e-05, step = 35163 (5.332 sec) 2021-10-19 08:51:46,478 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.207 INFO:tensorflow:global_step/sec: 5.04189 2021-10-19 08:51:49,271 [INFO] tensorflow: global_step/sec: 5.04189 INFO:tensorflow:epoch = 91.16580310880829, learning_rate = 0.00019992627, loss = 2.8900518e-05, step = 35190 (5.392 sec) 2021-10-19 08:51:49,689 [INFO] tensorflow: epoch = 91.16580310880829, learning_rate = 0.00019992627, loss = 2.8900518e-05, step = 35190 (5.392 sec) 2021-10-19 08:51:51,527 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.807 INFO:tensorflow:epoch = 91.23316062176166, learning_rate = 0.00019821082, loss = 1.0938278e-05, step = 35216 (5.238 sec) 2021-10-19 08:51:54,927 [INFO] tensorflow: epoch = 91.23316062176166, learning_rate = 0.00019821082, loss = 1.0938278e-05, step = 35216 (5.238 sec) 2021-10-19 08:51:56,466 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.248 INFO:tensorflow:global_step/sec: 5.00243 2021-10-19 08:51:56,867 [INFO] tensorflow: global_step/sec: 5.00243 INFO:tensorflow:epoch = 91.30310880829016, learning_rate = 0.00019644527, loss = 1.4720791e-05, step = 35243 (5.340 sec) 2021-10-19 08:52:00,266 [INFO] tensorflow: epoch = 91.30310880829016, learning_rate = 0.00019644527, loss = 1.4720791e-05, step = 35243 (5.340 sec) 2021-10-19 08:52:01,474 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.970 INFO:tensorflow:global_step/sec: 5.02865 2021-10-19 08:52:04,424 [INFO] tensorflow: global_step/sec: 5.02865 INFO:tensorflow:epoch = 91.37305699481865, learning_rate = 0.00019469544, loss = 1.1465001e-05, step = 35270 (5.344 sec) 2021-10-19 08:52:05,610 [INFO] tensorflow: epoch = 91.37305699481865, learning_rate = 0.00019469544, loss = 1.1465001e-05, step = 35270 (5.344 sec) 2021-10-19 08:52:06,397 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.315 INFO:tensorflow:epoch = 91.44300518134716, learning_rate = 0.00019296102, loss = 2.9720037e-05, step = 35297 (5.302 sec) 2021-10-19 08:52:10,912 [INFO] tensorflow: epoch = 91.44300518134716, learning_rate = 0.00019296102, loss = 2.9720037e-05, step = 35297 (5.302 sec) 2021-10-19 08:52:11,307 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.369 INFO:tensorflow:global_step/sec: 5.08442 2021-10-19 08:52:11,898 [INFO] tensorflow: global_step/sec: 5.08442 INFO:tensorflow:epoch = 91.51036269430053, learning_rate = 0.00019130552, loss = 1.4559758e-05, step = 35323 (5.223 sec) 2021-10-19 08:52:16,136 [INFO] tensorflow: epoch = 91.51036269430053, learning_rate = 0.00019130552, loss = 1.4559758e-05, step = 35323 (5.223 sec) 2021-10-19 08:52:16,334 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.894 INFO:tensorflow:global_step/sec: 4.96705 2021-10-19 08:52:19,548 [INFO] tensorflow: global_step/sec: 4.96705 2021-10-19 08:52:21,320 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.055 INFO:tensorflow:epoch = 91.58031088082902, learning_rate = 0.0001896013, loss = 1.1955743e-05, step = 35350 (5.389 sec) 2021-10-19 08:52:21,525 [INFO] tensorflow: epoch = 91.58031088082902, learning_rate = 0.0001896013, loss = 1.1955743e-05, step = 35350 (5.389 sec) 2021-10-19 08:52:26,288 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.134 INFO:tensorflow:epoch = 91.65025906735751, learning_rate = 0.00018791243, loss = 8.236276e-06, step = 35377 (5.367 sec) 2021-10-19 08:52:26,891 [INFO] tensorflow: epoch = 91.65025906735751, learning_rate = 0.00018791243, loss = 8.236276e-06, step = 35377 (5.367 sec) INFO:tensorflow:global_step/sec: 5.03941 2021-10-19 08:52:27,089 [INFO] tensorflow: global_step/sec: 5.03941 2021-10-19 08:52:31,299 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.958 INFO:tensorflow:epoch = 91.71761658031089, learning_rate = 0.00018630024, loss = 2.3720544e-05, step = 35403 (5.242 sec) 2021-10-19 08:52:32,133 [INFO] tensorflow: epoch = 91.71761658031089, learning_rate = 0.00018630024, loss = 2.3720544e-05, step = 35403 (5.242 sec) INFO:tensorflow:global_step/sec: 4.98548 2021-10-19 08:52:34,711 [INFO] tensorflow: global_step/sec: 4.98548 2021-10-19 08:52:36,371 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.718 INFO:tensorflow:epoch = 91.78497409326425, learning_rate = 0.00018470189, loss = 2.0945727e-05, step = 35429 (5.209 sec) 2021-10-19 08:52:37,342 [INFO] tensorflow: epoch = 91.78497409326425, learning_rate = 0.00018470189, loss = 2.0945727e-05, step = 35429 (5.209 sec) 2021-10-19 08:52:41,329 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.172 INFO:tensorflow:global_step/sec: 5.00138 2021-10-19 08:52:42,309 [INFO] tensorflow: global_step/sec: 5.00138 INFO:tensorflow:epoch = 91.85492227979275, learning_rate = 0.00018305649, loss = 2.9832046e-05, step = 35456 (5.374 sec) 2021-10-19 08:52:42,716 [INFO] tensorflow: epoch = 91.85492227979275, learning_rate = 0.00018305649, loss = 2.9832046e-05, step = 35456 (5.374 sec) 2021-10-19 08:52:46,237 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.374 INFO:tensorflow:epoch = 91.92487046632125, learning_rate = 0.00018142576, loss = 8.196058e-06, step = 35483 (5.306 sec) 2021-10-19 08:52:48,021 [INFO] tensorflow: epoch = 91.92487046632125, learning_rate = 0.00018142576, loss = 8.196058e-06, step = 35483 (5.306 sec) INFO:tensorflow:global_step/sec: 5.08328 2021-10-19 08:52:49,784 [INFO] tensorflow: global_step/sec: 5.08328 2021-10-19 08:52:51,183 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.222 INFO:tensorflow:epoch = 91.99481865284974, learning_rate = 0.00017980971, loss = 2.4461127e-05, step = 35510 (5.341 sec) 2021-10-19 08:52:53,362 [INFO] tensorflow: epoch = 91.99481865284974, learning_rate = 0.00017980971, loss = 2.4461127e-05, step = 35510 (5.341 sec) 2021-10-19 08:52:53,781 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 92/120: loss: 0.00001 learning rate: 0.00018 Time taken: 0:01:16.848626 ETA: 0:35:51.761518 2021-10-19 08:52:56,172 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.046 INFO:tensorflow:global_step/sec: 5.01159 2021-10-19 08:52:57,367 [INFO] tensorflow: global_step/sec: 5.01159 INFO:tensorflow:epoch = 92.06217616580311, learning_rate = 0.0001782672, loss = 2.7642816e-05, step = 35536 (5.196 sec) 2021-10-19 08:52:58,558 [INFO] tensorflow: epoch = 92.06217616580311, learning_rate = 0.0001782672, loss = 2.7642816e-05, step = 35536 (5.196 sec) 2021-10-19 08:53:01,140 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.131 INFO:tensorflow:epoch = 92.12953367875647, learning_rate = 0.00017673761, loss = 1.0463568e-05, step = 35562 (5.202 sec) 2021-10-19 08:53:03,760 [INFO] tensorflow: epoch = 92.12953367875647, learning_rate = 0.00017673761, loss = 1.0463568e-05, step = 35562 (5.202 sec) INFO:tensorflow:global_step/sec: 4.99786 2021-10-19 08:53:04,970 [INFO] tensorflow: global_step/sec: 4.99786 2021-10-19 08:53:06,155 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.944 INFO:tensorflow:epoch = 92.19948186528498, learning_rate = 0.00017516334, loss = 1.0663316e-05, step = 35589 (5.401 sec) 2021-10-19 08:53:09,162 [INFO] tensorflow: epoch = 92.19948186528498, learning_rate = 0.00017516334, loss = 1.0663316e-05, step = 35589 (5.401 sec) 2021-10-19 08:53:11,117 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.156 INFO:tensorflow:global_step/sec: 5.01453 2021-10-19 08:53:12,548 [INFO] tensorflow: global_step/sec: 5.01453 INFO:tensorflow:epoch = 92.26943005181347, learning_rate = 0.00017360291, loss = 8.563871e-06, step = 35616 (5.344 sec) 2021-10-19 08:53:14,506 [INFO] tensorflow: epoch = 92.26943005181347, learning_rate = 0.00017360291, loss = 8.563871e-06, step = 35616 (5.344 sec) 2021-10-19 08:53:16,090 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.109 INFO:tensorflow:epoch = 92.33937823834198, learning_rate = 0.00017205653, loss = 1.731095e-05, step = 35643 (5.348 sec) 2021-10-19 08:53:19,854 [INFO] tensorflow: epoch = 92.33937823834198, learning_rate = 0.00017205653, loss = 1.731095e-05, step = 35643 (5.348 sec) INFO:tensorflow:global_step/sec: 5.0386 2021-10-19 08:53:20,090 [INFO] tensorflow: global_step/sec: 5.0386 2021-10-19 08:53:21,095 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.982 INFO:tensorflow:epoch = 92.40673575129534, learning_rate = 0.00017058037, loss = 1.729191e-05, step = 35669 (5.331 sec) 2021-10-19 08:53:25,185 [INFO] tensorflow: epoch = 92.40673575129534, learning_rate = 0.00017058037, loss = 1.729191e-05, step = 35669 (5.331 sec) 2021-10-19 08:53:26,170 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.705 INFO:tensorflow:global_step/sec: 4.9324 2021-10-19 08:53:27,794 [INFO] tensorflow: global_step/sec: 4.9324 INFO:tensorflow:epoch = 92.47668393782384, learning_rate = 0.00016906079, loss = 1.560056e-05, step = 35696 (5.379 sec) 2021-10-19 08:53:30,564 [INFO] tensorflow: epoch = 92.47668393782384, learning_rate = 0.00016906079, loss = 1.560056e-05, step = 35696 (5.379 sec) 2021-10-19 08:53:31,165 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.022 INFO:tensorflow:global_step/sec: 5.03809 2021-10-19 08:53:35,336 [INFO] tensorflow: global_step/sec: 5.03809 INFO:tensorflow:epoch = 92.54663212435233, learning_rate = 0.00016755488, loss = 2.6845668e-05, step = 35723 (5.368 sec) 2021-10-19 08:53:35,932 [INFO] tensorflow: epoch = 92.54663212435233, learning_rate = 0.00016755488, loss = 2.6845668e-05, step = 35723 (5.368 sec) 2021-10-19 08:53:36,150 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.062 2021-10-19 08:53:41,095 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.227 INFO:tensorflow:epoch = 92.61658031088083, learning_rate = 0.00016606225, loss = 1.4865962e-05, step = 35750 (5.359 sec) 2021-10-19 08:53:41,291 [INFO] tensorflow: epoch = 92.61658031088083, learning_rate = 0.00016606225, loss = 1.4865962e-05, step = 35750 (5.359 sec) INFO:tensorflow:global_step/sec: 5.02498 2021-10-19 08:53:42,899 [INFO] tensorflow: global_step/sec: 5.02498 2021-10-19 08:53:46,108 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.949 INFO:tensorflow:epoch = 92.6839378238342, learning_rate = 0.00016463753, loss = 1.5750014e-05, step = 35776 (5.239 sec) 2021-10-19 08:53:46,530 [INFO] tensorflow: epoch = 92.6839378238342, learning_rate = 0.00016463753, loss = 1.5750014e-05, step = 35776 (5.239 sec) INFO:tensorflow:global_step/sec: 4.98244 2021-10-19 08:53:50,525 [INFO] tensorflow: global_step/sec: 4.98244 2021-10-19 08:53:51,127 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.924 INFO:tensorflow:epoch = 92.75388601036269, learning_rate = 0.00016317102, loss = 6.915979e-06, step = 35803 (5.383 sec) 2021-10-19 08:53:51,913 [INFO] tensorflow: epoch = 92.75388601036269, learning_rate = 0.00016317102, loss = 6.915979e-06, step = 35803 (5.383 sec) 2021-10-19 08:53:56,157 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.884 INFO:tensorflow:epoch = 92.82124352331607, learning_rate = 0.00016177095, loss = 3.1621024e-05, step = 35829 (5.234 sec) 2021-10-19 08:53:57,147 [INFO] tensorflow: epoch = 92.82124352331607, learning_rate = 0.00016177095, loss = 3.1621024e-05, step = 35829 (5.234 sec) INFO:tensorflow:global_step/sec: 4.99433 2021-10-19 08:53:58,134 [INFO] tensorflow: global_step/sec: 4.99433 2021-10-19 08:54:01,232 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.707 INFO:tensorflow:epoch = 92.88860103626943, learning_rate = 0.00016038305, loss = 6.846667e-06, step = 35855 (5.246 sec) 2021-10-19 08:54:02,394 [INFO] tensorflow: epoch = 92.88860103626943, learning_rate = 0.00016038305, loss = 6.846667e-06, step = 35855 (5.246 sec) INFO:tensorflow:global_step/sec: 4.93177 2021-10-19 08:54:05,839 [INFO] tensorflow: global_step/sec: 4.93177 2021-10-19 08:54:06,281 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.809 INFO:tensorflow:epoch = 92.9559585492228, learning_rate = 0.0001590072, loss = 2.6019654e-05, step = 35881 (5.295 sec) 2021-10-19 08:54:07,689 [INFO] tensorflow: epoch = 92.9559585492228, learning_rate = 0.0001590072, loss = 2.6019654e-05, step = 35881 (5.295 sec) 2021-10-19 08:54:11,095 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 93/120: loss: 0.00001 learning rate: 0.00016 Time taken: 0:01:17.320218 ETA: 0:34:47.645888 2021-10-19 08:54:11,294 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.948 INFO:tensorflow:epoch = 93.02590673575129, learning_rate = 0.00015759056, loss = 3.9137292e-05, step = 35908 (5.369 sec) 2021-10-19 08:54:13,058 [INFO] tensorflow: epoch = 93.02590673575129, learning_rate = 0.00015759056, loss = 3.9137292e-05, step = 35908 (5.369 sec) INFO:tensorflow:global_step/sec: 4.98377 2021-10-19 08:54:13,464 [INFO] tensorflow: global_step/sec: 4.98377 2021-10-19 08:54:16,277 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.071 INFO:tensorflow:epoch = 93.0958549222798, learning_rate = 0.00015618683, loss = 1.6183816e-05, step = 35935 (5.412 sec) 2021-10-19 08:54:18,470 [INFO] tensorflow: epoch = 93.0958549222798, learning_rate = 0.00015618683, loss = 1.6183816e-05, step = 35935 (5.412 sec) INFO:tensorflow:global_step/sec: 4.95146 2021-10-19 08:54:21,138 [INFO] tensorflow: global_step/sec: 4.95146 2021-10-19 08:54:21,335 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.774 INFO:tensorflow:epoch = 93.16321243523316, learning_rate = 0.00015484697, loss = 9.9047675e-06, step = 35961 (5.257 sec) 2021-10-19 08:54:23,727 [INFO] tensorflow: epoch = 93.16321243523316, learning_rate = 0.00015484697, loss = 9.9047675e-06, step = 35961 (5.257 sec) 2021-10-19 08:54:26,275 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.244 INFO:tensorflow:global_step/sec: 5.05425 2021-10-19 08:54:28,657 [INFO] tensorflow: global_step/sec: 5.05425 INFO:tensorflow:epoch = 93.23316062176166, learning_rate = 0.00015346754, loss = 1.5081101e-05, step = 35988 (5.338 sec) 2021-10-19 08:54:29,065 [INFO] tensorflow: epoch = 93.23316062176166, learning_rate = 0.00015346754, loss = 1.5081101e-05, step = 35988 (5.338 sec) 2021-10-19 08:54:31,249 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.105 INFO:tensorflow:epoch = 93.30310880829016, learning_rate = 0.00015210052, loss = 1.5894473e-05, step = 36015 (5.372 sec) 2021-10-19 08:54:34,437 [INFO] tensorflow: epoch = 93.30310880829016, learning_rate = 0.00015210052, loss = 1.5894473e-05, step = 36015 (5.372 sec) INFO:tensorflow:global_step/sec: 5.00308 2021-10-19 08:54:36,252 [INFO] tensorflow: global_step/sec: 5.00308 2021-10-19 08:54:36,253 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.988 INFO:tensorflow:epoch = 93.37046632124353, learning_rate = 0.00015079544, loss = 1.3353023e-05, step = 36041 (5.204 sec) 2021-10-19 08:54:39,641 [INFO] tensorflow: epoch = 93.37046632124353, learning_rate = 0.00015079544, loss = 1.3353023e-05, step = 36041 (5.204 sec) 2021-10-19 08:54:41,216 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.150 INFO:tensorflow:global_step/sec: 5.02719 2021-10-19 08:54:43,811 [INFO] tensorflow: global_step/sec: 5.02719 INFO:tensorflow:epoch = 93.44041450777202, learning_rate = 0.00014945224, loss = 3.076715e-05, step = 36068 (5.346 sec) 2021-10-19 08:54:44,988 [INFO] tensorflow: epoch = 93.44041450777202, learning_rate = 0.00014945224, loss = 3.076715e-05, step = 36068 (5.346 sec) 2021-10-19 08:54:46,168 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.196 INFO:tensorflow:epoch = 93.51036269430053, learning_rate = 0.000148121, loss = 1.3283834e-05, step = 36095 (5.311 sec) 2021-10-19 08:54:50,298 [INFO] tensorflow: epoch = 93.51036269430053, learning_rate = 0.000148121, loss = 1.3283834e-05, step = 36095 (5.311 sec) 2021-10-19 08:54:51,109 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.241 INFO:tensorflow:global_step/sec: 5.06828 2021-10-19 08:54:51,308 [INFO] tensorflow: global_step/sec: 5.06828 INFO:tensorflow:epoch = 93.58031088082902, learning_rate = 0.00014680148, loss = 2.7784949e-05, step = 36122 (5.310 sec) 2021-10-19 08:54:55,608 [INFO] tensorflow: epoch = 93.58031088082902, learning_rate = 0.00014680148, loss = 2.7784949e-05, step = 36122 (5.310 sec) 2021-10-19 08:54:56,003 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.435 INFO:tensorflow:global_step/sec: 5.08606 2021-10-19 08:54:58,780 [INFO] tensorflow: global_step/sec: 5.08606 INFO:tensorflow:epoch = 93.65025906735751, learning_rate = 0.00014549385, loss = 1.7647657e-05, step = 36149 (5.340 sec) 2021-10-19 08:55:00,948 [INFO] tensorflow: epoch = 93.65025906735751, learning_rate = 0.00014549385, loss = 1.7647657e-05, step = 36149 (5.340 sec) 2021-10-19 08:55:00,949 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.220 2021-10-19 08:55:05,975 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.897 INFO:tensorflow:epoch = 93.71761658031089, learning_rate = 0.00014424547, loss = 1.47696755e-05, step = 36175 (5.234 sec) 2021-10-19 08:55:06,182 [INFO] tensorflow: epoch = 93.71761658031089, learning_rate = 0.00014424547, loss = 1.47696755e-05, step = 36175 (5.234 sec) INFO:tensorflow:global_step/sec: 4.99276 2021-10-19 08:55:06,391 [INFO] tensorflow: global_step/sec: 4.99276 2021-10-19 08:55:10,973 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.010 INFO:tensorflow:epoch = 93.78497409326425, learning_rate = 0.00014300791, loss = 7.685747e-06, step = 36201 (5.226 sec) 2021-10-19 08:55:11,408 [INFO] tensorflow: epoch = 93.78497409326425, learning_rate = 0.00014300791, loss = 7.685747e-06, step = 36201 (5.226 sec) INFO:tensorflow:global_step/sec: 5.02283 2021-10-19 08:55:13,956 [INFO] tensorflow: global_step/sec: 5.02283 2021-10-19 08:55:15,949 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.099 INFO:tensorflow:epoch = 93.85492227979275, learning_rate = 0.00014173408, loss = 1.2019727e-05, step = 36228 (5.333 sec) 2021-10-19 08:55:16,740 [INFO] tensorflow: epoch = 93.85492227979275, learning_rate = 0.00014173408, loss = 1.2019727e-05, step = 36228 (5.333 sec) 2021-10-19 08:55:20,927 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.092 INFO:tensorflow:global_step/sec: 5.01717 2021-10-19 08:55:21,530 [INFO] tensorflow: global_step/sec: 5.01717 INFO:tensorflow:epoch = 93.92227979274612, learning_rate = 0.00014051795, loss = 9.66764e-06, step = 36254 (5.202 sec) 2021-10-19 08:55:21,943 [INFO] tensorflow: epoch = 93.92227979274612, learning_rate = 0.00014051795, loss = 9.66764e-06, step = 36254 (5.202 sec) 2021-10-19 08:55:25,930 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.990 INFO:tensorflow:epoch = 93.99222797927462, learning_rate = 0.00013926641, loss = 2.190076e-05, step = 36281 (5.361 sec) 2021-10-19 08:55:27,304 [INFO] tensorflow: epoch = 93.99222797927462, learning_rate = 0.00013926641, loss = 2.190076e-05, step = 36281 (5.361 sec) 2021-10-19 08:55:27,899 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 94/120: loss: 0.00002 learning rate: 0.00014 Time taken: 0:01:16.800320 ETA: 0:33:16.808318 INFO:tensorflow:global_step/sec: 5.03546 2021-10-19 08:55:29,077 [INFO] tensorflow: global_step/sec: 5.03546 2021-10-19 08:55:30,846 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.341 INFO:tensorflow:epoch = 94.06217616580311, learning_rate = 0.00013802564, loss = 7.1873646e-06, step = 36308 (5.308 sec) 2021-10-19 08:55:32,612 [INFO] tensorflow: epoch = 94.06217616580311, learning_rate = 0.00013802564, loss = 7.1873646e-06, step = 36308 (5.308 sec) 2021-10-19 08:55:35,825 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.087 INFO:tensorflow:global_step/sec: 5.02231 2021-10-19 08:55:36,643 [INFO] tensorflow: global_step/sec: 5.02231 INFO:tensorflow:epoch = 94.12953367875647, learning_rate = 0.00013684158, loss = 1.1873372e-05, step = 36334 (5.208 sec) 2021-10-19 08:55:37,820 [INFO] tensorflow: epoch = 94.12953367875647, learning_rate = 0.00013684158, loss = 1.1873372e-05, step = 36334 (5.208 sec) 2021-10-19 08:55:40,876 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.803 INFO:tensorflow:epoch = 94.19689119170985, learning_rate = 0.00013566756, loss = 1.4988162e-05, step = 36360 (5.226 sec) 2021-10-19 08:55:43,046 [INFO] tensorflow: epoch = 94.19689119170985, learning_rate = 0.00013566756, loss = 1.4988162e-05, step = 36360 (5.226 sec) INFO:tensorflow:global_step/sec: 4.99591 2021-10-19 08:55:44,249 [INFO] tensorflow: global_step/sec: 4.99591 2021-10-19 08:55:45,864 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.048 INFO:tensorflow:epoch = 94.26424870466322, learning_rate = 0.00013450361, loss = 2.3940614e-05, step = 36386 (5.271 sec) 2021-10-19 08:55:48,317 [INFO] tensorflow: epoch = 94.26424870466322, learning_rate = 0.00013450361, loss = 2.3940614e-05, step = 36386 (5.271 sec) 2021-10-19 08:55:50,877 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.950 INFO:tensorflow:global_step/sec: 4.9711 2021-10-19 08:55:51,893 [INFO] tensorflow: global_step/sec: 4.9711 INFO:tensorflow:epoch = 94.33419689119171, learning_rate = 0.00013330541, loss = 8.680683e-06, step = 36413 (5.331 sec) 2021-10-19 08:55:53,648 [INFO] tensorflow: epoch = 94.33419689119171, learning_rate = 0.00013330541, loss = 8.680683e-06, step = 36413 (5.331 sec) 2021-10-19 08:55:55,877 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.002 INFO:tensorflow:epoch = 94.40155440414509, learning_rate = 0.00013216173, loss = 5.8543287e-06, step = 36439 (5.283 sec) 2021-10-19 08:55:58,931 [INFO] tensorflow: epoch = 94.40155440414509, learning_rate = 0.00013216173, loss = 5.8543287e-06, step = 36439 (5.283 sec) INFO:tensorflow:global_step/sec: 4.95972 2021-10-19 08:55:59,555 [INFO] tensorflow: global_step/sec: 4.95972 2021-10-19 08:56:00,939 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.757 INFO:tensorflow:epoch = 94.47150259067358, learning_rate = 0.00013098436, loss = 1.8040766e-05, step = 36466 (5.378 sec) 2021-10-19 08:56:04,309 [INFO] tensorflow: epoch = 94.47150259067358, learning_rate = 0.00013098436, loss = 1.8040766e-05, step = 36466 (5.378 sec) 2021-10-19 08:56:05,895 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.181 INFO:tensorflow:global_step/sec: 5.03335 2021-10-19 08:56:07,105 [INFO] tensorflow: global_step/sec: 5.03335 INFO:tensorflow:epoch = 94.54145077720207, learning_rate = 0.00012981764, loss = 2.917508e-05, step = 36493 (5.386 sec) 2021-10-19 08:56:09,695 [INFO] tensorflow: epoch = 94.54145077720207, learning_rate = 0.00012981764, loss = 2.917508e-05, step = 36493 (5.386 sec) 2021-10-19 08:56:10,869 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.105 INFO:tensorflow:global_step/sec: 5.0115 2021-10-19 08:56:14,687 [INFO] tensorflow: global_step/sec: 5.0115 INFO:tensorflow:epoch = 94.61139896373058, learning_rate = 0.0001286613, loss = 1.541272e-05, step = 36520 (5.403 sec) 2021-10-19 08:56:15,099 [INFO] tensorflow: epoch = 94.61139896373058, learning_rate = 0.0001286613, loss = 1.541272e-05, step = 36520 (5.403 sec) 2021-10-19 08:56:15,924 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.788 INFO:tensorflow:epoch = 94.67875647668394, learning_rate = 0.00012755733, loss = 2.07607e-05, step = 36546 (5.215 sec) 2021-10-19 08:56:20,314 [INFO] tensorflow: epoch = 94.67875647668394, learning_rate = 0.00012755733, loss = 2.07607e-05, step = 36546 (5.215 sec) 2021-10-19 08:56:20,908 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.066 INFO:tensorflow:global_step/sec: 4.95626 2021-10-19 08:56:22,354 [INFO] tensorflow: global_step/sec: 4.95626 INFO:tensorflow:epoch = 94.74611398963731, learning_rate = 0.00012646295, loss = 2.2115459e-05, step = 36572 (5.271 sec) 2021-10-19 08:56:25,585 [INFO] tensorflow: epoch = 94.74611398963731, learning_rate = 0.00012646295, loss = 2.2115459e-05, step = 36572 (5.271 sec) 2021-10-19 08:56:25,986 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.691 INFO:tensorflow:global_step/sec: 5.00267 2021-10-19 08:56:29,950 [INFO] tensorflow: global_step/sec: 5.00267 INFO:tensorflow:epoch = 94.8160621761658, learning_rate = 0.00012533649, loss = 1.4340565e-05, step = 36599 (5.391 sec) 2021-10-19 08:56:30,976 [INFO] tensorflow: epoch = 94.8160621761658, learning_rate = 0.00012533649, loss = 1.4340565e-05, step = 36599 (5.391 sec) 2021-10-19 08:56:30,976 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.044 2021-10-19 08:56:36,010 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.866 INFO:tensorflow:epoch = 94.88341968911918, learning_rate = 0.00012426118, loss = 1.1717524e-05, step = 36625 (5.231 sec) 2021-10-19 08:56:36,207 [INFO] tensorflow: epoch = 94.88341968911918, learning_rate = 0.00012426118, loss = 1.1717524e-05, step = 36625 (5.231 sec) INFO:tensorflow:global_step/sec: 4.97239 2021-10-19 08:56:37,593 [INFO] tensorflow: global_step/sec: 4.97239 2021-10-19 08:56:40,956 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.217 INFO:tensorflow:epoch = 94.95336787564767, learning_rate = 0.00012315421, loss = 9.3269555e-06, step = 36652 (5.345 sec) 2021-10-19 08:56:41,552 [INFO] tensorflow: epoch = 94.95336787564767, learning_rate = 0.00012315421, loss = 9.3269555e-06, step = 36652 (5.345 sec) INFO:tensorflow:global_step/sec: 5.00646 2021-10-19 08:56:45,183 [INFO] tensorflow: global_step/sec: 5.00646 2021-10-19 08:56:45,184 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 95/120: loss: 0.00001 learning rate: 0.00012 Time taken: 0:01:17.276875 ETA: 0:32:11.921887 2021-10-19 08:56:45,954 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.013 INFO:tensorflow:epoch = 95.02072538860104, learning_rate = 0.00012209773, loss = 6.8059267e-06, step = 36678 (5.210 sec) 2021-10-19 08:56:46,762 [INFO] tensorflow: epoch = 95.02072538860104, learning_rate = 0.00012209773, loss = 6.8059267e-06, step = 36678 (5.210 sec) 2021-10-19 08:56:50,927 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.108 INFO:tensorflow:epoch = 95.09067357512954, learning_rate = 0.00012101003, loss = 5.124215e-06, step = 36705 (5.372 sec) 2021-10-19 08:56:52,134 [INFO] tensorflow: epoch = 95.09067357512954, learning_rate = 0.00012101003, loss = 5.124215e-06, step = 36705 (5.372 sec) INFO:tensorflow:global_step/sec: 5.04462 2021-10-19 08:56:52,716 [INFO] tensorflow: global_step/sec: 5.04462 2021-10-19 08:56:55,945 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.932 INFO:tensorflow:epoch = 95.15803108808291, learning_rate = 0.00011997183, loss = 1.5854173e-05, step = 36731 (5.216 sec) 2021-10-19 08:56:57,350 [INFO] tensorflow: epoch = 95.15803108808291, learning_rate = 0.00011997183, loss = 1.5854173e-05, step = 36731 (5.216 sec) INFO:tensorflow:global_step/sec: 4.99576 2021-10-19 08:57:00,322 [INFO] tensorflow: global_step/sec: 4.99576 2021-10-19 08:57:00,903 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.168 INFO:tensorflow:epoch = 95.2279792746114, learning_rate = 0.00011890308, loss = 2.089667e-05, step = 36758 (5.342 sec) 2021-10-19 08:57:02,691 [INFO] tensorflow: epoch = 95.2279792746114, learning_rate = 0.00011890308, loss = 2.089667e-05, step = 36758 (5.342 sec) 2021-10-19 08:57:05,851 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.212 INFO:tensorflow:global_step/sec: 5.03221 2021-10-19 08:57:07,873 [INFO] tensorflow: global_step/sec: 5.03221 INFO:tensorflow:epoch = 95.2979274611399, learning_rate = 0.000117843956, loss = 6.5442637e-06, step = 36785 (5.378 sec) 2021-10-19 08:57:08,070 [INFO] tensorflow: epoch = 95.2979274611399, learning_rate = 0.000117843956, loss = 6.5442637e-06, step = 36785 (5.378 sec) 2021-10-19 08:57:10,814 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.153 INFO:tensorflow:epoch = 95.3678756476684, learning_rate = 0.00011679415, loss = 1.4324537e-05, step = 36812 (5.336 sec) 2021-10-19 08:57:13,405 [INFO] tensorflow: epoch = 95.3678756476684, learning_rate = 0.00011679415, loss = 1.4324537e-05, step = 36812 (5.336 sec) INFO:tensorflow:global_step/sec: 5.04631 2021-10-19 08:57:15,404 [INFO] tensorflow: global_step/sec: 5.04631 2021-10-19 08:57:15,801 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.056 INFO:tensorflow:epoch = 95.43782383419689, learning_rate = 0.000115753806, loss = 6.9183548e-06, step = 36839 (5.305 sec) 2021-10-19 08:57:18,711 [INFO] tensorflow: epoch = 95.43782383419689, learning_rate = 0.000115753806, loss = 6.9183548e-06, step = 36839 (5.305 sec) 2021-10-19 08:57:20,700 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.414 INFO:tensorflow:global_step/sec: 5.05705 2021-10-19 08:57:22,918 [INFO] tensorflow: global_step/sec: 5.05705 INFO:tensorflow:epoch = 95.50518134715026, learning_rate = 0.00011476071, loss = 5.3679437e-06, step = 36865 (5.228 sec) 2021-10-19 08:57:23,939 [INFO] tensorflow: epoch = 95.50518134715026, learning_rate = 0.00011476071, loss = 5.3679437e-06, step = 36865 (5.228 sec) 2021-10-19 08:57:25,690 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.039 INFO:tensorflow:epoch = 95.57512953367876, learning_rate = 0.00011373837, loss = 2.058361e-05, step = 36892 (5.326 sec) 2021-10-19 08:57:29,265 [INFO] tensorflow: epoch = 95.57512953367876, learning_rate = 0.00011373837, loss = 2.058361e-05, step = 36892 (5.326 sec) INFO:tensorflow:global_step/sec: 5.03959 2021-10-19 08:57:30,458 [INFO] tensorflow: global_step/sec: 5.03959 2021-10-19 08:57:30,653 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.151 INFO:tensorflow:epoch = 95.64507772020725, learning_rate = 0.000112725254, loss = 1.0926993e-05, step = 36919 (5.325 sec) 2021-10-19 08:57:34,589 [INFO] tensorflow: epoch = 95.64507772020725, learning_rate = 0.000112725254, loss = 1.0926993e-05, step = 36919 (5.325 sec) 2021-10-19 08:57:35,582 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.289 INFO:tensorflow:global_step/sec: 5.0276 2021-10-19 08:57:38,016 [INFO] tensorflow: global_step/sec: 5.0276 INFO:tensorflow:epoch = 95.71243523316062, learning_rate = 0.000111758025, loss = 1.8304874e-05, step = 36945 (5.231 sec) 2021-10-19 08:57:39,821 [INFO] tensorflow: epoch = 95.71243523316062, learning_rate = 0.000111758025, loss = 1.8304874e-05, step = 36945 (5.231 sec) 2021-10-19 08:57:40,639 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.778 INFO:tensorflow:epoch = 95.78238341968913, learning_rate = 0.00011076254, loss = 2.4905588e-05, step = 36972 (5.398 sec) 2021-10-19 08:57:45,219 [INFO] tensorflow: epoch = 95.78238341968913, learning_rate = 0.00011076254, loss = 2.4905588e-05, step = 36972 (5.398 sec) INFO:tensorflow:global_step/sec: 4.99939 2021-10-19 08:57:45,617 [INFO] tensorflow: global_step/sec: 4.99939 2021-10-19 08:57:45,619 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.081 INFO:tensorflow:epoch = 95.84974093264249, learning_rate = 0.00010981226, loss = 1.1249982e-05, step = 36998 (5.285 sec) 2021-10-19 08:57:50,505 [INFO] tensorflow: epoch = 95.84974093264249, learning_rate = 0.00010981226, loss = 1.1249982e-05, step = 36998 (5.285 sec) 2021-10-19 08:57:50,744 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.514 INFO:tensorflow:global_step/sec: 4.94045 2021-10-19 08:57:53,309 [INFO] tensorflow: global_step/sec: 4.94045 2021-10-19 08:57:55,655 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.364 INFO:tensorflow:epoch = 95.91968911917098, learning_rate = 0.000108834014, loss = 1.8473427e-05, step = 37025 (5.365 sec) 2021-10-19 08:57:55,870 [INFO] tensorflow: epoch = 95.91968911917098, learning_rate = 0.000108834014, loss = 1.8473427e-05, step = 37025 (5.365 sec) 2021-10-19 08:58:00,631 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.099 INFO:tensorflow:global_step/sec: 5.05512 2021-10-19 08:58:00,826 [INFO] tensorflow: global_step/sec: 5.05512 INFO:tensorflow:epoch = 95.98963730569949, learning_rate = 0.00010786458, loss = 2.344423e-05, step = 37052 (5.351 sec) 2021-10-19 08:58:01,220 [INFO] tensorflow: epoch = 95.98963730569949, learning_rate = 0.00010786458, loss = 2.344423e-05, step = 37052 (5.351 sec) 2021-10-19 08:58:02,044 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 96/120: loss: 0.00000 learning rate: 0.00011 Time taken: 0:01:16.828503 ETA: 0:30:43.884075 2021-10-19 08:58:05,637 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.978 INFO:tensorflow:epoch = 96.05699481865285, learning_rate = 0.00010693927, loss = 3.0099232e-05, step = 37078 (5.233 sec) 2021-10-19 08:58:06,453 [INFO] tensorflow: epoch = 96.05699481865285, learning_rate = 0.00010693927, loss = 3.0099232e-05, step = 37078 (5.233 sec) INFO:tensorflow:global_step/sec: 4.97929 2021-10-19 08:58:08,458 [INFO] tensorflow: global_step/sec: 4.97929 2021-10-19 08:58:10,660 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.911 INFO:tensorflow:epoch = 96.12435233160622, learning_rate = 0.00010602168, loss = 6.2924746e-06, step = 37104 (5.211 sec) 2021-10-19 08:58:11,664 [INFO] tensorflow: epoch = 96.12435233160622, learning_rate = 0.00010602168, loss = 6.2924746e-06, step = 37104 (5.211 sec) 2021-10-19 08:58:15,685 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.901 INFO:tensorflow:global_step/sec: 4.98132 2021-10-19 08:58:16,086 [INFO] tensorflow: global_step/sec: 4.98132 INFO:tensorflow:epoch = 96.19170984455958, learning_rate = 0.00010511208, loss = 4.1988424e-06, step = 37130 (5.222 sec) 2021-10-19 08:58:16,886 [INFO] tensorflow: epoch = 96.19170984455958, learning_rate = 0.00010511208, loss = 4.1988424e-06, step = 37130 (5.222 sec) 2021-10-19 08:58:20,656 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.119 INFO:tensorflow:epoch = 96.26165803108809, learning_rate = 0.00010417579, loss = 1.023043e-05, step = 37157 (5.384 sec) 2021-10-19 08:58:22,270 [INFO] tensorflow: epoch = 96.26165803108809, learning_rate = 0.00010417579, loss = 1.023043e-05, step = 37157 (5.384 sec) INFO:tensorflow:global_step/sec: 4.98916 2021-10-19 08:58:23,703 [INFO] tensorflow: global_step/sec: 4.98916 2021-10-19 08:58:25,738 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.676 INFO:tensorflow:epoch = 96.32901554404145, learning_rate = 0.000103281935, loss = 9.763027e-06, step = 37183 (5.278 sec) 2021-10-19 08:58:27,549 [INFO] tensorflow: epoch = 96.32901554404145, learning_rate = 0.000103281935, loss = 9.763027e-06, step = 37183 (5.278 sec) 2021-10-19 08:58:30,748 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.965 INFO:tensorflow:global_step/sec: 4.96202 2021-10-19 08:58:31,361 [INFO] tensorflow: global_step/sec: 4.96202 INFO:tensorflow:epoch = 96.39896373056995, learning_rate = 0.000102361955, loss = 5.4344496e-06, step = 37210 (5.381 sec) 2021-10-19 08:58:32,929 [INFO] tensorflow: epoch = 96.39896373056995, learning_rate = 0.000102361955, loss = 5.4344496e-06, step = 37210 (5.381 sec) 2021-10-19 08:58:35,642 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.435 INFO:tensorflow:epoch = 96.46891191709845, learning_rate = 0.00010145007, loss = 2.3835095e-05, step = 37237 (5.338 sec) 2021-10-19 08:58:38,267 [INFO] tensorflow: epoch = 96.46891191709845, learning_rate = 0.00010145007, loss = 2.3835095e-05, step = 37237 (5.338 sec) INFO:tensorflow:global_step/sec: 5.05289 2021-10-19 08:58:38,881 [INFO] tensorflow: global_step/sec: 5.05289 2021-10-19 08:58:40,671 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.889 INFO:tensorflow:epoch = 96.53886010362694, learning_rate = 0.0001005464, loss = 2.115618e-05, step = 37264 (5.387 sec) 2021-10-19 08:58:43,654 [INFO] tensorflow: epoch = 96.53886010362694, learning_rate = 0.0001005464, loss = 2.115618e-05, step = 37264 (5.387 sec) 2021-10-19 08:58:45,629 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.171 INFO:tensorflow:global_step/sec: 5.05189 2021-10-19 08:58:46,403 [INFO] tensorflow: global_step/sec: 5.05189 INFO:tensorflow:epoch = 96.60880829015544, learning_rate = 9.965069e-05, loss = 1.5487032e-05, step = 37291 (5.360 sec) 2021-10-19 08:58:49,014 [INFO] tensorflow: epoch = 96.60880829015544, learning_rate = 9.965069e-05, loss = 1.5487032e-05, step = 37291 (5.360 sec) 2021-10-19 08:58:50,614 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.064 INFO:tensorflow:global_step/sec: 4.99645 2021-10-19 08:58:54,009 [INFO] tensorflow: global_step/sec: 4.99645 INFO:tensorflow:epoch = 96.67875647668394, learning_rate = 9.876306e-05, loss = 2.4351219e-05, step = 37318 (5.382 sec) 2021-10-19 08:58:54,396 [INFO] tensorflow: epoch = 96.67875647668394, learning_rate = 9.876306e-05, loss = 2.4351219e-05, step = 37318 (5.382 sec) 2021-10-19 08:58:55,633 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.923 INFO:tensorflow:epoch = 96.74870466321244, learning_rate = 9.7883334e-05, loss = 5.482138e-06, step = 37345 (5.379 sec) 2021-10-19 08:58:59,775 [INFO] tensorflow: epoch = 96.74870466321244, learning_rate = 9.7883334e-05, loss = 5.482138e-06, step = 37345 (5.379 sec) 2021-10-19 08:59:00,560 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.302 INFO:tensorflow:global_step/sec: 5.03037 2021-10-19 08:59:01,563 [INFO] tensorflow: global_step/sec: 5.03037 INFO:tensorflow:epoch = 96.81865284974094, learning_rate = 9.7011354e-05, loss = 1.9767584e-05, step = 37372 (5.386 sec) 2021-10-19 08:59:05,161 [INFO] tensorflow: epoch = 96.81865284974094, learning_rate = 9.7011354e-05, loss = 1.9767584e-05, step = 37372 (5.386 sec) 2021-10-19 08:59:05,553 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.026 INFO:tensorflow:global_step/sec: 5.0156 2021-10-19 08:59:09,139 [INFO] tensorflow: global_step/sec: 5.0156 INFO:tensorflow:epoch = 96.88601036269431, learning_rate = 9.617905e-05, loss = 1.0441205e-05, step = 37398 (5.210 sec) 2021-10-19 08:59:10,371 [INFO] tensorflow: epoch = 96.88601036269431, learning_rate = 9.617905e-05, loss = 1.0441205e-05, step = 37398 (5.210 sec) 2021-10-19 08:59:10,574 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.919 2021-10-19 08:59:15,564 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.041 INFO:tensorflow:epoch = 96.9559585492228, learning_rate = 9.532224e-05, loss = 1.1099461e-05, step = 37425 (5.389 sec) 2021-10-19 08:59:15,760 [INFO] tensorflow: epoch = 96.9559585492228, learning_rate = 9.532224e-05, loss = 1.1099461e-05, step = 37425 (5.389 sec) INFO:tensorflow:global_step/sec: 5.00021 2021-10-19 08:59:16,739 [INFO] tensorflow: global_step/sec: 5.00021 2021-10-19 08:59:19,155 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 97/120: loss: 0.00003 learning rate: 0.00009 Time taken: 0:01:17.142336 ETA: 0:29:34.273726 2021-10-19 08:59:20,566 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.995 INFO:tensorflow:epoch = 97.02331606217616, learning_rate = 9.450443e-05, loss = 9.2020655e-06, step = 37451 (5.231 sec) 2021-10-19 08:59:20,991 [INFO] tensorflow: epoch = 97.02331606217616, learning_rate = 9.450443e-05, loss = 9.2020655e-06, step = 37451 (5.231 sec) INFO:tensorflow:global_step/sec: 4.99145 2021-10-19 08:59:24,352 [INFO] tensorflow: global_step/sec: 4.99145 2021-10-19 08:59:25,509 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.232 INFO:tensorflow:epoch = 97.09326424870467, learning_rate = 9.366264e-05, loss = 1.1157204e-05, step = 37478 (5.311 sec) 2021-10-19 08:59:26,302 [INFO] tensorflow: epoch = 97.09326424870467, learning_rate = 9.366264e-05, loss = 1.1157204e-05, step = 37478 (5.311 sec) 2021-10-19 08:59:30,478 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.126 INFO:tensorflow:epoch = 97.16321243523316, learning_rate = 9.282825e-05, loss = 2.0046758e-05, step = 37505 (5.350 sec) 2021-10-19 08:59:31,653 [INFO] tensorflow: epoch = 97.16321243523316, learning_rate = 9.282825e-05, loss = 2.0046758e-05, step = 37505 (5.350 sec) INFO:tensorflow:global_step/sec: 5.0574 2021-10-19 08:59:31,866 [INFO] tensorflow: global_step/sec: 5.0574 2021-10-19 08:59:35,478 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.003 INFO:tensorflow:epoch = 97.23056994818653, learning_rate = 9.203184e-05, loss = 2.0275369e-05, step = 37531 (5.236 sec) 2021-10-19 08:59:36,888 [INFO] tensorflow: epoch = 97.23056994818653, learning_rate = 9.203184e-05, loss = 2.0275369e-05, step = 37531 (5.236 sec) INFO:tensorflow:global_step/sec: 4.99322 2021-10-19 08:59:39,476 [INFO] tensorflow: global_step/sec: 4.99322 2021-10-19 08:59:40,481 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.989 INFO:tensorflow:epoch = 97.30051813471503, learning_rate = 9.121198e-05, loss = 1.6680144e-05, step = 37558 (5.341 sec) 2021-10-19 08:59:42,230 [INFO] tensorflow: epoch = 97.30051813471503, learning_rate = 9.121198e-05, loss = 1.6680144e-05, step = 37558 (5.341 sec) 2021-10-19 08:59:45,366 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.474 INFO:tensorflow:global_step/sec: 5.07162 2021-10-19 08:59:46,969 [INFO] tensorflow: global_step/sec: 5.07162 INFO:tensorflow:epoch = 97.37046632124353, learning_rate = 9.039951e-05, loss = 1.09960365e-05, step = 37585 (5.379 sec) 2021-10-19 08:59:47,608 [INFO] tensorflow: epoch = 97.37046632124353, learning_rate = 9.039951e-05, loss = 1.09960365e-05, step = 37585 (5.379 sec) 2021-10-19 08:59:50,368 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.994 INFO:tensorflow:epoch = 97.44041450777202, learning_rate = 8.95942e-05, loss = 1.4261659e-05, step = 37612 (5.340 sec) 2021-10-19 08:59:52,948 [INFO] tensorflow: epoch = 97.44041450777202, learning_rate = 8.95942e-05, loss = 1.4261659e-05, step = 37612 (5.340 sec) INFO:tensorflow:global_step/sec: 5.02778 2021-10-19 08:59:54,527 [INFO] tensorflow: global_step/sec: 5.02778 2021-10-19 08:59:55,341 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.109 INFO:tensorflow:epoch = 97.5077720207254, learning_rate = 8.882553e-05, loss = 2.1972435e-05, step = 37638 (5.198 sec) 2021-10-19 08:59:58,145 [INFO] tensorflow: epoch = 97.5077720207254, learning_rate = 8.882553e-05, loss = 2.1972435e-05, step = 37638 (5.198 sec) 2021-10-19 09:00:00,278 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.257 INFO:tensorflow:global_step/sec: 5.00187 2021-10-19 09:00:02,124 [INFO] tensorflow: global_step/sec: 5.00187 INFO:tensorflow:epoch = 97.57772020725389, learning_rate = 8.803432e-05, loss = 8.887251e-06, step = 37665 (5.379 sec) 2021-10-19 09:00:03,524 [INFO] tensorflow: epoch = 97.57772020725389, learning_rate = 8.803432e-05, loss = 8.887251e-06, step = 37665 (5.379 sec) 2021-10-19 09:00:05,327 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.809 INFO:tensorflow:epoch = 97.64507772020725, learning_rate = 8.727895e-05, loss = 1.0121067e-05, step = 37691 (5.229 sec) 2021-10-19 09:00:08,754 [INFO] tensorflow: epoch = 97.64507772020725, learning_rate = 8.727895e-05, loss = 1.0121067e-05, step = 37691 (5.229 sec) INFO:tensorflow:global_step/sec: 4.9837 2021-10-19 09:00:09,749 [INFO] tensorflow: global_step/sec: 4.9837 2021-10-19 09:00:10,366 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.847 INFO:tensorflow:epoch = 97.71502590673576, learning_rate = 8.6501524e-05, loss = 1.1292641e-05, step = 37718 (5.386 sec) 2021-10-19 09:00:14,139 [INFO] tensorflow: epoch = 97.71502590673576, learning_rate = 8.6501524e-05, loss = 1.1292641e-05, step = 37718 (5.386 sec) 2021-10-19 09:00:15,333 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.134 INFO:tensorflow:global_step/sec: 5.01301 2021-10-19 09:00:17,329 [INFO] tensorflow: global_step/sec: 5.01301 INFO:tensorflow:epoch = 97.78497409326425, learning_rate = 8.573101e-05, loss = 9.648277e-06, step = 37745 (5.388 sec) 2021-10-19 09:00:19,528 [INFO] tensorflow: epoch = 97.78497409326425, learning_rate = 8.573101e-05, loss = 9.648277e-06, step = 37745 (5.388 sec) 2021-10-19 09:00:20,295 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.155 INFO:tensorflow:epoch = 97.85233160621762, learning_rate = 8.499548e-05, loss = 1.0285985e-05, step = 37771 (5.263 sec) 2021-10-19 09:00:24,791 [INFO] tensorflow: epoch = 97.85233160621762, learning_rate = 8.499548e-05, loss = 1.0285985e-05, step = 37771 (5.263 sec) INFO:tensorflow:global_step/sec: 4.96048 2021-10-19 09:00:24,989 [INFO] tensorflow: global_step/sec: 4.96048 2021-10-19 09:00:25,388 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.637 INFO:tensorflow:epoch = 97.92227979274612, learning_rate = 8.4238396e-05, loss = 1.9626623e-05, step = 37798 (5.327 sec) 2021-10-19 09:00:30,118 [INFO] tensorflow: epoch = 97.92227979274612, learning_rate = 8.4238396e-05, loss = 1.9626623e-05, step = 37798 (5.327 sec) 2021-10-19 09:00:30,322 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.267 INFO:tensorflow:global_step/sec: 5.04483 2021-10-19 09:00:32,522 [INFO] tensorflow: global_step/sec: 5.04483 2021-10-19 09:00:35,258 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.261 INFO:tensorflow:epoch = 97.99222797927462, learning_rate = 8.348796e-05, loss = 1.3189083e-05, step = 37825 (5.360 sec) 2021-10-19 09:00:35,479 [INFO] tensorflow: epoch = 97.99222797927462, learning_rate = 8.348796e-05, loss = 1.3189083e-05, step = 37825 (5.360 sec) 2021-10-19 09:00:36,122 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 98/120: loss: 0.00001 learning rate: 0.00008 Time taken: 0:01:16.959137 ETA: 0:28:13.101008 INFO:tensorflow:global_step/sec: 5.03785 2021-10-19 09:00:40,065 [INFO] tensorflow: global_step/sec: 5.03785 2021-10-19 09:00:40,263 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.983 INFO:tensorflow:epoch = 98.06217616580311, learning_rate = 8.274429e-05, loss = 1.20784025e-05, step = 37852 (5.385 sec) 2021-10-19 09:00:40,864 [INFO] tensorflow: epoch = 98.06217616580311, learning_rate = 8.274429e-05, loss = 1.20784025e-05, step = 37852 (5.385 sec) 2021-10-19 09:00:45,238 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.103 INFO:tensorflow:epoch = 98.13212435233162, learning_rate = 8.20071e-05, loss = 1.4357423e-05, step = 37879 (5.338 sec) 2021-10-19 09:00:46,202 [INFO] tensorflow: epoch = 98.13212435233162, learning_rate = 8.20071e-05, loss = 1.4357423e-05, step = 37879 (5.338 sec) INFO:tensorflow:global_step/sec: 5.05288 2021-10-19 09:00:47,585 [INFO] tensorflow: global_step/sec: 5.05288 2021-10-19 09:00:50,194 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.178 INFO:tensorflow:epoch = 98.19948186528498, learning_rate = 8.13036e-05, loss = 1.5986827e-05, step = 37905 (5.215 sec) 2021-10-19 09:00:51,417 [INFO] tensorflow: epoch = 98.19948186528498, learning_rate = 8.13036e-05, loss = 1.5986827e-05, step = 37905 (5.215 sec) INFO:tensorflow:global_step/sec: 4.97054 2021-10-19 09:00:55,230 [INFO] tensorflow: global_step/sec: 4.97054 2021-10-19 09:00:55,232 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.850 INFO:tensorflow:epoch = 98.26683937823834, learning_rate = 8.060606e-05, loss = 7.3053693e-06, step = 37931 (5.210 sec) 2021-10-19 09:00:56,627 [INFO] tensorflow: epoch = 98.26683937823834, learning_rate = 8.060606e-05, loss = 7.3053693e-06, step = 37931 (5.210 sec) 2021-10-19 09:01:00,173 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.242 INFO:tensorflow:epoch = 98.33678756476684, learning_rate = 7.988799e-05, loss = 1.0376067e-05, step = 37958 (5.324 sec) 2021-10-19 09:01:01,951 [INFO] tensorflow: epoch = 98.33678756476684, learning_rate = 7.988799e-05, loss = 1.0376067e-05, step = 37958 (5.324 sec) INFO:tensorflow:global_step/sec: 5.03774 2021-10-19 09:01:02,773 [INFO] tensorflow: global_step/sec: 5.03774 2021-10-19 09:01:05,169 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.018 INFO:tensorflow:epoch = 98.40414507772022, learning_rate = 7.92026e-05, loss = 2.0974061e-05, step = 37984 (5.209 sec) 2021-10-19 09:01:07,160 [INFO] tensorflow: epoch = 98.40414507772022, learning_rate = 7.92026e-05, loss = 2.0974061e-05, step = 37984 (5.209 sec) 2021-10-19 09:01:10,150 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.079 INFO:tensorflow:global_step/sec: 5.02019 2021-10-19 09:01:10,343 [INFO] tensorflow: global_step/sec: 5.02019 INFO:tensorflow:epoch = 98.47409326424871, learning_rate = 7.84971e-05, loss = 1.4358211e-05, step = 38011 (5.385 sec) 2021-10-19 09:01:12,546 [INFO] tensorflow: epoch = 98.47409326424871, learning_rate = 7.84971e-05, loss = 1.4358211e-05, step = 38011 (5.385 sec) 2021-10-19 09:01:15,166 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.938 INFO:tensorflow:epoch = 98.54145077720207, learning_rate = 7.782357e-05, loss = 2.5185172e-05, step = 38037 (5.231 sec) 2021-10-19 09:01:17,777 [INFO] tensorflow: epoch = 98.54145077720207, learning_rate = 7.782357e-05, loss = 2.5185172e-05, step = 38037 (5.231 sec) INFO:tensorflow:global_step/sec: 4.9805 2021-10-19 09:01:17,973 [INFO] tensorflow: global_step/sec: 4.9805 2021-10-19 09:01:20,202 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.857 INFO:tensorflow:epoch = 98.60880829015544, learning_rate = 7.715589e-05, loss = 1.108152e-05, step = 38063 (5.203 sec) 2021-10-19 09:01:22,979 [INFO] tensorflow: epoch = 98.60880829015544, learning_rate = 7.715589e-05, loss = 1.108152e-05, step = 38063 (5.203 sec) 2021-10-19 09:01:25,168 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.139 INFO:tensorflow:global_step/sec: 5.01031 2021-10-19 09:01:25,557 [INFO] tensorflow: global_step/sec: 5.01031 INFO:tensorflow:epoch = 98.67875647668394, learning_rate = 7.646863e-05, loss = 1.7102482e-05, step = 38090 (5.343 sec) 2021-10-19 09:01:28,323 [INFO] tensorflow: epoch = 98.67875647668394, learning_rate = 7.646863e-05, loss = 1.7102482e-05, step = 38090 (5.343 sec) 2021-10-19 09:01:30,081 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.355 INFO:tensorflow:global_step/sec: 5.01883 2021-10-19 09:01:33,128 [INFO] tensorflow: global_step/sec: 5.01883 INFO:tensorflow:epoch = 98.74611398963731, learning_rate = 7.581257e-05, loss = 2.268009e-05, step = 38116 (5.195 sec) 2021-10-19 09:01:33,517 [INFO] tensorflow: epoch = 98.74611398963731, learning_rate = 7.581257e-05, loss = 2.268009e-05, step = 38116 (5.195 sec) 2021-10-19 09:01:35,127 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.821 INFO:tensorflow:epoch = 98.81347150259067, learning_rate = 7.5162134e-05, loss = 1.0757229e-05, step = 38142 (5.227 sec) 2021-10-19 09:01:38,745 [INFO] tensorflow: epoch = 98.81347150259067, learning_rate = 7.5162134e-05, loss = 1.0757229e-05, step = 38142 (5.227 sec) 2021-10-19 09:01:40,181 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.788 INFO:tensorflow:global_step/sec: 4.97476 2021-10-19 09:01:40,767 [INFO] tensorflow: global_step/sec: 4.97476 INFO:tensorflow:epoch = 98.88082901554405, learning_rate = 7.451729e-05, loss = 1.0508014e-05, step = 38168 (5.215 sec) 2021-10-19 09:01:43,960 [INFO] tensorflow: epoch = 98.88082901554405, learning_rate = 7.451729e-05, loss = 1.0508014e-05, step = 38168 (5.215 sec) 2021-10-19 09:01:45,155 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.111 INFO:tensorflow:global_step/sec: 4.99785 2021-10-19 09:01:48,370 [INFO] tensorflow: global_step/sec: 4.99785 INFO:tensorflow:epoch = 98.94818652849742, learning_rate = 7.387797e-05, loss = 1.8275994e-05, step = 38194 (5.203 sec) 2021-10-19 09:01:49,163 [INFO] tensorflow: epoch = 98.94818652849742, learning_rate = 7.387797e-05, loss = 1.8275994e-05, step = 38194 (5.203 sec) 2021-10-19 09:01:50,168 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.946 2021-10-19 09:01:53,231 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 99/120: loss: 0.00001 learning rate: 0.00007 Time taken: 0:01:17.116220 ETA: 0:26:59.440620 INFO:tensorflow:epoch = 99.01554404145078, learning_rate = 7.324414e-05, loss = 1.00779e-05, step = 38220 (5.275 sec) 2021-10-19 09:01:54,437 [INFO] tensorflow: epoch = 99.01554404145078, learning_rate = 7.324414e-05, loss = 1.00779e-05, step = 38220 (5.275 sec) 2021-10-19 09:01:55,261 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.637 INFO:tensorflow:global_step/sec: 4.93562 2021-10-19 09:01:56,069 [INFO] tensorflow: global_step/sec: 4.93562 INFO:tensorflow:epoch = 99.08549222797927, learning_rate = 7.2591654e-05, loss = 5.7927277e-06, step = 38247 (5.392 sec) 2021-10-19 09:01:59,829 [INFO] tensorflow: epoch = 99.08549222797927, learning_rate = 7.2591654e-05, loss = 5.7927277e-06, step = 38247 (5.392 sec) 2021-10-19 09:02:00,229 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.131 INFO:tensorflow:global_step/sec: 5.01934 2021-10-19 09:02:03,640 [INFO] tensorflow: global_step/sec: 5.01934 INFO:tensorflow:epoch = 99.15284974093265, learning_rate = 7.196893e-05, loss = 1.0789016e-05, step = 38273 (5.197 sec) 2021-10-19 09:02:05,026 [INFO] tensorflow: epoch = 99.15284974093265, learning_rate = 7.196893e-05, loss = 1.0789016e-05, step = 38273 (5.197 sec) 2021-10-19 09:02:05,244 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.942 INFO:tensorflow:epoch = 99.22020725388602, learning_rate = 7.13514e-05, loss = 1.3433561e-05, step = 38299 (5.225 sec) 2021-10-19 09:02:10,251 [INFO] tensorflow: epoch = 99.22020725388602, learning_rate = 7.13514e-05, loss = 1.3433561e-05, step = 38299 (5.225 sec) 2021-10-19 09:02:10,251 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.974 INFO:tensorflow:global_step/sec: 4.98251 2021-10-19 09:02:11,267 [INFO] tensorflow: global_step/sec: 4.98251 2021-10-19 09:02:15,291 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.843 INFO:tensorflow:epoch = 99.28756476683938, learning_rate = 7.073932e-05, loss = 2.0875359e-05, step = 38325 (5.253 sec) 2021-10-19 09:02:15,504 [INFO] tensorflow: epoch = 99.28756476683938, learning_rate = 7.073932e-05, loss = 2.0875359e-05, step = 38325 (5.253 sec) INFO:tensorflow:global_step/sec: 4.97321 2021-10-19 09:02:18,908 [INFO] tensorflow: global_step/sec: 4.97321 2021-10-19 09:02:20,296 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.981 INFO:tensorflow:epoch = 99.35751295336787, learning_rate = 7.010914e-05, loss = 1.209274e-05, step = 38352 (5.393 sec) 2021-10-19 09:02:20,897 [INFO] tensorflow: epoch = 99.35751295336787, learning_rate = 7.010914e-05, loss = 1.209274e-05, step = 38352 (5.393 sec) 2021-10-19 09:02:25,259 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.152 INFO:tensorflow:epoch = 99.42746113989638, learning_rate = 6.948458e-05, loss = 1.7817607e-05, step = 38379 (5.364 sec) 2021-10-19 09:02:26,262 [INFO] tensorflow: epoch = 99.42746113989638, learning_rate = 6.948458e-05, loss = 1.7817607e-05, step = 38379 (5.364 sec) INFO:tensorflow:global_step/sec: 5.0227 2021-10-19 09:02:26,473 [INFO] tensorflow: global_step/sec: 5.0227 2021-10-19 09:02:30,266 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.975 INFO:tensorflow:epoch = 99.49740932642487, learning_rate = 6.886565e-05, loss = 2.1757038e-05, step = 38406 (5.373 sec) 2021-10-19 09:02:31,635 [INFO] tensorflow: epoch = 99.49740932642487, learning_rate = 6.886565e-05, loss = 2.1757038e-05, step = 38406 (5.373 sec) INFO:tensorflow:global_step/sec: 5.05903 2021-10-19 09:02:33,985 [INFO] tensorflow: global_step/sec: 5.05903 2021-10-19 09:02:35,137 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.530 INFO:tensorflow:epoch = 99.56735751295336, learning_rate = 6.825217e-05, loss = 4.520813e-06, step = 38433 (5.249 sec) 2021-10-19 09:02:36,883 [INFO] tensorflow: epoch = 99.56735751295336, learning_rate = 6.825217e-05, loss = 4.520813e-06, step = 38433 (5.249 sec) 2021-10-19 09:02:40,024 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.465 INFO:tensorflow:global_step/sec: 5.11642 2021-10-19 09:02:41,412 [INFO] tensorflow: global_step/sec: 5.11642 INFO:tensorflow:epoch = 99.63730569948187, learning_rate = 6.7644214e-05, loss = 5.1352927e-06, step = 38460 (5.330 sec) 2021-10-19 09:02:42,213 [INFO] tensorflow: epoch = 99.63730569948187, learning_rate = 6.7644214e-05, loss = 5.1352927e-06, step = 38460 (5.330 sec) 2021-10-19 09:02:45,005 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.077 INFO:tensorflow:epoch = 99.70725388601036, learning_rate = 6.704161e-05, loss = 9.0223575e-06, step = 38487 (5.359 sec) 2021-10-19 09:02:47,572 [INFO] tensorflow: epoch = 99.70725388601036, learning_rate = 6.704161e-05, loss = 9.0223575e-06, step = 38487 (5.359 sec) INFO:tensorflow:global_step/sec: 5.02697 2021-10-19 09:02:48,971 [INFO] tensorflow: global_step/sec: 5.02697 2021-10-19 09:02:49,939 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.272 INFO:tensorflow:epoch = 99.77720207253887, learning_rate = 6.644444e-05, loss = 1.6783337e-05, step = 38514 (5.344 sec) 2021-10-19 09:02:52,916 [INFO] tensorflow: epoch = 99.77720207253887, learning_rate = 6.644444e-05, loss = 1.6783337e-05, step = 38514 (5.344 sec) 2021-10-19 09:02:54,906 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.133 INFO:tensorflow:global_step/sec: 5.07116 2021-10-19 09:02:56,464 [INFO] tensorflow: global_step/sec: 5.07116 INFO:tensorflow:epoch = 99.84715025906736, learning_rate = 6.585253e-05, loss = 9.720164e-06, step = 38541 (5.325 sec) 2021-10-19 09:02:58,241 [INFO] tensorflow: epoch = 99.84715025906736, learning_rate = 6.585253e-05, loss = 9.720164e-06, step = 38541 (5.325 sec) 2021-10-19 09:02:59,846 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.247 INFO:tensorflow:epoch = 99.91709844559585, learning_rate = 6.5265944e-05, loss = 7.21607e-06, step = 38568 (5.359 sec) 2021-10-19 09:03:03,600 [INFO] tensorflow: epoch = 99.91709844559585, learning_rate = 6.5265944e-05, loss = 7.21607e-06, step = 38568 (5.359 sec) INFO:tensorflow:global_step/sec: 5.03874 2021-10-19 09:03:04,006 [INFO] tensorflow: global_step/sec: 5.03874 2021-10-19 09:03:04,792 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.220 INFO:tensorflow:epoch = 99.98704663212436, learning_rate = 6.468453e-05, loss = 5.1245643e-06, step = 38595 (5.376 sec) 2021-10-19 09:03:08,976 [INFO] tensorflow: epoch = 99.98704663212436, learning_rate = 6.468453e-05, loss = 5.1245643e-06, step = 38595 (5.376 sec) INFO:tensorflow:Saving checkpoints for step-38600. 2021-10-19 09:03:09,789 [INFO] tensorflow: Saving checkpoints for step-38600. WARNING:tensorflow:Ignoring: /tmp/tmpqz2ru27h; No such file or directory 2021-10-19 09:03:10,012 [WARNING] tensorflow: Ignoring: /tmp/tmpqz2ru27h; No such file or directory 2021-10-19 09:03:14,696 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 62, 0.00s/step 2021-10-19 09:03:15,694 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 62, 0.10s/step 2021-10-19 09:03:16,618 [INFO] iva.detectnet_v2.evaluation.evaluation: step 20 / 62, 0.09s/step 2021-10-19 09:03:17,580 [INFO] iva.detectnet_v2.evaluation.evaluation: step 30 / 62, 0.10s/step 2021-10-19 09:03:18,481 [INFO] iva.detectnet_v2.evaluation.evaluation: step 40 / 62, 0.09s/step 2021-10-19 09:03:19,422 [INFO] iva.detectnet_v2.evaluation.evaluation: step 50 / 62, 0.09s/step 2021-10-19 09:03:20,306 [INFO] iva.detectnet_v2.evaluation.evaluation: step 60 / 62, 0.09s/step Matching predictions to ground truth, class 1/7.: 100%|█| 396/396 [00:00<00:00, 15209.14it/s] Matching predictions to ground truth, class 2/7.: 100%|█| 185/185 [00:00<00:00, 13796.81it/s] Matching predictions to ground truth, class 7/7.: 100%|█| 175/175 [00:00<00:00, 33734.87it/s] Epoch 100/120 ========================= Validation cost: 0.000010 Mean average_precision (in %): 32.1867 class name average precision (in %) ------------- -------------------------- auto_rickshaw 0 bicycle 91.3426 bus 49.9171 car 84.0472 person 0 road_sign 0 truck 0 Median Inference Time: 0.018486 2021-10-19 09:03:20,541 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 6.350 INFO:tensorflow:epoch = 100.0, learning_rate = 6.4577405e-05, loss = 1.4049619e-05, step = 38600 (11.779 sec) 2021-10-19 09:03:20,755 [INFO] tensorflow: epoch = 100.0, learning_rate = 6.4577405e-05, loss = 1.4049619e-05, step = 38600 (11.779 sec) 2021-10-19 09:03:20,756 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 100/120: loss: 0.00001 learning rate: 0.00006 Time taken: 0:01:27.502457 ETA: 0:29:10.049148 INFO:tensorflow:global_step/sec: 2.07145 2021-10-19 09:03:22,351 [INFO] tensorflow: global_step/sec: 2.07145 2021-10-19 09:03:25,574 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.878 INFO:tensorflow:epoch = 100.06735751295336, learning_rate = 6.402343e-05, loss = 1.3863364e-05, step = 38626 (5.236 sec) 2021-10-19 09:03:25,992 [INFO] tensorflow: epoch = 100.06735751295336, learning_rate = 6.402343e-05, loss = 1.3863364e-05, step = 38626 (5.236 sec) INFO:tensorflow:global_step/sec: 5.00468 2021-10-19 09:03:29,943 [INFO] tensorflow: global_step/sec: 5.00468 2021-10-19 09:03:30,528 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.189 INFO:tensorflow:epoch = 100.13730569948187, learning_rate = 6.3453146e-05, loss = 9.546523e-06, step = 38653 (5.362 sec) 2021-10-19 09:03:31,354 [INFO] tensorflow: epoch = 100.13730569948187, learning_rate = 6.3453146e-05, loss = 9.546523e-06, step = 38653 (5.362 sec) 2021-10-19 09:03:35,554 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.897 INFO:tensorflow:epoch = 100.20466321243524, learning_rate = 6.290869e-05, loss = 1.6196935e-05, step = 38679 (5.247 sec) 2021-10-19 09:03:36,601 [INFO] tensorflow: epoch = 100.20466321243524, learning_rate = 6.290869e-05, loss = 1.6196935e-05, step = 38679 (5.247 sec) INFO:tensorflow:global_step/sec: 4.95255 2021-10-19 09:03:37,616 [INFO] tensorflow: global_step/sec: 4.95255 2021-10-19 09:03:40,575 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.920 INFO:tensorflow:epoch = 100.27461139896373, learning_rate = 6.2348336e-05, loss = 1.8495466e-05, step = 38706 (5.343 sec) 2021-10-19 09:03:41,944 [INFO] tensorflow: epoch = 100.27461139896373, learning_rate = 6.2348336e-05, loss = 1.8495466e-05, step = 38706 (5.343 sec) INFO:tensorflow:global_step/sec: 5.05447 2021-10-19 09:03:45,134 [INFO] tensorflow: global_step/sec: 5.05447 2021-10-19 09:03:45,564 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.044 INFO:tensorflow:epoch = 100.3419689119171, learning_rate = 6.1813415e-05, loss = 1.52538305e-05, step = 38732 (5.198 sec) 2021-10-19 09:03:47,143 [INFO] tensorflow: epoch = 100.3419689119171, learning_rate = 6.1813415e-05, loss = 1.52538305e-05, step = 38732 (5.198 sec) 2021-10-19 09:03:50,526 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.158 INFO:tensorflow:epoch = 100.4119170984456, learning_rate = 6.126276e-05, loss = 9.641982e-06, step = 38759 (5.410 sec) 2021-10-19 09:03:52,553 [INFO] tensorflow: epoch = 100.4119170984456, learning_rate = 6.126276e-05, loss = 9.641982e-06, step = 38759 (5.410 sec) INFO:tensorflow:global_step/sec: 4.9907 2021-10-19 09:03:52,749 [INFO] tensorflow: global_step/sec: 4.9907 2021-10-19 09:03:55,525 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.006 INFO:tensorflow:epoch = 100.48186528497409, learning_rate = 6.0717066e-05, loss = 4.618609e-06, step = 38786 (5.375 sec) 2021-10-19 09:03:57,927 [INFO] tensorflow: epoch = 100.48186528497409, learning_rate = 6.0717066e-05, loss = 4.618609e-06, step = 38786 (5.375 sec) INFO:tensorflow:global_step/sec: 5.04108 2021-10-19 09:04:00,287 [INFO] tensorflow: global_step/sec: 5.04108 2021-10-19 09:04:00,496 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.119 INFO:tensorflow:epoch = 100.5518134715026, learning_rate = 6.0176175e-05, loss = 1.2579238e-05, step = 38813 (5.365 sec) 2021-10-19 09:04:03,292 [INFO] tensorflow: epoch = 100.5518134715026, learning_rate = 6.0176175e-05, loss = 1.2579238e-05, step = 38813 (5.365 sec) 2021-10-19 09:04:05,525 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.884 INFO:tensorflow:global_step/sec: 4.95613 2021-10-19 09:04:07,954 [INFO] tensorflow: global_step/sec: 4.95613 INFO:tensorflow:epoch = 100.61917098445596, learning_rate = 5.96599e-05, loss = 1.3934765e-05, step = 38839 (5.243 sec) 2021-10-19 09:04:08,535 [INFO] tensorflow: epoch = 100.61917098445596, learning_rate = 5.96599e-05, loss = 1.3934765e-05, step = 38839 (5.243 sec) 2021-10-19 09:04:10,533 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.970 INFO:tensorflow:epoch = 100.68652849740933, learning_rate = 5.9148104e-05, loss = 2.3352268e-05, step = 38865 (5.199 sec) 2021-10-19 09:04:13,735 [INFO] tensorflow: epoch = 100.68652849740933, learning_rate = 5.9148104e-05, loss = 2.3352268e-05, step = 38865 (5.199 sec) INFO:tensorflow:global_step/sec: 5.01605 2021-10-19 09:04:15,530 [INFO] tensorflow: global_step/sec: 5.01605 2021-10-19 09:04:15,531 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.011 INFO:tensorflow:epoch = 100.75647668393783, learning_rate = 5.862113e-05, loss = 1.2764987e-05, step = 38892 (5.329 sec) 2021-10-19 09:04:19,064 [INFO] tensorflow: epoch = 100.75647668393783, learning_rate = 5.862113e-05, loss = 1.2764987e-05, step = 38892 (5.329 sec) 2021-10-19 09:04:20,444 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.355 INFO:tensorflow:global_step/sec: 5.09772 2021-10-19 09:04:22,984 [INFO] tensorflow: global_step/sec: 5.09772 INFO:tensorflow:epoch = 100.82642487046633, learning_rate = 5.8098965e-05, loss = 1.14563545e-05, step = 38919 (5.285 sec) 2021-10-19 09:04:24,349 [INFO] tensorflow: epoch = 100.82642487046633, learning_rate = 5.8098965e-05, loss = 1.14563545e-05, step = 38919 (5.285 sec) 2021-10-19 09:04:25,364 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.326 INFO:tensorflow:epoch = 100.89637305699482, learning_rate = 5.75814e-05, loss = 1.6487144e-05, step = 38946 (5.373 sec) 2021-10-19 09:04:29,722 [INFO] tensorflow: epoch = 100.89637305699482, learning_rate = 5.75814e-05, loss = 1.6487144e-05, step = 38946 (5.373 sec) 2021-10-19 09:04:30,306 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.236 INFO:tensorflow:global_step/sec: 5.03265 2021-10-19 09:04:30,535 [INFO] tensorflow: global_step/sec: 5.03265 INFO:tensorflow:epoch = 100.96632124352332, learning_rate = 5.7068493e-05, loss = 3.2093718e-05, step = 38973 (5.343 sec) 2021-10-19 09:04:35,065 [INFO] tensorflow: epoch = 100.96632124352332, learning_rate = 5.7068493e-05, loss = 3.2093718e-05, step = 38973 (5.343 sec) 2021-10-19 09:04:35,283 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.093 2021-10-19 09:04:37,662 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 101/120: loss: 0.00001 learning rate: 0.00006 Time taken: 0:01:16.925391 ETA: 0:24:21.582424 INFO:tensorflow:global_step/sec: 5.0472 2021-10-19 09:04:38,063 [INFO] tensorflow: global_step/sec: 5.0472 2021-10-19 09:04:40,242 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.168 INFO:tensorflow:epoch = 101.03626943005182, learning_rate = 5.6560108e-05, loss = 1.375614e-05, step = 39000 (5.376 sec) 2021-10-19 09:04:40,440 [INFO] tensorflow: epoch = 101.03626943005182, learning_rate = 5.6560108e-05, loss = 1.375614e-05, step = 39000 (5.376 sec) 2021-10-19 09:04:45,141 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.414 INFO:tensorflow:global_step/sec: 5.08055 2021-10-19 09:04:45,543 [INFO] tensorflow: global_step/sec: 5.08055 INFO:tensorflow:epoch = 101.10621761658031, learning_rate = 5.60563e-05, loss = 8.560511e-06, step = 39027 (5.301 sec) 2021-10-19 09:04:45,741 [INFO] tensorflow: epoch = 101.10621761658031, learning_rate = 5.60563e-05, loss = 8.560511e-06, step = 39027 (5.301 sec) 2021-10-19 09:04:50,032 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.446 INFO:tensorflow:epoch = 101.17616580310882, learning_rate = 5.5556982e-05, loss = 8.696787e-06, step = 39054 (5.328 sec) 2021-10-19 09:04:51,069 [INFO] tensorflow: epoch = 101.17616580310882, learning_rate = 5.5556982e-05, loss = 8.696787e-06, step = 39054 (5.328 sec) INFO:tensorflow:global_step/sec: 5.02096 2021-10-19 09:04:53,111 [INFO] tensorflow: global_step/sec: 5.02096 2021-10-19 09:04:55,114 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.679 INFO:tensorflow:epoch = 101.24352331606218, learning_rate = 5.508028e-05, loss = 8.181353e-06, step = 39080 (5.247 sec) 2021-10-19 09:04:56,316 [INFO] tensorflow: epoch = 101.24352331606218, learning_rate = 5.508028e-05, loss = 8.181353e-06, step = 39080 (5.247 sec) 2021-10-19 09:05:00,167 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.792 INFO:tensorflow:global_step/sec: 4.95735 2021-10-19 09:05:00,777 [INFO] tensorflow: global_step/sec: 4.95735 INFO:tensorflow:epoch = 101.31088082901555, learning_rate = 5.4607775e-05, loss = 1.7544124e-05, step = 39106 (5.271 sec) 2021-10-19 09:05:01,587 [INFO] tensorflow: epoch = 101.31088082901555, learning_rate = 5.4607775e-05, loss = 1.7544124e-05, step = 39106 (5.271 sec) 2021-10-19 09:05:05,221 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.791 INFO:tensorflow:epoch = 101.37823834196891, learning_rate = 5.413927e-05, loss = 1.6415976e-05, step = 39132 (5.242 sec) 2021-10-19 09:05:06,829 [INFO] tensorflow: epoch = 101.37823834196891, learning_rate = 5.413927e-05, loss = 1.6415976e-05, step = 39132 (5.242 sec) INFO:tensorflow:global_step/sec: 4.97988 2021-10-19 09:05:08,407 [INFO] tensorflow: global_step/sec: 4.97988 2021-10-19 09:05:10,192 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.118 INFO:tensorflow:epoch = 101.44818652849742, learning_rate = 5.3656975e-05, loss = 4.1600215e-06, step = 39159 (5.378 sec) 2021-10-19 09:05:12,207 [INFO] tensorflow: epoch = 101.44818652849742, learning_rate = 5.3656975e-05, loss = 4.1600215e-06, step = 39159 (5.378 sec) 2021-10-19 09:05:15,178 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.057 INFO:tensorflow:global_step/sec: 5.02185 2021-10-19 09:05:15,974 [INFO] tensorflow: global_step/sec: 5.02185 INFO:tensorflow:epoch = 101.51813471502591, learning_rate = 5.317903e-05, loss = 1.0406607e-05, step = 39186 (5.375 sec) 2021-10-19 09:05:17,582 [INFO] tensorflow: epoch = 101.51813471502591, learning_rate = 5.317903e-05, loss = 1.0406607e-05, step = 39186 (5.375 sec) 2021-10-19 09:05:20,200 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.913 INFO:tensorflow:epoch = 101.58549222797927, learning_rate = 5.2722782e-05, loss = 1.653571e-05, step = 39212 (5.356 sec) 2021-10-19 09:05:22,938 [INFO] tensorflow: epoch = 101.58549222797927, learning_rate = 5.2722782e-05, loss = 1.653571e-05, step = 39212 (5.356 sec) INFO:tensorflow:global_step/sec: 4.91026 2021-10-19 09:05:23,713 [INFO] tensorflow: global_step/sec: 4.91026 2021-10-19 09:05:25,317 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.545 INFO:tensorflow:epoch = 101.65284974093265, learning_rate = 5.22704e-05, loss = 1.9817864e-05, step = 39238 (5.193 sec) 2021-10-19 09:05:28,132 [INFO] tensorflow: epoch = 101.65284974093265, learning_rate = 5.22704e-05, loss = 1.9817864e-05, step = 39238 (5.193 sec) 2021-10-19 09:05:30,283 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.141 INFO:tensorflow:global_step/sec: 5.0087 2021-10-19 09:05:31,300 [INFO] tensorflow: global_step/sec: 5.0087 INFO:tensorflow:epoch = 101.72279792746114, learning_rate = 5.1804804e-05, loss = 1.0111454e-05, step = 39265 (5.374 sec) 2021-10-19 09:05:33,506 [INFO] tensorflow: epoch = 101.72279792746114, learning_rate = 5.1804804e-05, loss = 1.0111454e-05, step = 39265 (5.374 sec) 2021-10-19 09:05:35,307 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.904 INFO:tensorflow:epoch = 101.79274611398964, learning_rate = 5.1343308e-05, loss = 1.3284582e-05, step = 39292 (5.332 sec) 2021-10-19 09:05:38,838 [INFO] tensorflow: epoch = 101.79274611398964, learning_rate = 5.1343308e-05, loss = 1.3284582e-05, step = 39292 (5.332 sec) INFO:tensorflow:global_step/sec: 5.04002 2021-10-19 09:05:38,840 [INFO] tensorflow: global_step/sec: 5.04002 2021-10-19 09:05:40,201 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.437 INFO:tensorflow:epoch = 101.86269430051814, learning_rate = 5.0885967e-05, loss = 9.87402e-06, step = 39319 (5.366 sec) 2021-10-19 09:05:44,204 [INFO] tensorflow: epoch = 101.86269430051814, learning_rate = 5.0885967e-05, loss = 9.87402e-06, step = 39319 (5.366 sec) 2021-10-19 09:05:45,218 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.934 INFO:tensorflow:global_step/sec: 5.03156 2021-10-19 09:05:46,392 [INFO] tensorflow: global_step/sec: 5.03156 INFO:tensorflow:epoch = 101.93005181347151, learning_rate = 5.0449395e-05, loss = 1.6103357e-05, step = 39345 (5.194 sec) 2021-10-19 09:05:49,398 [INFO] tensorflow: epoch = 101.93005181347151, learning_rate = 5.0449395e-05, loss = 1.6103357e-05, step = 39345 (5.194 sec) 2021-10-19 09:05:50,194 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.100 INFO:tensorflow:global_step/sec: 4.99823 2021-10-19 09:05:53,995 [INFO] tensorflow: global_step/sec: 4.99823 INFO:tensorflow:epoch = 101.99740932642487, learning_rate = 5.001652e-05, loss = 1.4223015e-05, step = 39371 (5.209 sec) 2021-10-19 09:05:54,607 [INFO] tensorflow: epoch = 101.99740932642487, learning_rate = 5.001652e-05, loss = 1.4223015e-05, step = 39371 (5.209 sec) 2021-10-19 09:05:54,825 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 102/120: loss: 0.00000 learning rate: 0.00005 Time taken: 0:01:17.141241 ETA: 0:23:08.542331 2021-10-19 09:05:55,230 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.859 INFO:tensorflow:epoch = 102.06735751295338, learning_rate = 4.9571e-05, loss = 7.2226976e-06, step = 39398 (5.406 sec) 2021-10-19 09:06:00,012 [INFO] tensorflow: epoch = 102.06735751295338, learning_rate = 4.9571e-05, loss = 7.2226976e-06, step = 39398 (5.406 sec) 2021-10-19 09:06:00,221 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.035 INFO:tensorflow:global_step/sec: 4.99556 2021-10-19 09:06:01,601 [INFO] tensorflow: global_step/sec: 4.99556 INFO:tensorflow:epoch = 102.13471502590674, learning_rate = 4.9145707e-05, loss = 8.416366e-06, step = 39424 (5.202 sec) 2021-10-19 09:06:05,214 [INFO] tensorflow: epoch = 102.13471502590674, learning_rate = 4.9145707e-05, loss = 8.416366e-06, step = 39424 (5.202 sec) 2021-10-19 09:06:05,215 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.029 INFO:tensorflow:global_step/sec: 5.04806 2021-10-19 09:06:09,129 [INFO] tensorflow: global_step/sec: 5.04806 2021-10-19 09:06:10,125 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.368 INFO:tensorflow:epoch = 102.20466321243524, learning_rate = 4.8707945e-05, loss = 1.4015859e-05, step = 39451 (5.305 sec) 2021-10-19 09:06:10,519 [INFO] tensorflow: epoch = 102.20466321243524, learning_rate = 4.8707945e-05, loss = 1.4015859e-05, step = 39451 (5.305 sec) 2021-10-19 09:06:15,062 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.255 INFO:tensorflow:epoch = 102.27461139896373, learning_rate = 4.827408e-05, loss = 1.3149595e-05, step = 39478 (5.334 sec) 2021-10-19 09:06:15,853 [INFO] tensorflow: epoch = 102.27461139896373, learning_rate = 4.827408e-05, loss = 1.3149595e-05, step = 39478 (5.334 sec) INFO:tensorflow:global_step/sec: 5.05872 2021-10-19 09:06:16,641 [INFO] tensorflow: global_step/sec: 5.05872 2021-10-19 09:06:20,000 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.252 INFO:tensorflow:epoch = 102.34455958549223, learning_rate = 4.7843987e-05, loss = 1.17236905e-05, step = 39505 (5.323 sec) 2021-10-19 09:06:21,175 [INFO] tensorflow: epoch = 102.34455958549223, learning_rate = 4.7843987e-05, loss = 1.17236905e-05, step = 39505 (5.323 sec) INFO:tensorflow:global_step/sec: 4.99759 2021-10-19 09:06:24,245 [INFO] tensorflow: global_step/sec: 4.99759 2021-10-19 09:06:25,024 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.906 INFO:tensorflow:epoch = 102.4119170984456, learning_rate = 4.743356e-05, loss = 1.2144332e-05, step = 39531 (5.232 sec) 2021-10-19 09:06:26,408 [INFO] tensorflow: epoch = 102.4119170984456, learning_rate = 4.743356e-05, loss = 1.2144332e-05, step = 39531 (5.232 sec) 2021-10-19 09:06:29,964 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.245 INFO:tensorflow:epoch = 102.47927461139896, learning_rate = 4.7026606e-05, loss = 2.0301031e-05, step = 39557 (5.272 sec) 2021-10-19 09:06:31,680 [INFO] tensorflow: epoch = 102.47927461139896, learning_rate = 4.7026606e-05, loss = 2.0301031e-05, step = 39557 (5.272 sec) INFO:tensorflow:global_step/sec: 4.97233 2021-10-19 09:06:31,887 [INFO] tensorflow: global_step/sec: 4.97233 2021-10-19 09:06:35,025 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.762 INFO:tensorflow:epoch = 102.54922279792747, learning_rate = 4.6607674e-05, loss = 1.1187028e-05, step = 39584 (5.302 sec) 2021-10-19 09:06:36,982 [INFO] tensorflow: epoch = 102.54922279792747, learning_rate = 4.6607674e-05, loss = 1.1187028e-05, step = 39584 (5.302 sec) INFO:tensorflow:global_step/sec: 5.08129 2021-10-19 09:06:39,365 [INFO] tensorflow: global_step/sec: 5.08129 2021-10-19 09:06:39,962 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.255 INFO:tensorflow:epoch = 102.61917098445596, learning_rate = 4.6192516e-05, loss = 1.9318595e-05, step = 39611 (5.355 sec) 2021-10-19 09:06:42,337 [INFO] tensorflow: epoch = 102.61917098445596, learning_rate = 4.6192516e-05, loss = 1.9318595e-05, step = 39611 (5.355 sec) 2021-10-19 09:06:44,937 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.102 INFO:tensorflow:global_step/sec: 4.99212 2021-10-19 09:06:46,977 [INFO] tensorflow: global_step/sec: 4.99212 INFO:tensorflow:epoch = 102.68652849740933, learning_rate = 4.5796212e-05, loss = 4.213284e-06, step = 39637 (5.237 sec) 2021-10-19 09:06:47,574 [INFO] tensorflow: epoch = 102.68652849740933, learning_rate = 4.5796212e-05, loss = 4.213284e-06, step = 39637 (5.237 sec) 2021-10-19 09:06:50,026 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.655 INFO:tensorflow:epoch = 102.75647668393783, learning_rate = 4.5388242e-05, loss = 9.542479e-06, step = 39664 (5.397 sec) 2021-10-19 09:06:52,970 [INFO] tensorflow: epoch = 102.75647668393783, learning_rate = 4.5388242e-05, loss = 9.542479e-06, step = 39664 (5.397 sec) INFO:tensorflow:global_step/sec: 4.99866 2021-10-19 09:06:54,579 [INFO] tensorflow: global_step/sec: 4.99866 2021-10-19 09:06:54,970 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.228 INFO:tensorflow:epoch = 102.82642487046633, learning_rate = 4.498395e-05, loss = 6.5657914e-06, step = 39691 (5.372 sec) 2021-10-19 09:06:58,342 [INFO] tensorflow: epoch = 102.82642487046633, learning_rate = 4.498395e-05, loss = 6.5657914e-06, step = 39691 (5.372 sec) 2021-10-19 09:06:59,927 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.176 INFO:tensorflow:global_step/sec: 5.00891 2021-10-19 09:07:02,166 [INFO] tensorflow: global_step/sec: 5.00891 INFO:tensorflow:epoch = 102.89378238341969, learning_rate = 4.459797e-05, loss = 8.397109e-06, step = 39717 (5.248 sec) 2021-10-19 09:07:03,590 [INFO] tensorflow: epoch = 102.89378238341969, learning_rate = 4.459797e-05, loss = 8.397109e-06, step = 39717 (5.248 sec) 2021-10-19 09:07:04,947 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.924 INFO:tensorflow:epoch = 102.9637305699482, learning_rate = 4.4200715e-05, loss = 4.1744297e-06, step = 39744 (5.318 sec) 2021-10-19 09:07:08,907 [INFO] tensorflow: epoch = 102.9637305699482, learning_rate = 4.4200715e-05, loss = 4.1744297e-06, step = 39744 (5.318 sec) INFO:tensorflow:global_step/sec: 5.05657 2021-10-19 09:07:09,681 [INFO] tensorflow: global_step/sec: 5.05657 2021-10-19 09:07:09,900 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.190 2021-10-19 09:07:11,697 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 103/120: loss: 0.00003 learning rate: 0.00004 Time taken: 0:01:16.884483 ETA: 0:21:47.036209 INFO:tensorflow:epoch = 103.03108808290156, learning_rate = 4.38215e-05, loss = 7.600609e-06, step = 39770 (5.251 sec) 2021-10-19 09:07:14,158 [INFO] tensorflow: epoch = 103.03108808290156, learning_rate = 4.38215e-05, loss = 7.600609e-06, step = 39770 (5.251 sec) 2021-10-19 09:07:14,983 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.675 INFO:tensorflow:global_step/sec: 4.96872 2021-10-19 09:07:17,329 [INFO] tensorflow: global_step/sec: 4.96872 INFO:tensorflow:epoch = 103.10103626943005, learning_rate = 4.343112e-05, loss = 1.0117859e-05, step = 39797 (5.353 sec) 2021-10-19 09:07:19,511 [INFO] tensorflow: epoch = 103.10103626943005, learning_rate = 4.343112e-05, loss = 1.0117859e-05, step = 39797 (5.353 sec) 2021-10-19 09:07:19,901 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.337 INFO:tensorflow:epoch = 103.17098445595855, learning_rate = 4.3044256e-05, loss = 4.522198e-06, step = 39824 (5.291 sec) 2021-10-19 09:07:24,803 [INFO] tensorflow: epoch = 103.17098445595855, learning_rate = 4.3044256e-05, loss = 4.522198e-06, step = 39824 (5.291 sec) INFO:tensorflow:global_step/sec: 5.08286 2021-10-19 09:07:24,805 [INFO] tensorflow: global_step/sec: 5.08286 2021-10-19 09:07:24,806 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.389 2021-10-19 09:07:29,792 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.058 INFO:tensorflow:epoch = 103.24093264248705, learning_rate = 4.2660795e-05, loss = 1.258904e-05, step = 39851 (5.380 sec) 2021-10-19 09:07:30,183 [INFO] tensorflow: epoch = 103.24093264248705, learning_rate = 4.2660795e-05, loss = 1.258904e-05, step = 39851 (5.380 sec) INFO:tensorflow:global_step/sec: 5.0126 2021-10-19 09:07:32,386 [INFO] tensorflow: global_step/sec: 5.0126 2021-10-19 09:07:34,742 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.202 INFO:tensorflow:epoch = 103.30829015544042, learning_rate = 4.2294792e-05, loss = 1.6092312e-05, step = 39877 (5.204 sec) 2021-10-19 09:07:35,387 [INFO] tensorflow: epoch = 103.30829015544042, learning_rate = 4.2294792e-05, loss = 1.6092312e-05, step = 39877 (5.204 sec) 2021-10-19 09:07:39,866 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.517 INFO:tensorflow:global_step/sec: 4.94712 2021-10-19 09:07:40,067 [INFO] tensorflow: global_step/sec: 4.94712 INFO:tensorflow:epoch = 103.37564766839378, learning_rate = 4.1931966e-05, loss = 1.107103e-05, step = 39903 (5.309 sec) 2021-10-19 09:07:40,696 [INFO] tensorflow: epoch = 103.37564766839378, learning_rate = 4.1931966e-05, loss = 1.107103e-05, step = 39903 (5.309 sec) 2021-10-19 09:07:44,836 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.124 INFO:tensorflow:epoch = 103.44559585492229, learning_rate = 4.155842e-05, loss = 1.2385959e-05, step = 39930 (5.361 sec) 2021-10-19 09:07:46,057 [INFO] tensorflow: epoch = 103.44559585492229, learning_rate = 4.155842e-05, loss = 1.2385959e-05, step = 39930 (5.361 sec) INFO:tensorflow:global_step/sec: 5.01424 2021-10-19 09:07:47,645 [INFO] tensorflow: global_step/sec: 5.01424 2021-10-19 09:07:49,814 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.090 INFO:tensorflow:epoch = 103.51554404145078, learning_rate = 4.1188203e-05, loss = 1.1680718e-05, step = 39957 (5.356 sec) 2021-10-19 09:07:51,412 [INFO] tensorflow: epoch = 103.51554404145078, learning_rate = 4.1188203e-05, loss = 1.1680718e-05, step = 39957 (5.356 sec) 2021-10-19 09:07:54,760 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.222 INFO:tensorflow:global_step/sec: 5.05983 2021-10-19 09:07:55,155 [INFO] tensorflow: global_step/sec: 5.05983 INFO:tensorflow:epoch = 103.58549222797927, learning_rate = 4.082132e-05, loss = 8.802828e-06, step = 39984 (5.378 sec) 2021-10-19 09:07:56,790 [INFO] tensorflow: epoch = 103.58549222797927, learning_rate = 4.082132e-05, loss = 8.802828e-06, step = 39984 (5.378 sec) 2021-10-19 09:07:59,766 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.978 INFO:tensorflow:epoch = 103.65544041450778, learning_rate = 4.0457704e-05, loss = 1.11737045e-05, step = 40011 (5.352 sec) 2021-10-19 09:08:02,142 [INFO] tensorflow: epoch = 103.65544041450778, learning_rate = 4.0457704e-05, loss = 1.11737045e-05, step = 40011 (5.352 sec) INFO:tensorflow:global_step/sec: 4.9954 2021-10-19 09:08:02,762 [INFO] tensorflow: global_step/sec: 4.9954 2021-10-19 09:08:04,740 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.107 INFO:tensorflow:epoch = 103.72279792746114, learning_rate = 4.01106e-05, loss = 1.16844485e-05, step = 40037 (5.192 sec) 2021-10-19 09:08:07,334 [INFO] tensorflow: epoch = 103.72279792746114, learning_rate = 4.01106e-05, loss = 1.16844485e-05, step = 40037 (5.192 sec) 2021-10-19 09:08:09,699 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.167 INFO:tensorflow:global_step/sec: 5.01859 2021-10-19 09:08:10,334 [INFO] tensorflow: global_step/sec: 5.01859 INFO:tensorflow:epoch = 103.79274611398964, learning_rate = 3.975328e-05, loss = 1.0223861e-05, step = 40064 (5.402 sec) 2021-10-19 09:08:12,735 [INFO] tensorflow: epoch = 103.79274611398964, learning_rate = 3.975328e-05, loss = 1.0223861e-05, step = 40064 (5.402 sec) 2021-10-19 09:08:14,701 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.994 INFO:tensorflow:global_step/sec: 5.04111 2021-10-19 09:08:17,872 [INFO] tensorflow: global_step/sec: 5.04111 INFO:tensorflow:epoch = 103.86269430051814, learning_rate = 3.939918e-05, loss = 1.2561197e-05, step = 40091 (5.339 sec) 2021-10-19 09:08:18,075 [INFO] tensorflow: epoch = 103.86269430051814, learning_rate = 3.939918e-05, loss = 1.2561197e-05, step = 40091 (5.339 sec) 2021-10-19 09:08:19,707 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.978 INFO:tensorflow:epoch = 103.93005181347151, learning_rate = 3.9061117e-05, loss = 2.3314906e-05, step = 40117 (5.223 sec) 2021-10-19 09:08:23,298 [INFO] tensorflow: epoch = 103.93005181347151, learning_rate = 3.9061117e-05, loss = 2.3314906e-05, step = 40117 (5.223 sec) 2021-10-19 09:08:24,673 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.142 INFO:tensorflow:global_step/sec: 4.98283 2021-10-19 09:08:25,498 [INFO] tensorflow: global_step/sec: 4.98283 INFO:tensorflow:epoch = 104.0, learning_rate = 3.8713184e-05, loss = 1.107887e-05, step = 40144 (5.356 sec) 2021-10-19 09:08:28,654 [INFO] tensorflow: epoch = 104.0, learning_rate = 3.8713184e-05, loss = 1.107887e-05, step = 40144 (5.356 sec) 2021-10-19 09:08:28,655 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 104/120: loss: 0.00001 learning rate: 0.00004 Time taken: 0:01:16.953117 ETA: 0:20:31.249866 2021-10-19 09:08:29,659 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.055 INFO:tensorflow:global_step/sec: 5.01929 2021-10-19 09:08:33,069 [INFO] tensorflow: global_step/sec: 5.01929 INFO:tensorflow:epoch = 104.06735751295338, learning_rate = 3.8381044e-05, loss = 2.3043796e-05, step = 40170 (5.274 sec) 2021-10-19 09:08:33,928 [INFO] tensorflow: epoch = 104.06735751295338, learning_rate = 3.8381044e-05, loss = 2.3043796e-05, step = 40170 (5.274 sec) 2021-10-19 09:08:34,732 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.716 INFO:tensorflow:epoch = 104.13471502590674, learning_rate = 3.8051723e-05, loss = 1.5207839e-05, step = 40196 (5.234 sec) 2021-10-19 09:08:39,163 [INFO] tensorflow: epoch = 104.13471502590674, learning_rate = 3.8051723e-05, loss = 1.5207839e-05, step = 40196 (5.234 sec) 2021-10-19 09:08:39,759 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.892 INFO:tensorflow:global_step/sec: 4.9444 2021-10-19 09:08:40,755 [INFO] tensorflow: global_step/sec: 4.9444 INFO:tensorflow:epoch = 104.20466321243524, learning_rate = 3.7712776e-05, loss = 1.6710266e-05, step = 40223 (5.356 sec) 2021-10-19 09:08:44,518 [INFO] tensorflow: epoch = 104.20466321243524, learning_rate = 3.7712776e-05, loss = 1.6710266e-05, step = 40223 (5.356 sec) 2021-10-19 09:08:44,753 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.026 INFO:tensorflow:global_step/sec: 5.05728 2021-10-19 09:08:48,269 [INFO] tensorflow: global_step/sec: 5.05728 2021-10-19 09:08:49,711 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.173 INFO:tensorflow:epoch = 104.27461139896373, learning_rate = 3.737682e-05, loss = 4.5220618e-06, step = 40250 (5.407 sec) 2021-10-19 09:08:49,925 [INFO] tensorflow: epoch = 104.27461139896373, learning_rate = 3.737682e-05, loss = 4.5220618e-06, step = 40250 (5.407 sec) 2021-10-19 09:08:54,684 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.107 INFO:tensorflow:epoch = 104.34455958549223, learning_rate = 3.7043887e-05, loss = 1.0863252e-05, step = 40277 (5.373 sec) 2021-10-19 09:08:55,298 [INFO] tensorflow: epoch = 104.34455958549223, learning_rate = 3.7043887e-05, loss = 1.0863252e-05, step = 40277 (5.373 sec) INFO:tensorflow:global_step/sec: 4.94835 2021-10-19 09:08:55,948 [INFO] tensorflow: global_step/sec: 4.94835 2021-10-19 09:08:59,734 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.806 INFO:tensorflow:epoch = 104.4119170984456, learning_rate = 3.672607e-05, loss = 1.2989868e-05, step = 40303 (5.216 sec) 2021-10-19 09:09:00,514 [INFO] tensorflow: epoch = 104.4119170984456, learning_rate = 3.672607e-05, loss = 1.2989868e-05, step = 40303 (5.216 sec) INFO:tensorflow:global_step/sec: 5.0226 2021-10-19 09:09:03,514 [INFO] tensorflow: global_step/sec: 5.0226 2021-10-19 09:09:04,768 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.864 INFO:tensorflow:epoch = 104.47927461139896, learning_rate = 3.641098e-05, loss = 1.4391907e-05, step = 40329 (5.257 sec) 2021-10-19 09:09:05,771 [INFO] tensorflow: epoch = 104.47927461139896, learning_rate = 3.641098e-05, loss = 1.4391907e-05, step = 40329 (5.257 sec) 2021-10-19 09:09:09,734 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.139 INFO:tensorflow:epoch = 104.54922279792747, learning_rate = 3.6086614e-05, loss = 6.228531e-06, step = 40356 (5.382 sec) 2021-10-19 09:09:11,153 [INFO] tensorflow: epoch = 104.54922279792747, learning_rate = 3.6086614e-05, loss = 6.228531e-06, step = 40356 (5.382 sec) INFO:tensorflow:global_step/sec: 4.97329 2021-10-19 09:09:11,154 [INFO] tensorflow: global_step/sec: 4.97329 2021-10-19 09:09:14,697 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.154 INFO:tensorflow:epoch = 104.61917098445596, learning_rate = 3.5765173e-05, loss = 9.367926e-06, step = 40383 (5.310 sec) 2021-10-19 09:09:16,463 [INFO] tensorflow: epoch = 104.61917098445596, learning_rate = 3.5765173e-05, loss = 9.367926e-06, step = 40383 (5.310 sec) INFO:tensorflow:global_step/sec: 5.08654 2021-10-19 09:09:18,625 [INFO] tensorflow: global_step/sec: 5.08654 2021-10-19 09:09:19,624 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.298 INFO:tensorflow:epoch = 104.68911917098445, learning_rate = 3.5446566e-05, loss = 1.0908683e-05, step = 40410 (5.343 sec) 2021-10-19 09:09:21,806 [INFO] tensorflow: epoch = 104.68911917098445, learning_rate = 3.5446566e-05, loss = 1.0908683e-05, step = 40410 (5.343 sec) 2021-10-19 09:09:24,566 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.233 INFO:tensorflow:global_step/sec: 5.05216 2021-10-19 09:09:26,147 [INFO] tensorflow: global_step/sec: 5.05216 INFO:tensorflow:epoch = 104.75906735751296, learning_rate = 3.5130826e-05, loss = 1.6914326e-05, step = 40437 (5.313 sec) 2021-10-19 09:09:27,119 [INFO] tensorflow: epoch = 104.75906735751296, learning_rate = 3.5130826e-05, loss = 1.6914326e-05, step = 40437 (5.313 sec) 2021-10-19 09:09:29,509 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.233 INFO:tensorflow:epoch = 104.82901554404145, learning_rate = 3.4817866e-05, loss = 1.676573e-05, step = 40464 (5.348 sec) 2021-10-19 09:09:32,467 [INFO] tensorflow: epoch = 104.82901554404145, learning_rate = 3.4817866e-05, loss = 1.676573e-05, step = 40464 (5.348 sec) INFO:tensorflow:global_step/sec: 5.05727 2021-10-19 09:09:33,661 [INFO] tensorflow: global_step/sec: 5.05727 2021-10-19 09:09:34,449 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.247 INFO:tensorflow:epoch = 104.89896373056995, learning_rate = 3.4507728e-05, loss = 1.254934e-05, step = 40491 (5.364 sec) 2021-10-19 09:09:37,831 [INFO] tensorflow: epoch = 104.89896373056995, learning_rate = 3.4507728e-05, loss = 1.254934e-05, step = 40491 (5.364 sec) 2021-10-19 09:09:39,463 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.946 INFO:tensorflow:global_step/sec: 5.01783 2021-10-19 09:09:41,234 [INFO] tensorflow: global_step/sec: 5.01783 INFO:tensorflow:epoch = 104.96632124352332, learning_rate = 3.4211673e-05, loss = 9.690106e-06, step = 40517 (5.203 sec) 2021-10-19 09:09:43,033 [INFO] tensorflow: epoch = 104.96632124352332, learning_rate = 3.4211673e-05, loss = 9.690106e-06, step = 40517 (5.203 sec) 2021-10-19 09:09:44,434 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.118 2021-10-19 09:09:45,642 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 105/120: loss: 0.00000 learning rate: 0.00003 Time taken: 0:01:16.989336 ETA: 0:19:14.840040 INFO:tensorflow:epoch = 105.03367875647669, learning_rate = 3.3918157e-05, loss = 1.4374164e-05, step = 40543 (5.200 sec) 2021-10-19 09:09:48,233 [INFO] tensorflow: epoch = 105.03367875647669, learning_rate = 3.3918157e-05, loss = 1.4374164e-05, step = 40543 (5.200 sec) INFO:tensorflow:global_step/sec: 5.0126 2021-10-19 09:09:48,815 [INFO] tensorflow: global_step/sec: 5.0126 2021-10-19 09:09:49,427 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.032 INFO:tensorflow:epoch = 105.10362694300518, learning_rate = 3.3615997e-05, loss = 7.171793e-06, step = 40570 (5.362 sec) 2021-10-19 09:09:53,595 [INFO] tensorflow: epoch = 105.10362694300518, learning_rate = 3.3615997e-05, loss = 7.171793e-06, step = 40570 (5.362 sec) 2021-10-19 09:09:54,418 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.037 INFO:tensorflow:global_step/sec: 4.95997 2021-10-19 09:09:56,476 [INFO] tensorflow: global_step/sec: 4.95997 INFO:tensorflow:epoch = 105.17098445595855, learning_rate = 3.3327593e-05, loss = 9.796026e-06, step = 40596 (5.228 sec) 2021-10-19 09:09:58,823 [INFO] tensorflow: epoch = 105.17098445595855, learning_rate = 3.3327593e-05, loss = 9.796026e-06, step = 40596 (5.228 sec) 2021-10-19 09:09:59,456 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.850 INFO:tensorflow:epoch = 105.23834196891193, learning_rate = 3.304166e-05, loss = 1.1638509e-05, step = 40622 (5.255 sec) 2021-10-19 09:10:04,077 [INFO] tensorflow: epoch = 105.23834196891193, learning_rate = 3.304166e-05, loss = 1.1638509e-05, step = 40622 (5.255 sec) INFO:tensorflow:global_step/sec: 4.99818 2021-10-19 09:10:04,079 [INFO] tensorflow: global_step/sec: 4.99818 2021-10-19 09:10:04,488 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.875 INFO:tensorflow:epoch = 105.30829015544042, learning_rate = 3.2747343e-05, loss = 1.09630655e-05, step = 40649 (5.386 sec) 2021-10-19 09:10:09,464 [INFO] tensorflow: epoch = 105.30829015544042, learning_rate = 3.2747343e-05, loss = 1.09630655e-05, step = 40649 (5.386 sec) 2021-10-19 09:10:09,464 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.098 INFO:tensorflow:global_step/sec: 5.00309 2021-10-19 09:10:11,674 [INFO] tensorflow: global_step/sec: 5.00309 2021-10-19 09:10:14,442 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.090 INFO:tensorflow:epoch = 105.37823834196891, learning_rate = 3.2455617e-05, loss = 2.337958e-05, step = 40676 (5.419 sec) 2021-10-19 09:10:14,883 [INFO] tensorflow: epoch = 105.37823834196891, learning_rate = 3.2455617e-05, loss = 2.337958e-05, step = 40676 (5.419 sec) INFO:tensorflow:global_step/sec: 5.04721 2021-10-19 09:10:19,203 [INFO] tensorflow: global_step/sec: 5.04721 2021-10-19 09:10:19,399 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.177 INFO:tensorflow:epoch = 105.44818652849742, learning_rate = 3.2166517e-05, loss = 2.465878e-05, step = 40703 (5.301 sec) 2021-10-19 09:10:20,184 [INFO] tensorflow: epoch = 105.44818652849742, learning_rate = 3.2166517e-05, loss = 2.465878e-05, step = 40703 (5.301 sec) 2021-10-19 09:10:24,314 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.345 INFO:tensorflow:epoch = 105.51813471502591, learning_rate = 3.1879965e-05, loss = 1.9767407e-05, step = 40730 (5.333 sec) 2021-10-19 09:10:25,517 [INFO] tensorflow: epoch = 105.51813471502591, learning_rate = 3.1879965e-05, loss = 1.9767407e-05, step = 40730 (5.333 sec) INFO:tensorflow:global_step/sec: 5.04364 2021-10-19 09:10:26,737 [INFO] tensorflow: global_step/sec: 5.04364 2021-10-19 09:10:29,307 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.032 INFO:tensorflow:epoch = 105.58549222797927, learning_rate = 3.160645e-05, loss = 5.8386695e-06, step = 40756 (5.218 sec) 2021-10-19 09:10:30,734 [INFO] tensorflow: epoch = 105.58549222797927, learning_rate = 3.160645e-05, loss = 5.8386695e-06, step = 40756 (5.218 sec) INFO:tensorflow:global_step/sec: 4.97937 2021-10-19 09:10:34,369 [INFO] tensorflow: global_step/sec: 4.97937 2021-10-19 09:10:34,370 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.752 INFO:tensorflow:epoch = 105.65544041450778, learning_rate = 3.132492e-05, loss = 1.0657512e-05, step = 40783 (5.376 sec) 2021-10-19 09:10:36,110 [INFO] tensorflow: epoch = 105.65544041450778, learning_rate = 3.132492e-05, loss = 1.0657512e-05, step = 40783 (5.376 sec) 2021-10-19 09:10:39,293 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.314 INFO:tensorflow:epoch = 105.72538860103627, learning_rate = 3.1045864e-05, loss = 1.01262995e-05, step = 40810 (5.348 sec) 2021-10-19 09:10:41,458 [INFO] tensorflow: epoch = 105.72538860103627, learning_rate = 3.1045864e-05, loss = 1.01262995e-05, step = 40810 (5.348 sec) INFO:tensorflow:global_step/sec: 5.07246 2021-10-19 09:10:41,860 [INFO] tensorflow: global_step/sec: 5.07246 2021-10-19 09:10:44,249 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.181 INFO:tensorflow:epoch = 105.79533678756476, learning_rate = 3.0769326e-05, loss = 9.667462e-06, step = 40837 (5.364 sec) 2021-10-19 09:10:46,822 [INFO] tensorflow: epoch = 105.79533678756476, learning_rate = 3.0769326e-05, loss = 9.667462e-06, step = 40837 (5.364 sec) 2021-10-19 09:10:49,259 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.963 INFO:tensorflow:global_step/sec: 5.00361 2021-10-19 09:10:49,455 [INFO] tensorflow: global_step/sec: 5.00361 INFO:tensorflow:epoch = 105.86269430051814, learning_rate = 3.0505314e-05, loss = 6.853097e-06, step = 40863 (5.213 sec) 2021-10-19 09:10:52,035 [INFO] tensorflow: epoch = 105.86269430051814, learning_rate = 3.0505314e-05, loss = 6.853097e-06, step = 40863 (5.213 sec) 2021-10-19 09:10:54,324 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.745 INFO:tensorflow:epoch = 105.92746113989638, learning_rate = 3.0253632e-05, loss = 1.02824915e-05, step = 40888 (5.250 sec) 2021-10-19 09:10:57,284 [INFO] tensorflow: epoch = 105.92746113989638, learning_rate = 3.0253632e-05, loss = 1.02824915e-05, step = 40888 (5.250 sec) INFO:tensorflow:global_step/sec: 4.85191 2021-10-19 09:10:57,287 [INFO] tensorflow: global_step/sec: 4.85191 2021-10-19 09:10:59,539 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.177 INFO:tensorflow:epoch = 105.99481865284974, learning_rate = 2.9994073e-05, loss = 9.755025e-06, step = 40914 (5.354 sec) 2021-10-19 09:11:02,639 [INFO] tensorflow: epoch = 105.99481865284974, learning_rate = 2.9994073e-05, loss = 9.755025e-06, step = 40914 (5.354 sec) 2021-10-19 09:11:03,070 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 106/120: loss: 0.00001 learning rate: 0.00003 Time taken: 0:01:17.407902 ETA: 0:18:03.710631 2021-10-19 09:11:04,752 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.184 INFO:tensorflow:global_step/sec: 4.80969 2021-10-19 09:11:05,187 [INFO] tensorflow: global_step/sec: 4.80969 INFO:tensorflow:epoch = 106.05958549222798, learning_rate = 2.9746583e-05, loss = 3.6565193e-06, step = 40939 (5.304 sec) 2021-10-19 09:11:07,943 [INFO] tensorflow: epoch = 106.05958549222798, learning_rate = 2.9746583e-05, loss = 3.6565193e-06, step = 40939 (5.304 sec) 2021-10-19 09:11:10,086 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.749 INFO:tensorflow:epoch = 106.12435233160622, learning_rate = 2.9501167e-05, loss = 4.4114145e-06, step = 40964 (5.355 sec) 2021-10-19 09:11:13,298 [INFO] tensorflow: epoch = 106.12435233160622, learning_rate = 2.9501167e-05, loss = 4.4114145e-06, step = 40964 (5.355 sec) INFO:tensorflow:global_step/sec: 4.68271 2021-10-19 09:11:13,302 [INFO] tensorflow: global_step/sec: 4.68271 2021-10-19 09:11:15,419 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.754 INFO:tensorflow:epoch = 106.18911917098445, learning_rate = 2.925777e-05, loss = 8.908899e-06, step = 40989 (5.248 sec) 2021-10-19 09:11:18,545 [INFO] tensorflow: epoch = 106.18911917098445, learning_rate = 2.925777e-05, loss = 8.908899e-06, step = 40989 (5.248 sec) 2021-10-19 09:11:20,545 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.508 INFO:tensorflow:global_step/sec: 4.83868 2021-10-19 09:11:21,156 [INFO] tensorflow: global_step/sec: 4.83868 INFO:tensorflow:epoch = 106.25647668393783, learning_rate = 2.9006755e-05, loss = 1.4215003e-05, step = 41015 (5.318 sec) 2021-10-19 09:11:23,863 [INFO] tensorflow: epoch = 106.25647668393783, learning_rate = 2.9006755e-05, loss = 1.4215003e-05, step = 41015 (5.318 sec) 2021-10-19 09:11:25,775 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.125 INFO:tensorflow:epoch = 106.32124352331607, learning_rate = 2.8767412e-05, loss = 6.875432e-06, step = 41040 (5.258 sec) 2021-10-19 09:11:29,121 [INFO] tensorflow: epoch = 106.32124352331607, learning_rate = 2.8767412e-05, loss = 6.875432e-06, step = 41040 (5.258 sec) INFO:tensorflow:global_step/sec: 4.76967 2021-10-19 09:11:29,123 [INFO] tensorflow: global_step/sec: 4.76967 2021-10-19 09:11:31,051 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.953 INFO:tensorflow:epoch = 106.38860103626943, learning_rate = 2.8520604e-05, loss = 6.267109e-06, step = 41066 (5.439 sec) 2021-10-19 09:11:34,561 [INFO] tensorflow: epoch = 106.38860103626943, learning_rate = 2.8520604e-05, loss = 6.267109e-06, step = 41066 (5.439 sec) 2021-10-19 09:11:36,197 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.435 INFO:tensorflow:global_step/sec: 4.82635 2021-10-19 09:11:36,996 [INFO] tensorflow: global_step/sec: 4.82635 INFO:tensorflow:epoch = 106.4559585492228, learning_rate = 2.8275914e-05, loss = 4.834179e-06, step = 41092 (5.331 sec) 2021-10-19 09:11:39,892 [INFO] tensorflow: epoch = 106.4559585492228, learning_rate = 2.8275914e-05, loss = 4.834179e-06, step = 41092 (5.331 sec) 2021-10-19 09:11:41,324 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.506 INFO:tensorflow:global_step/sec: 4.82457 2021-10-19 09:11:44,872 [INFO] tensorflow: global_step/sec: 4.82457 INFO:tensorflow:epoch = 106.52331606217616, learning_rate = 2.8033322e-05, loss = 1.50929745e-05, step = 41118 (5.378 sec) 2021-10-19 09:11:45,270 [INFO] tensorflow: epoch = 106.52331606217616, learning_rate = 2.8033322e-05, loss = 1.50929745e-05, step = 41118 (5.378 sec) 2021-10-19 09:11:46,460 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.473 INFO:tensorflow:epoch = 106.59067357512954, learning_rate = 2.7792812e-05, loss = 6.0873003e-06, step = 41144 (5.310 sec) 2021-10-19 09:11:50,579 [INFO] tensorflow: epoch = 106.59067357512954, learning_rate = 2.7792812e-05, loss = 6.0873003e-06, step = 41144 (5.310 sec) 2021-10-19 09:11:51,667 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.205 INFO:tensorflow:global_step/sec: 4.83515 2021-10-19 09:11:52,732 [INFO] tensorflow: global_step/sec: 4.83515 INFO:tensorflow:epoch = 106.65803108808291, learning_rate = 2.7554366e-05, loss = 1.49501875e-05, step = 41170 (5.368 sec) 2021-10-19 09:11:55,948 [INFO] tensorflow: epoch = 106.65803108808291, learning_rate = 2.7554366e-05, loss = 1.49501875e-05, step = 41170 (5.368 sec) 2021-10-19 09:11:56,838 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.341 INFO:tensorflow:global_step/sec: 4.73525 2021-10-19 09:12:00,756 [INFO] tensorflow: global_step/sec: 4.73525 INFO:tensorflow:epoch = 106.72020725388602, learning_rate = 2.7336077e-05, loss = 1.2056772e-05, step = 41194 (5.211 sec) 2021-10-19 09:12:01,159 [INFO] tensorflow: epoch = 106.72020725388602, learning_rate = 2.7336077e-05, loss = 1.2056772e-05, step = 41194 (5.211 sec) 2021-10-19 09:12:02,199 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.655 INFO:tensorflow:epoch = 106.78497409326425, learning_rate = 2.7110518e-05, loss = 2.6684891e-05, step = 41219 (5.426 sec) 2021-10-19 09:12:06,584 [INFO] tensorflow: epoch = 106.78497409326425, learning_rate = 2.7110518e-05, loss = 2.6684891e-05, step = 41219 (5.426 sec) 2021-10-19 09:12:07,610 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.482 INFO:tensorflow:global_step/sec: 4.68597 2021-10-19 09:12:08,866 [INFO] tensorflow: global_step/sec: 4.68597 INFO:tensorflow:epoch = 106.84974093264249, learning_rate = 2.688682e-05, loss = 1.1970389e-05, step = 41244 (5.407 sec) 2021-10-19 09:12:11,992 [INFO] tensorflow: epoch = 106.84974093264249, learning_rate = 2.688682e-05, loss = 1.1970389e-05, step = 41244 (5.407 sec) 2021-10-19 09:12:13,063 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.341 INFO:tensorflow:global_step/sec: 4.69021 2021-10-19 09:12:16,968 [INFO] tensorflow: global_step/sec: 4.69021 INFO:tensorflow:epoch = 106.91709844559585, learning_rate = 2.6656175e-05, loss = 7.8403245e-06, step = 41270 (5.405 sec) 2021-10-19 09:12:17,397 [INFO] tensorflow: epoch = 106.91709844559585, learning_rate = 2.6656175e-05, loss = 7.8403245e-06, step = 41270 (5.405 sec) 2021-10-19 09:12:18,200 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.467 INFO:tensorflow:epoch = 106.98445595854923, learning_rate = 2.642748e-05, loss = 1.3838978e-05, step = 41296 (5.223 sec) 2021-10-19 09:12:22,620 [INFO] tensorflow: epoch = 106.98445595854923, learning_rate = 2.642748e-05, loss = 1.3838978e-05, step = 41296 (5.223 sec) 2021-10-19 09:12:23,215 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.945 2021-10-19 09:12:23,824 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 107/120: loss: 0.00001 learning rate: 0.00003 Time taken: 0:01:20.756490 ETA: 0:17:29.834376 INFO:tensorflow:global_step/sec: 4.95615 2021-10-19 09:12:24,636 [INFO] tensorflow: global_step/sec: 4.95615 INFO:tensorflow:epoch = 107.05440414507773, learning_rate = 2.6192052e-05, loss = 4.3947093e-06, step = 41323 (5.383 sec) 2021-10-19 09:12:28,004 [INFO] tensorflow: epoch = 107.05440414507773, learning_rate = 2.6192052e-05, loss = 4.3947093e-06, step = 41323 (5.383 sec) 2021-10-19 09:12:28,208 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.030 INFO:tensorflow:global_step/sec: 5.00857 2021-10-19 09:12:32,222 [INFO] tensorflow: global_step/sec: 5.00857 INFO:tensorflow:epoch = 107.12176165803109, learning_rate = 2.5967338e-05, loss = 5.934833e-06, step = 41349 (5.228 sec) 2021-10-19 09:12:33,231 [INFO] tensorflow: epoch = 107.12176165803109, learning_rate = 2.5967338e-05, loss = 5.934833e-06, step = 41349 (5.228 sec) 2021-10-19 09:12:33,232 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.905 2021-10-19 09:12:38,245 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.951 INFO:tensorflow:epoch = 107.18911917098445, learning_rate = 2.5744554e-05, loss = 1.2095853e-05, step = 41375 (5.261 sec) 2021-10-19 09:12:38,492 [INFO] tensorflow: epoch = 107.18911917098445, learning_rate = 2.5744554e-05, loss = 1.2095853e-05, step = 41375 (5.261 sec) INFO:tensorflow:global_step/sec: 4.95019 2021-10-19 09:12:39,898 [INFO] tensorflow: global_step/sec: 4.95019 2021-10-19 09:12:43,255 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.961 INFO:tensorflow:epoch = 107.25906735751296, learning_rate = 2.5515212e-05, loss = 1.0295192e-05, step = 41402 (5.358 sec) 2021-10-19 09:12:43,850 [INFO] tensorflow: epoch = 107.25906735751296, learning_rate = 2.5515212e-05, loss = 1.0295192e-05, step = 41402 (5.358 sec) INFO:tensorflow:global_step/sec: 5.08639 2021-10-19 09:12:47,369 [INFO] tensorflow: global_step/sec: 5.08639 2021-10-19 09:12:48,206 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.200 INFO:tensorflow:epoch = 107.32901554404145, learning_rate = 2.5287936e-05, loss = 1.170862e-05, step = 41429 (5.384 sec) 2021-10-19 09:12:49,234 [INFO] tensorflow: epoch = 107.32901554404145, learning_rate = 2.5287936e-05, loss = 1.170862e-05, step = 41429 (5.384 sec) 2021-10-19 09:12:53,371 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.364 INFO:tensorflow:epoch = 107.39637305699482, learning_rate = 2.507098e-05, loss = 3.5765383e-06, step = 41455 (5.423 sec) 2021-10-19 09:12:54,657 [INFO] tensorflow: epoch = 107.39637305699482, learning_rate = 2.507098e-05, loss = 3.5765383e-06, step = 41455 (5.423 sec) INFO:tensorflow:global_step/sec: 4.81004 2021-10-19 09:12:55,269 [INFO] tensorflow: global_step/sec: 4.81004 2021-10-19 09:12:58,616 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.066 INFO:tensorflow:epoch = 107.46113989637306, learning_rate = 2.4864112e-05, loss = 1.0527444e-05, step = 41480 (5.272 sec) 2021-10-19 09:12:59,929 [INFO] tensorflow: epoch = 107.46113989637306, learning_rate = 2.4864112e-05, loss = 1.0527444e-05, step = 41480 (5.272 sec) INFO:tensorflow:global_step/sec: 4.747 2021-10-19 09:13:03,275 [INFO] tensorflow: global_step/sec: 4.747 2021-10-19 09:13:03,914 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.875 INFO:tensorflow:epoch = 107.5259067357513, learning_rate = 2.4658973e-05, loss = 1.4653489e-05, step = 41505 (5.232 sec) 2021-10-19 09:13:05,162 [INFO] tensorflow: epoch = 107.5259067357513, learning_rate = 2.4658973e-05, loss = 1.4653489e-05, step = 41505 (5.232 sec) 2021-10-19 09:13:09,177 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.005 INFO:tensorflow:epoch = 107.59067357512954, learning_rate = 2.4455505e-05, loss = 2.247974e-05, step = 41530 (5.255 sec) 2021-10-19 09:13:10,416 [INFO] tensorflow: epoch = 107.59067357512954, learning_rate = 2.4455505e-05, loss = 2.247974e-05, step = 41530 (5.255 sec) INFO:tensorflow:global_step/sec: 4.78448 2021-10-19 09:13:11,217 [INFO] tensorflow: global_step/sec: 4.78448 2021-10-19 09:13:14,230 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.791 INFO:tensorflow:epoch = 107.65803108808291, learning_rate = 2.4245692e-05, loss = 1.2579808e-05, step = 41556 (5.210 sec) 2021-10-19 09:13:15,627 [INFO] tensorflow: epoch = 107.65803108808291, learning_rate = 2.4245692e-05, loss = 1.2579808e-05, step = 41556 (5.210 sec) INFO:tensorflow:global_step/sec: 4.92467 2021-10-19 09:13:18,933 [INFO] tensorflow: global_step/sec: 4.92467 2021-10-19 09:13:19,335 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.590 INFO:tensorflow:epoch = 107.72538860103627, learning_rate = 2.4037701e-05, loss = 6.735625e-06, step = 41582 (5.333 sec) 2021-10-19 09:13:20,960 [INFO] tensorflow: epoch = 107.72538860103627, learning_rate = 2.4037701e-05, loss = 6.735625e-06, step = 41582 (5.333 sec) 2021-10-19 09:13:24,296 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.159 INFO:tensorflow:epoch = 107.79533678756476, learning_rate = 2.382356e-05, loss = 2.034778e-05, step = 41609 (5.277 sec) 2021-10-19 09:13:26,237 [INFO] tensorflow: epoch = 107.79533678756476, learning_rate = 2.382356e-05, loss = 2.034778e-05, step = 41609 (5.277 sec) INFO:tensorflow:global_step/sec: 5.0507 2021-10-19 09:13:26,457 [INFO] tensorflow: global_step/sec: 5.0507 2021-10-19 09:13:29,238 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.238 INFO:tensorflow:epoch = 107.86528497409327, learning_rate = 2.3611332e-05, loss = 2.1280302e-05, step = 41636 (5.361 sec) 2021-10-19 09:13:31,598 [INFO] tensorflow: epoch = 107.86528497409327, learning_rate = 2.3611332e-05, loss = 2.1280302e-05, step = 41636 (5.361 sec) INFO:tensorflow:global_step/sec: 5.01775 2021-10-19 09:13:34,030 [INFO] tensorflow: global_step/sec: 5.01775 2021-10-19 09:13:34,235 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.014 INFO:tensorflow:epoch = 107.93264248704664, learning_rate = 2.3408784e-05, loss = 1.70036e-05, step = 41662 (5.229 sec) 2021-10-19 09:13:36,828 [INFO] tensorflow: epoch = 107.93264248704664, learning_rate = 2.3408784e-05, loss = 1.70036e-05, step = 41662 (5.229 sec) 2021-10-19 09:13:39,258 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.911 INFO:tensorflow:global_step/sec: 4.98976 2021-10-19 09:13:41,646 [INFO] tensorflow: global_step/sec: 4.98976 INFO:tensorflow:epoch = 108.0, learning_rate = 2.3207927e-05, loss = 4.109514e-06, step = 41688 (5.238 sec) 2021-10-19 09:13:42,066 [INFO] tensorflow: epoch = 108.0, learning_rate = 2.3207927e-05, loss = 4.109514e-06, step = 41688 (5.238 sec) 2021-10-19 09:13:42,066 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 108/120: loss: 0.00000 learning rate: 0.00002 Time taken: 0:01:18.240190 ETA: 0:15:38.882283 2021-10-19 09:13:44,238 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.081 INFO:tensorflow:epoch = 108.06735751295338, learning_rate = 2.3008815e-05, loss = 1.8074637e-05, step = 41714 (5.257 sec) 2021-10-19 09:13:47,323 [INFO] tensorflow: epoch = 108.06735751295338, learning_rate = 2.3008815e-05, loss = 1.8074637e-05, step = 41714 (5.257 sec) INFO:tensorflow:global_step/sec: 4.89237 2021-10-19 09:13:49,413 [INFO] tensorflow: global_step/sec: 4.89237 2021-10-19 09:13:49,414 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.320 INFO:tensorflow:epoch = 108.13471502590674, learning_rate = 2.2811413e-05, loss = 1.2863096e-05, step = 41740 (5.359 sec) 2021-10-19 09:13:52,682 [INFO] tensorflow: epoch = 108.13471502590674, learning_rate = 2.2811413e-05, loss = 1.2863096e-05, step = 41740 (5.359 sec) 2021-10-19 09:13:54,488 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.710 INFO:tensorflow:global_step/sec: 4.87823 2021-10-19 09:13:57,203 [INFO] tensorflow: global_step/sec: 4.87823 INFO:tensorflow:epoch = 108.20207253886011, learning_rate = 2.2615726e-05, loss = 1.3180073e-05, step = 41766 (5.409 sec) 2021-10-19 09:13:58,091 [INFO] tensorflow: epoch = 108.20207253886011, learning_rate = 2.2615726e-05, loss = 1.3180073e-05, step = 41766 (5.409 sec) 2021-10-19 09:13:59,675 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.283 INFO:tensorflow:epoch = 108.2720207253886, learning_rate = 2.2414257e-05, loss = 8.383021e-06, step = 41793 (5.379 sec) 2021-10-19 09:14:03,470 [INFO] tensorflow: epoch = 108.2720207253886, learning_rate = 2.2414257e-05, loss = 8.383021e-06, step = 41793 (5.379 sec) 2021-10-19 09:14:04,627 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.194 INFO:tensorflow:global_step/sec: 4.98109 2021-10-19 09:14:04,831 [INFO] tensorflow: global_step/sec: 4.98109 INFO:tensorflow:epoch = 108.3419689119171, learning_rate = 2.22146e-05, loss = 1.23458085e-05, step = 41820 (5.300 sec) 2021-10-19 09:14:08,770 [INFO] tensorflow: epoch = 108.3419689119171, learning_rate = 2.22146e-05, loss = 1.23458085e-05, step = 41820 (5.300 sec) 2021-10-19 09:14:09,559 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.281 INFO:tensorflow:global_step/sec: 5.04528 2021-10-19 09:14:12,363 [INFO] tensorflow: global_step/sec: 5.04528 INFO:tensorflow:epoch = 108.40932642487047, learning_rate = 2.2023993e-05, loss = 4.211488e-06, step = 41846 (5.312 sec) 2021-10-19 09:14:14,082 [INFO] tensorflow: epoch = 108.40932642487047, learning_rate = 2.2023993e-05, loss = 4.211488e-06, step = 41846 (5.312 sec) 2021-10-19 09:14:14,686 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.504 INFO:tensorflow:epoch = 108.47668393782384, learning_rate = 2.183504e-05, loss = 1.2353747e-05, step = 41872 (5.235 sec) 2021-10-19 09:14:19,317 [INFO] tensorflow: epoch = 108.47668393782384, learning_rate = 2.183504e-05, loss = 1.2353747e-05, step = 41872 (5.235 sec) 2021-10-19 09:14:19,717 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.879 INFO:tensorflow:global_step/sec: 4.90246 2021-10-19 09:14:20,114 [INFO] tensorflow: global_step/sec: 4.90246 INFO:tensorflow:epoch = 108.5440414507772, learning_rate = 2.1647707e-05, loss = 7.790315e-06, step = 41898 (5.234 sec) 2021-10-19 09:14:24,551 [INFO] tensorflow: epoch = 108.5440414507772, learning_rate = 2.1647707e-05, loss = 7.790315e-06, step = 41898 (5.234 sec) 2021-10-19 09:14:24,755 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.849 INFO:tensorflow:global_step/sec: 4.92291 2021-10-19 09:14:27,833 [INFO] tensorflow: global_step/sec: 4.92291 INFO:tensorflow:epoch = 108.61139896373058, learning_rate = 2.1461981e-05, loss = 9.460261e-06, step = 41924 (5.264 sec) 2021-10-19 09:14:29,815 [INFO] tensorflow: epoch = 108.61139896373058, learning_rate = 2.1461981e-05, loss = 9.460261e-06, step = 41924 (5.264 sec) 2021-10-19 09:14:29,815 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.765 2021-10-19 09:14:34,779 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.149 INFO:tensorflow:epoch = 108.68134715025907, learning_rate = 2.1270807e-05, loss = 1.9030156e-05, step = 41951 (5.352 sec) 2021-10-19 09:14:35,167 [INFO] tensorflow: epoch = 108.68134715025907, learning_rate = 2.1270807e-05, loss = 1.9030156e-05, step = 41951 (5.352 sec) INFO:tensorflow:global_step/sec: 5.04381 2021-10-19 09:14:35,367 [INFO] tensorflow: global_step/sec: 5.04381 2021-10-19 09:14:39,726 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.217 INFO:tensorflow:epoch = 108.75129533678756, learning_rate = 2.108134e-05, loss = 1.8749557e-05, step = 41978 (5.353 sec) 2021-10-19 09:14:40,520 [INFO] tensorflow: epoch = 108.75129533678756, learning_rate = 2.108134e-05, loss = 1.8749557e-05, step = 41978 (5.353 sec) INFO:tensorflow:global_step/sec: 5.07069 2021-10-19 09:14:42,861 [INFO] tensorflow: global_step/sec: 5.07069 2021-10-19 09:14:44,680 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.186 INFO:tensorflow:epoch = 108.82124352331607, learning_rate = 2.0893518e-05, loss = 4.1955846e-06, step = 42005 (5.412 sec) 2021-10-19 09:14:45,931 [INFO] tensorflow: epoch = 108.82124352331607, learning_rate = 2.0893518e-05, loss = 4.1955846e-06, step = 42005 (5.412 sec) 2021-10-19 09:14:49,688 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.970 INFO:tensorflow:global_step/sec: 4.97664 2021-10-19 09:14:50,497 [INFO] tensorflow: global_step/sec: 4.97664 INFO:tensorflow:epoch = 108.89119170984456, learning_rate = 2.0707412e-05, loss = 2.0198455e-05, step = 42032 (5.376 sec) 2021-10-19 09:14:51,307 [INFO] tensorflow: epoch = 108.89119170984456, learning_rate = 2.0707412e-05, loss = 2.0198455e-05, step = 42032 (5.376 sec) 2021-10-19 09:14:54,659 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.118 INFO:tensorflow:epoch = 108.96113989637306, learning_rate = 2.052294e-05, loss = 1.2132846e-05, step = 42059 (5.304 sec) 2021-10-19 09:14:56,611 [INFO] tensorflow: epoch = 108.96113989637306, learning_rate = 2.052294e-05, loss = 1.2132846e-05, step = 42059 (5.304 sec) INFO:tensorflow:global_step/sec: 5.05632 2021-10-19 09:14:58,012 [INFO] tensorflow: global_step/sec: 5.05632 2021-10-19 09:14:59,628 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 109/120: loss: 0.00002 learning rate: 0.00002 Time taken: 0:01:17.552427 ETA: 0:14:13.076700 2021-10-19 09:14:59,628 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.128 INFO:tensorflow:epoch = 109.02849740932643, learning_rate = 2.0346886e-05, loss = 6.6855537e-06, step = 42085 (5.237 sec) 2021-10-19 09:15:01,848 [INFO] tensorflow: epoch = 109.02849740932643, learning_rate = 2.0346886e-05, loss = 6.6855537e-06, step = 42085 (5.237 sec) 2021-10-19 09:15:04,635 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.974 INFO:tensorflow:global_step/sec: 4.95845 2021-10-19 09:15:05,676 [INFO] tensorflow: global_step/sec: 4.95845 INFO:tensorflow:epoch = 109.0958549222798, learning_rate = 2.0172321e-05, loss = 9.6767335e-06, step = 42111 (5.232 sec) 2021-10-19 09:15:07,080 [INFO] tensorflow: epoch = 109.0958549222798, learning_rate = 2.0172321e-05, loss = 9.6767335e-06, step = 42111 (5.232 sec) 2021-10-19 09:15:09,662 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.892 INFO:tensorflow:epoch = 109.16580310880829, learning_rate = 1.9992616e-05, loss = 1.2102897e-05, step = 42138 (5.326 sec) 2021-10-19 09:15:12,406 [INFO] tensorflow: epoch = 109.16580310880829, learning_rate = 1.9992616e-05, loss = 1.2102897e-05, step = 42138 (5.326 sec) INFO:tensorflow:global_step/sec: 5.04965 2021-10-19 09:15:13,201 [INFO] tensorflow: global_step/sec: 5.04965 2021-10-19 09:15:14,621 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.168 INFO:tensorflow:epoch = 109.23316062176166, learning_rate = 1.9821091e-05, loss = 1.07299875e-05, step = 42164 (5.205 sec) 2021-10-19 09:15:17,610 [INFO] tensorflow: epoch = 109.23316062176166, learning_rate = 1.9821091e-05, loss = 1.07299875e-05, step = 42164 (5.205 sec) 2021-10-19 09:15:19,589 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.130 INFO:tensorflow:global_step/sec: 5.01877 2021-10-19 09:15:20,773 [INFO] tensorflow: global_step/sec: 5.01877 INFO:tensorflow:epoch = 109.30310880829016, learning_rate = 1.9644536e-05, loss = 4.4484605e-06, step = 42191 (5.328 sec) 2021-10-19 09:15:22,939 [INFO] tensorflow: epoch = 109.30310880829016, learning_rate = 1.9644536e-05, loss = 4.4484605e-06, step = 42191 (5.328 sec) 2021-10-19 09:15:24,591 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.996 INFO:tensorflow:epoch = 109.37046632124353, learning_rate = 1.947598e-05, loss = 2.2182281e-05, step = 42217 (5.211 sec) 2021-10-19 09:15:28,150 [INFO] tensorflow: epoch = 109.37046632124353, learning_rate = 1.947598e-05, loss = 2.2182281e-05, step = 42217 (5.211 sec) INFO:tensorflow:global_step/sec: 5.00821 2021-10-19 09:15:28,360 [INFO] tensorflow: global_step/sec: 5.00821 2021-10-19 09:15:29,592 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.996 INFO:tensorflow:epoch = 109.43782383419689, learning_rate = 1.9308904e-05, loss = 1.1640109e-05, step = 42243 (5.217 sec) 2021-10-19 09:15:33,367 [INFO] tensorflow: epoch = 109.43782383419689, learning_rate = 1.9308904e-05, loss = 1.1640109e-05, step = 42243 (5.217 sec) 2021-10-19 09:15:34,592 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.003 INFO:tensorflow:global_step/sec: 4.99297 2021-10-19 09:15:35,971 [INFO] tensorflow: global_step/sec: 4.99297 INFO:tensorflow:epoch = 109.50518134715027, learning_rate = 1.9143245e-05, loss = 2.0049916e-05, step = 42269 (5.206 sec) 2021-10-19 09:15:38,573 [INFO] tensorflow: epoch = 109.50518134715027, learning_rate = 1.9143245e-05, loss = 2.0049916e-05, step = 42269 (5.206 sec) 2021-10-19 09:15:39,603 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.957 INFO:tensorflow:global_step/sec: 4.97709 2021-10-19 09:15:43,606 [INFO] tensorflow: global_step/sec: 4.97709 INFO:tensorflow:epoch = 109.57253886010363, learning_rate = 1.8979006e-05, loss = 9.676654e-06, step = 42295 (5.238 sec) 2021-10-19 09:15:43,811 [INFO] tensorflow: epoch = 109.57253886010363, learning_rate = 1.8979006e-05, loss = 9.676654e-06, step = 42295 (5.238 sec) 2021-10-19 09:15:44,628 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.903 INFO:tensorflow:epoch = 109.639896373057, learning_rate = 1.881616e-05, loss = 2.4580437e-05, step = 42321 (5.234 sec) 2021-10-19 09:15:49,045 [INFO] tensorflow: epoch = 109.639896373057, learning_rate = 1.881616e-05, loss = 2.4580437e-05, step = 42321 (5.234 sec) 2021-10-19 09:15:49,653 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.900 INFO:tensorflow:global_step/sec: 4.95851 2021-10-19 09:15:51,270 [INFO] tensorflow: global_step/sec: 4.95851 INFO:tensorflow:epoch = 109.70725388601036, learning_rate = 1.8654728e-05, loss = 1.0535168e-05, step = 42347 (5.225 sec) 2021-10-19 09:15:54,270 [INFO] tensorflow: epoch = 109.70725388601036, learning_rate = 1.8654728e-05, loss = 1.0535168e-05, step = 42347 (5.225 sec) 2021-10-19 09:15:54,682 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.890 INFO:tensorflow:global_step/sec: 5.01212 2021-10-19 09:15:58,851 [INFO] tensorflow: global_step/sec: 5.01212 INFO:tensorflow:epoch = 109.77720207253887, learning_rate = 1.8488561e-05, loss = 9.216061e-06, step = 42374 (5.382 sec) 2021-10-19 09:15:59,652 [INFO] tensorflow: epoch = 109.77720207253887, learning_rate = 1.8488561e-05, loss = 9.216061e-06, step = 42374 (5.382 sec) 2021-10-19 09:15:59,653 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.119 2021-10-19 09:16:04,608 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.180 INFO:tensorflow:epoch = 109.84715025906736, learning_rate = 1.8323857e-05, loss = 4.1411085e-06, step = 42401 (5.353 sec) 2021-10-19 09:16:05,005 [INFO] tensorflow: epoch = 109.84715025906736, learning_rate = 1.8323857e-05, loss = 4.1411085e-06, step = 42401 (5.353 sec) INFO:tensorflow:global_step/sec: 5.03448 2021-10-19 09:16:06,399 [INFO] tensorflow: global_step/sec: 5.03448 2021-10-19 09:16:09,606 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.011 INFO:tensorflow:epoch = 109.91709844559585, learning_rate = 1.8160637e-05, loss = 7.99406e-06, step = 42428 (5.385 sec) 2021-10-19 09:16:10,390 [INFO] tensorflow: epoch = 109.91709844559585, learning_rate = 1.8160637e-05, loss = 7.99406e-06, step = 42428 (5.385 sec) INFO:tensorflow:global_step/sec: 5.03781 2021-10-19 09:16:13,942 [INFO] tensorflow: global_step/sec: 5.03781 2021-10-19 09:16:14,549 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.234 INFO:tensorflow:epoch = 109.98704663212436, learning_rate = 1.7998856e-05, loss = 2.394046e-05, step = 42455 (5.340 sec) 2021-10-19 09:16:15,730 [INFO] tensorflow: epoch = 109.98704663212436, learning_rate = 1.7998856e-05, loss = 2.394046e-05, step = 42455 (5.340 sec) INFO:tensorflow:Saving checkpoints for step-42460. 2021-10-19 09:16:16,556 [INFO] tensorflow: Saving checkpoints for step-42460. WARNING:tensorflow:Ignoring: /tmp/tmpv1xwyzi3; No such file or directory 2021-10-19 09:16:16,802 [WARNING] tensorflow: Ignoring: /tmp/tmpv1xwyzi3; No such file or directory 2021-10-19 09:16:21,491 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 62, 0.00s/step 2021-10-19 09:16:22,467 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 62, 0.10s/step 2021-10-19 09:16:23,419 [INFO] iva.detectnet_v2.evaluation.evaluation: step 20 / 62, 0.10s/step 2021-10-19 09:16:24,352 [INFO] iva.detectnet_v2.evaluation.evaluation: step 30 / 62, 0.09s/step 2021-10-19 09:16:25,354 [INFO] iva.detectnet_v2.evaluation.evaluation: step 40 / 62, 0.10s/step 2021-10-19 09:16:26,321 [INFO] iva.detectnet_v2.evaluation.evaluation: step 50 / 62, 0.10s/step 2021-10-19 09:16:27,285 [INFO] iva.detectnet_v2.evaluation.evaluation: step 60 / 62, 0.10s/step Matching predictions to ground truth, class 1/7.: 100%|█| 423/423 [00:00<00:00, 13960.88it/s] Matching predictions to ground truth, class 2/7.: 100%|█| 182/182 [00:00<00:00, 12717.64it/s] Matching predictions to ground truth, class 7/7.: 100%|█| 216/216 [00:00<00:00, 25407.19it/s] Epoch 110/120 ========================= Validation cost: 0.000010 Mean average_precision (in %): 31.8790 class name average precision (in %) ------------- -------------------------- auto_rickshaw 0 bicycle 91.2032 bus 47.0271 car 84.9223 person 0 road_sign 0 truck 0 Median Inference Time: 0.019027 INFO:tensorflow:epoch = 110.0, learning_rate = 1.7969065e-05, loss = 7.0285205e-06, step = 42460 (12.036 sec) 2021-10-19 09:16:27,767 [INFO] tensorflow: epoch = 110.0, learning_rate = 1.7969065e-05, loss = 7.0285205e-06, step = 42460 (12.036 sec) 2021-10-19 09:16:27,767 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 110/120: loss: 0.00001 learning rate: 0.00002 Time taken: 0:01:28.134805 ETA: 0:14:41.348054 2021-10-19 09:16:30,631 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 6.218 INFO:tensorflow:global_step/sec: 2.03743 2021-10-19 09:16:32,593 [INFO] tensorflow: global_step/sec: 2.03743 INFO:tensorflow:epoch = 110.06735751295338, learning_rate = 1.78149e-05, loss = 1.0886186e-05, step = 42486 (5.230 sec) 2021-10-19 09:16:32,997 [INFO] tensorflow: epoch = 110.06735751295338, learning_rate = 1.78149e-05, loss = 1.0886186e-05, step = 42486 (5.230 sec) 2021-10-19 09:16:35,626 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.019 INFO:tensorflow:epoch = 110.13471502590674, learning_rate = 1.766206e-05, loss = 1.0125708e-05, step = 42512 (5.223 sec) 2021-10-19 09:16:38,220 [INFO] tensorflow: epoch = 110.13471502590674, learning_rate = 1.766206e-05, loss = 1.0125708e-05, step = 42512 (5.223 sec) INFO:tensorflow:global_step/sec: 4.96668 2021-10-19 09:16:40,244 [INFO] tensorflow: global_step/sec: 4.96668 2021-10-19 09:16:40,652 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.900 INFO:tensorflow:epoch = 110.20207253886011, learning_rate = 1.7510512e-05, loss = 1.3107904e-05, step = 42538 (5.275 sec) 2021-10-19 09:16:43,495 [INFO] tensorflow: epoch = 110.20207253886011, learning_rate = 1.7510512e-05, loss = 1.3107904e-05, step = 42538 (5.275 sec) 2021-10-19 09:16:45,670 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.930 INFO:tensorflow:global_step/sec: 4.9835 2021-10-19 09:16:47,869 [INFO] tensorflow: global_step/sec: 4.9835 INFO:tensorflow:epoch = 110.2720207253886, learning_rate = 1.7354538e-05, loss = 4.100623e-06, step = 42565 (5.388 sec) 2021-10-19 09:16:48,883 [INFO] tensorflow: epoch = 110.2720207253886, learning_rate = 1.7354538e-05, loss = 4.100623e-06, step = 42565 (5.388 sec) 2021-10-19 09:16:50,707 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.855 INFO:tensorflow:epoch = 110.33937823834198, learning_rate = 1.720566e-05, loss = 2.4404444e-05, step = 42591 (5.246 sec) 2021-10-19 09:16:54,129 [INFO] tensorflow: epoch = 110.33937823834198, learning_rate = 1.720566e-05, loss = 2.4404444e-05, step = 42591 (5.246 sec) INFO:tensorflow:global_step/sec: 4.944 2021-10-19 09:16:55,556 [INFO] tensorflow: global_step/sec: 4.944 2021-10-19 09:16:55,757 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.805 INFO:tensorflow:epoch = 110.40673575129534, learning_rate = 1.705803e-05, loss = 9.4845545e-06, step = 42617 (5.212 sec) 2021-10-19 09:16:59,340 [INFO] tensorflow: epoch = 110.40673575129534, learning_rate = 1.705803e-05, loss = 9.4845545e-06, step = 42617 (5.212 sec) 2021-10-19 09:17:00,751 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.025 INFO:tensorflow:global_step/sec: 4.97124 2021-10-19 09:17:03,199 [INFO] tensorflow: global_step/sec: 4.97124 INFO:tensorflow:epoch = 110.47409326424871, learning_rate = 1.6911683e-05, loss = 1.23129075e-05, step = 42643 (5.243 sec) 2021-10-19 09:17:04,583 [INFO] tensorflow: epoch = 110.47409326424871, learning_rate = 1.6911683e-05, loss = 1.23129075e-05, step = 42643 (5.243 sec) 2021-10-19 09:17:05,788 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.855 INFO:tensorflow:epoch = 110.5440414507772, learning_rate = 1.6761041e-05, loss = 8.844273e-06, step = 42670 (5.340 sec) 2021-10-19 09:17:09,923 [INFO] tensorflow: epoch = 110.5440414507772, learning_rate = 1.6761041e-05, loss = 8.844273e-06, step = 42670 (5.340 sec) INFO:tensorflow:global_step/sec: 5.05204 2021-10-19 09:17:10,721 [INFO] tensorflow: global_step/sec: 5.05204 2021-10-19 09:17:10,722 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.267 INFO:tensorflow:epoch = 110.6139896373057, learning_rate = 1.6611728e-05, loss = 2.4076513e-05, step = 42697 (5.342 sec) 2021-10-19 09:17:15,265 [INFO] tensorflow: epoch = 110.6139896373057, learning_rate = 1.6611728e-05, loss = 2.4076513e-05, step = 42697 (5.342 sec) 2021-10-19 09:17:15,682 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.162 INFO:tensorflow:global_step/sec: 5.04516 2021-10-19 09:17:18,253 [INFO] tensorflow: global_step/sec: 5.04516 INFO:tensorflow:epoch = 110.68134715025907, learning_rate = 1.6469208e-05, loss = 1.006431e-05, step = 42723 (5.272 sec) 2021-10-19 09:17:20,537 [INFO] tensorflow: epoch = 110.68134715025907, learning_rate = 1.6469208e-05, loss = 1.006431e-05, step = 42723 (5.272 sec) 2021-10-19 09:17:20,742 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.766 INFO:tensorflow:epoch = 110.74870466321244, learning_rate = 1.6327911e-05, loss = 1.694803e-05, step = 42749 (5.200 sec) 2021-10-19 09:17:25,737 [INFO] tensorflow: epoch = 110.74870466321244, learning_rate = 1.6327911e-05, loss = 1.694803e-05, step = 42749 (5.200 sec) 2021-10-19 09:17:25,737 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.022 INFO:tensorflow:global_step/sec: 4.9367 2021-10-19 09:17:25,951 [INFO] tensorflow: global_step/sec: 4.9367 2021-10-19 09:17:30,735 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.009 INFO:tensorflow:epoch = 110.81865284974094, learning_rate = 1.6182457e-05, loss = 4.6747105e-06, step = 42776 (5.395 sec) 2021-10-19 09:17:31,132 [INFO] tensorflow: epoch = 110.81865284974094, learning_rate = 1.6182457e-05, loss = 4.6747105e-06, step = 42776 (5.395 sec) INFO:tensorflow:global_step/sec: 4.97253 2021-10-19 09:17:33,593 [INFO] tensorflow: global_step/sec: 4.97253 2021-10-19 09:17:35,791 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.779 INFO:tensorflow:epoch = 110.88601036269431, learning_rate = 1.6043636e-05, loss = 1.12482285e-05, step = 42802 (5.244 sec) 2021-10-19 09:17:36,376 [INFO] tensorflow: epoch = 110.88601036269431, learning_rate = 1.6043636e-05, loss = 1.12482285e-05, step = 42802 (5.244 sec) 2021-10-19 09:17:40,755 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.149 INFO:tensorflow:global_step/sec: 5.02174 2021-10-19 09:17:41,160 [INFO] tensorflow: global_step/sec: 5.02174 INFO:tensorflow:epoch = 110.9559585492228, learning_rate = 1.5900712e-05, loss = 2.212189e-05, step = 42829 (5.380 sec) 2021-10-19 09:17:41,757 [INFO] tensorflow: epoch = 110.9559585492228, learning_rate = 1.5900712e-05, loss = 2.212189e-05, step = 42829 (5.380 sec) 2021-10-19 09:17:45,161 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 111/120: loss: 0.00002 learning rate: 0.00002 Time taken: 0:01:17.412051 ETA: 0:11:36.708457 2021-10-19 09:17:45,776 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.918 INFO:tensorflow:epoch = 111.02331606217616, learning_rate = 1.5764293e-05, loss = 2.0282447e-05, step = 42855 (5.219 sec) 2021-10-19 09:17:46,975 [INFO] tensorflow: epoch = 111.02331606217616, learning_rate = 1.5764293e-05, loss = 2.0282447e-05, step = 42855 (5.219 sec) INFO:tensorflow:global_step/sec: 4.95425 2021-10-19 09:17:48,830 [INFO] tensorflow: global_step/sec: 4.95425 2021-10-19 09:17:50,859 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.675 INFO:tensorflow:epoch = 111.09067357512954, learning_rate = 1.5629044e-05, loss = 1.0755752e-05, step = 42881 (5.284 sec) 2021-10-19 09:17:52,260 [INFO] tensorflow: epoch = 111.09067357512954, learning_rate = 1.5629044e-05, loss = 1.0755752e-05, step = 42881 (5.284 sec) 2021-10-19 09:17:55,736 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.505 INFO:tensorflow:global_step/sec: 5.07291 2021-10-19 09:17:56,321 [INFO] tensorflow: global_step/sec: 5.07291 INFO:tensorflow:epoch = 111.16062176165804, learning_rate = 1.5489813e-05, loss = 3.328595e-06, step = 42908 (5.292 sec) 2021-10-19 09:17:57,551 [INFO] tensorflow: epoch = 111.16062176165804, learning_rate = 1.5489813e-05, loss = 3.328595e-06, step = 42908 (5.292 sec) 2021-10-19 09:18:00,769 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.870 INFO:tensorflow:epoch = 111.2279792746114, learning_rate = 1.5356922e-05, loss = 4.583182e-06, step = 42934 (5.264 sec) 2021-10-19 09:18:02,815 [INFO] tensorflow: epoch = 111.2279792746114, learning_rate = 1.5356922e-05, loss = 4.583182e-06, step = 42934 (5.264 sec) INFO:tensorflow:global_step/sec: 4.92655 2021-10-19 09:18:04,034 [INFO] tensorflow: global_step/sec: 4.92655 2021-10-19 09:18:05,868 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.613 INFO:tensorflow:epoch = 111.29533678756476, learning_rate = 1.5225181e-05, loss = 1.7443283e-05, step = 42960 (5.233 sec) 2021-10-19 09:18:08,048 [INFO] tensorflow: epoch = 111.29533678756476, learning_rate = 1.5225181e-05, loss = 1.7443283e-05, step = 42960 (5.233 sec) 2021-10-19 09:18:10,823 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.183 INFO:tensorflow:global_step/sec: 5.01553 2021-10-19 09:18:11,610 [INFO] tensorflow: global_step/sec: 5.01553 INFO:tensorflow:epoch = 111.36528497409327, learning_rate = 1.5089535e-05, loss = 1.6019992e-05, step = 42987 (5.353 sec) 2021-10-19 09:18:13,401 [INFO] tensorflow: epoch = 111.36528497409327, learning_rate = 1.5089535e-05, loss = 1.6019992e-05, step = 42987 (5.353 sec) 2021-10-19 09:18:15,818 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.023 INFO:tensorflow:epoch = 111.43264248704664, learning_rate = 1.496009e-05, loss = 1.5106241e-05, step = 43013 (5.194 sec) 2021-10-19 09:18:18,595 [INFO] tensorflow: epoch = 111.43264248704664, learning_rate = 1.496009e-05, loss = 1.5106241e-05, step = 43013 (5.194 sec) INFO:tensorflow:global_step/sec: 5.00146 2021-10-19 09:18:19,208 [INFO] tensorflow: global_step/sec: 5.00146 2021-10-19 09:18:20,872 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.789 INFO:tensorflow:epoch = 111.5, learning_rate = 1.483174e-05, loss = 1.2411967e-05, step = 43039 (5.221 sec) 2021-10-19 09:18:23,817 [INFO] tensorflow: epoch = 111.5, learning_rate = 1.483174e-05, loss = 1.2411967e-05, step = 43039 (5.221 sec) 2021-10-19 09:18:25,833 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.158 INFO:tensorflow:global_step/sec: 4.95269 2021-10-19 09:18:26,881 [INFO] tensorflow: global_step/sec: 4.95269 INFO:tensorflow:epoch = 111.56735751295338, learning_rate = 1.4704493e-05, loss = 1.329845e-05, step = 43065 (5.317 sec) 2021-10-19 09:18:29,134 [INFO] tensorflow: epoch = 111.56735751295338, learning_rate = 1.4704493e-05, loss = 1.329845e-05, step = 43065 (5.317 sec) 2021-10-19 09:18:30,957 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.517 INFO:tensorflow:epoch = 111.63471502590674, learning_rate = 1.4578322e-05, loss = 1.4332612e-05, step = 43091 (5.226 sec) 2021-10-19 09:18:34,360 [INFO] tensorflow: epoch = 111.63471502590674, learning_rate = 1.4578322e-05, loss = 1.4332612e-05, step = 43091 (5.226 sec) INFO:tensorflow:global_step/sec: 4.92479 2021-10-19 09:18:34,597 [INFO] tensorflow: global_step/sec: 4.92479 2021-10-19 09:18:36,003 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.822 INFO:tensorflow:epoch = 111.70207253886011, learning_rate = 1.4453262e-05, loss = 1.7314604e-05, step = 43117 (5.222 sec) 2021-10-19 09:18:39,582 [INFO] tensorflow: epoch = 111.70207253886011, learning_rate = 1.4453262e-05, loss = 1.7314604e-05, step = 43117 (5.222 sec) 2021-10-19 09:18:40,991 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.049 INFO:tensorflow:global_step/sec: 5.00613 2021-10-19 09:18:42,188 [INFO] tensorflow: global_step/sec: 5.00613 INFO:tensorflow:epoch = 111.76943005181347, learning_rate = 1.4329263e-05, loss = 6.3844577e-06, step = 43143 (5.268 sec) 2021-10-19 09:18:44,850 [INFO] tensorflow: epoch = 111.76943005181347, learning_rate = 1.4329263e-05, loss = 6.3844577e-06, step = 43143 (5.268 sec) 2021-10-19 09:18:46,034 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.832 INFO:tensorflow:global_step/sec: 4.97593 2021-10-19 09:18:49,824 [INFO] tensorflow: global_step/sec: 4.97593 INFO:tensorflow:epoch = 111.83937823834198, learning_rate = 1.4201611e-05, loss = 1.1045381e-05, step = 43170 (5.363 sec) 2021-10-19 09:18:50,213 [INFO] tensorflow: epoch = 111.83937823834198, learning_rate = 1.4201611e-05, loss = 1.1045381e-05, step = 43170 (5.363 sec) 2021-10-19 09:18:51,015 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.077 INFO:tensorflow:epoch = 111.90932642487047, learning_rate = 1.407511e-05, loss = 9.394551e-06, step = 43197 (5.308 sec) 2021-10-19 09:18:55,521 [INFO] tensorflow: epoch = 111.90932642487047, learning_rate = 1.407511e-05, loss = 9.394551e-06, step = 43197 (5.308 sec) 2021-10-19 09:18:55,940 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.307 INFO:tensorflow:global_step/sec: 5.05637 2021-10-19 09:18:57,340 [INFO] tensorflow: global_step/sec: 5.05637 INFO:tensorflow:epoch = 111.97668393782384, learning_rate = 1.3954354e-05, loss = 1.1911725e-05, step = 43223 (5.232 sec) 2021-10-19 09:19:00,753 [INFO] tensorflow: epoch = 111.97668393782384, learning_rate = 1.3954354e-05, loss = 1.1911725e-05, step = 43223 (5.232 sec) 2021-10-19 09:19:00,957 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.935 2021-10-19 09:19:02,602 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 112/120: loss: 0.00002 learning rate: 0.00001 Time taken: 0:01:17.424354 ETA: 0:10:19.394829 INFO:tensorflow:global_step/sec: 4.90059 2021-10-19 09:19:05,094 [INFO] tensorflow: global_step/sec: 4.90059 INFO:tensorflow:epoch = 112.0440414507772, learning_rate = 1.383462e-05, loss = 5.370505e-06, step = 43249 (5.348 sec) 2021-10-19 09:19:06,101 [INFO] tensorflow: epoch = 112.0440414507772, learning_rate = 1.383462e-05, loss = 5.370505e-06, step = 43249 (5.348 sec) 2021-10-19 09:19:06,102 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.439 2021-10-19 09:19:11,071 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.123 INFO:tensorflow:epoch = 112.1139896373057, learning_rate = 1.3711389e-05, loss = 1.6708103e-05, step = 43276 (5.376 sec) 2021-10-19 09:19:11,478 [INFO] tensorflow: epoch = 112.1139896373057, learning_rate = 1.3711389e-05, loss = 1.6708103e-05, step = 43276 (5.376 sec) INFO:tensorflow:global_step/sec: 5.01199 2021-10-19 09:19:12,676 [INFO] tensorflow: global_step/sec: 5.01199 2021-10-19 09:19:16,068 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.015 INFO:tensorflow:epoch = 112.18134715025907, learning_rate = 1.35937535e-05, loss = 1.4722736e-05, step = 43302 (5.206 sec) 2021-10-19 09:19:16,683 [INFO] tensorflow: epoch = 112.18134715025907, learning_rate = 1.35937535e-05, loss = 1.4722736e-05, step = 43302 (5.206 sec) INFO:tensorflow:global_step/sec: 4.97973 2021-10-19 09:19:20,307 [INFO] tensorflow: global_step/sec: 4.97973 2021-10-19 09:19:21,120 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.797 INFO:tensorflow:epoch = 112.24870466321244, learning_rate = 1.3477126e-05, loss = 1.6733402e-05, step = 43328 (5.277 sec) 2021-10-19 09:19:21,960 [INFO] tensorflow: epoch = 112.24870466321244, learning_rate = 1.3477126e-05, loss = 1.6733402e-05, step = 43328 (5.277 sec) 2021-10-19 09:19:26,204 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.669 INFO:tensorflow:epoch = 112.3160621761658, learning_rate = 1.3361501e-05, loss = 1.6121438e-05, step = 43354 (5.305 sec) 2021-10-19 09:19:27,266 [INFO] tensorflow: epoch = 112.3160621761658, learning_rate = 1.3361501e-05, loss = 1.6121438e-05, step = 43354 (5.305 sec) INFO:tensorflow:global_step/sec: 4.89262 2021-10-19 09:19:28,073 [INFO] tensorflow: global_step/sec: 4.89262 2021-10-19 09:19:31,299 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.631 INFO:tensorflow:epoch = 112.38341968911918, learning_rate = 1.3246867e-05, loss = 7.717431e-06, step = 43380 (5.241 sec) 2021-10-19 09:19:32,506 [INFO] tensorflow: epoch = 112.38341968911918, learning_rate = 1.3246867e-05, loss = 7.717431e-06, step = 43380 (5.241 sec) INFO:tensorflow:global_step/sec: 4.99069 2021-10-19 09:19:35,688 [INFO] tensorflow: global_step/sec: 4.99069 2021-10-19 09:19:36,307 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.970 INFO:tensorflow:epoch = 112.45077720207254, learning_rate = 1.3133215e-05, loss = 1.7254675e-05, step = 43406 (5.194 sec) 2021-10-19 09:19:37,700 [INFO] tensorflow: epoch = 112.45077720207254, learning_rate = 1.3133215e-05, loss = 1.7254675e-05, step = 43406 (5.194 sec) 2021-10-19 09:19:41,335 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.886 INFO:tensorflow:epoch = 112.51813471502591, learning_rate = 1.302054e-05, loss = 2.6253272e-05, step = 43432 (5.228 sec) 2021-10-19 09:19:42,928 [INFO] tensorflow: epoch = 112.51813471502591, learning_rate = 1.302054e-05, loss = 2.6253272e-05, step = 43432 (5.228 sec) INFO:tensorflow:global_step/sec: 4.97899 2021-10-19 09:19:43,320 [INFO] tensorflow: global_step/sec: 4.97899 2021-10-19 09:19:46,284 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.211 INFO:tensorflow:epoch = 112.58808290155442, learning_rate = 1.290456e-05, loss = 2.5893689e-05, step = 43459 (5.363 sec) 2021-10-19 09:19:48,291 [INFO] tensorflow: epoch = 112.58808290155442, learning_rate = 1.290456e-05, loss = 2.5893689e-05, step = 43459 (5.363 sec) INFO:tensorflow:global_step/sec: 5.03001 2021-10-19 09:19:50,874 [INFO] tensorflow: global_step/sec: 5.03001 2021-10-19 09:19:51,270 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.056 INFO:tensorflow:epoch = 112.65803108808291, learning_rate = 1.2789601e-05, loss = 1.2569677e-05, step = 43486 (5.348 sec) 2021-10-19 09:19:53,639 [INFO] tensorflow: epoch = 112.65803108808291, learning_rate = 1.2789601e-05, loss = 1.2569677e-05, step = 43486 (5.348 sec) 2021-10-19 09:19:56,307 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.857 INFO:tensorflow:global_step/sec: 4.98434 2021-10-19 09:19:58,498 [INFO] tensorflow: global_step/sec: 4.98434 INFO:tensorflow:epoch = 112.72538860103627, learning_rate = 1.2679873e-05, loss = 3.643877e-05, step = 43512 (5.264 sec) 2021-10-19 09:19:58,904 [INFO] tensorflow: epoch = 112.72538860103627, learning_rate = 1.2679873e-05, loss = 3.643877e-05, step = 43512 (5.264 sec) 2021-10-19 09:20:01,339 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.873 INFO:tensorflow:epoch = 112.79274611398964, learning_rate = 1.2571087e-05, loss = 1.6365366e-05, step = 43538 (5.221 sec) 2021-10-19 09:20:04,125 [INFO] tensorflow: epoch = 112.79274611398964, learning_rate = 1.2571087e-05, loss = 1.6365366e-05, step = 43538 (5.221 sec) INFO:tensorflow:global_step/sec: 4.96163 2021-10-19 09:20:06,157 [INFO] tensorflow: global_step/sec: 4.96163 2021-10-19 09:20:06,362 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.909 INFO:tensorflow:epoch = 112.86010362694302, learning_rate = 1.2463234e-05, loss = 1.545576e-05, step = 43564 (5.258 sec) 2021-10-19 09:20:09,383 [INFO] tensorflow: epoch = 112.86010362694302, learning_rate = 1.2463234e-05, loss = 1.545576e-05, step = 43564 (5.258 sec) 2021-10-19 09:20:11,357 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.024 INFO:tensorflow:global_step/sec: 5.00815 2021-10-19 09:20:13,745 [INFO] tensorflow: global_step/sec: 5.00815 INFO:tensorflow:epoch = 112.93005181347151, learning_rate = 1.2352207e-05, loss = 4.096682e-06, step = 43591 (5.362 sec) 2021-10-19 09:20:14,745 [INFO] tensorflow: epoch = 112.93005181347151, learning_rate = 1.2352207e-05, loss = 4.096682e-06, step = 43591 (5.362 sec) 2021-10-19 09:20:16,337 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.082 INFO:tensorflow:epoch = 113.0, learning_rate = 1.2242181e-05, loss = 6.215476e-06, step = 43618 (5.361 sec) 2021-10-19 09:20:20,106 [INFO] tensorflow: epoch = 113.0, learning_rate = 1.2242181e-05, loss = 6.215476e-06, step = 43618 (5.361 sec) 2021-10-19 09:20:20,107 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 113/120: loss: 0.00001 learning rate: 0.00001 Time taken: 0:01:17.519182 ETA: 0:09:02.634277 INFO:tensorflow:global_step/sec: 5.00181 2021-10-19 09:20:21,342 [INFO] tensorflow: global_step/sec: 5.00181 2021-10-19 09:20:21,343 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.977 INFO:tensorflow:epoch = 113.06735751295338, learning_rate = 1.2137138e-05, loss = 1.3567958e-05, step = 43644 (5.257 sec) 2021-10-19 09:20:25,364 [INFO] tensorflow: epoch = 113.06735751295338, learning_rate = 1.2137138e-05, loss = 1.3567958e-05, step = 43644 (5.257 sec) 2021-10-19 09:20:26,399 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.779 INFO:tensorflow:global_step/sec: 4.94495 2021-10-19 09:20:29,027 [INFO] tensorflow: global_step/sec: 4.94495 INFO:tensorflow:epoch = 113.13471502590674, learning_rate = 1.203302e-05, loss = 2.9454728e-05, step = 43670 (5.257 sec) 2021-10-19 09:20:30,620 [INFO] tensorflow: epoch = 113.13471502590674, learning_rate = 1.203302e-05, loss = 2.9454728e-05, step = 43670 (5.257 sec) 2021-10-19 09:20:31,442 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.831 INFO:tensorflow:epoch = 113.20207253886011, learning_rate = 1.1929782e-05, loss = 9.0998e-06, step = 43696 (5.267 sec) 2021-10-19 09:20:35,888 [INFO] tensorflow: epoch = 113.20207253886011, learning_rate = 1.1929782e-05, loss = 9.0998e-06, step = 43696 (5.267 sec) 2021-10-19 09:20:36,472 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.883 INFO:tensorflow:global_step/sec: 4.95156 2021-10-19 09:20:36,701 [INFO] tensorflow: global_step/sec: 4.95156 INFO:tensorflow:epoch = 113.2720207253886, learning_rate = 1.18235075e-05, loss = 9.504865e-06, step = 43723 (5.324 sec) 2021-10-19 09:20:41,211 [INFO] tensorflow: epoch = 113.2720207253886, learning_rate = 1.18235075e-05, loss = 9.504865e-06, step = 43723 (5.324 sec) 2021-10-19 09:20:41,421 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.208 INFO:tensorflow:global_step/sec: 5.07373 2021-10-19 09:20:44,190 [INFO] tensorflow: global_step/sec: 5.07373 2021-10-19 09:20:46,398 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.097 INFO:tensorflow:epoch = 113.3419689119171, learning_rate = 1.1718191e-05, loss = 8.590988e-06, step = 43750 (5.419 sec) 2021-10-19 09:20:46,630 [INFO] tensorflow: epoch = 113.3419689119171, learning_rate = 1.1718191e-05, loss = 8.590988e-06, step = 43750 (5.419 sec) 2021-10-19 09:20:51,480 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.677 INFO:tensorflow:epoch = 113.40932642487047, learning_rate = 1.1617655e-05, loss = 1.3702695e-05, step = 43776 (5.257 sec) 2021-10-19 09:20:51,887 [INFO] tensorflow: epoch = 113.40932642487047, learning_rate = 1.1617655e-05, loss = 1.3702695e-05, step = 43776 (5.257 sec) INFO:tensorflow:global_step/sec: 4.93577 2021-10-19 09:20:51,889 [INFO] tensorflow: global_step/sec: 4.93577 2021-10-19 09:20:56,503 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.912 INFO:tensorflow:epoch = 113.47668393782384, learning_rate = 1.1517971e-05, loss = 1.3188601e-05, step = 43802 (5.237 sec) 2021-10-19 09:20:57,125 [INFO] tensorflow: epoch = 113.47668393782384, learning_rate = 1.1517971e-05, loss = 1.3188601e-05, step = 43802 (5.237 sec) INFO:tensorflow:global_step/sec: 5.01338 2021-10-19 09:20:59,469 [INFO] tensorflow: global_step/sec: 5.01338 2021-10-19 09:21:01,449 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.220 INFO:tensorflow:epoch = 113.54663212435234, learning_rate = 1.14153745e-05, loss = 9.008996e-06, step = 43829 (5.336 sec) 2021-10-19 09:21:02,460 [INFO] tensorflow: epoch = 113.54663212435234, learning_rate = 1.14153745e-05, loss = 9.008996e-06, step = 43829 (5.336 sec) 2021-10-19 09:21:06,390 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.241 INFO:tensorflow:global_step/sec: 5.0401 2021-10-19 09:21:07,008 [INFO] tensorflow: global_step/sec: 5.0401 INFO:tensorflow:epoch = 113.61658031088083, learning_rate = 1.1313692e-05, loss = 1.3575929e-05, step = 43856 (5.353 sec) 2021-10-19 09:21:07,813 [INFO] tensorflow: epoch = 113.61658031088083, learning_rate = 1.1313692e-05, loss = 1.3575929e-05, step = 43856 (5.353 sec) 2021-10-19 09:21:11,466 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.702 INFO:tensorflow:epoch = 113.6839378238342, learning_rate = 1.1216617e-05, loss = 8.366784e-06, step = 43882 (5.323 sec) 2021-10-19 09:21:13,136 [INFO] tensorflow: epoch = 113.6839378238342, learning_rate = 1.1216617e-05, loss = 8.366784e-06, step = 43882 (5.323 sec) INFO:tensorflow:global_step/sec: 4.94189 2021-10-19 09:21:14,698 [INFO] tensorflow: global_step/sec: 4.94189 2021-10-19 09:21:16,461 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.025 INFO:tensorflow:epoch = 113.75388601036269, learning_rate = 1.11167055e-05, loss = 4.7107337e-06, step = 43909 (5.330 sec) 2021-10-19 09:21:18,466 [INFO] tensorflow: epoch = 113.75388601036269, learning_rate = 1.11167055e-05, loss = 4.7107337e-06, step = 43909 (5.330 sec) 2021-10-19 09:21:21,482 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.917 INFO:tensorflow:global_step/sec: 5.02565 2021-10-19 09:21:22,259 [INFO] tensorflow: global_step/sec: 5.02565 INFO:tensorflow:epoch = 113.8238341968912, learning_rate = 1.10176725e-05, loss = 7.9979545e-06, step = 43936 (5.359 sec) 2021-10-19 09:21:23,825 [INFO] tensorflow: epoch = 113.8238341968912, learning_rate = 1.10176725e-05, loss = 7.9979545e-06, step = 43936 (5.359 sec) 2021-10-19 09:21:26,435 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.193 INFO:tensorflow:epoch = 113.89378238341969, learning_rate = 1.0919534e-05, loss = 9.722742e-06, step = 43963 (5.368 sec) 2021-10-19 09:21:29,193 [INFO] tensorflow: epoch = 113.89378238341969, learning_rate = 1.0919534e-05, loss = 9.722742e-06, step = 43963 (5.368 sec) INFO:tensorflow:global_step/sec: 5.04171 2021-10-19 09:21:29,796 [INFO] tensorflow: global_step/sec: 5.04171 2021-10-19 09:21:31,413 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.088 INFO:tensorflow:epoch = 113.9637305699482, learning_rate = 1.08222575e-05, loss = 1.0007804e-05, step = 43990 (5.398 sec) 2021-10-19 09:21:34,592 [INFO] tensorflow: epoch = 113.9637305699482, learning_rate = 1.08222575e-05, loss = 1.0007804e-05, step = 43990 (5.398 sec) 2021-10-19 09:21:36,338 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.308 INFO:tensorflow:global_step/sec: 5.03204 2021-10-19 09:21:37,348 [INFO] tensorflow: global_step/sec: 5.03204 2021-10-19 09:21:37,349 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 114/120: loss: 0.00001 learning rate: 0.00001 Time taken: 0:01:17.242207 ETA: 0:07:43.453241 INFO:tensorflow:epoch = 114.03367875647669, learning_rate = 1.0725859e-05, loss = 2.8082559e-05, step = 44017 (5.344 sec) 2021-10-19 09:21:39,936 [INFO] tensorflow: epoch = 114.03367875647669, learning_rate = 1.0725859e-05, loss = 2.8082559e-05, step = 44017 (5.344 sec) 2021-10-19 09:21:41,340 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.993 INFO:tensorflow:global_step/sec: 4.95652 2021-10-19 09:21:45,014 [INFO] tensorflow: global_step/sec: 4.95652 INFO:tensorflow:epoch = 114.10103626943005, learning_rate = 1.06338375e-05, loss = 3.8478647e-06, step = 44043 (5.292 sec) 2021-10-19 09:21:45,228 [INFO] tensorflow: epoch = 114.10103626943005, learning_rate = 1.06338375e-05, loss = 3.8478647e-06, step = 44043 (5.292 sec) 2021-10-19 09:21:46,413 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.714 INFO:tensorflow:epoch = 114.17098445595856, learning_rate = 1.0539107e-05, loss = 1.2597406e-05, step = 44070 (5.334 sec) 2021-10-19 09:21:50,563 [INFO] tensorflow: epoch = 114.17098445595856, learning_rate = 1.0539107e-05, loss = 1.2597406e-05, step = 44070 (5.334 sec) 2021-10-19 09:21:51,389 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.097 INFO:tensorflow:global_step/sec: 5.01132 2021-10-19 09:21:52,597 [INFO] tensorflow: global_step/sec: 5.01132 INFO:tensorflow:epoch = 114.23834196891193, learning_rate = 1.0448697e-05, loss = 5.9476324e-06, step = 44096 (5.224 sec) 2021-10-19 09:21:55,786 [INFO] tensorflow: epoch = 114.23834196891193, learning_rate = 1.0448697e-05, loss = 5.9476324e-06, step = 44096 (5.224 sec) 2021-10-19 09:21:56,401 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.956 INFO:tensorflow:global_step/sec: 5.01172 2021-10-19 09:22:00,179 [INFO] tensorflow: global_step/sec: 5.01172 INFO:tensorflow:epoch = 114.30569948186529, learning_rate = 1.0359053e-05, loss = 1.2455912e-05, step = 44122 (5.197 sec) 2021-10-19 09:22:00,984 [INFO] tensorflow: epoch = 114.30569948186529, learning_rate = 1.0359053e-05, loss = 1.2455912e-05, step = 44122 (5.197 sec) 2021-10-19 09:22:01,384 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.069 INFO:tensorflow:epoch = 114.37305699481865, learning_rate = 1.027017e-05, loss = 1.4630525e-05, step = 44148 (5.207 sec) 2021-10-19 09:22:06,191 [INFO] tensorflow: epoch = 114.37305699481865, learning_rate = 1.027017e-05, loss = 1.4630525e-05, step = 44148 (5.207 sec) 2021-10-19 09:22:06,427 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.830 INFO:tensorflow:global_step/sec: 4.98792 2021-10-19 09:22:07,798 [INFO] tensorflow: global_step/sec: 4.98792 2021-10-19 09:22:11,352 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.308 INFO:tensorflow:epoch = 114.44300518134716, learning_rate = 1.0178688e-05, loss = 3.8458497e-06, step = 44175 (5.364 sec) 2021-10-19 09:22:11,555 [INFO] tensorflow: epoch = 114.44300518134716, learning_rate = 1.0178688e-05, loss = 3.8458497e-06, step = 44175 (5.364 sec) INFO:tensorflow:global_step/sec: 5.0991 2021-10-19 09:22:15,250 [INFO] tensorflow: global_step/sec: 5.0991 2021-10-19 09:22:16,270 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.335 INFO:tensorflow:epoch = 114.51295336787565, learning_rate = 1.0088012e-05, loss = 1.2753398e-05, step = 44202 (5.298 sec) 2021-10-19 09:22:16,853 [INFO] tensorflow: epoch = 114.51295336787565, learning_rate = 1.0088012e-05, loss = 1.2753398e-05, step = 44202 (5.298 sec) 2021-10-19 09:22:21,140 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.536 INFO:tensorflow:epoch = 114.58290155440415, learning_rate = 9.9981535e-06, loss = 2.4114124e-05, step = 44229 (5.328 sec) 2021-10-19 09:22:22,180 [INFO] tensorflow: epoch = 114.58290155440415, learning_rate = 9.9981535e-06, loss = 2.4114124e-05, step = 44229 (5.328 sec) INFO:tensorflow:global_step/sec: 5.04961 2021-10-19 09:22:22,776 [INFO] tensorflow: global_step/sec: 5.04961 2021-10-19 09:22:26,237 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.622 INFO:tensorflow:epoch = 114.65025906735752, learning_rate = 9.912375e-06, loss = 1.9119467e-05, step = 44255 (5.253 sec) 2021-10-19 09:22:27,433 [INFO] tensorflow: epoch = 114.65025906735752, learning_rate = 9.912375e-06, loss = 1.9119467e-05, step = 44255 (5.253 sec) INFO:tensorflow:global_step/sec: 4.94143 2021-10-19 09:22:30,466 [INFO] tensorflow: global_step/sec: 4.94143 2021-10-19 09:22:31,248 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.957 INFO:tensorflow:epoch = 114.71761658031089, learning_rate = 9.827333e-06, loss = 2.2501798e-05, step = 44281 (5.219 sec) 2021-10-19 09:22:32,653 [INFO] tensorflow: epoch = 114.71761658031089, learning_rate = 9.827333e-06, loss = 2.2501798e-05, step = 44281 (5.219 sec) 2021-10-19 09:22:36,264 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.938 INFO:tensorflow:epoch = 114.78497409326425, learning_rate = 9.74302e-06, loss = 1.0084271e-05, step = 44307 (5.257 sec) 2021-10-19 09:22:37,910 [INFO] tensorflow: epoch = 114.78497409326425, learning_rate = 9.74302e-06, loss = 1.0084271e-05, step = 44307 (5.257 sec) INFO:tensorflow:global_step/sec: 4.9651 2021-10-19 09:22:38,119 [INFO] tensorflow: global_step/sec: 4.9651 2021-10-19 09:22:41,410 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.434 INFO:tensorflow:epoch = 114.85233160621762, learning_rate = 9.65943e-06, loss = 1.5593869e-05, step = 44333 (5.302 sec) 2021-10-19 09:22:43,212 [INFO] tensorflow: epoch = 114.85233160621762, learning_rate = 9.65943e-06, loss = 1.5593869e-05, step = 44333 (5.302 sec) INFO:tensorflow:global_step/sec: 4.91552 2021-10-19 09:22:45,850 [INFO] tensorflow: global_step/sec: 4.91552 2021-10-19 09:22:46,450 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.845 INFO:tensorflow:epoch = 114.91968911917098, learning_rate = 9.576558e-06, loss = 1.2558432e-05, step = 44359 (5.276 sec) 2021-10-19 09:22:48,488 [INFO] tensorflow: epoch = 114.91968911917098, learning_rate = 9.576558e-06, loss = 1.2558432e-05, step = 44359 (5.276 sec) 2021-10-19 09:22:51,553 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.597 INFO:tensorflow:global_step/sec: 4.9605 2021-10-19 09:22:53,510 [INFO] tensorflow: global_step/sec: 4.9605 INFO:tensorflow:epoch = 114.98704663212436, learning_rate = 9.494387e-06, loss = 9.105369e-06, step = 44385 (5.259 sec) 2021-10-19 09:22:53,747 [INFO] tensorflow: epoch = 114.98704663212436, learning_rate = 9.494387e-06, loss = 9.105369e-06, step = 44385 (5.259 sec) 2021-10-19 09:22:54,735 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 115/120: loss: 0.00001 learning rate: 0.00001 Time taken: 0:01:17.376921 ETA: 0:06:26.884605 2021-10-19 09:22:56,557 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.986 INFO:tensorflow:epoch = 115.05440414507773, learning_rate = 9.41294e-06, loss = 2.3035855e-05, step = 44411 (5.268 sec) 2021-10-19 09:22:59,015 [INFO] tensorflow: epoch = 115.05440414507773, learning_rate = 9.41294e-06, loss = 2.3035855e-05, step = 44411 (5.268 sec) INFO:tensorflow:global_step/sec: 4.95069 2021-10-19 09:23:01,186 [INFO] tensorflow: global_step/sec: 4.95069 2021-10-19 09:23:01,587 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.884 INFO:tensorflow:epoch = 115.12435233160622, learning_rate = 9.329095e-06, loss = 1.1047589e-05, step = 44438 (5.411 sec) 2021-10-19 09:23:04,426 [INFO] tensorflow: epoch = 115.12435233160622, learning_rate = 9.329095e-06, loss = 1.1047589e-05, step = 44438 (5.411 sec) 2021-10-19 09:23:06,675 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.658 INFO:tensorflow:global_step/sec: 4.93779 2021-10-19 09:23:08,882 [INFO] tensorflow: global_step/sec: 4.93779 INFO:tensorflow:epoch = 115.1917098445596, learning_rate = 9.249046e-06, loss = 9.984929e-06, step = 44464 (5.258 sec) 2021-10-19 09:23:09,684 [INFO] tensorflow: epoch = 115.1917098445596, learning_rate = 9.249046e-06, loss = 9.984929e-06, step = 44464 (5.258 sec) 2021-10-19 09:23:11,679 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.982 INFO:tensorflow:epoch = 115.26165803108809, learning_rate = 9.166662e-06, loss = 4.0152045e-06, step = 44491 (5.375 sec) 2021-10-19 09:23:15,059 [INFO] tensorflow: epoch = 115.26165803108809, learning_rate = 9.166662e-06, loss = 4.0152045e-06, step = 44491 (5.375 sec) INFO:tensorflow:global_step/sec: 5.04123 2021-10-19 09:23:16,419 [INFO] tensorflow: global_step/sec: 5.04123 2021-10-19 09:23:16,623 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.231 INFO:tensorflow:epoch = 115.33160621761658, learning_rate = 9.085009e-06, loss = 2.3445922e-05, step = 44518 (5.358 sec) 2021-10-19 09:23:20,417 [INFO] tensorflow: epoch = 115.33160621761658, learning_rate = 9.085009e-06, loss = 2.3445922e-05, step = 44518 (5.358 sec) 2021-10-19 09:23:21,628 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.983 INFO:tensorflow:global_step/sec: 5.01203 2021-10-19 09:23:24,001 [INFO] tensorflow: global_step/sec: 5.01203 INFO:tensorflow:epoch = 115.40155440414509, learning_rate = 9.004077e-06, loss = 9.883564e-06, step = 44545 (5.369 sec) 2021-10-19 09:23:25,786 [INFO] tensorflow: epoch = 115.40155440414509, learning_rate = 9.004077e-06, loss = 9.883564e-06, step = 44545 (5.369 sec) 2021-10-19 09:23:26,615 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.051 INFO:tensorflow:epoch = 115.46891191709845, learning_rate = 8.926827e-06, loss = 4.93774e-06, step = 44571 (5.218 sec) 2021-10-19 09:23:31,004 [INFO] tensorflow: epoch = 115.46891191709845, learning_rate = 8.926827e-06, loss = 4.93774e-06, step = 44571 (5.218 sec) INFO:tensorflow:global_step/sec: 4.98643 2021-10-19 09:23:31,622 [INFO] tensorflow: global_step/sec: 4.98643 2021-10-19 09:23:31,623 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.971 INFO:tensorflow:epoch = 115.53626943005182, learning_rate = 8.85024e-06, loss = 7.867396e-06, step = 44597 (5.280 sec) 2021-10-19 09:23:36,284 [INFO] tensorflow: epoch = 115.53626943005182, learning_rate = 8.85024e-06, loss = 7.867396e-06, step = 44597 (5.280 sec) 2021-10-19 09:23:36,681 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.773 INFO:tensorflow:global_step/sec: 4.95131 2021-10-19 09:23:39,297 [INFO] tensorflow: global_step/sec: 4.95131 INFO:tensorflow:epoch = 115.60362694300518, learning_rate = 8.774301e-06, loss = 1.829741e-05, step = 44623 (5.231 sec) 2021-10-19 09:23:41,515 [INFO] tensorflow: epoch = 115.60362694300518, learning_rate = 8.774301e-06, loss = 1.829741e-05, step = 44623 (5.231 sec) 2021-10-19 09:23:41,752 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.721 2021-10-19 09:23:46,702 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.202 INFO:tensorflow:epoch = 115.67357512953369, learning_rate = 8.6961445e-06, loss = 1.2186119e-05, step = 44650 (5.385 sec) 2021-10-19 09:23:46,900 [INFO] tensorflow: epoch = 115.67357512953369, learning_rate = 8.6961445e-06, loss = 1.2186119e-05, step = 44650 (5.385 sec) INFO:tensorflow:global_step/sec: 4.99636 2021-10-19 09:23:46,902 [INFO] tensorflow: global_step/sec: 4.99636 2021-10-19 09:23:51,680 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.092 INFO:tensorflow:epoch = 115.74352331606218, learning_rate = 8.618676e-06, loss = 2.8393806e-05, step = 44677 (5.367 sec) 2021-10-19 09:23:52,267 [INFO] tensorflow: epoch = 115.74352331606218, learning_rate = 8.618676e-06, loss = 2.8393806e-05, step = 44677 (5.367 sec) INFO:tensorflow:global_step/sec: 5.02976 2021-10-19 09:23:54,457 [INFO] tensorflow: global_step/sec: 5.02976 2021-10-19 09:23:56,675 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.020 INFO:tensorflow:epoch = 115.81088082901555, learning_rate = 8.54474e-06, loss = 1.6230084e-05, step = 44703 (5.231 sec) 2021-10-19 09:23:57,498 [INFO] tensorflow: epoch = 115.81088082901555, learning_rate = 8.54474e-06, loss = 1.6230084e-05, step = 44703 (5.231 sec) 2021-10-19 09:24:01,715 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.842 INFO:tensorflow:global_step/sec: 4.9431 2021-10-19 09:24:02,145 [INFO] tensorflow: global_step/sec: 4.9431 INFO:tensorflow:epoch = 115.87823834196891, learning_rate = 8.471431e-06, loss = 1.3083956e-05, step = 44729 (5.262 sec) 2021-10-19 09:24:02,760 [INFO] tensorflow: epoch = 115.87823834196891, learning_rate = 8.471431e-06, loss = 1.3083956e-05, step = 44729 (5.262 sec) 2021-10-19 09:24:06,659 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.231 INFO:tensorflow:epoch = 115.94818652849742, learning_rate = 8.395965e-06, loss = 2.0198777e-05, step = 44756 (5.301 sec) 2021-10-19 09:24:08,062 [INFO] tensorflow: epoch = 115.94818652849742, learning_rate = 8.395965e-06, loss = 2.0198777e-05, step = 44756 (5.301 sec) INFO:tensorflow:global_step/sec: 5.04329 2021-10-19 09:24:09,679 [INFO] tensorflow: global_step/sec: 5.04329 2021-10-19 09:24:11,680 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.916 2021-10-19 09:24:12,081 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 116/120: loss: 0.00001 learning rate: 0.00001 Time taken: 0:01:17.354632 ETA: 0:05:09.418528 INFO:tensorflow:epoch = 116.01554404145078, learning_rate = 8.323932e-06, loss = 8.3526e-06, step = 44782 (5.221 sec) 2021-10-19 09:24:13,283 [INFO] tensorflow: epoch = 116.01554404145078, learning_rate = 8.323932e-06, loss = 8.3526e-06, step = 44782 (5.221 sec) 2021-10-19 09:24:16,699 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.929 INFO:tensorflow:global_step/sec: 4.97127 2021-10-19 09:24:17,323 [INFO] tensorflow: global_step/sec: 4.97127 INFO:tensorflow:epoch = 116.08290155440415, learning_rate = 8.252518e-06, loss = 1.4859441e-05, step = 44808 (5.252 sec) 2021-10-19 09:24:18,535 [INFO] tensorflow: epoch = 116.08290155440415, learning_rate = 8.252518e-06, loss = 1.4859441e-05, step = 44808 (5.252 sec) 2021-10-19 09:24:21,752 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.790 INFO:tensorflow:epoch = 116.15284974093265, learning_rate = 8.179e-06, loss = 4.2289134e-06, step = 44835 (5.391 sec) 2021-10-19 09:24:23,926 [INFO] tensorflow: epoch = 116.15284974093265, learning_rate = 8.179e-06, loss = 4.2289134e-06, step = 44835 (5.391 sec) INFO:tensorflow:global_step/sec: 4.99607 2021-10-19 09:24:24,929 [INFO] tensorflow: global_step/sec: 4.99607 2021-10-19 09:24:26,761 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.969 INFO:tensorflow:epoch = 116.22020725388602, learning_rate = 8.108829e-06, loss = 1.3054295e-05, step = 44861 (5.268 sec) 2021-10-19 09:24:29,194 [INFO] tensorflow: epoch = 116.22020725388602, learning_rate = 8.108829e-06, loss = 1.3054295e-05, step = 44861 (5.268 sec) 2021-10-19 09:24:31,736 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.100 INFO:tensorflow:global_step/sec: 4.99854 2021-10-19 09:24:32,532 [INFO] tensorflow: global_step/sec: 4.99854 INFO:tensorflow:epoch = 116.29015544041451, learning_rate = 8.0366e-06, loss = 8.499214e-06, step = 44888 (5.340 sec) 2021-10-19 09:24:34,534 [INFO] tensorflow: epoch = 116.29015544041451, learning_rate = 8.0366e-06, loss = 8.499214e-06, step = 44888 (5.340 sec) 2021-10-19 09:24:36,735 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.005 INFO:tensorflow:epoch = 116.35751295336787, learning_rate = 7.967651e-06, loss = 6.506769e-06, step = 44914 (5.253 sec) 2021-10-19 09:24:39,787 [INFO] tensorflow: epoch = 116.35751295336787, learning_rate = 7.967651e-06, loss = 6.506769e-06, step = 44914 (5.253 sec) INFO:tensorflow:global_step/sec: 4.94194 2021-10-19 09:24:40,221 [INFO] tensorflow: global_step/sec: 4.94194 2021-10-19 09:24:41,835 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.612 INFO:tensorflow:epoch = 116.42487046632125, learning_rate = 7.8992925e-06, loss = 4.9653786e-06, step = 44940 (5.234 sec) 2021-10-19 09:24:45,022 [INFO] tensorflow: epoch = 116.42487046632125, learning_rate = 7.8992925e-06, loss = 4.9653786e-06, step = 44940 (5.234 sec) 2021-10-19 09:24:46,840 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.983 INFO:tensorflow:global_step/sec: 4.99683 2021-10-19 09:24:47,826 [INFO] tensorflow: global_step/sec: 4.99683 INFO:tensorflow:epoch = 116.49222797927462, learning_rate = 7.831521e-06, loss = 5.131905e-06, step = 44966 (5.227 sec) 2021-10-19 09:24:50,248 [INFO] tensorflow: epoch = 116.49222797927462, learning_rate = 7.831521e-06, loss = 5.131905e-06, step = 44966 (5.227 sec) 2021-10-19 09:24:51,806 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.137 INFO:tensorflow:global_step/sec: 5.0235 2021-10-19 09:24:55,390 [INFO] tensorflow: global_step/sec: 5.0235 INFO:tensorflow:epoch = 116.56217616580311, learning_rate = 7.761755e-06, loss = 1.3957904e-05, step = 44993 (5.361 sec) 2021-10-19 09:24:55,610 [INFO] tensorflow: epoch = 116.56217616580311, learning_rate = 7.761755e-06, loss = 1.3957904e-05, step = 44993 (5.361 sec) 2021-10-19 09:24:56,801 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.020 INFO:tensorflow:epoch = 116.63212435233162, learning_rate = 7.692617e-06, loss = 2.0209973e-05, step = 45020 (5.333 sec) 2021-10-19 09:25:00,943 [INFO] tensorflow: epoch = 116.63212435233162, learning_rate = 7.692617e-06, loss = 2.0209973e-05, step = 45020 (5.333 sec) 2021-10-19 09:25:01,771 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.123 INFO:tensorflow:global_step/sec: 4.99759 2021-10-19 09:25:02,994 [INFO] tensorflow: global_step/sec: 4.99759 INFO:tensorflow:epoch = 116.70207253886011, learning_rate = 7.6240885e-06, loss = 2.1761649e-05, step = 45047 (5.355 sec) 2021-10-19 09:25:06,298 [INFO] tensorflow: epoch = 116.70207253886011, learning_rate = 7.6240885e-06, loss = 2.1761649e-05, step = 45047 (5.355 sec) 2021-10-19 09:25:06,695 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.314 INFO:tensorflow:global_step/sec: 5.03426 2021-10-19 09:25:10,542 [INFO] tensorflow: global_step/sec: 5.03426 INFO:tensorflow:epoch = 116.76943005181347, learning_rate = 7.558678e-06, loss = 2.0755213e-05, step = 45073 (5.235 sec) 2021-10-19 09:25:11,532 [INFO] tensorflow: epoch = 116.76943005181347, learning_rate = 7.558678e-06, loss = 2.0755213e-05, step = 45073 (5.235 sec) 2021-10-19 09:25:11,738 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.828 INFO:tensorflow:epoch = 116.83678756476684, learning_rate = 7.493829e-06, loss = 1.0017405e-05, step = 45099 (5.238 sec) 2021-10-19 09:25:16,770 [INFO] tensorflow: epoch = 116.83678756476684, learning_rate = 7.493829e-06, loss = 1.0017405e-05, step = 45099 (5.238 sec) 2021-10-19 09:25:16,771 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.873 INFO:tensorflow:global_step/sec: 4.99118 2021-10-19 09:25:18,155 [INFO] tensorflow: global_step/sec: 4.99118 2021-10-19 09:25:21,785 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.944 INFO:tensorflow:epoch = 116.90414507772022, learning_rate = 7.4295426e-06, loss = 1.07499545e-05, step = 45125 (5.229 sec) 2021-10-19 09:25:21,999 [INFO] tensorflow: epoch = 116.90414507772022, learning_rate = 7.4295426e-06, loss = 1.07499545e-05, step = 45125 (5.229 sec) INFO:tensorflow:global_step/sec: 5.01812 2021-10-19 09:25:25,728 [INFO] tensorflow: global_step/sec: 5.01812 2021-10-19 09:25:26,727 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.236 INFO:tensorflow:epoch = 116.97409326424871, learning_rate = 7.363358e-06, loss = 1.1556602e-05, step = 45152 (5.325 sec) 2021-10-19 09:25:27,324 [INFO] tensorflow: epoch = 116.97409326424871, learning_rate = 7.363358e-06, loss = 1.1556602e-05, step = 45152 (5.325 sec) 2021-10-19 09:25:29,310 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 117/120: loss: 0.00002 learning rate: 0.00001 Time taken: 0:01:17.223925 ETA: 0:03:51.671776 2021-10-19 09:25:31,691 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.148 INFO:tensorflow:epoch = 117.0440414507772, learning_rate = 7.297762e-06, loss = 2.428388e-05, step = 45179 (5.356 sec) 2021-10-19 09:25:32,680 [INFO] tensorflow: epoch = 117.0440414507772, learning_rate = 7.297762e-06, loss = 2.428388e-05, step = 45179 (5.356 sec) INFO:tensorflow:global_step/sec: 5.01849 2021-10-19 09:25:33,300 [INFO] tensorflow: global_step/sec: 5.01849 2021-10-19 09:25:36,696 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.982 INFO:tensorflow:epoch = 117.11139896373058, learning_rate = 7.235151e-06, loss = 1.5194661e-05, step = 45205 (5.215 sec) 2021-10-19 09:25:37,895 [INFO] tensorflow: epoch = 117.11139896373058, learning_rate = 7.235151e-06, loss = 1.5194661e-05, step = 45205 (5.215 sec) INFO:tensorflow:global_step/sec: 5.04512 2021-10-19 09:25:40,832 [INFO] tensorflow: global_step/sec: 5.04512 2021-10-19 09:25:41,629 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.274 INFO:tensorflow:epoch = 117.18134715025907, learning_rate = 7.170704e-06, loss = 1.6112423e-05, step = 45232 (5.343 sec) 2021-10-19 09:25:43,238 [INFO] tensorflow: epoch = 117.18134715025907, learning_rate = 7.170704e-06, loss = 1.6112423e-05, step = 45232 (5.343 sec) 2021-10-19 09:25:46,652 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.908 INFO:tensorflow:epoch = 117.24870466321244, learning_rate = 7.1091836e-06, loss = 3.8340418e-06, step = 45258 (5.232 sec) 2021-10-19 09:25:48,469 [INFO] tensorflow: epoch = 117.24870466321244, learning_rate = 7.1091836e-06, loss = 3.8340418e-06, step = 45258 (5.232 sec) INFO:tensorflow:global_step/sec: 4.97477 2021-10-19 09:25:48,471 [INFO] tensorflow: global_step/sec: 4.97477 2021-10-19 09:25:51,680 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.892 INFO:tensorflow:epoch = 117.3160621761658, learning_rate = 7.048184e-06, loss = 4.095911e-06, step = 45284 (5.294 sec) 2021-10-19 09:25:53,763 [INFO] tensorflow: epoch = 117.3160621761658, learning_rate = 7.048184e-06, loss = 4.095911e-06, step = 45284 (5.294 sec) INFO:tensorflow:global_step/sec: 4.92854 2021-10-19 09:25:56,181 [INFO] tensorflow: global_step/sec: 4.92854 2021-10-19 09:25:56,770 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.649 INFO:tensorflow:epoch = 117.38601036269431, learning_rate = 6.9854027e-06, loss = 4.7872404e-06, step = 45311 (5.374 sec) 2021-10-19 09:25:59,137 [INFO] tensorflow: epoch = 117.38601036269431, learning_rate = 6.9854027e-06, loss = 4.7872404e-06, step = 45311 (5.374 sec) 2021-10-19 09:26:01,712 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.237 INFO:tensorflow:global_step/sec: 5.09384 2021-10-19 09:26:03,641 [INFO] tensorflow: global_step/sec: 5.09384 INFO:tensorflow:epoch = 117.4559585492228, learning_rate = 6.9231805e-06, loss = 5.5813434e-06, step = 45338 (5.284 sec) 2021-10-19 09:26:04,421 [INFO] tensorflow: epoch = 117.4559585492228, learning_rate = 6.9231805e-06, loss = 5.5813434e-06, step = 45338 (5.284 sec) 2021-10-19 09:26:06,587 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.512 INFO:tensorflow:epoch = 117.5259067357513, learning_rate = 6.8615063e-06, loss = 2.1295456e-05, step = 45365 (5.335 sec) 2021-10-19 09:26:09,757 [INFO] tensorflow: epoch = 117.5259067357513, learning_rate = 6.8615063e-06, loss = 2.1295456e-05, step = 45365 (5.335 sec) INFO:tensorflow:global_step/sec: 5.04847 2021-10-19 09:26:11,168 [INFO] tensorflow: global_step/sec: 5.04847 2021-10-19 09:26:11,586 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.006 INFO:tensorflow:epoch = 117.5958549222798, learning_rate = 6.800388e-06, loss = 1.5287427e-05, step = 45392 (5.373 sec) 2021-10-19 09:26:15,129 [INFO] tensorflow: epoch = 117.5958549222798, learning_rate = 6.800388e-06, loss = 1.5287427e-05, step = 45392 (5.373 sec) 2021-10-19 09:26:16,532 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.219 INFO:tensorflow:global_step/sec: 5.01973 2021-10-19 09:26:18,738 [INFO] tensorflow: global_step/sec: 5.01973 INFO:tensorflow:epoch = 117.66321243523316, learning_rate = 6.7420374e-06, loss = 1.1539818e-05, step = 45418 (5.240 sec) 2021-10-19 09:26:20,370 [INFO] tensorflow: epoch = 117.66321243523316, learning_rate = 6.7420374e-06, loss = 1.1539818e-05, step = 45418 (5.240 sec) 2021-10-19 09:26:21,540 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.970 INFO:tensorflow:epoch = 117.73316062176167, learning_rate = 6.681983e-06, loss = 1.839967e-05, step = 45445 (5.364 sec) 2021-10-19 09:26:25,734 [INFO] tensorflow: epoch = 117.73316062176167, learning_rate = 6.681983e-06, loss = 1.839967e-05, step = 45445 (5.364 sec) INFO:tensorflow:global_step/sec: 5.0124 2021-10-19 09:26:26,319 [INFO] tensorflow: global_step/sec: 5.0124 2021-10-19 09:26:26,522 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.074 INFO:tensorflow:epoch = 117.80310880829016, learning_rate = 6.6224575e-06, loss = 3.2722542e-06, step = 45472 (5.335 sec) 2021-10-19 09:26:31,069 [INFO] tensorflow: epoch = 117.80310880829016, learning_rate = 6.6224575e-06, loss = 3.2722542e-06, step = 45472 (5.335 sec) 2021-10-19 09:26:31,501 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.088 INFO:tensorflow:global_step/sec: 5.02529 2021-10-19 09:26:33,881 [INFO] tensorflow: global_step/sec: 5.02529 INFO:tensorflow:epoch = 117.87305699481865, learning_rate = 6.563468e-06, loss = 7.4475697e-06, step = 45499 (5.361 sec) 2021-10-19 09:26:36,430 [INFO] tensorflow: epoch = 117.87305699481865, learning_rate = 6.563468e-06, loss = 7.4475697e-06, step = 45499 (5.361 sec) 2021-10-19 09:26:36,430 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.287 INFO:tensorflow:global_step/sec: 5.02509 2021-10-19 09:26:41,443 [INFO] tensorflow: global_step/sec: 5.02509 2021-10-19 09:26:41,444 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.947 INFO:tensorflow:epoch = 117.94041450777202, learning_rate = 6.507157e-06, loss = 1.4258129e-05, step = 45525 (5.221 sec) 2021-10-19 09:26:41,651 [INFO] tensorflow: epoch = 117.94041450777202, learning_rate = 6.507157e-06, loss = 1.4258129e-05, step = 45525 (5.221 sec) 2021-10-19 09:26:46,161 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 118/120: loss: 0.00001 learning rate: 0.00001 Time taken: 0:01:16.851351 ETA: 0:02:33.702702 2021-10-19 09:26:46,365 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.325 INFO:tensorflow:epoch = 118.01036269430053, learning_rate = 6.4491887e-06, loss = 1.0102321e-05, step = 45552 (5.334 sec) 2021-10-19 09:26:46,985 [INFO] tensorflow: epoch = 118.01036269430053, learning_rate = 6.4491887e-06, loss = 1.0102321e-05, step = 45552 (5.334 sec) INFO:tensorflow:global_step/sec: 5.02402 2021-10-19 09:26:49,007 [INFO] tensorflow: global_step/sec: 5.02402 2021-10-19 09:26:51,373 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.968 INFO:tensorflow:epoch = 118.07772020725389, learning_rate = 6.3938583e-06, loss = 2.2323104e-05, step = 45578 (5.225 sec) 2021-10-19 09:26:52,210 [INFO] tensorflow: epoch = 118.07772020725389, learning_rate = 6.3938583e-06, loss = 2.2323104e-05, step = 45578 (5.225 sec) 2021-10-19 09:26:56,346 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.110 INFO:tensorflow:global_step/sec: 5.04216 2021-10-19 09:26:56,543 [INFO] tensorflow: global_step/sec: 5.04216 INFO:tensorflow:epoch = 118.14766839378238, learning_rate = 6.3368993e-06, loss = 5.616147e-06, step = 45605 (5.323 sec) 2021-10-19 09:26:57,534 [INFO] tensorflow: epoch = 118.14766839378238, learning_rate = 6.3368993e-06, loss = 5.616147e-06, step = 45605 (5.323 sec) 2021-10-19 09:27:01,291 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.224 INFO:tensorflow:epoch = 118.21761658031089, learning_rate = 6.2804534e-06, loss = 1.9332103e-05, step = 45632 (5.343 sec) 2021-10-19 09:27:02,876 [INFO] tensorflow: epoch = 118.21761658031089, learning_rate = 6.2804534e-06, loss = 1.9332103e-05, step = 45632 (5.343 sec) INFO:tensorflow:global_step/sec: 5.0118 2021-10-19 09:27:04,125 [INFO] tensorflow: global_step/sec: 5.0118 2021-10-19 09:27:06,317 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.900 INFO:tensorflow:epoch = 118.28497409326425, learning_rate = 6.2265767e-06, loss = 7.0089773e-06, step = 45658 (5.217 sec) 2021-10-19 09:27:08,093 [INFO] tensorflow: epoch = 118.28497409326425, learning_rate = 6.2265767e-06, loss = 7.0089773e-06, step = 45658 (5.217 sec) 2021-10-19 09:27:11,223 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.383 INFO:tensorflow:global_step/sec: 5.05375 2021-10-19 09:27:11,644 [INFO] tensorflow: global_step/sec: 5.05375 INFO:tensorflow:epoch = 118.35492227979275, learning_rate = 6.171102e-06, loss = 1.408786e-05, step = 45685 (5.337 sec) 2021-10-19 09:27:13,430 [INFO] tensorflow: epoch = 118.35492227979275, learning_rate = 6.171102e-06, loss = 1.408786e-05, step = 45685 (5.337 sec) 2021-10-19 09:27:16,294 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.722 INFO:tensorflow:epoch = 118.42227979274612, learning_rate = 6.1181636e-06, loss = 1.625124e-05, step = 45711 (5.255 sec) 2021-10-19 09:27:18,685 [INFO] tensorflow: epoch = 118.42227979274612, learning_rate = 6.1181636e-06, loss = 1.625124e-05, step = 45711 (5.255 sec) INFO:tensorflow:global_step/sec: 4.97454 2021-10-19 09:27:19,283 [INFO] tensorflow: global_step/sec: 4.97454 2021-10-19 09:27:21,252 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.170 INFO:tensorflow:epoch = 118.49222797927462, learning_rate = 6.0636658e-06, loss = 1.2351979e-05, step = 45738 (5.381 sec) 2021-10-19 09:27:24,066 [INFO] tensorflow: epoch = 118.49222797927462, learning_rate = 6.0636658e-06, loss = 1.2351979e-05, step = 45738 (5.381 sec) 2021-10-19 09:27:26,272 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.921 INFO:tensorflow:global_step/sec: 5.01764 2021-10-19 09:27:26,857 [INFO] tensorflow: global_step/sec: 5.01764 INFO:tensorflow:epoch = 118.55958549222798, learning_rate = 6.0116377e-06, loss = 1.0599466e-05, step = 45764 (5.233 sec) 2021-10-19 09:27:29,298 [INFO] tensorflow: epoch = 118.55958549222798, learning_rate = 6.0116377e-06, loss = 1.0599466e-05, step = 45764 (5.233 sec) 2021-10-19 09:27:31,261 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.046 INFO:tensorflow:global_step/sec: 5.01955 2021-10-19 09:27:34,427 [INFO] tensorflow: global_step/sec: 5.01955 INFO:tensorflow:epoch = 118.62953367875649, learning_rate = 5.958089e-06, loss = 1.3966674e-05, step = 45791 (5.335 sec) 2021-10-19 09:27:34,633 [INFO] tensorflow: epoch = 118.62953367875649, learning_rate = 5.958089e-06, loss = 1.3966674e-05, step = 45791 (5.335 sec) 2021-10-19 09:27:36,232 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.119 INFO:tensorflow:epoch = 118.69948186528498, learning_rate = 5.9050117e-06, loss = 2.3991612e-05, step = 45818 (5.357 sec) 2021-10-19 09:27:39,990 [INFO] tensorflow: epoch = 118.69948186528498, learning_rate = 5.9050117e-06, loss = 2.3991612e-05, step = 45818 (5.357 sec) 2021-10-19 09:27:41,197 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.145 INFO:tensorflow:global_step/sec: 5.03221 2021-10-19 09:27:41,978 [INFO] tensorflow: global_step/sec: 5.03221 INFO:tensorflow:epoch = 118.76943005181347, learning_rate = 5.8524133e-06, loss = 3.6583908e-05, step = 45845 (5.315 sec) 2021-10-19 09:27:45,305 [INFO] tensorflow: epoch = 118.76943005181347, learning_rate = 5.8524133e-06, loss = 3.6583908e-05, step = 45845 (5.315 sec) 2021-10-19 09:27:46,090 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.437 INFO:tensorflow:global_step/sec: 5.07572 2021-10-19 09:27:49,465 [INFO] tensorflow: global_step/sec: 5.07572 INFO:tensorflow:epoch = 118.83937823834198, learning_rate = 5.8002774e-06, loss = 2.8063208e-05, step = 45872 (5.337 sec) 2021-10-19 09:27:50,642 [INFO] tensorflow: epoch = 118.83937823834198, learning_rate = 5.8002774e-06, loss = 2.8063208e-05, step = 45872 (5.337 sec) 2021-10-19 09:27:51,080 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.042 INFO:tensorflow:epoch = 118.90673575129534, learning_rate = 5.7505144e-06, loss = 1.6539967e-05, step = 45898 (5.240 sec) 2021-10-19 09:27:55,882 [INFO] tensorflow: epoch = 118.90673575129534, learning_rate = 5.7505144e-06, loss = 1.6539967e-05, step = 45898 (5.240 sec) 2021-10-19 09:27:56,086 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.979 INFO:tensorflow:global_step/sec: 4.98777 2021-10-19 09:27:57,084 [INFO] tensorflow: global_step/sec: 4.98777 2021-10-19 09:28:01,049 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.150 INFO:tensorflow:epoch = 118.97668393782384, learning_rate = 5.699292e-06, loss = 6.534213e-06, step = 45925 (5.373 sec) 2021-10-19 09:28:01,255 [INFO] tensorflow: epoch = 118.97668393782384, learning_rate = 5.699292e-06, loss = 6.534213e-06, step = 45925 (5.373 sec) 2021-10-19 09:28:03,091 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 119/120: loss: 0.00001 learning rate: 0.00001 Time taken: 0:01:16.929274 ETA: 0:01:16.929274 INFO:tensorflow:global_step/sec: 4.97462 2021-10-19 09:28:04,722 [INFO] tensorflow: global_step/sec: 4.97462 2021-10-19 09:28:06,114 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.745 INFO:tensorflow:epoch = 119.0440414507772, learning_rate = 5.6503895e-06, loss = 4.2162983e-06, step = 45951 (5.254 sec) 2021-10-19 09:28:06,510 [INFO] tensorflow: epoch = 119.0440414507772, learning_rate = 5.6503895e-06, loss = 4.2162983e-06, step = 45951 (5.254 sec) 2021-10-19 09:28:11,123 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.968 INFO:tensorflow:epoch = 119.11139896373058, learning_rate = 5.6019126e-06, loss = 1.3723929e-05, step = 45977 (5.222 sec) 2021-10-19 09:28:11,732 [INFO] tensorflow: epoch = 119.11139896373058, learning_rate = 5.6019126e-06, loss = 1.3723929e-05, step = 45977 (5.222 sec) INFO:tensorflow:global_step/sec: 4.99576 2021-10-19 09:28:12,329 [INFO] tensorflow: global_step/sec: 4.99576 2021-10-19 09:28:16,143 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.920 INFO:tensorflow:epoch = 119.17875647668394, learning_rate = 5.5538567e-06, loss = 1.6826045e-05, step = 46003 (5.242 sec) 2021-10-19 09:28:16,974 [INFO] tensorflow: epoch = 119.17875647668394, learning_rate = 5.5538567e-06, loss = 1.6826045e-05, step = 46003 (5.242 sec) INFO:tensorflow:global_step/sec: 5.00728 2021-10-19 09:28:19,918 [INFO] tensorflow: global_step/sec: 5.00728 2021-10-19 09:28:21,117 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.106 INFO:tensorflow:epoch = 119.24870466321244, learning_rate = 5.504381e-06, loss = 1.5857655e-05, step = 46030 (5.389 sec) 2021-10-19 09:28:22,363 [INFO] tensorflow: epoch = 119.24870466321244, learning_rate = 5.504381e-06, loss = 1.5857655e-05, step = 46030 (5.389 sec) 2021-10-19 09:28:26,165 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.812 INFO:tensorflow:epoch = 119.3160621761658, learning_rate = 5.4571565e-06, loss = 8.279925e-06, step = 46056 (5.237 sec) 2021-10-19 09:28:27,600 [INFO] tensorflow: epoch = 119.3160621761658, learning_rate = 5.4571565e-06, loss = 8.279925e-06, step = 46056 (5.237 sec) INFO:tensorflow:global_step/sec: 4.94576 2021-10-19 09:28:27,601 [INFO] tensorflow: global_step/sec: 4.94576 2021-10-19 09:28:31,156 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.041 INFO:tensorflow:epoch = 119.38601036269431, learning_rate = 5.408547e-06, loss = 8.846857e-06, step = 46083 (5.346 sec) 2021-10-19 09:28:32,946 [INFO] tensorflow: epoch = 119.38601036269431, learning_rate = 5.408547e-06, loss = 8.846857e-06, step = 46083 (5.346 sec) INFO:tensorflow:global_step/sec: 5.04202 2021-10-19 09:28:35,138 [INFO] tensorflow: global_step/sec: 5.04202 2021-10-19 09:28:36,139 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.070 INFO:tensorflow:epoch = 119.45336787564767, learning_rate = 5.3621397e-06, loss = 8.558325e-06, step = 46109 (5.205 sec) 2021-10-19 09:28:38,150 [INFO] tensorflow: epoch = 119.45336787564767, learning_rate = 5.3621397e-06, loss = 8.558325e-06, step = 46109 (5.205 sec) 2021-10-19 09:28:41,189 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.803 INFO:tensorflow:global_step/sec: 4.9819 2021-10-19 09:28:42,765 [INFO] tensorflow: global_step/sec: 4.9819 INFO:tensorflow:epoch = 119.52072538860104, learning_rate = 5.3161352e-06, loss = 2.0656833e-05, step = 46135 (5.210 sec) 2021-10-19 09:28:43,361 [INFO] tensorflow: epoch = 119.52072538860104, learning_rate = 5.3161352e-06, loss = 2.0656833e-05, step = 46135 (5.210 sec) 2021-10-19 09:28:46,190 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.996 INFO:tensorflow:epoch = 119.58808290155442, learning_rate = 5.270526e-06, loss = 1.7167888e-05, step = 46161 (5.231 sec) 2021-10-19 09:28:48,591 [INFO] tensorflow: epoch = 119.58808290155442, learning_rate = 5.270526e-06, loss = 1.7167888e-05, step = 46161 (5.231 sec) INFO:tensorflow:global_step/sec: 4.9697 2021-10-19 09:28:50,412 [INFO] tensorflow: global_step/sec: 4.9697 2021-10-19 09:28:51,251 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.760 INFO:tensorflow:epoch = 119.65544041450778, learning_rate = 5.225313e-06, loss = 9.900321e-06, step = 46187 (5.264 sec) 2021-10-19 09:28:53,855 [INFO] tensorflow: epoch = 119.65544041450778, learning_rate = 5.225313e-06, loss = 9.900321e-06, step = 46187 (5.264 sec) 2021-10-19 09:28:56,245 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.025 INFO:tensorflow:global_step/sec: 4.99556 2021-10-19 09:28:58,018 [INFO] tensorflow: global_step/sec: 4.99556 INFO:tensorflow:epoch = 119.72538860103627, learning_rate = 5.1787633e-06, loss = 3.6267425e-06, step = 46214 (5.364 sec) 2021-10-19 09:28:59,219 [INFO] tensorflow: epoch = 119.72538860103627, learning_rate = 5.1787633e-06, loss = 3.6267425e-06, step = 46214 (5.364 sec) 2021-10-19 09:29:01,232 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.053 INFO:tensorflow:epoch = 119.79274611398964, learning_rate = 5.1343327e-06, loss = 1.1443032e-05, step = 46240 (5.278 sec) 2021-10-19 09:29:04,498 [INFO] tensorflow: epoch = 119.79274611398964, learning_rate = 5.1343327e-06, loss = 1.1443032e-05, step = 46240 (5.278 sec) INFO:tensorflow:global_step/sec: 4.93294 2021-10-19 09:29:05,722 [INFO] tensorflow: global_step/sec: 4.93294 2021-10-19 09:29:06,330 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.617 INFO:tensorflow:epoch = 119.86269430051814, learning_rate = 5.088594e-06, loss = 7.942395e-06, step = 46267 (5.358 sec) 2021-10-19 09:29:09,856 [INFO] tensorflow: epoch = 119.86269430051814, learning_rate = 5.088594e-06, loss = 7.942395e-06, step = 46267 (5.358 sec) 2021-10-19 09:29:11,239 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.375 INFO:tensorflow:global_step/sec: 5.04 2021-10-19 09:29:13,262 [INFO] tensorflow: global_step/sec: 5.04 INFO:tensorflow:epoch = 119.93264248704664, learning_rate = 5.0432677e-06, loss = 2.196906e-05, step = 46294 (5.337 sec) 2021-10-19 09:29:15,193 [INFO] tensorflow: epoch = 119.93264248704664, learning_rate = 5.0432677e-06, loss = 2.196906e-05, step = 46294 (5.337 sec) 2021-10-19 09:29:16,202 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.148 INFO:tensorflow:Saving checkpoints for step-46320. 2021-10-19 09:29:20,232 [INFO] tensorflow: Saving checkpoints for step-46320. WARNING:tensorflow:Ignoring: /tmp/tmpqjd0rk7f; No such file or directory 2021-10-19 09:29:20,446 [WARNING] tensorflow: Ignoring: /tmp/tmpqjd0rk7f; No such file or directory 2021-10-19 09:29:25,389 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 62, 0.00s/step 2021-10-19 09:29:26,332 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 62, 0.09s/step 2021-10-19 09:29:27,336 [INFO] iva.detectnet_v2.evaluation.evaluation: step 20 / 62, 0.10s/step 2021-10-19 09:29:28,282 [INFO] iva.detectnet_v2.evaluation.evaluation: step 30 / 62, 0.09s/step 2021-10-19 09:29:29,282 [INFO] iva.detectnet_v2.evaluation.evaluation: step 40 / 62, 0.10s/step 2021-10-19 09:29:30,189 [INFO] iva.detectnet_v2.evaluation.evaluation: step 50 / 62, 0.09s/step 2021-10-19 09:29:31,123 [INFO] iva.detectnet_v2.evaluation.evaluation: step 60 / 62, 0.09s/step Matching predictions to ground truth, class 1/7.: 100%|█| 410/410 [00:00<00:00, 12678.15it/s] Matching predictions to ground truth, class 2/7.: 100%|█| 180/180 [00:00<00:00, 13295.79it/s] Matching predictions to ground truth, class 7/7.: 100%|█| 203/203 [00:00<00:00, 30940.21it/s] Epoch 120/120 ========================= Validation cost: 0.000010 Mean average_precision (in %): 31.8961 class name average precision (in %) ------------- -------------------------- auto_rickshaw 0 bicycle 91.3138 bus 46.8785 car 85.0804 person 0 road_sign 0 truck 0 Median Inference Time: 0.019721 WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:95: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead. 2021-10-19 09:29:32,186 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:95: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:98: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead. 2021-10-19 09:29:32,187 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:98: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead. 2021-10-19 09:29:32,190 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.148 Time taken to run __main__:main: 2:37:56.199058. 2021-10-19 05:29:35,227 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.