Matplotlib created a temporary config/cache directory at /tmp/matplotlib-5ugbte98 because the default path (/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing. Using TensorFlow backend. WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them. WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them. Using TensorFlow backend. WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:43: The name tf.train.SessionRunHook is deprecated. Please use tf.estimator.SessionRunHook instead. 2022-04-22 17:54:14,311 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:43: The name tf.train.SessionRunHook is deprecated. Please use tf.estimator.SessionRunHook instead. Using TensorFlow backend. WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:43: The name tf.train.SessionRunHook is deprecated. Please use tf.estimator.SessionRunHook instead. 2022-04-22 17:54:14,327 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:43: The name tf.train.SessionRunHook is deprecated. Please use tf.estimator.SessionRunHook instead. WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/checkpoint_saver_hook.py:25: The name tf.train.CheckpointSaverHook is deprecated. Please use tf.estimator.CheckpointSaverHook instead. 2022-04-22 17:54:14,415 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/checkpoint_saver_hook.py:25: The name tf.train.CheckpointSaverHook is deprecated. Please use tf.estimator.CheckpointSaverHook instead. WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/checkpoint_saver_hook.py:25: The name tf.train.CheckpointSaverHook is deprecated. Please use tf.estimator.CheckpointSaverHook instead. 2022-04-22 17:54:14,415 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/checkpoint_saver_hook.py:25: The name tf.train.CheckpointSaverHook is deprecated. Please use tf.estimator.CheckpointSaverHook instead. WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/scripts/train.py:69: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead. 2022-04-22 17:54:14,416 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/scripts/train.py:69: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead. WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/scripts/train.py:69: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead. WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/scripts/train.py:69: The name tf.logging.INFO is deprecated. Please use tf.compat.v1.logging.INFO instead. 2022-04-22 17:54:14,416 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/scripts/train.py:69: The name tf.logging.INFO is deprecated. Please use tf.compat.v1.logging.INFO instead. 2022-04-22 17:54:14,416 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/scripts/train.py:69: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead. WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/scripts/train.py:69: The name tf.logging.INFO is deprecated. Please use tf.compat.v1.logging.INFO instead. 2022-04-22 17:54:14,416 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/scripts/train.py:69: The name tf.logging.INFO is deprecated. Please use tf.compat.v1.logging.INFO instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod/tensorflow/__init__.py:117: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead. 2022-04-22 17:54:14,474 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/horovod/tensorflow/__init__.py:117: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod/tensorflow/__init__.py:117: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead. 2022-04-22 17:54:14,474 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/horovod/tensorflow/__init__.py:117: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod/tensorflow/__init__.py:143: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead. 2022-04-22 17:54:14,474 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/horovod/tensorflow/__init__.py:143: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod/tensorflow/__init__.py:143: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead. 2022-04-22 17:54:14,474 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/horovod/tensorflow/__init__.py:143: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead. 2022-04-22 17:54:15,244 [INFO] __main__: Loading experiment spec at /workspace/tao-experiments/specs/peoplenet_spec.txt. 2022-04-22 17:54:15,244 [INFO] iva.detectnet_v2.spec_handler.spec_loader: Merging specification from /workspace/tao-experiments/specs/peoplenet_spec.txt 2022-04-22 17:54:15,247 [INFO] iva.common.logging.logging: Log file already exists at /workspace/tao-experiments/models/peoplenet_detector/status.json 2022-04-22 17:54:15,247 [INFO] __main__: Loading experiment spec at /workspace/tao-experiments/specs/peoplenet_spec.txt. 2022-04-22 17:54:15,248 [INFO] iva.detectnet_v2.spec_handler.spec_loader: Merging specification from /workspace/tao-experiments/specs/peoplenet_spec.txt WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:107: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead. 2022-04-22 17:54:15,501 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:107: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead. WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:110: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead. 2022-04-22 17:54:15,502 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:110: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead. WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:113: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead. 2022-04-22 17:54:15,505 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:113: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead. WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:107: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead. 2022-04-22 17:54:15,508 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:107: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead. WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:110: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead. 2022-04-22 17:54:15,509 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:110: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead. WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:113: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead. 2022-04-22 17:54:15,510 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:113: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead. 2022-04-22 17:54:15,526 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead. 2022-04-22 17:54:15,527 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead. 2022-04-22 17:54:15,531 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead. 2022-04-22 17:54:15,531 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1834: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead. 2022-04-22 17:54:15,548 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1834: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1834: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead. 2022-04-22 17:54:15,550 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1834: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead. 2022-04-22 17:54:17,758 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead. 2022-04-22 17:54:17,759 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead. 2022-04-22 17:54:17,759 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead. 2022-04-22 17:54:17,759 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead. 2022-04-22 17:54:18,037 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead. 2022-04-22 17:54:18,038 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead. /usr/local/lib/python3.6/dist-packages/keras/engine/saving.py:292: UserWarning: No training configuration found in save file: the model was *not* compiled. Compile it manually. warnings.warn('No training configuration found in save file: ' 2022-04-22 17:54:24,426 [INFO] iva.detectnet_v2.objectives.bbox_objective: Default L1 loss function will be used. /usr/local/lib/python3.6/dist-packages/keras/engine/saving.py:292: UserWarning: No training configuration found in save file: the model was *not* compiled. Compile it manually. warnings.warn('No training configuration found in save file: ' 2022-04-22 17:54:24,441 [INFO] iva.detectnet_v2.objectives.bbox_objective: Default L1 loss function will be used. __________________________________________________________________________________________________ Layer (type) Output Shape Param # Connected to ================================================================================================== input_1 (InputLayer) (None, 3, 544, 960) 0 __________________________________________________________________________________________________ conv1 (Conv2D) (None, 64, 272, 480) 9472 input_1[0][0] __________________________________________________________________________________________________ bn_conv1 (BatchNormalization) (None, 64, 272, 480) 256 conv1[0][0] __________________________________________________________________________________________________ activation_1 (Activation) (None, 64, 272, 480) 0 bn_conv1[0][0] __________________________________________________________________________________________________ block_1a_conv_1 (Conv2D) (None, 64, 136, 240) 36928 activation_1[0][0] __________________________________________________________________________________________________ block_1a_bn_1 (BatchNormalizati (None, 64, 136, 240) 256 block_1a_conv_1[0][0] __________________________________________________________________________________________________ block_1a_relu_1 (Activation) (None, 64, 136, 240) 0 block_1a_bn_1[0][0] __________________________________________________________________________________________________ block_1a_conv_2 (Conv2D) (None, 64, 136, 240) 36928 block_1a_relu_1[0][0] __________________________________________________________________________________________________ block_1a_conv_shortcut (Conv2D) (None, 64, 136, 240) 4160 activation_1[0][0] __________________________________________________________________________________________________ block_1a_bn_2 (BatchNormalizati (None, 64, 136, 240) 256 block_1a_conv_2[0][0] __________________________________________________________________________________________________ block_1a_bn_shortcut (BatchNorm (None, 64, 136, 240) 256 block_1a_conv_shortcut[0][0] __________________________________________________________________________________________________ add_1 (Add) (None, 64, 136, 240) 0 block_1a_bn_2[0][0] block_1a_bn_shortcut[0][0] __________________________________________________________________________________________________ block_1a_relu (Activation) (None, 64, 136, 240) 0 add_1[0][0] __________________________________________________________________________________________________ block_1b_conv_1 (Conv2D) (None, 64, 136, 240) 36928 block_1a_relu[0][0] __________________________________________________________________________________________________ block_1b_bn_1 (BatchNormalizati (None, 64, 136, 240) 256 block_1b_conv_1[0][0] __________________________________________________________________________________________________ block_1b_relu_1 (Activation) (None, 64, 136, 240) 0 block_1b_bn_1[0][0] __________________________________________________________________________________________________ block_1b_conv_2 (Conv2D) (None, 64, 136, 240) 36928 block_1b_relu_1[0][0] __________________________________________________________________________________________________ block_1b_bn_2 (BatchNormalizati (None, 64, 136, 240) 256 block_1b_conv_2[0][0] __________________________________________________________________________________________________ add_2 (Add) (None, 64, 136, 240) 0 block_1b_bn_2[0][0] block_1a_relu[0][0] __________________________________________________________________________________________________ block_1b_relu (Activation) (None, 64, 136, 240) 0 add_2[0][0] __________________________________________________________________________________________________ block_2a_conv_1 (Conv2D) (None, 128, 68, 120) 73856 block_1b_relu[0][0] __________________________________________________________________________________________________ block_2a_bn_1 (BatchNormalizati (None, 128, 68, 120) 512 block_2a_conv_1[0][0] __________________________________________________________________________________________________ block_2a_relu_1 (Activation) (None, 128, 68, 120) 0 block_2a_bn_1[0][0] __________________________________________________________________________________________________ block_2a_conv_2 (Conv2D) (None, 128, 68, 120) 147584 block_2a_relu_1[0][0] __________________________________________________________________________________________________ block_2a_conv_shortcut (Conv2D) (None, 128, 68, 120) 8320 block_1b_relu[0][0] __________________________________________________________________________________________________ block_2a_bn_2 (BatchNormalizati (None, 128, 68, 120) 512 block_2a_conv_2[0][0] __________________________________________________________________________________________________ block_2a_bn_shortcut (BatchNorm (None, 128, 68, 120) 512 block_2a_conv_shortcut[0][0] __________________________________________________________________________________________________ add_3 (Add) (None, 128, 68, 120) 0 block_2a_bn_2[0][0] block_2a_bn_shortcut[0][0] __________________________________________________________________________________________________ block_2a_relu (Activation) (None, 128, 68, 120) 0 add_3[0][0] __________________________________________________________________________________________________ block_2b_conv_1 (Conv2D) (None, 128, 68, 120) 147584 block_2a_relu[0][0] __________________________________________________________________________________________________ block_2b_bn_1 (BatchNormalizati (None, 128, 68, 120) 512 block_2b_conv_1[0][0] __________________________________________________________________________________________________ block_2b_relu_1 (Activation) (None, 128, 68, 120) 0 block_2b_bn_1[0][0] __________________________________________________________________________________________________ block_2b_conv_2 (Conv2D) (None, 128, 68, 120) 147584 block_2b_relu_1[0][0] __________________________________________________________________________________________________ block_2b_bn_2 (BatchNormalizati (None, 128, 68, 120) 512 block_2b_conv_2[0][0] __________________________________________________________________________________________________ add_4 (Add) (None, 128, 68, 120) 0 block_2b_bn_2[0][0] block_2a_relu[0][0] __________________________________________________________________________________________________ block_2b_relu (Activation) (None, 128, 68, 120) 0 add_4[0][0] __________________________________________________________________________________________________ block_3a_conv_1 (Conv2D) (None, 256, 34, 60) 295168 block_2b_relu[0][0] __________________________________________________________________________________________________ block_3a_bn_1 (BatchNormalizati (None, 256, 34, 60) 1024 block_3a_conv_1[0][0] __________________________________________________________________________________________________ block_3a_relu_1 (Activation) (None, 256, 34, 60) 0 block_3a_bn_1[0][0] __________________________________________________________________________________________________ block_3a_conv_2 (Conv2D) (None, 256, 34, 60) 590080 block_3a_relu_1[0][0] __________________________________________________________________________________________________ block_3a_conv_shortcut (Conv2D) (None, 256, 34, 60) 33024 block_2b_relu[0][0] __________________________________________________________________________________________________ block_3a_bn_2 (BatchNormalizati (None, 256, 34, 60) 1024 block_3a_conv_2[0][0] __________________________________________________________________________________________________ block_3a_bn_shortcut (BatchNorm (None, 256, 34, 60) 1024 block_3a_conv_shortcut[0][0] __________________________________________________________________________________________________ add_5 (Add) (None, 256, 34, 60) 0 block_3a_bn_2[0][0] block_3a_bn_shortcut[0][0] __________________________________________________________________________________________________ block_3a_relu (Activation) (None, 256, 34, 60) 0 add_5[0][0] __________________________________________________________________________________________________ block_3b_conv_1 (Conv2D) (None, 256, 34, 60) 590080 block_3a_relu[0][0] __________________________________________________________________________________________________ block_3b_bn_1 (BatchNormalizati (None, 256, 34, 60) 1024 block_3b_conv_1[0][0] __________________________________________________________________________________________________ block_3b_relu_1 (Activation) (None, 256, 34, 60) 0 block_3b_bn_1[0][0] __________________________________________________________________________________________________ block_3b_conv_2 (Conv2D) (None, 256, 34, 60) 590080 block_3b_relu_1[0][0] __________________________________________________________________________________________________ block_3b_bn_2 (BatchNormalizati (None, 256, 34, 60) 1024 block_3b_conv_2[0][0] __________________________________________________________________________________________________ add_6 (Add) (None, 256, 34, 60) 0 block_3b_bn_2[0][0] block_3a_relu[0][0] __________________________________________________________________________________________________ block_3b_relu (Activation) (None, 256, 34, 60) 0 add_6[0][0] __________________________________________________________________________________________________ block_4a_conv_1 (Conv2D) (None, 512, 34, 60) 1180160 block_3b_relu[0][0] __________________________________________________________________________________________________ block_4a_bn_1 (BatchNormalizati (None, 512, 34, 60) 2048 block_4a_conv_1[0][0] __________________________________________________________________________________________________ block_4a_relu_1 (Activation) (None, 512, 34, 60) 0 block_4a_bn_1[0][0] __________________________________________________________________________________________________ block_4a_conv_2 (Conv2D) (None, 512, 34, 60) 2359808 block_4a_relu_1[0][0] __________________________________________________________________________________________________ block_4a_conv_shortcut (Conv2D) (None, 512, 34, 60) 131584 block_3b_relu[0][0] __________________________________________________________________________________________________ block_4a_bn_2 (BatchNormalizati (None, 512, 34, 60) 2048 block_4a_conv_2[0][0] __________________________________________________________________________________________________ block_4a_bn_shortcut (BatchNorm (None, 512, 34, 60) 2048 block_4a_conv_shortcut[0][0] __________________________________________________________________________________________________ add_7 (Add) (None, 512, 34, 60) 0 block_4a_bn_2[0][0] block_4a_bn_shortcut[0][0] __________________________________________________________________________________________________ block_4a_relu (Activation) (None, 512, 34, 60) 0 add_7[0][0] __________________________________________________________________________________________________ block_4b_conv_1 (Conv2D) (None, 512, 34, 60) 2359808 block_4a_relu[0][0] __________________________________________________________________________________________________ block_4b_bn_1 (BatchNormalizati (None, 512, 34, 60) 2048 block_4b_conv_1[0][0] __________________________________________________________________________________________________ block_4b_relu_1 (Activation) (None, 512, 34, 60) 0 block_4b_bn_1[0][0] __________________________________________________________________________________________________ block_4b_conv_2 (Conv2D) (None, 512, 34, 60) 2359808 block_4b_relu_1[0][0] __________________________________________________________________________________________________ block_4b_bn_2 (BatchNormalizati (None, 512, 34, 60) 2048 block_4b_conv_2[0][0] __________________________________________________________________________________________________ add_8 (Add) (None, 512, 34, 60) 0 block_4b_bn_2[0][0] block_4a_relu[0][0] __________________________________________________________________________________________________ block_4b_relu (Activation) (None, 512, 34, 60) 0 add_8[0][0] __________________________________________________________________________________________________ output_bbox (Conv2D) (None, 4, 34, 60) 2052 block_4b_relu[0][0] __________________________________________________________________________________________________ output_cov (Conv2D) (None, 1, 34, 60) 513 block_4b_relu[0][0] ================================================================================================== Total params: 11,197,893 Trainable params: 11,026,821 Non-trainable params: 171,072 __________________________________________________________________________________________________ 2022-04-22 17:54:24,459 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Serial augmentation enabled = False 2022-04-22 17:54:24,459 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Pseudo sharding enabled = False 2022-04-22 17:54:24,459 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Max Image Dimensions (all sources): (0, 0) 2022-04-22 17:54:24,459 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: number of cpus: 12, io threads: 12, compute threads: 6, buffered batches: 4 2022-04-22 17:54:24,460 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: total dataset size 2400, number of sources: 1, batch size per gpu: 2, steps: 600 __________________________________________________________________________________________________ Layer (type) Output Shape Param # Connected to ================================================================================================== input_1 (InputLayer) (None, 3, 544, 960) 0 __________________________________________________________________________________________________ conv1 (Conv2D) (None, 64, 272, 480) 9472 input_1[0][0] __________________________________________________________________________________________________ bn_conv1 (BatchNormalization) (None, 64, 272, 480) 256 conv1[0][0] __________________________________________________________________________________________________ activation_1 (Activation) (None, 64, 272, 480) 0 bn_conv1[0][0] __________________________________________________________________________________________________ block_1a_conv_1 (Conv2D) (None, 64, 136, 240) 36928 activation_1[0][0] __________________________________________________________________________________________________ block_1a_bn_1 (BatchNormalizati (None, 64, 136, 240) 256 block_1a_conv_1[0][0] __________________________________________________________________________________________________ block_1a_relu_1 (Activation) (None, 64, 136, 240) 0 block_1a_bn_1[0][0] __________________________________________________________________________________________________ block_1a_conv_2 (Conv2D) (None, 64, 136, 240) 36928 block_1a_relu_1[0][0] __________________________________________________________________________________________________ block_1a_conv_shortcut (Conv2D) (None, 64, 136, 240) 4160 activation_1[0][0] __________________________________________________________________________________________________ block_1a_bn_2 (BatchNormalizati (None, 64, 136, 240) 256 block_1a_conv_2[0][0] __________________________________________________________________________________________________ block_1a_bn_shortcut (BatchNorm (None, 64, 136, 240) 256 block_1a_conv_shortcut[0][0] __________________________________________________________________________________________________ add_1 (Add) (None, 64, 136, 240) 0 block_1a_bn_2[0][0] block_1a_bn_shortcut[0][0] __________________________________________________________________________________________________ block_1a_relu (Activation) (None, 64, 136, 240) 0 add_1[0][0] __________________________________________________________________________________________________ block_1b_conv_1 (Conv2D) (None, 64, 136, 240) 36928 block_1a_relu[0][0] __________________________________________________________________________________________________ block_1b_bn_1 (BatchNormalizati (None, 64, 136, 240) 256 block_1b_conv_1[0][0] __________________________________________________________________________________________________ block_1b_relu_1 (Activation) (None, 64, 136, 240) 0 block_1b_bn_1[0][0] __________________________________________________________________________________________________ block_1b_conv_2 (Conv2D) (None, 64, 136, 240) 36928 block_1b_relu_1[0][0] __________________________________________________________________________________________________ block_1b_bn_2 (BatchNormalizati (None, 64, 136, 240) 256 block_1b_conv_2[0][0] __________________________________________________________________________________________________ add_2 (Add) (None, 64, 136, 240) 0 block_1b_bn_2[0][0] block_1a_relu[0][0] __________________________________________________________________________________________________ block_1b_relu (Activation) (None, 64, 136, 240) 0 add_2[0][0] __________________________________________________________________________________________________ block_2a_conv_1 (Conv2D) (None, 128, 68, 120) 73856 block_1b_relu[0][0] __________________________________________________________________________________________________ block_2a_bn_1 (BatchNormalizati (None, 128, 68, 120) 512 block_2a_conv_1[0][0] __________________________________________________________________________________________________ block_2a_relu_1 (Activation) (None, 128, 68, 120) 0 block_2a_bn_1[0][0] __________________________________________________________________________________________________ block_2a_conv_2 (Conv2D) (None, 128, 68, 120) 147584 block_2a_relu_1[0][0] __________________________________________________________________________________________________ block_2a_conv_shortcut (Conv2D) (None, 128, 68, 120) 8320 block_1b_relu[0][0] __________________________________________________________________________________________________ block_2a_bn_2 (BatchNormalizati (None, 128, 68, 120) 512 block_2a_conv_2[0][0] __________________________________________________________________________________________________ block_2a_bn_shortcut (BatchNorm (None, 128, 68, 120) 512 block_2a_conv_shortcut[0][0] __________________________________________________________________________________________________ add_3 (Add) (None, 128, 68, 120) 0 block_2a_bn_2[0][0] block_2a_bn_shortcut[0][0] __________________________________________________________________________________________________ block_2a_relu (Activation) (None, 128, 68, 120) 0 add_3[0][0] __________________________________________________________________________________________________ block_2b_conv_1 (Conv2D) (None, 128, 68, 120) 147584 block_2a_relu[0][0] __________________________________________________________________________________________________ block_2b_bn_1 (BatchNormalizati (None, 128, 68, 120) 512 block_2b_conv_1[0][0] __________________________________________________________________________________________________ block_2b_relu_1 (Activation) (None, 128, 68, 120) 0 block_2b_bn_1[0][0] __________________________________________________________________________________________________ block_2b_conv_2 (Conv2D) (None, 128, 68, 120) 147584 block_2b_relu_1[0][0] __________________________________________________________________________________________________ block_2b_bn_2 (BatchNormalizati (None, 128, 68, 120) 512 block_2b_conv_2[0][0] __________________________________________________________________________________________________ add_4 (Add) (None, 128, 68, 120) 0 block_2b_bn_2[0][0] block_2a_relu[0][0] __________________________________________________________________________________________________ block_2b_relu (Activation) (None, 128, 68, 120) 0 add_4[0][0] __________________________________________________________________________________________________ block_3a_conv_1 (Conv2D) (None, 256, 34, 60) 295168 block_2b_relu[0][0] __________________________________________________________________________________________________ block_3a_bn_1 (BatchNormalizati (None, 256, 34, 60) 1024 block_3a_conv_1[0][0] __________________________________________________________________________________________________ block_3a_relu_1 (Activation) (None, 256, 34, 60) 0 block_3a_bn_1[0][0] __________________________________________________________________________________________________ block_3a_conv_2 (Conv2D) (None, 256, 34, 60) 590080 block_3a_relu_1[0][0] __________________________________________________________________________________________________ block_3a_conv_shortcut (Conv2D) (None, 256, 34, 60) 33024 block_2b_relu[0][0] __________________________________________________________________________________________________ block_3a_bn_2 (BatchNormalizati (None, 256, 34, 60) 1024 block_3a_conv_2[0][0] __________________________________________________________________________________________________ block_3a_bn_shortcut (BatchNorm (None, 256, 34, 60) 1024 block_3a_conv_shortcut[0][0] __________________________________________________________________________________________________ add_5 (Add) (None, 256, 34, 60) 0 block_3a_bn_2[0][0] block_3a_bn_shortcut[0][0] __________________________________________________________________________________________________ block_3a_relu (Activation) (None, 256, 34, 60) 0 add_5[0][0] __________________________________________________________________________________________________ block_3b_conv_1 (Conv2D) (None, 256, 34, 60) 590080 block_3a_relu[0][0] __________________________________________________________________________________________________ block_3b_bn_1 (BatchNormalizati (None, 256, 34, 60) 1024 block_3b_conv_1[0][0] __________________________________________________________________________________________________ block_3b_relu_1 (Activation) (None, 256, 34, 60) 0 block_3b_bn_1[0][0] __________________________________________________________________________________________________ block_3b_conv_2 (Conv2D) (None, 256, 34, 60) 590080 block_3b_relu_1[0][0] __________________________________________________________________________________________________ block_3b_bn_2 (BatchNormalizati (None, 256, 34, 60) 1024 block_3b_conv_2[0][0] __________________________________________________________________________________________________ add_6 (Add) (None, 256, 34, 60) 0 block_3b_bn_2[0][0] block_3a_relu[0][0] __________________________________________________________________________________________________ block_3b_relu (Activation) (None, 256, 34, 60) 0 add_6[0][0] __________________________________________________________________________________________________ block_4a_conv_1 (Conv2D) (None, 512, 34, 60) 1180160 block_3b_relu[0][0] __________________________________________________________________________________________________ block_4a_bn_1 (BatchNormalizati (None, 512, 34, 60) 2048 block_4a_conv_1[0][0] __________________________________________________________________________________________________ block_4a_relu_1 (Activation) (None, 512, 34, 60) 0 block_4a_bn_1[0][0] __________________________________________________________________________________________________ block_4a_conv_2 (Conv2D) (None, 512, 34, 60) 2359808 block_4a_relu_1[0][0] __________________________________________________________________________________________________ block_4a_conv_shortcut (Conv2D) (None, 512, 34, 60) 131584 block_3b_relu[0][0] __________________________________________________________________________________________________ block_4a_bn_2 (BatchNormalizati (None, 512, 34, 60) 2048 block_4a_conv_2[0][0] __________________________________________________________________________________________________ block_4a_bn_shortcut (BatchNorm (None, 512, 34, 60) 2048 block_4a_conv_shortcut[0][0] __________________________________________________________________________________________________ add_7 (Add) (None, 512, 34, 60) 0 block_4a_bn_2[0][0] block_4a_bn_shortcut[0][0] __________________________________________________________________________________________________ block_4a_relu (Activation) (None, 512, 34, 60) 0 add_7[0][0] __________________________________________________________________________________________________ block_4b_conv_1 (Conv2D) (None, 512, 34, 60) 2359808 block_4a_relu[0][0] __________________________________________________________________________________________________ block_4b_bn_1 (BatchNormalizati (None, 512, 34, 60) 2048 block_4b_conv_1[0][0] __________________________________________________________________________________________________ block_4b_relu_1 (Activation) (None, 512, 34, 60) 0 block_4b_bn_1[0][0] __________________________________________________________________________________________________ block_4b_conv_2 (Conv2D) (None, 512, 34, 60) 2359808 block_4b_relu_1[0][0] __________________________________________________________________________________________________ block_4b_bn_2 (BatchNormalizati (None, 512, 34, 60) 2048 block_4b_conv_2[0][0] __________________________________________________________________________________________________ add_8 (Add) (None, 512, 34, 60) 0 block_4b_bn_2[0][0] block_4a_relu[0][0] __________________________________________________________________________________________________ block_4b_relu (Activation) (None, 512, 34, 60) 0 add_8[0][0] __________________________________________________________________________________________________ output_bbox (Conv2D) (None, 4, 34, 60) 2052 block_4b_relu[0][0] __________________________________________________________________________________________________ output_cov (Conv2D) (None, 1, 34, 60) 513 block_4b_relu[0][0] ================================================================================================== Total params: 11,197,893 Trainable params: 11,026,821 Non-trainable params: 171,072 __________________________________________________________________________________________________ 2022-04-22 17:54:24,474 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Serial augmentation enabled = False 2022-04-22 17:54:24,474 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Pseudo sharding enabled = False 2022-04-22 17:54:24,474 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Max Image Dimensions (all sources): (0, 0) 2022-04-22 17:54:24,474 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: number of cpus: 12, io threads: 12, compute threads: 6, buffered batches: 4 2022-04-22 17:54:24,474 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: total dataset size 2400, number of sources: 1, batch size per gpu: 2, steps: 600 WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead. 2022-04-22 17:54:24,529 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead. 2022-04-22 17:54:24,529 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead. WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2022-04-22 17:54:24,575 [WARNING] tensorflow: Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2022-04-22 17:54:24,575 [WARNING] tensorflow: Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2022-04-22 17:54:24,592 [INFO] iva.detectnet_v2.dataloader.default_dataloader: Bounding box coordinates were detected in the input specification! Bboxes will be automatically converted to polygon coordinates. 2022-04-22 17:54:24,592 [INFO] iva.detectnet_v2.dataloader.default_dataloader: Bounding box coordinates were detected in the input specification! Bboxes will be automatically converted to polygon coordinates. 2022-04-22 17:54:24,841 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: shuffle: True - shard 1 of 2 2022-04-22 17:54:24,846 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: sampling 1 datasets with weights: 2022-04-22 17:54:24,846 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: source: 0 weight: 1.000000 2022-04-22 17:54:24,847 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: shuffle: True - shard 0 of 2 2022-04-22 17:54:24,852 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: sampling 1 datasets with weights: 2022-04-22 17:54:24,852 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: source: 0 weight: 1.000000 WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2022-04-22 17:54:24,858 [WARNING] tensorflow: Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2022-04-22 17:54:24,864 [WARNING] tensorflow: Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2022-04-22 17:54:25,193 [INFO] __main__: Found 2400 samples in training set 2022-04-22 17:54:25,198 [INFO] __main__: Found 2400 samples in training set WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/rasterizers/bbox_rasterizer.py:347: The name tf.bincount is deprecated. Please use tf.math.bincount instead. 2022-04-22 17:54:25,284 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/rasterizers/bbox_rasterizer.py:347: The name tf.bincount is deprecated. Please use tf.math.bincount instead. WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/rasterizers/bbox_rasterizer.py:347: The name tf.bincount is deprecated. Please use tf.math.bincount instead. 2022-04-22 17:54:25,292 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/rasterizers/bbox_rasterizer.py:347: The name tf.bincount is deprecated. Please use tf.math.bincount instead. WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/training_proto_utilities.py:89: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead. 2022-04-22 17:54:25,380 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/training_proto_utilities.py:89: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead. WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/training_proto_utilities.py:89: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead. 2022-04-22 17:54:25,391 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/training_proto_utilities.py:89: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead. WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/training_proto_utilities.py:36: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead. 2022-04-22 17:54:25,394 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/training_proto_utilities.py:36: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead. WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/training_proto_utilities.py:36: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead. 2022-04-22 17:54:25,405 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/training_proto_utilities.py:36: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead. WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_functions.py:17: The name tf.log is deprecated. Please use tf.math.log instead. 2022-04-22 17:54:25,528 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_functions.py:17: The name tf.log is deprecated. Please use tf.math.log instead. WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:235: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead. 2022-04-22 17:54:25,537 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:235: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead. WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/model/detectnet_model.py:591: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead. 2022-04-22 17:54:25,540 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/model/detectnet_model.py:591: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead. WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_functions.py:17: The name tf.log is deprecated. Please use tf.math.log instead. 2022-04-22 17:54:25,545 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_functions.py:17: The name tf.log is deprecated. Please use tf.math.log instead. WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:235: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead. 2022-04-22 17:54:25,553 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:235: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead. WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/model/detectnet_model.py:591: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead. 2022-04-22 17:54:25,556 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/model/detectnet_model.py:591: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead. 2022-04-22 17:54:26,656 [INFO] __main__: Checkpoint interval: 10 WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/scripts/train.py:109: The name tf.train.Scaffold is deprecated. Please use tf.compat.v1.train.Scaffold instead. 2022-04-22 17:54:26,656 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/scripts/train.py:109: The name tf.train.Scaffold is deprecated. Please use tf.compat.v1.train.Scaffold instead. WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:14: The name tf.local_variables_initializer is deprecated. Please use tf.compat.v1.local_variables_initializer instead. 2022-04-22 17:54:26,656 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:14: The name tf.local_variables_initializer is deprecated. Please use tf.compat.v1.local_variables_initializer instead. WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:15: The name tf.tables_initializer is deprecated. Please use tf.compat.v1.tables_initializer instead. 2022-04-22 17:54:26,656 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:15: The name tf.tables_initializer is deprecated. Please use tf.compat.v1.tables_initializer instead. WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:16: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead. 2022-04-22 17:54:26,657 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:16: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead. WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/utils.py:59: The name tf.train.LoggingTensorHook is deprecated. Please use tf.estimator.LoggingTensorHook instead. 2022-04-22 17:54:26,658 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/utils.py:59: The name tf.train.LoggingTensorHook is deprecated. Please use tf.estimator.LoggingTensorHook instead. WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/utils.py:60: The name tf.train.StopAtStepHook is deprecated. Please use tf.estimator.StopAtStepHook instead. 2022-04-22 17:54:26,658 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/utils.py:60: The name tf.train.StopAtStepHook is deprecated. Please use tf.estimator.StopAtStepHook instead. WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/utilities.py:140: The name tf.train.SingularMonitoredSession is deprecated. Please use tf.compat.v1.train.SingularMonitoredSession instead. 2022-04-22 17:54:26,659 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/utilities.py:140: The name tf.train.SingularMonitoredSession is deprecated. Please use tf.compat.v1.train.SingularMonitoredSession instead. 2022-04-22 17:54:26,771 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Serial augmentation enabled = False 2022-04-22 17:54:26,771 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Pseudo sharding enabled = False 2022-04-22 17:54:26,771 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Max Image Dimensions (all sources): (0, 0) 2022-04-22 17:54:26,771 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: number of cpus: 12, io threads: 24, compute threads: 12, buffered batches: 4 2022-04-22 17:54:26,771 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: total dataset size 599, number of sources: 1, batch size per gpu: 2, steps: 300 WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2022-04-22 17:54:26,780 [WARNING] tensorflow: Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2022-04-22 17:54:26,796 [INFO] iva.detectnet_v2.dataloader.default_dataloader: Bounding box coordinates were detected in the input specification! Bboxes will be automatically converted to polygon coordinates. 2022-04-22 17:54:27,077 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: shuffle: False - shard 0 of 1 2022-04-22 17:54:27,082 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: sampling 1 datasets with weights: 2022-04-22 17:54:27,082 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: source: 0 weight: 1.000000 WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2022-04-22 17:54:27,094 [WARNING] tensorflow: Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2022-04-22 17:54:27,304 [INFO] __main__: Found 599 samples in validation set INFO:tensorflow:Graph was finalized. 2022-04-22 17:54:27,730 [INFO] tensorflow: Graph was finalized. WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/validation_hook.py:40: The name tf.summary.FileWriterCache is deprecated. Please use tf.compat.v1.summary.FileWriterCache instead. 2022-04-22 17:54:27,756 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/validation_hook.py:40: The name tf.summary.FileWriterCache is deprecated. Please use tf.compat.v1.summary.FileWriterCache instead. 2022-04-22 17:54:28,695 [INFO] __main__: Checkpoint interval: 10 WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/scripts/train.py:109: The name tf.train.Scaffold is deprecated. Please use tf.compat.v1.train.Scaffold instead. 2022-04-22 17:54:28,696 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/scripts/train.py:109: The name tf.train.Scaffold is deprecated. Please use tf.compat.v1.train.Scaffold instead. WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:14: The name tf.local_variables_initializer is deprecated. Please use tf.compat.v1.local_variables_initializer instead. 2022-04-22 17:54:28,696 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:14: The name tf.local_variables_initializer is deprecated. Please use tf.compat.v1.local_variables_initializer instead. WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:15: The name tf.tables_initializer is deprecated. Please use tf.compat.v1.tables_initializer instead. 2022-04-22 17:54:28,696 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:15: The name tf.tables_initializer is deprecated. Please use tf.compat.v1.tables_initializer instead. WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:16: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead. 2022-04-22 17:54:28,697 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:16: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead. WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/utils.py:59: The name tf.train.LoggingTensorHook is deprecated. Please use tf.estimator.LoggingTensorHook instead. 2022-04-22 17:54:28,698 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/utils.py:59: The name tf.train.LoggingTensorHook is deprecated. Please use tf.estimator.LoggingTensorHook instead. WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/utils.py:60: The name tf.train.StopAtStepHook is deprecated. Please use tf.estimator.StopAtStepHook instead. 2022-04-22 17:54:28,698 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/utils.py:60: The name tf.train.StopAtStepHook is deprecated. Please use tf.estimator.StopAtStepHook instead. WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/utils.py:73: The name tf.train.StepCounterHook is deprecated. Please use tf.estimator.StepCounterHook instead. 2022-04-22 17:54:28,698 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/utils.py:73: The name tf.train.StepCounterHook is deprecated. Please use tf.estimator.StepCounterHook instead. INFO:tensorflow:Create CheckpointSaverHook. 2022-04-22 17:54:28,699 [INFO] tensorflow: Create CheckpointSaverHook. WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/utils.py:99: The name tf.train.SummarySaverHook is deprecated. Please use tf.estimator.SummarySaverHook instead. 2022-04-22 17:54:28,699 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/utils.py:99: The name tf.train.SummarySaverHook is deprecated. Please use tf.estimator.SummarySaverHook instead. WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/utilities.py:140: The name tf.train.SingularMonitoredSession is deprecated. Please use tf.compat.v1.train.SingularMonitoredSession instead. 2022-04-22 17:54:28,699 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/utilities.py:140: The name tf.train.SingularMonitoredSession is deprecated. Please use tf.compat.v1.train.SingularMonitoredSession instead. INFO:tensorflow:Running local_init_op. 2022-04-22 17:54:29,038 [INFO] tensorflow: Running local_init_op. INFO:tensorflow:Done running local_init_op. 2022-04-22 17:54:29,355 [INFO] tensorflow: Done running local_init_op. INFO:tensorflow:Graph was finalized. 2022-04-22 17:54:29,735 [INFO] tensorflow: Graph was finalized. INFO:tensorflow:Running local_init_op. 2022-04-22 17:54:31,188 [INFO] tensorflow: Running local_init_op. INFO:tensorflow:Done running local_init_op. 2022-04-22 17:54:31,680 [INFO] tensorflow: Done running local_init_op. INFO:tensorflow:Saving checkpoints for step-0. 2022-04-22 17:54:38,080 [INFO] tensorflow: Saving checkpoints for step-0. fc30edcc1e06:56:109 [0] NCCL INFO Bootstrap : Using lo:127.0.0.1<0> fc30edcc1e06:56:109 [0] NCCL INFO NET/Plugin : Plugin load returned 0 : libnccl-net.so: cannot open shared object file: No such file or directory. fc30edcc1e06:56:109 [0] NCCL INFO NET/IB : No device found. fc30edcc1e06:56:109 [0] NCCL INFO NET/Socket : Using [0]lo:127.0.0.1<0> [1]eth0:172.17.0.2<0> fc30edcc1e06:56:109 [0] NCCL INFO Using network Socket NCCL version 2.9.9+cuda11.3 fc30edcc1e06:57:108 [1] NCCL INFO Bootstrap : Using lo:127.0.0.1<0> fc30edcc1e06:57:108 [1] NCCL INFO NET/Plugin : Plugin load returned 0 : libnccl-net.so: cannot open shared object file: No such file or directory. fc30edcc1e06:57:108 [1] NCCL INFO NET/IB : No device found. fc30edcc1e06:57:108 [1] NCCL INFO NET/Socket : Using [0]lo:127.0.0.1<0> [1]eth0:172.17.0.2<0> fc30edcc1e06:57:108 [1] NCCL INFO Using network Socket fc30edcc1e06:56:109 [0] NCCL INFO Channel 00/02 : 0 1 fc30edcc1e06:56:109 [0] NCCL INFO Channel 01/02 : 0 1 fc30edcc1e06:56:109 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 fc30edcc1e06:57:108 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 fc30edcc1e06:56:109 [0] NCCL INFO Channel 00 : 0[1000] -> 1[2000] via direct shared memory fc30edcc1e06:57:108 [1] NCCL INFO Channel 00 : 1[2000] -> 0[1000] via direct shared memory fc30edcc1e06:57:108 [1] NCCL INFO Channel 01 : 1[2000] -> 0[1000] via direct shared memory fc30edcc1e06:56:109 [0] NCCL INFO Channel 01 : 0[1000] -> 1[2000] via direct shared memory fc30edcc1e06:56:109 [0] NCCL INFO Connected all rings fc30edcc1e06:56:109 [0] NCCL INFO Connected all trees fc30edcc1e06:56:109 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 fc30edcc1e06:56:109 [0] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer fc30edcc1e06:57:108 [1] NCCL INFO Connected all rings fc30edcc1e06:57:108 [1] NCCL INFO Connected all trees fc30edcc1e06:57:108 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 fc30edcc1e06:57:108 [1] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer fc30edcc1e06:57:108 [1] NCCL INFO comm 0x7f5d00336000 rank 1 nranks 2 cudaDev 1 busId 2000 - Init COMPLETE fc30edcc1e06:56:109 [0] NCCL INFO comm 0x7faa08367000 rank 0 nranks 2 cudaDev 0 busId 1000 - Init COMPLETE fc30edcc1e06:56:109 [0] NCCL INFO Launch mode Parallel INFO:tensorflow:epoch = 0.0, learning_rate = 4.9999994e-06, loss = 0.04578698, step = 0 2022-04-22 17:54:54,512 [INFO] tensorflow: epoch = 0.0, learning_rate = 4.9999994e-06, loss = 0.04578698, step = 0 INFO:tensorflow:epoch = 0.0, learning_rate = 4.9999994e-06, loss = 0.04547847, step = 0 2022-04-22 17:54:54,512 [INFO] tensorflow: epoch = 0.0, learning_rate = 4.9999994e-06, loss = 0.04547847, step = 0 2022-04-22 17:54:54,513 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 0/100: loss: 0.04548 learning rate: 0.00000 Time taken: 0:00:00 ETA: 0:00:00 2022-04-22 17:54:54,513 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 1.254 2022-04-22 17:54:57,540 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 16.088 2022-04-22 17:54:59,363 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.889 INFO:tensorflow:epoch = 0.08833333333333333, learning_rate = 5.207588e-06, loss = 0.018597819, step = 53 (5.164 sec) 2022-04-22 17:54:59,676 [INFO] tensorflow: epoch = 0.08833333333333333, learning_rate = 5.207588e-06, loss = 0.018597819, step = 53 (5.164 sec) INFO:tensorflow:epoch = 0.08833333333333333, learning_rate = 5.207588e-06, loss = 0.01811045, step = 53 (5.165 sec) 2022-04-22 17:54:59,676 [INFO] tensorflow: epoch = 0.08833333333333333, learning_rate = 5.207588e-06, loss = 0.01811045, step = 53 (5.165 sec) INFO:tensorflow:global_step/sec: 10.6677 2022-04-22 17:55:00,137 [INFO] tensorflow: global_step/sec: 10.6677 2022-04-22 17:55:01,244 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.152 2022-04-22 17:55:03,012 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.580 INFO:tensorflow:global_step/sec: 13.1923 2022-04-22 17:55:04,685 [INFO] tensorflow: global_step/sec: 13.1923 INFO:tensorflow:epoch = 0.2016666666666667, learning_rate = 5.4865986e-06, loss = 0.0076788026, step = 121 (5.085 sec) 2022-04-22 17:55:04,760 [INFO] tensorflow: epoch = 0.2016666666666667, learning_rate = 5.4865986e-06, loss = 0.0076788026, step = 121 (5.085 sec) INFO:tensorflow:epoch = 0.2016666666666667, learning_rate = 5.4865986e-06, loss = 0.007791846, step = 121 (5.084 sec) 2022-04-22 17:55:04,760 [INFO] tensorflow: epoch = 0.2016666666666667, learning_rate = 5.4865986e-06, loss = 0.007791846, step = 121 (5.084 sec) 2022-04-22 17:55:04,978 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 50.868 2022-04-22 17:55:06,765 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.975 2022-04-22 17:55:08,617 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.996 INFO:tensorflow:global_step/sec: 13.843 2022-04-22 17:55:09,019 [INFO] tensorflow: global_step/sec: 13.843 INFO:tensorflow:epoch = 0.32, learning_rate = 5.7938864e-06, loss = 0.0048042987, step = 192 (5.081 sec) INFO:tensorflow:epoch = 0.32, learning_rate = 5.7938864e-06, loss = 0.0047421325, step = 192 (5.081 sec) 2022-04-22 17:55:09,841 [INFO] tensorflow: epoch = 0.32, learning_rate = 5.7938864e-06, loss = 0.0048042987, step = 192 (5.081 sec) 2022-04-22 17:55:09,841 [INFO] tensorflow: epoch = 0.32, learning_rate = 5.7938864e-06, loss = 0.0047421325, step = 192 (5.081 sec) 2022-04-22 17:55:10,351 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 57.655 2022-04-22 17:55:12,112 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.791 INFO:tensorflow:global_step/sec: 14.1711 2022-04-22 17:55:13,253 [INFO] tensorflow: global_step/sec: 14.1711 2022-04-22 17:55:13,863 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 57.134 INFO:tensorflow:epoch = 0.44, learning_rate = 6.123078e-06, loss = 0.0033901392, step = 264 (5.109 sec) 2022-04-22 17:55:14,950 [INFO] tensorflow: epoch = 0.44, learning_rate = 6.123078e-06, loss = 0.0033901392, step = 264 (5.109 sec) INFO:tensorflow:epoch = 0.44, learning_rate = 6.123078e-06, loss = 0.0034210077, step = 264 (5.109 sec) 2022-04-22 17:55:14,950 [INFO] tensorflow: epoch = 0.44, learning_rate = 6.123078e-06, loss = 0.0034210077, step = 264 (5.109 sec) 2022-04-22 17:55:15,633 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.490 2022-04-22 17:55:17,334 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 58.807 INFO:tensorflow:global_step/sec: 14.3895 2022-04-22 17:55:17,423 [INFO] tensorflow: global_step/sec: 14.3895 2022-04-22 17:55:19,122 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.926 INFO:tensorflow:epoch = 0.5633333333333334, learning_rate = 6.4809165e-06, loss = 0.0026806737, step = 338 (5.191 sec) 2022-04-22 17:55:20,141 [INFO] tensorflow: epoch = 0.5633333333333334, learning_rate = 6.4809165e-06, loss = 0.0026806737, step = 338 (5.191 sec) INFO:tensorflow:epoch = 0.5633333333333334, learning_rate = 6.4809165e-06, loss = 0.0026636806, step = 338 (5.191 sec) 2022-04-22 17:55:20,141 [INFO] tensorflow: epoch = 0.5633333333333334, learning_rate = 6.4809165e-06, loss = 0.0026636806, step = 338 (5.191 sec) 2022-04-22 17:55:21,044 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 52.034 INFO:tensorflow:global_step/sec: 13.6983 2022-04-22 17:55:21,803 [INFO] tensorflow: global_step/sec: 13.6983 2022-04-22 17:55:22,793 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 57.197 2022-04-22 17:55:24,604 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.206 INFO:tensorflow:epoch = 0.68, learning_rate = 6.8386416e-06, loss = 0.0023329454, step = 408 (5.089 sec) 2022-04-22 17:55:25,231 [INFO] tensorflow: epoch = 0.68, learning_rate = 6.8386416e-06, loss = 0.0023329454, step = 408 (5.089 sec) INFO:tensorflow:epoch = 0.68, learning_rate = 6.8386416e-06, loss = 0.002297821, step = 408 (5.089 sec) 2022-04-22 17:55:25,231 [INFO] tensorflow: epoch = 0.68, learning_rate = 6.8386416e-06, loss = 0.002297821, step = 408 (5.089 sec) INFO:tensorflow:global_step/sec: 13.8296 2022-04-22 17:55:26,142 [INFO] tensorflow: global_step/sec: 13.8296 2022-04-22 17:55:26,428 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.833 2022-04-22 17:55:28,270 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.313 2022-04-22 17:55:30,048 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.239 INFO:tensorflow:epoch = 0.7966666666666667, learning_rate = 7.2161115e-06, loss = 0.0021282993, step = 478 (5.105 sec) INFO:tensorflow:epoch = 0.7966666666666667, learning_rate = 7.2161115e-06, loss = 0.0020417802, step = 478 (5.105 sec) 2022-04-22 17:55:30,336 [INFO] tensorflow: epoch = 0.7966666666666667, learning_rate = 7.2161115e-06, loss = 0.0021282993, step = 478 (5.105 sec) 2022-04-22 17:55:30,336 [INFO] tensorflow: epoch = 0.7966666666666667, learning_rate = 7.2161115e-06, loss = 0.0020417802, step = 478 (5.105 sec) INFO:tensorflow:global_step/sec: 13.8755 2022-04-22 17:55:30,466 [INFO] tensorflow: global_step/sec: 13.8755 2022-04-22 17:55:31,817 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.529 2022-04-22 17:55:33,575 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.890 INFO:tensorflow:global_step/sec: 13.8309 2022-04-22 17:55:34,804 [INFO] tensorflow: global_step/sec: 13.8309 INFO:tensorflow:epoch = 0.9133333333333334, learning_rate = 7.614417e-06, loss = 0.0018395389, step = 548 (5.114 sec) INFO:tensorflow:epoch = 0.9133333333333334, learning_rate = 7.614417e-06, loss = 0.0018341392, step = 548 (5.114 sec) 2022-04-22 17:55:35,450 [INFO] tensorflow: epoch = 0.9133333333333334, learning_rate = 7.614417e-06, loss = 0.0018395389, step = 548 (5.114 sec) 2022-04-22 17:55:35,450 [INFO] tensorflow: epoch = 0.9133333333333334, learning_rate = 7.614417e-06, loss = 0.0018341392, step = 548 (5.114 sec) 2022-04-22 17:55:35,517 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 51.508 2022-04-22 17:55:37,293 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.310 2022-04-22 17:55:39,371 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 299, 0.00s/step 2022-04-22 17:55:42,546 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 299, 0.32s/step 2022-04-22 17:55:45,051 [INFO] iva.detectnet_v2.evaluation.evaluation: step 20 / 299, 0.25s/step 2022-04-22 17:55:47,600 [INFO] iva.detectnet_v2.evaluation.evaluation: step 30 / 299, 0.25s/step 2022-04-22 17:55:50,073 [INFO] iva.detectnet_v2.evaluation.evaluation: step 40 / 299, 0.25s/step 2022-04-22 17:55:52,608 [INFO] iva.detectnet_v2.evaluation.evaluation: step 50 / 299, 0.25s/step 2022-04-22 17:55:55,124 [INFO] iva.detectnet_v2.evaluation.evaluation: step 60 / 299, 0.25s/step 2022-04-22 17:55:57,597 [INFO] iva.detectnet_v2.evaluation.evaluation: step 70 / 299, 0.25s/step 2022-04-22 17:56:00,117 [INFO] iva.detectnet_v2.evaluation.evaluation: step 80 / 299, 0.25s/step 2022-04-22 17:56:02,604 [INFO] iva.detectnet_v2.evaluation.evaluation: step 90 / 299, 0.25s/step 2022-04-22 17:56:05,131 [INFO] iva.detectnet_v2.evaluation.evaluation: step 100 / 299, 0.25s/step 2022-04-22 17:56:07,693 [INFO] iva.detectnet_v2.evaluation.evaluation: step 110 / 299, 0.26s/step 2022-04-22 17:56:10,286 [INFO] iva.detectnet_v2.evaluation.evaluation: step 120 / 299, 0.26s/step 2022-04-22 17:56:12,880 [INFO] iva.detectnet_v2.evaluation.evaluation: step 130 / 299, 0.26s/step 2022-04-22 17:56:15,485 [INFO] iva.detectnet_v2.evaluation.evaluation: step 140 / 299, 0.26s/step 2022-04-22 17:56:18,072 [INFO] iva.detectnet_v2.evaluation.evaluation: step 150 / 299, 0.26s/step 2022-04-22 17:56:20,654 [INFO] iva.detectnet_v2.evaluation.evaluation: step 160 / 299, 0.26s/step 2022-04-22 17:56:23,225 [INFO] iva.detectnet_v2.evaluation.evaluation: step 170 / 299, 0.26s/step 2022-04-22 17:56:25,824 [INFO] iva.detectnet_v2.evaluation.evaluation: step 180 / 299, 0.26s/step 2022-04-22 17:56:28,415 [INFO] iva.detectnet_v2.evaluation.evaluation: step 190 / 299, 0.26s/step 2022-04-22 17:56:31,019 [INFO] iva.detectnet_v2.evaluation.evaluation: step 200 / 299, 0.26s/step 2022-04-22 17:56:33,592 [INFO] iva.detectnet_v2.evaluation.evaluation: step 210 / 299, 0.26s/step 2022-04-22 17:56:36,246 [INFO] iva.detectnet_v2.evaluation.evaluation: step 220 / 299, 0.27s/step 2022-04-22 17:56:38,790 [INFO] iva.detectnet_v2.evaluation.evaluation: step 230 / 299, 0.25s/step 2022-04-22 17:56:41,324 [INFO] iva.detectnet_v2.evaluation.evaluation: step 240 / 299, 0.25s/step 2022-04-22 17:56:43,937 [INFO] iva.detectnet_v2.evaluation.evaluation: step 250 / 299, 0.26s/step 2022-04-22 17:56:46,549 [INFO] iva.detectnet_v2.evaluation.evaluation: step 260 / 299, 0.26s/step 2022-04-22 17:56:49,184 [INFO] iva.detectnet_v2.evaluation.evaluation: step 270 / 299, 0.26s/step 2022-04-22 17:56:51,870 [INFO] iva.detectnet_v2.evaluation.evaluation: step 280 / 299, 0.27s/step 2022-04-22 17:56:54,599 [INFO] iva.detectnet_v2.evaluation.evaluation: step 290 / 299, 0.27s/step Matching predictions to ground truth, class 1/1.: 100% 345/345 [00:00<00:00, 13044.11it/s] Epoch 1/100 ========================= Validation cost: 0.018684 Mean average_precision (in %): 0.0000 class name average precision (in %) ------------ -------------------------- face 0 Median Inference Time: 0.011706 2022-04-22 17:56:57,099 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 1.253 INFO:tensorflow:epoch = 1.0, learning_rate = 7.924464e-06, loss = 0.01827196, step = 600 (81.721 sec) 2022-04-22 17:56:57,171 [INFO] tensorflow: epoch = 1.0, learning_rate = 7.924464e-06, loss = 0.01827196, step = 600 (81.721 sec) INFO:tensorflow:global_step/sec: 0.728442 2022-04-22 17:56:57,172 [INFO] tensorflow: global_step/sec: 0.728442 2022-04-22 17:56:57,173 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 1/100: loss: 0.01827 learning rate: 0.00001 Time taken: 0:02:05.773971 ETA: 3:27:31.623161 INFO:tensorflow:epoch = 1.0016666666666667, learning_rate = 7.93055e-06, loss = 0.017660026, step = 601 (81.792 sec) 2022-04-22 17:56:57,242 [INFO] tensorflow: epoch = 1.0016666666666667, learning_rate = 7.93055e-06, loss = 0.017660026, step = 601 (81.792 sec) 2022-04-22 17:56:58,898 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.583 2022-04-22 17:57:00,697 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.593 INFO:tensorflow:global_step/sec: 14.0169 2022-04-22 17:57:01,452 [INFO] tensorflow: global_step/sec: 14.0169 INFO:tensorflow:epoch = 1.12, learning_rate = 8.374716e-06, loss = 0.01070704, step = 672 (5.138 sec) 2022-04-22 17:57:02,309 [INFO] tensorflow: epoch = 1.12, learning_rate = 8.374716e-06, loss = 0.01070704, step = 672 (5.138 sec) INFO:tensorflow:epoch = 1.1216666666666668, learning_rate = 8.38114e-06, loss = 0.011095502, step = 673 (5.136 sec) 2022-04-22 17:57:02,378 [INFO] tensorflow: epoch = 1.1216666666666668, learning_rate = 8.38114e-06, loss = 0.011095502, step = 673 (5.136 sec) 2022-04-22 17:57:02,455 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.909 2022-04-22 17:57:04,335 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.207 INFO:tensorflow:global_step/sec: 13.7187 2022-04-22 17:57:05,826 [INFO] tensorflow: global_step/sec: 13.7187 2022-04-22 17:57:06,136 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.529 INFO:tensorflow:epoch = 1.235, learning_rate = 8.830191e-06, loss = 0.008896679, step = 741 (5.074 sec) 2022-04-22 17:57:07,383 [INFO] tensorflow: epoch = 1.235, learning_rate = 8.830191e-06, loss = 0.008896679, step = 741 (5.074 sec) INFO:tensorflow:epoch = 1.2366666666666668, learning_rate = 8.836964e-06, loss = 0.008461489, step = 742 (5.073 sec) 2022-04-22 17:57:07,451 [INFO] tensorflow: epoch = 1.2366666666666668, learning_rate = 8.836964e-06, loss = 0.008461489, step = 742 (5.073 sec) 2022-04-22 17:57:07,954 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.992 2022-04-22 17:57:09,783 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.680 INFO:tensorflow:global_step/sec: 13.7188 2022-04-22 17:57:10,199 [INFO] tensorflow: global_step/sec: 13.7188 2022-04-22 17:57:11,569 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.002 INFO:tensorflow:epoch = 1.3533333333333335, learning_rate = 9.324735e-06, loss = 0.0071027116, step = 812 (5.108 sec) 2022-04-22 17:57:12,491 [INFO] tensorflow: epoch = 1.3533333333333335, learning_rate = 9.324735e-06, loss = 0.0071027116, step = 812 (5.108 sec) INFO:tensorflow:epoch = 1.355, learning_rate = 9.331897e-06, loss = 0.0073552253, step = 813 (5.114 sec) 2022-04-22 17:57:12,565 [INFO] tensorflow: epoch = 1.355, learning_rate = 9.331897e-06, loss = 0.0073552253, step = 813 (5.114 sec) 2022-04-22 17:57:13,386 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.034 INFO:tensorflow:global_step/sec: 13.5188 2022-04-22 17:57:14,638 [INFO] tensorflow: global_step/sec: 13.5188 2022-04-22 17:57:15,397 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 49.742 2022-04-22 17:57:17,251 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.947 INFO:tensorflow:epoch = 1.465, learning_rate = 9.816804e-06, loss = 0.006249166, step = 879 (5.122 sec) 2022-04-22 17:57:17,613 [INFO] tensorflow: epoch = 1.465, learning_rate = 9.816804e-06, loss = 0.006249166, step = 879 (5.122 sec) INFO:tensorflow:epoch = 1.4666666666666668, learning_rate = 9.824334e-06, loss = 0.0062649045, step = 880 (5.116 sec) 2022-04-22 17:57:17,681 [INFO] tensorflow: epoch = 1.4666666666666668, learning_rate = 9.824334e-06, loss = 0.0062649045, step = 880 (5.116 sec) 2022-04-22 17:57:19,057 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.377 INFO:tensorflow:global_step/sec: 13.3595 2022-04-22 17:57:19,129 [INFO] tensorflow: global_step/sec: 13.3595 2022-04-22 17:57:20,880 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.870 2022-04-22 17:57:22,651 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.471 INFO:tensorflow:epoch = 1.5833333333333335, learning_rate = 1.0366603e-05, loss = 0.005764888, step = 950 (5.113 sec) 2022-04-22 17:57:22,726 [INFO] tensorflow: epoch = 1.5833333333333335, learning_rate = 1.0366603e-05, loss = 0.005764888, step = 950 (5.113 sec) INFO:tensorflow:epoch = 1.5850000000000002, learning_rate = 1.0374565e-05, loss = 0.005637334, step = 951 (5.111 sec) 2022-04-22 17:57:22,792 [INFO] tensorflow: epoch = 1.5850000000000002, learning_rate = 1.0374565e-05, loss = 0.005637334, step = 951 (5.111 sec) INFO:tensorflow:global_step/sec: 13.9431 2022-04-22 17:57:23,432 [INFO] tensorflow: global_step/sec: 13.9431 2022-04-22 17:57:24,525 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.357 2022-04-22 17:57:26,408 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.109 INFO:tensorflow:epoch = 1.6983333333333335, learning_rate = 1.0930411e-05, loss = 0.0050861062, step = 1019 (5.066 sec) 2022-04-22 17:57:27,857 [INFO] tensorflow: epoch = 1.6983333333333335, learning_rate = 1.0930411e-05, loss = 0.0050861062, step = 1019 (5.066 sec) INFO:tensorflow:epoch = 1.6983333333333335, learning_rate = 1.0930411e-05, loss = 0.0050339736, step = 1019 (5.145 sec) 2022-04-22 17:57:27,871 [INFO] tensorflow: epoch = 1.6983333333333335, learning_rate = 1.0930411e-05, loss = 0.0050339736, step = 1019 (5.145 sec) INFO:tensorflow:global_step/sec: 13.3076 2022-04-22 17:57:27,941 [INFO] tensorflow: global_step/sec: 13.3076 2022-04-22 17:57:28,232 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.830 2022-04-22 17:57:30,049 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.040 2022-04-22 17:57:31,926 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.290 INFO:tensorflow:global_step/sec: 13.5081 2022-04-22 17:57:32,382 [INFO] tensorflow: global_step/sec: 13.5081 INFO:tensorflow:epoch = 1.8116666666666668, learning_rate = 1.1516038e-05, loss = 0.0048852256, step = 1087 (5.118 sec) 2022-04-22 17:57:32,989 [INFO] tensorflow: epoch = 1.8116666666666668, learning_rate = 1.1516038e-05, loss = 0.0048852256, step = 1087 (5.118 sec) INFO:tensorflow:epoch = 1.8116666666666668, learning_rate = 1.1516038e-05, loss = 0.004540758, step = 1087 (5.133 sec) 2022-04-22 17:57:32,990 [INFO] tensorflow: epoch = 1.8116666666666668, learning_rate = 1.1516038e-05, loss = 0.004540758, step = 1087 (5.133 sec) 2022-04-22 17:57:33,838 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 52.303 2022-04-22 17:57:35,588 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 57.150 INFO:tensorflow:global_step/sec: 13.471 2022-04-22 17:57:36,836 [INFO] tensorflow: global_step/sec: 13.471 2022-04-22 17:57:37,462 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.369 INFO:tensorflow:epoch = 1.9300000000000002, learning_rate = 1.2161017e-05, loss = 0.0041347677, step = 1158 (5.124 sec) 2022-04-22 17:57:38,113 [INFO] tensorflow: epoch = 1.9300000000000002, learning_rate = 1.2161017e-05, loss = 0.0041347677, step = 1158 (5.124 sec) INFO:tensorflow:epoch = 1.9300000000000002, learning_rate = 1.2161017e-05, loss = 0.004095979, step = 1158 (5.123 sec) 2022-04-22 17:57:38,113 [INFO] tensorflow: epoch = 1.9300000000000002, learning_rate = 1.2161017e-05, loss = 0.004095979, step = 1158 (5.123 sec) 2022-04-22 17:57:39,208 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 57.298 2022-04-22 17:57:41,086 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.234 INFO:tensorflow:global_step/sec: 13.8745 2022-04-22 17:57:41,161 [INFO] tensorflow: global_step/sec: 13.8745 2022-04-22 17:57:41,162 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 2/100: loss: 0.00388 learning rate: 0.00001 Time taken: 0:00:43.986989 ETA: 1:11:50.724877 2022-04-22 17:57:42,887 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.551 INFO:tensorflow:epoch = 2.0483333333333333, learning_rate = 1.284212e-05, loss = 0.0038402404, step = 1229 (5.134 sec) 2022-04-22 17:57:43,247 [INFO] tensorflow: epoch = 2.0483333333333333, learning_rate = 1.284212e-05, loss = 0.0038402404, step = 1229 (5.134 sec) INFO:tensorflow:epoch = 2.0483333333333333, learning_rate = 1.284212e-05, loss = 0.003742469, step = 1229 (5.135 sec) 2022-04-22 17:57:43,248 [INFO] tensorflow: epoch = 2.0483333333333333, learning_rate = 1.284212e-05, loss = 0.003742469, step = 1229 (5.135 sec) 2022-04-22 17:57:44,639 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 57.088 INFO:tensorflow:global_step/sec: 14.0763 2022-04-22 17:57:45,423 [INFO] tensorflow: global_step/sec: 14.0763 2022-04-22 17:57:46,500 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.723 INFO:tensorflow:epoch = 2.1633333333333336, learning_rate = 1.3540562e-05, loss = 0.0034601449, step = 1298 (5.098 sec) 2022-04-22 17:57:48,345 [INFO] tensorflow: epoch = 2.1633333333333336, learning_rate = 1.3540562e-05, loss = 0.0034601449, step = 1298 (5.098 sec) INFO:tensorflow:epoch = 2.1633333333333336, learning_rate = 1.3540562e-05, loss = 0.0035446272, step = 1298 (5.098 sec) 2022-04-22 17:57:48,345 [INFO] tensorflow: epoch = 2.1633333333333336, learning_rate = 1.3540562e-05, loss = 0.0035446272, step = 1298 (5.098 sec) 2022-04-22 17:57:48,417 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 52.189 INFO:tensorflow:global_step/sec: 13.1438 2022-04-22 17:57:49,988 [INFO] tensorflow: global_step/sec: 13.1438 2022-04-22 17:57:50,295 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.259 2022-04-22 17:57:52,057 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.749 INFO:tensorflow:epoch = 2.2800000000000002, learning_rate = 1.4287955e-05, loss = 0.0031424165, step = 1368 (5.094 sec) 2022-04-22 17:57:53,439 [INFO] tensorflow: epoch = 2.2800000000000002, learning_rate = 1.4287955e-05, loss = 0.0031424165, step = 1368 (5.094 sec) INFO:tensorflow:epoch = 2.2800000000000002, learning_rate = 1.4287955e-05, loss = 0.0031738563, step = 1368 (5.096 sec) 2022-04-22 17:57:53,441 [INFO] tensorflow: epoch = 2.2800000000000002, learning_rate = 1.4287955e-05, loss = 0.0031738563, step = 1368 (5.096 sec) 2022-04-22 17:57:53,898 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.310 INFO:tensorflow:global_step/sec: 13.8187 2022-04-22 17:57:54,330 [INFO] tensorflow: global_step/sec: 13.8187 2022-04-22 17:57:55,662 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.706 2022-04-22 17:57:57,477 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.099 INFO:tensorflow:epoch = 2.395, learning_rate = 1.5065033e-05, loss = 0.0028804983, step = 1437 (5.076 sec) 2022-04-22 17:57:58,514 [INFO] tensorflow: epoch = 2.395, learning_rate = 1.5065033e-05, loss = 0.0028804983, step = 1437 (5.076 sec) INFO:tensorflow:epoch = 2.395, learning_rate = 1.5065033e-05, loss = 0.0029200683, step = 1437 (5.074 sec) 2022-04-22 17:57:58,515 [INFO] tensorflow: epoch = 2.395, learning_rate = 1.5065033e-05, loss = 0.0029200683, step = 1437 (5.074 sec) INFO:tensorflow:global_step/sec: 13.6036 2022-04-22 17:57:58,741 [INFO] tensorflow: global_step/sec: 13.6036 2022-04-22 17:57:59,378 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 52.609 2022-04-22 17:58:01,196 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.037 2022-04-22 17:58:02,949 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 57.067 INFO:tensorflow:global_step/sec: 14.0261 2022-04-22 17:58:03,019 [INFO] tensorflow: global_step/sec: 14.0261 INFO:tensorflow:epoch = 2.5133333333333336, learning_rate = 1.5908767e-05, loss = 0.0028891393, step = 1508 (5.074 sec) 2022-04-22 17:58:03,589 [INFO] tensorflow: epoch = 2.5133333333333336, learning_rate = 1.5908767e-05, loss = 0.0028891393, step = 1508 (5.074 sec) INFO:tensorflow:epoch = 2.5133333333333336, learning_rate = 1.5908767e-05, loss = 0.0026353449, step = 1508 (5.083 sec) 2022-04-22 17:58:03,598 [INFO] tensorflow: epoch = 2.5133333333333336, learning_rate = 1.5908767e-05, loss = 0.0026353449, step = 1508 (5.083 sec) 2022-04-22 17:58:04,748 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.590 2022-04-22 17:58:06,571 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.837 INFO:tensorflow:global_step/sec: 13.6421 2022-04-22 17:58:07,417 [INFO] tensorflow: global_step/sec: 13.6421 2022-04-22 17:58:08,452 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.169 INFO:tensorflow:epoch = 2.6283333333333334, learning_rate = 1.6773995e-05, loss = 0.002415518, step = 1577 (5.088 sec) 2022-04-22 17:58:08,677 [INFO] tensorflow: epoch = 2.6283333333333334, learning_rate = 1.6773995e-05, loss = 0.002415518, step = 1577 (5.088 sec) INFO:tensorflow:epoch = 2.6300000000000003, learning_rate = 1.6786877e-05, loss = 0.0024441676, step = 1578 (5.166 sec) 2022-04-22 17:58:08,764 [INFO] tensorflow: epoch = 2.6300000000000003, learning_rate = 1.6786877e-05, loss = 0.0024441676, step = 1578 (5.166 sec) 2022-04-22 17:58:10,261 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.309 INFO:tensorflow:global_step/sec: 13.6465 2022-04-22 17:58:11,813 [INFO] tensorflow: global_step/sec: 13.6465 2022-04-22 17:58:12,120 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.788 INFO:tensorflow:epoch = 2.745, learning_rate = 1.7699862e-05, loss = 0.0022070862, step = 1647 (5.090 sec) 2022-04-22 17:58:13,767 [INFO] tensorflow: epoch = 2.745, learning_rate = 1.7699862e-05, loss = 0.0022070862, step = 1647 (5.090 sec) INFO:tensorflow:epoch = 2.746666666666667, learning_rate = 1.7713457e-05, loss = 0.0022045716, step = 1648 (5.080 sec) 2022-04-22 17:58:13,844 [INFO] tensorflow: epoch = 2.746666666666667, learning_rate = 1.7713457e-05, loss = 0.0022045716, step = 1648 (5.080 sec) 2022-04-22 17:58:13,927 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.345 2022-04-22 17:58:15,949 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 49.479 INFO:tensorflow:global_step/sec: 12.8544 2022-04-22 17:58:16,481 [INFO] tensorflow: global_step/sec: 12.8544 2022-04-22 17:58:17,869 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 52.104 INFO:tensorflow:epoch = 2.855, learning_rate = 1.8619588e-05, loss = 0.0020586452, step = 1713 (5.075 sec) 2022-04-22 17:58:18,842 [INFO] tensorflow: epoch = 2.855, learning_rate = 1.8619588e-05, loss = 0.0020586452, step = 1713 (5.075 sec) INFO:tensorflow:epoch = 2.8583333333333334, learning_rate = 1.8648181e-05, loss = 0.0022371514, step = 1715 (5.140 sec) 2022-04-22 17:58:18,984 [INFO] tensorflow: epoch = 2.8583333333333334, learning_rate = 1.8648181e-05, loss = 0.0022371514, step = 1715 (5.140 sec) 2022-04-22 17:58:19,624 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.964 INFO:tensorflow:global_step/sec: 13.6325 2022-04-22 17:58:20,882 [INFO] tensorflow: global_step/sec: 13.6325 2022-04-22 17:58:21,511 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.011 2022-04-22 17:58:23,290 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.205 INFO:tensorflow:epoch = 2.9716666666666667, learning_rate = 1.9647308e-05, loss = 0.00196068, step = 1783 (5.110 sec) 2022-04-22 17:58:23,952 [INFO] tensorflow: epoch = 2.9716666666666667, learning_rate = 1.9647308e-05, loss = 0.00196068, step = 1783 (5.110 sec) INFO:tensorflow:epoch = 2.975, learning_rate = 1.9677498e-05, loss = 0.0019244102, step = 1785 (5.111 sec) 2022-04-22 17:58:24,095 [INFO] tensorflow: epoch = 2.975, learning_rate = 1.9677498e-05, loss = 0.0019244102, step = 1785 (5.111 sec) 2022-04-22 17:58:25,105 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.119 INFO:tensorflow:global_step/sec: 13.9796 2022-04-22 17:58:25,174 [INFO] tensorflow: global_step/sec: 13.9796 2022-04-22 17:58:25,175 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 3/100: loss: 0.00187 learning rate: 0.00002 Time taken: 0:00:44.018021 ETA: 1:11:09.748070 2022-04-22 17:58:26,940 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.504 2022-04-22 17:58:28,733 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.779 INFO:tensorflow:epoch = 3.0883333333333334, learning_rate = 2.0731793e-05, loss = 0.0017513045, step = 1853 (5.099 sec) 2022-04-22 17:58:29,052 [INFO] tensorflow: epoch = 3.0883333333333334, learning_rate = 2.0731793e-05, loss = 0.0017513045, step = 1853 (5.099 sec) INFO:tensorflow:epoch = 3.091666666666667, learning_rate = 2.076363e-05, loss = 0.0017574094, step = 1855 (5.142 sec) 2022-04-22 17:58:29,236 [INFO] tensorflow: epoch = 3.091666666666667, learning_rate = 2.076363e-05, loss = 0.0017574094, step = 1855 (5.142 sec) INFO:tensorflow:global_step/sec: 13.5987 2022-04-22 17:58:29,586 [INFO] tensorflow: global_step/sec: 13.5987 2022-04-22 17:58:30,662 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 51.845 2022-04-22 17:58:32,569 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 52.442 INFO:tensorflow:epoch = 3.1983333333333337, learning_rate = 2.1809046e-05, loss = 0.0016146587, step = 1919 (5.089 sec) 2022-04-22 17:58:34,141 [INFO] tensorflow: epoch = 3.1983333333333337, learning_rate = 2.1809046e-05, loss = 0.0016146587, step = 1919 (5.089 sec) INFO:tensorflow:global_step/sec: 12.9625 2022-04-22 17:58:34,215 [INFO] tensorflow: global_step/sec: 12.9625 INFO:tensorflow:epoch = 3.2033333333333336, learning_rate = 2.1859332e-05, loss = 0.001666348, step = 1922 (5.124 sec) 2022-04-22 17:58:34,360 [INFO] tensorflow: epoch = 3.2033333333333336, learning_rate = 2.1859332e-05, loss = 0.001666348, step = 1922 (5.124 sec) 2022-04-22 17:58:34,509 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 51.563 2022-04-22 17:58:36,451 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 51.493 2022-04-22 17:58:38,284 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.577 INFO:tensorflow:global_step/sec: 13.3579 2022-04-22 17:58:38,707 [INFO] tensorflow: global_step/sec: 13.3579 INFO:tensorflow:epoch = 3.311666666666667, learning_rate = 2.2977525e-05, loss = 0.0014765048, step = 1987 (5.087 sec) 2022-04-22 17:58:39,228 [INFO] tensorflow: epoch = 3.311666666666667, learning_rate = 2.2977525e-05, loss = 0.0014765048, step = 1987 (5.087 sec) INFO:tensorflow:epoch = 3.316666666666667, learning_rate = 2.3030485e-05, loss = 0.0014760353, step = 1990 (5.088 sec) 2022-04-22 17:58:39,448 [INFO] tensorflow: epoch = 3.316666666666667, learning_rate = 2.3030485e-05, loss = 0.0014760353, step = 1990 (5.088 sec) 2022-04-22 17:58:40,071 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.936 2022-04-22 17:58:41,986 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 52.241 INFO:tensorflow:global_step/sec: 13.4345 2022-04-22 17:58:43,173 [INFO] tensorflow: global_step/sec: 13.4345 2022-04-22 17:58:43,825 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.372 INFO:tensorflow:epoch = 3.4266666666666667, learning_rate = 2.42272e-05, loss = 0.0013307532, step = 2056 (5.097 sec) 2022-04-22 17:58:44,325 [INFO] tensorflow: epoch = 3.4266666666666667, learning_rate = 2.42272e-05, loss = 0.0013307532, step = 2056 (5.097 sec) INFO:tensorflow:epoch = 3.431666666666667, learning_rate = 2.428304e-05, loss = 0.0013336489, step = 2059 (5.089 sec) 2022-04-22 17:58:44,537 [INFO] tensorflow: epoch = 3.431666666666667, learning_rate = 2.428304e-05, loss = 0.0013336489, step = 2059 (5.089 sec) 2022-04-22 17:58:45,651 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.788 2022-04-22 17:58:47,497 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.155 INFO:tensorflow:global_step/sec: 13.6548 2022-04-22 17:58:47,567 [INFO] tensorflow: global_step/sec: 13.6548 2022-04-22 17:58:49,275 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.262 INFO:tensorflow:epoch = 3.5433333333333334, learning_rate = 2.556446e-05, loss = 0.001224135, step = 2126 (5.088 sec) 2022-04-22 17:58:49,413 [INFO] tensorflow: epoch = 3.5433333333333334, learning_rate = 2.556446e-05, loss = 0.001224135, step = 2126 (5.088 sec) INFO:tensorflow:epoch = 3.5483333333333333, learning_rate = 2.5623382e-05, loss = 0.0012395629, step = 2129 (5.083 sec) 2022-04-22 17:58:49,620 [INFO] tensorflow: epoch = 3.5483333333333333, learning_rate = 2.5623382e-05, loss = 0.0012395629, step = 2129 (5.083 sec) 2022-04-22 17:58:51,025 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 57.139 INFO:tensorflow:global_step/sec: 14.227 2022-04-22 17:58:51,784 [INFO] tensorflow: global_step/sec: 14.227 2022-04-22 17:58:52,826 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.551 INFO:tensorflow:epoch = 3.66, learning_rate = 2.6975533e-05, loss = 0.0011228205, step = 2196 (5.103 sec) 2022-04-22 17:58:54,516 [INFO] tensorflow: epoch = 3.66, learning_rate = 2.6975533e-05, loss = 0.0011228205, step = 2196 (5.103 sec) INFO:tensorflow:epoch = 3.665, learning_rate = 2.7037706e-05, loss = 0.0011459169, step = 2199 (5.144 sec) 2022-04-22 17:58:54,765 [INFO] tensorflow: epoch = 3.665, learning_rate = 2.7037706e-05, loss = 0.0011459169, step = 2199 (5.144 sec) 2022-04-22 17:58:54,765 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 51.581 INFO:tensorflow:global_step/sec: 13.4128 2022-04-22 17:58:56,258 [INFO] tensorflow: global_step/sec: 13.4128 2022-04-22 17:58:56,553 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.922 2022-04-22 17:58:58,370 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.057 INFO:tensorflow:epoch = 3.7783333333333338, learning_rate = 2.8486325e-05, loss = 0.0010430852, step = 2267 (5.134 sec) 2022-04-22 17:58:59,650 [INFO] tensorflow: epoch = 3.7783333333333338, learning_rate = 2.8486325e-05, loss = 0.0010430852, step = 2267 (5.134 sec) INFO:tensorflow:epoch = 3.7833333333333337, learning_rate = 2.855201e-05, loss = 0.0010504124, step = 2270 (5.111 sec) 2022-04-22 17:58:59,876 [INFO] tensorflow: epoch = 3.7833333333333337, learning_rate = 2.855201e-05, loss = 0.0010504124, step = 2270 (5.111 sec) 2022-04-22 17:59:00,155 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.021 INFO:tensorflow:global_step/sec: 13.6466 2022-04-22 17:59:00,654 [INFO] tensorflow: global_step/sec: 13.6466 2022-04-22 17:59:02,042 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.005 2022-04-22 17:59:03,826 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.050 INFO:tensorflow:epoch = 3.8933333333333335, learning_rate = 3.0035637e-05, loss = 0.00094492245, step = 2336 (5.149 sec) 2022-04-22 17:59:04,798 [INFO] tensorflow: epoch = 3.8933333333333335, learning_rate = 3.0035637e-05, loss = 0.00094492245, step = 2336 (5.149 sec) INFO:tensorflow:epoch = 3.8983333333333334, learning_rate = 3.0104864e-05, loss = 0.00094097387, step = 2339 (5.139 sec) 2022-04-22 17:59:05,015 [INFO] tensorflow: epoch = 3.8983333333333334, learning_rate = 3.0104864e-05, loss = 0.00094097387, step = 2339 (5.139 sec) INFO:tensorflow:global_step/sec: 13.5375 2022-04-22 17:59:05,087 [INFO] tensorflow: global_step/sec: 13.5375 2022-04-22 17:59:05,800 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 50.676 2022-04-22 17:59:07,646 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.169 2022-04-22 17:59:09,617 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 50.744 INFO:tensorflow:global_step/sec: 13.023 2022-04-22 17:59:09,694 [INFO] tensorflow: global_step/sec: 13.023 2022-04-22 17:59:09,695 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 4/100: loss: 0.00087 learning rate: 0.00003 Time taken: 0:00:44.511927 ETA: 1:11:13.145004 INFO:tensorflow:epoch = 4.005, learning_rate = 3.162059e-05, loss = 0.0008659351, step = 2403 (5.119 sec) 2022-04-22 17:59:09,917 [INFO] tensorflow: epoch = 4.005, learning_rate = 3.162059e-05, loss = 0.0008659351, step = 2403 (5.119 sec) INFO:tensorflow:epoch = 4.010000000000001, learning_rate = 3.1693475e-05, loss = 0.0008735097, step = 2406 (5.112 sec) 2022-04-22 17:59:10,127 [INFO] tensorflow: epoch = 4.010000000000001, learning_rate = 3.1693475e-05, loss = 0.0008735097, step = 2406 (5.112 sec) 2022-04-22 17:59:11,380 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.747 2022-04-22 17:59:13,134 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 57.010 INFO:tensorflow:global_step/sec: 14.1904 2022-04-22 17:59:13,922 [INFO] tensorflow: global_step/sec: 14.1904 INFO:tensorflow:epoch = 4.123333333333334, learning_rate = 3.3391567e-05, loss = 0.0008107077, step = 2474 (5.104 sec) 2022-04-22 17:59:15,021 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 52.999 2022-04-22 17:59:15,021 [INFO] tensorflow: epoch = 4.123333333333334, learning_rate = 3.3391567e-05, loss = 0.0008107077, step = 2474 (5.104 sec) INFO:tensorflow:epoch = 4.128333333333334, learning_rate = 3.346853e-05, loss = 0.00078763714, step = 2477 (5.133 sec) 2022-04-22 17:59:15,260 [INFO] tensorflow: epoch = 4.128333333333334, learning_rate = 3.346853e-05, loss = 0.00078763714, step = 2477 (5.133 sec) 2022-04-22 17:59:16,892 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.476 INFO:tensorflow:global_step/sec: 13.1332 2022-04-22 17:59:18,491 [INFO] tensorflow: global_step/sec: 13.1332 2022-04-22 17:59:18,797 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 52.481 INFO:tensorflow:epoch = 4.238333333333333, learning_rate = 3.520763e-05, loss = 0.0007301614, step = 2543 (5.135 sec) 2022-04-22 17:59:20,157 [INFO] tensorflow: epoch = 4.238333333333333, learning_rate = 3.520763e-05, loss = 0.0007301614, step = 2543 (5.135 sec) INFO:tensorflow:epoch = 4.243333333333333, learning_rate = 3.528878e-05, loss = 0.0007251326, step = 2546 (5.121 sec) 2022-04-22 17:59:20,381 [INFO] tensorflow: epoch = 4.243333333333333, learning_rate = 3.528878e-05, loss = 0.0007251326, step = 2546 (5.121 sec) 2022-04-22 17:59:20,601 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.454 2022-04-22 17:59:22,460 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.802 INFO:tensorflow:global_step/sec: 13.6487 2022-04-22 17:59:22,887 [INFO] tensorflow: global_step/sec: 13.6487 2022-04-22 17:59:24,237 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.274 INFO:tensorflow:epoch = 4.355, learning_rate = 3.7150938e-05, loss = 0.00069518865, step = 2613 (5.146 sec) 2022-04-22 17:59:25,302 [INFO] tensorflow: epoch = 4.355, learning_rate = 3.7150938e-05, loss = 0.00069518865, step = 2613 (5.146 sec) INFO:tensorflow:epoch = 4.36, learning_rate = 3.72366e-05, loss = 0.0006593655, step = 2616 (5.126 sec) 2022-04-22 17:59:25,507 [INFO] tensorflow: epoch = 4.36, learning_rate = 3.72366e-05, loss = 0.0006593655, step = 2616 (5.126 sec) 2022-04-22 17:59:26,165 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 51.881 INFO:tensorflow:global_step/sec: 13.4697 2022-04-22 17:59:27,341 [INFO] tensorflow: global_step/sec: 13.4697 2022-04-22 17:59:27,992 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.740 2022-04-22 17:59:29,904 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 52.324 INFO:tensorflow:epoch = 4.466666666666667, learning_rate = 3.9111404e-05, loss = 0.0006090282, step = 2680 (5.098 sec) 2022-04-22 17:59:30,400 [INFO] tensorflow: epoch = 4.466666666666667, learning_rate = 3.9111404e-05, loss = 0.0006090282, step = 2680 (5.098 sec) INFO:tensorflow:epoch = 4.471666666666667, learning_rate = 3.9201546e-05, loss = 0.0006255055, step = 2683 (5.107 sec) 2022-04-22 17:59:30,615 [INFO] tensorflow: epoch = 4.471666666666667, learning_rate = 3.9201546e-05, loss = 0.0006255055, step = 2683 (5.107 sec) 2022-04-22 17:59:31,925 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 49.474 INFO:tensorflow:global_step/sec: 12.8342 2022-04-22 17:59:32,016 [INFO] tensorflow: global_step/sec: 12.8342 2022-04-22 17:59:33,821 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 52.742 INFO:tensorflow:epoch = 4.58, learning_rate = 4.1206906e-05, loss = 0.0005572244, step = 2748 (5.124 sec) 2022-04-22 17:59:35,524 [INFO] tensorflow: epoch = 4.58, learning_rate = 4.1206906e-05, loss = 0.0005572244, step = 2748 (5.124 sec) 2022-04-22 17:59:35,594 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.423 INFO:tensorflow:epoch = 4.585, learning_rate = 4.130188e-05, loss = 0.0005591678, step = 2751 (5.114 sec) 2022-04-22 17:59:35,729 [INFO] tensorflow: epoch = 4.585, learning_rate = 4.130188e-05, loss = 0.0005591678, step = 2751 (5.114 sec) INFO:tensorflow:global_step/sec: 13.4222 2022-04-22 17:59:36,486 [INFO] tensorflow: global_step/sec: 13.4222 2022-04-22 17:59:37,558 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 50.930 2022-04-22 17:59:39,309 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 57.087 INFO:tensorflow:epoch = 4.696666666666667, learning_rate = 4.348139e-05, loss = 0.0005145955, step = 2818 (5.138 sec) 2022-04-22 17:59:40,663 [INFO] tensorflow: epoch = 4.696666666666667, learning_rate = 4.348139e-05, loss = 0.0005145955, step = 2818 (5.138 sec) INFO:tensorflow:epoch = 4.7, learning_rate = 4.354816e-05, loss = 0.0005208661, step = 2820 (5.079 sec) 2022-04-22 17:59:40,808 [INFO] tensorflow: epoch = 4.7, learning_rate = 4.354816e-05, loss = 0.0005208661, step = 2820 (5.079 sec) INFO:tensorflow:global_step/sec: 13.8808 2022-04-22 17:59:40,809 [INFO] tensorflow: global_step/sec: 13.8808 2022-04-22 17:59:41,101 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.826 2022-04-22 17:59:42,935 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.540 2022-04-22 17:59:44,739 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.432 INFO:tensorflow:global_step/sec: 13.6692 2022-04-22 17:59:45,198 [INFO] tensorflow: global_step/sec: 13.6692 INFO:tensorflow:epoch = 4.8116666666666665, learning_rate = 4.5846205e-05, loss = 0.00046869507, step = 2887 (5.081 sec) 2022-04-22 17:59:45,743 [INFO] tensorflow: epoch = 4.8116666666666665, learning_rate = 4.5846205e-05, loss = 0.00046869507, step = 2887 (5.081 sec) INFO:tensorflow:epoch = 4.815, learning_rate = 4.5916608e-05, loss = 0.0004679728, step = 2889 (5.080 sec) 2022-04-22 17:59:45,888 [INFO] tensorflow: epoch = 4.815, learning_rate = 4.5916608e-05, loss = 0.0004679728, step = 2889 (5.080 sec) 2022-04-22 17:59:46,652 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 52.281 2022-04-22 17:59:48,460 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.309 INFO:tensorflow:global_step/sec: 13.5834 2022-04-22 17:59:49,615 [INFO] tensorflow: global_step/sec: 13.5834 2022-04-22 17:59:50,243 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.084 INFO:tensorflow:epoch = 4.928333333333334, learning_rate = 4.837676e-05, loss = 0.00043759664, step = 2957 (5.111 sec) 2022-04-22 17:59:50,854 [INFO] tensorflow: epoch = 4.928333333333334, learning_rate = 4.837676e-05, loss = 0.00043759664, step = 2957 (5.111 sec) INFO:tensorflow:epoch = 4.931666666666667, learning_rate = 4.845105e-05, loss = 0.0004317015, step = 2959 (5.122 sec) 2022-04-22 17:59:51,010 [INFO] tensorflow: epoch = 4.931666666666667, learning_rate = 4.845105e-05, loss = 0.0004317015, step = 2959 (5.122 sec) 2022-04-22 17:59:52,180 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 51.644 2022-04-22 17:59:54,059 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.212 INFO:tensorflow:global_step/sec: 13.2873 2022-04-22 17:59:54,131 [INFO] tensorflow: global_step/sec: 13.2873 2022-04-22 17:59:54,132 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 5/100: loss: 0.00041 learning rate: 0.00005 Time taken: 0:00:44.441534 ETA: 1:10:21.945734 2022-04-22 17:59:55,886 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.766 INFO:tensorflow:epoch = 5.041666666666667, learning_rate = 5.096868e-05, loss = 0.0003985542, step = 3025 (5.107 sec) 2022-04-22 17:59:55,961 [INFO] tensorflow: epoch = 5.041666666666667, learning_rate = 5.096868e-05, loss = 0.0003985542, step = 3025 (5.107 sec) INFO:tensorflow:epoch = 5.045, learning_rate = 5.1046947e-05, loss = 0.00039906448, step = 3027 (5.077 sec) 2022-04-22 17:59:56,087 [INFO] tensorflow: epoch = 5.045, learning_rate = 5.1046947e-05, loss = 0.00039906448, step = 3027 (5.077 sec) 2022-04-22 17:59:57,679 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.763 INFO:tensorflow:global_step/sec: 13.9147 2022-04-22 17:59:58,443 [INFO] tensorflow: global_step/sec: 13.9147 2022-04-22 17:59:59,458 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.219 INFO:tensorflow:epoch = 5.16, learning_rate = 5.382323e-05, loss = 0.00037422194, step = 3096 (5.139 sec) 2022-04-22 18:00:01,100 [INFO] tensorflow: epoch = 5.16, learning_rate = 5.382323e-05, loss = 0.00037422194, step = 3096 (5.139 sec) INFO:tensorflow:epoch = 5.161666666666667, learning_rate = 5.3864565e-05, loss = 0.00035645984, step = 3097 (5.088 sec) 2022-04-22 18:00:01,174 [INFO] tensorflow: epoch = 5.161666666666667, learning_rate = 5.3864565e-05, loss = 0.00035645984, step = 3097 (5.088 sec) 2022-04-22 18:00:01,331 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.410 INFO:tensorflow:global_step/sec: 13.397 2022-04-22 18:00:02,922 [INFO] tensorflow: global_step/sec: 13.397 2022-04-22 18:00:03,207 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.311 2022-04-22 18:00:04,949 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 57.419 INFO:tensorflow:epoch = 5.278333333333333, learning_rate = 5.683771e-05, loss = 0.00034372657, step = 3167 (5.122 sec) 2022-04-22 18:00:06,222 [INFO] tensorflow: epoch = 5.278333333333333, learning_rate = 5.683771e-05, loss = 0.00034372657, step = 3167 (5.122 sec) INFO:tensorflow:epoch = 5.28, learning_rate = 5.6881363e-05, loss = 0.00032632204, step = 3168 (5.142 sec) 2022-04-22 18:00:06,316 [INFO] tensorflow: epoch = 5.28, learning_rate = 5.6881363e-05, loss = 0.00032632204, step = 3168 (5.142 sec) 2022-04-22 18:00:06,745 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.683 INFO:tensorflow:global_step/sec: 14.1282 2022-04-22 18:00:07,168 [INFO] tensorflow: global_step/sec: 14.1282 2022-04-22 18:00:08,447 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 58.739 2022-04-22 18:00:10,204 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.920 INFO:tensorflow:epoch = 5.4, learning_rate = 6.0113252e-05, loss = 0.00029672903, step = 3240 (5.098 sec) 2022-04-22 18:00:11,320 [INFO] tensorflow: epoch = 5.4, learning_rate = 6.0113252e-05, loss = 0.00029672903, step = 3240 (5.098 sec) INFO:tensorflow:global_step/sec: 14.4439 2022-04-22 18:00:11,323 [INFO] tensorflow: global_step/sec: 14.4439 INFO:tensorflow:epoch = 5.401666666666667, learning_rate = 6.015936e-05, loss = 0.00029765093, step = 3241 (5.102 sec) 2022-04-22 18:00:11,418 [INFO] tensorflow: epoch = 5.401666666666667, learning_rate = 6.015936e-05, loss = 0.00029765093, step = 3241 (5.102 sec) 2022-04-22 18:00:12,184 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 50.525 2022-04-22 18:00:14,013 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.673 2022-04-22 18:00:15,887 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.386 INFO:tensorflow:global_step/sec: 12.8886 2022-04-22 18:00:15,978 [INFO] tensorflow: global_step/sec: 12.8886 INFO:tensorflow:epoch = 5.510000000000001, learning_rate = 6.323681e-05, loss = 0.00027744475, step = 3306 (5.095 sec) 2022-04-22 18:00:16,415 [INFO] tensorflow: epoch = 5.510000000000001, learning_rate = 6.323681e-05, loss = 0.00027744475, step = 3306 (5.095 sec) INFO:tensorflow:epoch = 5.513333333333334, learning_rate = 6.333398e-05, loss = 0.00027489164, step = 3308 (5.150 sec) 2022-04-22 18:00:16,568 [INFO] tensorflow: epoch = 5.513333333333334, learning_rate = 6.333398e-05, loss = 0.00027489164, step = 3308 (5.150 sec) 2022-04-22 18:00:17,736 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.073 2022-04-22 18:00:19,571 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.526 INFO:tensorflow:global_step/sec: 13.8017 2022-04-22 18:00:20,325 [INFO] tensorflow: global_step/sec: 13.8017 2022-04-22 18:00:21,320 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 57.169 INFO:tensorflow:epoch = 5.628333333333334, learning_rate = 6.6778455e-05, loss = 0.0002519965, step = 3377 (5.117 sec) 2022-04-22 18:00:21,532 [INFO] tensorflow: epoch = 5.628333333333334, learning_rate = 6.6778455e-05, loss = 0.0002519965, step = 3377 (5.117 sec) INFO:tensorflow:epoch = 5.631666666666667, learning_rate = 6.688107e-05, loss = 0.0002525258, step = 3379 (5.107 sec) 2022-04-22 18:00:21,675 [INFO] tensorflow: epoch = 5.631666666666667, learning_rate = 6.688107e-05, loss = 0.0002525258, step = 3379 (5.107 sec) 2022-04-22 18:00:23,159 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.395 INFO:tensorflow:global_step/sec: 13.8455 2022-04-22 18:00:24,659 [INFO] tensorflow: global_step/sec: 13.8455 2022-04-22 18:00:24,925 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.629 INFO:tensorflow:epoch = 5.746666666666667, learning_rate = 7.051852e-05, loss = 0.0002401043, step = 3448 (5.124 sec) 2022-04-22 18:00:26,656 [INFO] tensorflow: epoch = 5.746666666666667, learning_rate = 7.051852e-05, loss = 0.0002401043, step = 3448 (5.124 sec) 2022-04-22 18:00:26,728 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.453 INFO:tensorflow:epoch = 5.75, learning_rate = 7.0626884e-05, loss = 0.00023462548, step = 3450 (5.123 sec) 2022-04-22 18:00:26,798 [INFO] tensorflow: epoch = 5.75, learning_rate = 7.0626884e-05, loss = 0.00023462548, step = 3450 (5.123 sec) 2022-04-22 18:00:28,576 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.137 INFO:tensorflow:global_step/sec: 13.8113 2022-04-22 18:00:29,003 [INFO] tensorflow: global_step/sec: 13.8113 2022-04-22 18:00:30,366 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.877 INFO:tensorflow:epoch = 5.865, learning_rate = 7.446806e-05, loss = 0.0002093789, step = 3519 (5.146 sec) 2022-04-22 18:00:31,802 [INFO] tensorflow: epoch = 5.865, learning_rate = 7.446806e-05, loss = 0.0002093789, step = 3519 (5.146 sec) INFO:tensorflow:epoch = 5.866666666666667, learning_rate = 7.4525255e-05, loss = 0.00021332706, step = 3520 (5.105 sec) 2022-04-22 18:00:31,902 [INFO] tensorflow: epoch = 5.866666666666667, learning_rate = 7.4525255e-05, loss = 0.00021332706, step = 3520 (5.105 sec) 2022-04-22 18:00:32,177 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.201 INFO:tensorflow:global_step/sec: 13.8012 2022-04-22 18:00:33,350 [INFO] tensorflow: global_step/sec: 13.8012 2022-04-22 18:00:34,045 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.536 2022-04-22 18:00:35,958 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 52.311 INFO:tensorflow:epoch = 5.9783333333333335, learning_rate = 7.845789e-05, loss = 0.000193691, step = 3587 (5.114 sec) 2022-04-22 18:00:36,916 [INFO] tensorflow: epoch = 5.9783333333333335, learning_rate = 7.845789e-05, loss = 0.000193691, step = 3587 (5.114 sec) INFO:tensorflow:epoch = 5.98, learning_rate = 7.851815e-05, loss = 0.00018976588, step = 3588 (5.086 sec) 2022-04-22 18:00:36,989 [INFO] tensorflow: epoch = 5.98, learning_rate = 7.851815e-05, loss = 0.00018976588, step = 3588 (5.086 sec) 2022-04-22 18:00:37,798 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 299, 0.00s/step 2022-04-22 18:00:38,021 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 299, 0.02s/step 2022-04-22 18:00:38,242 [INFO] iva.detectnet_v2.evaluation.evaluation: step 20 / 299, 0.02s/step 2022-04-22 18:00:38,465 [INFO] iva.detectnet_v2.evaluation.evaluation: step 30 / 299, 0.02s/step 2022-04-22 18:00:38,688 [INFO] iva.detectnet_v2.evaluation.evaluation: step 40 / 299, 0.02s/step 2022-04-22 18:00:38,913 [INFO] iva.detectnet_v2.evaluation.evaluation: step 50 / 299, 0.02s/step 2022-04-22 18:00:39,145 [INFO] iva.detectnet_v2.evaluation.evaluation: step 60 / 299, 0.02s/step 2022-04-22 18:00:39,380 [INFO] iva.detectnet_v2.evaluation.evaluation: step 70 / 299, 0.02s/step 2022-04-22 18:00:39,610 [INFO] iva.detectnet_v2.evaluation.evaluation: step 80 / 299, 0.02s/step 2022-04-22 18:00:39,844 [INFO] iva.detectnet_v2.evaluation.evaluation: step 90 / 299, 0.02s/step 2022-04-22 18:00:40,079 [INFO] iva.detectnet_v2.evaluation.evaluation: step 100 / 299, 0.02s/step 2022-04-22 18:00:40,313 [INFO] iva.detectnet_v2.evaluation.evaluation: step 110 / 299, 0.02s/step 2022-04-22 18:00:40,553 [INFO] iva.detectnet_v2.evaluation.evaluation: step 120 / 299, 0.02s/step 2022-04-22 18:00:40,788 [INFO] iva.detectnet_v2.evaluation.evaluation: step 130 / 299, 0.02s/step 2022-04-22 18:00:41,025 [INFO] iva.detectnet_v2.evaluation.evaluation: step 140 / 299, 0.02s/step 2022-04-22 18:00:41,260 [INFO] iva.detectnet_v2.evaluation.evaluation: step 150 / 299, 0.02s/step 2022-04-22 18:00:41,495 [INFO] iva.detectnet_v2.evaluation.evaluation: step 160 / 299, 0.02s/step 2022-04-22 18:00:41,733 [INFO] iva.detectnet_v2.evaluation.evaluation: step 170 / 299, 0.02s/step 2022-04-22 18:00:41,970 [INFO] iva.detectnet_v2.evaluation.evaluation: step 180 / 299, 0.02s/step 2022-04-22 18:00:42,206 [INFO] iva.detectnet_v2.evaluation.evaluation: step 190 / 299, 0.02s/step 2022-04-22 18:00:42,440 [INFO] iva.detectnet_v2.evaluation.evaluation: step 200 / 299, 0.02s/step 2022-04-22 18:00:42,679 [INFO] iva.detectnet_v2.evaluation.evaluation: step 210 / 299, 0.02s/step 2022-04-22 18:00:42,915 [INFO] iva.detectnet_v2.evaluation.evaluation: step 220 / 299, 0.02s/step 2022-04-22 18:00:43,148 [INFO] iva.detectnet_v2.evaluation.evaluation: step 230 / 299, 0.02s/step 2022-04-22 18:00:43,379 [INFO] iva.detectnet_v2.evaluation.evaluation: step 240 / 299, 0.02s/step 2022-04-22 18:00:43,620 [INFO] iva.detectnet_v2.evaluation.evaluation: step 250 / 299, 0.02s/step 2022-04-22 18:00:43,860 [INFO] iva.detectnet_v2.evaluation.evaluation: step 260 / 299, 0.02s/step 2022-04-22 18:00:44,098 [INFO] iva.detectnet_v2.evaluation.evaluation: step 270 / 299, 0.02s/step 2022-04-22 18:00:44,340 [INFO] iva.detectnet_v2.evaluation.evaluation: step 280 / 299, 0.02s/step 2022-04-22 18:00:44,577 [INFO] iva.detectnet_v2.evaluation.evaluation: step 290 / 299, 0.02s/step Epoch 6/100 ========================= Validation cost: 0.000159 Mean average_precision (in %): 0.0000 class name average precision (in %) ------------ -------------------------- face 0 Median Inference Time: 0.011576 2022-04-22 18:00:44,824 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 11.279 INFO:tensorflow:epoch = 6.0, learning_rate = 7.924468e-05, loss = 0.00018837611, step = 3600 (7.902 sec) 2022-04-22 18:00:44,891 [INFO] tensorflow: epoch = 6.0, learning_rate = 7.924468e-05, loss = 0.00018837611, step = 3600 (7.902 sec) INFO:tensorflow:global_step/sec: 5.19895 2022-04-22 18:00:44,891 [INFO] tensorflow: global_step/sec: 5.19895 2022-04-22 18:00:44,892 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 6/100: loss: 0.00019 learning rate: 0.00008 Time taken: 0:00:50.764453 ETA: 1:19:31.858576 INFO:tensorflow:epoch = 6.001666666666667, learning_rate = 7.9305464e-05, loss = 0.00018795645, step = 3601 (8.042 sec) 2022-04-22 18:00:44,958 [INFO] tensorflow: epoch = 6.001666666666667, learning_rate = 7.9305464e-05, loss = 0.00018795645, step = 3601 (8.042 sec) 2022-04-22 18:00:46,652 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.737 2022-04-22 18:00:48,400 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 57.223 INFO:tensorflow:global_step/sec: 14.0916 2022-04-22 18:00:49,149 [INFO] tensorflow: global_step/sec: 14.0916 INFO:tensorflow:epoch = 6.12, learning_rate = 8.374712e-05, loss = 0.00017144677, step = 3672 (5.139 sec) 2022-04-22 18:00:50,030 [INFO] tensorflow: epoch = 6.12, learning_rate = 8.374712e-05, loss = 0.00017144677, step = 3672 (5.139 sec) INFO:tensorflow:epoch = 6.12, learning_rate = 8.374712e-05, loss = 0.00017677648, step = 3672 (5.072 sec) 2022-04-22 18:00:50,030 [INFO] tensorflow: epoch = 6.12, learning_rate = 8.374712e-05, loss = 0.00017677648, step = 3672 (5.072 sec) 2022-04-22 18:00:50,174 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.350 2022-04-22 18:00:51,998 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.835 INFO:tensorflow:global_step/sec: 13.6826 2022-04-22 18:00:53,534 [INFO] tensorflow: global_step/sec: 13.6826 2022-04-22 18:00:53,819 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.930 INFO:tensorflow:epoch = 6.233333333333333, learning_rate = 8.823411e-05, loss = 0.00016543886, step = 3740 (5.082 sec) INFO:tensorflow:epoch = 6.233333333333333, learning_rate = 8.823411e-05, loss = 0.00015839975, step = 3740 (5.083 sec) 2022-04-22 18:00:55,113 [INFO] tensorflow: epoch = 6.233333333333333, learning_rate = 8.823411e-05, loss = 0.00016543886, step = 3740 (5.082 sec) 2022-04-22 18:00:55,113 [INFO] tensorflow: epoch = 6.233333333333333, learning_rate = 8.823411e-05, loss = 0.00015839975, step = 3740 (5.083 sec) 2022-04-22 18:00:55,765 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 51.403 2022-04-22 18:00:57,531 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.611 INFO:tensorflow:global_step/sec: 13.6206 2022-04-22 18:00:57,939 [INFO] tensorflow: global_step/sec: 13.6206 2022-04-22 18:00:59,402 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.467 INFO:tensorflow:epoch = 6.3500000000000005, learning_rate = 9.310434e-05, loss = 0.00014598109, step = 3810 (5.127 sec) 2022-04-22 18:01:00,240 [INFO] tensorflow: epoch = 6.3500000000000005, learning_rate = 9.310434e-05, loss = 0.00014598109, step = 3810 (5.127 sec) INFO:tensorflow:epoch = 6.3500000000000005, learning_rate = 9.310434e-05, loss = 0.00014460446, step = 3810 (5.127 sec) 2022-04-22 18:01:00,240 [INFO] tensorflow: epoch = 6.3500000000000005, learning_rate = 9.310434e-05, loss = 0.00014460446, step = 3810 (5.127 sec) 2022-04-22 18:01:01,249 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.152 INFO:tensorflow:global_step/sec: 13.4542 2022-04-22 18:01:02,399 [INFO] tensorflow: global_step/sec: 13.4542 2022-04-22 18:01:03,037 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.924 2022-04-22 18:01:04,802 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.656 INFO:tensorflow:epoch = 6.468333333333334, learning_rate = 9.831884e-05, loss = 0.00013609271, step = 3881 (5.125 sec) 2022-04-22 18:01:05,365 [INFO] tensorflow: epoch = 6.468333333333334, learning_rate = 9.831884e-05, loss = 0.00013609271, step = 3881 (5.125 sec) INFO:tensorflow:epoch = 6.468333333333334, learning_rate = 9.831884e-05, loss = 0.00013250294, step = 3881 (5.125 sec) 2022-04-22 18:01:05,365 [INFO] tensorflow: epoch = 6.468333333333334, learning_rate = 9.831884e-05, loss = 0.00013250294, step = 3881 (5.125 sec) 2022-04-22 18:01:06,652 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.101 INFO:tensorflow:global_step/sec: 13.8816 2022-04-22 18:01:06,721 [INFO] tensorflow: global_step/sec: 13.8816 2022-04-22 18:01:08,459 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.320 2022-04-22 18:01:10,270 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.242 INFO:tensorflow:epoch = 6.586666666666667, learning_rate = 0.00010382528, loss = 0.0001242502, step = 3952 (5.120 sec) 2022-04-22 18:01:10,484 [INFO] tensorflow: epoch = 6.586666666666667, learning_rate = 0.00010382528, loss = 0.0001242502, step = 3952 (5.120 sec) INFO:tensorflow:epoch = 6.586666666666667, learning_rate = 0.00010382528, loss = 0.00012270668, step = 3952 (5.119 sec) 2022-04-22 18:01:10,484 [INFO] tensorflow: epoch = 6.586666666666667, learning_rate = 0.00010382528, loss = 0.00012270668, step = 3952 (5.119 sec) INFO:tensorflow:global_step/sec: 13.8456 2022-04-22 18:01:11,055 [INFO] tensorflow: global_step/sec: 13.8456 2022-04-22 18:01:12,059 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.911 2022-04-22 18:01:13,887 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.686 INFO:tensorflow:global_step/sec: 13.9083 2022-04-22 18:01:15,369 [INFO] tensorflow: global_step/sec: 13.9083 INFO:tensorflow:epoch = 6.705, learning_rate = 0.000109640234, loss = 0.00011264731, step = 4023 (5.090 sec) INFO:tensorflow:epoch = 6.705, learning_rate = 0.000109640234, loss = 0.00011219685, step = 4023 (5.090 sec) 2022-04-22 18:01:15,574 [INFO] tensorflow: epoch = 6.705, learning_rate = 0.000109640234, loss = 0.00011264731, step = 4023 (5.090 sec) 2022-04-22 18:01:15,574 [INFO] tensorflow: epoch = 6.705, learning_rate = 0.000109640234, loss = 0.00011219685, step = 4023 (5.090 sec) 2022-04-22 18:01:15,646 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.876 2022-04-22 18:01:17,453 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.327 2022-04-22 18:01:19,258 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.429 INFO:tensorflow:global_step/sec: 13.7431 2022-04-22 18:01:19,734 [INFO] tensorflow: global_step/sec: 13.7431 INFO:tensorflow:epoch = 6.821666666666667, learning_rate = 0.000115692004, loss = 0.000102826634, step = 4093 (5.087 sec) 2022-04-22 18:01:20,661 [INFO] tensorflow: epoch = 6.821666666666667, learning_rate = 0.000115692004, loss = 0.000102826634, step = 4093 (5.087 sec) INFO:tensorflow:epoch = 6.821666666666667, learning_rate = 0.000115692004, loss = 0.00010982267, step = 4093 (5.087 sec) 2022-04-22 18:01:20,661 [INFO] tensorflow: epoch = 6.821666666666667, learning_rate = 0.000115692004, loss = 0.00010982267, step = 4093 (5.087 sec) 2022-04-22 18:01:21,128 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.470 2022-04-22 18:01:23,061 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 51.739 INFO:tensorflow:global_step/sec: 13.3219 2022-04-22 18:01:24,238 [INFO] tensorflow: global_step/sec: 13.3219 2022-04-22 18:01:24,850 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.903 INFO:tensorflow:epoch = 6.936666666666667, learning_rate = 0.000121984136, loss = 9.5754454e-05, step = 4162 (5.113 sec) INFO:tensorflow:epoch = 6.936666666666667, learning_rate = 0.000121984136, loss = 9.577615e-05, step = 4162 (5.112 sec) 2022-04-22 18:01:25,773 [INFO] tensorflow: epoch = 6.936666666666667, learning_rate = 0.000121984136, loss = 9.5754454e-05, step = 4162 (5.113 sec) 2022-04-22 18:01:25,773 [INFO] tensorflow: epoch = 6.936666666666667, learning_rate = 0.000121984136, loss = 9.577615e-05, step = 4162 (5.112 sec) 2022-04-22 18:01:26,621 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.485 2022-04-22 18:01:28,600 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 50.519 INFO:tensorflow:global_step/sec: 13.5302 2022-04-22 18:01:28,673 [INFO] tensorflow: global_step/sec: 13.5302 2022-04-22 18:01:28,673 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 7/100: loss: 0.00009 learning rate: 0.00013 Time taken: 0:00:43.775483 ETA: 1:07:51.119887 2022-04-22 18:01:30,560 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 51.047 INFO:tensorflow:epoch = 7.046666666666667, learning_rate = 0.0001283227, loss = 8.828945e-05, step = 4228 (5.077 sec) INFO:tensorflow:epoch = 7.046666666666667, learning_rate = 0.0001283227, loss = 8.803836e-05, step = 4228 (5.077 sec) 2022-04-22 18:01:30,850 [INFO] tensorflow: epoch = 7.046666666666667, learning_rate = 0.0001283227, loss = 8.828945e-05, step = 4228 (5.077 sec) 2022-04-22 18:01:30,850 [INFO] tensorflow: epoch = 7.046666666666667, learning_rate = 0.0001283227, loss = 8.803836e-05, step = 4228 (5.077 sec) 2022-04-22 18:01:32,361 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.521 INFO:tensorflow:global_step/sec: 13.1528 2022-04-22 18:01:33,235 [INFO] tensorflow: global_step/sec: 13.1528 2022-04-22 18:01:34,282 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 52.072 INFO:tensorflow:epoch = 7.158333333333334, learning_rate = 0.00013509419, loss = 8.198329e-05, step = 4295 (5.098 sec) 2022-04-22 18:01:35,948 [INFO] tensorflow: epoch = 7.158333333333334, learning_rate = 0.00013509419, loss = 8.198329e-05, step = 4295 (5.098 sec) INFO:tensorflow:epoch = 7.158333333333334, learning_rate = 0.00013509419, loss = 8.204325e-05, step = 4295 (5.108 sec) 2022-04-22 18:01:35,958 [INFO] tensorflow: epoch = 7.158333333333334, learning_rate = 0.00013509419, loss = 8.204325e-05, step = 4295 (5.108 sec) 2022-04-22 18:01:36,238 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 51.130 INFO:tensorflow:global_step/sec: 13.4622 2022-04-22 18:01:37,691 [INFO] tensorflow: global_step/sec: 13.4622 2022-04-22 18:01:37,977 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 57.525 2022-04-22 18:01:39,810 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.553 INFO:tensorflow:epoch = 7.275, learning_rate = 0.00014255094, loss = 7.7681616e-05, step = 4365 (5.086 sec) INFO:tensorflow:epoch = 7.275, learning_rate = 0.00014255094, loss = 7.545947e-05, step = 4365 (5.077 sec) 2022-04-22 18:01:41,035 [INFO] tensorflow: epoch = 7.275, learning_rate = 0.00014255094, loss = 7.7681616e-05, step = 4365 (5.086 sec) 2022-04-22 18:01:41,035 [INFO] tensorflow: epoch = 7.275, learning_rate = 0.00014255094, loss = 7.545947e-05, step = 4365 (5.077 sec) 2022-04-22 18:01:41,760 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 51.294 INFO:tensorflow:global_step/sec: 13.2422 2022-04-22 18:01:42,222 [INFO] tensorflow: global_step/sec: 13.2422 2022-04-22 18:01:43,556 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.681 2022-04-22 18:01:45,463 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 52.440 INFO:tensorflow:epoch = 7.386666666666667, learning_rate = 0.00015007325, loss = 7.1221766e-05, step = 4432 (5.120 sec) 2022-04-22 18:01:46,155 [INFO] tensorflow: epoch = 7.386666666666667, learning_rate = 0.00015007325, loss = 7.1221766e-05, step = 4432 (5.120 sec) INFO:tensorflow:epoch = 7.386666666666667, learning_rate = 0.00015007325, loss = 7.2234514e-05, step = 4432 (5.121 sec) 2022-04-22 18:01:46,156 [INFO] tensorflow: epoch = 7.386666666666667, learning_rate = 0.00015007325, loss = 7.2234514e-05, step = 4432 (5.121 sec) INFO:tensorflow:global_step/sec: 13.097 2022-04-22 18:01:46,804 [INFO] tensorflow: global_step/sec: 13.097 2022-04-22 18:01:47,466 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 49.940 2022-04-22 18:01:49,232 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.654 2022-04-22 18:01:51,150 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 52.158 INFO:tensorflow:global_step/sec: 13.5751 2022-04-22 18:01:51,223 [INFO] tensorflow: global_step/sec: 13.5751 INFO:tensorflow:epoch = 7.501666666666667, learning_rate = 0.00015823526, loss = 6.494456e-05, step = 4501 (5.134 sec) 2022-04-22 18:01:51,289 [INFO] tensorflow: epoch = 7.501666666666667, learning_rate = 0.00015823526, loss = 6.494456e-05, step = 4501 (5.134 sec) INFO:tensorflow:epoch = 7.501666666666667, learning_rate = 0.00015823526, loss = 6.575341e-05, step = 4501 (5.134 sec) 2022-04-22 18:01:51,290 [INFO] tensorflow: epoch = 7.501666666666667, learning_rate = 0.00015823526, loss = 6.575341e-05, step = 4501 (5.134 sec) 2022-04-22 18:01:52,927 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.277 2022-04-22 18:01:54,729 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.504 INFO:tensorflow:global_step/sec: 13.9647 2022-04-22 18:01:55,520 [INFO] tensorflow: global_step/sec: 13.9647 INFO:tensorflow:epoch = 7.62, learning_rate = 0.00016709755, loss = 6.0355127e-05, step = 4572 (5.084 sec) 2022-04-22 18:01:56,373 [INFO] tensorflow: epoch = 7.62, learning_rate = 0.00016709755, loss = 6.0355127e-05, step = 4572 (5.084 sec) INFO:tensorflow:epoch = 7.62, learning_rate = 0.00016709755, loss = 6.0342674e-05, step = 4572 (5.086 sec) 2022-04-22 18:01:56,374 [INFO] tensorflow: epoch = 7.62, learning_rate = 0.00016709755, loss = 6.0342674e-05, step = 4572 (5.086 sec) 2022-04-22 18:01:56,522 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.756 2022-04-22 18:01:58,383 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.749 INFO:tensorflow:global_step/sec: 13.7479 2022-04-22 18:01:59,884 [INFO] tensorflow: global_step/sec: 13.7479 2022-04-22 18:02:00,172 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.917 INFO:tensorflow:epoch = 7.736666666666667, learning_rate = 0.0001763206, loss = 5.6533903e-05, step = 4642 (5.119 sec) 2022-04-22 18:02:01,494 [INFO] tensorflow: epoch = 7.736666666666667, learning_rate = 0.0001763206, loss = 5.6533903e-05, step = 4642 (5.119 sec) INFO:tensorflow:epoch = 7.736666666666667, learning_rate = 0.0001763206, loss = 5.714028e-05, step = 4642 (5.121 sec) 2022-04-22 18:02:01,494 [INFO] tensorflow: epoch = 7.736666666666667, learning_rate = 0.0001763206, loss = 5.714028e-05, step = 4642 (5.121 sec) 2022-04-22 18:02:02,001 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.664 2022-04-22 18:02:03,855 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.942 INFO:tensorflow:global_step/sec: 13.6003 2022-04-22 18:02:04,296 [INFO] tensorflow: global_step/sec: 13.6003 2022-04-22 18:02:05,764 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 52.384 INFO:tensorflow:epoch = 7.8500000000000005, learning_rate = 0.00018576764, loss = 5.286043e-05, step = 4710 (5.125 sec) 2022-04-22 18:02:06,619 [INFO] tensorflow: epoch = 7.8500000000000005, learning_rate = 0.00018576764, loss = 5.286043e-05, step = 4710 (5.125 sec) INFO:tensorflow:epoch = 7.8500000000000005, learning_rate = 0.00018576764, loss = 5.2451815e-05, step = 4710 (5.126 sec) 2022-04-22 18:02:06,620 [INFO] tensorflow: epoch = 7.8500000000000005, learning_rate = 0.00018576764, loss = 5.2451815e-05, step = 4710 (5.126 sec) 2022-04-22 18:02:07,599 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.522 INFO:tensorflow:global_step/sec: 13.6464 2022-04-22 18:02:08,693 [INFO] tensorflow: global_step/sec: 13.6464 2022-04-22 18:02:09,315 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 58.284 2022-04-22 18:02:11,107 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.799 INFO:tensorflow:epoch = 7.970000000000001, learning_rate = 0.0001963224, loss = 4.8776274e-05, step = 4782 (5.084 sec) 2022-04-22 18:02:11,704 [INFO] tensorflow: epoch = 7.970000000000001, learning_rate = 0.0001963224, loss = 4.8776274e-05, step = 4782 (5.084 sec) INFO:tensorflow:epoch = 7.970000000000001, learning_rate = 0.0001963224, loss = 5.03274e-05, step = 4782 (5.085 sec) 2022-04-22 18:02:11,704 [INFO] tensorflow: epoch = 7.970000000000001, learning_rate = 0.0001963224, loss = 5.03274e-05, step = 4782 (5.085 sec) 2022-04-22 18:02:12,965 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.838 INFO:tensorflow:global_step/sec: 13.8059 2022-04-22 18:02:13,039 [INFO] tensorflow: global_step/sec: 13.8059 2022-04-22 18:02:13,039 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 8/100: loss: 0.00005 learning rate: 0.00020 Time taken: 0:00:44.363604 ETA: 1:08:01.451552 2022-04-22 18:02:14,804 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.392 INFO:tensorflow:epoch = 8.081666666666667, learning_rate = 0.0002066824, loss = 4.5607347e-05, step = 4849 (5.127 sec) 2022-04-22 18:02:16,831 [INFO] tensorflow: epoch = 8.081666666666667, learning_rate = 0.0002066824, loss = 4.5607347e-05, step = 4849 (5.127 sec) INFO:tensorflow:epoch = 8.081666666666667, learning_rate = 0.0002066824, loss = 4.6240813e-05, step = 4849 (5.127 sec) 2022-04-22 18:02:16,831 [INFO] tensorflow: epoch = 8.081666666666667, learning_rate = 0.0002066824, loss = 4.6240813e-05, step = 4849 (5.127 sec) 2022-04-22 18:02:16,831 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 49.338 INFO:tensorflow:global_step/sec: 13.0845 2022-04-22 18:02:17,624 [INFO] tensorflow: global_step/sec: 13.0845 2022-04-22 18:02:18,675 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.242 2022-04-22 18:02:20,529 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.940 INFO:tensorflow:epoch = 8.196666666666667, learning_rate = 0.00021792321, loss = 4.2552252e-05, step = 4918 (5.089 sec) 2022-04-22 18:02:21,919 [INFO] tensorflow: epoch = 8.196666666666667, learning_rate = 0.00021792321, loss = 4.2552252e-05, step = 4918 (5.089 sec) INFO:tensorflow:epoch = 8.196666666666667, learning_rate = 0.00021792321, loss = 4.2545158e-05, step = 4918 (5.088 sec) 2022-04-22 18:02:21,919 [INFO] tensorflow: epoch = 8.196666666666667, learning_rate = 0.00021792321, loss = 4.2545158e-05, step = 4918 (5.088 sec) INFO:tensorflow:global_step/sec: 13.5389 2022-04-22 18:02:22,056 [INFO] tensorflow: global_step/sec: 13.5389 2022-04-22 18:02:22,330 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.551 2022-04-22 18:02:24,054 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 57.994 2022-04-22 18:02:25,897 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.274 INFO:tensorflow:global_step/sec: 14.1198 2022-04-22 18:02:26,305 [INFO] tensorflow: global_step/sec: 14.1198 INFO:tensorflow:epoch = 8.315000000000001, learning_rate = 0.00023012821, loss = 4.041129e-05, step = 4989 (5.096 sec) 2022-04-22 18:02:27,015 [INFO] tensorflow: epoch = 8.315000000000001, learning_rate = 0.00023012821, loss = 4.041129e-05, step = 4989 (5.096 sec) INFO:tensorflow:epoch = 8.315000000000001, learning_rate = 0.00023012821, loss = 4.088147e-05, step = 4989 (5.096 sec) 2022-04-22 18:02:27,015 [INFO] tensorflow: epoch = 8.315000000000001, learning_rate = 0.00023012821, loss = 4.088147e-05, step = 4989 (5.096 sec) 2022-04-22 18:02:27,737 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.343 2022-04-22 18:02:29,502 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.689 INFO:tensorflow:global_step/sec: 13.6956 2022-04-22 18:02:30,686 [INFO] tensorflow: global_step/sec: 13.6956 2022-04-22 18:02:31,330 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.695 INFO:tensorflow:epoch = 8.433333333333334, learning_rate = 0.00024301701, loss = 3.76614e-05, step = 5060 (5.125 sec) 2022-04-22 18:02:32,140 [INFO] tensorflow: epoch = 8.433333333333334, learning_rate = 0.00024301701, loss = 3.76614e-05, step = 5060 (5.125 sec) INFO:tensorflow:epoch = 8.433333333333334, learning_rate = 0.00024301701, loss = 3.7493242e-05, step = 5060 (5.126 sec) 2022-04-22 18:02:32,141 [INFO] tensorflow: epoch = 8.433333333333334, learning_rate = 0.00024301701, loss = 3.7493242e-05, step = 5060 (5.126 sec) 2022-04-22 18:02:33,152 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.905 2022-04-22 18:02:34,965 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.155 INFO:tensorflow:global_step/sec: 13.7809 2022-04-22 18:02:35,040 [INFO] tensorflow: global_step/sec: 13.7809 2022-04-22 18:02:36,821 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.899 INFO:tensorflow:epoch = 8.55, learning_rate = 0.00025643074, loss = 3.524227e-05, step = 5130 (5.097 sec) 2022-04-22 18:02:37,238 [INFO] tensorflow: epoch = 8.55, learning_rate = 0.00025643074, loss = 3.524227e-05, step = 5130 (5.097 sec) INFO:tensorflow:epoch = 8.55, learning_rate = 0.00025643074, loss = 3.5974044e-05, step = 5130 (5.099 sec) 2022-04-22 18:02:37,239 [INFO] tensorflow: epoch = 8.55, learning_rate = 0.00025643074, loss = 3.5974044e-05, step = 5130 (5.099 sec) 2022-04-22 18:02:38,583 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.731 INFO:tensorflow:global_step/sec: 13.871 2022-04-22 18:02:39,366 [INFO] tensorflow: global_step/sec: 13.871 2022-04-22 18:02:40,456 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.416 2022-04-22 18:02:42,262 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.380 INFO:tensorflow:epoch = 8.666666666666668, learning_rate = 0.0002705846, loss = 3.288552e-05, step = 5200 (5.097 sec) 2022-04-22 18:02:42,336 [INFO] tensorflow: epoch = 8.666666666666668, learning_rate = 0.0002705846, loss = 3.288552e-05, step = 5200 (5.097 sec) INFO:tensorflow:epoch = 8.666666666666668, learning_rate = 0.0002705846, loss = 3.3155244e-05, step = 5200 (5.098 sec) 2022-04-22 18:02:42,336 [INFO] tensorflow: epoch = 8.666666666666668, learning_rate = 0.0002705846, loss = 3.3155244e-05, step = 5200 (5.098 sec) INFO:tensorflow:global_step/sec: 13.2701 2022-04-22 18:02:43,887 [INFO] tensorflow: global_step/sec: 13.2701 2022-04-22 18:02:44,163 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 52.593 2022-04-22 18:02:45,940 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.297 INFO:tensorflow:epoch = 8.781666666666668, learning_rate = 0.00028530083, loss = 3.1369258e-05, step = 5269 (5.157 sec) 2022-04-22 18:02:47,493 [INFO] tensorflow: epoch = 8.781666666666668, learning_rate = 0.00028530083, loss = 3.1369258e-05, step = 5269 (5.157 sec) INFO:tensorflow:epoch = 8.781666666666668, learning_rate = 0.00028530083, loss = 3.133208e-05, step = 5269 (5.158 sec) 2022-04-22 18:02:47,494 [INFO] tensorflow: epoch = 8.781666666666668, learning_rate = 0.00028530083, loss = 3.133208e-05, step = 5269 (5.158 sec) 2022-04-22 18:02:47,944 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 49.913 INFO:tensorflow:global_step/sec: 13.3686 2022-04-22 18:02:48,375 [INFO] tensorflow: global_step/sec: 13.3686 2022-04-22 18:02:49,840 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 52.731 2022-04-22 18:02:51,722 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.155 INFO:tensorflow:epoch = 8.895000000000001, learning_rate = 0.00030058692, loss = 2.9313986e-05, step = 5337 (5.133 sec) 2022-04-22 18:02:52,626 [INFO] tensorflow: epoch = 8.895000000000001, learning_rate = 0.00030058692, loss = 2.9313986e-05, step = 5337 (5.133 sec) INFO:tensorflow:epoch = 8.895000000000001, learning_rate = 0.00030058692, loss = 2.9268846e-05, step = 5337 (5.134 sec) 2022-04-22 18:02:52,627 [INFO] tensorflow: epoch = 8.895000000000001, learning_rate = 0.00030058692, loss = 2.9268846e-05, step = 5337 (5.134 sec) INFO:tensorflow:global_step/sec: 13.4962 2022-04-22 18:02:52,821 [INFO] tensorflow: global_step/sec: 13.4962 2022-04-22 18:02:53,476 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 57.004 2022-04-22 18:02:55,364 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 52.995 2022-04-22 18:02:57,236 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.421 INFO:tensorflow:global_step/sec: 13.3763 2022-04-22 18:02:57,307 [INFO] tensorflow: global_step/sec: 13.3763 2022-04-22 18:02:57,307 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 9/100: loss: 0.00003 learning rate: 0.00032 Time taken: 0:00:44.270721 ETA: 1:07:08.635651 INFO:tensorflow:epoch = 9.01, learning_rate = 0.00031693486, loss = 2.76238e-05, step = 5406 (5.096 sec) INFO:tensorflow:epoch = 9.01, learning_rate = 0.00031693486, loss = 2.7864251e-05, step = 5406 (5.096 sec) 2022-04-22 18:02:57,723 [INFO] tensorflow: epoch = 9.01, learning_rate = 0.00031693486, loss = 2.76238e-05, step = 5406 (5.096 sec) 2022-04-22 18:02:57,723 [INFO] tensorflow: epoch = 9.01, learning_rate = 0.00031693486, loss = 2.7864251e-05, step = 5406 (5.096 sec) 2022-04-22 18:02:59,009 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.430 2022-04-22 18:03:00,887 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.255 INFO:tensorflow:global_step/sec: 13.5531 2022-04-22 18:03:01,734 [INFO] tensorflow: global_step/sec: 13.5531 INFO:tensorflow:epoch = 9.123333333333333, learning_rate = 0.00033391555, loss = 2.61522e-05, step = 5474 (5.082 sec) 2022-04-22 18:03:02,805 [INFO] tensorflow: epoch = 9.123333333333333, learning_rate = 0.00033391555, loss = 2.61522e-05, step = 5474 (5.082 sec) INFO:tensorflow:epoch = 9.123333333333333, learning_rate = 0.00033391555, loss = 2.6664407e-05, step = 5474 (5.105 sec) 2022-04-22 18:03:02,828 [INFO] tensorflow: epoch = 9.123333333333333, learning_rate = 0.00033391555, loss = 2.6664407e-05, step = 5474 (5.105 sec) 2022-04-22 18:03:02,828 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 51.510 2022-04-22 18:03:04,633 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.428 INFO:tensorflow:global_step/sec: 13.4023 2022-04-22 18:03:06,210 [INFO] tensorflow: global_step/sec: 13.4023 2022-04-22 18:03:06,499 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.588 INFO:tensorflow:epoch = 9.238333333333333, learning_rate = 0.00035207617, loss = 2.4938825e-05, step = 5543 (5.095 sec) 2022-04-22 18:03:07,899 [INFO] tensorflow: epoch = 9.238333333333333, learning_rate = 0.00035207617, loss = 2.4938825e-05, step = 5543 (5.095 sec) INFO:tensorflow:epoch = 9.24, learning_rate = 0.00035234657, loss = 2.4856521e-05, step = 5544 (5.148 sec) 2022-04-22 18:03:07,976 [INFO] tensorflow: epoch = 9.24, learning_rate = 0.00035234657, loss = 2.4856521e-05, step = 5544 (5.148 sec) 2022-04-22 18:03:08,345 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.172 2022-04-22 18:03:10,144 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.592 INFO:tensorflow:global_step/sec: 13.7213 2022-04-22 18:03:10,583 [INFO] tensorflow: global_step/sec: 13.7213 2022-04-22 18:03:12,033 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 52.952 INFO:tensorflow:epoch = 9.353333333333333, learning_rate = 0.00037122448, loss = 2.3576555e-05, step = 5612 (5.083 sec) 2022-04-22 18:03:12,982 [INFO] tensorflow: epoch = 9.353333333333333, learning_rate = 0.00037122448, loss = 2.3576555e-05, step = 5612 (5.083 sec) INFO:tensorflow:epoch = 9.355, learning_rate = 0.00037150958, loss = 2.3502787e-05, step = 5613 (5.079 sec) 2022-04-22 18:03:13,055 [INFO] tensorflow: epoch = 9.355, learning_rate = 0.00037150958, loss = 2.3502787e-05, step = 5613 (5.079 sec) 2022-04-22 18:03:13,866 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.568 INFO:tensorflow:global_step/sec: 13.468 2022-04-22 18:03:15,038 [INFO] tensorflow: global_step/sec: 13.468 2022-04-22 18:03:15,664 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.632 2022-04-22 18:03:17,502 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.407 INFO:tensorflow:epoch = 9.47, learning_rate = 0.00039171486, loss = 2.2459966e-05, step = 5682 (5.131 sec) 2022-04-22 18:03:18,113 [INFO] tensorflow: epoch = 9.47, learning_rate = 0.00039171486, loss = 2.2459966e-05, step = 5682 (5.131 sec) INFO:tensorflow:epoch = 9.471666666666668, learning_rate = 0.0003920155, loss = 2.2303891e-05, step = 5683 (5.128 sec) 2022-04-22 18:03:18,183 [INFO] tensorflow: epoch = 9.471666666666668, learning_rate = 0.0003920155, loss = 2.2303891e-05, step = 5683 (5.128 sec) 2022-04-22 18:03:19,518 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 49.617 INFO:tensorflow:global_step/sec: 13.1881 2022-04-22 18:03:19,588 [INFO] tensorflow: global_step/sec: 13.1881 2022-04-22 18:03:21,371 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.977 INFO:tensorflow:epoch = 9.58, learning_rate = 0.00041206885, loss = 2.0489124e-05, step = 5748 (5.122 sec) 2022-04-22 18:03:23,235 [INFO] tensorflow: epoch = 9.58, learning_rate = 0.00041206885, loss = 2.0489124e-05, step = 5748 (5.122 sec) INFO:tensorflow:epoch = 9.581666666666667, learning_rate = 0.0004123853, loss = 2.0970694e-05, step = 5749 (5.154 sec) 2022-04-22 18:03:23,336 [INFO] tensorflow: epoch = 9.581666666666667, learning_rate = 0.0004123853, loss = 2.0970694e-05, step = 5749 (5.154 sec) 2022-04-22 18:03:23,337 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 50.870 INFO:tensorflow:global_step/sec: 12.7739 2022-04-22 18:03:24,285 [INFO] tensorflow: global_step/sec: 12.7739 2022-04-22 18:03:25,331 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 50.151 2022-04-22 18:03:27,270 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 51.573 INFO:tensorflow:epoch = 9.688333333333334, learning_rate = 0.00043314823, loss = 1.9756848e-05, step = 5813 (5.117 sec) 2022-04-22 18:03:28,352 [INFO] tensorflow: epoch = 9.688333333333334, learning_rate = 0.00043314823, loss = 1.9756848e-05, step = 5813 (5.117 sec) INFO:tensorflow:epoch = 9.690000000000001, learning_rate = 0.00043348092, loss = 1.9228075e-05, step = 5814 (5.095 sec) 2022-04-22 18:03:28,431 [INFO] tensorflow: epoch = 9.690000000000001, learning_rate = 0.00043348092, loss = 1.9228075e-05, step = 5814 (5.095 sec) INFO:tensorflow:global_step/sec: 12.7657 2022-04-22 18:03:28,985 [INFO] tensorflow: global_step/sec: 12.7657 2022-04-22 18:03:29,330 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 48.546 2022-04-22 18:03:31,183 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.987 2022-04-22 18:03:32,925 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 57.409 INFO:tensorflow:global_step/sec: 13.7851 2022-04-22 18:03:33,337 [INFO] tensorflow: global_step/sec: 13.7851 INFO:tensorflow:epoch = 9.803333333333335, learning_rate = 0.0004567058, loss = 1.8086243e-05, step = 5882 (5.122 sec) 2022-04-22 18:03:33,474 [INFO] tensorflow: epoch = 9.803333333333335, learning_rate = 0.0004567058, loss = 1.8086243e-05, step = 5882 (5.122 sec) INFO:tensorflow:epoch = 9.805000000000001, learning_rate = 0.00045705657, loss = 1.7964405e-05, step = 5883 (5.115 sec) 2022-04-22 18:03:33,546 [INFO] tensorflow: epoch = 9.805000000000001, learning_rate = 0.00045705657, loss = 1.7964405e-05, step = 5883 (5.115 sec) 2022-04-22 18:03:34,787 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.712 2022-04-22 18:03:36,908 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 47.153 INFO:tensorflow:global_step/sec: 12.6537 2022-04-22 18:03:38,079 [INFO] tensorflow: global_step/sec: 12.6537 INFO:tensorflow:epoch = 9.911666666666667, learning_rate = 0.00048006861, loss = 1.697526e-05, step = 5947 (5.114 sec) 2022-04-22 18:03:38,588 [INFO] tensorflow: epoch = 9.911666666666667, learning_rate = 0.00048006861, loss = 1.697526e-05, step = 5947 (5.114 sec) INFO:tensorflow:epoch = 9.913333333333334, learning_rate = 0.0004804373, loss = 1.7147126e-05, step = 5948 (5.111 sec) 2022-04-22 18:03:38,658 [INFO] tensorflow: epoch = 9.913333333333334, learning_rate = 0.0004804373, loss = 1.7147126e-05, step = 5948 (5.111 sec) 2022-04-22 18:03:38,725 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.045 2022-04-22 18:03:40,622 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 52.727 INFO:tensorflow:Saving checkpoints for step-6000. 2022-04-22 18:03:42,452 [INFO] tensorflow: Saving checkpoints for step-6000. 2022-04-22 18:03:47,865 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 13.807 INFO:tensorflow:epoch = 10.0, learning_rate = 0.00049999997, loss = 1.658898e-05, step = 6000 (9.353 sec) 2022-04-22 18:03:47,941 [INFO] tensorflow: epoch = 10.0, learning_rate = 0.00049999997, loss = 1.658898e-05, step = 6000 (9.353 sec) INFO:tensorflow:epoch = 10.0, learning_rate = 0.00049999997, loss = 1.6313992e-05, step = 6000 (9.283 sec) 2022-04-22 18:03:47,941 [INFO] tensorflow: epoch = 10.0, learning_rate = 0.00049999997, loss = 1.6313992e-05, step = 6000 (9.283 sec) INFO:tensorflow:global_step/sec: 6.08359 2022-04-22 18:03:47,942 [INFO] tensorflow: global_step/sec: 6.08359 2022-04-22 18:03:47,943 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 10/100: loss: 0.00002 learning rate: 0.00050 Time taken: 0:00:50.628206 ETA: 1:15:56.538541 2022-04-22 18:03:49,646 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.184 2022-04-22 18:03:51,457 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.236 INFO:tensorflow:global_step/sec: 13.7405 2022-04-22 18:03:52,308 [INFO] tensorflow: global_step/sec: 13.7405 INFO:tensorflow:epoch = 10.118333333333334, learning_rate = 0.00049999997, loss = 1.5303267e-05, step = 6071 (5.122 sec) INFO:tensorflow:epoch = 10.118333333333334, learning_rate = 0.00049999997, loss = 1.5243442e-05, step = 6071 (5.121 sec) 2022-04-22 18:03:53,062 [INFO] tensorflow: epoch = 10.118333333333334, learning_rate = 0.00049999997, loss = 1.5303267e-05, step = 6071 (5.122 sec) 2022-04-22 18:03:53,062 [INFO] tensorflow: epoch = 10.118333333333334, learning_rate = 0.00049999997, loss = 1.5243442e-05, step = 6071 (5.121 sec) 2022-04-22 18:03:53,277 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.931 2022-04-22 18:03:55,016 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 57.514 INFO:tensorflow:global_step/sec: 13.9755 2022-04-22 18:03:56,602 [INFO] tensorflow: global_step/sec: 13.9755 2022-04-22 18:03:56,894 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.266 INFO:tensorflow:epoch = 10.235000000000001, learning_rate = 0.00049999997, loss = 1.4424215e-05, step = 6141 (5.089 sec) 2022-04-22 18:03:58,151 [INFO] tensorflow: epoch = 10.235000000000001, learning_rate = 0.00049999997, loss = 1.4424215e-05, step = 6141 (5.089 sec) INFO:tensorflow:epoch = 10.235000000000001, learning_rate = 0.00049999997, loss = 1.4380725e-05, step = 6141 (5.089 sec) 2022-04-22 18:03:58,152 [INFO] tensorflow: epoch = 10.235000000000001, learning_rate = 0.00049999997, loss = 1.4380725e-05, step = 6141 (5.089 sec) 2022-04-22 18:03:58,890 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 50.100 2022-04-22 18:04:00,687 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.646 INFO:tensorflow:global_step/sec: 13.2291 2022-04-22 18:04:01,137 [INFO] tensorflow: global_step/sec: 13.2291 2022-04-22 18:04:02,576 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 52.945 INFO:tensorflow:epoch = 10.346666666666668, learning_rate = 0.00049999997, loss = 1.3703328e-05, step = 6208 (5.087 sec) INFO:tensorflow:epoch = 10.346666666666668, learning_rate = 0.00049999997, loss = 1.3705105e-05, step = 6208 (5.087 sec) 2022-04-22 18:04:03,239 [INFO] tensorflow: epoch = 10.346666666666668, learning_rate = 0.00049999997, loss = 1.3703328e-05, step = 6208 (5.087 sec) 2022-04-22 18:04:03,239 [INFO] tensorflow: epoch = 10.346666666666668, learning_rate = 0.00049999997, loss = 1.3705105e-05, step = 6208 (5.087 sec) 2022-04-22 18:04:04,502 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 51.937 INFO:tensorflow:global_step/sec: 13.3736 2022-04-22 18:04:05,624 [INFO] tensorflow: global_step/sec: 13.3736 2022-04-22 18:04:06,298 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.675 2022-04-22 18:04:08,163 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.628 INFO:tensorflow:epoch = 10.461666666666668, learning_rate = 0.00049999997, loss = 1.32106015e-05, step = 6277 (5.136 sec) 2022-04-22 18:04:08,375 [INFO] tensorflow: epoch = 10.461666666666668, learning_rate = 0.00049999997, loss = 1.32106015e-05, step = 6277 (5.136 sec) INFO:tensorflow:epoch = 10.461666666666668, learning_rate = 0.00049999997, loss = 1.3185015e-05, step = 6277 (5.137 sec) 2022-04-22 18:04:08,375 [INFO] tensorflow: epoch = 10.461666666666668, learning_rate = 0.00049999997, loss = 1.3185015e-05, step = 6277 (5.137 sec) 2022-04-22 18:04:09,999 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.472 INFO:tensorflow:global_step/sec: 13.5033 2022-04-22 18:04:10,067 [INFO] tensorflow: global_step/sec: 13.5033 2022-04-22 18:04:11,786 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.981 INFO:tensorflow:epoch = 10.578333333333333, learning_rate = 0.00049999997, loss = 1.2527104e-05, step = 6347 (5.072 sec) 2022-04-22 18:04:13,447 [INFO] tensorflow: epoch = 10.578333333333333, learning_rate = 0.00049999997, loss = 1.2527104e-05, step = 6347 (5.072 sec) INFO:tensorflow:epoch = 10.58, learning_rate = 0.00049999997, loss = 1.2554589e-05, step = 6348 (5.137 sec) 2022-04-22 18:04:13,511 [INFO] tensorflow: epoch = 10.58, learning_rate = 0.00049999997, loss = 1.2554589e-05, step = 6348 (5.137 sec) 2022-04-22 18:04:13,584 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.610 INFO:tensorflow:global_step/sec: 13.8685 2022-04-22 18:04:14,393 [INFO] tensorflow: global_step/sec: 13.8685 2022-04-22 18:04:15,414 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.672 2022-04-22 18:04:17,205 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.835 INFO:tensorflow:epoch = 10.696666666666667, learning_rate = 0.00049999997, loss = 1.2093553e-05, step = 6418 (5.137 sec) 2022-04-22 18:04:18,584 [INFO] tensorflow: epoch = 10.696666666666667, learning_rate = 0.00049999997, loss = 1.2093553e-05, step = 6418 (5.137 sec) INFO:tensorflow:epoch = 10.696666666666667, learning_rate = 0.00049999997, loss = 1.21062385e-05, step = 6418 (5.073 sec) 2022-04-22 18:04:18,584 [INFO] tensorflow: epoch = 10.696666666666667, learning_rate = 0.00049999997, loss = 1.21062385e-05, step = 6418 (5.073 sec) INFO:tensorflow:global_step/sec: 13.7616 2022-04-22 18:04:18,753 [INFO] tensorflow: global_step/sec: 13.7616 2022-04-22 18:04:19,042 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.440 2022-04-22 18:04:20,872 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.663 2022-04-22 18:04:22,781 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 52.380 INFO:tensorflow:global_step/sec: 13.431 2022-04-22 18:04:23,220 [INFO] tensorflow: global_step/sec: 13.431 INFO:tensorflow:epoch = 10.811666666666667, learning_rate = 0.00049999997, loss = 1.1597882e-05, step = 6487 (5.139 sec) INFO:tensorflow:epoch = 10.811666666666667, learning_rate = 0.00049999997, loss = 1.1615783e-05, step = 6487 (5.139 sec) 2022-04-22 18:04:23,723 [INFO] tensorflow: epoch = 10.811666666666667, learning_rate = 0.00049999997, loss = 1.1597882e-05, step = 6487 (5.139 sec) 2022-04-22 18:04:23,723 [INFO] tensorflow: epoch = 10.811666666666667, learning_rate = 0.00049999997, loss = 1.1615783e-05, step = 6487 (5.139 sec) 2022-04-22 18:04:24,581 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.551 2022-04-22 18:04:26,401 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.974 INFO:tensorflow:global_step/sec: 13.8363 2022-04-22 18:04:27,557 [INFO] tensorflow: global_step/sec: 13.8363 2022-04-22 18:04:28,180 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.199 INFO:tensorflow:epoch = 10.930000000000001, learning_rate = 0.00049999997, loss = 1.1338188e-05, step = 6558 (5.163 sec) 2022-04-22 18:04:28,887 [INFO] tensorflow: epoch = 10.930000000000001, learning_rate = 0.00049999997, loss = 1.1338188e-05, step = 6558 (5.163 sec) INFO:tensorflow:epoch = 10.930000000000001, learning_rate = 0.00049999997, loss = 1.1297319e-05, step = 6558 (5.164 sec) 2022-04-22 18:04:28,887 [INFO] tensorflow: epoch = 10.930000000000001, learning_rate = 0.00049999997, loss = 1.1297319e-05, step = 6558 (5.164 sec) 2022-04-22 18:04:29,980 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.566 2022-04-22 18:04:31,791 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 299, 0.00s/step 2022-04-22 18:04:32,006 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 299, 0.02s/step 2022-04-22 18:04:32,221 [INFO] iva.detectnet_v2.evaluation.evaluation: step 20 / 299, 0.02s/step 2022-04-22 18:04:32,437 [INFO] iva.detectnet_v2.evaluation.evaluation: step 30 / 299, 0.02s/step 2022-04-22 18:04:32,653 [INFO] iva.detectnet_v2.evaluation.evaluation: step 40 / 299, 0.02s/step 2022-04-22 18:04:32,874 [INFO] iva.detectnet_v2.evaluation.evaluation: step 50 / 299, 0.02s/step 2022-04-22 18:04:33,089 [INFO] iva.detectnet_v2.evaluation.evaluation: step 60 / 299, 0.02s/step 2022-04-22 18:04:33,306 [INFO] iva.detectnet_v2.evaluation.evaluation: step 70 / 299, 0.02s/step 2022-04-22 18:04:33,526 [INFO] iva.detectnet_v2.evaluation.evaluation: step 80 / 299, 0.02s/step 2022-04-22 18:04:33,749 [INFO] iva.detectnet_v2.evaluation.evaluation: step 90 / 299, 0.02s/step 2022-04-22 18:04:33,975 [INFO] iva.detectnet_v2.evaluation.evaluation: step 100 / 299, 0.02s/step 2022-04-22 18:04:34,205 [INFO] iva.detectnet_v2.evaluation.evaluation: step 110 / 299, 0.02s/step 2022-04-22 18:04:34,431 [INFO] iva.detectnet_v2.evaluation.evaluation: step 120 / 299, 0.02s/step 2022-04-22 18:04:34,660 [INFO] iva.detectnet_v2.evaluation.evaluation: step 130 / 299, 0.02s/step 2022-04-22 18:04:34,888 [INFO] iva.detectnet_v2.evaluation.evaluation: step 140 / 299, 0.02s/step 2022-04-22 18:04:35,113 [INFO] iva.detectnet_v2.evaluation.evaluation: step 150 / 299, 0.02s/step 2022-04-22 18:04:35,348 [INFO] iva.detectnet_v2.evaluation.evaluation: step 160 / 299, 0.02s/step 2022-04-22 18:04:35,578 [INFO] iva.detectnet_v2.evaluation.evaluation: step 170 / 299, 0.02s/step 2022-04-22 18:04:35,815 [INFO] iva.detectnet_v2.evaluation.evaluation: step 180 / 299, 0.02s/step 2022-04-22 18:04:36,053 [INFO] iva.detectnet_v2.evaluation.evaluation: step 190 / 299, 0.02s/step 2022-04-22 18:04:36,289 [INFO] iva.detectnet_v2.evaluation.evaluation: step 200 / 299, 0.02s/step 2022-04-22 18:04:36,524 [INFO] iva.detectnet_v2.evaluation.evaluation: step 210 / 299, 0.02s/step 2022-04-22 18:04:36,762 [INFO] iva.detectnet_v2.evaluation.evaluation: step 220 / 299, 0.02s/step 2022-04-22 18:04:36,996 [INFO] iva.detectnet_v2.evaluation.evaluation: step 230 / 299, 0.02s/step 2022-04-22 18:04:37,231 [INFO] iva.detectnet_v2.evaluation.evaluation: step 240 / 299, 0.02s/step 2022-04-22 18:04:37,467 [INFO] iva.detectnet_v2.evaluation.evaluation: step 250 / 299, 0.02s/step 2022-04-22 18:04:37,703 [INFO] iva.detectnet_v2.evaluation.evaluation: step 260 / 299, 0.02s/step 2022-04-22 18:04:37,941 [INFO] iva.detectnet_v2.evaluation.evaluation: step 270 / 299, 0.02s/step 2022-04-22 18:04:38,179 [INFO] iva.detectnet_v2.evaluation.evaluation: step 280 / 299, 0.02s/step 2022-04-22 18:04:38,418 [INFO] iva.detectnet_v2.evaluation.evaluation: step 290 / 299, 0.02s/step Epoch 11/100 ========================= Validation cost: -0.000007 Mean average_precision (in %): 0.0000 class name average precision (in %) ------------ -------------------------- face 0 Median Inference Time: 0.011355 2022-04-22 18:04:38,662 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 11.518 INFO:tensorflow:epoch = 11.0, learning_rate = 0.00049999997, loss = 1.1017615e-05, step = 6600 (9.846 sec) 2022-04-22 18:04:38,733 [INFO] tensorflow: epoch = 11.0, learning_rate = 0.00049999997, loss = 1.1017615e-05, step = 6600 (9.846 sec) INFO:tensorflow:global_step/sec: 5.36813 2022-04-22 18:04:38,734 [INFO] tensorflow: global_step/sec: 5.36813 2022-04-22 18:04:38,735 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 11/100: loss: 0.00001 learning rate: 0.00050 Time taken: 0:00:50.796215 ETA: 1:15:20.863119 INFO:tensorflow:epoch = 11.001666666666667, learning_rate = 0.00049999997, loss = 1.113364e-05, step = 6601 (9.918 sec) 2022-04-22 18:04:38,805 [INFO] tensorflow: epoch = 11.001666666666667, learning_rate = 0.00049999997, loss = 1.113364e-05, step = 6601 (9.918 sec) 2022-04-22 18:04:40,466 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.455 2022-04-22 18:04:42,282 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.070 INFO:tensorflow:global_step/sec: 13.726 2022-04-22 18:04:43,105 [INFO] tensorflow: global_step/sec: 13.726 INFO:tensorflow:epoch = 11.116666666666667, learning_rate = 0.00049999997, loss = 1.07686255e-05, step = 6670 (5.120 sec) 2022-04-22 18:04:43,853 [INFO] tensorflow: epoch = 11.116666666666667, learning_rate = 0.00049999997, loss = 1.07686255e-05, step = 6670 (5.120 sec) INFO:tensorflow:epoch = 11.118333333333334, learning_rate = 0.00049999997, loss = 1.0646634e-05, step = 6671 (5.127 sec) 2022-04-22 18:04:43,932 [INFO] tensorflow: epoch = 11.118333333333334, learning_rate = 0.00049999997, loss = 1.0646634e-05, step = 6671 (5.127 sec) 2022-04-22 18:04:44,166 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.088 2022-04-22 18:04:46,125 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 51.047 INFO:tensorflow:global_step/sec: 13.275 2022-04-22 18:04:47,625 [INFO] tensorflow: global_step/sec: 13.275 2022-04-22 18:04:47,893 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.564 INFO:tensorflow:epoch = 11.23, learning_rate = 0.00049999997, loss = 1.0302072e-05, step = 6738 (5.066 sec) 2022-04-22 18:04:48,919 [INFO] tensorflow: epoch = 11.23, learning_rate = 0.00049999997, loss = 1.0302072e-05, step = 6738 (5.066 sec) INFO:tensorflow:epoch = 11.233333333333334, learning_rate = 0.00049999997, loss = 1.0527552e-05, step = 6740 (5.125 sec) 2022-04-22 18:04:49,056 [INFO] tensorflow: epoch = 11.233333333333334, learning_rate = 0.00049999997, loss = 1.0527552e-05, step = 6740 (5.125 sec) 2022-04-22 18:04:49,750 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.858 2022-04-22 18:04:51,561 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.235 INFO:tensorflow:global_step/sec: 13.5863 2022-04-22 18:04:52,041 [INFO] tensorflow: global_step/sec: 13.5863 2022-04-22 18:04:53,462 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 52.630 INFO:tensorflow:epoch = 11.343333333333334, learning_rate = 0.00049999997, loss = 9.985208e-06, step = 6806 (5.075 sec) 2022-04-22 18:04:53,995 [INFO] tensorflow: epoch = 11.343333333333334, learning_rate = 0.00049999997, loss = 9.985208e-06, step = 6806 (5.075 sec) INFO:tensorflow:epoch = 11.346666666666668, learning_rate = 0.00049999997, loss = 9.945664e-06, step = 6808 (5.075 sec) 2022-04-22 18:04:54,132 [INFO] tensorflow: epoch = 11.346666666666668, learning_rate = 0.00049999997, loss = 9.945664e-06, step = 6808 (5.075 sec) 2022-04-22 18:04:55,339 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.261 INFO:tensorflow:global_step/sec: 13.3327 2022-04-22 18:04:56,542 [INFO] tensorflow: global_step/sec: 13.3327 2022-04-22 18:04:57,210 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.471 2022-04-22 18:04:59,004 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.728 INFO:tensorflow:epoch = 11.458333333333334, learning_rate = 0.00049999997, loss = 9.672716e-06, step = 6875 (5.082 sec) 2022-04-22 18:04:59,077 [INFO] tensorflow: epoch = 11.458333333333334, learning_rate = 0.00049999997, loss = 9.672716e-06, step = 6875 (5.082 sec) INFO:tensorflow:epoch = 11.461666666666668, learning_rate = 0.00049999997, loss = 9.6433805e-06, step = 6877 (5.097 sec) 2022-04-22 18:04:59,229 [INFO] tensorflow: epoch = 11.461666666666668, learning_rate = 0.00049999997, loss = 9.6433805e-06, step = 6877 (5.097 sec) 2022-04-22 18:05:00,793 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.925 INFO:tensorflow:global_step/sec: 13.8527 2022-04-22 18:05:00,873 [INFO] tensorflow: global_step/sec: 13.8527 2022-04-22 18:05:02,565 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.442 INFO:tensorflow:epoch = 11.578333333333335, learning_rate = 0.00049999997, loss = 9.278265e-06, step = 6947 (5.111 sec) 2022-04-22 18:05:04,188 [INFO] tensorflow: epoch = 11.578333333333335, learning_rate = 0.00049999997, loss = 9.278265e-06, step = 6947 (5.111 sec) INFO:tensorflow:epoch = 11.581666666666667, learning_rate = 0.00049999997, loss = 9.316716e-06, step = 6949 (5.101 sec) 2022-04-22 18:05:04,330 [INFO] tensorflow: epoch = 11.581666666666667, learning_rate = 0.00049999997, loss = 9.316716e-06, step = 6949 (5.101 sec) 2022-04-22 18:05:04,331 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.621 INFO:tensorflow:global_step/sec: 14.2187 2022-04-22 18:05:05,092 [INFO] tensorflow: global_step/sec: 14.2187 2022-04-22 18:05:06,066 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 57.638 2022-04-22 18:05:07,895 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.674 INFO:tensorflow:epoch = 11.696666666666667, learning_rate = 0.00049999997, loss = 9.003573e-06, step = 7018 (5.107 sec) 2022-04-22 18:05:09,295 [INFO] tensorflow: epoch = 11.696666666666667, learning_rate = 0.00049999997, loss = 9.003573e-06, step = 7018 (5.107 sec) INFO:tensorflow:epoch = 11.700000000000001, learning_rate = 0.00049999997, loss = 9.013468e-06, step = 7020 (5.106 sec) 2022-04-22 18:05:09,436 [INFO] tensorflow: epoch = 11.700000000000001, learning_rate = 0.00049999997, loss = 9.013468e-06, step = 7020 (5.106 sec) INFO:tensorflow:global_step/sec: 13.8129 2022-04-22 18:05:09,436 [INFO] tensorflow: global_step/sec: 13.8129 2022-04-22 18:05:09,728 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.556 2022-04-22 18:05:11,532 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.455 2022-04-22 18:05:13,360 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.698 INFO:tensorflow:global_step/sec: 13.7185 2022-04-22 18:05:13,810 [INFO] tensorflow: global_step/sec: 13.7185 INFO:tensorflow:epoch = 11.811666666666667, learning_rate = 0.00049999997, loss = 8.661489e-06, step = 7087 (5.113 sec) 2022-04-22 18:05:14,408 [INFO] tensorflow: epoch = 11.811666666666667, learning_rate = 0.00049999997, loss = 8.661489e-06, step = 7087 (5.113 sec) INFO:tensorflow:epoch = 11.815000000000001, learning_rate = 0.00049999997, loss = 8.704648e-06, step = 7089 (5.113 sec) 2022-04-22 18:05:14,549 [INFO] tensorflow: epoch = 11.815000000000001, learning_rate = 0.00049999997, loss = 8.704648e-06, step = 7089 (5.113 sec) 2022-04-22 18:05:15,255 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 52.774 2022-04-22 18:05:17,060 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.429 INFO:tensorflow:global_step/sec: 13.4982 2022-04-22 18:05:18,255 [INFO] tensorflow: global_step/sec: 13.4982 2022-04-22 18:05:18,865 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.399 INFO:tensorflow:epoch = 11.931666666666667, learning_rate = 0.00049999997, loss = 8.4070425e-06, step = 7159 (5.136 sec) 2022-04-22 18:05:19,544 [INFO] tensorflow: epoch = 11.931666666666667, learning_rate = 0.00049999997, loss = 8.4070425e-06, step = 7159 (5.136 sec) INFO:tensorflow:epoch = 11.935, learning_rate = 0.00049999997, loss = 8.430973e-06, step = 7161 (5.133 sec) 2022-04-22 18:05:19,682 [INFO] tensorflow: epoch = 11.935, learning_rate = 0.00049999997, loss = 8.430973e-06, step = 7161 (5.133 sec) 2022-04-22 18:05:20,681 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.060 2022-04-22 18:05:22,515 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.548 INFO:tensorflow:global_step/sec: 13.8549 2022-04-22 18:05:22,586 [INFO] tensorflow: global_step/sec: 13.8549 2022-04-22 18:05:22,586 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 12/100: loss: 0.00001 learning rate: 0.00050 Time taken: 0:00:43.852143 ETA: 1:04:18.988567 2022-04-22 18:05:24,374 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.804 INFO:tensorflow:epoch = 12.046666666666667, learning_rate = 0.00049999997, loss = 8.172997e-06, step = 7228 (5.142 sec) 2022-04-22 18:05:24,685 [INFO] tensorflow: epoch = 12.046666666666667, learning_rate = 0.00049999997, loss = 8.172997e-06, step = 7228 (5.142 sec) INFO:tensorflow:epoch = 12.048333333333334, learning_rate = 0.00049999997, loss = 8.212902e-06, step = 7229 (5.075 sec) 2022-04-22 18:05:24,757 [INFO] tensorflow: epoch = 12.048333333333334, learning_rate = 0.00049999997, loss = 8.212902e-06, step = 7229 (5.075 sec) 2022-04-22 18:05:26,204 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.652 INFO:tensorflow:global_step/sec: 13.5499 2022-04-22 18:05:27,014 [INFO] tensorflow: global_step/sec: 13.5499 2022-04-22 18:05:28,106 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 52.584 INFO:tensorflow:epoch = 12.158333333333335, learning_rate = 0.00049999997, loss = 7.900533e-06, step = 7295 (5.078 sec) 2022-04-22 18:05:29,764 [INFO] tensorflow: epoch = 12.158333333333335, learning_rate = 0.00049999997, loss = 7.900533e-06, step = 7295 (5.078 sec) INFO:tensorflow:epoch = 12.16, learning_rate = 0.00049999997, loss = 7.884057e-06, step = 7296 (5.082 sec) 2022-04-22 18:05:29,839 [INFO] tensorflow: epoch = 12.16, learning_rate = 0.00049999997, loss = 7.884057e-06, step = 7296 (5.082 sec) 2022-04-22 18:05:30,056 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 51.297 INFO:tensorflow:global_step/sec: 13.2779 2022-04-22 18:05:31,532 [INFO] tensorflow: global_step/sec: 13.2779 2022-04-22 18:05:31,810 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 57.024 2022-04-22 18:05:33,613 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.479 INFO:tensorflow:epoch = 12.276666666666667, learning_rate = 0.00049999997, loss = 7.99111e-06, step = 7366 (5.092 sec) 2022-04-22 18:05:34,856 [INFO] tensorflow: epoch = 12.276666666666667, learning_rate = 0.00049999997, loss = 7.99111e-06, step = 7366 (5.092 sec) INFO:tensorflow:epoch = 12.278333333333334, learning_rate = 0.00049999997, loss = 7.606467e-06, step = 7367 (5.098 sec) 2022-04-22 18:05:34,937 [INFO] tensorflow: epoch = 12.278333333333334, learning_rate = 0.00049999997, loss = 7.606467e-06, step = 7367 (5.098 sec) 2022-04-22 18:05:35,438 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.799 INFO:tensorflow:global_step/sec: 13.8769 2022-04-22 18:05:35,856 [INFO] tensorflow: global_step/sec: 13.8769 2022-04-22 18:05:37,328 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 52.901 2022-04-22 18:05:39,097 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.536 INFO:tensorflow:epoch = 12.393333333333334, learning_rate = 0.00049999997, loss = 7.4037407e-06, step = 7436 (5.095 sec) 2022-04-22 18:05:39,950 [INFO] tensorflow: epoch = 12.393333333333334, learning_rate = 0.00049999997, loss = 7.4037407e-06, step = 7436 (5.095 sec) INFO:tensorflow:epoch = 12.395000000000001, learning_rate = 0.00049999997, loss = 7.348064e-06, step = 7437 (5.087 sec) 2022-04-22 18:05:40,024 [INFO] tensorflow: epoch = 12.395000000000001, learning_rate = 0.00049999997, loss = 7.348064e-06, step = 7437 (5.087 sec) INFO:tensorflow:global_step/sec: 13.6902 2022-04-22 18:05:40,239 [INFO] tensorflow: global_step/sec: 13.6902 2022-04-22 18:05:40,885 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.952 2022-04-22 18:05:42,751 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.590 2022-04-22 18:05:44,617 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.598 INFO:tensorflow:global_step/sec: 13.4919 2022-04-22 18:05:44,686 [INFO] tensorflow: global_step/sec: 13.4919 INFO:tensorflow:epoch = 12.510000000000002, learning_rate = 0.00049999997, loss = 7.1756876e-06, step = 7506 (5.135 sec) 2022-04-22 18:05:45,085 [INFO] tensorflow: epoch = 12.510000000000002, learning_rate = 0.00049999997, loss = 7.1756876e-06, step = 7506 (5.135 sec) INFO:tensorflow:epoch = 12.511666666666667, learning_rate = 0.00049999997, loss = 7.166108e-06, step = 7507 (5.155 sec) 2022-04-22 18:05:45,179 [INFO] tensorflow: epoch = 12.511666666666667, learning_rate = 0.00049999997, loss = 7.166108e-06, step = 7507 (5.155 sec) 2022-04-22 18:05:46,386 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.538 2022-04-22 18:05:48,140 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 57.016 INFO:tensorflow:global_step/sec: 14.0101 2022-04-22 18:05:48,969 [INFO] tensorflow: global_step/sec: 14.0101 2022-04-22 18:05:50,097 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 51.121 INFO:tensorflow:epoch = 12.625, learning_rate = 0.00049999997, loss = 6.9879916e-06, step = 7575 (5.146 sec) 2022-04-22 18:05:50,231 [INFO] tensorflow: epoch = 12.625, learning_rate = 0.00049999997, loss = 6.9879916e-06, step = 7575 (5.146 sec) INFO:tensorflow:epoch = 12.626666666666667, learning_rate = 0.00049999997, loss = 6.9831503e-06, step = 7576 (5.135 sec) 2022-04-22 18:05:50,313 [INFO] tensorflow: epoch = 12.626666666666667, learning_rate = 0.00049999997, loss = 6.9831503e-06, step = 7576 (5.135 sec) 2022-04-22 18:05:52,058 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 50.994 INFO:tensorflow:global_step/sec: 12.9398 2022-04-22 18:05:53,605 [INFO] tensorflow: global_step/sec: 12.9398 2022-04-22 18:05:53,912 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.952 INFO:tensorflow:epoch = 12.738333333333333, learning_rate = 0.00049999997, loss = 6.7346828e-06, step = 7643 (5.081 sec) 2022-04-22 18:05:55,312 [INFO] tensorflow: epoch = 12.738333333333333, learning_rate = 0.00049999997, loss = 6.7346828e-06, step = 7643 (5.081 sec) INFO:tensorflow:epoch = 12.741666666666667, learning_rate = 0.00049999997, loss = 6.7878455e-06, step = 7645 (5.143 sec) 2022-04-22 18:05:55,456 [INFO] tensorflow: epoch = 12.741666666666667, learning_rate = 0.00049999997, loss = 6.7878455e-06, step = 7645 (5.143 sec) 2022-04-22 18:05:55,761 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.084 2022-04-22 18:05:57,609 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.130 INFO:tensorflow:global_step/sec: 13.4825 2022-04-22 18:05:58,056 [INFO] tensorflow: global_step/sec: 13.4825 2022-04-22 18:05:59,383 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.377 INFO:tensorflow:epoch = 12.856666666666667, learning_rate = 0.00049999997, loss = 6.5872982e-06, step = 7714 (5.099 sec) 2022-04-22 18:06:00,411 [INFO] tensorflow: epoch = 12.856666666666667, learning_rate = 0.00049999997, loss = 6.5872982e-06, step = 7714 (5.099 sec) INFO:tensorflow:epoch = 12.860000000000001, learning_rate = 0.00049999997, loss = 6.6093726e-06, step = 7716 (5.100 sec) 2022-04-22 18:06:00,556 [INFO] tensorflow: epoch = 12.860000000000001, learning_rate = 0.00049999997, loss = 6.6093726e-06, step = 7716 (5.100 sec) 2022-04-22 18:06:01,204 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.920 INFO:tensorflow:global_step/sec: 13.8531 2022-04-22 18:06:02,387 [INFO] tensorflow: global_step/sec: 13.8531 2022-04-22 18:06:03,044 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.353 2022-04-22 18:06:04,883 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.382 INFO:tensorflow:epoch = 12.971666666666668, learning_rate = 0.00049999997, loss = 6.325916e-06, step = 7783 (5.111 sec) 2022-04-22 18:06:05,522 [INFO] tensorflow: epoch = 12.971666666666668, learning_rate = 0.00049999997, loss = 6.325916e-06, step = 7783 (5.111 sec) INFO:tensorflow:epoch = 12.975000000000001, learning_rate = 0.00049999997, loss = 6.3349407e-06, step = 7785 (5.103 sec) 2022-04-22 18:06:05,659 [INFO] tensorflow: epoch = 12.975000000000001, learning_rate = 0.00049999997, loss = 6.3349407e-06, step = 7785 (5.103 sec) 2022-04-22 18:06:06,700 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.044 INFO:tensorflow:global_step/sec: 13.6853 2022-04-22 18:06:06,771 [INFO] tensorflow: global_step/sec: 13.6853 2022-04-22 18:06:06,772 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 13/100: loss: 0.00001 learning rate: 0.00050 Time taken: 0:00:44.184332 ETA: 1:04:04.036896 2022-04-22 18:06:08,536 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.493 2022-04-22 18:06:10,315 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.212 INFO:tensorflow:epoch = 13.088333333333335, learning_rate = 0.00049999997, loss = 6.1216615e-06, step = 7853 (5.132 sec) 2022-04-22 18:06:10,654 [INFO] tensorflow: epoch = 13.088333333333335, learning_rate = 0.00049999997, loss = 6.1216615e-06, step = 7853 (5.132 sec) INFO:tensorflow:epoch = 13.091666666666667, learning_rate = 0.00049999997, loss = 6.161771e-06, step = 7855 (5.161 sec) 2022-04-22 18:06:10,820 [INFO] tensorflow: epoch = 13.091666666666667, learning_rate = 0.00049999997, loss = 6.161771e-06, step = 7855 (5.161 sec) INFO:tensorflow:global_step/sec: 13.4868 2022-04-22 18:06:11,220 [INFO] tensorflow: global_step/sec: 13.4868 2022-04-22 18:06:12,255 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 51.558 2022-04-22 18:06:14,108 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.961 INFO:tensorflow:epoch = 13.200000000000001, learning_rate = 0.00049999997, loss = 5.982509e-06, step = 7920 (5.154 sec) 2022-04-22 18:06:15,808 [INFO] tensorflow: epoch = 13.200000000000001, learning_rate = 0.00049999997, loss = 5.982509e-06, step = 7920 (5.154 sec) INFO:tensorflow:global_step/sec: 13.0724 2022-04-22 18:06:15,810 [INFO] tensorflow: global_step/sec: 13.0724 INFO:tensorflow:epoch = 13.203333333333335, learning_rate = 0.00049999997, loss = 6.0116954e-06, step = 7922 (5.204 sec) 2022-04-22 18:06:16,023 [INFO] tensorflow: epoch = 13.203333333333335, learning_rate = 0.00049999997, loss = 6.0116954e-06, step = 7922 (5.204 sec) 2022-04-22 18:06:16,177 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 48.337 2022-04-22 18:06:18,204 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 49.355 2022-04-22 18:06:20,010 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.377 INFO:tensorflow:global_step/sec: 12.9221 2022-04-22 18:06:20,453 [INFO] tensorflow: global_step/sec: 12.9221 INFO:tensorflow:epoch = 13.311666666666667, learning_rate = 0.00049999997, loss = 5.7582065e-06, step = 7987 (5.161 sec) 2022-04-22 18:06:20,968 [INFO] tensorflow: epoch = 13.311666666666667, learning_rate = 0.00049999997, loss = 5.7582065e-06, step = 7987 (5.161 sec) INFO:tensorflow:epoch = 13.315000000000001, learning_rate = 0.00049999997, loss = 5.823499e-06, step = 7989 (5.081 sec) 2022-04-22 18:06:21,104 [INFO] tensorflow: epoch = 13.315000000000001, learning_rate = 0.00049999997, loss = 5.823499e-06, step = 7989 (5.081 sec) 2022-04-22 18:06:21,935 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 51.943 2022-04-22 18:06:23,715 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.186 INFO:tensorflow:global_step/sec: 13.4184 2022-04-22 18:06:24,925 [INFO] tensorflow: global_step/sec: 13.4184 2022-04-22 18:06:25,564 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.085 INFO:tensorflow:epoch = 13.426666666666668, learning_rate = 0.00049999997, loss = 5.6183035e-06, step = 8056 (5.101 sec) 2022-04-22 18:06:26,069 [INFO] tensorflow: epoch = 13.426666666666668, learning_rate = 0.00049999997, loss = 5.6183035e-06, step = 8056 (5.101 sec) INFO:tensorflow:epoch = 13.430000000000001, learning_rate = 0.00049999997, loss = 5.583526e-06, step = 8058 (5.094 sec) 2022-04-22 18:06:26,198 [INFO] tensorflow: epoch = 13.430000000000001, learning_rate = 0.00049999997, loss = 5.583526e-06, step = 8058 (5.094 sec) 2022-04-22 18:06:27,510 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 51.406 2022-04-22 18:06:29,386 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.308 INFO:tensorflow:global_step/sec: 13.2003 2022-04-22 18:06:29,470 [INFO] tensorflow: global_step/sec: 13.2003 INFO:tensorflow:epoch = 13.538333333333334, learning_rate = 0.00049999997, loss = 5.3888634e-06, step = 8123 (5.170 sec) 2022-04-22 18:06:31,239 [INFO] tensorflow: epoch = 13.538333333333334, learning_rate = 0.00049999997, loss = 5.3888634e-06, step = 8123 (5.170 sec) 2022-04-22 18:06:31,319 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 51.755 INFO:tensorflow:epoch = 13.540000000000001, learning_rate = 0.00049999997, loss = 5.4256634e-06, step = 8124 (5.125 sec) 2022-04-22 18:06:31,323 [INFO] tensorflow: epoch = 13.540000000000001, learning_rate = 0.00049999997, loss = 5.4256634e-06, step = 8124 (5.125 sec) 2022-04-22 18:06:33,308 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 50.282 INFO:tensorflow:global_step/sec: 12.8779 2022-04-22 18:06:34,129 [INFO] tensorflow: global_step/sec: 12.8779 2022-04-22 18:06:35,172 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.648 INFO:tensorflow:epoch = 13.65, learning_rate = 0.00049999997, loss = 5.2772293e-06, step = 8190 (5.078 sec) 2022-04-22 18:06:36,317 [INFO] tensorflow: epoch = 13.65, learning_rate = 0.00049999997, loss = 5.2772293e-06, step = 8190 (5.078 sec) INFO:tensorflow:epoch = 13.653333333333334, learning_rate = 0.00049999997, loss = 5.271453e-06, step = 8192 (5.156 sec) 2022-04-22 18:06:36,479 [INFO] tensorflow: epoch = 13.653333333333334, learning_rate = 0.00049999997, loss = 5.271453e-06, step = 8192 (5.156 sec) 2022-04-22 18:06:36,988 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.081 INFO:tensorflow:global_step/sec: 13.4884 2022-04-22 18:06:38,577 [INFO] tensorflow: global_step/sec: 13.4884 2022-04-22 18:06:38,864 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.309 2022-04-22 18:06:40,708 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.212 INFO:tensorflow:epoch = 13.765, learning_rate = 0.00049999997, loss = 5.1129045e-06, step = 8259 (5.131 sec) 2022-04-22 18:06:41,448 [INFO] tensorflow: epoch = 13.765, learning_rate = 0.00049999997, loss = 5.1129045e-06, step = 8259 (5.131 sec) INFO:tensorflow:epoch = 13.768333333333334, learning_rate = 0.00049999997, loss = 5.0948083e-06, step = 8261 (5.115 sec) 2022-04-22 18:06:41,593 [INFO] tensorflow: epoch = 13.768333333333334, learning_rate = 0.00049999997, loss = 5.0948083e-06, step = 8261 (5.115 sec) 2022-04-22 18:06:42,518 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.261 INFO:tensorflow:global_step/sec: 13.7055 2022-04-22 18:06:42,955 [INFO] tensorflow: global_step/sec: 13.7055 2022-04-22 18:06:44,343 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.815 2022-04-22 18:06:46,173 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.649 INFO:tensorflow:epoch = 13.881666666666668, learning_rate = 0.00049999997, loss = 4.9296323e-06, step = 8329 (5.107 sec) 2022-04-22 18:06:46,555 [INFO] tensorflow: epoch = 13.881666666666668, learning_rate = 0.00049999997, loss = 4.9296323e-06, step = 8329 (5.107 sec) INFO:tensorflow:epoch = 13.885000000000002, learning_rate = 0.00049999997, loss = 4.9249247e-06, step = 8331 (5.117 sec) 2022-04-22 18:06:46,710 [INFO] tensorflow: epoch = 13.885000000000002, learning_rate = 0.00049999997, loss = 4.9249247e-06, step = 8331 (5.117 sec) INFO:tensorflow:global_step/sec: 13.599 2022-04-22 18:06:47,367 [INFO] tensorflow: global_step/sec: 13.599 2022-04-22 18:06:48,059 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.020 2022-04-22 18:06:50,208 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 46.537 INFO:tensorflow:epoch = 13.986666666666668, learning_rate = 0.00049999997, loss = 4.7544927e-06, step = 8392 (5.138 sec) 2022-04-22 18:06:51,693 [INFO] tensorflow: epoch = 13.986666666666668, learning_rate = 0.00049999997, loss = 4.7544927e-06, step = 8392 (5.138 sec) INFO:tensorflow:epoch = 13.99, learning_rate = 0.00049999997, loss = 4.798979e-06, step = 8394 (5.140 sec) 2022-04-22 18:06:51,850 [INFO] tensorflow: epoch = 13.99, learning_rate = 0.00049999997, loss = 4.798979e-06, step = 8394 (5.140 sec) 2022-04-22 18:06:52,235 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 49.349 INFO:tensorflow:global_step/sec: 12.1538 2022-04-22 18:06:52,304 [INFO] tensorflow: global_step/sec: 12.1538 2022-04-22 18:06:52,305 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 14/100: loss: 0.00000 learning rate: 0.00050 Time taken: 0:00:45.534271 ETA: 1:05:15.947306 2022-04-22 18:06:54,078 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.280 2022-04-22 18:06:55,949 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.439 INFO:tensorflow:global_step/sec: 13.6552 2022-04-22 18:06:56,698 [INFO] tensorflow: global_step/sec: 13.6552 INFO:tensorflow:epoch = 14.101666666666668, learning_rate = 0.00049999997, loss = 4.6222603e-06, step = 8461 (5.081 sec) 2022-04-22 18:06:56,774 [INFO] tensorflow: epoch = 14.101666666666668, learning_rate = 0.00049999997, loss = 4.6222603e-06, step = 8461 (5.081 sec) INFO:tensorflow:epoch = 14.106666666666667, learning_rate = 0.00049999997, loss = 4.6685464e-06, step = 8464 (5.129 sec) 2022-04-22 18:06:56,979 [INFO] tensorflow: epoch = 14.106666666666667, learning_rate = 0.00049999997, loss = 4.6685464e-06, step = 8464 (5.129 sec) 2022-04-22 18:06:57,668 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 58.185 2022-04-22 18:06:59,485 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.032 INFO:tensorflow:global_step/sec: 14.2149 2022-04-22 18:07:00,919 [INFO] tensorflow: global_step/sec: 14.2149 2022-04-22 18:07:01,188 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 58.724 INFO:tensorflow:epoch = 14.221666666666668, learning_rate = 0.00049999997, loss = 4.4853896e-06, step = 8533 (5.094 sec) 2022-04-22 18:07:01,869 [INFO] tensorflow: epoch = 14.221666666666668, learning_rate = 0.00049999997, loss = 4.4853896e-06, step = 8533 (5.094 sec) INFO:tensorflow:epoch = 14.226666666666668, learning_rate = 0.00049999997, loss = 4.444012e-06, step = 8536 (5.105 sec) 2022-04-22 18:07:02,084 [INFO] tensorflow: epoch = 14.226666666666668, learning_rate = 0.00049999997, loss = 4.444012e-06, step = 8536 (5.105 sec) 2022-04-22 18:07:03,040 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.018 2022-04-22 18:07:04,881 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.334 INFO:tensorflow:global_step/sec: 13.5429 2022-04-22 18:07:05,349 [INFO] tensorflow: global_step/sec: 13.5429 2022-04-22 18:07:06,720 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.375 INFO:tensorflow:epoch = 14.336666666666668, learning_rate = 0.00049999997, loss = 4.3367827e-06, step = 8602 (5.068 sec) 2022-04-22 18:07:06,937 [INFO] tensorflow: epoch = 14.336666666666668, learning_rate = 0.00049999997, loss = 4.3367827e-06, step = 8602 (5.068 sec) INFO:tensorflow:epoch = 14.341666666666667, learning_rate = 0.00049999997, loss = 4.3119135e-06, step = 8605 (5.066 sec) 2022-04-22 18:07:07,150 [INFO] tensorflow: epoch = 14.341666666666667, learning_rate = 0.00049999997, loss = 4.3119135e-06, step = 8605 (5.066 sec) 2022-04-22 18:07:08,488 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.570 INFO:tensorflow:global_step/sec: 13.436 2022-04-22 18:07:09,815 [INFO] tensorflow: global_step/sec: 13.436 2022-04-22 18:07:10,489 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 49.980 INFO:tensorflow:epoch = 14.451666666666668, learning_rate = 0.00049999997, loss = 4.1466774e-06, step = 8671 (5.108 sec) 2022-04-22 18:07:12,044 [INFO] tensorflow: epoch = 14.451666666666668, learning_rate = 0.00049999997, loss = 4.1466774e-06, step = 8671 (5.108 sec) 2022-04-22 18:07:12,265 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.309 INFO:tensorflow:epoch = 14.456666666666667, learning_rate = 0.00049999997, loss = 4.213128e-06, step = 8674 (5.117 sec) 2022-04-22 18:07:12,267 [INFO] tensorflow: epoch = 14.456666666666667, learning_rate = 0.00049999997, loss = 4.213128e-06, step = 8674 (5.117 sec) 2022-04-22 18:07:14,060 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.736 INFO:tensorflow:global_step/sec: 13.9023 2022-04-22 18:07:14,130 [INFO] tensorflow: global_step/sec: 13.9023 2022-04-22 18:07:15,894 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.509 INFO:tensorflow:epoch = 14.57, learning_rate = 0.00049999997, loss = 3.98891e-06, step = 8742 (5.129 sec) 2022-04-22 18:07:17,173 [INFO] tensorflow: epoch = 14.57, learning_rate = 0.00049999997, loss = 3.98891e-06, step = 8742 (5.129 sec) INFO:tensorflow:epoch = 14.575000000000001, learning_rate = 0.00049999997, loss = 4.012867e-06, step = 8745 (5.144 sec) 2022-04-22 18:07:17,411 [INFO] tensorflow: epoch = 14.575000000000001, learning_rate = 0.00049999997, loss = 4.012867e-06, step = 8745 (5.144 sec) 2022-04-22 18:07:17,681 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.987 INFO:tensorflow:global_step/sec: 13.7618 2022-04-22 18:07:18,490 [INFO] tensorflow: global_step/sec: 13.7618 2022-04-22 18:07:19,538 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.853 2022-04-22 18:07:21,358 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.956 INFO:tensorflow:epoch = 14.686666666666667, learning_rate = 0.00049999997, loss = 3.819112e-06, step = 8812 (5.128 sec) 2022-04-22 18:07:22,301 [INFO] tensorflow: epoch = 14.686666666666667, learning_rate = 0.00049999997, loss = 3.819112e-06, step = 8812 (5.128 sec) INFO:tensorflow:epoch = 14.691666666666668, learning_rate = 0.00049999997, loss = 3.843694e-06, step = 8815 (5.127 sec) 2022-04-22 18:07:22,538 [INFO] tensorflow: epoch = 14.691666666666668, learning_rate = 0.00049999997, loss = 3.843694e-06, step = 8815 (5.127 sec) INFO:tensorflow:global_step/sec: 13.4023 2022-04-22 18:07:22,967 [INFO] tensorflow: global_step/sec: 13.4023 2022-04-22 18:07:23,305 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 51.349 2022-04-22 18:07:25,167 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.720 2022-04-22 18:07:26,942 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.350 INFO:tensorflow:epoch = 14.8, learning_rate = 0.00049999997, loss = 3.6764031e-06, step = 8880 (5.084 sec) 2022-04-22 18:07:27,385 [INFO] tensorflow: epoch = 14.8, learning_rate = 0.00049999997, loss = 3.6764031e-06, step = 8880 (5.084 sec) INFO:tensorflow:global_step/sec: 13.5737 2022-04-22 18:07:27,387 [INFO] tensorflow: global_step/sec: 13.5737 INFO:tensorflow:epoch = 14.806666666666668, learning_rate = 0.00049999997, loss = 3.6841102e-06, step = 8884 (5.133 sec) 2022-04-22 18:07:27,671 [INFO] tensorflow: epoch = 14.806666666666668, learning_rate = 0.00049999997, loss = 3.6841102e-06, step = 8884 (5.133 sec) 2022-04-22 18:07:28,885 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 51.474 2022-04-22 18:07:30,728 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.271 INFO:tensorflow:global_step/sec: 13.3806 2022-04-22 18:07:31,872 [INFO] tensorflow: global_step/sec: 13.3806 INFO:tensorflow:epoch = 14.915000000000001, learning_rate = 0.00049999997, loss = 3.5293006e-06, step = 8949 (5.097 sec) 2022-04-22 18:07:32,482 [INFO] tensorflow: epoch = 14.915000000000001, learning_rate = 0.00049999997, loss = 3.5293006e-06, step = 8949 (5.097 sec) 2022-04-22 18:07:32,482 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 57.014 INFO:tensorflow:epoch = 14.921666666666667, learning_rate = 0.00049999997, loss = 3.509008e-06, step = 8953 (5.118 sec) 2022-04-22 18:07:32,789 [INFO] tensorflow: epoch = 14.921666666666667, learning_rate = 0.00049999997, loss = 3.509008e-06, step = 8953 (5.118 sec) 2022-04-22 18:07:34,273 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.847 2022-04-22 18:07:36,092 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.980 INFO:tensorflow:global_step/sec: 13.9988 2022-04-22 18:07:36,158 [INFO] tensorflow: global_step/sec: 13.9988 2022-04-22 18:07:36,158 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 15/100: loss: 0.00000 learning rate: 0.00050 Time taken: 0:00:43.856464 ETA: 1:02:07.799432 INFO:tensorflow:epoch = 15.033333333333335, learning_rate = 0.00049999997, loss = 3.3900096e-06, step = 9020 (5.084 sec) 2022-04-22 18:07:37,566 [INFO] tensorflow: epoch = 15.033333333333335, learning_rate = 0.00049999997, loss = 3.3900096e-06, step = 9020 (5.084 sec) 2022-04-22 18:07:37,858 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.625 INFO:tensorflow:epoch = 15.041666666666668, learning_rate = 0.00049999997, loss = 3.411532e-06, step = 9025 (5.161 sec) 2022-04-22 18:07:37,950 [INFO] tensorflow: epoch = 15.041666666666668, learning_rate = 0.00049999997, loss = 3.411532e-06, step = 9025 (5.161 sec) 2022-04-22 18:07:39,733 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.375 INFO:tensorflow:global_step/sec: 13.7684 2022-04-22 18:07:40,515 [INFO] tensorflow: global_step/sec: 13.7684 2022-04-22 18:07:41,485 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 57.059 INFO:tensorflow:epoch = 15.15, learning_rate = 0.00049999997, loss = 3.2454182e-06, step = 9090 (5.073 sec) 2022-04-22 18:07:42,639 [INFO] tensorflow: epoch = 15.15, learning_rate = 0.00049999997, loss = 3.2454182e-06, step = 9090 (5.073 sec) INFO:tensorflow:epoch = 15.158333333333335, learning_rate = 0.00049999997, loss = 3.2104554e-06, step = 9095 (5.080 sec) 2022-04-22 18:07:43,029 [INFO] tensorflow: epoch = 15.158333333333335, learning_rate = 0.00049999997, loss = 3.2104554e-06, step = 9095 (5.080 sec) 2022-04-22 18:07:43,343 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.854 INFO:tensorflow:global_step/sec: 13.6953 2022-04-22 18:07:44,897 [INFO] tensorflow: global_step/sec: 13.6953 2022-04-22 18:07:45,183 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.338 2022-04-22 18:07:47,007 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.837 INFO:tensorflow:epoch = 15.263333333333334, learning_rate = 0.00049999997, loss = 3.1121544e-06, step = 9158 (5.087 sec) 2022-04-22 18:07:47,726 [INFO] tensorflow: epoch = 15.263333333333334, learning_rate = 0.00049999997, loss = 3.1121544e-06, step = 9158 (5.087 sec) INFO:tensorflow:epoch = 15.273333333333333, learning_rate = 0.00049999997, loss = 3.102953e-06, step = 9164 (5.150 sec) 2022-04-22 18:07:48,179 [INFO] tensorflow: epoch = 15.273333333333333, learning_rate = 0.00049999997, loss = 3.102953e-06, step = 9164 (5.150 sec) 2022-04-22 18:07:48,967 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 51.036 INFO:tensorflow:global_step/sec: 13.0329 2022-04-22 18:07:49,500 [INFO] tensorflow: global_step/sec: 13.0329 2022-04-22 18:07:50,837 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.484 2022-04-22 18:07:52,726 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 52.945 INFO:tensorflow:epoch = 15.376666666666667, learning_rate = 0.00049999997, loss = 2.983582e-06, step = 9226 (5.157 sec) 2022-04-22 18:07:52,882 [INFO] tensorflow: epoch = 15.376666666666667, learning_rate = 0.00049999997, loss = 2.983582e-06, step = 9226 (5.157 sec) INFO:tensorflow:epoch = 15.386666666666668, learning_rate = 0.00049999997, loss = 2.93654e-06, step = 9232 (5.166 sec) 2022-04-22 18:07:53,345 [INFO] tensorflow: epoch = 15.386666666666668, learning_rate = 0.00049999997, loss = 2.93654e-06, step = 9232 (5.166 sec) INFO:tensorflow:global_step/sec: 13.5669 2022-04-22 18:07:53,923 [INFO] tensorflow: global_step/sec: 13.5669 2022-04-22 18:07:54,609 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.108 2022-04-22 18:07:56,382 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.398 INFO:tensorflow:epoch = 15.49, learning_rate = 0.00049999997, loss = 2.8301993e-06, step = 9294 (5.078 sec) 2022-04-22 18:07:57,960 [INFO] tensorflow: epoch = 15.49, learning_rate = 0.00049999997, loss = 2.8301993e-06, step = 9294 (5.078 sec) 2022-04-22 18:07:58,423 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 48.998 INFO:tensorflow:epoch = 15.500000000000002, learning_rate = 0.00049999997, loss = 2.8423347e-06, step = 9300 (5.149 sec) 2022-04-22 18:07:58,494 [INFO] tensorflow: epoch = 15.500000000000002, learning_rate = 0.00049999997, loss = 2.8423347e-06, step = 9300 (5.149 sec) INFO:tensorflow:global_step/sec: 13.1243 2022-04-22 18:07:58,494 [INFO] tensorflow: global_step/sec: 13.1243 2022-04-22 18:08:00,357 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 51.709 2022-04-22 18:08:02,195 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.435 INFO:tensorflow:global_step/sec: 13.4951 2022-04-22 18:08:02,941 [INFO] tensorflow: global_step/sec: 13.4951 INFO:tensorflow:epoch = 15.603333333333333, learning_rate = 0.00049999997, loss = 2.6742337e-06, step = 9362 (5.130 sec) 2022-04-22 18:08:03,090 [INFO] tensorflow: epoch = 15.603333333333333, learning_rate = 0.00049999997, loss = 2.6742337e-06, step = 9362 (5.130 sec) INFO:tensorflow:epoch = 15.615, learning_rate = 0.00049999997, loss = 2.6838907e-06, step = 9369 (5.178 sec) 2022-04-22 18:08:03,672 [INFO] tensorflow: epoch = 15.615, learning_rate = 0.00049999997, loss = 2.6838907e-06, step = 9369 (5.178 sec) 2022-04-22 18:08:04,060 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.625 2022-04-22 18:08:05,901 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.308 INFO:tensorflow:global_step/sec: 13.0328 2022-04-22 18:08:07,544 [INFO] tensorflow: global_step/sec: 13.0328 2022-04-22 18:08:07,854 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 51.204 INFO:tensorflow:epoch = 15.715000000000002, learning_rate = 0.00049999997, loss = 2.5877662e-06, step = 9429 (5.105 sec) 2022-04-22 18:08:08,195 [INFO] tensorflow: epoch = 15.715000000000002, learning_rate = 0.00049999997, loss = 2.5877662e-06, step = 9429 (5.105 sec) INFO:tensorflow:epoch = 15.728333333333333, learning_rate = 0.00049999997, loss = 2.5283289e-06, step = 9437 (5.083 sec) 2022-04-22 18:08:08,755 [INFO] tensorflow: epoch = 15.728333333333333, learning_rate = 0.00049999997, loss = 2.5283289e-06, step = 9437 (5.083 sec) 2022-04-22 18:08:09,607 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 57.067 2022-04-22 18:08:11,521 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 52.237 INFO:tensorflow:global_step/sec: 13.5674 2022-04-22 18:08:11,967 [INFO] tensorflow: global_step/sec: 13.5674 INFO:tensorflow:epoch = 15.830000000000002, learning_rate = 0.00049999997, loss = 2.4724814e-06, step = 9498 (5.087 sec) 2022-04-22 18:08:13,282 [INFO] tensorflow: epoch = 15.830000000000002, learning_rate = 0.00049999997, loss = 2.4724814e-06, step = 9498 (5.087 sec) 2022-04-22 18:08:13,360 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.396 INFO:tensorflow:epoch = 15.843333333333334, learning_rate = 0.00049999997, loss = 2.467751e-06, step = 9506 (5.109 sec) 2022-04-22 18:08:13,863 [INFO] tensorflow: epoch = 15.843333333333334, learning_rate = 0.00049999997, loss = 2.467751e-06, step = 9506 (5.109 sec) 2022-04-22 18:08:15,155 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.706 INFO:tensorflow:global_step/sec: 13.6785 2022-04-22 18:08:16,353 [INFO] tensorflow: global_step/sec: 13.6785 2022-04-22 18:08:16,993 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.435 INFO:tensorflow:epoch = 15.948333333333334, learning_rate = 0.00049999997, loss = 2.352157e-06, step = 9569 (5.117 sec) 2022-04-22 18:08:18,399 [INFO] tensorflow: epoch = 15.948333333333334, learning_rate = 0.00049999997, loss = 2.352157e-06, step = 9569 (5.117 sec) 2022-04-22 18:08:18,770 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.261 INFO:tensorflow:epoch = 15.96, learning_rate = 0.00049999997, loss = 2.3484545e-06, step = 9576 (5.070 sec) 2022-04-22 18:08:18,933 [INFO] tensorflow: epoch = 15.96, learning_rate = 0.00049999997, loss = 2.3484545e-06, step = 9576 (5.070 sec) 2022-04-22 18:08:20,700 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 299, 0.00s/step 2022-04-22 18:08:20,922 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 299, 0.02s/step 2022-04-22 18:08:21,137 [INFO] iva.detectnet_v2.evaluation.evaluation: step 20 / 299, 0.02s/step 2022-04-22 18:08:21,352 [INFO] iva.detectnet_v2.evaluation.evaluation: step 30 / 299, 0.02s/step 2022-04-22 18:08:21,570 [INFO] iva.detectnet_v2.evaluation.evaluation: step 40 / 299, 0.02s/step 2022-04-22 18:08:21,788 [INFO] iva.detectnet_v2.evaluation.evaluation: step 50 / 299, 0.02s/step 2022-04-22 18:08:22,008 [INFO] iva.detectnet_v2.evaluation.evaluation: step 60 / 299, 0.02s/step 2022-04-22 18:08:22,229 [INFO] iva.detectnet_v2.evaluation.evaluation: step 70 / 299, 0.02s/step 2022-04-22 18:08:22,452 [INFO] iva.detectnet_v2.evaluation.evaluation: step 80 / 299, 0.02s/step 2022-04-22 18:08:22,675 [INFO] iva.detectnet_v2.evaluation.evaluation: step 90 / 299, 0.02s/step 2022-04-22 18:08:22,900 [INFO] iva.detectnet_v2.evaluation.evaluation: step 100 / 299, 0.02s/step 2022-04-22 18:08:23,130 [INFO] iva.detectnet_v2.evaluation.evaluation: step 110 / 299, 0.02s/step 2022-04-22 18:08:23,366 [INFO] iva.detectnet_v2.evaluation.evaluation: step 120 / 299, 0.02s/step 2022-04-22 18:08:23,600 [INFO] iva.detectnet_v2.evaluation.evaluation: step 130 / 299, 0.02s/step 2022-04-22 18:08:23,836 [INFO] iva.detectnet_v2.evaluation.evaluation: step 140 / 299, 0.02s/step 2022-04-22 18:08:24,073 [INFO] iva.detectnet_v2.evaluation.evaluation: step 150 / 299, 0.02s/step 2022-04-22 18:08:24,309 [INFO] iva.detectnet_v2.evaluation.evaluation: step 160 / 299, 0.02s/step 2022-04-22 18:08:24,544 [INFO] iva.detectnet_v2.evaluation.evaluation: step 170 / 299, 0.02s/step 2022-04-22 18:08:24,781 [INFO] iva.detectnet_v2.evaluation.evaluation: step 180 / 299, 0.02s/step 2022-04-22 18:08:25,018 [INFO] iva.detectnet_v2.evaluation.evaluation: step 190 / 299, 0.02s/step 2022-04-22 18:08:25,254 [INFO] iva.detectnet_v2.evaluation.evaluation: step 200 / 299, 0.02s/step 2022-04-22 18:08:25,490 [INFO] iva.detectnet_v2.evaluation.evaluation: step 210 / 299, 0.02s/step 2022-04-22 18:08:25,727 [INFO] iva.detectnet_v2.evaluation.evaluation: step 220 / 299, 0.02s/step 2022-04-22 18:08:25,961 [INFO] iva.detectnet_v2.evaluation.evaluation: step 230 / 299, 0.02s/step 2022-04-22 18:08:26,195 [INFO] iva.detectnet_v2.evaluation.evaluation: step 240 / 299, 0.02s/step 2022-04-22 18:08:26,432 [INFO] iva.detectnet_v2.evaluation.evaluation: step 250 / 299, 0.02s/step 2022-04-22 18:08:26,667 [INFO] iva.detectnet_v2.evaluation.evaluation: step 260 / 299, 0.02s/step 2022-04-22 18:08:26,902 [INFO] iva.detectnet_v2.evaluation.evaluation: step 270 / 299, 0.02s/step 2022-04-22 18:08:27,136 [INFO] iva.detectnet_v2.evaluation.evaluation: step 280 / 299, 0.02s/step 2022-04-22 18:08:27,380 [INFO] iva.detectnet_v2.evaluation.evaluation: step 290 / 299, 0.02s/step Epoch 16/100 ========================= Validation cost: -0.000009 Mean average_precision (in %): 0.0000 class name average precision (in %) ------------ -------------------------- face 0 Median Inference Time: 0.011558 2022-04-22 18:08:27,626 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 11.293 INFO:tensorflow:epoch = 16.0, learning_rate = 0.00049999997, loss = 2.283915e-06, step = 9600 (9.296 sec) 2022-04-22 18:08:27,695 [INFO] tensorflow: epoch = 16.0, learning_rate = 0.00049999997, loss = 2.283915e-06, step = 9600 (9.296 sec) INFO:tensorflow:global_step/sec: 5.28967 2022-04-22 18:08:27,696 [INFO] tensorflow: global_step/sec: 5.28967 2022-04-22 18:08:27,697 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 16/100: loss: 0.00000 learning rate: 0.00050 Time taken: 0:00:51.533303 ETA: 1:12:08.797494 INFO:tensorflow:epoch = 16.00166666666667, learning_rate = 0.00049999997, loss = 2.2854556e-06, step = 9601 (8.839 sec) 2022-04-22 18:08:27,772 [INFO] tensorflow: epoch = 16.00166666666667, learning_rate = 0.00049999997, loss = 2.2854556e-06, step = 9601 (8.839 sec) 2022-04-22 18:08:29,467 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.323 2022-04-22 18:08:31,270 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.477 INFO:tensorflow:global_step/sec: 13.7231 2022-04-22 18:08:32,068 [INFO] tensorflow: global_step/sec: 13.7231 INFO:tensorflow:epoch = 16.115000000000002, learning_rate = 0.00049999997, loss = 2.176842e-06, step = 9669 (5.080 sec) 2022-04-22 18:08:32,775 [INFO] tensorflow: epoch = 16.115000000000002, learning_rate = 0.00049999997, loss = 2.176842e-06, step = 9669 (5.080 sec) INFO:tensorflow:epoch = 16.116666666666667, learning_rate = 0.00049999997, loss = 2.1788437e-06, step = 9670 (5.072 sec) 2022-04-22 18:08:32,844 [INFO] tensorflow: epoch = 16.116666666666667, learning_rate = 0.00049999997, loss = 2.1788437e-06, step = 9670 (5.072 sec) 2022-04-22 18:08:33,130 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.769 2022-04-22 18:08:35,105 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 50.637 INFO:tensorflow:global_step/sec: 13.1952 2022-04-22 18:08:36,615 [INFO] tensorflow: global_step/sec: 13.1952 2022-04-22 18:08:36,891 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.994 INFO:tensorflow:epoch = 16.228333333333335, learning_rate = 0.00049999997, loss = 2.0531543e-06, step = 9737 (5.104 sec) 2022-04-22 18:08:37,880 [INFO] tensorflow: epoch = 16.228333333333335, learning_rate = 0.00049999997, loss = 2.0531543e-06, step = 9737 (5.104 sec) INFO:tensorflow:epoch = 16.23, learning_rate = 0.00049999997, loss = 2.0484858e-06, step = 9738 (5.106 sec) 2022-04-22 18:08:37,950 [INFO] tensorflow: epoch = 16.23, learning_rate = 0.00049999997, loss = 2.0484858e-06, step = 9738 (5.106 sec) 2022-04-22 18:08:38,740 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.097 2022-04-22 18:08:40,645 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 52.506 INFO:tensorflow:global_step/sec: 13.518 2022-04-22 18:08:41,054 [INFO] tensorflow: global_step/sec: 13.518 2022-04-22 18:08:42,474 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.684 INFO:tensorflow:epoch = 16.343333333333334, learning_rate = 0.00049999997, loss = 1.9796098e-06, step = 9806 (5.104 sec) 2022-04-22 18:08:42,983 [INFO] tensorflow: epoch = 16.343333333333334, learning_rate = 0.00049999997, loss = 1.9796098e-06, step = 9806 (5.104 sec) INFO:tensorflow:epoch = 16.345000000000002, learning_rate = 0.00049999997, loss = 1.9226482e-06, step = 9807 (5.115 sec) 2022-04-22 18:08:43,066 [INFO] tensorflow: epoch = 16.345000000000002, learning_rate = 0.00049999997, loss = 1.9226482e-06, step = 9807 (5.115 sec) 2022-04-22 18:08:44,255 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.148 INFO:tensorflow:global_step/sec: 13.7004 2022-04-22 18:08:45,433 [INFO] tensorflow: global_step/sec: 13.7004 2022-04-22 18:08:46,093 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.414 2022-04-22 18:08:47,959 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.595 INFO:tensorflow:epoch = 16.46, learning_rate = 0.00049999997, loss = 1.8413712e-06, step = 9876 (5.110 sec) 2022-04-22 18:08:48,093 [INFO] tensorflow: epoch = 16.46, learning_rate = 0.00049999997, loss = 1.8413712e-06, step = 9876 (5.110 sec) INFO:tensorflow:epoch = 16.461666666666666, learning_rate = 0.00049999997, loss = 1.8099681e-06, step = 9877 (5.101 sec) 2022-04-22 18:08:48,167 [INFO] tensorflow: epoch = 16.461666666666666, learning_rate = 0.00049999997, loss = 1.8099681e-06, step = 9877 (5.101 sec) 2022-04-22 18:08:49,701 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 57.420 INFO:tensorflow:global_step/sec: 13.8208 2022-04-22 18:08:49,774 [INFO] tensorflow: global_step/sec: 13.8208 2022-04-22 18:08:51,542 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.313 INFO:tensorflow:epoch = 16.576666666666668, learning_rate = 0.00049999997, loss = 1.742621e-06, step = 9946 (5.086 sec) 2022-04-22 18:08:53,179 [INFO] tensorflow: epoch = 16.576666666666668, learning_rate = 0.00049999997, loss = 1.742621e-06, step = 9946 (5.086 sec) INFO:tensorflow:epoch = 16.578333333333333, learning_rate = 0.00049999997, loss = 1.6904796e-06, step = 9947 (5.082 sec) 2022-04-22 18:08:53,249 [INFO] tensorflow: epoch = 16.578333333333333, learning_rate = 0.00049999997, loss = 1.6904796e-06, step = 9947 (5.082 sec) 2022-04-22 18:08:53,381 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.380 INFO:tensorflow:global_step/sec: 13.7255 2022-04-22 18:08:54,146 [INFO] tensorflow: global_step/sec: 13.7255 2022-04-22 18:08:55,122 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 57.477 2022-04-22 18:08:56,906 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.033 INFO:tensorflow:epoch = 16.698333333333334, learning_rate = 0.00049999997, loss = 1.6035792e-06, step = 10019 (5.133 sec) 2022-04-22 18:08:58,313 [INFO] tensorflow: epoch = 16.698333333333334, learning_rate = 0.00049999997, loss = 1.6035792e-06, step = 10019 (5.133 sec) INFO:tensorflow:epoch = 16.700000000000003, learning_rate = 0.00049999997, loss = 1.6200684e-06, step = 10020 (5.157 sec) 2022-04-22 18:08:58,405 [INFO] tensorflow: epoch = 16.700000000000003, learning_rate = 0.00049999997, loss = 1.6200684e-06, step = 10020 (5.157 sec) INFO:tensorflow:global_step/sec: 14.0846 2022-04-22 18:08:58,406 [INFO] tensorflow: global_step/sec: 14.0846 2022-04-22 18:08:58,697 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.860 2022-04-22 18:09:00,697 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 49.987 2022-04-22 18:09:02,539 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.321 INFO:tensorflow:global_step/sec: 13.0913 2022-04-22 18:09:02,989 [INFO] tensorflow: global_step/sec: 13.0913 INFO:tensorflow:epoch = 16.810000000000002, learning_rate = 0.00049999997, loss = 1.5035457e-06, step = 10086 (5.108 sec) 2022-04-22 18:09:03,421 [INFO] tensorflow: epoch = 16.810000000000002, learning_rate = 0.00049999997, loss = 1.5035457e-06, step = 10086 (5.108 sec) INFO:tensorflow:epoch = 16.811666666666667, learning_rate = 0.00049999997, loss = 1.4809048e-06, step = 10087 (5.091 sec) 2022-04-22 18:09:03,497 [INFO] tensorflow: epoch = 16.811666666666667, learning_rate = 0.00049999997, loss = 1.4809048e-06, step = 10087 (5.091 sec) 2022-04-22 18:09:04,425 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.027 2022-04-22 18:09:06,312 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.001 INFO:tensorflow:global_step/sec: 13.2806 2022-04-22 18:09:07,507 [INFO] tensorflow: global_step/sec: 13.2806 2022-04-22 18:09:08,133 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.916 INFO:tensorflow:epoch = 16.923333333333336, learning_rate = 0.00049999997, loss = 1.4058605e-06, step = 10154 (5.087 sec) 2022-04-22 18:09:08,508 [INFO] tensorflow: epoch = 16.923333333333336, learning_rate = 0.00049999997, loss = 1.4058605e-06, step = 10154 (5.087 sec) INFO:tensorflow:epoch = 16.925, learning_rate = 0.00049999997, loss = 1.4021853e-06, step = 10155 (5.091 sec) 2022-04-22 18:09:08,588 [INFO] tensorflow: epoch = 16.925, learning_rate = 0.00049999997, loss = 1.4021853e-06, step = 10155 (5.091 sec) 2022-04-22 18:09:09,933 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.556 2022-04-22 18:09:11,885 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 51.228 INFO:tensorflow:global_step/sec: 13.4651 2022-04-22 18:09:11,963 [INFO] tensorflow: global_step/sec: 13.4651 2022-04-22 18:09:11,964 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 17/100: loss: 0.00000 learning rate: 0.00050 Time taken: 0:00:44.258966 ETA: 1:01:13.494156 INFO:tensorflow:epoch = 17.038333333333334, learning_rate = 0.00049999997, loss = 1.2667315e-06, step = 10223 (5.097 sec) 2022-04-22 18:09:13,604 [INFO] tensorflow: epoch = 17.038333333333334, learning_rate = 0.00049999997, loss = 1.2667315e-06, step = 10223 (5.097 sec) INFO:tensorflow:epoch = 17.040000000000003, learning_rate = 0.00049999997, loss = 1.2948849e-06, step = 10224 (5.082 sec) 2022-04-22 18:09:13,669 [INFO] tensorflow: epoch = 17.040000000000003, learning_rate = 0.00049999997, loss = 1.2948849e-06, step = 10224 (5.082 sec) 2022-04-22 18:09:13,670 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.059 2022-04-22 18:09:15,466 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.678 INFO:tensorflow:global_step/sec: 13.9244 2022-04-22 18:09:16,272 [INFO] tensorflow: global_step/sec: 13.9244 2022-04-22 18:09:17,315 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.084 INFO:tensorflow:epoch = 17.153333333333336, learning_rate = 0.00049999997, loss = 1.1623406e-06, step = 10292 (5.178 sec) INFO:tensorflow:epoch = 17.153333333333336, learning_rate = 0.00049999997, loss = 1.1594266e-06, step = 10292 (5.114 sec) 2022-04-22 18:09:18,783 [INFO] tensorflow: epoch = 17.153333333333336, learning_rate = 0.00049999997, loss = 1.1623406e-06, step = 10292 (5.178 sec) 2022-04-22 18:09:18,783 [INFO] tensorflow: epoch = 17.153333333333336, learning_rate = 0.00049999997, loss = 1.1594266e-06, step = 10292 (5.114 sec) 2022-04-22 18:09:19,400 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 47.965 INFO:tensorflow:global_step/sec: 13.008 2022-04-22 18:09:20,884 [INFO] tensorflow: global_step/sec: 13.008 2022-04-22 18:09:21,156 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.963 2022-04-22 18:09:23,139 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 50.439 INFO:tensorflow:epoch = 17.26666666666667, learning_rate = 0.00049999997, loss = 1.0987242e-06, step = 10360 (5.095 sec) INFO:tensorflow:epoch = 17.26666666666667, learning_rate = 0.00049999997, loss = 1.1077982e-06, step = 10360 (5.095 sec) 2022-04-22 18:09:23,878 [INFO] tensorflow: epoch = 17.26666666666667, learning_rate = 0.00049999997, loss = 1.0987242e-06, step = 10360 (5.095 sec) 2022-04-22 18:09:23,878 [INFO] tensorflow: epoch = 17.26666666666667, learning_rate = 0.00049999997, loss = 1.1077982e-06, step = 10360 (5.095 sec) 2022-04-22 18:09:24,876 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 57.563 INFO:tensorflow:global_step/sec: 13.5122 2022-04-22 18:09:25,325 [INFO] tensorflow: global_step/sec: 13.5122 2022-04-22 18:09:26,738 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.733 2022-04-22 18:09:28,769 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 49.244 INFO:tensorflow:epoch = 17.378333333333334, learning_rate = 0.00049999997, loss = 9.2625396e-07, step = 10427 (5.101 sec) 2022-04-22 18:09:28,979 [INFO] tensorflow: epoch = 17.378333333333334, learning_rate = 0.00049999997, loss = 9.2625396e-07, step = 10427 (5.101 sec) INFO:tensorflow:epoch = 17.378333333333334, learning_rate = 0.00049999997, loss = 9.244768e-07, step = 10427 (5.101 sec) 2022-04-22 18:09:28,980 [INFO] tensorflow: epoch = 17.378333333333334, learning_rate = 0.00049999997, loss = 9.244768e-07, step = 10427 (5.101 sec) INFO:tensorflow:global_step/sec: 12.9397 2022-04-22 18:09:29,962 [INFO] tensorflow: global_step/sec: 12.9397 2022-04-22 18:09:30,612 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.258 2022-04-22 18:09:32,365 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 57.046 INFO:tensorflow:epoch = 17.495, learning_rate = 0.00049999997, loss = 8.422621e-07, step = 10497 (5.091 sec) 2022-04-22 18:09:34,070 [INFO] tensorflow: epoch = 17.495, learning_rate = 0.00049999997, loss = 8.422621e-07, step = 10497 (5.091 sec) INFO:tensorflow:epoch = 17.495, learning_rate = 0.00049999997, loss = 8.3116265e-07, step = 10497 (5.102 sec) 2022-04-22 18:09:34,081 [INFO] tensorflow: epoch = 17.495, learning_rate = 0.00049999997, loss = 8.3116265e-07, step = 10497 (5.102 sec) 2022-04-22 18:09:34,216 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.047 INFO:tensorflow:global_step/sec: 13.8833 2022-04-22 18:09:34,283 [INFO] tensorflow: global_step/sec: 13.8833 2022-04-22 18:09:36,117 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 52.589 2022-04-22 18:09:37,921 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.444 INFO:tensorflow:global_step/sec: 13.4966 2022-04-22 18:09:38,729 [INFO] tensorflow: global_step/sec: 13.4966 INFO:tensorflow:epoch = 17.61, learning_rate = 0.00049999997, loss = 7.627341e-07, step = 10566 (5.090 sec) 2022-04-22 18:09:39,160 [INFO] tensorflow: epoch = 17.61, learning_rate = 0.00049999997, loss = 7.627341e-07, step = 10566 (5.090 sec) INFO:tensorflow:epoch = 17.61, learning_rate = 0.00049999997, loss = 7.6803644e-07, step = 10566 (5.079 sec) 2022-04-22 18:09:39,160 [INFO] tensorflow: epoch = 17.61, learning_rate = 0.00049999997, loss = 7.6803644e-07, step = 10566 (5.079 sec) 2022-04-22 18:09:39,769 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.129 2022-04-22 18:09:41,729 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 51.021 INFO:tensorflow:global_step/sec: 13.1395 2022-04-22 18:09:43,295 [INFO] tensorflow: global_step/sec: 13.1395 2022-04-22 18:09:43,552 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.868 INFO:tensorflow:epoch = 17.721666666666668, learning_rate = 0.00049999997, loss = 6.783757e-07, step = 10633 (5.087 sec) 2022-04-22 18:09:44,247 [INFO] tensorflow: epoch = 17.721666666666668, learning_rate = 0.00049999997, loss = 6.783757e-07, step = 10633 (5.087 sec) INFO:tensorflow:epoch = 17.721666666666668, learning_rate = 0.00049999997, loss = 6.7718247e-07, step = 10633 (5.090 sec) 2022-04-22 18:09:44,250 [INFO] tensorflow: epoch = 17.721666666666668, learning_rate = 0.00049999997, loss = 6.7718247e-07, step = 10633 (5.090 sec) 2022-04-22 18:09:45,420 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.547 2022-04-22 18:09:47,235 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.079 INFO:tensorflow:global_step/sec: 13.649 2022-04-22 18:09:47,691 [INFO] tensorflow: global_step/sec: 13.649 2022-04-22 18:09:49,038 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.486 INFO:tensorflow:epoch = 17.838333333333335, learning_rate = 0.00049999997, loss = 5.6503904e-07, step = 10703 (5.083 sec) 2022-04-22 18:09:49,333 [INFO] tensorflow: epoch = 17.838333333333335, learning_rate = 0.00049999997, loss = 5.6503904e-07, step = 10703 (5.083 sec) INFO:tensorflow:epoch = 17.838333333333335, learning_rate = 0.00049999997, loss = 5.63291e-07, step = 10703 (5.090 sec) 2022-04-22 18:09:49,337 [INFO] tensorflow: epoch = 17.838333333333335, learning_rate = 0.00049999997, loss = 5.63291e-07, step = 10703 (5.090 sec) 2022-04-22 18:09:50,802 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.695 INFO:tensorflow:global_step/sec: 14.315 2022-04-22 18:09:51,883 [INFO] tensorflow: global_step/sec: 14.315 2022-04-22 18:09:52,562 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.815 2022-04-22 18:09:54,362 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.565 INFO:tensorflow:epoch = 17.958333333333336, learning_rate = 0.00049999997, loss = 5.2103314e-07, step = 10775 (5.096 sec) INFO:tensorflow:epoch = 17.958333333333336, learning_rate = 0.00049999997, loss = 5.2294854e-07, step = 10775 (5.100 sec) 2022-04-22 18:09:54,433 [INFO] tensorflow: epoch = 17.958333333333336, learning_rate = 0.00049999997, loss = 5.2103314e-07, step = 10775 (5.096 sec) 2022-04-22 18:09:54,433 [INFO] tensorflow: epoch = 17.958333333333336, learning_rate = 0.00049999997, loss = 5.2294854e-07, step = 10775 (5.100 sec) 2022-04-22 18:09:56,186 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.835 INFO:tensorflow:global_step/sec: 13.7197 2022-04-22 18:09:56,256 [INFO] tensorflow: global_step/sec: 13.7197 2022-04-22 18:09:56,257 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 18/100: loss: 0.00000 learning rate: 0.00050 Time taken: 0:00:44.299958 ETA: 1:00:32.596516 2022-04-22 18:09:58,046 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.780 INFO:tensorflow:epoch = 18.073333333333334, learning_rate = 0.00049999997, loss = 4.2666215e-07, step = 10844 (5.100 sec) 2022-04-22 18:09:59,533 [INFO] tensorflow: epoch = 18.073333333333334, learning_rate = 0.00049999997, loss = 4.2666215e-07, step = 10844 (5.100 sec) INFO:tensorflow:epoch = 18.073333333333334, learning_rate = 0.00049999997, loss = 4.1961175e-07, step = 10844 (5.100 sec) 2022-04-22 18:09:59,533 [INFO] tensorflow: epoch = 18.073333333333334, learning_rate = 0.00049999997, loss = 4.1961175e-07, step = 10844 (5.100 sec) 2022-04-22 18:09:59,927 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.161 INFO:tensorflow:global_step/sec: 13.4266 2022-04-22 18:10:00,725 [INFO] tensorflow: global_step/sec: 13.4266 2022-04-22 18:10:01,857 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 51.822 2022-04-22 18:10:03,667 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.263 INFO:tensorflow:epoch = 18.188333333333336, learning_rate = 0.00049999997, loss = 2.694551e-07, step = 10913 (5.128 sec) INFO:tensorflow:epoch = 18.188333333333336, learning_rate = 0.00049999997, loss = 2.7436818e-07, step = 10913 (5.128 sec) 2022-04-22 18:10:04,661 [INFO] tensorflow: epoch = 18.188333333333336, learning_rate = 0.00049999997, loss = 2.694551e-07, step = 10913 (5.128 sec) 2022-04-22 18:10:04,661 [INFO] tensorflow: epoch = 18.188333333333336, learning_rate = 0.00049999997, loss = 2.7436818e-07, step = 10913 (5.128 sec) INFO:tensorflow:global_step/sec: 13.5403 2022-04-22 18:10:05,156 [INFO] tensorflow: global_step/sec: 13.5403 2022-04-22 18:10:05,458 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.831 2022-04-22 18:10:07,380 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 52.034 2022-04-22 18:10:09,178 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.618 INFO:tensorflow:global_step/sec: 13.4455 2022-04-22 18:10:09,618 [INFO] tensorflow: global_step/sec: 13.4455 INFO:tensorflow:epoch = 18.303333333333335, learning_rate = 0.00049999997, loss = 1.9974141e-07, step = 10982 (5.113 sec) 2022-04-22 18:10:09,774 [INFO] tensorflow: epoch = 18.303333333333335, learning_rate = 0.00049999997, loss = 1.9974141e-07, step = 10982 (5.113 sec) INFO:tensorflow:epoch = 18.303333333333335, learning_rate = 0.00049999997, loss = 2.0149037e-07, step = 10982 (5.114 sec) 2022-04-22 18:10:09,775 [INFO] tensorflow: epoch = 18.303333333333335, learning_rate = 0.00049999997, loss = 2.0149037e-07, step = 10982 (5.114 sec) 2022-04-22 18:10:10,980 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.516 2022-04-22 18:10:12,784 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.431 INFO:tensorflow:global_step/sec: 13.3768 2022-04-22 18:10:14,104 [INFO] tensorflow: global_step/sec: 13.3768 2022-04-22 18:10:14,765 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 50.491 INFO:tensorflow:epoch = 18.418333333333333, learning_rate = 0.00049999997, loss = 1.2801684e-07, step = 11051 (5.122 sec) 2022-04-22 18:10:14,897 [INFO] tensorflow: epoch = 18.418333333333333, learning_rate = 0.00049999997, loss = 1.2801684e-07, step = 11051 (5.122 sec) INFO:tensorflow:epoch = 18.418333333333333, learning_rate = 0.00049999997, loss = 1.2038436e-07, step = 11051 (5.125 sec) 2022-04-22 18:10:14,899 [INFO] tensorflow: epoch = 18.418333333333333, learning_rate = 0.00049999997, loss = 1.2038436e-07, step = 11051 (5.125 sec) 2022-04-22 18:10:16,519 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 57.027 2022-04-22 18:10:18,375 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.888 INFO:tensorflow:global_step/sec: 13.803 2022-04-22 18:10:18,451 [INFO] tensorflow: global_step/sec: 13.803 INFO:tensorflow:epoch = 18.535, learning_rate = 0.00049999997, loss = 4.077083e-08, step = 11121 (5.097 sec) 2022-04-22 18:10:19,993 [INFO] tensorflow: epoch = 18.535, learning_rate = 0.00049999997, loss = 4.077083e-08, step = 11121 (5.097 sec) INFO:tensorflow:epoch = 18.535, learning_rate = 0.00049999997, loss = 4.254798e-08, step = 11121 (5.095 sec) 2022-04-22 18:10:19,994 [INFO] tensorflow: epoch = 18.535, learning_rate = 0.00049999997, loss = 4.254798e-08, step = 11121 (5.095 sec) 2022-04-22 18:10:20,249 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.372 2022-04-22 18:10:22,020 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.472 INFO:tensorflow:global_step/sec: 13.6826 2022-04-22 18:10:22,836 [INFO] tensorflow: global_step/sec: 13.6826 2022-04-22 18:10:23,872 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.994 INFO:tensorflow:epoch = 18.650000000000002, learning_rate = 0.00049999997, loss = -2.7835995e-08, step = 11190 (5.073 sec) 2022-04-22 18:10:25,066 [INFO] tensorflow: epoch = 18.650000000000002, learning_rate = 0.00049999997, loss = -2.7835995e-08, step = 11190 (5.073 sec) INFO:tensorflow:epoch = 18.650000000000002, learning_rate = 0.00049999997, loss = -3.313835e-08, step = 11190 (5.072 sec) 2022-04-22 18:10:25,066 [INFO] tensorflow: epoch = 18.650000000000002, learning_rate = 0.00049999997, loss = -3.313835e-08, step = 11190 (5.072 sec) 2022-04-22 18:10:25,733 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.725 INFO:tensorflow:global_step/sec: 12.7666 2022-04-22 18:10:27,536 [INFO] tensorflow: global_step/sec: 12.7666 2022-04-22 18:10:27,816 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 48.016 2022-04-22 18:10:29,613 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.677 INFO:tensorflow:epoch = 18.761666666666667, learning_rate = 0.00049999997, loss = -1.5310525e-07, step = 11257 (5.098 sec) 2022-04-22 18:10:30,164 [INFO] tensorflow: epoch = 18.761666666666667, learning_rate = 0.00049999997, loss = -1.5310525e-07, step = 11257 (5.098 sec) INFO:tensorflow:epoch = 18.761666666666667, learning_rate = 0.00049999997, loss = -1.5546357e-07, step = 11257 (5.098 sec) 2022-04-22 18:10:30,164 [INFO] tensorflow: epoch = 18.761666666666667, learning_rate = 0.00049999997, loss = -1.5546357e-07, step = 11257 (5.098 sec) 2022-04-22 18:10:31,508 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 52.751 INFO:tensorflow:global_step/sec: 13.4616 2022-04-22 18:10:31,993 [INFO] tensorflow: global_step/sec: 13.4616 2022-04-22 18:10:33,586 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 48.147 INFO:tensorflow:epoch = 18.87, learning_rate = 0.00049999997, loss = -2.4476594e-07, step = 11322 (5.093 sec) 2022-04-22 18:10:35,257 [INFO] tensorflow: epoch = 18.87, learning_rate = 0.00049999997, loss = -2.4476594e-07, step = 11322 (5.093 sec) INFO:tensorflow:epoch = 18.87, learning_rate = 0.00049999997, loss = -2.399347e-07, step = 11322 (5.093 sec) 2022-04-22 18:10:35,257 [INFO] tensorflow: epoch = 18.87, learning_rate = 0.00049999997, loss = -2.399347e-07, step = 11322 (5.093 sec) 2022-04-22 18:10:35,389 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.474 INFO:tensorflow:global_step/sec: 12.9233 2022-04-22 18:10:36,636 [INFO] tensorflow: global_step/sec: 12.9233 2022-04-22 18:10:37,382 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 50.172 2022-04-22 18:10:39,201 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.988 INFO:tensorflow:epoch = 18.98, learning_rate = 0.00049999997, loss = -2.9116745e-07, step = 11388 (5.102 sec) 2022-04-22 18:10:40,359 [INFO] tensorflow: epoch = 18.98, learning_rate = 0.00049999997, loss = -2.9116745e-07, step = 11388 (5.102 sec) INFO:tensorflow:epoch = 18.98, learning_rate = 0.00049999997, loss = -2.866418e-07, step = 11388 (5.119 sec) 2022-04-22 18:10:40,376 [INFO] tensorflow: epoch = 18.98, learning_rate = 0.00049999997, loss = -2.866418e-07, step = 11388 (5.119 sec) 2022-04-22 18:10:41,237 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 49.127 INFO:tensorflow:global_step/sec: 12.834 2022-04-22 18:10:41,311 [INFO] tensorflow: global_step/sec: 12.834 2022-04-22 18:10:41,311 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 19/100: loss: -0.00000 learning rate: 0.00050 Time taken: 0:00:45.050043 ETA: 1:00:49.053472 2022-04-22 18:10:43,223 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 50.349 2022-04-22 18:10:45,307 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 48.007 INFO:tensorflow:epoch = 19.085, learning_rate = 0.00049999997, loss = -3.510604e-07, step = 11451 (5.075 sec) 2022-04-22 18:10:45,451 [INFO] tensorflow: epoch = 19.085, learning_rate = 0.00049999997, loss = -3.510604e-07, step = 11451 (5.075 sec) INFO:tensorflow:epoch = 19.085, learning_rate = 0.00049999997, loss = -3.4684035e-07, step = 11451 (5.093 sec) 2022-04-22 18:10:45,451 [INFO] tensorflow: epoch = 19.085, learning_rate = 0.00049999997, loss = -3.4684035e-07, step = 11451 (5.093 sec) INFO:tensorflow:global_step/sec: 12.3761 2022-04-22 18:10:46,159 [INFO] tensorflow: global_step/sec: 12.3761 2022-04-22 18:10:47,260 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 51.205 2022-04-22 18:10:49,095 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.496 INFO:tensorflow:epoch = 19.198333333333334, learning_rate = 0.00049999997, loss = -4.2832653e-07, step = 11519 (5.078 sec) INFO:tensorflow:epoch = 19.198333333333334, learning_rate = 0.00049999997, loss = -4.3118507e-07, step = 11519 (5.078 sec) 2022-04-22 18:10:50,529 [INFO] tensorflow: epoch = 19.198333333333334, learning_rate = 0.00049999997, loss = -4.2832653e-07, step = 11519 (5.078 sec) 2022-04-22 18:10:50,529 [INFO] tensorflow: epoch = 19.198333333333334, learning_rate = 0.00049999997, loss = -4.3118507e-07, step = 11519 (5.078 sec) INFO:tensorflow:global_step/sec: 13.4904 2022-04-22 18:10:50,606 [INFO] tensorflow: global_step/sec: 13.4904 2022-04-22 18:10:50,983 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 52.982 2022-04-22 18:10:52,803 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.947 2022-04-22 18:10:54,563 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.814 INFO:tensorflow:global_step/sec: 13.7158 2022-04-22 18:10:54,981 [INFO] tensorflow: global_step/sec: 13.7158 INFO:tensorflow:epoch = 19.315, learning_rate = 0.00049999997, loss = -5.166339e-07, step = 11589 (5.079 sec) 2022-04-22 18:10:55,609 [INFO] tensorflow: epoch = 19.315, learning_rate = 0.00049999997, loss = -5.166339e-07, step = 11589 (5.079 sec) INFO:tensorflow:epoch = 19.316666666666666, learning_rate = 0.00049999997, loss = -4.9856135e-07, step = 11590 (5.152 sec) 2022-04-22 18:10:55,681 [INFO] tensorflow: epoch = 19.316666666666666, learning_rate = 0.00049999997, loss = -4.9856135e-07, step = 11590 (5.152 sec) 2022-04-22 18:10:56,362 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.605 2022-04-22 18:10:58,238 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.308 INFO:tensorflow:global_step/sec: 13.1812 2022-04-22 18:10:59,533 [INFO] tensorflow: global_step/sec: 13.1812 2022-04-22 18:11:00,313 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 48.199 INFO:tensorflow:epoch = 19.425, learning_rate = 0.00049999997, loss = -5.339598e-07, step = 11655 (5.111 sec) 2022-04-22 18:11:00,720 [INFO] tensorflow: epoch = 19.425, learning_rate = 0.00049999997, loss = -5.339598e-07, step = 11655 (5.111 sec) INFO:tensorflow:epoch = 19.42666666666667, learning_rate = 0.00049999997, loss = -5.3448457e-07, step = 11656 (5.113 sec) 2022-04-22 18:11:00,795 [INFO] tensorflow: epoch = 19.42666666666667, learning_rate = 0.00049999997, loss = -5.3448457e-07, step = 11656 (5.113 sec) 2022-04-22 18:11:02,077 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.713 2022-04-22 18:11:03,970 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 52.831 INFO:tensorflow:global_step/sec: 13.3239 2022-04-22 18:11:04,036 [INFO] tensorflow: global_step/sec: 13.3239 2022-04-22 18:11:05,790 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.955 INFO:tensorflow:epoch = 19.541666666666668, learning_rate = 0.00049999997, loss = -6.279315e-07, step = 11725 (5.138 sec) 2022-04-22 18:11:05,857 [INFO] tensorflow: epoch = 19.541666666666668, learning_rate = 0.00049999997, loss = -6.279315e-07, step = 11725 (5.138 sec) INFO:tensorflow:epoch = 19.543333333333333, learning_rate = 0.00049999997, loss = -6.388482e-07, step = 11726 (5.147 sec) 2022-04-22 18:11:05,942 [INFO] tensorflow: epoch = 19.543333333333333, learning_rate = 0.00049999997, loss = -6.388482e-07, step = 11726 (5.147 sec) 2022-04-22 18:11:07,659 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.487 INFO:tensorflow:global_step/sec: 13.4814 2022-04-22 18:11:08,486 [INFO] tensorflow: global_step/sec: 13.4814 2022-04-22 18:11:09,461 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.500 INFO:tensorflow:epoch = 19.658333333333335, learning_rate = 0.00049999997, loss = -5.96845e-07, step = 11795 (5.075 sec) 2022-04-22 18:11:10,932 [INFO] tensorflow: epoch = 19.658333333333335, learning_rate = 0.00049999997, loss = -5.96845e-07, step = 11795 (5.075 sec) INFO:tensorflow:epoch = 19.66166666666667, learning_rate = 0.00049999997, loss = -6.3610787e-07, step = 11797 (5.137 sec) 2022-04-22 18:11:11,079 [INFO] tensorflow: epoch = 19.66166666666667, learning_rate = 0.00049999997, loss = -6.3610787e-07, step = 11797 (5.137 sec) 2022-04-22 18:11:11,216 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.985 INFO:tensorflow:global_step/sec: 13.8077 2022-04-22 18:11:12,832 [INFO] tensorflow: global_step/sec: 13.8077 2022-04-22 18:11:13,185 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 50.811 2022-04-22 18:11:15,134 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 51.311 INFO:tensorflow:epoch = 19.77, learning_rate = 0.00049999997, loss = -7.8309404e-07, step = 11862 (5.081 sec) 2022-04-22 18:11:16,013 [INFO] tensorflow: epoch = 19.77, learning_rate = 0.00049999997, loss = -7.8309404e-07, step = 11862 (5.081 sec) INFO:tensorflow:epoch = 19.773333333333333, learning_rate = 0.00049999997, loss = -7.3721276e-07, step = 11864 (5.085 sec) 2022-04-22 18:11:16,165 [INFO] tensorflow: epoch = 19.773333333333333, learning_rate = 0.00049999997, loss = -7.3721276e-07, step = 11864 (5.085 sec) 2022-04-22 18:11:16,893 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.850 INFO:tensorflow:global_step/sec: 13.3179 2022-04-22 18:11:17,337 [INFO] tensorflow: global_step/sec: 13.3179 2022-04-22 18:11:18,778 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.051 2022-04-22 18:11:20,632 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.971 INFO:tensorflow:epoch = 19.885, learning_rate = 0.00049999997, loss = -8.5587635e-07, step = 11931 (5.085 sec) 2022-04-22 18:11:21,098 [INFO] tensorflow: epoch = 19.885, learning_rate = 0.00049999997, loss = -8.5587635e-07, step = 11931 (5.085 sec) INFO:tensorflow:epoch = 19.89, learning_rate = 0.00049999997, loss = -8.335037e-07, step = 11934 (5.139 sec) 2022-04-22 18:11:21,304 [INFO] tensorflow: epoch = 19.89, learning_rate = 0.00049999997, loss = -8.335037e-07, step = 11934 (5.139 sec) INFO:tensorflow:global_step/sec: 13.2114 2022-04-22 18:11:21,879 [INFO] tensorflow: global_step/sec: 13.2114 2022-04-22 18:11:22,589 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 51.093 2022-04-22 18:11:24,460 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.444 INFO:tensorflow:Saving checkpoints for step-12000. 2022-04-22 18:11:26,137 [INFO] tensorflow: Saving checkpoints for step-12000. 2022-04-22 18:11:30,562 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 16.389 INFO:tensorflow:epoch = 20.0, learning_rate = 0.00049999997, loss = -9.1325455e-07, step = 12000 (9.335 sec) 2022-04-22 18:11:30,639 [INFO] tensorflow: epoch = 20.0, learning_rate = 0.00049999997, loss = -9.1325455e-07, step = 12000 (9.335 sec) INFO:tensorflow:epoch = 20.0, learning_rate = 0.00049999997, loss = -9.160285e-07, step = 12000 (9.541 sec) 2022-04-22 18:11:30,639 [INFO] tensorflow: epoch = 20.0, learning_rate = 0.00049999997, loss = -9.160285e-07, step = 12000 (9.541 sec) INFO:tensorflow:global_step/sec: 6.84876 2022-04-22 18:11:30,639 [INFO] tensorflow: global_step/sec: 6.84876 2022-04-22 18:11:30,640 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 20/100: loss: -0.00000 learning rate: 0.00050 Time taken: 0:00:49.324923 ETA: 1:05:45.993843 2022-04-22 18:11:32,422 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.792 2022-04-22 18:11:34,391 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 50.801 INFO:tensorflow:global_step/sec: 13.0149 2022-04-22 18:11:35,249 [INFO] tensorflow: global_step/sec: 13.0149 INFO:tensorflow:epoch = 20.111666666666668, learning_rate = 0.00049999997, loss = -1.0301501e-06, step = 12067 (5.083 sec) 2022-04-22 18:11:35,721 [INFO] tensorflow: epoch = 20.111666666666668, learning_rate = 0.00049999997, loss = -1.0301501e-06, step = 12067 (5.083 sec) INFO:tensorflow:epoch = 20.111666666666668, learning_rate = 0.00049999997, loss = -1.0274298e-06, step = 12067 (5.083 sec) 2022-04-22 18:11:35,722 [INFO] tensorflow: epoch = 20.111666666666668, learning_rate = 0.00049999997, loss = -1.0274298e-06, step = 12067 (5.083 sec) 2022-04-22 18:11:36,251 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.761 2022-04-22 18:11:38,036 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.032 INFO:tensorflow:global_step/sec: 14.0579 2022-04-22 18:11:39,517 [INFO] tensorflow: global_step/sec: 14.0579 2022-04-22 18:11:39,812 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.313 INFO:tensorflow:epoch = 20.23166666666667, learning_rate = 0.00049999997, loss = -1.0700733e-06, step = 12139 (5.116 sec) 2022-04-22 18:11:40,837 [INFO] tensorflow: epoch = 20.23166666666667, learning_rate = 0.00049999997, loss = -1.0700733e-06, step = 12139 (5.116 sec) INFO:tensorflow:epoch = 20.23166666666667, learning_rate = 0.00049999997, loss = -1.0691874e-06, step = 12139 (5.115 sec) 2022-04-22 18:11:40,837 [INFO] tensorflow: epoch = 20.23166666666667, learning_rate = 0.00049999997, loss = -1.0691874e-06, step = 12139 (5.115 sec) 2022-04-22 18:11:41,588 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.301 2022-04-22 18:11:43,408 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.971 INFO:tensorflow:global_step/sec: 13.75 2022-04-22 18:11:43,881 [INFO] tensorflow: global_step/sec: 13.75 2022-04-22 18:11:45,248 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.356 INFO:tensorflow:epoch = 20.346666666666668, learning_rate = 0.00049999997, loss = -1.1445663e-06, step = 12208 (5.085 sec) 2022-04-22 18:11:45,922 [INFO] tensorflow: epoch = 20.346666666666668, learning_rate = 0.00049999997, loss = -1.1445663e-06, step = 12208 (5.085 sec) INFO:tensorflow:epoch = 20.348333333333336, learning_rate = 0.00049999997, loss = -1.1547527e-06, step = 12209 (5.156 sec) 2022-04-22 18:11:45,993 [INFO] tensorflow: epoch = 20.348333333333336, learning_rate = 0.00049999997, loss = -1.1547527e-06, step = 12209 (5.156 sec) 2022-04-22 18:11:47,090 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.278 INFO:tensorflow:global_step/sec: 13.6724 2022-04-22 18:11:48,269 [INFO] tensorflow: global_step/sec: 13.6724 2022-04-22 18:11:49,010 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 52.102 2022-04-22 18:11:50,944 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 51.706 INFO:tensorflow:epoch = 20.458333333333336, learning_rate = 0.00049999997, loss = -1.2022192e-06, step = 12275 (5.100 sec) 2022-04-22 18:11:51,022 [INFO] tensorflow: epoch = 20.458333333333336, learning_rate = 0.00049999997, loss = -1.2022192e-06, step = 12275 (5.100 sec) INFO:tensorflow:epoch = 20.46, learning_rate = 0.00049999997, loss = -1.2061446e-06, step = 12276 (5.105 sec) 2022-04-22 18:11:51,097 [INFO] tensorflow: epoch = 20.46, learning_rate = 0.00049999997, loss = -1.2061446e-06, step = 12276 (5.105 sec) 2022-04-22 18:11:52,829 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.068 INFO:tensorflow:global_step/sec: 12.8803 2022-04-22 18:11:52,928 [INFO] tensorflow: global_step/sec: 12.8803 2022-04-22 18:11:54,767 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 51.594 INFO:tensorflow:epoch = 20.57, learning_rate = 0.00049999997, loss = -1.2990013e-06, step = 12342 (5.069 sec) 2022-04-22 18:11:56,091 [INFO] tensorflow: epoch = 20.57, learning_rate = 0.00049999997, loss = -1.2990013e-06, step = 12342 (5.069 sec) INFO:tensorflow:epoch = 20.573333333333334, learning_rate = 0.00049999997, loss = -1.2676282e-06, step = 12344 (5.136 sec) 2022-04-22 18:11:56,233 [INFO] tensorflow: epoch = 20.573333333333334, learning_rate = 0.00049999997, loss = -1.2676282e-06, step = 12344 (5.136 sec) 2022-04-22 18:11:56,594 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.765 INFO:tensorflow:global_step/sec: 13.3514 2022-04-22 18:11:57,422 [INFO] tensorflow: global_step/sec: 13.3514 2022-04-22 18:11:58,461 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.546 2022-04-22 18:12:00,226 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.684 INFO:tensorflow:epoch = 20.686666666666667, learning_rate = 0.00049999997, loss = -1.3520175e-06, step = 12412 (5.089 sec) 2022-04-22 18:12:01,180 [INFO] tensorflow: epoch = 20.686666666666667, learning_rate = 0.00049999997, loss = -1.3520175e-06, step = 12412 (5.089 sec) INFO:tensorflow:epoch = 20.69, learning_rate = 0.00049999997, loss = -1.3429917e-06, step = 12414 (5.093 sec) 2022-04-22 18:12:01,327 [INFO] tensorflow: epoch = 20.69, learning_rate = 0.00049999997, loss = -1.3429917e-06, step = 12414 (5.093 sec) INFO:tensorflow:global_step/sec: 13.7761 2022-04-22 18:12:01,777 [INFO] tensorflow: global_step/sec: 13.7761 2022-04-22 18:12:02,070 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.232 2022-04-22 18:12:03,961 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 52.884 2022-04-22 18:12:05,924 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 50.954 INFO:tensorflow:epoch = 20.798333333333336, learning_rate = 0.00049999997, loss = -1.415483e-06, step = 12479 (5.118 sec) 2022-04-22 18:12:06,298 [INFO] tensorflow: epoch = 20.798333333333336, learning_rate = 0.00049999997, loss = -1.415483e-06, step = 12479 (5.118 sec) INFO:tensorflow:global_step/sec: 13.0564 2022-04-22 18:12:06,373 [INFO] tensorflow: global_step/sec: 13.0564 INFO:tensorflow:epoch = 20.80166666666667, learning_rate = 0.00049999997, loss = -1.3960416e-06, step = 12481 (5.116 sec) 2022-04-22 18:12:06,442 [INFO] tensorflow: epoch = 20.80166666666667, learning_rate = 0.00049999997, loss = -1.3960416e-06, step = 12481 (5.116 sec) 2022-04-22 18:12:07,754 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.641 2022-04-22 18:12:09,562 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.313 INFO:tensorflow:global_step/sec: 13.6928 2022-04-22 18:12:10,754 [INFO] tensorflow: global_step/sec: 13.6928 INFO:tensorflow:epoch = 20.915000000000003, learning_rate = 0.00049999997, loss = -1.4581719e-06, step = 12549 (5.092 sec) 2022-04-22 18:12:11,390 [INFO] tensorflow: epoch = 20.915000000000003, learning_rate = 0.00049999997, loss = -1.4581719e-06, step = 12549 (5.092 sec) 2022-04-22 18:12:11,391 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.687 INFO:tensorflow:epoch = 20.918333333333333, learning_rate = 0.00049999997, loss = -1.4735224e-06, step = 12551 (5.097 sec) 2022-04-22 18:12:11,539 [INFO] tensorflow: epoch = 20.918333333333333, learning_rate = 0.00049999997, loss = -1.4735224e-06, step = 12551 (5.097 sec) 2022-04-22 18:12:13,255 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.642 2022-04-22 18:12:15,069 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 299, 0.00s/step 2022-04-22 18:12:15,296 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 299, 0.02s/step 2022-04-22 18:12:15,531 [INFO] iva.detectnet_v2.evaluation.evaluation: step 20 / 299, 0.02s/step 2022-04-22 18:12:15,773 [INFO] iva.detectnet_v2.evaluation.evaluation: step 30 / 299, 0.02s/step 2022-04-22 18:12:16,015 [INFO] iva.detectnet_v2.evaluation.evaluation: step 40 / 299, 0.02s/step 2022-04-22 18:12:16,253 [INFO] iva.detectnet_v2.evaluation.evaluation: step 50 / 299, 0.02s/step 2022-04-22 18:12:16,491 [INFO] iva.detectnet_v2.evaluation.evaluation: step 60 / 299, 0.02s/step 2022-04-22 18:12:16,738 [INFO] iva.detectnet_v2.evaluation.evaluation: step 70 / 299, 0.02s/step 2022-04-22 18:12:16,994 [INFO] iva.detectnet_v2.evaluation.evaluation: step 80 / 299, 0.03s/step 2022-04-22 18:12:17,241 [INFO] iva.detectnet_v2.evaluation.evaluation: step 90 / 299, 0.02s/step 2022-04-22 18:12:17,489 [INFO] iva.detectnet_v2.evaluation.evaluation: step 100 / 299, 0.02s/step 2022-04-22 18:12:17,736 [INFO] iva.detectnet_v2.evaluation.evaluation: step 110 / 299, 0.02s/step 2022-04-22 18:12:17,980 [INFO] iva.detectnet_v2.evaluation.evaluation: step 120 / 299, 0.02s/step 2022-04-22 18:12:18,226 [INFO] iva.detectnet_v2.evaluation.evaluation: step 130 / 299, 0.02s/step 2022-04-22 18:12:18,480 [INFO] iva.detectnet_v2.evaluation.evaluation: step 140 / 299, 0.03s/step 2022-04-22 18:12:18,728 [INFO] iva.detectnet_v2.evaluation.evaluation: step 150 / 299, 0.02s/step 2022-04-22 18:12:18,972 [INFO] iva.detectnet_v2.evaluation.evaluation: step 160 / 299, 0.02s/step 2022-04-22 18:12:19,209 [INFO] iva.detectnet_v2.evaluation.evaluation: step 170 / 299, 0.02s/step 2022-04-22 18:12:19,445 [INFO] iva.detectnet_v2.evaluation.evaluation: step 180 / 299, 0.02s/step 2022-04-22 18:12:19,679 [INFO] iva.detectnet_v2.evaluation.evaluation: step 190 / 299, 0.02s/step 2022-04-22 18:12:19,914 [INFO] iva.detectnet_v2.evaluation.evaluation: step 200 / 299, 0.02s/step 2022-04-22 18:12:20,151 [INFO] iva.detectnet_v2.evaluation.evaluation: step 210 / 299, 0.02s/step 2022-04-22 18:12:20,385 [INFO] iva.detectnet_v2.evaluation.evaluation: step 220 / 299, 0.02s/step 2022-04-22 18:12:20,628 [INFO] iva.detectnet_v2.evaluation.evaluation: step 230 / 299, 0.02s/step 2022-04-22 18:12:20,862 [INFO] iva.detectnet_v2.evaluation.evaluation: step 240 / 299, 0.02s/step 2022-04-22 18:12:21,100 [INFO] iva.detectnet_v2.evaluation.evaluation: step 250 / 299, 0.02s/step 2022-04-22 18:12:21,339 [INFO] iva.detectnet_v2.evaluation.evaluation: step 260 / 299, 0.02s/step 2022-04-22 18:12:21,578 [INFO] iva.detectnet_v2.evaluation.evaluation: step 270 / 299, 0.02s/step 2022-04-22 18:12:21,813 [INFO] iva.detectnet_v2.evaluation.evaluation: step 280 / 299, 0.02s/step 2022-04-22 18:12:22,047 [INFO] iva.detectnet_v2.evaluation.evaluation: step 290 / 299, 0.02s/step Epoch 21/100 ========================= Validation cost: -0.000010 Mean average_precision (in %): 0.0000 class name average precision (in %) ------------ -------------------------- face 0 Median Inference Time: 0.011727 2022-04-22 18:12:22,316 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 11.037 INFO:tensorflow:epoch = 21.0, learning_rate = 0.00049999997, loss = -1.4865773e-06, step = 12600 (10.839 sec) 2022-04-22 18:12:22,379 [INFO] tensorflow: epoch = 21.0, learning_rate = 0.00049999997, loss = -1.4865773e-06, step = 12600 (10.839 sec) INFO:tensorflow:global_step/sec: 5.16134 2022-04-22 18:12:22,379 [INFO] tensorflow: global_step/sec: 5.16134 2022-04-22 18:12:22,380 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 21/100: loss: -0.00000 learning rate: 0.00050 Time taken: 0:00:51.753064 ETA: 1:08:08.492049 INFO:tensorflow:epoch = 21.00166666666667, learning_rate = 0.00049999997, loss = -1.5146106e-06, step = 12601 (11.065 sec) 2022-04-22 18:12:22,455 [INFO] tensorflow: epoch = 21.00166666666667, learning_rate = 0.00049999997, loss = -1.5146106e-06, step = 12601 (11.065 sec) 2022-04-22 18:12:24,031 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 58.328 2022-04-22 18:12:25,784 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 57.061 INFO:tensorflow:global_step/sec: 14.1597 2022-04-22 18:12:26,617 [INFO] tensorflow: global_step/sec: 14.1597 INFO:tensorflow:epoch = 21.118333333333336, learning_rate = 0.00049999997, loss = -1.5715177e-06, step = 12671 (5.080 sec) 2022-04-22 18:12:27,459 [INFO] tensorflow: epoch = 21.118333333333336, learning_rate = 0.00049999997, loss = -1.5715177e-06, step = 12671 (5.080 sec) INFO:tensorflow:epoch = 21.12, learning_rate = 0.00049999997, loss = -1.6068561e-06, step = 12672 (5.077 sec) 2022-04-22 18:12:27,532 [INFO] tensorflow: epoch = 21.12, learning_rate = 0.00049999997, loss = -1.6068561e-06, step = 12672 (5.077 sec) 2022-04-22 18:12:27,675 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 52.877 2022-04-22 18:12:29,421 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 57.288 INFO:tensorflow:global_step/sec: 13.7959 2022-04-22 18:12:30,966 [INFO] tensorflow: global_step/sec: 13.7959 2022-04-22 18:12:31,275 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.927 INFO:tensorflow:epoch = 21.236666666666668, learning_rate = 0.00049999997, loss = -1.6598797e-06, step = 12742 (5.077 sec) 2022-04-22 18:12:32,536 [INFO] tensorflow: epoch = 21.236666666666668, learning_rate = 0.00049999997, loss = -1.6598797e-06, step = 12742 (5.077 sec) INFO:tensorflow:epoch = 21.238333333333333, learning_rate = 0.00049999997, loss = -1.6597114e-06, step = 12743 (5.076 sec) 2022-04-22 18:12:32,608 [INFO] tensorflow: epoch = 21.238333333333333, learning_rate = 0.00049999997, loss = -1.6597114e-06, step = 12743 (5.076 sec) 2022-04-22 18:12:33,054 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.218 2022-04-22 18:12:34,860 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.394 INFO:tensorflow:global_step/sec: 13.948 2022-04-22 18:12:35,267 [INFO] tensorflow: global_step/sec: 13.948 2022-04-22 18:12:36,569 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 58.511 INFO:tensorflow:epoch = 21.353333333333335, learning_rate = 0.00049999997, loss = -1.7128677e-06, step = 12812 (5.088 sec) 2022-04-22 18:12:37,624 [INFO] tensorflow: epoch = 21.353333333333335, learning_rate = 0.00049999997, loss = -1.7128677e-06, step = 12812 (5.088 sec) INFO:tensorflow:epoch = 21.355, learning_rate = 0.00049999997, loss = -1.7237917e-06, step = 12813 (5.086 sec) 2022-04-22 18:12:37,695 [INFO] tensorflow: epoch = 21.355, learning_rate = 0.00049999997, loss = -1.7237917e-06, step = 12813 (5.086 sec) 2022-04-22 18:12:38,517 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 51.330 INFO:tensorflow:global_step/sec: 13.7449 2022-04-22 18:12:39,633 [INFO] tensorflow: global_step/sec: 13.7449 2022-04-22 18:12:40,253 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 57.629 2022-04-22 18:12:42,148 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 52.761 INFO:tensorflow:epoch = 21.470000000000002, learning_rate = 0.00049999997, loss = -1.7730144e-06, step = 12882 (5.120 sec) 2022-04-22 18:12:42,744 [INFO] tensorflow: epoch = 21.470000000000002, learning_rate = 0.00049999997, loss = -1.7730144e-06, step = 12882 (5.120 sec) INFO:tensorflow:epoch = 21.471666666666668, learning_rate = 0.00049999997, loss = -1.7707198e-06, step = 12883 (5.124 sec) 2022-04-22 18:12:42,818 [INFO] tensorflow: epoch = 21.471666666666668, learning_rate = 0.00049999997, loss = -1.7707198e-06, step = 12883 (5.124 sec) 2022-04-22 18:12:43,942 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.765 INFO:tensorflow:global_step/sec: 13.7273 2022-04-22 18:12:44,004 [INFO] tensorflow: global_step/sec: 13.7273 2022-04-22 18:12:45,741 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.580 2022-04-22 18:12:47,669 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 51.885 INFO:tensorflow:epoch = 21.58666666666667, learning_rate = 0.00049999997, loss = -1.829897e-06, step = 12952 (5.070 sec) 2022-04-22 18:12:47,888 [INFO] tensorflow: epoch = 21.58666666666667, learning_rate = 0.00049999997, loss = -1.829897e-06, step = 12952 (5.070 sec) INFO:tensorflow:epoch = 21.58666666666667, learning_rate = 0.00049999997, loss = -1.8298424e-06, step = 12952 (5.144 sec) 2022-04-22 18:12:47,888 [INFO] tensorflow: epoch = 21.58666666666667, learning_rate = 0.00049999997, loss = -1.8298424e-06, step = 12952 (5.144 sec) INFO:tensorflow:global_step/sec: 13.4309 2022-04-22 18:12:48,471 [INFO] tensorflow: global_step/sec: 13.4309 2022-04-22 18:12:49,451 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.111 2022-04-22 18:12:51,479 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 49.312 INFO:tensorflow:epoch = 21.695, learning_rate = 0.00049999997, loss = -1.8977166e-06, step = 13017 (5.096 sec) 2022-04-22 18:12:52,984 [INFO] tensorflow: epoch = 21.695, learning_rate = 0.00049999997, loss = -1.8977166e-06, step = 13017 (5.096 sec) INFO:tensorflow:epoch = 21.695, learning_rate = 0.00049999997, loss = -1.8990195e-06, step = 13017 (5.096 sec) 2022-04-22 18:12:52,984 [INFO] tensorflow: epoch = 21.695, learning_rate = 0.00049999997, loss = -1.8990195e-06, step = 13017 (5.096 sec) INFO:tensorflow:global_step/sec: 12.7107 2022-04-22 18:12:53,191 [INFO] tensorflow: global_step/sec: 12.7107 2022-04-22 18:12:53,456 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 50.599 2022-04-22 18:12:55,319 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.671 2022-04-22 18:12:57,096 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.286 INFO:tensorflow:global_step/sec: 13.8218 2022-04-22 18:12:57,532 [INFO] tensorflow: global_step/sec: 13.8218 INFO:tensorflow:epoch = 21.813333333333336, learning_rate = 0.00049999997, loss = -1.9232084e-06, step = 13088 (5.103 sec) 2022-04-22 18:12:58,087 [INFO] tensorflow: epoch = 21.813333333333336, learning_rate = 0.00049999997, loss = -1.9232084e-06, step = 13088 (5.103 sec) INFO:tensorflow:epoch = 21.813333333333336, learning_rate = 0.00049999997, loss = -1.9229587e-06, step = 13088 (5.105 sec) 2022-04-22 18:12:58,089 [INFO] tensorflow: epoch = 21.813333333333336, learning_rate = 0.00049999997, loss = -1.9229587e-06, step = 13088 (5.105 sec) 2022-04-22 18:12:58,917 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.918 2022-04-22 18:13:00,680 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.736 INFO:tensorflow:global_step/sec: 13.8568 2022-04-22 18:13:01,862 [INFO] tensorflow: global_step/sec: 13.8568 2022-04-22 18:13:02,462 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.128 INFO:tensorflow:epoch = 21.93166666666667, learning_rate = 0.00049999997, loss = -1.9642457e-06, step = 13159 (5.103 sec) 2022-04-22 18:13:03,190 [INFO] tensorflow: epoch = 21.93166666666667, learning_rate = 0.00049999997, loss = -1.9642457e-06, step = 13159 (5.103 sec) INFO:tensorflow:epoch = 21.93166666666667, learning_rate = 0.00049999997, loss = -1.9642994e-06, step = 13159 (5.101 sec) 2022-04-22 18:13:03,191 [INFO] tensorflow: epoch = 21.93166666666667, learning_rate = 0.00049999997, loss = -1.9642994e-06, step = 13159 (5.101 sec) 2022-04-22 18:13:04,314 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.976 2022-04-22 18:13:06,138 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.856 INFO:tensorflow:global_step/sec: 13.7761 2022-04-22 18:13:06,218 [INFO] tensorflow: global_step/sec: 13.7761 2022-04-22 18:13:06,218 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 22/100: loss: -0.00000 learning rate: 0.00050 Time taken: 0:00:43.821195 ETA: 0:56:58.053201 2022-04-22 18:13:07,998 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.768 INFO:tensorflow:epoch = 22.046666666666667, learning_rate = 0.00049999997, loss = -2.0230568e-06, step = 13228 (5.169 sec) INFO:tensorflow:epoch = 22.046666666666667, learning_rate = 0.00049999997, loss = -2.0231691e-06, step = 13228 (5.168 sec) 2022-04-22 18:13:08,359 [INFO] tensorflow: epoch = 22.046666666666667, learning_rate = 0.00049999997, loss = -2.0230568e-06, step = 13228 (5.169 sec) 2022-04-22 18:13:08,359 [INFO] tensorflow: epoch = 22.046666666666667, learning_rate = 0.00049999997, loss = -2.0231691e-06, step = 13228 (5.168 sec) 2022-04-22 18:13:09,875 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.299 INFO:tensorflow:global_step/sec: 13.4042 2022-04-22 18:13:10,694 [INFO] tensorflow: global_step/sec: 13.4042 2022-04-22 18:13:11,752 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.270 INFO:tensorflow:epoch = 22.16, learning_rate = 0.00049999997, loss = -2.0786852e-06, step = 13296 (5.068 sec) INFO:tensorflow:epoch = 22.16, learning_rate = 0.00049999997, loss = -2.0783532e-06, step = 13296 (5.068 sec) 2022-04-22 18:13:13,427 [INFO] tensorflow: epoch = 22.16, learning_rate = 0.00049999997, loss = -2.0786852e-06, step = 13296 (5.068 sec) 2022-04-22 18:13:13,427 [INFO] tensorflow: epoch = 22.16, learning_rate = 0.00049999997, loss = -2.0783532e-06, step = 13296 (5.068 sec) 2022-04-22 18:13:13,629 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.272 INFO:tensorflow:global_step/sec: 13.4138 2022-04-22 18:13:15,167 [INFO] tensorflow: global_step/sec: 13.4138 2022-04-22 18:13:15,454 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.812 2022-04-22 18:13:17,324 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.493 INFO:tensorflow:epoch = 22.276666666666667, learning_rate = 0.00049999997, loss = -2.126233e-06, step = 13366 (5.177 sec) 2022-04-22 18:13:18,604 [INFO] tensorflow: epoch = 22.276666666666667, learning_rate = 0.00049999997, loss = -2.126233e-06, step = 13366 (5.177 sec) INFO:tensorflow:epoch = 22.276666666666667, learning_rate = 0.00049999997, loss = -2.1263168e-06, step = 13366 (5.177 sec) 2022-04-22 18:13:18,604 [INFO] tensorflow: epoch = 22.276666666666667, learning_rate = 0.00049999997, loss = -2.1263168e-06, step = 13366 (5.177 sec) 2022-04-22 18:13:19,153 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.680 INFO:tensorflow:global_step/sec: 13.5739 2022-04-22 18:13:19,587 [INFO] tensorflow: global_step/sec: 13.5739 2022-04-22 18:13:20,948 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.710 2022-04-22 18:13:22,716 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.562 INFO:tensorflow:epoch = 22.39666666666667, learning_rate = 0.00049999997, loss = -2.1795418e-06, step = 13438 (5.163 sec) 2022-04-22 18:13:23,767 [INFO] tensorflow: epoch = 22.39666666666667, learning_rate = 0.00049999997, loss = -2.1795418e-06, step = 13438 (5.163 sec) INFO:tensorflow:epoch = 22.39666666666667, learning_rate = 0.00049999997, loss = -2.1794303e-06, step = 13438 (5.180 sec) 2022-04-22 18:13:23,784 [INFO] tensorflow: epoch = 22.39666666666667, learning_rate = 0.00049999997, loss = -2.1794303e-06, step = 13438 (5.180 sec) INFO:tensorflow:global_step/sec: 13.8071 2022-04-22 18:13:23,933 [INFO] tensorflow: global_step/sec: 13.8071 2022-04-22 18:13:24,588 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.439 2022-04-22 18:13:26,513 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 51.952 2022-04-22 18:13:28,292 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.208 INFO:tensorflow:global_step/sec: 13.5359 2022-04-22 18:13:28,365 [INFO] tensorflow: global_step/sec: 13.5359 INFO:tensorflow:epoch = 22.513333333333335, learning_rate = 0.00049999997, loss = -2.2297481e-06, step = 13508 (5.121 sec) 2022-04-22 18:13:28,904 [INFO] tensorflow: epoch = 22.513333333333335, learning_rate = 0.00049999997, loss = -2.2297481e-06, step = 13508 (5.121 sec) INFO:tensorflow:epoch = 22.513333333333335, learning_rate = 0.00049999997, loss = -2.22972e-06, step = 13508 (5.138 sec) 2022-04-22 18:13:28,905 [INFO] tensorflow: epoch = 22.513333333333335, learning_rate = 0.00049999997, loss = -2.22972e-06, step = 13508 (5.138 sec) 2022-04-22 18:13:30,010 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 58.208 2022-04-22 18:13:32,055 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 48.901 INFO:tensorflow:global_step/sec: 13.4794 2022-04-22 18:13:32,816 [INFO] tensorflow: global_step/sec: 13.4794 2022-04-22 18:13:33,858 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.492 INFO:tensorflow:epoch = 22.62666666666667, learning_rate = 0.00049999997, loss = -2.2780214e-06, step = 13576 (5.191 sec) 2022-04-22 18:13:34,097 [INFO] tensorflow: epoch = 22.62666666666667, learning_rate = 0.00049999997, loss = -2.2780214e-06, step = 13576 (5.191 sec) INFO:tensorflow:epoch = 22.62666666666667, learning_rate = 0.00049999997, loss = -2.2779382e-06, step = 13576 (5.205 sec) 2022-04-22 18:13:34,109 [INFO] tensorflow: epoch = 22.62666666666667, learning_rate = 0.00049999997, loss = -2.2779382e-06, step = 13576 (5.205 sec) 2022-04-22 18:13:35,765 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 52.453 INFO:tensorflow:global_step/sec: 13.5987 2022-04-22 18:13:37,229 [INFO] tensorflow: global_step/sec: 13.5987 2022-04-22 18:13:37,519 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 57.012 INFO:tensorflow:epoch = 22.741666666666667, learning_rate = 0.00049999997, loss = -2.3208277e-06, step = 13645 (5.091 sec) INFO:tensorflow:epoch = 22.741666666666667, learning_rate = 0.00049999997, loss = -2.320745e-06, step = 13645 (5.103 sec) 2022-04-22 18:13:39,200 [INFO] tensorflow: epoch = 22.741666666666667, learning_rate = 0.00049999997, loss = -2.3208277e-06, step = 13645 (5.091 sec) 2022-04-22 18:13:39,200 [INFO] tensorflow: epoch = 22.741666666666667, learning_rate = 0.00049999997, loss = -2.320745e-06, step = 13645 (5.103 sec) 2022-04-22 18:13:39,482 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 50.945 2022-04-22 18:13:41,378 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 52.761 INFO:tensorflow:global_step/sec: 13.0686 2022-04-22 18:13:41,820 [INFO] tensorflow: global_step/sec: 13.0686 2022-04-22 18:13:43,213 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.497 INFO:tensorflow:epoch = 22.85666666666667, learning_rate = 0.00049999997, loss = -2.354694e-06, step = 13714 (5.126 sec) 2022-04-22 18:13:44,326 [INFO] tensorflow: epoch = 22.85666666666667, learning_rate = 0.00049999997, loss = -2.354694e-06, step = 13714 (5.126 sec) INFO:tensorflow:epoch = 22.85666666666667, learning_rate = 0.00049999997, loss = -2.354804e-06, step = 13714 (5.127 sec) 2022-04-22 18:13:44,327 [INFO] tensorflow: epoch = 22.85666666666667, learning_rate = 0.00049999997, loss = -2.354804e-06, step = 13714 (5.127 sec) 2022-04-22 18:13:45,049 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.461 INFO:tensorflow:global_step/sec: 13.765 2022-04-22 18:13:46,179 [INFO] tensorflow: global_step/sec: 13.765 2022-04-22 18:13:46,883 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.543 2022-04-22 18:13:48,652 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.535 INFO:tensorflow:epoch = 22.975, learning_rate = 0.00049999997, loss = -2.4114188e-06, step = 13785 (5.087 sec) 2022-04-22 18:13:49,412 [INFO] tensorflow: epoch = 22.975, learning_rate = 0.00049999997, loss = -2.4114188e-06, step = 13785 (5.087 sec) INFO:tensorflow:epoch = 22.975, learning_rate = 0.00049999997, loss = -2.4117244e-06, step = 13785 (5.087 sec) 2022-04-22 18:13:49,414 [INFO] tensorflow: epoch = 22.975, learning_rate = 0.00049999997, loss = -2.4117244e-06, step = 13785 (5.087 sec) 2022-04-22 18:13:50,373 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 58.107 INFO:tensorflow:global_step/sec: 14.0647 2022-04-22 18:13:50,445 [INFO] tensorflow: global_step/sec: 14.0647 2022-04-22 18:13:50,445 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 23/100: loss: -0.00000 learning rate: 0.00050 Time taken: 0:00:44.234838 ETA: 0:56:46.082563 2022-04-22 18:13:52,204 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.647 2022-04-22 18:13:53,922 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 58.211 INFO:tensorflow:epoch = 23.096666666666668, learning_rate = 0.00049999997, loss = -2.4781361e-06, step = 13858 (5.110 sec) 2022-04-22 18:13:54,524 [INFO] tensorflow: epoch = 23.096666666666668, learning_rate = 0.00049999997, loss = -2.4781361e-06, step = 13858 (5.110 sec) INFO:tensorflow:epoch = 23.096666666666668, learning_rate = 0.00049999997, loss = -2.4780247e-06, step = 13858 (5.111 sec) 2022-04-22 18:13:54,524 [INFO] tensorflow: epoch = 23.096666666666668, learning_rate = 0.00049999997, loss = -2.4780247e-06, step = 13858 (5.111 sec) INFO:tensorflow:global_step/sec: 14.2217 2022-04-22 18:13:54,664 [INFO] tensorflow: global_step/sec: 14.2217 2022-04-22 18:13:55,673 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 57.109 2022-04-22 18:13:57,361 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 59.254 INFO:tensorflow:global_step/sec: 13.8093 2022-04-22 18:13:59,009 [INFO] tensorflow: global_step/sec: 13.8093 2022-04-22 18:13:59,302 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 51.513 INFO:tensorflow:epoch = 23.213333333333335, learning_rate = 0.00049999997, loss = -2.498985e-06, step = 13928 (5.114 sec) 2022-04-22 18:13:59,638 [INFO] tensorflow: epoch = 23.213333333333335, learning_rate = 0.00049999997, loss = -2.498985e-06, step = 13928 (5.114 sec) INFO:tensorflow:epoch = 23.213333333333335, learning_rate = 0.00049999997, loss = -2.4989859e-06, step = 13928 (5.116 sec) 2022-04-22 18:13:59,639 [INFO] tensorflow: epoch = 23.213333333333335, learning_rate = 0.00049999997, loss = -2.4989859e-06, step = 13928 (5.116 sec) 2022-04-22 18:14:01,154 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.006 2022-04-22 18:14:03,164 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 49.752 INFO:tensorflow:global_step/sec: 13.0574 2022-04-22 18:14:03,604 [INFO] tensorflow: global_step/sec: 13.0574 INFO:tensorflow:epoch = 23.326666666666668, learning_rate = 0.00049999997, loss = -2.5486243e-06, step = 13996 (5.116 sec) 2022-04-22 18:14:04,754 [INFO] tensorflow: epoch = 23.326666666666668, learning_rate = 0.00049999997, loss = -2.5486243e-06, step = 13996 (5.116 sec) INFO:tensorflow:epoch = 23.326666666666668, learning_rate = 0.00049999997, loss = -2.5486247e-06, step = 13996 (5.115 sec) 2022-04-22 18:14:04,754 [INFO] tensorflow: epoch = 23.326666666666668, learning_rate = 0.00049999997, loss = -2.5486247e-06, step = 13996 (5.115 sec) 2022-04-22 18:14:04,969 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.424 2022-04-22 18:14:06,780 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.225 INFO:tensorflow:global_step/sec: 13.667 2022-04-22 18:14:07,994 [INFO] tensorflow: global_step/sec: 13.667 2022-04-22 18:14:08,735 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 51.144 INFO:tensorflow:epoch = 23.44, learning_rate = 0.00049999997, loss = -2.5897702e-06, step = 14064 (5.103 sec) 2022-04-22 18:14:09,857 [INFO] tensorflow: epoch = 23.44, learning_rate = 0.00049999997, loss = -2.5897702e-06, step = 14064 (5.103 sec) INFO:tensorflow:epoch = 23.44, learning_rate = 0.00049999997, loss = -2.5897698e-06, step = 14064 (5.104 sec) 2022-04-22 18:14:09,858 [INFO] tensorflow: epoch = 23.44, learning_rate = 0.00049999997, loss = -2.5897698e-06, step = 14064 (5.104 sec) 2022-04-22 18:14:10,589 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.945 2022-04-22 18:14:12,350 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.786 INFO:tensorflow:global_step/sec: 13.4861 2022-04-22 18:14:12,443 [INFO] tensorflow: global_step/sec: 13.4861 2022-04-22 18:14:14,287 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 51.651 INFO:tensorflow:epoch = 23.555000000000003, learning_rate = 0.00049999997, loss = -2.6523612e-06, step = 14133 (5.067 sec) 2022-04-22 18:14:14,924 [INFO] tensorflow: epoch = 23.555000000000003, learning_rate = 0.00049999997, loss = -2.6523612e-06, step = 14133 (5.067 sec) INFO:tensorflow:epoch = 23.555000000000003, learning_rate = 0.00049999997, loss = -2.6523612e-06, step = 14133 (5.068 sec) 2022-04-22 18:14:14,925 [INFO] tensorflow: epoch = 23.555000000000003, learning_rate = 0.00049999997, loss = -2.6523612e-06, step = 14133 (5.068 sec) 2022-04-22 18:14:16,111 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.825 INFO:tensorflow:global_step/sec: 13.3653 2022-04-22 18:14:16,932 [INFO] tensorflow: global_step/sec: 13.3653 2022-04-22 18:14:17,926 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.102 2022-04-22 18:14:19,762 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.463 INFO:tensorflow:epoch = 23.671666666666667, learning_rate = 0.00049999997, loss = -2.6958419e-06, step = 14203 (5.126 sec) 2022-04-22 18:14:20,050 [INFO] tensorflow: epoch = 23.671666666666667, learning_rate = 0.00049999997, loss = -2.6958419e-06, step = 14203 (5.126 sec) INFO:tensorflow:epoch = 23.671666666666667, learning_rate = 0.00049999997, loss = -2.6958687e-06, step = 14203 (5.127 sec) 2022-04-22 18:14:20,051 [INFO] tensorflow: epoch = 23.671666666666667, learning_rate = 0.00049999997, loss = -2.6958687e-06, step = 14203 (5.127 sec) INFO:tensorflow:global_step/sec: 13.5934 2022-04-22 18:14:21,346 [INFO] tensorflow: global_step/sec: 13.5934 2022-04-22 18:14:21,625 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.697 2022-04-22 18:14:23,451 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.768 INFO:tensorflow:epoch = 23.790000000000003, learning_rate = 0.00049999997, loss = -2.7106007e-06, step = 14274 (5.097 sec) INFO:tensorflow:epoch = 23.790000000000003, learning_rate = 0.00049999997, loss = -2.7107108e-06, step = 14274 (5.099 sec) 2022-04-22 18:14:25,149 [INFO] tensorflow: epoch = 23.790000000000003, learning_rate = 0.00049999997, loss = -2.7106007e-06, step = 14274 (5.097 sec) 2022-04-22 18:14:25,149 [INFO] tensorflow: epoch = 23.790000000000003, learning_rate = 0.00049999997, loss = -2.7107108e-06, step = 14274 (5.099 sec) 2022-04-22 18:14:25,149 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 58.900 INFO:tensorflow:global_step/sec: 14.02 2022-04-22 18:14:25,626 [INFO] tensorflow: global_step/sec: 14.02 2022-04-22 18:14:27,082 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 51.745 2022-04-22 18:14:28,843 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.771 INFO:tensorflow:global_step/sec: 13.6501 2022-04-22 18:14:30,021 [INFO] tensorflow: global_step/sec: 13.6501 INFO:tensorflow:epoch = 23.905, learning_rate = 0.00049999997, loss = -2.769104e-06, step = 14343 (5.090 sec) 2022-04-22 18:14:30,239 [INFO] tensorflow: epoch = 23.905, learning_rate = 0.00049999997, loss = -2.769104e-06, step = 14343 (5.090 sec) INFO:tensorflow:epoch = 23.905, learning_rate = 0.00049999997, loss = -2.7690762e-06, step = 14343 (5.091 sec) 2022-04-22 18:14:30,240 [INFO] tensorflow: epoch = 23.905, learning_rate = 0.00049999997, loss = -2.7690762e-06, step = 14343 (5.091 sec) 2022-04-22 18:14:30,697 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.950 2022-04-22 18:14:32,608 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 52.351 2022-04-22 18:14:34,385 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.262 INFO:tensorflow:global_step/sec: 13.5245 2022-04-22 18:14:34,458 [INFO] tensorflow: global_step/sec: 13.5245 2022-04-22 18:14:34,458 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 24/100: loss: -0.00000 learning rate: 0.00050 Time taken: 0:00:44.011484 ETA: 0:55:44.872813 INFO:tensorflow:epoch = 24.01666666666667, learning_rate = 0.00049999997, loss = -2.7839897e-06, step = 14410 (5.106 sec) 2022-04-22 18:14:35,345 [INFO] tensorflow: epoch = 24.01666666666667, learning_rate = 0.00049999997, loss = -2.7839897e-06, step = 14410 (5.106 sec) INFO:tensorflow:epoch = 24.01666666666667, learning_rate = 0.00049999997, loss = -2.7839897e-06, step = 14410 (5.105 sec) 2022-04-22 18:14:35,345 [INFO] tensorflow: epoch = 24.01666666666667, learning_rate = 0.00049999997, loss = -2.7839897e-06, step = 14410 (5.105 sec) 2022-04-22 18:14:36,546 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 46.285 2022-04-22 18:14:38,430 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.105 INFO:tensorflow:global_step/sec: 12.6104 2022-04-22 18:14:39,216 [INFO] tensorflow: global_step/sec: 12.6104 2022-04-22 18:14:40,194 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.681 INFO:tensorflow:epoch = 24.130000000000003, learning_rate = 0.00049999997, loss = -2.8542609e-06, step = 14478 (5.137 sec) 2022-04-22 18:14:40,482 [INFO] tensorflow: epoch = 24.130000000000003, learning_rate = 0.00049999997, loss = -2.8542609e-06, step = 14478 (5.137 sec) INFO:tensorflow:epoch = 24.130000000000003, learning_rate = 0.00049999997, loss = -2.8542331e-06, step = 14478 (5.137 sec) 2022-04-22 18:14:40,482 [INFO] tensorflow: epoch = 24.130000000000003, learning_rate = 0.00049999997, loss = -2.8542331e-06, step = 14478 (5.137 sec) 2022-04-22 18:14:41,961 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.608 INFO:tensorflow:global_step/sec: 13.7886 2022-04-22 18:14:43,567 [INFO] tensorflow: global_step/sec: 13.7886 2022-04-22 18:14:43,856 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 52.779 INFO:tensorflow:epoch = 24.245, learning_rate = 0.00049999997, loss = -2.9029216e-06, step = 14547 (5.102 sec) 2022-04-22 18:14:45,583 [INFO] tensorflow: epoch = 24.245, learning_rate = 0.00049999997, loss = -2.9029216e-06, step = 14547 (5.102 sec) INFO:tensorflow:epoch = 24.245, learning_rate = 0.00049999997, loss = -2.9030043e-06, step = 14547 (5.103 sec) 2022-04-22 18:14:45,585 [INFO] tensorflow: epoch = 24.245, learning_rate = 0.00049999997, loss = -2.9030043e-06, step = 14547 (5.103 sec) 2022-04-22 18:14:45,749 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 52.810 2022-04-22 18:14:47,590 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.331 INFO:tensorflow:global_step/sec: 13.4455 2022-04-22 18:14:48,029 [INFO] tensorflow: global_step/sec: 13.4455 2022-04-22 18:14:49,422 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.599 INFO:tensorflow:epoch = 24.358333333333334, learning_rate = 0.00049999997, loss = -2.894923e-06, step = 14615 (5.110 sec) 2022-04-22 18:14:50,693 [INFO] tensorflow: epoch = 24.358333333333334, learning_rate = 0.00049999997, loss = -2.894923e-06, step = 14615 (5.110 sec) INFO:tensorflow:epoch = 24.358333333333334, learning_rate = 0.00049999997, loss = -2.8949498e-06, step = 14615 (5.118 sec) 2022-04-22 18:14:50,702 [INFO] tensorflow: epoch = 24.358333333333334, learning_rate = 0.00049999997, loss = -2.8949498e-06, step = 14615 (5.118 sec) 2022-04-22 18:14:51,316 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 52.787 INFO:tensorflow:global_step/sec: 13.3242 2022-04-22 18:14:52,533 [INFO] tensorflow: global_step/sec: 13.3242 2022-04-22 18:14:53,166 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.071 2022-04-22 18:14:54,943 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.292 INFO:tensorflow:epoch = 24.476666666666667, learning_rate = 0.00049999997, loss = -2.9378907e-06, step = 14686 (5.121 sec) 2022-04-22 18:14:55,814 [INFO] tensorflow: epoch = 24.476666666666667, learning_rate = 0.00049999997, loss = -2.9378907e-06, step = 14686 (5.121 sec) INFO:tensorflow:epoch = 24.476666666666667, learning_rate = 0.00049999997, loss = -2.937918e-06, step = 14686 (5.112 sec) 2022-04-22 18:14:55,814 [INFO] tensorflow: epoch = 24.476666666666667, learning_rate = 0.00049999997, loss = -2.937918e-06, step = 14686 (5.112 sec) 2022-04-22 18:14:56,705 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.780 INFO:tensorflow:global_step/sec: 14.1441 2022-04-22 18:14:56,775 [INFO] tensorflow: global_step/sec: 14.1441 2022-04-22 18:14:58,559 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.946 2022-04-22 18:15:00,420 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.738 INFO:tensorflow:epoch = 24.59166666666667, learning_rate = 0.00049999997, loss = -2.9714652e-06, step = 14755 (5.136 sec) 2022-04-22 18:15:00,950 [INFO] tensorflow: epoch = 24.59166666666667, learning_rate = 0.00049999997, loss = -2.9714652e-06, step = 14755 (5.136 sec) INFO:tensorflow:epoch = 24.59166666666667, learning_rate = 0.00049999997, loss = -2.9715202e-06, step = 14755 (5.140 sec) 2022-04-22 18:15:00,953 [INFO] tensorflow: epoch = 24.59166666666667, learning_rate = 0.00049999997, loss = -2.9715202e-06, step = 14755 (5.140 sec) INFO:tensorflow:global_step/sec: 13.2097 2022-04-22 18:15:01,317 [INFO] tensorflow: global_step/sec: 13.2097 2022-04-22 18:15:02,332 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 52.306 2022-04-22 18:15:04,116 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.072 INFO:tensorflow:global_step/sec: 13.876 2022-04-22 18:15:05,641 [INFO] tensorflow: global_step/sec: 13.876 2022-04-22 18:15:05,989 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.404 INFO:tensorflow:epoch = 24.708333333333336, learning_rate = 0.00049999997, loss = -3.0054339e-06, step = 14825 (5.118 sec) 2022-04-22 18:15:06,068 [INFO] tensorflow: epoch = 24.708333333333336, learning_rate = 0.00049999997, loss = -3.0054339e-06, step = 14825 (5.118 sec) INFO:tensorflow:epoch = 24.708333333333336, learning_rate = 0.00049999997, loss = -3.0054339e-06, step = 14825 (5.115 sec) 2022-04-22 18:15:06,068 [INFO] tensorflow: epoch = 24.708333333333336, learning_rate = 0.00049999997, loss = -3.0054339e-06, step = 14825 (5.115 sec) 2022-04-22 18:15:07,757 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.549 2022-04-22 18:15:09,544 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.991 INFO:tensorflow:global_step/sec: 13.8415 2022-04-22 18:15:09,975 [INFO] tensorflow: global_step/sec: 13.8415 INFO:tensorflow:epoch = 24.826666666666668, learning_rate = 0.00049999997, loss = -3.03575e-06, step = 14896 (5.108 sec) 2022-04-22 18:15:11,176 [INFO] tensorflow: epoch = 24.826666666666668, learning_rate = 0.00049999997, loss = -3.03575e-06, step = 14896 (5.108 sec) INFO:tensorflow:epoch = 24.826666666666668, learning_rate = 0.00049999997, loss = -3.035777e-06, step = 14896 (5.108 sec) 2022-04-22 18:15:11,176 [INFO] tensorflow: epoch = 24.826666666666668, learning_rate = 0.00049999997, loss = -3.035777e-06, step = 14896 (5.108 sec) 2022-04-22 18:15:11,400 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.865 2022-04-22 18:15:13,414 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 49.677 INFO:tensorflow:global_step/sec: 13.1633 2022-04-22 18:15:14,534 [INFO] tensorflow: global_step/sec: 13.1633 2022-04-22 18:15:15,188 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.375 INFO:tensorflow:epoch = 24.94, learning_rate = 0.00049999997, loss = -3.0880078e-06, step = 14964 (5.104 sec) 2022-04-22 18:15:16,280 [INFO] tensorflow: epoch = 24.94, learning_rate = 0.00049999997, loss = -3.0880078e-06, step = 14964 (5.104 sec) INFO:tensorflow:epoch = 24.94, learning_rate = 0.00049999997, loss = -3.0880637e-06, step = 14964 (5.104 sec) 2022-04-22 18:15:16,280 [INFO] tensorflow: epoch = 24.94, learning_rate = 0.00049999997, loss = -3.0880637e-06, step = 14964 (5.104 sec) 2022-04-22 18:15:17,008 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.942 2022-04-22 18:15:18,787 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.203 INFO:tensorflow:global_step/sec: 13.7714 2022-04-22 18:15:18,891 [INFO] tensorflow: global_step/sec: 13.7714 2022-04-22 18:15:18,892 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 25/100: loss: -0.00000 learning rate: 0.00050 Time taken: 0:00:44.401749 ETA: 0:55:30.131149 2022-04-22 18:15:20,736 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 51.364 INFO:tensorflow:epoch = 25.055000000000003, learning_rate = 0.00049999997, loss = -3.1393429e-06, step = 15033 (5.104 sec) 2022-04-22 18:15:21,384 [INFO] tensorflow: epoch = 25.055000000000003, learning_rate = 0.00049999997, loss = -3.1393429e-06, step = 15033 (5.104 sec) INFO:tensorflow:epoch = 25.055000000000003, learning_rate = 0.00049999997, loss = -3.1393433e-06, step = 15033 (5.104 sec) 2022-04-22 18:15:21,384 [INFO] tensorflow: epoch = 25.055000000000003, learning_rate = 0.00049999997, loss = -3.1393433e-06, step = 15033 (5.104 sec) 2022-04-22 18:15:22,547 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.240 INFO:tensorflow:global_step/sec: 13.4739 2022-04-22 18:15:23,344 [INFO] tensorflow: global_step/sec: 13.4739 2022-04-22 18:15:24,328 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.166 2022-04-22 18:15:26,124 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.697 INFO:tensorflow:epoch = 25.173333333333336, learning_rate = 0.00049999997, loss = -3.140608e-06, step = 15104 (5.075 sec) 2022-04-22 18:15:26,460 [INFO] tensorflow: epoch = 25.173333333333336, learning_rate = 0.00049999997, loss = -3.140608e-06, step = 15104 (5.075 sec) INFO:tensorflow:epoch = 25.173333333333336, learning_rate = 0.00049999997, loss = -3.140608e-06, step = 15104 (5.078 sec) 2022-04-22 18:15:26,462 [INFO] tensorflow: epoch = 25.173333333333336, learning_rate = 0.00049999997, loss = -3.140608e-06, step = 15104 (5.078 sec) INFO:tensorflow:global_step/sec: 13.9705 2022-04-22 18:15:27,638 [INFO] tensorflow: global_step/sec: 13.9705 2022-04-22 18:15:27,923 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.581 2022-04-22 18:15:29,970 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 48.870 INFO:tensorflow:epoch = 25.285, learning_rate = 0.00049999997, loss = -3.1940626e-06, step = 15171 (5.083 sec) 2022-04-22 18:15:31,545 [INFO] tensorflow: epoch = 25.285, learning_rate = 0.00049999997, loss = -3.1940626e-06, step = 15171 (5.083 sec) INFO:tensorflow:epoch = 25.285, learning_rate = 0.00049999997, loss = -3.1940626e-06, step = 15171 (5.087 sec) 2022-04-22 18:15:31,546 [INFO] tensorflow: epoch = 25.285, learning_rate = 0.00049999997, loss = -3.1940626e-06, step = 15171 (5.087 sec) 2022-04-22 18:15:31,751 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.140 INFO:tensorflow:global_step/sec: 13.1163 2022-04-22 18:15:32,213 [INFO] tensorflow: global_step/sec: 13.1163 2022-04-22 18:15:33,882 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 46.942 2022-04-22 18:15:35,753 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.437 INFO:tensorflow:epoch = 25.393333333333334, learning_rate = 0.00049999997, loss = -3.1662075e-06, step = 15236 (5.094 sec) 2022-04-22 18:15:36,640 [INFO] tensorflow: epoch = 25.393333333333334, learning_rate = 0.00049999997, loss = -3.1662075e-06, step = 15236 (5.094 sec) INFO:tensorflow:epoch = 25.393333333333334, learning_rate = 0.00049999997, loss = -3.1662894e-06, step = 15236 (5.096 sec) 2022-04-22 18:15:36,641 [INFO] tensorflow: epoch = 25.393333333333334, learning_rate = 0.00049999997, loss = -3.1662894e-06, step = 15236 (5.096 sec) INFO:tensorflow:global_step/sec: 12.6614 2022-04-22 18:15:36,952 [INFO] tensorflow: global_step/sec: 12.6614 2022-04-22 18:15:37,621 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.547 2022-04-22 18:15:39,576 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 51.154 2022-04-22 18:15:41,455 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.263 INFO:tensorflow:global_step/sec: 13.1047 2022-04-22 18:15:41,530 [INFO] tensorflow: global_step/sec: 13.1047 INFO:tensorflow:epoch = 25.505000000000003, learning_rate = 0.00049999997, loss = -3.2435796e-06, step = 15303 (5.108 sec) INFO:tensorflow:epoch = 25.505000000000003, learning_rate = 0.00049999997, loss = -3.2434127e-06, step = 15303 (5.106 sec) 2022-04-22 18:15:41,748 [INFO] tensorflow: epoch = 25.505000000000003, learning_rate = 0.00049999997, loss = -3.2435796e-06, step = 15303 (5.108 sec) 2022-04-22 18:15:41,748 [INFO] tensorflow: epoch = 25.505000000000003, learning_rate = 0.00049999997, loss = -3.2434127e-06, step = 15303 (5.106 sec) 2022-04-22 18:15:43,298 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.247 2022-04-22 18:15:45,133 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.501 INFO:tensorflow:global_step/sec: 13.6767 2022-04-22 18:15:45,917 [INFO] tensorflow: global_step/sec: 13.6767 INFO:tensorflow:epoch = 25.62, learning_rate = 0.00049999997, loss = -3.2792486e-06, step = 15372 (5.081 sec) 2022-04-22 18:15:46,828 [INFO] tensorflow: epoch = 25.62, learning_rate = 0.00049999997, loss = -3.2792486e-06, step = 15372 (5.081 sec) INFO:tensorflow:epoch = 25.62166666666667, learning_rate = 0.00049999997, loss = -3.2783041e-06, step = 15373 (5.216 sec) 2022-04-22 18:15:46,964 [INFO] tensorflow: epoch = 25.62166666666667, learning_rate = 0.00049999997, loss = -3.2783041e-06, step = 15373 (5.216 sec) 2022-04-22 18:15:47,058 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 51.953 2022-04-22 18:15:48,897 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.403 INFO:tensorflow:global_step/sec: 13.3498 2022-04-22 18:15:50,412 [INFO] tensorflow: global_step/sec: 13.3498 2022-04-22 18:15:50,704 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.330 INFO:tensorflow:epoch = 25.735000000000003, learning_rate = 0.00049999997, loss = -3.2992166e-06, step = 15441 (5.127 sec) 2022-04-22 18:15:51,955 [INFO] tensorflow: epoch = 25.735000000000003, learning_rate = 0.00049999997, loss = -3.2992166e-06, step = 15441 (5.127 sec) INFO:tensorflow:epoch = 25.738333333333333, learning_rate = 0.00049999997, loss = -3.309281e-06, step = 15443 (5.147 sec) 2022-04-22 18:15:52,111 [INFO] tensorflow: epoch = 25.738333333333333, learning_rate = 0.00049999997, loss = -3.309281e-06, step = 15443 (5.147 sec) 2022-04-22 18:15:52,557 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.969 2022-04-22 18:15:54,557 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 50.012 INFO:tensorflow:global_step/sec: 13.0945 2022-04-22 18:15:54,994 [INFO] tensorflow: global_step/sec: 13.0945 2022-04-22 18:15:56,394 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.449 INFO:tensorflow:epoch = 25.845000000000002, learning_rate = 0.00049999997, loss = -3.363145e-06, step = 15507 (5.067 sec) 2022-04-22 18:15:57,022 [INFO] tensorflow: epoch = 25.845000000000002, learning_rate = 0.00049999997, loss = -3.363145e-06, step = 15507 (5.067 sec) INFO:tensorflow:epoch = 25.85, learning_rate = 0.00049999997, loss = -3.3100928e-06, step = 15510 (5.135 sec) 2022-04-22 18:15:57,246 [INFO] tensorflow: epoch = 25.85, learning_rate = 0.00049999997, loss = -3.3100928e-06, step = 15510 (5.135 sec) 2022-04-22 18:15:58,245 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.045 INFO:tensorflow:global_step/sec: 13.3982 2022-04-22 18:15:59,472 [INFO] tensorflow: global_step/sec: 13.3982 2022-04-22 18:16:00,113 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.528 2022-04-22 18:16:02,012 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 52.672 INFO:tensorflow:epoch = 25.96, learning_rate = 0.00049999997, loss = -3.3389242e-06, step = 15576 (5.139 sec) 2022-04-22 18:16:02,161 [INFO] tensorflow: epoch = 25.96, learning_rate = 0.00049999997, loss = -3.3389242e-06, step = 15576 (5.139 sec) INFO:tensorflow:epoch = 25.965000000000003, learning_rate = 0.00049999997, loss = -3.3388642e-06, step = 15579 (5.131 sec) 2022-04-22 18:16:02,377 [INFO] tensorflow: epoch = 25.965000000000003, learning_rate = 0.00049999997, loss = -3.3388642e-06, step = 15579 (5.131 sec) 2022-04-22 18:16:03,894 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 299, 0.00s/step 2022-04-22 18:16:04,118 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 299, 0.02s/step 2022-04-22 18:16:04,340 [INFO] iva.detectnet_v2.evaluation.evaluation: step 20 / 299, 0.02s/step 2022-04-22 18:16:04,563 [INFO] iva.detectnet_v2.evaluation.evaluation: step 30 / 299, 0.02s/step 2022-04-22 18:16:04,785 [INFO] iva.detectnet_v2.evaluation.evaluation: step 40 / 299, 0.02s/step 2022-04-22 18:16:05,009 [INFO] iva.detectnet_v2.evaluation.evaluation: step 50 / 299, 0.02s/step 2022-04-22 18:16:05,240 [INFO] iva.detectnet_v2.evaluation.evaluation: step 60 / 299, 0.02s/step 2022-04-22 18:16:05,470 [INFO] iva.detectnet_v2.evaluation.evaluation: step 70 / 299, 0.02s/step 2022-04-22 18:16:05,700 [INFO] iva.detectnet_v2.evaluation.evaluation: step 80 / 299, 0.02s/step 2022-04-22 18:16:05,932 [INFO] iva.detectnet_v2.evaluation.evaluation: step 90 / 299, 0.02s/step 2022-04-22 18:16:06,162 [INFO] iva.detectnet_v2.evaluation.evaluation: step 100 / 299, 0.02s/step 2022-04-22 18:16:06,393 [INFO] iva.detectnet_v2.evaluation.evaluation: step 110 / 299, 0.02s/step 2022-04-22 18:16:06,626 [INFO] iva.detectnet_v2.evaluation.evaluation: step 120 / 299, 0.02s/step 2022-04-22 18:16:06,863 [INFO] iva.detectnet_v2.evaluation.evaluation: step 130 / 299, 0.02s/step 2022-04-22 18:16:07,097 [INFO] iva.detectnet_v2.evaluation.evaluation: step 140 / 299, 0.02s/step 2022-04-22 18:16:07,336 [INFO] iva.detectnet_v2.evaluation.evaluation: step 150 / 299, 0.02s/step 2022-04-22 18:16:07,570 [INFO] iva.detectnet_v2.evaluation.evaluation: step 160 / 299, 0.02s/step 2022-04-22 18:16:07,808 [INFO] iva.detectnet_v2.evaluation.evaluation: step 170 / 299, 0.02s/step 2022-04-22 18:16:08,047 [INFO] iva.detectnet_v2.evaluation.evaluation: step 180 / 299, 0.02s/step 2022-04-22 18:16:08,281 [INFO] iva.detectnet_v2.evaluation.evaluation: step 190 / 299, 0.02s/step 2022-04-22 18:16:08,517 [INFO] iva.detectnet_v2.evaluation.evaluation: step 200 / 299, 0.02s/step 2022-04-22 18:16:08,753 [INFO] iva.detectnet_v2.evaluation.evaluation: step 210 / 299, 0.02s/step 2022-04-22 18:16:08,988 [INFO] iva.detectnet_v2.evaluation.evaluation: step 220 / 299, 0.02s/step 2022-04-22 18:16:09,222 [INFO] iva.detectnet_v2.evaluation.evaluation: step 230 / 299, 0.02s/step 2022-04-22 18:16:09,457 [INFO] iva.detectnet_v2.evaluation.evaluation: step 240 / 299, 0.02s/step 2022-04-22 18:16:09,692 [INFO] iva.detectnet_v2.evaluation.evaluation: step 250 / 299, 0.02s/step 2022-04-22 18:16:09,929 [INFO] iva.detectnet_v2.evaluation.evaluation: step 260 / 299, 0.02s/step 2022-04-22 18:16:10,166 [INFO] iva.detectnet_v2.evaluation.evaluation: step 270 / 299, 0.02s/step 2022-04-22 18:16:10,404 [INFO] iva.detectnet_v2.evaluation.evaluation: step 280 / 299, 0.02s/step 2022-04-22 18:16:10,642 [INFO] iva.detectnet_v2.evaluation.evaluation: step 290 / 299, 0.02s/step Epoch 26/100 ========================= Validation cost: -0.000009 Mean average_precision (in %): 0.0000 class name average precision (in %) ------------ -------------------------- face 0 Median Inference Time: 0.011537 2022-04-22 18:16:10,888 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 11.266 INFO:tensorflow:epoch = 26.0, learning_rate = 0.00049999997, loss = -3.3351525e-06, step = 15600 (8.797 sec) 2022-04-22 18:16:10,958 [INFO] tensorflow: epoch = 26.0, learning_rate = 0.00049999997, loss = -3.3351525e-06, step = 15600 (8.797 sec) INFO:tensorflow:global_step/sec: 5.22331 2022-04-22 18:16:10,959 [INFO] tensorflow: global_step/sec: 5.22331 2022-04-22 18:16:10,959 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 26/100: loss: -0.00000 learning rate: 0.00050 Time taken: 0:00:52.098588 ETA: 1:04:15.295494 INFO:tensorflow:epoch = 26.00166666666667, learning_rate = 0.00049999997, loss = -3.3474553e-06, step = 15601 (8.657 sec) 2022-04-22 18:16:11,034 [INFO] tensorflow: epoch = 26.00166666666667, learning_rate = 0.00049999997, loss = -3.3474553e-06, step = 15601 (8.657 sec) 2022-04-22 18:16:12,658 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.507 2022-04-22 18:16:14,472 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.145 INFO:tensorflow:global_step/sec: 14.0798 2022-04-22 18:16:15,220 [INFO] tensorflow: global_step/sec: 14.0798 INFO:tensorflow:epoch = 26.12, learning_rate = 0.00049999997, loss = -3.4084478e-06, step = 15672 (5.149 sec) 2022-04-22 18:16:16,107 [INFO] tensorflow: epoch = 26.12, learning_rate = 0.00049999997, loss = -3.4084478e-06, step = 15672 (5.149 sec) INFO:tensorflow:epoch = 26.12166666666667, learning_rate = 0.00049999997, loss = -3.3923907e-06, step = 15673 (5.149 sec) 2022-04-22 18:16:16,183 [INFO] tensorflow: epoch = 26.12166666666667, learning_rate = 0.00049999997, loss = -3.3923907e-06, step = 15673 (5.149 sec) 2022-04-22 18:16:16,261 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.893 2022-04-22 18:16:18,006 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 57.313 INFO:tensorflow:global_step/sec: 14.1278 2022-04-22 18:16:19,467 [INFO] tensorflow: global_step/sec: 14.1278 2022-04-22 18:16:19,752 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 57.294 INFO:tensorflow:epoch = 26.240000000000002, learning_rate = 0.00049999997, loss = -3.3999959e-06, step = 15744 (5.154 sec) 2022-04-22 18:16:21,261 [INFO] tensorflow: epoch = 26.240000000000002, learning_rate = 0.00049999997, loss = -3.3999959e-06, step = 15744 (5.154 sec) INFO:tensorflow:epoch = 26.241666666666667, learning_rate = 0.00049999997, loss = -3.4048908e-06, step = 15745 (5.157 sec) 2022-04-22 18:16:21,339 [INFO] tensorflow: epoch = 26.241666666666667, learning_rate = 0.00049999997, loss = -3.4048908e-06, step = 15745 (5.157 sec) 2022-04-22 18:16:21,719 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 50.853 2022-04-22 18:16:23,544 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.793 INFO:tensorflow:global_step/sec: 13.335 2022-04-22 18:16:23,967 [INFO] tensorflow: global_step/sec: 13.335 2022-04-22 18:16:25,321 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.298 INFO:tensorflow:epoch = 26.355, learning_rate = 0.00049999997, loss = -3.4563554e-06, step = 15813 (5.089 sec) 2022-04-22 18:16:26,350 [INFO] tensorflow: epoch = 26.355, learning_rate = 0.00049999997, loss = -3.4563554e-06, step = 15813 (5.089 sec) INFO:tensorflow:epoch = 26.35666666666667, learning_rate = 0.00049999997, loss = -3.4288805e-06, step = 15814 (5.081 sec) 2022-04-22 18:16:26,420 [INFO] tensorflow: epoch = 26.35666666666667, learning_rate = 0.00049999997, loss = -3.4288805e-06, step = 15814 (5.081 sec) 2022-04-22 18:16:27,153 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.600 INFO:tensorflow:global_step/sec: 13.6781 2022-04-22 18:16:28,353 [INFO] tensorflow: global_step/sec: 13.6781 2022-04-22 18:16:29,041 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 52.958 2022-04-22 18:16:31,031 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 50.250 INFO:tensorflow:epoch = 26.46666666666667, learning_rate = 0.00049999997, loss = -3.450823e-06, step = 15880 (5.137 sec) 2022-04-22 18:16:31,486 [INFO] tensorflow: epoch = 26.46666666666667, learning_rate = 0.00049999997, loss = -3.450823e-06, step = 15880 (5.137 sec) INFO:tensorflow:epoch = 26.468333333333334, learning_rate = 0.00049999997, loss = -3.4847412e-06, step = 15881 (5.142 sec) 2022-04-22 18:16:31,561 [INFO] tensorflow: epoch = 26.468333333333334, learning_rate = 0.00049999997, loss = -3.4847412e-06, step = 15881 (5.142 sec) 2022-04-22 18:16:32,996 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 50.907 INFO:tensorflow:global_step/sec: 12.5762 2022-04-22 18:16:33,124 [INFO] tensorflow: global_step/sec: 12.5762 2022-04-22 18:16:34,943 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 51.397 INFO:tensorflow:epoch = 26.575000000000003, learning_rate = 0.00049999997, loss = -3.4866625e-06, step = 15945 (5.126 sec) 2022-04-22 18:16:36,612 [INFO] tensorflow: epoch = 26.575000000000003, learning_rate = 0.00049999997, loss = -3.4866625e-06, step = 15945 (5.126 sec) INFO:tensorflow:epoch = 26.576666666666668, learning_rate = 0.00049999997, loss = -3.4867217e-06, step = 15946 (5.138 sec) 2022-04-22 18:16:36,700 [INFO] tensorflow: epoch = 26.576666666666668, learning_rate = 0.00049999997, loss = -3.4867217e-06, step = 15946 (5.138 sec) 2022-04-22 18:16:36,941 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 50.040 INFO:tensorflow:global_step/sec: 12.8875 2022-04-22 18:16:37,780 [INFO] tensorflow: global_step/sec: 12.8875 2022-04-22 18:16:39,094 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 46.453 2022-04-22 18:16:40,930 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.471 INFO:tensorflow:epoch = 26.683333333333334, learning_rate = 0.00049999997, loss = -3.4946443e-06, step = 16010 (5.126 sec) 2022-04-22 18:16:41,738 [INFO] tensorflow: epoch = 26.683333333333334, learning_rate = 0.00049999997, loss = -3.4946443e-06, step = 16010 (5.126 sec) INFO:tensorflow:epoch = 26.685000000000002, learning_rate = 0.00049999997, loss = -3.5265475e-06, step = 16011 (5.111 sec) 2022-04-22 18:16:41,811 [INFO] tensorflow: epoch = 26.685000000000002, learning_rate = 0.00049999997, loss = -3.5265475e-06, step = 16011 (5.111 sec) INFO:tensorflow:global_step/sec: 12.8159 2022-04-22 18:16:42,462 [INFO] tensorflow: global_step/sec: 12.8159 2022-04-22 18:16:42,767 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.455 2022-04-22 18:16:44,520 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 57.044 2022-04-22 18:16:46,439 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 52.118 INFO:tensorflow:epoch = 26.8, learning_rate = 0.00049999997, loss = -3.5306166e-06, step = 16080 (5.134 sec) 2022-04-22 18:16:46,872 [INFO] tensorflow: epoch = 26.8, learning_rate = 0.00049999997, loss = -3.5306166e-06, step = 16080 (5.134 sec) INFO:tensorflow:global_step/sec: 13.6009 2022-04-22 18:16:46,873 [INFO] tensorflow: global_step/sec: 13.6009 INFO:tensorflow:epoch = 26.80166666666667, learning_rate = 0.00049999997, loss = -3.539818e-06, step = 16081 (5.136 sec) 2022-04-22 18:16:46,947 [INFO] tensorflow: epoch = 26.80166666666667, learning_rate = 0.00049999997, loss = -3.539818e-06, step = 16081 (5.136 sec) 2022-04-22 18:16:48,322 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.107 2022-04-22 18:16:50,224 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 52.582 INFO:tensorflow:global_step/sec: 13.0702 2022-04-22 18:16:51,464 [INFO] tensorflow: global_step/sec: 13.0702 INFO:tensorflow:epoch = 26.91166666666667, learning_rate = 0.00049999997, loss = -3.6083961e-06, step = 16147 (5.080 sec) 2022-04-22 18:16:51,952 [INFO] tensorflow: epoch = 26.91166666666667, learning_rate = 0.00049999997, loss = -3.6083961e-06, step = 16147 (5.080 sec) INFO:tensorflow:epoch = 26.913333333333334, learning_rate = 0.00049999997, loss = -3.576829e-06, step = 16148 (5.076 sec) 2022-04-22 18:16:52,023 [INFO] tensorflow: epoch = 26.913333333333334, learning_rate = 0.00049999997, loss = -3.576829e-06, step = 16148 (5.076 sec) 2022-04-22 18:16:52,091 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.575 2022-04-22 18:16:53,954 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.676 2022-04-22 18:16:55,785 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.625 INFO:tensorflow:global_step/sec: 13.6773 2022-04-22 18:16:55,850 [INFO] tensorflow: global_step/sec: 13.6773 2022-04-22 18:16:55,851 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 27/100: loss: -0.00000 learning rate: 0.00050 Time taken: 0:00:44.896282 ETA: 0:54:37.428566 INFO:tensorflow:epoch = 27.028333333333336, learning_rate = 0.00049999997, loss = -3.6149263e-06, step = 16217 (5.107 sec) 2022-04-22 18:16:57,059 [INFO] tensorflow: epoch = 27.028333333333336, learning_rate = 0.00049999997, loss = -3.6149263e-06, step = 16217 (5.107 sec) INFO:tensorflow:epoch = 27.03, learning_rate = 0.00049999997, loss = -3.5979283e-06, step = 16218 (5.116 sec) 2022-04-22 18:16:57,139 [INFO] tensorflow: epoch = 27.03, learning_rate = 0.00049999997, loss = -3.5979283e-06, step = 16218 (5.116 sec) 2022-04-22 18:16:57,547 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.783 2022-04-22 18:16:59,379 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.569 INFO:tensorflow:global_step/sec: 13.7411 2022-04-22 18:17:00,217 [INFO] tensorflow: global_step/sec: 13.7411 2022-04-22 18:17:01,272 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 52.842 INFO:tensorflow:epoch = 27.143333333333334, learning_rate = 0.00049999997, loss = -3.6025667e-06, step = 16286 (5.117 sec) 2022-04-22 18:17:02,176 [INFO] tensorflow: epoch = 27.143333333333334, learning_rate = 0.00049999997, loss = -3.6025667e-06, step = 16286 (5.117 sec) INFO:tensorflow:epoch = 27.145000000000003, learning_rate = 0.00049999997, loss = -3.6498186e-06, step = 16287 (5.133 sec) 2022-04-22 18:17:02,271 [INFO] tensorflow: epoch = 27.145000000000003, learning_rate = 0.00049999997, loss = -3.6498186e-06, step = 16287 (5.133 sec) 2022-04-22 18:17:03,145 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.392 INFO:tensorflow:global_step/sec: 13.1241 2022-04-22 18:17:04,789 [INFO] tensorflow: global_step/sec: 13.1241 2022-04-22 18:17:05,107 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 50.986 2022-04-22 18:17:06,906 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.571 INFO:tensorflow:epoch = 27.256666666666668, learning_rate = 0.00049999997, loss = -3.623244e-06, step = 16354 (5.137 sec) 2022-04-22 18:17:07,314 [INFO] tensorflow: epoch = 27.256666666666668, learning_rate = 0.00049999997, loss = -3.623244e-06, step = 16354 (5.137 sec) INFO:tensorflow:epoch = 27.258333333333336, learning_rate = 0.00049999997, loss = -3.6387069e-06, step = 16355 (5.108 sec) 2022-04-22 18:17:07,379 [INFO] tensorflow: epoch = 27.258333333333336, learning_rate = 0.00049999997, loss = -3.6387069e-06, step = 16355 (5.108 sec) 2022-04-22 18:17:08,741 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.534 INFO:tensorflow:global_step/sec: 13.7018 2022-04-22 18:17:09,168 [INFO] tensorflow: global_step/sec: 13.7018 2022-04-22 18:17:10,524 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.068 2022-04-22 18:17:12,339 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.122 INFO:tensorflow:epoch = 27.375, learning_rate = 0.00049999997, loss = -3.662914e-06, step = 16425 (5.098 sec) 2022-04-22 18:17:12,411 [INFO] tensorflow: epoch = 27.375, learning_rate = 0.00049999997, loss = -3.662914e-06, step = 16425 (5.098 sec) INFO:tensorflow:epoch = 27.37666666666667, learning_rate = 0.00049999997, loss = -3.6549081e-06, step = 16426 (5.104 sec) 2022-04-22 18:17:12,483 [INFO] tensorflow: epoch = 27.37666666666667, learning_rate = 0.00049999997, loss = -3.6549081e-06, step = 16426 (5.104 sec) INFO:tensorflow:global_step/sec: 13.9929 2022-04-22 18:17:13,456 [INFO] tensorflow: global_step/sec: 13.9929 2022-04-22 18:17:14,102 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.723 2022-04-22 18:17:16,069 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 50.854 INFO:tensorflow:epoch = 27.491666666666667, learning_rate = 0.00049999997, loss = -3.666075e-06, step = 16495 (5.122 sec) 2022-04-22 18:17:17,534 [INFO] tensorflow: epoch = 27.491666666666667, learning_rate = 0.00049999997, loss = -3.666075e-06, step = 16495 (5.122 sec) INFO:tensorflow:epoch = 27.493333333333336, learning_rate = 0.00049999997, loss = -3.6590523e-06, step = 16496 (5.128 sec) 2022-04-22 18:17:17,611 [INFO] tensorflow: epoch = 27.493333333333336, learning_rate = 0.00049999997, loss = -3.6590523e-06, step = 16496 (5.128 sec) 2022-04-22 18:17:17,914 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.214 INFO:tensorflow:global_step/sec: 13.2154 2022-04-22 18:17:17,996 [INFO] tensorflow: global_step/sec: 13.2154 2022-04-22 18:17:19,971 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 48.621 2022-04-22 18:17:21,748 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.284 INFO:tensorflow:global_step/sec: 13.1755 2022-04-22 18:17:22,550 [INFO] tensorflow: global_step/sec: 13.1755 INFO:tensorflow:epoch = 27.60166666666667, learning_rate = 0.00049999997, loss = -3.6993297e-06, step = 16561 (5.081 sec) 2022-04-22 18:17:22,615 [INFO] tensorflow: epoch = 27.60166666666667, learning_rate = 0.00049999997, loss = -3.6993297e-06, step = 16561 (5.081 sec) INFO:tensorflow:epoch = 27.603333333333335, learning_rate = 0.00049999997, loss = -3.7204009e-06, step = 16562 (5.069 sec) 2022-04-22 18:17:22,679 [INFO] tensorflow: epoch = 27.603333333333335, learning_rate = 0.00049999997, loss = -3.7204009e-06, step = 16562 (5.069 sec) 2022-04-22 18:17:23,502 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 57.009 2022-04-22 18:17:25,422 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 52.086 INFO:tensorflow:global_step/sec: 13.6208 2022-04-22 18:17:26,955 [INFO] tensorflow: global_step/sec: 13.6208 2022-04-22 18:17:27,231 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.285 INFO:tensorflow:epoch = 27.718333333333334, learning_rate = 0.00049999997, loss = -3.7160821e-06, step = 16631 (5.115 sec) 2022-04-22 18:17:27,730 [INFO] tensorflow: epoch = 27.718333333333334, learning_rate = 0.00049999997, loss = -3.7160821e-06, step = 16631 (5.115 sec) INFO:tensorflow:epoch = 27.720000000000002, learning_rate = 0.00049999997, loss = -3.7414015e-06, step = 16632 (5.120 sec) 2022-04-22 18:17:27,799 [INFO] tensorflow: epoch = 27.720000000000002, learning_rate = 0.00049999997, loss = -3.7414015e-06, step = 16632 (5.120 sec) 2022-04-22 18:17:29,054 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.887 2022-04-22 18:17:31,174 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 47.155 INFO:tensorflow:global_step/sec: 12.943 2022-04-22 18:17:31,590 [INFO] tensorflow: global_step/sec: 12.943 INFO:tensorflow:epoch = 27.825000000000003, learning_rate = 0.00049999997, loss = -3.724681e-06, step = 16695 (5.103 sec) 2022-04-22 18:17:32,834 [INFO] tensorflow: epoch = 27.825000000000003, learning_rate = 0.00049999997, loss = -3.724681e-06, step = 16695 (5.103 sec) INFO:tensorflow:epoch = 27.826666666666668, learning_rate = 0.00049999997, loss = -3.7210789e-06, step = 16696 (5.104 sec) 2022-04-22 18:17:32,903 [INFO] tensorflow: epoch = 27.826666666666668, learning_rate = 0.00049999997, loss = -3.7210789e-06, step = 16696 (5.104 sec) 2022-04-22 18:17:33,130 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 51.132 2022-04-22 18:17:34,957 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.760 INFO:tensorflow:global_step/sec: 13.4475 2022-04-22 18:17:36,052 [INFO] tensorflow: global_step/sec: 13.4475 2022-04-22 18:17:36,654 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 58.942 INFO:tensorflow:epoch = 27.945, learning_rate = 0.00049999997, loss = -3.7985224e-06, step = 16767 (5.078 sec) 2022-04-22 18:17:37,912 [INFO] tensorflow: epoch = 27.945, learning_rate = 0.00049999997, loss = -3.7985224e-06, step = 16767 (5.078 sec) INFO:tensorflow:epoch = 27.94666666666667, learning_rate = 0.00049999997, loss = -3.7795685e-06, step = 16768 (5.080 sec) 2022-04-22 18:17:37,983 [INFO] tensorflow: epoch = 27.94666666666667, learning_rate = 0.00049999997, loss = -3.7795685e-06, step = 16768 (5.080 sec) 2022-04-22 18:17:38,401 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 57.250 2022-04-22 18:17:40,145 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 57.324 INFO:tensorflow:global_step/sec: 14.4026 2022-04-22 18:17:40,218 [INFO] tensorflow: global_step/sec: 14.4026 2022-04-22 18:17:40,219 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 28/100: loss: -0.00000 learning rate: 0.00050 Time taken: 0:00:44.359648 ETA: 0:53:13.894621 2022-04-22 18:17:41,863 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 58.239 INFO:tensorflow:epoch = 28.068333333333335, learning_rate = 0.00049999997, loss = -3.7845648e-06, step = 16841 (5.124 sec) 2022-04-22 18:17:43,036 [INFO] tensorflow: epoch = 28.068333333333335, learning_rate = 0.00049999997, loss = -3.7845648e-06, step = 16841 (5.124 sec) INFO:tensorflow:epoch = 28.07, learning_rate = 0.00049999997, loss = -3.8042112e-06, step = 16842 (5.125 sec) 2022-04-22 18:17:43,108 [INFO] tensorflow: epoch = 28.07, learning_rate = 0.00049999997, loss = -3.8042112e-06, step = 16842 (5.125 sec) 2022-04-22 18:17:43,617 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 57.025 INFO:tensorflow:global_step/sec: 14.1285 2022-04-22 18:17:44,465 [INFO] tensorflow: global_step/sec: 14.1285 2022-04-22 18:17:45,467 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.051 2022-04-22 18:17:47,239 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.427 INFO:tensorflow:epoch = 28.185000000000002, learning_rate = 0.00049999997, loss = -3.7993182e-06, step = 16911 (5.162 sec) 2022-04-22 18:17:48,198 [INFO] tensorflow: epoch = 28.185000000000002, learning_rate = 0.00049999997, loss = -3.7993182e-06, step = 16911 (5.162 sec) INFO:tensorflow:epoch = 28.186666666666667, learning_rate = 0.00049999997, loss = -3.8151184e-06, step = 16912 (5.160 sec) 2022-04-22 18:17:48,268 [INFO] tensorflow: epoch = 28.186666666666667, learning_rate = 0.00049999997, loss = -3.8151184e-06, step = 16912 (5.160 sec) INFO:tensorflow:global_step/sec: 13.4051 2022-04-22 18:17:48,941 [INFO] tensorflow: global_step/sec: 13.4051 2022-04-22 18:17:49,267 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 49.331 2022-04-22 18:17:51,163 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 52.754 2022-04-22 18:17:52,992 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 54.668 INFO:tensorflow:epoch = 28.296666666666667, learning_rate = 0.00049999997, loss = -3.841181e-06, step = 16978 (5.075 sec) 2022-04-22 18:17:53,272 [INFO] tensorflow: epoch = 28.296666666666667, learning_rate = 0.00049999997, loss = -3.841181e-06, step = 16978 (5.075 sec) INFO:tensorflow:epoch = 28.298333333333336, learning_rate = 0.00049999997, loss = -3.834964e-06, step = 16979 (5.070 sec) 2022-04-22 18:17:53,338 [INFO] tensorflow: epoch = 28.298333333333336, learning_rate = 0.00049999997, loss = -3.834964e-06, step = 16979 (5.070 sec) INFO:tensorflow:global_step/sec: 13.3933 2022-04-22 18:17:53,421 [INFO] tensorflow: global_step/sec: 13.3933 2022-04-22 18:17:54,852 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.785 2022-04-22 18:17:56,669 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.018 INFO:tensorflow:global_step/sec: 13.7 2022-04-22 18:17:57,800 [INFO] tensorflow: global_step/sec: 13.7 INFO:tensorflow:epoch = 28.413333333333334, learning_rate = 0.00049999997, loss = -3.861151e-06, step = 17048 (5.102 sec) 2022-04-22 18:17:58,374 [INFO] tensorflow: epoch = 28.413333333333334, learning_rate = 0.00049999997, loss = -3.861151e-06, step = 17048 (5.102 sec) INFO:tensorflow:epoch = 28.415000000000003, learning_rate = 0.00049999997, loss = -3.8406924e-06, step = 17049 (5.116 sec) 2022-04-22 18:17:58,454 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.032 2022-04-22 18:17:58,454 [INFO] tensorflow: epoch = 28.415000000000003, learning_rate = 0.00049999997, loss = -3.8406924e-06, step = 17049 (5.116 sec) 2022-04-22 18:18:00,241 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.976 2022-04-22 18:18:02,032 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 55.842 INFO:tensorflow:global_step/sec: 13.9395 2022-04-22 18:18:02,104 [INFO] tensorflow: global_step/sec: 13.9395 INFO:tensorflow:epoch = 28.528333333333336, learning_rate = 0.00049999997, loss = -3.883557e-06, step = 17117 (5.090 sec) 2022-04-22 18:18:03,464 [INFO] tensorflow: epoch = 28.528333333333336, learning_rate = 0.00049999997, loss = -3.883557e-06, step = 17117 (5.090 sec) INFO:tensorflow:epoch = 28.53, learning_rate = 0.00049999997, loss = -3.87267e-06, step = 17118 (5.080 sec) 2022-04-22 18:18:03,535 [INFO] tensorflow: epoch = 28.53, learning_rate = 0.00049999997, loss = -3.87267e-06, step = 17118 (5.080 sec) 2022-04-22 18:18:03,969 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 51.621 2022-04-22 18:18:05,741 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 56.445 INFO:tensorflow:global_step/sec: 13.5054 2022-04-22 18:18:06,547 [INFO] tensorflow: global_step/sec: 13.5054 2022-04-22 18:18:07,597 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.890 INFO:tensorflow:epoch = 28.645000000000003, learning_rate = 0.00049999997, loss = -3.9137303e-06, step = 17187 (5.064 sec) 2022-04-22 18:18:08,528 [INFO] tensorflow: epoch = 28.645000000000003, learning_rate = 0.00049999997, loss = -3.9137303e-06, step = 17187 (5.064 sec) INFO:tensorflow:epoch = 28.648333333333333, learning_rate = 0.00049999997, loss = -3.8726253e-06, step = 17189 (5.144 sec) 2022-04-22 18:18:08,679 [INFO] tensorflow: epoch = 28.648333333333333, learning_rate = 0.00049999997, loss = -3.8726253e-06, step = 17189 (5.144 sec) 2022-04-22 18:18:09,488 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 52.876 INFO:tensorflow:global_step/sec: 12.9413 2022-04-22 18:18:11,183 [INFO] tensorflow: global_step/sec: 12.9413 2022-04-22 18:18:11,481 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 50.184 2022-04-22 18:18:13,365 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 53.094 INFO:tensorflow:epoch = 28.755000000000003, learning_rate = 0.00049999997, loss = -3.910015e-06, step = 17253 (5.137 sec) 2022-04-22 18:18:13,665 [INFO] tensorflow: epoch = 28.755000000000003, learning_rate = 0.00049999997, loss = -3.910015e-06, step = 17253 (5.137 sec) INFO:tensorflow:epoch = 28.758333333333336, learning_rate = 0.00049999997, loss = -3.878547e-06, step = 17255 (5.125 sec) 2022-04-22 18:18:13,803 [INFO] tensorflow: epoch = 28.758333333333336, learning_rate = 0.00049999997, loss = -3.878547e-06, step = 17255 (5.125 sec)