2021-09-25 23:03:21,831 [INFO] root: Registry: ['nvcr.io'] Matplotlib created a temporary config/cache directory at /tmp/matplotlib-y3_fqod7 because the default path (/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing. Using TensorFlow backend. WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them. Using TensorFlow backend. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:43: The name tf.train.SessionRunHook is deprecated. Please use tf.estimator.SessionRunHook instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/checkpoint_saver_hook.py:25: The name tf.train.CheckpointSaverHook is deprecated. Please use tf.estimator.CheckpointSaverHook instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/scripts/train.py:68: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/scripts/train.py:68: The name tf.logging.INFO is deprecated. Please use tf.compat.v1.logging.INFO instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod/tensorflow/__init__.py:117: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead. 2021-09-25 23:03:30,375 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/horovod/tensorflow/__init__.py:117: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod/tensorflow/__init__.py:143: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead. 2021-09-25 23:03:30,375 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/horovod/tensorflow/__init__.py:143: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead. 2021-09-25 23:03:31,245 [INFO] __main__: Loading experiment spec at /workspace/tao-experiments/platesNet/Lab/specs/platesNet_train_resnet18_kitti.txt. 2021-09-25 23:03:31,248 [INFO] iva.detectnet_v2.spec_handler.spec_loader: Merging specification from /workspace/tao-experiments/platesNet/Lab/specs/platesNet_train_resnet18_kitti.txt 2021-09-25 23:03:31,408 [INFO] __main__: Cannot iterate over exactly 297 samples with a batch size of 64; each epoch will therefore take one extra step. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:107: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead. 2021-09-25 23:03:31,410 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:107: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:110: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead. 2021-09-25 23:03:31,411 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:110: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:113: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead. 2021-09-25 23:03:31,414 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:113: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead. 2021-09-25 23:03:31,921 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead. 2021-09-25 23:03:31,932 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1834: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead. 2021-09-25 23:03:31,962 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1834: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead. 2021-09-25 23:03:32,918 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead. 2021-09-25 23:03:32,919 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead. 2021-09-25 23:03:33,129 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead. /usr/local/lib/python3.6/dist-packages/keras/engine/saving.py:292: UserWarning: No training configuration found in save file: the model was *not* compiled. Compile it manually. warnings.warn('No training configuration found in save file: ' 2021-09-25 23:03:33,516 [INFO] iva.detectnet_v2.objectives.bbox_objective: Default L1 loss function will be used. __________________________________________________________________________________________________ Layer (type) Output Shape Param # Connected to ================================================================================================== input_1 (InputLayer) (None, 3, 480, 640) 0 __________________________________________________________________________________________________ conv1 (Conv2D) (None, 64, 240, 320) 9472 input_1[0][0] __________________________________________________________________________________________________ bn_conv1 (BatchNormalization) (None, 64, 240, 320) 256 conv1[0][0] __________________________________________________________________________________________________ activation_1 (Activation) (None, 64, 240, 320) 0 bn_conv1[0][0] __________________________________________________________________________________________________ block_1a_conv_1 (Conv2D) (None, 64, 120, 160) 36928 activation_1[0][0] __________________________________________________________________________________________________ block_1a_bn_1 (BatchNormalizati (None, 64, 120, 160) 256 block_1a_conv_1[0][0] __________________________________________________________________________________________________ block_1a_relu_1 (Activation) (None, 64, 120, 160) 0 block_1a_bn_1[0][0] __________________________________________________________________________________________________ block_1a_conv_2 (Conv2D) (None, 64, 120, 160) 36928 block_1a_relu_1[0][0] __________________________________________________________________________________________________ block_1a_conv_shortcut (Conv2D) (None, 64, 120, 160) 4160 activation_1[0][0] __________________________________________________________________________________________________ block_1a_bn_2 (BatchNormalizati (None, 64, 120, 160) 256 block_1a_conv_2[0][0] __________________________________________________________________________________________________ block_1a_bn_shortcut (BatchNorm (None, 64, 120, 160) 256 block_1a_conv_shortcut[0][0] __________________________________________________________________________________________________ add_1 (Add) (None, 64, 120, 160) 0 block_1a_bn_2[0][0] block_1a_bn_shortcut[0][0] __________________________________________________________________________________________________ block_1a_relu (Activation) (None, 64, 120, 160) 0 add_1[0][0] __________________________________________________________________________________________________ block_1b_conv_1 (Conv2D) (None, 64, 120, 160) 36928 block_1a_relu[0][0] __________________________________________________________________________________________________ block_1b_bn_1 (BatchNormalizati (None, 64, 120, 160) 256 block_1b_conv_1[0][0] __________________________________________________________________________________________________ block_1b_relu_1 (Activation) (None, 64, 120, 160) 0 block_1b_bn_1[0][0] __________________________________________________________________________________________________ block_1b_conv_2 (Conv2D) (None, 64, 120, 160) 36928 block_1b_relu_1[0][0] __________________________________________________________________________________________________ block_1b_bn_2 (BatchNormalizati (None, 64, 120, 160) 256 block_1b_conv_2[0][0] __________________________________________________________________________________________________ add_2 (Add) (None, 64, 120, 160) 0 block_1b_bn_2[0][0] block_1a_relu[0][0] __________________________________________________________________________________________________ block_1b_relu (Activation) (None, 64, 120, 160) 0 add_2[0][0] __________________________________________________________________________________________________ block_2a_conv_1 (Conv2D) (None, 128, 60, 80) 73856 block_1b_relu[0][0] __________________________________________________________________________________________________ block_2a_bn_1 (BatchNormalizati (None, 128, 60, 80) 512 block_2a_conv_1[0][0] __________________________________________________________________________________________________ block_2a_relu_1 (Activation) (None, 128, 60, 80) 0 block_2a_bn_1[0][0] __________________________________________________________________________________________________ block_2a_conv_2 (Conv2D) (None, 128, 60, 80) 147584 block_2a_relu_1[0][0] __________________________________________________________________________________________________ block_2a_conv_shortcut (Conv2D) (None, 128, 60, 80) 8320 block_1b_relu[0][0] __________________________________________________________________________________________________ block_2a_bn_2 (BatchNormalizati (None, 128, 60, 80) 512 block_2a_conv_2[0][0] __________________________________________________________________________________________________ block_2a_bn_shortcut (BatchNorm (None, 128, 60, 80) 512 block_2a_conv_shortcut[0][0] __________________________________________________________________________________________________ add_3 (Add) (None, 128, 60, 80) 0 block_2a_bn_2[0][0] block_2a_bn_shortcut[0][0] __________________________________________________________________________________________________ block_2a_relu (Activation) (None, 128, 60, 80) 0 add_3[0][0] __________________________________________________________________________________________________ block_2b_conv_1 (Conv2D) (None, 128, 60, 80) 147584 block_2a_relu[0][0] __________________________________________________________________________________________________ block_2b_bn_1 (BatchNormalizati (None, 128, 60, 80) 512 block_2b_conv_1[0][0] __________________________________________________________________________________________________ block_2b_relu_1 (Activation) (None, 128, 60, 80) 0 block_2b_bn_1[0][0] __________________________________________________________________________________________________ block_2b_conv_2 (Conv2D) (None, 128, 60, 80) 147584 block_2b_relu_1[0][0] __________________________________________________________________________________________________ block_2b_bn_2 (BatchNormalizati (None, 128, 60, 80) 512 block_2b_conv_2[0][0] __________________________________________________________________________________________________ add_4 (Add) (None, 128, 60, 80) 0 block_2b_bn_2[0][0] block_2a_relu[0][0] __________________________________________________________________________________________________ block_2b_relu (Activation) (None, 128, 60, 80) 0 add_4[0][0] __________________________________________________________________________________________________ block_3a_conv_1 (Conv2D) (None, 256, 30, 40) 295168 block_2b_relu[0][0] __________________________________________________________________________________________________ block_3a_bn_1 (BatchNormalizati (None, 256, 30, 40) 1024 block_3a_conv_1[0][0] __________________________________________________________________________________________________ block_3a_relu_1 (Activation) (None, 256, 30, 40) 0 block_3a_bn_1[0][0] __________________________________________________________________________________________________ block_3a_conv_2 (Conv2D) (None, 256, 30, 40) 590080 block_3a_relu_1[0][0] __________________________________________________________________________________________________ block_3a_conv_shortcut (Conv2D) (None, 256, 30, 40) 33024 block_2b_relu[0][0] __________________________________________________________________________________________________ block_3a_bn_2 (BatchNormalizati (None, 256, 30, 40) 1024 block_3a_conv_2[0][0] __________________________________________________________________________________________________ block_3a_bn_shortcut (BatchNorm (None, 256, 30, 40) 1024 block_3a_conv_shortcut[0][0] __________________________________________________________________________________________________ add_5 (Add) (None, 256, 30, 40) 0 block_3a_bn_2[0][0] block_3a_bn_shortcut[0][0] __________________________________________________________________________________________________ block_3a_relu (Activation) (None, 256, 30, 40) 0 add_5[0][0] __________________________________________________________________________________________________ block_3b_conv_1 (Conv2D) (None, 256, 30, 40) 590080 block_3a_relu[0][0] __________________________________________________________________________________________________ block_3b_bn_1 (BatchNormalizati (None, 256, 30, 40) 1024 block_3b_conv_1[0][0] __________________________________________________________________________________________________ block_3b_relu_1 (Activation) (None, 256, 30, 40) 0 block_3b_bn_1[0][0] __________________________________________________________________________________________________ block_3b_conv_2 (Conv2D) (None, 256, 30, 40) 590080 block_3b_relu_1[0][0] __________________________________________________________________________________________________ block_3b_bn_2 (BatchNormalizati (None, 256, 30, 40) 1024 block_3b_conv_2[0][0] __________________________________________________________________________________________________ add_6 (Add) (None, 256, 30, 40) 0 block_3b_bn_2[0][0] block_3a_relu[0][0] __________________________________________________________________________________________________ block_3b_relu (Activation) (None, 256, 30, 40) 0 add_6[0][0] __________________________________________________________________________________________________ block_4a_conv_1 (Conv2D) (None, 512, 30, 40) 1180160 block_3b_relu[0][0] __________________________________________________________________________________________________ block_4a_bn_1 (BatchNormalizati (None, 512, 30, 40) 2048 block_4a_conv_1[0][0] __________________________________________________________________________________________________ block_4a_relu_1 (Activation) (None, 512, 30, 40) 0 block_4a_bn_1[0][0] __________________________________________________________________________________________________ block_4a_conv_2 (Conv2D) (None, 512, 30, 40) 2359808 block_4a_relu_1[0][0] __________________________________________________________________________________________________ block_4a_conv_shortcut (Conv2D) (None, 512, 30, 40) 131584 block_3b_relu[0][0] __________________________________________________________________________________________________ block_4a_bn_2 (BatchNormalizati (None, 512, 30, 40) 2048 block_4a_conv_2[0][0] __________________________________________________________________________________________________ block_4a_bn_shortcut (BatchNorm (None, 512, 30, 40) 2048 block_4a_conv_shortcut[0][0] __________________________________________________________________________________________________ add_7 (Add) (None, 512, 30, 40) 0 block_4a_bn_2[0][0] block_4a_bn_shortcut[0][0] __________________________________________________________________________________________________ block_4a_relu (Activation) (None, 512, 30, 40) 0 add_7[0][0] __________________________________________________________________________________________________ block_4b_conv_1 (Conv2D) (None, 512, 30, 40) 2359808 block_4a_relu[0][0] __________________________________________________________________________________________________ block_4b_bn_1 (BatchNormalizati (None, 512, 30, 40) 2048 block_4b_conv_1[0][0] __________________________________________________________________________________________________ block_4b_relu_1 (Activation) (None, 512, 30, 40) 0 block_4b_bn_1[0][0] __________________________________________________________________________________________________ block_4b_conv_2 (Conv2D) (None, 512, 30, 40) 2359808 block_4b_relu_1[0][0] __________________________________________________________________________________________________ block_4b_bn_2 (BatchNormalizati (None, 512, 30, 40) 2048 block_4b_conv_2[0][0] __________________________________________________________________________________________________ add_8 (Add) (None, 512, 30, 40) 0 block_4b_bn_2[0][0] block_4a_relu[0][0] __________________________________________________________________________________________________ block_4b_relu (Activation) (None, 512, 30, 40) 0 add_8[0][0] __________________________________________________________________________________________________ output_bbox (Conv2D) (None, 4, 30, 40) 2052 block_4b_relu[0][0] __________________________________________________________________________________________________ output_cov (Conv2D) (None, 1, 30, 40) 513 block_4b_relu[0][0] ================================================================================================== Total params: 11,197,893 Trainable params: 11,188,165 Non-trainable params: 9,728 __________________________________________________________________________________________________ 2021-09-25 23:03:35,380 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Serial augmentation enabled = False 2021-09-25 23:03:35,380 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Pseudo sharding enabled = False 2021-09-25 23:03:35,380 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Max Image Dimensions (all sources): (0, 0) 2021-09-25 23:03:35,381 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: number of cpus: 8, io threads: 16, compute threads: 8, buffered batches: 4 2021-09-25 23:03:35,381 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: total dataset size 297, number of sources: 1, batch size per gpu: 64, steps: 5 WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead. 2021-09-25 23:03:35,432 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead. WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2021-09-25 23:03:35,482 [WARNING] tensorflow: Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2021-09-25 23:03:35,511 [INFO] iva.detectnet_v2.dataloader.default_dataloader: Bounding box coordinates were detected in the input specification! Bboxes will be automatically converted to polygon coordinates. 2021-09-25 23:03:35,874 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: shuffle: True - shard 0 of 1 2021-09-25 23:03:35,882 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: sampling 1 datasets with weights: 2021-09-25 23:03:35,882 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: source: 0 weight: 1.000000 WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2021-09-25 23:03:35,904 [WARNING] tensorflow: Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2021-09-25 23:03:36,437 [INFO] __main__: Found 297 samples in training set WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/rasterizers/bbox_rasterizer.py:347: The name tf.bincount is deprecated. Please use tf.math.bincount instead. 2021-09-25 23:03:36,592 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/rasterizers/bbox_rasterizer.py:347: The name tf.bincount is deprecated. Please use tf.math.bincount instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/training_proto_utilities.py:89: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead. 2021-09-25 23:03:36,753 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/training_proto_utilities.py:89: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/training_proto_utilities.py:36: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead. 2021-09-25 23:03:36,776 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/training_proto_utilities.py:36: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_functions.py:17: The name tf.log is deprecated. Please use tf.math.log instead. 2021-09-25 23:03:36,999 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_functions.py:17: The name tf.log is deprecated. Please use tf.math.log instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:235: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead. 2021-09-25 23:03:37,014 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:235: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/model/detectnet_model.py:587: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead. 2021-09-25 23:03:37,019 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/model/detectnet_model.py:587: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead. 2021-09-25 23:03:39,166 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Serial augmentation enabled = False 2021-09-25 23:03:39,167 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Pseudo sharding enabled = False 2021-09-25 23:03:39,167 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Max Image Dimensions (all sources): (0, 0) 2021-09-25 23:03:39,167 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: number of cpus: 8, io threads: 16, compute threads: 8, buffered batches: 4 2021-09-25 23:03:39,167 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: total dataset size 33, number of sources: 1, batch size per gpu: 64, steps: 1 WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2021-09-25 23:03:39,183 [WARNING] tensorflow: Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2021-09-25 23:03:39,211 [INFO] iva.detectnet_v2.dataloader.default_dataloader: Bounding box coordinates were detected in the input specification! Bboxes will be automatically converted to polygon coordinates. 2021-09-25 23:03:39,553 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: shuffle: False - shard 0 of 1 2021-09-25 23:03:39,561 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: sampling 1 datasets with weights: 2021-09-25 23:03:39,561 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: source: 0 weight: 1.000000 WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2021-09-25 23:03:39,582 [WARNING] tensorflow: Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2021-09-25 23:03:39,919 [INFO] __main__: Found 33 samples in validation set WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/validation_hook.py:40: The name tf.summary.FileWriterCache is deprecated. Please use tf.compat.v1.summary.FileWriterCache instead. 2021-09-25 23:03:40,643 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/validation_hook.py:40: The name tf.summary.FileWriterCache is deprecated. Please use tf.compat.v1.summary.FileWriterCache instead. 2021-09-25 23:03:42,008 [INFO] __main__: Checkpoint interval: 20 WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/scripts/train.py:108: The name tf.train.Scaffold is deprecated. Please use tf.compat.v1.train.Scaffold instead. 2021-09-25 23:03:42,009 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/scripts/train.py:108: The name tf.train.Scaffold is deprecated. Please use tf.compat.v1.train.Scaffold instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:14: The name tf.local_variables_initializer is deprecated. Please use tf.compat.v1.local_variables_initializer instead. 2021-09-25 23:03:42,009 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:14: The name tf.local_variables_initializer is deprecated. Please use tf.compat.v1.local_variables_initializer instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:15: The name tf.tables_initializer is deprecated. Please use tf.compat.v1.tables_initializer instead. 2021-09-25 23:03:42,010 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:15: The name tf.tables_initializer is deprecated. Please use tf.compat.v1.tables_initializer instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:16: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead. 2021-09-25 23:03:42,011 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:16: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/utils.py:59: The name tf.train.LoggingTensorHook is deprecated. Please use tf.estimator.LoggingTensorHook instead. 2021-09-25 23:03:42,014 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/utils.py:59: The name tf.train.LoggingTensorHook is deprecated. Please use tf.estimator.LoggingTensorHook instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/utils.py:60: The name tf.train.StopAtStepHook is deprecated. Please use tf.estimator.StopAtStepHook instead. 2021-09-25 23:03:42,014 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/utils.py:60: The name tf.train.StopAtStepHook is deprecated. Please use tf.estimator.StopAtStepHook instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/utils.py:73: The name tf.train.StepCounterHook is deprecated. Please use tf.estimator.StepCounterHook instead. 2021-09-25 23:03:42,014 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/utils.py:73: The name tf.train.StepCounterHook is deprecated. Please use tf.estimator.StepCounterHook instead. INFO:tensorflow:Create CheckpointSaverHook. 2021-09-25 23:03:42,014 [INFO] tensorflow: Create CheckpointSaverHook. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/utils.py:99: The name tf.train.SummarySaverHook is deprecated. Please use tf.estimator.SummarySaverHook instead. 2021-09-25 23:03:42,015 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/utils.py:99: The name tf.train.SummarySaverHook is deprecated. Please use tf.estimator.SummarySaverHook instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/utilities.py:140: The name tf.train.SingularMonitoredSession is deprecated. Please use tf.compat.v1.train.SingularMonitoredSession instead. 2021-09-25 23:03:42,015 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/utilities.py:140: The name tf.train.SingularMonitoredSession is deprecated. Please use tf.compat.v1.train.SingularMonitoredSession instead. INFO:tensorflow:Graph was finalized. 2021-09-25 23:03:43,361 [INFO] tensorflow: Graph was finalized. INFO:tensorflow:Running local_init_op. 2021-09-25 23:03:45,656 [INFO] tensorflow: Running local_init_op. INFO:tensorflow:Done running local_init_op. 2021-09-25 23:03:46,523 [INFO] tensorflow: Done running local_init_op. INFO:tensorflow:Saving checkpoints for step-0. 2021-09-25 23:03:53,596 [INFO] tensorflow: Saving checkpoints for step-0. INFO:tensorflow:epoch = 0.0, learning_rate = 4.9999994e-06, loss = 0.0017903168, step = 0 2021-09-25 23:09:44,445 [INFO] tensorflow: epoch = 0.0, learning_rate = 4.9999994e-06, loss = 0.0017903168, step = 0 2021-09-25 23:09:44,451 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 0/500: loss: 0.00179 learning rate: 0.00000 Time taken: 0:00:00 ETA: 0:00:00 2021-09-25 23:09:44,451 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 0.194 INFO:tensorflow:global_step/sec: 1.72349 2021-09-25 23:09:45,029 [INFO] tensorflow: global_step/sec: 1.72349 INFO:tensorflow:global_step/sec: 1.81085 2021-09-25 23:09:45,582 [INFO] tensorflow: global_step/sec: 1.81085 INFO:tensorflow:global_step/sec: 1.73125 2021-09-25 23:09:46,159 [INFO] tensorflow: global_step/sec: 1.73125 INFO:tensorflow:global_step/sec: 1.78943 2021-09-25 23:09:46,718 [INFO] tensorflow: global_step/sec: 1.78943 dad7739149f8:43:63 [0] NCCL INFO Bootstrap : Using [0]lo:127.0.0.1<0> [1]eth0:172.17.0.8<0> dad7739149f8:43:63 [0] NCCL INFO NET/Plugin : Plugin load returned 0 : libnccl-net.so: cannot open shared object file: No such file or directory. dad7739149f8:43:63 [0] NCCL INFO NET/IB : No device found. dad7739149f8:43:63 [0] NCCL INFO NET/Socket : Using [0]lo:127.0.0.1<0> [1]eth0:172.17.0.8<0> dad7739149f8:43:63 [0] NCCL INFO Using network Socket NCCL version 2.7.8+cuda11.1 dad7739149f8:43:63 [0] NCCL INFO Channel 00/32 : 0 dad7739149f8:43:63 [0] NCCL INFO Channel 01/32 : 0 dad7739149f8:43:63 [0] NCCL INFO Channel 02/32 : 0 dad7739149f8:43:63 [0] NCCL INFO Channel 03/32 : 0 dad7739149f8:43:63 [0] NCCL INFO Channel 04/32 : 0 dad7739149f8:43:63 [0] NCCL INFO Channel 05/32 : 0 dad7739149f8:43:63 [0] NCCL INFO Channel 06/32 : 0 dad7739149f8:43:63 [0] NCCL INFO Channel 07/32 : 0 dad7739149f8:43:63 [0] NCCL INFO Channel 08/32 : 0 dad7739149f8:43:63 [0] NCCL INFO Channel 09/32 : 0 dad7739149f8:43:63 [0] NCCL INFO Channel 10/32 : 0 dad7739149f8:43:63 [0] NCCL INFO Channel 11/32 : 0 dad7739149f8:43:63 [0] NCCL INFO Channel 12/32 : 0 dad7739149f8:43:63 [0] NCCL INFO Channel 13/32 : 0 dad7739149f8:43:63 [0] NCCL INFO Channel 14/32 : 0 dad7739149f8:43:63 [0] NCCL INFO Channel 15/32 : 0 dad7739149f8:43:63 [0] NCCL INFO Channel 16/32 : 0 dad7739149f8:43:63 [0] NCCL INFO Channel 17/32 : 0 dad7739149f8:43:63 [0] NCCL INFO Channel 18/32 : 0 dad7739149f8:43:63 [0] NCCL INFO Channel 19/32 : 0 dad7739149f8:43:63 [0] NCCL INFO Channel 20/32 : 0 dad7739149f8:43:63 [0] NCCL INFO Channel 21/32 : 0 dad7739149f8:43:63 [0] NCCL INFO Channel 22/32 : 0 dad7739149f8:43:63 [0] NCCL INFO Channel 23/32 : 0 dad7739149f8:43:63 [0] NCCL INFO Channel 24/32 : 0 dad7739149f8:43:63 [0] NCCL INFO Channel 25/32 : 0 dad7739149f8:43:63 [0] NCCL INFO Channel 26/32 : 0 dad7739149f8:43:63 [0] NCCL INFO Channel 27/32 : 0 dad7739149f8:43:63 [0] NCCL INFO Channel 28/32 : 0 dad7739149f8:43:63 [0] NCCL INFO Channel 29/32 : 0 dad7739149f8:43:63 [0] NCCL INFO Channel 30/32 : 0 dad7739149f8:43:63 [0] NCCL INFO Channel 31/32 : 0 dad7739149f8:43:63 [0] NCCL INFO Trees [0] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [1] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [2] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [3] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [4] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [5] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [6] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [7] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [8] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [9] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [10] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [11] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [12] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [13] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [14] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [15] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [16] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [17] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [18] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [19] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [20] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [21] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [22] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [23] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [24] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [25] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [26] -1/-1/-1->0->-1|-1->0->-1/- dad7739149f8:43:63 [0] NCCL INFO 32 coll channels, 32 p2p channels, 32 p2p channels per peer dad7739149f8:43:63 [0] NCCL INFO comm 0x7f4e02fef940 rank 0 nranks 1 cudaDev 0 busId 1e0 - Init COMPLETE /usr/local/lib/python3.6/dist-packages/numpy/core/fromnumeric.py:3257: RuntimeWarning: Mean of empty slice. out=out, **kwargs) /usr/local/lib/python3.6/dist-packages/numpy/core/_methods.py:161: RuntimeWarning: invalid value encountered in double_scalars ret = ret.dtype.type(ret / rcount) Epoch 1/500 ========================= Validation cost: 0.000000 Mean average_precision (in %): 0.0000 class name average precision (in %) ------------ -------------------------- lpd 0 Median Inference Time: nan INFO:tensorflow:global_step/sec: 0.928682 2021-09-25 23:09:47,795 [INFO] tensorflow: global_step/sec: 0.928682 2021-09-25 23:09:47,796 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 1/500: loss: 0.00043 learning rate: 0.00001 Time taken: 0:05:32.866569 ETA: 1 day, 22:08:20.417714 INFO:tensorflow:global_step/sec: 1.77195 2021-09-25 23:09:48,359 [INFO] tensorflow: global_step/sec: 1.77195 INFO:tensorflow:global_step/sec: 1.75554 2021-09-25 23:09:48,929 [INFO] tensorflow: global_step/sec: 1.75554 INFO:tensorflow:global_step/sec: 1.81659 2021-09-25 23:09:49,479 [INFO] tensorflow: global_step/sec: 1.81659 INFO:tensorflow:epoch = 1.8, learning_rate = 5.9016e-06, loss = 0.00047852763, step = 9 (5.584 sec) 2021-09-25 23:09:50,028 [INFO] tensorflow: epoch = 1.8, learning_rate = 5.9016e-06, loss = 0.00047852763, step = 9 (5.584 sec) INFO:tensorflow:global_step/sec: 1.81716 2021-09-25 23:09:50,030 [INFO] tensorflow: global_step/sec: 1.81716 INFO:tensorflow:global_step/sec: 1.69633 2021-09-25 23:09:50,619 [INFO] tensorflow: global_step/sec: 1.69633 2021-09-25 23:09:50,621 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 2/500: loss: 0.00041 learning rate: 0.00001 Time taken: 0:00:02.806251 ETA: 0:23:17.512785 INFO:tensorflow:global_step/sec: 1.76941 2021-09-25 23:09:51,184 [INFO] tensorflow: global_step/sec: 1.76941 INFO:tensorflow:global_step/sec: 1.77589 2021-09-25 23:09:51,747 [INFO] tensorflow: global_step/sec: 1.77589 INFO:tensorflow:global_step/sec: 1.78344 2021-09-25 23:09:52,308 [INFO] tensorflow: global_step/sec: 1.78344 INFO:tensorflow:global_step/sec: 1.71597 2021-09-25 23:09:52,891 [INFO] tensorflow: global_step/sec: 1.71597 INFO:tensorflow:global_step/sec: 1.76006 2021-09-25 23:09:53,459 [INFO] tensorflow: global_step/sec: 1.76006 2021-09-25 23:09:53,460 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 3/500: loss: 0.00046 learning rate: 0.00001 Time taken: 0:00:02.860372 ETA: 0:23:41.604680 INFO:tensorflow:global_step/sec: 1.7808 2021-09-25 23:09:54,021 [INFO] tensorflow: global_step/sec: 1.7808 INFO:tensorflow:global_step/sec: 1.69813 2021-09-25 23:09:54,609 [INFO] tensorflow: global_step/sec: 1.69813 INFO:tensorflow:global_step/sec: 1.83142 2021-09-25 23:09:55,155 [INFO] tensorflow: global_step/sec: 1.83142 INFO:tensorflow:epoch = 3.8000000000000003, learning_rate = 7.095285e-06, loss = 0.00040653208, step = 19 (5.676 sec) 2021-09-25 23:09:55,704 [INFO] tensorflow: epoch = 3.8000000000000003, learning_rate = 7.095285e-06, loss = 0.00040653208, step = 19 (5.676 sec) INFO:tensorflow:global_step/sec: 1.81773 2021-09-25 23:09:55,706 [INFO] tensorflow: global_step/sec: 1.81773 INFO:tensorflow:global_step/sec: 1.78112 2021-09-25 23:09:56,267 [INFO] tensorflow: global_step/sec: 1.78112 2021-09-25 23:09:56,269 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 4/500: loss: 0.00042 learning rate: 0.00001 Time taken: 0:00:02.813027 ETA: 0:23:15.261227 INFO:tensorflow:global_step/sec: 1.82353 2021-09-25 23:09:56,815 [INFO] tensorflow: global_step/sec: 1.82353 INFO:tensorflow:global_step/sec: 1.81117 2021-09-25 23:09:57,368 [INFO] tensorflow: global_step/sec: 1.81117 INFO:tensorflow:global_step/sec: 1.8501 2021-09-25 23:09:57,908 [INFO] tensorflow: global_step/sec: 1.8501 INFO:tensorflow:global_step/sec: 1.79823 2021-09-25 23:09:58,464 [INFO] tensorflow: global_step/sec: 1.79823 2021-09-25 23:09:58,467 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 4.650 INFO:tensorflow:global_step/sec: 1.77921 2021-09-25 23:09:59,026 [INFO] tensorflow: global_step/sec: 1.77921 2021-09-25 23:09:59,028 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 5/500: loss: 0.00044 learning rate: 0.00001 Time taken: 0:00:02.757093 ETA: 0:22:44.761248 INFO:tensorflow:global_step/sec: 1.73641 2021-09-25 23:09:59,602 [INFO] tensorflow: global_step/sec: 1.73641 INFO:tensorflow:global_step/sec: 1.76691 2021-09-25 23:10:00,168 [INFO] tensorflow: global_step/sec: 1.76691 INFO:tensorflow:global_step/sec: 1.80711 2021-09-25 23:10:00,721 [INFO] tensorflow: global_step/sec: 1.80711 INFO:tensorflow:epoch = 5.800000000000001, learning_rate = 8.53041e-06, loss = 0.0003588209, step = 29 (5.602 sec) 2021-09-25 23:10:01,306 [INFO] tensorflow: epoch = 5.800000000000001, learning_rate = 8.53041e-06, loss = 0.0003588209, step = 29 (5.602 sec) INFO:tensorflow:global_step/sec: 1.7057 2021-09-25 23:10:01,308 [INFO] tensorflow: global_step/sec: 1.7057 INFO:tensorflow:global_step/sec: 1.82366 2021-09-25 23:10:01,856 [INFO] tensorflow: global_step/sec: 1.82366 2021-09-25 23:10:01,858 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 6/500: loss: 0.00037 learning rate: 0.00001 Time taken: 0:00:02.845276 ETA: 0:23:25.566520 INFO:tensorflow:global_step/sec: 1.79372 2021-09-25 23:10:02,414 [INFO] tensorflow: global_step/sec: 1.79372 INFO:tensorflow:global_step/sec: 1.77109 2021-09-25 23:10:02,978 [INFO] tensorflow: global_step/sec: 1.77109 INFO:tensorflow:global_step/sec: 1.76278 2021-09-25 23:10:03,545 [INFO] tensorflow: global_step/sec: 1.76278 INFO:tensorflow:global_step/sec: 1.77427 2021-09-25 23:10:04,109 [INFO] tensorflow: global_step/sec: 1.77427 INFO:tensorflow:global_step/sec: 1.36133 2021-09-25 23:10:04,844 [INFO] tensorflow: global_step/sec: 1.36133 2021-09-25 23:10:04,851 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 7/500: loss: 0.00040 learning rate: 0.00001 Time taken: 0:00:02.807297 ETA: 0:23:03.997299 INFO:tensorflow:global_step/sec: 1.48587 2021-09-25 23:10:05,517 [INFO] tensorflow: global_step/sec: 1.48587 INFO:tensorflow:global_step/sec: 1.31469 2021-09-25 23:10:06,277 [INFO] tensorflow: global_step/sec: 1.31469 INFO:tensorflow:global_step/sec: 1.73982 2021-09-25 23:10:06,852 [INFO] tensorflow: global_step/sec: 1.73982 INFO:tensorflow:epoch = 7.800000000000001, learning_rate = 1.0255811e-05, loss = 0.00039235008, step = 39 (6.110 sec) 2021-09-25 23:10:07,417 [INFO] tensorflow: epoch = 7.800000000000001, learning_rate = 1.0255811e-05, loss = 0.00039235008, step = 39 (6.110 sec) INFO:tensorflow:global_step/sec: 1.7666 2021-09-25 23:10:07,418 [INFO] tensorflow: global_step/sec: 1.7666 INFO:tensorflow:global_step/sec: 1.82082 2021-09-25 23:10:07,967 [INFO] tensorflow: global_step/sec: 1.82082 2021-09-25 23:10:07,969 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 8/500: loss: 0.00035 learning rate: 0.00001 Time taken: 0:00:03.298422 ETA: 0:27:02.823555 INFO:tensorflow:global_step/sec: 1.72131 2021-09-25 23:10:08,548 [INFO] tensorflow: global_step/sec: 1.72131 INFO:tensorflow:global_step/sec: 1.84165 2021-09-25 23:10:09,091 [INFO] tensorflow: global_step/sec: 1.84165 INFO:tensorflow:global_step/sec: 1.81173 2021-09-25 23:10:09,643 [INFO] tensorflow: global_step/sec: 1.81173 INFO:tensorflow:global_step/sec: 1.82185 2021-09-25 23:10:10,192 [INFO] tensorflow: global_step/sec: 1.82185 INFO:tensorflow:global_step/sec: 1.77563 2021-09-25 23:10:10,755 [INFO] tensorflow: global_step/sec: 1.77563 2021-09-25 23:10:10,757 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 9/500: loss: 0.00041 learning rate: 0.00001 Time taken: 0:00:02.772554 ETA: 0:22:41.324092 INFO:tensorflow:global_step/sec: 1.80668 2021-09-25 23:10:11,309 [INFO] tensorflow: global_step/sec: 1.80668 INFO:tensorflow:global_step/sec: 1.82293 2021-09-25 23:10:11,857 [INFO] tensorflow: global_step/sec: 1.82293 INFO:tensorflow:global_step/sec: 1.74871 2021-09-25 23:10:12,429 [INFO] tensorflow: global_step/sec: 1.74871 INFO:tensorflow:epoch = 9.8, learning_rate = 1.2330198e-05, loss = 0.00038022132, step = 49 (5.572 sec) 2021-09-25 23:10:12,989 [INFO] tensorflow: epoch = 9.8, learning_rate = 1.2330198e-05, loss = 0.00038022132, step = 49 (5.572 sec) INFO:tensorflow:global_step/sec: 1.78177 2021-09-25 23:10:12,990 [INFO] tensorflow: global_step/sec: 1.78177 2021-09-25 23:10:12,995 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 110.145 INFO:tensorflow:global_step/sec: 1.76599 2021-09-25 23:10:13,557 [INFO] tensorflow: global_step/sec: 1.76599 2021-09-25 23:10:13,558 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 10/500: loss: 0.00036 learning rate: 0.00001 Time taken: 0:00:02.795147 ETA: 0:22:49.622118 INFO:tensorflow:global_step/sec: 1.80585 2021-09-25 23:10:14,111 [INFO] tensorflow: global_step/sec: 1.80585 INFO:tensorflow:global_step/sec: 1.80283 2021-09-25 23:10:14,665 [INFO] tensorflow: global_step/sec: 1.80283 INFO:tensorflow:global_step/sec: 1.77876 2021-09-25 23:10:15,227 [INFO] tensorflow: global_step/sec: 1.77876 INFO:tensorflow:global_step/sec: 1.7953 2021-09-25 23:10:15,784 [INFO] tensorflow: global_step/sec: 1.7953 INFO:tensorflow:global_step/sec: 1.7196 2021-09-25 23:10:16,366 [INFO] tensorflow: global_step/sec: 1.7196 2021-09-25 23:10:16,367 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 11/500: loss: 0.00034 learning rate: 0.00001 Time taken: 0:00:02.795902 ETA: 0:22:47.196085 INFO:tensorflow:global_step/sec: 1.79993 2021-09-25 23:10:16,922 [INFO] tensorflow: global_step/sec: 1.79993 INFO:tensorflow:global_step/sec: 1.85234 2021-09-25 23:10:17,461 [INFO] tensorflow: global_step/sec: 1.85234 INFO:tensorflow:global_step/sec: 1.83078 2021-09-25 23:10:18,008 [INFO] tensorflow: global_step/sec: 1.83078 INFO:tensorflow:epoch = 11.8, learning_rate = 1.4824147e-05, loss = 0.00030940206, step = 59 (5.577 sec) 2021-09-25 23:10:18,566 [INFO] tensorflow: epoch = 11.8, learning_rate = 1.4824147e-05, loss = 0.00030940206, step = 59 (5.577 sec) INFO:tensorflow:global_step/sec: 1.78659 2021-09-25 23:10:18,567 [INFO] tensorflow: global_step/sec: 1.78659 INFO:tensorflow:global_step/sec: 1.81504 2021-09-25 23:10:19,118 [INFO] tensorflow: global_step/sec: 1.81504 2021-09-25 23:10:19,120 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 12/500: loss: 0.00035 learning rate: 0.00002 Time taken: 0:00:02.779264 ETA: 0:22:36.280703 INFO:tensorflow:global_step/sec: 1.77834 2021-09-25 23:10:19,681 [INFO] tensorflow: global_step/sec: 1.77834 INFO:tensorflow:global_step/sec: 1.72063 2021-09-25 23:10:20,262 [INFO] tensorflow: global_step/sec: 1.72063 INFO:tensorflow:global_step/sec: 1.84098 2021-09-25 23:10:20,805 [INFO] tensorflow: global_step/sec: 1.84098 INFO:tensorflow:global_step/sec: 1.76452 2021-09-25 23:10:21,372 [INFO] tensorflow: global_step/sec: 1.76452 INFO:tensorflow:global_step/sec: 1.76292 2021-09-25 23:10:21,939 [INFO] tensorflow: global_step/sec: 1.76292 2021-09-25 23:10:21,940 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 13/500: loss: 0.00034 learning rate: 0.00002 Time taken: 0:00:02.803322 ETA: 0:22:45.217968 INFO:tensorflow:global_step/sec: 1.80954 2021-09-25 23:10:22,492 [INFO] tensorflow: global_step/sec: 1.80954 INFO:tensorflow:global_step/sec: 1.75736 2021-09-25 23:10:23,061 [INFO] tensorflow: global_step/sec: 1.75736 INFO:tensorflow:global_step/sec: 1.80556 2021-09-25 23:10:23,614 [INFO] tensorflow: global_step/sec: 1.80556 INFO:tensorflow:epoch = 13.8, learning_rate = 1.7822547e-05, loss = 0.0002921686, step = 69 (5.608 sec) 2021-09-25 23:10:24,174 [INFO] tensorflow: epoch = 13.8, learning_rate = 1.7822547e-05, loss = 0.0002921686, step = 69 (5.608 sec) INFO:tensorflow:global_step/sec: 1.78238 2021-09-25 23:10:24,175 [INFO] tensorflow: global_step/sec: 1.78238 INFO:tensorflow:global_step/sec: 1.78763 2021-09-25 23:10:24,735 [INFO] tensorflow: global_step/sec: 1.78763 2021-09-25 23:10:24,736 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 14/500: loss: 0.00031 learning rate: 0.00002 Time taken: 0:00:02.801478 ETA: 0:22:41.518148 INFO:tensorflow:global_step/sec: 1.84593 2021-09-25 23:10:25,277 [INFO] tensorflow: global_step/sec: 1.84593 INFO:tensorflow:global_step/sec: 1.74911 2021-09-25 23:10:25,848 [INFO] tensorflow: global_step/sec: 1.74911 INFO:tensorflow:global_step/sec: 1.78892 2021-09-25 23:10:26,407 [INFO] tensorflow: global_step/sec: 1.78892 INFO:tensorflow:global_step/sec: 1.8316 2021-09-25 23:10:26,953 [INFO] tensorflow: global_step/sec: 1.8316 2021-09-25 23:10:26,956 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 114.609 INFO:tensorflow:global_step/sec: 1.8174 2021-09-25 23:10:27,504 [INFO] tensorflow: global_step/sec: 1.8174 2021-09-25 23:10:27,505 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 15/500: loss: 0.00036 learning rate: 0.00002 Time taken: 0:00:02.777539 ETA: 0:22:27.106191 INFO:tensorflow:global_step/sec: 1.80811 2021-09-25 23:10:28,057 [INFO] tensorflow: global_step/sec: 1.80811 INFO:tensorflow:global_step/sec: 1.79303 2021-09-25 23:10:28,614 [INFO] tensorflow: global_step/sec: 1.79303 INFO:tensorflow:global_step/sec: 1.8189 2021-09-25 23:10:29,164 [INFO] tensorflow: global_step/sec: 1.8189 INFO:tensorflow:global_step/sec: 1.76376 2021-09-25 23:10:29,731 [INFO] tensorflow: global_step/sec: 1.76376 INFO:tensorflow:epoch = 16.0, learning_rate = 2.1825796e-05, loss = 0.0002987127, step = 80 (6.129 sec) 2021-09-25 23:10:30,303 [INFO] tensorflow: epoch = 16.0, learning_rate = 2.1825796e-05, loss = 0.0002987127, step = 80 (6.129 sec) INFO:tensorflow:global_step/sec: 1.74509 2021-09-25 23:10:30,304 [INFO] tensorflow: global_step/sec: 1.74509 2021-09-25 23:10:30,305 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 16/500: loss: 0.00030 learning rate: 0.00002 Time taken: 0:00:02.777981 ETA: 0:22:24.542594 INFO:tensorflow:global_step/sec: 1.74713 2021-09-25 23:10:30,876 [INFO] tensorflow: global_step/sec: 1.74713 INFO:tensorflow:global_step/sec: 1.83478 2021-09-25 23:10:31,421 [INFO] tensorflow: global_step/sec: 1.83478 INFO:tensorflow:global_step/sec: 1.68705 2021-09-25 23:10:32,014 [INFO] tensorflow: global_step/sec: 1.68705 INFO:tensorflow:global_step/sec: 1.7879 2021-09-25 23:10:32,574 [INFO] tensorflow: global_step/sec: 1.7879 INFO:tensorflow:global_step/sec: 1.77902 2021-09-25 23:10:33,136 [INFO] tensorflow: global_step/sec: 1.77902 2021-09-25 23:10:33,137 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 17/500: loss: 0.00032 learning rate: 0.00002 Time taken: 0:00:02.841294 ETA: 0:22:52.345026 INFO:tensorflow:global_step/sec: 1.77321 2021-09-25 23:10:33,700 [INFO] tensorflow: global_step/sec: 1.77321 INFO:tensorflow:global_step/sec: 1.8377 2021-09-25 23:10:34,244 [INFO] tensorflow: global_step/sec: 1.8377 INFO:tensorflow:global_step/sec: 1.74003 2021-09-25 23:10:34,818 [INFO] tensorflow: global_step/sec: 1.74003 INFO:tensorflow:global_step/sec: 1.70827 2021-09-25 23:10:35,404 [INFO] tensorflow: global_step/sec: 1.70827 INFO:tensorflow:epoch = 18.0, learning_rate = 2.6240356e-05, loss = 0.00021556068, step = 90 (5.666 sec) 2021-09-25 23:10:35,969 [INFO] tensorflow: epoch = 18.0, learning_rate = 2.6240356e-05, loss = 0.00021556068, step = 90 (5.666 sec) INFO:tensorflow:global_step/sec: 1.7654 2021-09-25 23:10:35,970 [INFO] tensorflow: global_step/sec: 1.7654 2021-09-25 23:10:35,972 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 18/500: loss: 0.00022 learning rate: 0.00003 Time taken: 0:00:02.831376 ETA: 0:22:44.723154 INFO:tensorflow:global_step/sec: 1.73944 2021-09-25 23:10:36,545 [INFO] tensorflow: global_step/sec: 1.73944 INFO:tensorflow:global_step/sec: 1.80512 2021-09-25 23:10:37,099 [INFO] tensorflow: global_step/sec: 1.80512 INFO:tensorflow:global_step/sec: 1.74354 2021-09-25 23:10:37,673 [INFO] tensorflow: global_step/sec: 1.74354 INFO:tensorflow:global_step/sec: 1.84218 2021-09-25 23:10:38,216 [INFO] tensorflow: global_step/sec: 1.84218 INFO:tensorflow:global_step/sec: 1.71938 2021-09-25 23:10:38,797 [INFO] tensorflow: global_step/sec: 1.71938 2021-09-25 23:10:38,799 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 19/500: loss: 0.00021 learning rate: 0.00003 Time taken: 0:00:02.813073 ETA: 0:22:33.088074 INFO:tensorflow:global_step/sec: 1.7745 2021-09-25 23:10:39,361 [INFO] tensorflow: global_step/sec: 1.7745 INFO:tensorflow:global_step/sec: 1.7409 2021-09-25 23:10:39,935 [INFO] tensorflow: global_step/sec: 1.7409 INFO:tensorflow:global_step/sec: 1.75994 2021-09-25 23:10:40,503 [INFO] tensorflow: global_step/sec: 1.75994 INFO:tensorflow:global_step/sec: 1.72395 2021-09-25 23:10:41,083 [INFO] tensorflow: global_step/sec: 1.72395 INFO:tensorflow:Saving checkpoints for step-100. 2021-09-25 23:10:41,084 [INFO] tensorflow: Saving checkpoints for step-100. 2021-09-25 23:10:45,137 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 88.009 INFO:tensorflow:epoch = 20.0, learning_rate = 3.154785e-05, loss = 0.00022334616, step = 100 (9.749 sec) 2021-09-25 23:10:45,718 [INFO] tensorflow: epoch = 20.0, learning_rate = 3.154785e-05, loss = 0.00022334616, step = 100 (9.749 sec) INFO:tensorflow:global_step/sec: 0.215704 2021-09-25 23:10:45,719 [INFO] tensorflow: global_step/sec: 0.215704 2021-09-25 23:10:45,721 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 20/500: loss: 0.00022 learning rate: 0.00003 Time taken: 0:00:06.913314 ETA: 0:55:18.390770 INFO:tensorflow:global_step/sec: 1.81788 2021-09-25 23:10:46,269 [INFO] tensorflow: global_step/sec: 1.81788 INFO:tensorflow:global_step/sec: 1.76989 2021-09-25 23:10:46,834 [INFO] tensorflow: global_step/sec: 1.76989 INFO:tensorflow:global_step/sec: 1.80421 2021-09-25 23:10:47,389 [INFO] tensorflow: global_step/sec: 1.80421 INFO:tensorflow:global_step/sec: 1.8016 2021-09-25 23:10:47,944 [INFO] tensorflow: global_step/sec: 1.8016 Epoch 21/500 ========================= Validation cost: 0.000000 Mean average_precision (in %): 0.0000 class name average precision (in %) ------------ -------------------------- lpd 0 Median Inference Time: nan INFO:tensorflow:global_step/sec: 1.8092 2021-09-25 23:10:48,497 [INFO] tensorflow: global_step/sec: 1.8092 2021-09-25 23:10:48,498 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 21/500: loss: 0.00023 learning rate: 0.00003 Time taken: 0:00:02.810050 ETA: 0:22:26.013955 INFO:tensorflow:global_step/sec: 1.79134 2021-09-25 23:10:49,055 [INFO] tensorflow: global_step/sec: 1.79134 INFO:tensorflow:global_step/sec: 1.75066 2021-09-25 23:10:49,626 [INFO] tensorflow: global_step/sec: 1.75066 INFO:tensorflow:global_step/sec: 1.73718 2021-09-25 23:10:50,202 [INFO] tensorflow: global_step/sec: 1.73718 INFO:tensorflow:global_step/sec: 1.80299 2021-09-25 23:10:50,756 [INFO] tensorflow: global_step/sec: 1.80299 INFO:tensorflow:epoch = 22.0, learning_rate = 3.792886e-05, loss = 0.00021500375, step = 110 (5.601 sec) 2021-09-25 23:10:51,319 [INFO] tensorflow: epoch = 22.0, learning_rate = 3.792886e-05, loss = 0.00021500375, step = 110 (5.601 sec) INFO:tensorflow:global_step/sec: 1.77366 2021-09-25 23:10:51,320 [INFO] tensorflow: global_step/sec: 1.77366 2021-09-25 23:10:51,321 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 22/500: loss: 0.00022 learning rate: 0.00004 Time taken: 0:00:02.810559 ETA: 0:22:23.447332 INFO:tensorflow:global_step/sec: 1.77802 2021-09-25 23:10:51,882 [INFO] tensorflow: global_step/sec: 1.77802 INFO:tensorflow:global_step/sec: 1.80023 2021-09-25 23:10:52,438 [INFO] tensorflow: global_step/sec: 1.80023 INFO:tensorflow:global_step/sec: 1.83536 2021-09-25 23:10:52,983 [INFO] tensorflow: global_step/sec: 1.83536 INFO:tensorflow:global_step/sec: 1.77853 2021-09-25 23:10:53,545 [INFO] tensorflow: global_step/sec: 1.77853 INFO:tensorflow:global_step/sec: 1.74373 2021-09-25 23:10:54,119 [INFO] tensorflow: global_step/sec: 1.74373 2021-09-25 23:10:54,120 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 23/500: loss: 0.00019 learning rate: 0.00004 Time taken: 0:00:02.789656 ETA: 0:22:10.666103 INFO:tensorflow:global_step/sec: 1.77192 2021-09-25 23:10:54,683 [INFO] tensorflow: global_step/sec: 1.77192 INFO:tensorflow:global_step/sec: 1.76315 2021-09-25 23:10:55,250 [INFO] tensorflow: global_step/sec: 1.76315 INFO:tensorflow:global_step/sec: 1.71243 2021-09-25 23:10:55,834 [INFO] tensorflow: global_step/sec: 1.71243 INFO:tensorflow:global_step/sec: 1.78599 2021-09-25 23:10:56,394 [INFO] tensorflow: global_step/sec: 1.78599 INFO:tensorflow:epoch = 24.0, learning_rate = 4.5600533e-05, loss = 0.00017960329, step = 120 (5.652 sec) 2021-09-25 23:10:56,971 [INFO] tensorflow: epoch = 24.0, learning_rate = 4.5600533e-05, loss = 0.00017960329, step = 120 (5.652 sec) INFO:tensorflow:global_step/sec: 1.72926 2021-09-25 23:10:56,972 [INFO] tensorflow: global_step/sec: 1.72926 2021-09-25 23:10:56,974 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 24/500: loss: 0.00018 learning rate: 0.00005 Time taken: 0:00:02.846298 ETA: 0:22:34.837611 INFO:tensorflow:global_step/sec: 1.80562 2021-09-25 23:10:57,526 [INFO] tensorflow: global_step/sec: 1.80562 INFO:tensorflow:global_step/sec: 1.74145 2021-09-25 23:10:58,100 [INFO] tensorflow: global_step/sec: 1.74145 INFO:tensorflow:global_step/sec: 1.83679 2021-09-25 23:10:58,645 [INFO] tensorflow: global_step/sec: 1.83679 INFO:tensorflow:global_step/sec: 1.76213 2021-09-25 23:10:59,212 [INFO] tensorflow: global_step/sec: 1.76213 2021-09-25 23:10:59,216 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 113.654 INFO:tensorflow:global_step/sec: 1.81967 2021-09-25 23:10:59,762 [INFO] tensorflow: global_step/sec: 1.81967 2021-09-25 23:10:59,763 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 25/500: loss: 0.00017 learning rate: 0.00005 Time taken: 0:00:02.818211 ETA: 0:22:18.650036 INFO:tensorflow:global_step/sec: 1.79345 2021-09-25 23:11:00,319 [INFO] tensorflow: global_step/sec: 1.79345 INFO:tensorflow:global_step/sec: 1.78929 2021-09-25 23:11:00,878 [INFO] tensorflow: global_step/sec: 1.78929 INFO:tensorflow:global_step/sec: 1.75639 2021-09-25 23:11:01,448 [INFO] tensorflow: global_step/sec: 1.75639 INFO:tensorflow:global_step/sec: 1.71483 2021-09-25 23:11:02,031 [INFO] tensorflow: global_step/sec: 1.71483 INFO:tensorflow:epoch = 26.0, learning_rate = 5.4823908e-05, loss = 0.00016212248, step = 130 (5.628 sec) 2021-09-25 23:11:02,599 [INFO] tensorflow: epoch = 26.0, learning_rate = 5.4823908e-05, loss = 0.00016212248, step = 130 (5.628 sec) INFO:tensorflow:global_step/sec: 1.75433 2021-09-25 23:11:02,601 [INFO] tensorflow: global_step/sec: 1.75433 2021-09-25 23:11:02,602 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 26/500: loss: 0.00016 learning rate: 0.00005 Time taken: 0:00:02.818636 ETA: 0:22:16.033323 INFO:tensorflow:global_step/sec: 1.72376 2021-09-25 23:11:03,181 [INFO] tensorflow: global_step/sec: 1.72376 INFO:tensorflow:global_step/sec: 1.7866 2021-09-25 23:11:03,741 [INFO] tensorflow: global_step/sec: 1.7866 INFO:tensorflow:global_step/sec: 1.80012 2021-09-25 23:11:04,296 [INFO] tensorflow: global_step/sec: 1.80012 INFO:tensorflow:global_step/sec: 1.80054 2021-09-25 23:11:04,851 [INFO] tensorflow: global_step/sec: 1.80054 INFO:tensorflow:global_step/sec: 1.83814 2021-09-25 23:11:05,396 [INFO] tensorflow: global_step/sec: 1.83814 2021-09-25 23:11:05,397 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 27/500: loss: 0.00019 learning rate: 0.00006 Time taken: 0:00:02.821662 ETA: 0:22:14.646215 INFO:tensorflow:global_step/sec: 1.84165 2021-09-25 23:11:05,939 [INFO] tensorflow: global_step/sec: 1.84165 INFO:tensorflow:global_step/sec: 1.80989 2021-09-25 23:11:06,491 [INFO] tensorflow: global_step/sec: 1.80989 INFO:tensorflow:global_step/sec: 1.79374 2021-09-25 23:11:07,049 [INFO] tensorflow: global_step/sec: 1.79374 INFO:tensorflow:global_step/sec: 1.81894 2021-09-25 23:11:07,598 [INFO] tensorflow: global_step/sec: 1.81894 INFO:tensorflow:epoch = 28.0, learning_rate = 6.591285e-05, loss = 0.00014989186, step = 140 (5.564 sec) 2021-09-25 23:11:08,163 [INFO] tensorflow: epoch = 28.0, learning_rate = 6.591285e-05, loss = 0.00014989186, step = 140 (5.564 sec) INFO:tensorflow:global_step/sec: 1.76505 2021-09-25 23:11:08,165 [INFO] tensorflow: global_step/sec: 1.76505 2021-09-25 23:11:08,166 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 28/500: loss: 0.00015 learning rate: 0.00007 Time taken: 0:00:02.743579 ETA: 0:21:34.969246 INFO:tensorflow:global_step/sec: 1.76294 2021-09-25 23:11:08,732 [INFO] tensorflow: global_step/sec: 1.76294 INFO:tensorflow:global_step/sec: 1.7515 2021-09-25 23:11:09,303 [INFO] tensorflow: global_step/sec: 1.7515 INFO:tensorflow:global_step/sec: 1.73496 2021-09-25 23:11:09,879 [INFO] tensorflow: global_step/sec: 1.73496 INFO:tensorflow:global_step/sec: 1.75213 2021-09-25 23:11:10,450 [INFO] tensorflow: global_step/sec: 1.75213 INFO:tensorflow:global_step/sec: 1.74706 2021-09-25 23:11:11,023 [INFO] tensorflow: global_step/sec: 1.74706 2021-09-25 23:11:11,024 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 29/500: loss: 0.00013 learning rate: 0.00007 Time taken: 0:00:02.852885 ETA: 0:22:23.708726 INFO:tensorflow:global_step/sec: 1.83018 2021-09-25 23:11:11,569 [INFO] tensorflow: global_step/sec: 1.83018 INFO:tensorflow:global_step/sec: 1.84536 2021-09-25 23:11:12,111 [INFO] tensorflow: global_step/sec: 1.84536 INFO:tensorflow:global_step/sec: 1.82504 2021-09-25 23:11:12,659 [INFO] tensorflow: global_step/sec: 1.82504 INFO:tensorflow:global_step/sec: 1.68453 2021-09-25 23:11:13,252 [INFO] tensorflow: global_step/sec: 1.68453 2021-09-25 23:11:13,256 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 113.969 INFO:tensorflow:epoch = 30.0, learning_rate = 7.924468e-05, loss = 0.000140509, step = 150 (5.652 sec) 2021-09-25 23:11:13,815 [INFO] tensorflow: epoch = 30.0, learning_rate = 7.924468e-05, loss = 0.000140509, step = 150 (5.652 sec) INFO:tensorflow:global_step/sec: 1.77309 2021-09-25 23:11:13,816 [INFO] tensorflow: global_step/sec: 1.77309 2021-09-25 23:11:13,818 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 30/500: loss: 0.00014 learning rate: 0.00008 Time taken: 0:00:02.799314 ETA: 0:21:55.677590 INFO:tensorflow:global_step/sec: 1.80694 2021-09-25 23:11:14,370 [INFO] tensorflow: global_step/sec: 1.80694 INFO:tensorflow:global_step/sec: 1.76065 2021-09-25 23:11:14,938 [INFO] tensorflow: global_step/sec: 1.76065 INFO:tensorflow:global_step/sec: 1.78853 2021-09-25 23:11:15,497 [INFO] tensorflow: global_step/sec: 1.78853 INFO:tensorflow:global_step/sec: 1.79811 2021-09-25 23:11:16,053 [INFO] tensorflow: global_step/sec: 1.79811 INFO:tensorflow:global_step/sec: 1.81262 2021-09-25 23:11:16,605 [INFO] tensorflow: global_step/sec: 1.81262 2021-09-25 23:11:16,606 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 31/500: loss: 0.00011 learning rate: 0.00009 Time taken: 0:00:02.803754 ETA: 0:21:54.960445 INFO:tensorflow:global_step/sec: 1.77778 2021-09-25 23:11:17,167 [INFO] tensorflow: global_step/sec: 1.77778 INFO:tensorflow:global_step/sec: 1.781 2021-09-25 23:11:17,729 [INFO] tensorflow: global_step/sec: 1.781 INFO:tensorflow:global_step/sec: 1.80866 2021-09-25 23:11:18,282 [INFO] tensorflow: global_step/sec: 1.80866 INFO:tensorflow:global_step/sec: 1.79415 2021-09-25 23:11:18,839 [INFO] tensorflow: global_step/sec: 1.79415 INFO:tensorflow:epoch = 32.0, learning_rate = 9.527299e-05, loss = 0.0001376642, step = 160 (5.592 sec) 2021-09-25 23:11:19,407 [INFO] tensorflow: epoch = 32.0, learning_rate = 9.527299e-05, loss = 0.0001376642, step = 160 (5.592 sec) INFO:tensorflow:global_step/sec: 1.75651 2021-09-25 23:11:19,408 [INFO] tensorflow: global_step/sec: 1.75651 2021-09-25 23:11:19,410 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 32/500: loss: 0.00014 learning rate: 0.00010 Time taken: 0:00:02.782604 ETA: 0:21:42.258551 INFO:tensorflow:global_step/sec: 1.84748 2021-09-25 23:11:19,950 [INFO] tensorflow: global_step/sec: 1.84748 INFO:tensorflow:global_step/sec: 1.73598 2021-09-25 23:11:20,526 [INFO] tensorflow: global_step/sec: 1.73598 INFO:tensorflow:global_step/sec: 1.79679 2021-09-25 23:11:21,082 [INFO] tensorflow: global_step/sec: 1.79679 INFO:tensorflow:global_step/sec: 1.80854 2021-09-25 23:11:21,635 [INFO] tensorflow: global_step/sec: 1.80854 INFO:tensorflow:global_step/sec: 1.79809 2021-09-25 23:11:22,191 [INFO] tensorflow: global_step/sec: 1.79809 2021-09-25 23:11:22,193 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 33/500: loss: 0.00013 learning rate: 0.00010 Time taken: 0:00:02.797597 ETA: 0:21:46.477878 INFO:tensorflow:global_step/sec: 1.79411 2021-09-25 23:11:22,749 [INFO] tensorflow: global_step/sec: 1.79411 INFO:tensorflow:global_step/sec: 1.77567 2021-09-25 23:11:23,312 [INFO] tensorflow: global_step/sec: 1.77567 INFO:tensorflow:global_step/sec: 1.82523 2021-09-25 23:11:23,860 [INFO] tensorflow: global_step/sec: 1.82523 INFO:tensorflow:global_step/sec: 1.80567 2021-09-25 23:11:24,413 [INFO] tensorflow: global_step/sec: 1.80567 INFO:tensorflow:epoch = 34.0, learning_rate = 0.00011454334, loss = 0.00012562987, step = 170 (5.553 sec) 2021-09-25 23:11:24,960 [INFO] tensorflow: epoch = 34.0, learning_rate = 0.00011454334, loss = 0.00012562987, step = 170 (5.553 sec) INFO:tensorflow:global_step/sec: 1.82493 2021-09-25 23:11:24,961 [INFO] tensorflow: global_step/sec: 1.82493 2021-09-25 23:11:24,963 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 34/500: loss: 0.00013 learning rate: 0.00011 Time taken: 0:00:02.776813 ETA: 0:21:33.994872 INFO:tensorflow:global_step/sec: 1.75299 2021-09-25 23:11:25,532 [INFO] tensorflow: global_step/sec: 1.75299 INFO:tensorflow:global_step/sec: 1.82106 2021-09-25 23:11:26,081 [INFO] tensorflow: global_step/sec: 1.82106 INFO:tensorflow:global_step/sec: 1.76232 2021-09-25 23:11:26,648 [INFO] tensorflow: global_step/sec: 1.76232 INFO:tensorflow:global_step/sec: 1.71698 2021-09-25 23:11:27,231 [INFO] tensorflow: global_step/sec: 1.71698 2021-09-25 23:11:27,238 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 114.439 INFO:tensorflow:global_step/sec: 1.79711 2021-09-25 23:11:27,787 [INFO] tensorflow: global_step/sec: 1.79711 2021-09-25 23:11:27,789 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 35/500: loss: 0.00010 learning rate: 0.00013 Time taken: 0:00:02.817377 ETA: 0:21:50.080347 INFO:tensorflow:global_step/sec: 1.72985 2021-09-25 23:11:28,365 [INFO] tensorflow: global_step/sec: 1.72985 INFO:tensorflow:global_step/sec: 1.78077 2021-09-25 23:11:28,927 [INFO] tensorflow: global_step/sec: 1.78077 INFO:tensorflow:global_step/sec: 1.74724 2021-09-25 23:11:29,499 [INFO] tensorflow: global_step/sec: 1.74724 INFO:tensorflow:global_step/sec: 1.80742 2021-09-25 23:11:30,053 [INFO] tensorflow: global_step/sec: 1.80742 INFO:tensorflow:epoch = 36.0, learning_rate = 0.00013771141, loss = 0.00016013291, step = 180 (5.638 sec) 2021-09-25 23:11:30,598 [INFO] tensorflow: epoch = 36.0, learning_rate = 0.00013771141, loss = 0.00016013291, step = 180 (5.638 sec) INFO:tensorflow:global_step/sec: 1.82744 2021-09-25 23:11:30,600 [INFO] tensorflow: global_step/sec: 1.82744 2021-09-25 23:11:30,601 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 36/500: loss: 0.00016 learning rate: 0.00014 Time taken: 0:00:02.817540 ETA: 0:21:47.338528 INFO:tensorflow:global_step/sec: 1.78216 2021-09-25 23:11:31,161 [INFO] tensorflow: global_step/sec: 1.78216 INFO:tensorflow:global_step/sec: 1.77945 2021-09-25 23:11:31,723 [INFO] tensorflow: global_step/sec: 1.77945 INFO:tensorflow:global_step/sec: 1.74961 2021-09-25 23:11:32,294 [INFO] tensorflow: global_step/sec: 1.74961 INFO:tensorflow:global_step/sec: 1.81181 2021-09-25 23:11:32,846 [INFO] tensorflow: global_step/sec: 1.81181 INFO:tensorflow:global_step/sec: 1.74987 2021-09-25 23:11:33,418 [INFO] tensorflow: global_step/sec: 1.74987 2021-09-25 23:11:33,419 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 37/500: loss: 0.00012 learning rate: 0.00015 Time taken: 0:00:02.792537 ETA: 0:21:32.944619 INFO:tensorflow:global_step/sec: 1.71779 2021-09-25 23:11:34,000 [INFO] tensorflow: global_step/sec: 1.71779 INFO:tensorflow:global_step/sec: 1.78528 2021-09-25 23:11:34,560 [INFO] tensorflow: global_step/sec: 1.78528 INFO:tensorflow:global_step/sec: 1.76781 2021-09-25 23:11:35,126 [INFO] tensorflow: global_step/sec: 1.76781 INFO:tensorflow:global_step/sec: 1.78014 2021-09-25 23:11:35,688 [INFO] tensorflow: global_step/sec: 1.78014 INFO:tensorflow:epoch = 38.0, learning_rate = 0.00016556555, loss = 9.8479795e-05, step = 190 (5.656 sec) 2021-09-25 23:11:36,255 [INFO] tensorflow: epoch = 38.0, learning_rate = 0.00016556555, loss = 9.8479795e-05, step = 190 (5.656 sec) INFO:tensorflow:global_step/sec: 1.7589 2021-09-25 23:11:36,256 [INFO] tensorflow: global_step/sec: 1.7589 2021-09-25 23:11:36,257 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 38/500: loss: 0.00010 learning rate: 0.00017 Time taken: 0:00:02.841649 ETA: 0:21:52.841753 INFO:tensorflow:global_step/sec: 1.73088 2021-09-25 23:11:36,834 [INFO] tensorflow: global_step/sec: 1.73088 INFO:tensorflow:global_step/sec: 1.79764 2021-09-25 23:11:37,390 [INFO] tensorflow: global_step/sec: 1.79764 INFO:tensorflow:global_step/sec: 1.83146 2021-09-25 23:11:37,936 [INFO] tensorflow: global_step/sec: 1.83146 INFO:tensorflow:global_step/sec: 1.8123 2021-09-25 23:11:38,488 [INFO] tensorflow: global_step/sec: 1.8123 INFO:tensorflow:global_step/sec: 1.77956 2021-09-25 23:11:39,050 [INFO] tensorflow: global_step/sec: 1.77956 2021-09-25 23:11:39,051 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 39/500: loss: 0.00011 learning rate: 0.00018 Time taken: 0:00:02.802198 ETA: 0:21:31.813357 INFO:tensorflow:global_step/sec: 1.74894 2021-09-25 23:11:39,622 [INFO] tensorflow: global_step/sec: 1.74894 INFO:tensorflow:global_step/sec: 1.84482 2021-09-25 23:11:40,164 [INFO] tensorflow: global_step/sec: 1.84482 INFO:tensorflow:global_step/sec: 1.76007 2021-09-25 23:11:40,732 [INFO] tensorflow: global_step/sec: 1.76007 INFO:tensorflow:global_step/sec: 1.80755 2021-09-25 23:11:41,285 [INFO] tensorflow: global_step/sec: 1.80755 INFO:tensorflow:Saving checkpoints for step-200. 2021-09-25 23:11:41,286 [INFO] tensorflow: Saving checkpoints for step-200. 2021-09-25 23:11:45,336 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 88.413 INFO:tensorflow:epoch = 40.0, learning_rate = 0.0001990536, loss = 9.3442206e-05, step = 200 (9.648 sec) 2021-09-25 23:11:45,903 [INFO] tensorflow: epoch = 40.0, learning_rate = 0.0001990536, loss = 9.3442206e-05, step = 200 (9.648 sec) INFO:tensorflow:global_step/sec: 0.21648 2021-09-25 23:11:45,904 [INFO] tensorflow: global_step/sec: 0.21648 2021-09-25 23:11:45,906 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 40/500: loss: 0.00009 learning rate: 0.00020 Time taken: 0:00:06.840180 ETA: 0:52:26.482873 INFO:tensorflow:global_step/sec: 1.80744 2021-09-25 23:11:46,458 [INFO] tensorflow: global_step/sec: 1.80744 INFO:tensorflow:global_step/sec: 1.8256 2021-09-25 23:11:47,005 [INFO] tensorflow: global_step/sec: 1.8256 INFO:tensorflow:global_step/sec: 1.825 2021-09-25 23:11:47,553 [INFO] tensorflow: global_step/sec: 1.825 INFO:tensorflow:global_step/sec: 1.85369 2021-09-25 23:11:48,093 [INFO] tensorflow: global_step/sec: 1.85369 Epoch 41/500 ========================= Validation cost: 0.000000 Mean average_precision (in %): 0.0000 class name average precision (in %) ------------ -------------------------- lpd 0 Median Inference Time: nan INFO:tensorflow:global_step/sec: 1.78837 2021-09-25 23:11:48,652 [INFO] tensorflow: global_step/sec: 1.78837 2021-09-25 23:11:48,654 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 41/500: loss: 0.00014 learning rate: 0.00022 Time taken: 0:00:02.759999 ETA: 0:21:06.839339 INFO:tensorflow:global_step/sec: 1.81684 2021-09-25 23:11:49,202 [INFO] tensorflow: global_step/sec: 1.81684 INFO:tensorflow:global_step/sec: 1.74601 2021-09-25 23:11:49,775 [INFO] tensorflow: global_step/sec: 1.74601 INFO:tensorflow:global_step/sec: 1.77401 2021-09-25 23:11:50,339 [INFO] tensorflow: global_step/sec: 1.77401 INFO:tensorflow:global_step/sec: 1.85131 2021-09-25 23:11:50,879 [INFO] tensorflow: global_step/sec: 1.85131 INFO:tensorflow:global_step/sec: 1.72488 2021-09-25 23:11:51,459 [INFO] tensorflow: global_step/sec: 1.72488 2021-09-25 23:11:51,460 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 42/500: loss: 0.00008 learning rate: 0.00024 Time taken: 0:00:02.785980 ETA: 0:21:15.978834 INFO:tensorflow:epoch = 42.2, learning_rate = 0.0002437642, loss = 7.892033e-05, step = 211 (6.118 sec) 2021-09-25 23:11:52,021 [INFO] tensorflow: epoch = 42.2, learning_rate = 0.0002437642, loss = 7.892033e-05, step = 211 (6.118 sec) INFO:tensorflow:global_step/sec: 1.77388 2021-09-25 23:11:52,022 [INFO] tensorflow: global_step/sec: 1.77388 INFO:tensorflow:global_step/sec: 1.85142 2021-09-25 23:11:52,563 [INFO] tensorflow: global_step/sec: 1.85142 INFO:tensorflow:global_step/sec: 1.72895 2021-09-25 23:11:53,141 [INFO] tensorflow: global_step/sec: 1.72895 INFO:tensorflow:global_step/sec: 1.73037 2021-09-25 23:11:53,719 [INFO] tensorflow: global_step/sec: 1.73037 INFO:tensorflow:global_step/sec: 1.74441 2021-09-25 23:11:54,292 [INFO] tensorflow: global_step/sec: 1.74441 2021-09-25 23:11:54,294 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 43/500: loss: 0.00009 learning rate: 0.00026 Time taken: 0:00:02.836654 ETA: 0:21:36.350854 INFO:tensorflow:global_step/sec: 1.87081 2021-09-25 23:11:54,827 [INFO] tensorflow: global_step/sec: 1.87081 INFO:tensorflow:global_step/sec: 1.80412 2021-09-25 23:11:55,381 [INFO] tensorflow: global_step/sec: 1.80412 INFO:tensorflow:global_step/sec: 1.74487 2021-09-25 23:11:55,954 [INFO] tensorflow: global_step/sec: 1.74487 INFO:tensorflow:global_step/sec: 1.84395 2021-09-25 23:11:56,496 [INFO] tensorflow: global_step/sec: 1.84395 INFO:tensorflow:global_step/sec: 1.79058 2021-09-25 23:11:57,055 [INFO] tensorflow: global_step/sec: 1.79058 2021-09-25 23:11:57,056 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 44/500: loss: 0.00009 learning rate: 0.00029 Time taken: 0:00:02.777616 ETA: 0:21:06.592798 INFO:tensorflow:epoch = 44.2, learning_rate = 0.00029306905, loss = 9.461112e-05, step = 221 (5.616 sec) 2021-09-25 23:11:57,637 [INFO] tensorflow: epoch = 44.2, learning_rate = 0.00029306905, loss = 9.461112e-05, step = 221 (5.616 sec) INFO:tensorflow:global_step/sec: 1.71292 2021-09-25 23:11:57,639 [INFO] tensorflow: global_step/sec: 1.71292 INFO:tensorflow:global_step/sec: 1.85374 2021-09-25 23:11:58,178 [INFO] tensorflow: global_step/sec: 1.85374 INFO:tensorflow:global_step/sec: 1.83727 2021-09-25 23:11:58,722 [INFO] tensorflow: global_step/sec: 1.83727 INFO:tensorflow:global_step/sec: 1.70704 2021-09-25 23:11:59,308 [INFO] tensorflow: global_step/sec: 1.70704 2021-09-25 23:11:59,315 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 114.485 INFO:tensorflow:global_step/sec: 1.75888 2021-09-25 23:11:59,877 [INFO] tensorflow: global_step/sec: 1.75888 2021-09-25 23:11:59,878 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 45/500: loss: 0.00008 learning rate: 0.00032 Time taken: 0:00:02.812458 ETA: 0:21:19.668516 INFO:tensorflow:global_step/sec: 1.77588 2021-09-25 23:12:00,440 [INFO] tensorflow: global_step/sec: 1.77588 INFO:tensorflow:global_step/sec: 1.79878 2021-09-25 23:12:00,996 [INFO] tensorflow: global_step/sec: 1.79878 INFO:tensorflow:global_step/sec: 1.78353 2021-09-25 23:12:01,557 [INFO] tensorflow: global_step/sec: 1.78353 INFO:tensorflow:global_step/sec: 1.72836 2021-09-25 23:12:02,135 [INFO] tensorflow: global_step/sec: 1.72836 INFO:tensorflow:global_step/sec: 1.75111 2021-09-25 23:12:02,706 [INFO] tensorflow: global_step/sec: 1.75111 2021-09-25 23:12:02,708 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 46/500: loss: 0.00010 learning rate: 0.00035 Time taken: 0:00:02.823256 ETA: 0:21:21.758123 INFO:tensorflow:epoch = 46.2, learning_rate = 0.00035234657, loss = 9.585165e-05, step = 231 (5.627 sec) 2021-09-25 23:12:03,265 [INFO] tensorflow: epoch = 46.2, learning_rate = 0.00035234657, loss = 9.585165e-05, step = 231 (5.627 sec) INFO:tensorflow:global_step/sec: 1.78543 2021-09-25 23:12:03,266 [INFO] tensorflow: global_step/sec: 1.78543 INFO:tensorflow:global_step/sec: 1.80337 2021-09-25 23:12:03,821 [INFO] tensorflow: global_step/sec: 1.80337 INFO:tensorflow:global_step/sec: 1.76805 2021-09-25 23:12:04,386 [INFO] tensorflow: global_step/sec: 1.76805 INFO:tensorflow:global_step/sec: 1.71975 2021-09-25 23:12:04,968 [INFO] tensorflow: global_step/sec: 1.71975 INFO:tensorflow:global_step/sec: 1.82899 2021-09-25 23:12:05,515 [INFO] tensorflow: global_step/sec: 1.82899 2021-09-25 23:12:05,516 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 47/500: loss: 0.00008 learning rate: 0.00038 Time taken: 0:00:02.832244 ETA: 0:21:23.006387 INFO:tensorflow:global_step/sec: 1.77216 2021-09-25 23:12:06,079 [INFO] tensorflow: global_step/sec: 1.77216 INFO:tensorflow:global_step/sec: 1.77181 2021-09-25 23:12:06,643 [INFO] tensorflow: global_step/sec: 1.77181 INFO:tensorflow:global_step/sec: 1.80096 2021-09-25 23:12:07,199 [INFO] tensorflow: global_step/sec: 1.80096 INFO:tensorflow:global_step/sec: 1.75646 2021-09-25 23:12:07,768 [INFO] tensorflow: global_step/sec: 1.75646 INFO:tensorflow:global_step/sec: 1.72537 2021-09-25 23:12:08,347 [INFO] tensorflow: global_step/sec: 1.72537 2021-09-25 23:12:08,349 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 48/500: loss: 0.00008 learning rate: 0.00042 Time taken: 0:00:02.799880 ETA: 0:21:05.545665 INFO:tensorflow:epoch = 48.2, learning_rate = 0.0004236136, loss = 6.705224e-05, step = 241 (5.655 sec) 2021-09-25 23:12:08,920 [INFO] tensorflow: epoch = 48.2, learning_rate = 0.0004236136, loss = 6.705224e-05, step = 241 (5.655 sec) INFO:tensorflow:global_step/sec: 1.74204 2021-09-25 23:12:08,921 [INFO] tensorflow: global_step/sec: 1.74204 INFO:tensorflow:global_step/sec: 1.82182 2021-09-25 23:12:09,470 [INFO] tensorflow: global_step/sec: 1.82182 INFO:tensorflow:global_step/sec: 1.78369 2021-09-25 23:12:10,031 [INFO] tensorflow: global_step/sec: 1.78369 INFO:tensorflow:global_step/sec: 1.80964 2021-09-25 23:12:10,584 [INFO] tensorflow: global_step/sec: 1.80964 INFO:tensorflow:global_step/sec: 1.74955 2021-09-25 23:12:11,155 [INFO] tensorflow: global_step/sec: 1.74955 2021-09-25 23:12:11,157 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 49/500: loss: 0.00008 learning rate: 0.00046 Time taken: 0:00:02.814327 ETA: 0:21:09.261478 INFO:tensorflow:global_step/sec: 1.76056 2021-09-25 23:12:11,723 [INFO] tensorflow: global_step/sec: 1.76056 INFO:tensorflow:global_step/sec: 1.78691 2021-09-25 23:12:12,283 [INFO] tensorflow: global_step/sec: 1.78691 INFO:tensorflow:global_step/sec: 1.78578 2021-09-25 23:12:12,843 [INFO] tensorflow: global_step/sec: 1.78578 INFO:tensorflow:global_step/sec: 1.79871 2021-09-25 23:12:13,399 [INFO] tensorflow: global_step/sec: 1.79871 2021-09-25 23:12:13,402 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 113.590 INFO:tensorflow:global_step/sec: 1.76147 2021-09-25 23:12:13,966 [INFO] tensorflow: global_step/sec: 1.76147 2021-09-25 23:12:13,968 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 50/500: loss: 0.00007 learning rate: 0.00050 Time taken: 0:00:02.812961 ETA: 0:21:05.832281 INFO:tensorflow:epoch = 50.2, learning_rate = 0.00049999997, loss = 7.6497985e-05, step = 251 (5.590 sec) 2021-09-25 23:12:14,510 [INFO] tensorflow: epoch = 50.2, learning_rate = 0.00049999997, loss = 7.6497985e-05, step = 251 (5.590 sec) INFO:tensorflow:global_step/sec: 1.83619 2021-09-25 23:12:14,511 [INFO] tensorflow: global_step/sec: 1.83619 INFO:tensorflow:global_step/sec: 1.71137 2021-09-25 23:12:15,095 [INFO] tensorflow: global_step/sec: 1.71137 INFO:tensorflow:global_step/sec: 1.73078 2021-09-25 23:12:15,673 [INFO] tensorflow: global_step/sec: 1.73078 INFO:tensorflow:global_step/sec: 1.80598 2021-09-25 23:12:16,227 [INFO] tensorflow: global_step/sec: 1.80598 INFO:tensorflow:global_step/sec: 1.8005 2021-09-25 23:12:16,782 [INFO] tensorflow: global_step/sec: 1.8005 2021-09-25 23:12:16,784 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 51/500: loss: 0.00009 learning rate: 0.00050 Time taken: 0:00:02.827434 ETA: 0:21:09.518001 INFO:tensorflow:global_step/sec: 1.78679 2021-09-25 23:12:17,342 [INFO] tensorflow: global_step/sec: 1.78679 INFO:tensorflow:global_step/sec: 1.73174 2021-09-25 23:12:17,919 [INFO] tensorflow: global_step/sec: 1.73174 INFO:tensorflow:global_step/sec: 1.74788 2021-09-25 23:12:18,492 [INFO] tensorflow: global_step/sec: 1.74788 INFO:tensorflow:global_step/sec: 1.74848 2021-09-25 23:12:19,063 [INFO] tensorflow: global_step/sec: 1.74848 INFO:tensorflow:global_step/sec: 1.84137 2021-09-25 23:12:19,607 [INFO] tensorflow: global_step/sec: 1.84137 2021-09-25 23:12:19,608 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 52/500: loss: 0.00009 learning rate: 0.00050 Time taken: 0:00:02.839274 ETA: 0:21:11.994614 INFO:tensorflow:epoch = 52.2, learning_rate = 0.00049999997, loss = 7.954775e-05, step = 261 (5.660 sec) 2021-09-25 23:12:20,170 [INFO] tensorflow: epoch = 52.2, learning_rate = 0.00049999997, loss = 7.954775e-05, step = 261 (5.660 sec) INFO:tensorflow:global_step/sec: 1.77152 2021-09-25 23:12:20,171 [INFO] tensorflow: global_step/sec: 1.77152 INFO:tensorflow:global_step/sec: 1.78245 2021-09-25 23:12:20,732 [INFO] tensorflow: global_step/sec: 1.78245 INFO:tensorflow:global_step/sec: 1.86089 2021-09-25 23:12:21,269 [INFO] tensorflow: global_step/sec: 1.86089 INFO:tensorflow:global_step/sec: 1.75431 2021-09-25 23:12:21,839 [INFO] tensorflow: global_step/sec: 1.75431 INFO:tensorflow:global_step/sec: 1.76105 2021-09-25 23:12:22,407 [INFO] tensorflow: global_step/sec: 1.76105 2021-09-25 23:12:22,409 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 53/500: loss: 0.00007 learning rate: 0.00050 Time taken: 0:00:02.775716 ETA: 0:20:40.745082 INFO:tensorflow:global_step/sec: 1.7971 2021-09-25 23:12:22,964 [INFO] tensorflow: global_step/sec: 1.7971 INFO:tensorflow:global_step/sec: 1.78676 2021-09-25 23:12:23,523 [INFO] tensorflow: global_step/sec: 1.78676 INFO:tensorflow:global_step/sec: 1.78173 2021-09-25 23:12:24,085 [INFO] tensorflow: global_step/sec: 1.78173 INFO:tensorflow:global_step/sec: 1.85491 2021-09-25 23:12:24,624 [INFO] tensorflow: global_step/sec: 1.85491 INFO:tensorflow:global_step/sec: 1.79458 2021-09-25 23:12:25,181 [INFO] tensorflow: global_step/sec: 1.79458 2021-09-25 23:12:25,182 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 54/500: loss: 0.00008 learning rate: 0.00050 Time taken: 0:00:02.782166 ETA: 0:20:40.846251 INFO:tensorflow:epoch = 54.2, learning_rate = 0.00049999997, loss = 5.921806e-05, step = 271 (5.596 sec) 2021-09-25 23:12:25,766 [INFO] tensorflow: epoch = 54.2, learning_rate = 0.00049999997, loss = 5.921806e-05, step = 271 (5.596 sec) INFO:tensorflow:global_step/sec: 1.70529 2021-09-25 23:12:25,767 [INFO] tensorflow: global_step/sec: 1.70529 INFO:tensorflow:global_step/sec: 1.75104 2021-09-25 23:12:26,339 [INFO] tensorflow: global_step/sec: 1.75104 INFO:tensorflow:global_step/sec: 1.69054 2021-09-25 23:12:26,930 [INFO] tensorflow: global_step/sec: 1.69054 INFO:tensorflow:global_step/sec: 1.7526 2021-09-25 23:12:27,501 [INFO] tensorflow: global_step/sec: 1.7526 2021-09-25 23:12:27,507 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 113.446 INFO:tensorflow:global_step/sec: 1.79684 2021-09-25 23:12:28,057 [INFO] tensorflow: global_step/sec: 1.79684 2021-09-25 23:12:28,059 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 55/500: loss: 0.00006 learning rate: 0.00050 Time taken: 0:00:02.876195 ETA: 0:21:19.906754 INFO:tensorflow:global_step/sec: 1.78205 2021-09-25 23:12:28,618 [INFO] tensorflow: global_step/sec: 1.78205 INFO:tensorflow:global_step/sec: 1.73972 2021-09-25 23:12:29,193 [INFO] tensorflow: global_step/sec: 1.73972 INFO:tensorflow:global_step/sec: 1.8392 2021-09-25 23:12:29,737 [INFO] tensorflow: global_step/sec: 1.8392 INFO:tensorflow:global_step/sec: 1.76663 2021-09-25 23:12:30,303 [INFO] tensorflow: global_step/sec: 1.76663 INFO:tensorflow:global_step/sec: 1.73986 2021-09-25 23:12:30,878 [INFO] tensorflow: global_step/sec: 1.73986 2021-09-25 23:12:30,879 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 56/500: loss: 0.00006 learning rate: 0.00050 Time taken: 0:00:02.799806 ETA: 0:20:43.113705 INFO:tensorflow:epoch = 56.2, learning_rate = 0.00049999997, loss = 7.271419e-05, step = 281 (5.689 sec) 2021-09-25 23:12:31,455 [INFO] tensorflow: epoch = 56.2, learning_rate = 0.00049999997, loss = 7.271419e-05, step = 281 (5.689 sec) INFO:tensorflow:global_step/sec: 1.72798 2021-09-25 23:12:31,456 [INFO] tensorflow: global_step/sec: 1.72798 INFO:tensorflow:global_step/sec: 1.80271 2021-09-25 23:12:32,011 [INFO] tensorflow: global_step/sec: 1.80271 INFO:tensorflow:global_step/sec: 1.81596 2021-09-25 23:12:32,562 [INFO] tensorflow: global_step/sec: 1.81596 INFO:tensorflow:global_step/sec: 1.77187 2021-09-25 23:12:33,126 [INFO] tensorflow: global_step/sec: 1.77187 INFO:tensorflow:global_step/sec: 1.75952 2021-09-25 23:12:33,694 [INFO] tensorflow: global_step/sec: 1.75952 2021-09-25 23:12:33,696 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 57/500: loss: 0.00007 learning rate: 0.00050 Time taken: 0:00:02.822462 ETA: 0:20:50.350702 INFO:tensorflow:global_step/sec: 1.78796 2021-09-25 23:12:34,254 [INFO] tensorflow: global_step/sec: 1.78796 INFO:tensorflow:global_step/sec: 1.79659 2021-09-25 23:12:34,810 [INFO] tensorflow: global_step/sec: 1.79659 INFO:tensorflow:global_step/sec: 1.75883 2021-09-25 23:12:35,379 [INFO] tensorflow: global_step/sec: 1.75883 INFO:tensorflow:global_step/sec: 1.74364 2021-09-25 23:12:35,952 [INFO] tensorflow: global_step/sec: 1.74364 INFO:tensorflow:global_step/sec: 1.81309 2021-09-25 23:12:36,504 [INFO] tensorflow: global_step/sec: 1.81309 2021-09-25 23:12:36,505 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 58/500: loss: 0.00008 learning rate: 0.00050 Time taken: 0:00:02.825893 ETA: 0:20:49.044568 INFO:tensorflow:epoch = 58.2, learning_rate = 0.00049999997, loss = 7.8275916e-05, step = 291 (5.631 sec) 2021-09-25 23:12:37,086 [INFO] tensorflow: epoch = 58.2, learning_rate = 0.00049999997, loss = 7.8275916e-05, step = 291 (5.631 sec) INFO:tensorflow:global_step/sec: 1.71348 2021-09-25 23:12:37,088 [INFO] tensorflow: global_step/sec: 1.71348 INFO:tensorflow:global_step/sec: 1.72966 2021-09-25 23:12:37,666 [INFO] tensorflow: global_step/sec: 1.72966 INFO:tensorflow:global_step/sec: 1.78301 2021-09-25 23:12:38,227 [INFO] tensorflow: global_step/sec: 1.78301 INFO:tensorflow:global_step/sec: 1.82761 2021-09-25 23:12:38,774 [INFO] tensorflow: global_step/sec: 1.82761 INFO:tensorflow:global_step/sec: 1.73069 2021-09-25 23:12:39,352 [INFO] tensorflow: global_step/sec: 1.73069 2021-09-25 23:12:39,353 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 59/500: loss: 0.00006 learning rate: 0.00050 Time taken: 0:00:02.822210 ETA: 0:20:44.594432 INFO:tensorflow:global_step/sec: 1.8012 2021-09-25 23:12:39,907 [INFO] tensorflow: global_step/sec: 1.8012 INFO:tensorflow:global_step/sec: 1.72434 2021-09-25 23:12:40,487 [INFO] tensorflow: global_step/sec: 1.72434 INFO:tensorflow:global_step/sec: 1.76865 2021-09-25 23:12:41,052 [INFO] tensorflow: global_step/sec: 1.76865 INFO:tensorflow:global_step/sec: 1.78725 2021-09-25 23:12:41,612 [INFO] tensorflow: global_step/sec: 1.78725 INFO:tensorflow:Saving checkpoints for step-300. 2021-09-25 23:12:41,613 [INFO] tensorflow: Saving checkpoints for step-300. 2021-09-25 23:12:45,712 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 87.891 INFO:tensorflow:epoch = 60.0, learning_rate = 0.00049999997, loss = 6.976402e-05, step = 300 (9.187 sec) 2021-09-25 23:12:46,273 [INFO] tensorflow: epoch = 60.0, learning_rate = 0.00049999997, loss = 6.976402e-05, step = 300 (9.187 sec) INFO:tensorflow:global_step/sec: 0.214472 2021-09-25 23:12:46,274 [INFO] tensorflow: global_step/sec: 0.214472 2021-09-25 23:12:46,276 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 60/500: loss: 0.00007 learning rate: 0.00050 Time taken: 0:00:06.931045 ETA: 0:50:49.659929 INFO:tensorflow:global_step/sec: 1.80529 2021-09-25 23:12:46,828 [INFO] tensorflow: global_step/sec: 1.80529 INFO:tensorflow:global_step/sec: 1.71307 2021-09-25 23:12:47,412 [INFO] tensorflow: global_step/sec: 1.71307 INFO:tensorflow:global_step/sec: 1.81381 2021-09-25 23:12:47,963 [INFO] tensorflow: global_step/sec: 1.81381 INFO:tensorflow:global_step/sec: 1.7992 2021-09-25 23:12:48,519 [INFO] tensorflow: global_step/sec: 1.7992 Epoch 61/500 ========================= Validation cost: 0.000000 Mean average_precision (in %): 0.0000 class name average precision (in %) ------------ -------------------------- lpd 0 Median Inference Time: nan INFO:tensorflow:global_step/sec: 1.73693 2021-09-25 23:12:49,095 [INFO] tensorflow: global_step/sec: 1.73693 2021-09-25 23:12:49,096 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 61/500: loss: 0.00007 learning rate: 0.00050 Time taken: 0:00:02.812761 ETA: 0:20:34.802004 INFO:tensorflow:global_step/sec: 1.7467 2021-09-25 23:12:49,667 [INFO] tensorflow: global_step/sec: 1.7467 INFO:tensorflow:global_step/sec: 1.77883 2021-09-25 23:12:50,229 [INFO] tensorflow: global_step/sec: 1.77883 INFO:tensorflow:global_step/sec: 1.73623 2021-09-25 23:12:50,805 [INFO] tensorflow: global_step/sec: 1.73623 INFO:tensorflow:global_step/sec: 1.76032 2021-09-25 23:12:51,373 [INFO] tensorflow: global_step/sec: 1.76032 INFO:tensorflow:epoch = 62.0, learning_rate = 0.00049999997, loss = 5.8550475e-05, step = 310 (5.671 sec) 2021-09-25 23:12:51,944 [INFO] tensorflow: epoch = 62.0, learning_rate = 0.00049999997, loss = 5.8550475e-05, step = 310 (5.671 sec) INFO:tensorflow:global_step/sec: 1.74907 2021-09-25 23:12:51,945 [INFO] tensorflow: global_step/sec: 1.74907 2021-09-25 23:12:51,947 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 62/500: loss: 0.00006 learning rate: 0.00050 Time taken: 0:00:02.853058 ETA: 0:20:49.639552 INFO:tensorflow:global_step/sec: 1.78316 2021-09-25 23:12:52,506 [INFO] tensorflow: global_step/sec: 1.78316 INFO:tensorflow:global_step/sec: 1.81626 2021-09-25 23:12:53,057 [INFO] tensorflow: global_step/sec: 1.81626 INFO:tensorflow:global_step/sec: 1.82846 2021-09-25 23:12:53,603 [INFO] tensorflow: global_step/sec: 1.82846 INFO:tensorflow:global_step/sec: 1.79599 2021-09-25 23:12:54,160 [INFO] tensorflow: global_step/sec: 1.79599 INFO:tensorflow:global_step/sec: 1.75124 2021-09-25 23:12:54,731 [INFO] tensorflow: global_step/sec: 1.75124 2021-09-25 23:12:54,733 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 63/500: loss: 0.00006 learning rate: 0.00050 Time taken: 0:00:02.784973 ETA: 0:20:17.033264 INFO:tensorflow:global_step/sec: 1.80156 2021-09-25 23:12:55,286 [INFO] tensorflow: global_step/sec: 1.80156 INFO:tensorflow:global_step/sec: 1.77122 2021-09-25 23:12:55,851 [INFO] tensorflow: global_step/sec: 1.77122 INFO:tensorflow:global_step/sec: 1.79142 2021-09-25 23:12:56,409 [INFO] tensorflow: global_step/sec: 1.79142 INFO:tensorflow:global_step/sec: 1.78071 2021-09-25 23:12:56,971 [INFO] tensorflow: global_step/sec: 1.78071 INFO:tensorflow:epoch = 64.0, learning_rate = 0.00049999997, loss = 6.341858e-05, step = 320 (5.609 sec) 2021-09-25 23:12:57,552 [INFO] tensorflow: epoch = 64.0, learning_rate = 0.00049999997, loss = 6.341858e-05, step = 320 (5.609 sec) INFO:tensorflow:global_step/sec: 1.71462 2021-09-25 23:12:57,554 [INFO] tensorflow: global_step/sec: 1.71462 2021-09-25 23:12:57,555 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 64/500: loss: 0.00006 learning rate: 0.00050 Time taken: 0:00:02.811985 ETA: 0:20:26.025571 INFO:tensorflow:global_step/sec: 1.74442 2021-09-25 23:12:58,127 [INFO] tensorflow: global_step/sec: 1.74442 INFO:tensorflow:global_step/sec: 1.77163 2021-09-25 23:12:58,692 [INFO] tensorflow: global_step/sec: 1.77163 INFO:tensorflow:global_step/sec: 1.73372 2021-09-25 23:12:59,268 [INFO] tensorflow: global_step/sec: 1.73372 INFO:tensorflow:global_step/sec: 1.74678 2021-09-25 23:12:59,841 [INFO] tensorflow: global_step/sec: 1.74678 2021-09-25 23:12:59,846 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 113.211 INFO:tensorflow:global_step/sec: 1.73529 2021-09-25 23:13:00,417 [INFO] tensorflow: global_step/sec: 1.73529 2021-09-25 23:13:00,419 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 65/500: loss: 0.00006 learning rate: 0.00050 Time taken: 0:00:02.867301 ETA: 0:20:47.276033 INFO:tensorflow:global_step/sec: 1.8058 2021-09-25 23:13:00,971 [INFO] tensorflow: global_step/sec: 1.8058 INFO:tensorflow:global_step/sec: 1.844 2021-09-25 23:13:01,513 [INFO] tensorflow: global_step/sec: 1.844 INFO:tensorflow:global_step/sec: 1.83475 2021-09-25 23:13:02,058 [INFO] tensorflow: global_step/sec: 1.83475 INFO:tensorflow:global_step/sec: 1.77086 2021-09-25 23:13:02,623 [INFO] tensorflow: global_step/sec: 1.77086 INFO:tensorflow:epoch = 66.0, learning_rate = 0.00049999997, loss = 6.0171267e-05, step = 330 (5.642 sec) 2021-09-25 23:13:03,194 [INFO] tensorflow: epoch = 66.0, learning_rate = 0.00049999997, loss = 6.0171267e-05, step = 330 (5.642 sec) INFO:tensorflow:global_step/sec: 1.74563 2021-09-25 23:13:03,196 [INFO] tensorflow: global_step/sec: 1.74563 2021-09-25 23:13:03,197 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 66/500: loss: 0.00006 learning rate: 0.00050 Time taken: 0:00:02.782292 ETA: 0:20:07.514783 INFO:tensorflow:global_step/sec: 1.75827 2021-09-25 23:13:03,765 [INFO] tensorflow: global_step/sec: 1.75827 INFO:tensorflow:global_step/sec: 1.84049 2021-09-25 23:13:04,308 [INFO] tensorflow: global_step/sec: 1.84049 INFO:tensorflow:global_step/sec: 1.80757 2021-09-25 23:13:04,861 [INFO] tensorflow: global_step/sec: 1.80757 INFO:tensorflow:global_step/sec: 1.78645 2021-09-25 23:13:05,421 [INFO] tensorflow: global_step/sec: 1.78645 INFO:tensorflow:global_step/sec: 1.74477 2021-09-25 23:13:05,994 [INFO] tensorflow: global_step/sec: 1.74477 2021-09-25 23:13:05,995 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 67/500: loss: 0.00008 learning rate: 0.00050 Time taken: 0:00:02.798606 ETA: 0:20:11.796466 INFO:tensorflow:global_step/sec: 1.84631 2021-09-25 23:13:06,536 [INFO] tensorflow: global_step/sec: 1.84631 INFO:tensorflow:global_step/sec: 1.80131 2021-09-25 23:13:07,091 [INFO] tensorflow: global_step/sec: 1.80131 INFO:tensorflow:global_step/sec: 1.70468 2021-09-25 23:13:07,677 [INFO] tensorflow: global_step/sec: 1.70468 INFO:tensorflow:global_step/sec: 1.8231 2021-09-25 23:13:08,226 [INFO] tensorflow: global_step/sec: 1.8231 INFO:tensorflow:epoch = 68.0, learning_rate = 0.00049999997, loss = 5.2866802e-05, step = 340 (5.591 sec) 2021-09-25 23:13:08,786 [INFO] tensorflow: epoch = 68.0, learning_rate = 0.00049999997, loss = 5.2866802e-05, step = 340 (5.591 sec) INFO:tensorflow:global_step/sec: 1.78202 2021-09-25 23:13:08,787 [INFO] tensorflow: global_step/sec: 1.78202 2021-09-25 23:13:08,788 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 68/500: loss: 0.00005 learning rate: 0.00050 Time taken: 0:00:02.803793 ETA: 0:20:11.238762 INFO:tensorflow:global_step/sec: 1.76729 2021-09-25 23:13:09,353 [INFO] tensorflow: global_step/sec: 1.76729 INFO:tensorflow:global_step/sec: 1.83111 2021-09-25 23:13:09,899 [INFO] tensorflow: global_step/sec: 1.83111 INFO:tensorflow:global_step/sec: 1.77011 2021-09-25 23:13:10,464 [INFO] tensorflow: global_step/sec: 1.77011 INFO:tensorflow:global_step/sec: 1.77682 2021-09-25 23:13:11,027 [INFO] tensorflow: global_step/sec: 1.77682 INFO:tensorflow:global_step/sec: 1.69918 2021-09-25 23:13:11,615 [INFO] tensorflow: global_step/sec: 1.69918 2021-09-25 23:13:11,617 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 69/500: loss: 0.00005 learning rate: 0.00050 Time taken: 0:00:02.799521 ETA: 0:20:06.593641 INFO:tensorflow:global_step/sec: 1.79519 2021-09-25 23:13:12,172 [INFO] tensorflow: global_step/sec: 1.79519 INFO:tensorflow:global_step/sec: 1.79721 2021-09-25 23:13:12,729 [INFO] tensorflow: global_step/sec: 1.79721 INFO:tensorflow:global_step/sec: 1.83457 2021-09-25 23:13:13,274 [INFO] tensorflow: global_step/sec: 1.83457 INFO:tensorflow:global_step/sec: 1.79692 2021-09-25 23:13:13,830 [INFO] tensorflow: global_step/sec: 1.79692 2021-09-25 23:13:13,838 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 114.362 INFO:tensorflow:epoch = 70.0, learning_rate = 0.00049999997, loss = 5.285801e-05, step = 350 (5.621 sec) 2021-09-25 23:13:14,407 [INFO] tensorflow: epoch = 70.0, learning_rate = 0.00049999997, loss = 5.285801e-05, step = 350 (5.621 sec) INFO:tensorflow:global_step/sec: 1.73038 2021-09-25 23:13:14,408 [INFO] tensorflow: global_step/sec: 1.73038 2021-09-25 23:13:14,410 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 70/500: loss: 0.00005 learning rate: 0.00050 Time taken: 0:00:02.803405 ETA: 0:20:05.464170 INFO:tensorflow:global_step/sec: 1.78174 2021-09-25 23:13:14,970 [INFO] tensorflow: global_step/sec: 1.78174 INFO:tensorflow:global_step/sec: 1.84829 2021-09-25 23:13:15,511 [INFO] tensorflow: global_step/sec: 1.84829 INFO:tensorflow:global_step/sec: 1.80384 2021-09-25 23:13:16,065 [INFO] tensorflow: global_step/sec: 1.80384 INFO:tensorflow:global_step/sec: 1.71138 2021-09-25 23:13:16,649 [INFO] tensorflow: global_step/sec: 1.71138 INFO:tensorflow:global_step/sec: 1.7904 2021-09-25 23:13:17,208 [INFO] tensorflow: global_step/sec: 1.7904 2021-09-25 23:13:17,209 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 71/500: loss: 0.00005 learning rate: 0.00050 Time taken: 0:00:02.818079 ETA: 0:20:08.955684 INFO:tensorflow:global_step/sec: 1.69727 2021-09-25 23:13:17,797 [INFO] tensorflow: global_step/sec: 1.69727 INFO:tensorflow:global_step/sec: 1.74117 2021-09-25 23:13:18,371 [INFO] tensorflow: global_step/sec: 1.74117 INFO:tensorflow:global_step/sec: 1.79042 2021-09-25 23:13:18,930 [INFO] tensorflow: global_step/sec: 1.79042 INFO:tensorflow:global_step/sec: 1.74344 2021-09-25 23:13:19,503 [INFO] tensorflow: global_step/sec: 1.74344 INFO:tensorflow:epoch = 72.0, learning_rate = 0.00049999997, loss = 5.954635e-05, step = 360 (5.648 sec) 2021-09-25 23:13:20,055 [INFO] tensorflow: epoch = 72.0, learning_rate = 0.00049999997, loss = 5.954635e-05, step = 360 (5.648 sec) INFO:tensorflow:global_step/sec: 1.80845 2021-09-25 23:13:20,056 [INFO] tensorflow: global_step/sec: 1.80845 2021-09-25 23:13:20,058 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 72/500: loss: 0.00006 learning rate: 0.00050 Time taken: 0:00:02.851971 ETA: 0:20:20.643754 INFO:tensorflow:global_step/sec: 1.84931 2021-09-25 23:13:20,597 [INFO] tensorflow: global_step/sec: 1.84931 INFO:tensorflow:global_step/sec: 1.68663 2021-09-25 23:13:21,190 [INFO] tensorflow: global_step/sec: 1.68663 INFO:tensorflow:global_step/sec: 1.72971 2021-09-25 23:13:21,768 [INFO] tensorflow: global_step/sec: 1.72971 INFO:tensorflow:global_step/sec: 1.74043 2021-09-25 23:13:22,343 [INFO] tensorflow: global_step/sec: 1.74043 INFO:tensorflow:global_step/sec: 1.76108 2021-09-25 23:13:22,911 [INFO] tensorflow: global_step/sec: 1.76108 2021-09-25 23:13:22,912 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 73/500: loss: 0.00006 learning rate: 0.00050 Time taken: 0:00:02.836735 ETA: 0:20:11.285748 INFO:tensorflow:global_step/sec: 1.78887 2021-09-25 23:13:23,470 [INFO] tensorflow: global_step/sec: 1.78887 INFO:tensorflow:global_step/sec: 1.79162 2021-09-25 23:13:24,028 [INFO] tensorflow: global_step/sec: 1.79162 INFO:tensorflow:global_step/sec: 1.7682 2021-09-25 23:13:24,593 [INFO] tensorflow: global_step/sec: 1.7682 INFO:tensorflow:global_step/sec: 1.82379 2021-09-25 23:13:25,142 [INFO] tensorflow: global_step/sec: 1.82379 INFO:tensorflow:epoch = 74.0, learning_rate = 0.00049999997, loss = 4.5800694e-05, step = 370 (5.677 sec) 2021-09-25 23:13:25,732 [INFO] tensorflow: epoch = 74.0, learning_rate = 0.00049999997, loss = 4.5800694e-05, step = 370 (5.677 sec) INFO:tensorflow:global_step/sec: 1.68907 2021-09-25 23:13:25,734 [INFO] tensorflow: global_step/sec: 1.68907 2021-09-25 23:13:25,735 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 74/500: loss: 0.00005 learning rate: 0.00050 Time taken: 0:00:02.800864 ETA: 0:19:53.168056 INFO:tensorflow:global_step/sec: 1.73973 2021-09-25 23:13:26,308 [INFO] tensorflow: global_step/sec: 1.73973 INFO:tensorflow:global_step/sec: 1.82103 2021-09-25 23:13:26,858 [INFO] tensorflow: global_step/sec: 1.82103 INFO:tensorflow:global_step/sec: 1.85056 2021-09-25 23:13:27,398 [INFO] tensorflow: global_step/sec: 1.85056 INFO:tensorflow:global_step/sec: 1.74102 2021-09-25 23:13:27,972 [INFO] tensorflow: global_step/sec: 1.74102 2021-09-25 23:13:27,976 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 113.172 INFO:tensorflow:global_step/sec: 1.80257 2021-09-25 23:13:28,527 [INFO] tensorflow: global_step/sec: 1.80257 2021-09-25 23:13:28,529 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 75/500: loss: 0.00006 learning rate: 0.00050 Time taken: 0:00:02.827128 ETA: 0:20:01.529270 INFO:tensorflow:global_step/sec: 1.80838 2021-09-25 23:13:29,080 [INFO] tensorflow: global_step/sec: 1.80838 INFO:tensorflow:global_step/sec: 1.86298 2021-09-25 23:13:29,617 [INFO] tensorflow: global_step/sec: 1.86298 INFO:tensorflow:global_step/sec: 1.77242 2021-09-25 23:13:30,181 [INFO] tensorflow: global_step/sec: 1.77242 INFO:tensorflow:global_step/sec: 1.7697 2021-09-25 23:13:30,746 [INFO] tensorflow: global_step/sec: 1.7697 INFO:tensorflow:epoch = 76.0, learning_rate = 0.00049999997, loss = 5.4861535e-05, step = 380 (5.570 sec) 2021-09-25 23:13:31,302 [INFO] tensorflow: epoch = 76.0, learning_rate = 0.00049999997, loss = 5.4861535e-05, step = 380 (5.570 sec) INFO:tensorflow:global_step/sec: 1.79389 2021-09-25 23:13:31,304 [INFO] tensorflow: global_step/sec: 1.79389 2021-09-25 23:13:31,305 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 76/500: loss: 0.00005 learning rate: 0.00050 Time taken: 0:00:02.774533 ETA: 0:19:36.401804 INFO:tensorflow:global_step/sec: 1.82145 2021-09-25 23:13:31,853 [INFO] tensorflow: global_step/sec: 1.82145 INFO:tensorflow:global_step/sec: 1.77233 2021-09-25 23:13:32,417 [INFO] tensorflow: global_step/sec: 1.77233 INFO:tensorflow:global_step/sec: 1.78963 2021-09-25 23:13:32,976 [INFO] tensorflow: global_step/sec: 1.78963 INFO:tensorflow:global_step/sec: 1.83468 2021-09-25 23:13:33,521 [INFO] tensorflow: global_step/sec: 1.83468 INFO:tensorflow:global_step/sec: 1.8181 2021-09-25 23:13:34,071 [INFO] tensorflow: global_step/sec: 1.8181 2021-09-25 23:13:34,072 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 77/500: loss: 0.00005 learning rate: 0.00050 Time taken: 0:00:02.773291 ETA: 0:19:33.102241 INFO:tensorflow:global_step/sec: 1.8005 2021-09-25 23:13:34,626 [INFO] tensorflow: global_step/sec: 1.8005 INFO:tensorflow:global_step/sec: 1.79194 2021-09-25 23:13:35,184 [INFO] tensorflow: global_step/sec: 1.79194 INFO:tensorflow:global_step/sec: 1.79129 2021-09-25 23:13:35,742 [INFO] tensorflow: global_step/sec: 1.79129 INFO:tensorflow:global_step/sec: 1.75893 2021-09-25 23:13:36,311 [INFO] tensorflow: global_step/sec: 1.75893 INFO:tensorflow:epoch = 78.0, learning_rate = 0.00049999997, loss = 5.806661e-05, step = 390 (5.570 sec) 2021-09-25 23:13:36,873 [INFO] tensorflow: epoch = 78.0, learning_rate = 0.00049999997, loss = 5.806661e-05, step = 390 (5.570 sec) INFO:tensorflow:global_step/sec: 1.77602 2021-09-25 23:13:36,874 [INFO] tensorflow: global_step/sec: 1.77602 2021-09-25 23:13:36,875 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 78/500: loss: 0.00006 learning rate: 0.00050 Time taken: 0:00:02.788690 ETA: 0:19:36.827017 INFO:tensorflow:global_step/sec: 1.77061 2021-09-25 23:13:37,439 [INFO] tensorflow: global_step/sec: 1.77061 INFO:tensorflow:global_step/sec: 1.72723 2021-09-25 23:13:38,018 [INFO] tensorflow: global_step/sec: 1.72723 INFO:tensorflow:global_step/sec: 1.64508 2021-09-25 23:13:38,626 [INFO] tensorflow: global_step/sec: 1.64508 INFO:tensorflow:global_step/sec: 1.84474 2021-09-25 23:13:39,168 [INFO] tensorflow: global_step/sec: 1.84474 INFO:tensorflow:global_step/sec: 1.71969 2021-09-25 23:13:39,749 [INFO] tensorflow: global_step/sec: 1.71969 2021-09-25 23:13:39,751 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 79/500: loss: 0.00005 learning rate: 0.00050 Time taken: 0:00:02.857256 ETA: 0:20:02.904749 INFO:tensorflow:global_step/sec: 1.76963 2021-09-25 23:13:40,314 [INFO] tensorflow: global_step/sec: 1.76963 INFO:tensorflow:global_step/sec: 1.7691 2021-09-25 23:13:40,880 [INFO] tensorflow: global_step/sec: 1.7691 INFO:tensorflow:global_step/sec: 1.82769 2021-09-25 23:13:41,427 [INFO] tensorflow: global_step/sec: 1.82769 INFO:tensorflow:global_step/sec: 1.8468 2021-09-25 23:13:41,968 [INFO] tensorflow: global_step/sec: 1.8468 INFO:tensorflow:Saving checkpoints for step-400. 2021-09-25 23:13:41,969 [INFO] tensorflow: Saving checkpoints for step-400. 2021-09-25 23:13:45,970 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 88.924 INFO:tensorflow:epoch = 80.0, learning_rate = 0.00049999997, loss = 5.706604e-05, step = 400 (9.672 sec) 2021-09-25 23:13:46,544 [INFO] tensorflow: epoch = 80.0, learning_rate = 0.00049999997, loss = 5.706604e-05, step = 400 (9.672 sec) INFO:tensorflow:global_step/sec: 0.218453 2021-09-25 23:13:46,546 [INFO] tensorflow: global_step/sec: 0.218453 2021-09-25 23:13:46,547 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 80/500: loss: 0.00006 learning rate: 0.00050 Time taken: 0:00:06.796166 ETA: 0:47:34.389896 INFO:tensorflow:global_step/sec: 1.81036 2021-09-25 23:13:47,098 [INFO] tensorflow: global_step/sec: 1.81036 INFO:tensorflow:global_step/sec: 1.81698 2021-09-25 23:13:47,649 [INFO] tensorflow: global_step/sec: 1.81698 INFO:tensorflow:global_step/sec: 1.6646 2021-09-25 23:13:48,249 [INFO] tensorflow: global_step/sec: 1.6646 INFO:tensorflow:global_step/sec: 1.85252 2021-09-25 23:13:48,789 [INFO] tensorflow: global_step/sec: 1.85252 Epoch 81/500 ========================= Validation cost: 0.000000 Mean average_precision (in %): 0.0000 class name average precision (in %) ------------ -------------------------- lpd 0 Median Inference Time: nan INFO:tensorflow:global_step/sec: 1.81051 2021-09-25 23:13:49,341 [INFO] tensorflow: global_step/sec: 1.81051 2021-09-25 23:13:49,343 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 81/500: loss: 0.00005 learning rate: 0.00050 Time taken: 0:00:02.820121 ETA: 0:19:41.630520 INFO:tensorflow:global_step/sec: 1.80616 2021-09-25 23:13:49,895 [INFO] tensorflow: global_step/sec: 1.80616 INFO:tensorflow:global_step/sec: 1.70816 2021-09-25 23:13:50,480 [INFO] tensorflow: global_step/sec: 1.70816 INFO:tensorflow:global_step/sec: 1.80499 2021-09-25 23:13:51,034 [INFO] tensorflow: global_step/sec: 1.80499 INFO:tensorflow:global_step/sec: 1.78316 2021-09-25 23:13:51,595 [INFO] tensorflow: global_step/sec: 1.78316 INFO:tensorflow:epoch = 82.0, learning_rate = 0.00049999997, loss = 5.299235e-05, step = 410 (5.610 sec) 2021-09-25 23:13:52,155 [INFO] tensorflow: epoch = 82.0, learning_rate = 0.00049999997, loss = 5.299235e-05, step = 410 (5.610 sec) INFO:tensorflow:global_step/sec: 1.78359 2021-09-25 23:13:52,156 [INFO] tensorflow: global_step/sec: 1.78359 2021-09-25 23:13:52,157 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 82/500: loss: 0.00005 learning rate: 0.00050 Time taken: 0:00:02.805982 ETA: 0:19:32.900324 INFO:tensorflow:global_step/sec: 1.81906 2021-09-25 23:13:52,706 [INFO] tensorflow: global_step/sec: 1.81906 INFO:tensorflow:global_step/sec: 1.72961 2021-09-25 23:13:53,284 [INFO] tensorflow: global_step/sec: 1.72961 INFO:tensorflow:global_step/sec: 1.79865 2021-09-25 23:13:53,840 [INFO] tensorflow: global_step/sec: 1.79865 INFO:tensorflow:global_step/sec: 1.7945 2021-09-25 23:13:54,397 [INFO] tensorflow: global_step/sec: 1.7945 INFO:tensorflow:global_step/sec: 1.8425 2021-09-25 23:13:54,940 [INFO] tensorflow: global_step/sec: 1.8425 2021-09-25 23:13:54,941 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 83/500: loss: 0.00006 learning rate: 0.00050 Time taken: 0:00:02.799311 ETA: 0:19:27.312655 INFO:tensorflow:global_step/sec: 1.75566 2021-09-25 23:13:55,509 [INFO] tensorflow: global_step/sec: 1.75566 INFO:tensorflow:global_step/sec: 1.81729 2021-09-25 23:13:56,060 [INFO] tensorflow: global_step/sec: 1.81729 INFO:tensorflow:global_step/sec: 1.77784 2021-09-25 23:13:56,622 [INFO] tensorflow: global_step/sec: 1.77784 INFO:tensorflow:global_step/sec: 1.83947 2021-09-25 23:13:57,166 [INFO] tensorflow: global_step/sec: 1.83947 INFO:tensorflow:epoch = 84.0, learning_rate = 0.00049999997, loss = 5.2913863e-05, step = 420 (5.566 sec) 2021-09-25 23:13:57,720 [INFO] tensorflow: epoch = 84.0, learning_rate = 0.00049999997, loss = 5.2913863e-05, step = 420 (5.566 sec) INFO:tensorflow:global_step/sec: 1.79827 2021-09-25 23:13:57,722 [INFO] tensorflow: global_step/sec: 1.79827 2021-09-25 23:13:57,723 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 84/500: loss: 0.00005 learning rate: 0.00050 Time taken: 0:00:02.771119 ETA: 0:19:12.785355 INFO:tensorflow:global_step/sec: 1.74624 2021-09-25 23:13:58,295 [INFO] tensorflow: global_step/sec: 1.74624 INFO:tensorflow:global_step/sec: 1.8303 2021-09-25 23:13:58,841 [INFO] tensorflow: global_step/sec: 1.8303 INFO:tensorflow:global_step/sec: 1.84358 2021-09-25 23:13:59,383 [INFO] tensorflow: global_step/sec: 1.84358 INFO:tensorflow:global_step/sec: 1.70677 2021-09-25 23:13:59,969 [INFO] tensorflow: global_step/sec: 1.70677 2021-09-25 23:13:59,975 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 114.277 INFO:tensorflow:global_step/sec: 1.77944 2021-09-25 23:14:00,531 [INFO] tensorflow: global_step/sec: 1.77944 2021-09-25 23:14:00,533 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 85/500: loss: 0.00006 learning rate: 0.00050 Time taken: 0:00:02.801475 ETA: 0:19:22.612046 INFO:tensorflow:global_step/sec: 1.75205 2021-09-25 23:14:01,102 [INFO] tensorflow: global_step/sec: 1.75205 INFO:tensorflow:global_step/sec: 1.70874 2021-09-25 23:14:01,687 [INFO] tensorflow: global_step/sec: 1.70874 INFO:tensorflow:global_step/sec: 1.82102 2021-09-25 23:14:02,236 [INFO] tensorflow: global_step/sec: 1.82102 INFO:tensorflow:global_step/sec: 1.79213 2021-09-25 23:14:02,794 [INFO] tensorflow: global_step/sec: 1.79213 INFO:tensorflow:epoch = 86.0, learning_rate = 0.00049999997, loss = 5.042077e-05, step = 430 (5.615 sec) 2021-09-25 23:14:03,336 [INFO] tensorflow: epoch = 86.0, learning_rate = 0.00049999997, loss = 5.042077e-05, step = 430 (5.615 sec) INFO:tensorflow:global_step/sec: 1.84306 2021-09-25 23:14:03,337 [INFO] tensorflow: global_step/sec: 1.84306 2021-09-25 23:14:03,338 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 86/500: loss: 0.00005 learning rate: 0.00050 Time taken: 0:00:02.821111 ETA: 0:19:27.939840 INFO:tensorflow:global_step/sec: 1.74629 2021-09-25 23:14:03,910 [INFO] tensorflow: global_step/sec: 1.74629 INFO:tensorflow:global_step/sec: 1.82985 2021-09-25 23:14:04,456 [INFO] tensorflow: global_step/sec: 1.82985 INFO:tensorflow:global_step/sec: 1.70123 2021-09-25 23:14:05,044 [INFO] tensorflow: global_step/sec: 1.70123 INFO:tensorflow:global_step/sec: 1.77481 2021-09-25 23:14:05,607 [INFO] tensorflow: global_step/sec: 1.77481 INFO:tensorflow:global_step/sec: 1.78609 2021-09-25 23:14:06,167 [INFO] tensorflow: global_step/sec: 1.78609 2021-09-25 23:14:06,169 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 87/500: loss: 0.00005 learning rate: 0.00050 Time taken: 0:00:02.813016 ETA: 0:19:21.775681 INFO:tensorflow:global_step/sec: 1.80367 2021-09-25 23:14:06,722 [INFO] tensorflow: global_step/sec: 1.80367 INFO:tensorflow:global_step/sec: 1.81906 2021-09-25 23:14:07,271 [INFO] tensorflow: global_step/sec: 1.81906 INFO:tensorflow:global_step/sec: 1.78278 2021-09-25 23:14:07,832 [INFO] tensorflow: global_step/sec: 1.78278 INFO:tensorflow:global_step/sec: 1.77828 2021-09-25 23:14:08,395 [INFO] tensorflow: global_step/sec: 1.77828 INFO:tensorflow:epoch = 88.0, learning_rate = 0.00049999997, loss = 4.813468e-05, step = 440 (5.615 sec) 2021-09-25 23:14:08,950 [INFO] tensorflow: epoch = 88.0, learning_rate = 0.00049999997, loss = 4.813468e-05, step = 440 (5.615 sec) INFO:tensorflow:global_step/sec: 1.79557 2021-09-25 23:14:08,952 [INFO] tensorflow: global_step/sec: 1.79557 2021-09-25 23:14:08,953 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 88/500: loss: 0.00005 learning rate: 0.00050 Time taken: 0:00:02.789787 ETA: 0:19:09.392168 INFO:tensorflow:global_step/sec: 1.77943 2021-09-25 23:14:09,514 [INFO] tensorflow: global_step/sec: 1.77943 INFO:tensorflow:global_step/sec: 1.80803 2021-09-25 23:14:10,067 [INFO] tensorflow: global_step/sec: 1.80803 INFO:tensorflow:global_step/sec: 1.84556 2021-09-25 23:14:10,608 [INFO] tensorflow: global_step/sec: 1.84556 INFO:tensorflow:global_step/sec: 1.81003 2021-09-25 23:14:11,161 [INFO] tensorflow: global_step/sec: 1.81003 INFO:tensorflow:global_step/sec: 1.7836 2021-09-25 23:14:11,722 [INFO] tensorflow: global_step/sec: 1.7836 2021-09-25 23:14:11,723 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 89/500: loss: 0.00005 learning rate: 0.00050 Time taken: 0:00:02.763165 ETA: 0:18:55.660618 INFO:tensorflow:global_step/sec: 1.75736 2021-09-25 23:14:12,291 [INFO] tensorflow: global_step/sec: 1.75736 INFO:tensorflow:global_step/sec: 1.79902 2021-09-25 23:14:12,846 [INFO] tensorflow: global_step/sec: 1.79902 INFO:tensorflow:global_step/sec: 1.80631 2021-09-25 23:14:13,400 [INFO] tensorflow: global_step/sec: 1.80631 INFO:tensorflow:global_step/sec: 1.80051 2021-09-25 23:14:13,956 [INFO] tensorflow: global_step/sec: 1.80051 2021-09-25 23:14:13,961 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 114.406 INFO:tensorflow:epoch = 90.0, learning_rate = 0.00049999997, loss = 6.0779865e-05, step = 450 (5.573 sec) 2021-09-25 23:14:14,523 [INFO] tensorflow: epoch = 90.0, learning_rate = 0.00049999997, loss = 6.0779865e-05, step = 450 (5.573 sec) INFO:tensorflow:global_step/sec: 1.758 2021-09-25 23:14:14,524 [INFO] tensorflow: global_step/sec: 1.758 2021-09-25 23:14:14,526 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 90/500: loss: 0.00006 learning rate: 0.00050 Time taken: 0:00:02.795098 ETA: 0:19:05.990012 INFO:tensorflow:global_step/sec: 1.8239 2021-09-25 23:14:15,073 [INFO] tensorflow: global_step/sec: 1.8239 INFO:tensorflow:global_step/sec: 1.71452 2021-09-25 23:14:15,656 [INFO] tensorflow: global_step/sec: 1.71452 INFO:tensorflow:global_step/sec: 1.807 2021-09-25 23:14:16,209 [INFO] tensorflow: global_step/sec: 1.807 INFO:tensorflow:global_step/sec: 1.79751 2021-09-25 23:14:16,766 [INFO] tensorflow: global_step/sec: 1.79751 INFO:tensorflow:global_step/sec: 1.66599 2021-09-25 23:14:17,366 [INFO] tensorflow: global_step/sec: 1.66599 2021-09-25 23:14:17,367 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 91/500: loss: 0.00005 learning rate: 0.00050 Time taken: 0:00:02.807545 ETA: 0:19:08.285883 INFO:tensorflow:global_step/sec: 1.74775 2021-09-25 23:14:17,938 [INFO] tensorflow: global_step/sec: 1.74775 INFO:tensorflow:global_step/sec: 1.81009 2021-09-25 23:14:18,490 [INFO] tensorflow: global_step/sec: 1.81009 INFO:tensorflow:global_step/sec: 1.80093 2021-09-25 23:14:19,046 [INFO] tensorflow: global_step/sec: 1.80093 INFO:tensorflow:global_step/sec: 1.72507 2021-09-25 23:14:19,625 [INFO] tensorflow: global_step/sec: 1.72507 INFO:tensorflow:epoch = 92.0, learning_rate = 0.00049999997, loss = 5.0058974e-05, step = 460 (5.669 sec) 2021-09-25 23:14:20,192 [INFO] tensorflow: epoch = 92.0, learning_rate = 0.00049999997, loss = 5.0058974e-05, step = 460 (5.669 sec) INFO:tensorflow:global_step/sec: 1.76144 2021-09-25 23:14:20,193 [INFO] tensorflow: global_step/sec: 1.76144 2021-09-25 23:14:20,195 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 92/500: loss: 0.00005 learning rate: 0.00050 Time taken: 0:00:02.860562 ETA: 0:19:27.109331 INFO:tensorflow:global_step/sec: 1.77869 2021-09-25 23:14:20,755 [INFO] tensorflow: global_step/sec: 1.77869 INFO:tensorflow:global_step/sec: 1.80204 2021-09-25 23:14:21,310 [INFO] tensorflow: global_step/sec: 1.80204 INFO:tensorflow:global_step/sec: 1.83246 2021-09-25 23:14:21,856 [INFO] tensorflow: global_step/sec: 1.83246 INFO:tensorflow:global_step/sec: 1.73478 2021-09-25 23:14:22,432 [INFO] tensorflow: global_step/sec: 1.73478 INFO:tensorflow:global_step/sec: 1.7948 2021-09-25 23:14:22,990 [INFO] tensorflow: global_step/sec: 1.7948 2021-09-25 23:14:22,991 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 93/500: loss: 0.00005 learning rate: 0.00050 Time taken: 0:00:02.807614 ETA: 0:19:02.698837 INFO:tensorflow:global_step/sec: 1.80334 2021-09-25 23:14:23,544 [INFO] tensorflow: global_step/sec: 1.80334 INFO:tensorflow:global_step/sec: 1.80005 2021-09-25 23:14:24,100 [INFO] tensorflow: global_step/sec: 1.80005 INFO:tensorflow:global_step/sec: 1.76029 2021-09-25 23:14:24,668 [INFO] tensorflow: global_step/sec: 1.76029 INFO:tensorflow:global_step/sec: 1.78028 2021-09-25 23:14:25,229 [INFO] tensorflow: global_step/sec: 1.78028 INFO:tensorflow:epoch = 94.0, learning_rate = 0.00049999997, loss = 4.7001922e-05, step = 470 (5.609 sec) 2021-09-25 23:14:25,800 [INFO] tensorflow: epoch = 94.0, learning_rate = 0.00049999997, loss = 4.7001922e-05, step = 470 (5.609 sec) INFO:tensorflow:global_step/sec: 1.74747 2021-09-25 23:14:25,802 [INFO] tensorflow: global_step/sec: 1.74747 2021-09-25 23:14:25,803 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 94/500: loss: 0.00005 learning rate: 0.00050 Time taken: 0:00:02.796810 ETA: 0:18:55.504824 INFO:tensorflow:global_step/sec: 1.8022 2021-09-25 23:14:26,357 [INFO] tensorflow: global_step/sec: 1.8022 INFO:tensorflow:global_step/sec: 1.77751 2021-09-25 23:14:26,919 [INFO] tensorflow: global_step/sec: 1.77751 INFO:tensorflow:global_step/sec: 1.8359 2021-09-25 23:14:27,464 [INFO] tensorflow: global_step/sec: 1.8359 INFO:tensorflow:global_step/sec: 1.87178 2021-09-25 23:14:27,998 [INFO] tensorflow: global_step/sec: 1.87178 2021-09-25 23:14:28,005 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 113.937 INFO:tensorflow:global_step/sec: 1.76251 2021-09-25 23:14:28,565 [INFO] tensorflow: global_step/sec: 1.76251 2021-09-25 23:14:28,567 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 95/500: loss: 0.00005 learning rate: 0.00050 Time taken: 0:00:02.767404 ETA: 0:18:40.798749 INFO:tensorflow:global_step/sec: 1.82388 2021-09-25 23:14:29,114 [INFO] tensorflow: global_step/sec: 1.82388 INFO:tensorflow:global_step/sec: 1.75245 2021-09-25 23:14:29,684 [INFO] tensorflow: global_step/sec: 1.75245 INFO:tensorflow:global_step/sec: 1.76677 2021-09-25 23:14:30,250 [INFO] tensorflow: global_step/sec: 1.76677 INFO:tensorflow:global_step/sec: 1.7292 2021-09-25 23:14:30,829 [INFO] tensorflow: global_step/sec: 1.7292 INFO:tensorflow:epoch = 96.0, learning_rate = 0.00049999997, loss = 4.9544484e-05, step = 480 (5.597 sec) 2021-09-25 23:14:31,397 [INFO] tensorflow: epoch = 96.0, learning_rate = 0.00049999997, loss = 4.9544484e-05, step = 480 (5.597 sec) INFO:tensorflow:global_step/sec: 1.75399 2021-09-25 23:14:31,399 [INFO] tensorflow: global_step/sec: 1.75399 2021-09-25 23:14:31,400 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 96/500: loss: 0.00005 learning rate: 0.00050 Time taken: 0:00:02.829522 ETA: 0:19:03.126942 INFO:tensorflow:global_step/sec: 1.84883 2021-09-25 23:14:31,940 [INFO] tensorflow: global_step/sec: 1.84883 INFO:tensorflow:global_step/sec: 1.77586 2021-09-25 23:14:32,503 [INFO] tensorflow: global_step/sec: 1.77586 INFO:tensorflow:global_step/sec: 1.81742 2021-09-25 23:14:33,053 [INFO] tensorflow: global_step/sec: 1.81742 INFO:tensorflow:global_step/sec: 1.73637 2021-09-25 23:14:33,629 [INFO] tensorflow: global_step/sec: 1.73637 INFO:tensorflow:global_step/sec: 1.72935 2021-09-25 23:14:34,207 [INFO] tensorflow: global_step/sec: 1.72935 2021-09-25 23:14:34,209 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 97/500: loss: 0.00005 learning rate: 0.00050 Time taken: 0:00:02.799388 ETA: 0:18:48.153337 INFO:tensorflow:global_step/sec: 1.75594 2021-09-25 23:14:34,777 [INFO] tensorflow: global_step/sec: 1.75594 INFO:tensorflow:global_step/sec: 1.8064 2021-09-25 23:14:35,330 [INFO] tensorflow: global_step/sec: 1.8064 INFO:tensorflow:global_step/sec: 1.7455 2021-09-25 23:14:35,903 [INFO] tensorflow: global_step/sec: 1.7455 INFO:tensorflow:global_step/sec: 1.77453 2021-09-25 23:14:36,467 [INFO] tensorflow: global_step/sec: 1.77453 INFO:tensorflow:epoch = 98.0, learning_rate = 0.00049999997, loss = 4.56287e-05, step = 490 (5.614 sec) 2021-09-25 23:14:37,011 [INFO] tensorflow: epoch = 98.0, learning_rate = 0.00049999997, loss = 4.56287e-05, step = 490 (5.614 sec) INFO:tensorflow:global_step/sec: 1.83153 2021-09-25 23:14:37,013 [INFO] tensorflow: global_step/sec: 1.83153 2021-09-25 23:14:37,014 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 98/500: loss: 0.00005 learning rate: 0.00050 Time taken: 0:00:02.833063 ETA: 0:18:58.891472 INFO:tensorflow:global_step/sec: 1.75865 2021-09-25 23:14:37,581 [INFO] tensorflow: global_step/sec: 1.75865 INFO:tensorflow:global_step/sec: 1.81746 2021-09-25 23:14:38,132 [INFO] tensorflow: global_step/sec: 1.81746 INFO:tensorflow:global_step/sec: 1.80947 2021-09-25 23:14:38,684 [INFO] tensorflow: global_step/sec: 1.80947 INFO:tensorflow:global_step/sec: 1.75797 2021-09-25 23:14:39,253 [INFO] tensorflow: global_step/sec: 1.75797 INFO:tensorflow:global_step/sec: 1.66494 2021-09-25 23:14:39,854 [INFO] tensorflow: global_step/sec: 1.66494 2021-09-25 23:14:39,855 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 99/500: loss: 0.00005 learning rate: 0.00050 Time taken: 0:00:02.787236 ETA: 0:18:37.681817 INFO:tensorflow:global_step/sec: 1.68605 2021-09-25 23:14:40,447 [INFO] tensorflow: global_step/sec: 1.68605 INFO:tensorflow:global_step/sec: 1.82257 2021-09-25 23:14:40,995 [INFO] tensorflow: global_step/sec: 1.82257 INFO:tensorflow:global_step/sec: 1.82457 2021-09-25 23:14:41,544 [INFO] tensorflow: global_step/sec: 1.82457 INFO:tensorflow:global_step/sec: 1.76192 2021-09-25 23:14:42,111 [INFO] tensorflow: global_step/sec: 1.76192 INFO:tensorflow:Saving checkpoints for step-500. 2021-09-25 23:14:42,112 [INFO] tensorflow: Saving checkpoints for step-500. WARNING:tensorflow:Ignoring: /tmp/tmpv5zfadox; No such file or directory 2021-09-25 23:14:42,334 [WARNING] tensorflow: Ignoring: /tmp/tmpv5zfadox; No such file or directory 2021-09-25 23:14:46,223 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 87.828 INFO:tensorflow:epoch = 100.0, learning_rate = 0.00049999997, loss = 6.0369777e-05, step = 500 (9.761 sec) 2021-09-25 23:14:46,773 [INFO] tensorflow: epoch = 100.0, learning_rate = 0.00049999997, loss = 6.0369777e-05, step = 500 (9.761 sec) INFO:tensorflow:global_step/sec: 0.214456 2021-09-25 23:14:46,774 [INFO] tensorflow: global_step/sec: 0.214456 2021-09-25 23:14:46,775 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 100/500: loss: 0.00006 learning rate: 0.00050 Time taken: 0:00:06.965335 ETA: 0:46:26.134148 INFO:tensorflow:global_step/sec: 1.8161 2021-09-25 23:14:47,325 [INFO] tensorflow: global_step/sec: 1.8161 INFO:tensorflow:global_step/sec: 1.86689 2021-09-25 23:14:47,860 [INFO] tensorflow: global_step/sec: 1.86689 INFO:tensorflow:global_step/sec: 1.824 2021-09-25 23:14:48,409 [INFO] tensorflow: global_step/sec: 1.824 INFO:tensorflow:global_step/sec: 1.80316 2021-09-25 23:14:48,963 [INFO] tensorflow: global_step/sec: 1.80316 Epoch 101/500 ========================= Validation cost: 0.000000 Mean average_precision (in %): 0.0000 class name average precision (in %) ------------ -------------------------- lpd 0 Median Inference Time: nan INFO:tensorflow:global_step/sec: 1.78866 2021-09-25 23:14:49,522 [INFO] tensorflow: global_step/sec: 1.78866 2021-09-25 23:14:49,524 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 101/500: loss: 0.00005 learning rate: 0.00050 Time taken: 0:00:02.747023 ETA: 0:18:16.062314 INFO:tensorflow:global_step/sec: 1.76445 2021-09-25 23:14:50,089 [INFO] tensorflow: global_step/sec: 1.76445 INFO:tensorflow:global_step/sec: 1.74523 2021-09-25 23:14:50,662 [INFO] tensorflow: global_step/sec: 1.74523 INFO:tensorflow:global_step/sec: 1.76045 2021-09-25 23:14:51,230 [INFO] tensorflow: global_step/sec: 1.76045 INFO:tensorflow:global_step/sec: 1.82745 2021-09-25 23:14:51,777 [INFO] tensorflow: global_step/sec: 1.82745 INFO:tensorflow:epoch = 102.0, learning_rate = 0.00049999997, loss = 6.417774e-05, step = 510 (5.552 sec) 2021-09-25 23:14:52,324 [INFO] tensorflow: epoch = 102.0, learning_rate = 0.00049999997, loss = 6.417774e-05, step = 510 (5.552 sec) INFO:tensorflow:global_step/sec: 1.82401 2021-09-25 23:14:52,325 [INFO] tensorflow: global_step/sec: 1.82401 2021-09-25 23:14:52,327 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 102/500: loss: 0.00006 learning rate: 0.00050 Time taken: 0:00:02.810910 ETA: 0:18:38.742080 INFO:tensorflow:global_step/sec: 1.72583 2021-09-25 23:14:52,905 [INFO] tensorflow: global_step/sec: 1.72583 INFO:tensorflow:global_step/sec: 1.75298 2021-09-25 23:14:53,475 [INFO] tensorflow: global_step/sec: 1.75298 INFO:tensorflow:global_step/sec: 1.85148 2021-09-25 23:14:54,015 [INFO] tensorflow: global_step/sec: 1.85148 INFO:tensorflow:global_step/sec: 1.82235 2021-09-25 23:14:54,564 [INFO] tensorflow: global_step/sec: 1.82235 INFO:tensorflow:global_step/sec: 1.73575 2021-09-25 23:14:55,140 [INFO] tensorflow: global_step/sec: 1.73575 2021-09-25 23:14:55,142 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 103/500: loss: 0.00005 learning rate: 0.00050 Time taken: 0:00:02.786352 ETA: 0:18:26.181901 INFO:tensorflow:global_step/sec: 1.77374 2021-09-25 23:14:55,704 [INFO] tensorflow: global_step/sec: 1.77374 INFO:tensorflow:global_step/sec: 1.778 2021-09-25 23:14:56,267 [INFO] tensorflow: global_step/sec: 1.778 INFO:tensorflow:global_step/sec: 1.82346 2021-09-25 23:14:56,815 [INFO] tensorflow: global_step/sec: 1.82346 INFO:tensorflow:global_step/sec: 1.79902 2021-09-25 23:14:57,371 [INFO] tensorflow: global_step/sec: 1.79902 INFO:tensorflow:epoch = 104.0, learning_rate = 0.00049999997, loss = 5.251474e-05, step = 520 (5.612 sec) 2021-09-25 23:14:57,936 [INFO] tensorflow: epoch = 104.0, learning_rate = 0.00049999997, loss = 5.251474e-05, step = 520 (5.612 sec) INFO:tensorflow:global_step/sec: 1.76524 2021-09-25 23:14:57,937 [INFO] tensorflow: global_step/sec: 1.76524 2021-09-25 23:14:57,939 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 104/500: loss: 0.00005 learning rate: 0.00050 Time taken: 0:00:02.804096 ETA: 0:18:30.422104 INFO:tensorflow:global_step/sec: 1.74246 2021-09-25 23:14:58,511 [INFO] tensorflow: global_step/sec: 1.74246 INFO:tensorflow:global_step/sec: 1.80006 2021-09-25 23:14:59,067 [INFO] tensorflow: global_step/sec: 1.80006 INFO:tensorflow:global_step/sec: 1.76816 2021-09-25 23:14:59,632 [INFO] tensorflow: global_step/sec: 1.76816 INFO:tensorflow:global_step/sec: 1.78387 2021-09-25 23:15:00,193 [INFO] tensorflow: global_step/sec: 1.78387 2021-09-25 23:15:00,200 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 114.488 INFO:tensorflow:global_step/sec: 1.70424 2021-09-25 23:15:00,780 [INFO] tensorflow: global_step/sec: 1.70424 2021-09-25 23:15:00,781 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 105/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.824028 ETA: 0:18:35.490878 INFO:tensorflow:global_step/sec: 1.75866 2021-09-25 23:15:01,348 [INFO] tensorflow: global_step/sec: 1.75866 INFO:tensorflow:global_step/sec: 1.79123 2021-09-25 23:15:01,907 [INFO] tensorflow: global_step/sec: 1.79123 INFO:tensorflow:global_step/sec: 1.72927 2021-09-25 23:15:02,485 [INFO] tensorflow: global_step/sec: 1.72927 INFO:tensorflow:global_step/sec: 1.82337 2021-09-25 23:15:03,033 [INFO] tensorflow: global_step/sec: 1.82337 INFO:tensorflow:epoch = 106.0, learning_rate = 0.00049999997, loss = 5.4008822e-05, step = 530 (5.647 sec) 2021-09-25 23:15:03,583 [INFO] tensorflow: epoch = 106.0, learning_rate = 0.00049999997, loss = 5.4008822e-05, step = 530 (5.647 sec) INFO:tensorflow:global_step/sec: 1.81433 2021-09-25 23:15:03,584 [INFO] tensorflow: global_step/sec: 1.81433 2021-09-25 23:15:03,586 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 106/500: loss: 0.00005 learning rate: 0.00050 Time taken: 0:00:02.835688 ETA: 0:18:37.261117 INFO:tensorflow:global_step/sec: 1.772 2021-09-25 23:15:04,149 [INFO] tensorflow: global_step/sec: 1.772 INFO:tensorflow:global_step/sec: 1.77838 2021-09-25 23:15:04,711 [INFO] tensorflow: global_step/sec: 1.77838 INFO:tensorflow:global_step/sec: 1.75628 2021-09-25 23:15:05,280 [INFO] tensorflow: global_step/sec: 1.75628 INFO:tensorflow:global_step/sec: 1.83276 2021-09-25 23:15:05,826 [INFO] tensorflow: global_step/sec: 1.83276 INFO:tensorflow:global_step/sec: 1.79346 2021-09-25 23:15:06,384 [INFO] tensorflow: global_step/sec: 1.79346 2021-09-25 23:15:06,385 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 107/500: loss: 0.00005 learning rate: 0.00050 Time taken: 0:00:02.792725 ETA: 0:18:17.540959 INFO:tensorflow:global_step/sec: 1.81634 2021-09-25 23:15:06,934 [INFO] tensorflow: global_step/sec: 1.81634 INFO:tensorflow:global_step/sec: 1.8085 2021-09-25 23:15:07,487 [INFO] tensorflow: global_step/sec: 1.8085 INFO:tensorflow:global_step/sec: 1.78683 2021-09-25 23:15:08,047 [INFO] tensorflow: global_step/sec: 1.78683 INFO:tensorflow:global_step/sec: 1.77947 2021-09-25 23:15:08,609 [INFO] tensorflow: global_step/sec: 1.77947 INFO:tensorflow:epoch = 108.0, learning_rate = 0.00049999997, loss = 4.4143213e-05, step = 540 (5.592 sec) 2021-09-25 23:15:09,175 [INFO] tensorflow: epoch = 108.0, learning_rate = 0.00049999997, loss = 4.4143213e-05, step = 540 (5.592 sec) INFO:tensorflow:global_step/sec: 1.76291 2021-09-25 23:15:09,176 [INFO] tensorflow: global_step/sec: 1.76291 2021-09-25 23:15:09,177 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 108/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.783970 ETA: 0:18:11.316286 INFO:tensorflow:global_step/sec: 1.81937 2021-09-25 23:15:09,726 [INFO] tensorflow: global_step/sec: 1.81937 INFO:tensorflow:global_step/sec: 1.82786 2021-09-25 23:15:10,273 [INFO] tensorflow: global_step/sec: 1.82786 INFO:tensorflow:global_step/sec: 1.77628 2021-09-25 23:15:10,836 [INFO] tensorflow: global_step/sec: 1.77628 INFO:tensorflow:global_step/sec: 1.80726 2021-09-25 23:15:11,389 [INFO] tensorflow: global_step/sec: 1.80726 INFO:tensorflow:global_step/sec: 1.77301 2021-09-25 23:15:11,953 [INFO] tensorflow: global_step/sec: 1.77301 2021-09-25 23:15:11,955 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 109/500: loss: 0.00005 learning rate: 0.00050 Time taken: 0:00:02.780058 ETA: 0:18:07.002828 INFO:tensorflow:global_step/sec: 1.7857 2021-09-25 23:15:12,513 [INFO] tensorflow: global_step/sec: 1.7857 INFO:tensorflow:global_step/sec: 1.80926 2021-09-25 23:15:13,066 [INFO] tensorflow: global_step/sec: 1.80926 INFO:tensorflow:global_step/sec: 1.8125 2021-09-25 23:15:13,617 [INFO] tensorflow: global_step/sec: 1.8125 INFO:tensorflow:global_step/sec: 1.83411 2021-09-25 23:15:14,163 [INFO] tensorflow: global_step/sec: 1.83411 2021-09-25 23:15:14,170 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 114.540 INFO:tensorflow:global_step/sec: 1.73226 2021-09-25 23:15:14,740 [INFO] tensorflow: global_step/sec: 1.73226 2021-09-25 23:15:14,742 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 110/500: loss: 0.00005 learning rate: 0.00050 Time taken: 0:00:02.773428 ETA: 0:18:01.636906 INFO:tensorflow:epoch = 110.2, learning_rate = 0.00049999997, loss = 4.6333596e-05, step = 551 (6.109 sec) 2021-09-25 23:15:15,283 [INFO] tensorflow: epoch = 110.2, learning_rate = 0.00049999997, loss = 4.6333596e-05, step = 551 (6.109 sec) INFO:tensorflow:global_step/sec: 1.83595 2021-09-25 23:15:15,285 [INFO] tensorflow: global_step/sec: 1.83595 INFO:tensorflow:global_step/sec: 1.77948 2021-09-25 23:15:15,847 [INFO] tensorflow: global_step/sec: 1.77948 INFO:tensorflow:global_step/sec: 1.76101 2021-09-25 23:15:16,414 [INFO] tensorflow: global_step/sec: 1.76101 INFO:tensorflow:global_step/sec: 1.7267 2021-09-25 23:15:16,994 [INFO] tensorflow: global_step/sec: 1.7267 INFO:tensorflow:global_step/sec: 1.75127 2021-09-25 23:15:17,565 [INFO] tensorflow: global_step/sec: 1.75127 2021-09-25 23:15:17,566 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 111/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.828647 ETA: 0:18:20.343736 INFO:tensorflow:global_step/sec: 1.69666 2021-09-25 23:15:18,154 [INFO] tensorflow: global_step/sec: 1.69666 INFO:tensorflow:global_step/sec: 1.77078 2021-09-25 23:15:18,719 [INFO] tensorflow: global_step/sec: 1.77078 INFO:tensorflow:global_step/sec: 1.75312 2021-09-25 23:15:19,289 [INFO] tensorflow: global_step/sec: 1.75312 INFO:tensorflow:global_step/sec: 1.77502 2021-09-25 23:15:19,853 [INFO] tensorflow: global_step/sec: 1.77502 INFO:tensorflow:global_step/sec: 1.77019 2021-09-25 23:15:20,417 [INFO] tensorflow: global_step/sec: 1.77019 2021-09-25 23:15:20,419 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 112/500: loss: 0.00005 learning rate: 0.00050 Time taken: 0:00:02.856925 ETA: 0:18:28.487089 INFO:tensorflow:epoch = 112.2, learning_rate = 0.00049999997, loss = 4.7446374e-05, step = 561 (5.713 sec) 2021-09-25 23:15:20,996 [INFO] tensorflow: epoch = 112.2, learning_rate = 0.00049999997, loss = 4.7446374e-05, step = 561 (5.713 sec) INFO:tensorflow:global_step/sec: 1.72326 2021-09-25 23:15:20,998 [INFO] tensorflow: global_step/sec: 1.72326 INFO:tensorflow:global_step/sec: 1.78654 2021-09-25 23:15:21,557 [INFO] tensorflow: global_step/sec: 1.78654 INFO:tensorflow:global_step/sec: 1.78177 2021-09-25 23:15:22,119 [INFO] tensorflow: global_step/sec: 1.78177 INFO:tensorflow:global_step/sec: 1.71263 2021-09-25 23:15:22,703 [INFO] tensorflow: global_step/sec: 1.71263 INFO:tensorflow:global_step/sec: 1.69304 2021-09-25 23:15:23,293 [INFO] tensorflow: global_step/sec: 1.69304 2021-09-25 23:15:23,295 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 113/500: loss: 0.00005 learning rate: 0.00050 Time taken: 0:00:02.850344 ETA: 0:18:23.083014 INFO:tensorflow:global_step/sec: 1.80209 2021-09-25 23:15:23,848 [INFO] tensorflow: global_step/sec: 1.80209 INFO:tensorflow:global_step/sec: 1.76496 2021-09-25 23:15:24,415 [INFO] tensorflow: global_step/sec: 1.76496 INFO:tensorflow:global_step/sec: 1.83584 2021-09-25 23:15:24,959 [INFO] tensorflow: global_step/sec: 1.83584 INFO:tensorflow:global_step/sec: 1.78964 2021-09-25 23:15:25,518 [INFO] tensorflow: global_step/sec: 1.78964 INFO:tensorflow:global_step/sec: 1.75723 2021-09-25 23:15:26,087 [INFO] tensorflow: global_step/sec: 1.75723 2021-09-25 23:15:26,089 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 114/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.812464 ETA: 0:18:05.611196 INFO:tensorflow:epoch = 114.2, learning_rate = 0.00049999997, loss = 4.2345368e-05, step = 571 (5.642 sec) 2021-09-25 23:15:26,639 [INFO] tensorflow: epoch = 114.2, learning_rate = 0.00049999997, loss = 4.2345368e-05, step = 571 (5.642 sec) INFO:tensorflow:global_step/sec: 1.80925 2021-09-25 23:15:26,640 [INFO] tensorflow: global_step/sec: 1.80925 INFO:tensorflow:global_step/sec: 1.77964 2021-09-25 23:15:27,202 [INFO] tensorflow: global_step/sec: 1.77964 INFO:tensorflow:global_step/sec: 1.79085 2021-09-25 23:15:27,760 [INFO] tensorflow: global_step/sec: 1.79085 INFO:tensorflow:global_step/sec: 1.70512 2021-09-25 23:15:28,347 [INFO] tensorflow: global_step/sec: 1.70512 2021-09-25 23:15:28,350 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 112.838 INFO:tensorflow:global_step/sec: 1.82249 2021-09-25 23:15:28,896 [INFO] tensorflow: global_step/sec: 1.82249 2021-09-25 23:15:28,897 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 115/500: loss: 0.00006 learning rate: 0.00050 Time taken: 0:00:02.828257 ETA: 0:18:08.878794 INFO:tensorflow:global_step/sec: 1.74424 2021-09-25 23:15:29,469 [INFO] tensorflow: global_step/sec: 1.74424 INFO:tensorflow:global_step/sec: 1.76551 2021-09-25 23:15:30,035 [INFO] tensorflow: global_step/sec: 1.76551 INFO:tensorflow:global_step/sec: 1.76719 2021-09-25 23:15:30,601 [INFO] tensorflow: global_step/sec: 1.76719 INFO:tensorflow:global_step/sec: 1.77626 2021-09-25 23:15:31,164 [INFO] tensorflow: global_step/sec: 1.77626 INFO:tensorflow:global_step/sec: 1.79904 2021-09-25 23:15:31,720 [INFO] tensorflow: global_step/sec: 1.79904 2021-09-25 23:15:31,722 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 116/500: loss: 0.00005 learning rate: 0.00050 Time taken: 0:00:02.816168 ETA: 0:18:01.408447 INFO:tensorflow:epoch = 116.2, learning_rate = 0.00049999997, loss = 4.7749218e-05, step = 581 (5.643 sec) 2021-09-25 23:15:32,281 [INFO] tensorflow: epoch = 116.2, learning_rate = 0.00049999997, loss = 4.7749218e-05, step = 581 (5.643 sec) INFO:tensorflow:global_step/sec: 1.7768 2021-09-25 23:15:32,283 [INFO] tensorflow: global_step/sec: 1.7768 INFO:tensorflow:global_step/sec: 1.76585 2021-09-25 23:15:32,849 [INFO] tensorflow: global_step/sec: 1.76585 INFO:tensorflow:global_step/sec: 1.79944 2021-09-25 23:15:33,405 [INFO] tensorflow: global_step/sec: 1.79944 INFO:tensorflow:global_step/sec: 1.73669 2021-09-25 23:15:33,981 [INFO] tensorflow: global_step/sec: 1.73669 INFO:tensorflow:global_step/sec: 1.76474 2021-09-25 23:15:34,547 [INFO] tensorflow: global_step/sec: 1.76474 2021-09-25 23:15:34,549 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 117/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.816208 ETA: 0:17:58.607620 INFO:tensorflow:global_step/sec: 1.84799 2021-09-25 23:15:35,088 [INFO] tensorflow: global_step/sec: 1.84799 INFO:tensorflow:global_step/sec: 1.80806 2021-09-25 23:15:35,641 [INFO] tensorflow: global_step/sec: 1.80806 INFO:tensorflow:global_step/sec: 1.76013 2021-09-25 23:15:36,210 [INFO] tensorflow: global_step/sec: 1.76013 INFO:tensorflow:global_step/sec: 1.78789 2021-09-25 23:15:36,769 [INFO] tensorflow: global_step/sec: 1.78789 INFO:tensorflow:global_step/sec: 1.78729 2021-09-25 23:15:37,328 [INFO] tensorflow: global_step/sec: 1.78729 2021-09-25 23:15:37,330 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 118/500: loss: 0.00005 learning rate: 0.00050 Time taken: 0:00:02.787693 ETA: 0:17:44.898826 INFO:tensorflow:epoch = 118.2, learning_rate = 0.00049999997, loss = 4.2768694e-05, step = 591 (5.625 sec) 2021-09-25 23:15:37,906 [INFO] tensorflow: epoch = 118.2, learning_rate = 0.00049999997, loss = 4.2768694e-05, step = 591 (5.625 sec) INFO:tensorflow:global_step/sec: 1.72756 2021-09-25 23:15:37,907 [INFO] tensorflow: global_step/sec: 1.72756 INFO:tensorflow:global_step/sec: 1.8356 2021-09-25 23:15:38,452 [INFO] tensorflow: global_step/sec: 1.8356 INFO:tensorflow:global_step/sec: 1.8392 2021-09-25 23:15:38,996 [INFO] tensorflow: global_step/sec: 1.8392 INFO:tensorflow:global_step/sec: 1.805 2021-09-25 23:15:39,550 [INFO] tensorflow: global_step/sec: 1.805 INFO:tensorflow:global_step/sec: 1.77118 2021-09-25 23:15:40,114 [INFO] tensorflow: global_step/sec: 1.77118 2021-09-25 23:15:40,116 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 119/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.782173 ETA: 0:17:40.008064 INFO:tensorflow:global_step/sec: 1.81691 2021-09-25 23:15:40,665 [INFO] tensorflow: global_step/sec: 1.81691 INFO:tensorflow:global_step/sec: 1.71849 2021-09-25 23:15:41,247 [INFO] tensorflow: global_step/sec: 1.71849 INFO:tensorflow:global_step/sec: 1.72593 2021-09-25 23:15:41,826 [INFO] tensorflow: global_step/sec: 1.72593 INFO:tensorflow:global_step/sec: 1.8394 2021-09-25 23:15:42,370 [INFO] tensorflow: global_step/sec: 1.8394 INFO:tensorflow:Saving checkpoints for step-600. 2021-09-25 23:15:42,371 [INFO] tensorflow: Saving checkpoints for step-600. WARNING:tensorflow:Ignoring: /tmp/tmp_4kjpzt_; No such file or directory 2021-09-25 23:15:42,595 [WARNING] tensorflow: Ignoring: /tmp/tmp_4kjpzt_; No such file or directory 2021-09-25 23:15:46,457 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 88.372 INFO:tensorflow:epoch = 120.0, learning_rate = 0.00049999997, loss = 4.2346033e-05, step = 600 (9.137 sec) 2021-09-25 23:15:47,042 [INFO] tensorflow: epoch = 120.0, learning_rate = 0.00049999997, loss = 4.2346033e-05, step = 600 (9.137 sec) INFO:tensorflow:global_step/sec: 0.213941 2021-09-25 23:15:47,044 [INFO] tensorflow: global_step/sec: 0.213941 2021-09-25 23:15:47,045 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 120/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:06.900107 ETA: 0:43:42.040806 INFO:tensorflow:global_step/sec: 1.77691 2021-09-25 23:15:47,607 [INFO] tensorflow: global_step/sec: 1.77691 INFO:tensorflow:global_step/sec: 1.84835 2021-09-25 23:15:48,148 [INFO] tensorflow: global_step/sec: 1.84835 INFO:tensorflow:global_step/sec: 1.77386 2021-09-25 23:15:48,711 [INFO] tensorflow: global_step/sec: 1.77386 INFO:tensorflow:global_step/sec: 1.80317 2021-09-25 23:15:49,266 [INFO] tensorflow: global_step/sec: 1.80317 Epoch 121/500 ========================= Validation cost: 0.000000 Mean average_precision (in %): 0.0000 class name average precision (in %) ------------ -------------------------- lpd 0 Median Inference Time: nan INFO:tensorflow:global_step/sec: 1.80033 2021-09-25 23:15:49,821 [INFO] tensorflow: global_step/sec: 1.80033 2021-09-25 23:15:49,823 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 121/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.811321 ETA: 0:17:45.490576 INFO:tensorflow:global_step/sec: 1.80185 2021-09-25 23:15:50,376 [INFO] tensorflow: global_step/sec: 1.80185 INFO:tensorflow:global_step/sec: 1.75933 2021-09-25 23:15:50,945 [INFO] tensorflow: global_step/sec: 1.75933 INFO:tensorflow:global_step/sec: 1.82311 2021-09-25 23:15:51,493 [INFO] tensorflow: global_step/sec: 1.82311 INFO:tensorflow:global_step/sec: 1.73331 2021-09-25 23:15:52,070 [INFO] tensorflow: global_step/sec: 1.73331 INFO:tensorflow:epoch = 122.0, learning_rate = 0.00049999997, loss = 3.776655e-05, step = 610 (5.600 sec) 2021-09-25 23:15:52,642 [INFO] tensorflow: epoch = 122.0, learning_rate = 0.00049999997, loss = 3.776655e-05, step = 610 (5.600 sec) INFO:tensorflow:global_step/sec: 1.74457 2021-09-25 23:15:52,643 [INFO] tensorflow: global_step/sec: 1.74457 2021-09-25 23:15:52,645 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 122/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.801725 ETA: 0:17:39.052197 INFO:tensorflow:global_step/sec: 1.7137 2021-09-25 23:15:53,227 [INFO] tensorflow: global_step/sec: 1.7137 INFO:tensorflow:global_step/sec: 1.75143 2021-09-25 23:15:53,798 [INFO] tensorflow: global_step/sec: 1.75143 INFO:tensorflow:global_step/sec: 1.75092 2021-09-25 23:15:54,369 [INFO] tensorflow: global_step/sec: 1.75092 INFO:tensorflow:global_step/sec: 1.82148 2021-09-25 23:15:54,918 [INFO] tensorflow: global_step/sec: 1.82148 INFO:tensorflow:global_step/sec: 1.82272 2021-09-25 23:15:55,467 [INFO] tensorflow: global_step/sec: 1.82272 2021-09-25 23:15:55,468 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 123/500: loss: 0.00005 learning rate: 0.00050 Time taken: 0:00:02.846659 ETA: 0:17:53.190602 INFO:tensorflow:global_step/sec: 1.79042 2021-09-25 23:15:56,025 [INFO] tensorflow: global_step/sec: 1.79042 INFO:tensorflow:global_step/sec: 1.82748 2021-09-25 23:15:56,572 [INFO] tensorflow: global_step/sec: 1.82748 INFO:tensorflow:global_step/sec: 1.82668 2021-09-25 23:15:57,120 [INFO] tensorflow: global_step/sec: 1.82668 INFO:tensorflow:global_step/sec: 1.71026 2021-09-25 23:15:57,705 [INFO] tensorflow: global_step/sec: 1.71026 INFO:tensorflow:epoch = 124.0, learning_rate = 0.00049999997, loss = 4.8216352e-05, step = 620 (5.635 sec) 2021-09-25 23:15:58,277 [INFO] tensorflow: epoch = 124.0, learning_rate = 0.00049999997, loss = 4.8216352e-05, step = 620 (5.635 sec) INFO:tensorflow:global_step/sec: 1.74268 2021-09-25 23:15:58,278 [INFO] tensorflow: global_step/sec: 1.74268 2021-09-25 23:15:58,280 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 124/500: loss: 0.00005 learning rate: 0.00050 Time taken: 0:00:02.784998 ETA: 0:17:27.159225 INFO:tensorflow:global_step/sec: 1.84866 2021-09-25 23:15:58,819 [INFO] tensorflow: global_step/sec: 1.84866 INFO:tensorflow:global_step/sec: 1.81914 2021-09-25 23:15:59,369 [INFO] tensorflow: global_step/sec: 1.81914 INFO:tensorflow:global_step/sec: 1.78404 2021-09-25 23:15:59,930 [INFO] tensorflow: global_step/sec: 1.78404 INFO:tensorflow:global_step/sec: 1.75784 2021-09-25 23:16:00,499 [INFO] tensorflow: global_step/sec: 1.75784 2021-09-25 23:16:00,504 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 113.929 INFO:tensorflow:global_step/sec: 1.8095 2021-09-25 23:16:01,051 [INFO] tensorflow: global_step/sec: 1.8095 2021-09-25 23:16:01,053 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 125/500: loss: 0.00005 learning rate: 0.00050 Time taken: 0:00:02.796149 ETA: 0:17:28.555791 INFO:tensorflow:global_step/sec: 1.81474 2021-09-25 23:16:01,602 [INFO] tensorflow: global_step/sec: 1.81474 INFO:tensorflow:global_step/sec: 1.83846 2021-09-25 23:16:02,146 [INFO] tensorflow: global_step/sec: 1.83846 INFO:tensorflow:global_step/sec: 1.77482 2021-09-25 23:16:02,710 [INFO] tensorflow: global_step/sec: 1.77482 INFO:tensorflow:global_step/sec: 1.79164 2021-09-25 23:16:03,268 [INFO] tensorflow: global_step/sec: 1.79164 INFO:tensorflow:global_step/sec: 1.79583 2021-09-25 23:16:03,825 [INFO] tensorflow: global_step/sec: 1.79583 2021-09-25 23:16:03,826 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 126/500: loss: 0.00005 learning rate: 0.00050 Time taken: 0:00:02.769565 ETA: 0:17:15.817349 INFO:tensorflow:epoch = 126.2, learning_rate = 0.00049999997, loss = 4.5154455e-05, step = 631 (6.101 sec) 2021-09-25 23:16:04,378 [INFO] tensorflow: epoch = 126.2, learning_rate = 0.00049999997, loss = 4.5154455e-05, step = 631 (6.101 sec) INFO:tensorflow:global_step/sec: 1.80159 2021-09-25 23:16:04,380 [INFO] tensorflow: global_step/sec: 1.80159 INFO:tensorflow:global_step/sec: 1.75424 2021-09-25 23:16:04,950 [INFO] tensorflow: global_step/sec: 1.75424 INFO:tensorflow:global_step/sec: 1.85066 2021-09-25 23:16:05,490 [INFO] tensorflow: global_step/sec: 1.85066 INFO:tensorflow:global_step/sec: 1.75869 2021-09-25 23:16:06,059 [INFO] tensorflow: global_step/sec: 1.75869 INFO:tensorflow:global_step/sec: 1.8401 2021-09-25 23:16:06,602 [INFO] tensorflow: global_step/sec: 1.8401 2021-09-25 23:16:06,604 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 127/500: loss: 0.00005 learning rate: 0.00050 Time taken: 0:00:02.787884 ETA: 0:17:19.880642 INFO:tensorflow:global_step/sec: 1.65893 2021-09-25 23:16:07,205 [INFO] tensorflow: global_step/sec: 1.65893 INFO:tensorflow:global_step/sec: 1.75196 2021-09-25 23:16:07,776 [INFO] tensorflow: global_step/sec: 1.75196 INFO:tensorflow:global_step/sec: 1.7844 2021-09-25 23:16:08,336 [INFO] tensorflow: global_step/sec: 1.7844 INFO:tensorflow:global_step/sec: 1.76602 2021-09-25 23:16:08,902 [INFO] tensorflow: global_step/sec: 1.76602 INFO:tensorflow:global_step/sec: 1.79761 2021-09-25 23:16:09,459 [INFO] tensorflow: global_step/sec: 1.79761 2021-09-25 23:16:09,460 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 128/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.845261 ETA: 0:17:38.437128 INFO:tensorflow:epoch = 128.20000000000002, learning_rate = 0.00049999997, loss = 4.4067576e-05, step = 641 (5.639 sec) 2021-09-25 23:16:10,018 [INFO] tensorflow: epoch = 128.20000000000002, learning_rate = 0.00049999997, loss = 4.4067576e-05, step = 641 (5.639 sec) INFO:tensorflow:global_step/sec: 1.78444 2021-09-25 23:16:10,019 [INFO] tensorflow: global_step/sec: 1.78444 INFO:tensorflow:global_step/sec: 1.77951 2021-09-25 23:16:10,581 [INFO] tensorflow: global_step/sec: 1.77951 INFO:tensorflow:global_step/sec: 1.78562 2021-09-25 23:16:11,141 [INFO] tensorflow: global_step/sec: 1.78562 INFO:tensorflow:global_step/sec: 1.78935 2021-09-25 23:16:11,700 [INFO] tensorflow: global_step/sec: 1.78935 INFO:tensorflow:global_step/sec: 1.8199 2021-09-25 23:16:12,249 [INFO] tensorflow: global_step/sec: 1.8199 2021-09-25 23:16:12,251 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 129/500: loss: 0.00006 learning rate: 0.00050 Time taken: 0:00:02.793263 ETA: 0:17:16.300558 INFO:tensorflow:global_step/sec: 1.73097 2021-09-25 23:16:12,827 [INFO] tensorflow: global_step/sec: 1.73097 INFO:tensorflow:global_step/sec: 1.72413 2021-09-25 23:16:13,407 [INFO] tensorflow: global_step/sec: 1.72413 INFO:tensorflow:global_step/sec: 1.78456 2021-09-25 23:16:13,967 [INFO] tensorflow: global_step/sec: 1.78456 INFO:tensorflow:global_step/sec: 1.8195 2021-09-25 23:16:14,517 [INFO] tensorflow: global_step/sec: 1.8195 2021-09-25 23:16:14,522 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 114.150 INFO:tensorflow:global_step/sec: 1.82059 2021-09-25 23:16:15,066 [INFO] tensorflow: global_step/sec: 1.82059 2021-09-25 23:16:15,068 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 130/500: loss: 0.00005 learning rate: 0.00050 Time taken: 0:00:02.817843 ETA: 0:17:22.601895 INFO:tensorflow:epoch = 130.20000000000002, learning_rate = 0.00049999997, loss = 5.446093e-05, step = 651 (5.585 sec) 2021-09-25 23:16:15,603 [INFO] tensorflow: epoch = 130.20000000000002, learning_rate = 0.00049999997, loss = 5.446093e-05, step = 651 (5.585 sec) INFO:tensorflow:global_step/sec: 1.86014 2021-09-25 23:16:15,604 [INFO] tensorflow: global_step/sec: 1.86014 INFO:tensorflow:global_step/sec: 1.78812 2021-09-25 23:16:16,163 [INFO] tensorflow: global_step/sec: 1.78812 INFO:tensorflow:global_step/sec: 1.73175 2021-09-25 23:16:16,741 [INFO] tensorflow: global_step/sec: 1.73175 INFO:tensorflow:global_step/sec: 1.81332 2021-09-25 23:16:17,292 [INFO] tensorflow: global_step/sec: 1.81332 INFO:tensorflow:global_step/sec: 1.81551 2021-09-25 23:16:17,843 [INFO] tensorflow: global_step/sec: 1.81551 2021-09-25 23:16:17,844 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 131/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.773104 ETA: 0:17:03.275270 INFO:tensorflow:global_step/sec: 1.80098 2021-09-25 23:16:18,398 [INFO] tensorflow: global_step/sec: 1.80098 INFO:tensorflow:global_step/sec: 1.81034 2021-09-25 23:16:18,951 [INFO] tensorflow: global_step/sec: 1.81034 INFO:tensorflow:global_step/sec: 1.76543 2021-09-25 23:16:19,517 [INFO] tensorflow: global_step/sec: 1.76543 INFO:tensorflow:global_step/sec: 1.71814 2021-09-25 23:16:20,099 [INFO] tensorflow: global_step/sec: 1.71814 INFO:tensorflow:global_step/sec: 1.74228 2021-09-25 23:16:20,673 [INFO] tensorflow: global_step/sec: 1.74228 2021-09-25 23:16:20,674 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 132/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.808664 ETA: 0:17:13.588295 INFO:tensorflow:epoch = 132.20000000000002, learning_rate = 0.00049999997, loss = 4.575665e-05, step = 661 (5.646 sec) 2021-09-25 23:16:21,249 [INFO] tensorflow: epoch = 132.20000000000002, learning_rate = 0.00049999997, loss = 4.575665e-05, step = 661 (5.646 sec) INFO:tensorflow:global_step/sec: 1.73184 2021-09-25 23:16:21,250 [INFO] tensorflow: global_step/sec: 1.73184 INFO:tensorflow:global_step/sec: 1.80062 2021-09-25 23:16:21,806 [INFO] tensorflow: global_step/sec: 1.80062 INFO:tensorflow:global_step/sec: 1.682 2021-09-25 23:16:22,400 [INFO] tensorflow: global_step/sec: 1.682 INFO:tensorflow:global_step/sec: 1.84363 2021-09-25 23:16:22,943 [INFO] tensorflow: global_step/sec: 1.84363 INFO:tensorflow:global_step/sec: 1.79473 2021-09-25 23:16:23,500 [INFO] tensorflow: global_step/sec: 1.79473 2021-09-25 23:16:23,501 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 133/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.840073 ETA: 0:17:22.306743 INFO:tensorflow:global_step/sec: 1.78502 2021-09-25 23:16:24,060 [INFO] tensorflow: global_step/sec: 1.78502 INFO:tensorflow:global_step/sec: 1.81341 2021-09-25 23:16:24,611 [INFO] tensorflow: global_step/sec: 1.81341 INFO:tensorflow:global_step/sec: 1.72549 2021-09-25 23:16:25,191 [INFO] tensorflow: global_step/sec: 1.72549 INFO:tensorflow:global_step/sec: 1.73536 2021-09-25 23:16:25,767 [INFO] tensorflow: global_step/sec: 1.73536 INFO:tensorflow:global_step/sec: 1.75262 2021-09-25 23:16:26,338 [INFO] tensorflow: global_step/sec: 1.75262 2021-09-25 23:16:26,339 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 134/500: loss: 0.00005 learning rate: 0.00050 Time taken: 0:00:02.824462 ETA: 0:17:13.753069 INFO:tensorflow:epoch = 134.20000000000002, learning_rate = 0.00049999997, loss = 4.5628254e-05, step = 671 (5.655 sec) 2021-09-25 23:16:26,904 [INFO] tensorflow: epoch = 134.20000000000002, learning_rate = 0.00049999997, loss = 4.5628254e-05, step = 671 (5.655 sec) INFO:tensorflow:global_step/sec: 1.7634 2021-09-25 23:16:26,905 [INFO] tensorflow: global_step/sec: 1.7634 INFO:tensorflow:global_step/sec: 1.86818 2021-09-25 23:16:27,440 [INFO] tensorflow: global_step/sec: 1.86818 INFO:tensorflow:global_step/sec: 1.79667 2021-09-25 23:16:27,997 [INFO] tensorflow: global_step/sec: 1.79667 INFO:tensorflow:global_step/sec: 1.82545 2021-09-25 23:16:28,545 [INFO] tensorflow: global_step/sec: 1.82545 2021-09-25 23:16:28,552 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 114.049 INFO:tensorflow:global_step/sec: 1.74268 2021-09-25 23:16:29,118 [INFO] tensorflow: global_step/sec: 1.74268 2021-09-25 23:16:29,120 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 135/500: loss: 0.00005 learning rate: 0.00050 Time taken: 0:00:02.779943 ETA: 0:16:54.679365 INFO:tensorflow:global_step/sec: 1.69478 2021-09-25 23:16:29,709 [INFO] tensorflow: global_step/sec: 1.69478 INFO:tensorflow:global_step/sec: 1.75244 2021-09-25 23:16:30,279 [INFO] tensorflow: global_step/sec: 1.75244 INFO:tensorflow:global_step/sec: 1.81156 2021-09-25 23:16:30,831 [INFO] tensorflow: global_step/sec: 1.81156 INFO:tensorflow:global_step/sec: 1.80132 2021-09-25 23:16:31,386 [INFO] tensorflow: global_step/sec: 1.80132 INFO:tensorflow:global_step/sec: 1.78996 2021-09-25 23:16:31,945 [INFO] tensorflow: global_step/sec: 1.78996 2021-09-25 23:16:31,946 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 136/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.839682 ETA: 0:17:13.644198 INFO:tensorflow:epoch = 136.20000000000002, learning_rate = 0.00049999997, loss = 4.178118e-05, step = 681 (5.583 sec) 2021-09-25 23:16:32,486 [INFO] tensorflow: epoch = 136.20000000000002, learning_rate = 0.00049999997, loss = 4.178118e-05, step = 681 (5.583 sec) INFO:tensorflow:global_step/sec: 1.84244 2021-09-25 23:16:32,488 [INFO] tensorflow: global_step/sec: 1.84244 INFO:tensorflow:global_step/sec: 1.82427 2021-09-25 23:16:33,036 [INFO] tensorflow: global_step/sec: 1.82427 INFO:tensorflow:global_step/sec: 1.77351 2021-09-25 23:16:33,600 [INFO] tensorflow: global_step/sec: 1.77351 INFO:tensorflow:global_step/sec: 1.73569 2021-09-25 23:16:34,176 [INFO] tensorflow: global_step/sec: 1.73569 INFO:tensorflow:global_step/sec: 1.77308 2021-09-25 23:16:34,740 [INFO] tensorflow: global_step/sec: 1.77308 2021-09-25 23:16:34,741 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 137/500: loss: 0.00005 learning rate: 0.00050 Time taken: 0:00:02.785463 ETA: 0:16:51.123190 INFO:tensorflow:global_step/sec: 1.7724 2021-09-25 23:16:35,304 [INFO] tensorflow: global_step/sec: 1.7724 INFO:tensorflow:global_step/sec: 1.76307 2021-09-25 23:16:35,871 [INFO] tensorflow: global_step/sec: 1.76307 INFO:tensorflow:global_step/sec: 1.77336 2021-09-25 23:16:36,435 [INFO] tensorflow: global_step/sec: 1.77336 INFO:tensorflow:global_step/sec: 1.82787 2021-09-25 23:16:36,982 [INFO] tensorflow: global_step/sec: 1.82787 INFO:tensorflow:global_step/sec: 1.78575 2021-09-25 23:16:37,542 [INFO] tensorflow: global_step/sec: 1.78575 2021-09-25 23:16:37,544 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 138/500: loss: 0.00005 learning rate: 0.00050 Time taken: 0:00:02.808770 ETA: 0:16:56.774719 INFO:tensorflow:epoch = 138.20000000000002, learning_rate = 0.00049999997, loss = 5.956295e-05, step = 691 (5.619 sec) 2021-09-25 23:16:38,105 [INFO] tensorflow: epoch = 138.20000000000002, learning_rate = 0.00049999997, loss = 5.956295e-05, step = 691 (5.619 sec) INFO:tensorflow:global_step/sec: 1.77179 2021-09-25 23:16:38,107 [INFO] tensorflow: global_step/sec: 1.77179 INFO:tensorflow:global_step/sec: 1.7379 2021-09-25 23:16:38,682 [INFO] tensorflow: global_step/sec: 1.7379 INFO:tensorflow:global_step/sec: 1.81314 2021-09-25 23:16:39,234 [INFO] tensorflow: global_step/sec: 1.81314 INFO:tensorflow:global_step/sec: 1.77251 2021-09-25 23:16:39,798 [INFO] tensorflow: global_step/sec: 1.77251 INFO:tensorflow:global_step/sec: 1.74717 2021-09-25 23:16:40,370 [INFO] tensorflow: global_step/sec: 1.74717 2021-09-25 23:16:40,372 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 139/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.813033 ETA: 0:16:55.504778 INFO:tensorflow:global_step/sec: 1.77711 2021-09-25 23:16:40,933 [INFO] tensorflow: global_step/sec: 1.77711 INFO:tensorflow:global_step/sec: 1.83743 2021-09-25 23:16:41,477 [INFO] tensorflow: global_step/sec: 1.83743 INFO:tensorflow:global_step/sec: 1.76486 2021-09-25 23:16:42,044 [INFO] tensorflow: global_step/sec: 1.76486 INFO:tensorflow:global_step/sec: 1.7459 2021-09-25 23:16:42,616 [INFO] tensorflow: global_step/sec: 1.7459 INFO:tensorflow:Saving checkpoints for step-700. 2021-09-25 23:16:42,617 [INFO] tensorflow: Saving checkpoints for step-700. WARNING:tensorflow:Ignoring: /tmp/tmphdvvyefp; No such file or directory 2021-09-25 23:16:42,838 [WARNING] tensorflow: Ignoring: /tmp/tmphdvvyefp; No such file or directory 2021-09-25 23:16:46,738 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 87.988 INFO:tensorflow:epoch = 140.0, learning_rate = 0.00049999997, loss = 4.3598728e-05, step = 700 (9.206 sec) 2021-09-25 23:16:47,311 [INFO] tensorflow: epoch = 140.0, learning_rate = 0.00049999997, loss = 4.3598728e-05, step = 700 (9.206 sec) INFO:tensorflow:global_step/sec: 0.212949 2021-09-25 23:16:47,312 [INFO] tensorflow: global_step/sec: 0.212949 2021-09-25 23:16:47,314 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 140/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:06.934322 ETA: 0:41:36.355791 INFO:tensorflow:global_step/sec: 1.82246 2021-09-25 23:16:47,861 [INFO] tensorflow: global_step/sec: 1.82246 INFO:tensorflow:global_step/sec: 1.75306 2021-09-25 23:16:48,432 [INFO] tensorflow: global_step/sec: 1.75306 INFO:tensorflow:global_step/sec: 1.75468 2021-09-25 23:16:49,001 [INFO] tensorflow: global_step/sec: 1.75468 INFO:tensorflow:global_step/sec: 1.85197 2021-09-25 23:16:49,541 [INFO] tensorflow: global_step/sec: 1.85197 Epoch 141/500 ========================= Validation cost: 0.000000 Mean average_precision (in %): 0.0000 class name average precision (in %) ------------ -------------------------- lpd 0 Median Inference Time: nan INFO:tensorflow:global_step/sec: 1.73734 2021-09-25 23:16:50,117 [INFO] tensorflow: global_step/sec: 1.73734 2021-09-25 23:16:50,118 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 141/500: loss: 0.00005 learning rate: 0.00050 Time taken: 0:00:02.807356 ETA: 0:16:47.840676 INFO:tensorflow:global_step/sec: 1.84268 2021-09-25 23:16:50,660 [INFO] tensorflow: global_step/sec: 1.84268 INFO:tensorflow:global_step/sec: 1.79814 2021-09-25 23:16:51,216 [INFO] tensorflow: global_step/sec: 1.79814 INFO:tensorflow:global_step/sec: 1.71606 2021-09-25 23:16:51,799 [INFO] tensorflow: global_step/sec: 1.71606 INFO:tensorflow:global_step/sec: 1.75462 2021-09-25 23:16:52,368 [INFO] tensorflow: global_step/sec: 1.75462 INFO:tensorflow:epoch = 142.0, learning_rate = 0.00049999997, loss = 3.7249494e-05, step = 710 (5.617 sec) 2021-09-25 23:16:52,928 [INFO] tensorflow: epoch = 142.0, learning_rate = 0.00049999997, loss = 3.7249494e-05, step = 710 (5.617 sec) INFO:tensorflow:global_step/sec: 1.78202 2021-09-25 23:16:52,930 [INFO] tensorflow: global_step/sec: 1.78202 2021-09-25 23:16:52,931 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 142/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.826251 ETA: 0:16:51.797698 INFO:tensorflow:global_step/sec: 1.74333 2021-09-25 23:16:53,503 [INFO] tensorflow: global_step/sec: 1.74333 INFO:tensorflow:global_step/sec: 1.76045 2021-09-25 23:16:54,071 [INFO] tensorflow: global_step/sec: 1.76045 INFO:tensorflow:global_step/sec: 1.84599 2021-09-25 23:16:54,613 [INFO] tensorflow: global_step/sec: 1.84599 INFO:tensorflow:global_step/sec: 1.78545 2021-09-25 23:16:55,173 [INFO] tensorflow: global_step/sec: 1.78545 INFO:tensorflow:global_step/sec: 1.76343 2021-09-25 23:16:55,740 [INFO] tensorflow: global_step/sec: 1.76343 2021-09-25 23:16:55,742 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 143/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.803848 ETA: 0:16:40.973576 INFO:tensorflow:global_step/sec: 1.77187 2021-09-25 23:16:56,305 [INFO] tensorflow: global_step/sec: 1.77187 INFO:tensorflow:global_step/sec: 1.73048 2021-09-25 23:16:56,882 [INFO] tensorflow: global_step/sec: 1.73048 INFO:tensorflow:global_step/sec: 1.77844 2021-09-25 23:16:57,445 [INFO] tensorflow: global_step/sec: 1.77844 INFO:tensorflow:global_step/sec: 1.87408 2021-09-25 23:16:57,978 [INFO] tensorflow: global_step/sec: 1.87408 INFO:tensorflow:epoch = 144.0, learning_rate = 0.00049999997, loss = 4.4160293e-05, step = 720 (5.609 sec) 2021-09-25 23:16:58,537 [INFO] tensorflow: epoch = 144.0, learning_rate = 0.00049999997, loss = 4.4160293e-05, step = 720 (5.609 sec) INFO:tensorflow:global_step/sec: 1.78597 2021-09-25 23:16:58,538 [INFO] tensorflow: global_step/sec: 1.78597 2021-09-25 23:16:58,540 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 144/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.805395 ETA: 0:16:38.720750 INFO:tensorflow:global_step/sec: 1.77944 2021-09-25 23:16:59,100 [INFO] tensorflow: global_step/sec: 1.77944 INFO:tensorflow:global_step/sec: 1.7381 2021-09-25 23:16:59,676 [INFO] tensorflow: global_step/sec: 1.7381 INFO:tensorflow:global_step/sec: 1.71306 2021-09-25 23:17:00,259 [INFO] tensorflow: global_step/sec: 1.71306 INFO:tensorflow:global_step/sec: 1.83343 2021-09-25 23:17:00,805 [INFO] tensorflow: global_step/sec: 1.83343 2021-09-25 23:17:00,811 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 113.698 INFO:tensorflow:global_step/sec: 1.76249 2021-09-25 23:17:01,372 [INFO] tensorflow: global_step/sec: 1.76249 2021-09-25 23:17:01,374 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 145/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.826444 ETA: 0:16:43.387673 INFO:tensorflow:global_step/sec: 1.79073 2021-09-25 23:17:01,931 [INFO] tensorflow: global_step/sec: 1.79073 INFO:tensorflow:global_step/sec: 1.74918 2021-09-25 23:17:02,502 [INFO] tensorflow: global_step/sec: 1.74918 INFO:tensorflow:global_step/sec: 1.71638 2021-09-25 23:17:03,085 [INFO] tensorflow: global_step/sec: 1.71638 INFO:tensorflow:global_step/sec: 1.79807 2021-09-25 23:17:03,641 [INFO] tensorflow: global_step/sec: 1.79807 INFO:tensorflow:epoch = 146.0, learning_rate = 0.00049999997, loss = 6.0065668e-05, step = 730 (5.677 sec) 2021-09-25 23:17:04,214 [INFO] tensorflow: epoch = 146.0, learning_rate = 0.00049999997, loss = 6.0065668e-05, step = 730 (5.677 sec) INFO:tensorflow:global_step/sec: 1.74227 2021-09-25 23:17:04,215 [INFO] tensorflow: global_step/sec: 1.74227 2021-09-25 23:17:04,216 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 146/500: loss: 0.00006 learning rate: 0.00050 Time taken: 0:00:02.836272 ETA: 0:16:44.040373 INFO:tensorflow:global_step/sec: 1.74288 2021-09-25 23:17:04,789 [INFO] tensorflow: global_step/sec: 1.74288 INFO:tensorflow:global_step/sec: 1.78727 2021-09-25 23:17:05,348 [INFO] tensorflow: global_step/sec: 1.78727 INFO:tensorflow:global_step/sec: 1.72198 2021-09-25 23:17:05,929 [INFO] tensorflow: global_step/sec: 1.72198 INFO:tensorflow:global_step/sec: 1.83929 2021-09-25 23:17:06,473 [INFO] tensorflow: global_step/sec: 1.83929 INFO:tensorflow:global_step/sec: 1.75411 2021-09-25 23:17:07,043 [INFO] tensorflow: global_step/sec: 1.75411 2021-09-25 23:17:07,044 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 147/500: loss: 0.00006 learning rate: 0.00050 Time taken: 0:00:02.830224 ETA: 0:16:39.069169 INFO:tensorflow:global_step/sec: 1.78899 2021-09-25 23:17:07,602 [INFO] tensorflow: global_step/sec: 1.78899 INFO:tensorflow:global_step/sec: 1.76433 2021-09-25 23:17:08,168 [INFO] tensorflow: global_step/sec: 1.76433 INFO:tensorflow:global_step/sec: 1.77282 2021-09-25 23:17:08,733 [INFO] tensorflow: global_step/sec: 1.77282 INFO:tensorflow:global_step/sec: 1.73772 2021-09-25 23:17:09,308 [INFO] tensorflow: global_step/sec: 1.73772 INFO:tensorflow:epoch = 148.0, learning_rate = 0.00049999997, loss = 4.2036583e-05, step = 740 (5.657 sec) 2021-09-25 23:17:09,871 [INFO] tensorflow: epoch = 148.0, learning_rate = 0.00049999997, loss = 4.2036583e-05, step = 740 (5.657 sec) INFO:tensorflow:global_step/sec: 1.77316 2021-09-25 23:17:09,872 [INFO] tensorflow: global_step/sec: 1.77316 2021-09-25 23:17:09,873 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 148/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.832002 ETA: 0:16:36.864594 INFO:tensorflow:global_step/sec: 1.81885 2021-09-25 23:17:10,422 [INFO] tensorflow: global_step/sec: 1.81885 INFO:tensorflow:global_step/sec: 1.7903 2021-09-25 23:17:10,980 [INFO] tensorflow: global_step/sec: 1.7903 INFO:tensorflow:global_step/sec: 1.85042 2021-09-25 23:17:11,521 [INFO] tensorflow: global_step/sec: 1.85042 INFO:tensorflow:global_step/sec: 1.80629 2021-09-25 23:17:12,074 [INFO] tensorflow: global_step/sec: 1.80629 INFO:tensorflow:global_step/sec: 1.82426 2021-09-25 23:17:12,623 [INFO] tensorflow: global_step/sec: 1.82426 2021-09-25 23:17:12,624 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 149/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.767480 ETA: 0:16:11.385527 INFO:tensorflow:global_step/sec: 1.7479 2021-09-25 23:17:13,195 [INFO] tensorflow: global_step/sec: 1.7479 INFO:tensorflow:global_step/sec: 1.75409 2021-09-25 23:17:13,765 [INFO] tensorflow: global_step/sec: 1.75409 INFO:tensorflow:global_step/sec: 1.79081 2021-09-25 23:17:14,323 [INFO] tensorflow: global_step/sec: 1.79081 INFO:tensorflow:global_step/sec: 1.76058 2021-09-25 23:17:14,891 [INFO] tensorflow: global_step/sec: 1.76058 2021-09-25 23:17:14,895 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 113.608 INFO:tensorflow:epoch = 150.0, learning_rate = 0.00049999997, loss = 4.408918e-05, step = 750 (5.575 sec) 2021-09-25 23:17:15,446 [INFO] tensorflow: epoch = 150.0, learning_rate = 0.00049999997, loss = 4.408918e-05, step = 750 (5.575 sec) INFO:tensorflow:global_step/sec: 1.79814 2021-09-25 23:17:15,447 [INFO] tensorflow: global_step/sec: 1.79814 2021-09-25 23:17:15,449 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 150/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.814786 ETA: 0:16:25.175252 INFO:tensorflow:global_step/sec: 1.81359 2021-09-25 23:17:15,999 [INFO] tensorflow: global_step/sec: 1.81359 INFO:tensorflow:global_step/sec: 1.7461 2021-09-25 23:17:16,571 [INFO] tensorflow: global_step/sec: 1.7461 INFO:tensorflow:global_step/sec: 1.79762 2021-09-25 23:17:17,128 [INFO] tensorflow: global_step/sec: 1.79762 INFO:tensorflow:global_step/sec: 1.82369 2021-09-25 23:17:17,676 [INFO] tensorflow: global_step/sec: 1.82369 INFO:tensorflow:global_step/sec: 1.78326 2021-09-25 23:17:18,237 [INFO] tensorflow: global_step/sec: 1.78326 2021-09-25 23:17:18,238 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 151/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.785010 ETA: 0:16:11.968358 INFO:tensorflow:global_step/sec: 1.80559 2021-09-25 23:17:18,791 [INFO] tensorflow: global_step/sec: 1.80559 INFO:tensorflow:global_step/sec: 1.7427 2021-09-25 23:17:19,364 [INFO] tensorflow: global_step/sec: 1.7427 INFO:tensorflow:global_step/sec: 1.75961 2021-09-25 23:17:19,933 [INFO] tensorflow: global_step/sec: 1.75961 INFO:tensorflow:global_step/sec: 1.76598 2021-09-25 23:17:20,499 [INFO] tensorflow: global_step/sec: 1.76598 INFO:tensorflow:epoch = 152.0, learning_rate = 0.00049999997, loss = 4.2033244e-05, step = 760 (5.622 sec) 2021-09-25 23:17:21,068 [INFO] tensorflow: epoch = 152.0, learning_rate = 0.00049999997, loss = 4.2033244e-05, step = 760 (5.622 sec) INFO:tensorflow:global_step/sec: 1.7539 2021-09-25 23:17:21,069 [INFO] tensorflow: global_step/sec: 1.7539 2021-09-25 23:17:21,070 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 152/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.820759 ETA: 0:16:21.623986 INFO:tensorflow:global_step/sec: 1.82908 2021-09-25 23:17:21,616 [INFO] tensorflow: global_step/sec: 1.82908 INFO:tensorflow:global_step/sec: 1.64918 2021-09-25 23:17:22,222 [INFO] tensorflow: global_step/sec: 1.64918 INFO:tensorflow:global_step/sec: 1.73439 2021-09-25 23:17:22,799 [INFO] tensorflow: global_step/sec: 1.73439 INFO:tensorflow:global_step/sec: 1.8116 2021-09-25 23:17:23,351 [INFO] tensorflow: global_step/sec: 1.8116 INFO:tensorflow:global_step/sec: 1.77442 2021-09-25 23:17:23,914 [INFO] tensorflow: global_step/sec: 1.77442 2021-09-25 23:17:23,916 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 153/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.854058 ETA: 0:16:30.358218 INFO:tensorflow:global_step/sec: 1.69713 2021-09-25 23:17:24,504 [INFO] tensorflow: global_step/sec: 1.69713 INFO:tensorflow:global_step/sec: 1.74993 2021-09-25 23:17:25,075 [INFO] tensorflow: global_step/sec: 1.74993 INFO:tensorflow:global_step/sec: 1.77582 2021-09-25 23:17:25,638 [INFO] tensorflow: global_step/sec: 1.77582 INFO:tensorflow:global_step/sec: 1.7691 2021-09-25 23:17:26,203 [INFO] tensorflow: global_step/sec: 1.7691 INFO:tensorflow:epoch = 154.0, learning_rate = 0.00049999997, loss = 4.016337e-05, step = 770 (5.684 sec) 2021-09-25 23:17:26,752 [INFO] tensorflow: epoch = 154.0, learning_rate = 0.00049999997, loss = 4.016337e-05, step = 770 (5.684 sec) INFO:tensorflow:global_step/sec: 1.81989 2021-09-25 23:17:26,753 [INFO] tensorflow: global_step/sec: 1.81989 2021-09-25 23:17:26,754 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 154/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.848027 ETA: 0:16:25.417339 INFO:tensorflow:global_step/sec: 1.84949 2021-09-25 23:17:27,294 [INFO] tensorflow: global_step/sec: 1.84949 INFO:tensorflow:global_step/sec: 1.75853 2021-09-25 23:17:27,862 [INFO] tensorflow: global_step/sec: 1.75853 INFO:tensorflow:global_step/sec: 1.71867 2021-09-25 23:17:28,444 [INFO] tensorflow: global_step/sec: 1.71867 INFO:tensorflow:global_step/sec: 1.70102 2021-09-25 23:17:29,032 [INFO] tensorflow: global_step/sec: 1.70102 2021-09-25 23:17:29,037 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 113.148 INFO:tensorflow:global_step/sec: 1.81115 2021-09-25 23:17:29,584 [INFO] tensorflow: global_step/sec: 1.81115 2021-09-25 23:17:29,586 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 155/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.830167 ETA: 0:16:16.407470 INFO:tensorflow:global_step/sec: 1.83697 2021-09-25 23:17:30,129 [INFO] tensorflow: global_step/sec: 1.83697 INFO:tensorflow:global_step/sec: 1.76933 2021-09-25 23:17:30,694 [INFO] tensorflow: global_step/sec: 1.76933 INFO:tensorflow:global_step/sec: 1.86032 2021-09-25 23:17:31,231 [INFO] tensorflow: global_step/sec: 1.86032 INFO:tensorflow:global_step/sec: 1.82511 2021-09-25 23:17:31,779 [INFO] tensorflow: global_step/sec: 1.82511 INFO:tensorflow:epoch = 156.0, learning_rate = 0.00049999997, loss = 4.8055612e-05, step = 780 (5.597 sec) 2021-09-25 23:17:32,348 [INFO] tensorflow: epoch = 156.0, learning_rate = 0.00049999997, loss = 4.8055612e-05, step = 780 (5.597 sec) INFO:tensorflow:global_step/sec: 1.75266 2021-09-25 23:17:32,350 [INFO] tensorflow: global_step/sec: 1.75266 2021-09-25 23:17:32,351 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 156/500: loss: 0.00005 learning rate: 0.00050 Time taken: 0:00:02.743764 ETA: 0:15:43.854790 INFO:tensorflow:global_step/sec: 1.6959 2021-09-25 23:17:32,939 [INFO] tensorflow: global_step/sec: 1.6959 INFO:tensorflow:global_step/sec: 1.83797 2021-09-25 23:17:33,483 [INFO] tensorflow: global_step/sec: 1.83797 INFO:tensorflow:global_step/sec: 1.81254 2021-09-25 23:17:34,035 [INFO] tensorflow: global_step/sec: 1.81254 INFO:tensorflow:global_step/sec: 1.85586 2021-09-25 23:17:34,574 [INFO] tensorflow: global_step/sec: 1.85586 INFO:tensorflow:global_step/sec: 1.70543 2021-09-25 23:17:35,160 [INFO] tensorflow: global_step/sec: 1.70543 2021-09-25 23:17:35,162 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 157/500: loss: 0.00005 learning rate: 0.00050 Time taken: 0:00:02.797155 ETA: 0:15:59.424295 INFO:tensorflow:global_step/sec: 1.75318 2021-09-25 23:17:35,731 [INFO] tensorflow: global_step/sec: 1.75318 INFO:tensorflow:global_step/sec: 1.84391 2021-09-25 23:17:36,273 [INFO] tensorflow: global_step/sec: 1.84391 INFO:tensorflow:global_step/sec: 1.79269 2021-09-25 23:17:36,831 [INFO] tensorflow: global_step/sec: 1.79269 INFO:tensorflow:global_step/sec: 1.72593 2021-09-25 23:17:37,410 [INFO] tensorflow: global_step/sec: 1.72593 INFO:tensorflow:epoch = 158.0, learning_rate = 0.00049999997, loss = 4.9013255e-05, step = 790 (5.630 sec) 2021-09-25 23:17:37,978 [INFO] tensorflow: epoch = 158.0, learning_rate = 0.00049999997, loss = 4.9013255e-05, step = 790 (5.630 sec) INFO:tensorflow:global_step/sec: 1.75628 2021-09-25 23:17:37,980 [INFO] tensorflow: global_step/sec: 1.75628 2021-09-25 23:17:37,981 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 158/500: loss: 0.00005 learning rate: 0.00050 Time taken: 0:00:02.835047 ETA: 0:16:09.586076 INFO:tensorflow:global_step/sec: 1.77529 2021-09-25 23:17:38,543 [INFO] tensorflow: global_step/sec: 1.77529 INFO:tensorflow:global_step/sec: 1.7971 2021-09-25 23:17:39,099 [INFO] tensorflow: global_step/sec: 1.7971 INFO:tensorflow:global_step/sec: 1.79875 2021-09-25 23:17:39,655 [INFO] tensorflow: global_step/sec: 1.79875 INFO:tensorflow:global_step/sec: 1.76546 2021-09-25 23:17:40,222 [INFO] tensorflow: global_step/sec: 1.76546 INFO:tensorflow:global_step/sec: 1.74729 2021-09-25 23:17:40,794 [INFO] tensorflow: global_step/sec: 1.74729 2021-09-25 23:17:40,796 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 159/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.812964 ETA: 0:15:59.220793 INFO:tensorflow:global_step/sec: 1.74517 2021-09-25 23:17:41,367 [INFO] tensorflow: global_step/sec: 1.74517 INFO:tensorflow:global_step/sec: 1.8056 2021-09-25 23:17:41,921 [INFO] tensorflow: global_step/sec: 1.8056 INFO:tensorflow:global_step/sec: 1.80921 2021-09-25 23:17:42,474 [INFO] tensorflow: global_step/sec: 1.80921 INFO:tensorflow:global_step/sec: 1.78381 2021-09-25 23:17:43,034 [INFO] tensorflow: global_step/sec: 1.78381 INFO:tensorflow:Saving checkpoints for step-800. 2021-09-25 23:17:43,035 [INFO] tensorflow: Saving checkpoints for step-800. WARNING:tensorflow:Ignoring: /tmp/tmpmlu18xr3; No such file or directory 2021-09-25 23:17:43,259 [WARNING] tensorflow: Ignoring: /tmp/tmpmlu18xr3; No such file or directory 2021-09-25 23:17:47,147 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 88.355 INFO:tensorflow:epoch = 160.0, learning_rate = 0.00049999997, loss = 4.2859363e-05, step = 800 (9.743 sec) 2021-09-25 23:17:47,721 [INFO] tensorflow: epoch = 160.0, learning_rate = 0.00049999997, loss = 4.2859363e-05, step = 800 (9.743 sec) INFO:tensorflow:global_step/sec: 0.213295 2021-09-25 23:17:47,723 [INFO] tensorflow: global_step/sec: 0.213295 2021-09-25 23:17:47,724 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 160/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:06.917067 ETA: 0:39:11.802797 INFO:tensorflow:global_step/sec: 1.79597 2021-09-25 23:17:48,279 [INFO] tensorflow: global_step/sec: 1.79597 INFO:tensorflow:global_step/sec: 1.79841 2021-09-25 23:17:48,835 [INFO] tensorflow: global_step/sec: 1.79841 INFO:tensorflow:global_step/sec: 1.81642 2021-09-25 23:17:49,386 [INFO] tensorflow: global_step/sec: 1.81642 INFO:tensorflow:global_step/sec: 1.76894 2021-09-25 23:17:49,951 [INFO] tensorflow: global_step/sec: 1.76894 Epoch 161/500 ========================= Validation cost: 0.000000 Mean average_precision (in %): 0.0000 class name average precision (in %) ------------ -------------------------- lpd 0 Median Inference Time: nan INFO:tensorflow:global_step/sec: 1.7982 2021-09-25 23:17:50,507 [INFO] tensorflow: global_step/sec: 1.7982 2021-09-25 23:17:50,509 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 161/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.808129 ETA: 0:15:51.955594 INFO:tensorflow:global_step/sec: 1.84321 2021-09-25 23:17:51,050 [INFO] tensorflow: global_step/sec: 1.84321 INFO:tensorflow:global_step/sec: 1.80966 2021-09-25 23:17:51,603 [INFO] tensorflow: global_step/sec: 1.80966 INFO:tensorflow:global_step/sec: 1.76512 2021-09-25 23:17:52,169 [INFO] tensorflow: global_step/sec: 1.76512 INFO:tensorflow:global_step/sec: 1.77791 2021-09-25 23:17:52,732 [INFO] tensorflow: global_step/sec: 1.77791 INFO:tensorflow:epoch = 162.0, learning_rate = 0.00049999997, loss = 4.3843254e-05, step = 810 (5.569 sec) 2021-09-25 23:17:53,290 [INFO] tensorflow: epoch = 162.0, learning_rate = 0.00049999997, loss = 4.3843254e-05, step = 810 (5.569 sec) INFO:tensorflow:global_step/sec: 1.78594 2021-09-25 23:17:53,291 [INFO] tensorflow: global_step/sec: 1.78594 2021-09-25 23:17:53,293 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 162/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.775723 ETA: 0:15:38.194367 INFO:tensorflow:global_step/sec: 1.77631 2021-09-25 23:17:53,854 [INFO] tensorflow: global_step/sec: 1.77631 INFO:tensorflow:global_step/sec: 1.80763 2021-09-25 23:17:54,408 [INFO] tensorflow: global_step/sec: 1.80763 INFO:tensorflow:global_step/sec: 1.67295 2021-09-25 23:17:55,005 [INFO] tensorflow: global_step/sec: 1.67295 INFO:tensorflow:global_step/sec: 1.75606 2021-09-25 23:17:55,575 [INFO] tensorflow: global_step/sec: 1.75606 INFO:tensorflow:global_step/sec: 1.71939 2021-09-25 23:17:56,156 [INFO] tensorflow: global_step/sec: 1.71939 2021-09-25 23:17:56,158 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 163/500: loss: 0.00005 learning rate: 0.00050 Time taken: 0:00:02.845474 ETA: 0:15:58.924579 INFO:tensorflow:global_step/sec: 1.85295 2021-09-25 23:17:56,696 [INFO] tensorflow: global_step/sec: 1.85295 INFO:tensorflow:global_step/sec: 1.80188 2021-09-25 23:17:57,251 [INFO] tensorflow: global_step/sec: 1.80188 INFO:tensorflow:global_step/sec: 1.7885 2021-09-25 23:17:57,810 [INFO] tensorflow: global_step/sec: 1.7885 INFO:tensorflow:global_step/sec: 1.82884 2021-09-25 23:17:58,357 [INFO] tensorflow: global_step/sec: 1.82884 INFO:tensorflow:epoch = 164.0, learning_rate = 0.00049999997, loss = 4.4826083e-05, step = 820 (5.633 sec) 2021-09-25 23:17:58,923 [INFO] tensorflow: epoch = 164.0, learning_rate = 0.00049999997, loss = 4.4826083e-05, step = 820 (5.633 sec) INFO:tensorflow:global_step/sec: 1.76348 2021-09-25 23:17:58,924 [INFO] tensorflow: global_step/sec: 1.76348 2021-09-25 23:17:58,925 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 164/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.777376 ETA: 0:15:33.198395 INFO:tensorflow:global_step/sec: 1.70054 2021-09-25 23:17:59,512 [INFO] tensorflow: global_step/sec: 1.70054 INFO:tensorflow:global_step/sec: 1.78765 2021-09-25 23:18:00,072 [INFO] tensorflow: global_step/sec: 1.78765 INFO:tensorflow:global_step/sec: 1.82157 2021-09-25 23:18:00,621 [INFO] tensorflow: global_step/sec: 1.82157 INFO:tensorflow:global_step/sec: 1.74491 2021-09-25 23:18:01,194 [INFO] tensorflow: global_step/sec: 1.74491 2021-09-25 23:18:01,199 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 113.898 INFO:tensorflow:global_step/sec: 1.80736 2021-09-25 23:18:01,747 [INFO] tensorflow: global_step/sec: 1.80736 2021-09-25 23:18:01,748 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 165/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.838019 ETA: 0:15:50.736489 INFO:tensorflow:global_step/sec: 1.77909 2021-09-25 23:18:02,309 [INFO] tensorflow: global_step/sec: 1.77909 INFO:tensorflow:global_step/sec: 1.82601 2021-09-25 23:18:02,857 [INFO] tensorflow: global_step/sec: 1.82601 INFO:tensorflow:global_step/sec: 1.77391 2021-09-25 23:18:03,420 [INFO] tensorflow: global_step/sec: 1.77391 INFO:tensorflow:global_step/sec: 1.78803 2021-09-25 23:18:03,980 [INFO] tensorflow: global_step/sec: 1.78803 INFO:tensorflow:epoch = 166.0, learning_rate = 0.00049999997, loss = 4.1091003e-05, step = 830 (5.623 sec) 2021-09-25 23:18:04,546 [INFO] tensorflow: epoch = 166.0, learning_rate = 0.00049999997, loss = 4.1091003e-05, step = 830 (5.623 sec) INFO:tensorflow:global_step/sec: 1.76194 2021-09-25 23:18:04,547 [INFO] tensorflow: global_step/sec: 1.76194 2021-09-25 23:18:04,548 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 166/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.784338 ETA: 0:15:29.968812 INFO:tensorflow:global_step/sec: 1.74411 2021-09-25 23:18:05,121 [INFO] tensorflow: global_step/sec: 1.74411 INFO:tensorflow:global_step/sec: 1.74663 2021-09-25 23:18:05,693 [INFO] tensorflow: global_step/sec: 1.74663 INFO:tensorflow:global_step/sec: 1.71424 2021-09-25 23:18:06,276 [INFO] tensorflow: global_step/sec: 1.71424 INFO:tensorflow:global_step/sec: 1.81345 2021-09-25 23:18:06,828 [INFO] tensorflow: global_step/sec: 1.81345 INFO:tensorflow:global_step/sec: 1.77643 2021-09-25 23:18:07,391 [INFO] tensorflow: global_step/sec: 1.77643 2021-09-25 23:18:07,392 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 167/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.846505 ETA: 0:15:47.886220 INFO:tensorflow:global_step/sec: 1.81095 2021-09-25 23:18:07,943 [INFO] tensorflow: global_step/sec: 1.81095 INFO:tensorflow:global_step/sec: 1.80824 2021-09-25 23:18:08,496 [INFO] tensorflow: global_step/sec: 1.80824 INFO:tensorflow:global_step/sec: 1.80099 2021-09-25 23:18:09,051 [INFO] tensorflow: global_step/sec: 1.80099 INFO:tensorflow:global_step/sec: 1.81051 2021-09-25 23:18:09,604 [INFO] tensorflow: global_step/sec: 1.81051 INFO:tensorflow:epoch = 168.0, learning_rate = 0.00049999997, loss = 4.834427e-05, step = 840 (5.612 sec) 2021-09-25 23:18:10,157 [INFO] tensorflow: epoch = 168.0, learning_rate = 0.00049999997, loss = 4.834427e-05, step = 840 (5.612 sec) INFO:tensorflow:global_step/sec: 1.80113 2021-09-25 23:18:10,159 [INFO] tensorflow: global_step/sec: 1.80113 2021-09-25 23:18:10,160 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 168/500: loss: 0.00005 learning rate: 0.00050 Time taken: 0:00:02.775060 ETA: 0:15:21.319820 INFO:tensorflow:global_step/sec: 1.80717 2021-09-25 23:18:10,712 [INFO] tensorflow: global_step/sec: 1.80717 INFO:tensorflow:global_step/sec: 1.78521 2021-09-25 23:18:11,272 [INFO] tensorflow: global_step/sec: 1.78521 INFO:tensorflow:global_step/sec: 1.69746 2021-09-25 23:18:11,861 [INFO] tensorflow: global_step/sec: 1.69746 INFO:tensorflow:global_step/sec: 1.8641 2021-09-25 23:18:12,398 [INFO] tensorflow: global_step/sec: 1.8641 INFO:tensorflow:global_step/sec: 1.8347 2021-09-25 23:18:12,943 [INFO] tensorflow: global_step/sec: 1.8347 2021-09-25 23:18:12,944 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 169/500: loss: 0.00005 learning rate: 0.00050 Time taken: 0:00:02.794352 ETA: 0:15:24.930451 INFO:tensorflow:global_step/sec: 1.83943 2021-09-25 23:18:13,487 [INFO] tensorflow: global_step/sec: 1.83943 INFO:tensorflow:global_step/sec: 1.79158 2021-09-25 23:18:14,045 [INFO] tensorflow: global_step/sec: 1.79158 INFO:tensorflow:global_step/sec: 1.78485 2021-09-25 23:18:14,605 [INFO] tensorflow: global_step/sec: 1.78485 INFO:tensorflow:global_step/sec: 1.7779 2021-09-25 23:18:15,167 [INFO] tensorflow: global_step/sec: 1.7779 2021-09-25 23:18:15,171 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 114.521 INFO:tensorflow:epoch = 170.0, learning_rate = 0.00049999997, loss = 3.9201906e-05, step = 850 (5.587 sec) 2021-09-25 23:18:15,744 [INFO] tensorflow: epoch = 170.0, learning_rate = 0.00049999997, loss = 3.9201906e-05, step = 850 (5.587 sec) INFO:tensorflow:global_step/sec: 1.72992 2021-09-25 23:18:15,745 [INFO] tensorflow: global_step/sec: 1.72992 2021-09-25 23:18:15,747 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 170/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.767884 ETA: 0:15:13.401725 INFO:tensorflow:global_step/sec: 1.752 2021-09-25 23:18:16,316 [INFO] tensorflow: global_step/sec: 1.752 INFO:tensorflow:global_step/sec: 1.80005 2021-09-25 23:18:16,872 [INFO] tensorflow: global_step/sec: 1.80005 INFO:tensorflow:global_step/sec: 1.85357 2021-09-25 23:18:17,411 [INFO] tensorflow: global_step/sec: 1.85357 INFO:tensorflow:global_step/sec: 1.78708 2021-09-25 23:18:17,971 [INFO] tensorflow: global_step/sec: 1.78708 INFO:tensorflow:global_step/sec: 1.78129 2021-09-25 23:18:18,532 [INFO] tensorflow: global_step/sec: 1.78129 2021-09-25 23:18:18,534 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 171/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.806883 ETA: 0:15:23.464539 INFO:tensorflow:global_step/sec: 1.75616 2021-09-25 23:18:19,102 [INFO] tensorflow: global_step/sec: 1.75616 INFO:tensorflow:global_step/sec: 1.83802 2021-09-25 23:18:19,646 [INFO] tensorflow: global_step/sec: 1.83802 INFO:tensorflow:global_step/sec: 1.68837 2021-09-25 23:18:20,238 [INFO] tensorflow: global_step/sec: 1.68837 INFO:tensorflow:global_step/sec: 1.74035 2021-09-25 23:18:20,813 [INFO] tensorflow: global_step/sec: 1.74035 INFO:tensorflow:epoch = 172.0, learning_rate = 0.00049999997, loss = 4.1608357e-05, step = 860 (5.620 sec) 2021-09-25 23:18:21,364 [INFO] tensorflow: epoch = 172.0, learning_rate = 0.00049999997, loss = 4.1608357e-05, step = 860 (5.620 sec) INFO:tensorflow:global_step/sec: 1.80894 2021-09-25 23:18:21,365 [INFO] tensorflow: global_step/sec: 1.80894 2021-09-25 23:18:21,367 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 172/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.838022 ETA: 0:15:30.871058 INFO:tensorflow:global_step/sec: 1.79757 2021-09-25 23:18:21,922 [INFO] tensorflow: global_step/sec: 1.79757 INFO:tensorflow:global_step/sec: 1.82091 2021-09-25 23:18:22,471 [INFO] tensorflow: global_step/sec: 1.82091 INFO:tensorflow:global_step/sec: 1.76363 2021-09-25 23:18:23,038 [INFO] tensorflow: global_step/sec: 1.76363 INFO:tensorflow:global_step/sec: 1.85869 2021-09-25 23:18:23,576 [INFO] tensorflow: global_step/sec: 1.85869 INFO:tensorflow:global_step/sec: 1.7773 2021-09-25 23:18:24,139 [INFO] tensorflow: global_step/sec: 1.7773 2021-09-25 23:18:24,140 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 173/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.764759 ETA: 0:15:04.076214 INFO:tensorflow:global_step/sec: 1.82706 2021-09-25 23:18:24,686 [INFO] tensorflow: global_step/sec: 1.82706 INFO:tensorflow:global_step/sec: 1.80707 2021-09-25 23:18:25,239 [INFO] tensorflow: global_step/sec: 1.80707 INFO:tensorflow:global_step/sec: 1.83521 2021-09-25 23:18:25,784 [INFO] tensorflow: global_step/sec: 1.83521 INFO:tensorflow:global_step/sec: 1.79293 2021-09-25 23:18:26,342 [INFO] tensorflow: global_step/sec: 1.79293 INFO:tensorflow:global_step/sec: 1.81785 2021-09-25 23:18:26,892 [INFO] tensorflow: global_step/sec: 1.81785 2021-09-25 23:18:26,894 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 174/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.763970 ETA: 0:15:01.054265 INFO:tensorflow:epoch = 174.20000000000002, learning_rate = 0.00049999997, loss = 3.8340928e-05, step = 871 (6.084 sec) 2021-09-25 23:18:27,448 [INFO] tensorflow: epoch = 174.20000000000002, learning_rate = 0.00049999997, loss = 3.8340928e-05, step = 871 (6.084 sec) INFO:tensorflow:global_step/sec: 1.79548 2021-09-25 23:18:27,449 [INFO] tensorflow: global_step/sec: 1.79548 INFO:tensorflow:global_step/sec: 1.83902 2021-09-25 23:18:27,993 [INFO] tensorflow: global_step/sec: 1.83902 INFO:tensorflow:global_step/sec: 1.8355 2021-09-25 23:18:28,538 [INFO] tensorflow: global_step/sec: 1.8355 INFO:tensorflow:global_step/sec: 1.79501 2021-09-25 23:18:29,095 [INFO] tensorflow: global_step/sec: 1.79501 2021-09-25 23:18:29,102 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 114.858 INFO:tensorflow:global_step/sec: 1.7633 2021-09-25 23:18:29,662 [INFO] tensorflow: global_step/sec: 1.7633 2021-09-25 23:18:29,663 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 175/500: loss: 0.00005 learning rate: 0.00050 Time taken: 0:00:02.753701 ETA: 0:14:54.952816 INFO:tensorflow:global_step/sec: 1.76948 2021-09-25 23:18:30,227 [INFO] tensorflow: global_step/sec: 1.76948 INFO:tensorflow:global_step/sec: 1.81706 2021-09-25 23:18:30,777 [INFO] tensorflow: global_step/sec: 1.81706 INFO:tensorflow:global_step/sec: 1.80908 2021-09-25 23:18:31,330 [INFO] tensorflow: global_step/sec: 1.80908 INFO:tensorflow:global_step/sec: 1.79474 2021-09-25 23:18:31,887 [INFO] tensorflow: global_step/sec: 1.79474 INFO:tensorflow:global_step/sec: 1.75699 2021-09-25 23:18:32,456 [INFO] tensorflow: global_step/sec: 1.75699 2021-09-25 23:18:32,458 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 176/500: loss: 0.00005 learning rate: 0.00050 Time taken: 0:00:02.787445 ETA: 0:15:03.132279 INFO:tensorflow:epoch = 176.20000000000002, learning_rate = 0.00049999997, loss = 4.392748e-05, step = 881 (5.565 sec) 2021-09-25 23:18:33,013 [INFO] tensorflow: epoch = 176.20000000000002, learning_rate = 0.00049999997, loss = 4.392748e-05, step = 881 (5.565 sec) INFO:tensorflow:global_step/sec: 1.79344 2021-09-25 23:18:33,014 [INFO] tensorflow: global_step/sec: 1.79344 INFO:tensorflow:global_step/sec: 1.7516 2021-09-25 23:18:33,585 [INFO] tensorflow: global_step/sec: 1.7516 INFO:tensorflow:global_step/sec: 1.76082 2021-09-25 23:18:34,153 [INFO] tensorflow: global_step/sec: 1.76082 INFO:tensorflow:global_step/sec: 1.82935 2021-09-25 23:18:34,699 [INFO] tensorflow: global_step/sec: 1.82935 INFO:tensorflow:global_step/sec: 1.7996 2021-09-25 23:18:35,255 [INFO] tensorflow: global_step/sec: 1.7996 2021-09-25 23:18:35,257 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 177/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.815812 ETA: 0:15:09.507389 INFO:tensorflow:global_step/sec: 1.82261 2021-09-25 23:18:35,804 [INFO] tensorflow: global_step/sec: 1.82261 INFO:tensorflow:global_step/sec: 1.76086 2021-09-25 23:18:36,372 [INFO] tensorflow: global_step/sec: 1.76086 INFO:tensorflow:global_step/sec: 1.76623 2021-09-25 23:18:36,938 [INFO] tensorflow: global_step/sec: 1.76623 INFO:tensorflow:global_step/sec: 1.84475 2021-09-25 23:18:37,480 [INFO] tensorflow: global_step/sec: 1.84475 INFO:tensorflow:global_step/sec: 1.7767 2021-09-25 23:18:38,043 [INFO] tensorflow: global_step/sec: 1.7767 2021-09-25 23:18:38,044 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 178/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.779770 ETA: 0:14:55.085907 INFO:tensorflow:epoch = 178.20000000000002, learning_rate = 0.00049999997, loss = 3.630761e-05, step = 891 (5.574 sec) 2021-09-25 23:18:38,587 [INFO] tensorflow: epoch = 178.20000000000002, learning_rate = 0.00049999997, loss = 3.630761e-05, step = 891 (5.574 sec) INFO:tensorflow:global_step/sec: 1.83456 2021-09-25 23:18:38,588 [INFO] tensorflow: global_step/sec: 1.83456 INFO:tensorflow:global_step/sec: 1.8345 2021-09-25 23:18:39,133 [INFO] tensorflow: global_step/sec: 1.8345 INFO:tensorflow:global_step/sec: 1.75588 2021-09-25 23:18:39,703 [INFO] tensorflow: global_step/sec: 1.75588 INFO:tensorflow:global_step/sec: 1.80258 2021-09-25 23:18:40,257 [INFO] tensorflow: global_step/sec: 1.80258 INFO:tensorflow:global_step/sec: 1.72566 2021-09-25 23:18:40,837 [INFO] tensorflow: global_step/sec: 1.72566 2021-09-25 23:18:40,838 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 179/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.776331 ETA: 0:14:51.202158 INFO:tensorflow:global_step/sec: 1.84773 2021-09-25 23:18:41,378 [INFO] tensorflow: global_step/sec: 1.84773 INFO:tensorflow:global_step/sec: 1.83775 2021-09-25 23:18:41,922 [INFO] tensorflow: global_step/sec: 1.83775 INFO:tensorflow:global_step/sec: 1.73987 2021-09-25 23:18:42,497 [INFO] tensorflow: global_step/sec: 1.73987 INFO:tensorflow:global_step/sec: 1.82382 2021-09-25 23:18:43,045 [INFO] tensorflow: global_step/sec: 1.82382 INFO:tensorflow:Saving checkpoints for step-900. 2021-09-25 23:18:43,046 [INFO] tensorflow: Saving checkpoints for step-900. WARNING:tensorflow:Ignoring: /tmp/tmpy1v7qamn; No such file or directory 2021-09-25 23:18:43,268 [WARNING] tensorflow: Ignoring: /tmp/tmpy1v7qamn; No such file or directory 2021-09-25 23:18:47,131 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 88.752 INFO:tensorflow:epoch = 180.0, learning_rate = 0.00049999997, loss = 3.8471568e-05, step = 900 (9.124 sec) 2021-09-25 23:18:47,711 [INFO] tensorflow: epoch = 180.0, learning_rate = 0.00049999997, loss = 3.8471568e-05, step = 900 (9.124 sec) INFO:tensorflow:global_step/sec: 0.21427 2021-09-25 23:18:47,712 [INFO] tensorflow: global_step/sec: 0.21427 2021-09-25 23:18:47,713 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 180/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:06.865471 ETA: 0:36:36.950607 INFO:tensorflow:global_step/sec: 1.81326 2021-09-25 23:18:48,264 [INFO] tensorflow: global_step/sec: 1.81326 INFO:tensorflow:global_step/sec: 1.80454 2021-09-25 23:18:48,818 [INFO] tensorflow: global_step/sec: 1.80454 INFO:tensorflow:global_step/sec: 1.76591 2021-09-25 23:18:49,384 [INFO] tensorflow: global_step/sec: 1.76591 INFO:tensorflow:global_step/sec: 1.77885 2021-09-25 23:18:49,946 [INFO] tensorflow: global_step/sec: 1.77885 Epoch 181/500 ========================= Validation cost: 0.000000 Mean average_precision (in %): 0.0000 class name average precision (in %) ------------ -------------------------- lpd 0 Median Inference Time: nan INFO:tensorflow:global_step/sec: 1.80088 2021-09-25 23:18:50,502 [INFO] tensorflow: global_step/sec: 1.80088 2021-09-25 23:18:50,503 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 181/500: loss: 0.00005 learning rate: 0.00050 Time taken: 0:00:02.821031 ETA: 0:14:59.908767 INFO:tensorflow:global_step/sec: 1.81042 2021-09-25 23:18:51,054 [INFO] tensorflow: global_step/sec: 1.81042 INFO:tensorflow:global_step/sec: 1.80582 2021-09-25 23:18:51,608 [INFO] tensorflow: global_step/sec: 1.80582 INFO:tensorflow:global_step/sec: 1.75898 2021-09-25 23:18:52,176 [INFO] tensorflow: global_step/sec: 1.75898 INFO:tensorflow:global_step/sec: 1.78701 2021-09-25 23:18:52,736 [INFO] tensorflow: global_step/sec: 1.78701 INFO:tensorflow:epoch = 182.0, learning_rate = 0.00049999997, loss = 3.8985774e-05, step = 910 (5.576 sec) 2021-09-25 23:18:53,287 [INFO] tensorflow: epoch = 182.0, learning_rate = 0.00049999997, loss = 3.8985774e-05, step = 910 (5.576 sec) INFO:tensorflow:global_step/sec: 1.81115 2021-09-25 23:18:53,288 [INFO] tensorflow: global_step/sec: 1.81115 2021-09-25 23:18:53,289 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 182/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.788572 ETA: 0:14:46.765919 INFO:tensorflow:global_step/sec: 1.7926 2021-09-25 23:18:53,846 [INFO] tensorflow: global_step/sec: 1.7926 INFO:tensorflow:global_step/sec: 1.78601 2021-09-25 23:18:54,406 [INFO] tensorflow: global_step/sec: 1.78601 INFO:tensorflow:global_step/sec: 1.83962 2021-09-25 23:18:54,949 [INFO] tensorflow: global_step/sec: 1.83962 INFO:tensorflow:global_step/sec: 1.77539 2021-09-25 23:18:55,513 [INFO] tensorflow: global_step/sec: 1.77539 INFO:tensorflow:global_step/sec: 1.764 2021-09-25 23:18:56,079 [INFO] tensorflow: global_step/sec: 1.764 2021-09-25 23:18:56,081 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 183/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.775144 ETA: 0:14:39.720529 INFO:tensorflow:global_step/sec: 1.83299 2021-09-25 23:18:56,625 [INFO] tensorflow: global_step/sec: 1.83299 INFO:tensorflow:global_step/sec: 1.77669 2021-09-25 23:18:57,188 [INFO] tensorflow: global_step/sec: 1.77669 INFO:tensorflow:global_step/sec: 1.85214 2021-09-25 23:18:57,728 [INFO] tensorflow: global_step/sec: 1.85214 INFO:tensorflow:global_step/sec: 1.82826 2021-09-25 23:18:58,275 [INFO] tensorflow: global_step/sec: 1.82826 INFO:tensorflow:global_step/sec: 1.84134 2021-09-25 23:18:58,818 [INFO] tensorflow: global_step/sec: 1.84134 2021-09-25 23:18:58,819 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 184/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.760387 ETA: 0:14:32.282425 INFO:tensorflow:epoch = 184.20000000000002, learning_rate = 0.00049999997, loss = 4.2891912e-05, step = 921 (6.063 sec) 2021-09-25 23:18:59,349 [INFO] tensorflow: epoch = 184.20000000000002, learning_rate = 0.00049999997, loss = 4.2891912e-05, step = 921 (6.063 sec) INFO:tensorflow:global_step/sec: 1.87653 2021-09-25 23:18:59,351 [INFO] tensorflow: global_step/sec: 1.87653 INFO:tensorflow:global_step/sec: 1.80473 2021-09-25 23:18:59,905 [INFO] tensorflow: global_step/sec: 1.80473 INFO:tensorflow:global_step/sec: 1.76448 2021-09-25 23:19:00,472 [INFO] tensorflow: global_step/sec: 1.76448 INFO:tensorflow:global_step/sec: 1.74585 2021-09-25 23:19:01,044 [INFO] tensorflow: global_step/sec: 1.74585 2021-09-25 23:19:01,050 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 114.954 INFO:tensorflow:global_step/sec: 1.68443 2021-09-25 23:19:01,638 [INFO] tensorflow: global_step/sec: 1.68443 2021-09-25 23:19:01,640 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 185/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.770429 ETA: 0:14:32.685252 INFO:tensorflow:global_step/sec: 1.78988 2021-09-25 23:19:02,197 [INFO] tensorflow: global_step/sec: 1.78988 INFO:tensorflow:global_step/sec: 1.78867 2021-09-25 23:19:02,756 [INFO] tensorflow: global_step/sec: 1.78867 INFO:tensorflow:global_step/sec: 1.83947 2021-09-25 23:19:03,299 [INFO] tensorflow: global_step/sec: 1.83947 INFO:tensorflow:global_step/sec: 1.78624 2021-09-25 23:19:03,859 [INFO] tensorflow: global_step/sec: 1.78624 INFO:tensorflow:global_step/sec: 1.7915 2021-09-25 23:19:04,417 [INFO] tensorflow: global_step/sec: 1.7915 2021-09-25 23:19:04,419 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 186/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.815504 ETA: 0:14:44.068130 INFO:tensorflow:epoch = 186.20000000000002, learning_rate = 0.00049999997, loss = 4.240246e-05, step = 931 (5.644 sec) 2021-09-25 23:19:04,993 [INFO] tensorflow: epoch = 186.20000000000002, learning_rate = 0.00049999997, loss = 4.240246e-05, step = 931 (5.644 sec) INFO:tensorflow:global_step/sec: 1.73328 2021-09-25 23:19:04,994 [INFO] tensorflow: global_step/sec: 1.73328 INFO:tensorflow:global_step/sec: 1.78097 2021-09-25 23:19:05,556 [INFO] tensorflow: global_step/sec: 1.78097 INFO:tensorflow:global_step/sec: 1.83408 2021-09-25 23:19:06,101 [INFO] tensorflow: global_step/sec: 1.83408 INFO:tensorflow:global_step/sec: 1.75893 2021-09-25 23:19:06,670 [INFO] tensorflow: global_step/sec: 1.75893 INFO:tensorflow:global_step/sec: 1.7986 2021-09-25 23:19:07,226 [INFO] tensorflow: global_step/sec: 1.7986 2021-09-25 23:19:07,227 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 187/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.808920 ETA: 0:14:39.192005 INFO:tensorflow:global_step/sec: 1.73402 2021-09-25 23:19:07,802 [INFO] tensorflow: global_step/sec: 1.73402 INFO:tensorflow:global_step/sec: 1.7796 2021-09-25 23:19:08,364 [INFO] tensorflow: global_step/sec: 1.7796 INFO:tensorflow:global_step/sec: 1.82133 2021-09-25 23:19:08,913 [INFO] tensorflow: global_step/sec: 1.82133 INFO:tensorflow:global_step/sec: 1.65307 2021-09-25 23:19:09,518 [INFO] tensorflow: global_step/sec: 1.65307 INFO:tensorflow:global_step/sec: 1.73022 2021-09-25 23:19:10,096 [INFO] tensorflow: global_step/sec: 1.73022 2021-09-25 23:19:10,098 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 188/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.848597 ETA: 0:14:48.762354 INFO:tensorflow:epoch = 188.20000000000002, learning_rate = 0.00049999997, loss = 4.4375978e-05, step = 941 (5.647 sec) 2021-09-25 23:19:10,640 [INFO] tensorflow: epoch = 188.20000000000002, learning_rate = 0.00049999997, loss = 4.4375978e-05, step = 941 (5.647 sec) INFO:tensorflow:global_step/sec: 1.83341 2021-09-25 23:19:10,642 [INFO] tensorflow: global_step/sec: 1.83341 INFO:tensorflow:global_step/sec: 1.71621 2021-09-25 23:19:11,224 [INFO] tensorflow: global_step/sec: 1.71621 INFO:tensorflow:global_step/sec: 1.63771 2021-09-25 23:19:11,835 [INFO] tensorflow: global_step/sec: 1.63771 INFO:tensorflow:global_step/sec: 1.83366 2021-09-25 23:19:12,380 [INFO] tensorflow: global_step/sec: 1.83366 INFO:tensorflow:global_step/sec: 1.79719 2021-09-25 23:19:12,937 [INFO] tensorflow: global_step/sec: 1.79719 2021-09-25 23:19:12,938 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 189/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.859088 ETA: 0:14:49.176425 INFO:tensorflow:global_step/sec: 1.78874 2021-09-25 23:19:13,496 [INFO] tensorflow: global_step/sec: 1.78874 INFO:tensorflow:global_step/sec: 1.78381 2021-09-25 23:19:14,056 [INFO] tensorflow: global_step/sec: 1.78381 INFO:tensorflow:global_step/sec: 1.7662 2021-09-25 23:19:14,623 [INFO] tensorflow: global_step/sec: 1.7662 INFO:tensorflow:global_step/sec: 1.79792 2021-09-25 23:19:15,179 [INFO] tensorflow: global_step/sec: 1.79792 2021-09-25 23:19:15,184 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 113.210 INFO:tensorflow:global_step/sec: 1.81718 2021-09-25 23:19:15,729 [INFO] tensorflow: global_step/sec: 1.81718 2021-09-25 23:19:15,731 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 190/500: loss: 0.00006 learning rate: 0.00050 Time taken: 0:00:02.797907 ETA: 0:14:27.351131 INFO:tensorflow:epoch = 190.20000000000002, learning_rate = 0.00049999997, loss = 4.1011222e-05, step = 951 (5.637 sec) 2021-09-25 23:19:16,277 [INFO] tensorflow: epoch = 190.20000000000002, learning_rate = 0.00049999997, loss = 4.1011222e-05, step = 951 (5.637 sec) INFO:tensorflow:global_step/sec: 1.81933 2021-09-25 23:19:16,279 [INFO] tensorflow: global_step/sec: 1.81933 INFO:tensorflow:global_step/sec: 1.85808 2021-09-25 23:19:16,817 [INFO] tensorflow: global_step/sec: 1.85808 INFO:tensorflow:global_step/sec: 1.82483 2021-09-25 23:19:17,365 [INFO] tensorflow: global_step/sec: 1.82483 INFO:tensorflow:global_step/sec: 1.79322 2021-09-25 23:19:17,923 [INFO] tensorflow: global_step/sec: 1.79322 INFO:tensorflow:global_step/sec: 1.77926 2021-09-25 23:19:18,485 [INFO] tensorflow: global_step/sec: 1.77926 2021-09-25 23:19:18,486 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 191/500: loss: 0.00006 learning rate: 0.00050 Time taken: 0:00:02.741550 ETA: 0:14:07.138940 INFO:tensorflow:global_step/sec: 1.76545 2021-09-25 23:19:19,051 [INFO] tensorflow: global_step/sec: 1.76545 INFO:tensorflow:global_step/sec: 1.84046 2021-09-25 23:19:19,594 [INFO] tensorflow: global_step/sec: 1.84046 INFO:tensorflow:global_step/sec: 1.78082 2021-09-25 23:19:20,156 [INFO] tensorflow: global_step/sec: 1.78082 INFO:tensorflow:global_step/sec: 1.79778 2021-09-25 23:19:20,712 [INFO] tensorflow: global_step/sec: 1.79778 INFO:tensorflow:global_step/sec: 1.79017 2021-09-25 23:19:21,271 [INFO] tensorflow: global_step/sec: 1.79017 2021-09-25 23:19:21,272 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 192/500: loss: 0.00005 learning rate: 0.00050 Time taken: 0:00:02.789352 ETA: 0:14:19.120398 INFO:tensorflow:global_step/sec: 1.78659 2021-09-25 23:19:21,830 [INFO] tensorflow: global_step/sec: 1.78659 INFO:tensorflow:epoch = 192.4, learning_rate = 0.00049999997, loss = 3.981519e-05, step = 962 (6.117 sec) 2021-09-25 23:19:22,394 [INFO] tensorflow: epoch = 192.4, learning_rate = 0.00049999997, loss = 3.981519e-05, step = 962 (6.117 sec) INFO:tensorflow:global_step/sec: 1.76967 2021-09-25 23:19:22,395 [INFO] tensorflow: global_step/sec: 1.76967 INFO:tensorflow:global_step/sec: 1.79888 2021-09-25 23:19:22,951 [INFO] tensorflow: global_step/sec: 1.79888 INFO:tensorflow:global_step/sec: 1.77745 2021-09-25 23:19:23,514 [INFO] tensorflow: global_step/sec: 1.77745 INFO:tensorflow:global_step/sec: 1.74625 2021-09-25 23:19:24,087 [INFO] tensorflow: global_step/sec: 1.74625 2021-09-25 23:19:24,088 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 193/500: loss: 0.00005 learning rate: 0.00050 Time taken: 0:00:02.799922 ETA: 0:14:19.576051 INFO:tensorflow:global_step/sec: 1.65982 2021-09-25 23:19:24,689 [INFO] tensorflow: global_step/sec: 1.65982 INFO:tensorflow:global_step/sec: 1.73211 2021-09-25 23:19:25,266 [INFO] tensorflow: global_step/sec: 1.73211 INFO:tensorflow:global_step/sec: 1.83377 2021-09-25 23:19:25,812 [INFO] tensorflow: global_step/sec: 1.83377 INFO:tensorflow:global_step/sec: 1.73995 2021-09-25 23:19:26,387 [INFO] tensorflow: global_step/sec: 1.73995 INFO:tensorflow:global_step/sec: 1.78057 2021-09-25 23:19:26,948 [INFO] tensorflow: global_step/sec: 1.78057 2021-09-25 23:19:26,950 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 194/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.872322 ETA: 0:14:38.930630 INFO:tensorflow:global_step/sec: 1.78907 2021-09-25 23:19:27,507 [INFO] tensorflow: global_step/sec: 1.78907 INFO:tensorflow:epoch = 194.4, learning_rate = 0.00049999997, loss = 4.755936e-05, step = 972 (5.682 sec) 2021-09-25 23:19:28,076 [INFO] tensorflow: epoch = 194.4, learning_rate = 0.00049999997, loss = 4.755936e-05, step = 972 (5.682 sec) INFO:tensorflow:global_step/sec: 1.75431 2021-09-25 23:19:28,077 [INFO] tensorflow: global_step/sec: 1.75431 INFO:tensorflow:global_step/sec: 1.7322 2021-09-25 23:19:28,654 [INFO] tensorflow: global_step/sec: 1.7322 INFO:tensorflow:global_step/sec: 1.76243 2021-09-25 23:19:29,222 [INFO] tensorflow: global_step/sec: 1.76243 2021-09-25 23:19:29,227 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 113.943 INFO:tensorflow:global_step/sec: 1.74875 2021-09-25 23:19:29,794 [INFO] tensorflow: global_step/sec: 1.74875 2021-09-25 23:19:29,795 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 195/500: loss: 0.00005 learning rate: 0.00050 Time taken: 0:00:02.835969 ETA: 0:14:24.970464 INFO:tensorflow:global_step/sec: 1.81047 2021-09-25 23:19:30,346 [INFO] tensorflow: global_step/sec: 1.81047 INFO:tensorflow:global_step/sec: 1.7712 2021-09-25 23:19:30,911 [INFO] tensorflow: global_step/sec: 1.7712 INFO:tensorflow:global_step/sec: 1.78321 2021-09-25 23:19:31,471 [INFO] tensorflow: global_step/sec: 1.78321 INFO:tensorflow:global_step/sec: 1.76513 2021-09-25 23:19:32,038 [INFO] tensorflow: global_step/sec: 1.76513 INFO:tensorflow:global_step/sec: 1.74597 2021-09-25 23:19:32,611 [INFO] tensorflow: global_step/sec: 1.74597 2021-09-25 23:19:32,612 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 196/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.814604 ETA: 0:14:15.639702 INFO:tensorflow:global_step/sec: 1.7942 2021-09-25 23:19:33,168 [INFO] tensorflow: global_step/sec: 1.7942 INFO:tensorflow:epoch = 196.4, learning_rate = 0.00049999997, loss = 4.8891598e-05, step = 982 (5.629 sec) 2021-09-25 23:19:33,705 [INFO] tensorflow: epoch = 196.4, learning_rate = 0.00049999997, loss = 4.8891598e-05, step = 982 (5.629 sec) INFO:tensorflow:global_step/sec: 1.85806 2021-09-25 23:19:33,706 [INFO] tensorflow: global_step/sec: 1.85806 INFO:tensorflow:global_step/sec: 1.80441 2021-09-25 23:19:34,260 [INFO] tensorflow: global_step/sec: 1.80441 INFO:tensorflow:global_step/sec: 1.78221 2021-09-25 23:19:34,821 [INFO] tensorflow: global_step/sec: 1.78221 INFO:tensorflow:global_step/sec: 1.78209 2021-09-25 23:19:35,383 [INFO] tensorflow: global_step/sec: 1.78209 2021-09-25 23:19:35,384 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 197/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.782808 ETA: 0:14:03.190916 INFO:tensorflow:global_step/sec: 1.7794 2021-09-25 23:19:35,945 [INFO] tensorflow: global_step/sec: 1.7794 INFO:tensorflow:global_step/sec: 1.79281 2021-09-25 23:19:36,502 [INFO] tensorflow: global_step/sec: 1.79281 INFO:tensorflow:global_step/sec: 1.76801 2021-09-25 23:19:37,068 [INFO] tensorflow: global_step/sec: 1.76801 INFO:tensorflow:global_step/sec: 1.82864 2021-09-25 23:19:37,615 [INFO] tensorflow: global_step/sec: 1.82864 INFO:tensorflow:global_step/sec: 1.74692 2021-09-25 23:19:38,187 [INFO] tensorflow: global_step/sec: 1.74692 2021-09-25 23:19:38,189 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 198/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.794390 ETA: 0:14:03.905913 INFO:tensorflow:global_step/sec: 1.80045 2021-09-25 23:19:38,743 [INFO] tensorflow: global_step/sec: 1.80045 INFO:tensorflow:epoch = 198.4, learning_rate = 0.00049999997, loss = 3.9899493e-05, step = 992 (5.581 sec) 2021-09-25 23:19:39,286 [INFO] tensorflow: epoch = 198.4, learning_rate = 0.00049999997, loss = 3.9899493e-05, step = 992 (5.581 sec) INFO:tensorflow:global_step/sec: 1.83693 2021-09-25 23:19:39,287 [INFO] tensorflow: global_step/sec: 1.83693 INFO:tensorflow:global_step/sec: 1.78634 2021-09-25 23:19:39,847 [INFO] tensorflow: global_step/sec: 1.78634 INFO:tensorflow:global_step/sec: 1.82861 2021-09-25 23:19:40,394 [INFO] tensorflow: global_step/sec: 1.82861 INFO:tensorflow:global_step/sec: 1.80016 2021-09-25 23:19:40,949 [INFO] tensorflow: global_step/sec: 1.80016 2021-09-25 23:19:40,951 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 199/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.775015 ETA: 0:13:55.279622 INFO:tensorflow:global_step/sec: 1.78066 2021-09-25 23:19:41,511 [INFO] tensorflow: global_step/sec: 1.78066 INFO:tensorflow:global_step/sec: 1.6919 2021-09-25 23:19:42,102 [INFO] tensorflow: global_step/sec: 1.6919 INFO:tensorflow:global_step/sec: 1.77896 2021-09-25 23:19:42,664 [INFO] tensorflow: global_step/sec: 1.77896 INFO:tensorflow:global_step/sec: 1.79318 2021-09-25 23:19:43,222 [INFO] tensorflow: global_step/sec: 1.79318 INFO:tensorflow:Saving checkpoints for step-1000. 2021-09-25 23:19:43,223 [INFO] tensorflow: Saving checkpoints for step-1000. WARNING:tensorflow:Ignoring: /tmp/tmpcply2eb_; No such file or directory 2021-09-25 23:19:43,443 [WARNING] tensorflow: Ignoring: /tmp/tmpcply2eb_; No such file or directory 2021-09-25 23:19:47,259 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 88.738 INFO:tensorflow:epoch = 200.0, learning_rate = 0.00049999997, loss = 4.126194e-05, step = 1000 (8.543 sec) 2021-09-25 23:19:47,829 [INFO] tensorflow: epoch = 200.0, learning_rate = 0.00049999997, loss = 4.126194e-05, step = 1000 (8.543 sec) INFO:tensorflow:global_step/sec: 0.216974 2021-09-25 23:19:47,831 [INFO] tensorflow: global_step/sec: 0.216974 2021-09-25 23:19:47,832 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 200/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:06.860865 ETA: 0:34:18.259535 INFO:tensorflow:global_step/sec: 1.75445 2021-09-25 23:19:48,401 [INFO] tensorflow: global_step/sec: 1.75445 INFO:tensorflow:global_step/sec: 1.77772 2021-09-25 23:19:48,963 [INFO] tensorflow: global_step/sec: 1.77772 INFO:tensorflow:global_step/sec: 1.70325 2021-09-25 23:19:49,550 [INFO] tensorflow: global_step/sec: 1.70325 INFO:tensorflow:global_step/sec: 1.84348 2021-09-25 23:19:50,093 [INFO] tensorflow: global_step/sec: 1.84348 Epoch 201/500 ========================= Validation cost: 0.000000 Mean average_precision (in %): 0.0000 class name average precision (in %) ------------ -------------------------- lpd 0 Median Inference Time: nan INFO:tensorflow:global_step/sec: 1.74799 2021-09-25 23:19:50,665 [INFO] tensorflow: global_step/sec: 1.74799 2021-09-25 23:19:50,666 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 201/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.838303 ETA: 0:14:08.652552 INFO:tensorflow:global_step/sec: 1.83095 2021-09-25 23:19:51,211 [INFO] tensorflow: global_step/sec: 1.83095 INFO:tensorflow:global_step/sec: 1.84451 2021-09-25 23:19:51,753 [INFO] tensorflow: global_step/sec: 1.84451 INFO:tensorflow:global_step/sec: 1.80916 2021-09-25 23:19:52,306 [INFO] tensorflow: global_step/sec: 1.80916 INFO:tensorflow:global_step/sec: 1.78757 2021-09-25 23:19:52,865 [INFO] tensorflow: global_step/sec: 1.78757 INFO:tensorflow:epoch = 202.0, learning_rate = 0.00049999997, loss = 4.2139694e-05, step = 1010 (5.616 sec) 2021-09-25 23:19:53,445 [INFO] tensorflow: epoch = 202.0, learning_rate = 0.00049999997, loss = 4.2139694e-05, step = 1010 (5.616 sec) INFO:tensorflow:global_step/sec: 1.72028 2021-09-25 23:19:53,446 [INFO] tensorflow: global_step/sec: 1.72028 2021-09-25 23:19:53,448 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 202/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.768299 ETA: 0:13:44.953133 INFO:tensorflow:global_step/sec: 1.73148 2021-09-25 23:19:54,024 [INFO] tensorflow: global_step/sec: 1.73148 INFO:tensorflow:global_step/sec: 1.71017 2021-09-25 23:19:54,609 [INFO] tensorflow: global_step/sec: 1.71017 INFO:tensorflow:global_step/sec: 1.83434 2021-09-25 23:19:55,154 [INFO] tensorflow: global_step/sec: 1.83434 INFO:tensorflow:global_step/sec: 1.78413 2021-09-25 23:19:55,714 [INFO] tensorflow: global_step/sec: 1.78413 INFO:tensorflow:global_step/sec: 1.81681 2021-09-25 23:19:56,265 [INFO] tensorflow: global_step/sec: 1.81681 2021-09-25 23:19:56,266 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 203/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.850872 ETA: 0:14:06.708996 INFO:tensorflow:global_step/sec: 1.67425 2021-09-25 23:19:56,862 [INFO] tensorflow: global_step/sec: 1.67425 INFO:tensorflow:global_step/sec: 1.77731 2021-09-25 23:19:57,425 [INFO] tensorflow: global_step/sec: 1.77731 INFO:tensorflow:global_step/sec: 1.79358 2021-09-25 23:19:57,982 [INFO] tensorflow: global_step/sec: 1.79358 INFO:tensorflow:global_step/sec: 1.76735 2021-09-25 23:19:58,548 [INFO] tensorflow: global_step/sec: 1.76735 INFO:tensorflow:epoch = 204.0, learning_rate = 0.00049999997, loss = 3.981275e-05, step = 1020 (5.658 sec) 2021-09-25 23:19:59,103 [INFO] tensorflow: epoch = 204.0, learning_rate = 0.00049999997, loss = 3.981275e-05, step = 1020 (5.658 sec) INFO:tensorflow:global_step/sec: 1.79738 2021-09-25 23:19:59,104 [INFO] tensorflow: global_step/sec: 1.79738 2021-09-25 23:19:59,106 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 204/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.832317 ETA: 0:13:58.365866 INFO:tensorflow:global_step/sec: 1.78416 2021-09-25 23:19:59,665 [INFO] tensorflow: global_step/sec: 1.78416 INFO:tensorflow:global_step/sec: 1.81228 2021-09-25 23:20:00,217 [INFO] tensorflow: global_step/sec: 1.81228 INFO:tensorflow:global_step/sec: 1.68992 2021-09-25 23:20:00,809 [INFO] tensorflow: global_step/sec: 1.68992 INFO:tensorflow:global_step/sec: 1.73399 2021-09-25 23:20:01,385 [INFO] tensorflow: global_step/sec: 1.73399 2021-09-25 23:20:01,388 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 113.270 INFO:tensorflow:global_step/sec: 1.7406 2021-09-25 23:20:01,960 [INFO] tensorflow: global_step/sec: 1.7406 2021-09-25 23:20:01,961 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 205/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.832779 ETA: 0:13:55.669925 INFO:tensorflow:global_step/sec: 1.74765 2021-09-25 23:20:02,532 [INFO] tensorflow: global_step/sec: 1.74765 INFO:tensorflow:global_step/sec: 1.83351 2021-09-25 23:20:03,077 [INFO] tensorflow: global_step/sec: 1.83351 INFO:tensorflow:global_step/sec: 1.77178 2021-09-25 23:20:03,642 [INFO] tensorflow: global_step/sec: 1.77178 INFO:tensorflow:global_step/sec: 1.78719 2021-09-25 23:20:04,201 [INFO] tensorflow: global_step/sec: 1.78719 INFO:tensorflow:epoch = 206.0, learning_rate = 0.00049999997, loss = 4.0844858e-05, step = 1030 (5.667 sec) 2021-09-25 23:20:04,770 [INFO] tensorflow: epoch = 206.0, learning_rate = 0.00049999997, loss = 4.0844858e-05, step = 1030 (5.667 sec) INFO:tensorflow:global_step/sec: 1.75375 2021-09-25 23:20:04,771 [INFO] tensorflow: global_step/sec: 1.75375 2021-09-25 23:20:04,773 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 206/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.819843 ETA: 0:13:49.033718 INFO:tensorflow:global_step/sec: 1.74881 2021-09-25 23:20:05,343 [INFO] tensorflow: global_step/sec: 1.74881 INFO:tensorflow:global_step/sec: 1.74267 2021-09-25 23:20:05,917 [INFO] tensorflow: global_step/sec: 1.74267 INFO:tensorflow:global_step/sec: 1.80617 2021-09-25 23:20:06,471 [INFO] tensorflow: global_step/sec: 1.80617 INFO:tensorflow:global_step/sec: 1.82471 2021-09-25 23:20:07,019 [INFO] tensorflow: global_step/sec: 1.82471 INFO:tensorflow:global_step/sec: 1.78656 2021-09-25 23:20:07,579 [INFO] tensorflow: global_step/sec: 1.78656 2021-09-25 23:20:07,580 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 207/500: loss: 0.00005 learning rate: 0.00050 Time taken: 0:00:02.812348 ETA: 0:13:44.018001 INFO:tensorflow:global_step/sec: 1.80226 2021-09-25 23:20:08,133 [INFO] tensorflow: global_step/sec: 1.80226 INFO:tensorflow:global_step/sec: 1.81074 2021-09-25 23:20:08,686 [INFO] tensorflow: global_step/sec: 1.81074 INFO:tensorflow:global_step/sec: 1.7485 2021-09-25 23:20:09,258 [INFO] tensorflow: global_step/sec: 1.7485 INFO:tensorflow:global_step/sec: 1.77611 2021-09-25 23:20:09,821 [INFO] tensorflow: global_step/sec: 1.77611 INFO:tensorflow:epoch = 208.0, learning_rate = 0.00049999997, loss = 4.1337196e-05, step = 1040 (5.610 sec) 2021-09-25 23:20:10,380 [INFO] tensorflow: epoch = 208.0, learning_rate = 0.00049999997, loss = 4.1337196e-05, step = 1040 (5.610 sec) INFO:tensorflow:global_step/sec: 1.78257 2021-09-25 23:20:10,382 [INFO] tensorflow: global_step/sec: 1.78257 2021-09-25 23:20:10,383 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 208/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.803759 ETA: 0:13:38.697726 INFO:tensorflow:global_step/sec: 1.8106 2021-09-25 23:20:10,934 [INFO] tensorflow: global_step/sec: 1.8106 INFO:tensorflow:global_step/sec: 1.7306 2021-09-25 23:20:11,512 [INFO] tensorflow: global_step/sec: 1.7306 INFO:tensorflow:global_step/sec: 1.74069 2021-09-25 23:20:12,086 [INFO] tensorflow: global_step/sec: 1.74069 INFO:tensorflow:global_step/sec: 1.82032 2021-09-25 23:20:12,636 [INFO] tensorflow: global_step/sec: 1.82032 INFO:tensorflow:global_step/sec: 1.75336 2021-09-25 23:20:13,206 [INFO] tensorflow: global_step/sec: 1.75336 2021-09-25 23:20:13,207 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 209/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.813790 ETA: 0:13:38.812775 INFO:tensorflow:global_step/sec: 1.79301 2021-09-25 23:20:13,764 [INFO] tensorflow: global_step/sec: 1.79301 INFO:tensorflow:global_step/sec: 1.71389 2021-09-25 23:20:14,347 [INFO] tensorflow: global_step/sec: 1.71389 INFO:tensorflow:global_step/sec: 1.72797 2021-09-25 23:20:14,926 [INFO] tensorflow: global_step/sec: 1.72797 INFO:tensorflow:global_step/sec: 1.85854 2021-09-25 23:20:15,464 [INFO] tensorflow: global_step/sec: 1.85854 2021-09-25 23:20:15,469 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 113.635 INFO:tensorflow:epoch = 210.0, learning_rate = 0.00049999997, loss = 4.088131e-05, step = 1050 (5.626 sec) 2021-09-25 23:20:16,006 [INFO] tensorflow: epoch = 210.0, learning_rate = 0.00049999997, loss = 4.088131e-05, step = 1050 (5.626 sec) INFO:tensorflow:global_step/sec: 1.83951 2021-09-25 23:20:16,007 [INFO] tensorflow: global_step/sec: 1.83951 2021-09-25 23:20:16,009 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 210/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.827331 ETA: 0:13:39.925871 INFO:tensorflow:global_step/sec: 1.82944 2021-09-25 23:20:16,554 [INFO] tensorflow: global_step/sec: 1.82944 INFO:tensorflow:global_step/sec: 1.78448 2021-09-25 23:20:17,114 [INFO] tensorflow: global_step/sec: 1.78448 INFO:tensorflow:global_step/sec: 1.80502 2021-09-25 23:20:17,669 [INFO] tensorflow: global_step/sec: 1.80502 INFO:tensorflow:global_step/sec: 1.73972 2021-09-25 23:20:18,243 [INFO] tensorflow: global_step/sec: 1.73972 INFO:tensorflow:global_step/sec: 1.76291 2021-09-25 23:20:18,811 [INFO] tensorflow: global_step/sec: 1.76291 2021-09-25 23:20:18,812 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 211/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.778062 ETA: 0:13:22.859811 INFO:tensorflow:global_step/sec: 1.77011 2021-09-25 23:20:19,376 [INFO] tensorflow: global_step/sec: 1.77011 INFO:tensorflow:global_step/sec: 1.83218 2021-09-25 23:20:19,921 [INFO] tensorflow: global_step/sec: 1.83218 INFO:tensorflow:global_step/sec: 1.76989 2021-09-25 23:20:20,486 [INFO] tensorflow: global_step/sec: 1.76989 INFO:tensorflow:global_step/sec: 1.79523 2021-09-25 23:20:21,043 [INFO] tensorflow: global_step/sec: 1.79523 INFO:tensorflow:epoch = 212.0, learning_rate = 0.00049999997, loss = 3.914346e-05, step = 1060 (5.596 sec) 2021-09-25 23:20:21,602 [INFO] tensorflow: epoch = 212.0, learning_rate = 0.00049999997, loss = 3.914346e-05, step = 1060 (5.596 sec) INFO:tensorflow:global_step/sec: 1.78457 2021-09-25 23:20:21,604 [INFO] tensorflow: global_step/sec: 1.78457 2021-09-25 23:20:21,605 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 212/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.797476 ETA: 0:13:25.673172 INFO:tensorflow:global_step/sec: 1.74213 2021-09-25 23:20:22,178 [INFO] tensorflow: global_step/sec: 1.74213 INFO:tensorflow:global_step/sec: 1.83948 2021-09-25 23:20:22,721 [INFO] tensorflow: global_step/sec: 1.83948 INFO:tensorflow:global_step/sec: 1.81208 2021-09-25 23:20:23,273 [INFO] tensorflow: global_step/sec: 1.81208 INFO:tensorflow:global_step/sec: 1.75587 2021-09-25 23:20:23,843 [INFO] tensorflow: global_step/sec: 1.75587 INFO:tensorflow:global_step/sec: 1.79896 2021-09-25 23:20:24,399 [INFO] tensorflow: global_step/sec: 1.79896 2021-09-25 23:20:24,400 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 213/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.801346 ETA: 0:13:23.986389 INFO:tensorflow:global_step/sec: 1.79444 2021-09-25 23:20:24,956 [INFO] tensorflow: global_step/sec: 1.79444 INFO:tensorflow:global_step/sec: 1.78873 2021-09-25 23:20:25,515 [INFO] tensorflow: global_step/sec: 1.78873 INFO:tensorflow:global_step/sec: 1.71543 2021-09-25 23:20:26,098 [INFO] tensorflow: global_step/sec: 1.71543 INFO:tensorflow:global_step/sec: 1.75586 2021-09-25 23:20:26,667 [INFO] tensorflow: global_step/sec: 1.75586 INFO:tensorflow:epoch = 214.0, learning_rate = 0.00049999997, loss = 3.7268357e-05, step = 1070 (5.613 sec) 2021-09-25 23:20:27,215 [INFO] tensorflow: epoch = 214.0, learning_rate = 0.00049999997, loss = 3.7268357e-05, step = 1070 (5.613 sec) INFO:tensorflow:global_step/sec: 1.82167 2021-09-25 23:20:27,216 [INFO] tensorflow: global_step/sec: 1.82167 2021-09-25 23:20:27,218 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 214/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.826169 ETA: 0:13:28.284270 INFO:tensorflow:global_step/sec: 1.79411 2021-09-25 23:20:27,774 [INFO] tensorflow: global_step/sec: 1.79411 INFO:tensorflow:global_step/sec: 1.79854 2021-09-25 23:20:28,330 [INFO] tensorflow: global_step/sec: 1.79854 INFO:tensorflow:global_step/sec: 1.84554 2021-09-25 23:20:28,872 [INFO] tensorflow: global_step/sec: 1.84554 INFO:tensorflow:global_step/sec: 1.75455 2021-09-25 23:20:29,442 [INFO] tensorflow: global_step/sec: 1.75455 2021-09-25 23:20:29,449 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 114.459 INFO:tensorflow:global_step/sec: 1.84632 2021-09-25 23:20:29,983 [INFO] tensorflow: global_step/sec: 1.84632 2021-09-25 23:20:29,985 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 215/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.772879 ETA: 0:13:10.270618 INFO:tensorflow:global_step/sec: 1.77855 2021-09-25 23:20:30,545 [INFO] tensorflow: global_step/sec: 1.77855 INFO:tensorflow:global_step/sec: 1.75858 2021-09-25 23:20:31,114 [INFO] tensorflow: global_step/sec: 1.75858 INFO:tensorflow:global_step/sec: 1.74046 2021-09-25 23:20:31,689 [INFO] tensorflow: global_step/sec: 1.74046 INFO:tensorflow:global_step/sec: 1.78887 2021-09-25 23:20:32,248 [INFO] tensorflow: global_step/sec: 1.78887 INFO:tensorflow:epoch = 216.0, learning_rate = 0.00049999997, loss = 4.5351786e-05, step = 1080 (5.639 sec) 2021-09-25 23:20:32,854 [INFO] tensorflow: epoch = 216.0, learning_rate = 0.00049999997, loss = 4.5351786e-05, step = 1080 (5.639 sec) INFO:tensorflow:global_step/sec: 1.64436 2021-09-25 23:20:32,856 [INFO] tensorflow: global_step/sec: 1.64436 2021-09-25 23:20:32,857 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 216/500: loss: 0.00005 learning rate: 0.00050 Time taken: 0:00:02.804928 ETA: 0:13:16.599435 INFO:tensorflow:global_step/sec: 1.82471 2021-09-25 23:20:33,404 [INFO] tensorflow: global_step/sec: 1.82471 INFO:tensorflow:global_step/sec: 1.71575 2021-09-25 23:20:33,987 [INFO] tensorflow: global_step/sec: 1.71575 INFO:tensorflow:global_step/sec: 1.8136 2021-09-25 23:20:34,538 [INFO] tensorflow: global_step/sec: 1.8136 INFO:tensorflow:global_step/sec: 1.77358 2021-09-25 23:20:35,102 [INFO] tensorflow: global_step/sec: 1.77358 INFO:tensorflow:global_step/sec: 1.79253 2021-09-25 23:20:35,660 [INFO] tensorflow: global_step/sec: 1.79253 2021-09-25 23:20:35,661 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 217/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.853608 ETA: 0:13:27.571169 INFO:tensorflow:global_step/sec: 1.76515 2021-09-25 23:20:36,226 [INFO] tensorflow: global_step/sec: 1.76515 INFO:tensorflow:global_step/sec: 1.81656 2021-09-25 23:20:36,777 [INFO] tensorflow: global_step/sec: 1.81656 INFO:tensorflow:global_step/sec: 1.77522 2021-09-25 23:20:37,340 [INFO] tensorflow: global_step/sec: 1.77522 INFO:tensorflow:global_step/sec: 1.77411 2021-09-25 23:20:37,904 [INFO] tensorflow: global_step/sec: 1.77411 INFO:tensorflow:epoch = 218.0, learning_rate = 0.00049999997, loss = 4.8636117e-05, step = 1090 (5.623 sec) 2021-09-25 23:20:38,477 [INFO] tensorflow: epoch = 218.0, learning_rate = 0.00049999997, loss = 4.8636117e-05, step = 1090 (5.623 sec) INFO:tensorflow:global_step/sec: 1.73918 2021-09-25 23:20:38,479 [INFO] tensorflow: global_step/sec: 1.73918 2021-09-25 23:20:38,480 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 218/500: loss: 0.00005 learning rate: 0.00050 Time taken: 0:00:02.800333 ETA: 0:13:09.693845 INFO:tensorflow:global_step/sec: 1.7649 2021-09-25 23:20:39,045 [INFO] tensorflow: global_step/sec: 1.7649 INFO:tensorflow:global_step/sec: 1.75803 2021-09-25 23:20:39,614 [INFO] tensorflow: global_step/sec: 1.75803 INFO:tensorflow:global_step/sec: 1.76202 2021-09-25 23:20:40,182 [INFO] tensorflow: global_step/sec: 1.76202 INFO:tensorflow:global_step/sec: 1.72761 2021-09-25 23:20:40,760 [INFO] tensorflow: global_step/sec: 1.72761 INFO:tensorflow:global_step/sec: 1.77213 2021-09-25 23:20:41,325 [INFO] tensorflow: global_step/sec: 1.77213 2021-09-25 23:20:41,326 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 219/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.855183 ETA: 0:13:22.306324 INFO:tensorflow:global_step/sec: 1.76953 2021-09-25 23:20:41,890 [INFO] tensorflow: global_step/sec: 1.76953 INFO:tensorflow:global_step/sec: 1.75673 2021-09-25 23:20:42,459 [INFO] tensorflow: global_step/sec: 1.75673 INFO:tensorflow:global_step/sec: 1.79489 2021-09-25 23:20:43,016 [INFO] tensorflow: global_step/sec: 1.79489 INFO:tensorflow:global_step/sec: 1.72227 2021-09-25 23:20:43,597 [INFO] tensorflow: global_step/sec: 1.72227 INFO:tensorflow:Saving checkpoints for step-1100. 2021-09-25 23:20:43,598 [INFO] tensorflow: Saving checkpoints for step-1100. WARNING:tensorflow:Ignoring: /tmp/tmpxndqs_at; No such file or directory 2021-09-25 23:20:43,817 [WARNING] tensorflow: Ignoring: /tmp/tmpxndqs_at; No such file or directory 2021-09-25 23:20:47,656 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 87.881 INFO:tensorflow:epoch = 220.0, learning_rate = 0.00049999997, loss = 3.7459213e-05, step = 1100 (9.735 sec) 2021-09-25 23:20:48,213 [INFO] tensorflow: epoch = 220.0, learning_rate = 0.00049999997, loss = 3.7459213e-05, step = 1100 (9.735 sec) INFO:tensorflow:global_step/sec: 0.216584 2021-09-25 23:20:48,214 [INFO] tensorflow: global_step/sec: 0.216584 2021-09-25 23:20:48,215 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 220/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:06.889599 ETA: 0:32:09.087744 INFO:tensorflow:global_step/sec: 1.83604 2021-09-25 23:20:48,759 [INFO] tensorflow: global_step/sec: 1.83604 INFO:tensorflow:global_step/sec: 1.81884 2021-09-25 23:20:49,308 [INFO] tensorflow: global_step/sec: 1.81884 INFO:tensorflow:global_step/sec: 1.77516 2021-09-25 23:20:49,872 [INFO] tensorflow: global_step/sec: 1.77516 INFO:tensorflow:global_step/sec: 1.75282 2021-09-25 23:20:50,442 [INFO] tensorflow: global_step/sec: 1.75282 Epoch 221/500 ========================= Validation cost: 0.000000 Mean average_precision (in %): 0.0000 class name average precision (in %) ------------ -------------------------- lpd 0 Median Inference Time: nan INFO:tensorflow:global_step/sec: 1.77736 2021-09-25 23:20:51,005 [INFO] tensorflow: global_step/sec: 1.77736 2021-09-25 23:20:51,007 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 221/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.789421 ETA: 0:12:58.248349 INFO:tensorflow:global_step/sec: 1.70757 2021-09-25 23:20:51,591 [INFO] tensorflow: global_step/sec: 1.70757 INFO:tensorflow:global_step/sec: 1.79589 2021-09-25 23:20:52,147 [INFO] tensorflow: global_step/sec: 1.79589 INFO:tensorflow:global_step/sec: 1.74026 2021-09-25 23:20:52,722 [INFO] tensorflow: global_step/sec: 1.74026 INFO:tensorflow:global_step/sec: 1.75453 2021-09-25 23:20:53,292 [INFO] tensorflow: global_step/sec: 1.75453 INFO:tensorflow:epoch = 222.0, learning_rate = 0.00049999997, loss = 3.7678026e-05, step = 1110 (5.659 sec) 2021-09-25 23:20:53,872 [INFO] tensorflow: epoch = 222.0, learning_rate = 0.00049999997, loss = 3.7678026e-05, step = 1110 (5.659 sec) INFO:tensorflow:global_step/sec: 1.72046 2021-09-25 23:20:53,873 [INFO] tensorflow: global_step/sec: 1.72046 2021-09-25 23:20:53,875 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 222/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.852074 ETA: 0:13:12.876480 INFO:tensorflow:global_step/sec: 1.76906 2021-09-25 23:20:54,438 [INFO] tensorflow: global_step/sec: 1.76906 INFO:tensorflow:global_step/sec: 1.67707 2021-09-25 23:20:55,035 [INFO] tensorflow: global_step/sec: 1.67707 INFO:tensorflow:global_step/sec: 1.78854 2021-09-25 23:20:55,594 [INFO] tensorflow: global_step/sec: 1.78854 INFO:tensorflow:global_step/sec: 1.77219 2021-09-25 23:20:56,158 [INFO] tensorflow: global_step/sec: 1.77219 INFO:tensorflow:global_step/sec: 1.68249 2021-09-25 23:20:56,752 [INFO] tensorflow: global_step/sec: 1.68249 2021-09-25 23:20:56,754 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 223/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.862886 ETA: 0:13:13.019541 INFO:tensorflow:global_step/sec: 1.79341 2021-09-25 23:20:57,310 [INFO] tensorflow: global_step/sec: 1.79341 INFO:tensorflow:global_step/sec: 1.71821 2021-09-25 23:20:57,892 [INFO] tensorflow: global_step/sec: 1.71821 INFO:tensorflow:global_step/sec: 1.78084 2021-09-25 23:20:58,454 [INFO] tensorflow: global_step/sec: 1.78084 INFO:tensorflow:global_step/sec: 1.80219 2021-09-25 23:20:59,009 [INFO] tensorflow: global_step/sec: 1.80219 INFO:tensorflow:epoch = 224.0, learning_rate = 0.00049999997, loss = 3.8755596e-05, step = 1120 (5.694 sec) 2021-09-25 23:20:59,566 [INFO] tensorflow: epoch = 224.0, learning_rate = 0.00049999997, loss = 3.8755596e-05, step = 1120 (5.694 sec) INFO:tensorflow:global_step/sec: 1.79026 2021-09-25 23:20:59,567 [INFO] tensorflow: global_step/sec: 1.79026 2021-09-25 23:20:59,568 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 224/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.848699 ETA: 0:13:06.241015 INFO:tensorflow:global_step/sec: 1.77275 2021-09-25 23:21:00,131 [INFO] tensorflow: global_step/sec: 1.77275 INFO:tensorflow:global_step/sec: 1.78423 2021-09-25 23:21:00,692 [INFO] tensorflow: global_step/sec: 1.78423 INFO:tensorflow:global_step/sec: 1.73929 2021-09-25 23:21:01,267 [INFO] tensorflow: global_step/sec: 1.73929 INFO:tensorflow:global_step/sec: 1.86808 2021-09-25 23:21:01,802 [INFO] tensorflow: global_step/sec: 1.86808 2021-09-25 23:21:01,805 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 113.089 INFO:tensorflow:global_step/sec: 1.76427 2021-09-25 23:21:02,369 [INFO] tensorflow: global_step/sec: 1.76427 2021-09-25 23:21:02,370 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 225/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.792041 ETA: 0:12:47.811358 INFO:tensorflow:global_step/sec: 1.78821 2021-09-25 23:21:02,928 [INFO] tensorflow: global_step/sec: 1.78821 INFO:tensorflow:global_step/sec: 1.79876 2021-09-25 23:21:03,484 [INFO] tensorflow: global_step/sec: 1.79876 INFO:tensorflow:global_step/sec: 1.85657 2021-09-25 23:21:04,023 [INFO] tensorflow: global_step/sec: 1.85657 INFO:tensorflow:global_step/sec: 1.75147 2021-09-25 23:21:04,593 [INFO] tensorflow: global_step/sec: 1.75147 INFO:tensorflow:epoch = 226.0, learning_rate = 0.00049999997, loss = 4.2711014e-05, step = 1130 (5.597 sec) 2021-09-25 23:21:05,163 [INFO] tensorflow: epoch = 226.0, learning_rate = 0.00049999997, loss = 4.2711014e-05, step = 1130 (5.597 sec) INFO:tensorflow:global_step/sec: 1.75283 2021-09-25 23:21:05,164 [INFO] tensorflow: global_step/sec: 1.75283 2021-09-25 23:21:05,165 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 226/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.792118 ETA: 0:12:45.040287 INFO:tensorflow:global_step/sec: 1.73887 2021-09-25 23:21:05,739 [INFO] tensorflow: global_step/sec: 1.73887 INFO:tensorflow:global_step/sec: 1.81441 2021-09-25 23:21:06,290 [INFO] tensorflow: global_step/sec: 1.81441 INFO:tensorflow:global_step/sec: 1.81019 2021-09-25 23:21:06,843 [INFO] tensorflow: global_step/sec: 1.81019 INFO:tensorflow:global_step/sec: 1.82319 2021-09-25 23:21:07,391 [INFO] tensorflow: global_step/sec: 1.82319 INFO:tensorflow:global_step/sec: 1.72406 2021-09-25 23:21:07,971 [INFO] tensorflow: global_step/sec: 1.72406 2021-09-25 23:21:07,973 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 227/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.799181 ETA: 0:12:44.176539 INFO:tensorflow:global_step/sec: 1.73729 2021-09-25 23:21:08,547 [INFO] tensorflow: global_step/sec: 1.73729 INFO:tensorflow:global_step/sec: 1.84133 2021-09-25 23:21:09,090 [INFO] tensorflow: global_step/sec: 1.84133 INFO:tensorflow:global_step/sec: 1.82757 2021-09-25 23:21:09,637 [INFO] tensorflow: global_step/sec: 1.82757 INFO:tensorflow:global_step/sec: 1.77043 2021-09-25 23:21:10,202 [INFO] tensorflow: global_step/sec: 1.77043 INFO:tensorflow:epoch = 228.0, learning_rate = 0.00049999997, loss = 3.8533326e-05, step = 1140 (5.588 sec) 2021-09-25 23:21:10,751 [INFO] tensorflow: epoch = 228.0, learning_rate = 0.00049999997, loss = 3.8533326e-05, step = 1140 (5.588 sec) INFO:tensorflow:global_step/sec: 1.81704 2021-09-25 23:21:10,752 [INFO] tensorflow: global_step/sec: 1.81704 2021-09-25 23:21:10,753 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 228/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.806924 ETA: 0:12:43.483227 INFO:tensorflow:global_step/sec: 1.80377 2021-09-25 23:21:11,307 [INFO] tensorflow: global_step/sec: 1.80377 INFO:tensorflow:global_step/sec: 1.83908 2021-09-25 23:21:11,850 [INFO] tensorflow: global_step/sec: 1.83908 INFO:tensorflow:global_step/sec: 1.79519 2021-09-25 23:21:12,407 [INFO] tensorflow: global_step/sec: 1.79519 INFO:tensorflow:global_step/sec: 1.73524 2021-09-25 23:21:12,984 [INFO] tensorflow: global_step/sec: 1.73524 INFO:tensorflow:global_step/sec: 1.82098 2021-09-25 23:21:13,533 [INFO] tensorflow: global_step/sec: 1.82098 2021-09-25 23:21:13,534 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 229/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.780067 ETA: 0:12:33.398213 INFO:tensorflow:global_step/sec: 1.70474 2021-09-25 23:21:14,119 [INFO] tensorflow: global_step/sec: 1.70474 INFO:tensorflow:global_step/sec: 1.73853 2021-09-25 23:21:14,695 [INFO] tensorflow: global_step/sec: 1.73853 INFO:tensorflow:global_step/sec: 1.74339 2021-09-25 23:21:15,268 [INFO] tensorflow: global_step/sec: 1.74339 INFO:tensorflow:global_step/sec: 1.8125 2021-09-25 23:21:15,820 [INFO] tensorflow: global_step/sec: 1.8125 2021-09-25 23:21:15,826 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 114.123 INFO:tensorflow:epoch = 230.0, learning_rate = 0.00049999997, loss = 4.1307405e-05, step = 1150 (5.625 sec) 2021-09-25 23:21:16,376 [INFO] tensorflow: epoch = 230.0, learning_rate = 0.00049999997, loss = 4.1307405e-05, step = 1150 (5.625 sec) INFO:tensorflow:global_step/sec: 1.79459 2021-09-25 23:21:16,377 [INFO] tensorflow: global_step/sec: 1.79459 2021-09-25 23:21:16,378 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 230/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.838678 ETA: 0:12:46.443093 INFO:tensorflow:global_step/sec: 1.74638 2021-09-25 23:21:16,950 [INFO] tensorflow: global_step/sec: 1.74638 INFO:tensorflow:global_step/sec: 1.8085 2021-09-25 23:21:17,503 [INFO] tensorflow: global_step/sec: 1.8085 INFO:tensorflow:global_step/sec: 1.79248 2021-09-25 23:21:18,061 [INFO] tensorflow: global_step/sec: 1.79248 INFO:tensorflow:global_step/sec: 1.78451 2021-09-25 23:21:18,621 [INFO] tensorflow: global_step/sec: 1.78451 INFO:tensorflow:global_step/sec: 1.78905 2021-09-25 23:21:19,180 [INFO] tensorflow: global_step/sec: 1.78905 2021-09-25 23:21:19,181 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 231/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.800768 ETA: 0:12:33.406501 INFO:tensorflow:global_step/sec: 1.72675 2021-09-25 23:21:19,759 [INFO] tensorflow: global_step/sec: 1.72675 INFO:tensorflow:global_step/sec: 1.72466 2021-09-25 23:21:20,339 [INFO] tensorflow: global_step/sec: 1.72466 INFO:tensorflow:global_step/sec: 1.75667 2021-09-25 23:21:20,908 [INFO] tensorflow: global_step/sec: 1.75667 INFO:tensorflow:global_step/sec: 1.76451 2021-09-25 23:21:21,475 [INFO] tensorflow: global_step/sec: 1.76451 INFO:tensorflow:epoch = 232.0, learning_rate = 0.00049999997, loss = 3.859796e-05, step = 1160 (5.670 sec) 2021-09-25 23:21:22,046 [INFO] tensorflow: epoch = 232.0, learning_rate = 0.00049999997, loss = 3.859796e-05, step = 1160 (5.670 sec) INFO:tensorflow:global_step/sec: 1.74819 2021-09-25 23:21:22,047 [INFO] tensorflow: global_step/sec: 1.74819 2021-09-25 23:21:22,048 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 232/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.851606 ETA: 0:12:44.230507 INFO:tensorflow:global_step/sec: 1.81607 2021-09-25 23:21:22,598 [INFO] tensorflow: global_step/sec: 1.81607 INFO:tensorflow:global_step/sec: 1.80045 2021-09-25 23:21:23,153 [INFO] tensorflow: global_step/sec: 1.80045 INFO:tensorflow:global_step/sec: 1.76866 2021-09-25 23:21:23,718 [INFO] tensorflow: global_step/sec: 1.76866 INFO:tensorflow:global_step/sec: 1.79398 2021-09-25 23:21:24,276 [INFO] tensorflow: global_step/sec: 1.79398 INFO:tensorflow:global_step/sec: 1.77691 2021-09-25 23:21:24,839 [INFO] tensorflow: global_step/sec: 1.77691 2021-09-25 23:21:24,840 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 233/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.800601 ETA: 0:12:27.760468 INFO:tensorflow:global_step/sec: 1.77305 2021-09-25 23:21:25,403 [INFO] tensorflow: global_step/sec: 1.77305 INFO:tensorflow:global_step/sec: 1.81649 2021-09-25 23:21:25,953 [INFO] tensorflow: global_step/sec: 1.81649 INFO:tensorflow:global_step/sec: 1.74365 2021-09-25 23:21:26,527 [INFO] tensorflow: global_step/sec: 1.74365 INFO:tensorflow:global_step/sec: 1.79538 2021-09-25 23:21:27,084 [INFO] tensorflow: global_step/sec: 1.79538 INFO:tensorflow:epoch = 234.0, learning_rate = 0.00049999997, loss = 5.0051167e-05, step = 1170 (5.590 sec) 2021-09-25 23:21:27,635 [INFO] tensorflow: epoch = 234.0, learning_rate = 0.00049999997, loss = 5.0051167e-05, step = 1170 (5.590 sec) INFO:tensorflow:global_step/sec: 1.80792 2021-09-25 23:21:27,637 [INFO] tensorflow: global_step/sec: 1.80792 2021-09-25 23:21:27,638 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 234/500: loss: 0.00005 learning rate: 0.00050 Time taken: 0:00:02.808989 ETA: 0:12:27.190960 INFO:tensorflow:global_step/sec: 1.83145 2021-09-25 23:21:28,183 [INFO] tensorflow: global_step/sec: 1.83145 INFO:tensorflow:global_step/sec: 1.73395 2021-09-25 23:21:28,759 [INFO] tensorflow: global_step/sec: 1.73395 INFO:tensorflow:global_step/sec: 1.78231 2021-09-25 23:21:29,320 [INFO] tensorflow: global_step/sec: 1.78231 INFO:tensorflow:global_step/sec: 1.78698 2021-09-25 23:21:29,880 [INFO] tensorflow: global_step/sec: 1.78698 2021-09-25 23:21:29,887 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 113.795 INFO:tensorflow:global_step/sec: 1.76196 2021-09-25 23:21:30,448 [INFO] tensorflow: global_step/sec: 1.76196 2021-09-25 23:21:30,449 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 235/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.795499 ETA: 0:12:20.807132 INFO:tensorflow:global_step/sec: 1.81492 2021-09-25 23:21:30,999 [INFO] tensorflow: global_step/sec: 1.81492 INFO:tensorflow:global_step/sec: 1.77379 2021-09-25 23:21:31,562 [INFO] tensorflow: global_step/sec: 1.77379 INFO:tensorflow:global_step/sec: 1.78143 2021-09-25 23:21:32,124 [INFO] tensorflow: global_step/sec: 1.78143 INFO:tensorflow:global_step/sec: 1.79347 2021-09-25 23:21:32,681 [INFO] tensorflow: global_step/sec: 1.79347 INFO:tensorflow:epoch = 236.0, learning_rate = 0.00049999997, loss = 3.7092705e-05, step = 1180 (5.596 sec) 2021-09-25 23:21:33,231 [INFO] tensorflow: epoch = 236.0, learning_rate = 0.00049999997, loss = 3.7092705e-05, step = 1180 (5.596 sec) INFO:tensorflow:global_step/sec: 1.81493 2021-09-25 23:21:33,232 [INFO] tensorflow: global_step/sec: 1.81493 2021-09-25 23:21:33,234 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 236/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.795940 ETA: 0:12:18.128077 INFO:tensorflow:global_step/sec: 1.84874 2021-09-25 23:21:33,773 [INFO] tensorflow: global_step/sec: 1.84874 INFO:tensorflow:global_step/sec: 1.70444 2021-09-25 23:21:34,360 [INFO] tensorflow: global_step/sec: 1.70444 INFO:tensorflow:global_step/sec: 1.83172 2021-09-25 23:21:34,906 [INFO] tensorflow: global_step/sec: 1.83172 INFO:tensorflow:global_step/sec: 1.86959 2021-09-25 23:21:35,441 [INFO] tensorflow: global_step/sec: 1.86959 INFO:tensorflow:global_step/sec: 1.76858 2021-09-25 23:21:36,006 [INFO] tensorflow: global_step/sec: 1.76858 2021-09-25 23:21:36,008 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 237/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.762868 ETA: 0:12:06.634265 INFO:tensorflow:global_step/sec: 1.82533 2021-09-25 23:21:36,554 [INFO] tensorflow: global_step/sec: 1.82533 INFO:tensorflow:global_step/sec: 1.7984 2021-09-25 23:21:37,110 [INFO] tensorflow: global_step/sec: 1.7984 INFO:tensorflow:global_step/sec: 1.74182 2021-09-25 23:21:37,684 [INFO] tensorflow: global_step/sec: 1.74182 INFO:tensorflow:global_step/sec: 1.81667 2021-09-25 23:21:38,235 [INFO] tensorflow: global_step/sec: 1.81667 INFO:tensorflow:epoch = 238.0, learning_rate = 0.00049999997, loss = 3.8950482e-05, step = 1190 (5.561 sec) 2021-09-25 23:21:38,792 [INFO] tensorflow: epoch = 238.0, learning_rate = 0.00049999997, loss = 3.8950482e-05, step = 1190 (5.561 sec) INFO:tensorflow:global_step/sec: 1.78811 2021-09-25 23:21:38,794 [INFO] tensorflow: global_step/sec: 1.78811 2021-09-25 23:21:38,795 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 238/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.791925 ETA: 0:12:11.484338 INFO:tensorflow:global_step/sec: 1.81834 2021-09-25 23:21:39,344 [INFO] tensorflow: global_step/sec: 1.81834 INFO:tensorflow:global_step/sec: 1.82566 2021-09-25 23:21:39,892 [INFO] tensorflow: global_step/sec: 1.82566 INFO:tensorflow:global_step/sec: 1.80077 2021-09-25 23:21:40,447 [INFO] tensorflow: global_step/sec: 1.80077 INFO:tensorflow:global_step/sec: 1.72558 2021-09-25 23:21:41,026 [INFO] tensorflow: global_step/sec: 1.72558 INFO:tensorflow:global_step/sec: 1.64247 2021-09-25 23:21:41,635 [INFO] tensorflow: global_step/sec: 1.64247 2021-09-25 23:21:41,637 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 239/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.788237 ETA: 0:12:07.729944 INFO:tensorflow:global_step/sec: 1.78627 2021-09-25 23:21:42,195 [INFO] tensorflow: global_step/sec: 1.78627 INFO:tensorflow:global_step/sec: 1.79658 2021-09-25 23:21:42,752 [INFO] tensorflow: global_step/sec: 1.79658 INFO:tensorflow:global_step/sec: 1.77744 2021-09-25 23:21:43,314 [INFO] tensorflow: global_step/sec: 1.77744 INFO:tensorflow:global_step/sec: 1.78673 2021-09-25 23:21:43,874 [INFO] tensorflow: global_step/sec: 1.78673 INFO:tensorflow:Saving checkpoints for step-1200. 2021-09-25 23:21:43,875 [INFO] tensorflow: Saving checkpoints for step-1200. WARNING:tensorflow:Ignoring: /tmp/tmpnmxbkxvp; No such file or directory 2021-09-25 23:21:44,098 [WARNING] tensorflow: Ignoring: /tmp/tmpnmxbkxvp; No such file or directory 2021-09-25 23:21:47,897 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 88.847 INFO:tensorflow:epoch = 240.0, learning_rate = 0.00049999997, loss = 3.8900507e-05, step = 1200 (9.677 sec) 2021-09-25 23:21:48,469 [INFO] tensorflow: epoch = 240.0, learning_rate = 0.00049999997, loss = 3.8900507e-05, step = 1200 (9.677 sec) INFO:tensorflow:global_step/sec: 0.217543 2021-09-25 23:21:48,471 [INFO] tensorflow: global_step/sec: 0.217543 2021-09-25 23:21:48,472 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 240/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:06.866653 ETA: 0:29:45.329709 INFO:tensorflow:global_step/sec: 1.75501 2021-09-25 23:21:49,041 [INFO] tensorflow: global_step/sec: 1.75501 INFO:tensorflow:global_step/sec: 1.81223 2021-09-25 23:21:49,592 [INFO] tensorflow: global_step/sec: 1.81223 INFO:tensorflow:global_step/sec: 1.80246 2021-09-25 23:21:50,147 [INFO] tensorflow: global_step/sec: 1.80246 INFO:tensorflow:global_step/sec: 1.6533 2021-09-25 23:21:50,752 [INFO] tensorflow: global_step/sec: 1.6533 Epoch 241/500 ========================= Validation cost: 0.000000 Mean average_precision (in %): 0.0000 class name average precision (in %) ------------ -------------------------- lpd 0 Median Inference Time: nan INFO:tensorflow:global_step/sec: 1.79036 2021-09-25 23:21:51,311 [INFO] tensorflow: global_step/sec: 1.79036 2021-09-25 23:21:51,312 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 241/500: loss: 0.00003 learning rate: 0.00050 Time taken: 0:00:02.860338 ETA: 0:12:20.827473 INFO:tensorflow:global_step/sec: 1.77939 2021-09-25 23:21:51,873 [INFO] tensorflow: global_step/sec: 1.77939 INFO:tensorflow:global_step/sec: 1.70732 2021-09-25 23:21:52,458 [INFO] tensorflow: global_step/sec: 1.70732 INFO:tensorflow:global_step/sec: 1.79469 2021-09-25 23:21:53,015 [INFO] tensorflow: global_step/sec: 1.79469 INFO:tensorflow:global_step/sec: 1.7931 2021-09-25 23:21:53,573 [INFO] tensorflow: global_step/sec: 1.7931 INFO:tensorflow:epoch = 242.0, learning_rate = 0.00049999997, loss = 3.8639955e-05, step = 1210 (5.658 sec) 2021-09-25 23:21:54,127 [INFO] tensorflow: epoch = 242.0, learning_rate = 0.00049999997, loss = 3.8639955e-05, step = 1210 (5.658 sec) INFO:tensorflow:global_step/sec: 1.80158 2021-09-25 23:21:54,128 [INFO] tensorflow: global_step/sec: 1.80158 2021-09-25 23:21:54,130 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 242/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.818604 ETA: 0:12:07.199892 INFO:tensorflow:global_step/sec: 1.74042 2021-09-25 23:21:54,703 [INFO] tensorflow: global_step/sec: 1.74042 INFO:tensorflow:global_step/sec: 1.81369 2021-09-25 23:21:55,254 [INFO] tensorflow: global_step/sec: 1.81369 INFO:tensorflow:global_step/sec: 1.81663 2021-09-25 23:21:55,805 [INFO] tensorflow: global_step/sec: 1.81663 INFO:tensorflow:global_step/sec: 1.76915 2021-09-25 23:21:56,370 [INFO] tensorflow: global_step/sec: 1.76915 INFO:tensorflow:global_step/sec: 1.75143 2021-09-25 23:21:56,941 [INFO] tensorflow: global_step/sec: 1.75143 2021-09-25 23:21:56,942 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 243/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.795619 ETA: 0:11:58.473963 INFO:tensorflow:global_step/sec: 1.76824 2021-09-25 23:21:57,506 [INFO] tensorflow: global_step/sec: 1.76824 INFO:tensorflow:global_step/sec: 1.84582 2021-09-25 23:21:58,048 [INFO] tensorflow: global_step/sec: 1.84582 INFO:tensorflow:global_step/sec: 1.81195 2021-09-25 23:21:58,600 [INFO] tensorflow: global_step/sec: 1.81195 INFO:tensorflow:global_step/sec: 1.74707 2021-09-25 23:21:59,172 [INFO] tensorflow: global_step/sec: 1.74707 INFO:tensorflow:epoch = 244.0, learning_rate = 0.00049999997, loss = 4.116776e-05, step = 1220 (5.624 sec) 2021-09-25 23:21:59,751 [INFO] tensorflow: epoch = 244.0, learning_rate = 0.00049999997, loss = 4.116776e-05, step = 1220 (5.624 sec) INFO:tensorflow:global_step/sec: 1.72522 2021-09-25 23:21:59,752 [INFO] tensorflow: global_step/sec: 1.72522 2021-09-25 23:21:59,753 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 244/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.803788 ETA: 0:11:57.769714 INFO:tensorflow:global_step/sec: 1.77622 2021-09-25 23:22:00,315 [INFO] tensorflow: global_step/sec: 1.77622 INFO:tensorflow:global_step/sec: 1.72112 2021-09-25 23:22:00,896 [INFO] tensorflow: global_step/sec: 1.72112 INFO:tensorflow:global_step/sec: 1.80443 2021-09-25 23:22:01,450 [INFO] tensorflow: global_step/sec: 1.80443 INFO:tensorflow:global_step/sec: 1.83346 2021-09-25 23:22:01,996 [INFO] tensorflow: global_step/sec: 1.83346 2021-09-25 23:22:01,999 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 113.470 INFO:tensorflow:global_step/sec: 1.69747 2021-09-25 23:22:02,585 [INFO] tensorflow: global_step/sec: 1.69747 2021-09-25 23:22:02,586 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 245/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.819583 ETA: 0:11:58.993649 INFO:tensorflow:global_step/sec: 1.86433 2021-09-25 23:22:03,121 [INFO] tensorflow: global_step/sec: 1.86433 INFO:tensorflow:global_step/sec: 1.74137 2021-09-25 23:22:03,695 [INFO] tensorflow: global_step/sec: 1.74137 INFO:tensorflow:global_step/sec: 1.73666 2021-09-25 23:22:04,271 [INFO] tensorflow: global_step/sec: 1.73666 INFO:tensorflow:global_step/sec: 1.81434 2021-09-25 23:22:04,822 [INFO] tensorflow: global_step/sec: 1.81434 INFO:tensorflow:epoch = 246.0, learning_rate = 0.00049999997, loss = 3.6658304e-05, step = 1230 (5.665 sec) 2021-09-25 23:22:05,415 [INFO] tensorflow: epoch = 246.0, learning_rate = 0.00049999997, loss = 3.6658304e-05, step = 1230 (5.665 sec) INFO:tensorflow:global_step/sec: 1.68289 2021-09-25 23:22:05,417 [INFO] tensorflow: global_step/sec: 1.68289 2021-09-25 23:22:05,418 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 246/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.828387 ETA: 0:11:58.410182 INFO:tensorflow:global_step/sec: 1.78714 2021-09-25 23:22:05,976 [INFO] tensorflow: global_step/sec: 1.78714 INFO:tensorflow:global_step/sec: 1.73422 2021-09-25 23:22:06,553 [INFO] tensorflow: global_step/sec: 1.73422 INFO:tensorflow:global_step/sec: 1.82332 2021-09-25 23:22:07,101 [INFO] tensorflow: global_step/sec: 1.82332 INFO:tensorflow:global_step/sec: 1.79703 2021-09-25 23:22:07,658 [INFO] tensorflow: global_step/sec: 1.79703 INFO:tensorflow:global_step/sec: 1.75608 2021-09-25 23:22:08,227 [INFO] tensorflow: global_step/sec: 1.75608 2021-09-25 23:22:08,229 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 247/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.836044 ETA: 0:11:57.519150 INFO:tensorflow:global_step/sec: 1.72216 2021-09-25 23:22:08,808 [INFO] tensorflow: global_step/sec: 1.72216 INFO:tensorflow:global_step/sec: 1.83585 2021-09-25 23:22:09,353 [INFO] tensorflow: global_step/sec: 1.83585 INFO:tensorflow:global_step/sec: 1.82811 2021-09-25 23:22:09,900 [INFO] tensorflow: global_step/sec: 1.82811 INFO:tensorflow:global_step/sec: 1.77831 2021-09-25 23:22:10,462 [INFO] tensorflow: global_step/sec: 1.77831 INFO:tensorflow:epoch = 248.0, learning_rate = 0.00049999997, loss = 4.1725987e-05, step = 1240 (5.611 sec) 2021-09-25 23:22:11,026 [INFO] tensorflow: epoch = 248.0, learning_rate = 0.00049999997, loss = 4.1725987e-05, step = 1240 (5.611 sec) INFO:tensorflow:global_step/sec: 1.76901 2021-09-25 23:22:11,027 [INFO] tensorflow: global_step/sec: 1.76901 2021-09-25 23:22:11,028 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 248/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.799888 ETA: 0:11:45.571810 INFO:tensorflow:global_step/sec: 1.84077 2021-09-25 23:22:11,570 [INFO] tensorflow: global_step/sec: 1.84077 INFO:tensorflow:global_step/sec: 1.73178 2021-09-25 23:22:12,148 [INFO] tensorflow: global_step/sec: 1.73178 INFO:tensorflow:global_step/sec: 1.85275 2021-09-25 23:22:12,688 [INFO] tensorflow: global_step/sec: 1.85275 INFO:tensorflow:global_step/sec: 1.80531 2021-09-25 23:22:13,242 [INFO] tensorflow: global_step/sec: 1.80531 INFO:tensorflow:global_step/sec: 1.78943 2021-09-25 23:22:13,800 [INFO] tensorflow: global_step/sec: 1.78943 2021-09-25 23:22:13,802 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 249/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.777605 ETA: 0:11:37.178869 INFO:tensorflow:global_step/sec: 1.75047 2021-09-25 23:22:14,372 [INFO] tensorflow: global_step/sec: 1.75047 INFO:tensorflow:global_step/sec: 1.7625 2021-09-25 23:22:14,939 [INFO] tensorflow: global_step/sec: 1.7625 INFO:tensorflow:global_step/sec: 1.78814 2021-09-25 23:22:15,498 [INFO] tensorflow: global_step/sec: 1.78814 INFO:tensorflow:global_step/sec: 1.77831 2021-09-25 23:22:16,061 [INFO] tensorflow: global_step/sec: 1.77831 2021-09-25 23:22:16,067 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 113.743 INFO:tensorflow:epoch = 250.0, learning_rate = 0.00049999997, loss = 3.6107558e-05, step = 1250 (5.579 sec) 2021-09-25 23:22:16,605 [INFO] tensorflow: epoch = 250.0, learning_rate = 0.00049999997, loss = 3.6107558e-05, step = 1250 (5.579 sec) INFO:tensorflow:global_step/sec: 1.83324 2021-09-25 23:22:16,606 [INFO] tensorflow: global_step/sec: 1.83324 2021-09-25 23:22:16,607 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 250/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.821841 ETA: 0:11:45.460370 INFO:tensorflow:global_step/sec: 1.84558 2021-09-25 23:22:17,148 [INFO] tensorflow: global_step/sec: 1.84558 INFO:tensorflow:global_step/sec: 1.72121 2021-09-25 23:22:17,729 [INFO] tensorflow: global_step/sec: 1.72121 INFO:tensorflow:global_step/sec: 1.76481 2021-09-25 23:22:18,296 [INFO] tensorflow: global_step/sec: 1.76481 INFO:tensorflow:global_step/sec: 1.77441 2021-09-25 23:22:18,859 [INFO] tensorflow: global_step/sec: 1.77441 INFO:tensorflow:global_step/sec: 1.81621 2021-09-25 23:22:19,410 [INFO] tensorflow: global_step/sec: 1.81621 2021-09-25 23:22:19,411 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 251/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.797620 ETA: 0:11:36.607454 INFO:tensorflow:global_step/sec: 1.80285 2021-09-25 23:22:19,964 [INFO] tensorflow: global_step/sec: 1.80285 INFO:tensorflow:global_step/sec: 1.83424 2021-09-25 23:22:20,510 [INFO] tensorflow: global_step/sec: 1.83424 INFO:tensorflow:global_step/sec: 1.77239 2021-09-25 23:22:21,074 [INFO] tensorflow: global_step/sec: 1.77239 INFO:tensorflow:global_step/sec: 1.72449 2021-09-25 23:22:21,654 [INFO] tensorflow: global_step/sec: 1.72449 INFO:tensorflow:epoch = 252.0, learning_rate = 0.00049999997, loss = 3.669558e-05, step = 1260 (5.607 sec) 2021-09-25 23:22:22,212 [INFO] tensorflow: epoch = 252.0, learning_rate = 0.00049999997, loss = 3.669558e-05, step = 1260 (5.607 sec) INFO:tensorflow:global_step/sec: 1.78733 2021-09-25 23:22:22,213 [INFO] tensorflow: global_step/sec: 1.78733 2021-09-25 23:22:22,214 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 252/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.790785 ETA: 0:11:32.114639 INFO:tensorflow:global_step/sec: 1.73238 2021-09-25 23:22:22,790 [INFO] tensorflow: global_step/sec: 1.73238 INFO:tensorflow:global_step/sec: 1.8081 2021-09-25 23:22:23,343 [INFO] tensorflow: global_step/sec: 1.8081 INFO:tensorflow:global_step/sec: 1.78371 2021-09-25 23:22:23,904 [INFO] tensorflow: global_step/sec: 1.78371 INFO:tensorflow:global_step/sec: 1.79285 2021-09-25 23:22:24,462 [INFO] tensorflow: global_step/sec: 1.79285 INFO:tensorflow:global_step/sec: 1.80604 2021-09-25 23:22:25,016 [INFO] tensorflow: global_step/sec: 1.80604 2021-09-25 23:22:25,017 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 253/500: loss: 0.00003 learning rate: 0.00050 Time taken: 0:00:02.809700 ETA: 0:11:33.996021 INFO:tensorflow:global_step/sec: 1.82272 2021-09-25 23:22:25,564 [INFO] tensorflow: global_step/sec: 1.82272 INFO:tensorflow:global_step/sec: 1.79918 2021-09-25 23:22:26,120 [INFO] tensorflow: global_step/sec: 1.79918 INFO:tensorflow:global_step/sec: 1.75805 2021-09-25 23:22:26,689 [INFO] tensorflow: global_step/sec: 1.75805 INFO:tensorflow:global_step/sec: 1.73153 2021-09-25 23:22:27,266 [INFO] tensorflow: global_step/sec: 1.73153 INFO:tensorflow:epoch = 254.0, learning_rate = 0.00049999997, loss = 3.7187176e-05, step = 1270 (5.612 sec) 2021-09-25 23:22:27,823 [INFO] tensorflow: epoch = 254.0, learning_rate = 0.00049999997, loss = 3.7187176e-05, step = 1270 (5.612 sec) INFO:tensorflow:global_step/sec: 1.79034 2021-09-25 23:22:27,825 [INFO] tensorflow: global_step/sec: 1.79034 2021-09-25 23:22:27,826 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 254/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.804660 ETA: 0:11:29.946439 INFO:tensorflow:global_step/sec: 1.76826 2021-09-25 23:22:28,390 [INFO] tensorflow: global_step/sec: 1.76826 INFO:tensorflow:global_step/sec: 1.70801 2021-09-25 23:22:28,976 [INFO] tensorflow: global_step/sec: 1.70801 INFO:tensorflow:global_step/sec: 1.75206 2021-09-25 23:22:29,547 [INFO] tensorflow: global_step/sec: 1.75206 INFO:tensorflow:global_step/sec: 1.77096 2021-09-25 23:22:30,111 [INFO] tensorflow: global_step/sec: 1.77096 2021-09-25 23:22:30,115 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 113.905 INFO:tensorflow:global_step/sec: 1.83959 2021-09-25 23:22:30,655 [INFO] tensorflow: global_step/sec: 1.83959 2021-09-25 23:22:30,656 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 255/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.841129 ETA: 0:11:36.076504 INFO:tensorflow:global_step/sec: 1.75491 2021-09-25 23:22:31,225 [INFO] tensorflow: global_step/sec: 1.75491 INFO:tensorflow:global_step/sec: 1.83777 2021-09-25 23:22:31,769 [INFO] tensorflow: global_step/sec: 1.83777 INFO:tensorflow:global_step/sec: 1.74362 2021-09-25 23:22:32,342 [INFO] tensorflow: global_step/sec: 1.74362 INFO:tensorflow:global_step/sec: 1.73289 2021-09-25 23:22:32,919 [INFO] tensorflow: global_step/sec: 1.73289 INFO:tensorflow:epoch = 256.0, learning_rate = 0.00049999997, loss = 5.5887664e-05, step = 1280 (5.641 sec) 2021-09-25 23:22:33,465 [INFO] tensorflow: epoch = 256.0, learning_rate = 0.00049999997, loss = 5.5887664e-05, step = 1280 (5.641 sec) INFO:tensorflow:global_step/sec: 1.82893 2021-09-25 23:22:33,466 [INFO] tensorflow: global_step/sec: 1.82893 2021-09-25 23:22:33,468 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 256/500: loss: 0.00006 learning rate: 0.00050 Time taken: 0:00:02.806541 ETA: 0:11:24.796112 INFO:tensorflow:global_step/sec: 1.74441 2021-09-25 23:22:34,039 [INFO] tensorflow: global_step/sec: 1.74441 INFO:tensorflow:global_step/sec: 1.79651 2021-09-25 23:22:34,596 [INFO] tensorflow: global_step/sec: 1.79651 INFO:tensorflow:global_step/sec: 1.74184 2021-09-25 23:22:35,170 [INFO] tensorflow: global_step/sec: 1.74184 INFO:tensorflow:global_step/sec: 1.80932 2021-09-25 23:22:35,723 [INFO] tensorflow: global_step/sec: 1.80932 INFO:tensorflow:global_step/sec: 1.73919 2021-09-25 23:22:36,298 [INFO] tensorflow: global_step/sec: 1.73919 2021-09-25 23:22:36,299 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 257/500: loss: 0.00003 learning rate: 0.00050 Time taken: 0:00:02.806695 ETA: 0:11:22.026765 INFO:tensorflow:global_step/sec: 1.83899 2021-09-25 23:22:36,842 [INFO] tensorflow: global_step/sec: 1.83899 INFO:tensorflow:global_step/sec: 1.7762 2021-09-25 23:22:37,405 [INFO] tensorflow: global_step/sec: 1.7762 INFO:tensorflow:global_step/sec: 1.771 2021-09-25 23:22:37,969 [INFO] tensorflow: global_step/sec: 1.771 INFO:tensorflow:global_step/sec: 1.78049 2021-09-25 23:22:38,531 [INFO] tensorflow: global_step/sec: 1.78049 INFO:tensorflow:epoch = 258.0, learning_rate = 0.00049999997, loss = 3.9431456e-05, step = 1290 (5.656 sec) 2021-09-25 23:22:39,121 [INFO] tensorflow: epoch = 258.0, learning_rate = 0.00049999997, loss = 3.9431456e-05, step = 1290 (5.656 sec) INFO:tensorflow:global_step/sec: 1.69082 2021-09-25 23:22:39,122 [INFO] tensorflow: global_step/sec: 1.69082 2021-09-25 23:22:39,124 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 258/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.805922 ETA: 0:11:19.033189 INFO:tensorflow:global_step/sec: 1.83456 2021-09-25 23:22:39,668 [INFO] tensorflow: global_step/sec: 1.83456 INFO:tensorflow:global_step/sec: 1.72125 2021-09-25 23:22:40,248 [INFO] tensorflow: global_step/sec: 1.72125 INFO:tensorflow:global_step/sec: 1.812 2021-09-25 23:22:40,800 [INFO] tensorflow: global_step/sec: 1.812 INFO:tensorflow:global_step/sec: 1.7567 2021-09-25 23:22:41,370 [INFO] tensorflow: global_step/sec: 1.7567 INFO:tensorflow:global_step/sec: 1.80631 2021-09-25 23:22:41,923 [INFO] tensorflow: global_step/sec: 1.80631 2021-09-25 23:22:41,925 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 259/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.836982 ETA: 0:11:23.712665 INFO:tensorflow:global_step/sec: 1.82639 2021-09-25 23:22:42,471 [INFO] tensorflow: global_step/sec: 1.82639 INFO:tensorflow:global_step/sec: 1.81556 2021-09-25 23:22:43,022 [INFO] tensorflow: global_step/sec: 1.81556 INFO:tensorflow:global_step/sec: 1.73811 2021-09-25 23:22:43,597 [INFO] tensorflow: global_step/sec: 1.73811 INFO:tensorflow:global_step/sec: 1.72075 2021-09-25 23:22:44,178 [INFO] tensorflow: global_step/sec: 1.72075 INFO:tensorflow:Saving checkpoints for step-1300. 2021-09-25 23:22:44,179 [INFO] tensorflow: Saving checkpoints for step-1300. WARNING:tensorflow:Ignoring: /tmp/tmpqgb6qodn; No such file or directory 2021-09-25 23:22:44,401 [WARNING] tensorflow: Ignoring: /tmp/tmpqgb6qodn; No such file or directory 2021-09-25 23:22:48,274 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 88.112 INFO:tensorflow:epoch = 260.0, learning_rate = 0.00049999997, loss = 3.4647397e-05, step = 1300 (9.742 sec) 2021-09-25 23:22:48,863 [INFO] tensorflow: epoch = 260.0, learning_rate = 0.00049999997, loss = 3.4647397e-05, step = 1300 (9.742 sec) INFO:tensorflow:global_step/sec: 0.213384 2021-09-25 23:22:48,864 [INFO] tensorflow: global_step/sec: 0.213384 2021-09-25 23:22:48,866 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 260/500: loss: 0.00003 learning rate: 0.00050 Time taken: 0:00:06.899693 ETA: 0:27:35.926380 INFO:tensorflow:global_step/sec: 1.79919 2021-09-25 23:22:49,420 [INFO] tensorflow: global_step/sec: 1.79919 INFO:tensorflow:global_step/sec: 1.75148 2021-09-25 23:22:49,991 [INFO] tensorflow: global_step/sec: 1.75148 INFO:tensorflow:global_step/sec: 1.84117 2021-09-25 23:22:50,534 [INFO] tensorflow: global_step/sec: 1.84117 INFO:tensorflow:global_step/sec: 1.7775 2021-09-25 23:22:51,097 [INFO] tensorflow: global_step/sec: 1.7775 Epoch 261/500 ========================= Validation cost: 0.000000 Mean average_precision (in %): 0.0000 class name average precision (in %) ------------ -------------------------- lpd 0 Median Inference Time: nan INFO:tensorflow:global_step/sec: 1.75397 2021-09-25 23:22:51,667 [INFO] tensorflow: global_step/sec: 1.75397 2021-09-25 23:22:51,668 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 261/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.829882 ETA: 0:11:16.341890 INFO:tensorflow:global_step/sec: 1.76979 2021-09-25 23:22:52,232 [INFO] tensorflow: global_step/sec: 1.76979 INFO:tensorflow:global_step/sec: 1.79624 2021-09-25 23:22:52,789 [INFO] tensorflow: global_step/sec: 1.79624 INFO:tensorflow:global_step/sec: 1.77501 2021-09-25 23:22:53,352 [INFO] tensorflow: global_step/sec: 1.77501 INFO:tensorflow:global_step/sec: 1.83988 2021-09-25 23:22:53,896 [INFO] tensorflow: global_step/sec: 1.83988 INFO:tensorflow:epoch = 262.0, learning_rate = 0.00049999997, loss = 3.9791477e-05, step = 1310 (5.586 sec) 2021-09-25 23:22:54,449 [INFO] tensorflow: epoch = 262.0, learning_rate = 0.00049999997, loss = 3.9791477e-05, step = 1310 (5.586 sec) INFO:tensorflow:global_step/sec: 1.80149 2021-09-25 23:22:54,451 [INFO] tensorflow: global_step/sec: 1.80149 2021-09-25 23:22:54,452 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 262/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.794458 ETA: 0:11:05.081040 INFO:tensorflow:global_step/sec: 1.7354 2021-09-25 23:22:55,027 [INFO] tensorflow: global_step/sec: 1.7354 INFO:tensorflow:global_step/sec: 1.80828 2021-09-25 23:22:55,580 [INFO] tensorflow: global_step/sec: 1.80828 INFO:tensorflow:global_step/sec: 1.75327 2021-09-25 23:22:56,150 [INFO] tensorflow: global_step/sec: 1.75327 INFO:tensorflow:global_step/sec: 1.81065 2021-09-25 23:22:56,703 [INFO] tensorflow: global_step/sec: 1.81065 INFO:tensorflow:global_step/sec: 1.7918 2021-09-25 23:22:57,261 [INFO] tensorflow: global_step/sec: 1.7918 2021-09-25 23:22:57,262 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 263/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.805536 ETA: 0:11:04.911983 INFO:tensorflow:global_step/sec: 1.83622 2021-09-25 23:22:57,805 [INFO] tensorflow: global_step/sec: 1.83622 INFO:tensorflow:global_step/sec: 1.71505 2021-09-25 23:22:58,388 [INFO] tensorflow: global_step/sec: 1.71505 INFO:tensorflow:global_step/sec: 1.81029 2021-09-25 23:22:58,941 [INFO] tensorflow: global_step/sec: 1.81029 INFO:tensorflow:global_step/sec: 1.78619 2021-09-25 23:22:59,501 [INFO] tensorflow: global_step/sec: 1.78619 INFO:tensorflow:epoch = 264.0, learning_rate = 0.00049999997, loss = 3.595731e-05, step = 1320 (5.595 sec) 2021-09-25 23:23:00,044 [INFO] tensorflow: epoch = 264.0, learning_rate = 0.00049999997, loss = 3.595731e-05, step = 1320 (5.595 sec) INFO:tensorflow:global_step/sec: 1.83538 2021-09-25 23:23:00,046 [INFO] tensorflow: global_step/sec: 1.83538 2021-09-25 23:23:00,047 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 264/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.796356 ETA: 0:10:59.940007 INFO:tensorflow:global_step/sec: 1.80791 2021-09-25 23:23:00,599 [INFO] tensorflow: global_step/sec: 1.80791 INFO:tensorflow:global_step/sec: 1.74547 2021-09-25 23:23:01,172 [INFO] tensorflow: global_step/sec: 1.74547 INFO:tensorflow:global_step/sec: 1.75617 2021-09-25 23:23:01,741 [INFO] tensorflow: global_step/sec: 1.75617 INFO:tensorflow:global_step/sec: 1.74635 2021-09-25 23:23:02,314 [INFO] tensorflow: global_step/sec: 1.74635 2021-09-25 23:23:02,321 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 113.917 INFO:tensorflow:global_step/sec: 1.74714 2021-09-25 23:23:02,886 [INFO] tensorflow: global_step/sec: 1.74714 2021-09-25 23:23:02,887 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 265/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.816190 ETA: 0:11:01.804651 INFO:tensorflow:global_step/sec: 1.83108 2021-09-25 23:23:03,432 [INFO] tensorflow: global_step/sec: 1.83108 INFO:tensorflow:global_step/sec: 1.80853 2021-09-25 23:23:03,985 [INFO] tensorflow: global_step/sec: 1.80853 INFO:tensorflow:global_step/sec: 1.83001 2021-09-25 23:23:04,531 [INFO] tensorflow: global_step/sec: 1.83001 INFO:tensorflow:global_step/sec: 1.71469 2021-09-25 23:23:05,115 [INFO] tensorflow: global_step/sec: 1.71469 INFO:tensorflow:epoch = 266.0, learning_rate = 0.00049999997, loss = 3.6946305e-05, step = 1330 (5.632 sec) 2021-09-25 23:23:05,676 [INFO] tensorflow: epoch = 266.0, learning_rate = 0.00049999997, loss = 3.6946305e-05, step = 1330 (5.632 sec) INFO:tensorflow:global_step/sec: 1.77752 2021-09-25 23:23:05,677 [INFO] tensorflow: global_step/sec: 1.77752 2021-09-25 23:23:05,679 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 266/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.796588 ETA: 0:10:54.401635 INFO:tensorflow:global_step/sec: 1.83137 2021-09-25 23:23:06,223 [INFO] tensorflow: global_step/sec: 1.83137 INFO:tensorflow:global_step/sec: 1.78506 2021-09-25 23:23:06,783 [INFO] tensorflow: global_step/sec: 1.78506 INFO:tensorflow:global_step/sec: 1.7178 2021-09-25 23:23:07,366 [INFO] tensorflow: global_step/sec: 1.7178 INFO:tensorflow:global_step/sec: 1.80181 2021-09-25 23:23:07,921 [INFO] tensorflow: global_step/sec: 1.80181 INFO:tensorflow:global_step/sec: 1.71678 2021-09-25 23:23:08,503 [INFO] tensorflow: global_step/sec: 1.71678 2021-09-25 23:23:08,505 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 267/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.808807 ETA: 0:10:54.452007 INFO:tensorflow:global_step/sec: 1.66346 2021-09-25 23:23:09,104 [INFO] tensorflow: global_step/sec: 1.66346 INFO:tensorflow:global_step/sec: 1.61772 2021-09-25 23:23:09,723 [INFO] tensorflow: global_step/sec: 1.61772 INFO:tensorflow:global_step/sec: 1.35828 2021-09-25 23:23:10,459 [INFO] tensorflow: global_step/sec: 1.35828 INFO:tensorflow:global_step/sec: 1.79726 2021-09-25 23:23:11,015 [INFO] tensorflow: global_step/sec: 1.79726 INFO:tensorflow:epoch = 268.0, learning_rate = 0.00049999997, loss = 4.3298038e-05, step = 1340 (5.894 sec) 2021-09-25 23:23:11,570 [INFO] tensorflow: epoch = 268.0, learning_rate = 0.00049999997, loss = 4.3298038e-05, step = 1340 (5.894 sec) INFO:tensorflow:global_step/sec: 1.79694 2021-09-25 23:23:11,572 [INFO] tensorflow: global_step/sec: 1.79694 2021-09-25 23:23:11,573 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 268/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:03.090533 ETA: 0:11:57.003660 INFO:tensorflow:global_step/sec: 1.73366 2021-09-25 23:23:12,148 [INFO] tensorflow: global_step/sec: 1.73366 INFO:tensorflow:global_step/sec: 1.7467 2021-09-25 23:23:12,721 [INFO] tensorflow: global_step/sec: 1.7467 INFO:tensorflow:global_step/sec: 1.74727 2021-09-25 23:23:13,293 [INFO] tensorflow: global_step/sec: 1.74727 INFO:tensorflow:global_step/sec: 1.80109 2021-09-25 23:23:13,848 [INFO] tensorflow: global_step/sec: 1.80109 INFO:tensorflow:global_step/sec: 1.72281 2021-09-25 23:23:14,429 [INFO] tensorflow: global_step/sec: 1.72281 2021-09-25 23:23:14,430 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 269/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.836114 ETA: 0:10:55.142263 INFO:tensorflow:global_step/sec: 1.78265 2021-09-25 23:23:14,990 [INFO] tensorflow: global_step/sec: 1.78265 INFO:tensorflow:global_step/sec: 1.81116 2021-09-25 23:23:15,542 [INFO] tensorflow: global_step/sec: 1.81116 INFO:tensorflow:global_step/sec: 1.77603 2021-09-25 23:23:16,105 [INFO] tensorflow: global_step/sec: 1.77603 INFO:tensorflow:global_step/sec: 1.79826 2021-09-25 23:23:16,661 [INFO] tensorflow: global_step/sec: 1.79826 2021-09-25 23:23:16,668 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 111.532 INFO:tensorflow:epoch = 270.0, learning_rate = 0.00049999997, loss = 3.7295635e-05, step = 1350 (5.671 sec) 2021-09-25 23:23:17,241 [INFO] tensorflow: epoch = 270.0, learning_rate = 0.00049999997, loss = 3.7295635e-05, step = 1350 (5.671 sec) INFO:tensorflow:global_step/sec: 1.71998 2021-09-25 23:23:17,242 [INFO] tensorflow: global_step/sec: 1.71998 2021-09-25 23:23:17,244 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 270/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.810751 ETA: 0:10:46.472666 INFO:tensorflow:global_step/sec: 1.87244 2021-09-25 23:23:17,777 [INFO] tensorflow: global_step/sec: 1.87244 INFO:tensorflow:global_step/sec: 1.75586 2021-09-25 23:23:18,346 [INFO] tensorflow: global_step/sec: 1.75586 INFO:tensorflow:global_step/sec: 1.84506 2021-09-25 23:23:18,888 [INFO] tensorflow: global_step/sec: 1.84506 INFO:tensorflow:global_step/sec: 1.81696 2021-09-25 23:23:19,438 [INFO] tensorflow: global_step/sec: 1.81696 INFO:tensorflow:global_step/sec: 1.73787 2021-09-25 23:23:20,014 [INFO] tensorflow: global_step/sec: 1.73787 2021-09-25 23:23:20,015 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 271/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.776235 ETA: 0:10:35.757893 INFO:tensorflow:global_step/sec: 1.83959 2021-09-25 23:23:20,557 [INFO] tensorflow: global_step/sec: 1.83959 INFO:tensorflow:global_step/sec: 1.82102 2021-09-25 23:23:21,107 [INFO] tensorflow: global_step/sec: 1.82102 INFO:tensorflow:global_step/sec: 1.81195 2021-09-25 23:23:21,658 [INFO] tensorflow: global_step/sec: 1.81195 INFO:tensorflow:global_step/sec: 1.82024 2021-09-25 23:23:22,208 [INFO] tensorflow: global_step/sec: 1.82024 INFO:tensorflow:global_step/sec: 1.71032 2021-09-25 23:23:22,793 [INFO] tensorflow: global_step/sec: 1.71032 2021-09-25 23:23:22,794 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 272/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.769499 ETA: 0:10:31.445678 INFO:tensorflow:epoch = 272.2, learning_rate = 0.00049999997, loss = 3.590149e-05, step = 1361 (6.104 sec) 2021-09-25 23:23:23,346 [INFO] tensorflow: epoch = 272.2, learning_rate = 0.00049999997, loss = 3.590149e-05, step = 1361 (6.104 sec) INFO:tensorflow:global_step/sec: 1.80398 2021-09-25 23:23:23,347 [INFO] tensorflow: global_step/sec: 1.80398 INFO:tensorflow:global_step/sec: 1.8173 2021-09-25 23:23:23,897 [INFO] tensorflow: global_step/sec: 1.8173 INFO:tensorflow:global_step/sec: 1.83207 2021-09-25 23:23:24,443 [INFO] tensorflow: global_step/sec: 1.83207 INFO:tensorflow:global_step/sec: 1.7551 2021-09-25 23:23:25,013 [INFO] tensorflow: global_step/sec: 1.7551 INFO:tensorflow:global_step/sec: 1.77041 2021-09-25 23:23:25,578 [INFO] tensorflow: global_step/sec: 1.77041 2021-09-25 23:23:25,579 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 273/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.800246 ETA: 0:10:35.655842 INFO:tensorflow:global_step/sec: 1.74556 2021-09-25 23:23:26,150 [INFO] tensorflow: global_step/sec: 1.74556 INFO:tensorflow:global_step/sec: 1.75579 2021-09-25 23:23:26,720 [INFO] tensorflow: global_step/sec: 1.75579 INFO:tensorflow:global_step/sec: 1.76112 2021-09-25 23:23:27,288 [INFO] tensorflow: global_step/sec: 1.76112 INFO:tensorflow:global_step/sec: 1.81196 2021-09-25 23:23:27,840 [INFO] tensorflow: global_step/sec: 1.81196 INFO:tensorflow:global_step/sec: 1.72935 2021-09-25 23:23:28,418 [INFO] tensorflow: global_step/sec: 1.72935 2021-09-25 23:23:28,419 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 274/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.828185 ETA: 0:10:39.169775 INFO:tensorflow:epoch = 274.2, learning_rate = 0.00049999997, loss = 3.4914683e-05, step = 1371 (5.643 sec) 2021-09-25 23:23:28,989 [INFO] tensorflow: epoch = 274.2, learning_rate = 0.00049999997, loss = 3.4914683e-05, step = 1371 (5.643 sec) INFO:tensorflow:global_step/sec: 1.74788 2021-09-25 23:23:28,990 [INFO] tensorflow: global_step/sec: 1.74788 INFO:tensorflow:global_step/sec: 1.78545 2021-09-25 23:23:29,550 [INFO] tensorflow: global_step/sec: 1.78545 INFO:tensorflow:global_step/sec: 1.7241 2021-09-25 23:23:30,130 [INFO] tensorflow: global_step/sec: 1.7241 INFO:tensorflow:global_step/sec: 1.69366 2021-09-25 23:23:30,721 [INFO] tensorflow: global_step/sec: 1.69366 2021-09-25 23:23:30,728 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 113.804 INFO:tensorflow:global_step/sec: 1.78691 2021-09-25 23:23:31,280 [INFO] tensorflow: global_step/sec: 1.78691 2021-09-25 23:23:31,282 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 275/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.882154 ETA: 0:10:48.484647 INFO:tensorflow:global_step/sec: 1.80511 2021-09-25 23:23:31,834 [INFO] tensorflow: global_step/sec: 1.80511 INFO:tensorflow:global_step/sec: 1.79577 2021-09-25 23:23:32,391 [INFO] tensorflow: global_step/sec: 1.79577 INFO:tensorflow:global_step/sec: 1.7935 2021-09-25 23:23:32,949 [INFO] tensorflow: global_step/sec: 1.7935 INFO:tensorflow:global_step/sec: 1.78737 2021-09-25 23:23:33,508 [INFO] tensorflow: global_step/sec: 1.78737 INFO:tensorflow:global_step/sec: 1.77352 2021-09-25 23:23:34,072 [INFO] tensorflow: global_step/sec: 1.77352 2021-09-25 23:23:34,074 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 276/500: loss: 0.00003 learning rate: 0.00050 Time taken: 0:00:02.784654 ETA: 0:10:23.762581 INFO:tensorflow:epoch = 276.2, learning_rate = 0.00049999997, loss = 3.837004e-05, step = 1381 (5.633 sec) 2021-09-25 23:23:34,621 [INFO] tensorflow: epoch = 276.2, learning_rate = 0.00049999997, loss = 3.837004e-05, step = 1381 (5.633 sec) INFO:tensorflow:global_step/sec: 1.81632 2021-09-25 23:23:34,623 [INFO] tensorflow: global_step/sec: 1.81632 INFO:tensorflow:global_step/sec: 1.842 2021-09-25 23:23:35,165 [INFO] tensorflow: global_step/sec: 1.842 INFO:tensorflow:global_step/sec: 1.68619 2021-09-25 23:23:35,758 [INFO] tensorflow: global_step/sec: 1.68619 INFO:tensorflow:global_step/sec: 1.75702 2021-09-25 23:23:36,328 [INFO] tensorflow: global_step/sec: 1.75702 INFO:tensorflow:global_step/sec: 1.78809 2021-09-25 23:23:36,887 [INFO] tensorflow: global_step/sec: 1.78809 2021-09-25 23:23:36,888 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 277/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.817326 ETA: 0:10:28.263767 INFO:tensorflow:global_step/sec: 1.83873 2021-09-25 23:23:37,431 [INFO] tensorflow: global_step/sec: 1.83873 INFO:tensorflow:global_step/sec: 1.74736 2021-09-25 23:23:38,003 [INFO] tensorflow: global_step/sec: 1.74736 INFO:tensorflow:global_step/sec: 1.8137 2021-09-25 23:23:38,554 [INFO] tensorflow: global_step/sec: 1.8137 INFO:tensorflow:global_step/sec: 1.78191 2021-09-25 23:23:39,116 [INFO] tensorflow: global_step/sec: 1.78191 INFO:tensorflow:global_step/sec: 1.73457 2021-09-25 23:23:39,692 [INFO] tensorflow: global_step/sec: 1.73457 2021-09-25 23:23:39,694 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 278/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.789267 ETA: 0:10:19.217288 INFO:tensorflow:epoch = 278.2, learning_rate = 0.00049999997, loss = 3.8643375e-05, step = 1391 (5.615 sec) 2021-09-25 23:23:40,237 [INFO] tensorflow: epoch = 278.2, learning_rate = 0.00049999997, loss = 3.8643375e-05, step = 1391 (5.615 sec) INFO:tensorflow:global_step/sec: 1.83192 2021-09-25 23:23:40,238 [INFO] tensorflow: global_step/sec: 1.83192 INFO:tensorflow:global_step/sec: 1.84457 2021-09-25 23:23:40,780 [INFO] tensorflow: global_step/sec: 1.84457 INFO:tensorflow:global_step/sec: 1.74965 2021-09-25 23:23:41,352 [INFO] tensorflow: global_step/sec: 1.74965 INFO:tensorflow:global_step/sec: 1.79128 2021-09-25 23:23:41,910 [INFO] tensorflow: global_step/sec: 1.79128 INFO:tensorflow:global_step/sec: 1.8134 2021-09-25 23:23:42,461 [INFO] tensorflow: global_step/sec: 1.8134 2021-09-25 23:23:42,463 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 279/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.792540 ETA: 0:10:17.151356 INFO:tensorflow:global_step/sec: 1.79725 2021-09-25 23:23:43,018 [INFO] tensorflow: global_step/sec: 1.79725 INFO:tensorflow:global_step/sec: 1.78444 2021-09-25 23:23:43,578 [INFO] tensorflow: global_step/sec: 1.78444 INFO:tensorflow:global_step/sec: 1.72997 2021-09-25 23:23:44,156 [INFO] tensorflow: global_step/sec: 1.72997 INFO:tensorflow:global_step/sec: 1.83558 2021-09-25 23:23:44,701 [INFO] tensorflow: global_step/sec: 1.83558 INFO:tensorflow:Saving checkpoints for step-1400. 2021-09-25 23:23:44,702 [INFO] tensorflow: Saving checkpoints for step-1400. WARNING:tensorflow:Ignoring: /tmp/tmpuirwuo19; No such file or directory 2021-09-25 23:23:44,924 [WARNING] tensorflow: Ignoring: /tmp/tmpuirwuo19; No such file or directory 2021-09-25 23:23:48,801 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 88.532 INFO:tensorflow:epoch = 280.0, learning_rate = 0.00049999997, loss = 3.5875604e-05, step = 1400 (9.145 sec) 2021-09-25 23:23:49,382 [INFO] tensorflow: epoch = 280.0, learning_rate = 0.00049999997, loss = 3.5875604e-05, step = 1400 (9.145 sec) INFO:tensorflow:global_step/sec: 0.213563 2021-09-25 23:23:49,383 [INFO] tensorflow: global_step/sec: 0.213563 2021-09-25 23:23:49,385 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 280/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:06.885805 ETA: 0:25:14.877129 INFO:tensorflow:global_step/sec: 1.70621 2021-09-25 23:23:49,970 [INFO] tensorflow: global_step/sec: 1.70621 INFO:tensorflow:global_step/sec: 1.77723 2021-09-25 23:23:50,532 [INFO] tensorflow: global_step/sec: 1.77723 INFO:tensorflow:global_step/sec: 1.77464 2021-09-25 23:23:51,096 [INFO] tensorflow: global_step/sec: 1.77464 INFO:tensorflow:global_step/sec: 1.75107 2021-09-25 23:23:51,667 [INFO] tensorflow: global_step/sec: 1.75107 Epoch 281/500 ========================= Validation cost: 0.000000 Mean average_precision (in %): 0.0000 class name average precision (in %) ------------ -------------------------- lpd 0 Median Inference Time: nan INFO:tensorflow:global_step/sec: 1.77503 2021-09-25 23:23:52,230 [INFO] tensorflow: global_step/sec: 1.77503 2021-09-25 23:23:52,232 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 281/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.869945 ETA: 0:10:28.517966 INFO:tensorflow:global_step/sec: 1.73549 2021-09-25 23:23:52,806 [INFO] tensorflow: global_step/sec: 1.73549 INFO:tensorflow:global_step/sec: 1.77301 2021-09-25 23:23:53,370 [INFO] tensorflow: global_step/sec: 1.77301 INFO:tensorflow:global_step/sec: 1.79967 2021-09-25 23:23:53,926 [INFO] tensorflow: global_step/sec: 1.79967 INFO:tensorflow:global_step/sec: 1.77399 2021-09-25 23:23:54,490 [INFO] tensorflow: global_step/sec: 1.77399 INFO:tensorflow:epoch = 282.0, learning_rate = 0.00049999997, loss = 3.7413905e-05, step = 1410 (5.694 sec) 2021-09-25 23:23:55,076 [INFO] tensorflow: epoch = 282.0, learning_rate = 0.00049999997, loss = 3.7413905e-05, step = 1410 (5.694 sec) INFO:tensorflow:global_step/sec: 1.70196 2021-09-25 23:23:55,077 [INFO] tensorflow: global_step/sec: 1.70196 2021-09-25 23:23:55,079 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 282/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.819573 ETA: 0:10:14.666898 INFO:tensorflow:global_step/sec: 1.80696 2021-09-25 23:23:55,631 [INFO] tensorflow: global_step/sec: 1.80696 INFO:tensorflow:global_step/sec: 1.79723 2021-09-25 23:23:56,187 [INFO] tensorflow: global_step/sec: 1.79723 INFO:tensorflow:global_step/sec: 1.75924 2021-09-25 23:23:56,756 [INFO] tensorflow: global_step/sec: 1.75924 INFO:tensorflow:global_step/sec: 1.74166 2021-09-25 23:23:57,330 [INFO] tensorflow: global_step/sec: 1.74166 INFO:tensorflow:global_step/sec: 1.78893 2021-09-25 23:23:57,889 [INFO] tensorflow: global_step/sec: 1.78893 2021-09-25 23:23:57,890 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 283/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.838495 ETA: 0:10:15.953470 INFO:tensorflow:global_step/sec: 1.79791 2021-09-25 23:23:58,445 [INFO] tensorflow: global_step/sec: 1.79791 INFO:tensorflow:global_step/sec: 1.72883 2021-09-25 23:23:59,023 [INFO] tensorflow: global_step/sec: 1.72883 INFO:tensorflow:global_step/sec: 1.8099 2021-09-25 23:23:59,576 [INFO] tensorflow: global_step/sec: 1.8099 INFO:tensorflow:global_step/sec: 1.81913 2021-09-25 23:24:00,126 [INFO] tensorflow: global_step/sec: 1.81913 INFO:tensorflow:epoch = 284.0, learning_rate = 0.00049999997, loss = 3.6476617e-05, step = 1420 (5.610 sec) 2021-09-25 23:24:00,686 [INFO] tensorflow: epoch = 284.0, learning_rate = 0.00049999997, loss = 3.6476617e-05, step = 1420 (5.610 sec) INFO:tensorflow:global_step/sec: 1.78096 2021-09-25 23:24:00,687 [INFO] tensorflow: global_step/sec: 1.78096 2021-09-25 23:24:00,688 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 284/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.794781 ETA: 0:10:03.672741 INFO:tensorflow:global_step/sec: 1.81325 2021-09-25 23:24:01,239 [INFO] tensorflow: global_step/sec: 1.81325 INFO:tensorflow:global_step/sec: 1.71664 2021-09-25 23:24:01,821 [INFO] tensorflow: global_step/sec: 1.71664 INFO:tensorflow:global_step/sec: 1.78255 2021-09-25 23:24:02,382 [INFO] tensorflow: global_step/sec: 1.78255 INFO:tensorflow:global_step/sec: 1.74925 2021-09-25 23:24:02,954 [INFO] tensorflow: global_step/sec: 1.74925 2021-09-25 23:24:02,957 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 113.044 INFO:tensorflow:global_step/sec: 1.77426 2021-09-25 23:24:03,517 [INFO] tensorflow: global_step/sec: 1.77426 2021-09-25 23:24:03,519 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 285/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.828098 ETA: 0:10:08.041134 INFO:tensorflow:global_step/sec: 1.82995 2021-09-25 23:24:04,064 [INFO] tensorflow: global_step/sec: 1.82995 INFO:tensorflow:global_step/sec: 1.77056 2021-09-25 23:24:04,629 [INFO] tensorflow: global_step/sec: 1.77056 INFO:tensorflow:global_step/sec: 1.69973 2021-09-25 23:24:05,217 [INFO] tensorflow: global_step/sec: 1.69973 INFO:tensorflow:global_step/sec: 1.76755 2021-09-25 23:24:05,783 [INFO] tensorflow: global_step/sec: 1.76755 INFO:tensorflow:epoch = 286.0, learning_rate = 0.00049999997, loss = 3.672106e-05, step = 1430 (5.653 sec) 2021-09-25 23:24:06,338 [INFO] tensorflow: epoch = 286.0, learning_rate = 0.00049999997, loss = 3.672106e-05, step = 1430 (5.653 sec) INFO:tensorflow:global_step/sec: 1.79472 2021-09-25 23:24:06,340 [INFO] tensorflow: global_step/sec: 1.79472 2021-09-25 23:24:06,341 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 286/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.827539 ETA: 0:10:05.093441 INFO:tensorflow:global_step/sec: 1.79121 2021-09-25 23:24:06,898 [INFO] tensorflow: global_step/sec: 1.79121 INFO:tensorflow:global_step/sec: 1.82991 2021-09-25 23:24:07,445 [INFO] tensorflow: global_step/sec: 1.82991 INFO:tensorflow:global_step/sec: 1.74764 2021-09-25 23:24:08,017 [INFO] tensorflow: global_step/sec: 1.74764 INFO:tensorflow:global_step/sec: 1.74922 2021-09-25 23:24:08,589 [INFO] tensorflow: global_step/sec: 1.74922 INFO:tensorflow:global_step/sec: 1.74972 2021-09-25 23:24:09,160 [INFO] tensorflow: global_step/sec: 1.74972 2021-09-25 23:24:09,161 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 287/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.805372 ETA: 0:09:57.544236 INFO:tensorflow:global_step/sec: 1.75797 2021-09-25 23:24:09,729 [INFO] tensorflow: global_step/sec: 1.75797 INFO:tensorflow:global_step/sec: 1.77326 2021-09-25 23:24:10,293 [INFO] tensorflow: global_step/sec: 1.77326 INFO:tensorflow:global_step/sec: 1.75432 2021-09-25 23:24:10,863 [INFO] tensorflow: global_step/sec: 1.75432 INFO:tensorflow:global_step/sec: 1.81117 2021-09-25 23:24:11,415 [INFO] tensorflow: global_step/sec: 1.81117 INFO:tensorflow:epoch = 288.0, learning_rate = 0.00049999997, loss = 3.8504317e-05, step = 1440 (5.641 sec) 2021-09-25 23:24:11,979 [INFO] tensorflow: epoch = 288.0, learning_rate = 0.00049999997, loss = 3.8504317e-05, step = 1440 (5.641 sec) INFO:tensorflow:global_step/sec: 1.76728 2021-09-25 23:24:11,981 [INFO] tensorflow: global_step/sec: 1.76728 2021-09-25 23:24:11,982 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 288/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.830007 ETA: 0:09:59.961450 INFO:tensorflow:global_step/sec: 1.76295 2021-09-25 23:24:12,548 [INFO] tensorflow: global_step/sec: 1.76295 INFO:tensorflow:global_step/sec: 1.748 2021-09-25 23:24:13,120 [INFO] tensorflow: global_step/sec: 1.748 INFO:tensorflow:global_step/sec: 1.78612 2021-09-25 23:24:13,680 [INFO] tensorflow: global_step/sec: 1.78612 INFO:tensorflow:global_step/sec: 1.7961 2021-09-25 23:24:14,237 [INFO] tensorflow: global_step/sec: 1.7961 INFO:tensorflow:global_step/sec: 1.75834 2021-09-25 23:24:14,805 [INFO] tensorflow: global_step/sec: 1.75834 2021-09-25 23:24:14,807 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 289/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.819992 ETA: 0:09:55.018326 INFO:tensorflow:global_step/sec: 1.7423 2021-09-25 23:24:15,379 [INFO] tensorflow: global_step/sec: 1.7423 INFO:tensorflow:global_step/sec: 1.78246 2021-09-25 23:24:15,940 [INFO] tensorflow: global_step/sec: 1.78246 INFO:tensorflow:global_step/sec: 1.75554 2021-09-25 23:24:16,510 [INFO] tensorflow: global_step/sec: 1.75554 INFO:tensorflow:global_step/sec: 1.74812 2021-09-25 23:24:17,082 [INFO] tensorflow: global_step/sec: 1.74812 2021-09-25 23:24:17,088 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 113.239 INFO:tensorflow:epoch = 290.0, learning_rate = 0.00049999997, loss = 3.2745957e-05, step = 1450 (5.657 sec) 2021-09-25 23:24:17,636 [INFO] tensorflow: epoch = 290.0, learning_rate = 0.00049999997, loss = 3.2745957e-05, step = 1450 (5.657 sec) INFO:tensorflow:global_step/sec: 1.80031 2021-09-25 23:24:17,638 [INFO] tensorflow: global_step/sec: 1.80031 2021-09-25 23:24:17,639 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 290/500: loss: 0.00003 learning rate: 0.00050 Time taken: 0:00:02.843693 ETA: 0:09:57.175534 INFO:tensorflow:global_step/sec: 1.77508 2021-09-25 23:24:18,201 [INFO] tensorflow: global_step/sec: 1.77508 INFO:tensorflow:global_step/sec: 1.74442 2021-09-25 23:24:18,774 [INFO] tensorflow: global_step/sec: 1.74442 INFO:tensorflow:global_step/sec: 1.83355 2021-09-25 23:24:19,320 [INFO] tensorflow: global_step/sec: 1.83355 INFO:tensorflow:global_step/sec: 1.76186 2021-09-25 23:24:19,887 [INFO] tensorflow: global_step/sec: 1.76186 INFO:tensorflow:global_step/sec: 1.81331 2021-09-25 23:24:20,439 [INFO] tensorflow: global_step/sec: 1.81331 2021-09-25 23:24:20,440 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 291/500: loss: 0.00003 learning rate: 0.00050 Time taken: 0:00:02.802698 ETA: 0:09:45.763910 INFO:tensorflow:global_step/sec: 1.79668 2021-09-25 23:24:20,995 [INFO] tensorflow: global_step/sec: 1.79668 INFO:tensorflow:global_step/sec: 1.81247 2021-09-25 23:24:21,547 [INFO] tensorflow: global_step/sec: 1.81247 INFO:tensorflow:global_step/sec: 1.79665 2021-09-25 23:24:22,104 [INFO] tensorflow: global_step/sec: 1.79665 INFO:tensorflow:global_step/sec: 1.79103 2021-09-25 23:24:22,662 [INFO] tensorflow: global_step/sec: 1.79103 INFO:tensorflow:epoch = 292.0, learning_rate = 0.00049999997, loss = 3.5408484e-05, step = 1460 (5.606 sec) 2021-09-25 23:24:23,242 [INFO] tensorflow: epoch = 292.0, learning_rate = 0.00049999997, loss = 3.5408484e-05, step = 1460 (5.606 sec) INFO:tensorflow:global_step/sec: 1.71847 2021-09-25 23:24:23,244 [INFO] tensorflow: global_step/sec: 1.71847 2021-09-25 23:24:23,245 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 292/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.774610 ETA: 0:09:37.118938 INFO:tensorflow:global_step/sec: 1.79545 2021-09-25 23:24:23,801 [INFO] tensorflow: global_step/sec: 1.79545 INFO:tensorflow:global_step/sec: 1.79214 2021-09-25 23:24:24,359 [INFO] tensorflow: global_step/sec: 1.79214 INFO:tensorflow:global_step/sec: 1.81719 2021-09-25 23:24:24,909 [INFO] tensorflow: global_step/sec: 1.81719 INFO:tensorflow:global_step/sec: 1.83205 2021-09-25 23:24:25,455 [INFO] tensorflow: global_step/sec: 1.83205 INFO:tensorflow:global_step/sec: 1.69557 2021-09-25 23:24:26,045 [INFO] tensorflow: global_step/sec: 1.69557 2021-09-25 23:24:26,046 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 293/500: loss: 0.00005 learning rate: 0.00050 Time taken: 0:00:02.791111 ETA: 0:09:37.759975 INFO:tensorflow:global_step/sec: 1.7267 2021-09-25 23:24:26,624 [INFO] tensorflow: global_step/sec: 1.7267 INFO:tensorflow:global_step/sec: 1.78864 2021-09-25 23:24:27,183 [INFO] tensorflow: global_step/sec: 1.78864 INFO:tensorflow:global_step/sec: 1.8012 2021-09-25 23:24:27,738 [INFO] tensorflow: global_step/sec: 1.8012 INFO:tensorflow:global_step/sec: 1.75193 2021-09-25 23:24:28,309 [INFO] tensorflow: global_step/sec: 1.75193 INFO:tensorflow:epoch = 294.0, learning_rate = 0.00049999997, loss = 3.947318e-05, step = 1470 (5.619 sec) 2021-09-25 23:24:28,861 [INFO] tensorflow: epoch = 294.0, learning_rate = 0.00049999997, loss = 3.947318e-05, step = 1470 (5.619 sec) INFO:tensorflow:global_step/sec: 1.80544 2021-09-25 23:24:28,863 [INFO] tensorflow: global_step/sec: 1.80544 2021-09-25 23:24:28,864 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 294/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.853123 ETA: 0:09:47.743426 INFO:tensorflow:global_step/sec: 1.8425 2021-09-25 23:24:29,406 [INFO] tensorflow: global_step/sec: 1.8425 INFO:tensorflow:global_step/sec: 1.7665 2021-09-25 23:24:29,972 [INFO] tensorflow: global_step/sec: 1.7665 INFO:tensorflow:global_step/sec: 1.79339 2021-09-25 23:24:30,529 [INFO] tensorflow: global_step/sec: 1.79339 INFO:tensorflow:global_step/sec: 1.75398 2021-09-25 23:24:31,099 [INFO] tensorflow: global_step/sec: 1.75398 2021-09-25 23:24:31,107 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 114.137 INFO:tensorflow:global_step/sec: 1.77724 2021-09-25 23:24:31,662 [INFO] tensorflow: global_step/sec: 1.77724 2021-09-25 23:24:31,663 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 295/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.793624 ETA: 0:09:32.693002 INFO:tensorflow:global_step/sec: 1.81031 2021-09-25 23:24:32,214 [INFO] tensorflow: global_step/sec: 1.81031 INFO:tensorflow:global_step/sec: 1.72325 2021-09-25 23:24:32,795 [INFO] tensorflow: global_step/sec: 1.72325 INFO:tensorflow:global_step/sec: 1.78897 2021-09-25 23:24:33,354 [INFO] tensorflow: global_step/sec: 1.78897 INFO:tensorflow:global_step/sec: 1.78539 2021-09-25 23:24:33,914 [INFO] tensorflow: global_step/sec: 1.78539 INFO:tensorflow:epoch = 296.0, learning_rate = 0.00049999997, loss = 3.8828606e-05, step = 1480 (5.624 sec) 2021-09-25 23:24:34,485 [INFO] tensorflow: epoch = 296.0, learning_rate = 0.00049999997, loss = 3.8828606e-05, step = 1480 (5.624 sec) INFO:tensorflow:global_step/sec: 1.74572 2021-09-25 23:24:34,487 [INFO] tensorflow: global_step/sec: 1.74572 2021-09-25 23:24:34,488 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 296/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.811822 ETA: 0:09:33.611724 INFO:tensorflow:global_step/sec: 1.80977 2021-09-25 23:24:35,039 [INFO] tensorflow: global_step/sec: 1.80977 INFO:tensorflow:global_step/sec: 1.77773 2021-09-25 23:24:35,602 [INFO] tensorflow: global_step/sec: 1.77773 INFO:tensorflow:global_step/sec: 1.78943 2021-09-25 23:24:36,161 [INFO] tensorflow: global_step/sec: 1.78943 INFO:tensorflow:global_step/sec: 1.68137 2021-09-25 23:24:36,755 [INFO] tensorflow: global_step/sec: 1.68137 INFO:tensorflow:global_step/sec: 1.69176 2021-09-25 23:24:37,346 [INFO] tensorflow: global_step/sec: 1.69176 2021-09-25 23:24:37,348 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 297/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.839581 ETA: 0:09:36.434897 INFO:tensorflow:global_step/sec: 1.82941 2021-09-25 23:24:37,893 [INFO] tensorflow: global_step/sec: 1.82941 INFO:tensorflow:global_step/sec: 1.80539 2021-09-25 23:24:38,447 [INFO] tensorflow: global_step/sec: 1.80539 INFO:tensorflow:global_step/sec: 1.82344 2021-09-25 23:24:38,995 [INFO] tensorflow: global_step/sec: 1.82344 INFO:tensorflow:global_step/sec: 1.81977 2021-09-25 23:24:39,545 [INFO] tensorflow: global_step/sec: 1.81977 INFO:tensorflow:epoch = 298.0, learning_rate = 0.00049999997, loss = 3.4812652e-05, step = 1490 (5.616 sec) 2021-09-25 23:24:40,101 [INFO] tensorflow: epoch = 298.0, learning_rate = 0.00049999997, loss = 3.4812652e-05, step = 1490 (5.616 sec) INFO:tensorflow:global_step/sec: 1.79289 2021-09-25 23:24:40,103 [INFO] tensorflow: global_step/sec: 1.79289 2021-09-25 23:24:40,104 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 298/500: loss: 0.00003 learning rate: 0.00050 Time taken: 0:00:02.787143 ETA: 0:09:23.002981 INFO:tensorflow:global_step/sec: 1.79673 2021-09-25 23:24:40,659 [INFO] tensorflow: global_step/sec: 1.79673 INFO:tensorflow:global_step/sec: 1.72715 2021-09-25 23:24:41,238 [INFO] tensorflow: global_step/sec: 1.72715 INFO:tensorflow:global_step/sec: 1.75739 2021-09-25 23:24:41,807 [INFO] tensorflow: global_step/sec: 1.75739 INFO:tensorflow:global_step/sec: 1.795 2021-09-25 23:24:42,364 [INFO] tensorflow: global_step/sec: 1.795 INFO:tensorflow:global_step/sec: 1.74148 2021-09-25 23:24:42,938 [INFO] tensorflow: global_step/sec: 1.74148 2021-09-25 23:24:42,940 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 299/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.818532 ETA: 0:09:26.524834 INFO:tensorflow:global_step/sec: 1.7604 2021-09-25 23:24:43,507 [INFO] tensorflow: global_step/sec: 1.7604 INFO:tensorflow:global_step/sec: 1.80002 2021-09-25 23:24:44,062 [INFO] tensorflow: global_step/sec: 1.80002 INFO:tensorflow:global_step/sec: 1.83868 2021-09-25 23:24:44,606 [INFO] tensorflow: global_step/sec: 1.83868 INFO:tensorflow:global_step/sec: 1.83753 2021-09-25 23:24:45,150 [INFO] tensorflow: global_step/sec: 1.83753 INFO:tensorflow:Saving checkpoints for step-1500. 2021-09-25 23:24:45,151 [INFO] tensorflow: Saving checkpoints for step-1500. WARNING:tensorflow:Ignoring: /tmp/tmp3zxb_m70; No such file or directory 2021-09-25 23:24:45,371 [WARNING] tensorflow: Ignoring: /tmp/tmp3zxb_m70; No such file or directory 2021-09-25 23:24:49,198 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 88.445 INFO:tensorflow:epoch = 300.0, learning_rate = 0.00049999997, loss = 3.6587986e-05, step = 1500 (9.662 sec) 2021-09-25 23:24:49,763 [INFO] tensorflow: epoch = 300.0, learning_rate = 0.00049999997, loss = 3.6587986e-05, step = 1500 (9.662 sec) INFO:tensorflow:global_step/sec: 0.216703 2021-09-25 23:24:49,765 [INFO] tensorflow: global_step/sec: 0.216703 2021-09-25 23:24:49,766 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 300/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:06.830268 ETA: 0:22:46.053581 INFO:tensorflow:global_step/sec: 1.76791 2021-09-25 23:24:50,330 [INFO] tensorflow: global_step/sec: 1.76791 INFO:tensorflow:global_step/sec: 1.82516 2021-09-25 23:24:50,878 [INFO] tensorflow: global_step/sec: 1.82516 INFO:tensorflow:global_step/sec: 1.76638 2021-09-25 23:24:51,444 [INFO] tensorflow: global_step/sec: 1.76638 INFO:tensorflow:global_step/sec: 1.72971 2021-09-25 23:24:52,023 [INFO] tensorflow: global_step/sec: 1.72971 Epoch 301/500 ========================= Validation cost: 0.000000 Mean average_precision (in %): 0.0000 class name average precision (in %) ------------ -------------------------- lpd 0 Median Inference Time: nan INFO:tensorflow:global_step/sec: 1.77917 2021-09-25 23:24:52,585 [INFO] tensorflow: global_step/sec: 1.77917 2021-09-25 23:24:52,586 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 301/500: loss: 0.00003 learning rate: 0.00050 Time taken: 0:00:02.831744 ETA: 0:09:23.517142 INFO:tensorflow:global_step/sec: 1.78746 2021-09-25 23:24:53,144 [INFO] tensorflow: global_step/sec: 1.78746 INFO:tensorflow:global_step/sec: 1.8127 2021-09-25 23:24:53,696 [INFO] tensorflow: global_step/sec: 1.8127 INFO:tensorflow:global_step/sec: 1.81708 2021-09-25 23:24:54,246 [INFO] tensorflow: global_step/sec: 1.81708 INFO:tensorflow:global_step/sec: 1.842 2021-09-25 23:24:54,789 [INFO] tensorflow: global_step/sec: 1.842 INFO:tensorflow:epoch = 302.0, learning_rate = 0.00049999997, loss = 3.871284e-05, step = 1510 (5.578 sec) 2021-09-25 23:24:55,342 [INFO] tensorflow: epoch = 302.0, learning_rate = 0.00049999997, loss = 3.871284e-05, step = 1510 (5.578 sec) INFO:tensorflow:global_step/sec: 1.80458 2021-09-25 23:24:55,343 [INFO] tensorflow: global_step/sec: 1.80458 2021-09-25 23:24:55,344 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 302/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.763484 ETA: 0:09:07.169927 INFO:tensorflow:global_step/sec: 1.83925 2021-09-25 23:24:55,887 [INFO] tensorflow: global_step/sec: 1.83925 INFO:tensorflow:global_step/sec: 1.79859 2021-09-25 23:24:56,443 [INFO] tensorflow: global_step/sec: 1.79859 INFO:tensorflow:global_step/sec: 1.848 2021-09-25 23:24:56,984 [INFO] tensorflow: global_step/sec: 1.848 INFO:tensorflow:global_step/sec: 1.69772 2021-09-25 23:24:57,573 [INFO] tensorflow: global_step/sec: 1.69772 INFO:tensorflow:global_step/sec: 1.77566 2021-09-25 23:24:58,136 [INFO] tensorflow: global_step/sec: 1.77566 2021-09-25 23:24:58,138 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 303/500: loss: 0.00003 learning rate: 0.00050 Time taken: 0:00:02.783335 ETA: 0:09:08.317036 INFO:tensorflow:global_step/sec: 1.81892 2021-09-25 23:24:58,686 [INFO] tensorflow: global_step/sec: 1.81892 INFO:tensorflow:global_step/sec: 1.76812 2021-09-25 23:24:59,251 [INFO] tensorflow: global_step/sec: 1.76812 INFO:tensorflow:global_step/sec: 1.77566 2021-09-25 23:24:59,815 [INFO] tensorflow: global_step/sec: 1.77566 INFO:tensorflow:global_step/sec: 1.8434 2021-09-25 23:25:00,357 [INFO] tensorflow: global_step/sec: 1.8434 INFO:tensorflow:epoch = 304.0, learning_rate = 0.00049999997, loss = 3.9892806e-05, step = 1520 (5.603 sec) 2021-09-25 23:25:00,944 [INFO] tensorflow: epoch = 304.0, learning_rate = 0.00049999997, loss = 3.9892806e-05, step = 1520 (5.603 sec) INFO:tensorflow:global_step/sec: 1.69884 2021-09-25 23:25:00,946 [INFO] tensorflow: global_step/sec: 1.69884 2021-09-25 23:25:00,947 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 304/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.779746 ETA: 0:09:04.830274 INFO:tensorflow:global_step/sec: 1.79244 2021-09-25 23:25:01,504 [INFO] tensorflow: global_step/sec: 1.79244 INFO:tensorflow:global_step/sec: 1.76398 2021-09-25 23:25:02,071 [INFO] tensorflow: global_step/sec: 1.76398 INFO:tensorflow:global_step/sec: 1.77867 2021-09-25 23:25:02,633 [INFO] tensorflow: global_step/sec: 1.77867 INFO:tensorflow:global_step/sec: 1.80029 2021-09-25 23:25:03,188 [INFO] tensorflow: global_step/sec: 1.80029 2021-09-25 23:25:03,194 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 114.326 INFO:tensorflow:global_step/sec: 1.76394 2021-09-25 23:25:03,755 [INFO] tensorflow: global_step/sec: 1.76394 2021-09-25 23:25:03,757 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 305/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.833866 ETA: 0:09:12.603893 INFO:tensorflow:global_step/sec: 1.73197 2021-09-25 23:25:04,333 [INFO] tensorflow: global_step/sec: 1.73197 INFO:tensorflow:global_step/sec: 1.7717 2021-09-25 23:25:04,897 [INFO] tensorflow: global_step/sec: 1.7717 INFO:tensorflow:global_step/sec: 1.81778 2021-09-25 23:25:05,447 [INFO] tensorflow: global_step/sec: 1.81778 INFO:tensorflow:global_step/sec: 1.86228 2021-09-25 23:25:05,984 [INFO] tensorflow: global_step/sec: 1.86228 INFO:tensorflow:epoch = 306.0, learning_rate = 0.00049999997, loss = 3.387738e-05, step = 1530 (5.604 sec) 2021-09-25 23:25:06,548 [INFO] tensorflow: epoch = 306.0, learning_rate = 0.00049999997, loss = 3.387738e-05, step = 1530 (5.604 sec) INFO:tensorflow:global_step/sec: 1.76744 2021-09-25 23:25:06,550 [INFO] tensorflow: global_step/sec: 1.76744 2021-09-25 23:25:06,551 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 306/500: loss: 0.00003 learning rate: 0.00050 Time taken: 0:00:02.792797 ETA: 0:09:01.802635 INFO:tensorflow:global_step/sec: 1.78765 2021-09-25 23:25:07,109 [INFO] tensorflow: global_step/sec: 1.78765 INFO:tensorflow:global_step/sec: 1.83767 2021-09-25 23:25:07,653 [INFO] tensorflow: global_step/sec: 1.83767 INFO:tensorflow:global_step/sec: 1.76096 2021-09-25 23:25:08,221 [INFO] tensorflow: global_step/sec: 1.76096 INFO:tensorflow:global_step/sec: 1.81273 2021-09-25 23:25:08,773 [INFO] tensorflow: global_step/sec: 1.81273 INFO:tensorflow:global_step/sec: 1.79605 2021-09-25 23:25:09,330 [INFO] tensorflow: global_step/sec: 1.79605 2021-09-25 23:25:09,331 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 307/500: loss: 0.00005 learning rate: 0.00050 Time taken: 0:00:02.789318 ETA: 0:08:58.338390 INFO:tensorflow:global_step/sec: 1.7328 2021-09-25 23:25:09,907 [INFO] tensorflow: global_step/sec: 1.7328 INFO:tensorflow:global_step/sec: 1.78809 2021-09-25 23:25:10,466 [INFO] tensorflow: global_step/sec: 1.78809 INFO:tensorflow:global_step/sec: 1.81016 2021-09-25 23:25:11,019 [INFO] tensorflow: global_step/sec: 1.81016 INFO:tensorflow:global_step/sec: 1.80653 2021-09-25 23:25:11,572 [INFO] tensorflow: global_step/sec: 1.80653 INFO:tensorflow:epoch = 308.0, learning_rate = 0.00049999997, loss = 3.560615e-05, step = 1540 (5.581 sec) 2021-09-25 23:25:12,129 [INFO] tensorflow: epoch = 308.0, learning_rate = 0.00049999997, loss = 3.560615e-05, step = 1540 (5.581 sec) INFO:tensorflow:global_step/sec: 1.79015 2021-09-25 23:25:12,131 [INFO] tensorflow: global_step/sec: 1.79015 2021-09-25 23:25:12,132 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 308/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.799719 ETA: 0:08:57.546021 INFO:tensorflow:global_step/sec: 1.82441 2021-09-25 23:25:12,679 [INFO] tensorflow: global_step/sec: 1.82441 INFO:tensorflow:global_step/sec: 1.69135 2021-09-25 23:25:13,270 [INFO] tensorflow: global_step/sec: 1.69135 INFO:tensorflow:global_step/sec: 1.81419 2021-09-25 23:25:13,821 [INFO] tensorflow: global_step/sec: 1.81419 INFO:tensorflow:global_step/sec: 1.76699 2021-09-25 23:25:14,387 [INFO] tensorflow: global_step/sec: 1.76699 INFO:tensorflow:global_step/sec: 1.6964 2021-09-25 23:25:14,977 [INFO] tensorflow: global_step/sec: 1.6964 2021-09-25 23:25:14,978 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 309/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.812196 ETA: 0:08:57.129394 INFO:tensorflow:global_step/sec: 1.78177 2021-09-25 23:25:15,538 [INFO] tensorflow: global_step/sec: 1.78177 INFO:tensorflow:global_step/sec: 1.81043 2021-09-25 23:25:16,090 [INFO] tensorflow: global_step/sec: 1.81043 INFO:tensorflow:global_step/sec: 1.78601 2021-09-25 23:25:16,650 [INFO] tensorflow: global_step/sec: 1.78601 INFO:tensorflow:global_step/sec: 1.74855 2021-09-25 23:25:17,222 [INFO] tensorflow: global_step/sec: 1.74855 2021-09-25 23:25:17,227 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 114.024 INFO:tensorflow:epoch = 310.0, learning_rate = 0.00049999997, loss = 3.9510567e-05, step = 1550 (5.667 sec) 2021-09-25 23:25:17,796 [INFO] tensorflow: epoch = 310.0, learning_rate = 0.00049999997, loss = 3.9510567e-05, step = 1550 (5.667 sec) INFO:tensorflow:global_step/sec: 1.73888 2021-09-25 23:25:17,797 [INFO] tensorflow: global_step/sec: 1.73888 2021-09-25 23:25:17,798 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 310/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.835184 ETA: 0:08:58.684933 INFO:tensorflow:global_step/sec: 1.73401 2021-09-25 23:25:18,374 [INFO] tensorflow: global_step/sec: 1.73401 INFO:tensorflow:global_step/sec: 1.68575 2021-09-25 23:25:18,967 [INFO] tensorflow: global_step/sec: 1.68575 INFO:tensorflow:global_step/sec: 1.81102 2021-09-25 23:25:19,519 [INFO] tensorflow: global_step/sec: 1.81102 INFO:tensorflow:global_step/sec: 1.76954 2021-09-25 23:25:20,084 [INFO] tensorflow: global_step/sec: 1.76954 INFO:tensorflow:global_step/sec: 1.69458 2021-09-25 23:25:20,674 [INFO] tensorflow: global_step/sec: 1.69458 2021-09-25 23:25:20,676 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 311/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.863839 ETA: 0:09:01.265644 INFO:tensorflow:global_step/sec: 1.79909 2021-09-25 23:25:21,230 [INFO] tensorflow: global_step/sec: 1.79909 INFO:tensorflow:global_step/sec: 1.78264 2021-09-25 23:25:21,791 [INFO] tensorflow: global_step/sec: 1.78264 INFO:tensorflow:global_step/sec: 1.7349 2021-09-25 23:25:22,368 [INFO] tensorflow: global_step/sec: 1.7349 INFO:tensorflow:global_step/sec: 1.70244 2021-09-25 23:25:22,955 [INFO] tensorflow: global_step/sec: 1.70244 INFO:tensorflow:epoch = 312.0, learning_rate = 0.00049999997, loss = 4.6185647e-05, step = 1560 (5.714 sec) 2021-09-25 23:25:23,510 [INFO] tensorflow: epoch = 312.0, learning_rate = 0.00049999997, loss = 4.6185647e-05, step = 1560 (5.714 sec) INFO:tensorflow:global_step/sec: 1.7981 2021-09-25 23:25:23,511 [INFO] tensorflow: global_step/sec: 1.7981 2021-09-25 23:25:23,512 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 312/500: loss: 0.00005 learning rate: 0.00050 Time taken: 0:00:02.868140 ETA: 0:08:59.210227 INFO:tensorflow:global_step/sec: 1.75156 2021-09-25 23:25:24,082 [INFO] tensorflow: global_step/sec: 1.75156 INFO:tensorflow:global_step/sec: 1.75276 2021-09-25 23:25:24,653 [INFO] tensorflow: global_step/sec: 1.75276 INFO:tensorflow:global_step/sec: 1.82521 2021-09-25 23:25:25,201 [INFO] tensorflow: global_step/sec: 1.82521 INFO:tensorflow:global_step/sec: 1.73196 2021-09-25 23:25:25,778 [INFO] tensorflow: global_step/sec: 1.73196 INFO:tensorflow:global_step/sec: 1.68981 2021-09-25 23:25:26,370 [INFO] tensorflow: global_step/sec: 1.68981 2021-09-25 23:25:26,371 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 313/500: loss: 0.00003 learning rate: 0.00050 Time taken: 0:00:02.820724 ETA: 0:08:47.475390 INFO:tensorflow:global_step/sec: 1.82442 2021-09-25 23:25:26,918 [INFO] tensorflow: global_step/sec: 1.82442 INFO:tensorflow:global_step/sec: 1.80605 2021-09-25 23:25:27,472 [INFO] tensorflow: global_step/sec: 1.80605 INFO:tensorflow:global_step/sec: 1.74113 2021-09-25 23:25:28,046 [INFO] tensorflow: global_step/sec: 1.74113 INFO:tensorflow:global_step/sec: 1.73801 2021-09-25 23:25:28,621 [INFO] tensorflow: global_step/sec: 1.73801 INFO:tensorflow:epoch = 314.0, learning_rate = 0.00049999997, loss = 3.154311e-05, step = 1570 (5.664 sec) 2021-09-25 23:25:29,173 [INFO] tensorflow: epoch = 314.0, learning_rate = 0.00049999997, loss = 3.154311e-05, step = 1570 (5.664 sec) INFO:tensorflow:global_step/sec: 1.80649 2021-09-25 23:25:29,175 [INFO] tensorflow: global_step/sec: 1.80649 2021-09-25 23:25:29,176 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 314/500: loss: 0.00003 learning rate: 0.00050 Time taken: 0:00:02.845644 ETA: 0:08:49.289872 INFO:tensorflow:global_step/sec: 1.7783 2021-09-25 23:25:29,737 [INFO] tensorflow: global_step/sec: 1.7783 INFO:tensorflow:global_step/sec: 1.82903 2021-09-25 23:25:30,284 [INFO] tensorflow: global_step/sec: 1.82903 INFO:tensorflow:global_step/sec: 1.79652 2021-09-25 23:25:30,840 [INFO] tensorflow: global_step/sec: 1.79652 INFO:tensorflow:global_step/sec: 1.76233 2021-09-25 23:25:31,408 [INFO] tensorflow: global_step/sec: 1.76233 2021-09-25 23:25:31,414 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 112.789 INFO:tensorflow:global_step/sec: 1.72502 2021-09-25 23:25:31,988 [INFO] tensorflow: global_step/sec: 1.72502 2021-09-25 23:25:31,989 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 315/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.784505 ETA: 0:08:35.133493 INFO:tensorflow:global_step/sec: 1.84685 2021-09-25 23:25:32,529 [INFO] tensorflow: global_step/sec: 1.84685 INFO:tensorflow:global_step/sec: 1.7029 2021-09-25 23:25:33,116 [INFO] tensorflow: global_step/sec: 1.7029 INFO:tensorflow:global_step/sec: 1.78797 2021-09-25 23:25:33,676 [INFO] tensorflow: global_step/sec: 1.78797 INFO:tensorflow:global_step/sec: 1.79266 2021-09-25 23:25:34,233 [INFO] tensorflow: global_step/sec: 1.79266 INFO:tensorflow:epoch = 316.0, learning_rate = 0.00049999997, loss = 3.9029826e-05, step = 1580 (5.602 sec) 2021-09-25 23:25:34,775 [INFO] tensorflow: epoch = 316.0, learning_rate = 0.00049999997, loss = 3.9029826e-05, step = 1580 (5.602 sec) INFO:tensorflow:global_step/sec: 1.84023 2021-09-25 23:25:34,777 [INFO] tensorflow: global_step/sec: 1.84023 2021-09-25 23:25:34,778 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 316/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.821447 ETA: 0:08:39.146273 INFO:tensorflow:global_step/sec: 1.80816 2021-09-25 23:25:35,330 [INFO] tensorflow: global_step/sec: 1.80816 INFO:tensorflow:global_step/sec: 1.77776 2021-09-25 23:25:35,892 [INFO] tensorflow: global_step/sec: 1.77776 INFO:tensorflow:global_step/sec: 1.80752 2021-09-25 23:25:36,446 [INFO] tensorflow: global_step/sec: 1.80752 INFO:tensorflow:global_step/sec: 1.80699 2021-09-25 23:25:36,999 [INFO] tensorflow: global_step/sec: 1.80699 INFO:tensorflow:global_step/sec: 1.82268 2021-09-25 23:25:37,548 [INFO] tensorflow: global_step/sec: 1.82268 2021-09-25 23:25:37,549 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 317/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.768518 ETA: 0:08:26.638745 INFO:tensorflow:global_step/sec: 1.80277 2021-09-25 23:25:38,102 [INFO] tensorflow: global_step/sec: 1.80277 INFO:tensorflow:global_step/sec: 1.828 2021-09-25 23:25:38,649 [INFO] tensorflow: global_step/sec: 1.828 INFO:tensorflow:global_step/sec: 1.83152 2021-09-25 23:25:39,195 [INFO] tensorflow: global_step/sec: 1.83152 INFO:tensorflow:global_step/sec: 1.82613 2021-09-25 23:25:39,743 [INFO] tensorflow: global_step/sec: 1.82613 INFO:tensorflow:global_step/sec: 1.7656 2021-09-25 23:25:40,309 [INFO] tensorflow: global_step/sec: 1.7656 2021-09-25 23:25:40,311 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 318/500: loss: 0.00003 learning rate: 0.00050 Time taken: 0:00:02.740907 ETA: 0:08:18.845022 INFO:tensorflow:epoch = 318.20000000000005, learning_rate = 0.00049999997, loss = 3.7523216e-05, step = 1591 (6.086 sec) 2021-09-25 23:25:40,862 [INFO] tensorflow: epoch = 318.20000000000005, learning_rate = 0.00049999997, loss = 3.7523216e-05, step = 1591 (6.086 sec) INFO:tensorflow:global_step/sec: 1.80628 2021-09-25 23:25:40,863 [INFO] tensorflow: global_step/sec: 1.80628 INFO:tensorflow:global_step/sec: 1.7409 2021-09-25 23:25:41,437 [INFO] tensorflow: global_step/sec: 1.7409 INFO:tensorflow:global_step/sec: 1.73752 2021-09-25 23:25:42,013 [INFO] tensorflow: global_step/sec: 1.73752 INFO:tensorflow:global_step/sec: 1.76539 2021-09-25 23:25:42,579 [INFO] tensorflow: global_step/sec: 1.76539 INFO:tensorflow:global_step/sec: 1.74114 2021-09-25 23:25:43,154 [INFO] tensorflow: global_step/sec: 1.74114 2021-09-25 23:25:43,155 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 319/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.837798 ETA: 0:08:33.641373 INFO:tensorflow:global_step/sec: 1.70779 2021-09-25 23:25:43,739 [INFO] tensorflow: global_step/sec: 1.70779 INFO:tensorflow:global_step/sec: 1.74582 2021-09-25 23:25:44,312 [INFO] tensorflow: global_step/sec: 1.74582 INFO:tensorflow:global_step/sec: 1.72893 2021-09-25 23:25:44,891 [INFO] tensorflow: global_step/sec: 1.72893 INFO:tensorflow:global_step/sec: 1.76812 2021-09-25 23:25:45,456 [INFO] tensorflow: global_step/sec: 1.76812 INFO:tensorflow:Saving checkpoints for step-1600. 2021-09-25 23:25:45,457 [INFO] tensorflow: Saving checkpoints for step-1600. WARNING:tensorflow:Ignoring: /tmp/tmp02lf4ulk; No such file or directory 2021-09-25 23:25:45,681 [WARNING] tensorflow: Ignoring: /tmp/tmp02lf4ulk; No such file or directory 2021-09-25 23:25:49,558 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 88.190 INFO:tensorflow:epoch = 320.0, learning_rate = 0.00049999997, loss = 3.8122867e-05, step = 1600 (9.274 sec) 2021-09-25 23:25:50,136 [INFO] tensorflow: epoch = 320.0, learning_rate = 0.00049999997, loss = 3.8122867e-05, step = 1600 (9.274 sec) INFO:tensorflow:global_step/sec: 0.213618 2021-09-25 23:25:50,137 [INFO] tensorflow: global_step/sec: 0.213618 2021-09-25 23:25:50,139 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 320/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:06.971150 ETA: 0:20:54.807072 INFO:tensorflow:global_step/sec: 1.80493 2021-09-25 23:25:50,691 [INFO] tensorflow: global_step/sec: 1.80493 INFO:tensorflow:global_step/sec: 1.83612 2021-09-25 23:25:51,236 [INFO] tensorflow: global_step/sec: 1.83612 INFO:tensorflow:global_step/sec: 1.78025 2021-09-25 23:25:51,798 [INFO] tensorflow: global_step/sec: 1.78025 INFO:tensorflow:global_step/sec: 1.83242 2021-09-25 23:25:52,343 [INFO] tensorflow: global_step/sec: 1.83242 Epoch 321/500 ========================= Validation cost: 0.000000 Mean average_precision (in %): 0.0000 class name average precision (in %) ------------ -------------------------- lpd 0 Median Inference Time: nan INFO:tensorflow:global_step/sec: 1.76866 2021-09-25 23:25:52,909 [INFO] tensorflow: global_step/sec: 1.76866 2021-09-25 23:25:52,910 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 321/500: loss: 0.00003 learning rate: 0.00050 Time taken: 0:00:02.786989 ETA: 0:08:18.871069 INFO:tensorflow:global_step/sec: 1.77524 2021-09-25 23:25:53,472 [INFO] tensorflow: global_step/sec: 1.77524 INFO:tensorflow:global_step/sec: 1.72469 2021-09-25 23:25:54,052 [INFO] tensorflow: global_step/sec: 1.72469 INFO:tensorflow:global_step/sec: 1.74474 2021-09-25 23:25:54,625 [INFO] tensorflow: global_step/sec: 1.74474 INFO:tensorflow:global_step/sec: 1.79174 2021-09-25 23:25:55,183 [INFO] tensorflow: global_step/sec: 1.79174 INFO:tensorflow:epoch = 322.0, learning_rate = 0.00049999997, loss = 3.651581e-05, step = 1610 (5.607 sec) 2021-09-25 23:25:55,743 [INFO] tensorflow: epoch = 322.0, learning_rate = 0.00049999997, loss = 3.651581e-05, step = 1610 (5.607 sec) INFO:tensorflow:global_step/sec: 1.78114 2021-09-25 23:25:55,745 [INFO] tensorflow: global_step/sec: 1.78114 2021-09-25 23:25:55,746 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 322/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.839370 ETA: 0:08:25.407905 INFO:tensorflow:global_step/sec: 1.77949 2021-09-25 23:25:56,307 [INFO] tensorflow: global_step/sec: 1.77949 INFO:tensorflow:global_step/sec: 1.82738 2021-09-25 23:25:56,854 [INFO] tensorflow: global_step/sec: 1.82738 INFO:tensorflow:global_step/sec: 1.79545 2021-09-25 23:25:57,411 [INFO] tensorflow: global_step/sec: 1.79545 INFO:tensorflow:global_step/sec: 1.75912 2021-09-25 23:25:57,979 [INFO] tensorflow: global_step/sec: 1.75912 INFO:tensorflow:global_step/sec: 1.82147 2021-09-25 23:25:58,528 [INFO] tensorflow: global_step/sec: 1.82147 2021-09-25 23:25:58,530 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 323/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.796884 ETA: 0:08:15.048479 INFO:tensorflow:global_step/sec: 1.77823 2021-09-25 23:25:59,091 [INFO] tensorflow: global_step/sec: 1.77823 INFO:tensorflow:global_step/sec: 1.78608 2021-09-25 23:25:59,651 [INFO] tensorflow: global_step/sec: 1.78608 INFO:tensorflow:global_step/sec: 1.76112 2021-09-25 23:26:00,218 [INFO] tensorflow: global_step/sec: 1.76112 INFO:tensorflow:global_step/sec: 1.74333 2021-09-25 23:26:00,792 [INFO] tensorflow: global_step/sec: 1.74333 INFO:tensorflow:epoch = 324.0, learning_rate = 0.00049999997, loss = 3.543109e-05, step = 1620 (5.605 sec) 2021-09-25 23:26:01,348 [INFO] tensorflow: epoch = 324.0, learning_rate = 0.00049999997, loss = 3.543109e-05, step = 1620 (5.605 sec) INFO:tensorflow:global_step/sec: 1.793 2021-09-25 23:26:01,350 [INFO] tensorflow: global_step/sec: 1.793 2021-09-25 23:26:01,351 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 324/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.809474 ETA: 0:08:14.467464 INFO:tensorflow:global_step/sec: 1.81211 2021-09-25 23:26:01,902 [INFO] tensorflow: global_step/sec: 1.81211 INFO:tensorflow:global_step/sec: 1.62471 2021-09-25 23:26:02,517 [INFO] tensorflow: global_step/sec: 1.62471 INFO:tensorflow:global_step/sec: 1.82053 2021-09-25 23:26:03,066 [INFO] tensorflow: global_step/sec: 1.82053 INFO:tensorflow:global_step/sec: 1.74817 2021-09-25 23:26:03,638 [INFO] tensorflow: global_step/sec: 1.74817 2021-09-25 23:26:03,644 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 113.617 INFO:tensorflow:global_step/sec: 1.74844 2021-09-25 23:26:04,210 [INFO] tensorflow: global_step/sec: 1.74844 2021-09-25 23:26:04,212 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 325/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.846703 ETA: 0:08:18.173076 INFO:tensorflow:global_step/sec: 1.79302 2021-09-25 23:26:04,768 [INFO] tensorflow: global_step/sec: 1.79302 INFO:tensorflow:global_step/sec: 1.82126 2021-09-25 23:26:05,317 [INFO] tensorflow: global_step/sec: 1.82126 INFO:tensorflow:global_step/sec: 1.79718 2021-09-25 23:26:05,874 [INFO] tensorflow: global_step/sec: 1.79718 INFO:tensorflow:global_step/sec: 1.75422 2021-09-25 23:26:06,444 [INFO] tensorflow: global_step/sec: 1.75422 INFO:tensorflow:epoch = 326.0, learning_rate = 0.00049999997, loss = 3.4542274e-05, step = 1630 (5.682 sec) 2021-09-25 23:26:07,030 [INFO] tensorflow: epoch = 326.0, learning_rate = 0.00049999997, loss = 3.4542274e-05, step = 1630 (5.682 sec) INFO:tensorflow:global_step/sec: 1.70011 2021-09-25 23:26:07,032 [INFO] tensorflow: global_step/sec: 1.70011 2021-09-25 23:26:07,033 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 326/500: loss: 0.00003 learning rate: 0.00050 Time taken: 0:00:02.803663 ETA: 0:08:07.837406 INFO:tensorflow:global_step/sec: 1.81583 2021-09-25 23:26:07,583 [INFO] tensorflow: global_step/sec: 1.81583 INFO:tensorflow:global_step/sec: 1.77338 2021-09-25 23:26:08,146 [INFO] tensorflow: global_step/sec: 1.77338 INFO:tensorflow:global_step/sec: 1.66984 2021-09-25 23:26:08,745 [INFO] tensorflow: global_step/sec: 1.66984 INFO:tensorflow:global_step/sec: 1.83335 2021-09-25 23:26:09,291 [INFO] tensorflow: global_step/sec: 1.83335 INFO:tensorflow:global_step/sec: 1.76636 2021-09-25 23:26:09,857 [INFO] tensorflow: global_step/sec: 1.76636 2021-09-25 23:26:09,863 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 327/500: loss: 0.00003 learning rate: 0.00050 Time taken: 0:00:02.848499 ETA: 0:08:12.790379 INFO:tensorflow:global_step/sec: 1.47296 2021-09-25 23:26:10,536 [INFO] tensorflow: global_step/sec: 1.47296 INFO:tensorflow:global_step/sec: 1.41071 2021-09-25 23:26:11,245 [INFO] tensorflow: global_step/sec: 1.41071 INFO:tensorflow:global_step/sec: 1.56228 2021-09-25 23:26:11,885 [INFO] tensorflow: global_step/sec: 1.56228 INFO:tensorflow:global_step/sec: 1.60181 2021-09-25 23:26:12,509 [INFO] tensorflow: global_step/sec: 1.60181 INFO:tensorflow:epoch = 328.0, learning_rate = 0.00049999997, loss = 3.5569654e-05, step = 1640 (6.048 sec) 2021-09-25 23:26:13,079 [INFO] tensorflow: epoch = 328.0, learning_rate = 0.00049999997, loss = 3.5569654e-05, step = 1640 (6.048 sec) INFO:tensorflow:global_step/sec: 1.75059 2021-09-25 23:26:13,080 [INFO] tensorflow: global_step/sec: 1.75059 2021-09-25 23:26:13,082 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 328/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:03.216990 ETA: 0:09:13.322361 INFO:tensorflow:global_step/sec: 1.83693 2021-09-25 23:26:13,625 [INFO] tensorflow: global_step/sec: 1.83693 INFO:tensorflow:global_step/sec: 1.77254 2021-09-25 23:26:14,189 [INFO] tensorflow: global_step/sec: 1.77254 INFO:tensorflow:global_step/sec: 1.72587 2021-09-25 23:26:14,768 [INFO] tensorflow: global_step/sec: 1.72587 INFO:tensorflow:global_step/sec: 1.82609 2021-09-25 23:26:15,316 [INFO] tensorflow: global_step/sec: 1.82609 INFO:tensorflow:global_step/sec: 1.77896 2021-09-25 23:26:15,878 [INFO] tensorflow: global_step/sec: 1.77896 2021-09-25 23:26:15,879 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 329/500: loss: 0.00003 learning rate: 0.00050 Time taken: 0:00:02.803961 ETA: 0:07:59.477297 INFO:tensorflow:global_step/sec: 1.80477 2021-09-25 23:26:16,432 [INFO] tensorflow: global_step/sec: 1.80477 INFO:tensorflow:global_step/sec: 1.81969 2021-09-25 23:26:16,982 [INFO] tensorflow: global_step/sec: 1.81969 INFO:tensorflow:global_step/sec: 1.78123 2021-09-25 23:26:17,543 [INFO] tensorflow: global_step/sec: 1.78123 INFO:tensorflow:global_step/sec: 1.82731 2021-09-25 23:26:18,090 [INFO] tensorflow: global_step/sec: 1.82731 2021-09-25 23:26:18,093 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 110.735 INFO:tensorflow:epoch = 330.0, learning_rate = 0.00049999997, loss = 3.3138578e-05, step = 1650 (5.572 sec) 2021-09-25 23:26:18,651 [INFO] tensorflow: epoch = 330.0, learning_rate = 0.00049999997, loss = 3.3138578e-05, step = 1650 (5.572 sec) INFO:tensorflow:global_step/sec: 1.77988 2021-09-25 23:26:18,652 [INFO] tensorflow: global_step/sec: 1.77988 2021-09-25 23:26:18,653 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 330/500: loss: 0.00003 learning rate: 0.00050 Time taken: 0:00:02.772965 ETA: 0:07:51.404083 INFO:tensorflow:global_step/sec: 1.78745 2021-09-25 23:26:19,212 [INFO] tensorflow: global_step/sec: 1.78745 INFO:tensorflow:global_step/sec: 1.81011 2021-09-25 23:26:19,764 [INFO] tensorflow: global_step/sec: 1.81011 INFO:tensorflow:global_step/sec: 1.73381 2021-09-25 23:26:20,341 [INFO] tensorflow: global_step/sec: 1.73381 INFO:tensorflow:global_step/sec: 1.84559 2021-09-25 23:26:20,883 [INFO] tensorflow: global_step/sec: 1.84559 INFO:tensorflow:global_step/sec: 1.78392 2021-09-25 23:26:21,443 [INFO] tensorflow: global_step/sec: 1.78392 2021-09-25 23:26:21,445 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 331/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.792068 ETA: 0:07:51.859412 INFO:tensorflow:global_step/sec: 1.73198 2021-09-25 23:26:22,021 [INFO] tensorflow: global_step/sec: 1.73198 INFO:tensorflow:global_step/sec: 1.73218 2021-09-25 23:26:22,598 [INFO] tensorflow: global_step/sec: 1.73218 INFO:tensorflow:global_step/sec: 1.79615 2021-09-25 23:26:23,155 [INFO] tensorflow: global_step/sec: 1.79615 INFO:tensorflow:global_step/sec: 1.77766 2021-09-25 23:26:23,717 [INFO] tensorflow: global_step/sec: 1.77766 INFO:tensorflow:epoch = 332.0, learning_rate = 0.00049999997, loss = 3.57621e-05, step = 1660 (5.625 sec) 2021-09-25 23:26:24,276 [INFO] tensorflow: epoch = 332.0, learning_rate = 0.00049999997, loss = 3.57621e-05, step = 1660 (5.625 sec) INFO:tensorflow:global_step/sec: 1.78484 2021-09-25 23:26:24,277 [INFO] tensorflow: global_step/sec: 1.78484 2021-09-25 23:26:24,279 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 332/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.833371 ETA: 0:07:56.006395 INFO:tensorflow:global_step/sec: 1.73382 2021-09-25 23:26:24,854 [INFO] tensorflow: global_step/sec: 1.73382 INFO:tensorflow:global_step/sec: 1.78177 2021-09-25 23:26:25,415 [INFO] tensorflow: global_step/sec: 1.78177 INFO:tensorflow:global_step/sec: 1.84101 2021-09-25 23:26:25,959 [INFO] tensorflow: global_step/sec: 1.84101 INFO:tensorflow:global_step/sec: 1.72009 2021-09-25 23:26:26,540 [INFO] tensorflow: global_step/sec: 1.72009 INFO:tensorflow:global_step/sec: 1.81621 2021-09-25 23:26:27,091 [INFO] tensorflow: global_step/sec: 1.81621 2021-09-25 23:26:27,092 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 333/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.822623 ETA: 0:07:51.378123 INFO:tensorflow:global_step/sec: 1.84544 2021-09-25 23:26:27,632 [INFO] tensorflow: global_step/sec: 1.84544 INFO:tensorflow:global_step/sec: 1.78068 2021-09-25 23:26:28,194 [INFO] tensorflow: global_step/sec: 1.78068 INFO:tensorflow:global_step/sec: 1.70446 2021-09-25 23:26:28,781 [INFO] tensorflow: global_step/sec: 1.70446 INFO:tensorflow:global_step/sec: 1.82364 2021-09-25 23:26:29,329 [INFO] tensorflow: global_step/sec: 1.82364 INFO:tensorflow:epoch = 334.0, learning_rate = 0.00049999997, loss = 4.215227e-05, step = 1670 (5.595 sec) 2021-09-25 23:26:29,871 [INFO] tensorflow: epoch = 334.0, learning_rate = 0.00049999997, loss = 4.215227e-05, step = 1670 (5.595 sec) INFO:tensorflow:global_step/sec: 1.83931 2021-09-25 23:26:29,873 [INFO] tensorflow: global_step/sec: 1.83931 2021-09-25 23:26:29,874 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 334/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.789585 ETA: 0:07:43.071089 INFO:tensorflow:global_step/sec: 1.80467 2021-09-25 23:26:30,427 [INFO] tensorflow: global_step/sec: 1.80467 INFO:tensorflow:global_step/sec: 1.77549 2021-09-25 23:26:30,990 [INFO] tensorflow: global_step/sec: 1.77549 INFO:tensorflow:global_step/sec: 1.72824 2021-09-25 23:26:31,569 [INFO] tensorflow: global_step/sec: 1.72824 INFO:tensorflow:global_step/sec: 1.6931 2021-09-25 23:26:32,159 [INFO] tensorflow: global_step/sec: 1.6931 2021-09-25 23:26:32,165 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 113.708 INFO:tensorflow:global_step/sec: 1.75905 2021-09-25 23:26:32,728 [INFO] tensorflow: global_step/sec: 1.75905 2021-09-25 23:26:32,729 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 335/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.830646 ETA: 0:07:47.056596 INFO:tensorflow:global_step/sec: 1.76213 2021-09-25 23:26:33,295 [INFO] tensorflow: global_step/sec: 1.76213 INFO:tensorflow:global_step/sec: 1.79041 2021-09-25 23:26:33,854 [INFO] tensorflow: global_step/sec: 1.79041 INFO:tensorflow:global_step/sec: 1.82529 2021-09-25 23:26:34,402 [INFO] tensorflow: global_step/sec: 1.82529 INFO:tensorflow:global_step/sec: 1.75934 2021-09-25 23:26:34,970 [INFO] tensorflow: global_step/sec: 1.75934 INFO:tensorflow:epoch = 336.0, learning_rate = 0.00049999997, loss = 3.6214602e-05, step = 1680 (5.670 sec) 2021-09-25 23:26:35,542 [INFO] tensorflow: epoch = 336.0, learning_rate = 0.00049999997, loss = 3.6214602e-05, step = 1680 (5.670 sec) INFO:tensorflow:global_step/sec: 1.7455 2021-09-25 23:26:35,543 [INFO] tensorflow: global_step/sec: 1.7455 2021-09-25 23:26:35,544 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 336/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.810782 ETA: 0:07:40.968280 INFO:tensorflow:global_step/sec: 1.79751 2021-09-25 23:26:36,099 [INFO] tensorflow: global_step/sec: 1.79751 INFO:tensorflow:global_step/sec: 1.75758 2021-09-25 23:26:36,668 [INFO] tensorflow: global_step/sec: 1.75758 INFO:tensorflow:global_step/sec: 1.8182 2021-09-25 23:26:37,218 [INFO] tensorflow: global_step/sec: 1.8182 INFO:tensorflow:global_step/sec: 1.72672 2021-09-25 23:26:37,797 [INFO] tensorflow: global_step/sec: 1.72672 INFO:tensorflow:global_step/sec: 1.74812 2021-09-25 23:26:38,369 [INFO] tensorflow: global_step/sec: 1.74812 2021-09-25 23:26:38,371 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 337/500: loss: 0.00003 learning rate: 0.00050 Time taken: 0:00:02.822236 ETA: 0:07:40.024478 INFO:tensorflow:global_step/sec: 1.78472 2021-09-25 23:26:38,930 [INFO] tensorflow: global_step/sec: 1.78472 INFO:tensorflow:global_step/sec: 1.78534 2021-09-25 23:26:39,490 [INFO] tensorflow: global_step/sec: 1.78534 INFO:tensorflow:global_step/sec: 1.77331 2021-09-25 23:26:40,054 [INFO] tensorflow: global_step/sec: 1.77331 INFO:tensorflow:global_step/sec: 1.75548 2021-09-25 23:26:40,623 [INFO] tensorflow: global_step/sec: 1.75548 INFO:tensorflow:epoch = 338.0, learning_rate = 0.00049999997, loss = 4.32515e-05, step = 1690 (5.661 sec) 2021-09-25 23:26:41,202 [INFO] tensorflow: epoch = 338.0, learning_rate = 0.00049999997, loss = 4.32515e-05, step = 1690 (5.661 sec) INFO:tensorflow:global_step/sec: 1.72345 2021-09-25 23:26:41,204 [INFO] tensorflow: global_step/sec: 1.72345 2021-09-25 23:26:41,205 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 338/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.828802 ETA: 0:07:38.265942 INFO:tensorflow:global_step/sec: 1.78785 2021-09-25 23:26:41,763 [INFO] tensorflow: global_step/sec: 1.78785 INFO:tensorflow:global_step/sec: 1.79001 2021-09-25 23:26:42,322 [INFO] tensorflow: global_step/sec: 1.79001 INFO:tensorflow:global_step/sec: 1.7946 2021-09-25 23:26:42,879 [INFO] tensorflow: global_step/sec: 1.7946 INFO:tensorflow:global_step/sec: 1.81781 2021-09-25 23:26:43,429 [INFO] tensorflow: global_step/sec: 1.81781 INFO:tensorflow:global_step/sec: 1.78694 2021-09-25 23:26:43,989 [INFO] tensorflow: global_step/sec: 1.78694 2021-09-25 23:26:43,990 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 339/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.805025 ETA: 0:07:31.609080 INFO:tensorflow:global_step/sec: 1.76714 2021-09-25 23:26:44,555 [INFO] tensorflow: global_step/sec: 1.76714 INFO:tensorflow:global_step/sec: 1.83645 2021-09-25 23:26:45,099 [INFO] tensorflow: global_step/sec: 1.83645 INFO:tensorflow:global_step/sec: 1.81157 2021-09-25 23:26:45,651 [INFO] tensorflow: global_step/sec: 1.81157 INFO:tensorflow:global_step/sec: 1.76364 2021-09-25 23:26:46,218 [INFO] tensorflow: global_step/sec: 1.76364 INFO:tensorflow:Saving checkpoints for step-1700. 2021-09-25 23:26:46,219 [INFO] tensorflow: Saving checkpoints for step-1700. WARNING:tensorflow:Ignoring: /tmp/tmpbxob5ip9; No such file or directory 2021-09-25 23:26:46,440 [WARNING] tensorflow: Ignoring: /tmp/tmpbxob5ip9; No such file or directory 2021-09-25 23:26:50,272 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 88.370 INFO:tensorflow:epoch = 340.0, learning_rate = 0.00049999997, loss = 3.8557966e-05, step = 1700 (9.633 sec) 2021-09-25 23:26:50,835 [INFO] tensorflow: epoch = 340.0, learning_rate = 0.00049999997, loss = 3.8557966e-05, step = 1700 (9.633 sec) INFO:tensorflow:global_step/sec: 0.216515 2021-09-25 23:26:50,837 [INFO] tensorflow: global_step/sec: 0.216515 2021-09-25 23:26:50,838 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 340/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:06.836038 ETA: 0:18:13.766022 INFO:tensorflow:global_step/sec: 1.70006 2021-09-25 23:26:51,425 [INFO] tensorflow: global_step/sec: 1.70006 INFO:tensorflow:global_step/sec: 1.76544 2021-09-25 23:26:51,991 [INFO] tensorflow: global_step/sec: 1.76544 INFO:tensorflow:global_step/sec: 1.78023 2021-09-25 23:26:52,553 [INFO] tensorflow: global_step/sec: 1.78023 INFO:tensorflow:global_step/sec: 1.80647 2021-09-25 23:26:53,107 [INFO] tensorflow: global_step/sec: 1.80647 Epoch 341/500 ========================= Validation cost: 0.000000 Mean average_precision (in %): 0.0000 class name average precision (in %) ------------ -------------------------- lpd 0 Median Inference Time: nan INFO:tensorflow:global_step/sec: 1.78075 2021-09-25 23:26:53,668 [INFO] tensorflow: global_step/sec: 1.78075 2021-09-25 23:26:53,670 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 341/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.838707 ETA: 0:07:31.354484 INFO:tensorflow:global_step/sec: 1.71109 2021-09-25 23:26:54,253 [INFO] tensorflow: global_step/sec: 1.71109 INFO:tensorflow:global_step/sec: 1.76378 2021-09-25 23:26:54,820 [INFO] tensorflow: global_step/sec: 1.76378 INFO:tensorflow:global_step/sec: 1.86692 2021-09-25 23:26:55,355 [INFO] tensorflow: global_step/sec: 1.86692 INFO:tensorflow:global_step/sec: 1.77368 2021-09-25 23:26:55,919 [INFO] tensorflow: global_step/sec: 1.77368 INFO:tensorflow:epoch = 342.0, learning_rate = 0.00049999997, loss = 3.685812e-05, step = 1710 (5.651 sec) 2021-09-25 23:26:56,486 [INFO] tensorflow: epoch = 342.0, learning_rate = 0.00049999997, loss = 3.685812e-05, step = 1710 (5.651 sec) INFO:tensorflow:global_step/sec: 1.75864 2021-09-25 23:26:56,488 [INFO] tensorflow: global_step/sec: 1.75864 2021-09-25 23:26:56,489 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 342/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.813879 ETA: 0:07:24.592846 INFO:tensorflow:global_step/sec: 1.83256 2021-09-25 23:26:57,033 [INFO] tensorflow: global_step/sec: 1.83256 INFO:tensorflow:global_step/sec: 1.79391 2021-09-25 23:26:57,591 [INFO] tensorflow: global_step/sec: 1.79391 INFO:tensorflow:global_step/sec: 1.80597 2021-09-25 23:26:58,144 [INFO] tensorflow: global_step/sec: 1.80597 INFO:tensorflow:global_step/sec: 1.78218 2021-09-25 23:26:58,706 [INFO] tensorflow: global_step/sec: 1.78218 INFO:tensorflow:global_step/sec: 1.80012 2021-09-25 23:26:59,261 [INFO] tensorflow: global_step/sec: 1.80012 2021-09-25 23:26:59,263 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 343/500: loss: 0.00003 learning rate: 0.00050 Time taken: 0:00:02.781487 ETA: 0:07:16.693457 INFO:tensorflow:global_step/sec: 1.83574 2021-09-25 23:26:59,806 [INFO] tensorflow: global_step/sec: 1.83574 INFO:tensorflow:global_step/sec: 1.7752 2021-09-25 23:27:00,369 [INFO] tensorflow: global_step/sec: 1.7752 INFO:tensorflow:global_step/sec: 1.6732 2021-09-25 23:27:00,967 [INFO] tensorflow: global_step/sec: 1.6732 INFO:tensorflow:global_step/sec: 1.79016 2021-09-25 23:27:01,525 [INFO] tensorflow: global_step/sec: 1.79016 INFO:tensorflow:epoch = 344.0, learning_rate = 0.00049999997, loss = 3.464575e-05, step = 1720 (5.622 sec) 2021-09-25 23:27:02,108 [INFO] tensorflow: epoch = 344.0, learning_rate = 0.00049999997, loss = 3.464575e-05, step = 1720 (5.622 sec) INFO:tensorflow:global_step/sec: 1.71081 2021-09-25 23:27:02,110 [INFO] tensorflow: global_step/sec: 1.71081 2021-09-25 23:27:02,111 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 344/500: loss: 0.00003 learning rate: 0.00050 Time taken: 0:00:02.822828 ETA: 0:07:20.361214 INFO:tensorflow:global_step/sec: 1.80042 2021-09-25 23:27:02,665 [INFO] tensorflow: global_step/sec: 1.80042 INFO:tensorflow:global_step/sec: 1.75063 2021-09-25 23:27:03,237 [INFO] tensorflow: global_step/sec: 1.75063 INFO:tensorflow:global_step/sec: 1.80368 2021-09-25 23:27:03,791 [INFO] tensorflow: global_step/sec: 1.80368 INFO:tensorflow:global_step/sec: 1.82133 2021-09-25 23:27:04,340 [INFO] tensorflow: global_step/sec: 1.82133 2021-09-25 23:27:04,343 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 113.718 INFO:tensorflow:global_step/sec: 1.78824 2021-09-25 23:27:04,899 [INFO] tensorflow: global_step/sec: 1.78824 2021-09-25 23:27:04,901 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 345/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.809991 ETA: 0:07:15.548587 INFO:tensorflow:global_step/sec: 1.81012 2021-09-25 23:27:05,452 [INFO] tensorflow: global_step/sec: 1.81012 INFO:tensorflow:global_step/sec: 1.8053 2021-09-25 23:27:06,006 [INFO] tensorflow: global_step/sec: 1.8053 INFO:tensorflow:global_step/sec: 1.80293 2021-09-25 23:27:06,560 [INFO] tensorflow: global_step/sec: 1.80293 INFO:tensorflow:global_step/sec: 1.84161 2021-09-25 23:27:07,103 [INFO] tensorflow: global_step/sec: 1.84161 INFO:tensorflow:global_step/sec: 1.73795 2021-09-25 23:27:07,679 [INFO] tensorflow: global_step/sec: 1.73795 2021-09-25 23:27:07,680 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 346/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.762083 ETA: 0:07:05.360717 INFO:tensorflow:epoch = 346.20000000000005, learning_rate = 0.00049999997, loss = 3.2058313e-05, step = 1731 (6.162 sec) 2021-09-25 23:27:08,271 [INFO] tensorflow: epoch = 346.20000000000005, learning_rate = 0.00049999997, loss = 3.2058313e-05, step = 1731 (6.162 sec) INFO:tensorflow:global_step/sec: 1.6847 2021-09-25 23:27:08,272 [INFO] tensorflow: global_step/sec: 1.6847 INFO:tensorflow:global_step/sec: 1.81038 2021-09-25 23:27:08,825 [INFO] tensorflow: global_step/sec: 1.81038 INFO:tensorflow:global_step/sec: 1.7498 2021-09-25 23:27:09,396 [INFO] tensorflow: global_step/sec: 1.7498 INFO:tensorflow:global_step/sec: 1.79051 2021-09-25 23:27:09,955 [INFO] tensorflow: global_step/sec: 1.79051 INFO:tensorflow:global_step/sec: 1.72565 2021-09-25 23:27:10,534 [INFO] tensorflow: global_step/sec: 1.72565 2021-09-25 23:27:10,536 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 347/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.852290 ETA: 0:07:16.400357 INFO:tensorflow:global_step/sec: 1.78365 2021-09-25 23:27:11,095 [INFO] tensorflow: global_step/sec: 1.78365 INFO:tensorflow:global_step/sec: 1.7727 2021-09-25 23:27:11,659 [INFO] tensorflow: global_step/sec: 1.7727 INFO:tensorflow:global_step/sec: 1.77759 2021-09-25 23:27:12,221 [INFO] tensorflow: global_step/sec: 1.77759 INFO:tensorflow:global_step/sec: 1.78544 2021-09-25 23:27:12,782 [INFO] tensorflow: global_step/sec: 1.78544 INFO:tensorflow:global_step/sec: 1.80981 2021-09-25 23:27:13,334 [INFO] tensorflow: global_step/sec: 1.80981 2021-09-25 23:27:13,336 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 348/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.825314 ETA: 0:07:09.447699 INFO:tensorflow:epoch = 348.20000000000005, learning_rate = 0.00049999997, loss = 3.7827114e-05, step = 1741 (5.609 sec) 2021-09-25 23:27:13,880 [INFO] tensorflow: epoch = 348.20000000000005, learning_rate = 0.00049999997, loss = 3.7827114e-05, step = 1741 (5.609 sec) INFO:tensorflow:global_step/sec: 1.82597 2021-09-25 23:27:13,882 [INFO] tensorflow: global_step/sec: 1.82597 INFO:tensorflow:global_step/sec: 1.79424 2021-09-25 23:27:14,439 [INFO] tensorflow: global_step/sec: 1.79424 INFO:tensorflow:global_step/sec: 1.78393 2021-09-25 23:27:15,000 [INFO] tensorflow: global_step/sec: 1.78393 INFO:tensorflow:global_step/sec: 1.78946 2021-09-25 23:27:15,558 [INFO] tensorflow: global_step/sec: 1.78946 INFO:tensorflow:global_step/sec: 1.76091 2021-09-25 23:27:16,126 [INFO] tensorflow: global_step/sec: 1.76091 2021-09-25 23:27:16,128 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 349/500: loss: 0.00003 learning rate: 0.00050 Time taken: 0:00:02.779503 ETA: 0:06:59.704933 INFO:tensorflow:global_step/sec: 1.81095 2021-09-25 23:27:16,679 [INFO] tensorflow: global_step/sec: 1.81095 INFO:tensorflow:global_step/sec: 1.85581 2021-09-25 23:27:17,217 [INFO] tensorflow: global_step/sec: 1.85581 INFO:tensorflow:global_step/sec: 1.80976 2021-09-25 23:27:17,770 [INFO] tensorflow: global_step/sec: 1.80976 INFO:tensorflow:global_step/sec: 1.70473 2021-09-25 23:27:18,357 [INFO] tensorflow: global_step/sec: 1.70473 2021-09-25 23:27:18,364 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 114.127 INFO:tensorflow:global_step/sec: 1.80772 2021-09-25 23:27:18,910 [INFO] tensorflow: global_step/sec: 1.80772 2021-09-25 23:27:18,911 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 350/500: loss: 0.00004 learning rate: 0.00050 Time taken: 0:00:02.797456 ETA: 0:06:59.618368 INFO:tensorflow:epoch = 350.20000000000005, learning_rate = 0.00049693923, loss = 3.4922738e-05, step = 1751 (5.593 sec) 2021-09-25 23:27:19,473 [INFO] tensorflow: epoch = 350.20000000000005, learning_rate = 0.00049693923, loss = 3.4922738e-05, step = 1751 (5.593 sec) INFO:tensorflow:global_step/sec: 1.77146 2021-09-25 23:27:19,474 [INFO] tensorflow: global_step/sec: 1.77146 INFO:tensorflow:global_step/sec: 1.70559 2021-09-25 23:27:20,061 [INFO] tensorflow: global_step/sec: 1.70559 INFO:tensorflow:global_step/sec: 1.82318 2021-09-25 23:27:20,609 [INFO] tensorflow: global_step/sec: 1.82318 INFO:tensorflow:global_step/sec: 1.85743 2021-09-25 23:27:21,147 [INFO] tensorflow: global_step/sec: 1.85743 INFO:tensorflow:global_step/sec: 1.8227 2021-09-25 23:27:21,696 [INFO] tensorflow: global_step/sec: 1.8227 2021-09-25 23:27:21,698 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 351/500: loss: 0.00004 learning rate: 0.00048 Time taken: 0:00:02.786689 ETA: 0:06:55.216596 INFO:tensorflow:global_step/sec: 1.72429 2021-09-25 23:27:22,276 [INFO] tensorflow: global_step/sec: 1.72429 INFO:tensorflow:global_step/sec: 1.8143 2021-09-25 23:27:22,827 [INFO] tensorflow: global_step/sec: 1.8143 INFO:tensorflow:global_step/sec: 1.72015 2021-09-25 23:27:23,409 [INFO] tensorflow: global_step/sec: 1.72015 INFO:tensorflow:global_step/sec: 1.82471 2021-09-25 23:27:23,957 [INFO] tensorflow: global_step/sec: 1.82471 INFO:tensorflow:global_step/sec: 1.78661 2021-09-25 23:27:24,516 [INFO] tensorflow: global_step/sec: 1.78661 2021-09-25 23:27:24,518 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 352/500: loss: 0.00003 learning rate: 0.00047 Time taken: 0:00:02.809839 ETA: 0:06:55.856138 INFO:tensorflow:epoch = 352.20000000000005, learning_rate = 0.00046734393, loss = 3.4515306e-05, step = 1761 (5.600 sec) 2021-09-25 23:27:25,073 [INFO] tensorflow: epoch = 352.20000000000005, learning_rate = 0.00046734393, loss = 3.4515306e-05, step = 1761 (5.600 sec) INFO:tensorflow:global_step/sec: 1.79138 2021-09-25 23:27:25,074 [INFO] tensorflow: global_step/sec: 1.79138 INFO:tensorflow:global_step/sec: 1.70167 2021-09-25 23:27:25,662 [INFO] tensorflow: global_step/sec: 1.70167 INFO:tensorflow:global_step/sec: 1.69831 2021-09-25 23:27:26,251 [INFO] tensorflow: global_step/sec: 1.69831 INFO:tensorflow:global_step/sec: 1.71528 2021-09-25 23:27:26,834 [INFO] tensorflow: global_step/sec: 1.71528 INFO:tensorflow:global_step/sec: 1.7827 2021-09-25 23:27:27,395 [INFO] tensorflow: global_step/sec: 1.7827 2021-09-25 23:27:27,396 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 353/500: loss: 0.00004 learning rate: 0.00046 Time taken: 0:00:02.877571 ETA: 0:07:03.002882 INFO:tensorflow:global_step/sec: 1.7519 2021-09-25 23:27:27,966 [INFO] tensorflow: global_step/sec: 1.7519 INFO:tensorflow:global_step/sec: 1.85043 2021-09-25 23:27:28,506 [INFO] tensorflow: global_step/sec: 1.85043 INFO:tensorflow:global_step/sec: 1.79129 2021-09-25 23:27:29,064 [INFO] tensorflow: global_step/sec: 1.79129 INFO:tensorflow:global_step/sec: 1.73672 2021-09-25 23:27:29,640 [INFO] tensorflow: global_step/sec: 1.73672 INFO:tensorflow:global_step/sec: 1.66824 2021-09-25 23:27:30,240 [INFO] tensorflow: global_step/sec: 1.66824 2021-09-25 23:27:30,241 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 354/500: loss: 0.00003 learning rate: 0.00044 Time taken: 0:00:02.806443 ETA: 0:06:49.740709 INFO:tensorflow:epoch = 354.20000000000005, learning_rate = 0.00043951118, loss = 3.4810157e-05, step = 1771 (5.732 sec) 2021-09-25 23:27:30,805 [INFO] tensorflow: epoch = 354.20000000000005, learning_rate = 0.00043951118, loss = 3.4810157e-05, step = 1771 (5.732 sec) INFO:tensorflow:global_step/sec: 1.76324 2021-09-25 23:27:30,807 [INFO] tensorflow: global_step/sec: 1.76324 INFO:tensorflow:global_step/sec: 1.81381 2021-09-25 23:27:31,358 [INFO] tensorflow: global_step/sec: 1.81381 INFO:tensorflow:global_step/sec: 1.8478 2021-09-25 23:27:31,899 [INFO] tensorflow: global_step/sec: 1.8478 INFO:tensorflow:global_step/sec: 1.73457 2021-09-25 23:27:32,476 [INFO] tensorflow: global_step/sec: 1.73457 2021-09-25 23:27:32,480 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 113.350 INFO:tensorflow:global_step/sec: 1.81113 2021-09-25 23:27:33,028 [INFO] tensorflow: global_step/sec: 1.81113 2021-09-25 23:27:33,029 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 355/500: loss: 0.00003 learning rate: 0.00043 Time taken: 0:00:02.832058 ETA: 0:06:50.648403 INFO:tensorflow:global_step/sec: 1.78979 2021-09-25 23:27:33,587 [INFO] tensorflow: global_step/sec: 1.78979 INFO:tensorflow:global_step/sec: 1.83869 2021-09-25 23:27:34,131 [INFO] tensorflow: global_step/sec: 1.83869 INFO:tensorflow:global_step/sec: 1.72158 2021-09-25 23:27:34,711 [INFO] tensorflow: global_step/sec: 1.72158 INFO:tensorflow:global_step/sec: 1.81088 2021-09-25 23:27:35,264 [INFO] tensorflow: global_step/sec: 1.81088 INFO:tensorflow:global_step/sec: 1.76727 2021-09-25 23:27:35,829 [INFO] tensorflow: global_step/sec: 1.76727 2021-09-25 23:27:35,831 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 356/500: loss: 0.00003 learning rate: 0.00042 Time taken: 0:00:02.785885 ETA: 0:06:41.167488 INFO:tensorflow:epoch = 356.20000000000005, learning_rate = 0.00041333638, loss = 3.5417957e-05, step = 1781 (5.579 sec) 2021-09-25 23:27:36,384 [INFO] tensorflow: epoch = 356.20000000000005, learning_rate = 0.00041333638, loss = 3.5417957e-05, step = 1781 (5.579 sec) INFO:tensorflow:global_step/sec: 1.79729 2021-09-25 23:27:36,386 [INFO] tensorflow: global_step/sec: 1.79729 INFO:tensorflow:global_step/sec: 1.80783 2021-09-25 23:27:36,939 [INFO] tensorflow: global_step/sec: 1.80783 INFO:tensorflow:global_step/sec: 1.78377 2021-09-25 23:27:37,500 [INFO] tensorflow: global_step/sec: 1.78377 INFO:tensorflow:global_step/sec: 1.79182 2021-09-25 23:27:38,058 [INFO] tensorflow: global_step/sec: 1.79182 INFO:tensorflow:global_step/sec: 1.79695 2021-09-25 23:27:38,614 [INFO] tensorflow: global_step/sec: 1.79695 2021-09-25 23:27:38,616 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 357/500: loss: 0.00003 learning rate: 0.00040 Time taken: 0:00:02.796924 ETA: 0:06:39.960182 INFO:tensorflow:global_step/sec: 1.72041 2021-09-25 23:27:39,195 [INFO] tensorflow: global_step/sec: 1.72041 INFO:tensorflow:global_step/sec: 1.80188 2021-09-25 23:27:39,750 [INFO] tensorflow: global_step/sec: 1.80188 INFO:tensorflow:global_step/sec: 1.77356 2021-09-25 23:27:40,314 [INFO] tensorflow: global_step/sec: 1.77356 INFO:tensorflow:global_step/sec: 1.82932 2021-09-25 23:27:40,861 [INFO] tensorflow: global_step/sec: 1.82932 INFO:tensorflow:global_step/sec: 1.71971 2021-09-25 23:27:41,442 [INFO] tensorflow: global_step/sec: 1.71971 2021-09-25 23:27:41,444 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 358/500: loss: 0.00004 learning rate: 0.00039 Time taken: 0:00:02.799512 ETA: 0:06:37.530691 INFO:tensorflow:epoch = 358.20000000000005, learning_rate = 0.00038871987, loss = 3.629031e-05, step = 1791 (5.606 sec) 2021-09-25 23:27:41,991 [INFO] tensorflow: epoch = 358.20000000000005, learning_rate = 0.00038871987, loss = 3.629031e-05, step = 1791 (5.606 sec) INFO:tensorflow:global_step/sec: 1.81894 2021-09-25 23:27:41,992 [INFO] tensorflow: global_step/sec: 1.81894 INFO:tensorflow:global_step/sec: 1.7967 2021-09-25 23:27:42,549 [INFO] tensorflow: global_step/sec: 1.7967 INFO:tensorflow:global_step/sec: 1.78713 2021-09-25 23:27:43,108 [INFO] tensorflow: global_step/sec: 1.78713 INFO:tensorflow:global_step/sec: 1.82583 2021-09-25 23:27:43,656 [INFO] tensorflow: global_step/sec: 1.82583 INFO:tensorflow:global_step/sec: 1.82468 2021-09-25 23:27:44,204 [INFO] tensorflow: global_step/sec: 1.82468 2021-09-25 23:27:44,205 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 359/500: loss: 0.00004 learning rate: 0.00038 Time taken: 0:00:02.794752 ETA: 0:06:34.060049 INFO:tensorflow:global_step/sec: 1.85142 2021-09-25 23:27:44,744 [INFO] tensorflow: global_step/sec: 1.85142 INFO:tensorflow:global_step/sec: 1.76024 2021-09-25 23:27:45,312 [INFO] tensorflow: global_step/sec: 1.76024 INFO:tensorflow:global_step/sec: 1.82929 2021-09-25 23:27:45,859 [INFO] tensorflow: global_step/sec: 1.82929 INFO:tensorflow:global_step/sec: 1.78702 2021-09-25 23:27:46,419 [INFO] tensorflow: global_step/sec: 1.78702 INFO:tensorflow:Saving checkpoints for step-1800. 2021-09-25 23:27:46,420 [INFO] tensorflow: Saving checkpoints for step-1800. WARNING:tensorflow:Ignoring: /tmp/tmpngw_6vgs; No such file or directory 2021-09-25 23:27:46,640 [WARNING] tensorflow: Ignoring: /tmp/tmpngw_6vgs; No such file or directory 2021-09-25 23:27:50,510 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 88.746 INFO:tensorflow:epoch = 360.0, learning_rate = 0.0003678212, loss = 3.868658e-05, step = 1800 (9.081 sec) 2021-09-25 23:27:51,072 [INFO] tensorflow: epoch = 360.0, learning_rate = 0.0003678212, loss = 3.868658e-05, step = 1800 (9.081 sec) INFO:tensorflow:global_step/sec: 0.214824 2021-09-25 23:27:51,073 [INFO] tensorflow: global_step/sec: 0.214824 2021-09-25 23:27:51,075 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 360/500: loss: 0.00004 learning rate: 0.00037 Time taken: 0:00:06.848904 ETA: 0:15:58.846512 INFO:tensorflow:global_step/sec: 1.79811 2021-09-25 23:27:51,630 [INFO] tensorflow: global_step/sec: 1.79811 INFO:tensorflow:global_step/sec: 1.78225 2021-09-25 23:27:52,191 [INFO] tensorflow: global_step/sec: 1.78225 INFO:tensorflow:global_step/sec: 1.78547 2021-09-25 23:27:52,751 [INFO] tensorflow: global_step/sec: 1.78547 INFO:tensorflow:global_step/sec: 1.82507 2021-09-25 23:27:53,299 [INFO] tensorflow: global_step/sec: 1.82507 Epoch 361/500 ========================= Validation cost: 0.000000 Mean average_precision (in %): 0.0000 class name average precision (in %) ------------ -------------------------- lpd 0 Median Inference Time: nan INFO:tensorflow:global_step/sec: 1.8035 2021-09-25 23:27:53,853 [INFO] tensorflow: global_step/sec: 1.8035 2021-09-25 23:27:53,855 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 361/500: loss: 0.00004 learning rate: 0.00036 Time taken: 0:00:02.792535 ETA: 0:06:28.162374 INFO:tensorflow:global_step/sec: 1.69232 2021-09-25 23:27:54,444 [INFO] tensorflow: global_step/sec: 1.69232 INFO:tensorflow:global_step/sec: 1.80051 2021-09-25 23:27:55,000 [INFO] tensorflow: global_step/sec: 1.80051 INFO:tensorflow:global_step/sec: 1.79337 2021-09-25 23:27:55,557 [INFO] tensorflow: global_step/sec: 1.79337 INFO:tensorflow:global_step/sec: 1.70263 2021-09-25 23:27:56,144 [INFO] tensorflow: global_step/sec: 1.70263 INFO:tensorflow:epoch = 362.0, learning_rate = 0.0003459155, loss = 3.368066e-05, step = 1810 (5.639 sec) 2021-09-25 23:27:56,711 [INFO] tensorflow: epoch = 362.0, learning_rate = 0.0003459155, loss = 3.368066e-05, step = 1810 (5.639 sec) INFO:tensorflow:global_step/sec: 1.76003 2021-09-25 23:27:56,713 [INFO] tensorflow: global_step/sec: 1.76003 2021-09-25 23:27:56,714 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 362/500: loss: 0.00003 learning rate: 0.00035 Time taken: 0:00:02.845120 ETA: 0:06:32.626554 INFO:tensorflow:global_step/sec: 1.76157 2021-09-25 23:27:57,280 [INFO] tensorflow: global_step/sec: 1.76157 INFO:tensorflow:global_step/sec: 1.81234 2021-09-25 23:27:57,832 [INFO] tensorflow: global_step/sec: 1.81234 INFO:tensorflow:global_step/sec: 1.78699 2021-09-25 23:27:58,392 [INFO] tensorflow: global_step/sec: 1.78699 INFO:tensorflow:global_step/sec: 1.75518 2021-09-25 23:27:58,961 [INFO] tensorflow: global_step/sec: 1.75518 INFO:tensorflow:global_step/sec: 1.68092 2021-09-25 23:27:59,556 [INFO] tensorflow: global_step/sec: 1.68092 2021-09-25 23:27:59,558 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 363/500: loss: 0.00003 learning rate: 0.00034 Time taken: 0:00:02.814030 ETA: 0:06:25.522101 INFO:tensorflow:global_step/sec: 1.71939 2021-09-25 23:28:00,138 [INFO] tensorflow: global_step/sec: 1.71939 INFO:tensorflow:global_step/sec: 1.73085 2021-09-25 23:28:00,716 [INFO] tensorflow: global_step/sec: 1.73085 INFO:tensorflow:global_step/sec: 1.78936 2021-09-25 23:28:01,275 [INFO] tensorflow: global_step/sec: 1.78936 INFO:tensorflow:global_step/sec: 1.83788 2021-09-25 23:28:01,819 [INFO] tensorflow: global_step/sec: 1.83788 INFO:tensorflow:epoch = 364.0, learning_rate = 0.0003253146, loss = 3.3167442e-05, step = 1820 (5.656 sec) 2021-09-25 23:28:02,367 [INFO] tensorflow: epoch = 364.0, learning_rate = 0.0003253146, loss = 3.3167442e-05, step = 1820 (5.656 sec) INFO:tensorflow:global_step/sec: 1.81797 2021-09-25 23:28:02,369 [INFO] tensorflow: global_step/sec: 1.81797 2021-09-25 23:28:02,370 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 364/500: loss: 0.00003 learning rate: 0.00033 Time taken: 0:00:02.856663 ETA: 0:06:28.506231 INFO:tensorflow:global_step/sec: 1.83317 2021-09-25 23:28:02,914 [INFO] tensorflow: global_step/sec: 1.83317 INFO:tensorflow:global_step/sec: 1.77368 2021-09-25 23:28:03,478 [INFO] tensorflow: global_step/sec: 1.77368 INFO:tensorflow:global_step/sec: 1.70305 2021-09-25 23:28:04,065 [INFO] tensorflow: global_step/sec: 1.70305 INFO:tensorflow:global_step/sec: 1.85814 2021-09-25 23:28:04,603 [INFO] tensorflow: global_step/sec: 1.85814 2021-09-25 23:28:04,610 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 113.485 INFO:tensorflow:global_step/sec: 1.82289 2021-09-25 23:28:05,152 [INFO] tensorflow: global_step/sec: 1.82289 2021-09-25 23:28:05,153 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 365/500: loss: 0.00003 learning rate: 0.00032 Time taken: 0:00:02.787623 ETA: 0:06:16.329160 INFO:tensorflow:global_step/sec: 1.8028 2021-09-25 23:28:05,707 [INFO] tensorflow: global_step/sec: 1.8028 INFO:tensorflow:global_step/sec: 1.79132 2021-09-25 23:28:06,265 [INFO] tensorflow: global_step/sec: 1.79132 INFO:tensorflow:global_step/sec: 1.76433 2021-09-25 23:28:06,832 [INFO] tensorflow: global_step/sec: 1.76433 INFO:tensorflow:global_step/sec: 1.7589 2021-09-25 23:28:07,400 [INFO] tensorflow: global_step/sec: 1.7589 INFO:tensorflow:epoch = 366.0, learning_rate = 0.00030594014, loss = 3.1299987e-05, step = 1830 (5.594 sec) 2021-09-25 23:28:07,961 [INFO] tensorflow: epoch = 366.0, learning_rate = 0.00030594014, loss = 3.1299987e-05, step = 1830 (5.594 sec) INFO:tensorflow:global_step/sec: 1.77812 2021-09-25 23:28:07,963 [INFO] tensorflow: global_step/sec: 1.77812 2021-09-25 23:28:07,964 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 366/500: loss: 0.00003 learning rate: 0.00031 Time taken: 0:00:02.794618 ETA: 0:06:14.478797 INFO:tensorflow:global_step/sec: 1.84032 2021-09-25 23:28:08,506 [INFO] tensorflow: global_step/sec: 1.84032 INFO:tensorflow:global_step/sec: 1.7816 2021-09-25 23:28:09,067 [INFO] tensorflow: global_step/sec: 1.7816 INFO:tensorflow:global_step/sec: 1.82045 2021-09-25 23:28:09,617 [INFO] tensorflow: global_step/sec: 1.82045 INFO:tensorflow:global_step/sec: 1.85067 2021-09-25 23:28:10,157 [INFO] tensorflow: global_step/sec: 1.85067 INFO:tensorflow:global_step/sec: 1.77535 2021-09-25 23:28:10,720 [INFO] tensorflow: global_step/sec: 1.77535 2021-09-25 23:28:10,722 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 367/500: loss: 0.00004 learning rate: 0.00030 Time taken: 0:00:02.756087 ETA: 0:06:06.559611 INFO:tensorflow:global_step/sec: 1.69815 2021-09-25 23:28:11,309 [INFO] tensorflow: global_step/sec: 1.69815 INFO:tensorflow:global_step/sec: 1.78446 2021-09-25 23:28:11,869 [INFO] tensorflow: global_step/sec: 1.78446 INFO:tensorflow:global_step/sec: 1.76981 2021-09-25 23:28:12,434 [INFO] tensorflow: global_step/sec: 1.76981 INFO:tensorflow:global_step/sec: 1.79778 2021-09-25 23:28:12,991 [INFO] tensorflow: global_step/sec: 1.79778 INFO:tensorflow:epoch = 368.0, learning_rate = 0.0002877201, loss = 2.972272e-05, step = 1840 (5.583 sec) 2021-09-25 23:28:13,544 [INFO] tensorflow: epoch = 368.0, learning_rate = 0.0002877201, loss = 2.972272e-05, step = 1840 (5.583 sec) INFO:tensorflow:global_step/sec: 1.80265 2021-09-25 23:28:13,545 [INFO] tensorflow: global_step/sec: 1.80265 2021-09-25 23:28:13,547 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 368/500: loss: 0.00003 learning rate: 0.00029 Time taken: 0:00:02.830862 ETA: 0:06:13.673727 INFO:tensorflow:global_step/sec: 1.83002 2021-09-25 23:28:14,092 [INFO] tensorflow: global_step/sec: 1.83002 INFO:tensorflow:global_step/sec: 1.71378 2021-09-25 23:28:14,675 [INFO] tensorflow: global_step/sec: 1.71378 INFO:tensorflow:global_step/sec: 1.71364 2021-09-25 23:28:15,259 [INFO] tensorflow: global_step/sec: 1.71364 INFO:tensorflow:global_step/sec: 1.81346 2021-09-25 23:28:15,810 [INFO] tensorflow: global_step/sec: 1.81346 INFO:tensorflow:global_step/sec: 1.77105 2021-09-25 23:28:16,375 [INFO] tensorflow: global_step/sec: 1.77105 2021-09-25 23:28:16,376 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 369/500: loss: 0.00003 learning rate: 0.00028 Time taken: 0:00:02.821145 ETA: 0:06:09.570003 INFO:tensorflow:global_step/sec: 1.78287 2021-09-25 23:28:16,936 [INFO] tensorflow: global_step/sec: 1.78287 INFO:tensorflow:global_step/sec: 1.78703 2021-09-25 23:28:17,496 [INFO] tensorflow: global_step/sec: 1.78703 INFO:tensorflow:global_step/sec: 1.71002 2021-09-25 23:28:18,080 [INFO] tensorflow: global_step/sec: 1.71002 INFO:tensorflow:global_step/sec: 1.70759 2021-09-25 23:28:18,666 [INFO] tensorflow: global_step/sec: 1.70759 2021-09-25 23:28:18,670 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 113.800 INFO:tensorflow:epoch = 370.0, learning_rate = 0.0002705846, loss = 3.5297424e-05, step = 1850 (5.702 sec) 2021-09-25 23:28:19,246 [INFO] tensorflow: epoch = 370.0, learning_rate = 0.0002705846, loss = 3.5297424e-05, step = 1850 (5.702 sec) INFO:tensorflow:global_step/sec: 1.72041 2021-09-25 23:28:19,247 [INFO] tensorflow: global_step/sec: 1.72041 2021-09-25 23:28:19,248 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 370/500: loss: 0.00004 learning rate: 0.00027 Time taken: 0:00:02.854024 ETA: 0:06:11.023111 INFO:tensorflow:global_step/sec: 1.78058 2021-09-25 23:28:19,809 [INFO] tensorflow: global_step/sec: 1.78058 INFO:tensorflow:global_step/sec: 1.75549 2021-09-25 23:28:20,378 [INFO] tensorflow: global_step/sec: 1.75549 INFO:tensorflow:global_step/sec: 1.8235 2021-09-25 23:28:20,927 [INFO] tensorflow: global_step/sec: 1.8235 INFO:tensorflow:global_step/sec: 1.7858 2021-09-25 23:28:21,487 [INFO] tensorflow: global_step/sec: 1.7858 INFO:tensorflow:global_step/sec: 1.75412 2021-09-25 23:28:22,057 [INFO] tensorflow: global_step/sec: 1.75412 2021-09-25 23:28:22,058 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 371/500: loss: 0.00003 learning rate: 0.00026 Time taken: 0:00:02.817993 ETA: 0:06:03.521057 INFO:tensorflow:global_step/sec: 1.80075 2021-09-25 23:28:22,612 [INFO] tensorflow: global_step/sec: 1.80075 INFO:tensorflow:global_step/sec: 1.69421 2021-09-25 23:28:23,202 [INFO] tensorflow: global_step/sec: 1.69421 INFO:tensorflow:global_step/sec: 1.82177 2021-09-25 23:28:23,751 [INFO] tensorflow: global_step/sec: 1.82177 INFO:tensorflow:global_step/sec: 1.70433 2021-09-25 23:28:24,338 [INFO] tensorflow: global_step/sec: 1.70433 INFO:tensorflow:epoch = 372.0, learning_rate = 0.00025447013, loss = 3.365844e-05, step = 1860 (5.656 sec) 2021-09-25 23:28:24,902 [INFO] tensorflow: epoch = 372.0, learning_rate = 0.00025447013, loss = 3.365844e-05, step = 1860 (5.656 sec) INFO:tensorflow:global_step/sec: 1.76928 2021-09-25 23:28:24,903 [INFO] tensorflow: global_step/sec: 1.76928 2021-09-25 23:28:24,905 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 372/500: loss: 0.00003 learning rate: 0.00025 Time taken: 0:00:02.855245 ETA: 0:06:05.471313 INFO:tensorflow:global_step/sec: 1.74692 2021-09-25 23:28:25,476 [INFO] tensorflow: global_step/sec: 1.74692 INFO:tensorflow:global_step/sec: 1.82936 2021-09-25 23:28:26,022 [INFO] tensorflow: global_step/sec: 1.82936 INFO:tensorflow:global_step/sec: 1.80536 2021-09-25 23:28:26,576 [INFO] tensorflow: global_step/sec: 1.80536 INFO:tensorflow:global_step/sec: 1.75112 2021-09-25 23:28:27,147 [INFO] tensorflow: global_step/sec: 1.75112 INFO:tensorflow:global_step/sec: 1.74792 2021-09-25 23:28:27,719 [INFO] tensorflow: global_step/sec: 1.74792 2021-09-25 23:28:27,721 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 373/500: loss: 0.00003 learning rate: 0.00025 Time taken: 0:00:02.806190 ETA: 0:05:56.386101 INFO:tensorflow:global_step/sec: 1.7024 2021-09-25 23:28:28,307 [INFO] tensorflow: global_step/sec: 1.7024 INFO:tensorflow:global_step/sec: 1.79684 2021-09-25 23:28:28,863 [INFO] tensorflow: global_step/sec: 1.79684 INFO:tensorflow:global_step/sec: 1.79253 2021-09-25 23:28:29,421 [INFO] tensorflow: global_step/sec: 1.79253 INFO:tensorflow:global_step/sec: 1.77858 2021-09-25 23:28:29,984 [INFO] tensorflow: global_step/sec: 1.77858 INFO:tensorflow:epoch = 374.0, learning_rate = 0.00023931511, loss = 3.2994234e-05, step = 1870 (5.640 sec) 2021-09-25 23:28:30,542 [INFO] tensorflow: epoch = 374.0, learning_rate = 0.00023931511, loss = 3.2994234e-05, step = 1870 (5.640 sec) INFO:tensorflow:global_step/sec: 1.78587 2021-09-25 23:28:30,543 [INFO] tensorflow: global_step/sec: 1.78587 2021-09-25 23:28:30,545 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 374/500: loss: 0.00003 learning rate: 0.00024 Time taken: 0:00:02.835374 ETA: 0:05:57.257169 INFO:tensorflow:global_step/sec: 1.81535 2021-09-25 23:28:31,094 [INFO] tensorflow: global_step/sec: 1.81535 INFO:tensorflow:global_step/sec: 1.80953 2021-09-25 23:28:31,647 [INFO] tensorflow: global_step/sec: 1.80953 INFO:tensorflow:global_step/sec: 1.79027 2021-09-25 23:28:32,206 [INFO] tensorflow: global_step/sec: 1.79027 INFO:tensorflow:global_step/sec: 1.81135 2021-09-25 23:28:32,758 [INFO] tensorflow: global_step/sec: 1.81135 2021-09-25 23:28:32,763 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 113.543 INFO:tensorflow:global_step/sec: 1.6968 2021-09-25 23:28:33,347 [INFO] tensorflow: global_step/sec: 1.6968 2021-09-25 23:28:33,348 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 375/500: loss: 0.00003 learning rate: 0.00023 Time taken: 0:00:02.773723 ETA: 0:05:46.715391 INFO:tensorflow:global_step/sec: 1.83517 2021-09-25 23:28:33,892 [INFO] tensorflow: global_step/sec: 1.83517 INFO:tensorflow:global_step/sec: 1.81511 2021-09-25 23:28:34,443 [INFO] tensorflow: global_step/sec: 1.81511 INFO:tensorflow:global_step/sec: 1.77863 2021-09-25 23:28:35,005 [INFO] tensorflow: global_step/sec: 1.77863 INFO:tensorflow:global_step/sec: 1.828 2021-09-25 23:28:35,552 [INFO] tensorflow: global_step/sec: 1.828 INFO:tensorflow:epoch = 376.0, learning_rate = 0.00022506263, loss = 3.3376222e-05, step = 1880 (5.582 sec) 2021-09-25 23:28:36,124 [INFO] tensorflow: epoch = 376.0, learning_rate = 0.00022506263, loss = 3.3376222e-05, step = 1880 (5.582 sec) INFO:tensorflow:global_step/sec: 1.74464 2021-09-25 23:28:36,125 [INFO] tensorflow: global_step/sec: 1.74464 2021-09-25 23:28:36,127 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 376/500: loss: 0.00003 learning rate: 0.00023 Time taken: 0:00:02.793921 ETA: 0:05:46.446144 INFO:tensorflow:global_step/sec: 1.80048 2021-09-25 23:28:36,681 [INFO] tensorflow: global_step/sec: 1.80048 INFO:tensorflow:global_step/sec: 1.77016 2021-09-25 23:28:37,246 [INFO] tensorflow: global_step/sec: 1.77016 INFO:tensorflow:global_step/sec: 1.76428 2021-09-25 23:28:37,812 [INFO] tensorflow: global_step/sec: 1.76428 INFO:tensorflow:global_step/sec: 1.79296 2021-09-25 23:28:38,370 [INFO] tensorflow: global_step/sec: 1.79296 INFO:tensorflow:global_step/sec: 1.78171 2021-09-25 23:28:38,931 [INFO] tensorflow: global_step/sec: 1.78171 2021-09-25 23:28:38,933 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 377/500: loss: 0.00003 learning rate: 0.00022 Time taken: 0:00:02.815260 ETA: 0:05:46.276972 INFO:tensorflow:global_step/sec: 1.74923 2021-09-25 23:28:39,503 [INFO] tensorflow: global_step/sec: 1.74923 INFO:tensorflow:global_step/sec: 1.71463 2021-09-25 23:28:40,086 [INFO] tensorflow: global_step/sec: 1.71463 INFO:tensorflow:global_step/sec: 1.70665 2021-09-25 23:28:40,672 [INFO] tensorflow: global_step/sec: 1.70665 INFO:tensorflow:global_step/sec: 1.80581 2021-09-25 23:28:41,226 [INFO] tensorflow: global_step/sec: 1.80581 INFO:tensorflow:epoch = 378.0, learning_rate = 0.00021165899, loss = 3.6368358e-05, step = 1890 (5.659 sec) 2021-09-25 23:28:41,783 [INFO] tensorflow: epoch = 378.0, learning_rate = 0.00021165899, loss = 3.6368358e-05, step = 1890 (5.659 sec) INFO:tensorflow:global_step/sec: 1.79266 2021-09-25 23:28:41,784 [INFO] tensorflow: global_step/sec: 1.79266 2021-09-25 23:28:41,785 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 378/500: loss: 0.00004 learning rate: 0.00021 Time taken: 0:00:02.854974 ETA: 0:05:48.306774 INFO:tensorflow:global_step/sec: 1.74068 2021-09-25 23:28:42,358 [INFO] tensorflow: global_step/sec: 1.74068 INFO:tensorflow:global_step/sec: 1.78706 2021-09-25 23:28:42,918 [INFO] tensorflow: global_step/sec: 1.78706 INFO:tensorflow:global_step/sec: 1.82965 2021-09-25 23:28:43,464 [INFO] tensorflow: global_step/sec: 1.82965 INFO:tensorflow:global_step/sec: 1.76085 2021-09-25 23:28:44,032 [INFO] tensorflow: global_step/sec: 1.76085 INFO:tensorflow:global_step/sec: 1.79379 2021-09-25 23:28:44,590 [INFO] tensorflow: global_step/sec: 1.79379 2021-09-25 23:28:44,591 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 379/500: loss: 0.00004 learning rate: 0.00021 Time taken: 0:00:02.809160 ETA: 0:05:39.908359 INFO:tensorflow:global_step/sec: 1.8041 2021-09-25 23:28:45,144 [INFO] tensorflow: global_step/sec: 1.8041 INFO:tensorflow:global_step/sec: 1.79349 2021-09-25 23:28:45,702 [INFO] tensorflow: global_step/sec: 1.79349 INFO:tensorflow:global_step/sec: 1.77189 2021-09-25 23:28:46,266 [INFO] tensorflow: global_step/sec: 1.77189 INFO:tensorflow:global_step/sec: 1.77953 2021-09-25 23:28:46,828 [INFO] tensorflow: global_step/sec: 1.77953 INFO:tensorflow:Saving checkpoints for step-1900. 2021-09-25 23:28:46,829 [INFO] tensorflow: Saving checkpoints for step-1900. WARNING:tensorflow:Ignoring: /tmp/tmpepkd1i5_; No such file or directory 2021-09-25 23:28:47,050 [WARNING] tensorflow: Ignoring: /tmp/tmpepkd1i5_; No such file or directory 2021-09-25 23:28:50,867 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 88.381 INFO:tensorflow:epoch = 380.0, learning_rate = 0.0001990536, loss = 2.870571e-05, step = 1900 (9.672 sec) 2021-09-25 23:28:51,454 [INFO] tensorflow: epoch = 380.0, learning_rate = 0.0001990536, loss = 2.870571e-05, step = 1900 (9.672 sec) INFO:tensorflow:global_step/sec: 0.2161 2021-09-25 23:28:51,455 [INFO] tensorflow: global_step/sec: 0.2161 2021-09-25 23:28:51,457 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 380/500: loss: 0.00003 learning rate: 0.00020 Time taken: 0:00:06.826987 ETA: 0:13:39.238472 INFO:tensorflow:global_step/sec: 1.75694 2021-09-25 23:28:52,025 [INFO] tensorflow: global_step/sec: 1.75694 INFO:tensorflow:global_step/sec: 1.72687 2021-09-25 23:28:52,604 [INFO] tensorflow: global_step/sec: 1.72687 INFO:tensorflow:global_step/sec: 1.84538 2021-09-25 23:28:53,146 [INFO] tensorflow: global_step/sec: 1.84538 INFO:tensorflow:global_step/sec: 1.80653 2021-09-25 23:28:53,699 [INFO] tensorflow: global_step/sec: 1.80653 Epoch 381/500 ========================= Validation cost: 0.000000 Mean average_precision (in %): 0.0000 class name average precision (in %) ------------ -------------------------- lpd 0 Median Inference Time: nan INFO:tensorflow:global_step/sec: 1.7523 2021-09-25 23:28:54,270 [INFO] tensorflow: global_step/sec: 1.7523 2021-09-25 23:28:54,271 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 381/500: loss: 0.00003 learning rate: 0.00019 Time taken: 0:00:02.837285 ETA: 0:05:37.636863 INFO:tensorflow:global_step/sec: 1.8279 2021-09-25 23:28:54,817 [INFO] tensorflow: global_step/sec: 1.8279 INFO:tensorflow:global_step/sec: 1.78615 2021-09-25 23:28:55,377 [INFO] tensorflow: global_step/sec: 1.78615 INFO:tensorflow:global_step/sec: 1.82396 2021-09-25 23:28:55,925 [INFO] tensorflow: global_step/sec: 1.82396 INFO:tensorflow:global_step/sec: 1.8388 2021-09-25 23:28:56,469 [INFO] tensorflow: global_step/sec: 1.8388 INFO:tensorflow:epoch = 382.0, learning_rate = 0.00018719892, loss = 3.4193854e-05, step = 1910 (5.583 sec) 2021-09-25 23:28:57,037 [INFO] tensorflow: epoch = 382.0, learning_rate = 0.00018719892, loss = 3.4193854e-05, step = 1910 (5.583 sec) INFO:tensorflow:global_step/sec: 1.75529 2021-09-25 23:28:57,039 [INFO] tensorflow: global_step/sec: 1.75529 2021-09-25 23:28:57,040 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 382/500: loss: 0.00003 learning rate: 0.00019 Time taken: 0:00:02.766044 ETA: 0:05:26.393180 INFO:tensorflow:global_step/sec: 1.78531 2021-09-25 23:28:57,599 [INFO] tensorflow: global_step/sec: 1.78531 INFO:tensorflow:global_step/sec: 1.76555 2021-09-25 23:28:58,165 [INFO] tensorflow: global_step/sec: 1.76555 INFO:tensorflow:global_step/sec: 1.7625 2021-09-25 23:28:58,733 [INFO] tensorflow: global_step/sec: 1.7625 INFO:tensorflow:global_step/sec: 1.76105 2021-09-25 23:28:59,300 [INFO] tensorflow: global_step/sec: 1.76105 INFO:tensorflow:global_step/sec: 1.81345 2021-09-25 23:28:59,852 [INFO] tensorflow: global_step/sec: 1.81345 2021-09-25 23:28:59,853 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 383/500: loss: 0.00003 learning rate: 0.00018 Time taken: 0:00:02.834675 ETA: 0:05:31.656984 INFO:tensorflow:global_step/sec: 1.78692 2021-09-25 23:29:00,411 [INFO] tensorflow: global_step/sec: 1.78692 INFO:tensorflow:global_step/sec: 1.82232 2021-09-25 23:29:00,960 [INFO] tensorflow: global_step/sec: 1.82232 INFO:tensorflow:global_step/sec: 1.74906 2021-09-25 23:29:01,532 [INFO] tensorflow: global_step/sec: 1.74906 INFO:tensorflow:global_step/sec: 1.80718 2021-09-25 23:29:02,085 [INFO] tensorflow: global_step/sec: 1.80718 INFO:tensorflow:epoch = 384.0, learning_rate = 0.00017605026, loss = 3.2301425e-05, step = 1920 (5.621 sec) 2021-09-25 23:29:02,658 [INFO] tensorflow: epoch = 384.0, learning_rate = 0.00017605026, loss = 3.2301425e-05, step = 1920 (5.621 sec) INFO:tensorflow:global_step/sec: 1.74236 2021-09-25 23:29:02,659 [INFO] tensorflow: global_step/sec: 1.74236 2021-09-25 23:29:02,660 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 384/500: loss: 0.00003 learning rate: 0.00018 Time taken: 0:00:02.783921 ETA: 0:05:22.934864 INFO:tensorflow:global_step/sec: 1.81963 2021-09-25 23:29:03,209 [INFO] tensorflow: global_step/sec: 1.81963 INFO:tensorflow:global_step/sec: 1.83975 2021-09-25 23:29:03,752 [INFO] tensorflow: global_step/sec: 1.83975 INFO:tensorflow:global_step/sec: 1.80145 2021-09-25 23:29:04,307 [INFO] tensorflow: global_step/sec: 1.80145 INFO:tensorflow:global_step/sec: 1.81836 2021-09-25 23:29:04,857 [INFO] tensorflow: global_step/sec: 1.81836 2021-09-25 23:29:04,860 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 114.352 INFO:tensorflow:global_step/sec: 1.73865 2021-09-25 23:29:05,433 [INFO] tensorflow: global_step/sec: 1.73865 2021-09-25 23:29:05,434 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 385/500: loss: 0.00003 learning rate: 0.00017 Time taken: 0:00:02.767000 ETA: 0:05:18.205050 INFO:tensorflow:global_step/sec: 1.79356 2021-09-25 23:29:05,990 [INFO] tensorflow: global_step/sec: 1.79356 INFO:tensorflow:global_step/sec: 1.83868 2021-09-25 23:29:06,534 [INFO] tensorflow: global_step/sec: 1.83868 INFO:tensorflow:global_step/sec: 1.81281 2021-09-25 23:29:07,086 [INFO] tensorflow: global_step/sec: 1.81281 INFO:tensorflow:global_step/sec: 1.76334 2021-09-25 23:29:07,653 [INFO] tensorflow: global_step/sec: 1.76334 INFO:tensorflow:global_step/sec: 1.78918 2021-09-25 23:29:08,212 [INFO] tensorflow: global_step/sec: 1.78918 2021-09-25 23:29:08,213 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 386/500: loss: 0.00003 learning rate: 0.00017 Time taken: 0:00:02.795020 ETA: 0:05:18.632237 INFO:tensorflow:epoch = 386.20000000000005, learning_rate = 0.00016455214, loss = 3.470662e-05, step = 1931 (6.123 sec) 2021-09-25 23:29:08,781 [INFO] tensorflow: epoch = 386.20000000000005, learning_rate = 0.00016455214, loss = 3.470662e-05, step = 1931 (6.123 sec) INFO:tensorflow:global_step/sec: 1.75232 2021-09-25 23:29:08,782 [INFO] tensorflow: global_step/sec: 1.75232 INFO:tensorflow:global_step/sec: 1.82515 2021-09-25 23:29:09,330 [INFO] tensorflow: global_step/sec: 1.82515 INFO:tensorflow:global_step/sec: 1.7821 2021-09-25 23:29:09,891 [INFO] tensorflow: global_step/sec: 1.7821 INFO:tensorflow:global_step/sec: 1.77677 2021-09-25 23:29:10,454 [INFO] tensorflow: global_step/sec: 1.77677 INFO:tensorflow:global_step/sec: 1.76291 2021-09-25 23:29:11,021 [INFO] tensorflow: global_step/sec: 1.76291 2021-09-25 23:29:11,023 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 387/500: loss: 0.00003 learning rate: 0.00016 Time taken: 0:00:02.804051 ETA: 0:05:16.857781 INFO:tensorflow:global_step/sec: 1.7445 2021-09-25 23:29:11,595 [INFO] tensorflow: global_step/sec: 1.7445 INFO:tensorflow:global_step/sec: 1.77044 2021-09-25 23:29:12,159 [INFO] tensorflow: global_step/sec: 1.77044 INFO:tensorflow:global_step/sec: 1.82128 2021-09-25 23:29:12,708 [INFO] tensorflow: global_step/sec: 1.82128 INFO:tensorflow:global_step/sec: 1.73552 2021-09-25 23:29:13,285 [INFO] tensorflow: global_step/sec: 1.73552 INFO:tensorflow:global_step/sec: 1.76132 2021-09-25 23:29:13,852 [INFO] tensorflow: global_step/sec: 1.76132 2021-09-25 23:29:13,854 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 388/500: loss: 0.00003 learning rate: 0.00016 Time taken: 0:00:02.829464 ETA: 0:05:16.899963 INFO:tensorflow:epoch = 388.20000000000005, learning_rate = 0.0001547522, loss = 4.208921e-05, step = 1941 (5.632 sec) 2021-09-25 23:29:14,413 [INFO] tensorflow: epoch = 388.20000000000005, learning_rate = 0.0001547522, loss = 4.208921e-05, step = 1941 (5.632 sec) INFO:tensorflow:global_step/sec: 1.78027 2021-09-25 23:29:14,414 [INFO] tensorflow: global_step/sec: 1.78027 INFO:tensorflow:global_step/sec: 1.78706 2021-09-25 23:29:14,974 [INFO] tensorflow: global_step/sec: 1.78706 INFO:tensorflow:global_step/sec: 1.82979 2021-09-25 23:29:15,520 [INFO] tensorflow: global_step/sec: 1.82979 INFO:tensorflow:global_step/sec: 1.74175 2021-09-25 23:29:16,094 [INFO] tensorflow: global_step/sec: 1.74175 INFO:tensorflow:global_step/sec: 1.79862 2021-09-25 23:29:16,650 [INFO] tensorflow: global_step/sec: 1.79862 2021-09-25 23:29:16,652 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 389/500: loss: 0.00004 learning rate: 0.00015 Time taken: 0:00:02.808451 ETA: 0:05:11.738081 INFO:tensorflow:global_step/sec: 1.81598 2021-09-25 23:29:17,201 [INFO] tensorflow: global_step/sec: 1.81598 INFO:tensorflow:global_step/sec: 1.81277 2021-09-25 23:29:17,753 [INFO] tensorflow: global_step/sec: 1.81277 INFO:tensorflow:global_step/sec: 1.84364 2021-09-25 23:29:18,295 [INFO] tensorflow: global_step/sec: 1.84364 INFO:tensorflow:global_step/sec: 1.77139 2021-09-25 23:29:18,860 [INFO] tensorflow: global_step/sec: 1.77139 2021-09-25 23:29:18,866 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 114.243 INFO:tensorflow:global_step/sec: 1.75565 2021-09-25 23:29:19,429 [INFO] tensorflow: global_step/sec: 1.75565 2021-09-25 23:29:19,431 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 390/500: loss: 0.00004 learning rate: 0.00015 Time taken: 0:00:02.765245 ETA: 0:05:04.176946 INFO:tensorflow:epoch = 390.20000000000005, learning_rate = 0.00014553589, loss = 3.9108767e-05, step = 1951 (5.581 sec) 2021-09-25 23:29:19,994 [INFO] tensorflow: epoch = 390.20000000000005, learning_rate = 0.00014553589, loss = 3.9108767e-05, step = 1951 (5.581 sec) INFO:tensorflow:global_step/sec: 1.76649 2021-09-25 23:29:19,995 [INFO] tensorflow: global_step/sec: 1.76649 INFO:tensorflow:global_step/sec: 1.78475 2021-09-25 23:29:20,556 [INFO] tensorflow: global_step/sec: 1.78475 INFO:tensorflow:global_step/sec: 1.80002 2021-09-25 23:29:21,111 [INFO] tensorflow: global_step/sec: 1.80002 INFO:tensorflow:global_step/sec: 1.72678 2021-09-25 23:29:21,690 [INFO] tensorflow: global_step/sec: 1.72678 INFO:tensorflow:global_step/sec: 1.7494 2021-09-25 23:29:22,262 [INFO] tensorflow: global_step/sec: 1.7494 2021-09-25 23:29:22,263 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 391/500: loss: 0.00004 learning rate: 0.00014 Time taken: 0:00:02.830309 ETA: 0:05:08.503672 INFO:tensorflow:global_step/sec: 1.79536 2021-09-25 23:29:22,819 [INFO] tensorflow: global_step/sec: 1.79536 INFO:tensorflow:global_step/sec: 1.76731 2021-09-25 23:29:23,385 [INFO] tensorflow: global_step/sec: 1.76731 INFO:tensorflow:global_step/sec: 1.71713 2021-09-25 23:29:23,967 [INFO] tensorflow: global_step/sec: 1.71713 INFO:tensorflow:global_step/sec: 1.76848 2021-09-25 23:29:24,533 [INFO] tensorflow: global_step/sec: 1.76848 INFO:tensorflow:global_step/sec: 1.82044 2021-09-25 23:29:25,082 [INFO] tensorflow: global_step/sec: 1.82044 2021-09-25 23:29:25,083 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 392/500: loss: 0.00003 learning rate: 0.00014 Time taken: 0:00:02.838585 ETA: 0:05:06.567143 INFO:tensorflow:epoch = 392.20000000000005, learning_rate = 0.00013686848, loss = 3.1473122e-05, step = 1961 (5.639 sec) 2021-09-25 23:29:25,633 [INFO] tensorflow: epoch = 392.20000000000005, learning_rate = 0.00013686848, loss = 3.1473122e-05, step = 1961 (5.639 sec) INFO:tensorflow:global_step/sec: 1.80863 2021-09-25 23:29:25,635 [INFO] tensorflow: global_step/sec: 1.80863 INFO:tensorflow:global_step/sec: 1.7788 2021-09-25 23:29:26,197 [INFO] tensorflow: global_step/sec: 1.7788 INFO:tensorflow:global_step/sec: 1.81504 2021-09-25 23:29:26,748 [INFO] tensorflow: global_step/sec: 1.81504 INFO:tensorflow:global_step/sec: 1.81482 2021-09-25 23:29:27,299 [INFO] tensorflow: global_step/sec: 1.81482 INFO:tensorflow:global_step/sec: 1.74916 2021-09-25 23:29:27,871 [INFO] tensorflow: global_step/sec: 1.74916 2021-09-25 23:29:27,872 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 393/500: loss: 0.00003 learning rate: 0.00013 Time taken: 0:00:02.767330 ETA: 0:04:56.104328 INFO:tensorflow:global_step/sec: 1.80982 2021-09-25 23:29:28,423 [INFO] tensorflow: global_step/sec: 1.80982 INFO:tensorflow:global_step/sec: 1.75996 2021-09-25 23:29:28,991 [INFO] tensorflow: global_step/sec: 1.75996 INFO:tensorflow:global_step/sec: 1.82263 2021-09-25 23:29:29,540 [INFO] tensorflow: global_step/sec: 1.82263 INFO:tensorflow:global_step/sec: 1.77518 2021-09-25 23:29:30,103 [INFO] tensorflow: global_step/sec: 1.77518 INFO:tensorflow:global_step/sec: 1.70985 2021-09-25 23:29:30,688 [INFO] tensorflow: global_step/sec: 1.70985 2021-09-25 23:29:30,690 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 394/500: loss: 0.00003 learning rate: 0.00013 Time taken: 0:00:02.805119 ETA: 0:04:57.342618 INFO:tensorflow:epoch = 394.20000000000005, learning_rate = 0.00012871726, loss = 3.1220145e-05, step = 1971 (5.593 sec) 2021-09-25 23:29:31,227 [INFO] tensorflow: epoch = 394.20000000000005, learning_rate = 0.00012871726, loss = 3.1220145e-05, step = 1971 (5.593 sec) INFO:tensorflow:global_step/sec: 1.85239 2021-09-25 23:29:31,228 [INFO] tensorflow: global_step/sec: 1.85239 INFO:tensorflow:global_step/sec: 1.74586 2021-09-25 23:29:31,801 [INFO] tensorflow: global_step/sec: 1.74586 INFO:tensorflow:global_step/sec: 1.76477 2021-09-25 23:29:32,367 [INFO] tensorflow: global_step/sec: 1.76477 INFO:tensorflow:global_step/sec: 1.79085 2021-09-25 23:29:32,926 [INFO] tensorflow: global_step/sec: 1.79085 2021-09-25 23:29:32,932 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 113.758 INFO:tensorflow:global_step/sec: 1.76569 2021-09-25 23:29:33,492 [INFO] tensorflow: global_step/sec: 1.76569 2021-09-25 23:29:33,494 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 395/500: loss: 0.00003 learning rate: 0.00013 Time taken: 0:00:02.820737 ETA: 0:04:56.177398 INFO:tensorflow:global_step/sec: 1.75077 2021-09-25 23:29:34,063 [INFO] tensorflow: global_step/sec: 1.75077 INFO:tensorflow:global_step/sec: 1.81515 2021-09-25 23:29:34,614 [INFO] tensorflow: global_step/sec: 1.81515 INFO:tensorflow:global_step/sec: 1.7301 2021-09-25 23:29:35,192 [INFO] tensorflow: global_step/sec: 1.7301 INFO:tensorflow:global_step/sec: 1.76268 2021-09-25 23:29:35,760 [INFO] tensorflow: global_step/sec: 1.76268 INFO:tensorflow:global_step/sec: 1.75762 2021-09-25 23:29:36,329 [INFO] tensorflow: global_step/sec: 1.75762 2021-09-25 23:29:36,330 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 396/500: loss: 0.00003 learning rate: 0.00012 Time taken: 0:00:02.833473 ETA: 0:04:54.681238 INFO:tensorflow:epoch = 396.20000000000005, learning_rate = 0.00012105146, loss = 3.3922453e-05, step = 1981 (5.654 sec) 2021-09-25 23:29:36,880 [INFO] tensorflow: epoch = 396.20000000000005, learning_rate = 0.00012105146, loss = 3.3922453e-05, step = 1981 (5.654 sec) INFO:tensorflow:global_step/sec: 1.80735 2021-09-25 23:29:36,882 [INFO] tensorflow: global_step/sec: 1.80735 INFO:tensorflow:global_step/sec: 1.75043 2021-09-25 23:29:37,453 [INFO] tensorflow: global_step/sec: 1.75043 INFO:tensorflow:global_step/sec: 1.83272 2021-09-25 23:29:37,999 [INFO] tensorflow: global_step/sec: 1.83272 INFO:tensorflow:global_step/sec: 1.81118 2021-09-25 23:29:38,551 [INFO] tensorflow: global_step/sec: 1.81118 INFO:tensorflow:global_step/sec: 1.75237 2021-09-25 23:29:39,122 [INFO] tensorflow: global_step/sec: 1.75237 2021-09-25 23:29:39,123 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 397/500: loss: 0.00003 learning rate: 0.00012 Time taken: 0:00:02.789446 ETA: 0:04:47.312974 INFO:tensorflow:global_step/sec: 1.81659 2021-09-25 23:29:39,672 [INFO] tensorflow: global_step/sec: 1.81659 INFO:tensorflow:global_step/sec: 1.82719 2021-09-25 23:29:40,219 [INFO] tensorflow: global_step/sec: 1.82719 INFO:tensorflow:global_step/sec: 1.75112 2021-09-25 23:29:40,790 [INFO] tensorflow: global_step/sec: 1.75112 INFO:tensorflow:global_step/sec: 1.81466 2021-09-25 23:29:41,341 [INFO] tensorflow: global_step/sec: 1.81466 INFO:tensorflow:global_step/sec: 1.75267 2021-09-25 23:29:41,912 [INFO] tensorflow: global_step/sec: 1.75267 2021-09-25 23:29:41,914 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 398/500: loss: 0.00003 learning rate: 0.00011 Time taken: 0:00:02.791162 ETA: 0:04:44.698477 INFO:tensorflow:epoch = 398.20000000000005, learning_rate = 0.00011384234, loss = 3.2294163e-05, step = 1991 (5.582 sec) 2021-09-25 23:29:42,463 [INFO] tensorflow: epoch = 398.20000000000005, learning_rate = 0.00011384234, loss = 3.2294163e-05, step = 1991 (5.582 sec) INFO:tensorflow:global_step/sec: 1.81111 2021-09-25 23:29:42,464 [INFO] tensorflow: global_step/sec: 1.81111 INFO:tensorflow:global_step/sec: 1.8311 2021-09-25 23:29:43,010 [INFO] tensorflow: global_step/sec: 1.8311 INFO:tensorflow:global_step/sec: 1.72699 2021-09-25 23:29:43,589 [INFO] tensorflow: global_step/sec: 1.72699 INFO:tensorflow:global_step/sec: 1.77716 2021-09-25 23:29:44,152 [INFO] tensorflow: global_step/sec: 1.77716 INFO:tensorflow:global_step/sec: 1.71102 2021-09-25 23:29:44,736 [INFO] tensorflow: global_step/sec: 1.71102 2021-09-25 23:29:44,738 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 399/500: loss: 0.00003 learning rate: 0.00011 Time taken: 0:00:02.805859 ETA: 0:04:43.391720 INFO:tensorflow:global_step/sec: 1.79854 2021-09-25 23:29:45,292 [INFO] tensorflow: global_step/sec: 1.79854 INFO:tensorflow:global_step/sec: 1.78885 2021-09-25 23:29:45,851 [INFO] tensorflow: global_step/sec: 1.78885 INFO:tensorflow:global_step/sec: 1.86297 2021-09-25 23:29:46,388 [INFO] tensorflow: global_step/sec: 1.86297 INFO:tensorflow:global_step/sec: 1.80224 2021-09-25 23:29:46,943 [INFO] tensorflow: global_step/sec: 1.80224 INFO:tensorflow:Saving checkpoints for step-2000. 2021-09-25 23:29:46,944 [INFO] tensorflow: Saving checkpoints for step-2000. WARNING:tensorflow:Ignoring: /tmp/tmpx1htte72; No such file or directory 2021-09-25 23:29:47,164 [WARNING] tensorflow: Ignoring: /tmp/tmpx1htte72; No such file or directory 2021-09-25 23:29:51,000 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 88.562 INFO:tensorflow:epoch = 400.0, learning_rate = 0.00010772179, loss = 3.0193198e-05, step = 2000 (9.091 sec) 2021-09-25 23:29:51,554 [INFO] tensorflow: epoch = 400.0, learning_rate = 0.00010772179, loss = 3.0193198e-05, step = 2000 (9.091 sec) INFO:tensorflow:global_step/sec: 0.216808 2021-09-25 23:29:51,555 [INFO] tensorflow: global_step/sec: 0.216808 2021-09-25 23:29:51,557 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 400/500: loss: 0.00003 learning rate: 0.00011 Time taken: 0:00:06.843541 ETA: 0:11:24.354067 INFO:tensorflow:global_step/sec: 1.79626 2021-09-25 23:29:52,112 [INFO] tensorflow: global_step/sec: 1.79626 INFO:tensorflow:global_step/sec: 1.7636 2021-09-25 23:29:52,679 [INFO] tensorflow: global_step/sec: 1.7636 INFO:tensorflow:global_step/sec: 1.80489 2021-09-25 23:29:53,233 [INFO] tensorflow: global_step/sec: 1.80489 INFO:tensorflow:global_step/sec: 1.75776 2021-09-25 23:29:53,802 [INFO] tensorflow: global_step/sec: 1.75776 Epoch 401/500 ========================= Validation cost: 0.000000 Mean average_precision (in %): 0.0000 class name average precision (in %) ------------ -------------------------- lpd 0 Median Inference Time: nan INFO:tensorflow:global_step/sec: 1.74616 2021-09-25 23:29:54,375 [INFO] tensorflow: global_step/sec: 1.74616 2021-09-25 23:29:54,376 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 401/500: loss: 0.00003 learning rate: 0.00010 Time taken: 0:00:02.804448 ETA: 0:04:37.640317 INFO:tensorflow:global_step/sec: 1.70588 2021-09-25 23:29:54,961 [INFO] tensorflow: global_step/sec: 1.70588 INFO:tensorflow:global_step/sec: 1.77907 2021-09-25 23:29:55,523 [INFO] tensorflow: global_step/sec: 1.77907 INFO:tensorflow:global_step/sec: 1.7907 2021-09-25 23:29:56,082 [INFO] tensorflow: global_step/sec: 1.7907 INFO:tensorflow:global_step/sec: 1.75039 2021-09-25 23:29:56,653 [INFO] tensorflow: global_step/sec: 1.75039 INFO:tensorflow:epoch = 402.0, learning_rate = 0.000101306396, loss = 3.5799712e-05, step = 2010 (5.658 sec) 2021-09-25 23:29:57,212 [INFO] tensorflow: epoch = 402.0, learning_rate = 0.000101306396, loss = 3.5799712e-05, step = 2010 (5.658 sec) INFO:tensorflow:global_step/sec: 1.78437 2021-09-25 23:29:57,213 [INFO] tensorflow: global_step/sec: 1.78437 2021-09-25 23:29:57,215 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 402/500: loss: 0.00004 learning rate: 0.00010 Time taken: 0:00:02.850719 ETA: 0:04:39.370483 INFO:tensorflow:global_step/sec: 1.6957 2021-09-25 23:29:57,803 [INFO] tensorflow: global_step/sec: 1.6957 INFO:tensorflow:global_step/sec: 1.7642 2021-09-25 23:29:58,370 [INFO] tensorflow: global_step/sec: 1.7642 INFO:tensorflow:global_step/sec: 1.8027 2021-09-25 23:29:58,925 [INFO] tensorflow: global_step/sec: 1.8027 INFO:tensorflow:global_step/sec: 1.7857 2021-09-25 23:29:59,485 [INFO] tensorflow: global_step/sec: 1.7857 INFO:tensorflow:global_step/sec: 1.65011 2021-09-25 23:30:00,091 [INFO] tensorflow: global_step/sec: 1.65011 2021-09-25 23:30:00,092 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 403/500: loss: 0.00003 learning rate: 0.00010 Time taken: 0:00:02.828058 ETA: 0:04:34.321603 INFO:tensorflow:global_step/sec: 1.79129 2021-09-25 23:30:00,649 [INFO] tensorflow: global_step/sec: 1.79129 INFO:tensorflow:global_step/sec: 1.7837 2021-09-25 23:30:01,210 [INFO] tensorflow: global_step/sec: 1.7837 INFO:tensorflow:global_step/sec: 1.68273 2021-09-25 23:30:01,804 [INFO] tensorflow: global_step/sec: 1.68273 INFO:tensorflow:global_step/sec: 1.81907 2021-09-25 23:30:02,354 [INFO] tensorflow: global_step/sec: 1.81907 INFO:tensorflow:epoch = 404.0, learning_rate = 9.527307e-05, loss = 3.4426557e-05, step = 2020 (5.703 sec) 2021-09-25 23:30:02,915 [INFO] tensorflow: epoch = 404.0, learning_rate = 9.527307e-05, loss = 3.4426557e-05, step = 2020 (5.703 sec) INFO:tensorflow:global_step/sec: 1.7767 2021-09-25 23:30:02,916 [INFO] tensorflow: global_step/sec: 1.7767 2021-09-25 23:30:02,918 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 404/500: loss: 0.00003 learning rate: 0.00010 Time taken: 0:00:02.868691 ETA: 0:04:35.394379 INFO:tensorflow:global_step/sec: 1.69227 2021-09-25 23:30:03,507 [INFO] tensorflow: global_step/sec: 1.69227 INFO:tensorflow:global_step/sec: 1.74321 2021-09-25 23:30:04,081 [INFO] tensorflow: global_step/sec: 1.74321 INFO:tensorflow:global_step/sec: 1.80173 2021-09-25 23:30:04,636 [INFO] tensorflow: global_step/sec: 1.80173 INFO:tensorflow:global_step/sec: 1.76699 2021-09-25 23:30:05,202 [INFO] tensorflow: global_step/sec: 1.76699 2021-09-25 23:30:05,208 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 112.644 INFO:tensorflow:global_step/sec: 1.76125 2021-09-25 23:30:05,770 [INFO] tensorflow: global_step/sec: 1.76125 2021-09-25 23:30:05,771 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 405/500: loss: 0.00003 learning rate: 0.00009 Time taken: 0:00:02.849010 ETA: 0:04:30.655926 INFO:tensorflow:global_step/sec: 1.72755 2021-09-25 23:30:06,349 [INFO] tensorflow: global_step/sec: 1.72755 INFO:tensorflow:global_step/sec: 1.78623 2021-09-25 23:30:06,908 [INFO] tensorflow: global_step/sec: 1.78623 INFO:tensorflow:global_step/sec: 1.76539 2021-09-25 23:30:07,475 [INFO] tensorflow: global_step/sec: 1.76539 INFO:tensorflow:global_step/sec: 1.77235 2021-09-25 23:30:08,039 [INFO] tensorflow: global_step/sec: 1.77235 INFO:tensorflow:epoch = 406.0, learning_rate = 8.9599074e-05, loss = 3.5776808e-05, step = 2030 (5.674 sec) 2021-09-25 23:30:08,589 [INFO] tensorflow: epoch = 406.0, learning_rate = 8.9599074e-05, loss = 3.5776808e-05, step = 2030 (5.674 sec) INFO:tensorflow:global_step/sec: 1.81371 2021-09-25 23:30:08,590 [INFO] tensorflow: global_step/sec: 1.81371 2021-09-25 23:30:08,592 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 406/500: loss: 0.00004 learning rate: 0.00009 Time taken: 0:00:02.837272 ETA: 0:04:26.703539 INFO:tensorflow:global_step/sec: 1.76868 2021-09-25 23:30:09,156 [INFO] tensorflow: global_step/sec: 1.76868 INFO:tensorflow:global_step/sec: 1.82396 2021-09-25 23:30:09,704 [INFO] tensorflow: global_step/sec: 1.82396 INFO:tensorflow:global_step/sec: 1.79818 2021-09-25 23:30:10,260 [INFO] tensorflow: global_step/sec: 1.79818 INFO:tensorflow:global_step/sec: 1.74655 2021-09-25 23:30:10,833 [INFO] tensorflow: global_step/sec: 1.74655 INFO:tensorflow:global_step/sec: 1.73361 2021-09-25 23:30:11,410 [INFO] tensorflow: global_step/sec: 1.73361 2021-09-25 23:30:11,411 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 407/500: loss: 0.00004 learning rate: 0.00009 Time taken: 0:00:02.788887 ETA: 0:04:19.366449 INFO:tensorflow:global_step/sec: 1.82809 2021-09-25 23:30:11,957 [INFO] tensorflow: global_step/sec: 1.82809 INFO:tensorflow:global_step/sec: 1.74008 2021-09-25 23:30:12,531 [INFO] tensorflow: global_step/sec: 1.74008 INFO:tensorflow:global_step/sec: 1.75748 2021-09-25 23:30:13,100 [INFO] tensorflow: global_step/sec: 1.75748 INFO:tensorflow:global_step/sec: 1.65227 2021-09-25 23:30:13,705 [INFO] tensorflow: global_step/sec: 1.65227 INFO:tensorflow:epoch = 408.0, learning_rate = 8.42629e-05, loss = 3.32024e-05, step = 2040 (5.685 sec) 2021-09-25 23:30:14,274 [INFO] tensorflow: epoch = 408.0, learning_rate = 8.42629e-05, loss = 3.32024e-05, step = 2040 (5.685 sec) INFO:tensorflow:global_step/sec: 1.75584 2021-09-25 23:30:14,275 [INFO] tensorflow: global_step/sec: 1.75584 2021-09-25 23:30:14,276 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 408/500: loss: 0.00003 learning rate: 0.00008 Time taken: 0:00:02.874571 ETA: 0:04:24.460518 INFO:tensorflow:global_step/sec: 1.7908 2021-09-25 23:30:14,833 [INFO] tensorflow: global_step/sec: 1.7908 INFO:tensorflow:global_step/sec: 1.81075 2021-09-25 23:30:15,386 [INFO] tensorflow: global_step/sec: 1.81075 INFO:tensorflow:global_step/sec: 1.7981 2021-09-25 23:30:15,942 [INFO] tensorflow: global_step/sec: 1.7981 INFO:tensorflow:global_step/sec: 1.7611 2021-09-25 23:30:16,510 [INFO] tensorflow: global_step/sec: 1.7611 INFO:tensorflow:global_step/sec: 1.7521 2021-09-25 23:30:17,080 [INFO] tensorflow: global_step/sec: 1.7521 2021-09-25 23:30:17,082 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 409/500: loss: 0.00003 learning rate: 0.00008 Time taken: 0:00:02.802740 ETA: 0:04:15.049349 INFO:tensorflow:global_step/sec: 1.78038 2021-09-25 23:30:17,642 [INFO] tensorflow: global_step/sec: 1.78038 INFO:tensorflow:global_step/sec: 1.69695 2021-09-25 23:30:18,231 [INFO] tensorflow: global_step/sec: 1.69695 INFO:tensorflow:global_step/sec: 1.75447 2021-09-25 23:30:18,801 [INFO] tensorflow: global_step/sec: 1.75447 INFO:tensorflow:global_step/sec: 1.75686 2021-09-25 23:30:19,371 [INFO] tensorflow: global_step/sec: 1.75686 2021-09-25 23:30:19,377 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 112.933 INFO:tensorflow:epoch = 410.0, learning_rate = 7.924468e-05, loss = 2.9450055e-05, step = 2050 (5.670 sec) 2021-09-25 23:30:19,944 [INFO] tensorflow: epoch = 410.0, learning_rate = 7.924468e-05, loss = 2.9450055e-05, step = 2050 (5.670 sec) INFO:tensorflow:global_step/sec: 1.74039 2021-09-25 23:30:19,945 [INFO] tensorflow: global_step/sec: 1.74039 2021-09-25 23:30:19,947 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 410/500: loss: 0.00003 learning rate: 0.00008 Time taken: 0:00:02.861137 ETA: 0:04:17.502365 INFO:tensorflow:global_step/sec: 1.80874 2021-09-25 23:30:20,498 [INFO] tensorflow: global_step/sec: 1.80874 INFO:tensorflow:global_step/sec: 1.82081 2021-09-25 23:30:21,047 [INFO] tensorflow: global_step/sec: 1.82081 INFO:tensorflow:global_step/sec: 1.83449 2021-09-25 23:30:21,592 [INFO] tensorflow: global_step/sec: 1.83449 INFO:tensorflow:global_step/sec: 1.76251 2021-09-25 23:30:22,160 [INFO] tensorflow: global_step/sec: 1.76251 INFO:tensorflow:global_step/sec: 1.72761 2021-09-25 23:30:22,739 [INFO] tensorflow: global_step/sec: 1.72761 2021-09-25 23:30:22,740 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 411/500: loss: 0.00003 learning rate: 0.00008 Time taken: 0:00:02.788525 ETA: 0:04:08.178756 INFO:tensorflow:global_step/sec: 1.75683 2021-09-25 23:30:23,308 [INFO] tensorflow: global_step/sec: 1.75683 INFO:tensorflow:global_step/sec: 1.78935 2021-09-25 23:30:23,867 [INFO] tensorflow: global_step/sec: 1.78935 INFO:tensorflow:global_step/sec: 1.77833 2021-09-25 23:30:24,429 [INFO] tensorflow: global_step/sec: 1.77833 INFO:tensorflow:global_step/sec: 1.82903 2021-09-25 23:30:24,976 [INFO] tensorflow: global_step/sec: 1.82903 INFO:tensorflow:epoch = 412.0, learning_rate = 7.4525175e-05, loss = 3.2231466e-05, step = 2060 (5.610 sec) 2021-09-25 23:30:25,554 [INFO] tensorflow: epoch = 412.0, learning_rate = 7.4525175e-05, loss = 3.2231466e-05, step = 2060 (5.610 sec) INFO:tensorflow:global_step/sec: 1.72524 2021-09-25 23:30:25,555 [INFO] tensorflow: global_step/sec: 1.72524 2021-09-25 23:30:25,557 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 412/500: loss: 0.00003 learning rate: 0.00007 Time taken: 0:00:02.812746 ETA: 0:04:07.521631 INFO:tensorflow:global_step/sec: 1.76954 2021-09-25 23:30:26,120 [INFO] tensorflow: global_step/sec: 1.76954 INFO:tensorflow:global_step/sec: 1.80201 2021-09-25 23:30:26,675 [INFO] tensorflow: global_step/sec: 1.80201 INFO:tensorflow:global_step/sec: 1.80122 2021-09-25 23:30:27,230 [INFO] tensorflow: global_step/sec: 1.80122 INFO:tensorflow:global_step/sec: 1.83453 2021-09-25 23:30:27,776 [INFO] tensorflow: global_step/sec: 1.83453 INFO:tensorflow:global_step/sec: 1.79594 2021-09-25 23:30:28,332 [INFO] tensorflow: global_step/sec: 1.79594 2021-09-25 23:30:28,334 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 413/500: loss: 0.00003 learning rate: 0.00007 Time taken: 0:00:02.802708 ETA: 0:04:03.835609 INFO:tensorflow:global_step/sec: 1.68943 2021-09-25 23:30:28,924 [INFO] tensorflow: global_step/sec: 1.68943 INFO:tensorflow:global_step/sec: 1.79924 2021-09-25 23:30:29,480 [INFO] tensorflow: global_step/sec: 1.79924 INFO:tensorflow:global_step/sec: 1.81275 2021-09-25 23:30:30,032 [INFO] tensorflow: global_step/sec: 1.81275 INFO:tensorflow:global_step/sec: 1.79617 2021-09-25 23:30:30,588 [INFO] tensorflow: global_step/sec: 1.79617 INFO:tensorflow:epoch = 414.0, learning_rate = 7.0086884e-05, loss = 2.997806e-05, step = 2070 (5.587 sec) 2021-09-25 23:30:31,141 [INFO] tensorflow: epoch = 414.0, learning_rate = 7.0086884e-05, loss = 2.997806e-05, step = 2070 (5.587 sec) INFO:tensorflow:global_step/sec: 1.80469 2021-09-25 23:30:31,143 [INFO] tensorflow: global_step/sec: 1.80469 2021-09-25 23:30:31,144 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 414/500: loss: 0.00003 learning rate: 0.00007 Time taken: 0:00:02.810854 ETA: 0:04:01.733481 INFO:tensorflow:global_step/sec: 1.81288 2021-09-25 23:30:31,694 [INFO] tensorflow: global_step/sec: 1.81288 INFO:tensorflow:global_step/sec: 1.79557 2021-09-25 23:30:32,251 [INFO] tensorflow: global_step/sec: 1.79557 INFO:tensorflow:global_step/sec: 1.84638 2021-09-25 23:30:32,793 [INFO] tensorflow: global_step/sec: 1.84638 INFO:tensorflow:global_step/sec: 1.75099 2021-09-25 23:30:33,364 [INFO] tensorflow: global_step/sec: 1.75099 2021-09-25 23:30:33,369 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 114.351 INFO:tensorflow:global_step/sec: 1.71337 2021-09-25 23:30:33,947 [INFO] tensorflow: global_step/sec: 1.71337 2021-09-25 23:30:33,949 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 415/500: loss: 0.00003 learning rate: 0.00007 Time taken: 0:00:02.774291 ETA: 0:03:55.814738 INFO:tensorflow:global_step/sec: 1.7377 2021-09-25 23:30:34,523 [INFO] tensorflow: global_step/sec: 1.7377 INFO:tensorflow:global_step/sec: 1.77627 2021-09-25 23:30:35,086 [INFO] tensorflow: global_step/sec: 1.77627 INFO:tensorflow:global_step/sec: 1.76647 2021-09-25 23:30:35,652 [INFO] tensorflow: global_step/sec: 1.76647 INFO:tensorflow:global_step/sec: 1.7513 2021-09-25 23:30:36,223 [INFO] tensorflow: global_step/sec: 1.7513 INFO:tensorflow:epoch = 416.0, learning_rate = 6.591285e-05, loss = 3.1990603e-05, step = 2080 (5.682 sec) 2021-09-25 23:30:36,824 [INFO] tensorflow: epoch = 416.0, learning_rate = 6.591285e-05, loss = 3.1990603e-05, step = 2080 (5.682 sec) INFO:tensorflow:global_step/sec: 1.66099 2021-09-25 23:30:36,825 [INFO] tensorflow: global_step/sec: 1.66099 2021-09-25 23:30:36,827 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 416/500: loss: 0.00003 learning rate: 0.00007 Time taken: 0:00:02.856516 ETA: 0:03:59.947334 INFO:tensorflow:global_step/sec: 1.76434 2021-09-25 23:30:37,392 [INFO] tensorflow: global_step/sec: 1.76434 INFO:tensorflow:global_step/sec: 1.84761 2021-09-25 23:30:37,933 [INFO] tensorflow: global_step/sec: 1.84761 INFO:tensorflow:global_step/sec: 1.76989 2021-09-25 23:30:38,498 [INFO] tensorflow: global_step/sec: 1.76989 INFO:tensorflow:global_step/sec: 1.80508 2021-09-25 23:30:39,052 [INFO] tensorflow: global_step/sec: 1.80508 INFO:tensorflow:global_step/sec: 1.71349 2021-09-25 23:30:39,636 [INFO] tensorflow: global_step/sec: 1.71349 2021-09-25 23:30:39,637 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 417/500: loss: 0.00004 learning rate: 0.00006 Time taken: 0:00:02.829127 ETA: 0:03:54.817527 INFO:tensorflow:global_step/sec: 1.80798 2021-09-25 23:30:40,189 [INFO] tensorflow: global_step/sec: 1.80798 INFO:tensorflow:global_step/sec: 1.70098 2021-09-25 23:30:40,777 [INFO] tensorflow: global_step/sec: 1.70098 INFO:tensorflow:global_step/sec: 1.7916 2021-09-25 23:30:41,335 [INFO] tensorflow: global_step/sec: 1.7916 INFO:tensorflow:global_step/sec: 1.7046 2021-09-25 23:30:41,922 [INFO] tensorflow: global_step/sec: 1.7046 INFO:tensorflow:epoch = 418.0, learning_rate = 6.198739e-05, loss = 3.1609772e-05, step = 2090 (5.670 sec) 2021-09-25 23:30:42,494 [INFO] tensorflow: epoch = 418.0, learning_rate = 6.198739e-05, loss = 3.1609772e-05, step = 2090 (5.670 sec) INFO:tensorflow:global_step/sec: 1.74362 2021-09-25 23:30:42,495 [INFO] tensorflow: global_step/sec: 1.74362 2021-09-25 23:30:42,496 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 418/500: loss: 0.00003 learning rate: 0.00006 Time taken: 0:00:02.869367 ETA: 0:03:55.288124 INFO:tensorflow:global_step/sec: 1.73559 2021-09-25 23:30:43,071 [INFO] tensorflow: global_step/sec: 1.73559 INFO:tensorflow:global_step/sec: 1.74705 2021-09-25 23:30:43,644 [INFO] tensorflow: global_step/sec: 1.74705 INFO:tensorflow:global_step/sec: 1.70148 2021-09-25 23:30:44,231 [INFO] tensorflow: global_step/sec: 1.70148 INFO:tensorflow:global_step/sec: 1.81314 2021-09-25 23:30:44,783 [INFO] tensorflow: global_step/sec: 1.81314 INFO:tensorflow:global_step/sec: 1.7657 2021-09-25 23:30:45,349 [INFO] tensorflow: global_step/sec: 1.7657 2021-09-25 23:30:45,351 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 419/500: loss: 0.00003 learning rate: 0.00006 Time taken: 0:00:02.859041 ETA: 0:03:51.582319 INFO:tensorflow:global_step/sec: 1.82339 2021-09-25 23:30:45,898 [INFO] tensorflow: global_step/sec: 1.82339 INFO:tensorflow:global_step/sec: 1.78914 2021-09-25 23:30:46,457 [INFO] tensorflow: global_step/sec: 1.78914 INFO:tensorflow:global_step/sec: 1.7971 2021-09-25 23:30:47,013 [INFO] tensorflow: global_step/sec: 1.7971 INFO:tensorflow:global_step/sec: 1.71711 2021-09-25 23:30:47,595 [INFO] tensorflow: global_step/sec: 1.71711 INFO:tensorflow:Saving checkpoints for step-2100. 2021-09-25 23:30:47,596 [INFO] tensorflow: Saving checkpoints for step-2100. WARNING:tensorflow:Ignoring: /tmp/tmpv5jutrrt; No such file or directory 2021-09-25 23:30:47,817 [WARNING] tensorflow: Ignoring: /tmp/tmpv5jutrrt; No such file or directory 2021-09-25 23:30:51,612 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 87.713 INFO:tensorflow:epoch = 420.0, learning_rate = 5.829572e-05, loss = 2.8639577e-05, step = 2100 (9.687 sec) 2021-09-25 23:30:52,181 [INFO] tensorflow: epoch = 420.0, learning_rate = 5.829572e-05, loss = 2.8639577e-05, step = 2100 (9.687 sec) INFO:tensorflow:global_step/sec: 0.218021 2021-09-25 23:30:52,182 [INFO] tensorflow: global_step/sec: 0.218021 2021-09-25 23:30:52,183 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 420/500: loss: 0.00003 learning rate: 0.00006 Time taken: 0:00:06.823987 ETA: 0:09:05.918941 INFO:tensorflow:global_step/sec: 1.70557 2021-09-25 23:30:52,768 [INFO] tensorflow: global_step/sec: 1.70557 INFO:tensorflow:global_step/sec: 1.75135 2021-09-25 23:30:53,339 [INFO] tensorflow: global_step/sec: 1.75135 INFO:tensorflow:global_step/sec: 1.66818 2021-09-25 23:30:53,939 [INFO] tensorflow: global_step/sec: 1.66818 INFO:tensorflow:global_step/sec: 1.78713 2021-09-25 23:30:54,498 [INFO] tensorflow: global_step/sec: 1.78713 Epoch 421/500 ========================= Validation cost: 0.000000 Mean average_precision (in %): 0.0000 class name average precision (in %) ------------ -------------------------- lpd 0 Median Inference Time: nan INFO:tensorflow:global_step/sec: 1.71686 2021-09-25 23:30:55,081 [INFO] tensorflow: global_step/sec: 1.71686 2021-09-25 23:30:55,082 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 421/500: loss: 0.00003 learning rate: 0.00006 Time taken: 0:00:02.893901 ETA: 0:03:48.618150 INFO:tensorflow:global_step/sec: 1.84932 2021-09-25 23:30:55,622 [INFO] tensorflow: global_step/sec: 1.84932 INFO:tensorflow:global_step/sec: 1.81488 2021-09-25 23:30:56,173 [INFO] tensorflow: global_step/sec: 1.81488 INFO:tensorflow:global_step/sec: 1.81788 2021-09-25 23:30:56,723 [INFO] tensorflow: global_step/sec: 1.81788 INFO:tensorflow:global_step/sec: 1.81574 2021-09-25 23:30:57,273 [INFO] tensorflow: global_step/sec: 1.81574 INFO:tensorflow:epoch = 422.0, learning_rate = 5.4823908e-05, loss = 3.596406e-05, step = 2110 (5.631 sec) 2021-09-25 23:30:57,812 [INFO] tensorflow: epoch = 422.0, learning_rate = 5.4823908e-05, loss = 3.596406e-05, step = 2110 (5.631 sec) INFO:tensorflow:global_step/sec: 1.85215 2021-09-25 23:30:57,813 [INFO] tensorflow: global_step/sec: 1.85215 2021-09-25 23:30:57,815 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 422/500: loss: 0.00004 learning rate: 0.00005 Time taken: 0:00:02.771879 ETA: 0:03:36.206596 INFO:tensorflow:global_step/sec: 1.76564 2021-09-25 23:30:58,380 [INFO] tensorflow: global_step/sec: 1.76564 INFO:tensorflow:global_step/sec: 1.84203 2021-09-25 23:30:58,923 [INFO] tensorflow: global_step/sec: 1.84203 INFO:tensorflow:global_step/sec: 1.76644 2021-09-25 23:30:59,489 [INFO] tensorflow: global_step/sec: 1.76644 INFO:tensorflow:global_step/sec: 1.74886 2021-09-25 23:31:00,061 [INFO] tensorflow: global_step/sec: 1.74886 INFO:tensorflow:global_step/sec: 1.81183 2021-09-25 23:31:00,612 [INFO] tensorflow: global_step/sec: 1.81183 2021-09-25 23:31:00,614 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 423/500: loss: 0.00003 learning rate: 0.00005 Time taken: 0:00:02.785321 ETA: 0:03:34.469680 INFO:tensorflow:global_step/sec: 1.83028 2021-09-25 23:31:01,159 [INFO] tensorflow: global_step/sec: 1.83028 INFO:tensorflow:global_step/sec: 1.80931 2021-09-25 23:31:01,712 [INFO] tensorflow: global_step/sec: 1.80931 INFO:tensorflow:global_step/sec: 1.76243 2021-09-25 23:31:02,279 [INFO] tensorflow: global_step/sec: 1.76243 INFO:tensorflow:global_step/sec: 1.77344 2021-09-25 23:31:02,843 [INFO] tensorflow: global_step/sec: 1.77344 INFO:tensorflow:epoch = 424.0, learning_rate = 5.155886e-05, loss = 3.0517876e-05, step = 2120 (5.580 sec) 2021-09-25 23:31:03,392 [INFO] tensorflow: epoch = 424.0, learning_rate = 5.155886e-05, loss = 3.0517876e-05, step = 2120 (5.580 sec) INFO:tensorflow:global_step/sec: 1.8176 2021-09-25 23:31:03,393 [INFO] tensorflow: global_step/sec: 1.8176 2021-09-25 23:31:03,394 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 424/500: loss: 0.00003 learning rate: 0.00005 Time taken: 0:00:02.780206 ETA: 0:03:31.295635 INFO:tensorflow:global_step/sec: 1.76517 2021-09-25 23:31:03,960 [INFO] tensorflow: global_step/sec: 1.76517 INFO:tensorflow:global_step/sec: 1.76374 2021-09-25 23:31:04,526 [INFO] tensorflow: global_step/sec: 1.76374 INFO:tensorflow:global_step/sec: 1.7516 2021-09-25 23:31:05,097 [INFO] tensorflow: global_step/sec: 1.7516 INFO:tensorflow:global_step/sec: 1.77069 2021-09-25 23:31:05,662 [INFO] tensorflow: global_step/sec: 1.77069 2021-09-25 23:31:05,667 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 113.847 INFO:tensorflow:global_step/sec: 1.81481 2021-09-25 23:31:06,213 [INFO] tensorflow: global_step/sec: 1.81481 2021-09-25 23:31:06,215 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 425/500: loss: 0.00003 learning rate: 0.00005 Time taken: 0:00:02.819580 ETA: 0:03:31.468470 INFO:tensorflow:global_step/sec: 1.82413 2021-09-25 23:31:06,761 [INFO] tensorflow: global_step/sec: 1.82413 INFO:tensorflow:global_step/sec: 1.73283 2021-09-25 23:31:07,338 [INFO] tensorflow: global_step/sec: 1.73283 INFO:tensorflow:global_step/sec: 1.73677 2021-09-25 23:31:07,914 [INFO] tensorflow: global_step/sec: 1.73677 INFO:tensorflow:global_step/sec: 1.78967 2021-09-25 23:31:08,473 [INFO] tensorflow: global_step/sec: 1.78967 INFO:tensorflow:epoch = 426.0, learning_rate = 4.8488262e-05, loss = 3.2638032e-05, step = 2130 (5.656 sec) 2021-09-25 23:31:09,048 [INFO] tensorflow: epoch = 426.0, learning_rate = 4.8488262e-05, loss = 3.2638032e-05, step = 2130 (5.656 sec) INFO:tensorflow:global_step/sec: 1.7347 2021-09-25 23:31:09,049 [INFO] tensorflow: global_step/sec: 1.7347 2021-09-25 23:31:09,051 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 426/500: loss: 0.00003 learning rate: 0.00005 Time taken: 0:00:02.811626 ETA: 0:03:28.060338 INFO:tensorflow:global_step/sec: 1.78157 2021-09-25 23:31:09,611 [INFO] tensorflow: global_step/sec: 1.78157 INFO:tensorflow:global_step/sec: 1.84147 2021-09-25 23:31:10,154 [INFO] tensorflow: global_step/sec: 1.84147 INFO:tensorflow:global_step/sec: 1.82526 2021-09-25 23:31:10,702 [INFO] tensorflow: global_step/sec: 1.82526 INFO:tensorflow:global_step/sec: 1.80749 2021-09-25 23:31:11,255 [INFO] tensorflow: global_step/sec: 1.80749 INFO:tensorflow:global_step/sec: 1.80307 2021-09-25 23:31:11,810 [INFO] tensorflow: global_step/sec: 1.80307 2021-09-25 23:31:11,811 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 427/500: loss: 0.00003 learning rate: 0.00005 Time taken: 0:00:02.780354 ETA: 0:03:22.965826 INFO:tensorflow:global_step/sec: 1.78623 2021-09-25 23:31:12,369 [INFO] tensorflow: global_step/sec: 1.78623 INFO:tensorflow:global_step/sec: 1.84927 2021-09-25 23:31:12,910 [INFO] tensorflow: global_step/sec: 1.84927 INFO:tensorflow:global_step/sec: 1.76174 2021-09-25 23:31:13,478 [INFO] tensorflow: global_step/sec: 1.76174 INFO:tensorflow:global_step/sec: 1.75283 2021-09-25 23:31:14,048 [INFO] tensorflow: global_step/sec: 1.75283 INFO:tensorflow:epoch = 428.0, learning_rate = 4.5600533e-05, loss = 3.1226013e-05, step = 2140 (5.552 sec) 2021-09-25 23:31:14,600 [INFO] tensorflow: epoch = 428.0, learning_rate = 4.5600533e-05, loss = 3.1226013e-05, step = 2140 (5.552 sec) INFO:tensorflow:global_step/sec: 1.80916 2021-09-25 23:31:14,601 [INFO] tensorflow: global_step/sec: 1.80916 2021-09-25 23:31:14,602 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 428/500: loss: 0.00003 learning rate: 0.00005 Time taken: 0:00:02.793884 ETA: 0:03:21.159634 INFO:tensorflow:global_step/sec: 1.76445 2021-09-25 23:31:15,168 [INFO] tensorflow: global_step/sec: 1.76445 INFO:tensorflow:global_step/sec: 1.74687 2021-09-25 23:31:15,740 [INFO] tensorflow: global_step/sec: 1.74687 INFO:tensorflow:global_step/sec: 1.75617 2021-09-25 23:31:16,310 [INFO] tensorflow: global_step/sec: 1.75617 INFO:tensorflow:global_step/sec: 1.75947 2021-09-25 23:31:16,878 [INFO] tensorflow: global_step/sec: 1.75947 INFO:tensorflow:global_step/sec: 1.77079 2021-09-25 23:31:17,443 [INFO] tensorflow: global_step/sec: 1.77079 2021-09-25 23:31:17,444 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 429/500: loss: 0.00003 learning rate: 0.00004 Time taken: 0:00:02.828098 ETA: 0:03:20.794979 INFO:tensorflow:global_step/sec: 1.84434 2021-09-25 23:31:17,985 [INFO] tensorflow: global_step/sec: 1.84434 INFO:tensorflow:global_step/sec: 1.74753 2021-09-25 23:31:18,557 [INFO] tensorflow: global_step/sec: 1.74753 INFO:tensorflow:global_step/sec: 1.69801 2021-09-25 23:31:19,146 [INFO] tensorflow: global_step/sec: 1.69801 INFO:tensorflow:global_step/sec: 1.70305 2021-09-25 23:31:19,733 [INFO] tensorflow: global_step/sec: 1.70305 2021-09-25 23:31:19,739 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 113.702 INFO:tensorflow:epoch = 430.0, learning_rate = 4.2884825e-05, loss = 3.105383e-05, step = 2150 (5.698 sec) 2021-09-25 23:31:20,298 [INFO] tensorflow: epoch = 430.0, learning_rate = 4.2884825e-05, loss = 3.105383e-05, step = 2150 (5.698 sec) INFO:tensorflow:global_step/sec: 1.76747 2021-09-25 23:31:20,299 [INFO] tensorflow: global_step/sec: 1.76747 2021-09-25 23:31:20,300 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 430/500: loss: 0.00003 learning rate: 0.00004 Time taken: 0:00:02.854934 ETA: 0:03:19.845362 INFO:tensorflow:global_step/sec: 1.76203 2021-09-25 23:31:20,867 [INFO] tensorflow: global_step/sec: 1.76203 INFO:tensorflow:global_step/sec: 1.83571 2021-09-25 23:31:21,411 [INFO] tensorflow: global_step/sec: 1.83571 INFO:tensorflow:global_step/sec: 1.85898 2021-09-25 23:31:21,949 [INFO] tensorflow: global_step/sec: 1.85898 INFO:tensorflow:global_step/sec: 1.7887 2021-09-25 23:31:22,508 [INFO] tensorflow: global_step/sec: 1.7887 INFO:tensorflow:global_step/sec: 1.70587 2021-09-25 23:31:23,095 [INFO] tensorflow: global_step/sec: 1.70587 2021-09-25 23:31:23,096 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 431/500: loss: 0.00003 learning rate: 0.00004 Time taken: 0:00:02.773027 ETA: 0:03:11.338892 INFO:tensorflow:global_step/sec: 1.83289 2021-09-25 23:31:23,640 [INFO] tensorflow: global_step/sec: 1.83289 INFO:tensorflow:global_step/sec: 1.79335 2021-09-25 23:31:24,198 [INFO] tensorflow: global_step/sec: 1.79335 INFO:tensorflow:global_step/sec: 1.78669 2021-09-25 23:31:24,757 [INFO] tensorflow: global_step/sec: 1.78669 INFO:tensorflow:global_step/sec: 1.7879 2021-09-25 23:31:25,317 [INFO] tensorflow: global_step/sec: 1.7879 INFO:tensorflow:epoch = 432.0, learning_rate = 4.033081e-05, loss = 3.472213e-05, step = 2160 (5.584 sec) 2021-09-25 23:31:25,881 [INFO] tensorflow: epoch = 432.0, learning_rate = 4.033081e-05, loss = 3.472213e-05, step = 2160 (5.584 sec) INFO:tensorflow:global_step/sec: 1.76756 2021-09-25 23:31:25,882 [INFO] tensorflow: global_step/sec: 1.76756 2021-09-25 23:31:25,884 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 432/500: loss: 0.00003 learning rate: 0.00004 Time taken: 0:00:02.808784 ETA: 0:03:10.997280 INFO:tensorflow:global_step/sec: 1.71091 2021-09-25 23:31:26,467 [INFO] tensorflow: global_step/sec: 1.71091 INFO:tensorflow:global_step/sec: 1.79992 2021-09-25 23:31:27,023 [INFO] tensorflow: global_step/sec: 1.79992 INFO:tensorflow:global_step/sec: 1.79129 2021-09-25 23:31:27,581 [INFO] tensorflow: global_step/sec: 1.79129 INFO:tensorflow:global_step/sec: 1.80488 2021-09-25 23:31:28,135 [INFO] tensorflow: global_step/sec: 1.80488 INFO:tensorflow:global_step/sec: 1.80471 2021-09-25 23:31:28,689 [INFO] tensorflow: global_step/sec: 1.80471 2021-09-25 23:31:28,691 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 433/500: loss: 0.00004 learning rate: 0.00004 Time taken: 0:00:02.815675 ETA: 0:03:08.650194 INFO:tensorflow:global_step/sec: 1.81661 2021-09-25 23:31:29,239 [INFO] tensorflow: global_step/sec: 1.81661 INFO:tensorflow:global_step/sec: 1.7692 2021-09-25 23:31:29,805 [INFO] tensorflow: global_step/sec: 1.7692 INFO:tensorflow:global_step/sec: 1.69994 2021-09-25 23:31:30,393 [INFO] tensorflow: global_step/sec: 1.69994 INFO:tensorflow:global_step/sec: 1.80962 2021-09-25 23:31:30,946 [INFO] tensorflow: global_step/sec: 1.80962 INFO:tensorflow:epoch = 434.0, learning_rate = 3.79289e-05, loss = 3.003758e-05, step = 2170 (5.628 sec) 2021-09-25 23:31:31,510 [INFO] tensorflow: epoch = 434.0, learning_rate = 3.79289e-05, loss = 3.003758e-05, step = 2170 (5.628 sec) INFO:tensorflow:global_step/sec: 1.76829 2021-09-25 23:31:31,511 [INFO] tensorflow: global_step/sec: 1.76829 2021-09-25 23:31:31,512 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 434/500: loss: 0.00003 learning rate: 0.00004 Time taken: 0:00:02.810992 ETA: 0:03:05.525488 INFO:tensorflow:global_step/sec: 1.75725 2021-09-25 23:31:32,080 [INFO] tensorflow: global_step/sec: 1.75725 INFO:tensorflow:global_step/sec: 1.79654 2021-09-25 23:31:32,637 [INFO] tensorflow: global_step/sec: 1.79654 INFO:tensorflow:global_step/sec: 1.73116 2021-09-25 23:31:33,214 [INFO] tensorflow: global_step/sec: 1.73116 INFO:tensorflow:global_step/sec: 1.77903 2021-09-25 23:31:33,776 [INFO] tensorflow: global_step/sec: 1.77903 2021-09-25 23:31:33,779 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 113.967 INFO:tensorflow:global_step/sec: 1.78262 2021-09-25 23:31:34,337 [INFO] tensorflow: global_step/sec: 1.78262 2021-09-25 23:31:34,339 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 435/500: loss: 0.00004 learning rate: 0.00004 Time taken: 0:00:02.827578 ETA: 0:03:03.792590 INFO:tensorflow:global_step/sec: 1.75556 2021-09-25 23:31:34,907 [INFO] tensorflow: global_step/sec: 1.75556 INFO:tensorflow:global_step/sec: 1.78211 2021-09-25 23:31:35,468 [INFO] tensorflow: global_step/sec: 1.78211 INFO:tensorflow:global_step/sec: 1.80018 2021-09-25 23:31:36,024 [INFO] tensorflow: global_step/sec: 1.80018 INFO:tensorflow:global_step/sec: 1.75561 2021-09-25 23:31:36,593 [INFO] tensorflow: global_step/sec: 1.75561 INFO:tensorflow:epoch = 436.0, learning_rate = 3.5670037e-05, loss = 3.3011907e-05, step = 2180 (5.650 sec) 2021-09-25 23:31:37,160 [INFO] tensorflow: epoch = 436.0, learning_rate = 3.5670037e-05, loss = 3.3011907e-05, step = 2180 (5.650 sec) INFO:tensorflow:global_step/sec: 1.76164 2021-09-25 23:31:37,161 [INFO] tensorflow: global_step/sec: 1.76164 2021-09-25 23:31:37,162 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 436/500: loss: 0.00003 learning rate: 0.00004 Time taken: 0:00:02.819353 ETA: 0:03:00.438583 INFO:tensorflow:global_step/sec: 1.7978 2021-09-25 23:31:37,717 [INFO] tensorflow: global_step/sec: 1.7978 INFO:tensorflow:global_step/sec: 1.76233 2021-09-25 23:31:38,285 [INFO] tensorflow: global_step/sec: 1.76233 INFO:tensorflow:global_step/sec: 1.80572 2021-09-25 23:31:38,838 [INFO] tensorflow: global_step/sec: 1.80572 INFO:tensorflow:global_step/sec: 1.78171 2021-09-25 23:31:39,400 [INFO] tensorflow: global_step/sec: 1.78171 INFO:tensorflow:global_step/sec: 1.79244 2021-09-25 23:31:39,958 [INFO] tensorflow: global_step/sec: 1.79244 2021-09-25 23:31:39,959 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 437/500: loss: 0.00003 learning rate: 0.00003 Time taken: 0:00:02.806846 ETA: 0:02:56.831292 INFO:tensorflow:global_step/sec: 1.73891 2021-09-25 23:31:40,533 [INFO] tensorflow: global_step/sec: 1.73891 INFO:tensorflow:global_step/sec: 1.82492 2021-09-25 23:31:41,081 [INFO] tensorflow: global_step/sec: 1.82492 INFO:tensorflow:global_step/sec: 1.82433 2021-09-25 23:31:41,629 [INFO] tensorflow: global_step/sec: 1.82433 INFO:tensorflow:global_step/sec: 1.79606 2021-09-25 23:31:42,186 [INFO] tensorflow: global_step/sec: 1.79606 INFO:tensorflow:epoch = 438.0, learning_rate = 3.354567e-05, loss = 3.072808e-05, step = 2190 (5.578 sec) 2021-09-25 23:31:42,738 [INFO] tensorflow: epoch = 438.0, learning_rate = 3.354567e-05, loss = 3.072808e-05, step = 2190 (5.578 sec) INFO:tensorflow:global_step/sec: 1.8055 2021-09-25 23:31:42,739 [INFO] tensorflow: global_step/sec: 1.8055 2021-09-25 23:31:42,741 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 438/500: loss: 0.00003 learning rate: 0.00003 Time taken: 0:00:02.781869 ETA: 0:02:52.475889 INFO:tensorflow:global_step/sec: 1.80455 2021-09-25 23:31:43,294 [INFO] tensorflow: global_step/sec: 1.80455 INFO:tensorflow:global_step/sec: 1.80111 2021-09-25 23:31:43,849 [INFO] tensorflow: global_step/sec: 1.80111 INFO:tensorflow:global_step/sec: 1.70597 2021-09-25 23:31:44,435 [INFO] tensorflow: global_step/sec: 1.70597 INFO:tensorflow:global_step/sec: 1.7613 2021-09-25 23:31:45,003 [INFO] tensorflow: global_step/sec: 1.7613 INFO:tensorflow:global_step/sec: 1.73556 2021-09-25 23:31:45,579 [INFO] tensorflow: global_step/sec: 1.73556 2021-09-25 23:31:45,580 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 439/500: loss: 0.00003 learning rate: 0.00003 Time taken: 0:00:02.818600 ETA: 0:02:51.934625 INFO:tensorflow:global_step/sec: 1.81168 2021-09-25 23:31:46,131 [INFO] tensorflow: global_step/sec: 1.81168 INFO:tensorflow:global_step/sec: 1.8285 2021-09-25 23:31:46,678 [INFO] tensorflow: global_step/sec: 1.8285 INFO:tensorflow:global_step/sec: 1.81022 2021-09-25 23:31:47,230 [INFO] tensorflow: global_step/sec: 1.81022 INFO:tensorflow:global_step/sec: 1.74337 2021-09-25 23:31:47,804 [INFO] tensorflow: global_step/sec: 1.74337 INFO:tensorflow:Saving checkpoints for step-2200. 2021-09-25 23:31:47,805 [INFO] tensorflow: Saving checkpoints for step-2200. WARNING:tensorflow:Ignoring: /tmp/tmposd1m_bd; No such file or directory 2021-09-25 23:31:48,031 [WARNING] tensorflow: Ignoring: /tmp/tmposd1m_bd; No such file or directory 2021-09-25 23:31:51,877 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 88.413 INFO:tensorflow:epoch = 440.0, learning_rate = 3.154785e-05, loss = 3.0746192e-05, step = 2200 (9.726 sec) 2021-09-25 23:31:52,464 [INFO] tensorflow: epoch = 440.0, learning_rate = 3.154785e-05, loss = 3.0746192e-05, step = 2200 (9.726 sec) INFO:tensorflow:global_step/sec: 0.214508 2021-09-25 23:31:52,466 [INFO] tensorflow: global_step/sec: 0.214508 2021-09-25 23:31:52,467 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 440/500: loss: 0.00003 learning rate: 0.00003 Time taken: 0:00:06.867193 ETA: 0:06:52.031579 INFO:tensorflow:global_step/sec: 1.71552 2021-09-25 23:31:53,049 [INFO] tensorflow: global_step/sec: 1.71552 INFO:tensorflow:global_step/sec: 1.79388 2021-09-25 23:31:53,606 [INFO] tensorflow: global_step/sec: 1.79388 INFO:tensorflow:global_step/sec: 1.74459 2021-09-25 23:31:54,179 [INFO] tensorflow: global_step/sec: 1.74459 INFO:tensorflow:global_step/sec: 1.80408 2021-09-25 23:31:54,733 [INFO] tensorflow: global_step/sec: 1.80408 Epoch 441/500 ========================= Validation cost: 0.000000 Mean average_precision (in %): 0.0000 class name average precision (in %) ------------ -------------------------- lpd 0 Median Inference Time: nan INFO:tensorflow:global_step/sec: 1.80166 2021-09-25 23:31:55,289 [INFO] tensorflow: global_step/sec: 1.80166 2021-09-25 23:31:55,290 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 441/500: loss: 0.00004 learning rate: 0.00003 Time taken: 0:00:02.859078 ETA: 0:02:48.685598 INFO:tensorflow:global_step/sec: 1.67584 2021-09-25 23:31:55,885 [INFO] tensorflow: global_step/sec: 1.67584 INFO:tensorflow:global_step/sec: 1.68435 2021-09-25 23:31:56,479 [INFO] tensorflow: global_step/sec: 1.68435 INFO:tensorflow:global_step/sec: 1.69308 2021-09-25 23:31:57,070 [INFO] tensorflow: global_step/sec: 1.69308 INFO:tensorflow:global_step/sec: 1.70493 2021-09-25 23:31:57,656 [INFO] tensorflow: global_step/sec: 1.70493 INFO:tensorflow:epoch = 442.0, learning_rate = 2.9669012e-05, loss = 3.786678e-05, step = 2210 (5.776 sec) 2021-09-25 23:31:58,240 [INFO] tensorflow: epoch = 442.0, learning_rate = 2.9669012e-05, loss = 3.786678e-05, step = 2210 (5.776 sec) INFO:tensorflow:global_step/sec: 1.70933 2021-09-25 23:31:58,241 [INFO] tensorflow: global_step/sec: 1.70933 2021-09-25 23:31:58,243 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 442/500: loss: 0.00004 learning rate: 0.00003 Time taken: 0:00:02.919402 ETA: 0:02:49.325337 INFO:tensorflow:global_step/sec: 1.79517 2021-09-25 23:31:58,798 [INFO] tensorflow: global_step/sec: 1.79517 INFO:tensorflow:global_step/sec: 1.68235 2021-09-25 23:31:59,393 [INFO] tensorflow: global_step/sec: 1.68235 INFO:tensorflow:global_step/sec: 1.82958 2021-09-25 23:31:59,939 [INFO] tensorflow: global_step/sec: 1.82958 INFO:tensorflow:global_step/sec: 1.70175 2021-09-25 23:32:00,527 [INFO] tensorflow: global_step/sec: 1.70175 INFO:tensorflow:global_step/sec: 1.72102 2021-09-25 23:32:01,108 [INFO] tensorflow: global_step/sec: 1.72102 2021-09-25 23:32:01,110 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 443/500: loss: 0.00003 learning rate: 0.00003 Time taken: 0:00:02.873171 ETA: 0:02:43.770752 INFO:tensorflow:global_step/sec: 1.78681 2021-09-25 23:32:01,668 [INFO] tensorflow: global_step/sec: 1.78681 INFO:tensorflow:global_step/sec: 1.85126 2021-09-25 23:32:02,208 [INFO] tensorflow: global_step/sec: 1.85126 INFO:tensorflow:global_step/sec: 1.81013 2021-09-25 23:32:02,760 [INFO] tensorflow: global_step/sec: 1.81013 INFO:tensorflow:global_step/sec: 1.83219 2021-09-25 23:32:03,306 [INFO] tensorflow: global_step/sec: 1.83219 INFO:tensorflow:epoch = 444.0, learning_rate = 2.7902097e-05, loss = 3.635897e-05, step = 2220 (5.622 sec) 2021-09-25 23:32:03,862 [INFO] tensorflow: epoch = 444.0, learning_rate = 2.7902097e-05, loss = 3.635897e-05, step = 2220 (5.622 sec) INFO:tensorflow:global_step/sec: 1.79395 2021-09-25 23:32:03,863 [INFO] tensorflow: global_step/sec: 1.79395 2021-09-25 23:32:03,865 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 444/500: loss: 0.00004 learning rate: 0.00003 Time taken: 0:00:02.778360 ETA: 0:02:35.588181 INFO:tensorflow:global_step/sec: 1.68929 2021-09-25 23:32:04,455 [INFO] tensorflow: global_step/sec: 1.68929 INFO:tensorflow:global_step/sec: 1.74822 2021-09-25 23:32:05,027 [INFO] tensorflow: global_step/sec: 1.74822 INFO:tensorflow:global_step/sec: 1.79004 2021-09-25 23:32:05,586 [INFO] tensorflow: global_step/sec: 1.79004 INFO:tensorflow:global_step/sec: 1.77627 2021-09-25 23:32:06,149 [INFO] tensorflow: global_step/sec: 1.77627 2021-09-25 23:32:06,154 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 112.104 INFO:tensorflow:global_step/sec: 1.7768 2021-09-25 23:32:06,712 [INFO] tensorflow: global_step/sec: 1.7768 2021-09-25 23:32:06,713 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 445/500: loss: 0.00003 learning rate: 0.00003 Time taken: 0:00:02.839491 ETA: 0:02:36.171986 INFO:tensorflow:global_step/sec: 1.75524 2021-09-25 23:32:07,281 [INFO] tensorflow: global_step/sec: 1.75524 INFO:tensorflow:global_step/sec: 1.77972 2021-09-25 23:32:07,843 [INFO] tensorflow: global_step/sec: 1.77972 INFO:tensorflow:global_step/sec: 1.79872 2021-09-25 23:32:08,399 [INFO] tensorflow: global_step/sec: 1.79872 INFO:tensorflow:global_step/sec: 1.75738 2021-09-25 23:32:08,968 [INFO] tensorflow: global_step/sec: 1.75738 INFO:tensorflow:epoch = 446.0, learning_rate = 2.6240381e-05, loss = 2.95144e-05, step = 2230 (5.661 sec) 2021-09-25 23:32:09,523 [INFO] tensorflow: epoch = 446.0, learning_rate = 2.6240381e-05, loss = 2.95144e-05, step = 2230 (5.661 sec) INFO:tensorflow:global_step/sec: 1.79773 2021-09-25 23:32:09,525 [INFO] tensorflow: global_step/sec: 1.79773 2021-09-25 23:32:09,526 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 446/500: loss: 0.00003 learning rate: 0.00003 Time taken: 0:00:02.818399 ETA: 0:02:32.193543 INFO:tensorflow:global_step/sec: 1.85306 2021-09-25 23:32:10,064 [INFO] tensorflow: global_step/sec: 1.85306 INFO:tensorflow:global_step/sec: 1.78309 2021-09-25 23:32:10,625 [INFO] tensorflow: global_step/sec: 1.78309 INFO:tensorflow:global_step/sec: 1.73789 2021-09-25 23:32:11,200 [INFO] tensorflow: global_step/sec: 1.73789 INFO:tensorflow:global_step/sec: 1.80164 2021-09-25 23:32:11,756 [INFO] tensorflow: global_step/sec: 1.80164 INFO:tensorflow:global_step/sec: 1.82385 2021-09-25 23:32:12,304 [INFO] tensorflow: global_step/sec: 1.82385 2021-09-25 23:32:12,305 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 447/500: loss: 0.00003 learning rate: 0.00003 Time taken: 0:00:02.787400 ETA: 0:02:27.732226 INFO:tensorflow:global_step/sec: 1.78369 2021-09-25 23:32:12,864 [INFO] tensorflow: global_step/sec: 1.78369 INFO:tensorflow:global_step/sec: 1.80574 2021-09-25 23:32:13,418 [INFO] tensorflow: global_step/sec: 1.80574 INFO:tensorflow:global_step/sec: 1.7856 2021-09-25 23:32:13,978 [INFO] tensorflow: global_step/sec: 1.7856 INFO:tensorflow:global_step/sec: 1.7615 2021-09-25 23:32:14,546 [INFO] tensorflow: global_step/sec: 1.7615 INFO:tensorflow:epoch = 448.0, learning_rate = 2.467763e-05, loss = 3.3392986e-05, step = 2240 (5.589 sec) 2021-09-25 23:32:15,112 [INFO] tensorflow: epoch = 448.0, learning_rate = 2.467763e-05, loss = 3.3392986e-05, step = 2240 (5.589 sec) INFO:tensorflow:global_step/sec: 1.76205 2021-09-25 23:32:15,113 [INFO] tensorflow: global_step/sec: 1.76205 2021-09-25 23:32:15,115 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 448/500: loss: 0.00003 learning rate: 0.00002 Time taken: 0:00:02.790663 ETA: 0:02:25.114489 INFO:tensorflow:global_step/sec: 1.80725 2021-09-25 23:32:15,667 [INFO] tensorflow: global_step/sec: 1.80725 INFO:tensorflow:global_step/sec: 1.7539 2021-09-25 23:32:16,237 [INFO] tensorflow: global_step/sec: 1.7539 INFO:tensorflow:global_step/sec: 1.84842 2021-09-25 23:32:16,778 [INFO] tensorflow: global_step/sec: 1.84842 INFO:tensorflow:global_step/sec: 1.72916 2021-09-25 23:32:17,356 [INFO] tensorflow: global_step/sec: 1.72916 INFO:tensorflow:global_step/sec: 1.73895 2021-09-25 23:32:17,931 [INFO] tensorflow: global_step/sec: 1.73895 2021-09-25 23:32:17,933 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 449/500: loss: 0.00003 learning rate: 0.00002 Time taken: 0:00:02.808042 ETA: 0:02:23.210157 INFO:tensorflow:global_step/sec: 1.85622 2021-09-25 23:32:18,470 [INFO] tensorflow: global_step/sec: 1.85622 INFO:tensorflow:global_step/sec: 1.80508 2021-09-25 23:32:19,024 [INFO] tensorflow: global_step/sec: 1.80508 INFO:tensorflow:global_step/sec: 1.84258 2021-09-25 23:32:19,567 [INFO] tensorflow: global_step/sec: 1.84258 INFO:tensorflow:global_step/sec: 1.83107 2021-09-25 23:32:20,113 [INFO] tensorflow: global_step/sec: 1.83107 2021-09-25 23:32:20,117 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 114.589 INFO:tensorflow:epoch = 450.0, learning_rate = 2.320795e-05, loss = 3.220236e-05, step = 2250 (5.558 sec) 2021-09-25 23:32:20,670 [INFO] tensorflow: epoch = 450.0, learning_rate = 2.320795e-05, loss = 3.220236e-05, step = 2250 (5.558 sec) INFO:tensorflow:global_step/sec: 1.7907 2021-09-25 23:32:20,671 [INFO] tensorflow: global_step/sec: 1.7907 2021-09-25 23:32:20,673 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 450/500: loss: 0.00003 learning rate: 0.00002 Time taken: 0:00:02.755286 ETA: 0:02:17.764299 INFO:tensorflow:global_step/sec: 1.80438 2021-09-25 23:32:21,226 [INFO] tensorflow: global_step/sec: 1.80438 INFO:tensorflow:global_step/sec: 1.81601 2021-09-25 23:32:21,776 [INFO] tensorflow: global_step/sec: 1.81601 INFO:tensorflow:global_step/sec: 1.83578 2021-09-25 23:32:22,321 [INFO] tensorflow: global_step/sec: 1.83578 INFO:tensorflow:global_step/sec: 1.82763 2021-09-25 23:32:22,868 [INFO] tensorflow: global_step/sec: 1.82763 INFO:tensorflow:global_step/sec: 1.8232 2021-09-25 23:32:23,417 [INFO] tensorflow: global_step/sec: 1.8232 2021-09-25 23:32:23,418 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 451/500: loss: 0.00003 learning rate: 0.00002 Time taken: 0:00:02.757052 ETA: 0:02:15.095534 INFO:tensorflow:global_step/sec: 1.8173 2021-09-25 23:32:23,967 [INFO] tensorflow: global_step/sec: 1.8173 INFO:tensorflow:global_step/sec: 1.73184 2021-09-25 23:32:24,544 [INFO] tensorflow: global_step/sec: 1.73184 INFO:tensorflow:global_step/sec: 1.74633 2021-09-25 23:32:25,117 [INFO] tensorflow: global_step/sec: 1.74633 INFO:tensorflow:global_step/sec: 1.83656 2021-09-25 23:32:25,661 [INFO] tensorflow: global_step/sec: 1.83656 INFO:tensorflow:global_step/sec: 1.76608 2021-09-25 23:32:26,228 [INFO] tensorflow: global_step/sec: 1.76608 2021-09-25 23:32:26,229 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 452/500: loss: 0.00003 learning rate: 0.00002 Time taken: 0:00:02.791833 ETA: 0:02:14.007992 INFO:tensorflow:epoch = 452.20000000000005, learning_rate = 2.169218e-05, loss = 3.092233e-05, step = 2261 (6.134 sec) 2021-09-25 23:32:26,804 [INFO] tensorflow: epoch = 452.20000000000005, learning_rate = 2.169218e-05, loss = 3.092233e-05, step = 2261 (6.134 sec) INFO:tensorflow:global_step/sec: 1.72968 2021-09-25 23:32:26,806 [INFO] tensorflow: global_step/sec: 1.72968 INFO:tensorflow:global_step/sec: 1.85315 2021-09-25 23:32:27,345 [INFO] tensorflow: global_step/sec: 1.85315 INFO:tensorflow:global_step/sec: 1.78793 2021-09-25 23:32:27,905 [INFO] tensorflow: global_step/sec: 1.78793 INFO:tensorflow:global_step/sec: 1.78913 2021-09-25 23:32:28,464 [INFO] tensorflow: global_step/sec: 1.78913 INFO:tensorflow:global_step/sec: 1.75078 2021-09-25 23:32:29,035 [INFO] tensorflow: global_step/sec: 1.75078 2021-09-25 23:32:29,036 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 453/500: loss: 0.00003 learning rate: 0.00002 Time taken: 0:00:02.800908 ETA: 0:02:11.642680 INFO:tensorflow:global_step/sec: 1.73557 2021-09-25 23:32:29,611 [INFO] tensorflow: global_step/sec: 1.73557 INFO:tensorflow:global_step/sec: 1.75026 2021-09-25 23:32:30,182 [INFO] tensorflow: global_step/sec: 1.75026 INFO:tensorflow:global_step/sec: 1.76361 2021-09-25 23:32:30,749 [INFO] tensorflow: global_step/sec: 1.76361 INFO:tensorflow:global_step/sec: 1.82643 2021-09-25 23:32:31,297 [INFO] tensorflow: global_step/sec: 1.82643 INFO:tensorflow:global_step/sec: 1.8197 2021-09-25 23:32:31,846 [INFO] tensorflow: global_step/sec: 1.8197 2021-09-25 23:32:31,848 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 454/500: loss: 0.00003 learning rate: 0.00002 Time taken: 0:00:02.831491 ETA: 0:02:10.248597 INFO:tensorflow:epoch = 454.20000000000005, learning_rate = 2.0400297e-05, loss = 2.9040815e-05, step = 2271 (5.589 sec) 2021-09-25 23:32:32,394 [INFO] tensorflow: epoch = 454.20000000000005, learning_rate = 2.0400297e-05, loss = 2.9040815e-05, step = 2271 (5.589 sec) INFO:tensorflow:global_step/sec: 1.82226 2021-09-25 23:32:32,395 [INFO] tensorflow: global_step/sec: 1.82226 INFO:tensorflow:global_step/sec: 1.73869 2021-09-25 23:32:32,970 [INFO] tensorflow: global_step/sec: 1.73869 INFO:tensorflow:global_step/sec: 1.83255 2021-09-25 23:32:33,516 [INFO] tensorflow: global_step/sec: 1.83255 INFO:tensorflow:global_step/sec: 1.72466 2021-09-25 23:32:34,096 [INFO] tensorflow: global_step/sec: 1.72466 2021-09-25 23:32:34,100 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 114.438 INFO:tensorflow:global_step/sec: 1.80034 2021-09-25 23:32:34,651 [INFO] tensorflow: global_step/sec: 1.80034 2021-09-25 23:32:34,653 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 455/500: loss: 0.00003 learning rate: 0.00002 Time taken: 0:00:02.797137 ETA: 0:02:05.871187 INFO:tensorflow:global_step/sec: 1.73903 2021-09-25 23:32:35,226 [INFO] tensorflow: global_step/sec: 1.73903 INFO:tensorflow:global_step/sec: 1.82698 2021-09-25 23:32:35,774 [INFO] tensorflow: global_step/sec: 1.82698 INFO:tensorflow:global_step/sec: 1.82266 2021-09-25 23:32:36,322 [INFO] tensorflow: global_step/sec: 1.82266 INFO:tensorflow:global_step/sec: 1.69763 2021-09-25 23:32:36,911 [INFO] tensorflow: global_step/sec: 1.69763 INFO:tensorflow:global_step/sec: 1.75597 2021-09-25 23:32:37,481 [INFO] tensorflow: global_step/sec: 1.75597 2021-09-25 23:32:37,482 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 456/500: loss: 0.00003 learning rate: 0.00002 Time taken: 0:00:02.813496 ETA: 0:02:03.793808 INFO:tensorflow:epoch = 456.20000000000005, learning_rate = 1.9185372e-05, loss = 3.3191063e-05, step = 2281 (5.650 sec) 2021-09-25 23:32:38,044 [INFO] tensorflow: epoch = 456.20000000000005, learning_rate = 1.9185372e-05, loss = 3.3191063e-05, step = 2281 (5.650 sec) INFO:tensorflow:global_step/sec: 1.7722 2021-09-25 23:32:38,045 [INFO] tensorflow: global_step/sec: 1.7722 INFO:tensorflow:global_step/sec: 1.72687 2021-09-25 23:32:38,624 [INFO] tensorflow: global_step/sec: 1.72687 INFO:tensorflow:global_step/sec: 1.75904 2021-09-25 23:32:39,193 [INFO] tensorflow: global_step/sec: 1.75904 INFO:tensorflow:global_step/sec: 1.75997 2021-09-25 23:32:39,761 [INFO] tensorflow: global_step/sec: 1.75997 INFO:tensorflow:global_step/sec: 1.80458 2021-09-25 23:32:40,315 [INFO] tensorflow: global_step/sec: 1.80458 2021-09-25 23:32:40,317 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 457/500: loss: 0.00004 learning rate: 0.00002 Time taken: 0:00:02.851551 ETA: 0:02:02.616695 INFO:tensorflow:global_step/sec: 1.80299 2021-09-25 23:32:40,870 [INFO] tensorflow: global_step/sec: 1.80299 INFO:tensorflow:global_step/sec: 1.77527 2021-09-25 23:32:41,433 [INFO] tensorflow: global_step/sec: 1.77527 INFO:tensorflow:global_step/sec: 1.78317 2021-09-25 23:32:41,994 [INFO] tensorflow: global_step/sec: 1.78317 INFO:tensorflow:global_step/sec: 1.73076 2021-09-25 23:32:42,572 [INFO] tensorflow: global_step/sec: 1.73076 INFO:tensorflow:global_step/sec: 1.75221 2021-09-25 23:32:43,142 [INFO] tensorflow: global_step/sec: 1.75221 2021-09-25 23:32:43,144 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 458/500: loss: 0.00003 learning rate: 0.00002 Time taken: 0:00:02.807814 ETA: 0:01:57.928203 INFO:tensorflow:epoch = 458.20000000000005, learning_rate = 1.8042783e-05, loss = 3.15598e-05, step = 2291 (5.639 sec) 2021-09-25 23:32:43,683 [INFO] tensorflow: epoch = 458.20000000000005, learning_rate = 1.8042783e-05, loss = 3.15598e-05, step = 2291 (5.639 sec) INFO:tensorflow:global_step/sec: 1.84488 2021-09-25 23:32:43,684 [INFO] tensorflow: global_step/sec: 1.84488 INFO:tensorflow:global_step/sec: 1.78532 2021-09-25 23:32:44,244 [INFO] tensorflow: global_step/sec: 1.78532 INFO:tensorflow:global_step/sec: 1.70747 2021-09-25 23:32:44,830 [INFO] tensorflow: global_step/sec: 1.70747 INFO:tensorflow:global_step/sec: 1.76803 2021-09-25 23:32:45,396 [INFO] tensorflow: global_step/sec: 1.76803 INFO:tensorflow:global_step/sec: 1.80843 2021-09-25 23:32:45,949 [INFO] tensorflow: global_step/sec: 1.80843 2021-09-25 23:32:45,950 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 459/500: loss: 0.00003 learning rate: 0.00002 Time taken: 0:00:02.822858 ETA: 0:01:55.737162 INFO:tensorflow:global_step/sec: 1.77596 2021-09-25 23:32:46,512 [INFO] tensorflow: global_step/sec: 1.77596 INFO:tensorflow:global_step/sec: 1.85124 2021-09-25 23:32:47,052 [INFO] tensorflow: global_step/sec: 1.85124 INFO:tensorflow:global_step/sec: 1.73488 2021-09-25 23:32:47,628 [INFO] tensorflow: global_step/sec: 1.73488 INFO:tensorflow:global_step/sec: 1.82749 2021-09-25 23:32:48,176 [INFO] tensorflow: global_step/sec: 1.82749 INFO:tensorflow:Saving checkpoints for step-2300. 2021-09-25 23:32:48,177 [INFO] tensorflow: Saving checkpoints for step-2300. WARNING:tensorflow:Ignoring: /tmp/tmpmujdyk6s; No such file or directory 2021-09-25 23:32:48,395 [WARNING] tensorflow: Ignoring: /tmp/tmpmujdyk6s; No such file or directory 2021-09-25 23:32:52,300 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 87.916 INFO:tensorflow:epoch = 460.0, learning_rate = 1.707276e-05, loss = 3.0833224e-05, step = 2300 (9.216 sec) 2021-09-25 23:32:52,898 [INFO] tensorflow: epoch = 460.0, learning_rate = 1.707276e-05, loss = 3.0833224e-05, step = 2300 (9.216 sec) INFO:tensorflow:global_step/sec: 0.211676 2021-09-25 23:32:52,900 [INFO] tensorflow: global_step/sec: 0.211676 2021-09-25 23:32:52,901 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 460/500: loss: 0.00003 learning rate: 0.00002 Time taken: 0:00:06.900390 ETA: 0:04:36.015606 INFO:tensorflow:global_step/sec: 1.79379 2021-09-25 23:32:53,457 [INFO] tensorflow: global_step/sec: 1.79379 INFO:tensorflow:global_step/sec: 1.78172 2021-09-25 23:32:54,018 [INFO] tensorflow: global_step/sec: 1.78172 INFO:tensorflow:global_step/sec: 1.80157 2021-09-25 23:32:54,574 [INFO] tensorflow: global_step/sec: 1.80157 INFO:tensorflow:global_step/sec: 1.78573 2021-09-25 23:32:55,134 [INFO] tensorflow: global_step/sec: 1.78573 Epoch 461/500 ========================= Validation cost: 0.000000 Mean average_precision (in %): 0.0000 class name average precision (in %) ------------ -------------------------- lpd 0 Median Inference Time: nan INFO:tensorflow:global_step/sec: 1.78286 2021-09-25 23:32:55,694 [INFO] tensorflow: global_step/sec: 1.78286 2021-09-25 23:32:55,696 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 461/500: loss: 0.00003 learning rate: 0.00002 Time taken: 0:00:02.839993 ETA: 0:01:50.759746 INFO:tensorflow:global_step/sec: 1.7422 2021-09-25 23:32:56,268 [INFO] tensorflow: global_step/sec: 1.7422 INFO:tensorflow:global_step/sec: 1.84992 2021-09-25 23:32:56,809 [INFO] tensorflow: global_step/sec: 1.84992 INFO:tensorflow:global_step/sec: 1.84198 2021-09-25 23:32:57,352 [INFO] tensorflow: global_step/sec: 1.84198 INFO:tensorflow:global_step/sec: 1.76478 2021-09-25 23:32:57,919 [INFO] tensorflow: global_step/sec: 1.76478 INFO:tensorflow:epoch = 462.0, learning_rate = 1.6055987e-05, loss = 3.1555428e-05, step = 2310 (5.580 sec) 2021-09-25 23:32:58,478 [INFO] tensorflow: epoch = 462.0, learning_rate = 1.6055987e-05, loss = 3.1555428e-05, step = 2310 (5.580 sec) INFO:tensorflow:global_step/sec: 1.78316 2021-09-25 23:32:58,479 [INFO] tensorflow: global_step/sec: 1.78316 2021-09-25 23:32:58,481 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 462/500: loss: 0.00003 learning rate: 0.00002 Time taken: 0:00:02.780489 ETA: 0:01:45.658599 INFO:tensorflow:global_step/sec: 1.81748 2021-09-25 23:32:59,030 [INFO] tensorflow: global_step/sec: 1.81748 INFO:tensorflow:global_step/sec: 1.81957 2021-09-25 23:32:59,579 [INFO] tensorflow: global_step/sec: 1.81957 INFO:tensorflow:global_step/sec: 1.72418 2021-09-25 23:33:00,159 [INFO] tensorflow: global_step/sec: 1.72418 INFO:tensorflow:global_step/sec: 1.74172 2021-09-25 23:33:00,733 [INFO] tensorflow: global_step/sec: 1.74172 INFO:tensorflow:global_step/sec: 1.80229 2021-09-25 23:33:01,288 [INFO] tensorflow: global_step/sec: 1.80229 2021-09-25 23:33:01,290 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 463/500: loss: 0.00003 learning rate: 0.00002 Time taken: 0:00:02.816715 ETA: 0:01:44.218464 INFO:tensorflow:global_step/sec: 1.75648 2021-09-25 23:33:01,857 [INFO] tensorflow: global_step/sec: 1.75648 INFO:tensorflow:global_step/sec: 1.79539 2021-09-25 23:33:02,414 [INFO] tensorflow: global_step/sec: 1.79539 INFO:tensorflow:global_step/sec: 1.6876 2021-09-25 23:33:03,007 [INFO] tensorflow: global_step/sec: 1.6876 INFO:tensorflow:global_step/sec: 1.77836 2021-09-25 23:33:03,569 [INFO] tensorflow: global_step/sec: 1.77836 INFO:tensorflow:epoch = 464.0, learning_rate = 1.5099757e-05, loss = 2.7343654e-05, step = 2320 (5.650 sec) 2021-09-25 23:33:04,128 [INFO] tensorflow: epoch = 464.0, learning_rate = 1.5099757e-05, loss = 2.7343654e-05, step = 2320 (5.650 sec) INFO:tensorflow:global_step/sec: 1.78528 2021-09-25 23:33:04,129 [INFO] tensorflow: global_step/sec: 1.78528 2021-09-25 23:33:04,131 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 464/500: loss: 0.00003 learning rate: 0.00002 Time taken: 0:00:02.831926 ETA: 0:01:41.949323 INFO:tensorflow:global_step/sec: 1.75585 2021-09-25 23:33:04,699 [INFO] tensorflow: global_step/sec: 1.75585 INFO:tensorflow:global_step/sec: 1.84075 2021-09-25 23:33:05,242 [INFO] tensorflow: global_step/sec: 1.84075 INFO:tensorflow:global_step/sec: 1.82193 2021-09-25 23:33:05,791 [INFO] tensorflow: global_step/sec: 1.82193 INFO:tensorflow:global_step/sec: 1.81893 2021-09-25 23:33:06,341 [INFO] tensorflow: global_step/sec: 1.81893 2021-09-25 23:33:06,348 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 113.902 INFO:tensorflow:global_step/sec: 1.77062 2021-09-25 23:33:06,906 [INFO] tensorflow: global_step/sec: 1.77062 2021-09-25 23:33:06,907 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 465/500: loss: 0.00003 learning rate: 0.00001 Time taken: 0:00:02.774994 ETA: 0:01:37.124795 INFO:tensorflow:global_step/sec: 1.70506 2021-09-25 23:33:07,492 [INFO] tensorflow: global_step/sec: 1.70506 INFO:tensorflow:global_step/sec: 1.73438 2021-09-25 23:33:08,069 [INFO] tensorflow: global_step/sec: 1.73438 INFO:tensorflow:global_step/sec: 1.81765 2021-09-25 23:33:08,619 [INFO] tensorflow: global_step/sec: 1.81765 INFO:tensorflow:global_step/sec: 1.61981 2021-09-25 23:33:09,236 [INFO] tensorflow: global_step/sec: 1.61981 INFO:tensorflow:epoch = 466.0, learning_rate = 1.42004865e-05, loss = 2.8781942e-05, step = 2330 (5.674 sec) 2021-09-25 23:33:09,802 [INFO] tensorflow: epoch = 466.0, learning_rate = 1.42004865e-05, loss = 2.8781942e-05, step = 2330 (5.674 sec) INFO:tensorflow:global_step/sec: 1.76243 2021-09-25 23:33:09,804 [INFO] tensorflow: global_step/sec: 1.76243 2021-09-25 23:33:09,805 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 466/500: loss: 0.00003 learning rate: 0.00001 Time taken: 0:00:02.890973 ETA: 0:01:38.293085 INFO:tensorflow:global_step/sec: 1.78678 2021-09-25 23:33:10,363 [INFO] tensorflow: global_step/sec: 1.78678 INFO:tensorflow:global_step/sec: 1.71949 2021-09-25 23:33:10,945 [INFO] tensorflow: global_step/sec: 1.71949 INFO:tensorflow:global_step/sec: 1.80455 2021-09-25 23:33:11,499 [INFO] tensorflow: global_step/sec: 1.80455 INFO:tensorflow:global_step/sec: 1.75372 2021-09-25 23:33:12,069 [INFO] tensorflow: global_step/sec: 1.75372 INFO:tensorflow:global_step/sec: 1.7569 2021-09-25 23:33:12,638 [INFO] tensorflow: global_step/sec: 1.7569 2021-09-25 23:33:12,640 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 467/500: loss: 0.00003 learning rate: 0.00001 Time taken: 0:00:02.833332 ETA: 0:01:33.499966 INFO:tensorflow:global_step/sec: 1.70858 2021-09-25 23:33:13,224 [INFO] tensorflow: global_step/sec: 1.70858 INFO:tensorflow:global_step/sec: 1.82858 2021-09-25 23:33:13,771 [INFO] tensorflow: global_step/sec: 1.82858 INFO:tensorflow:global_step/sec: 1.72326 2021-09-25 23:33:14,351 [INFO] tensorflow: global_step/sec: 1.72326 INFO:tensorflow:global_step/sec: 1.73948 2021-09-25 23:33:14,926 [INFO] tensorflow: global_step/sec: 1.73948 INFO:tensorflow:epoch = 468.0, learning_rate = 1.3354775e-05, loss = 2.7730472e-05, step = 2340 (5.681 sec) 2021-09-25 23:33:15,483 [INFO] tensorflow: epoch = 468.0, learning_rate = 1.3354775e-05, loss = 2.7730472e-05, step = 2340 (5.681 sec) INFO:tensorflow:global_step/sec: 1.79011 2021-09-25 23:33:15,484 [INFO] tensorflow: global_step/sec: 1.79011 2021-09-25 23:33:15,486 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 468/500: loss: 0.00003 learning rate: 0.00001 Time taken: 0:00:02.855705 ETA: 0:01:31.382553 INFO:tensorflow:global_step/sec: 1.72755 2021-09-25 23:33:16,063 [INFO] tensorflow: global_step/sec: 1.72755 INFO:tensorflow:global_step/sec: 1.73712 2021-09-25 23:33:16,639 [INFO] tensorflow: global_step/sec: 1.73712 INFO:tensorflow:global_step/sec: 1.81919 2021-09-25 23:33:17,189 [INFO] tensorflow: global_step/sec: 1.81919 INFO:tensorflow:global_step/sec: 1.7843 2021-09-25 23:33:17,749 [INFO] tensorflow: global_step/sec: 1.7843 INFO:tensorflow:global_step/sec: 1.72072 2021-09-25 23:33:18,330 [INFO] tensorflow: global_step/sec: 1.72072 2021-09-25 23:33:18,332 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 469/500: loss: 0.00003 learning rate: 0.00001 Time taken: 0:00:02.823141 ETA: 0:01:27.517374 INFO:tensorflow:global_step/sec: 1.70874 2021-09-25 23:33:18,915 [INFO] tensorflow: global_step/sec: 1.70874 INFO:tensorflow:global_step/sec: 1.75223 2021-09-25 23:33:19,486 [INFO] tensorflow: global_step/sec: 1.75223 INFO:tensorflow:global_step/sec: 1.81563 2021-09-25 23:33:20,037 [INFO] tensorflow: global_step/sec: 1.81563 INFO:tensorflow:global_step/sec: 1.81584 2021-09-25 23:33:20,588 [INFO] tensorflow: global_step/sec: 1.81584 2021-09-25 23:33:20,591 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 112.345 INFO:tensorflow:epoch = 470.0, learning_rate = 1.2559428e-05, loss = 2.9367922e-05, step = 2350 (5.678 sec) 2021-09-25 23:33:21,160 [INFO] tensorflow: epoch = 470.0, learning_rate = 1.2559428e-05, loss = 2.9367922e-05, step = 2350 (5.678 sec) INFO:tensorflow:global_step/sec: 1.74121 2021-09-25 23:33:21,162 [INFO] tensorflow: global_step/sec: 1.74121 2021-09-25 23:33:21,163 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 470/500: loss: 0.00003 learning rate: 0.00001 Time taken: 0:00:02.835820 ETA: 0:01:25.074592 INFO:tensorflow:global_step/sec: 1.79914 2021-09-25 23:33:21,718 [INFO] tensorflow: global_step/sec: 1.79914 INFO:tensorflow:global_step/sec: 1.74192 2021-09-25 23:33:22,292 [INFO] tensorflow: global_step/sec: 1.74192 INFO:tensorflow:global_step/sec: 1.83337 2021-09-25 23:33:22,837 [INFO] tensorflow: global_step/sec: 1.83337 INFO:tensorflow:global_step/sec: 1.81923 2021-09-25 23:33:23,387 [INFO] tensorflow: global_step/sec: 1.81923 INFO:tensorflow:global_step/sec: 1.6426 2021-09-25 23:33:23,996 [INFO] tensorflow: global_step/sec: 1.6426 2021-09-25 23:33:23,997 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 471/500: loss: 0.00003 learning rate: 0.00001 Time taken: 0:00:02.799870 ETA: 0:01:21.196223 INFO:tensorflow:global_step/sec: 1.80616 2021-09-25 23:33:24,549 [INFO] tensorflow: global_step/sec: 1.80616 INFO:tensorflow:global_step/sec: 1.83019 2021-09-25 23:33:25,096 [INFO] tensorflow: global_step/sec: 1.83019 INFO:tensorflow:global_step/sec: 1.76864 2021-09-25 23:33:25,661 [INFO] tensorflow: global_step/sec: 1.76864 INFO:tensorflow:global_step/sec: 1.77717 2021-09-25 23:33:26,224 [INFO] tensorflow: global_step/sec: 1.77717 INFO:tensorflow:epoch = 472.0, learning_rate = 1.18114485e-05, loss = 3.0081674e-05, step = 2360 (5.635 sec) 2021-09-25 23:33:26,796 [INFO] tensorflow: epoch = 472.0, learning_rate = 1.18114485e-05, loss = 3.0081674e-05, step = 2360 (5.635 sec) INFO:tensorflow:global_step/sec: 1.74383 2021-09-25 23:33:26,797 [INFO] tensorflow: global_step/sec: 1.74383 2021-09-25 23:33:26,799 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 472/500: loss: 0.00003 learning rate: 0.00001 Time taken: 0:00:02.838575 ETA: 0:01:19.480103 INFO:tensorflow:global_step/sec: 1.81746 2021-09-25 23:33:27,347 [INFO] tensorflow: global_step/sec: 1.81746 INFO:tensorflow:global_step/sec: 1.82585 2021-09-25 23:33:27,895 [INFO] tensorflow: global_step/sec: 1.82585 INFO:tensorflow:global_step/sec: 1.7903 2021-09-25 23:33:28,454 [INFO] tensorflow: global_step/sec: 1.7903 INFO:tensorflow:global_step/sec: 1.72568 2021-09-25 23:33:29,033 [INFO] tensorflow: global_step/sec: 1.72568 INFO:tensorflow:global_step/sec: 1.72239 2021-09-25 23:33:29,614 [INFO] tensorflow: global_step/sec: 1.72239 2021-09-25 23:33:29,615 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 473/500: loss: 0.00004 learning rate: 0.00001 Time taken: 0:00:02.804809 ETA: 0:01:15.729839 INFO:tensorflow:global_step/sec: 1.75338 2021-09-25 23:33:30,184 [INFO] tensorflow: global_step/sec: 1.75338 INFO:tensorflow:global_step/sec: 1.79486 2021-09-25 23:33:30,741 [INFO] tensorflow: global_step/sec: 1.79486 INFO:tensorflow:global_step/sec: 1.76431 2021-09-25 23:33:31,308 [INFO] tensorflow: global_step/sec: 1.76431 INFO:tensorflow:global_step/sec: 1.76009 2021-09-25 23:33:31,876 [INFO] tensorflow: global_step/sec: 1.76009 INFO:tensorflow:epoch = 474.0, learning_rate = 1.1108026e-05, loss = 3.0103605e-05, step = 2370 (5.669 sec) 2021-09-25 23:33:32,464 [INFO] tensorflow: epoch = 474.0, learning_rate = 1.1108026e-05, loss = 3.0103605e-05, step = 2370 (5.669 sec) INFO:tensorflow:global_step/sec: 1.69607 2021-09-25 23:33:32,466 [INFO] tensorflow: global_step/sec: 1.69607 2021-09-25 23:33:32,467 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 474/500: loss: 0.00003 learning rate: 0.00001 Time taken: 0:00:02.842318 ETA: 0:01:13.900270 INFO:tensorflow:global_step/sec: 1.71077 2021-09-25 23:33:33,050 [INFO] tensorflow: global_step/sec: 1.71077 INFO:tensorflow:global_step/sec: 1.7874 2021-09-25 23:33:33,610 [INFO] tensorflow: global_step/sec: 1.7874 INFO:tensorflow:global_step/sec: 1.75176 2021-09-25 23:33:34,181 [INFO] tensorflow: global_step/sec: 1.75176 INFO:tensorflow:global_step/sec: 1.74034 2021-09-25 23:33:34,755 [INFO] tensorflow: global_step/sec: 1.74034 2021-09-25 23:33:34,759 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 112.944 INFO:tensorflow:global_step/sec: 1.80723 2021-09-25 23:33:35,309 [INFO] tensorflow: global_step/sec: 1.80723 2021-09-25 23:33:35,310 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 475/500: loss: 0.00003 learning rate: 0.00001 Time taken: 0:00:02.877898 ETA: 0:01:11.947461 INFO:tensorflow:global_step/sec: 1.82478 2021-09-25 23:33:35,857 [INFO] tensorflow: global_step/sec: 1.82478 INFO:tensorflow:global_step/sec: 1.7912 2021-09-25 23:33:36,415 [INFO] tensorflow: global_step/sec: 1.7912 INFO:tensorflow:global_step/sec: 1.77289 2021-09-25 23:33:36,979 [INFO] tensorflow: global_step/sec: 1.77289 INFO:tensorflow:global_step/sec: 1.78838 2021-09-25 23:33:37,538 [INFO] tensorflow: global_step/sec: 1.78838 INFO:tensorflow:epoch = 476.0, learning_rate = 1.0446485e-05, loss = 3.0119369e-05, step = 2380 (5.624 sec) 2021-09-25 23:33:38,088 [INFO] tensorflow: epoch = 476.0, learning_rate = 1.0446485e-05, loss = 3.0119369e-05, step = 2380 (5.624 sec) INFO:tensorflow:global_step/sec: 1.81332 2021-09-25 23:33:38,090 [INFO] tensorflow: global_step/sec: 1.81332 2021-09-25 23:33:38,091 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 476/500: loss: 0.00003 learning rate: 0.00001 Time taken: 0:00:02.785070 ETA: 0:01:06.841679 INFO:tensorflow:global_step/sec: 1.80921 2021-09-25 23:33:38,642 [INFO] tensorflow: global_step/sec: 1.80921 INFO:tensorflow:global_step/sec: 1.84963 2021-09-25 23:33:39,183 [INFO] tensorflow: global_step/sec: 1.84963 INFO:tensorflow:global_step/sec: 1.71339 2021-09-25 23:33:39,767 [INFO] tensorflow: global_step/sec: 1.71339 INFO:tensorflow:global_step/sec: 1.76924 2021-09-25 23:33:40,332 [INFO] tensorflow: global_step/sec: 1.76924 INFO:tensorflow:global_step/sec: 1.7657 2021-09-25 23:33:40,898 [INFO] tensorflow: global_step/sec: 1.7657 2021-09-25 23:33:40,900 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 477/500: loss: 0.00003 learning rate: 0.00001 Time taken: 0:00:02.792388 ETA: 0:01:04.224923 INFO:tensorflow:global_step/sec: 1.79022 2021-09-25 23:33:41,457 [INFO] tensorflow: global_step/sec: 1.79022 INFO:tensorflow:global_step/sec: 1.81511 2021-09-25 23:33:42,008 [INFO] tensorflow: global_step/sec: 1.81511 INFO:tensorflow:global_step/sec: 1.73807 2021-09-25 23:33:42,583 [INFO] tensorflow: global_step/sec: 1.73807 INFO:tensorflow:global_step/sec: 1.75393 2021-09-25 23:33:43,153 [INFO] tensorflow: global_step/sec: 1.75393 INFO:tensorflow:epoch = 478.0, learning_rate = 9.824344e-06, loss = 3.3936623e-05, step = 2390 (5.616 sec) 2021-09-25 23:33:43,704 [INFO] tensorflow: epoch = 478.0, learning_rate = 9.824344e-06, loss = 3.3936623e-05, step = 2390 (5.616 sec) INFO:tensorflow:global_step/sec: 1.81137 2021-09-25 23:33:43,705 [INFO] tensorflow: global_step/sec: 1.81137 2021-09-25 23:33:43,707 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 478/500: loss: 0.00003 learning rate: 0.00001 Time taken: 0:00:02.817608 ETA: 0:01:01.987368 INFO:tensorflow:global_step/sec: 1.75603 2021-09-25 23:33:44,275 [INFO] tensorflow: global_step/sec: 1.75603 INFO:tensorflow:global_step/sec: 1.74571 2021-09-25 23:33:44,848 [INFO] tensorflow: global_step/sec: 1.74571 INFO:tensorflow:global_step/sec: 1.85723 2021-09-25 23:33:45,386 [INFO] tensorflow: global_step/sec: 1.85723 INFO:tensorflow:global_step/sec: 1.7642 2021-09-25 23:33:45,953 [INFO] tensorflow: global_step/sec: 1.7642 INFO:tensorflow:global_step/sec: 1.79032 2021-09-25 23:33:46,511 [INFO] tensorflow: global_step/sec: 1.79032 2021-09-25 23:33:46,513 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 479/500: loss: 0.00004 learning rate: 0.00001 Time taken: 0:00:02.802204 ETA: 0:00:58.846287 INFO:tensorflow:global_step/sec: 1.81746 2021-09-25 23:33:47,062 [INFO] tensorflow: global_step/sec: 1.81746 INFO:tensorflow:global_step/sec: 1.79311 2021-09-25 23:33:47,619 [INFO] tensorflow: global_step/sec: 1.79311 INFO:tensorflow:global_step/sec: 1.81527 2021-09-25 23:33:48,170 [INFO] tensorflow: global_step/sec: 1.81527 INFO:tensorflow:global_step/sec: 1.706 2021-09-25 23:33:48,756 [INFO] tensorflow: global_step/sec: 1.706 INFO:tensorflow:Saving checkpoints for step-2400. 2021-09-25 23:33:48,757 [INFO] tensorflow: Saving checkpoints for step-2400. WARNING:tensorflow:Ignoring: /tmp/tmpikf_oj40; No such file or directory 2021-09-25 23:33:48,977 [WARNING] tensorflow: Ignoring: /tmp/tmpikf_oj40; No such file or directory 2021-09-25 23:33:52,814 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 88.619 INFO:tensorflow:epoch = 480.0, learning_rate = 9.239252e-06, loss = 4.059716e-05, step = 2400 (9.682 sec) 2021-09-25 23:33:53,385 [INFO] tensorflow: epoch = 480.0, learning_rate = 9.239252e-06, loss = 4.059716e-05, step = 2400 (9.682 sec) INFO:tensorflow:global_step/sec: 0.215966 2021-09-25 23:33:53,387 [INFO] tensorflow: global_step/sec: 0.215966 2021-09-25 23:33:53,388 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 480/500: loss: 0.00004 learning rate: 0.00001 Time taken: 0:00:06.854614 ETA: 0:02:17.092290 INFO:tensorflow:global_step/sec: 1.74214 2021-09-25 23:33:53,961 [INFO] tensorflow: global_step/sec: 1.74214 INFO:tensorflow:global_step/sec: 1.78817 2021-09-25 23:33:54,520 [INFO] tensorflow: global_step/sec: 1.78817 INFO:tensorflow:global_step/sec: 1.78296 2021-09-25 23:33:55,081 [INFO] tensorflow: global_step/sec: 1.78296 INFO:tensorflow:global_step/sec: 1.85501 2021-09-25 23:33:55,620 [INFO] tensorflow: global_step/sec: 1.85501 Epoch 481/500 ========================= Validation cost: 0.000000 Mean average_precision (in %): 0.0000 class name average precision (in %) ------------ -------------------------- lpd 0 Median Inference Time: nan INFO:tensorflow:global_step/sec: 1.77777 2021-09-25 23:33:56,182 [INFO] tensorflow: global_step/sec: 1.77777 2021-09-25 23:33:56,184 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 481/500: loss: 0.00003 learning rate: 0.00001 Time taken: 0:00:02.809333 ETA: 0:00:53.377329 INFO:tensorflow:global_step/sec: 1.78656 2021-09-25 23:33:56,742 [INFO] tensorflow: global_step/sec: 1.78656 INFO:tensorflow:global_step/sec: 1.80654 2021-09-25 23:33:57,296 [INFO] tensorflow: global_step/sec: 1.80654 INFO:tensorflow:global_step/sec: 1.77188 2021-09-25 23:33:57,860 [INFO] tensorflow: global_step/sec: 1.77188 INFO:tensorflow:global_step/sec: 1.83923 2021-09-25 23:33:58,404 [INFO] tensorflow: global_step/sec: 1.83923 INFO:tensorflow:epoch = 482.0, learning_rate = 8.689007e-06, loss = 3.1041203e-05, step = 2410 (5.574 sec) 2021-09-25 23:33:58,959 [INFO] tensorflow: epoch = 482.0, learning_rate = 8.689007e-06, loss = 3.1041203e-05, step = 2410 (5.574 sec) INFO:tensorflow:global_step/sec: 1.79541 2021-09-25 23:33:58,961 [INFO] tensorflow: global_step/sec: 1.79541 2021-09-25 23:33:58,962 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 482/500: loss: 0.00003 learning rate: 0.00001 Time taken: 0:00:02.779314 ETA: 0:00:50.027653 INFO:tensorflow:global_step/sec: 1.80797 2021-09-25 23:33:59,514 [INFO] tensorflow: global_step/sec: 1.80797 INFO:tensorflow:global_step/sec: 1.80152 2021-09-25 23:34:00,069 [INFO] tensorflow: global_step/sec: 1.80152 INFO:tensorflow:global_step/sec: 1.77606 2021-09-25 23:34:00,632 [INFO] tensorflow: global_step/sec: 1.77606 INFO:tensorflow:global_step/sec: 1.68617 2021-09-25 23:34:01,225 [INFO] tensorflow: global_step/sec: 1.68617 INFO:tensorflow:global_step/sec: 1.86398 2021-09-25 23:34:01,762 [INFO] tensorflow: global_step/sec: 1.86398 2021-09-25 23:34:01,763 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 483/500: loss: 0.00003 learning rate: 0.00001 Time taken: 0:00:02.819407 ETA: 0:00:47.929915 INFO:tensorflow:global_step/sec: 1.79173 2021-09-25 23:34:02,320 [INFO] tensorflow: global_step/sec: 1.79173 INFO:tensorflow:global_step/sec: 1.8017 2021-09-25 23:34:02,875 [INFO] tensorflow: global_step/sec: 1.8017 INFO:tensorflow:global_step/sec: 1.81714 2021-09-25 23:34:03,425 [INFO] tensorflow: global_step/sec: 1.81714 INFO:tensorflow:global_step/sec: 1.71992 2021-09-25 23:34:04,006 [INFO] tensorflow: global_step/sec: 1.71992 INFO:tensorflow:epoch = 484.0, learning_rate = 8.171532e-06, loss = 3.203607e-05, step = 2420 (5.596 sec) 2021-09-25 23:34:04,556 [INFO] tensorflow: epoch = 484.0, learning_rate = 8.171532e-06, loss = 3.203607e-05, step = 2420 (5.596 sec) INFO:tensorflow:global_step/sec: 1.81658 2021-09-25 23:34:04,557 [INFO] tensorflow: global_step/sec: 1.81658 2021-09-25 23:34:04,558 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 484/500: loss: 0.00003 learning rate: 0.00001 Time taken: 0:00:02.780647 ETA: 0:00:44.490356 INFO:tensorflow:global_step/sec: 1.7265 2021-09-25 23:34:05,136 [INFO] tensorflow: global_step/sec: 1.7265 INFO:tensorflow:global_step/sec: 1.75967 2021-09-25 23:34:05,704 [INFO] tensorflow: global_step/sec: 1.75967 INFO:tensorflow:global_step/sec: 1.81681 2021-09-25 23:34:06,255 [INFO] tensorflow: global_step/sec: 1.81681 INFO:tensorflow:global_step/sec: 1.72963 2021-09-25 23:34:06,833 [INFO] tensorflow: global_step/sec: 1.72963 2021-09-25 23:34:06,841 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 114.106 INFO:tensorflow:global_step/sec: 1.72292 2021-09-25 23:34:07,413 [INFO] tensorflow: global_step/sec: 1.72292 2021-09-25 23:34:07,415 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 485/500: loss: 0.00003 learning rate: 0.00001 Time taken: 0:00:02.830271 ETA: 0:00:42.454072 INFO:tensorflow:global_step/sec: 1.8083 2021-09-25 23:34:07,966 [INFO] tensorflow: global_step/sec: 1.8083 INFO:tensorflow:global_step/sec: 1.74095 2021-09-25 23:34:08,541 [INFO] tensorflow: global_step/sec: 1.74095 INFO:tensorflow:global_step/sec: 1.81784 2021-09-25 23:34:09,091 [INFO] tensorflow: global_step/sec: 1.81784 INFO:tensorflow:global_step/sec: 1.81654 2021-09-25 23:34:09,641 [INFO] tensorflow: global_step/sec: 1.81654 INFO:tensorflow:epoch = 486.0, learning_rate = 7.684875e-06, loss = 3.047175e-05, step = 2430 (5.659 sec) 2021-09-25 23:34:10,214 [INFO] tensorflow: epoch = 486.0, learning_rate = 7.684875e-06, loss = 3.047175e-05, step = 2430 (5.659 sec) INFO:tensorflow:global_step/sec: 1.74165 2021-09-25 23:34:10,216 [INFO] tensorflow: global_step/sec: 1.74165 2021-09-25 23:34:10,217 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 486/500: loss: 0.00003 learning rate: 0.00001 Time taken: 0:00:02.804403 ETA: 0:00:39.261643 INFO:tensorflow:global_step/sec: 1.8304 2021-09-25 23:34:10,762 [INFO] tensorflow: global_step/sec: 1.8304 INFO:tensorflow:global_step/sec: 1.80034 2021-09-25 23:34:11,317 [INFO] tensorflow: global_step/sec: 1.80034 INFO:tensorflow:global_step/sec: 1.77636 2021-09-25 23:34:11,880 [INFO] tensorflow: global_step/sec: 1.77636 INFO:tensorflow:global_step/sec: 1.81634 2021-09-25 23:34:12,431 [INFO] tensorflow: global_step/sec: 1.81634 INFO:tensorflow:global_step/sec: 1.73654 2021-09-25 23:34:13,007 [INFO] tensorflow: global_step/sec: 1.73654 2021-09-25 23:34:13,008 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 487/500: loss: 0.00003 learning rate: 0.00001 Time taken: 0:00:02.791057 ETA: 0:00:36.283736 INFO:tensorflow:global_step/sec: 1.79348 2021-09-25 23:34:13,564 [INFO] tensorflow: global_step/sec: 1.79348 INFO:tensorflow:global_step/sec: 1.72197 2021-09-25 23:34:14,145 [INFO] tensorflow: global_step/sec: 1.72197 INFO:tensorflow:global_step/sec: 1.71925 2021-09-25 23:34:14,727 [INFO] tensorflow: global_step/sec: 1.71925 INFO:tensorflow:global_step/sec: 1.78339 2021-09-25 23:34:15,287 [INFO] tensorflow: global_step/sec: 1.78339 INFO:tensorflow:epoch = 488.0, learning_rate = 7.2271996e-06, loss = 3.259816e-05, step = 2440 (5.670 sec) 2021-09-25 23:34:15,885 [INFO] tensorflow: epoch = 488.0, learning_rate = 7.2271996e-06, loss = 3.259816e-05, step = 2440 (5.670 sec) INFO:tensorflow:global_step/sec: 1.67076 2021-09-25 23:34:15,886 [INFO] tensorflow: global_step/sec: 1.67076 2021-09-25 23:34:15,887 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 488/500: loss: 0.00003 learning rate: 0.00001 Time taken: 0:00:02.855269 ETA: 0:00:34.263233 INFO:tensorflow:global_step/sec: 1.70234 2021-09-25 23:34:16,473 [INFO] tensorflow: global_step/sec: 1.70234 INFO:tensorflow:global_step/sec: 1.80515 2021-09-25 23:34:17,027 [INFO] tensorflow: global_step/sec: 1.80515 INFO:tensorflow:global_step/sec: 1.70306 2021-09-25 23:34:17,614 [INFO] tensorflow: global_step/sec: 1.70306 INFO:tensorflow:global_step/sec: 1.7761 2021-09-25 23:34:18,178 [INFO] tensorflow: global_step/sec: 1.7761 INFO:tensorflow:global_step/sec: 1.80062 2021-09-25 23:34:18,733 [INFO] tensorflow: global_step/sec: 1.80062 2021-09-25 23:34:18,734 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 489/500: loss: 0.00003 learning rate: 0.00001 Time taken: 0:00:02.890357 ETA: 0:00:31.793930 INFO:tensorflow:global_step/sec: 1.80602 2021-09-25 23:34:19,287 [INFO] tensorflow: global_step/sec: 1.80602 INFO:tensorflow:global_step/sec: 1.80829 2021-09-25 23:34:19,840 [INFO] tensorflow: global_step/sec: 1.80829 INFO:tensorflow:global_step/sec: 1.7426 2021-09-25 23:34:20,413 [INFO] tensorflow: global_step/sec: 1.7426 INFO:tensorflow:global_step/sec: 1.77316 2021-09-25 23:34:20,977 [INFO] tensorflow: global_step/sec: 1.77316 2021-09-25 23:34:20,985 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 113.128 INFO:tensorflow:epoch = 490.0, learning_rate = 6.796783e-06, loss = 3.09117e-05, step = 2450 (5.644 sec) 2021-09-25 23:34:21,529 [INFO] tensorflow: epoch = 490.0, learning_rate = 6.796783e-06, loss = 3.09117e-05, step = 2450 (5.644 sec) INFO:tensorflow:global_step/sec: 1.8089 2021-09-25 23:34:21,530 [INFO] tensorflow: global_step/sec: 1.8089 2021-09-25 23:34:21,532 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 490/500: loss: 0.00003 learning rate: 0.00001 Time taken: 0:00:02.799331 ETA: 0:00:27.993312 INFO:tensorflow:global_step/sec: 1.78559 2021-09-25 23:34:22,090 [INFO] tensorflow: global_step/sec: 1.78559 INFO:tensorflow:global_step/sec: 1.71649 2021-09-25 23:34:22,673 [INFO] tensorflow: global_step/sec: 1.71649 INFO:tensorflow:global_step/sec: 1.83046 2021-09-25 23:34:23,219 [INFO] tensorflow: global_step/sec: 1.83046 INFO:tensorflow:global_step/sec: 1.80728 2021-09-25 23:34:23,772 [INFO] tensorflow: global_step/sec: 1.80728 INFO:tensorflow:global_step/sec: 1.72863 2021-09-25 23:34:24,351 [INFO] tensorflow: global_step/sec: 1.72863 2021-09-25 23:34:24,352 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 491/500: loss: 0.00004 learning rate: 0.00001 Time taken: 0:00:02.789456 ETA: 0:00:25.105101 INFO:tensorflow:global_step/sec: 1.79711 2021-09-25 23:34:24,907 [INFO] tensorflow: global_step/sec: 1.79711 INFO:tensorflow:global_step/sec: 1.7171 2021-09-25 23:34:25,490 [INFO] tensorflow: global_step/sec: 1.7171 INFO:tensorflow:global_step/sec: 1.77792 2021-09-25 23:34:26,052 [INFO] tensorflow: global_step/sec: 1.77792 INFO:tensorflow:global_step/sec: 1.77101 2021-09-25 23:34:26,617 [INFO] tensorflow: global_step/sec: 1.77101 INFO:tensorflow:epoch = 492.0, learning_rate = 6.391999e-06, loss = 2.76546e-05, step = 2460 (5.656 sec) 2021-09-25 23:34:27,185 [INFO] tensorflow: epoch = 492.0, learning_rate = 6.391999e-06, loss = 2.76546e-05, step = 2460 (5.656 sec) INFO:tensorflow:global_step/sec: 1.75694 2021-09-25 23:34:27,186 [INFO] tensorflow: global_step/sec: 1.75694 2021-09-25 23:34:27,187 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 492/500: loss: 0.00003 learning rate: 0.00001 Time taken: 0:00:02.843832 ETA: 0:00:22.750652 INFO:tensorflow:global_step/sec: 1.78648 2021-09-25 23:34:27,746 [INFO] tensorflow: global_step/sec: 1.78648 INFO:tensorflow:global_step/sec: 1.82114 2021-09-25 23:34:28,295 [INFO] tensorflow: global_step/sec: 1.82114 INFO:tensorflow:global_step/sec: 1.69906 2021-09-25 23:34:28,884 [INFO] tensorflow: global_step/sec: 1.69906 INFO:tensorflow:global_step/sec: 1.75753 2021-09-25 23:34:29,452 [INFO] tensorflow: global_step/sec: 1.75753 INFO:tensorflow:global_step/sec: 1.72883 2021-09-25 23:34:30,031 [INFO] tensorflow: global_step/sec: 1.72883 2021-09-25 23:34:30,033 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 493/500: loss: 0.00003 learning rate: 0.00001 Time taken: 0:00:02.835054 ETA: 0:00:19.845379 INFO:tensorflow:global_step/sec: 1.79332 2021-09-25 23:34:30,589 [INFO] tensorflow: global_step/sec: 1.79332 INFO:tensorflow:global_step/sec: 1.78878 2021-09-25 23:34:31,148 [INFO] tensorflow: global_step/sec: 1.78878 INFO:tensorflow:global_step/sec: 1.81338 2021-09-25 23:34:31,699 [INFO] tensorflow: global_step/sec: 1.81338 INFO:tensorflow:global_step/sec: 1.79617 2021-09-25 23:34:32,256 [INFO] tensorflow: global_step/sec: 1.79617 INFO:tensorflow:epoch = 494.0, learning_rate = 6.0113225e-06, loss = 2.9471146e-05, step = 2470 (5.656 sec) 2021-09-25 23:34:32,841 [INFO] tensorflow: epoch = 494.0, learning_rate = 6.0113225e-06, loss = 2.9471146e-05, step = 2470 (5.656 sec) INFO:tensorflow:global_step/sec: 1.70468 2021-09-25 23:34:32,842 [INFO] tensorflow: global_step/sec: 1.70468 2021-09-25 23:34:32,844 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 494/500: loss: 0.00003 learning rate: 0.00001 Time taken: 0:00:02.802184 ETA: 0:00:16.813103 INFO:tensorflow:global_step/sec: 1.82269 2021-09-25 23:34:33,391 [INFO] tensorflow: global_step/sec: 1.82269 INFO:tensorflow:global_step/sec: 1.69994 2021-09-25 23:34:33,979 [INFO] tensorflow: global_step/sec: 1.69994 INFO:tensorflow:global_step/sec: 1.80772 2021-09-25 23:34:34,532 [INFO] tensorflow: global_step/sec: 1.80772 INFO:tensorflow:global_step/sec: 1.69442 2021-09-25 23:34:35,123 [INFO] tensorflow: global_step/sec: 1.69442 2021-09-25 23:34:35,126 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 113.153 INFO:tensorflow:global_step/sec: 1.78256 2021-09-25 23:34:35,684 [INFO] tensorflow: global_step/sec: 1.78256 2021-09-25 23:34:35,685 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 495/500: loss: 0.00003 learning rate: 0.00001 Time taken: 0:00:02.866799 ETA: 0:00:14.333997 INFO:tensorflow:global_step/sec: 1.80573 2021-09-25 23:34:36,237 [INFO] tensorflow: global_step/sec: 1.80573 INFO:tensorflow:global_step/sec: 1.73658 2021-09-25 23:34:36,813 [INFO] tensorflow: global_step/sec: 1.73658 INFO:tensorflow:global_step/sec: 1.84342 2021-09-25 23:34:37,356 [INFO] tensorflow: global_step/sec: 1.84342 INFO:tensorflow:global_step/sec: 1.70192 2021-09-25 23:34:37,943 [INFO] tensorflow: global_step/sec: 1.70192 INFO:tensorflow:epoch = 496.0, learning_rate = 5.6533167e-06, loss = 3.0675183e-05, step = 2480 (5.664 sec) 2021-09-25 23:34:38,505 [INFO] tensorflow: epoch = 496.0, learning_rate = 5.6533167e-06, loss = 3.0675183e-05, step = 2480 (5.664 sec) INFO:tensorflow:global_step/sec: 1.77481 2021-09-25 23:34:38,507 [INFO] tensorflow: global_step/sec: 1.77481 2021-09-25 23:34:38,508 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 496/500: loss: 0.00003 learning rate: 0.00001 Time taken: 0:00:02.819109 ETA: 0:00:11.276436 INFO:tensorflow:global_step/sec: 1.76231 2021-09-25 23:34:39,074 [INFO] tensorflow: global_step/sec: 1.76231 INFO:tensorflow:global_step/sec: 1.8615 2021-09-25 23:34:39,611 [INFO] tensorflow: global_step/sec: 1.8615 INFO:tensorflow:global_step/sec: 1.79572 2021-09-25 23:34:40,168 [INFO] tensorflow: global_step/sec: 1.79572 INFO:tensorflow:global_step/sec: 1.75962 2021-09-25 23:34:40,737 [INFO] tensorflow: global_step/sec: 1.75962 INFO:tensorflow:global_step/sec: 1.76449 2021-09-25 23:34:41,303 [INFO] tensorflow: global_step/sec: 1.76449 2021-09-25 23:34:41,305 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 497/500: loss: 0.00003 learning rate: 0.00001 Time taken: 0:00:02.796330 ETA: 0:00:08.388989 INFO:tensorflow:global_step/sec: 1.81616 2021-09-25 23:34:41,854 [INFO] tensorflow: global_step/sec: 1.81616 INFO:tensorflow:global_step/sec: 1.7222 2021-09-25 23:34:42,435 [INFO] tensorflow: global_step/sec: 1.7222 INFO:tensorflow:global_step/sec: 1.79983 2021-09-25 23:34:42,990 [INFO] tensorflow: global_step/sec: 1.79983 INFO:tensorflow:global_step/sec: 1.76686 2021-09-25 23:34:43,556 [INFO] tensorflow: global_step/sec: 1.76686 INFO:tensorflow:epoch = 498.0, learning_rate = 5.3166323e-06, loss = 3.074537e-05, step = 2490 (5.620 sec) 2021-09-25 23:34:44,126 [INFO] tensorflow: epoch = 498.0, learning_rate = 5.3166323e-06, loss = 3.074537e-05, step = 2490 (5.620 sec) INFO:tensorflow:global_step/sec: 1.75078 2021-09-25 23:34:44,127 [INFO] tensorflow: global_step/sec: 1.75078 2021-09-25 23:34:44,129 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 498/500: loss: 0.00003 learning rate: 0.00001 Time taken: 0:00:02.816127 ETA: 0:00:05.632253 INFO:tensorflow:global_step/sec: 1.78514 2021-09-25 23:34:44,688 [INFO] tensorflow: global_step/sec: 1.78514 INFO:tensorflow:global_step/sec: 1.80635 2021-09-25 23:34:45,241 [INFO] tensorflow: global_step/sec: 1.80635 INFO:tensorflow:global_step/sec: 1.82188 2021-09-25 23:34:45,790 [INFO] tensorflow: global_step/sec: 1.82188 INFO:tensorflow:global_step/sec: 1.76806 2021-09-25 23:34:46,356 [INFO] tensorflow: global_step/sec: 1.76806 INFO:tensorflow:global_step/sec: 1.77836 2021-09-25 23:34:46,918 [INFO] tensorflow: global_step/sec: 1.77836 2021-09-25 23:34:46,919 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 499/500: loss: 0.00003 learning rate: 0.00001 Time taken: 0:00:02.802248 ETA: 0:00:02.802248 INFO:tensorflow:global_step/sec: 1.84217 2021-09-25 23:34:47,461 [INFO] tensorflow: global_step/sec: 1.84217 INFO:tensorflow:global_step/sec: 1.78556 2021-09-25 23:34:48,021 [INFO] tensorflow: global_step/sec: 1.78556 INFO:tensorflow:global_step/sec: 1.73599 2021-09-25 23:34:48,597 [INFO] tensorflow: global_step/sec: 1.73599 INFO:tensorflow:global_step/sec: 1.85348 2021-09-25 23:34:49,136 [INFO] tensorflow: global_step/sec: 1.85348 INFO:tensorflow:Saving checkpoints for step-2500. 2021-09-25 23:34:49,137 [INFO] tensorflow: Saving checkpoints for step-2500. WARNING:tensorflow:Ignoring: /tmp/tmpu9rqq9lf; No such file or directory 2021-09-25 23:34:49,354 [WARNING] tensorflow: Ignoring: /tmp/tmpu9rqq9lf; No such file or directory Epoch 500/500 ========================= Validation cost: 0.000000 Mean average_precision (in %): 0.0000 class name average precision (in %) ------------ -------------------------- lpd 0 Median Inference Time: nan 2021-09-25 23:34:53,147 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 88.789 WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:95: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead. 2021-09-25 23:34:53,791 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:95: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:98: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead. 2021-09-25 23:34:53,792 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:98: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead. 2021-09-25 23:34:53,795 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 88.789 Time taken to run __main__:main: 0:31:23.597851. 2021-09-25 23:34:56,545 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.