023-02-20 14:13:17,756 [INFO] root: Registry: ['nvcr.io'] 2023-02-20 14:13:17,801 [INFO] tlt.components.instance_handler.local_instance: Running command in container: nvcr.io/nvidia/tao/tao-toolkit:4.0.0-tf1.15.5 Using TensorFlow backend. 2023-02-20 14:13:18.733316: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0 WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them. /usr/local/lib/python3.6/dist-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.26.5) or chardet (3.0.4) doesn't match a supported version! RequestsDependencyWarning) Matplotlib created a temporary config/cache directory at /tmp/matplotlib-zt7i62y7 because the default path (/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing. WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them. /usr/local/lib/python3.6/dist-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.26.5) or chardet (3.0.4) doesn't match a supported version! RequestsDependencyWarning) Using TensorFlow backend. [1676902406.217430] [9f2dc1f0fa08:245 :f] vfs_fuse.c:424 UCX WARN failed to connect to vfs socket '��������': Invalid argument 2023-02-20 14:13:26,750 [INFO] root: Starting DetectNet_v2 Training job 2023-02-20 14:13:26,750 [INFO] __main__: Loading experiment spec at /workspace/tao-experiments/detectnet_v2/specs/detectnet_v2_train_resnet18_kitti.txt. 2023-02-20 14:13:26,752 [INFO] iva.detectnet_v2.spec_handler.spec_loader: Merging specification from /workspace/tao-experiments/detectnet_v2/specs/detectnet_v2_train_resnet18_kitti.txt 2023-02-20 14:13:26,758 [INFO] iva.common.mlops.wandb: Initializing wandb. 2023-02-20 14:13:26,758 [WARNING] iva.common.mlops.wandb: Wandb logging failed with error WandB client wasn't logged in. Please make sure to set the WANDB_API_KEY env variable or run `wandb login` in over the CLI and copy the ~/.netrc file to the container. 2023-02-20 14:13:26,759 [INFO] __main__: Integrating with clearml. 2023-02-20 14:13:27,227 [WARNING] iva.common.mlops.clearml: ClearML task init failed with error ClearML configuration could not be found (missing `~/clearml.conf` or Environment CLEARML_API_HOST) To get started with ClearML: setup your own `clearml-server`, or create a free account at https://app.clear.ml 2023-02-20 14:13:27,227 [WARNING] iva.common.mlops.clearml: Training will still continue. 2023-02-20 14:13:27,228 [INFO] root: Training gridbox model. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:153: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead. 2023-02-20 14:13:27,228 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:153: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead. 2023-02-20 14:13:27,478 [INFO] root: Sampling mode of the dataloader was set to user_defined. 2023-02-20 14:13:27,478 [INFO] __main__: Cannot iterate over exactly 2846 samples with a batch size of 4; each epoch will therefore take one extra step. 2023-02-20 14:13:27,548 [INFO] root: Building DetectNet V2 model WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead. 2023-02-20 14:13:27,548 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead. 2023-02-20 14:13:27,549 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1834: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead. 2023-02-20 14:13:27,567 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1834: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/third_party/keras/tensorflow_backend.py:187: The name tf.nn.avg_pool is deprecated. Please use tf.nn.avg_pool2d instead. 2023-02-20 14:13:28,469 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/third_party/keras/tensorflow_backend.py:187: The name tf.nn.avg_pool is deprecated. Please use tf.nn.avg_pool2d instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead. 2023-02-20 14:13:28,667 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:190: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead. 2023-02-20 14:13:28,668 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:190: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead. 2023-02-20 14:13:28,668 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead. 2023-02-20 14:13:29,119 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead. 2023-02-20 14:13:36,562 [INFO] iva.detectnet_v2.objectives.bbox_objective: Default L1 loss function will be used. __________________________________________________________________________________________________ Layer (type) Output Shape Param # Connected to ================================================================================================== input_1 (InputLayer) (None, 3, 1088, 1920 0 __________________________________________________________________________________________________ conv1 (Conv2D) (None, 64, 544, 960) 9472 input_1[0][0] __________________________________________________________________________________________________ bn_conv1 (BatchNormalization) (None, 64, 544, 960) 256 conv1[0][0] __________________________________________________________________________________________________ activation_1 (Activation) (None, 64, 544, 960) 0 bn_conv1[0][0] __________________________________________________________________________________________________ block_1a_conv_1 (Conv2D) (None, 64, 272, 480) 36928 activation_1[0][0] __________________________________________________________________________________________________ block_1a_bn_1 (BatchNormalizati (None, 64, 272, 480) 256 block_1a_conv_1[0][0] __________________________________________________________________________________________________ block_1a_relu_1 (Activation) (None, 64, 272, 480) 0 block_1a_bn_1[0][0] __________________________________________________________________________________________________ block_1a_conv_2 (Conv2D) (None, 64, 272, 480) 36928 block_1a_relu_1[0][0] __________________________________________________________________________________________________ block_1a_conv_shortcut (Conv2D) (None, 64, 272, 480) 4160 activation_1[0][0] __________________________________________________________________________________________________ block_1a_bn_2 (BatchNormalizati (None, 64, 272, 480) 256 block_1a_conv_2[0][0] __________________________________________________________________________________________________ block_1a_bn_shortcut (BatchNorm (None, 64, 272, 480) 256 block_1a_conv_shortcut[0][0] __________________________________________________________________________________________________ add_1 (Add) (None, 64, 272, 480) 0 block_1a_bn_2[0][0] block_1a_bn_shortcut[0][0] __________________________________________________________________________________________________ block_1a_relu (Activation) (None, 64, 272, 480) 0 add_1[0][0] __________________________________________________________________________________________________ block_1b_conv_1 (Conv2D) (None, 64, 272, 480) 36928 block_1a_relu[0][0] __________________________________________________________________________________________________ block_1b_bn_1 (BatchNormalizati (None, 64, 272, 480) 256 block_1b_conv_1[0][0] __________________________________________________________________________________________________ block_1b_relu_1 (Activation) (None, 64, 272, 480) 0 block_1b_bn_1[0][0] __________________________________________________________________________________________________ block_1b_conv_2 (Conv2D) (None, 64, 272, 480) 36928 block_1b_relu_1[0][0] __________________________________________________________________________________________________ block_1b_bn_2 (BatchNormalizati (None, 64, 272, 480) 256 block_1b_conv_2[0][0] __________________________________________________________________________________________________ add_2 (Add) (None, 64, 272, 480) 0 block_1b_bn_2[0][0] block_1a_relu[0][0] __________________________________________________________________________________________________ block_1b_relu (Activation) (None, 64, 272, 480) 0 add_2[0][0] __________________________________________________________________________________________________ block_2a_conv_1 (Conv2D) (None, 128, 136, 240 73856 block_1b_relu[0][0] __________________________________________________________________________________________________ block_2a_bn_1 (BatchNormalizati (None, 128, 136, 240 512 block_2a_conv_1[0][0] __________________________________________________________________________________________________ block_2a_relu_1 (Activation) (None, 128, 136, 240 0 block_2a_bn_1[0][0] __________________________________________________________________________________________________ block_2a_conv_2 (Conv2D) (None, 128, 136, 240 147584 block_2a_relu_1[0][0] __________________________________________________________________________________________________ block_2a_conv_shortcut (Conv2D) (None, 128, 136, 240 8320 block_1b_relu[0][0] __________________________________________________________________________________________________ block_2a_bn_2 (BatchNormalizati (None, 128, 136, 240 512 block_2a_conv_2[0][0] __________________________________________________________________________________________________ block_2a_bn_shortcut (BatchNorm (None, 128, 136, 240 512 block_2a_conv_shortcut[0][0] __________________________________________________________________________________________________ add_3 (Add) (None, 128, 136, 240 0 block_2a_bn_2[0][0] block_2a_bn_shortcut[0][0] __________________________________________________________________________________________________ block_2a_relu (Activation) (None, 128, 136, 240 0 add_3[0][0] __________________________________________________________________________________________________ block_2b_conv_1 (Conv2D) (None, 128, 136, 240 147584 block_2a_relu[0][0] __________________________________________________________________________________________________ block_2b_bn_1 (BatchNormalizati (None, 128, 136, 240 512 block_2b_conv_1[0][0] __________________________________________________________________________________________________ block_2b_relu_1 (Activation) (None, 128, 136, 240 0 block_2b_bn_1[0][0] __________________________________________________________________________________________________ block_2b_conv_2 (Conv2D) (None, 128, 136, 240 147584 block_2b_relu_1[0][0] __________________________________________________________________________________________________ block_2b_bn_2 (BatchNormalizati (None, 128, 136, 240 512 block_2b_conv_2[0][0] __________________________________________________________________________________________________ add_4 (Add) (None, 128, 136, 240 0 block_2b_bn_2[0][0] block_2a_relu[0][0] __________________________________________________________________________________________________ block_2b_relu (Activation) (None, 128, 136, 240 0 add_4[0][0] __________________________________________________________________________________________________ block_3a_conv_1 (Conv2D) (None, 256, 68, 120) 295168 block_2b_relu[0][0] __________________________________________________________________________________________________ block_3a_bn_1 (BatchNormalizati (None, 256, 68, 120) 1024 block_3a_conv_1[0][0] __________________________________________________________________________________________________ block_3a_relu_1 (Activation) (None, 256, 68, 120) 0 block_3a_bn_1[0][0] __________________________________________________________________________________________________ block_3a_conv_2 (Conv2D) (None, 256, 68, 120) 590080 block_3a_relu_1[0][0] __________________________________________________________________________________________________ block_3a_conv_shortcut (Conv2D) (None, 256, 68, 120) 33024 block_2b_relu[0][0] __________________________________________________________________________________________________ block_3a_bn_2 (BatchNormalizati (None, 256, 68, 120) 1024 block_3a_conv_2[0][0] __________________________________________________________________________________________________ block_3a_bn_shortcut (BatchNorm (None, 256, 68, 120) 1024 block_3a_conv_shortcut[0][0] __________________________________________________________________________________________________ add_5 (Add) (None, 256, 68, 120) 0 block_3a_bn_2[0][0] block_3a_bn_shortcut[0][0] __________________________________________________________________________________________________ block_3a_relu (Activation) (None, 256, 68, 120) 0 add_5[0][0] __________________________________________________________________________________________________ block_3b_conv_1 (Conv2D) (None, 256, 68, 120) 590080 block_3a_relu[0][0] __________________________________________________________________________________________________ block_3b_bn_1 (BatchNormalizati (None, 256, 68, 120) 1024 block_3b_conv_1[0][0] __________________________________________________________________________________________________ block_3b_relu_1 (Activation) (None, 256, 68, 120) 0 block_3b_bn_1[0][0] __________________________________________________________________________________________________ block_3b_conv_2 (Conv2D) (None, 256, 68, 120) 590080 block_3b_relu_1[0][0] __________________________________________________________________________________________________ block_3b_bn_2 (BatchNormalizati (None, 256, 68, 120) 1024 block_3b_conv_2[0][0] __________________________________________________________________________________________________ add_6 (Add) (None, 256, 68, 120) 0 block_3b_bn_2[0][0] block_3a_relu[0][0] __________________________________________________________________________________________________ block_3b_relu (Activation) (None, 256, 68, 120) 0 add_6[0][0] __________________________________________________________________________________________________ block_4a_conv_1 (Conv2D) (None, 512, 68, 120) 1180160 block_3b_relu[0][0] __________________________________________________________________________________________________ block_4a_bn_1 (BatchNormalizati (None, 512, 68, 120) 2048 block_4a_conv_1[0][0] __________________________________________________________________________________________________ block_4a_relu_1 (Activation) (None, 512, 68, 120) 0 block_4a_bn_1[0][0] __________________________________________________________________________________________________ block_4a_conv_2 (Conv2D) (None, 512, 68, 120) 2359808 block_4a_relu_1[0][0] __________________________________________________________________________________________________ block_4a_conv_shortcut (Conv2D) (None, 512, 68, 120) 131584 block_3b_relu[0][0] __________________________________________________________________________________________________ block_4a_bn_2 (BatchNormalizati (None, 512, 68, 120) 2048 block_4a_conv_2[0][0] __________________________________________________________________________________________________ block_4a_bn_shortcut (BatchNorm (None, 512, 68, 120) 2048 block_4a_conv_shortcut[0][0] __________________________________________________________________________________________________ add_7 (Add) (None, 512, 68, 120) 0 block_4a_bn_2[0][0] block_4a_bn_shortcut[0][0] __________________________________________________________________________________________________ block_4a_relu (Activation) (None, 512, 68, 120) 0 add_7[0][0] __________________________________________________________________________________________________ block_4b_conv_1 (Conv2D) (None, 512, 68, 120) 2359808 block_4a_relu[0][0] __________________________________________________________________________________________________ block_4b_bn_1 (BatchNormalizati (None, 512, 68, 120) 2048 block_4b_conv_1[0][0] __________________________________________________________________________________________________ block_4b_relu_1 (Activation) (None, 512, 68, 120) 0 block_4b_bn_1[0][0] __________________________________________________________________________________________________ block_4b_conv_2 (Conv2D) (None, 512, 68, 120) 2359808 block_4b_relu_1[0][0] __________________________________________________________________________________________________ block_4b_bn_2 (BatchNormalizati (None, 512, 68, 120) 2048 block_4b_conv_2[0][0] __________________________________________________________________________________________________ add_8 (Add) (None, 512, 68, 120) 0 block_4b_bn_2[0][0] block_4a_relu[0][0] __________________________________________________________________________________________________ block_4b_relu (Activation) (None, 512, 68, 120) 0 add_8[0][0] __________________________________________________________________________________________________ output_bbox (Conv2D) (None, 12, 68, 120) 6156 block_4b_relu[0][0] __________________________________________________________________________________________________ output_cov (Conv2D) (None, 3, 68, 120) 1539 block_4b_relu[0][0] ================================================================================================== Total params: 11,203,023 Trainable params: 11,193,295 Non-trainable params: 9,728 __________________________________________________________________________________________________ 2023-02-20 14:13:36,593 [INFO] root: DetectNet V2 model built. 2023-02-20 14:13:36,593 [INFO] root: Building rasterizer. 2023-02-20 14:13:36,593 [INFO] root: Rasterizers built. 2023-02-20 14:13:36,610 [INFO] root: Building training graph. 2023-02-20 14:13:36,612 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Serial augmentation enabled = False 2023-02-20 14:13:36,612 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Pseudo sharding enabled = False 2023-02-20 14:13:36,612 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Max Image Dimensions (all sources): (0, 0) 2023-02-20 14:13:36,612 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: number of cpus: 80, io threads: 160, compute threads: 80, buffered batches: 4 2023-02-20 14:13:36,612 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: total dataset size 2846, number of sources: 1, batch size per gpu: 4, steps: 712 WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead. 2023-02-20 14:13:36,655 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead. WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2023-02-20 14:13:36,693 [WARNING] tensorflow: Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2023-02-20 14:13:36,715 [INFO] iva.detectnet_v2.dataloader.default_dataloader: Bounding box coordinates were detected in the input specification! Bboxes will be automatically converted to polygon coordinates. 2023-02-20 14:13:36,953 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: shuffle: True - shard 0 of 1 2023-02-20 14:13:36,958 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: sampling 1 datasets with weights: 2023-02-20 14:13:36,958 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: source: 0 weight: 1.000000 WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2023-02-20 14:13:36,972 [WARNING] tensorflow: Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2023-02-20 14:13:37,316 [INFO] __main__: Found 2846 samples in training set 2023-02-20 14:13:37,321 [INFO] root: Rasterizing tensors. 2023-02-20 14:13:37,535 [INFO] root: Tensors rasterized. 2023-02-20 14:13:40,378 [INFO] root: Training graph built. 2023-02-20 14:13:40,378 [INFO] root: Building validation graph. 2023-02-20 14:13:40,379 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Serial augmentation enabled = False 2023-02-20 14:13:40,379 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Pseudo sharding enabled = False 2023-02-20 14:13:40,379 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Max Image Dimensions (all sources): (0, 0) 2023-02-20 14:13:40,379 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: number of cpus: 80, io threads: 160, compute threads: 80, buffered batches: 4 2023-02-20 14:13:40,379 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: total dataset size 463, number of sources: 1, batch size per gpu: 4, steps: 116 WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2023-02-20 14:13:40,391 [WARNING] tensorflow: Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2023-02-20 14:13:40,410 [INFO] iva.detectnet_v2.dataloader.default_dataloader: Bounding box coordinates were detected in the input specification! Bboxes will be automatically converted to polygon coordinates. 2023-02-20 14:13:40,637 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: shuffle: False - shard 0 of 1 2023-02-20 14:13:40,641 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: sampling 1 datasets with weights: 2023-02-20 14:13:40,641 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: source: 0 weight: 1.000000 WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2023-02-20 14:13:40,655 [WARNING] tensorflow: Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2023-02-20 14:13:40,877 [INFO] __main__: Found 463 samples in validation set 2023-02-20 14:13:40,877 [INFO] root: Rasterizing tensors. 2023-02-20 14:13:41,086 [INFO] root: Tensors rasterized. 2023-02-20 14:13:41,403 [INFO] root: Validation graph built. 2023-02-20 14:13:42,630 [INFO] root: Running training loop. 2023-02-20 14:13:42,630 [INFO] __main__: Checkpoint interval: 10 2023-02-20 14:13:42,630 [INFO] __main__: Scalars logged at every 14 steps 2023-02-20 14:13:42,630 [INFO] __main__: Images logged at every 3560 steps INFO:tensorflow:Create CheckpointSaverHook. 2023-02-20 14:13:42,633 [INFO] tensorflow: Create CheckpointSaverHook. INFO:tensorflow:Graph was finalized. 2023-02-20 14:13:44,678 [INFO] tensorflow: Graph was finalized. INFO:tensorflow:Running local_init_op. 2023-02-20 14:13:47,181 [INFO] tensorflow: Running local_init_op. INFO:tensorflow:Done running local_init_op. 2023-02-20 14:13:47,913 [INFO] tensorflow: Done running local_init_op. INFO:tensorflow:Saving checkpoints for step-0. 2023-02-20 14:13:55,191 [INFO] tensorflow: Saving checkpoints for step-0. INFO:tensorflow:epoch = 0.0, learning_rate = 4.9999994e-06, loss = 0.086082466, step = 0 2023-02-20 14:14:31,869 [INFO] tensorflow: epoch = 0.0, learning_rate = 4.9999994e-06, loss = 0.086082466, step = 0 2023-02-20 14:14:31,881 [INFO] root: None 2023-02-20 14:14:31,895 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 0/120: loss: 0.08608 learning rate: 0.00000 Time taken: 0:00:00 ETA: 0:00:00 2023-02-20 14:14:31,895 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 0.219 INFO:tensorflow:global_step/sec: 2.14366 2023-02-20 14:14:38,402 [INFO] tensorflow: global_step/sec: 2.14366 INFO:tensorflow:epoch = 0.021067415730337078, learning_rate = 5.020253e-06, loss = 0.07583617, step = 15 (6.704 sec) 2023-02-20 14:14:38,573 [INFO] tensorflow: epoch = 0.021067415730337078, learning_rate = 5.020253e-06, loss = 0.07583617, step = 15 (6.704 sec) 2023-02-20 14:14:40,054 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 3.783 INFO:tensorflow:global_step/sec: 5.98751 2023-02-20 14:14:40,741 [INFO] tensorflow: global_step/sec: 5.98751 INFO:tensorflow:global_step/sec: 6.0294 2023-02-20 14:14:43,062 [INFO] tensorflow: global_step/sec: 6.0294 INFO:tensorflow:epoch = 0.06601123595505617, learning_rate = 5.0637354e-06, loss = 0.07107144, step = 47 (5.320 sec) 2023-02-20 14:14:43,894 [INFO] tensorflow: epoch = 0.06601123595505617, learning_rate = 5.0637354e-06, loss = 0.07107144, step = 47 (5.320 sec) 2023-02-20 14:14:44,228 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 23.965 INFO:tensorflow:global_step/sec: 5.99842 2023-02-20 14:14:45,396 [INFO] tensorflow: global_step/sec: 5.99842 INFO:tensorflow:global_step/sec: 6.00963 2023-02-20 14:14:47,726 [INFO] tensorflow: global_step/sec: 6.00963 2023-02-20 14:14:48,387 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.046 INFO:tensorflow:epoch = 0.11095505617977527, learning_rate = 5.1075895e-06, loss = 0.0616019, step = 79 (5.328 sec) 2023-02-20 14:14:49,222 [INFO] tensorflow: epoch = 0.11095505617977527, learning_rate = 5.1075895e-06, loss = 0.0616019, step = 79 (5.328 sec) INFO:tensorflow:global_step/sec: 6.02388 2023-02-20 14:14:50,050 [INFO] tensorflow: global_step/sec: 6.02388 INFO:tensorflow:global_step/sec: 6.00219 2023-02-20 14:14:52,382 [INFO] tensorflow: global_step/sec: 6.00219 2023-02-20 14:14:52,549 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.028 INFO:tensorflow:epoch = 0.15589887640449437, learning_rate = 5.1518286e-06, loss = 0.054857057, step = 111 (5.315 sec) 2023-02-20 14:14:54,537 [INFO] tensorflow: epoch = 0.15589887640449437, learning_rate = 5.1518286e-06, loss = 0.054857057, step = 111 (5.315 sec) INFO:tensorflow:global_step/sec: 6.03291 2023-02-20 14:14:54,703 [INFO] tensorflow: global_step/sec: 6.03291 2023-02-20 14:14:56,704 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.067 INFO:tensorflow:global_step/sec: 6.00053 2023-02-20 14:14:57,036 [INFO] tensorflow: global_step/sec: 6.00053 INFO:tensorflow:global_step/sec: 5.98053 2023-02-20 14:14:59,377 [INFO] tensorflow: global_step/sec: 5.98053 INFO:tensorflow:epoch = 0.199438202247191, learning_rate = 5.195053e-06, loss = 0.036142357, step = 142 (5.177 sec) 2023-02-20 14:14:59,714 [INFO] tensorflow: epoch = 0.199438202247191, learning_rate = 5.195053e-06, loss = 0.036142357, step = 142 (5.177 sec) 2023-02-20 14:15:00,880 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 23.949 INFO:tensorflow:global_step/sec: 6.00841 2023-02-20 14:15:01,707 [INFO] tensorflow: global_step/sec: 6.00841 INFO:tensorflow:global_step/sec: 6.0461 2023-02-20 14:15:04,022 [INFO] tensorflow: global_step/sec: 6.0461 INFO:tensorflow:epoch = 0.2443820224719101, learning_rate = 5.240044e-06, loss = 0.029754661, step = 174 (5.299 sec) 2023-02-20 14:15:05,013 [INFO] tensorflow: epoch = 0.2443820224719101, learning_rate = 5.240044e-06, loss = 0.029754661, step = 174 (5.299 sec) 2023-02-20 14:15:05,013 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.198 INFO:tensorflow:global_step/sec: 6.06027 2023-02-20 14:15:06,333 [INFO] tensorflow: global_step/sec: 6.06027 INFO:tensorflow:global_step/sec: 6.05593 2023-02-20 14:15:08,645 [INFO] tensorflow: global_step/sec: 6.05593 2023-02-20 14:15:09,148 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.186 INFO:tensorflow:epoch = 0.2893258426966292, learning_rate = 5.2854302e-06, loss = 0.019679958, step = 206 (5.290 sec) 2023-02-20 14:15:10,302 [INFO] tensorflow: epoch = 0.2893258426966292, learning_rate = 5.2854302e-06, loss = 0.019679958, step = 206 (5.290 sec) INFO:tensorflow:global_step/sec: 6.03434 2023-02-20 14:15:10,964 [INFO] tensorflow: global_step/sec: 6.03434 INFO:tensorflow:global_step/sec: 6.05815 2023-02-20 14:15:13,276 [INFO] tensorflow: global_step/sec: 6.05815 2023-02-20 14:15:13,276 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.223 INFO:tensorflow:epoch = 0.3342696629213483, learning_rate = 5.331209e-06, loss = 0.015600432, step = 238 (5.296 sec) 2023-02-20 14:15:15,599 [INFO] tensorflow: epoch = 0.3342696629213483, learning_rate = 5.331209e-06, loss = 0.015600432, step = 238 (5.296 sec) INFO:tensorflow:global_step/sec: 6.02326 2023-02-20 14:15:15,600 [INFO] tensorflow: global_step/sec: 6.02326 2023-02-20 14:15:17,423 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.119 INFO:tensorflow:global_step/sec: 6.01202 2023-02-20 14:15:17,928 [INFO] tensorflow: global_step/sec: 6.01202 INFO:tensorflow:global_step/sec: 6.02707 2023-02-20 14:15:20,251 [INFO] tensorflow: global_step/sec: 6.02707 INFO:tensorflow:epoch = 0.3792134831460674, learning_rate = 5.3773847e-06, loss = 0.01669768, step = 270 (5.316 sec) 2023-02-20 14:15:20,915 [INFO] tensorflow: epoch = 0.3792134831460674, learning_rate = 5.3773847e-06, loss = 0.01669768, step = 270 (5.316 sec) 2023-02-20 14:15:21,587 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.021 INFO:tensorflow:global_step/sec: 6.00048 2023-02-20 14:15:22,584 [INFO] tensorflow: global_step/sec: 6.00048 INFO:tensorflow:global_step/sec: 6.04687 2023-02-20 14:15:24,900 [INFO] tensorflow: global_step/sec: 6.04687 2023-02-20 14:15:25,731 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.133 INFO:tensorflow:epoch = 0.4241573033707865, learning_rate = 5.4239604e-06, loss = 0.013035655, step = 302 (5.314 sec) 2023-02-20 14:15:26,229 [INFO] tensorflow: epoch = 0.4241573033707865, learning_rate = 5.4239604e-06, loss = 0.013035655, step = 302 (5.314 sec) INFO:tensorflow:global_step/sec: 6.02479 2023-02-20 14:15:27,223 [INFO] tensorflow: global_step/sec: 6.02479 INFO:tensorflow:global_step/sec: 6.04029 2023-02-20 14:15:29,541 [INFO] tensorflow: global_step/sec: 6.04029 2023-02-20 14:15:29,873 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.144 INFO:tensorflow:epoch = 0.4691011235955056, learning_rate = 5.4709394e-06, loss = 0.013312144, step = 334 (5.304 sec) 2023-02-20 14:15:31,533 [INFO] tensorflow: epoch = 0.4691011235955056, learning_rate = 5.4709394e-06, loss = 0.013312144, step = 334 (5.304 sec) INFO:tensorflow:global_step/sec: 6.02853 2023-02-20 14:15:31,863 [INFO] tensorflow: global_step/sec: 6.02853 2023-02-20 14:15:34,024 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.093 INFO:tensorflow:global_step/sec: 6.01824 2023-02-20 14:15:34,190 [INFO] tensorflow: global_step/sec: 6.01824 INFO:tensorflow:global_step/sec: 6.04208 2023-02-20 14:15:36,507 [INFO] tensorflow: global_step/sec: 6.04208 INFO:tensorflow:epoch = 0.5140449438202247, learning_rate = 5.51832e-06, loss = 0.009219557, step = 366 (5.306 sec) 2023-02-20 14:15:36,839 [INFO] tensorflow: epoch = 0.5140449438202247, learning_rate = 5.51832e-06, loss = 0.009219557, step = 366 (5.306 sec) 2023-02-20 14:15:38,163 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.160 INFO:tensorflow:global_step/sec: 5.99623 2023-02-20 14:15:38,842 [INFO] tensorflow: global_step/sec: 5.99623 INFO:tensorflow:global_step/sec: 6.03049 2023-02-20 14:15:41,163 [INFO] tensorflow: global_step/sec: 6.03049 INFO:tensorflow:epoch = 0.5589887640449438, learning_rate = 5.5661158e-06, loss = 0.005553379, step = 398 (5.317 sec) 2023-02-20 14:15:42,156 [INFO] tensorflow: epoch = 0.5589887640449438, learning_rate = 5.5661158e-06, loss = 0.005553379, step = 398 (5.317 sec) 2023-02-20 14:15:42,322 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.047 INFO:tensorflow:global_step/sec: 6.02625 2023-02-20 14:15:43,486 [INFO] tensorflow: global_step/sec: 6.02625 INFO:tensorflow:global_step/sec: 6.04235 2023-02-20 14:15:45,803 [INFO] tensorflow: global_step/sec: 6.04235 2023-02-20 14:15:46,466 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.133 INFO:tensorflow:epoch = 0.6039325842696629, learning_rate = 5.6143263e-06, loss = 0.008922378, step = 430 (5.303 sec) 2023-02-20 14:15:47,460 [INFO] tensorflow: epoch = 0.6039325842696629, learning_rate = 5.6143263e-06, loss = 0.008922378, step = 430 (5.303 sec) INFO:tensorflow:global_step/sec: 6.03759 2023-02-20 14:15:48,122 [INFO] tensorflow: global_step/sec: 6.03759 INFO:tensorflow:global_step/sec: 6.02487 2023-02-20 14:15:50,446 [INFO] tensorflow: global_step/sec: 6.02487 2023-02-20 14:15:50,613 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.117 INFO:tensorflow:epoch = 0.648876404494382, learning_rate = 5.662954e-06, loss = 0.0036467286, step = 462 (5.300 sec) 2023-02-20 14:15:52,760 [INFO] tensorflow: epoch = 0.648876404494382, learning_rate = 5.662954e-06, loss = 0.0036467286, step = 462 (5.300 sec) INFO:tensorflow:global_step/sec: 6.04708 2023-02-20 14:15:52,761 [INFO] tensorflow: global_step/sec: 6.04708 2023-02-20 14:15:54,751 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.166 INFO:tensorflow:global_step/sec: 6.02932 2023-02-20 14:15:55,083 [INFO] tensorflow: global_step/sec: 6.02932 INFO:tensorflow:global_step/sec: 6.04567 2023-02-20 14:15:57,399 [INFO] tensorflow: global_step/sec: 6.04567 INFO:tensorflow:epoch = 0.6938202247191011, learning_rate = 5.712003e-06, loss = 0.006119725, step = 494 (5.301 sec) 2023-02-20 14:15:58,061 [INFO] tensorflow: epoch = 0.6938202247191011, learning_rate = 5.712003e-06, loss = 0.006119725, step = 494 (5.301 sec) 2023-02-20 14:15:58,887 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.184 INFO:tensorflow:global_step/sec: 6.04901 2023-02-20 14:15:59,713 [INFO] tensorflow: global_step/sec: 6.04901 INFO:tensorflow:global_step/sec: 6.04708 2023-02-20 14:16:02,028 [INFO] tensorflow: global_step/sec: 6.04708 2023-02-20 14:16:03,019 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.199 INFO:tensorflow:epoch = 0.7387640449438202, learning_rate = 5.7614766e-06, loss = 0.0035466033, step = 526 (5.293 sec) 2023-02-20 14:16:03,353 [INFO] tensorflow: epoch = 0.7387640449438202, learning_rate = 5.7614766e-06, loss = 0.0035466033, step = 526 (5.293 sec) INFO:tensorflow:global_step/sec: 6.04439 2023-02-20 14:16:04,344 [INFO] tensorflow: global_step/sec: 6.04439 INFO:tensorflow:global_step/sec: 6.04177 2023-02-20 14:16:06,662 [INFO] tensorflow: global_step/sec: 6.04177 2023-02-20 14:16:07,159 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.160 INFO:tensorflow:epoch = 0.7837078651685393, learning_rate = 5.8113733e-06, loss = 0.004909405, step = 558 (5.298 sec) 2023-02-20 14:16:08,651 [INFO] tensorflow: epoch = 0.7837078651685393, learning_rate = 5.8113733e-06, loss = 0.004909405, step = 558 (5.298 sec) INFO:tensorflow:global_step/sec: 6.03389 2023-02-20 14:16:08,982 [INFO] tensorflow: global_step/sec: 6.03389 INFO:tensorflow:global_step/sec: 6.03151 2023-02-20 14:16:11,303 [INFO] tensorflow: global_step/sec: 6.03151 2023-02-20 14:16:11,304 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.127 INFO:tensorflow:global_step/sec: 6.04563 2023-02-20 14:16:13,619 [INFO] tensorflow: global_step/sec: 6.04563 INFO:tensorflow:epoch = 0.8286516853932584, learning_rate = 5.8617084e-06, loss = 0.0026412555, step = 590 (5.300 sec) 2023-02-20 14:16:13,951 [INFO] tensorflow: epoch = 0.8286516853932584, learning_rate = 5.8617084e-06, loss = 0.0026412555, step = 590 (5.300 sec) 2023-02-20 14:16:15,444 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.155 INFO:tensorflow:global_step/sec: 6.03191 2023-02-20 14:16:15,940 [INFO] tensorflow: global_step/sec: 6.03191 INFO:tensorflow:global_step/sec: 6.03155 2023-02-20 14:16:18,261 [INFO] tensorflow: global_step/sec: 6.03155 INFO:tensorflow:epoch = 0.8735955056179775, learning_rate = 5.912478e-06, loss = 0.0036212872, step = 622 (5.302 sec) 2023-02-20 14:16:19,253 [INFO] tensorflow: epoch = 0.8735955056179775, learning_rate = 5.912478e-06, loss = 0.0036212872, step = 622 (5.302 sec) 2023-02-20 14:16:19,583 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.160 INFO:tensorflow:global_step/sec: 6.04469 2023-02-20 14:16:20,577 [INFO] tensorflow: global_step/sec: 6.04469 INFO:tensorflow:global_step/sec: 6.03926 2023-02-20 14:16:22,895 [INFO] tensorflow: global_step/sec: 6.03926 2023-02-20 14:16:23,725 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.150 INFO:tensorflow:epoch = 0.9185393258426966, learning_rate = 5.963688e-06, loss = 0.002865105, step = 654 (5.297 sec) 2023-02-20 14:16:24,551 [INFO] tensorflow: epoch = 0.9185393258426966, learning_rate = 5.963688e-06, loss = 0.002865105, step = 654 (5.297 sec) INFO:tensorflow:global_step/sec: 6.04186 2023-02-20 14:16:25,212 [INFO] tensorflow: global_step/sec: 6.04186 INFO:tensorflow:global_step/sec: 6.04391 2023-02-20 14:16:27,529 [INFO] tensorflow: global_step/sec: 6.04391 2023-02-20 14:16:27,863 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.167 INFO:tensorflow:epoch = 0.9634831460674157, learning_rate = 6.015342e-06, loss = 0.003400401, step = 686 (5.295 sec) 2023-02-20 14:16:29,845 [INFO] tensorflow: epoch = 0.9634831460674157, learning_rate = 6.015342e-06, loss = 0.003400401, step = 686 (5.295 sec) INFO:tensorflow:global_step/sec: 6.03988 2023-02-20 14:16:29,847 [INFO] tensorflow: global_step/sec: 6.03988 2023-02-20 14:16:31,993 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.213 INFO:tensorflow:global_step/sec: 6.053 2023-02-20 14:16:32,159 [INFO] tensorflow: global_step/sec: 6.053 9f2dc1f0fa08:245:396 [0] NCCL INFO Bootstrap : Using eth0:172.17.0.4<0> 9f2dc1f0fa08:245:396 [0] NCCL INFO NET/Plugin: Failed to find ncclNetPlugin_v6 symbol. 9f2dc1f0fa08:245:396 [0] NCCL INFO NET/Plugin: Loaded net plugin NCCL RDMA Plugin (v5) 9f2dc1f0fa08:245:396 [0] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v6 symbol. 9f2dc1f0fa08:245:396 [0] NCCL INFO NET/Plugin: Loaded coll plugin SHARP (v5) 9f2dc1f0fa08:245:396 [0] NCCL INFO cudaDriverVersion 12000 NCCL version 2.15.1+cuda11.8 9f2dc1f0fa08:245:396 [0] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so 9f2dc1f0fa08:245:396 [0] NCCL INFO P2P plugin IBext 9f2dc1f0fa08:245:396 [0] NCCL INFO NET/IB : No device found. 9f2dc1f0fa08:245:396 [0] NCCL INFO NET/IB : No device found. 9f2dc1f0fa08:245:396 [0] NCCL INFO NET/Socket : Using [0]eth0:172.17.0.4<0> 9f2dc1f0fa08:245:396 [0] NCCL INFO Using network Socket 9f2dc1f0fa08:245:396 [0] NCCL INFO Setting affinity for GPU 0 to 0fffff00,000fffff 9f2dc1f0fa08:245:396 [0] NCCL INFO Channel 00/32 : 0 9f2dc1f0fa08:245:396 [0] NCCL INFO Channel 01/32 : 0 9f2dc1f0fa08:245:396 [0] NCCL INFO Channel 02/32 : 0 9f2dc1f0fa08:245:396 [0] NCCL INFO Channel 03/32 : 0 9f2dc1f0fa08:245:396 [0] NCCL INFO Channel 04/32 : 0 9f2dc1f0fa08:245:396 [0] NCCL INFO Channel 05/32 : 0 9f2dc1f0fa08:245:396 [0] NCCL INFO Channel 06/32 : 0 9f2dc1f0fa08:245:396 [0] NCCL INFO Channel 07/32 : 0 9f2dc1f0fa08:245:396 [0] NCCL INFO Channel 08/32 : 0 9f2dc1f0fa08:245:396 [0] NCCL INFO Channel 09/32 : 0 9f2dc1f0fa08:245:396 [0] NCCL INFO Channel 10/32 : 0 9f2dc1f0fa08:245:396 [0] NCCL INFO Channel 11/32 : 0 9f2dc1f0fa08:245:396 [0] NCCL INFO Channel 12/32 : 0 9f2dc1f0fa08:245:396 [0] NCCL INFO Channel 13/32 : 0 9f2dc1f0fa08:245:396 [0] NCCL INFO Channel 14/32 : 0 9f2dc1f0fa08:245:396 [0] NCCL INFO Channel 15/32 : 0 9f2dc1f0fa08:245:396 [0] NCCL INFO Channel 16/32 : 0 9f2dc1f0fa08:245:396 [0] NCCL INFO Channel 17/32 : 0 9f2dc1f0fa08:245:396 [0] NCCL INFO Channel 18/32 : 0 9f2dc1f0fa08:245:396 [0] NCCL INFO Channel 19/32 : 0 9f2dc1f0fa08:245:396 [0] NCCL INFO Channel 20/32 : 0 9f2dc1f0fa08:245:396 [0] NCCL INFO Channel 21/32 : 0 9f2dc1f0fa08:245:396 [0] NCCL INFO Channel 22/32 : 0 9f2dc1f0fa08:245:396 [0] NCCL INFO Channel 23/32 : 0 9f2dc1f0fa08:245:396 [0] NCCL INFO Channel 24/32 : 0 9f2dc1f0fa08:245:396 [0] NCCL INFO Channel 25/32 : 0 9f2dc1f0fa08:245:396 [0] NCCL INFO Channel 26/32 : 0 9f2dc1f0fa08:245:396 [0] NCCL INFO Channel 27/32 : 0 9f2dc1f0fa08:245:396 [0] NCCL INFO Channel 28/32 : 0 9f2dc1f0fa08:245:396 [0] NCCL INFO Channel 29/32 : 0 9f2dc1f0fa08:245:396 [0] NCCL INFO Channel 30/32 : 0 9f2dc1f0fa08:245:396 [0] NCCL INFO Channel 31/32 : 0 9f2dc1f0fa08:245:396 [0] NCCL INFO Trees [0] -1/-1/-1->0->-1 [1] -1/-1/-1->0->-1 [2] -1/-1/-1->0->-1 [3] -1/-1/-1->0->-1 [4] -1/-1/-1->0->-1 [5] -1/-1/-1->0->-1 [6] -1/-1/-1->0->-1 [7] -1/-1/-1->0->-1 [8] -1/-1/-1->0->-1 [9] -1/-1/-1->0->-1 [10] -1/-1/-1->0->-1 [11] -1/-1/-1->0->-1 [12] -1/-1/-1->0->-1 [13] -1/-1/-1->0->-1 [14] -1/-1/-1->0->-1 [15] -1/-1/-1->0->-1 [16] -1/-1/-1->0->-1 [17] -1/-1/-1->0->-1 [18] -1/-1/-1->0->-1 [19] -1/-1/-1->0->-1 [20] -1/-1/-1->0->-1 [21] -1/-1/-1->0->-1 [22] -1/-1/-1->0->-1 [23] -1/-1/-1->0->-1 [24] -1/-1/-1->0->-1 [25] -1/-1/-1->0->-1 [26] -1/-1/-1->0->-1 [27] -1/-1/-1->0->-1 [28] -1/-1/-1->0->-1 [29] -1/-1/-1->0->-1 [30] -1/-1/-1->0->-1 [31] -1/-1/-1->0->-1 9f2dc1f0fa08:245:396 [0] NCCL INFO Connected all rings 9f2dc1f0fa08:245:396 [0] NCCL INFO Connected all trees 9f2dc1f0fa08:245:396 [0] NCCL INFO 32 coll channels, 32 p2p channels, 32 p2p channels per peer 9f2dc1f0fa08:245:396 [0] NCCL INFO comm 0x7f4d1fbbadb0 rank 0 nranks 1 cudaDev 0 busId 3b000 - Init COMPLETE 2023-02-20 14:16:34,741 [INFO] root: None 2023-02-20 14:16:34,741 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 1/120: loss: 0.00062 learning rate: 0.00001 Time taken: 0:02:20.952470 ETA: 4:39:33.343938 INFO:tensorflow:epoch = 1.002808988764045, learning_rate = 6.0609023e-06, loss = 0.0006871463, step = 714 (5.225 sec) 2023-02-20 14:16:35,071 [INFO] tensorflow: epoch = 1.002808988764045, learning_rate = 6.0609023e-06, loss = 0.0006871463, step = 714 (5.225 sec) INFO:tensorflow:global_step/sec: 4.80679 2023-02-20 14:16:35,072 [INFO] tensorflow: global_step/sec: 4.80679 2023-02-20 14:16:36,725 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 21.134 INFO:tensorflow:global_step/sec: 6.04595 2023-02-20 14:16:37,388 [INFO] tensorflow: global_step/sec: 6.04595 INFO:tensorflow:global_step/sec: 6.05252 2023-02-20 14:16:39,701 [INFO] tensorflow: global_step/sec: 6.05252 INFO:tensorflow:epoch = 1.047752808988764, learning_rate = 6.1133983e-06, loss = 0.000748874, step = 746 (5.297 sec) 2023-02-20 14:16:40,367 [INFO] tensorflow: epoch = 1.047752808988764, learning_rate = 6.1133983e-06, loss = 0.000748874, step = 746 (5.297 sec) 2023-02-20 14:16:40,862 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.172 INFO:tensorflow:global_step/sec: 6.03826 2023-02-20 14:16:42,019 [INFO] tensorflow: global_step/sec: 6.03826 INFO:tensorflow:global_step/sec: 6.04266 2023-02-20 14:16:44,336 [INFO] tensorflow: global_step/sec: 6.04266 2023-02-20 14:16:44,999 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.177 INFO:tensorflow:epoch = 1.0926966292134832, learning_rate = 6.1663486e-06, loss = 0.0006153536, step = 778 (5.293 sec) 2023-02-20 14:16:45,661 [INFO] tensorflow: epoch = 1.0926966292134832, learning_rate = 6.1663486e-06, loss = 0.0006153536, step = 778 (5.293 sec) INFO:tensorflow:global_step/sec: 6.04271 2023-02-20 14:16:46,653 [INFO] tensorflow: global_step/sec: 6.04271 INFO:tensorflow:global_step/sec: 6.04553 2023-02-20 14:16:48,969 [INFO] tensorflow: global_step/sec: 6.04553 2023-02-20 14:16:49,135 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.181 INFO:tensorflow:epoch = 1.1376404494382022, learning_rate = 6.2197573e-06, loss = 0.00068229024, step = 810 (5.294 sec) 2023-02-20 14:16:50,954 [INFO] tensorflow: epoch = 1.1376404494382022, learning_rate = 6.2197573e-06, loss = 0.00068229024, step = 810 (5.294 sec) INFO:tensorflow:global_step/sec: 6.04374 2023-02-20 14:16:51,285 [INFO] tensorflow: global_step/sec: 6.04374 2023-02-20 14:16:53,269 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.188 INFO:tensorflow:global_step/sec: 6.04624 2023-02-20 14:16:53,601 [INFO] tensorflow: global_step/sec: 6.04624 INFO:tensorflow:global_step/sec: 6.04359 2023-02-20 14:16:55,917 [INFO] tensorflow: global_step/sec: 6.04359 INFO:tensorflow:epoch = 1.1825842696629214, learning_rate = 6.2736235e-06, loss = 0.0007883493, step = 842 (5.293 sec) 2023-02-20 14:16:56,247 [INFO] tensorflow: epoch = 1.1825842696629214, learning_rate = 6.2736235e-06, loss = 0.0007883493, step = 842 (5.293 sec) 2023-02-20 14:16:57,403 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.194 INFO:tensorflow:global_step/sec: 6.05488 2023-02-20 14:16:58,229 [INFO] tensorflow: global_step/sec: 6.05488 INFO:tensorflow:global_step/sec: 6.04707 2023-02-20 14:17:00,545 [INFO] tensorflow: global_step/sec: 6.04707 INFO:tensorflow:epoch = 1.2275280898876404, learning_rate = 6.3279613e-06, loss = 0.0006778587, step = 874 (5.290 sec) 2023-02-20 14:17:01,538 [INFO] tensorflow: epoch = 1.2275280898876404, learning_rate = 6.3279613e-06, loss = 0.0006778587, step = 874 (5.290 sec) 2023-02-20 14:17:01,538 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.185 INFO:tensorflow:global_step/sec: 6.04427 2023-02-20 14:17:02,861 [INFO] tensorflow: global_step/sec: 6.04427 INFO:tensorflow:global_step/sec: 6.04196 2023-02-20 14:17:05,178 [INFO] tensorflow: global_step/sec: 6.04196 2023-02-20 14:17:05,673 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.183 INFO:tensorflow:epoch = 1.2724719101123596, learning_rate = 6.3827706e-06, loss = 0.0006121363, step = 906 (5.294 sec) 2023-02-20 14:17:06,832 [INFO] tensorflow: epoch = 1.2724719101123596, learning_rate = 6.3827706e-06, loss = 0.0006121363, step = 906 (5.294 sec) INFO:tensorflow:global_step/sec: 6.03633 2023-02-20 14:17:07,497 [INFO] tensorflow: global_step/sec: 6.03633 INFO:tensorflow:global_step/sec: 6.02801 2023-02-20 14:17:09,820 [INFO] tensorflow: global_step/sec: 6.02801 2023-02-20 14:17:09,820 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.116 INFO:tensorflow:epoch = 1.3174157303370786, learning_rate = 6.438054e-06, loss = 0.0007114267, step = 938 (5.315 sec) 2023-02-20 14:17:12,147 [INFO] tensorflow: epoch = 1.3174157303370786, learning_rate = 6.438054e-06, loss = 0.0007114267, step = 938 (5.315 sec) INFO:tensorflow:global_step/sec: 6.01303 2023-02-20 14:17:12,148 [INFO] tensorflow: global_step/sec: 6.01303 2023-02-20 14:17:13,973 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.084 INFO:tensorflow:global_step/sec: 6.0299 2023-02-20 14:17:14,470 [INFO] tensorflow: global_step/sec: 6.0299 INFO:tensorflow:global_step/sec: 6.02673 2023-02-20 14:17:16,793 [INFO] tensorflow: global_step/sec: 6.02673 INFO:tensorflow:epoch = 1.3623595505617978, learning_rate = 6.493816e-06, loss = 0.0007019404, step = 970 (5.307 sec) 2023-02-20 14:17:17,454 [INFO] tensorflow: epoch = 1.3623595505617978, learning_rate = 6.493816e-06, loss = 0.0007019404, step = 970 (5.307 sec) 2023-02-20 14:17:18,117 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.130 INFO:tensorflow:global_step/sec: 6.03965 2023-02-20 14:17:19,111 [INFO] tensorflow: global_step/sec: 6.03965 INFO:tensorflow:global_step/sec: 6.01982 2023-02-20 14:17:21,436 [INFO] tensorflow: global_step/sec: 6.01982 2023-02-20 14:17:22,263 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.125 INFO:tensorflow:epoch = 1.4073033707865168, learning_rate = 6.5500617e-06, loss = 0.00061316806, step = 1002 (5.305 sec) 2023-02-20 14:17:22,759 [INFO] tensorflow: epoch = 1.4073033707865168, learning_rate = 6.5500617e-06, loss = 0.00061316806, step = 1002 (5.305 sec) INFO:tensorflow:global_step/sec: 6.04493 2023-02-20 14:17:23,752 [INFO] tensorflow: global_step/sec: 6.04493 INFO:tensorflow:global_step/sec: 6.03041 2023-02-20 14:17:26,074 [INFO] tensorflow: global_step/sec: 6.03041 2023-02-20 14:17:26,405 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.141 INFO:tensorflow:epoch = 1.452247191011236, learning_rate = 6.606794e-06, loss = 0.000614454, step = 1034 (5.303 sec) 2023-02-20 14:17:28,062 [INFO] tensorflow: epoch = 1.452247191011236, learning_rate = 6.606794e-06, loss = 0.000614454, step = 1034 (5.303 sec) INFO:tensorflow:global_step/sec: 6.03494 2023-02-20 14:17:28,394 [INFO] tensorflow: global_step/sec: 6.03494 2023-02-20 14:17:30,545 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.162 INFO:tensorflow:global_step/sec: 6.03911 2023-02-20 14:17:30,712 [INFO] tensorflow: global_step/sec: 6.03911 INFO:tensorflow:global_step/sec: 6.05191 2023-02-20 14:17:33,025 [INFO] tensorflow: global_step/sec: 6.05191 INFO:tensorflow:epoch = 1.497191011235955, learning_rate = 6.664012e-06, loss = 0.00071158505, step = 1066 (5.294 sec) 2023-02-20 14:17:33,356 [INFO] tensorflow: epoch = 1.497191011235955, learning_rate = 6.664012e-06, loss = 0.00071158505, step = 1066 (5.294 sec) 2023-02-20 14:17:34,677 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.199 INFO:tensorflow:global_step/sec: 6.05111 2023-02-20 14:17:35,339 [INFO] tensorflow: global_step/sec: 6.05111 INFO:tensorflow:global_step/sec: 6.04639 2023-02-20 14:17:37,654 [INFO] tensorflow: global_step/sec: 6.04639 INFO:tensorflow:epoch = 1.5421348314606742, learning_rate = 6.721731e-06, loss = 0.0008581015, step = 1098 (5.290 sec) 2023-02-20 14:17:38,646 [INFO] tensorflow: epoch = 1.5421348314606742, learning_rate = 6.721731e-06, loss = 0.0008581015, step = 1098 (5.290 sec) 2023-02-20 14:17:38,811 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.193 INFO:tensorflow:global_step/sec: 6.04797 2023-02-20 14:17:39,969 [INFO] tensorflow: global_step/sec: 6.04797 INFO:tensorflow:global_step/sec: 6.01541 2023-02-20 14:17:42,297 [INFO] tensorflow: global_step/sec: 6.01541 2023-02-20 14:17:42,965 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.078 INFO:tensorflow:epoch = 1.5870786516853932, learning_rate = 6.779951e-06, loss = 0.00061222067, step = 1130 (5.315 sec) 2023-02-20 14:17:43,961 [INFO] tensorflow: epoch = 1.5870786516853932, learning_rate = 6.779951e-06, loss = 0.00061222067, step = 1130 (5.315 sec) INFO:tensorflow:global_step/sec: 6.0062 2023-02-20 14:17:44,627 [INFO] tensorflow: global_step/sec: 6.0062 INFO:tensorflow:global_step/sec: 6.03704 2023-02-20 14:17:46,946 [INFO] tensorflow: global_step/sec: 6.03704 2023-02-20 14:17:47,113 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.111 INFO:tensorflow:epoch = 1.6320224719101124, learning_rate = 6.8386744e-06, loss = 0.00065834745, step = 1162 (5.306 sec) 2023-02-20 14:17:49,267 [INFO] tensorflow: epoch = 1.6320224719101124, learning_rate = 6.8386744e-06, loss = 0.00065834745, step = 1162 (5.306 sec) INFO:tensorflow:global_step/sec: 6.03156 2023-02-20 14:17:49,268 [INFO] tensorflow: global_step/sec: 6.03156 2023-02-20 14:17:51,253 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.154 INFO:tensorflow:global_step/sec: 6.04226 2023-02-20 14:17:51,585 [INFO] tensorflow: global_step/sec: 6.04226 INFO:tensorflow:global_step/sec: 6.03704 2023-02-20 14:17:53,904 [INFO] tensorflow: global_step/sec: 6.03704 INFO:tensorflow:epoch = 1.6769662921348314, learning_rate = 6.8979066e-06, loss = 0.00061381416, step = 1194 (5.300 sec) 2023-02-20 14:17:54,566 [INFO] tensorflow: epoch = 1.6769662921348314, learning_rate = 6.8979066e-06, loss = 0.00061381416, step = 1194 (5.300 sec) 2023-02-20 14:17:55,394 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.153 INFO:tensorflow:global_step/sec: 6.0381 2023-02-20 14:17:56,222 [INFO] tensorflow: global_step/sec: 6.0381 INFO:tensorflow:global_step/sec: 6.04336 2023-02-20 14:17:58,539 [INFO] tensorflow: global_step/sec: 6.04336 2023-02-20 14:17:59,530 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.178 INFO:tensorflow:epoch = 1.7219101123595506, learning_rate = 6.9576454e-06, loss = 0.00071252364, step = 1226 (5.295 sec) 2023-02-20 14:17:59,861 [INFO] tensorflow: epoch = 1.7219101123595506, learning_rate = 6.9576454e-06, loss = 0.00071252364, step = 1226 (5.295 sec) INFO:tensorflow:global_step/sec: 6.041 2023-02-20 14:18:00,856 [INFO] tensorflow: global_step/sec: 6.041 INFO:tensorflow:global_step/sec: 6.04495 2023-02-20 14:18:03,172 [INFO] tensorflow: global_step/sec: 6.04495 2023-02-20 14:18:03,669 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.162 INFO:tensorflow:epoch = 1.7668539325842696, learning_rate = 7.0179076e-06, loss = 0.0006125018, step = 1258 (5.294 sec) 2023-02-20 14:18:05,156 [INFO] tensorflow: epoch = 1.7668539325842696, learning_rate = 7.0179076e-06, loss = 0.0006125018, step = 1258 (5.294 sec) INFO:tensorflow:global_step/sec: 6.04339 2023-02-20 14:18:05,489 [INFO] tensorflow: global_step/sec: 6.04339 INFO:tensorflow:global_step/sec: 6.04945 2023-02-20 14:18:07,803 [INFO] tensorflow: global_step/sec: 6.04945 2023-02-20 14:18:07,804 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.185 INFO:tensorflow:global_step/sec: 6.03504 2023-02-20 14:18:10,123 [INFO] tensorflow: global_step/sec: 6.03504 INFO:tensorflow:epoch = 1.8117977528089888, learning_rate = 7.0786928e-06, loss = 0.0008352039, step = 1290 (5.298 sec) 2023-02-20 14:18:10,454 [INFO] tensorflow: epoch = 1.8117977528089888, learning_rate = 7.0786928e-06, loss = 0.0008352039, step = 1290 (5.298 sec) 2023-02-20 14:18:11,943 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.161 INFO:tensorflow:global_step/sec: 6.03821 2023-02-20 14:18:12,441 [INFO] tensorflow: global_step/sec: 6.03821 INFO:tensorflow:global_step/sec: 6.03101 2023-02-20 14:18:14,763 [INFO] tensorflow: global_step/sec: 6.03101 INFO:tensorflow:epoch = 1.8567415730337078, learning_rate = 7.1400036e-06, loss = 0.0006924549, step = 1322 (5.302 sec) 2023-02-20 14:18:15,756 [INFO] tensorflow: epoch = 1.8567415730337078, learning_rate = 7.1400036e-06, loss = 0.0006924549, step = 1322 (5.302 sec) 2023-02-20 14:18:16,087 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.132 INFO:tensorflow:global_step/sec: 6.03656 2023-02-20 14:18:17,082 [INFO] tensorflow: global_step/sec: 6.03656 INFO:tensorflow:global_step/sec: 6.03617 2023-02-20 14:18:19,401 [INFO] tensorflow: global_step/sec: 6.03617 2023-02-20 14:18:20,230 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.142 INFO:tensorflow:epoch = 1.901685393258427, learning_rate = 7.201846e-06, loss = 0.0006627094, step = 1354 (5.301 sec) 2023-02-20 14:18:21,057 [INFO] tensorflow: epoch = 1.901685393258427, learning_rate = 7.201846e-06, loss = 0.0006627094, step = 1354 (5.301 sec) INFO:tensorflow:global_step/sec: 6.03663 2023-02-20 14:18:21,721 [INFO] tensorflow: global_step/sec: 6.03663 INFO:tensorflow:global_step/sec: 6.04995 2023-02-20 14:18:24,035 [INFO] tensorflow: global_step/sec: 6.04995 2023-02-20 14:18:24,365 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.183 INFO:tensorflow:epoch = 1.946629213483146, learning_rate = 7.264224e-06, loss = 0.00061243767, step = 1386 (5.293 sec) 2023-02-20 14:18:26,350 [INFO] tensorflow: epoch = 1.946629213483146, learning_rate = 7.264224e-06, loss = 0.00061243767, step = 1386 (5.293 sec) INFO:tensorflow:global_step/sec: 6.04449 2023-02-20 14:18:26,351 [INFO] tensorflow: global_step/sec: 6.04449 2023-02-20 14:18:28,505 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.161 INFO:tensorflow:global_step/sec: 6.03395 2023-02-20 14:18:28,671 [INFO] tensorflow: global_step/sec: 6.03395 INFO:tensorflow:global_step/sec: 6.04897 2023-02-20 14:18:30,985 [INFO] tensorflow: global_step/sec: 6.04897 INFO:tensorflow:epoch = 1.9915730337078652, learning_rate = 7.327135e-06, loss = 0.000719743, step = 1418 (5.299 sec) 2023-02-20 14:18:31,648 [INFO] tensorflow: epoch = 1.9915730337078652, learning_rate = 7.327135e-06, loss = 0.000719743, step = 1418 (5.299 sec) 2023-02-20 14:18:32,649 [INFO] root: None 2023-02-20 14:18:32,649 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 2/120: loss: 0.00069 learning rate: 0.00001 Time taken: 0:01:57.904982 ETA: 3:51:52.787887 2023-02-20 14:18:32,649 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.128 INFO:tensorflow:global_step/sec: 6.01885 2023-02-20 14:18:33,311 [INFO] tensorflow: global_step/sec: 6.01885 INFO:tensorflow:global_step/sec: 6.03794 2023-02-20 14:18:35,630 [INFO] tensorflow: global_step/sec: 6.03794 2023-02-20 14:18:36,790 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.155 INFO:tensorflow:epoch = 2.0365168539325844, learning_rate = 7.3905985e-06, loss = 0.00084629684, step = 1450 (5.307 sec) 2023-02-20 14:18:36,955 [INFO] tensorflow: epoch = 2.0365168539325844, learning_rate = 7.3905985e-06, loss = 0.00084629684, step = 1450 (5.307 sec) INFO:tensorflow:global_step/sec: 6.03028 2023-02-20 14:18:37,952 [INFO] tensorflow: global_step/sec: 6.03028 INFO:tensorflow:global_step/sec: 6.04207 2023-02-20 14:18:40,269 [INFO] tensorflow: global_step/sec: 6.04207 2023-02-20 14:18:40,932 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.144 INFO:tensorflow:epoch = 2.081460674157303, learning_rate = 7.4546106e-06, loss = 0.00076313806, step = 1482 (5.298 sec) 2023-02-20 14:18:42,253 [INFO] tensorflow: epoch = 2.081460674157303, learning_rate = 7.4546106e-06, loss = 0.00076313806, step = 1482 (5.298 sec) INFO:tensorflow:global_step/sec: 6.04412 2023-02-20 14:18:42,585 [INFO] tensorflow: global_step/sec: 6.04412 INFO:tensorflow:global_step/sec: 6.03094 2023-02-20 14:18:44,907 [INFO] tensorflow: global_step/sec: 6.03094 2023-02-20 14:18:45,073 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.150 INFO:tensorflow:global_step/sec: 6.04908 2023-02-20 14:18:47,221 [INFO] tensorflow: global_step/sec: 6.04908 INFO:tensorflow:epoch = 2.1264044943820224, learning_rate = 7.519178e-06, loss = 0.0007720459, step = 1514 (5.298 sec) 2023-02-20 14:18:47,551 [INFO] tensorflow: epoch = 2.1264044943820224, learning_rate = 7.519178e-06, loss = 0.0007720459, step = 1514 (5.298 sec) 2023-02-20 14:18:49,207 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.195 INFO:tensorflow:global_step/sec: 6.03976 2023-02-20 14:18:49,539 [INFO] tensorflow: global_step/sec: 6.03976 INFO:tensorflow:global_step/sec: 6.03929 2023-02-20 14:18:51,857 [INFO] tensorflow: global_step/sec: 6.03929 INFO:tensorflow:epoch = 2.1713483146067416, learning_rate = 7.5843045e-06, loss = 0.00070182676, step = 1546 (5.300 sec) 2023-02-20 14:18:52,851 [INFO] tensorflow: epoch = 2.1713483146067416, learning_rate = 7.5843045e-06, loss = 0.00070182676, step = 1546 (5.300 sec) 2023-02-20 14:18:53,348 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.149 INFO:tensorflow:global_step/sec: 6.03669 2023-02-20 14:18:54,176 [INFO] tensorflow: global_step/sec: 6.03669 INFO:tensorflow:global_step/sec: 6.02866 2023-02-20 14:18:56,498 [INFO] tensorflow: global_step/sec: 6.02866 2023-02-20 14:18:57,492 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.132 INFO:tensorflow:epoch = 2.216292134831461, learning_rate = 7.649995e-06, loss = 0.00074223755, step = 1578 (5.302 sec) 2023-02-20 14:18:58,154 [INFO] tensorflow: epoch = 2.216292134831461, learning_rate = 7.649995e-06, loss = 0.00074223755, step = 1578 (5.302 sec) INFO:tensorflow:global_step/sec: 6.02869 2023-02-20 14:18:58,821 [INFO] tensorflow: global_step/sec: 6.02869 INFO:tensorflow:global_step/sec: 6.04539 2023-02-20 14:19:01,136 [INFO] tensorflow: global_step/sec: 6.04539 2023-02-20 14:19:01,636 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.138 INFO:tensorflow:epoch = 2.2612359550561796, learning_rate = 7.716247e-06, loss = 0.0008791398, step = 1610 (5.302 sec) 2023-02-20 14:19:03,456 [INFO] tensorflow: epoch = 2.2612359550561796, learning_rate = 7.716247e-06, loss = 0.0008791398, step = 1610 (5.302 sec) INFO:tensorflow:global_step/sec: 6.03382 2023-02-20 14:19:03,457 [INFO] tensorflow: global_step/sec: 6.03382 INFO:tensorflow:global_step/sec: 6.03669 2023-02-20 14:19:05,776 [INFO] tensorflow: global_step/sec: 6.03669 2023-02-20 14:19:05,777 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.147 INFO:tensorflow:global_step/sec: 6.04474 2023-02-20 14:19:08,092 [INFO] tensorflow: global_step/sec: 6.04474 INFO:tensorflow:epoch = 2.306179775280899, learning_rate = 7.78308e-06, loss = 0.0007017899, step = 1642 (5.300 sec) 2023-02-20 14:19:08,756 [INFO] tensorflow: epoch = 2.306179775280899, learning_rate = 7.78308e-06, loss = 0.0007017899, step = 1642 (5.300 sec) 2023-02-20 14:19:09,915 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.167 INFO:tensorflow:global_step/sec: 6.03492 2023-02-20 14:19:10,412 [INFO] tensorflow: global_step/sec: 6.03492 INFO:tensorflow:global_step/sec: 6.03418 2023-02-20 14:19:12,732 [INFO] tensorflow: global_step/sec: 6.03418 INFO:tensorflow:epoch = 2.351123595505618, learning_rate = 7.850492e-06, loss = 0.0007241551, step = 1674 (5.298 sec) 2023-02-20 14:19:14,054 [INFO] tensorflow: epoch = 2.351123595505618, learning_rate = 7.850492e-06, loss = 0.0007241551, step = 1674 (5.298 sec) 2023-02-20 14:19:14,055 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.161 INFO:tensorflow:global_step/sec: 6.04961 2023-02-20 14:19:15,046 [INFO] tensorflow: global_step/sec: 6.04961 INFO:tensorflow:global_step/sec: 6.03678 2023-02-20 14:19:17,365 [INFO] tensorflow: global_step/sec: 6.03678 2023-02-20 14:19:18,195 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.152 INFO:tensorflow:epoch = 2.396067415730337, learning_rate = 7.918488e-06, loss = 0.00062493, step = 1706 (5.307 sec) 2023-02-20 14:19:19,361 [INFO] tensorflow: epoch = 2.396067415730337, learning_rate = 7.918488e-06, loss = 0.00062493, step = 1706 (5.307 sec) INFO:tensorflow:global_step/sec: 6.0095 2023-02-20 14:19:19,695 [INFO] tensorflow: global_step/sec: 6.0095 INFO:tensorflow:global_step/sec: 6.04727 2023-02-20 14:19:22,010 [INFO] tensorflow: global_step/sec: 6.04727 2023-02-20 14:19:22,342 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.118 INFO:tensorflow:global_step/sec: 6.04065 2023-02-20 14:19:24,328 [INFO] tensorflow: global_step/sec: 6.04065 INFO:tensorflow:epoch = 2.441011235955056, learning_rate = 7.987073e-06, loss = 0.00065162405, step = 1738 (5.299 sec) 2023-02-20 14:19:24,661 [INFO] tensorflow: epoch = 2.441011235955056, learning_rate = 7.987073e-06, loss = 0.00065162405, step = 1738 (5.299 sec) 2023-02-20 14:19:26,491 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.106 INFO:tensorflow:global_step/sec: 6.00792 2023-02-20 14:19:26,658 [INFO] tensorflow: global_step/sec: 6.00792 INFO:tensorflow:global_step/sec: 6.02125 2023-02-20 14:19:28,983 [INFO] tensorflow: global_step/sec: 6.02125 INFO:tensorflow:epoch = 2.485955056179775, learning_rate = 8.056252e-06, loss = 0.0006878772, step = 1770 (5.315 sec) 2023-02-20 14:19:29,975 [INFO] tensorflow: epoch = 2.485955056179775, learning_rate = 8.056252e-06, loss = 0.0006878772, step = 1770 (5.315 sec) 2023-02-20 14:19:30,638 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.115 INFO:tensorflow:global_step/sec: 6.04322 2023-02-20 14:19:31,300 [INFO] tensorflow: global_step/sec: 6.04322 INFO:tensorflow:global_step/sec: 6.04862 2023-02-20 14:19:33,614 [INFO] tensorflow: global_step/sec: 6.04862 2023-02-20 14:19:34,776 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.166 INFO:tensorflow:epoch = 2.5308988764044944, learning_rate = 8.126023e-06, loss = 0.0006548204, step = 1802 (5.296 sec) 2023-02-20 14:19:35,272 [INFO] tensorflow: epoch = 2.5308988764044944, learning_rate = 8.126023e-06, loss = 0.0006548204, step = 1802 (5.296 sec) INFO:tensorflow:global_step/sec: 6.03617 2023-02-20 14:19:35,934 [INFO] tensorflow: global_step/sec: 6.03617 INFO:tensorflow:global_step/sec: 6.03526 2023-02-20 14:19:38,253 [INFO] tensorflow: global_step/sec: 6.03526 2023-02-20 14:19:38,914 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.173 INFO:tensorflow:epoch = 2.5758426966292136, learning_rate = 8.196405e-06, loss = 0.00062437926, step = 1834 (5.299 sec) 2023-02-20 14:19:40,570 [INFO] tensorflow: epoch = 2.5758426966292136, learning_rate = 8.196405e-06, loss = 0.00062437926, step = 1834 (5.299 sec) INFO:tensorflow:global_step/sec: 6.03934 2023-02-20 14:19:40,571 [INFO] tensorflow: global_step/sec: 6.03934 INFO:tensorflow:global_step/sec: 6.06235 2023-02-20 14:19:42,881 [INFO] tensorflow: global_step/sec: 6.06235 2023-02-20 14:19:43,046 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.202 INFO:tensorflow:global_step/sec: 6.06133 2023-02-20 14:19:45,190 [INFO] tensorflow: global_step/sec: 6.06133 INFO:tensorflow:epoch = 2.6207865168539324, learning_rate = 8.267398e-06, loss = 0.0006274468, step = 1866 (5.280 sec) 2023-02-20 14:19:45,851 [INFO] tensorflow: epoch = 2.6207865168539324, learning_rate = 8.267398e-06, loss = 0.0006274468, step = 1866 (5.280 sec) 2023-02-20 14:19:47,179 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.198 INFO:tensorflow:global_step/sec: 6.03292 2023-02-20 14:19:47,511 [INFO] tensorflow: global_step/sec: 6.03292 INFO:tensorflow:global_step/sec: 6.07513 2023-02-20 14:19:49,815 [INFO] tensorflow: global_step/sec: 6.07513 INFO:tensorflow:epoch = 2.6657303370786516, learning_rate = 8.339004e-06, loss = 0.0006284113, step = 1898 (5.283 sec) 2023-02-20 14:19:51,134 [INFO] tensorflow: epoch = 2.6657303370786516, learning_rate = 8.339004e-06, loss = 0.0006284113, step = 1898 (5.283 sec) 2023-02-20 14:19:51,299 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.275 INFO:tensorflow:global_step/sec: 6.06104 2023-02-20 14:19:52,125 [INFO] tensorflow: global_step/sec: 6.06104 INFO:tensorflow:global_step/sec: 6.06747 2023-02-20 14:19:54,433 [INFO] tensorflow: global_step/sec: 6.06747 2023-02-20 14:19:55,424 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.244 INFO:tensorflow:epoch = 2.710674157303371, learning_rate = 8.411232e-06, loss = 0.000625564, step = 1930 (5.280 sec) 2023-02-20 14:19:56,414 [INFO] tensorflow: epoch = 2.710674157303371, learning_rate = 8.411232e-06, loss = 0.000625564, step = 1930 (5.280 sec) INFO:tensorflow:global_step/sec: 6.05415 2023-02-20 14:19:56,745 [INFO] tensorflow: global_step/sec: 6.05415 INFO:tensorflow:global_step/sec: 6.07053 2023-02-20 14:19:59,051 [INFO] tensorflow: global_step/sec: 6.07053 2023-02-20 14:19:59,550 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.239 INFO:tensorflow:global_step/sec: 6.03141 2023-02-20 14:20:01,373 [INFO] tensorflow: global_step/sec: 6.03141 INFO:tensorflow:epoch = 2.75561797752809, learning_rate = 8.484085e-06, loss = 0.0006965685, step = 1962 (5.289 sec) 2023-02-20 14:20:01,704 [INFO] tensorflow: epoch = 2.75561797752809, learning_rate = 8.484085e-06, loss = 0.0006965685, step = 1962 (5.289 sec) INFO:tensorflow:global_step/sec: 6.06601 2023-02-20 14:20:03,680 [INFO] tensorflow: global_step/sec: 6.06601 2023-02-20 14:20:03,681 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.208 INFO:tensorflow:global_step/sec: 6.05937 2023-02-20 14:20:05,991 [INFO] tensorflow: global_step/sec: 6.05937 INFO:tensorflow:epoch = 2.800561797752809, learning_rate = 8.55756e-06, loss = 0.0006394651, step = 1994 (5.283 sec) 2023-02-20 14:20:06,986 [INFO] tensorflow: epoch = 2.800561797752809, learning_rate = 8.55756e-06, loss = 0.0006394651, step = 1994 (5.283 sec) 2023-02-20 14:20:07,812 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.211 INFO:tensorflow:global_step/sec: 6.04077 2023-02-20 14:20:08,309 [INFO] tensorflow: global_step/sec: 6.04077 INFO:tensorflow:global_step/sec: 6.06461 2023-02-20 14:20:10,617 [INFO] tensorflow: global_step/sec: 6.06461 2023-02-20 14:20:11,936 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.249 INFO:tensorflow:epoch = 2.845505617977528, learning_rate = 8.63168e-06, loss = 0.0006911514, step = 2026 (5.281 sec) 2023-02-20 14:20:12,267 [INFO] tensorflow: epoch = 2.845505617977528, learning_rate = 8.63168e-06, loss = 0.0006911514, step = 2026 (5.281 sec) INFO:tensorflow:global_step/sec: 6.06099 2023-02-20 14:20:12,927 [INFO] tensorflow: global_step/sec: 6.06099 INFO:tensorflow:global_step/sec: 6.0595 2023-02-20 14:20:15,237 [INFO] tensorflow: global_step/sec: 6.0595 2023-02-20 14:20:16,063 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.233 INFO:tensorflow:epoch = 2.890449438202247, learning_rate = 8.706442e-06, loss = 0.0006300831, step = 2058 (5.280 sec) 2023-02-20 14:20:17,547 [INFO] tensorflow: epoch = 2.890449438202247, learning_rate = 8.706442e-06, loss = 0.0006300831, step = 2058 (5.280 sec) INFO:tensorflow:global_step/sec: 6.05928 2023-02-20 14:20:17,548 [INFO] tensorflow: global_step/sec: 6.05928 INFO:tensorflow:global_step/sec: 6.07489 2023-02-20 14:20:19,852 [INFO] tensorflow: global_step/sec: 6.07489 2023-02-20 14:20:20,182 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.278 INFO:tensorflow:global_step/sec: 6.05808 2023-02-20 14:20:22,163 [INFO] tensorflow: global_step/sec: 6.05808 INFO:tensorflow:epoch = 2.9353932584269664, learning_rate = 8.781853e-06, loss = 0.00068887253, step = 2090 (5.276 sec) 2023-02-20 14:20:22,823 [INFO] tensorflow: epoch = 2.9353932584269664, learning_rate = 8.781853e-06, loss = 0.00068887253, step = 2090 (5.276 sec) 2023-02-20 14:20:24,311 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.224 INFO:tensorflow:global_step/sec: 6.0523 2023-02-20 14:20:24,477 [INFO] tensorflow: global_step/sec: 6.0523 INFO:tensorflow:global_step/sec: 6.05852 2023-02-20 14:20:26,787 [INFO] tensorflow: global_step/sec: 6.05852 INFO:tensorflow:epoch = 2.980337078651685, learning_rate = 8.857915e-06, loss = 0.00062035176, step = 2122 (5.292 sec) 2023-02-20 14:20:28,114 [INFO] tensorflow: epoch = 2.980337078651685, learning_rate = 8.857915e-06, loss = 0.00062035176, step = 2122 (5.292 sec) 2023-02-20 14:20:28,444 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.198 INFO:tensorflow:global_step/sec: 6.03837 2023-02-20 14:20:29,106 [INFO] tensorflow: global_step/sec: 6.03837 2023-02-20 14:20:30,429 [INFO] root: None 2023-02-20 14:20:30,429 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 3/120: loss: 0.00075 learning rate: 0.00001 Time taken: 0:01:57.781043 ETA: 3:49:40.382037 INFO:tensorflow:global_step/sec: 6.05033 2023-02-20 14:20:31,420 [INFO] tensorflow: global_step/sec: 6.05033 2023-02-20 14:20:32,574 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.213 INFO:tensorflow:epoch = 3.0252808988764044, learning_rate = 8.934637e-06, loss = 0.00070233457, step = 2154 (5.285 sec) 2023-02-20 14:20:33,399 [INFO] tensorflow: epoch = 3.0252808988764044, learning_rate = 8.934637e-06, loss = 0.00070233457, step = 2154 (5.285 sec) INFO:tensorflow:global_step/sec: 6.05733 2023-02-20 14:20:33,731 [INFO] tensorflow: global_step/sec: 6.05733 INFO:tensorflow:global_step/sec: 6.05612 2023-02-20 14:20:36,043 [INFO] tensorflow: global_step/sec: 6.05612 2023-02-20 14:20:36,706 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.202 INFO:tensorflow:global_step/sec: 6.05487 2023-02-20 14:20:38,355 [INFO] tensorflow: global_step/sec: 6.05487 INFO:tensorflow:epoch = 3.0702247191011236, learning_rate = 9.012015e-06, loss = 0.0006926184, step = 2186 (5.285 sec) 2023-02-20 14:20:38,684 [INFO] tensorflow: epoch = 3.0702247191011236, learning_rate = 9.012015e-06, loss = 0.0006926184, step = 2186 (5.285 sec) INFO:tensorflow:global_step/sec: 6.06507 2023-02-20 14:20:40,663 [INFO] tensorflow: global_step/sec: 6.06507 2023-02-20 14:20:40,830 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.254 INFO:tensorflow:global_step/sec: 6.06369 2023-02-20 14:20:42,972 [INFO] tensorflow: global_step/sec: 6.06369 INFO:tensorflow:epoch = 3.115168539325843, learning_rate = 9.09007e-06, loss = 0.0008806425, step = 2218 (5.278 sec) 2023-02-20 14:20:43,963 [INFO] tensorflow: epoch = 3.115168539325843, learning_rate = 9.09007e-06, loss = 0.0008806425, step = 2218 (5.278 sec) 2023-02-20 14:20:44,951 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.269 INFO:tensorflow:global_step/sec: 6.06096 2023-02-20 14:20:45,282 [INFO] tensorflow: global_step/sec: 6.06096 INFO:tensorflow:global_step/sec: 6.05817 2023-02-20 14:20:47,593 [INFO] tensorflow: global_step/sec: 6.05817 2023-02-20 14:20:49,075 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.250 INFO:tensorflow:epoch = 3.1601123595505616, learning_rate = 9.168803e-06, loss = 0.00068933313, step = 2250 (5.277 sec) 2023-02-20 14:20:49,240 [INFO] tensorflow: epoch = 3.1601123595505616, learning_rate = 9.168803e-06, loss = 0.00068933313, step = 2250 (5.277 sec) INFO:tensorflow:global_step/sec: 6.06709 2023-02-20 14:20:49,900 [INFO] tensorflow: global_step/sec: 6.06709 INFO:tensorflow:global_step/sec: 6.06337 2023-02-20 14:20:52,209 [INFO] tensorflow: global_step/sec: 6.06337 2023-02-20 14:20:53,202 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.229 INFO:tensorflow:epoch = 3.205056179775281, learning_rate = 9.248218e-06, loss = 0.0007825411, step = 2282 (5.280 sec) 2023-02-20 14:20:54,520 [INFO] tensorflow: epoch = 3.205056179775281, learning_rate = 9.248218e-06, loss = 0.0007825411, step = 2282 (5.280 sec) INFO:tensorflow:global_step/sec: 6.0547 2023-02-20 14:20:54,522 [INFO] tensorflow: global_step/sec: 6.0547 INFO:tensorflow:global_step/sec: 6.04508 2023-02-20 14:20:56,838 [INFO] tensorflow: global_step/sec: 6.04508 2023-02-20 14:20:57,334 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.206 INFO:tensorflow:global_step/sec: 6.05799 2023-02-20 14:20:59,148 [INFO] tensorflow: global_step/sec: 6.05799 INFO:tensorflow:epoch = 3.25, learning_rate = 9.328321e-06, loss = 0.0006292864, step = 2314 (5.287 sec) 2023-02-20 14:20:59,808 [INFO] tensorflow: epoch = 3.25, learning_rate = 9.328321e-06, loss = 0.0006292864, step = 2314 (5.287 sec) INFO:tensorflow:global_step/sec: 6.06792 2023-02-20 14:21:01,456 [INFO] tensorflow: global_step/sec: 6.06792 2023-02-20 14:21:01,457 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.257 INFO:tensorflow:global_step/sec: 6.04908 2023-02-20 14:21:03,770 [INFO] tensorflow: global_step/sec: 6.04908 INFO:tensorflow:epoch = 3.294943820224719, learning_rate = 9.4091165e-06, loss = 0.00063093536, step = 2346 (5.282 sec) 2023-02-20 14:21:05,089 [INFO] tensorflow: epoch = 3.294943820224719, learning_rate = 9.4091165e-06, loss = 0.00063093536, step = 2346 (5.282 sec) 2023-02-20 14:21:05,584 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.229 INFO:tensorflow:global_step/sec: 6.05333 2023-02-20 14:21:06,083 [INFO] tensorflow: global_step/sec: 6.05333 INFO:tensorflow:global_step/sec: 6.06963 2023-02-20 14:21:08,389 [INFO] tensorflow: global_step/sec: 6.06963 2023-02-20 14:21:09,708 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.250 INFO:tensorflow:epoch = 3.339887640449438, learning_rate = 9.490603e-06, loss = 0.0006459126, step = 2378 (5.285 sec) 2023-02-20 14:21:10,374 [INFO] tensorflow: epoch = 3.339887640449438, learning_rate = 9.490603e-06, loss = 0.0006459126, step = 2378 (5.285 sec) INFO:tensorflow:global_step/sec: 6.0477 2023-02-20 14:21:10,704 [INFO] tensorflow: global_step/sec: 6.0477 INFO:tensorflow:global_step/sec: 6.06106 2023-02-20 14:21:13,014 [INFO] tensorflow: global_step/sec: 6.06106 2023-02-20 14:21:13,840 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.207 INFO:tensorflow:global_step/sec: 6.05918 2023-02-20 14:21:15,325 [INFO] tensorflow: global_step/sec: 6.05918 INFO:tensorflow:epoch = 3.384831460674157, learning_rate = 9.572805e-06, loss = 0.0006322762, step = 2410 (5.282 sec) 2023-02-20 14:21:15,656 [INFO] tensorflow: epoch = 3.384831460674157, learning_rate = 9.572805e-06, loss = 0.0006322762, step = 2410 (5.282 sec) INFO:tensorflow:global_step/sec: 6.05674 2023-02-20 14:21:17,636 [INFO] tensorflow: global_step/sec: 6.05674 2023-02-20 14:21:17,969 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.221 INFO:tensorflow:global_step/sec: 6.05648 2023-02-20 14:21:19,948 [INFO] tensorflow: global_step/sec: 6.05648 INFO:tensorflow:epoch = 3.4297752808988764, learning_rate = 9.655718e-06, loss = 0.0006841825, step = 2442 (5.281 sec) 2023-02-20 14:21:20,937 [INFO] tensorflow: epoch = 3.4297752808988764, learning_rate = 9.655718e-06, loss = 0.0006841825, step = 2442 (5.281 sec) 2023-02-20 14:21:22,096 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.231 INFO:tensorflow:global_step/sec: 6.05348 2023-02-20 14:21:22,260 [INFO] tensorflow: global_step/sec: 6.05348 INFO:tensorflow:global_step/sec: 6.07624 2023-02-20 14:21:24,565 [INFO] tensorflow: global_step/sec: 6.07624 INFO:tensorflow:epoch = 3.4747191011235956, learning_rate = 9.73935e-06, loss = 0.00062670826, step = 2474 (5.276 sec) 2023-02-20 14:21:26,213 [INFO] tensorflow: epoch = 3.4747191011235956, learning_rate = 9.73935e-06, loss = 0.00062670826, step = 2474 (5.276 sec) 2023-02-20 14:21:26,213 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.290 INFO:tensorflow:global_step/sec: 6.06729 2023-02-20 14:21:26,872 [INFO] tensorflow: global_step/sec: 6.06729 INFO:tensorflow:global_step/sec: 6.05467 2023-02-20 14:21:29,184 [INFO] tensorflow: global_step/sec: 6.05467 2023-02-20 14:21:30,339 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.236 INFO:tensorflow:epoch = 3.5196629213483144, learning_rate = 9.823706e-06, loss = 0.0006542338, step = 2506 (5.281 sec) 2023-02-20 14:21:31,494 [INFO] tensorflow: epoch = 3.5196629213483144, learning_rate = 9.823706e-06, loss = 0.0006542338, step = 2506 (5.281 sec) INFO:tensorflow:global_step/sec: 6.05732 2023-02-20 14:21:31,496 [INFO] tensorflow: global_step/sec: 6.05732 INFO:tensorflow:global_step/sec: 6.06623 2023-02-20 14:21:33,803 [INFO] tensorflow: global_step/sec: 6.06623 2023-02-20 14:21:34,464 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.249 INFO:tensorflow:global_step/sec: 6.0604 2023-02-20 14:21:36,113 [INFO] tensorflow: global_step/sec: 6.0604 INFO:tensorflow:epoch = 3.5646067415730336, learning_rate = 9.908792e-06, loss = 0.0006386304, step = 2538 (5.279 sec) 2023-02-20 14:21:36,773 [INFO] tensorflow: epoch = 3.5646067415730336, learning_rate = 9.908792e-06, loss = 0.0006386304, step = 2538 (5.279 sec) INFO:tensorflow:global_step/sec: 6.06363 2023-02-20 14:21:38,422 [INFO] tensorflow: global_step/sec: 6.06363 2023-02-20 14:21:38,588 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.251 INFO:tensorflow:global_step/sec: 6.05423 2023-02-20 14:21:40,735 [INFO] tensorflow: global_step/sec: 6.05423 INFO:tensorflow:epoch = 3.609550561797753, learning_rate = 9.994606e-06, loss = 0.00076482876, step = 2570 (5.281 sec) 2023-02-20 14:21:42,054 [INFO] tensorflow: epoch = 3.609550561797753, learning_rate = 9.994606e-06, loss = 0.00076482876, step = 2570 (5.281 sec) 2023-02-20 14:21:42,714 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.234 INFO:tensorflow:global_step/sec: 6.06003 2023-02-20 14:21:43,045 [INFO] tensorflow: global_step/sec: 6.06003 INFO:tensorflow:global_step/sec: 6.05957 2023-02-20 14:21:45,355 [INFO] tensorflow: global_step/sec: 6.05957 2023-02-20 14:21:46,838 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.250 INFO:tensorflow:epoch = 3.654494382022472, learning_rate = 1.0081174e-05, loss = 0.0006806165, step = 2602 (5.279 sec) 2023-02-20 14:21:47,334 [INFO] tensorflow: epoch = 3.654494382022472, learning_rate = 1.0081174e-05, loss = 0.0006806165, step = 2602 (5.279 sec) INFO:tensorflow:global_step/sec: 6.06276 2023-02-20 14:21:47,664 [INFO] tensorflow: global_step/sec: 6.06276 INFO:tensorflow:global_step/sec: 6.07252 2023-02-20 14:21:49,970 [INFO] tensorflow: global_step/sec: 6.07252 2023-02-20 14:21:50,960 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.262 INFO:tensorflow:global_step/sec: 6.06227 2023-02-20 14:21:52,279 [INFO] tensorflow: global_step/sec: 6.06227 INFO:tensorflow:epoch = 3.6994382022471908, learning_rate = 1.0168491e-05, loss = 0.00066726934, step = 2634 (5.277 sec) 2023-02-20 14:21:52,611 [INFO] tensorflow: epoch = 3.6994382022471908, learning_rate = 1.0168491e-05, loss = 0.00066726934, step = 2634 (5.277 sec) INFO:tensorflow:global_step/sec: 6.06498 2023-02-20 14:21:54,588 [INFO] tensorflow: global_step/sec: 6.06498 2023-02-20 14:21:55,082 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.264 INFO:tensorflow:global_step/sec: 6.06768 2023-02-20 14:21:56,895 [INFO] tensorflow: global_step/sec: 6.06768 INFO:tensorflow:epoch = 3.74438202247191, learning_rate = 1.0256564e-05, loss = 0.0006331152, step = 2666 (5.274 sec) 2023-02-20 14:21:57,885 [INFO] tensorflow: epoch = 3.74438202247191, learning_rate = 1.0256564e-05, loss = 0.0006331152, step = 2666 (5.274 sec) INFO:tensorflow:global_step/sec: 6.05366 2023-02-20 14:21:59,208 [INFO] tensorflow: global_step/sec: 6.05366 2023-02-20 14:21:59,208 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.237 INFO:tensorflow:global_step/sec: 6.06012 2023-02-20 14:22:01,518 [INFO] tensorflow: global_step/sec: 6.06012 INFO:tensorflow:epoch = 3.789325842696629, learning_rate = 1.03453995e-05, loss = 0.0007059852, step = 2698 (5.279 sec) 2023-02-20 14:22:03,164 [INFO] tensorflow: epoch = 3.789325842696629, learning_rate = 1.03453995e-05, loss = 0.0007059852, step = 2698 (5.279 sec) 2023-02-20 14:22:03,329 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.268 INFO:tensorflow:global_step/sec: 6.06634 2023-02-20 14:22:03,826 [INFO] tensorflow: global_step/sec: 6.06634 INFO:tensorflow:global_step/sec: 6.0666 2023-02-20 14:22:06,133 [INFO] tensorflow: global_step/sec: 6.0666 2023-02-20 14:22:07,454 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.248 INFO:tensorflow:epoch = 3.8342696629213484, learning_rate = 1.0435006e-05, loss = 0.000693309, step = 2730 (5.278 sec) 2023-02-20 14:22:08,442 [INFO] tensorflow: epoch = 3.8342696629213484, learning_rate = 1.0435006e-05, loss = 0.000693309, step = 2730 (5.278 sec) INFO:tensorflow:global_step/sec: 6.06163 2023-02-20 14:22:08,443 [INFO] tensorflow: global_step/sec: 6.06163 INFO:tensorflow:global_step/sec: 6.06225 2023-02-20 14:22:10,752 [INFO] tensorflow: global_step/sec: 6.06225 2023-02-20 14:22:11,577 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.257 INFO:tensorflow:global_step/sec: 6.06335 2023-02-20 14:22:13,061 [INFO] tensorflow: global_step/sec: 6.06335 INFO:tensorflow:epoch = 3.879213483146067, learning_rate = 1.0525377e-05, loss = 0.0007490726, step = 2762 (5.280 sec) 2023-02-20 14:22:13,722 [INFO] tensorflow: epoch = 3.879213483146067, learning_rate = 1.0525377e-05, loss = 0.0007490726, step = 2762 (5.280 sec) INFO:tensorflow:global_step/sec: 6.04888 2023-02-20 14:22:15,376 [INFO] tensorflow: global_step/sec: 6.04888 2023-02-20 14:22:15,707 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.215 INFO:tensorflow:global_step/sec: 6.06394 2023-02-20 14:22:17,685 [INFO] tensorflow: global_step/sec: 6.06394 INFO:tensorflow:epoch = 3.9241573033707864, learning_rate = 1.0616541e-05, loss = 0.00069377734, step = 2794 (5.283 sec) 2023-02-20 14:22:19,005 [INFO] tensorflow: epoch = 3.9241573033707864, learning_rate = 1.0616541e-05, loss = 0.00069377734, step = 2794 (5.283 sec) 2023-02-20 14:22:19,830 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.256 INFO:tensorflow:global_step/sec: 6.06069 2023-02-20 14:22:19,994 [INFO] tensorflow: global_step/sec: 6.06069 INFO:tensorflow:global_step/sec: 6.06767 2023-02-20 14:22:22,302 [INFO] tensorflow: global_step/sec: 6.06767 2023-02-20 14:22:23,957 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.229 INFO:tensorflow:epoch = 3.9691011235955056, learning_rate = 1.0708494e-05, loss = 0.000627422, step = 2826 (5.281 sec) 2023-02-20 14:22:24,287 [INFO] tensorflow: epoch = 3.9691011235955056, learning_rate = 1.0708494e-05, loss = 0.000627422, step = 2826 (5.281 sec) INFO:tensorflow:global_step/sec: 6.0466 2023-02-20 14:22:24,617 [INFO] tensorflow: global_step/sec: 6.0466 INFO:tensorflow:global_step/sec: 6.06873 2023-02-20 14:22:26,924 [INFO] tensorflow: global_step/sec: 6.06873 2023-02-20 14:22:27,916 [INFO] root: None 2023-02-20 14:22:27,916 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 4/120: loss: 0.00082 learning rate: 0.00001 Time taken: 0:01:57.487695 ETA: 3:47:08.572618 2023-02-20 14:22:28,081 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.254 INFO:tensorflow:global_step/sec: 6.05867 2023-02-20 14:22:29,235 [INFO] tensorflow: global_step/sec: 6.05867 INFO:tensorflow:epoch = 4.014044943820225, learning_rate = 1.0801244e-05, loss = 0.0007227327, step = 2858 (5.280 sec) 2023-02-20 14:22:29,566 [INFO] tensorflow: epoch = 4.014044943820225, learning_rate = 1.0801244e-05, loss = 0.0007227327, step = 2858 (5.280 sec) INFO:tensorflow:global_step/sec: 6.0508 2023-02-20 14:22:31,548 [INFO] tensorflow: global_step/sec: 6.0508 2023-02-20 14:22:32,210 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.221 INFO:tensorflow:global_step/sec: 6.05754 2023-02-20 14:22:33,860 [INFO] tensorflow: global_step/sec: 6.05754 INFO:tensorflow:epoch = 4.058988764044944, learning_rate = 1.0894799e-05, loss = 0.0007009201, step = 2890 (5.282 sec) 2023-02-20 14:22:34,848 [INFO] tensorflow: epoch = 4.058988764044944, learning_rate = 1.0894799e-05, loss = 0.0007009201, step = 2890 (5.282 sec) INFO:tensorflow:global_step/sec: 6.06011 2023-02-20 14:22:36,170 [INFO] tensorflow: global_step/sec: 6.06011 2023-02-20 14:22:36,337 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.231 INFO:tensorflow:global_step/sec: 6.06382 2023-02-20 14:22:38,479 [INFO] tensorflow: global_step/sec: 6.06382 INFO:tensorflow:epoch = 4.103932584269663, learning_rate = 1.0989163e-05, loss = 0.00062653795, step = 2922 (5.281 sec) 2023-02-20 14:22:40,130 [INFO] tensorflow: epoch = 4.103932584269663, learning_rate = 1.0989163e-05, loss = 0.00062653795, step = 2922 (5.281 sec) 2023-02-20 14:22:40,459 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.261 INFO:tensorflow:global_step/sec: 6.05983 2023-02-20 14:22:40,789 [INFO] tensorflow: global_step/sec: 6.05983 INFO:tensorflow:global_step/sec: 6.03897 2023-02-20 14:22:43,107 [INFO] tensorflow: global_step/sec: 6.03897 2023-02-20 14:22:44,592 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.197 INFO:tensorflow:epoch = 4.1488764044943816, learning_rate = 1.1084333e-05, loss = 0.0007121931, step = 2954 (5.290 sec) 2023-02-20 14:22:45,419 [INFO] tensorflow: epoch = 4.1488764044943816, learning_rate = 1.1084333e-05, loss = 0.0007121931, step = 2954 (5.290 sec) INFO:tensorflow:global_step/sec: 6.05274 2023-02-20 14:22:45,420 [INFO] tensorflow: global_step/sec: 6.05274 INFO:tensorflow:global_step/sec: 6.05297 2023-02-20 14:22:47,733 [INFO] tensorflow: global_step/sec: 6.05297 2023-02-20 14:22:48,722 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.216 INFO:tensorflow:global_step/sec: 6.05511 2023-02-20 14:22:50,045 [INFO] tensorflow: global_step/sec: 6.05511 INFO:tensorflow:epoch = 4.193820224719101, learning_rate = 1.1180338e-05, loss = 0.00076864124, step = 2986 (5.288 sec) 2023-02-20 14:22:50,707 [INFO] tensorflow: epoch = 4.193820224719101, learning_rate = 1.1180338e-05, loss = 0.00076864124, step = 2986 (5.288 sec) INFO:tensorflow:global_step/sec: 6.05497 2023-02-20 14:22:52,357 [INFO] tensorflow: global_step/sec: 6.05497 2023-02-20 14:22:52,853 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.206 INFO:tensorflow:global_step/sec: 6.04965 2023-02-20 14:22:54,672 [INFO] tensorflow: global_step/sec: 6.04965 INFO:tensorflow:epoch = 4.23876404494382, learning_rate = 1.1277175e-05, loss = 0.00064133346, step = 3018 (5.284 sec) 2023-02-20 14:22:55,990 [INFO] tensorflow: epoch = 4.23876404494382, learning_rate = 1.1277175e-05, loss = 0.00064133346, step = 3018 (5.284 sec) INFO:tensorflow:global_step/sec: 6.06545 2023-02-20 14:22:56,980 [INFO] tensorflow: global_step/sec: 6.06545 2023-02-20 14:22:56,980 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.233 INFO:tensorflow:global_step/sec: 6.05565 2023-02-20 14:22:59,292 [INFO] tensorflow: global_step/sec: 6.05565 2023-02-20 14:23:01,106 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.240 INFO:tensorflow:epoch = 4.283707865168539, learning_rate = 1.1374851e-05, loss = 0.0006239356, step = 3050 (5.282 sec) 2023-02-20 14:23:01,272 [INFO] tensorflow: epoch = 4.283707865168539, learning_rate = 1.1374851e-05, loss = 0.0006239356, step = 3050 (5.282 sec) INFO:tensorflow:global_step/sec: 6.0503 2023-02-20 14:23:01,606 [INFO] tensorflow: global_step/sec: 6.0503 INFO:tensorflow:global_step/sec: 6.05734 2023-02-20 14:23:03,917 [INFO] tensorflow: global_step/sec: 6.05734 2023-02-20 14:23:05,237 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.210 INFO:tensorflow:global_step/sec: 6.05964 2023-02-20 14:23:06,227 [INFO] tensorflow: global_step/sec: 6.05964 INFO:tensorflow:epoch = 4.328651685393258, learning_rate = 1.1473373e-05, loss = 0.0008132492, step = 3082 (5.286 sec) 2023-02-20 14:23:06,558 [INFO] tensorflow: epoch = 4.328651685393258, learning_rate = 1.1473373e-05, loss = 0.0008132492, step = 3082 (5.286 sec) INFO:tensorflow:global_step/sec: 6.05932 2023-02-20 14:23:08,538 [INFO] tensorflow: global_step/sec: 6.05932 2023-02-20 14:23:09,362 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.243 INFO:tensorflow:global_step/sec: 6.05718 2023-02-20 14:23:10,849 [INFO] tensorflow: global_step/sec: 6.05718 INFO:tensorflow:epoch = 4.373595505617978, learning_rate = 1.1572747e-05, loss = 0.0006440804, step = 3114 (5.282 sec) 2023-02-20 14:23:11,840 [INFO] tensorflow: epoch = 4.373595505617978, learning_rate = 1.1572747e-05, loss = 0.0006440804, step = 3114 (5.282 sec) INFO:tensorflow:global_step/sec: 6.05063 2023-02-20 14:23:13,163 [INFO] tensorflow: global_step/sec: 6.05063 2023-02-20 14:23:13,495 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.202 INFO:tensorflow:global_step/sec: 6.06322 2023-02-20 14:23:15,472 [INFO] tensorflow: global_step/sec: 6.06322 INFO:tensorflow:epoch = 4.418539325842697, learning_rate = 1.16729725e-05, loss = 0.0007289707, step = 3146 (5.282 sec) 2023-02-20 14:23:17,122 [INFO] tensorflow: epoch = 4.418539325842697, learning_rate = 1.16729725e-05, loss = 0.0007289707, step = 3146 (5.282 sec) 2023-02-20 14:23:17,617 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.262 INFO:tensorflow:global_step/sec: 6.0607 2023-02-20 14:23:17,782 [INFO] tensorflow: global_step/sec: 6.0607 INFO:tensorflow:global_step/sec: 6.06585 2023-02-20 14:23:20,090 [INFO] tensorflow: global_step/sec: 6.06585 2023-02-20 14:23:21,741 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.246 INFO:tensorflow:epoch = 4.463483146067416, learning_rate = 1.1774077e-05, loss = 0.0006753914, step = 3178 (5.280 sec) 2023-02-20 14:23:22,402 [INFO] tensorflow: epoch = 4.463483146067416, learning_rate = 1.1774077e-05, loss = 0.0006753914, step = 3178 (5.280 sec) INFO:tensorflow:global_step/sec: 6.05184 2023-02-20 14:23:22,403 [INFO] tensorflow: global_step/sec: 6.05184 INFO:tensorflow:global_step/sec: 6.05802 2023-02-20 14:23:24,714 [INFO] tensorflow: global_step/sec: 6.05802 2023-02-20 14:23:25,866 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.245 INFO:tensorflow:global_step/sec: 6.06315 2023-02-20 14:23:27,023 [INFO] tensorflow: global_step/sec: 6.06315 INFO:tensorflow:epoch = 4.508426966292134, learning_rate = 1.1876056e-05, loss = 0.0006876475, step = 3210 (5.281 sec) 2023-02-20 14:23:27,683 [INFO] tensorflow: epoch = 4.508426966292134, learning_rate = 1.1876056e-05, loss = 0.0006876475, step = 3210 (5.281 sec) INFO:tensorflow:global_step/sec: 6.06128 2023-02-20 14:23:29,333 [INFO] tensorflow: global_step/sec: 6.06128 2023-02-20 14:23:29,992 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.237 INFO:tensorflow:global_step/sec: 6.05456 2023-02-20 14:23:31,645 [INFO] tensorflow: global_step/sec: 6.05456 INFO:tensorflow:epoch = 4.553370786516854, learning_rate = 1.1978919e-05, loss = 0.00062513305, step = 3242 (5.282 sec) 2023-02-20 14:23:32,965 [INFO] tensorflow: epoch = 4.553370786516854, learning_rate = 1.1978919e-05, loss = 0.00062513305, step = 3242 (5.282 sec) INFO:tensorflow:global_step/sec: 6.0597 2023-02-20 14:23:33,955 [INFO] tensorflow: global_step/sec: 6.0597 2023-02-20 14:23:34,122 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.219 INFO:tensorflow:global_step/sec: 6.06001 2023-02-20 14:23:36,266 [INFO] tensorflow: global_step/sec: 6.06001 INFO:tensorflow:epoch = 4.598314606741573, learning_rate = 1.20826735e-05, loss = 0.0007741887, step = 3274 (5.278 sec) 2023-02-20 14:23:38,243 [INFO] tensorflow: epoch = 4.598314606741573, learning_rate = 1.20826735e-05, loss = 0.0007741887, step = 3274 (5.278 sec) 2023-02-20 14:23:38,244 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.263 INFO:tensorflow:global_step/sec: 6.06235 2023-02-20 14:23:38,575 [INFO] tensorflow: global_step/sec: 6.06235 INFO:tensorflow:global_step/sec: 6.0643 2023-02-20 14:23:40,884 [INFO] tensorflow: global_step/sec: 6.0643 2023-02-20 14:23:42,369 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.238 INFO:tensorflow:global_step/sec: 6.05722 2023-02-20 14:23:43,195 [INFO] tensorflow: global_step/sec: 6.05722 INFO:tensorflow:epoch = 4.643258426966292, learning_rate = 1.2187314e-05, loss = 0.0008060336, step = 3306 (5.283 sec) 2023-02-20 14:23:43,526 [INFO] tensorflow: epoch = 4.643258426966292, learning_rate = 1.2187314e-05, loss = 0.0008060336, step = 3306 (5.283 sec) INFO:tensorflow:global_step/sec: 6.04834 2023-02-20 14:23:45,510 [INFO] tensorflow: global_step/sec: 6.04834 2023-02-20 14:23:46,498 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.221 INFO:tensorflow:global_step/sec: 6.06971 2023-02-20 14:23:47,816 [INFO] tensorflow: global_step/sec: 6.06971 INFO:tensorflow:epoch = 4.688202247191011, learning_rate = 1.2292873e-05, loss = 0.00075097906, step = 3338 (5.280 sec) 2023-02-20 14:23:48,806 [INFO] tensorflow: epoch = 4.688202247191011, learning_rate = 1.2292873e-05, loss = 0.00075097906, step = 3338 (5.280 sec) INFO:tensorflow:global_step/sec: 6.06058 2023-02-20 14:23:50,126 [INFO] tensorflow: global_step/sec: 6.06058 2023-02-20 14:23:50,620 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.263 INFO:tensorflow:global_step/sec: 6.06901 2023-02-20 14:23:52,433 [INFO] tensorflow: global_step/sec: 6.06901 INFO:tensorflow:epoch = 4.73314606741573, learning_rate = 1.2399346e-05, loss = 0.00062943425, step = 3370 (5.278 sec) 2023-02-20 14:23:54,084 [INFO] tensorflow: epoch = 4.73314606741573, learning_rate = 1.2399346e-05, loss = 0.00062943425, step = 3370 (5.278 sec) INFO:tensorflow:global_step/sec: 6.05409 2023-02-20 14:23:54,745 [INFO] tensorflow: global_step/sec: 6.05409 2023-02-20 14:23:54,746 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.238 INFO:tensorflow:global_step/sec: 6.06844 2023-02-20 14:23:57,052 [INFO] tensorflow: global_step/sec: 6.06844 2023-02-20 14:23:58,865 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.282 INFO:tensorflow:epoch = 4.77808988764045, learning_rate = 1.2506741e-05, loss = 0.00077516184, step = 3402 (5.275 sec) 2023-02-20 14:23:59,359 [INFO] tensorflow: epoch = 4.77808988764045, learning_rate = 1.2506741e-05, loss = 0.00077516184, step = 3402 (5.275 sec) INFO:tensorflow:global_step/sec: 6.06654 2023-02-20 14:23:59,360 [INFO] tensorflow: global_step/sec: 6.06654 INFO:tensorflow:global_step/sec: 6.05846 2023-02-20 14:24:01,671 [INFO] tensorflow: global_step/sec: 6.05846 2023-02-20 14:24:02,991 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.239 INFO:tensorflow:global_step/sec: 6.0605 2023-02-20 14:24:03,981 [INFO] tensorflow: global_step/sec: 6.0605 INFO:tensorflow:epoch = 4.823033707865169, learning_rate = 1.2615067e-05, loss = 0.0006307417, step = 3434 (5.282 sec) 2023-02-20 14:24:04,641 [INFO] tensorflow: epoch = 4.823033707865169, learning_rate = 1.2615067e-05, loss = 0.0006307417, step = 3434 (5.282 sec) INFO:tensorflow:global_step/sec: 6.064 2023-02-20 14:24:06,290 [INFO] tensorflow: global_step/sec: 6.064 2023-02-20 14:24:07,115 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.248 INFO:tensorflow:global_step/sec: 6.05664 2023-02-20 14:24:08,601 [INFO] tensorflow: global_step/sec: 6.05664 INFO:tensorflow:epoch = 4.867977528089887, learning_rate = 1.272433e-05, loss = 0.00072308246, step = 3466 (5.281 sec) 2023-02-20 14:24:09,923 [INFO] tensorflow: epoch = 4.867977528089887, learning_rate = 1.272433e-05, loss = 0.00072308246, step = 3466 (5.281 sec) INFO:tensorflow:global_step/sec: 6.05732 2023-02-20 14:24:10,913 [INFO] tensorflow: global_step/sec: 6.05732 2023-02-20 14:24:11,243 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.228 INFO:tensorflow:global_step/sec: 6.05752 2023-02-20 14:24:13,224 [INFO] tensorflow: global_step/sec: 6.05752 INFO:tensorflow:epoch = 4.912921348314606, learning_rate = 1.2834541e-05, loss = 0.0007209304, step = 3498 (5.279 sec) 2023-02-20 14:24:15,202 [INFO] tensorflow: epoch = 4.912921348314606, learning_rate = 1.2834541e-05, loss = 0.0007209304, step = 3498 (5.279 sec) 2023-02-20 14:24:15,367 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.250 INFO:tensorflow:global_step/sec: 6.06108 2023-02-20 14:24:15,534 [INFO] tensorflow: global_step/sec: 6.06108 INFO:tensorflow:global_step/sec: 6.05684 2023-02-20 14:24:17,845 [INFO] tensorflow: global_step/sec: 6.05684 2023-02-20 14:24:19,496 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.222 INFO:tensorflow:global_step/sec: 6.05236 2023-02-20 14:24:20,158 [INFO] tensorflow: global_step/sec: 6.05236 INFO:tensorflow:epoch = 4.957865168539326, learning_rate = 1.2945693e-05, loss = 0.0007347187, step = 3530 (5.287 sec) 2023-02-20 14:24:20,490 [INFO] tensorflow: epoch = 4.957865168539326, learning_rate = 1.2945693e-05, loss = 0.0007347187, step = 3530 (5.287 sec) INFO:tensorflow:global_step/sec: 6.05542 2023-02-20 14:24:22,470 [INFO] tensorflow: global_step/sec: 6.05542 2023-02-20 14:24:23,625 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.219 INFO:tensorflow:global_step/sec: 6.05057 2023-02-20 14:24:24,784 [INFO] tensorflow: global_step/sec: 6.05057 2023-02-20 14:24:32,605 [INFO] root: None 2023-02-20 14:24:32,620 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 5/120: loss: 0.00062 learning rate: 0.00001 Time taken: 0:01:57.528665 ETA: 3:45:15.796510 INFO:tensorflow:epoch = 5.001404494382022, learning_rate = 1.3054309e-05, loss = 0.0007069, step = 3561 (12.296 sec) 2023-02-20 14:24:32,786 [INFO] tensorflow: epoch = 5.001404494382022, learning_rate = 1.3054309e-05, loss = 0.0007069, step = 3561 (12.296 sec) INFO:tensorflow:global_step/sec: 1.47523 2023-02-20 14:24:34,274 [INFO] tensorflow: global_step/sec: 1.47523 2023-02-20 14:24:34,936 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 8.842 INFO:tensorflow:global_step/sec: 6.05074 2023-02-20 14:24:36,588 [INFO] tensorflow: global_step/sec: 6.05074 INFO:tensorflow:epoch = 5.046348314606742, learning_rate = 1.3167366e-05, loss = 0.0006834601, step = 3593 (5.288 sec) 2023-02-20 14:24:38,074 [INFO] tensorflow: epoch = 5.046348314606742, learning_rate = 1.3167366e-05, loss = 0.0006834601, step = 3593 (5.288 sec) INFO:tensorflow:global_step/sec: 6.05239 2023-02-20 14:24:38,901 [INFO] tensorflow: global_step/sec: 6.05239 2023-02-20 14:24:39,066 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.212 INFO:tensorflow:global_step/sec: 6.05427 2023-02-20 14:24:41,213 [INFO] tensorflow: global_step/sec: 6.05427 2023-02-20 14:24:43,194 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.227 INFO:tensorflow:epoch = 5.091292134831461, learning_rate = 1.3281413e-05, loss = 0.0007188899, step = 3625 (5.285 sec) 2023-02-20 14:24:43,359 [INFO] tensorflow: epoch = 5.091292134831461, learning_rate = 1.3281413e-05, loss = 0.0007188899, step = 3625 (5.285 sec) INFO:tensorflow:global_step/sec: 6.05212 2023-02-20 14:24:43,527 [INFO] tensorflow: global_step/sec: 6.05212 INFO:tensorflow:global_step/sec: 6.0525 2023-02-20 14:24:45,840 [INFO] tensorflow: global_step/sec: 6.0525 2023-02-20 14:24:47,327 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.196 INFO:tensorflow:global_step/sec: 6.04211 2023-02-20 14:24:48,157 [INFO] tensorflow: global_step/sec: 6.04211 INFO:tensorflow:epoch = 5.13623595505618, learning_rate = 1.3396448e-05, loss = 0.00062247756, step = 3657 (5.293 sec) 2023-02-20 14:24:48,653 [INFO] tensorflow: epoch = 5.13623595505618, learning_rate = 1.3396448e-05, loss = 0.00062247756, step = 3657 (5.293 sec) INFO:tensorflow:global_step/sec: 6.05147 2023-02-20 14:24:50,470 [INFO] tensorflow: global_step/sec: 6.05147 2023-02-20 14:24:51,461 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.197 INFO:tensorflow:global_step/sec: 6.05586 2023-02-20 14:24:52,782 [INFO] tensorflow: global_step/sec: 6.05586 INFO:tensorflow:epoch = 5.181179775280898, learning_rate = 1.3512478e-05, loss = 0.0006741384, step = 3689 (5.287 sec) 2023-02-20 14:24:53,940 [INFO] tensorflow: epoch = 5.181179775280898, learning_rate = 1.3512478e-05, loss = 0.0006741384, step = 3689 (5.287 sec) INFO:tensorflow:global_step/sec: 6.04996 2023-02-20 14:24:55,096 [INFO] tensorflow: global_step/sec: 6.04996 2023-02-20 14:24:55,592 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.207 INFO:tensorflow:global_step/sec: 6.05561 2023-02-20 14:24:57,408 [INFO] tensorflow: global_step/sec: 6.05561 INFO:tensorflow:epoch = 5.226123595505618, learning_rate = 1.3629517e-05, loss = 0.0006620317, step = 3721 (5.284 sec) 2023-02-20 14:24:59,224 [INFO] tensorflow: epoch = 5.226123595505618, learning_rate = 1.3629517e-05, loss = 0.0006620317, step = 3721 (5.284 sec) INFO:tensorflow:global_step/sec: 6.05813 2023-02-20 14:24:59,719 [INFO] tensorflow: global_step/sec: 6.05813 2023-02-20 14:24:59,720 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.228 INFO:tensorflow:global_step/sec: 6.05513 2023-02-20 14:25:02,031 [INFO] tensorflow: global_step/sec: 6.05513 2023-02-20 14:25:03,848 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.223 INFO:tensorflow:global_step/sec: 6.04602 2023-02-20 14:25:04,347 [INFO] tensorflow: global_step/sec: 6.04602 INFO:tensorflow:epoch = 5.271067415730337, learning_rate = 1.3747566e-05, loss = 0.0007482467, step = 3753 (5.289 sec) 2023-02-20 14:25:04,513 [INFO] tensorflow: epoch = 5.271067415730337, learning_rate = 1.3747566e-05, loss = 0.0007482467, step = 3753 (5.289 sec) INFO:tensorflow:global_step/sec: 6.05506 2023-02-20 14:25:06,659 [INFO] tensorflow: global_step/sec: 6.05506 2023-02-20 14:25:07,980 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.203 INFO:tensorflow:global_step/sec: 6.05249 2023-02-20 14:25:08,972 [INFO] tensorflow: global_step/sec: 6.05249 INFO:tensorflow:epoch = 5.316011235955056, learning_rate = 1.38666255e-05, loss = 0.0007197628, step = 3785 (5.286 sec) 2023-02-20 14:25:09,799 [INFO] tensorflow: epoch = 5.316011235955056, learning_rate = 1.38666255e-05, loss = 0.0007197628, step = 3785 (5.286 sec) INFO:tensorflow:global_step/sec: 6.05135 2023-02-20 14:25:11,285 [INFO] tensorflow: global_step/sec: 6.05135 2023-02-20 14:25:12,113 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.201 INFO:tensorflow:global_step/sec: 6.05154 2023-02-20 14:25:13,599 [INFO] tensorflow: global_step/sec: 6.05154 INFO:tensorflow:epoch = 5.360955056179775, learning_rate = 1.3986729e-05, loss = 0.0006231534, step = 3817 (5.284 sec) 2023-02-20 14:25:15,083 [INFO] tensorflow: epoch = 5.360955056179775, learning_rate = 1.3986729e-05, loss = 0.0006231534, step = 3817 (5.284 sec) INFO:tensorflow:global_step/sec: 6.06251 2023-02-20 14:25:15,908 [INFO] tensorflow: global_step/sec: 6.06251 2023-02-20 14:25:16,238 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.243 INFO:tensorflow:global_step/sec: 6.05121 2023-02-20 14:25:18,222 [INFO] tensorflow: global_step/sec: 6.05121 INFO:tensorflow:epoch = 5.405898876404494, learning_rate = 1.4107874e-05, loss = 0.00065890444, step = 3849 (5.295 sec) 2023-02-20 14:25:20,378 [INFO] tensorflow: epoch = 5.405898876404494, learning_rate = 1.4107874e-05, loss = 0.00065890444, step = 3849 (5.295 sec) 2023-02-20 14:25:20,378 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.155 INFO:tensorflow:global_step/sec: 6.02996 2023-02-20 14:25:20,543 [INFO] tensorflow: global_step/sec: 6.02996 INFO:tensorflow:global_step/sec: 6.05071 2023-02-20 14:25:22,857 [INFO] tensorflow: global_step/sec: 6.05071 2023-02-20 14:25:24,508 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.216 INFO:tensorflow:global_step/sec: 6.05736 2023-02-20 14:25:25,168 [INFO] tensorflow: global_step/sec: 6.05736 INFO:tensorflow:epoch = 5.450842696629214, learning_rate = 1.4230068e-05, loss = 0.0006188302, step = 3881 (5.287 sec) 2023-02-20 14:25:25,665 [INFO] tensorflow: epoch = 5.450842696629214, learning_rate = 1.4230068e-05, loss = 0.0006188302, step = 3881 (5.287 sec) INFO:tensorflow:global_step/sec: 6.05795 2023-02-20 14:25:27,479 [INFO] tensorflow: global_step/sec: 6.05795 2023-02-20 14:25:28,638 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.215 INFO:tensorflow:global_step/sec: 6.04091 2023-02-20 14:25:29,797 [INFO] tensorflow: global_step/sec: 6.04091 INFO:tensorflow:epoch = 5.495786516853933, learning_rate = 1.435332e-05, loss = 0.0006463074, step = 3913 (5.288 sec) 2023-02-20 14:25:30,953 [INFO] tensorflow: epoch = 5.495786516853933, learning_rate = 1.435332e-05, loss = 0.0006463074, step = 3913 (5.288 sec) INFO:tensorflow:global_step/sec: 6.05303 2023-02-20 14:25:32,110 [INFO] tensorflow: global_step/sec: 6.05303 2023-02-20 14:25:32,771 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.196 INFO:tensorflow:global_step/sec: 6.05268 2023-02-20 14:25:34,423 [INFO] tensorflow: global_step/sec: 6.05268 INFO:tensorflow:epoch = 5.540730337078651, learning_rate = 1.44776395e-05, loss = 0.0007194592, step = 3945 (5.283 sec) 2023-02-20 14:25:36,236 [INFO] tensorflow: epoch = 5.540730337078651, learning_rate = 1.44776395e-05, loss = 0.0007194592, step = 3945 (5.283 sec) INFO:tensorflow:global_step/sec: 6.06415 2023-02-20 14:25:36,732 [INFO] tensorflow: global_step/sec: 6.06415 2023-02-20 14:25:36,897 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.239 INFO:tensorflow:global_step/sec: 6.05553 2023-02-20 14:25:39,043 [INFO] tensorflow: global_step/sec: 6.05553 2023-02-20 14:25:41,031 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.191 INFO:tensorflow:global_step/sec: 6.03944 2023-02-20 14:25:41,362 [INFO] tensorflow: global_step/sec: 6.03944 INFO:tensorflow:epoch = 5.58567415730337, learning_rate = 1.4603035e-05, loss = 0.00082161126, step = 3977 (5.291 sec) 2023-02-20 14:25:41,527 [INFO] tensorflow: epoch = 5.58567415730337, learning_rate = 1.4603035e-05, loss = 0.00082161126, step = 3977 (5.291 sec) INFO:tensorflow:global_step/sec: 6.05079 2023-02-20 14:25:43,675 [INFO] tensorflow: global_step/sec: 6.05079 2023-02-20 14:25:45,164 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.201 INFO:tensorflow:global_step/sec: 6.0494 2023-02-20 14:25:45,990 [INFO] tensorflow: global_step/sec: 6.0494 INFO:tensorflow:epoch = 5.63061797752809, learning_rate = 1.4729504e-05, loss = 0.0006246363, step = 4009 (5.294 sec) 2023-02-20 14:25:46,822 [INFO] tensorflow: epoch = 5.63061797752809, learning_rate = 1.4729504e-05, loss = 0.0006246363, step = 4009 (5.294 sec) INFO:tensorflow:global_step/sec: 6.01805 2023-02-20 14:25:48,316 [INFO] tensorflow: global_step/sec: 6.01805 2023-02-20 14:25:49,306 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.141 INFO:tensorflow:global_step/sec: 6.05259 2023-02-20 14:25:50,629 [INFO] tensorflow: global_step/sec: 6.05259 INFO:tensorflow:epoch = 5.675561797752809, learning_rate = 1.4857081e-05, loss = 0.00063704205, step = 4041 (5.291 sec) 2023-02-20 14:25:52,112 [INFO] tensorflow: epoch = 5.675561797752809, learning_rate = 1.4857081e-05, loss = 0.00063704205, step = 4041 (5.291 sec) INFO:tensorflow:global_step/sec: 6.06164 2023-02-20 14:25:52,939 [INFO] tensorflow: global_step/sec: 6.06164 2023-02-20 14:25:53,436 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.216 INFO:tensorflow:global_step/sec: 6.04922 2023-02-20 14:25:55,253 [INFO] tensorflow: global_step/sec: 6.04922 INFO:tensorflow:epoch = 5.720505617977528, learning_rate = 1.4985763e-05, loss = 0.00062185334, step = 4073 (5.287 sec) 2023-02-20 14:25:57,400 [INFO] tensorflow: epoch = 5.720505617977528, learning_rate = 1.4985763e-05, loss = 0.00062185334, step = 4073 (5.287 sec) INFO:tensorflow:global_step/sec: 6.05348 2023-02-20 14:25:57,566 [INFO] tensorflow: global_step/sec: 6.05348 2023-02-20 14:25:57,566 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.213 INFO:tensorflow:global_step/sec: 6.05028 2023-02-20 14:25:59,880 [INFO] tensorflow: global_step/sec: 6.05028 2023-02-20 14:26:01,699 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.198 INFO:tensorflow:global_step/sec: 6.04634 2023-02-20 14:26:02,195 [INFO] tensorflow: global_step/sec: 6.04634 INFO:tensorflow:epoch = 5.765449438202247, learning_rate = 1.5115561e-05, loss = 0.0008432205, step = 4105 (5.292 sec) 2023-02-20 14:26:02,691 [INFO] tensorflow: epoch = 5.765449438202247, learning_rate = 1.5115561e-05, loss = 0.0008432205, step = 4105 (5.292 sec) INFO:tensorflow:global_step/sec: 6.06352 2023-02-20 14:26:04,504 [INFO] tensorflow: global_step/sec: 6.06352 2023-02-20 14:26:05,824 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.245 INFO:tensorflow:global_step/sec: 6.05293 2023-02-20 14:26:06,817 [INFO] tensorflow: global_step/sec: 6.05293 INFO:tensorflow:epoch = 5.810393258426966, learning_rate = 1.5246483e-05, loss = 0.00061848934, step = 4137 (5.286 sec) 2023-02-20 14:26:07,977 [INFO] tensorflow: epoch = 5.810393258426966, learning_rate = 1.5246483e-05, loss = 0.00061848934, step = 4137 (5.286 sec) INFO:tensorflow:global_step/sec: 6.0452 2023-02-20 14:26:09,133 [INFO] tensorflow: global_step/sec: 6.0452 2023-02-20 14:26:09,959 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.187 INFO:tensorflow:global_step/sec: 6.05389 2023-02-20 14:26:11,445 [INFO] tensorflow: global_step/sec: 6.05389 INFO:tensorflow:epoch = 5.855337078651686, learning_rate = 1.5378524e-05, loss = 0.00073441875, step = 4169 (5.286 sec) 2023-02-20 14:26:13,263 [INFO] tensorflow: epoch = 5.855337078651686, learning_rate = 1.5378524e-05, loss = 0.00073441875, step = 4169 (5.286 sec) INFO:tensorflow:global_step/sec: 6.05123 2023-02-20 14:26:13,759 [INFO] tensorflow: global_step/sec: 6.05123 2023-02-20 14:26:14,091 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.204 INFO:tensorflow:global_step/sec: 6.04114 2023-02-20 14:26:16,076 [INFO] tensorflow: global_step/sec: 6.04114 2023-02-20 14:26:18,222 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.212 INFO:tensorflow:global_step/sec: 6.05825 2023-02-20 14:26:18,387 [INFO] tensorflow: global_step/sec: 6.05825 INFO:tensorflow:epoch = 5.900280898876404, learning_rate = 1.5511721e-05, loss = 0.0006306187, step = 4201 (5.290 sec) 2023-02-20 14:26:18,553 [INFO] tensorflow: epoch = 5.900280898876404, learning_rate = 1.5511721e-05, loss = 0.0006306187, step = 4201 (5.290 sec) INFO:tensorflow:global_step/sec: 6.05706 2023-02-20 14:26:20,699 [INFO] tensorflow: global_step/sec: 6.05706 2023-02-20 14:26:22,353 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.209 INFO:tensorflow:global_step/sec: 6.04393 2023-02-20 14:26:23,015 [INFO] tensorflow: global_step/sec: 6.04393 INFO:tensorflow:epoch = 5.945224719101123, learning_rate = 1.5646076e-05, loss = 0.0006800833, step = 4233 (5.289 sec) 2023-02-20 14:26:23,841 [INFO] tensorflow: epoch = 5.945224719101123, learning_rate = 1.5646076e-05, loss = 0.0006800833, step = 4233 (5.289 sec) INFO:tensorflow:global_step/sec: 6.06025 2023-02-20 14:26:25,325 [INFO] tensorflow: global_step/sec: 6.06025 2023-02-20 14:26:26,480 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.230 INFO:tensorflow:global_step/sec: 6.05916 2023-02-20 14:26:27,636 [INFO] tensorflow: global_step/sec: 6.05916 INFO:tensorflow:epoch = 5.990168539325842, learning_rate = 1.5781592e-05, loss = 0.0006622885, step = 4265 (5.280 sec) 2023-02-20 14:26:29,121 [INFO] tensorflow: epoch = 5.990168539325842, learning_rate = 1.5781592e-05, loss = 0.0006622885, step = 4265 (5.280 sec) INFO:tensorflow:global_step/sec: 6.05347 2023-02-20 14:26:29,948 [INFO] tensorflow: global_step/sec: 6.05347 2023-02-20 14:26:30,287 [INFO] root: None 2023-02-20 14:26:30,287 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 6/120: loss: 0.00065 learning rate: 0.00002 Time taken: 0:02:04.839720 ETA: 3:57:11.728108 2023-02-20 14:26:30,618 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.166 INFO:tensorflow:global_step/sec: 6.03921 2023-02-20 14:26:32,267 [INFO] tensorflow: global_step/sec: 6.03921 INFO:tensorflow:epoch = 6.035112359550562, learning_rate = 1.5918282e-05, loss = 0.00076737534, step = 4297 (5.297 sec) 2023-02-20 14:26:34,418 [INFO] tensorflow: epoch = 6.035112359550562, learning_rate = 1.5918282e-05, loss = 0.00076737534, step = 4297 (5.297 sec) INFO:tensorflow:global_step/sec: 6.04345 2023-02-20 14:26:34,583 [INFO] tensorflow: global_step/sec: 6.04345 2023-02-20 14:26:34,749 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.210 INFO:tensorflow:global_step/sec: 6.0597 2023-02-20 14:26:36,893 [INFO] tensorflow: global_step/sec: 6.0597 2023-02-20 14:26:38,876 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.233 INFO:tensorflow:global_step/sec: 6.04994 2023-02-20 14:26:39,208 [INFO] tensorflow: global_step/sec: 6.04994 INFO:tensorflow:epoch = 6.080056179775281, learning_rate = 1.6056156e-05, loss = 0.0006151105, step = 4329 (5.285 sec) 2023-02-20 14:26:39,702 [INFO] tensorflow: epoch = 6.080056179775281, learning_rate = 1.6056156e-05, loss = 0.0006151105, step = 4329 (5.285 sec) INFO:tensorflow:global_step/sec: 6.04801 2023-02-20 14:26:41,522 [INFO] tensorflow: global_step/sec: 6.04801 2023-02-20 14:26:43,008 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.207 INFO:tensorflow:global_step/sec: 6.05652 2023-02-20 14:26:43,834 [INFO] tensorflow: global_step/sec: 6.05652 INFO:tensorflow:epoch = 6.125, learning_rate = 1.6195208e-05, loss = 0.0006192234, step = 4361 (5.287 sec) 2023-02-20 14:26:44,989 [INFO] tensorflow: epoch = 6.125, learning_rate = 1.6195208e-05, loss = 0.0006192234, step = 4361 (5.287 sec) INFO:tensorflow:global_step/sec: 6.06228 2023-02-20 14:26:46,143 [INFO] tensorflow: global_step/sec: 6.06228 2023-02-20 14:26:47,138 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.214 INFO:tensorflow:global_step/sec: 6.04575 2023-02-20 14:26:48,459 [INFO] tensorflow: global_step/sec: 6.04575 INFO:tensorflow:epoch = 6.169943820224719, learning_rate = 1.6335482e-05, loss = 0.0006540829, step = 4393 (5.289 sec) 2023-02-20 14:26:50,278 [INFO] tensorflow: epoch = 6.169943820224719, learning_rate = 1.6335482e-05, loss = 0.0006540829, step = 4393 (5.289 sec) INFO:tensorflow:global_step/sec: 6.04501 2023-02-20 14:26:50,775 [INFO] tensorflow: global_step/sec: 6.04501 2023-02-20 14:26:51,270 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.201 INFO:tensorflow:global_step/sec: 6.056 2023-02-20 14:26:53,087 [INFO] tensorflow: global_step/sec: 6.056 INFO:tensorflow:global_step/sec: 6.04574 2023-02-20 14:26:55,402 [INFO] tensorflow: global_step/sec: 6.04574 2023-02-20 14:26:55,404 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.193 INFO:tensorflow:epoch = 6.214887640449438, learning_rate = 1.647697e-05, loss = 0.00072514644, step = 4425 (5.291 sec) 2023-02-20 14:26:55,569 [INFO] tensorflow: epoch = 6.214887640449438, learning_rate = 1.647697e-05, loss = 0.00072514644, step = 4425 (5.291 sec) INFO:tensorflow:global_step/sec: 6.04422 2023-02-20 14:26:57,719 [INFO] tensorflow: global_step/sec: 6.04422 2023-02-20 14:26:59,535 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.210 INFO:tensorflow:global_step/sec: 6.05624 2023-02-20 14:27:00,030 [INFO] tensorflow: global_step/sec: 6.05624 INFO:tensorflow:epoch = 6.259831460674157, learning_rate = 1.6619682e-05, loss = 0.0006313047, step = 4457 (5.289 sec) 2023-02-20 14:27:00,858 [INFO] tensorflow: epoch = 6.259831460674157, learning_rate = 1.6619682e-05, loss = 0.0006313047, step = 4457 (5.289 sec) INFO:tensorflow:global_step/sec: 6.04955 2023-02-20 14:27:02,344 [INFO] tensorflow: global_step/sec: 6.04955 2023-02-20 14:27:03,668 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.199 INFO:tensorflow:global_step/sec: 6.03454 2023-02-20 14:27:04,664 [INFO] tensorflow: global_step/sec: 6.03454 INFO:tensorflow:epoch = 6.304775280898876, learning_rate = 1.6763632e-05, loss = 0.00062378676, step = 4489 (5.294 sec) 2023-02-20 14:27:06,153 [INFO] tensorflow: epoch = 6.304775280898876, learning_rate = 1.6763632e-05, loss = 0.00062378676, step = 4489 (5.294 sec) INFO:tensorflow:global_step/sec: 6.03895 2023-02-20 14:27:06,983 [INFO] tensorflow: global_step/sec: 6.03895 2023-02-20 14:27:07,810 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.143 INFO:tensorflow:global_step/sec: 6.04567 2023-02-20 14:27:09,298 [INFO] tensorflow: global_step/sec: 6.04567 INFO:tensorflow:epoch = 6.349719101123595, learning_rate = 1.6908827e-05, loss = 0.0007088872, step = 4521 (5.291 sec) 2023-02-20 14:27:11,443 [INFO] tensorflow: epoch = 6.349719101123595, learning_rate = 1.6908827e-05, loss = 0.0007088872, step = 4521 (5.291 sec) INFO:tensorflow:global_step/sec: 6.05638 2023-02-20 14:27:11,610 [INFO] tensorflow: global_step/sec: 6.05638 2023-02-20 14:27:11,943 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.196 INFO:tensorflow:global_step/sec: 6.03364 2023-02-20 14:27:13,931 [INFO] tensorflow: global_step/sec: 6.03364 2023-02-20 14:27:16,082 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.167 INFO:tensorflow:global_step/sec: 6.04188 2023-02-20 14:27:16,248 [INFO] tensorflow: global_step/sec: 6.04188 INFO:tensorflow:epoch = 6.394662921348314, learning_rate = 1.7055265e-05, loss = 0.0007403327, step = 4553 (5.303 sec) 2023-02-20 14:27:16,746 [INFO] tensorflow: epoch = 6.394662921348314, learning_rate = 1.7055265e-05, loss = 0.0007403327, step = 4553 (5.303 sec) INFO:tensorflow:global_step/sec: 6.03277 2023-02-20 14:27:18,568 [INFO] tensorflow: global_step/sec: 6.03277 2023-02-20 14:27:20,220 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.167 INFO:tensorflow:global_step/sec: 6.05717 2023-02-20 14:27:20,880 [INFO] tensorflow: global_step/sec: 6.05717 INFO:tensorflow:epoch = 6.439606741573034, learning_rate = 1.7202987e-05, loss = 0.00069699885, step = 4585 (5.291 sec) 2023-02-20 14:27:22,038 [INFO] tensorflow: epoch = 6.439606741573034, learning_rate = 1.7202987e-05, loss = 0.00069699885, step = 4585 (5.291 sec) INFO:tensorflow:global_step/sec: 6.05016 2023-02-20 14:27:23,193 [INFO] tensorflow: global_step/sec: 6.05016 2023-02-20 14:27:24,351 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.208 INFO:tensorflow:global_step/sec: 6.05584 2023-02-20 14:27:25,505 [INFO] tensorflow: global_step/sec: 6.05584 INFO:tensorflow:epoch = 6.484550561797753, learning_rate = 1.7351987e-05, loss = 0.0007305511, step = 4617 (5.285 sec) 2023-02-20 14:27:27,323 [INFO] tensorflow: epoch = 6.484550561797753, learning_rate = 1.7351987e-05, loss = 0.0007305511, step = 4617 (5.285 sec) INFO:tensorflow:global_step/sec: 6.05164 2023-02-20 14:27:27,819 [INFO] tensorflow: global_step/sec: 6.05164 2023-02-20 14:27:28,481 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.214 INFO:tensorflow:global_step/sec: 6.04254 2023-02-20 14:27:30,136 [INFO] tensorflow: global_step/sec: 6.04254 INFO:tensorflow:global_step/sec: 6.05419 2023-02-20 14:27:32,448 [INFO] tensorflow: global_step/sec: 6.05419 INFO:tensorflow:epoch = 6.529494382022472, learning_rate = 1.7502281e-05, loss = 0.0007277253, step = 4649 (5.291 sec) 2023-02-20 14:27:32,614 [INFO] tensorflow: epoch = 6.529494382022472, learning_rate = 1.7502281e-05, loss = 0.0007277253, step = 4649 (5.291 sec) 2023-02-20 14:27:32,615 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.194 INFO:tensorflow:global_step/sec: 6.04967 2023-02-20 14:27:34,762 [INFO] tensorflow: global_step/sec: 6.04967 2023-02-20 14:27:36,748 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.195 INFO:tensorflow:global_step/sec: 6.04433 2023-02-20 14:27:37,078 [INFO] tensorflow: global_step/sec: 6.04433 INFO:tensorflow:epoch = 6.574438202247191, learning_rate = 1.7653874e-05, loss = 0.0006251802, step = 4681 (5.293 sec) 2023-02-20 14:27:37,907 [INFO] tensorflow: epoch = 6.574438202247191, learning_rate = 1.7653874e-05, loss = 0.0006251802, step = 4681 (5.293 sec) INFO:tensorflow:global_step/sec: 6.0503 2023-02-20 14:27:39,392 [INFO] tensorflow: global_step/sec: 6.0503 2023-02-20 14:27:40,887 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.164 INFO:tensorflow:global_step/sec: 6.03614 2023-02-20 14:27:41,712 [INFO] tensorflow: global_step/sec: 6.03614 INFO:tensorflow:epoch = 6.6193820224719095, learning_rate = 1.7806764e-05, loss = 0.000632491, step = 4713 (5.293 sec) 2023-02-20 14:27:43,201 [INFO] tensorflow: epoch = 6.6193820224719095, learning_rate = 1.7806764e-05, loss = 0.000632491, step = 4713 (5.293 sec) INFO:tensorflow:global_step/sec: 6.04465 2023-02-20 14:27:44,028 [INFO] tensorflow: global_step/sec: 6.04465 2023-02-20 14:27:45,023 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.179 INFO:tensorflow:global_step/sec: 6.04178 2023-02-20 14:27:46,345 [INFO] tensorflow: global_step/sec: 6.04178 INFO:tensorflow:epoch = 6.664325842696629, learning_rate = 1.7960996e-05, loss = 0.000676952, step = 4745 (5.296 sec) 2023-02-20 14:27:48,496 [INFO] tensorflow: epoch = 6.664325842696629, learning_rate = 1.7960996e-05, loss = 0.000676952, step = 4745 (5.296 sec) INFO:tensorflow:global_step/sec: 6.04344 2023-02-20 14:27:48,662 [INFO] tensorflow: global_step/sec: 6.04344 2023-02-20 14:27:49,157 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.189 INFO:tensorflow:global_step/sec: 6.04956 2023-02-20 14:27:50,976 [INFO] tensorflow: global_step/sec: 6.04956 INFO:tensorflow:global_step/sec: 6.05547 2023-02-20 14:27:53,288 [INFO] tensorflow: global_step/sec: 6.05547 2023-02-20 14:27:53,289 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.208 INFO:tensorflow:epoch = 6.709269662921348, learning_rate = 1.8116561e-05, loss = 0.0006195324, step = 4777 (5.287 sec) 2023-02-20 14:27:53,783 [INFO] tensorflow: epoch = 6.709269662921348, learning_rate = 1.8116561e-05, loss = 0.0006195324, step = 4777 (5.287 sec) INFO:tensorflow:global_step/sec: 6.05699 2023-02-20 14:27:55,599 [INFO] tensorflow: global_step/sec: 6.05699 2023-02-20 14:27:57,416 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.232 INFO:tensorflow:global_step/sec: 6.05607 2023-02-20 14:27:57,911 [INFO] tensorflow: global_step/sec: 6.05607 INFO:tensorflow:epoch = 6.754213483146067, learning_rate = 1.8273477e-05, loss = 0.00067246985, step = 4809 (5.286 sec) 2023-02-20 14:27:59,069 [INFO] tensorflow: epoch = 6.754213483146067, learning_rate = 1.8273477e-05, loss = 0.00067246985, step = 4809 (5.286 sec) INFO:tensorflow:global_step/sec: 6.04855 2023-02-20 14:28:00,226 [INFO] tensorflow: global_step/sec: 6.04855 2023-02-20 14:28:01,546 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.212 INFO:tensorflow:global_step/sec: 6.05267 2023-02-20 14:28:02,539 [INFO] tensorflow: global_step/sec: 6.05267 INFO:tensorflow:epoch = 6.799157303370786, learning_rate = 1.8431749e-05, loss = 0.0007026701, step = 4841 (5.286 sec) 2023-02-20 14:28:04,355 [INFO] tensorflow: epoch = 6.799157303370786, learning_rate = 1.8431749e-05, loss = 0.0007026701, step = 4841 (5.286 sec) INFO:tensorflow:global_step/sec: 6.05101 2023-02-20 14:28:04,852 [INFO] tensorflow: global_step/sec: 6.05101 2023-02-20 14:28:05,678 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.206 INFO:tensorflow:global_step/sec: 6.06101 2023-02-20 14:28:07,162 [INFO] tensorflow: global_step/sec: 6.06101 INFO:tensorflow:global_step/sec: 6.05317 2023-02-20 14:28:09,475 [INFO] tensorflow: global_step/sec: 6.05317 INFO:tensorflow:epoch = 6.844101123595506, learning_rate = 1.8591394e-05, loss = 0.00063266506, step = 4873 (5.287 sec) 2023-02-20 14:28:09,642 [INFO] tensorflow: epoch = 6.844101123595506, learning_rate = 1.8591394e-05, loss = 0.00063266506, step = 4873 (5.287 sec) 2023-02-20 14:28:09,808 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.215 INFO:tensorflow:global_step/sec: 6.04444 2023-02-20 14:28:11,791 [INFO] tensorflow: global_step/sec: 6.04444 2023-02-20 14:28:13,939 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.211 INFO:tensorflow:global_step/sec: 6.05331 2023-02-20 14:28:14,104 [INFO] tensorflow: global_step/sec: 6.05331 INFO:tensorflow:epoch = 6.889044943820225, learning_rate = 1.8752422e-05, loss = 0.0006724474, step = 4905 (5.288 sec) 2023-02-20 14:28:14,930 [INFO] tensorflow: epoch = 6.889044943820225, learning_rate = 1.8752422e-05, loss = 0.0006724474, step = 4905 (5.288 sec) INFO:tensorflow:global_step/sec: 6.04935 2023-02-20 14:28:16,418 [INFO] tensorflow: global_step/sec: 6.04935 2023-02-20 14:28:18,073 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.190 INFO:tensorflow:global_step/sec: 6.03881 2023-02-20 14:28:18,737 [INFO] tensorflow: global_step/sec: 6.03881 INFO:tensorflow:epoch = 6.933988764044944, learning_rate = 1.8914825e-05, loss = 0.00061747176, step = 4937 (5.295 sec) 2023-02-20 14:28:20,225 [INFO] tensorflow: epoch = 6.933988764044944, learning_rate = 1.8914825e-05, loss = 0.00061747176, step = 4937 (5.295 sec) INFO:tensorflow:global_step/sec: 6.04755 2023-02-20 14:28:21,051 [INFO] tensorflow: global_step/sec: 6.04755 2023-02-20 14:28:22,206 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.199 INFO:tensorflow:global_step/sec: 6.0591 2023-02-20 14:28:23,362 [INFO] tensorflow: global_step/sec: 6.0591 INFO:tensorflow:epoch = 6.978932584269662, learning_rate = 1.9078652e-05, loss = 0.00064717536, step = 4969 (5.285 sec) 2023-02-20 14:28:25,510 [INFO] tensorflow: epoch = 6.978932584269662, learning_rate = 1.9078652e-05, loss = 0.00064717536, step = 4969 (5.285 sec) INFO:tensorflow:global_step/sec: 6.05126 2023-02-20 14:28:25,676 [INFO] tensorflow: global_step/sec: 6.05126 2023-02-20 14:28:26,338 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.202 INFO:tensorflow:global_step/sec: 6.00387 2023-02-20 14:28:28,007 [INFO] tensorflow: global_step/sec: 6.00387 2023-02-20 14:28:28,008 [INFO] root: None 2023-02-20 14:28:28,009 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 7/120: loss: 0.00068 learning rate: 0.00002 Time taken: 0:01:57.719738 ETA: 3:41:42.330368 INFO:tensorflow:global_step/sec: 6.04314 2023-02-20 14:28:30,324 [INFO] tensorflow: global_step/sec: 6.04314 2023-02-20 14:28:30,491 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.081 INFO:tensorflow:epoch = 7.0238764044943816, learning_rate = 1.92439e-05, loss = 0.0008170311, step = 5001 (5.311 sec) 2023-02-20 14:28:30,821 [INFO] tensorflow: epoch = 7.0238764044943816, learning_rate = 1.92439e-05, loss = 0.0008170311, step = 5001 (5.311 sec) INFO:tensorflow:global_step/sec: 6.03623 2023-02-20 14:28:32,643 [INFO] tensorflow: global_step/sec: 6.03623 2023-02-20 14:28:34,625 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.194 INFO:tensorflow:global_step/sec: 6.05061 2023-02-20 14:28:34,957 [INFO] tensorflow: global_step/sec: 6.05061 INFO:tensorflow:epoch = 7.068820224719101, learning_rate = 1.941058e-05, loss = 0.00069821713, step = 5033 (5.296 sec) 2023-02-20 14:28:36,117 [INFO] tensorflow: epoch = 7.068820224719101, learning_rate = 1.941058e-05, loss = 0.00069821713, step = 5033 (5.296 sec) INFO:tensorflow:global_step/sec: 6.04387 2023-02-20 14:28:37,274 [INFO] tensorflow: global_step/sec: 6.04387 2023-02-20 14:28:38,761 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.177 INFO:tensorflow:global_step/sec: 6.05037 2023-02-20 14:28:39,588 [INFO] tensorflow: global_step/sec: 6.05037 INFO:tensorflow:epoch = 7.11376404494382, learning_rate = 1.9578702e-05, loss = 0.0006383697, step = 5065 (5.289 sec) 2023-02-20 14:28:41,406 [INFO] tensorflow: epoch = 7.11376404494382, learning_rate = 1.9578702e-05, loss = 0.0006383697, step = 5065 (5.289 sec) INFO:tensorflow:global_step/sec: 6.04825 2023-02-20 14:28:41,902 [INFO] tensorflow: global_step/sec: 6.04825 2023-02-20 14:28:42,894 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.196 INFO:tensorflow:global_step/sec: 6.05019 2023-02-20 14:28:44,216 [INFO] tensorflow: global_step/sec: 6.05019 INFO:tensorflow:global_step/sec: 6.04789 2023-02-20 14:28:46,531 [INFO] tensorflow: global_step/sec: 6.04789 INFO:tensorflow:epoch = 7.158707865168539, learning_rate = 1.9748262e-05, loss = 0.0006741505, step = 5097 (5.292 sec) 2023-02-20 14:28:46,698 [INFO] tensorflow: epoch = 7.158707865168539, learning_rate = 1.9748262e-05, loss = 0.0006741505, step = 5097 (5.292 sec) 2023-02-20 14:28:47,029 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.189 INFO:tensorflow:global_step/sec: 6.04113 2023-02-20 14:28:48,849 [INFO] tensorflow: global_step/sec: 6.04113 INFO:tensorflow:global_step/sec: 6.0564 2023-02-20 14:28:51,160 [INFO] tensorflow: global_step/sec: 6.0564 2023-02-20 14:28:51,161 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.202 INFO:tensorflow:epoch = 7.203651685393258, learning_rate = 1.9919307e-05, loss = 0.0007783923, step = 5129 (5.288 sec) 2023-02-20 14:28:51,987 [INFO] tensorflow: epoch = 7.203651685393258, learning_rate = 1.9919307e-05, loss = 0.0007783923, step = 5129 (5.288 sec) INFO:tensorflow:global_step/sec: 6.04998 2023-02-20 14:28:53,474 [INFO] tensorflow: global_step/sec: 6.04998 2023-02-20 14:28:55,292 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.207 INFO:tensorflow:global_step/sec: 6.04251 2023-02-20 14:28:55,791 [INFO] tensorflow: global_step/sec: 6.04251 INFO:tensorflow:epoch = 7.248595505617978, learning_rate = 2.0091835e-05, loss = 0.0007057291, step = 5161 (5.291 sec) 2023-02-20 14:28:57,278 [INFO] tensorflow: epoch = 7.248595505617978, learning_rate = 2.0091835e-05, loss = 0.0007057291, step = 5161 (5.291 sec) INFO:tensorflow:global_step/sec: 6.0541 2023-02-20 14:28:58,104 [INFO] tensorflow: global_step/sec: 6.0541 2023-02-20 14:28:59,426 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.193 INFO:tensorflow:global_step/sec: 6.04215 2023-02-20 14:29:00,421 [INFO] tensorflow: global_step/sec: 6.04215 INFO:tensorflow:epoch = 7.293539325842697, learning_rate = 2.026586e-05, loss = 0.00065692427, step = 5193 (5.291 sec) 2023-02-20 14:29:02,568 [INFO] tensorflow: epoch = 7.293539325842697, learning_rate = 2.026586e-05, loss = 0.00065692427, step = 5193 (5.291 sec) INFO:tensorflow:global_step/sec: 6.05226 2023-02-20 14:29:02,734 [INFO] tensorflow: global_step/sec: 6.05226 2023-02-20 14:29:03,560 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.190 INFO:tensorflow:global_step/sec: 6.04031 2023-02-20 14:29:05,052 [INFO] tensorflow: global_step/sec: 6.04031 INFO:tensorflow:global_step/sec: 6.04041 2023-02-20 14:29:07,369 [INFO] tensorflow: global_step/sec: 6.04041 2023-02-20 14:29:07,700 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.156 INFO:tensorflow:epoch = 7.338483146067415, learning_rate = 2.0441388e-05, loss = 0.0006175303, step = 5225 (5.299 sec) 2023-02-20 14:29:07,867 [INFO] tensorflow: epoch = 7.338483146067415, learning_rate = 2.0441388e-05, loss = 0.0006175303, step = 5225 (5.299 sec) INFO:tensorflow:global_step/sec: 6.0269 2023-02-20 14:29:09,692 [INFO] tensorflow: global_step/sec: 6.0269 2023-02-20 14:29:11,837 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.177 INFO:tensorflow:global_step/sec: 6.05816 2023-02-20 14:29:12,003 [INFO] tensorflow: global_step/sec: 6.05816 INFO:tensorflow:epoch = 7.383426966292134, learning_rate = 2.061844e-05, loss = 0.0006855989, step = 5257 (5.292 sec) 2023-02-20 14:29:13,159 [INFO] tensorflow: epoch = 7.383426966292134, learning_rate = 2.061844e-05, loss = 0.0006855989, step = 5257 (5.292 sec) INFO:tensorflow:global_step/sec: 6.05603 2023-02-20 14:29:14,315 [INFO] tensorflow: global_step/sec: 6.05603 2023-02-20 14:29:15,966 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.223 INFO:tensorflow:global_step/sec: 6.04942 2023-02-20 14:29:16,629 [INFO] tensorflow: global_step/sec: 6.04942 INFO:tensorflow:epoch = 7.428370786516854, learning_rate = 2.0797024e-05, loss = 0.0006153443, step = 5289 (5.289 sec) 2023-02-20 14:29:18,448 [INFO] tensorflow: epoch = 7.428370786516854, learning_rate = 2.0797024e-05, loss = 0.0006153443, step = 5289 (5.289 sec) INFO:tensorflow:global_step/sec: 6.04886 2023-02-20 14:29:18,944 [INFO] tensorflow: global_step/sec: 6.04886 2023-02-20 14:29:20,100 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.190 INFO:tensorflow:global_step/sec: 6.0573 2023-02-20 14:29:21,255 [INFO] tensorflow: global_step/sec: 6.0573 INFO:tensorflow:global_step/sec: 6.04815 2023-02-20 14:29:23,570 [INFO] tensorflow: global_step/sec: 6.04815 INFO:tensorflow:epoch = 7.473314606741573, learning_rate = 2.0977135e-05, loss = 0.00068659947, step = 5321 (5.288 sec) 2023-02-20 14:29:23,736 [INFO] tensorflow: epoch = 7.473314606741573, learning_rate = 2.0977135e-05, loss = 0.00068659947, step = 5321 (5.288 sec) 2023-02-20 14:29:24,232 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.202 INFO:tensorflow:global_step/sec: 6.05248 2023-02-20 14:29:25,883 [INFO] tensorflow: global_step/sec: 6.05248 INFO:tensorflow:global_step/sec: 6.04918 2023-02-20 14:29:28,197 [INFO] tensorflow: global_step/sec: 6.04918 2023-02-20 14:29:28,364 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.202 INFO:tensorflow:epoch = 7.518258426966292, learning_rate = 2.1158825e-05, loss = 0.0006515081, step = 5353 (5.290 sec) 2023-02-20 14:29:29,026 [INFO] tensorflow: epoch = 7.518258426966292, learning_rate = 2.1158825e-05, loss = 0.0006515081, step = 5353 (5.290 sec) INFO:tensorflow:global_step/sec: 6.02644 2023-02-20 14:29:30,520 [INFO] tensorflow: global_step/sec: 6.02644 2023-02-20 14:29:32,509 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.129 INFO:tensorflow:global_step/sec: 6.03088 2023-02-20 14:29:32,842 [INFO] tensorflow: global_step/sec: 6.03088 INFO:tensorflow:epoch = 7.563202247191011, learning_rate = 2.1342088e-05, loss = 0.00061667187, step = 5385 (5.305 sec) 2023-02-20 14:29:34,331 [INFO] tensorflow: epoch = 7.563202247191011, learning_rate = 2.1342088e-05, loss = 0.00061667187, step = 5385 (5.305 sec) INFO:tensorflow:global_step/sec: 6.04298 2023-02-20 14:29:35,158 [INFO] tensorflow: global_step/sec: 6.04298 2023-02-20 14:29:36,644 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.185 INFO:tensorflow:global_step/sec: 6.05011 2023-02-20 14:29:37,472 [INFO] tensorflow: global_step/sec: 6.05011 INFO:tensorflow:epoch = 7.60814606741573, learning_rate = 2.1526941e-05, loss = 0.000644413, step = 5417 (5.291 sec) 2023-02-20 14:29:39,622 [INFO] tensorflow: epoch = 7.60814606741573, learning_rate = 2.1526941e-05, loss = 0.000644413, step = 5417 (5.291 sec) INFO:tensorflow:global_step/sec: 6.03457 2023-02-20 14:29:39,792 [INFO] tensorflow: global_step/sec: 6.03457 2023-02-20 14:29:40,785 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.150 INFO:tensorflow:global_step/sec: 6.04501 2023-02-20 14:29:42,108 [INFO] tensorflow: global_step/sec: 6.04501 INFO:tensorflow:global_step/sec: 6.04831 2023-02-20 14:29:44,423 [INFO] tensorflow: global_step/sec: 6.04831 INFO:tensorflow:epoch = 7.65308988764045, learning_rate = 2.1713395e-05, loss = 0.00062185305, step = 5449 (5.300 sec) 2023-02-20 14:29:44,922 [INFO] tensorflow: epoch = 7.65308988764045, learning_rate = 2.1713395e-05, loss = 0.00062185305, step = 5449 (5.300 sec) 2023-02-20 14:29:44,923 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.172 INFO:tensorflow:global_step/sec: 6.04406 2023-02-20 14:29:46,739 [INFO] tensorflow: global_step/sec: 6.04406 INFO:tensorflow:global_step/sec: 6.0577 2023-02-20 14:29:49,051 [INFO] tensorflow: global_step/sec: 6.0577 2023-02-20 14:29:49,051 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.222 INFO:tensorflow:epoch = 7.698033707865168, learning_rate = 2.1901442e-05, loss = 0.0006651256, step = 5481 (5.287 sec) 2023-02-20 14:29:50,209 [INFO] tensorflow: epoch = 7.698033707865168, learning_rate = 2.1901442e-05, loss = 0.0006651256, step = 5481 (5.287 sec) INFO:tensorflow:global_step/sec: 6.04479 2023-02-20 14:29:51,367 [INFO] tensorflow: global_step/sec: 6.04479 2023-02-20 14:29:53,192 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.153 INFO:tensorflow:global_step/sec: 6.03077 2023-02-20 14:29:53,688 [INFO] tensorflow: global_step/sec: 6.03077 INFO:tensorflow:epoch = 7.742977528089887, learning_rate = 2.2091137e-05, loss = 0.0006529116, step = 5513 (5.298 sec) 2023-02-20 14:29:55,507 [INFO] tensorflow: epoch = 7.742977528089887, learning_rate = 2.2091137e-05, loss = 0.0006529116, step = 5513 (5.298 sec) INFO:tensorflow:global_step/sec: 6.04621 2023-02-20 14:29:56,003 [INFO] tensorflow: global_step/sec: 6.04621 2023-02-20 14:29:57,328 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.179 INFO:tensorflow:global_step/sec: 6.04161 2023-02-20 14:29:58,321 [INFO] tensorflow: global_step/sec: 6.04161 INFO:tensorflow:global_step/sec: 6.0527 2023-02-20 14:30:00,634 [INFO] tensorflow: global_step/sec: 6.0527 INFO:tensorflow:epoch = 7.787921348314606, learning_rate = 2.2282478e-05, loss = 0.00060951867, step = 5545 (5.294 sec) 2023-02-20 14:30:00,801 [INFO] tensorflow: epoch = 7.787921348314606, learning_rate = 2.2282478e-05, loss = 0.00060951867, step = 5545 (5.294 sec) 2023-02-20 14:30:01,461 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.197 INFO:tensorflow:global_step/sec: 6.04578 2023-02-20 14:30:02,949 [INFO] tensorflow: global_step/sec: 6.04578 INFO:tensorflow:global_step/sec: 6.04361 2023-02-20 14:30:05,266 [INFO] tensorflow: global_step/sec: 6.04361 2023-02-20 14:30:05,597 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.179 INFO:tensorflow:epoch = 7.832865168539326, learning_rate = 2.2475475e-05, loss = 0.0006161908, step = 5577 (5.293 sec) 2023-02-20 14:30:06,094 [INFO] tensorflow: epoch = 7.832865168539326, learning_rate = 2.2475475e-05, loss = 0.0006161908, step = 5577 (5.293 sec) INFO:tensorflow:global_step/sec: 6.05077 2023-02-20 14:30:07,580 [INFO] tensorflow: global_step/sec: 6.05077 2023-02-20 14:30:09,727 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.218 INFO:tensorflow:global_step/sec: 6.05114 2023-02-20 14:30:09,893 [INFO] tensorflow: global_step/sec: 6.05114 INFO:tensorflow:epoch = 7.877808988764045, learning_rate = 2.2670143e-05, loss = 0.000695784, step = 5609 (5.289 sec) 2023-02-20 14:30:11,383 [INFO] tensorflow: epoch = 7.877808988764045, learning_rate = 2.2670143e-05, loss = 0.000695784, step = 5609 (5.289 sec) INFO:tensorflow:global_step/sec: 6.0423 2023-02-20 14:30:12,210 [INFO] tensorflow: global_step/sec: 6.0423 2023-02-20 14:30:13,864 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.171 INFO:tensorflow:global_step/sec: 6.04481 2023-02-20 14:30:14,526 [INFO] tensorflow: global_step/sec: 6.04481 INFO:tensorflow:epoch = 7.922752808988764, learning_rate = 2.2866498e-05, loss = 0.0006843742, step = 5641 (5.297 sec) 2023-02-20 14:30:16,680 [INFO] tensorflow: epoch = 7.922752808988764, learning_rate = 2.2866498e-05, loss = 0.0006843742, step = 5641 (5.297 sec) INFO:tensorflow:global_step/sec: 6.0355 2023-02-20 14:30:16,846 [INFO] tensorflow: global_step/sec: 6.0355 2023-02-20 14:30:18,002 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.172 INFO:tensorflow:global_step/sec: 6.04406 2023-02-20 14:30:19,162 [INFO] tensorflow: global_step/sec: 6.04406 INFO:tensorflow:global_step/sec: 6.04494 2023-02-20 14:30:21,478 [INFO] tensorflow: global_step/sec: 6.04494 INFO:tensorflow:epoch = 7.967696629213483, learning_rate = 2.3064553e-05, loss = 0.00061251334, step = 5673 (5.295 sec) 2023-02-20 14:30:21,975 [INFO] tensorflow: epoch = 7.967696629213483, learning_rate = 2.3064553e-05, loss = 0.00061251334, step = 5673 (5.295 sec) 2023-02-20 14:30:22,140 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.166 INFO:tensorflow:global_step/sec: 6.04054 2023-02-20 14:30:23,796 [INFO] tensorflow: global_step/sec: 6.04054 2023-02-20 14:30:25,781 [INFO] root: None 2023-02-20 14:30:25,781 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 8/120: loss: 0.00061 learning rate: 0.00002 Time taken: 0:01:57.771625 ETA: 3:39:50.422005 INFO:tensorflow:global_step/sec: 6.046 2023-02-20 14:30:26,112 [INFO] tensorflow: global_step/sec: 6.046 2023-02-20 14:30:26,278 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.170 INFO:tensorflow:epoch = 8.012640449438202, learning_rate = 2.3264301e-05, loss = 0.0006798366, step = 5705 (5.294 sec) 2023-02-20 14:30:27,269 [INFO] tensorflow: epoch = 8.012640449438202, learning_rate = 2.3264301e-05, loss = 0.0006798366, step = 5705 (5.294 sec) INFO:tensorflow:global_step/sec: 6.04804 2023-02-20 14:30:28,426 [INFO] tensorflow: global_step/sec: 6.04804 2023-02-20 14:30:30,413 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.185 INFO:tensorflow:global_step/sec: 6.03879 2023-02-20 14:30:30,745 [INFO] tensorflow: global_step/sec: 6.03879 INFO:tensorflow:epoch = 8.057584269662922, learning_rate = 2.3465802e-05, loss = 0.0006144468, step = 5737 (5.293 sec) 2023-02-20 14:30:32,563 [INFO] tensorflow: epoch = 8.057584269662922, learning_rate = 2.3465802e-05, loss = 0.0006144468, step = 5737 (5.293 sec) INFO:tensorflow:global_step/sec: 6.04922 2023-02-20 14:30:33,059 [INFO] tensorflow: global_step/sec: 6.04922 2023-02-20 14:30:34,547 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.193 INFO:tensorflow:global_step/sec: 6.0487 2023-02-20 14:30:35,374 [INFO] tensorflow: global_step/sec: 6.0487 INFO:tensorflow:global_step/sec: 6.05763 2023-02-20 14:30:37,685 [INFO] tensorflow: global_step/sec: 6.05763 INFO:tensorflow:epoch = 8.10252808988764, learning_rate = 2.3669047e-05, loss = 0.0006438581, step = 5769 (5.288 sec) 2023-02-20 14:30:37,850 [INFO] tensorflow: epoch = 8.10252808988764, learning_rate = 2.3669047e-05, loss = 0.0006438581, step = 5769 (5.288 sec) 2023-02-20 14:30:38,675 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.226 INFO:tensorflow:global_step/sec: 6.03933 2023-02-20 14:30:40,003 [INFO] tensorflow: global_step/sec: 6.03933 INFO:tensorflow:global_step/sec: 6.04581 2023-02-20 14:30:42,318 [INFO] tensorflow: global_step/sec: 6.04581 2023-02-20 14:30:42,816 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.148 INFO:tensorflow:epoch = 8.14747191011236, learning_rate = 2.3874054e-05, loss = 0.0006496763, step = 5801 (5.298 sec) 2023-02-20 14:30:43,148 [INFO] tensorflow: epoch = 8.14747191011236, learning_rate = 2.3874054e-05, loss = 0.0006496763, step = 5801 (5.298 sec) INFO:tensorflow:global_step/sec: 6.03756 2023-02-20 14:30:44,637 [INFO] tensorflow: global_step/sec: 6.03756 INFO:tensorflow:global_step/sec: 6.04117 2023-02-20 14:30:46,955 [INFO] tensorflow: global_step/sec: 6.04117 2023-02-20 14:30:46,955 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.163 INFO:tensorflow:epoch = 8.192415730337078, learning_rate = 2.4080837e-05, loss = 0.000659537, step = 5833 (5.294 sec) 2023-02-20 14:30:48,443 [INFO] tensorflow: epoch = 8.192415730337078, learning_rate = 2.4080837e-05, loss = 0.000659537, step = 5833 (5.294 sec) INFO:tensorflow:global_step/sec: 6.04867 2023-02-20 14:30:49,269 [INFO] tensorflow: global_step/sec: 6.04867 2023-02-20 14:30:51,091 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.182 INFO:tensorflow:global_step/sec: 6.03826 2023-02-20 14:30:51,588 [INFO] tensorflow: global_step/sec: 6.03826 INFO:tensorflow:epoch = 8.237359550561797, learning_rate = 2.4289386e-05, loss = 0.0006105274, step = 5865 (5.292 sec) 2023-02-20 14:30:53,735 [INFO] tensorflow: epoch = 8.237359550561797, learning_rate = 2.4289386e-05, loss = 0.0006105274, step = 5865 (5.292 sec) INFO:tensorflow:global_step/sec: 6.0536 2023-02-20 14:30:53,900 [INFO] tensorflow: global_step/sec: 6.0536 2023-02-20 14:30:55,223 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.205 INFO:tensorflow:global_step/sec: 6.05301 2023-02-20 14:30:56,213 [INFO] tensorflow: global_step/sec: 6.05301 INFO:tensorflow:global_step/sec: 6.05427 2023-02-20 14:30:58,526 [INFO] tensorflow: global_step/sec: 6.05427 INFO:tensorflow:epoch = 8.282303370786517, learning_rate = 2.4499766e-05, loss = 0.0006521086, step = 5897 (5.288 sec) 2023-02-20 14:30:59,023 [INFO] tensorflow: epoch = 8.282303370786517, learning_rate = 2.4499766e-05, loss = 0.0006521086, step = 5897 (5.288 sec) 2023-02-20 14:30:59,353 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.212 INFO:tensorflow:global_step/sec: 6.04957 2023-02-20 14:31:00,840 [INFO] tensorflow: global_step/sec: 6.04957 INFO:tensorflow:global_step/sec: 6.0212 2023-02-20 14:31:03,165 [INFO] tensorflow: global_step/sec: 6.0212 2023-02-20 14:31:03,498 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.132 INFO:tensorflow:epoch = 8.327247191011235, learning_rate = 2.4711968e-05, loss = 0.0006386633, step = 5929 (5.299 sec) 2023-02-20 14:31:04,322 [INFO] tensorflow: epoch = 8.327247191011235, learning_rate = 2.4711968e-05, loss = 0.0006386633, step = 5929 (5.299 sec) INFO:tensorflow:global_step/sec: 6.04957 2023-02-20 14:31:05,480 [INFO] tensorflow: global_step/sec: 6.04957 2023-02-20 14:31:07,630 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.202 INFO:tensorflow:global_step/sec: 6.04451 2023-02-20 14:31:07,795 [INFO] tensorflow: global_step/sec: 6.04451 INFO:tensorflow:epoch = 8.372191011235955, learning_rate = 2.4926007e-05, loss = 0.00060933694, step = 5961 (5.288 sec) 2023-02-20 14:31:09,610 [INFO] tensorflow: epoch = 8.372191011235955, learning_rate = 2.4926007e-05, loss = 0.00060933694, step = 5961 (5.288 sec) INFO:tensorflow:global_step/sec: 6.05727 2023-02-20 14:31:10,107 [INFO] tensorflow: global_step/sec: 6.05727 2023-02-20 14:31:11,757 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.234 INFO:tensorflow:global_step/sec: 6.06129 2023-02-20 14:31:12,417 [INFO] tensorflow: global_step/sec: 6.06129 INFO:tensorflow:global_step/sec: 6.04966 2023-02-20 14:31:14,731 [INFO] tensorflow: global_step/sec: 6.04966 INFO:tensorflow:epoch = 8.417134831460674, learning_rate = 2.5141899e-05, loss = 0.00061329006, step = 5993 (5.287 sec) 2023-02-20 14:31:14,897 [INFO] tensorflow: epoch = 8.417134831460674, learning_rate = 2.5141899e-05, loss = 0.00061329006, step = 5993 (5.287 sec) 2023-02-20 14:31:15,888 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.209 INFO:tensorflow:global_step/sec: 6.05018 2023-02-20 14:31:17,045 [INFO] tensorflow: global_step/sec: 6.05018 INFO:tensorflow:global_step/sec: 6.04707 2023-02-20 14:31:19,360 [INFO] tensorflow: global_step/sec: 6.04707 2023-02-20 14:31:20,021 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.195 INFO:tensorflow:epoch = 8.462078651685394, learning_rate = 2.5359663e-05, loss = 0.0007254092, step = 6025 (5.290 sec) 2023-02-20 14:31:20,187 [INFO] tensorflow: epoch = 8.462078651685394, learning_rate = 2.5359663e-05, loss = 0.0007254092, step = 6025 (5.290 sec) INFO:tensorflow:global_step/sec: 6.05092 2023-02-20 14:31:21,674 [INFO] tensorflow: global_step/sec: 6.05092 INFO:tensorflow:global_step/sec: 6.04449 2023-02-20 14:31:23,990 [INFO] tensorflow: global_step/sec: 6.04449 2023-02-20 14:31:24,156 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.189 INFO:tensorflow:epoch = 8.507022471910112, learning_rate = 2.5579313e-05, loss = 0.000685206, step = 6057 (5.290 sec) 2023-02-20 14:31:25,477 [INFO] tensorflow: epoch = 8.507022471910112, learning_rate = 2.5579313e-05, loss = 0.000685206, step = 6057 (5.290 sec) INFO:tensorflow:global_step/sec: 6.05362 2023-02-20 14:31:26,302 [INFO] tensorflow: global_step/sec: 6.05362 2023-02-20 14:31:28,288 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.202 INFO:tensorflow:global_step/sec: 6.03932 2023-02-20 14:31:28,620 [INFO] tensorflow: global_step/sec: 6.03932 INFO:tensorflow:epoch = 8.551966292134832, learning_rate = 2.580084e-05, loss = 0.0006704955, step = 6089 (5.295 sec) 2023-02-20 14:31:30,772 [INFO] tensorflow: epoch = 8.551966292134832, learning_rate = 2.580084e-05, loss = 0.0006704955, step = 6089 (5.295 sec) INFO:tensorflow:global_step/sec: 6.04104 2023-02-20 14:31:30,938 [INFO] tensorflow: global_step/sec: 6.04104 2023-02-20 14:31:32,431 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.138 INFO:tensorflow:global_step/sec: 6.03414 2023-02-20 14:31:33,258 [INFO] tensorflow: global_step/sec: 6.03414 INFO:tensorflow:global_step/sec: 6.05855 2023-02-20 14:31:35,569 [INFO] tensorflow: global_step/sec: 6.05855 INFO:tensorflow:epoch = 8.59691011235955, learning_rate = 2.602431e-05, loss = 0.0006819298, step = 6121 (5.292 sec) 2023-02-20 14:31:36,064 [INFO] tensorflow: epoch = 8.59691011235955, learning_rate = 2.602431e-05, loss = 0.0006819298, step = 6121 (5.292 sec) 2023-02-20 14:31:36,560 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.220 INFO:tensorflow:global_step/sec: 6.0591 2023-02-20 14:31:37,879 [INFO] tensorflow: global_step/sec: 6.0591 INFO:tensorflow:global_step/sec: 6.04342 2023-02-20 14:31:40,196 [INFO] tensorflow: global_step/sec: 6.04342 2023-02-20 14:31:40,695 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.189 INFO:tensorflow:epoch = 8.641853932584269, learning_rate = 2.6249716e-05, loss = 0.0006893443, step = 6153 (5.292 sec) 2023-02-20 14:31:41,356 [INFO] tensorflow: epoch = 8.641853932584269, learning_rate = 2.6249716e-05, loss = 0.0006893443, step = 6153 (5.292 sec) INFO:tensorflow:global_step/sec: 6.04467 2023-02-20 14:31:42,512 [INFO] tensorflow: global_step/sec: 6.04467 INFO:tensorflow:global_step/sec: 6.04437 2023-02-20 14:31:44,828 [INFO] tensorflow: global_step/sec: 6.04437 2023-02-20 14:31:44,830 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.186 INFO:tensorflow:epoch = 8.686797752808989, learning_rate = 2.6477077e-05, loss = 0.0006502186, step = 6185 (5.290 sec) 2023-02-20 14:31:46,646 [INFO] tensorflow: epoch = 8.686797752808989, learning_rate = 2.6477077e-05, loss = 0.0006502186, step = 6185 (5.290 sec) INFO:tensorflow:global_step/sec: 6.04964 2023-02-20 14:31:47,143 [INFO] tensorflow: global_step/sec: 6.04964 2023-02-20 14:31:48,961 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.210 INFO:tensorflow:global_step/sec: 6.04619 2023-02-20 14:31:49,458 [INFO] tensorflow: global_step/sec: 6.04619 INFO:tensorflow:global_step/sec: 6.04274 2023-02-20 14:31:51,775 [INFO] tensorflow: global_step/sec: 6.04274 INFO:tensorflow:epoch = 8.731741573033707, learning_rate = 2.6706404e-05, loss = 0.00065696967, step = 6217 (5.296 sec) 2023-02-20 14:31:51,942 [INFO] tensorflow: epoch = 8.731741573033707, learning_rate = 2.6706404e-05, loss = 0.00065696967, step = 6217 (5.296 sec) 2023-02-20 14:31:53,098 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.171 INFO:tensorflow:global_step/sec: 6.05012 2023-02-20 14:31:54,089 [INFO] tensorflow: global_step/sec: 6.05012 INFO:tensorflow:global_step/sec: 6.05258 2023-02-20 14:31:56,402 [INFO] tensorflow: global_step/sec: 6.05258 INFO:tensorflow:epoch = 8.776685393258427, learning_rate = 2.6937692e-05, loss = 0.00061288604, step = 6249 (5.285 sec) 2023-02-20 14:31:57,226 [INFO] tensorflow: epoch = 8.776685393258427, learning_rate = 2.6937692e-05, loss = 0.00061288604, step = 6249 (5.285 sec) 2023-02-20 14:31:57,227 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.223 INFO:tensorflow:global_step/sec: 6.05639 2023-02-20 14:31:58,714 [INFO] tensorflow: global_step/sec: 6.05639 INFO:tensorflow:global_step/sec: 6.05411 2023-02-20 14:32:01,026 [INFO] tensorflow: global_step/sec: 6.05411 2023-02-20 14:32:01,357 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.210 INFO:tensorflow:epoch = 8.821629213483146, learning_rate = 2.7171009e-05, loss = 0.0006094512, step = 6281 (5.288 sec) 2023-02-20 14:32:02,514 [INFO] tensorflow: epoch = 8.821629213483146, learning_rate = 2.7171009e-05, loss = 0.0006094512, step = 6281 (5.288 sec) INFO:tensorflow:global_step/sec: 6.04656 2023-02-20 14:32:03,342 [INFO] tensorflow: global_step/sec: 6.04656 2023-02-20 14:32:05,486 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.223 INFO:tensorflow:global_step/sec: 6.05802 2023-02-20 14:32:05,652 [INFO] tensorflow: global_step/sec: 6.05802 INFO:tensorflow:epoch = 8.866573033707866, learning_rate = 2.7406346e-05, loss = 0.0007338632, step = 6313 (5.289 sec) 2023-02-20 14:32:07,803 [INFO] tensorflow: epoch = 8.866573033707866, learning_rate = 2.7406346e-05, loss = 0.0007338632, step = 6313 (5.289 sec) INFO:tensorflow:global_step/sec: 6.04422 2023-02-20 14:32:07,969 [INFO] tensorflow: global_step/sec: 6.04422 2023-02-20 14:32:09,622 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.179 INFO:tensorflow:global_step/sec: 6.03591 2023-02-20 14:32:10,288 [INFO] tensorflow: global_step/sec: 6.03591 INFO:tensorflow:global_step/sec: 6.04785 2023-02-20 14:32:12,603 [INFO] tensorflow: global_step/sec: 6.04785 INFO:tensorflow:epoch = 8.911516853932584, learning_rate = 2.7643724e-05, loss = 0.00067429163, step = 6345 (5.296 sec) 2023-02-20 14:32:13,099 [INFO] tensorflow: epoch = 8.911516853932584, learning_rate = 2.7643724e-05, loss = 0.00067429163, step = 6345 (5.296 sec) 2023-02-20 14:32:13,761 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.167 INFO:tensorflow:global_step/sec: 6.03708 2023-02-20 14:32:14,922 [INFO] tensorflow: global_step/sec: 6.03708 INFO:tensorflow:global_step/sec: 6.05897 2023-02-20 14:32:17,233 [INFO] tensorflow: global_step/sec: 6.05897 2023-02-20 14:32:17,897 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.180 INFO:tensorflow:epoch = 8.956460674157302, learning_rate = 2.7883156e-05, loss = 0.0007316895, step = 6377 (5.293 sec) 2023-02-20 14:32:18,392 [INFO] tensorflow: epoch = 8.956460674157302, learning_rate = 2.7883156e-05, loss = 0.0007316895, step = 6377 (5.293 sec) INFO:tensorflow:global_step/sec: 6.04738 2023-02-20 14:32:19,548 [INFO] tensorflow: global_step/sec: 6.04738 INFO:tensorflow:global_step/sec: 6.05168 2023-02-20 14:32:21,861 [INFO] tensorflow: global_step/sec: 6.05168 2023-02-20 14:32:22,027 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.214 2023-02-20 14:32:23,515 [INFO] root: None 2023-02-20 14:32:23,515 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 9/120: loss: 0.00069 learning rate: 0.00003 Time taken: 0:01:57.734667 ETA: 3:37:48.548044 INFO:tensorflow:epoch = 9.001404494382022, learning_rate = 2.8124661e-05, loss = 0.00060742785, step = 6409 (5.288 sec) 2023-02-20 14:32:23,681 [INFO] tensorflow: epoch = 9.001404494382022, learning_rate = 2.8124661e-05, loss = 0.00060742785, step = 6409 (5.288 sec) INFO:tensorflow:global_step/sec: 6.04435 2023-02-20 14:32:24,177 [INFO] tensorflow: global_step/sec: 6.04435 2023-02-20 14:32:26,160 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.198 INFO:tensorflow:global_step/sec: 6.05081 2023-02-20 14:32:26,491 [INFO] tensorflow: global_step/sec: 6.05081 INFO:tensorflow:global_step/sec: 6.05172 2023-02-20 14:32:28,804 [INFO] tensorflow: global_step/sec: 6.05172 INFO:tensorflow:epoch = 9.04634831460674, learning_rate = 2.8368233e-05, loss = 0.0006064688, step = 6441 (5.290 sec) 2023-02-20 14:32:28,970 [INFO] tensorflow: epoch = 9.04634831460674, learning_rate = 2.8368233e-05, loss = 0.0006064688, step = 6441 (5.290 sec) 2023-02-20 14:32:30,292 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.202 INFO:tensorflow:global_step/sec: 6.04344 2023-02-20 14:32:31,125 [INFO] tensorflow: global_step/sec: 6.04344 INFO:tensorflow:global_step/sec: 6.03735 2023-02-20 14:32:33,440 [INFO] tensorflow: global_step/sec: 6.03735 INFO:tensorflow:epoch = 9.09129213483146, learning_rate = 2.861394e-05, loss = 0.0006485408, step = 6473 (5.298 sec) 2023-02-20 14:32:34,268 [INFO] tensorflow: epoch = 9.09129213483146, learning_rate = 2.861394e-05, loss = 0.0006485408, step = 6473 (5.298 sec) 2023-02-20 14:32:34,434 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.145 INFO:tensorflow:global_step/sec: 6.03927 2023-02-20 14:32:35,758 [INFO] tensorflow: global_step/sec: 6.03927 INFO:tensorflow:global_step/sec: 6.05116 2023-02-20 14:32:38,072 [INFO] tensorflow: global_step/sec: 6.05116 2023-02-20 14:32:38,569 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.182 INFO:tensorflow:epoch = 9.13623595505618, learning_rate = 2.8861778e-05, loss = 0.00064827234, step = 6505 (5.293 sec) 2023-02-20 14:32:39,561 [INFO] tensorflow: epoch = 9.13623595505618, learning_rate = 2.8861778e-05, loss = 0.00064827234, step = 6505 (5.293 sec) INFO:tensorflow:global_step/sec: 6.04541 2023-02-20 14:32:40,387 [INFO] tensorflow: global_step/sec: 6.04541 INFO:tensorflow:global_step/sec: 6.04395 2023-02-20 14:32:42,704 [INFO] tensorflow: global_step/sec: 6.04395 2023-02-20 14:32:42,705 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.185 INFO:tensorflow:epoch = 9.1811797752809, learning_rate = 2.911176e-05, loss = 0.00072643324, step = 6537 (5.295 sec) 2023-02-20 14:32:44,856 [INFO] tensorflow: epoch = 9.1811797752809, learning_rate = 2.911176e-05, loss = 0.00072643324, step = 6537 (5.295 sec) INFO:tensorflow:global_step/sec: 6.03938 2023-02-20 14:32:45,022 [INFO] tensorflow: global_step/sec: 6.03938 2023-02-20 14:32:46,838 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.195 INFO:tensorflow:global_step/sec: 6.05423 2023-02-20 14:32:47,334 [INFO] tensorflow: global_step/sec: 6.05423 INFO:tensorflow:global_step/sec: 6.0491 2023-02-20 14:32:49,649 [INFO] tensorflow: global_step/sec: 6.0491 INFO:tensorflow:epoch = 9.226123595505618, learning_rate = 2.9363908e-05, loss = 0.0006063669, step = 6569 (5.289 sec) 2023-02-20 14:32:50,145 [INFO] tensorflow: epoch = 9.226123595505618, learning_rate = 2.9363908e-05, loss = 0.0006063669, step = 6569 (5.289 sec) 2023-02-20 14:32:50,970 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.201 INFO:tensorflow:global_step/sec: 6.05229 2023-02-20 14:32:51,962 [INFO] tensorflow: global_step/sec: 6.05229 INFO:tensorflow:global_step/sec: 6.041 2023-02-20 14:32:54,279 [INFO] tensorflow: global_step/sec: 6.041 2023-02-20 14:32:55,106 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.180 INFO:tensorflow:epoch = 9.271067415730338, learning_rate = 2.9618239e-05, loss = 0.00067419617, step = 6601 (5.291 sec) 2023-02-20 14:32:55,436 [INFO] tensorflow: epoch = 9.271067415730338, learning_rate = 2.9618239e-05, loss = 0.00067419617, step = 6601 (5.291 sec) INFO:tensorflow:global_step/sec: 6.04906 2023-02-20 14:32:56,594 [INFO] tensorflow: global_step/sec: 6.04906 INFO:tensorflow:global_step/sec: 6.04848 2023-02-20 14:32:58,908 [INFO] tensorflow: global_step/sec: 6.04848 2023-02-20 14:32:59,240 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.195 INFO:tensorflow:epoch = 9.316011235955056, learning_rate = 2.9874747e-05, loss = 0.0006058643, step = 6633 (5.292 sec) 2023-02-20 14:33:00,729 [INFO] tensorflow: epoch = 9.316011235955056, learning_rate = 2.9874747e-05, loss = 0.0006058643, step = 6633 (5.292 sec) INFO:tensorflow:global_step/sec: 6.04453 2023-02-20 14:33:01,225 [INFO] tensorflow: global_step/sec: 6.04453 2023-02-20 14:33:03,378 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.168 INFO:tensorflow:global_step/sec: 6.03309 2023-02-20 14:33:03,545 [INFO] tensorflow: global_step/sec: 6.03309 INFO:tensorflow:global_step/sec: 6.02048 2023-02-20 14:33:05,871 [INFO] tensorflow: global_step/sec: 6.02048 INFO:tensorflow:epoch = 9.360955056179774, learning_rate = 3.0133502e-05, loss = 0.00066152617, step = 6665 (5.308 sec) 2023-02-20 14:33:06,037 [INFO] tensorflow: epoch = 9.360955056179774, learning_rate = 3.0133502e-05, loss = 0.00066152617, step = 6665 (5.308 sec) 2023-02-20 14:33:07,524 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.122 INFO:tensorflow:global_step/sec: 6.04407 2023-02-20 14:33:08,187 [INFO] tensorflow: global_step/sec: 6.04407 INFO:tensorflow:global_step/sec: 6.0516 2023-02-20 14:33:10,500 [INFO] tensorflow: global_step/sec: 6.0516 INFO:tensorflow:epoch = 9.405898876404494, learning_rate = 3.0394498e-05, loss = 0.0006803571, step = 6697 (5.289 sec) 2023-02-20 14:33:11,326 [INFO] tensorflow: epoch = 9.405898876404494, learning_rate = 3.0394498e-05, loss = 0.0006803571, step = 6697 (5.289 sec) 2023-02-20 14:33:11,657 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.197 INFO:tensorflow:global_step/sec: 6.04822 2023-02-20 14:33:12,815 [INFO] tensorflow: global_step/sec: 6.04822 INFO:tensorflow:global_step/sec: 6.03645 2023-02-20 14:33:15,134 [INFO] tensorflow: global_step/sec: 6.03645 2023-02-20 14:33:15,797 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.154 INFO:tensorflow:epoch = 9.450842696629213, learning_rate = 3.0657757e-05, loss = 0.0006671229, step = 6729 (5.297 sec) 2023-02-20 14:33:16,624 [INFO] tensorflow: epoch = 9.450842696629213, learning_rate = 3.0657757e-05, loss = 0.0006671229, step = 6729 (5.297 sec) INFO:tensorflow:global_step/sec: 6.04615 2023-02-20 14:33:17,450 [INFO] tensorflow: global_step/sec: 6.04615 INFO:tensorflow:global_step/sec: 6.04866 2023-02-20 14:33:19,764 [INFO] tensorflow: global_step/sec: 6.04866 2023-02-20 14:33:19,931 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.195 INFO:tensorflow:epoch = 9.495786516853933, learning_rate = 3.0923293e-05, loss = 0.00064643455, step = 6761 (5.290 sec) 2023-02-20 14:33:21,914 [INFO] tensorflow: epoch = 9.495786516853933, learning_rate = 3.0923293e-05, loss = 0.00064643455, step = 6761 (5.290 sec) INFO:tensorflow:global_step/sec: 6.04732 2023-02-20 14:33:22,079 [INFO] tensorflow: global_step/sec: 6.04732 2023-02-20 14:33:24,065 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.192 INFO:tensorflow:global_step/sec: 6.04407 2023-02-20 14:33:24,396 [INFO] tensorflow: global_step/sec: 6.04407 INFO:tensorflow:global_step/sec: 6.03844 2023-02-20 14:33:26,714 [INFO] tensorflow: global_step/sec: 6.03844 INFO:tensorflow:epoch = 9.540730337078651, learning_rate = 3.1191135e-05, loss = 0.0006240263, step = 6793 (5.296 sec) 2023-02-20 14:33:27,210 [INFO] tensorflow: epoch = 9.540730337078651, learning_rate = 3.1191135e-05, loss = 0.0006240263, step = 6793 (5.296 sec) 2023-02-20 14:33:28,201 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.178 INFO:tensorflow:global_step/sec: 6.05097 2023-02-20 14:33:29,028 [INFO] tensorflow: global_step/sec: 6.05097 INFO:tensorflow:global_step/sec: 6.03114 2023-02-20 14:33:31,349 [INFO] tensorflow: global_step/sec: 6.03114 2023-02-20 14:33:32,341 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.158 INFO:tensorflow:epoch = 9.585674157303371, learning_rate = 3.1461263e-05, loss = 0.00060278986, step = 6825 (5.296 sec) 2023-02-20 14:33:32,506 [INFO] tensorflow: epoch = 9.585674157303371, learning_rate = 3.1461263e-05, loss = 0.00060278986, step = 6825 (5.296 sec) INFO:tensorflow:global_step/sec: 6.03419 2023-02-20 14:33:33,669 [INFO] tensorflow: global_step/sec: 6.03419 INFO:tensorflow:global_step/sec: 6.04949 2023-02-20 14:33:35,983 [INFO] tensorflow: global_step/sec: 6.04949 2023-02-20 14:33:36,481 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.157 INFO:tensorflow:epoch = 9.63061797752809, learning_rate = 3.1733758e-05, loss = 0.0007032074, step = 6857 (5.297 sec) 2023-02-20 14:33:37,803 [INFO] tensorflow: epoch = 9.63061797752809, learning_rate = 3.1733758e-05, loss = 0.0007032074, step = 6857 (5.297 sec) INFO:tensorflow:global_step/sec: 6.04323 2023-02-20 14:33:38,300 [INFO] tensorflow: global_step/sec: 6.04323 INFO:tensorflow:global_step/sec: 6.05154 2023-02-20 14:33:40,614 [INFO] tensorflow: global_step/sec: 6.05154 2023-02-20 14:33:40,614 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.195 INFO:tensorflow:global_step/sec: 6.05715 2023-02-20 14:33:42,925 [INFO] tensorflow: global_step/sec: 6.05715 INFO:tensorflow:epoch = 9.675561797752808, learning_rate = 3.2008615e-05, loss = 0.0006534157, step = 6889 (5.289 sec) 2023-02-20 14:33:43,092 [INFO] tensorflow: epoch = 9.675561797752808, learning_rate = 3.2008615e-05, loss = 0.0006534157, step = 6889 (5.289 sec) 2023-02-20 14:33:44,749 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.189 INFO:tensorflow:global_step/sec: 6.03783 2023-02-20 14:33:45,244 [INFO] tensorflow: global_step/sec: 6.03783 INFO:tensorflow:global_step/sec: 6.03471 2023-02-20 14:33:47,564 [INFO] tensorflow: global_step/sec: 6.03471 INFO:tensorflow:epoch = 9.720505617977528, learning_rate = 3.2285858e-05, loss = 0.0006647013, step = 6921 (5.300 sec) 2023-02-20 14:33:48,392 [INFO] tensorflow: epoch = 9.720505617977528, learning_rate = 3.2285858e-05, loss = 0.0006647013, step = 6921 (5.300 sec) 2023-02-20 14:33:48,888 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.161 INFO:tensorflow:global_step/sec: 6.04399 2023-02-20 14:33:49,880 [INFO] tensorflow: global_step/sec: 6.04399 INFO:tensorflow:global_step/sec: 6.03559 2023-02-20 14:33:52,199 [INFO] tensorflow: global_step/sec: 6.03559 2023-02-20 14:33:53,026 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.170 INFO:tensorflow:epoch = 9.765449438202246, learning_rate = 3.2565495e-05, loss = 0.0007400005, step = 6953 (5.294 sec) 2023-02-20 14:33:53,686 [INFO] tensorflow: epoch = 9.765449438202246, learning_rate = 3.2565495e-05, loss = 0.0007400005, step = 6953 (5.294 sec) INFO:tensorflow:global_step/sec: 6.05177 2023-02-20 14:33:54,513 [INFO] tensorflow: global_step/sec: 6.05177 INFO:tensorflow:global_step/sec: 6.03502 2023-02-20 14:33:56,833 [INFO] tensorflow: global_step/sec: 6.03502 2023-02-20 14:33:57,164 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.167 INFO:tensorflow:epoch = 9.810393258426966, learning_rate = 3.2847558e-05, loss = 0.0006337852, step = 6985 (5.297 sec) 2023-02-20 14:33:58,983 [INFO] tensorflow: epoch = 9.810393258426966, learning_rate = 3.2847558e-05, loss = 0.0006337852, step = 6985 (5.297 sec) INFO:tensorflow:global_step/sec: 6.03792 2023-02-20 14:33:59,151 [INFO] tensorflow: global_step/sec: 6.03792 2023-02-20 14:34:01,307 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.137 INFO:tensorflow:global_step/sec: 6.02931 2023-02-20 14:34:01,473 [INFO] tensorflow: global_step/sec: 6.02931 INFO:tensorflow:global_step/sec: 6.04684 2023-02-20 14:34:03,789 [INFO] tensorflow: global_step/sec: 6.04684 INFO:tensorflow:epoch = 9.855337078651685, learning_rate = 3.313203e-05, loss = 0.0006319451, step = 7017 (5.303 sec) 2023-02-20 14:34:04,286 [INFO] tensorflow: epoch = 9.855337078651685, learning_rate = 3.313203e-05, loss = 0.0006319451, step = 7017 (5.303 sec) 2023-02-20 14:34:05,444 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.179 INFO:tensorflow:global_step/sec: 6.03895 2023-02-20 14:34:06,107 [INFO] tensorflow: global_step/sec: 6.03895 INFO:tensorflow:global_step/sec: 6.04558 2023-02-20 14:34:08,423 [INFO] tensorflow: global_step/sec: 6.04558 INFO:tensorflow:epoch = 9.900280898876405, learning_rate = 3.3418997e-05, loss = 0.0006744367, step = 7049 (5.297 sec) 2023-02-20 14:34:09,583 [INFO] tensorflow: epoch = 9.900280898876405, learning_rate = 3.3418997e-05, loss = 0.0006744367, step = 7049 (5.297 sec) 2023-02-20 14:34:09,583 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.160 INFO:tensorflow:global_step/sec: 6.03753 2023-02-20 14:34:10,741 [INFO] tensorflow: global_step/sec: 6.03753 INFO:tensorflow:global_step/sec: 6.03778 2023-02-20 14:34:13,060 [INFO] tensorflow: global_step/sec: 6.03778 2023-02-20 14:34:13,723 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.158 INFO:tensorflow:epoch = 9.945224719101123, learning_rate = 3.3708453e-05, loss = 0.0006045232, step = 7081 (5.297 sec) 2023-02-20 14:34:14,880 [INFO] tensorflow: epoch = 9.945224719101123, learning_rate = 3.3708453e-05, loss = 0.0006045232, step = 7081 (5.297 sec) INFO:tensorflow:global_step/sec: 6.04429 2023-02-20 14:34:15,376 [INFO] tensorflow: global_step/sec: 6.04429 INFO:tensorflow:global_step/sec: 6.04677 2023-02-20 14:34:17,692 [INFO] tensorflow: global_step/sec: 6.04677 2023-02-20 14:34:17,858 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.185 INFO:tensorflow:global_step/sec: 6.04552 2023-02-20 14:34:20,007 [INFO] tensorflow: global_step/sec: 6.04552 INFO:tensorflow:epoch = 9.990168539325843, learning_rate = 3.4000415e-05, loss = 0.00060086907, step = 7113 (5.293 sec) 2023-02-20 14:34:20,173 [INFO] tensorflow: epoch = 9.990168539325843, learning_rate = 3.4000415e-05, loss = 0.00060086907, step = 7113 (5.293 sec) INFO:tensorflow:Saving checkpoints for step-7120. 2023-02-20 14:34:21,169 [INFO] tensorflow: Saving checkpoints for step-7120. 2023-02-20 14:34:24,069 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 115, 0.00s/step 2023-02-20 14:37:49,594 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 115, 20.55s/step 2023-02-20 14:41:11,558 [INFO] iva.detectnet_v2.evaluation.evaluation: step 20 / 115, 20.20s/step 2023-02-20 14:44:32,615 [INFO] iva.detectnet_v2.evaluation.evaluation: step 30 / 115, 20.11s/step 2023-02-20 14:47:53,845 [INFO] iva.detectnet_v2.evaluation.evaluation: step 40 / 115, 20.12s/step 2023-02-20 14:51:15,970 [INFO] iva.detectnet_v2.evaluation.evaluation: step 50 / 115, 20.21s/step 2023-02-20 14:54:40,178 [INFO] iva.detectnet_v2.evaluation.evaluation: step 60 / 115, 20.42s/step 2023-02-20 14:58:02,419 [INFO] iva.detectnet_v2.evaluation.evaluation: step 70 / 115, 20.22s/step 2023-02-20 15:01:23,995 [INFO] iva.detectnet_v2.evaluation.evaluation: step 80 / 115, 20.16s/step 2023-02-20 15:04:45,495 [INFO] iva.detectnet_v2.evaluation.evaluation: step 90 / 115, 20.15s/step 2023-02-20 15:08:06,245 [INFO] iva.detectnet_v2.evaluation.evaluation: step 100 / 115, 20.08s/step 2023-02-20 15:11:29,401 [INFO] iva.detectnet_v2.evaluation.evaluation: step 110 / 115, 20.32s/step Matching predictions to ground truth, class 1/3.: 100%|█| 495/495 [00:00<00:00, 17202.16it/s] Matching predictions to ground truth, class 2/3.: 100%|█| 662/662 [00:00<00:00, 17982.88it/s] Matching predictions to ground truth, class 3/3.: 100%|█| 481/481 [00:00<00:00, 18005.64it/s] Epoch 10/120 ========================= Validation cost: 0.013203 Mean average_precision (in %): 0.0000 class name average precision (in %) ------------ -------------------------- cracked 0 offset 0 standard 0 Median Inference Time: 0.021099 2023-02-20 15:13:10,477 [INFO] root: Evaluation metrics generated. INFO:tensorflow:epoch = 10.0, learning_rate = 3.406461e-05, loss = 0.0006015986, step = 7120 (2334.807 sec) 2023-02-20 15:13:14,980 [INFO] tensorflow: epoch = 10.0, learning_rate = 3.406461e-05, loss = 0.0006015986, step = 7120 (2334.807 sec) 2023-02-20 15:13:14,986 [INFO] root: None 2023-02-20 15:13:14,997 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 10/120: loss: 0.00060 learning rate: 0.00003 Time taken: 0:40:47.127430 ETA: 3 days, 2:46:24.017296 2023-02-20 15:13:15,657 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 0.043 INFO:tensorflow:global_step/sec: 0.0059932 2023-02-20 15:13:15,989 [INFO] tensorflow: global_step/sec: 0.0059932 INFO:tensorflow:global_step/sec: 6.06278 2023-02-20 15:13:18,298 [INFO] tensorflow: global_step/sec: 6.06278 2023-02-20 15:13:19,782 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.244 INFO:tensorflow:epoch = 10.044943820224718, learning_rate = 3.435966e-05, loss = 0.0006485666, step = 7152 (5.298 sec) 2023-02-20 15:13:20,278 [INFO] tensorflow: epoch = 10.044943820224718, learning_rate = 3.435966e-05, loss = 0.0006485666, step = 7152 (5.298 sec) INFO:tensorflow:global_step/sec: 6.05468 2023-02-20 15:13:20,610 [INFO] tensorflow: global_step/sec: 6.05468 INFO:tensorflow:global_step/sec: 6.05233 2023-02-20 15:13:22,923 [INFO] tensorflow: global_step/sec: 6.05233 2023-02-20 15:13:23,914 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.206 INFO:tensorflow:global_step/sec: 6.0589 2023-02-20 15:13:25,234 [INFO] tensorflow: global_step/sec: 6.0589