{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Object Detection using TAO DetectNet_v2\n",
"\n",
"Transfer learning is the process of transferring learned features from one application to another. It is a commonly used training technique where you use a model trained on one task and re-train to use it on a different task. \n",
"\n",
"Train Adapt Optimize (TAO) Toolkit is a simple and easy-to-use Python based AI toolkit for taking purpose-built AI models and customizing them with users' own data.\n",
"\n",
" "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Learning Objectives\n",
"In this notebook, you will learn how to leverage the simplicity and convenience of TAO to:\n",
"\n",
"* Take a pretrained resnet18 model and train a ResNet-18 DetectNet_v2 model on the KITTI dataset\n",
"* Prune the trained detectnet_v2 model\n",
"* Retrain the pruned model to recover lost accuracy\n",
"* Export the pruned model\n",
"* Quantize the pruned model using QAT\n",
"* Run Inference on the trained model\n",
"* Export the pruned, quantized and retrained model to a .etlt file for deployment to DeepStream\n",
"* Run inference on the exported. etlt model to verify deployment using TensorRT\n",
"\n",
"### Table of Contents\n",
"\n",
"This notebook shows an example usecase of Object Detection using DetectNet_v2 in the Train Adapt Optimize (TAO) Toolkit.\n",
"\n",
"0. [Set up env variables and map drives](#head-0)\n",
"1. [Install the TAO Launcher](#head-1)\n",
"1. [Prepare dataset and pre-trained model](#head-2)\n",
" 1. [Download the dataset](#head-2-1)\n",
" 1. [Verify downloaded dataset](#head-2-2)\n",
" 1. [Prepare tfrecords from kitti format dataset](#head-2-3)\n",
" 2. [Download pre-trained model](#head-2-4)\n",
"2. [Provide training specification](#head-3)\n",
"3. [Run TAO training](#head-4)\n",
"4. [Evaluate trained models](#head-5)\n",
"5. [Prune trained models](#head-6)\n",
"6. [Retrain pruned models](#head-7)\n",
"7. [Evaluate retrained model](#head-8)\n",
"8. [Visualize inferences](#head-9)\n",
"9. [Model Export](#head-10)\n",
" 1. [Int8 Optimization](#head-10-1)\n",
" 2. [Generate TensorRT engine](#head-10-2)\n",
"10. [Verify Deployed Model](#head-11)\n",
" 1. [Inference using TensorRT engine](#head-11-1)\n",
"11. [QAT workflow](#head-12)\n",
" 1. [Convert pruned model to QAT and retrain](#head-12-1)\n",
" 2. [Evaluate QAT converted model](#head-12-2)\n",
" 3. [Export QAT trained model to int8](#head-12-3)\n",
" 4. [Evaluate a QAT trained model using the exported TensorRT engine](#head-12-4)\n",
" 5. [Inference using QAT engine](#head-12-5)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 0. Set up env variables and map drives \n",
"When using the purpose-built pretrained models from NGC, please make sure to set the `$KEY` environment variable to the key as mentioned in the model overview. Failing to do so, can lead to errors when trying to load them as pretrained models.\n",
"\n",
"The following notebook requires the user to set an env variable called the `$LOCAL_PROJECT_DIR` as the path to the users workspace. Please note that the dataset to run this notebook is expected to reside in the `$LOCAL_PROJECT_DIR/data`, while the TAO experiment generated collaterals will be output to `$LOCAL_PROJECT_DIR/detectnet_v2`. More information on how to set up the dataset and the supported steps in the TAO workflow are provided in the subsequent cells.\n",
"\n",
"*Note: Please make sure to remove any stray artifacts/files from the `$USER_EXPERIMENT_DIR` or `$DATA_DOWNLOAD_DIR` paths as mentioned below, that may have been generated from previous experiments. Having checkpoint files etc may interfere with creating a training graph for a new experiment.*\n",
"\n",
"*Note: This notebook currently is by default set up to run training using 1 GPU. To use more GPU's please update the env variable `$NUM_GPUS` accordingly*"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'/home/guest/taotoolkit/cv_samples_v1.2.0/detectnet_v2_car'"
]
},
"execution_count": 1,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"pwd"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"env: KEY=tlt_encode\n",
"env: NUM_GPUS=1\n",
"env: USER_EXPERIMENT_DIR=/workspace/tao-experiments/experiment\n",
"env: DATA_DOWNLOAD_DIR=/workspace/tao-experiments/data\n",
"env: LOCAL_PROJECT_DIR=/home/guest/taotoolkit/cv_samples_v1.2.0/detectnet_v2_car\n",
"env: SPECS_DIR=/workspace/tao-experiments/specs\n",
"total 28\r\n",
"-rw-r--r-- 1 guest guest 2436 Nov 18 10:00 detectnet_v2_inference_kitti_etlt.txt\r\n",
"-rw-r--r-- 1 guest guest 2445 Nov 18 10:00 detectnet_v2_inference_kitti_etlt_qat.txt\r\n",
"-rw-rw-r-- 1 guest guest 3172 Jan 6 15:56 detectnet_v2_train_resnet18_kitti.txt\r\n",
"-rw-rw-r-- 1 guest guest 370 Jan 6 16:06 detectnet_v2_tfrecords_kitti_trainval.txt\r\n",
"-rw-r--r-- 1 guest guest 3172 Jan 6 16:17 detectnet_v2_retrain_resnet18_kitti_qat.txt\r\n",
"-rw-rw-r-- 1 guest guest 3172 Jan 6 16:19 detectnet_v2_retrain_resnet18_kitti.txt\r\n",
"-rw-r--r-- 1 guest guest 990 Jan 20 10:38 detectnet_v2_inference_kitti_tlt.txt\r\n"
]
}
],
"source": [
"# Setting up env variables for cleaner command line commands.\n",
"import os\n",
"\n",
"# %env KEY=nvidia_tlt\n",
"%env KEY=tlt_encode\n",
"%env NUM_GPUS=1\n",
"%env USER_EXPERIMENT_DIR=/workspace/tao-experiments/experiment\n",
"%env DATA_DOWNLOAD_DIR=/workspace/tao-experiments/data\n",
"\n",
"# Set this path if you don't run the notebook from the samples directory.\n",
"# %env NOTEBOOK_ROOT=~/tao-samples/detectnet_v2\n",
"\n",
"# Please define this local project directory that needs to be mapped to the TAO docker session.\n",
"# The dataset expected to be present in $LOCAL_PROJECT_DIR/data, while the results for the steps\n",
"# in this notebook will be stored at $LOCAL_PROJECT_DIR/detectnet_v2\n",
"# !PLEASE MAKE SURE TO UPDATE THIS PATH!.\n",
"\n",
"%env LOCAL_PROJECT_DIR =/home/guest/taotoolkit/cv_samples_v1.2.0/detectnet_v2_car\n",
"# os.environ[\"LOCAL_PROJECT_DIR\"] = FIXME\n",
"\n",
"os.environ[\"LOCAL_DATA_DIR\"] = os.path.join(\n",
" os.getenv(\"LOCAL_PROJECT_DIR\", os.getcwd()),\n",
" \"data\"\n",
")\n",
"os.environ[\"LOCAL_EXPERIMENT_DIR\"] = os.path.join(\n",
" os.getenv(\"LOCAL_PROJECT_DIR\", os.getcwd()),\n",
" \"experiment\"\n",
")\n",
"\n",
"# The sample spec files are present in the same path as the downloaded samples.\n",
"os.environ[\"LOCAL_SPECS_DIR\"] = os.path.join(\n",
" os.getenv(\"NOTEBOOK_ROOT\", os.getcwd()),\n",
" \"specs\"\n",
")\n",
"%env SPECS_DIR=/workspace/tao-experiments/specs\n",
"\n",
"# Showing list of specification files.\n",
"!ls -rlt $LOCAL_SPECS_DIR"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The cell below maps the project directory on your local host to a workspace directory in the TAO docker instance, so that the data and the results are mapped from in and out of the docker. For more information please refer to the [launcher instance](https://docs.nvidia.com/tao/tao-toolkit/tao_launcher.html) in the user guide.\n",
"\n",
"When running this cell on AWS, update the drive_map entry with the dictionary defined below, so that you don't have permission issues when writing data into folders created by the TAO docker.\n",
"\n",
"```json\n",
"drive_map = {\n",
" \"Mounts\": [\n",
" # Mapping the data directory\n",
" {\n",
" \"source\": os.environ[\"LOCAL_PROJECT_DIR\"],\n",
" \"destination\": \"/workspace/tao-experiments\"\n",
" },\n",
" # Mapping the specs directory.\n",
" {\n",
" \"source\": os.environ[\"LOCAL_SPECS_DIR\"],\n",
" \"destination\": os.environ[\"SPECS_DIR\"]\n",
" },\n",
" ],\n",
" \"DockerOptions\": {\n",
" \"user\": \"{}:{}\".format(os.getuid(), os.getgid())\n",
" }\n",
"}\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"# Mapping up the local directories to the TAO docker.\n",
"import json\n",
"mounts_file = os.path.expanduser(\"~/.tao_mounts.json\")\n",
"\n",
"# Define the dictionary with the mapped drives\n",
"drive_map = {\n",
" \"Mounts\": [\n",
" # Mapping the data directory\n",
" {\n",
" \"source\": os.environ[\"LOCAL_PROJECT_DIR\"],\n",
" \"destination\": \"/workspace/tao-experiments\"\n",
" },\n",
" # Mapping the specs directory.\n",
" {\n",
" \"source\": os.environ[\"LOCAL_SPECS_DIR\"],\n",
" \"destination\": os.environ[\"SPECS_DIR\"]\n",
" },\n",
" ],\n",
" \"DockerOptions\": {\n",
" \"user\": \"{}:{}\".format(os.getuid(), os.getgid())\n",
" }\n",
"}\n",
"\n",
"# Writing the mounts file.\n",
"with open(mounts_file, \"w\") as mfile:\n",
" json.dump(drive_map, mfile, indent=4)"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{\r\n",
" \"Mounts\": [\r\n",
" {\r\n",
" \"source\": \"/home/guest/taotoolkit/cv_samples_v1.2.0/detectnet_v2_car\",\r\n",
" \"destination\": \"/workspace/tao-experiments\"\r\n",
" },\r\n",
" {\r\n",
" \"source\": \"/home/guest/taotoolkit/cv_samples_v1.2.0/detectnet_v2_car/specs\",\r\n",
" \"destination\": \"/workspace/tao-experiments/specs\"\r\n",
" }\r\n",
" ],\r\n",
" \"DockerOptions\": {\r\n",
" \"user\": \"1001:1001\"\r\n",
" }\r\n",
"}"
]
}
],
"source": [
"!cat ~/.tao_mounts.json"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 1. Install the TAO launcher \n",
"The TAO launcher is a python package distributed as a python wheel listed in the `nvidia-pyindex` python index. You may install the launcher by executing the following cell.\n",
"\n",
"Please note that TAO Toolkit recommends users to run the TAO launcher in a virtual env with python 3.6.9. You may follow the instruction in this [page](https://virtualenvwrapper.readthedocs.io/en/latest/install.html) to set up a python virtual env using the `virtualenv` and `virtualenvwrapper` packages. Once you have setup virtualenvwrapper, please set the version of python to be used in the virtual env by using the `VIRTUALENVWRAPPER_PYTHON` variable. You may do so by running\n",
"\n",
"```sh\n",
"export VIRTUALENVWRAPPER_PYTHON=/path/to/bin/python3.x\n",
"```\n",
"where x >= 6 and <= 8\n",
"\n",
"We recommend performing this step first and then launching the notebook from the virtual environment. In addition to installing TAO python package, please make sure of the following software requirements:\n",
"* python >=3.6.9 < 3.8.x\n",
"* docker-ce > 19.03.5\n",
"* docker-API 1.40\n",
"* nvidia-container-toolkit > 1.3.0-1\n",
"* nvidia-container-runtime > 3.4.0-1\n",
"* nvidia-docker2 > 2.5.0-1\n",
"* nvidia-driver > 455+\n",
"\n",
"Once you have installed the pre-requisites, please log in to the docker registry nvcr.io by following the command below\n",
"\n",
"```sh\n",
"docker login nvcr.io\n",
"```\n",
"\n",
"You will be trigerred to enter a username and password. The username is `$oauthtoken` and the password is the API key generated from `ngc.nvidia.com`. Please follow the instructions in the [NGC setup guide](https://docs.nvidia.com/ngc/ngc-overview/index.html#generating-api-key) to generate your own API key."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com\n",
"Requirement already satisfied: nvidia-pyindex in /home/guest/miniconda3/envs/taotoolkit/lib/python3.6/site-packages (1.0.9)\n",
"Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com\n",
"Requirement already satisfied: nvidia-tao in /home/guest/miniconda3/envs/taotoolkit/lib/python3.6/site-packages (0.1.19)\n",
"Requirement already satisfied: certifi==2020.6.20 in /home/guest/miniconda3/envs/taotoolkit/lib/python3.6/site-packages (from nvidia-tao) (2020.6.20)\n",
"Requirement already satisfied: six==1.15.0 in /home/guest/miniconda3/envs/taotoolkit/lib/python3.6/site-packages (from nvidia-tao) (1.15.0)\n",
"Requirement already satisfied: idna==2.10 in /home/guest/miniconda3/envs/taotoolkit/lib/python3.6/site-packages (from nvidia-tao) (2.10)\n",
"Requirement already satisfied: requests==2.24.0 in /home/guest/miniconda3/envs/taotoolkit/lib/python3.6/site-packages (from nvidia-tao) (2.24.0)\n",
"Requirement already satisfied: chardet==3.0.4 in /home/guest/miniconda3/envs/taotoolkit/lib/python3.6/site-packages (from nvidia-tao) (3.0.4)\n",
"Requirement already satisfied: docker-pycreds==0.4.0 in /home/guest/miniconda3/envs/taotoolkit/lib/python3.6/site-packages (from nvidia-tao) (0.4.0)\n",
"Requirement already satisfied: docker==4.3.1 in /home/guest/miniconda3/envs/taotoolkit/lib/python3.6/site-packages (from nvidia-tao) (4.3.1)\n",
"Requirement already satisfied: tabulate==0.8.7 in /home/guest/miniconda3/envs/taotoolkit/lib/python3.6/site-packages (from nvidia-tao) (0.8.7)\n",
"Requirement already satisfied: urllib3==1.25.10 in /home/guest/miniconda3/envs/taotoolkit/lib/python3.6/site-packages (from nvidia-tao) (1.25.10)\n",
"Requirement already satisfied: websocket-client==0.57.0 in /home/guest/miniconda3/envs/taotoolkit/lib/python3.6/site-packages (from nvidia-tao) (0.57.0)\n"
]
}
],
"source": [
"# SKIP this step IF you have already installed the TAO launcher wheel.\n",
"!pip3 install nvidia-pyindex\n",
"!pip3 install nvidia-tao"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Configuration of the TAO Toolkit Instance\r\n",
"dockers: ['nvidia/tao/tao-toolkit-tf', 'nvidia/tao/tao-toolkit-pyt', 'nvidia/tao/tao-toolkit-lm']\r\n",
"format_version: 1.0\r\n",
"toolkit_version: 3.21.08\r\n",
"published_date: 08/17/2021\r\n"
]
}
],
"source": [
"# View the versions of the TAO launcher\n",
"!tao info"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 2. Prepare dataset and pre-trained model "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We will be using the kitti object detection dataset for this example. To find more details, please visit http://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=2d. Please download both, the left color images of the object dataset from [here](http://www.cvlibs.net/download.php?file=data_object_image_2.zip) and, the training labels for the object dataset from [here](http://www.cvlibs.net/download.php?file=data_object_label_2.zip), and place the zip files in `$LOCAL_DATA_DIR`\n",
"\n",
"The data will then be extracted to have\n",
"* training images in `$LOCAL_DATA_DIR/training/image_2`\n",
"* training labels in `$LOCAL_DATA_DIR/training/label_2`\n",
"* testing images in `$LOCAL_DATA_DIR/testing/image_2`\n",
"\n",
"You may use this notebook with your own dataset as well. To use this example with your own dataset, please follow the same directory structure as mentioned below.\n",
"\n",
"*Note: There are no labels for the testing images, therefore we use it just to visualize inferences for the trained model.*"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### A. Download the dataset \n",
"Once you have gotten the download links in your email, please populate them in place of the `KITTI_IMAGES_DOWNLOAD_URL` and the `KITTI_LABELS_DOWNLOAD_URL`. This next cell, will download the data and place in `$LOCAL_DATA_DIR`"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"total 8\r\n",
"drwxrwxr-x 3 guest guest 4096 Jan 6 16:06 tfrecords\r\n",
"drwxrwxr-x 8 guest guest 4096 Dec 21 15:15 training\r\n"
]
}
],
"source": [
"!ls -l $LOCAL_DATA_DIR/"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"ename": "NameError",
"evalue": "name 'KITTI_IMAGES_DOWNLOAD_URL' is not defined",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mNameError\u001b[0m Traceback (most recent call last)",
"\u001b[0;32m/tmp/ipykernel_1372662/4144454426.py\u001b[0m in \u001b[0;36m\u001b[0;34m\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mos\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2\u001b[0m \u001b[0mget_ipython\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msystem\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'mkdir -p $LOCAL_DATA_DIR'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m \u001b[0mos\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0menviron\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"URL_IMAGES\"\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mKITTI_IMAGES_DOWNLOAD_URL\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 4\u001b[0m \u001b[0mget_ipython\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msystem\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'if [ ! -f $LOCAL_DATA_DIR/data_object_image_2.zip ]; then wget $URL_IMAGES -O $LOCAL_DATA_DIR/data_object_image_2.zip; else echo \"image archive already downloaded\"; fi'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 5\u001b[0m \u001b[0mos\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0menviron\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"URL_LABELS\"\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mKITTI_LABELS_DOWNLOAD_URL\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;31mNameError\u001b[0m: name 'KITTI_IMAGES_DOWNLOAD_URL' is not defined"
]
}
],
"source": [
"import os\n",
"!mkdir -p $LOCAL_DATA_DIR\n",
"os.environ[\"URL_IMAGES\"]=KITTI_IMAGES_DOWNLOAD_URL\n",
"!if [ ! -f $LOCAL_DATA_DIR/data_object_image_2.zip ]; then wget $URL_IMAGES -O $LOCAL_DATA_DIR/data_object_image_2.zip; else echo \"image archive already downloaded\"; fi \n",
"os.environ[\"URL_LABELS\"]=KITTI_LABELS_DOWNLOAD_URL\n",
"!if [ ! -f $LOCAL_DATA_DIR/data_object_label_2.zip ]; then wget $URL_LABELS -O $LOCAL_DATA_DIR/data_object_label_2.zip; else \\ echo \"label archive already downloaded\"; fi "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### B. Verify downloaded dataset "
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Found Image zip file.\n",
"Found Labels zip file.\n"
]
}
],
"source": [
"# Check the dataset is present\n",
"!if [ ! -f $LOCAL_DATA_DIR/data_object_image_2.zip ]; then echo 'Image zip file not found, please download.'; else echo 'Found Image zip file.';fi\n",
"!if [ ! -f $LOCAL_DATA_DIR/data_object_label_2.zip ]; then echo 'Label zip file not found, please download.'; else echo 'Found Labels zip file.';fi"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"images corrupt, redownload!\n",
"labels corrupt, redownload!\n"
]
}
],
"source": [
"# This may take a while: verify integrity of zip files \n",
"!sha256sum $LOCAL_DATA_DIR/data_object_image_2.zip | cut -d ' ' -f 1 | grep -xq '^351c5a2aa0cd9238b50174a3a62b846bc5855da256b82a196431d60ff8d43617$' ; \\\n",
"if test $? -eq 0; then echo \"images OK\"; else echo \"images corrupt, redownload!\" && rm -f $LOCAL_DATA_DIR/data_object_image_2.zip; fi \n",
"!sha256sum $LOCAL_DATA_DIR/data_object_label_2.zip | cut -d ' ' -f 1 | grep -xq '^4efc76220d867e1c31bb980bbf8cbc02599f02a9cb4350effa98dbb04aaed880$' ; \\\n",
"if test $? -eq 0; then echo \"labels OK\"; else echo \"labels corrupt, redownload!\" && rm -f $LOCAL_DATA_DIR/data_object_label_2.zip; fi "
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"/home/guest/taotoolkit/cv_samples_v1.2.0/detectnet_v2/car_data\n"
]
}
],
"source": [
"DATA_DIR = os.environ.get('LOCAL_DATA_DIR')\n",
"print(DATA_DIR)"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"ename": "FileNotFoundError",
"evalue": "[Errno 2] No such file or directory: '/home/guest/taotoolkit/cv_samples_v1.2.0/detectnet_v2/car_data/testing/images'",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mFileNotFoundError\u001b[0m Traceback (most recent call last)",
"\u001b[0;32m/tmp/ipykernel_3911868/2437139951.py\u001b[0m in \u001b[0;36m\u001b[0;34m\u001b[0m\n\u001b[1;32m 5\u001b[0m \u001b[0mnum_training_images\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mos\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlistdir\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mos\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpath\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjoin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mDATA_DIR\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"training/image_2\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 6\u001b[0m \u001b[0mnum_training_labels\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mos\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlistdir\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mos\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpath\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjoin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mDATA_DIR\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"training/label_2\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 7\u001b[0;31m \u001b[0mnum_testing_images\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mos\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlistdir\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mos\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpath\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjoin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mDATA_DIR\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"testing/images\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 8\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Number of images in the train/val set. {}\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnum_training_images\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 9\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Number of labels in the train/val set. {}\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnum_training_labels\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;31mFileNotFoundError\u001b[0m: [Errno 2] No such file or directory: '/home/guest/taotoolkit/cv_samples_v1.2.0/detectnet_v2/car_data/testing/images'"
]
}
],
"source": [
"# verify\n",
"import os\n",
"\n",
"DATA_DIR = os.environ.get('LOCAL_DATA_DIR')\n",
"num_training_images = len(os.listdir(os.path.join(DATA_DIR, \"training/image_2\")))\n",
"num_training_labels = len(os.listdir(os.path.join(DATA_DIR, \"training/label_2\")))\n",
"num_testing_images = len(os.listdir(os.path.join(DATA_DIR, \"testing/images\")))\n",
"print(\"Number of images in the train/val set. {}\".format(num_training_images))\n",
"print(\"Number of labels in the train/val set. {}\".format(num_training_labels))\n",
"print(\"Number of images in the test set. {}\".format(num_testing_images))"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"cat: /home/guest/taotoolkit/cv_samples_v1.2.0/detectnet_v2/car_data/training/label_2/1.txt: No such file or directory\r\n"
]
}
],
"source": [
"# Sample kitti label.\n",
"!cat $LOCAL_DATA_DIR/training/label_2/1.txt"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### C. Prepare tf records from kitti format dataset \n",
"\n",
"* Update the tfrecords spec file to take in your kitti format dataset\n",
"* Create the tfrecords using the detectnet_v2 dataset_convert \n",
"\n",
"*Note: TfRecords only need to be generated once.*"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"TFrecords conversion spec file for kitti training\n",
"kitti_config {\r\n",
" root_directory_path: \"/workspace/tao-experiments/data/training\"\r\n",
" image_dir_name: \"image_2\"\r\n",
" label_dir_name: \"label_2\"\r\n",
" image_extension: \".png\"\r\n",
" partition_mode: \"random\"\r\n",
" num_partitions: 2\r\n",
" val_split: 20\r\n",
" num_shards: 10\r\n",
"}\r\n",
"image_directory_path: \"/workspace/tao-experiments/car_data/training\"\r\n",
"target_class_mapping {\r\n",
" key: \"car\"\r\n",
" value: \"car\"\r\n",
"}"
]
}
],
"source": [
"print(\"TFrecords conversion spec file for kitti training\")\n",
"!cat $LOCAL_SPECS_DIR/detectnet_v2_tfrecords_kitti_trainval.txt"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Converting Tfrecords for kitti trainval dataset\n",
"2022-01-06 16:06:11,395 [INFO] root: Registry: ['nvcr.io']\n",
"Matplotlib created a temporary config/cache directory at /tmp/matplotlib-6_mb0qvf because the default path (/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.\n",
"Using TensorFlow backend.\n",
"WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.\n",
"Using TensorFlow backend.\n",
"2022-01-06 08:06:16,899 - iva.detectnet_v2.dataio.build_converter - INFO - Instantiating a kitti converter\n",
"2022-01-06 08:06:16,900 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Creating output directory /workspace/tao-experiments/data/tfrecords/kitti_trainval\n",
"2022-01-06 08:06:16,903 - iva.detectnet_v2.dataio.kitti_converter_lib - INFO - Num images in\n",
"Train: 761\tVal: 190\n",
"2022-01-06 08:06:16,903 - iva.detectnet_v2.dataio.kitti_converter_lib - INFO - Validation data in partition 0. Hence, while choosing the validationset during training choose validation_fold 0.\n",
"2022-01-06 08:06:16,904 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 0\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/dataio/dataset_converter_lib.py:142: The name tf.python_io.TFRecordWriter is deprecated. Please use tf.io.TFRecordWriter instead.\n",
"\n",
"2022-01-06 08:06:16,904 - tensorflow - WARNING - From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/dataio/dataset_converter_lib.py:142: The name tf.python_io.TFRecordWriter is deprecated. Please use tf.io.TFRecordWriter instead.\n",
"\n",
"/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:283: VisibleDeprecationWarning: Reading unicode strings without specifying the encoding argument is deprecated. Set the encoding, use None for the system default.\n",
"2022-01-06 08:06:16,933 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 1\n",
"2022-01-06 08:06:16,954 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 2\n",
"2022-01-06 08:06:16,978 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 3\n",
"2022-01-06 08:06:17,000 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 4\n",
"2022-01-06 08:06:17,023 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 5\n",
"2022-01-06 08:06:17,045 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 6\n",
"2022-01-06 08:06:17,067 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 7\n",
"2022-01-06 08:06:17,089 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 8\n",
"2022-01-06 08:06:17,112 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 9\n",
"2022-01-06 08:06:17,134 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - \n",
"Wrote the following numbers of objects:\n",
"b'car': 661\n",
"\n",
"2022-01-06 08:06:17,134 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 0\n",
"2022-01-06 08:06:17,221 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 1\n",
"2022-01-06 08:06:17,309 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 2\n",
"2022-01-06 08:06:17,396 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 3\n",
"2022-01-06 08:06:17,483 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 4\n",
"/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:283: UserWarning: genfromtxt: Empty input file: \"/workspace/tao-experiments/data/training/label_2/490.txt\"\n",
"2022-01-06 08:06:17,571 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 5\n",
"/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:283: UserWarning: genfromtxt: Empty input file: \"/workspace/tao-experiments/data/training/label_2/624.txt\"\n",
"2022-01-06 08:06:17,658 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 6\n",
"/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:283: UserWarning: genfromtxt: Empty input file: \"/workspace/tao-experiments/data/training/label_2/491.txt\"\n",
"2022-01-06 08:06:17,746 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 7\n",
"2022-01-06 08:06:17,833 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 8\n",
"2022-01-06 08:06:17,922 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 9\n",
"/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:283: UserWarning: genfromtxt: Empty input file: \"/workspace/tao-experiments/data/training/label_2/447.txt\"\n",
"2022-01-06 08:06:18,011 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - \n",
"Wrote the following numbers of objects:\n",
"b'car': 2624\n",
"\n",
"2022-01-06 08:06:18,012 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Cumulative object statistics\n",
"2022-01-06 08:06:18,012 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - \n",
"Wrote the following numbers of objects:\n",
"b'car': 3285\n",
"\n",
"2022-01-06 08:06:18,012 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Class map. \n",
"Label in GT: Label in tfrecords file \n",
"b'car': b'car'\n",
"For the dataset_config in the experiment_spec, please use labels in the tfrecords file, while writing the classmap.\n",
"\n",
"2022-01-06 08:06:18,012 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Tfrecords generation complete.\n",
"2022-01-06 16:06:18,566 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.\n"
]
}
],
"source": [
"# Creating a new directory for the output tfrecords dump.\n",
"print(\"Converting Tfrecords for kitti trainval dataset\")\n",
"!mkdir -p $LOCAL_DATA_DIR/tfrecords && rm -rf $LOCAL_DATA_DIR/tfrecords/*\n",
"!tao detectnet_v2 dataset_convert \\\n",
" -d $SPECS_DIR/detectnet_v2_tfrecords_kitti_trainval.txt \\\n",
" -o $DATA_DOWNLOAD_DIR/tfrecords/kitti_trainval/kitti_trainval"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"total 724\r\n",
"-rw-r--r-- 1 guest guest 14257 Jan 6 16:06 kitti_trainval-fold-000-of-002-shard-00000-of-00010\r\n",
"-rw-r--r-- 1 guest guest 14490 Jan 6 16:06 kitti_trainval-fold-000-of-002-shard-00001-of-00010\r\n",
"-rw-r--r-- 1 guest guest 14491 Jan 6 16:06 kitti_trainval-fold-000-of-002-shard-00002-of-00010\r\n",
"-rw-r--r-- 1 guest guest 14084 Jan 6 16:06 kitti_trainval-fold-000-of-002-shard-00003-of-00010\r\n",
"-rw-r--r-- 1 guest guest 14492 Jan 6 16:06 kitti_trainval-fold-000-of-002-shard-00004-of-00010\r\n",
"-rw-r--r-- 1 guest guest 14259 Jan 6 16:06 kitti_trainval-fold-000-of-002-shard-00005-of-00010\r\n",
"-rw-r--r-- 1 guest guest 13449 Jan 6 16:06 kitti_trainval-fold-000-of-002-shard-00006-of-00010\r\n",
"-rw-r--r-- 1 guest guest 13564 Jan 6 16:06 kitti_trainval-fold-000-of-002-shard-00007-of-00010\r\n",
"-rw-r--r-- 1 guest guest 14433 Jan 6 16:06 kitti_trainval-fold-000-of-002-shard-00008-of-00010\r\n",
"-rw-r--r-- 1 guest guest 13392 Jan 6 16:06 kitti_trainval-fold-000-of-002-shard-00009-of-00010\r\n",
"-rw-r--r-- 1 guest guest 56053 Jan 6 16:06 kitti_trainval-fold-001-of-002-shard-00000-of-00010\r\n",
"-rw-r--r-- 1 guest guest 56289 Jan 6 16:06 kitti_trainval-fold-001-of-002-shard-00001-of-00010\r\n",
"-rw-r--r-- 1 guest guest 56109 Jan 6 16:06 kitti_trainval-fold-001-of-002-shard-00002-of-00010\r\n",
"-rw-r--r-- 1 guest guest 55824 Jan 6 16:06 kitti_trainval-fold-001-of-002-shard-00003-of-00010\r\n",
"-rw-r--r-- 1 guest guest 56197 Jan 6 16:06 kitti_trainval-fold-001-of-002-shard-00004-of-00010\r\n",
"-rw-r--r-- 1 guest guest 57410 Jan 6 16:06 kitti_trainval-fold-001-of-002-shard-00005-of-00010\r\n",
"-rw-r--r-- 1 guest guest 56260 Jan 6 16:06 kitti_trainval-fold-001-of-002-shard-00006-of-00010\r\n",
"-rw-r--r-- 1 guest guest 55942 Jan 6 16:06 kitti_trainval-fold-001-of-002-shard-00007-of-00010\r\n",
"-rw-r--r-- 1 guest guest 56227 Jan 6 16:06 kitti_trainval-fold-001-of-002-shard-00008-of-00010\r\n",
"-rw-r--r-- 1 guest guest 56628 Jan 6 16:06 kitti_trainval-fold-001-of-002-shard-00009-of-00010\r\n"
]
}
],
"source": [
"!ls -rlt $LOCAL_DATA_DIR/tfrecords/kitti_trainval/"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### D. Download pre-trained model \n",
"Download the correct pretrained model from the NGC model registry for your experiment. Please note that for DetectNet_v2, the input is expected to be 0-1 normalized with input channels in RGB order. Therefore, for optimum results please download model templates from `nvidia/tao/pretrained_detectnet_v2`. The templates are now organized as version strings. For example, to download a resnet18 model suitable for detectnet please resolve to the ngc object shown as `nvidia/tao/pretrained_detectnet_v2:resnet18`. \n",
"\n",
"All other models are in BGR order expect input preprocessing with mean subtraction and input channels. Using them as pretrained weights may result in suboptimal performance.\n",
"\n",
"You may also use this notebook with the following purpose-built pretrained models \n",
"* [PeopleNet](https://ngc.nvidia.com/catalog/models/nvidia:tao:peoplenet)\n",
"* [TrafficCamNet](https://ngc.nvidia.com/catalog/models/nvidia:tao:trafficcamnet)\n",
"* [DashCamNet](https://ngc.nvidia.com/catalog/models/nvidia:tao:dashcamnet)\n",
"* [FaceDetect-IR](https://ngc.nvidia.com/catalog/models/nvidia:tao:facedetectir) "
]
},
{
"cell_type": "code",
"execution_count": 33,
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"env: CLI=ngccli_cat_linux.zip\n",
"--2021-12-13 17:12:19-- https://ngc.nvidia.com/downloads/ngccli_cat_linux.zip\n",
"Resolving ngc.nvidia.com (ngc.nvidia.com)... 13.225.99.60, 13.225.99.28, 13.225.99.8, ...\n",
"Connecting to ngc.nvidia.com (ngc.nvidia.com)|13.225.99.60|:443... connected.\n",
"HTTP request sent, awaiting response... 200 OK\n",
"Length: 25122731 (24M) [application/zip]\n",
"Saving to: ‘/home/guest/taotoolkit/cv_samples_v1.2.0/detectnet_v2/ngccli/ngccli_cat_linux.zip’\n",
"\n",
"ngccli_cat_linux.zi 100%[===================>] 23.96M 10.4MB/s in 2.3s \n",
"\n",
"2021-12-13 17:12:21 (10.4 MB/s) - ‘/home/guest/taotoolkit/cv_samples_v1.2.0/detectnet_v2/ngccli/ngccli_cat_linux.zip’ saved [25122731/25122731]\n",
"\n",
"Archive: /home/guest/taotoolkit/cv_samples_v1.2.0/detectnet_v2/ngccli/ngccli_cat_linux.zip\n",
" inflating: /home/guest/taotoolkit/cv_samples_v1.2.0/detectnet_v2/ngccli/ngc \n",
" extracting: /home/guest/taotoolkit/cv_samples_v1.2.0/detectnet_v2/ngccli/ngc.md5 \n"
]
}
],
"source": [
"# Installing NGC CLI on the local machine.\n",
"## Download and install\n",
"%env CLI=ngccli_cat_linux.zip\n",
"!mkdir -p $LOCAL_PROJECT_DIR/ngccli\n",
"\n",
"# Remove any previously existing CLI installations\n",
"!rm -rf $LOCAL_PROJECT_DIR/ngccli/*\n",
"!wget \"https://ngc.nvidia.com/downloads/$CLI\" -P $LOCAL_PROJECT_DIR/ngccli\n",
"!unzip -u \"$LOCAL_PROJECT_DIR/ngccli/$CLI\" -d $LOCAL_PROJECT_DIR/ngccli/\n",
"!rm $LOCAL_PROJECT_DIR/ngccli/*.zip \n",
"os.environ[\"PATH\"]=\"{}/ngccli:{}\".format(os.getenv(\"LOCAL_PROJECT_DIR\", \"\"), os.getenv(\"PATH\", \"\"))"
]
},
{
"cell_type": "code",
"execution_count": 34,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"+-------+-------+-------+-------+-------+-------+-------+-------+-------+\r\n",
"| Versi | Accur | Epoch | Batch | GPU | Memor | File | Statu | Creat |\r\n",
"| on | acy | s | Size | Model | y Foo | Size | s | ed |\r\n",
"| | | | | | tprin | | | Date |\r\n",
"| | | | | | t | | | |\r\n",
"+-------+-------+-------+-------+-------+-------+-------+-------+-------+\r\n",
"| vgg19 | 82.6 | 80 | 1 | V100 | 153.8 | 153.7 | UPLOA | Aug |\r\n",
"| | | | | | | 7 MB | D_COM | 24, |\r\n",
"| | | | | | | | PLETE | 2021 |\r\n",
"| vgg16 | 82.2 | 80 | 1 | V100 | 113.2 | 113.2 | UPLOA | Aug |\r\n",
"| | | | | | | MB | D_COM | 24, |\r\n",
"| | | | | | | | PLETE | 2021 |\r\n",
"| squee | 65.67 | 80 | 1 | V100 | 6.5 | 6.46 | UPLOA | Aug |\r\n",
"| zenet | | | | | | MB | D_COM | 24, |\r\n",
"| | | | | | | | PLETE | 2021 |\r\n",
"| resne | 82.7 | 80 | 1 | V100 | 294.5 | 294.5 | UPLOA | Aug |\r\n",
"| t50 | | | | | | 3 MB | D_COM | 24, |\r\n",
"| | | | | | | | PLETE | 2021 |\r\n",
"| resne | 79.0 | 80 | 1 | V100 | 89.0 | 89.02 | UPLOA | Aug |\r\n",
"| t18 | | | | | | MB | D_COM | 24, |\r\n",
"| | | | | | | | PLETE | 2021 |\r\n",
"| resne | 79.2 | 80 | 1 | V100 | 38.3 | 38.34 | UPLOA | Aug |\r\n",
"| t10 | | | | | | MB | D_COM | 24, |\r\n",
"| | | | | | | | PLETE | 2021 |\r\n",
"| mobil | 77.5 | 80 | 1 | V100 | 5.1 | 5.1 | UPLOA | Aug |\r\n",
"| enet_ | | | | | | MB | D_COM | 24, |\r\n",
"| v2 | | | | | | | PLETE | 2021 |\r\n",
"| mobil | 79.5 | 80 | 1 | V100 | 13.4 | 13.37 | UPLOA | Aug |\r\n",
"| enet_ | | | | | | MB | D_COM | 24, |\r\n",
"| v1 | | | | | | | PLETE | 2021 |\r\n",
"| googl | 82.2 | 80 | 1 | V100 | 47.7 | 47.74 | UPLOA | Aug |\r\n",
"| enet | | | | | | MB | D_COM | 24, |\r\n",
"| | | | | | | | PLETE | 2021 |\r\n",
"| effic | 77.11 | 80 | 1 | V100 | 16.9 | 16.9 | UPLOA | Aug |\r\n",
"| ientn | | | | | | MB | D_COM | 18, |\r\n",
"| et_b0 | | | | | | | PLETE | 2021 |\r\n",
"| _swis | | | | | | | | |\r\n",
"| h | | | | | | | | |\r\n",
"| effic | 77.11 | 80 | 1 | V100 | 16.9 | 16.9 | UPLOA | Aug |\r\n",
"| ientn | | | | | | MB | D_COM | 18, |\r\n",
"| et_b0 | | | | | | | PLETE | 2021 |\r\n",
"| _relu | | | | | | | | |\r\n",
"| darkn | 76.44 | 80 | 1 | V100 | 467.3 | 467.3 | UPLOA | Aug |\r\n",
"| et53 | | | | | | 2 MB | D_COM | 24, |\r\n",
"| | | | | | | | PLETE | 2021 |\r\n",
"| darkn | 77.52 | 80 | 1 | V100 | 229.1 | 229.1 | UPLOA | Aug |\r\n",
"| et19 | | | | | | 5 MB | D_COM | 24, |\r\n",
"| | | | | | | | PLETE | 2021 |\r\n",
"+-------+-------+-------+-------+-------+-------+-------+-------+-------+\r\n"
]
}
],
"source": [
"# List models available in the model registry.\n",
"!ngc registry model list nvidia/tao/pretrained_detectnet_v2:*"
]
},
{
"cell_type": "code",
"execution_count": 35,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"mkdir: cannot create directory ‘/home/guest/taotoolkit/cv_samples_v1.2.0/detectnet_v2/detectnet_v2_car/pretrained_resnet18/’: File exists\r\n"
]
}
],
"source": [
"# Create the target destination to download the model.\n",
"!mkdir $LOCAL_EXPERIMENT_DIR/pretrained_resnet18/"
]
},
{
"cell_type": "code",
"execution_count": 36,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Downloaded 82.28 MB in 40s, Download speed: 2.05 MB/s \n",
"----------------------------------------------------\n",
"Transfer id: pretrained_detectnet_v2_vresnet18 Download status: Completed.\n",
"Downloaded local path: /home/guest/taotoolkit/cv_samples_v1.2.0/detectnet_v2/detectnet_v2_car/pretrained_resnet18/pretrained_detectnet_v2_vresnet18-2\n",
"Total files downloaded: 1 \n",
"Total downloaded size: 82.28 MB\n",
"Started at: 2021-12-13 17:12:47.431104\n",
"Completed at: 2021-12-13 17:13:27.485671\n",
"Duration taken: 40s\n",
"----------------------------------------------------\n"
]
}
],
"source": [
"# Download the pretrained model from NGC\n",
"!ngc registry model download-version nvidia/tao/pretrained_detectnet_v2:resnet18 \\\n",
" --dest $LOCAL_EXPERIMENT_DIR/pretrained_resnet18"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"total 91160\r\n",
"-rw------- 1 guest guest 93345248 Dec 13 17:07 resnet18.hdf5\r\n"
]
}
],
"source": [
"!ls -rlt $LOCAL_EXPERIMENT_DIR/pretrained_resnet18/pretrained_detectnet_v2_vresnet18"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 3. Provide training specification \n",
"* Tfrecords for the train datasets\n",
" * To use the newly generated tfrecords, update the dataset_config parameter in the spec file at `$SPECS_DIR/detectnet_v2_train_resnet18_kitti.txt` \n",
" * Update the fold number to use for evaluation. In case of random data split, please use fold `0` only\n",
" * For sequence-wise split, you may use any fold generated from the dataset convert tool\n",
"* Pre-trained models\n",
"* Augmentation parameters for on the fly data augmentation\n",
"* Other training (hyper-)parameters such as batch size, number of epochs, learning rate etc."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"random_seed: 42\r\n",
"dataset_config {\r\n",
" data_sources {\r\n",
" tfrecords_path: \"/workspace/tao-experiments/car_data/tfrecords/kitti_trainval/*\"\r\n",
" image_directory_path: \"/workspace/tao-experiments/car_data/training/\"\r\n",
" }\r\n",
" image_extension: \"png\"\r\n",
" target_class_mapping{\r\n",
" key:\"car\"\r\n",
" value:\"car\"\r\n",
" }\r\n",
" validation_fold: 0\r\n",
"}\r\n",
"augmentation_config {\r\n",
" preprocessing {\r\n",
" output_image_width: 960\r\n",
" output_image_height: 544\r\n",
" min_bbox_width: 1.0\r\n",
" min_bbox_height: 1.0\r\n",
" output_image_channel: 3\r\n",
" enable_auto_resize: true\r\n",
" }\r\n",
" spatial_augmentation {\r\n",
" hflip_probability: 0.5\r\n",
" vflip_probability: 0.0\r\n",
" zoom_min: 1.0\r\n",
" zoom_max: 1.0\r\n",
" translate_max_x: 8.0\r\n",
" translate_max_y: 8.0\r\n",
" }\r\n",
" color_augmentation {\r\n",
" hue_rotation_max: 25.0\r\n",
" saturation_shift_max: 0.20000000298\r\n",
" contrast_scale_max: 0.10000000149\r\n",
" contrast_center: 0.5\r\n",
" }\r\n",
"}\r\n",
"\r\n",
"postprocessing_config {\r\n",
" target_class_config {\r\n",
" key: \"car\"\r\n",
" value {\r\n",
" clustering_config {\r\n",
" clustering_algorithm: DBSCAN\r\n",
" coverage_threshold: 0.005\r\n",
" dbscan_eps: 0.15\r\n",
" dbscan_min_samples: 0.05\r\n",
" minimum_bounding_box_height: 4\r\n",
" dbscan_confidence_threshold: 0.9\r\n",
" }\r\n",
" }\r\n",
" }\r\n",
"}\r\n",
"model_config {\r\n",
" pretrained_model_file: \"/workspace/tao-experiments/detectnet_v2_car/pretrained_trafficcamnet/resnet18_trafficcamnet.tlt\"\r\n",
" num_layers: 18\r\n",
" use_batch_norm: true\r\n",
" objective_set {\r\n",
" bbox {\r\n",
" scale: 35.0\r\n",
" offset: 0.5\r\n",
" }\r\n",
" cov {\r\n",
" }\r\n",
" }\r\n",
" training_precision {\r\n",
" backend_floatx: FLOAT32\r\n",
" }\r\n",
" arch: \"resnet\"\r\n",
" all_projections: true\r\n",
"}\r\n",
"evaluation_config {\r\n",
" validation_period_during_training: 10\r\n",
" first_validation_epoch: 20\r\n",
" minimum_detection_ground_truth_overlap {\r\n",
" key: \"car\"\r\n",
" value: 0.5\r\n",
" }\r\n",
" evaluation_box_config {\r\n",
" key: \"car\"\r\n",
" value {\r\n",
" minimum_height: 20\r\n",
" maximum_height: 9999\r\n",
" minimum_width: 10\r\n",
" maximum_width: 9999\r\n",
" }\r\n",
" }\r\n",
" average_precision_mode: INTEGRATE\r\n",
"}\r\n",
"\r\n",
"cost_function_config {\r\n",
" target_classes {\r\n",
" name: \"car\"\r\n",
" class_weight: 1.0\r\n",
" coverage_foreground_weight: 0.05\r\n",
" objectives {\r\n",
" name: \"cov\"\r\n",
" initial_weight: 1.0\r\n",
" weight_target: 1.0\r\n",
" }\r\n",
" objectives {\r\n",
" name: \"bbox\"\r\n",
" initial_weight: 10.0\r\n",
" weight_target: 10.0\r\n",
" }\r\n",
" }\r\n",
" enable_autoweighting: true\r\n",
" max_objective_weight: 0.999899983406\r\n",
" min_objective_weight: 9.99999974738e-05\r\n",
"}\r\n",
"training_config {\r\n",
" batch_size_per_gpu: 8\r\n",
" num_epochs:120\r\n",
" enable_qat:true\r\n",
" learning_rate {\r\n",
" soft_start_annealing_schedule {\r\n",
" min_learning_rate: 5e-06\r\n",
" max_learning_rate: 1e-03\r\n",
" soft_start: 0.10000000149\r\n",
" annealing: 0.699999988079\r\n",
" }\r\n",
" }\r\n",
" regularizer {\r\n",
" type: L1\r\n",
" weight: 3.00000002618e-09\r\n",
" }\r\n",
" optimizer {\r\n",
" adam {\r\n",
" epsilon: 9.99999993923e-09\r\n",
" beta1: 0.899999976158\r\n",
" beta2: 0.999000012875\r\n",
" }\r\n",
" }\r\n",
" cost_scaling {\r\n",
" enabled: False\r\n",
" initial_exponent: 20.0\r\n",
" increment: 0.005\r\n",
" decrement: 1.0\r\n",
" }\r\n",
" checkpoint_interval: 10\r\n",
"}\r\n",
"bbox_rasterizer_config {\r\n",
" target_class_config {\r\n",
" key: \"car\"\r\n",
" value: {\r\n",
" cov_center_x: 0.5\r\n",
" cov_center_y: 0.5\r\n",
" cov_radius_x: 0.4\r\n",
" cov_radius_y: 0.4\r\n",
" bbox_min_radius: 1.0\r\n",
" }\r\n",
" }\r\n",
" deadzone_radius: 0.4\r\n",
"}\r\n",
"\r\n"
]
}
],
"source": [
"!cat $LOCAL_SPECS_DIR/detectnet_v2_train_resnet18_kitti_car.txt"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 4. Run TAO training \n",
"* Provide the sample spec file and the output directory location for models\n",
"\n",
"*Note: The training may take hours to complete. Also, the remaining notebook, assumes that the training was done in single-GPU mode. When run in multi-GPU mode, please expect to update the pruning and inference steps with new pruning thresholds and updated parameters in the clusterfile.json accordingly for optimum performance.*\n",
"\n",
"*Detectnet_v2 now supports restart from checkpoint. Incase the training job is killed prematurely, you may resume training from the closest checkpoint by simply re-running the **same** command line. Please do make sure to use the **same number of GPUs** when restarting the training.*\n",
"\n",
"*When running the training with NUM_GPUs>1, you may need to modify the `batc_size_per_gpu` and `learning_rate` to get similar mAP as a 1GPU training run. In most cases, scaling down the batch-size by a factor of NUM_GPU's or scaling up the learning rate by a factor of NUM_GPU's would be a good place to start.* "
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
" !$SPECS_DIR"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"2021-12-30 17:07:03,018 [INFO] root: Registry: ['nvcr.io']\n",
"Matplotlib created a temporary config/cache directory at /tmp/matplotlib-8goo7gpw because the default path (/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.\n",
"Using TensorFlow backend.\n",
"WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.\n",
"Using TensorFlow backend.\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:43: The name tf.train.SessionRunHook is deprecated. Please use tf.estimator.SessionRunHook instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/checkpoint_saver_hook.py:25: The name tf.train.CheckpointSaverHook is deprecated. Please use tf.estimator.CheckpointSaverHook instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/scripts/train.py:68: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/scripts/train.py:68: The name tf.logging.INFO is deprecated. Please use tf.compat.v1.logging.INFO instead.\n",
"\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod/tensorflow/__init__.py:117: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.\n",
"\n",
"2021-12-30 09:07:08,663 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/horovod/tensorflow/__init__.py:117: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.\n",
"\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod/tensorflow/__init__.py:143: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.\n",
"\n",
"2021-12-30 09:07:08,663 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/horovod/tensorflow/__init__.py:143: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.\n",
"\n",
"2021-12-30 09:07:09,036 [INFO] __main__: Loading experiment spec at /workspace/tao-experiments/detectnet_v2_car/specs/detectnet_v2_train_resnet18_kitti_car.txt.\n",
"2021-12-30 09:07:09,037 [INFO] iva.detectnet_v2.spec_handler.spec_loader: Merging specification from /workspace/tao-experiments/detectnet_v2_car/specs/detectnet_v2_train_resnet18_kitti_car.txt\n",
"2021-12-30 09:07:09,151 [INFO] __main__: Cannot iterate over exactly 761 samples with a batch size of 8; each epoch will therefore take one extra step.\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:107: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.\n",
"\n",
"2021-12-30 09:07:09,152 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:107: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:110: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.\n",
"\n",
"2021-12-30 09:07:09,153 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:110: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:113: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead.\n",
"\n",
"2021-12-30 09:07:09,155 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:113: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead.\n",
"\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.\n",
"\n",
"2021-12-30 09:07:09,173 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.\n",
"\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.\n",
"\n",
"2021-12-30 09:07:09,174 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.\n",
"\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1834: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead.\n",
"\n",
"2021-12-30 09:07:09,190 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1834: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead.\n",
"\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.\n",
"\n",
"2021-12-30 09:07:10,692 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.\n",
"\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead.\n",
"\n",
"2021-12-30 09:07:10,692 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead.\n",
"\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead.\n",
"\n",
"2021-12-30 09:07:10,936 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead.\n",
"\n",
"/usr/local/lib/python3.6/dist-packages/keras/engine/saving.py:292: UserWarning: No training configuration found in save file: the model was *not* compiled. Compile it manually.\n",
" warnings.warn('No training configuration found in save file: '\n",
"2021-12-30 09:07:16,935 [INFO] iva.detectnet_v2.objectives.bbox_objective: Default L1 loss function will be used.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"2021-12-30 09:07:16,957 [INFO] iva.detectnet_v2.model.detectnet_model: Converting the keras model to quantize keras model.\n",
"__________________________________________________________________________________________________\n",
"Layer (type) Output Shape Param # Connected to \n",
"==================================================================================================\n",
"input_1 (InputLayer) (None, 3, 544, 960) 0 \n",
"__________________________________________________________________________________________________\n",
"input_1_qdq (QDQ) (None, 3, 544, 960) 1 input_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"conv1 (QuantizedConv2D) (None, 64, 272, 480) 9472 input_1_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"bn_conv1 (BatchNormalization) (None, 64, 272, 480) 256 conv1[0][0] \n",
"__________________________________________________________________________________________________\n",
"activation_1 (ReLU) (None, 64, 272, 480) 0 bn_conv1[0][0] \n",
"__________________________________________________________________________________________________\n",
"activation_1_qdq (QDQ) (None, 64, 272, 480) 1 activation_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1a_conv_1 (QuantizedConv2 (None, 64, 136, 240) 36928 activation_1_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1a_bn_1 (BatchNormalizati (None, 64, 136, 240) 256 block_1a_conv_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1a_relu_1 (ReLU) (None, 64, 136, 240) 0 block_1a_bn_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1a_relu_1_qdq (QDQ) (None, 64, 136, 240) 1 block_1a_relu_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1a_conv_2 (QuantizedConv2 (None, 64, 136, 240) 36928 block_1a_relu_1_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1a_conv_shortcut (Quantiz (None, 64, 136, 240) 4160 activation_1_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1a_bn_2 (BatchNormalizati (None, 64, 136, 240) 256 block_1a_conv_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1a_bn_shortcut (BatchNorm (None, 64, 136, 240) 256 block_1a_conv_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1a_bn_2_qdq (QDQ) (None, 64, 136, 240) 1 block_1a_bn_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1a_bn_shortcut_qdq (QDQ) (None, 64, 136, 240) 1 block_1a_bn_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_1 (Add) (None, 64, 136, 240) 0 block_1a_bn_2_qdq[0][0] \n",
" block_1a_bn_shortcut_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_1_qdq (QDQ) (None, 64, 136, 240) 1 add_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1a_relu (ReLU) (None, 64, 136, 240) 0 add_1_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1a_relu_qdq (QDQ) (None, 64, 136, 240) 1 block_1a_relu[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1b_conv_1 (QuantizedConv2 (None, 64, 136, 240) 36928 block_1a_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1b_bn_1 (BatchNormalizati (None, 64, 136, 240) 256 block_1b_conv_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1b_relu_1 (ReLU) (None, 64, 136, 240) 0 block_1b_bn_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1b_relu_1_qdq (QDQ) (None, 64, 136, 240) 1 block_1b_relu_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1b_conv_2 (QuantizedConv2 (None, 64, 136, 240) 36928 block_1b_relu_1_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1b_conv_shortcut (Quantiz (None, 64, 136, 240) 4160 block_1a_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1b_bn_2 (BatchNormalizati (None, 64, 136, 240) 256 block_1b_conv_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1b_bn_shortcut (BatchNorm (None, 64, 136, 240) 256 block_1b_conv_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1b_bn_2_qdq (QDQ) (None, 64, 136, 240) 1 block_1b_bn_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1b_bn_shortcut_qdq (QDQ) (None, 64, 136, 240) 1 block_1b_bn_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_2 (Add) (None, 64, 136, 240) 0 block_1b_bn_2_qdq[0][0] \n",
" block_1b_bn_shortcut_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_2_qdq (QDQ) (None, 64, 136, 240) 1 add_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1b_relu (ReLU) (None, 64, 136, 240) 0 add_2_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1b_relu_qdq (QDQ) (None, 64, 136, 240) 1 block_1b_relu[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2a_conv_1 (QuantizedConv2 (None, 128, 68, 120) 73856 block_1b_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2a_bn_1 (BatchNormalizati (None, 128, 68, 120) 512 block_2a_conv_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2a_relu_1 (ReLU) (None, 128, 68, 120) 0 block_2a_bn_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2a_relu_1_qdq (QDQ) (None, 128, 68, 120) 1 block_2a_relu_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2a_conv_2 (QuantizedConv2 (None, 128, 68, 120) 147584 block_2a_relu_1_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2a_conv_shortcut (Quantiz (None, 128, 68, 120) 8320 block_1b_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2a_bn_2 (BatchNormalizati (None, 128, 68, 120) 512 block_2a_conv_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2a_bn_shortcut (BatchNorm (None, 128, 68, 120) 512 block_2a_conv_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2a_bn_2_qdq (QDQ) (None, 128, 68, 120) 1 block_2a_bn_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2a_bn_shortcut_qdq (QDQ) (None, 128, 68, 120) 1 block_2a_bn_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_3 (Add) (None, 128, 68, 120) 0 block_2a_bn_2_qdq[0][0] \n",
" block_2a_bn_shortcut_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_3_qdq (QDQ) (None, 128, 68, 120) 1 add_3[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2a_relu (ReLU) (None, 128, 68, 120) 0 add_3_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2a_relu_qdq (QDQ) (None, 128, 68, 120) 1 block_2a_relu[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2b_conv_1 (QuantizedConv2 (None, 128, 68, 120) 147584 block_2a_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2b_bn_1 (BatchNormalizati (None, 128, 68, 120) 512 block_2b_conv_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2b_relu_1 (ReLU) (None, 128, 68, 120) 0 block_2b_bn_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2b_relu_1_qdq (QDQ) (None, 128, 68, 120) 1 block_2b_relu_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2b_conv_2 (QuantizedConv2 (None, 128, 68, 120) 147584 block_2b_relu_1_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2b_conv_shortcut (Quantiz (None, 128, 68, 120) 16512 block_2a_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2b_bn_2 (BatchNormalizati (None, 128, 68, 120) 512 block_2b_conv_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2b_bn_shortcut (BatchNorm (None, 128, 68, 120) 512 block_2b_conv_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2b_bn_2_qdq (QDQ) (None, 128, 68, 120) 1 block_2b_bn_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2b_bn_shortcut_qdq (QDQ) (None, 128, 68, 120) 1 block_2b_bn_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_4 (Add) (None, 128, 68, 120) 0 block_2b_bn_2_qdq[0][0] \n",
" block_2b_bn_shortcut_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_4_qdq (QDQ) (None, 128, 68, 120) 1 add_4[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2b_relu (ReLU) (None, 128, 68, 120) 0 add_4_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2b_relu_qdq (QDQ) (None, 128, 68, 120) 1 block_2b_relu[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3a_conv_1 (QuantizedConv2 (None, 256, 34, 60) 295168 block_2b_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3a_bn_1 (BatchNormalizati (None, 256, 34, 60) 1024 block_3a_conv_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3a_relu_1 (ReLU) (None, 256, 34, 60) 0 block_3a_bn_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3a_relu_1_qdq (QDQ) (None, 256, 34, 60) 1 block_3a_relu_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3a_conv_2 (QuantizedConv2 (None, 256, 34, 60) 590080 block_3a_relu_1_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3a_conv_shortcut (Quantiz (None, 256, 34, 60) 33024 block_2b_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3a_bn_2 (BatchNormalizati (None, 256, 34, 60) 1024 block_3a_conv_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3a_bn_shortcut (BatchNorm (None, 256, 34, 60) 1024 block_3a_conv_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3a_bn_2_qdq (QDQ) (None, 256, 34, 60) 1 block_3a_bn_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3a_bn_shortcut_qdq (QDQ) (None, 256, 34, 60) 1 block_3a_bn_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_5 (Add) (None, 256, 34, 60) 0 block_3a_bn_2_qdq[0][0] \n",
" block_3a_bn_shortcut_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_5_qdq (QDQ) (None, 256, 34, 60) 1 add_5[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3a_relu (ReLU) (None, 256, 34, 60) 0 add_5_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3a_relu_qdq (QDQ) (None, 256, 34, 60) 1 block_3a_relu[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3b_conv_1 (QuantizedConv2 (None, 256, 34, 60) 590080 block_3a_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3b_bn_1 (BatchNormalizati (None, 256, 34, 60) 1024 block_3b_conv_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3b_relu_1 (ReLU) (None, 256, 34, 60) 0 block_3b_bn_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3b_relu_1_qdq (QDQ) (None, 256, 34, 60) 1 block_3b_relu_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3b_conv_2 (QuantizedConv2 (None, 256, 34, 60) 590080 block_3b_relu_1_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3b_conv_shortcut (Quantiz (None, 256, 34, 60) 65792 block_3a_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3b_bn_2 (BatchNormalizati (None, 256, 34, 60) 1024 block_3b_conv_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3b_bn_shortcut (BatchNorm (None, 256, 34, 60) 1024 block_3b_conv_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3b_bn_2_qdq (QDQ) (None, 256, 34, 60) 1 block_3b_bn_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3b_bn_shortcut_qdq (QDQ) (None, 256, 34, 60) 1 block_3b_bn_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_6 (Add) (None, 256, 34, 60) 0 block_3b_bn_2_qdq[0][0] \n",
" block_3b_bn_shortcut_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_6_qdq (QDQ) (None, 256, 34, 60) 1 add_6[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3b_relu (ReLU) (None, 256, 34, 60) 0 add_6_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3b_relu_qdq (QDQ) (None, 256, 34, 60) 1 block_3b_relu[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4a_conv_1 (QuantizedConv2 (None, 512, 34, 60) 1180160 block_3b_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4a_bn_1 (BatchNormalizati (None, 512, 34, 60) 2048 block_4a_conv_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4a_relu_1 (ReLU) (None, 512, 34, 60) 0 block_4a_bn_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4a_relu_1_qdq (QDQ) (None, 512, 34, 60) 1 block_4a_relu_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4a_conv_2 (QuantizedConv2 (None, 512, 34, 60) 2359808 block_4a_relu_1_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4a_conv_shortcut (Quantiz (None, 512, 34, 60) 131584 block_3b_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4a_bn_2 (BatchNormalizati (None, 512, 34, 60) 2048 block_4a_conv_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4a_bn_shortcut (BatchNorm (None, 512, 34, 60) 2048 block_4a_conv_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4a_bn_2_qdq (QDQ) (None, 512, 34, 60) 1 block_4a_bn_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4a_bn_shortcut_qdq (QDQ) (None, 512, 34, 60) 1 block_4a_bn_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_7 (Add) (None, 512, 34, 60) 0 block_4a_bn_2_qdq[0][0] \n",
" block_4a_bn_shortcut_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_7_qdq (QDQ) (None, 512, 34, 60) 1 add_7[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4a_relu (ReLU) (None, 512, 34, 60) 0 add_7_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4a_relu_qdq (QDQ) (None, 512, 34, 60) 1 block_4a_relu[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4b_conv_1 (QuantizedConv2 (None, 512, 34, 60) 2359808 block_4a_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4b_bn_1 (BatchNormalizati (None, 512, 34, 60) 2048 block_4b_conv_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4b_relu_1 (ReLU) (None, 512, 34, 60) 0 block_4b_bn_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4b_relu_1_qdq (QDQ) (None, 512, 34, 60) 1 block_4b_relu_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4b_conv_2 (QuantizedConv2 (None, 512, 34, 60) 2359808 block_4b_relu_1_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4b_conv_shortcut (Quantiz (None, 512, 34, 60) 262656 block_4a_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4b_bn_2 (BatchNormalizati (None, 512, 34, 60) 2048 block_4b_conv_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4b_bn_shortcut (BatchNorm (None, 512, 34, 60) 2048 block_4b_conv_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4b_bn_2_qdq (QDQ) (None, 512, 34, 60) 1 block_4b_bn_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4b_bn_shortcut_qdq (QDQ) (None, 512, 34, 60) 1 block_4b_bn_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_8 (Add) (None, 512, 34, 60) 0 block_4b_bn_2_qdq[0][0] \n",
" block_4b_bn_shortcut_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_8_qdq (QDQ) (None, 512, 34, 60) 1 add_8[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4b_relu (ReLU) (None, 512, 34, 60) 0 add_8_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4b_relu_qdq (QDQ) (None, 512, 34, 60) 1 block_4b_relu[0][0] \n",
"__________________________________________________________________________________________________\n",
"output_bbox (Conv2D) (None, 4, 34, 60) 2052 block_4b_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"output_cov (Conv2D) (None, 1, 34, 60) 513 block_4b_relu_qdq[0][0] \n",
"==================================================================================================\n",
"Total params: 11,550,895\n",
"Trainable params: 11,539,205\n",
"Non-trainable params: 11,690\n",
"__________________________________________________________________________________________________\n",
"2021-12-30 09:07:41,158 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Serial augmentation enabled = False\n",
"2021-12-30 09:07:41,158 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Pseudo sharding enabled = False\n",
"2021-12-30 09:07:41,158 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Max Image Dimensions (all sources): (0, 0)\n",
"2021-12-30 09:07:41,158 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: number of cpus: 16, io threads: 32, compute threads: 16, buffered batches: 4\n",
"2021-12-30 09:07:41,158 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: total dataset size 761, number of sources: 1, batch size per gpu: 8, steps: 96\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.\n",
"\n",
"2021-12-30 09:07:41,189 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.\n",
"\n",
"WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code\n",
"2021-12-30 09:07:41,230 [WARNING] tensorflow: Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code\n",
"2021-12-30 09:07:41,248 [INFO] iva.detectnet_v2.dataloader.default_dataloader: Bounding box coordinates were detected in the input specification! Bboxes will be automatically converted to polygon coordinates.\n",
"2021-12-30 09:07:41,450 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: shuffle: True - shard 0 of 1\n",
"2021-12-30 09:07:41,456 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: sampling 1 datasets with weights:\n",
"2021-12-30 09:07:41,456 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: source: 0 weight: 1.000000\n",
"WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code\n",
"2021-12-30 09:07:41,467 [WARNING] tensorflow: Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/core/build_wheel.runfiles/ai_infra/moduluspy/modulus/blocks/data_loaders/multi_source_loader/types/images2d_reference.py:427: The name tf.image.resize_images is deprecated. Please use tf.image.resize instead.\n",
"\n",
"2021-12-30 09:07:41,486 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/core/build_wheel.runfiles/ai_infra/moduluspy/modulus/blocks/data_loaders/multi_source_loader/types/images2d_reference.py:427: The name tf.image.resize_images is deprecated. Please use tf.image.resize instead.\n",
"\n",
"2021-12-30 09:07:41,762 [INFO] __main__: Found 761 samples in training set\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/rasterizers/bbox_rasterizer.py:347: The name tf.bincount is deprecated. Please use tf.math.bincount instead.\n",
"\n",
"2021-12-30 09:07:41,845 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/rasterizers/bbox_rasterizer.py:347: The name tf.bincount is deprecated. Please use tf.math.bincount instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/training_proto_utilities.py:89: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.\n",
"\n",
"2021-12-30 09:07:41,999 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/training_proto_utilities.py:89: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/training_proto_utilities.py:36: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead.\n",
"\n",
"2021-12-30 09:07:42,011 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/training_proto_utilities.py:36: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_functions.py:17: The name tf.log is deprecated. Please use tf.math.log instead.\n",
"\n",
"2021-12-30 09:07:42,674 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_functions.py:17: The name tf.log is deprecated. Please use tf.math.log instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:235: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead.\n",
"\n",
"2021-12-30 09:07:42,682 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:235: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/model/detectnet_model.py:587: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead.\n",
"\n",
"2021-12-30 09:07:42,685 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/model/detectnet_model.py:587: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead.\n",
"\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"2021-12-30 09:07:43,967 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Serial augmentation enabled = False\n",
"2021-12-30 09:07:43,967 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Pseudo sharding enabled = False\n",
"2021-12-30 09:07:43,967 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Max Image Dimensions (all sources): (0, 0)\n",
"2021-12-30 09:07:43,967 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: number of cpus: 16, io threads: 32, compute threads: 16, buffered batches: 4\n",
"2021-12-30 09:07:43,968 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: total dataset size 190, number of sources: 1, batch size per gpu: 8, steps: 24\n",
"WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code\n",
"2021-12-30 09:07:43,975 [WARNING] tensorflow: Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code\n",
"2021-12-30 09:07:43,992 [INFO] iva.detectnet_v2.dataloader.default_dataloader: Bounding box coordinates were detected in the input specification! Bboxes will be automatically converted to polygon coordinates.\n",
"2021-12-30 09:07:44,187 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: shuffle: False - shard 0 of 1\n",
"2021-12-30 09:07:44,191 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: sampling 1 datasets with weights:\n",
"2021-12-30 09:07:44,191 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: source: 0 weight: 1.000000\n",
"WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code\n",
"2021-12-30 09:07:44,202 [WARNING] tensorflow: Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code\n",
"2021-12-30 09:07:44,394 [INFO] __main__: Found 190 samples in validation set\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/validation_hook.py:40: The name tf.summary.FileWriterCache is deprecated. Please use tf.compat.v1.summary.FileWriterCache instead.\n",
"\n",
"2021-12-30 09:07:44,948 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/validation_hook.py:40: The name tf.summary.FileWriterCache is deprecated. Please use tf.compat.v1.summary.FileWriterCache instead.\n",
"\n",
"2021-12-30 09:07:46,229 [INFO] __main__: Checkpoint interval: 10\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/scripts/train.py:108: The name tf.train.Scaffold is deprecated. Please use tf.compat.v1.train.Scaffold instead.\n",
"\n",
"2021-12-30 09:07:46,230 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/scripts/train.py:108: The name tf.train.Scaffold is deprecated. Please use tf.compat.v1.train.Scaffold instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:14: The name tf.local_variables_initializer is deprecated. Please use tf.compat.v1.local_variables_initializer instead.\n",
"\n",
"2021-12-30 09:07:46,230 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:14: The name tf.local_variables_initializer is deprecated. Please use tf.compat.v1.local_variables_initializer instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:15: The name tf.tables_initializer is deprecated. Please use tf.compat.v1.tables_initializer instead.\n",
"\n",
"2021-12-30 09:07:46,230 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:15: The name tf.tables_initializer is deprecated. Please use tf.compat.v1.tables_initializer instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:16: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead.\n",
"\n",
"2021-12-30 09:07:46,231 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:16: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/utils.py:59: The name tf.train.LoggingTensorHook is deprecated. Please use tf.estimator.LoggingTensorHook instead.\n",
"\n",
"2021-12-30 09:07:46,233 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/utils.py:59: The name tf.train.LoggingTensorHook is deprecated. Please use tf.estimator.LoggingTensorHook instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/utils.py:60: The name tf.train.StopAtStepHook is deprecated. Please use tf.estimator.StopAtStepHook instead.\n",
"\n",
"2021-12-30 09:07:46,233 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/utils.py:60: The name tf.train.StopAtStepHook is deprecated. Please use tf.estimator.StopAtStepHook instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/utils.py:73: The name tf.train.StepCounterHook is deprecated. Please use tf.estimator.StepCounterHook instead.\n",
"\n",
"2021-12-30 09:07:46,233 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/utils.py:73: The name tf.train.StepCounterHook is deprecated. Please use tf.estimator.StepCounterHook instead.\n",
"\n",
"INFO:tensorflow:Create CheckpointSaverHook.\n",
"2021-12-30 09:07:46,233 [INFO] tensorflow: Create CheckpointSaverHook.\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/utils.py:99: The name tf.train.SummarySaverHook is deprecated. Please use tf.estimator.SummarySaverHook instead.\n",
"\n",
"2021-12-30 09:07:46,233 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/utils.py:99: The name tf.train.SummarySaverHook is deprecated. Please use tf.estimator.SummarySaverHook instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/utilities.py:140: The name tf.train.SingularMonitoredSession is deprecated. Please use tf.compat.v1.train.SingularMonitoredSession instead.\n",
"\n",
"2021-12-30 09:07:46,234 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/utilities.py:140: The name tf.train.SingularMonitoredSession is deprecated. Please use tf.compat.v1.train.SingularMonitoredSession instead.\n",
"\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO:tensorflow:Graph was finalized.\n",
"2021-12-30 09:07:47,404 [INFO] tensorflow: Graph was finalized.\n",
"INFO:tensorflow:Running local_init_op.\n",
"2021-12-30 09:07:49,205 [INFO] tensorflow: Running local_init_op.\n",
"INFO:tensorflow:Done running local_init_op.\n",
"2021-12-30 09:07:49,736 [INFO] tensorflow: Done running local_init_op.\n",
"INFO:tensorflow:Saving checkpoints for step-0.\n",
"2021-12-30 09:07:58,014 [INFO] tensorflow: Saving checkpoints for step-0.\n",
"INFO:tensorflow:epoch = 0.0, learning_rate = 4.9999994e-06, loss = 0.058969934, step = 0\n",
"2021-12-30 09:09:54,118 [INFO] tensorflow: epoch = 0.0, learning_rate = 4.9999994e-06, loss = 0.058969934, step = 0\n",
"2021-12-30 09:09:54,120 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 0/120: loss: 0.05897 learning rate: 0.00000 Time taken: 0:00:00 ETA: 0:00:00\n",
"2021-12-30 09:09:54,120 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 0.091\n",
"INFO:tensorflow:global_step/sec: 2.07329\n",
"2021-12-30 09:09:58,460 [INFO] tensorflow: global_step/sec: 2.07329\n",
"INFO:tensorflow:epoch = 0.13541666666666666, learning_rate = 5.308065e-06, loss = 0.058166612, step = 13 (5.617 sec)\n",
"2021-12-30 09:09:59,735 [INFO] tensorflow: epoch = 0.13541666666666666, learning_rate = 5.308065e-06, loss = 0.058166612, step = 13 (5.617 sec)\n",
"INFO:tensorflow:global_step/sec: 3.16088\n",
"2021-12-30 09:10:01,307 [INFO] tensorflow: global_step/sec: 3.16088\n",
"2021-12-30 09:10:03,203 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 2.054\n",
"INFO:tensorflow:global_step/sec: 3.16292\n",
"2021-12-30 09:10:04,153 [INFO] tensorflow: global_step/sec: 3.16292\n",
"INFO:tensorflow:epoch = 0.3125, learning_rate = 5.739743e-06, loss = 0.05658871, step = 30 (5.351 sec)\n",
"2021-12-30 09:10:05,086 [INFO] tensorflow: epoch = 0.3125, learning_rate = 5.739743e-06, loss = 0.05658871, step = 30 (5.351 sec)\n",
"INFO:tensorflow:global_step/sec: 3.20974\n",
"2021-12-30 09:10:06,957 [INFO] tensorflow: global_step/sec: 3.20974\n",
"INFO:tensorflow:global_step/sec: 3.12829\n",
"2021-12-30 09:10:09,833 [INFO] tensorflow: global_step/sec: 3.12829\n",
"INFO:tensorflow:epoch = 0.4895833333333333, learning_rate = 6.2065265e-06, loss = 0.05632484, step = 47 (5.397 sec)\n",
"2021-12-30 09:10:10,483 [INFO] tensorflow: epoch = 0.4895833333333333, learning_rate = 6.2065265e-06, loss = 0.05632484, step = 47 (5.397 sec)\n",
"2021-12-30 09:10:11,117 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.272\n",
"INFO:tensorflow:global_step/sec: 3.13273\n",
"2021-12-30 09:10:12,706 [INFO] tensorflow: global_step/sec: 3.13273\n",
"INFO:tensorflow:global_step/sec: 3.17394\n",
"2021-12-30 09:10:15,542 [INFO] tensorflow: global_step/sec: 3.17394\n",
"INFO:tensorflow:epoch = 0.6666666666666666, learning_rate = 6.711271e-06, loss = 0.054740362, step = 64 (5.353 sec)\n",
"2021-12-30 09:10:15,836 [INFO] tensorflow: epoch = 0.6666666666666666, learning_rate = 6.711271e-06, loss = 0.054740362, step = 64 (5.353 sec)\n",
"INFO:tensorflow:global_step/sec: 3.24324\n",
"2021-12-30 09:10:18,317 [INFO] tensorflow: global_step/sec: 3.24324\n",
"2021-12-30 09:10:18,963 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.493\n",
"INFO:tensorflow:global_step/sec: 3.1651\n",
"2021-12-30 09:10:21,160 [INFO] tensorflow: global_step/sec: 3.1651\n",
"INFO:tensorflow:epoch = 0.8541666666666666, learning_rate = 7.290521e-06, loss = 0.05351142, step = 82 (5.631 sec)\n",
"2021-12-30 09:10:21,467 [INFO] tensorflow: epoch = 0.8541666666666666, learning_rate = 7.290521e-06, loss = 0.05351142, step = 82 (5.631 sec)\n",
"INFO:tensorflow:global_step/sec: 3.14904\n",
"2021-12-30 09:10:24,019 [INFO] tensorflow: global_step/sec: 3.14904\n",
"65e7325bdf4f:62:98 [0] NCCL INFO Bootstrap : Using [0]lo:127.0.0.1<0> [1]eth0:172.17.0.30<0>\n",
"65e7325bdf4f:62:98 [0] NCCL INFO NET/Plugin : Plugin load returned 0 : libnccl-net.so: cannot open shared object file: No such file or directory.\n",
"65e7325bdf4f:62:98 [0] NCCL INFO NET/IB : No device found.\n",
"65e7325bdf4f:62:98 [0] NCCL INFO NET/Socket : Using [0]lo:127.0.0.1<0> [1]eth0:172.17.0.30<0>\n",
"65e7325bdf4f:62:98 [0] NCCL INFO Using network Socket\n",
"NCCL version 2.7.8+cuda11.1\n",
"65e7325bdf4f:62:98 [0] NCCL INFO Channel 00/32 : 0\n",
"65e7325bdf4f:62:98 [0] NCCL INFO Channel 01/32 : 0\n",
"65e7325bdf4f:62:98 [0] NCCL INFO Channel 02/32 : 0\n",
"65e7325bdf4f:62:98 [0] NCCL INFO Channel 03/32 : 0\n",
"65e7325bdf4f:62:98 [0] NCCL INFO Channel 04/32 : 0\n",
"65e7325bdf4f:62:98 [0] NCCL INFO Channel 05/32 : 0\n",
"65e7325bdf4f:62:98 [0] NCCL INFO Channel 06/32 : 0\n",
"65e7325bdf4f:62:98 [0] NCCL INFO Channel 07/32 : 0\n",
"65e7325bdf4f:62:98 [0] NCCL INFO Channel 08/32 : 0\n",
"65e7325bdf4f:62:98 [0] NCCL INFO Channel 09/32 : 0\n",
"65e7325bdf4f:62:98 [0] NCCL INFO Channel 10/32 : 0\n",
"65e7325bdf4f:62:98 [0] NCCL INFO Channel 11/32 : 0\n",
"65e7325bdf4f:62:98 [0] NCCL INFO Channel 12/32 : 0\n",
"65e7325bdf4f:62:98 [0] NCCL INFO Channel 13/32 : 0\n",
"65e7325bdf4f:62:98 [0] NCCL INFO Channel 14/32 : 0\n",
"65e7325bdf4f:62:98 [0] NCCL INFO Channel 15/32 : 0\n",
"65e7325bdf4f:62:98 [0] NCCL INFO Channel 16/32 : 0\n",
"65e7325bdf4f:62:98 [0] NCCL INFO Channel 17/32 : 0\n",
"65e7325bdf4f:62:98 [0] NCCL INFO Channel 18/32 : 0\n",
"65e7325bdf4f:62:98 [0] NCCL INFO Channel 19/32 : 0\n",
"65e7325bdf4f:62:98 [0] NCCL INFO Channel 20/32 : 0\n",
"65e7325bdf4f:62:98 [0] NCCL INFO Channel 21/32 : 0\n",
"65e7325bdf4f:62:98 [0] NCCL INFO Channel 22/32 : 0\n",
"65e7325bdf4f:62:98 [0] NCCL INFO Channel 23/32 : 0\n",
"65e7325bdf4f:62:98 [0] NCCL INFO Channel 24/32 : 0\n",
"65e7325bdf4f:62:98 [0] NCCL INFO Channel 25/32 : 0\n",
"65e7325bdf4f:62:98 [0] NCCL INFO Channel 26/32 : 0\n",
"65e7325bdf4f:62:98 [0] NCCL INFO Channel 27/32 : 0\n",
"65e7325bdf4f:62:98 [0] NCCL INFO Channel 28/32 : 0\n",
"65e7325bdf4f:62:98 [0] NCCL INFO Channel 29/32 : 0\n",
"65e7325bdf4f:62:98 [0] NCCL INFO Channel 30/32 : 0\n",
"65e7325bdf4f:62:98 [0] NCCL INFO Channel 31/32 : 0\n",
"65e7325bdf4f:62:98 [0] NCCL INFO Trees [0] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [1] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [2] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [3] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [4] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [5] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [6] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [7] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [8] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [9] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [10] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [11] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [12] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [13] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [14] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [15] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [16] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [17] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [18] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [19] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [20] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [21] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [22] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [23] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [24] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [25] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [26] -1/-1/-1->0->-1|-1->0->-1/-\n",
"65e7325bdf4f:62:98 [0] NCCL INFO 32 coll channels, 32 p2p channels, 32 p2p channels per peer\n",
"65e7325bdf4f:62:98 [0] NCCL INFO comm 0x7f22943257e0 rank 0 nranks 1 cudaDev 0 busId 1000 - Init COMPLETE\n",
"2021-12-30 09:10:26,254 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 1/120: loss: 0.00233 learning rate: 0.00001 Time taken: 0:02:00.077063 ETA: 3:58:09.170450\n",
"INFO:tensorflow:epoch = 1.0208333333333333, learning_rate = 7.847244e-06, loss = 0.0024794308, step = 98 (5.417 sec)\n",
"2021-12-30 09:10:26,884 [INFO] tensorflow: epoch = 1.0208333333333333, learning_rate = 7.847244e-06, loss = 0.0024794308, step = 98 (5.417 sec)\n",
"INFO:tensorflow:global_step/sec: 2.8276\n",
"2021-12-30 09:10:27,201 [INFO] tensorflow: global_step/sec: 2.8276\n",
"2021-12-30 09:10:27,202 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.275\n",
"INFO:tensorflow:global_step/sec: 3.12407\n",
"2021-12-30 09:10:30,082 [INFO] tensorflow: global_step/sec: 3.12407\n",
"INFO:tensorflow:epoch = 1.1979166666666665, learning_rate = 8.485419e-06, loss = 0.0027224442, step = 115 (5.440 sec)\n",
"2021-12-30 09:10:32,324 [INFO] tensorflow: epoch = 1.1979166666666665, learning_rate = 8.485419e-06, loss = 0.0027224442, step = 115 (5.440 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10813\n",
"2021-12-30 09:10:32,978 [INFO] tensorflow: global_step/sec: 3.10813\n",
"2021-12-30 09:10:35,211 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.973\n",
"INFO:tensorflow:global_step/sec: 3.13479\n",
"2021-12-30 09:10:35,849 [INFO] tensorflow: global_step/sec: 3.13479\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO:tensorflow:epoch = 1.375, learning_rate = 9.175495e-06, loss = 0.0029616202, step = 132 (5.456 sec)\n",
"2021-12-30 09:10:37,781 [INFO] tensorflow: epoch = 1.375, learning_rate = 9.175495e-06, loss = 0.0029616202, step = 132 (5.456 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12555\n",
"2021-12-30 09:10:38,728 [INFO] tensorflow: global_step/sec: 3.12555\n",
"INFO:tensorflow:global_step/sec: 3.13134\n",
"2021-12-30 09:10:41,603 [INFO] tensorflow: global_step/sec: 3.13134\n",
"INFO:tensorflow:epoch = 1.5520833333333333, learning_rate = 9.92169e-06, loss = 0.0019230827, step = 149 (5.429 sec)\n",
"2021-12-30 09:10:43,210 [INFO] tensorflow: epoch = 1.5520833333333333, learning_rate = 9.92169e-06, loss = 0.0019230827, step = 149 (5.429 sec)\n",
"2021-12-30 09:10:43,210 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.003\n",
"INFO:tensorflow:global_step/sec: 3.11009\n",
"2021-12-30 09:10:44,496 [INFO] tensorflow: global_step/sec: 3.11009\n",
"INFO:tensorflow:global_step/sec: 3.09484\n",
"2021-12-30 09:10:47,404 [INFO] tensorflow: global_step/sec: 3.09484\n",
"INFO:tensorflow:epoch = 1.7291666666666665, learning_rate = 1.07285705e-05, loss = 0.0025290225, step = 166 (5.464 sec)\n",
"2021-12-30 09:10:48,674 [INFO] tensorflow: epoch = 1.7291666666666665, learning_rate = 1.07285705e-05, loss = 0.0025290225, step = 166 (5.464 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11639\n",
"2021-12-30 09:10:50,292 [INFO] tensorflow: global_step/sec: 3.11639\n",
"2021-12-30 09:10:51,235 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.925\n",
"INFO:tensorflow:global_step/sec: 3.10336\n",
"2021-12-30 09:10:53,192 [INFO] tensorflow: global_step/sec: 3.10336\n",
"INFO:tensorflow:epoch = 1.90625, learning_rate = 1.1601069e-05, loss = 0.0023484863, step = 183 (5.485 sec)\n",
"2021-12-30 09:10:54,159 [INFO] tensorflow: epoch = 1.90625, learning_rate = 1.1601069e-05, loss = 0.0023484863, step = 183 (5.485 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09429\n",
"2021-12-30 09:10:56,101 [INFO] tensorflow: global_step/sec: 3.09429\n",
"2021-12-30 09:10:57,064 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 2/120: loss: 0.00264 learning rate: 0.00001 Time taken: 0:00:30.822253 ETA: 1:00:37.025825\n",
"INFO:tensorflow:global_step/sec: 3.16367\n",
"2021-12-30 09:10:58,946 [INFO] tensorflow: global_step/sec: 3.16367\n",
"2021-12-30 09:10:59,264 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.911\n",
"INFO:tensorflow:epoch = 2.083333333333333, learning_rate = 1.2544524e-05, loss = 0.001824951, step = 200 (5.445 sec)\n",
"2021-12-30 09:10:59,604 [INFO] tensorflow: epoch = 2.083333333333333, learning_rate = 1.2544524e-05, loss = 0.001824951, step = 200 (5.445 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10605\n",
"2021-12-30 09:11:01,843 [INFO] tensorflow: global_step/sec: 3.10605\n",
"INFO:tensorflow:global_step/sec: 3.11283\n",
"2021-12-30 09:11:04,735 [INFO] tensorflow: global_step/sec: 3.11283\n",
"INFO:tensorflow:epoch = 2.2604166666666665, learning_rate = 1.3564706e-05, loss = 0.0022552179, step = 217 (5.477 sec)\n",
"2021-12-30 09:11:05,081 [INFO] tensorflow: epoch = 2.2604166666666665, learning_rate = 1.3564706e-05, loss = 0.0022552179, step = 217 (5.477 sec)\n",
"2021-12-30 09:11:07,362 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.697\n",
"INFO:tensorflow:global_step/sec: 3.04643\n",
"2021-12-30 09:11:07,689 [INFO] tensorflow: global_step/sec: 3.04643\n",
"INFO:tensorflow:epoch = 2.4375, learning_rate = 1.4667854e-05, loss = 0.0023462, step = 234 (5.492 sec)\n",
"2021-12-30 09:11:10,573 [INFO] tensorflow: epoch = 2.4375, learning_rate = 1.4667854e-05, loss = 0.0023462, step = 234 (5.492 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11979\n",
"2021-12-30 09:11:10,574 [INFO] tensorflow: global_step/sec: 3.11979\n",
"INFO:tensorflow:global_step/sec: 3.10684\n",
"2021-12-30 09:11:13,471 [INFO] tensorflow: global_step/sec: 3.10684\n",
"2021-12-30 09:11:15,430 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.791\n",
"INFO:tensorflow:epoch = 2.614583333333333, learning_rate = 1.5860714e-05, loss = 0.0025271776, step = 251 (5.502 sec)\n",
"2021-12-30 09:11:16,075 [INFO] tensorflow: epoch = 2.614583333333333, learning_rate = 1.5860714e-05, loss = 0.0025271776, step = 251 (5.502 sec)\n",
"INFO:tensorflow:global_step/sec: 3.06435\n",
"2021-12-30 09:11:16,408 [INFO] tensorflow: global_step/sec: 3.06435\n",
"INFO:tensorflow:global_step/sec: 3.07721\n",
"2021-12-30 09:11:19,332 [INFO] tensorflow: global_step/sec: 3.07721\n",
"INFO:tensorflow:epoch = 2.7916666666666665, learning_rate = 1.7150585e-05, loss = 0.0026860358, step = 268 (5.526 sec)\n",
"2021-12-30 09:11:21,601 [INFO] tensorflow: epoch = 2.7916666666666665, learning_rate = 1.7150585e-05, loss = 0.0026860358, step = 268 (5.526 sec)\n",
"INFO:tensorflow:global_step/sec: 3.04013\n",
"2021-12-30 09:11:22,293 [INFO] tensorflow: global_step/sec: 3.04013\n",
"2021-12-30 09:11:23,559 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.605\n",
"INFO:tensorflow:global_step/sec: 3.09671\n",
"2021-12-30 09:11:25,199 [INFO] tensorflow: global_step/sec: 3.09671\n",
"INFO:tensorflow:epoch = 2.96875, learning_rate = 1.8545352e-05, loss = 0.0019626052, step = 285 (5.503 sec)\n",
"2021-12-30 09:11:27,105 [INFO] tensorflow: epoch = 2.96875, learning_rate = 1.8545352e-05, loss = 0.0019626052, step = 285 (5.503 sec)\n",
"INFO:tensorflow:global_step/sec: 3.1422\n",
"2021-12-30 09:11:28,063 [INFO] tensorflow: global_step/sec: 3.1422\n",
"2021-12-30 09:11:28,064 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 3/120: loss: 0.00229 learning rate: 0.00002 Time taken: 0:00:30.995686 ETA: 1:00:26.495268\n",
"INFO:tensorflow:global_step/sec: 3.13525\n",
"2021-12-30 09:11:30,934 [INFO] tensorflow: global_step/sec: 3.13525\n",
"2021-12-30 09:11:31,586 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.914\n",
"INFO:tensorflow:epoch = 3.145833333333333, learning_rate = 2.005355e-05, loss = 0.0022796728, step = 302 (5.443 sec)\n",
"2021-12-30 09:11:32,548 [INFO] tensorflow: epoch = 3.145833333333333, learning_rate = 2.005355e-05, loss = 0.0022796728, step = 302 (5.443 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10642\n",
"2021-12-30 09:11:33,831 [INFO] tensorflow: global_step/sec: 3.10642\n",
"INFO:tensorflow:global_step/sec: 3.10113\n",
"2021-12-30 09:11:36,733 [INFO] tensorflow: global_step/sec: 3.10113\n",
"INFO:tensorflow:epoch = 3.3229166666666665, learning_rate = 2.1684402e-05, loss = 0.002564355, step = 319 (5.501 sec)\n",
"2021-12-30 09:11:38,049 [INFO] tensorflow: epoch = 3.3229166666666665, learning_rate = 2.1684402e-05, loss = 0.002564355, step = 319 (5.501 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07008\n",
"2021-12-30 09:11:39,665 [INFO] tensorflow: global_step/sec: 3.07008\n",
"2021-12-30 09:11:39,665 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.756\n",
"INFO:tensorflow:global_step/sec: 3.03836\n",
"2021-12-30 09:11:42,627 [INFO] tensorflow: global_step/sec: 3.03836\n",
"INFO:tensorflow:epoch = 3.5, learning_rate = 2.3447883e-05, loss = 0.0018386168, step = 336 (5.549 sec)\n",
"2021-12-30 09:11:43,598 [INFO] tensorflow: epoch = 3.5, learning_rate = 2.3447883e-05, loss = 0.0018386168, step = 336 (5.549 sec)\n",
"INFO:tensorflow:global_step/sec: 3.15806\n",
"2021-12-30 09:11:45,477 [INFO] tensorflow: global_step/sec: 3.15806\n",
"2021-12-30 09:11:47,762 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.704\n",
"INFO:tensorflow:global_step/sec: 3.0733\n",
"2021-12-30 09:11:48,405 [INFO] tensorflow: global_step/sec: 3.0733\n",
"INFO:tensorflow:epoch = 3.677083333333333, learning_rate = 2.5354779e-05, loss = 0.0018915869, step = 353 (5.446 sec)\n",
"2021-12-30 09:11:49,044 [INFO] tensorflow: epoch = 3.677083333333333, learning_rate = 2.5354779e-05, loss = 0.0018915869, step = 353 (5.446 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10039\n",
"2021-12-30 09:11:51,308 [INFO] tensorflow: global_step/sec: 3.10039\n",
"INFO:tensorflow:global_step/sec: 3.05674\n",
"2021-12-30 09:11:54,252 [INFO] tensorflow: global_step/sec: 3.05674\n",
"INFO:tensorflow:epoch = 3.8541666666666665, learning_rate = 2.7416752e-05, loss = 0.001492907, step = 370 (5.540 sec)\n",
"2021-12-30 09:11:54,584 [INFO] tensorflow: epoch = 3.8541666666666665, learning_rate = 2.7416752e-05, loss = 0.001492907, step = 370 (5.540 sec)\n",
"2021-12-30 09:11:55,884 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.624\n",
"INFO:tensorflow:global_step/sec: 3.09219\n",
"2021-12-30 09:11:57,163 [INFO] tensorflow: global_step/sec: 3.09219\n",
"2021-12-30 09:11:59,129 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 4/120: loss: 0.00146 learning rate: 0.00003 Time taken: 0:00:31.048055 ETA: 1:00:01.574345\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO:tensorflow:epoch = 4.03125, learning_rate = 2.9646415e-05, loss = 0.0008774643, step = 387 (5.533 sec)\n",
"2021-12-30 09:12:00,118 [INFO] tensorflow: epoch = 4.03125, learning_rate = 2.9646415e-05, loss = 0.0008774643, step = 387 (5.533 sec)\n",
"INFO:tensorflow:global_step/sec: 3.04534\n",
"2021-12-30 09:12:00,118 [INFO] tensorflow: global_step/sec: 3.04534\n",
"INFO:tensorflow:global_step/sec: 3.12508\n",
"2021-12-30 09:12:02,998 [INFO] tensorflow: global_step/sec: 3.12508\n",
"2021-12-30 09:12:03,971 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.732\n",
"INFO:tensorflow:epoch = 4.208333333333333, learning_rate = 3.2057404e-05, loss = 0.00095430797, step = 404 (5.478 sec)\n",
"2021-12-30 09:12:05,596 [INFO] tensorflow: epoch = 4.208333333333333, learning_rate = 3.2057404e-05, loss = 0.00095430797, step = 404 (5.478 sec)\n",
"INFO:tensorflow:global_step/sec: 3.06631\n",
"2021-12-30 09:12:05,933 [INFO] tensorflow: global_step/sec: 3.06631\n",
"INFO:tensorflow:global_step/sec: 3.09654\n",
"2021-12-30 09:12:08,840 [INFO] tensorflow: global_step/sec: 3.09654\n",
"INFO:tensorflow:epoch = 4.385416666666666, learning_rate = 3.4664467e-05, loss = 0.001014581, step = 421 (5.531 sec)\n",
"2021-12-30 09:12:11,127 [INFO] tensorflow: epoch = 4.385416666666666, learning_rate = 3.4664467e-05, loss = 0.001014581, step = 421 (5.531 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07919\n",
"2021-12-30 09:12:11,763 [INFO] tensorflow: global_step/sec: 3.07919\n",
"2021-12-30 09:12:12,094 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.621\n",
"INFO:tensorflow:global_step/sec: 3.11595\n",
"2021-12-30 09:12:14,651 [INFO] tensorflow: global_step/sec: 3.11595\n",
"INFO:tensorflow:epoch = 4.5625, learning_rate = 3.7483547e-05, loss = 0.0011078119, step = 438 (5.456 sec)\n",
"2021-12-30 09:12:16,583 [INFO] tensorflow: epoch = 4.5625, learning_rate = 3.7483547e-05, loss = 0.0011078119, step = 438 (5.456 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07855\n",
"2021-12-30 09:12:17,574 [INFO] tensorflow: global_step/sec: 3.07855\n",
"2021-12-30 09:12:20,179 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.739\n",
"INFO:tensorflow:global_step/sec: 3.08055\n",
"2021-12-30 09:12:20,496 [INFO] tensorflow: global_step/sec: 3.08055\n",
"INFO:tensorflow:epoch = 4.739583333333333, learning_rate = 4.0531893e-05, loss = 0.0016165904, step = 455 (5.551 sec)\n",
"2021-12-30 09:12:22,134 [INFO] tensorflow: epoch = 4.739583333333333, learning_rate = 4.0531893e-05, loss = 0.0016165904, step = 455 (5.551 sec)\n",
"INFO:tensorflow:global_step/sec: 3.05113\n",
"2021-12-30 09:12:23,446 [INFO] tensorflow: global_step/sec: 3.05113\n",
"INFO:tensorflow:global_step/sec: 3.09308\n",
"2021-12-30 09:12:26,355 [INFO] tensorflow: global_step/sec: 3.09308\n",
"INFO:tensorflow:epoch = 4.916666666666666, learning_rate = 4.3828146e-05, loss = 0.0011321991, step = 472 (5.498 sec)\n",
"2021-12-30 09:12:27,632 [INFO] tensorflow: epoch = 4.916666666666666, learning_rate = 4.3828146e-05, loss = 0.0011321991, step = 472 (5.498 sec)\n",
"2021-12-30 09:12:28,298 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.633\n",
"INFO:tensorflow:global_step/sec: 3.08691\n",
"2021-12-30 09:12:29,271 [INFO] tensorflow: global_step/sec: 3.08691\n",
"2021-12-30 09:12:30,234 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 5/120: loss: 0.00069 learning rate: 0.00005 Time taken: 0:00:31.124520 ETA: 0:59:39.319835\n",
"INFO:tensorflow:global_step/sec: 3.06114\n",
"2021-12-30 09:12:32,211 [INFO] tensorflow: global_step/sec: 3.06114\n",
"INFO:tensorflow:epoch = 5.09375, learning_rate = 4.739246e-05, loss = 0.0011419817, step = 489 (5.554 sec)\n",
"2021-12-30 09:12:33,186 [INFO] tensorflow: epoch = 5.09375, learning_rate = 4.739246e-05, loss = 0.0011419817, step = 489 (5.554 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12226\n",
"2021-12-30 09:12:35,094 [INFO] tensorflow: global_step/sec: 3.12226\n",
"2021-12-30 09:12:36,391 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.714\n",
"INFO:tensorflow:global_step/sec: 3.06699\n",
"2021-12-30 09:12:38,028 [INFO] tensorflow: global_step/sec: 3.06699\n",
"INFO:tensorflow:epoch = 5.270833333333333, learning_rate = 5.124664e-05, loss = 0.0009845321, step = 506 (5.502 sec)\n",
"2021-12-30 09:12:38,688 [INFO] tensorflow: epoch = 5.270833333333333, learning_rate = 5.124664e-05, loss = 0.0009845321, step = 506 (5.502 sec)\n",
"INFO:tensorflow:global_step/sec: 3.04676\n",
"2021-12-30 09:12:40,982 [INFO] tensorflow: global_step/sec: 3.04676\n",
"INFO:tensorflow:global_step/sec: 3.11489\n",
"2021-12-30 09:12:43,871 [INFO] tensorflow: global_step/sec: 3.11489\n",
"INFO:tensorflow:epoch = 5.447916666666666, learning_rate = 5.5414268e-05, loss = 0.0008429837, step = 523 (5.513 sec)\n",
"2021-12-30 09:12:44,201 [INFO] tensorflow: epoch = 5.447916666666666, learning_rate = 5.5414268e-05, loss = 0.0008429837, step = 523 (5.513 sec)\n",
"2021-12-30 09:12:44,533 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.565\n",
"INFO:tensorflow:global_step/sec: 3.08255\n",
"2021-12-30 09:12:46,791 [INFO] tensorflow: global_step/sec: 3.08255\n",
"INFO:tensorflow:epoch = 5.625, learning_rate = 5.992082e-05, loss = 0.0008237965, step = 540 (5.488 sec)\n",
"2021-12-30 09:12:49,689 [INFO] tensorflow: epoch = 5.625, learning_rate = 5.992082e-05, loss = 0.0008237965, step = 540 (5.488 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10472\n",
"2021-12-30 09:12:49,690 [INFO] tensorflow: global_step/sec: 3.10472\n",
"INFO:tensorflow:global_step/sec: 3.12475\n",
"2021-12-30 09:12:52,570 [INFO] tensorflow: global_step/sec: 3.12475\n",
"2021-12-30 09:12:52,571 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.884\n",
"INFO:tensorflow:epoch = 5.802083333333333, learning_rate = 6.479388e-05, loss = 0.0006237829, step = 557 (5.475 sec)\n",
"2021-12-30 09:12:55,164 [INFO] tensorflow: epoch = 5.802083333333333, learning_rate = 6.479388e-05, loss = 0.0006237829, step = 557 (5.475 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08319\n",
"2021-12-30 09:12:55,489 [INFO] tensorflow: global_step/sec: 3.08319\n",
"INFO:tensorflow:global_step/sec: 3.07126\n",
"2021-12-30 09:12:58,420 [INFO] tensorflow: global_step/sec: 3.07126\n",
"INFO:tensorflow:epoch = 5.979166666666666, learning_rate = 7.006322e-05, loss = 0.0007467545, step = 574 (5.521 sec)\n",
"2021-12-30 09:13:00,685 [INFO] tensorflow: epoch = 5.979166666666666, learning_rate = 7.006322e-05, loss = 0.0007467545, step = 574 (5.521 sec)\n",
"2021-12-30 09:13:00,685 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.649\n",
"INFO:tensorflow:global_step/sec: 3.09098\n",
"2021-12-30 09:13:01,331 [INFO] tensorflow: global_step/sec: 3.09098\n",
"2021-12-30 09:13:01,332 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 6/120: loss: 0.00066 learning rate: 0.00007 Time taken: 0:00:31.092568 ETA: 0:59:04.552797\n",
"INFO:tensorflow:global_step/sec: 3.12717\n",
"2021-12-30 09:13:04,209 [INFO] tensorflow: global_step/sec: 3.12717\n",
"INFO:tensorflow:epoch = 6.15625, learning_rate = 7.5761105e-05, loss = 0.00052929117, step = 591 (5.477 sec)\n",
"2021-12-30 09:13:06,161 [INFO] tensorflow: epoch = 6.15625, learning_rate = 7.5761105e-05, loss = 0.00052929117, step = 591 (5.477 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10945\n",
"2021-12-30 09:13:07,104 [INFO] tensorflow: global_step/sec: 3.10945\n",
"2021-12-30 09:13:08,731 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.857\n",
"INFO:tensorflow:global_step/sec: 3.07674\n",
"2021-12-30 09:13:10,029 [INFO] tensorflow: global_step/sec: 3.07674\n",
"INFO:tensorflow:epoch = 6.333333333333333, learning_rate = 8.1922364e-05, loss = 0.0006967305, step = 608 (5.494 sec)\n",
"2021-12-30 09:13:11,655 [INFO] tensorflow: epoch = 6.333333333333333, learning_rate = 8.1922364e-05, loss = 0.0006967305, step = 608 (5.494 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08141\n",
"2021-12-30 09:13:12,950 [INFO] tensorflow: global_step/sec: 3.08141\n",
"INFO:tensorflow:global_step/sec: 3.0755\n",
"2021-12-30 09:13:15,876 [INFO] tensorflow: global_step/sec: 3.0755\n",
"2021-12-30 09:13:16,818 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.732\n",
"INFO:tensorflow:epoch = 6.510416666666666, learning_rate = 8.858469e-05, loss = 0.00051924575, step = 625 (5.485 sec)\n",
"2021-12-30 09:13:17,140 [INFO] tensorflow: epoch = 6.510416666666666, learning_rate = 8.858469e-05, loss = 0.00051924575, step = 625 (5.485 sec)\n",
"INFO:tensorflow:global_step/sec: 3.16351\n",
"2021-12-30 09:13:18,721 [INFO] tensorflow: global_step/sec: 3.16351\n",
"INFO:tensorflow:global_step/sec: 3.09748\n",
"2021-12-30 09:13:21,626 [INFO] tensorflow: global_step/sec: 3.09748\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO:tensorflow:epoch = 6.6875, learning_rate = 9.578882e-05, loss = 0.0009072368, step = 642 (5.445 sec)\n",
"2021-12-30 09:13:22,586 [INFO] tensorflow: epoch = 6.6875, learning_rate = 9.578882e-05, loss = 0.0009072368, step = 642 (5.445 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08532\n",
"2021-12-30 09:13:24,543 [INFO] tensorflow: global_step/sec: 3.08532\n",
"2021-12-30 09:13:24,881 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.805\n",
"INFO:tensorflow:global_step/sec: 3.10785\n",
"2021-12-30 09:13:27,439 [INFO] tensorflow: global_step/sec: 3.10785\n",
"INFO:tensorflow:epoch = 6.864583333333333, learning_rate = 0.00010357884, loss = 0.0005257707, step = 659 (5.514 sec)\n",
"2021-12-30 09:13:28,100 [INFO] tensorflow: epoch = 6.864583333333333, learning_rate = 0.00010357884, loss = 0.0005257707, step = 659 (5.514 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09983\n",
"2021-12-30 09:13:30,343 [INFO] tensorflow: global_step/sec: 3.09983\n",
"2021-12-30 09:13:32,300 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 7/120: loss: 0.00058 learning rate: 0.00011 Time taken: 0:00:30.963935 ETA: 0:58:18.924644\n",
"2021-12-30 09:13:32,957 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.765\n",
"INFO:tensorflow:global_step/sec: 3.08106\n",
"2021-12-30 09:13:33,264 [INFO] tensorflow: global_step/sec: 3.08106\n",
"INFO:tensorflow:epoch = 7.041666666666666, learning_rate = 0.00011200236, loss = 0.0003908295, step = 676 (5.472 sec)\n",
"2021-12-30 09:13:33,572 [INFO] tensorflow: epoch = 7.041666666666666, learning_rate = 0.00011200236, loss = 0.0003908295, step = 676 (5.472 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08287\n",
"2021-12-30 09:13:36,183 [INFO] tensorflow: global_step/sec: 3.08287\n",
"INFO:tensorflow:epoch = 7.21875, learning_rate = 0.000121110934, loss = 0.00044059957, step = 693 (5.531 sec)\n",
"2021-12-30 09:13:39,103 [INFO] tensorflow: epoch = 7.21875, learning_rate = 0.000121110934, loss = 0.00044059957, step = 693 (5.531 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08153\n",
"2021-12-30 09:13:39,104 [INFO] tensorflow: global_step/sec: 3.08153\n",
"2021-12-30 09:13:41,097 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.573\n",
"INFO:tensorflow:global_step/sec: 3.0576\n",
"2021-12-30 09:13:42,047 [INFO] tensorflow: global_step/sec: 3.0576\n",
"INFO:tensorflow:epoch = 7.395833333333333, learning_rate = 0.00013096027, loss = 0.00065735384, step = 710 (5.516 sec)\n",
"2021-12-30 09:13:44,619 [INFO] tensorflow: epoch = 7.395833333333333, learning_rate = 0.00013096027, loss = 0.00065735384, step = 710 (5.516 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11417\n",
"2021-12-30 09:13:44,937 [INFO] tensorflow: global_step/sec: 3.11417\n",
"INFO:tensorflow:global_step/sec: 3.07701\n",
"2021-12-30 09:13:47,862 [INFO] tensorflow: global_step/sec: 3.07701\n",
"2021-12-30 09:13:49,155 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.820\n",
"INFO:tensorflow:epoch = 7.572916666666666, learning_rate = 0.0001416106, loss = 0.00037334624, step = 727 (5.518 sec)\n",
"2021-12-30 09:13:50,138 [INFO] tensorflow: epoch = 7.572916666666666, learning_rate = 0.0001416106, loss = 0.00037334624, step = 727 (5.518 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07443\n",
"2021-12-30 09:13:50,790 [INFO] tensorflow: global_step/sec: 3.07443\n",
"INFO:tensorflow:global_step/sec: 3.10274\n",
"2021-12-30 09:13:53,690 [INFO] tensorflow: global_step/sec: 3.10274\n",
"INFO:tensorflow:epoch = 7.75, learning_rate = 0.00015312705, loss = 0.0002881634, step = 744 (5.522 sec)\n",
"2021-12-30 09:13:55,659 [INFO] tensorflow: epoch = 7.75, learning_rate = 0.00015312705, loss = 0.0002881634, step = 744 (5.522 sec)\n",
"INFO:tensorflow:global_step/sec: 3.05142\n",
"2021-12-30 09:13:56,640 [INFO] tensorflow: global_step/sec: 3.05142\n",
"2021-12-30 09:13:57,304 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.543\n",
"INFO:tensorflow:global_step/sec: 3.07196\n",
"2021-12-30 09:13:59,569 [INFO] tensorflow: global_step/sec: 3.07196\n",
"INFO:tensorflow:epoch = 7.927083333333333, learning_rate = 0.00016558006, loss = 0.0003021504, step = 761 (5.497 sec)\n",
"2021-12-30 09:14:01,157 [INFO] tensorflow: epoch = 7.927083333333333, learning_rate = 0.00016558006, loss = 0.0003021504, step = 761 (5.497 sec)\n",
"INFO:tensorflow:global_step/sec: 3.13407\n",
"2021-12-30 09:14:02,441 [INFO] tensorflow: global_step/sec: 3.13407\n",
"2021-12-30 09:14:03,387 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 8/120: loss: 0.00042 learning rate: 0.00017 Time taken: 0:00:31.103408 ETA: 0:58:03.581707\n",
"INFO:tensorflow:global_step/sec: 3.12901\n",
"2021-12-30 09:14:05,317 [INFO] tensorflow: global_step/sec: 3.12901\n",
"2021-12-30 09:14:05,318 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.957\n",
"INFO:tensorflow:epoch = 8.104166666666666, learning_rate = 0.00017904585, loss = 0.00038615311, step = 778 (5.446 sec)\n",
"2021-12-30 09:14:06,603 [INFO] tensorflow: epoch = 8.104166666666666, learning_rate = 0.00017904585, loss = 0.00038615311, step = 778 (5.446 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08039\n",
"2021-12-30 09:14:08,239 [INFO] tensorflow: global_step/sec: 3.08039\n",
"INFO:tensorflow:global_step/sec: 3.13171\n",
"2021-12-30 09:14:11,113 [INFO] tensorflow: global_step/sec: 3.13171\n",
"INFO:tensorflow:epoch = 8.28125, learning_rate = 0.00019360673, loss = 0.0004922912, step = 795 (5.467 sec)\n",
"2021-12-30 09:14:12,070 [INFO] tensorflow: epoch = 8.28125, learning_rate = 0.00019360673, loss = 0.0004922912, step = 795 (5.467 sec)\n",
"2021-12-30 09:14:13,374 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.826\n",
"INFO:tensorflow:global_step/sec: 3.09487\n",
"2021-12-30 09:14:14,021 [INFO] tensorflow: global_step/sec: 3.09487\n",
"INFO:tensorflow:global_step/sec: 3.09997\n",
"2021-12-30 09:14:16,924 [INFO] tensorflow: global_step/sec: 3.09997\n",
"INFO:tensorflow:epoch = 8.458333333333332, learning_rate = 0.00020935175, loss = 0.00041331578, step = 812 (5.493 sec)\n",
"2021-12-30 09:14:17,563 [INFO] tensorflow: epoch = 8.458333333333332, learning_rate = 0.00020935175, loss = 0.00041331578, step = 812 (5.493 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09701\n",
"2021-12-30 09:14:19,830 [INFO] tensorflow: global_step/sec: 3.09701\n",
"2021-12-30 09:14:21,450 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.768\n",
"INFO:tensorflow:global_step/sec: 3.08913\n",
"2021-12-30 09:14:22,744 [INFO] tensorflow: global_step/sec: 3.08913\n",
"INFO:tensorflow:epoch = 8.635416666666666, learning_rate = 0.00022637726, loss = 0.00029083283, step = 829 (5.513 sec)\n",
"2021-12-30 09:14:23,077 [INFO] tensorflow: epoch = 8.635416666666666, learning_rate = 0.00022637726, loss = 0.00029083283, step = 829 (5.513 sec)\n",
"INFO:tensorflow:global_step/sec: 3.06325\n",
"2021-12-30 09:14:25,682 [INFO] tensorflow: global_step/sec: 3.06325\n",
"INFO:tensorflow:epoch = 8.8125, learning_rate = 0.00024478737, loss = 0.00044524414, step = 846 (5.497 sec)\n",
"2021-12-30 09:14:28,574 [INFO] tensorflow: epoch = 8.8125, learning_rate = 0.00024478737, loss = 0.00044524414, step = 846 (5.497 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11129\n",
"2021-12-30 09:14:28,574 [INFO] tensorflow: global_step/sec: 3.11129\n",
"2021-12-30 09:14:29,536 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.734\n",
"INFO:tensorflow:global_step/sec: 3.09472\n",
"2021-12-30 09:14:31,483 [INFO] tensorflow: global_step/sec: 3.09472\n",
"INFO:tensorflow:epoch = 8.989583333333332, learning_rate = 0.00026469462, loss = 0.00033640015, step = 863 (5.543 sec)\n",
"2021-12-30 09:14:34,117 [INFO] tensorflow: epoch = 8.989583333333332, learning_rate = 0.00026469462, loss = 0.00033640015, step = 863 (5.543 sec)\n",
"INFO:tensorflow:global_step/sec: 3.0541\n",
"2021-12-30 09:14:34,430 [INFO] tensorflow: global_step/sec: 3.0541\n",
"2021-12-30 09:14:34,430 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 9/120: loss: 0.00041 learning rate: 0.00027 Time taken: 0:00:31.038409 ETA: 0:57:25.263425\n",
"INFO:tensorflow:global_step/sec: 3.05258\n",
"2021-12-30 09:14:37,378 [INFO] tensorflow: global_step/sec: 3.05258\n",
"2021-12-30 09:14:37,698 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.504\n",
"INFO:tensorflow:epoch = 9.166666666666666, learning_rate = 0.00028622086, loss = 0.00038506495, step = 880 (5.498 sec)\n",
"2021-12-30 09:14:39,614 [INFO] tensorflow: epoch = 9.166666666666666, learning_rate = 0.00028622086, loss = 0.00038506495, step = 880 (5.498 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11605\n",
"2021-12-30 09:14:40,266 [INFO] tensorflow: global_step/sec: 3.11605\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO:tensorflow:global_step/sec: 3.10518\n",
"2021-12-30 09:14:43,164 [INFO] tensorflow: global_step/sec: 3.10518\n",
"INFO:tensorflow:epoch = 9.34375, learning_rate = 0.00030949776, loss = 0.00034103278, step = 897 (5.463 sec)\n",
"2021-12-30 09:14:45,078 [INFO] tensorflow: epoch = 9.34375, learning_rate = 0.00030949776, loss = 0.00034103278, step = 897 (5.463 sec)\n",
"2021-12-30 09:14:45,713 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.955\n",
"INFO:tensorflow:global_step/sec: 3.13285\n",
"2021-12-30 09:14:46,037 [INFO] tensorflow: global_step/sec: 3.13285\n",
"INFO:tensorflow:global_step/sec: 3.07157\n",
"2021-12-30 09:14:48,967 [INFO] tensorflow: global_step/sec: 3.07157\n",
"INFO:tensorflow:epoch = 9.520833333333332, learning_rate = 0.0003346676, loss = 0.0004047746, step = 914 (5.546 sec)\n",
"2021-12-30 09:14:50,624 [INFO] tensorflow: epoch = 9.520833333333332, learning_rate = 0.0003346676, loss = 0.0004047746, step = 914 (5.546 sec)\n",
"INFO:tensorflow:global_step/sec: 3.04281\n",
"2021-12-30 09:14:51,925 [INFO] tensorflow: global_step/sec: 3.04281\n",
"2021-12-30 09:14:53,915 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.385\n",
"INFO:tensorflow:global_step/sec: 3.04974\n",
"2021-12-30 09:14:54,876 [INFO] tensorflow: global_step/sec: 3.04974\n",
"INFO:tensorflow:epoch = 9.697916666666666, learning_rate = 0.00036188422, loss = 0.00043012225, step = 931 (5.529 sec)\n",
"2021-12-30 09:14:56,153 [INFO] tensorflow: epoch = 9.697916666666666, learning_rate = 0.00036188422, loss = 0.00043012225, step = 931 (5.529 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10235\n",
"2021-12-30 09:14:57,777 [INFO] tensorflow: global_step/sec: 3.10235\n",
"INFO:tensorflow:global_step/sec: 3.05101\n",
"2021-12-30 09:15:00,727 [INFO] tensorflow: global_step/sec: 3.05101\n",
"INFO:tensorflow:epoch = 9.875, learning_rate = 0.00039131456, loss = 0.00036340297, step = 948 (5.520 sec)\n",
"2021-12-30 09:15:01,673 [INFO] tensorflow: epoch = 9.875, learning_rate = 0.00039131456, loss = 0.00036340297, step = 948 (5.520 sec)\n",
"2021-12-30 09:15:01,983 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.789\n",
"INFO:tensorflow:global_step/sec: 3.14216\n",
"2021-12-30 09:15:03,591 [INFO] tensorflow: global_step/sec: 3.14216\n",
"INFO:tensorflow:Saving checkpoints for step-960.\n",
"2021-12-30 09:15:05,199 [INFO] tensorflow: Saving checkpoints for step-960.\n",
"INFO:tensorflow:epoch = 10.0, learning_rate = 0.00041351854, loss = 0.0003040716, step = 960 (7.512 sec)\n",
"2021-12-30 09:15:09,185 [INFO] tensorflow: epoch = 10.0, learning_rate = 0.00041351854, loss = 0.0003040716, step = 960 (7.512 sec)\n",
"2021-12-30 09:15:09,185 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 10/120: loss: 0.00030 learning rate: 0.00041 Time taken: 0:00:34.717591 ETA: 1:03:38.934989\n",
"INFO:tensorflow:global_step/sec: 1.37342\n",
"2021-12-30 09:15:10,144 [INFO] tensorflow: global_step/sec: 1.37342\n",
"INFO:tensorflow:global_step/sec: 3.12345\n",
"2021-12-30 09:15:13,026 [INFO] tensorflow: global_step/sec: 3.12345\n",
"2021-12-30 09:15:13,677 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 17.103\n",
"INFO:tensorflow:epoch = 10.177083333333332, learning_rate = 0.00044714788, loss = 0.0004248642, step = 977 (5.463 sec)\n",
"2021-12-30 09:15:14,648 [INFO] tensorflow: epoch = 10.177083333333332, learning_rate = 0.00044714788, loss = 0.0004248642, step = 977 (5.463 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09411\n",
"2021-12-30 09:15:15,935 [INFO] tensorflow: global_step/sec: 3.09411\n",
"INFO:tensorflow:global_step/sec: 3.08696\n",
"2021-12-30 09:15:18,850 [INFO] tensorflow: global_step/sec: 3.08696\n",
"INFO:tensorflow:epoch = 10.354166666666666, learning_rate = 0.0004835121, loss = 0.0004037608, step = 994 (5.490 sec)\n",
"2021-12-30 09:15:20,139 [INFO] tensorflow: epoch = 10.354166666666666, learning_rate = 0.0004835121, loss = 0.0004037608, step = 994 (5.490 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10685\n",
"2021-12-30 09:15:21,747 [INFO] tensorflow: global_step/sec: 3.10685\n",
"2021-12-30 09:15:21,747 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.782\n",
"INFO:tensorflow:global_step/sec: 3.10577\n",
"2021-12-30 09:15:24,645 [INFO] tensorflow: global_step/sec: 3.10577\n",
"INFO:tensorflow:epoch = 10.53125, learning_rate = 0.0005228336, loss = 0.00035080727, step = 1011 (5.488 sec)\n",
"2021-12-30 09:15:25,627 [INFO] tensorflow: epoch = 10.53125, learning_rate = 0.0005228336, loss = 0.00035080727, step = 1011 (5.488 sec)\n",
"INFO:tensorflow:global_step/sec: 3.0835\n",
"2021-12-30 09:15:27,563 [INFO] tensorflow: global_step/sec: 3.0835\n",
"2021-12-30 09:15:29,839 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.716\n",
"INFO:tensorflow:global_step/sec: 3.09473\n",
"2021-12-30 09:15:30,472 [INFO] tensorflow: global_step/sec: 3.09473\n",
"INFO:tensorflow:epoch = 10.708333333333332, learning_rate = 0.000565353, loss = 0.00046681424, step = 1028 (5.487 sec)\n",
"2021-12-30 09:15:31,114 [INFO] tensorflow: epoch = 10.708333333333332, learning_rate = 0.000565353, loss = 0.00046681424, step = 1028 (5.487 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09049\n",
"2021-12-30 09:15:33,384 [INFO] tensorflow: global_step/sec: 3.09049\n",
"INFO:tensorflow:global_step/sec: 3.06925\n",
"2021-12-30 09:15:36,316 [INFO] tensorflow: global_step/sec: 3.06925\n",
"INFO:tensorflow:epoch = 10.885416666666666, learning_rate = 0.00061133027, loss = 0.00040922334, step = 1045 (5.516 sec)\n",
"2021-12-30 09:15:36,630 [INFO] tensorflow: epoch = 10.885416666666666, learning_rate = 0.00061133027, loss = 0.00040922334, step = 1045 (5.516 sec)\n",
"2021-12-30 09:15:37,912 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.775\n",
"INFO:tensorflow:global_step/sec: 3.11242\n",
"2021-12-30 09:15:39,208 [INFO] tensorflow: global_step/sec: 3.11242\n",
"2021-12-30 09:15:40,179 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 11/120: loss: 0.00039 learning rate: 0.00064 Time taken: 0:00:31.013131 ETA: 0:56:20.431242\n",
"INFO:tensorflow:epoch = 11.0625, learning_rate = 0.0006610466, loss = 0.0005401101, step = 1062 (5.512 sec)\n",
"2021-12-30 09:15:42,142 [INFO] tensorflow: epoch = 11.0625, learning_rate = 0.0006610466, loss = 0.0005401101, step = 1062 (5.512 sec)\n",
"INFO:tensorflow:global_step/sec: 3.06686\n",
"2021-12-30 09:15:42,142 [INFO] tensorflow: global_step/sec: 3.06686\n",
"INFO:tensorflow:global_step/sec: 3.09046\n",
"2021-12-30 09:15:45,054 [INFO] tensorflow: global_step/sec: 3.09046\n",
"2021-12-30 09:15:46,023 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.659\n",
"INFO:tensorflow:epoch = 11.239583333333332, learning_rate = 0.00071480573, loss = 0.00035397316, step = 1079 (5.496 sec)\n",
"2021-12-30 09:15:47,637 [INFO] tensorflow: epoch = 11.239583333333332, learning_rate = 0.00071480573, loss = 0.00035397316, step = 1079 (5.496 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09567\n",
"2021-12-30 09:15:47,962 [INFO] tensorflow: global_step/sec: 3.09567\n",
"INFO:tensorflow:global_step/sec: 3.10523\n",
"2021-12-30 09:15:50,860 [INFO] tensorflow: global_step/sec: 3.10523\n",
"INFO:tensorflow:epoch = 11.416666666666666, learning_rate = 0.0007729376, loss = 0.00043676898, step = 1096 (5.462 sec)\n",
"2021-12-30 09:15:53,099 [INFO] tensorflow: epoch = 11.416666666666666, learning_rate = 0.0007729376, loss = 0.00043676898, step = 1096 (5.462 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11684\n",
"2021-12-30 09:15:53,748 [INFO] tensorflow: global_step/sec: 3.11684\n",
"2021-12-30 09:15:54,075 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.839\n",
"INFO:tensorflow:global_step/sec: 3.07593\n",
"2021-12-30 09:15:56,674 [INFO] tensorflow: global_step/sec: 3.07593\n",
"INFO:tensorflow:epoch = 11.59375, learning_rate = 0.00083579664, loss = 0.00030932913, step = 1113 (5.531 sec)\n",
"2021-12-30 09:15:58,630 [INFO] tensorflow: epoch = 11.59375, learning_rate = 0.00083579664, loss = 0.00030932913, step = 1113 (5.531 sec)\n",
"INFO:tensorflow:global_step/sec: 3.0804\n",
"2021-12-30 09:15:59,595 [INFO] tensorflow: global_step/sec: 3.0804\n",
"2021-12-30 09:16:02,147 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.777\n",
"INFO:tensorflow:global_step/sec: 3.1198\n",
"2021-12-30 09:16:02,480 [INFO] tensorflow: global_step/sec: 3.1198\n",
"INFO:tensorflow:epoch = 11.770833333333332, learning_rate = 0.0009037676, loss = 0.0005003148, step = 1130 (5.495 sec)\n",
"2021-12-30 09:16:04,125 [INFO] tensorflow: epoch = 11.770833333333332, learning_rate = 0.0009037676, loss = 0.0005003148, step = 1130 (5.495 sec)\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO:tensorflow:global_step/sec: 3.03403\n",
"2021-12-30 09:16:05,446 [INFO] tensorflow: global_step/sec: 3.03403\n",
"INFO:tensorflow:global_step/sec: 3.03002\n",
"2021-12-30 09:16:08,417 [INFO] tensorflow: global_step/sec: 3.03002\n",
"INFO:tensorflow:epoch = 11.947916666666666, learning_rate = 0.0009772663, loss = 0.0003278718, step = 1147 (5.580 sec)\n",
"2021-12-30 09:16:09,705 [INFO] tensorflow: epoch = 11.947916666666666, learning_rate = 0.0009772663, loss = 0.0003278718, step = 1147 (5.580 sec)\n",
"2021-12-30 09:16:10,339 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.416\n",
"INFO:tensorflow:global_step/sec: 3.1438\n",
"2021-12-30 09:16:11,280 [INFO] tensorflow: global_step/sec: 3.1438\n",
"2021-12-30 09:16:11,280 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 12/120: loss: 0.00029 learning rate: 0.00100 Time taken: 0:00:31.128277 ETA: 0:56:01.853871\n",
"INFO:tensorflow:global_step/sec: 3.07406\n",
"2021-12-30 09:16:14,207 [INFO] tensorflow: global_step/sec: 3.07406\n",
"INFO:tensorflow:epoch = 12.125, learning_rate = 0.0009999999, loss = 0.0003636689, step = 1164 (5.467 sec)\n",
"2021-12-30 09:16:15,172 [INFO] tensorflow: epoch = 12.125, learning_rate = 0.0009999999, loss = 0.0003636689, step = 1164 (5.467 sec)\n",
"INFO:tensorflow:global_step/sec: 3.06048\n",
"2021-12-30 09:16:17,148 [INFO] tensorflow: global_step/sec: 3.06048\n",
"2021-12-30 09:16:18,456 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.640\n",
"INFO:tensorflow:global_step/sec: 3.05372\n",
"2021-12-30 09:16:20,095 [INFO] tensorflow: global_step/sec: 3.05372\n",
"INFO:tensorflow:epoch = 12.302083333333332, learning_rate = 0.0009999999, loss = 0.00023650996, step = 1181 (5.585 sec)\n",
"2021-12-30 09:16:20,757 [INFO] tensorflow: epoch = 12.302083333333332, learning_rate = 0.0009999999, loss = 0.00023650996, step = 1181 (5.585 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07674\n",
"2021-12-30 09:16:23,020 [INFO] tensorflow: global_step/sec: 3.07674\n",
"INFO:tensorflow:global_step/sec: 3.0895\n",
"2021-12-30 09:16:25,933 [INFO] tensorflow: global_step/sec: 3.0895\n",
"INFO:tensorflow:epoch = 12.479166666666666, learning_rate = 0.0009999999, loss = 0.00035956915, step = 1198 (5.481 sec)\n",
"2021-12-30 09:16:26,238 [INFO] tensorflow: epoch = 12.479166666666666, learning_rate = 0.0009999999, loss = 0.00035956915, step = 1198 (5.481 sec)\n",
"2021-12-30 09:16:26,562 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.674\n",
"INFO:tensorflow:global_step/sec: 3.10277\n",
"2021-12-30 09:16:28,834 [INFO] tensorflow: global_step/sec: 3.10277\n",
"INFO:tensorflow:epoch = 12.65625, learning_rate = 0.0009999999, loss = 0.00030963708, step = 1215 (5.498 sec)\n",
"2021-12-30 09:16:31,736 [INFO] tensorflow: epoch = 12.65625, learning_rate = 0.0009999999, loss = 0.00030963708, step = 1215 (5.498 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10092\n",
"2021-12-30 09:16:31,736 [INFO] tensorflow: global_step/sec: 3.10092\n",
"INFO:tensorflow:global_step/sec: 3.11042\n",
"2021-12-30 09:16:34,630 [INFO] tensorflow: global_step/sec: 3.11042\n",
"2021-12-30 09:16:34,631 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.789\n",
"INFO:tensorflow:epoch = 12.833333333333332, learning_rate = 0.0009999999, loss = 0.0004294164, step = 1232 (5.453 sec)\n",
"2021-12-30 09:16:37,189 [INFO] tensorflow: epoch = 12.833333333333332, learning_rate = 0.0009999999, loss = 0.0004294164, step = 1232 (5.453 sec)\n",
"INFO:tensorflow:global_step/sec: 3.13119\n",
"2021-12-30 09:16:37,504 [INFO] tensorflow: global_step/sec: 3.13119\n",
"INFO:tensorflow:global_step/sec: 3.08661\n",
"2021-12-30 09:16:40,420 [INFO] tensorflow: global_step/sec: 3.08661\n",
"2021-12-30 09:16:42,374 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 13/120: loss: 0.00039 learning rate: 0.00100 Time taken: 0:00:31.055643 ETA: 0:55:22.953759\n",
"INFO:tensorflow:epoch = 13.010416666666666, learning_rate = 0.0009999999, loss = 0.00029456703, step = 1249 (5.505 sec)\n",
"2021-12-30 09:16:42,694 [INFO] tensorflow: epoch = 13.010416666666666, learning_rate = 0.0009999999, loss = 0.00029456703, step = 1249 (5.505 sec)\n",
"2021-12-30 09:16:42,695 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.802\n",
"INFO:tensorflow:global_step/sec: 3.06389\n",
"2021-12-30 09:16:43,358 [INFO] tensorflow: global_step/sec: 3.06389\n",
"INFO:tensorflow:global_step/sec: 3.08783\n",
"2021-12-30 09:16:46,272 [INFO] tensorflow: global_step/sec: 3.08783\n",
"INFO:tensorflow:epoch = 13.1875, learning_rate = 0.0009999999, loss = 0.00044958707, step = 1266 (5.505 sec)\n",
"2021-12-30 09:16:48,199 [INFO] tensorflow: epoch = 13.1875, learning_rate = 0.0009999999, loss = 0.00044958707, step = 1266 (5.505 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08769\n",
"2021-12-30 09:16:49,187 [INFO] tensorflow: global_step/sec: 3.08769\n",
"2021-12-30 09:16:50,800 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.677\n",
"INFO:tensorflow:global_step/sec: 3.09528\n",
"2021-12-30 09:16:52,095 [INFO] tensorflow: global_step/sec: 3.09528\n",
"INFO:tensorflow:epoch = 13.364583333333332, learning_rate = 0.0009999999, loss = 0.00023957531, step = 1283 (5.497 sec)\n",
"2021-12-30 09:16:53,696 [INFO] tensorflow: epoch = 13.364583333333332, learning_rate = 0.0009999999, loss = 0.00023957531, step = 1283 (5.497 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12479\n",
"2021-12-30 09:16:54,975 [INFO] tensorflow: global_step/sec: 3.12479\n",
"INFO:tensorflow:global_step/sec: 3.06509\n",
"2021-12-30 09:16:57,911 [INFO] tensorflow: global_step/sec: 3.06509\n",
"2021-12-30 09:16:58,862 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.807\n",
"INFO:tensorflow:epoch = 13.541666666666666, learning_rate = 0.0009999999, loss = 0.00023781369, step = 1300 (5.482 sec)\n",
"2021-12-30 09:16:59,178 [INFO] tensorflow: epoch = 13.541666666666666, learning_rate = 0.0009999999, loss = 0.00023781369, step = 1300 (5.482 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09944\n",
"2021-12-30 09:17:00,815 [INFO] tensorflow: global_step/sec: 3.09944\n",
"INFO:tensorflow:global_step/sec: 3.12408\n",
"2021-12-30 09:17:03,696 [INFO] tensorflow: global_step/sec: 3.12408\n",
"INFO:tensorflow:epoch = 13.71875, learning_rate = 0.0009999999, loss = 0.00025056925, step = 1317 (5.506 sec)\n",
"2021-12-30 09:17:04,684 [INFO] tensorflow: epoch = 13.71875, learning_rate = 0.0009999999, loss = 0.00025056925, step = 1317 (5.506 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08915\n",
"2021-12-30 09:17:06,609 [INFO] tensorflow: global_step/sec: 3.08915\n",
"2021-12-30 09:17:06,922 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.814\n",
"INFO:tensorflow:global_step/sec: 3.13825\n",
"2021-12-30 09:17:09,477 [INFO] tensorflow: global_step/sec: 3.13825\n",
"INFO:tensorflow:epoch = 13.895833333333332, learning_rate = 0.0009999999, loss = 0.00028820074, step = 1334 (5.431 sec)\n",
"2021-12-30 09:17:10,115 [INFO] tensorflow: epoch = 13.895833333333332, learning_rate = 0.0009999999, loss = 0.00028820074, step = 1334 (5.431 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07749\n",
"2021-12-30 09:17:12,401 [INFO] tensorflow: global_step/sec: 3.07749\n",
"2021-12-30 09:17:13,382 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 14/120: loss: 0.00027 learning rate: 0.00100 Time taken: 0:00:31.013524 ETA: 0:54:47.433525\n",
"2021-12-30 09:17:14,999 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.764\n",
"INFO:tensorflow:global_step/sec: 3.08259\n",
"2021-12-30 09:17:15,321 [INFO] tensorflow: global_step/sec: 3.08259\n",
"INFO:tensorflow:epoch = 14.072916666666666, learning_rate = 0.0009999999, loss = 0.00026307395, step = 1351 (5.530 sec)\n",
"2021-12-30 09:17:15,646 [INFO] tensorflow: epoch = 14.072916666666666, learning_rate = 0.0009999999, loss = 0.00026307395, step = 1351 (5.530 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08079\n",
"2021-12-30 09:17:18,242 [INFO] tensorflow: global_step/sec: 3.08079\n",
"INFO:tensorflow:epoch = 14.25, learning_rate = 0.0009999999, loss = 0.00039107597, step = 1368 (5.495 sec)\n",
"2021-12-30 09:17:21,141 [INFO] tensorflow: epoch = 14.25, learning_rate = 0.0009999999, loss = 0.00039107597, step = 1368 (5.495 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10471\n",
"2021-12-30 09:17:21,141 [INFO] tensorflow: global_step/sec: 3.10471\n",
"2021-12-30 09:17:23,093 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.710\n",
"INFO:tensorflow:global_step/sec: 3.08879\n",
"2021-12-30 09:17:24,055 [INFO] tensorflow: global_step/sec: 3.08879\n",
"INFO:tensorflow:epoch = 14.427083333333332, learning_rate = 0.0009999999, loss = 0.00028784072, step = 1385 (5.452 sec)\n",
"2021-12-30 09:17:26,593 [INFO] tensorflow: epoch = 14.427083333333332, learning_rate = 0.0009999999, loss = 0.00028784072, step = 1385 (5.452 sec)\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO:tensorflow:global_step/sec: 3.16184\n",
"2021-12-30 09:17:26,901 [INFO] tensorflow: global_step/sec: 3.16184\n",
"INFO:tensorflow:global_step/sec: 3.10493\n",
"2021-12-30 09:17:29,800 [INFO] tensorflow: global_step/sec: 3.10493\n",
"2021-12-30 09:17:31,053 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.125\n",
"INFO:tensorflow:epoch = 14.604166666666666, learning_rate = 0.0009999999, loss = 0.00024648634, step = 1402 (5.454 sec)\n",
"2021-12-30 09:17:32,046 [INFO] tensorflow: epoch = 14.604166666666666, learning_rate = 0.0009999999, loss = 0.00024648634, step = 1402 (5.454 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08606\n",
"2021-12-30 09:17:32,716 [INFO] tensorflow: global_step/sec: 3.08606\n",
"INFO:tensorflow:global_step/sec: 3.11472\n",
"2021-12-30 09:17:35,606 [INFO] tensorflow: global_step/sec: 3.11472\n",
"INFO:tensorflow:epoch = 14.78125, learning_rate = 0.0009999999, loss = 0.00022511775, step = 1419 (5.496 sec)\n",
"2021-12-30 09:17:37,543 [INFO] tensorflow: epoch = 14.78125, learning_rate = 0.0009999999, loss = 0.00022511775, step = 1419 (5.496 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09056\n",
"2021-12-30 09:17:38,518 [INFO] tensorflow: global_step/sec: 3.09056\n",
"2021-12-30 09:17:39,168 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.645\n",
"INFO:tensorflow:global_step/sec: 3.12196\n",
"2021-12-30 09:17:41,401 [INFO] tensorflow: global_step/sec: 3.12196\n",
"INFO:tensorflow:epoch = 14.958333333333332, learning_rate = 0.0009999999, loss = 0.00031049515, step = 1436 (5.432 sec)\n",
"2021-12-30 09:17:42,975 [INFO] tensorflow: epoch = 14.958333333333332, learning_rate = 0.0009999999, loss = 0.00031049515, step = 1436 (5.432 sec)\n",
"INFO:tensorflow:global_step/sec: 3.14744\n",
"2021-12-30 09:17:44,260 [INFO] tensorflow: global_step/sec: 3.14744\n",
"2021-12-30 09:17:44,261 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 15/120: loss: 0.00026 learning rate: 0.00100 Time taken: 0:00:30.879579 ETA: 0:54:02.355827\n",
"INFO:tensorflow:global_step/sec: 3.09042\n",
"2021-12-30 09:17:47,172 [INFO] tensorflow: global_step/sec: 3.09042\n",
"2021-12-30 09:17:47,173 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.986\n",
"INFO:tensorflow:epoch = 15.135416666666666, learning_rate = 0.0009999999, loss = 0.00024893726, step = 1453 (5.522 sec)\n",
"2021-12-30 09:17:48,497 [INFO] tensorflow: epoch = 15.135416666666666, learning_rate = 0.0009999999, loss = 0.00024893726, step = 1453 (5.522 sec)\n",
"INFO:tensorflow:global_step/sec: 3.04773\n",
"2021-12-30 09:17:50,126 [INFO] tensorflow: global_step/sec: 3.04773\n",
"INFO:tensorflow:global_step/sec: 3.06645\n",
"2021-12-30 09:17:53,060 [INFO] tensorflow: global_step/sec: 3.06645\n",
"INFO:tensorflow:epoch = 15.3125, learning_rate = 0.0009999999, loss = 0.0002423692, step = 1470 (5.530 sec)\n",
"2021-12-30 09:17:54,027 [INFO] tensorflow: epoch = 15.3125, learning_rate = 0.0009999999, loss = 0.0002423692, step = 1470 (5.530 sec)\n",
"2021-12-30 09:17:55,309 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.583\n",
"INFO:tensorflow:global_step/sec: 3.10178\n",
"2021-12-30 09:17:55,962 [INFO] tensorflow: global_step/sec: 3.10178\n",
"INFO:tensorflow:global_step/sec: 3.06197\n",
"2021-12-30 09:17:58,901 [INFO] tensorflow: global_step/sec: 3.06197\n",
"INFO:tensorflow:epoch = 15.489583333333332, learning_rate = 0.0009999999, loss = 0.00025314765, step = 1487 (5.518 sec)\n",
"2021-12-30 09:17:59,545 [INFO] tensorflow: epoch = 15.489583333333332, learning_rate = 0.0009999999, loss = 0.00025314765, step = 1487 (5.518 sec)\n",
"INFO:tensorflow:global_step/sec: 3.1406\n",
"2021-12-30 09:18:01,767 [INFO] tensorflow: global_step/sec: 3.1406\n",
"2021-12-30 09:18:03,425 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.645\n",
"INFO:tensorflow:global_step/sec: 3.08152\n",
"2021-12-30 09:18:04,688 [INFO] tensorflow: global_step/sec: 3.08152\n",
"INFO:tensorflow:epoch = 15.666666666666666, learning_rate = 0.0009999999, loss = 0.00025017513, step = 1504 (5.469 sec)\n",
"2021-12-30 09:18:05,014 [INFO] tensorflow: epoch = 15.666666666666666, learning_rate = 0.0009999999, loss = 0.00025017513, step = 1504 (5.469 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11047\n",
"2021-12-30 09:18:07,581 [INFO] tensorflow: global_step/sec: 3.11047\n",
"INFO:tensorflow:epoch = 15.84375, learning_rate = 0.0009999999, loss = 0.00030859778, step = 1521 (5.469 sec)\n",
"2021-12-30 09:18:10,483 [INFO] tensorflow: epoch = 15.84375, learning_rate = 0.0009999999, loss = 0.00030859778, step = 1521 (5.469 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10003\n",
"2021-12-30 09:18:10,484 [INFO] tensorflow: global_step/sec: 3.10003\n",
"2021-12-30 09:18:11,476 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.839\n",
"INFO:tensorflow:global_step/sec: 3.07145\n",
"2021-12-30 09:18:13,415 [INFO] tensorflow: global_step/sec: 3.07145\n",
"2021-12-30 09:18:15,368 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 16/120: loss: 0.00032 learning rate: 0.00100 Time taken: 0:00:31.108582 ETA: 0:53:55.292530\n",
"INFO:tensorflow:epoch = 16.020833333333332, learning_rate = 0.0009999999, loss = 0.00042072224, step = 1538 (5.539 sec)\n",
"2021-12-30 09:18:16,022 [INFO] tensorflow: epoch = 16.020833333333332, learning_rate = 0.0009999999, loss = 0.00042072224, step = 1538 (5.539 sec)\n",
"INFO:tensorflow:global_step/sec: 3.05371\n",
"2021-12-30 09:18:16,362 [INFO] tensorflow: global_step/sec: 3.05371\n",
"INFO:tensorflow:global_step/sec: 3.14906\n",
"2021-12-30 09:18:19,220 [INFO] tensorflow: global_step/sec: 3.14906\n",
"2021-12-30 09:18:19,536 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.816\n",
"INFO:tensorflow:epoch = 16.197916666666664, learning_rate = 0.0009999999, loss = 0.00020862516, step = 1555 (5.417 sec)\n",
"2021-12-30 09:18:21,440 [INFO] tensorflow: epoch = 16.197916666666664, learning_rate = 0.0009999999, loss = 0.00020862516, step = 1555 (5.417 sec)\n",
"INFO:tensorflow:global_step/sec: 3.13273\n",
"2021-12-30 09:18:22,093 [INFO] tensorflow: global_step/sec: 3.13273\n",
"INFO:tensorflow:global_step/sec: 3.11179\n",
"2021-12-30 09:18:24,985 [INFO] tensorflow: global_step/sec: 3.11179\n",
"INFO:tensorflow:epoch = 16.375, learning_rate = 0.0009999999, loss = 0.00031568634, step = 1572 (5.517 sec)\n",
"2021-12-30 09:18:26,956 [INFO] tensorflow: epoch = 16.375, learning_rate = 0.0009999999, loss = 0.00031568634, step = 1572 (5.517 sec)\n",
"2021-12-30 09:18:27,617 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.751\n",
"INFO:tensorflow:global_step/sec: 3.04059\n",
"2021-12-30 09:18:27,945 [INFO] tensorflow: global_step/sec: 3.04059\n",
"INFO:tensorflow:global_step/sec: 3.01534\n",
"2021-12-30 09:18:30,930 [INFO] tensorflow: global_step/sec: 3.01534\n",
"INFO:tensorflow:epoch = 16.552083333333332, learning_rate = 0.0009999999, loss = 0.0005011543, step = 1589 (5.602 sec)\n",
"2021-12-30 09:18:32,558 [INFO] tensorflow: epoch = 16.552083333333332, learning_rate = 0.0009999999, loss = 0.0005011543, step = 1589 (5.602 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08467\n",
"2021-12-30 09:18:33,847 [INFO] tensorflow: global_step/sec: 3.08467\n",
"2021-12-30 09:18:35,769 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.533\n",
"INFO:tensorflow:global_step/sec: 3.11263\n",
"2021-12-30 09:18:36,739 [INFO] tensorflow: global_step/sec: 3.11263\n",
"INFO:tensorflow:epoch = 16.729166666666664, learning_rate = 0.0009999999, loss = 0.0002468747, step = 1606 (5.504 sec)\n",
"2021-12-30 09:18:38,062 [INFO] tensorflow: epoch = 16.729166666666664, learning_rate = 0.0009999999, loss = 0.0002468747, step = 1606 (5.504 sec)\n",
"INFO:tensorflow:global_step/sec: 3.04707\n",
"2021-12-30 09:18:39,692 [INFO] tensorflow: global_step/sec: 3.04707\n",
"INFO:tensorflow:global_step/sec: 3.0979\n",
"2021-12-30 09:18:42,598 [INFO] tensorflow: global_step/sec: 3.0979\n",
"INFO:tensorflow:epoch = 16.90625, learning_rate = 0.0009999999, loss = 0.00021843152, step = 1623 (5.527 sec)\n",
"2021-12-30 09:18:43,590 [INFO] tensorflow: epoch = 16.90625, learning_rate = 0.0009999999, loss = 0.00021843152, step = 1623 (5.527 sec)\n",
"2021-12-30 09:18:43,915 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.553\n",
"INFO:tensorflow:global_step/sec: 3.06056\n",
"2021-12-30 09:18:45,538 [INFO] tensorflow: global_step/sec: 3.06056\n",
"2021-12-30 09:18:46,505 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 17/120: loss: 0.00021 learning rate: 0.00100 Time taken: 0:00:31.148093 ETA: 0:53:28.253528\n",
"INFO:tensorflow:global_step/sec: 3.08705\n",
"2021-12-30 09:18:48,454 [INFO] tensorflow: global_step/sec: 3.08705\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO:tensorflow:epoch = 17.083333333333332, learning_rate = 0.0009999999, loss = 0.00025058712, step = 1640 (5.510 sec)\n",
"2021-12-30 09:18:49,100 [INFO] tensorflow: epoch = 17.083333333333332, learning_rate = 0.0009999999, loss = 0.00025058712, step = 1640 (5.510 sec)\n",
"INFO:tensorflow:global_step/sec: 3.14287\n",
"2021-12-30 09:18:51,317 [INFO] tensorflow: global_step/sec: 3.14287\n",
"2021-12-30 09:18:51,961 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.858\n",
"INFO:tensorflow:global_step/sec: 3.06136\n",
"2021-12-30 09:18:54,257 [INFO] tensorflow: global_step/sec: 3.06136\n",
"INFO:tensorflow:epoch = 17.260416666666664, learning_rate = 0.0009999999, loss = 0.0002757479, step = 1657 (5.465 sec)\n",
"2021-12-30 09:18:54,564 [INFO] tensorflow: epoch = 17.260416666666664, learning_rate = 0.0009999999, loss = 0.0002757479, step = 1657 (5.465 sec)\n",
"INFO:tensorflow:global_step/sec: 3.1104\n",
"2021-12-30 09:18:57,151 [INFO] tensorflow: global_step/sec: 3.1104\n",
"INFO:tensorflow:epoch = 17.4375, learning_rate = 0.0009999999, loss = 0.00025238888, step = 1674 (5.495 sec)\n",
"2021-12-30 09:19:00,060 [INFO] tensorflow: epoch = 17.4375, learning_rate = 0.0009999999, loss = 0.00025238888, step = 1674 (5.495 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09303\n",
"2021-12-30 09:19:00,060 [INFO] tensorflow: global_step/sec: 3.09303\n",
"2021-12-30 09:19:00,061 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.692\n",
"INFO:tensorflow:global_step/sec: 3.04305\n",
"2021-12-30 09:19:03,018 [INFO] tensorflow: global_step/sec: 3.04305\n",
"INFO:tensorflow:epoch = 17.614583333333332, learning_rate = 0.0009999999, loss = 0.00025216828, step = 1691 (5.548 sec)\n",
"2021-12-30 09:19:05,608 [INFO] tensorflow: epoch = 17.614583333333332, learning_rate = 0.0009999999, loss = 0.00025216828, step = 1691 (5.548 sec)\n",
"INFO:tensorflow:global_step/sec: 3.0959\n",
"2021-12-30 09:19:05,925 [INFO] tensorflow: global_step/sec: 3.0959\n",
"2021-12-30 09:19:08,170 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.666\n",
"INFO:tensorflow:global_step/sec: 3.11783\n",
"2021-12-30 09:19:08,812 [INFO] tensorflow: global_step/sec: 3.11783\n",
"INFO:tensorflow:epoch = 17.791666666666664, learning_rate = 0.0009999999, loss = 0.00040681113, step = 1708 (5.508 sec)\n",
"2021-12-30 09:19:11,116 [INFO] tensorflow: epoch = 17.791666666666664, learning_rate = 0.0009999999, loss = 0.00040681113, step = 1708 (5.508 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09565\n",
"2021-12-30 09:19:11,719 [INFO] tensorflow: global_step/sec: 3.09565\n",
"INFO:tensorflow:global_step/sec: 3.0891\n",
"2021-12-30 09:19:14,632 [INFO] tensorflow: global_step/sec: 3.0891\n",
"2021-12-30 09:19:16,262 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.716\n",
"INFO:tensorflow:epoch = 17.96875, learning_rate = 0.0009999999, loss = 0.00027708645, step = 1725 (5.463 sec)\n",
"2021-12-30 09:19:16,579 [INFO] tensorflow: epoch = 17.96875, learning_rate = 0.0009999999, loss = 0.00027708645, step = 1725 (5.463 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08574\n",
"2021-12-30 09:19:17,549 [INFO] tensorflow: global_step/sec: 3.08574\n",
"2021-12-30 09:19:17,550 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 18/120: loss: 0.00026 learning rate: 0.00100 Time taken: 0:00:31.038749 ETA: 0:52:45.952396\n",
"INFO:tensorflow:global_step/sec: 3.05138\n",
"2021-12-30 09:19:20,499 [INFO] tensorflow: global_step/sec: 3.05138\n",
"INFO:tensorflow:epoch = 18.145833333333332, learning_rate = 0.0009999999, loss = 0.0003496712, step = 1742 (5.496 sec)\n",
"2021-12-30 09:19:22,075 [INFO] tensorflow: epoch = 18.145833333333332, learning_rate = 0.0009999999, loss = 0.0003496712, step = 1742 (5.496 sec)\n",
"INFO:tensorflow:global_step/sec: 3.13478\n",
"2021-12-30 09:19:23,370 [INFO] tensorflow: global_step/sec: 3.13478\n",
"2021-12-30 09:19:24,345 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.743\n",
"INFO:tensorflow:global_step/sec: 3.10192\n",
"2021-12-30 09:19:26,271 [INFO] tensorflow: global_step/sec: 3.10192\n",
"INFO:tensorflow:epoch = 18.322916666666664, learning_rate = 0.0009999999, loss = 0.00019805644, step = 1759 (5.477 sec)\n",
"2021-12-30 09:19:27,552 [INFO] tensorflow: epoch = 18.322916666666664, learning_rate = 0.0009999999, loss = 0.00019805644, step = 1759 (5.477 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12455\n",
"2021-12-30 09:19:29,151 [INFO] tensorflow: global_step/sec: 3.12455\n",
"INFO:tensorflow:global_step/sec: 3.05983\n",
"2021-12-30 09:19:32,093 [INFO] tensorflow: global_step/sec: 3.05983\n",
"2021-12-30 09:19:32,405 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.815\n",
"INFO:tensorflow:epoch = 18.5, learning_rate = 0.0009999999, loss = 0.0002630841, step = 1776 (5.490 sec)\n",
"2021-12-30 09:19:33,042 [INFO] tensorflow: epoch = 18.5, learning_rate = 0.0009999999, loss = 0.0002630841, step = 1776 (5.490 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11647\n",
"2021-12-30 09:19:34,981 [INFO] tensorflow: global_step/sec: 3.11647\n",
"INFO:tensorflow:global_step/sec: 3.02217\n",
"2021-12-30 09:19:37,959 [INFO] tensorflow: global_step/sec: 3.02217\n",
"INFO:tensorflow:epoch = 18.677083333333332, learning_rate = 0.0009999999, loss = 0.00026733126, step = 1793 (5.548 sec)\n",
"2021-12-30 09:19:38,590 [INFO] tensorflow: epoch = 18.677083333333332, learning_rate = 0.0009999999, loss = 0.00026733126, step = 1793 (5.548 sec)\n",
"2021-12-30 09:19:40,552 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.550\n",
"INFO:tensorflow:global_step/sec: 3.0995\n",
"2021-12-30 09:19:40,862 [INFO] tensorflow: global_step/sec: 3.0995\n",
"INFO:tensorflow:global_step/sec: 3.14478\n",
"2021-12-30 09:19:43,724 [INFO] tensorflow: global_step/sec: 3.14478\n",
"INFO:tensorflow:epoch = 18.854166666666664, learning_rate = 0.0009999999, loss = 0.00026158406, step = 1810 (5.453 sec)\n",
"2021-12-30 09:19:44,043 [INFO] tensorflow: epoch = 18.854166666666664, learning_rate = 0.0009999999, loss = 0.00026158406, step = 1810 (5.453 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11514\n",
"2021-12-30 09:19:46,613 [INFO] tensorflow: global_step/sec: 3.11514\n",
"2021-12-30 09:19:48,596 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 19/120: loss: 0.00023 learning rate: 0.00100 Time taken: 0:00:31.030901 ETA: 0:52:14.121045\n",
"2021-12-30 09:19:48,596 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.865\n",
"INFO:tensorflow:epoch = 19.03125, learning_rate = 0.0009999999, loss = 0.00021037235, step = 1827 (5.512 sec)\n",
"2021-12-30 09:19:49,555 [INFO] tensorflow: epoch = 19.03125, learning_rate = 0.0009999999, loss = 0.00021037235, step = 1827 (5.512 sec)\n",
"INFO:tensorflow:global_step/sec: 3.05896\n",
"2021-12-30 09:19:49,555 [INFO] tensorflow: global_step/sec: 3.05896\n",
"INFO:tensorflow:global_step/sec: 3.09345\n",
"2021-12-30 09:19:52,465 [INFO] tensorflow: global_step/sec: 3.09345\n",
"INFO:tensorflow:epoch = 19.208333333333332, learning_rate = 0.0009999999, loss = 0.00029713588, step = 1844 (5.487 sec)\n",
"2021-12-30 09:19:55,042 [INFO] tensorflow: epoch = 19.208333333333332, learning_rate = 0.0009999999, loss = 0.00029713588, step = 1844 (5.487 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10827\n",
"2021-12-30 09:19:55,360 [INFO] tensorflow: global_step/sec: 3.10827\n",
"2021-12-30 09:19:56,652 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.827\n",
"INFO:tensorflow:global_step/sec: 3.06344\n",
"2021-12-30 09:19:58,298 [INFO] tensorflow: global_step/sec: 3.06344\n",
"INFO:tensorflow:epoch = 19.385416666666664, learning_rate = 0.0009999999, loss = 0.00036884216, step = 1861 (5.549 sec)\n",
"2021-12-30 09:20:00,591 [INFO] tensorflow: epoch = 19.385416666666664, learning_rate = 0.0009999999, loss = 0.00036884216, step = 1861 (5.549 sec)\n",
"INFO:tensorflow:global_step/sec: 3.04786\n",
"2021-12-30 09:20:01,251 [INFO] tensorflow: global_step/sec: 3.04786\n",
"INFO:tensorflow:global_step/sec: 3.10696\n",
"2021-12-30 09:20:04,148 [INFO] tensorflow: global_step/sec: 3.10696\n",
"2021-12-30 09:20:04,783 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.597\n",
"INFO:tensorflow:epoch = 19.5625, learning_rate = 0.0009999999, loss = 0.00030112942, step = 1878 (5.486 sec)\n",
"2021-12-30 09:20:06,077 [INFO] tensorflow: epoch = 19.5625, learning_rate = 0.0009999999, loss = 0.00030112942, step = 1878 (5.486 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10195\n",
"2021-12-30 09:20:07,049 [INFO] tensorflow: global_step/sec: 3.10195\n",
"INFO:tensorflow:global_step/sec: 3.12243\n",
"2021-12-30 09:20:09,932 [INFO] tensorflow: global_step/sec: 3.12243\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO:tensorflow:epoch = 19.739583333333332, learning_rate = 0.0009999999, loss = 0.0002879659, step = 1895 (5.474 sec)\n",
"2021-12-30 09:20:11,550 [INFO] tensorflow: epoch = 19.739583333333332, learning_rate = 0.0009999999, loss = 0.0002879659, step = 1895 (5.474 sec)\n",
"INFO:tensorflow:global_step/sec: 3.04048\n",
"2021-12-30 09:20:12,892 [INFO] tensorflow: global_step/sec: 3.04048\n",
"2021-12-30 09:20:12,892 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.663\n",
"INFO:tensorflow:global_step/sec: 3.17794\n",
"2021-12-30 09:20:15,724 [INFO] tensorflow: global_step/sec: 3.17794\n",
"INFO:tensorflow:epoch = 19.916666666666664, learning_rate = 0.0009999999, loss = 0.00027038585, step = 1912 (5.450 sec)\n",
"2021-12-30 09:20:17,000 [INFO] tensorflow: epoch = 19.916666666666664, learning_rate = 0.0009999999, loss = 0.00027038585, step = 1912 (5.450 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10279\n",
"2021-12-30 09:20:18,624 [INFO] tensorflow: global_step/sec: 3.10279\n",
"INFO:tensorflow:Saving checkpoints for step-1920.\n",
"2021-12-30 09:20:19,283 [INFO] tensorflow: Saving checkpoints for step-1920.\n",
"2021-12-30 09:20:22,835 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 23, 0.00s/step\n",
"2021-12-30 09:20:40,153 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 23, 1.73s/step\n",
"2021-12-30 09:20:56,052 [INFO] iva.detectnet_v2.evaluation.evaluation: step 20 / 23, 1.59s/step\n",
"Matching predictions to ground truth, class 1/1.: 100%|█| 217535/217535 [00:13<00:00, 15853.04it/s]\n",
"Epoch 20/120\n",
"=========================\n",
"\n",
"Validation cost: 0.000260\n",
"Mean average_precision (in %): 39.8133\n",
"\n",
"class name average precision (in %)\n",
"------------ --------------------------\n",
"car 39.8133\n",
"\n",
"Median Inference Time: 0.015636\n",
"INFO:tensorflow:epoch = 20.0, learning_rate = 0.0009999999, loss = 0.00028473453, step = 1920 (61.699 sec)\n",
"2021-12-30 09:21:18,699 [INFO] tensorflow: epoch = 20.0, learning_rate = 0.0009999999, loss = 0.00028473453, step = 1920 (61.699 sec)\n",
"2021-12-30 09:21:18,699 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 20/120: loss: 0.00028 learning rate: 0.00100 Time taken: 0:01:30.055663 ETA: 2:30:05.566287\n",
"2021-12-30 09:21:19,949 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 2.983\n",
"INFO:tensorflow:global_step/sec: 0.145258\n",
"2021-12-30 09:21:20,583 [INFO] tensorflow: global_step/sec: 0.145258\n",
"INFO:tensorflow:global_step/sec: 3.04139\n",
"2021-12-30 09:21:23,542 [INFO] tensorflow: global_step/sec: 3.04139\n",
"INFO:tensorflow:epoch = 20.177083333333332, learning_rate = 0.0009999999, loss = 0.0001932427, step = 1937 (5.491 sec)\n",
"2021-12-30 09:21:24,190 [INFO] tensorflow: epoch = 20.177083333333332, learning_rate = 0.0009999999, loss = 0.0001932427, step = 1937 (5.491 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08021\n",
"2021-12-30 09:21:26,464 [INFO] tensorflow: global_step/sec: 3.08021\n",
"2021-12-30 09:21:28,073 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.618\n",
"INFO:tensorflow:global_step/sec: 3.11201\n",
"2021-12-30 09:21:29,356 [INFO] tensorflow: global_step/sec: 3.11201\n",
"INFO:tensorflow:epoch = 20.354166666666664, learning_rate = 0.0009999999, loss = 0.00030855078, step = 1954 (5.498 sec)\n",
"2021-12-30 09:21:29,688 [INFO] tensorflow: epoch = 20.354166666666664, learning_rate = 0.0009999999, loss = 0.00030855078, step = 1954 (5.498 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09237\n",
"2021-12-30 09:21:32,267 [INFO] tensorflow: global_step/sec: 3.09237\n",
"INFO:tensorflow:epoch = 20.53125, learning_rate = 0.0009999999, loss = 0.0001480727, step = 1971 (5.462 sec)\n",
"2021-12-30 09:21:35,150 [INFO] tensorflow: epoch = 20.53125, learning_rate = 0.0009999999, loss = 0.0001480727, step = 1971 (5.462 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12046\n",
"2021-12-30 09:21:35,151 [INFO] tensorflow: global_step/sec: 3.12046\n",
"2021-12-30 09:21:36,128 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.828\n",
"INFO:tensorflow:global_step/sec: 3.06863\n",
"2021-12-30 09:21:38,084 [INFO] tensorflow: global_step/sec: 3.06863\n",
"INFO:tensorflow:epoch = 20.708333333333332, learning_rate = 0.0009999999, loss = 0.00019896444, step = 1988 (5.502 sec)\n",
"2021-12-30 09:21:40,652 [INFO] tensorflow: epoch = 20.708333333333332, learning_rate = 0.0009999999, loss = 0.00019896444, step = 1988 (5.502 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11116\n",
"2021-12-30 09:21:40,977 [INFO] tensorflow: global_step/sec: 3.11116\n",
"INFO:tensorflow:global_step/sec: 3.10801\n",
"2021-12-30 09:21:43,872 [INFO] tensorflow: global_step/sec: 3.10801\n",
"2021-12-30 09:21:44,186 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.824\n",
"INFO:tensorflow:epoch = 20.885416666666664, learning_rate = 0.0009999999, loss = 0.00020042778, step = 2005 (5.418 sec)\n",
"2021-12-30 09:21:46,070 [INFO] tensorflow: epoch = 20.885416666666664, learning_rate = 0.0009999999, loss = 0.00020042778, step = 2005 (5.418 sec)\n",
"INFO:tensorflow:global_step/sec: 3.18026\n",
"2021-12-30 09:21:46,702 [INFO] tensorflow: global_step/sec: 3.18026\n",
"INFO:tensorflow:global_step/sec: 3.10773\n",
"2021-12-30 09:21:49,598 [INFO] tensorflow: global_step/sec: 3.10773\n",
"2021-12-30 09:21:49,599 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 21/120: loss: 0.00022 learning rate: 0.00100 Time taken: 0:00:30.919105 ETA: 0:51:00.991400\n",
"INFO:tensorflow:epoch = 21.0625, learning_rate = 0.0009999999, loss = 0.00027568548, step = 2022 (5.481 sec)\n",
"2021-12-30 09:21:51,551 [INFO] tensorflow: epoch = 21.0625, learning_rate = 0.0009999999, loss = 0.00027568548, step = 2022 (5.481 sec)\n",
"2021-12-30 09:21:52,195 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.972\n",
"INFO:tensorflow:global_step/sec: 3.08587\n",
"2021-12-30 09:21:52,515 [INFO] tensorflow: global_step/sec: 3.08587\n",
"INFO:tensorflow:global_step/sec: 3.12001\n",
"2021-12-30 09:21:55,399 [INFO] tensorflow: global_step/sec: 3.12001\n",
"INFO:tensorflow:epoch = 21.239583333333332, learning_rate = 0.0009999999, loss = 0.00020342169, step = 2039 (5.449 sec)\n",
"2021-12-30 09:21:57,000 [INFO] tensorflow: epoch = 21.239583333333332, learning_rate = 0.0009999999, loss = 0.00020342169, step = 2039 (5.449 sec)\n",
"INFO:tensorflow:global_step/sec: 3.1371\n",
"2021-12-30 09:21:58,268 [INFO] tensorflow: global_step/sec: 3.1371\n",
"2021-12-30 09:22:00,202 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.979\n",
"INFO:tensorflow:global_step/sec: 3.11999\n",
"2021-12-30 09:22:01,153 [INFO] tensorflow: global_step/sec: 3.11999\n",
"INFO:tensorflow:epoch = 21.416666666666664, learning_rate = 0.0009999999, loss = 0.00023206923, step = 2056 (5.439 sec)\n",
"2021-12-30 09:22:02,438 [INFO] tensorflow: epoch = 21.416666666666664, learning_rate = 0.0009999999, loss = 0.00023206923, step = 2056 (5.439 sec)\n",
"INFO:tensorflow:global_step/sec: 3.0912\n",
"2021-12-30 09:22:04,064 [INFO] tensorflow: global_step/sec: 3.0912\n",
"INFO:tensorflow:global_step/sec: 3.09343\n",
"2021-12-30 09:22:06,974 [INFO] tensorflow: global_step/sec: 3.09343\n",
"INFO:tensorflow:epoch = 21.59375, learning_rate = 0.0009999999, loss = 0.00022103539, step = 2073 (5.491 sec)\n",
"2021-12-30 09:22:07,929 [INFO] tensorflow: epoch = 21.59375, learning_rate = 0.0009999999, loss = 0.00022103539, step = 2073 (5.491 sec)\n",
"2021-12-30 09:22:08,255 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.834\n",
"INFO:tensorflow:global_step/sec: 3.07633\n",
"2021-12-30 09:22:09,899 [INFO] tensorflow: global_step/sec: 3.07633\n",
"INFO:tensorflow:global_step/sec: 3.12428\n",
"2021-12-30 09:22:12,780 [INFO] tensorflow: global_step/sec: 3.12428\n",
"INFO:tensorflow:epoch = 21.770833333333332, learning_rate = 0.0009999999, loss = 0.00019521004, step = 2090 (5.484 sec)\n",
"2021-12-30 09:22:13,412 [INFO] tensorflow: epoch = 21.770833333333332, learning_rate = 0.0009999999, loss = 0.00019521004, step = 2090 (5.484 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07794\n",
"2021-12-30 09:22:15,704 [INFO] tensorflow: global_step/sec: 3.07794\n",
"2021-12-30 09:22:16,358 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.683\n",
"INFO:tensorflow:global_step/sec: 3.10774\n",
"2021-12-30 09:22:18,600 [INFO] tensorflow: global_step/sec: 3.10774\n",
"INFO:tensorflow:epoch = 21.947916666666664, learning_rate = 0.0009999999, loss = 0.00029071316, step = 2107 (5.506 sec)\n",
"2021-12-30 09:22:18,919 [INFO] tensorflow: epoch = 21.947916666666664, learning_rate = 0.0009999999, loss = 0.00029071316, step = 2107 (5.506 sec)\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"2021-12-30 09:22:20,564 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 22/120: loss: 0.00027 learning rate: 0.00100 Time taken: 0:00:30.983797 ETA: 0:50:36.412137\n",
"INFO:tensorflow:global_step/sec: 3.06592\n",
"2021-12-30 09:22:21,535 [INFO] tensorflow: global_step/sec: 3.06592\n",
"INFO:tensorflow:epoch = 22.125, learning_rate = 0.0009999999, loss = 0.0002627346, step = 2124 (5.517 sec)\n",
"2021-12-30 09:22:24,436 [INFO] tensorflow: epoch = 22.125, learning_rate = 0.0009999999, loss = 0.0002627346, step = 2124 (5.517 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10224\n",
"2021-12-30 09:22:24,437 [INFO] tensorflow: global_step/sec: 3.10224\n",
"2021-12-30 09:22:24,437 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.757\n",
"INFO:tensorflow:global_step/sec: 3.01358\n",
"2021-12-30 09:22:27,423 [INFO] tensorflow: global_step/sec: 3.01358\n",
"INFO:tensorflow:epoch = 22.302083333333332, learning_rate = 0.0009999999, loss = 0.0002619341, step = 2141 (5.552 sec)\n",
"2021-12-30 09:22:29,988 [INFO] tensorflow: epoch = 22.302083333333332, learning_rate = 0.0009999999, loss = 0.0002619341, step = 2141 (5.552 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11816\n",
"2021-12-30 09:22:30,309 [INFO] tensorflow: global_step/sec: 3.11816\n",
"2021-12-30 09:22:32,553 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.645\n",
"INFO:tensorflow:global_step/sec: 3.1099\n",
"2021-12-30 09:22:33,203 [INFO] tensorflow: global_step/sec: 3.1099\n",
"INFO:tensorflow:epoch = 22.479166666666664, learning_rate = 0.0009999999, loss = 0.00021975051, step = 2158 (5.473 sec)\n",
"2021-12-30 09:22:35,460 [INFO] tensorflow: epoch = 22.479166666666664, learning_rate = 0.0009999999, loss = 0.00021975051, step = 2158 (5.473 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08542\n",
"2021-12-30 09:22:36,120 [INFO] tensorflow: global_step/sec: 3.08542\n",
"INFO:tensorflow:global_step/sec: 3.10117\n",
"2021-12-30 09:22:39,022 [INFO] tensorflow: global_step/sec: 3.10117\n",
"2021-12-30 09:22:40,659 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.672\n",
"INFO:tensorflow:epoch = 22.65625, learning_rate = 0.0009999999, loss = 0.00021660054, step = 2175 (5.506 sec)\n",
"2021-12-30 09:22:40,966 [INFO] tensorflow: epoch = 22.65625, learning_rate = 0.0009999999, loss = 0.00021660054, step = 2175 (5.506 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09896\n",
"2021-12-30 09:22:41,927 [INFO] tensorflow: global_step/sec: 3.09896\n",
"INFO:tensorflow:global_step/sec: 3.10934\n",
"2021-12-30 09:22:44,821 [INFO] tensorflow: global_step/sec: 3.10934\n",
"INFO:tensorflow:epoch = 22.833333333333332, learning_rate = 0.0009999999, loss = 0.00018921171, step = 2192 (5.457 sec)\n",
"2021-12-30 09:22:46,422 [INFO] tensorflow: epoch = 22.833333333333332, learning_rate = 0.0009999999, loss = 0.00018921171, step = 2192 (5.457 sec)\n",
"INFO:tensorflow:global_step/sec: 3.13345\n",
"2021-12-30 09:22:47,693 [INFO] tensorflow: global_step/sec: 3.13345\n",
"2021-12-30 09:22:48,681 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.934\n",
"INFO:tensorflow:global_step/sec: 3.06987\n",
"2021-12-30 09:22:50,625 [INFO] tensorflow: global_step/sec: 3.06987\n",
"2021-12-30 09:22:51,594 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 23/120: loss: 0.00024 learning rate: 0.00100 Time taken: 0:00:31.004936 ETA: 0:50:07.478767\n",
"INFO:tensorflow:epoch = 23.010416666666664, learning_rate = 0.0009999999, loss = 0.0002902623, step = 2209 (5.511 sec)\n",
"2021-12-30 09:22:51,934 [INFO] tensorflow: epoch = 23.010416666666664, learning_rate = 0.0009999999, loss = 0.0002902623, step = 2209 (5.511 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07389\n",
"2021-12-30 09:22:53,553 [INFO] tensorflow: global_step/sec: 3.07389\n",
"INFO:tensorflow:global_step/sec: 3.09855\n",
"2021-12-30 09:22:56,458 [INFO] tensorflow: global_step/sec: 3.09855\n",
"2021-12-30 09:22:56,776 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.707\n",
"INFO:tensorflow:epoch = 23.1875, learning_rate = 0.0009999999, loss = 0.00024422532, step = 2226 (5.490 sec)\n",
"2021-12-30 09:22:57,423 [INFO] tensorflow: epoch = 23.1875, learning_rate = 0.0009999999, loss = 0.00024422532, step = 2226 (5.490 sec)\n",
"INFO:tensorflow:global_step/sec: 3.03964\n",
"2021-12-30 09:22:59,418 [INFO] tensorflow: global_step/sec: 3.03964\n",
"INFO:tensorflow:global_step/sec: 3.01577\n",
"2021-12-30 09:23:02,403 [INFO] tensorflow: global_step/sec: 3.01577\n",
"INFO:tensorflow:epoch = 23.364583333333332, learning_rate = 0.0009999999, loss = 0.00029435047, step = 2243 (5.629 sec)\n",
"2021-12-30 09:23:03,052 [INFO] tensorflow: epoch = 23.364583333333332, learning_rate = 0.0009999999, loss = 0.00029435047, step = 2243 (5.629 sec)\n",
"2021-12-30 09:23:04,957 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.447\n",
"INFO:tensorflow:global_step/sec: 3.11152\n",
"2021-12-30 09:23:05,295 [INFO] tensorflow: global_step/sec: 3.11152\n",
"INFO:tensorflow:global_step/sec: 3.05512\n",
"2021-12-30 09:23:08,241 [INFO] tensorflow: global_step/sec: 3.05512\n",
"INFO:tensorflow:epoch = 23.541666666666664, learning_rate = 0.0009999999, loss = 0.00038957768, step = 2260 (5.523 sec)\n",
"2021-12-30 09:23:08,575 [INFO] tensorflow: epoch = 23.541666666666664, learning_rate = 0.0009999999, loss = 0.00038957768, step = 2260 (5.523 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09516\n",
"2021-12-30 09:23:11,149 [INFO] tensorflow: global_step/sec: 3.09516\n",
"2021-12-30 09:23:13,053 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.703\n",
"INFO:tensorflow:epoch = 23.71875, learning_rate = 0.0009999999, loss = 0.00024690735, step = 2277 (5.445 sec)\n",
"2021-12-30 09:23:14,020 [INFO] tensorflow: epoch = 23.71875, learning_rate = 0.0009999999, loss = 0.00024690735, step = 2277 (5.445 sec)\n",
"INFO:tensorflow:global_step/sec: 3.1341\n",
"2021-12-30 09:23:14,021 [INFO] tensorflow: global_step/sec: 3.1341\n",
"INFO:tensorflow:global_step/sec: 3.09135\n",
"2021-12-30 09:23:16,932 [INFO] tensorflow: global_step/sec: 3.09135\n",
"INFO:tensorflow:epoch = 23.895833333333332, learning_rate = 0.0009999999, loss = 0.00016519395, step = 2294 (5.464 sec)\n",
"2021-12-30 09:23:19,484 [INFO] tensorflow: epoch = 23.895833333333332, learning_rate = 0.0009999999, loss = 0.00016519395, step = 2294 (5.464 sec)\n",
"INFO:tensorflow:global_step/sec: 3.13561\n",
"2021-12-30 09:23:19,802 [INFO] tensorflow: global_step/sec: 3.13561\n",
"2021-12-30 09:23:21,101 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.852\n",
"INFO:tensorflow:global_step/sec: 3.07644\n",
"2021-12-30 09:23:22,728 [INFO] tensorflow: global_step/sec: 3.07644\n",
"2021-12-30 09:23:22,728 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 24/120: loss: 0.00019 learning rate: 0.00100 Time taken: 0:00:31.138706 ETA: 0:49:49.315750\n",
"INFO:tensorflow:epoch = 24.072916666666664, learning_rate = 0.0009999999, loss = 0.00020297745, step = 2311 (5.495 sec)\n",
"2021-12-30 09:23:24,979 [INFO] tensorflow: epoch = 24.072916666666664, learning_rate = 0.0009999999, loss = 0.00020297745, step = 2311 (5.495 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11187\n",
"2021-12-30 09:23:25,620 [INFO] tensorflow: global_step/sec: 3.11187\n",
"INFO:tensorflow:global_step/sec: 3.10994\n",
"2021-12-30 09:23:28,514 [INFO] tensorflow: global_step/sec: 3.10994\n",
"2021-12-30 09:23:29,153 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.841\n",
"INFO:tensorflow:epoch = 24.25, learning_rate = 0.0009999999, loss = 0.00022251203, step = 2328 (5.474 sec)\n",
"2021-12-30 09:23:30,453 [INFO] tensorflow: epoch = 24.25, learning_rate = 0.0009999999, loss = 0.00022251203, step = 2328 (5.474 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08311\n",
"2021-12-30 09:23:31,433 [INFO] tensorflow: global_step/sec: 3.08311\n",
"INFO:tensorflow:global_step/sec: 3.09899\n",
"2021-12-30 09:23:34,337 [INFO] tensorflow: global_step/sec: 3.09899\n",
"INFO:tensorflow:epoch = 24.427083333333332, learning_rate = 0.0009999999, loss = 0.0002522865, step = 2345 (5.492 sec)\n",
"2021-12-30 09:23:35,945 [INFO] tensorflow: epoch = 24.427083333333332, learning_rate = 0.0009999999, loss = 0.0002522865, step = 2345 (5.492 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12769\n",
"2021-12-30 09:23:37,215 [INFO] tensorflow: global_step/sec: 3.12769\n",
"2021-12-30 09:23:37,215 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.807\n",
"INFO:tensorflow:global_step/sec: 3.06906\n",
"2021-12-30 09:23:40,147 [INFO] tensorflow: global_step/sec: 3.06906\n",
"INFO:tensorflow:epoch = 24.604166666666664, learning_rate = 0.0009999999, loss = 0.0002667546, step = 2362 (5.486 sec)\n",
"2021-12-30 09:23:41,430 [INFO] tensorflow: epoch = 24.604166666666664, learning_rate = 0.0009999999, loss = 0.0002667546, step = 2362 (5.486 sec)\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO:tensorflow:global_step/sec: 3.14969\n",
"2021-12-30 09:23:43,004 [INFO] tensorflow: global_step/sec: 3.14969\n",
"2021-12-30 09:23:45,272 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.826\n",
"INFO:tensorflow:global_step/sec: 3.10588\n",
"2021-12-30 09:23:45,902 [INFO] tensorflow: global_step/sec: 3.10588\n",
"INFO:tensorflow:epoch = 24.78125, learning_rate = 0.0009999999, loss = 0.00018793397, step = 2379 (5.437 sec)\n",
"2021-12-30 09:23:46,867 [INFO] tensorflow: epoch = 24.78125, learning_rate = 0.0009999999, loss = 0.00018793397, step = 2379 (5.437 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09285\n",
"2021-12-30 09:23:48,812 [INFO] tensorflow: global_step/sec: 3.09285\n",
"INFO:tensorflow:global_step/sec: 3.06163\n",
"2021-12-30 09:23:51,752 [INFO] tensorflow: global_step/sec: 3.06163\n",
"INFO:tensorflow:epoch = 24.958333333333332, learning_rate = 0.0009999999, loss = 0.00019745229, step = 2396 (5.533 sec)\n",
"2021-12-30 09:23:52,399 [INFO] tensorflow: epoch = 24.958333333333332, learning_rate = 0.0009999999, loss = 0.00019745229, step = 2396 (5.533 sec)\n",
"2021-12-30 09:23:53,392 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.629\n",
"2021-12-30 09:23:53,722 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 25/120: loss: 0.00022 learning rate: 0.00100 Time taken: 0:00:30.990731 ETA: 0:49:04.119422\n",
"INFO:tensorflow:global_step/sec: 3.0661\n",
"2021-12-30 09:23:54,687 [INFO] tensorflow: global_step/sec: 3.0661\n",
"INFO:tensorflow:global_step/sec: 3.11623\n",
"2021-12-30 09:23:57,575 [INFO] tensorflow: global_step/sec: 3.11623\n",
"INFO:tensorflow:epoch = 25.135416666666664, learning_rate = 0.0009999999, loss = 0.00023486838, step = 2413 (5.494 sec)\n",
"2021-12-30 09:23:57,893 [INFO] tensorflow: epoch = 25.135416666666664, learning_rate = 0.0009999999, loss = 0.00023486838, step = 2413 (5.494 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10098\n",
"2021-12-30 09:24:00,477 [INFO] tensorflow: global_step/sec: 3.10098\n",
"2021-12-30 09:24:01,464 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.779\n",
"INFO:tensorflow:epoch = 25.3125, learning_rate = 0.0009999999, loss = 0.00021245792, step = 2430 (5.551 sec)\n",
"2021-12-30 09:24:03,444 [INFO] tensorflow: epoch = 25.3125, learning_rate = 0.0009999999, loss = 0.00021245792, step = 2430 (5.551 sec)\n",
"INFO:tensorflow:global_step/sec: 3.03272\n",
"2021-12-30 09:24:03,445 [INFO] tensorflow: global_step/sec: 3.03272\n",
"INFO:tensorflow:global_step/sec: 3.09359\n",
"2021-12-30 09:24:06,355 [INFO] tensorflow: global_step/sec: 3.09359\n",
"INFO:tensorflow:epoch = 25.489583333333332, learning_rate = 0.0009999999, loss = 0.00023107036, step = 2447 (5.499 sec)\n",
"2021-12-30 09:24:08,944 [INFO] tensorflow: epoch = 25.489583333333332, learning_rate = 0.0009999999, loss = 0.00023107036, step = 2447 (5.499 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09445\n",
"2021-12-30 09:24:09,263 [INFO] tensorflow: global_step/sec: 3.09445\n",
"2021-12-30 09:24:09,605 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.568\n",
"INFO:tensorflow:global_step/sec: 3.0411\n",
"2021-12-30 09:24:12,222 [INFO] tensorflow: global_step/sec: 3.0411\n",
"INFO:tensorflow:epoch = 25.666666666666664, learning_rate = 0.0009999999, loss = 0.00033157878, step = 2464 (5.535 sec)\n",
"2021-12-30 09:24:14,478 [INFO] tensorflow: epoch = 25.666666666666664, learning_rate = 0.0009999999, loss = 0.00033157878, step = 2464 (5.535 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10768\n",
"2021-12-30 09:24:15,118 [INFO] tensorflow: global_step/sec: 3.10768\n",
"2021-12-30 09:24:17,691 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.737\n",
"INFO:tensorflow:global_step/sec: 3.09965\n",
"2021-12-30 09:24:18,022 [INFO] tensorflow: global_step/sec: 3.09965\n",
"INFO:tensorflow:epoch = 25.84375, learning_rate = 0.0009999999, loss = 0.00028338737, step = 2481 (5.463 sec)\n",
"2021-12-30 09:24:19,941 [INFO] tensorflow: epoch = 25.84375, learning_rate = 0.0009999999, loss = 0.00028338737, step = 2481 (5.463 sec)\n",
"INFO:tensorflow:global_step/sec: 3.15276\n",
"2021-12-30 09:24:20,877 [INFO] tensorflow: global_step/sec: 3.15276\n",
"INFO:tensorflow:global_step/sec: 3.08876\n",
"2021-12-30 09:24:23,790 [INFO] tensorflow: global_step/sec: 3.08876\n",
"2021-12-30 09:24:24,765 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 26/120: loss: 0.00028 learning rate: 0.00100 Time taken: 0:00:31.025936 ETA: 0:48:36.437996\n",
"INFO:tensorflow:epoch = 26.020833333333332, learning_rate = 0.0009999999, loss = 0.00022633652, step = 2498 (5.460 sec)\n",
"2021-12-30 09:24:25,400 [INFO] tensorflow: epoch = 26.020833333333332, learning_rate = 0.0009999999, loss = 0.00022633652, step = 2498 (5.460 sec)\n",
"2021-12-30 09:24:25,722 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.903\n",
"INFO:tensorflow:global_step/sec: 3.11961\n",
"2021-12-30 09:24:26,675 [INFO] tensorflow: global_step/sec: 3.11961\n",
"INFO:tensorflow:global_step/sec: 3.08642\n",
"2021-12-30 09:24:29,591 [INFO] tensorflow: global_step/sec: 3.08642\n",
"INFO:tensorflow:epoch = 26.197916666666664, learning_rate = 0.0009999999, loss = 0.00017174613, step = 2515 (5.489 sec)\n",
"2021-12-30 09:24:30,889 [INFO] tensorflow: epoch = 26.197916666666664, learning_rate = 0.0009999999, loss = 0.00017174613, step = 2515 (5.489 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10502\n",
"2021-12-30 09:24:32,490 [INFO] tensorflow: global_step/sec: 3.10502\n",
"2021-12-30 09:24:33,808 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.736\n",
"INFO:tensorflow:global_step/sec: 3.02264\n",
"2021-12-30 09:24:35,467 [INFO] tensorflow: global_step/sec: 3.02264\n",
"INFO:tensorflow:epoch = 26.375, learning_rate = 0.0009999999, loss = 0.00026308306, step = 2532 (5.519 sec)\n",
"2021-12-30 09:24:36,408 [INFO] tensorflow: epoch = 26.375, learning_rate = 0.0009999999, loss = 0.00026308306, step = 2532 (5.519 sec)\n",
"INFO:tensorflow:global_step/sec: 3.13587\n",
"2021-12-30 09:24:38,337 [INFO] tensorflow: global_step/sec: 3.13587\n",
"INFO:tensorflow:global_step/sec: 3.07517\n",
"2021-12-30 09:24:41,264 [INFO] tensorflow: global_step/sec: 3.07517\n",
"INFO:tensorflow:epoch = 26.552083333333332, learning_rate = 0.0009999999, loss = 0.00024978447, step = 2549 (5.479 sec)\n",
"2021-12-30 09:24:41,888 [INFO] tensorflow: epoch = 26.552083333333332, learning_rate = 0.0009999999, loss = 0.00024978447, step = 2549 (5.479 sec)\n",
"2021-12-30 09:24:41,888 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.754\n",
"INFO:tensorflow:global_step/sec: 3.11653\n",
"2021-12-30 09:24:44,152 [INFO] tensorflow: global_step/sec: 3.11653\n",
"INFO:tensorflow:global_step/sec: 3.0352\n",
"2021-12-30 09:24:47,117 [INFO] tensorflow: global_step/sec: 3.0352\n",
"INFO:tensorflow:epoch = 26.729166666666664, learning_rate = 0.0009999999, loss = 0.00022606281, step = 2566 (5.559 sec)\n",
"2021-12-30 09:24:47,447 [INFO] tensorflow: epoch = 26.729166666666664, learning_rate = 0.0009999999, loss = 0.00022606281, step = 2566 (5.559 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10221\n",
"2021-12-30 09:24:50,018 [INFO] tensorflow: global_step/sec: 3.10221\n",
"2021-12-30 09:24:50,019 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.598\n",
"INFO:tensorflow:epoch = 26.90625, learning_rate = 0.0009999999, loss = 0.0002169539, step = 2583 (5.463 sec)\n",
"2021-12-30 09:24:52,910 [INFO] tensorflow: epoch = 26.90625, learning_rate = 0.0009999999, loss = 0.0002169539, step = 2583 (5.463 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11176\n",
"2021-12-30 09:24:52,910 [INFO] tensorflow: global_step/sec: 3.11176\n",
"INFO:tensorflow:global_step/sec: 3.10631\n",
"2021-12-30 09:24:55,808 [INFO] tensorflow: global_step/sec: 3.10631\n",
"2021-12-30 09:24:55,809 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 27/120: loss: 0.00023 learning rate: 0.00100 Time taken: 0:00:31.062483 ETA: 0:48:08.810926\n",
"2021-12-30 09:24:58,089 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.784\n",
"INFO:tensorflow:epoch = 27.083333333333332, learning_rate = 0.0009999999, loss = 0.00020600448, step = 2600 (5.497 sec)\n",
"2021-12-30 09:24:58,406 [INFO] tensorflow: epoch = 27.083333333333332, learning_rate = 0.0009999999, loss = 0.00020600448, step = 2600 (5.497 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10773\n",
"2021-12-30 09:24:58,704 [INFO] tensorflow: global_step/sec: 3.10773\n",
"INFO:tensorflow:global_step/sec: 3.07771\n",
"2021-12-30 09:25:01,628 [INFO] tensorflow: global_step/sec: 3.07771\n",
"INFO:tensorflow:epoch = 27.260416666666664, learning_rate = 0.0009999999, loss = 0.00020237922, step = 2617 (5.515 sec)\n",
"2021-12-30 09:25:03,921 [INFO] tensorflow: epoch = 27.260416666666664, learning_rate = 0.0009999999, loss = 0.00020237922, step = 2617 (5.515 sec)\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO:tensorflow:global_step/sec: 3.0688\n",
"2021-12-30 09:25:04,561 [INFO] tensorflow: global_step/sec: 3.0688\n",
"2021-12-30 09:25:06,153 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.801\n",
"INFO:tensorflow:global_step/sec: 3.11328\n",
"2021-12-30 09:25:07,452 [INFO] tensorflow: global_step/sec: 3.11328\n",
"INFO:tensorflow:epoch = 27.4375, learning_rate = 0.0009999999, loss = 0.00020225496, step = 2634 (5.469 sec)\n",
"2021-12-30 09:25:09,390 [INFO] tensorflow: epoch = 27.4375, learning_rate = 0.0009999999, loss = 0.00020225496, step = 2634 (5.469 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08255\n",
"2021-12-30 09:25:10,371 [INFO] tensorflow: global_step/sec: 3.08255\n",
"INFO:tensorflow:global_step/sec: 3.12082\n",
"2021-12-30 09:25:13,255 [INFO] tensorflow: global_step/sec: 3.12082\n",
"2021-12-30 09:25:14,261 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.668\n",
"INFO:tensorflow:epoch = 27.614583333333332, learning_rate = 0.0009999999, loss = 0.00019464636, step = 2651 (5.512 sec)\n",
"2021-12-30 09:25:14,902 [INFO] tensorflow: epoch = 27.614583333333332, learning_rate = 0.0009999999, loss = 0.00019464636, step = 2651 (5.512 sec)\n",
"INFO:tensorflow:global_step/sec: 3.06061\n",
"2021-12-30 09:25:16,196 [INFO] tensorflow: global_step/sec: 3.06061\n",
"INFO:tensorflow:global_step/sec: 3.10351\n",
"2021-12-30 09:25:19,096 [INFO] tensorflow: global_step/sec: 3.10351\n",
"INFO:tensorflow:epoch = 27.791666666666664, learning_rate = 0.0009999999, loss = 0.00020519846, step = 2668 (5.475 sec)\n",
"2021-12-30 09:25:20,377 [INFO] tensorflow: epoch = 27.791666666666664, learning_rate = 0.0009999999, loss = 0.00020519846, step = 2668 (5.475 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12443\n",
"2021-12-30 09:25:21,976 [INFO] tensorflow: global_step/sec: 3.12443\n",
"2021-12-30 09:25:22,296 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.893\n",
"INFO:tensorflow:global_step/sec: 3.03942\n",
"2021-12-30 09:25:24,937 [INFO] tensorflow: global_step/sec: 3.03942\n",
"INFO:tensorflow:epoch = 27.96875, learning_rate = 0.0009999999, loss = 0.00022801806, step = 2685 (5.558 sec)\n",
"2021-12-30 09:25:25,936 [INFO] tensorflow: epoch = 27.96875, learning_rate = 0.0009999999, loss = 0.00022801806, step = 2685 (5.558 sec)\n",
"2021-12-30 09:25:26,907 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 28/120: loss: 0.00022 learning rate: 0.00100 Time taken: 0:00:31.103107 ETA: 0:47:41.485864\n",
"INFO:tensorflow:global_step/sec: 3.05727\n",
"2021-12-30 09:25:27,881 [INFO] tensorflow: global_step/sec: 3.05727\n",
"2021-12-30 09:25:30,502 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.372\n",
"INFO:tensorflow:global_step/sec: 3.05643\n",
"2021-12-30 09:25:30,826 [INFO] tensorflow: global_step/sec: 3.05643\n",
"INFO:tensorflow:epoch = 28.145833333333332, learning_rate = 0.0009999999, loss = 0.0002726924, step = 2702 (5.524 sec)\n",
"2021-12-30 09:25:31,460 [INFO] tensorflow: epoch = 28.145833333333332, learning_rate = 0.0009999999, loss = 0.0002726924, step = 2702 (5.524 sec)\n",
"INFO:tensorflow:global_step/sec: 3.1096\n",
"2021-12-30 09:25:33,720 [INFO] tensorflow: global_step/sec: 3.1096\n",
"INFO:tensorflow:global_step/sec: 3.07511\n",
"2021-12-30 09:25:36,647 [INFO] tensorflow: global_step/sec: 3.07511\n",
"INFO:tensorflow:epoch = 28.322916666666664, learning_rate = 0.0009999999, loss = 0.00021864222, step = 2719 (5.517 sec)\n",
"2021-12-30 09:25:36,976 [INFO] tensorflow: epoch = 28.322916666666664, learning_rate = 0.0009999999, loss = 0.00021864222, step = 2719 (5.517 sec)\n",
"2021-12-30 09:25:38,584 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.747\n",
"INFO:tensorflow:global_step/sec: 3.08678\n",
"2021-12-30 09:25:39,562 [INFO] tensorflow: global_step/sec: 3.08678\n",
"INFO:tensorflow:epoch = 28.5, learning_rate = 0.0009999999, loss = 0.00026540094, step = 2736 (5.457 sec)\n",
"2021-12-30 09:25:42,433 [INFO] tensorflow: epoch = 28.5, learning_rate = 0.0009999999, loss = 0.00026540094, step = 2736 (5.457 sec)\n",
"INFO:tensorflow:global_step/sec: 3.13404\n",
"2021-12-30 09:25:42,434 [INFO] tensorflow: global_step/sec: 3.13404\n",
"INFO:tensorflow:global_step/sec: 3.09261\n",
"2021-12-30 09:25:45,344 [INFO] tensorflow: global_step/sec: 3.09261\n",
"2021-12-30 09:25:46,625 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.875\n",
"INFO:tensorflow:epoch = 28.677083333333332, learning_rate = 0.0009999999, loss = 0.00023510354, step = 2753 (5.529 sec)\n",
"2021-12-30 09:25:47,963 [INFO] tensorflow: epoch = 28.677083333333332, learning_rate = 0.0009999999, loss = 0.00023510354, step = 2753 (5.529 sec)\n",
"INFO:tensorflow:global_step/sec: 3.05573\n",
"2021-12-30 09:25:48,290 [INFO] tensorflow: global_step/sec: 3.05573\n",
"INFO:tensorflow:global_step/sec: 3.06439\n",
"2021-12-30 09:25:51,226 [INFO] tensorflow: global_step/sec: 3.06439\n",
"INFO:tensorflow:epoch = 28.854166666666664, learning_rate = 0.0009999999, loss = 0.00020199601, step = 2770 (5.561 sec)\n",
"2021-12-30 09:25:53,524 [INFO] tensorflow: epoch = 28.854166666666664, learning_rate = 0.0009999999, loss = 0.00020199601, step = 2770 (5.561 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07831\n",
"2021-12-30 09:25:54,150 [INFO] tensorflow: global_step/sec: 3.07831\n",
"2021-12-30 09:25:54,788 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.502\n",
"INFO:tensorflow:global_step/sec: 3.07514\n",
"2021-12-30 09:25:57,077 [INFO] tensorflow: global_step/sec: 3.07514\n",
"2021-12-30 09:25:58,052 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 29/120: loss: 0.00018 learning rate: 0.00100 Time taken: 0:00:31.132174 ETA: 0:47:13.027835\n",
"INFO:tensorflow:epoch = 29.03125, learning_rate = 0.0009999999, loss = 0.00025057, step = 2787 (5.481 sec)\n",
"2021-12-30 09:25:59,005 [INFO] tensorflow: epoch = 29.03125, learning_rate = 0.0009999999, loss = 0.00025057, step = 2787 (5.481 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12869\n",
"2021-12-30 09:25:59,953 [INFO] tensorflow: global_step/sec: 3.12869\n",
"INFO:tensorflow:global_step/sec: 3.07195\n",
"2021-12-30 09:26:02,883 [INFO] tensorflow: global_step/sec: 3.07195\n",
"2021-12-30 09:26:02,884 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.703\n",
"INFO:tensorflow:epoch = 29.208333333333332, learning_rate = 0.0009999999, loss = 0.00019148839, step = 2804 (5.519 sec)\n",
"2021-12-30 09:26:04,524 [INFO] tensorflow: epoch = 29.208333333333332, learning_rate = 0.0009999999, loss = 0.00019148839, step = 2804 (5.519 sec)\n",
"INFO:tensorflow:global_step/sec: 3.03047\n",
"2021-12-30 09:26:05,853 [INFO] tensorflow: global_step/sec: 3.03047\n",
"INFO:tensorflow:global_step/sec: 3.04308\n",
"2021-12-30 09:26:08,811 [INFO] tensorflow: global_step/sec: 3.04308\n",
"INFO:tensorflow:epoch = 29.385416666666664, learning_rate = 0.0009999999, loss = 0.00026509794, step = 2821 (5.590 sec)\n",
"2021-12-30 09:26:10,114 [INFO] tensorflow: epoch = 29.385416666666664, learning_rate = 0.0009999999, loss = 0.00026509794, step = 2821 (5.590 sec)\n",
"2021-12-30 09:26:11,073 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.424\n",
"INFO:tensorflow:global_step/sec: 3.10875\n",
"2021-12-30 09:26:11,706 [INFO] tensorflow: global_step/sec: 3.10875\n",
"INFO:tensorflow:global_step/sec: 3.0692\n",
"2021-12-30 09:26:14,638 [INFO] tensorflow: global_step/sec: 3.0692\n",
"INFO:tensorflow:epoch = 29.5625, learning_rate = 0.0009999999, loss = 0.00028035458, step = 2838 (5.508 sec)\n",
"2021-12-30 09:26:15,621 [INFO] tensorflow: epoch = 29.5625, learning_rate = 0.0009999999, loss = 0.00028035458, step = 2838 (5.508 sec)\n",
"INFO:tensorflow:global_step/sec: 3.06801\n",
"2021-12-30 09:26:17,571 [INFO] tensorflow: global_step/sec: 3.06801\n",
"2021-12-30 09:26:19,192 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.635\n",
"INFO:tensorflow:global_step/sec: 3.0826\n",
"2021-12-30 09:26:20,491 [INFO] tensorflow: global_step/sec: 3.0826\n",
"INFO:tensorflow:epoch = 29.739583333333332, learning_rate = 0.0009999999, loss = 0.00019909868, step = 2855 (5.537 sec)\n",
"2021-12-30 09:26:21,159 [INFO] tensorflow: epoch = 29.739583333333332, learning_rate = 0.0009999999, loss = 0.00019909868, step = 2855 (5.537 sec)\n",
"INFO:tensorflow:global_step/sec: 3.02725\n",
"2021-12-30 09:26:23,464 [INFO] tensorflow: global_step/sec: 3.02725\n",
"INFO:tensorflow:global_step/sec: 3.07814\n",
"2021-12-30 09:26:26,388 [INFO] tensorflow: global_step/sec: 3.07814\n",
"INFO:tensorflow:epoch = 29.916666666666664, learning_rate = 0.0009999999, loss = 0.00021463874, step = 2872 (5.537 sec)\n",
"2021-12-30 09:26:26,696 [INFO] tensorflow: epoch = 29.916666666666664, learning_rate = 0.0009999999, loss = 0.00021463874, step = 2872 (5.537 sec)\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"2021-12-30 09:26:27,351 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.512\n",
"INFO:tensorflow:Saving checkpoints for step-2880.\n",
"2021-12-30 09:26:28,954 [INFO] tensorflow: Saving checkpoints for step-2880.\n",
"2021-12-30 09:26:32,639 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 23, 0.00s/step\n",
"2021-12-30 09:26:44,331 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 23, 1.17s/step\n",
"2021-12-30 09:26:56,370 [INFO] iva.detectnet_v2.evaluation.evaluation: step 20 / 23, 1.20s/step\n",
"Matching predictions to ground truth, class 1/1.: 100%|█| 115313/115313 [00:07<00:00, 15500.79it/s]\n",
"Epoch 30/120\n",
"=========================\n",
"\n",
"Validation cost: 0.000706\n",
"Mean average_precision (in %): 24.3751\n",
"\n",
"class name average precision (in %)\n",
"------------ --------------------------\n",
"car 24.3751\n",
"\n",
"Median Inference Time: 0.017064\n",
"INFO:tensorflow:epoch = 30.0, learning_rate = 0.0009999999, loss = 0.00014131374, step = 2880 (42.947 sec)\n",
"2021-12-30 09:27:09,643 [INFO] tensorflow: epoch = 30.0, learning_rate = 0.0009999999, loss = 0.00014131374, step = 2880 (42.947 sec)\n",
"INFO:tensorflow:global_step/sec: 0.208065\n",
"2021-12-30 09:27:09,644 [INFO] tensorflow: global_step/sec: 0.208065\n",
"2021-12-30 09:27:09,644 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 30/120: loss: 0.00014 learning rate: 0.00100 Time taken: 0:01:11.589033 ETA: 1:47:23.012938\n",
"INFO:tensorflow:global_step/sec: 3.07416\n",
"2021-12-30 09:27:12,571 [INFO] tensorflow: global_step/sec: 3.07416\n",
"INFO:tensorflow:epoch = 30.177083333333332, learning_rate = 0.0009999999, loss = 0.00023352925, step = 2897 (5.549 sec)\n",
"2021-12-30 09:27:15,192 [INFO] tensorflow: epoch = 30.177083333333332, learning_rate = 0.0009999999, loss = 0.00023352925, step = 2897 (5.549 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07575\n",
"2021-12-30 09:27:15,497 [INFO] tensorflow: global_step/sec: 3.07575\n",
"2021-12-30 09:27:15,823 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 4.126\n",
"INFO:tensorflow:global_step/sec: 3.09973\n",
"2021-12-30 09:27:18,401 [INFO] tensorflow: global_step/sec: 3.09973\n",
"INFO:tensorflow:epoch = 30.354166666666664, learning_rate = 0.0009999999, loss = 0.00019757828, step = 2914 (5.495 sec)\n",
"2021-12-30 09:27:20,687 [INFO] tensorflow: epoch = 30.354166666666664, learning_rate = 0.0009999999, loss = 0.00019757828, step = 2914 (5.495 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08029\n",
"2021-12-30 09:27:21,323 [INFO] tensorflow: global_step/sec: 3.08029\n",
"2021-12-30 09:27:23,927 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.680\n",
"INFO:tensorflow:global_step/sec: 3.06368\n",
"2021-12-30 09:27:24,260 [INFO] tensorflow: global_step/sec: 3.06368\n",
"INFO:tensorflow:epoch = 30.53125, learning_rate = 0.0009999999, loss = 0.00025697678, step = 2931 (5.458 sec)\n",
"2021-12-30 09:27:26,144 [INFO] tensorflow: epoch = 30.53125, learning_rate = 0.0009999999, loss = 0.00025697678, step = 2931 (5.458 sec)\n",
"INFO:tensorflow:global_step/sec: 3.14125\n",
"2021-12-30 09:27:27,125 [INFO] tensorflow: global_step/sec: 3.14125\n",
"INFO:tensorflow:global_step/sec: 3.03668\n",
"2021-12-30 09:27:30,089 [INFO] tensorflow: global_step/sec: 3.03668\n",
"INFO:tensorflow:epoch = 30.708333333333332, learning_rate = 0.0009999999, loss = 0.00024946325, step = 2948 (5.551 sec)\n",
"2021-12-30 09:27:31,695 [INFO] tensorflow: epoch = 30.708333333333332, learning_rate = 0.0009999999, loss = 0.00024946325, step = 2948 (5.551 sec)\n",
"2021-12-30 09:27:32,012 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.738\n",
"INFO:tensorflow:global_step/sec: 3.17043\n",
"2021-12-30 09:27:32,928 [INFO] tensorflow: global_step/sec: 3.17043\n",
"INFO:tensorflow:global_step/sec: 3.10173\n",
"2021-12-30 09:27:35,829 [INFO] tensorflow: global_step/sec: 3.10173\n",
"INFO:tensorflow:epoch = 30.885416666666664, learning_rate = 0.0009999999, loss = 0.00015269626, step = 2965 (5.439 sec)\n",
"2021-12-30 09:27:37,134 [INFO] tensorflow: epoch = 30.885416666666664, learning_rate = 0.0009999999, loss = 0.00015269626, step = 2965 (5.439 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10673\n",
"2021-12-30 09:27:38,726 [INFO] tensorflow: global_step/sec: 3.10673\n",
"2021-12-30 09:27:40,025 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.961\n",
"2021-12-30 09:27:40,634 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 31/120: loss: 0.00021 learning rate: 0.00100 Time taken: 0:00:31.003367 ETA: 0:45:59.299658\n",
"INFO:tensorflow:global_step/sec: 3.15323\n",
"2021-12-30 09:27:41,581 [INFO] tensorflow: global_step/sec: 3.15323\n",
"INFO:tensorflow:epoch = 31.0625, learning_rate = 0.0009999999, loss = 0.0002627888, step = 2982 (5.391 sec)\n",
"2021-12-30 09:27:42,525 [INFO] tensorflow: epoch = 31.0625, learning_rate = 0.0009999999, loss = 0.0002627888, step = 2982 (5.391 sec)\n",
"INFO:tensorflow:global_step/sec: 3.14111\n",
"2021-12-30 09:27:44,446 [INFO] tensorflow: global_step/sec: 3.14111\n",
"INFO:tensorflow:global_step/sec: 3.11424\n",
"2021-12-30 09:27:47,336 [INFO] tensorflow: global_step/sec: 3.11424\n",
"INFO:tensorflow:epoch = 31.239583333333332, learning_rate = 0.0009999999, loss = 0.00022978173, step = 2999 (5.465 sec)\n",
"2021-12-30 09:27:47,990 [INFO] tensorflow: epoch = 31.239583333333332, learning_rate = 0.0009999999, loss = 0.00022978173, step = 2999 (5.465 sec)\n",
"2021-12-30 09:27:47,990 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.109\n",
"INFO:tensorflow:global_step/sec: 3.09226\n",
"2021-12-30 09:27:50,246 [INFO] tensorflow: global_step/sec: 3.09226\n",
"INFO:tensorflow:global_step/sec: 3.09626\n",
"2021-12-30 09:27:53,153 [INFO] tensorflow: global_step/sec: 3.09626\n",
"INFO:tensorflow:epoch = 31.416666666666664, learning_rate = 0.0009999999, loss = 0.00023965671, step = 3016 (5.488 sec)\n",
"2021-12-30 09:27:53,478 [INFO] tensorflow: epoch = 31.416666666666664, learning_rate = 0.0009999999, loss = 0.00023965671, step = 3016 (5.488 sec)\n",
"INFO:tensorflow:global_step/sec: 3.06865\n",
"2021-12-30 09:27:56,086 [INFO] tensorflow: global_step/sec: 3.06865\n",
"2021-12-30 09:27:56,087 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.704\n",
"INFO:tensorflow:epoch = 31.59375, learning_rate = 0.0009999999, loss = 0.0002688293, step = 3033 (5.512 sec)\n",
"2021-12-30 09:27:58,991 [INFO] tensorflow: epoch = 31.59375, learning_rate = 0.0009999999, loss = 0.0002688293, step = 3033 (5.512 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09751\n",
"2021-12-30 09:27:58,991 [INFO] tensorflow: global_step/sec: 3.09751\n",
"INFO:tensorflow:global_step/sec: 3.13141\n",
"2021-12-30 09:28:01,865 [INFO] tensorflow: global_step/sec: 3.13141\n",
"2021-12-30 09:28:04,141 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.833\n",
"INFO:tensorflow:epoch = 31.770833333333332, learning_rate = 0.0009999999, loss = 0.0002415931, step = 3050 (5.471 sec)\n",
"2021-12-30 09:28:04,461 [INFO] tensorflow: epoch = 31.770833333333332, learning_rate = 0.0009999999, loss = 0.0002415931, step = 3050 (5.471 sec)\n",
"INFO:tensorflow:global_step/sec: 3.0905\n",
"2021-12-30 09:28:04,778 [INFO] tensorflow: global_step/sec: 3.0905\n",
"INFO:tensorflow:global_step/sec: 3.09204\n",
"2021-12-30 09:28:07,688 [INFO] tensorflow: global_step/sec: 3.09204\n",
"INFO:tensorflow:epoch = 31.947916666666664, learning_rate = 0.0009999999, loss = 0.0002589122, step = 3067 (5.454 sec)\n",
"2021-12-30 09:28:09,915 [INFO] tensorflow: epoch = 31.947916666666664, learning_rate = 0.0009999999, loss = 0.0002589122, step = 3067 (5.454 sec)\n",
"INFO:tensorflow:global_step/sec: 3.13906\n",
"2021-12-30 09:28:10,555 [INFO] tensorflow: global_step/sec: 3.13906\n",
"2021-12-30 09:28:11,534 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 32/120: loss: 0.00022 learning rate: 0.00100 Time taken: 0:00:30.903376 ETA: 0:45:19.497097\n",
"2021-12-30 09:28:12,182 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.872\n",
"INFO:tensorflow:global_step/sec: 3.09171\n",
"2021-12-30 09:28:13,466 [INFO] tensorflow: global_step/sec: 3.09171\n",
"INFO:tensorflow:epoch = 32.125, learning_rate = 0.0009999999, loss = 0.00019548139, step = 3084 (5.526 sec)\n",
"2021-12-30 09:28:15,440 [INFO] tensorflow: epoch = 32.125, learning_rate = 0.0009999999, loss = 0.00019548139, step = 3084 (5.526 sec)\n",
"INFO:tensorflow:global_step/sec: 3.05925\n",
"2021-12-30 09:28:16,408 [INFO] tensorflow: global_step/sec: 3.05925\n",
"INFO:tensorflow:global_step/sec: 3.11002\n",
"2021-12-30 09:28:19,302 [INFO] tensorflow: global_step/sec: 3.11002\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"2021-12-30 09:28:20,273 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.720\n",
"INFO:tensorflow:epoch = 32.30208333333333, learning_rate = 0.0009999999, loss = 0.00028350917, step = 3101 (5.490 sec)\n",
"2021-12-30 09:28:20,931 [INFO] tensorflow: epoch = 32.30208333333333, learning_rate = 0.0009999999, loss = 0.00028350917, step = 3101 (5.490 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09948\n",
"2021-12-30 09:28:22,206 [INFO] tensorflow: global_step/sec: 3.09948\n",
"INFO:tensorflow:global_step/sec: 3.02581\n",
"2021-12-30 09:28:25,180 [INFO] tensorflow: global_step/sec: 3.02581\n",
"INFO:tensorflow:epoch = 32.479166666666664, learning_rate = 0.0009999999, loss = 0.00022941935, step = 3118 (5.557 sec)\n",
"2021-12-30 09:28:26,488 [INFO] tensorflow: epoch = 32.479166666666664, learning_rate = 0.0009999999, loss = 0.00022941935, step = 3118 (5.557 sec)\n",
"INFO:tensorflow:global_step/sec: 3.06921\n",
"2021-12-30 09:28:28,113 [INFO] tensorflow: global_step/sec: 3.06921\n",
"2021-12-30 09:28:28,442 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.485\n",
"INFO:tensorflow:global_step/sec: 3.10484\n",
"2021-12-30 09:28:31,011 [INFO] tensorflow: global_step/sec: 3.10484\n",
"INFO:tensorflow:epoch = 32.65625, learning_rate = 0.0009999999, loss = 0.00024051953, step = 3135 (5.481 sec)\n",
"2021-12-30 09:28:31,969 [INFO] tensorflow: epoch = 32.65625, learning_rate = 0.0009999999, loss = 0.00024051953, step = 3135 (5.481 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10434\n",
"2021-12-30 09:28:33,911 [INFO] tensorflow: global_step/sec: 3.10434\n",
"2021-12-30 09:28:36,526 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.739\n",
"INFO:tensorflow:global_step/sec: 3.05004\n",
"2021-12-30 09:28:36,861 [INFO] tensorflow: global_step/sec: 3.05004\n",
"INFO:tensorflow:epoch = 32.83333333333333, learning_rate = 0.0009999999, loss = 0.00021725355, step = 3152 (5.550 sec)\n",
"2021-12-30 09:28:37,519 [INFO] tensorflow: epoch = 32.83333333333333, learning_rate = 0.0009999999, loss = 0.00021725355, step = 3152 (5.550 sec)\n",
"INFO:tensorflow:global_step/sec: 3.04211\n",
"2021-12-30 09:28:39,820 [INFO] tensorflow: global_step/sec: 3.04211\n",
"INFO:tensorflow:global_step/sec: 3.09068\n",
"2021-12-30 09:28:42,732 [INFO] tensorflow: global_step/sec: 3.09068\n",
"2021-12-30 09:28:42,733 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 33/120: loss: 0.00025 learning rate: 0.00100 Time taken: 0:00:31.183628 ETA: 0:45:12.975602\n",
"INFO:tensorflow:epoch = 33.010416666666664, learning_rate = 0.0009999999, loss = 0.00023190457, step = 3169 (5.551 sec)\n",
"2021-12-30 09:28:43,069 [INFO] tensorflow: epoch = 33.010416666666664, learning_rate = 0.0009999999, loss = 0.00023190457, step = 3169 (5.551 sec)\n",
"2021-12-30 09:28:44,722 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.407\n",
"INFO:tensorflow:global_step/sec: 3.05832\n",
"2021-12-30 09:28:45,675 [INFO] tensorflow: global_step/sec: 3.05832\n",
"INFO:tensorflow:epoch = 33.1875, learning_rate = 0.0009999999, loss = 0.00040032167, step = 3186 (5.543 sec)\n",
"2021-12-30 09:28:48,612 [INFO] tensorflow: epoch = 33.1875, learning_rate = 0.0009999999, loss = 0.00040032167, step = 3186 (5.543 sec)\n",
"INFO:tensorflow:global_step/sec: 3.06315\n",
"2021-12-30 09:28:48,613 [INFO] tensorflow: global_step/sec: 3.06315\n",
"INFO:tensorflow:global_step/sec: 3.16781\n",
"2021-12-30 09:28:51,454 [INFO] tensorflow: global_step/sec: 3.16781\n",
"2021-12-30 09:28:52,729 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.977\n",
"INFO:tensorflow:epoch = 33.36458333333333, learning_rate = 0.0009999999, loss = 0.00024281528, step = 3203 (5.412 sec)\n",
"2021-12-30 09:28:54,024 [INFO] tensorflow: epoch = 33.36458333333333, learning_rate = 0.0009999999, loss = 0.00024281528, step = 3203 (5.412 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09139\n",
"2021-12-30 09:28:54,365 [INFO] tensorflow: global_step/sec: 3.09139\n",
"INFO:tensorflow:global_step/sec: 3.10879\n",
"2021-12-30 09:28:57,260 [INFO] tensorflow: global_step/sec: 3.10879\n",
"INFO:tensorflow:epoch = 33.541666666666664, learning_rate = 0.0009999999, loss = 0.00023487242, step = 3220 (5.563 sec)\n",
"2021-12-30 09:28:59,587 [INFO] tensorflow: epoch = 33.541666666666664, learning_rate = 0.0009999999, loss = 0.00023487242, step = 3220 (5.563 sec)\n",
"INFO:tensorflow:global_step/sec: 3.04211\n",
"2021-12-30 09:29:00,219 [INFO] tensorflow: global_step/sec: 3.04211\n",
"2021-12-30 09:29:00,857 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.607\n",
"INFO:tensorflow:global_step/sec: 3.09684\n",
"2021-12-30 09:29:03,125 [INFO] tensorflow: global_step/sec: 3.09684\n",
"INFO:tensorflow:epoch = 33.71875, learning_rate = 0.0009999999, loss = 0.00023589985, step = 3237 (5.454 sec)\n",
"2021-12-30 09:29:05,042 [INFO] tensorflow: epoch = 33.71875, learning_rate = 0.0009999999, loss = 0.00023589985, step = 3237 (5.454 sec)\n",
"INFO:tensorflow:global_step/sec: 3.14455\n",
"2021-12-30 09:29:05,987 [INFO] tensorflow: global_step/sec: 3.14455\n",
"INFO:tensorflow:global_step/sec: 3.10952\n",
"2021-12-30 09:29:08,881 [INFO] tensorflow: global_step/sec: 3.10952\n",
"2021-12-30 09:29:08,882 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.924\n",
"INFO:tensorflow:epoch = 33.89583333333333, learning_rate = 0.0009999999, loss = 0.00030039815, step = 3254 (5.472 sec)\n",
"2021-12-30 09:29:10,514 [INFO] tensorflow: epoch = 33.89583333333333, learning_rate = 0.0009999999, loss = 0.00030039815, step = 3254 (5.472 sec)\n",
"INFO:tensorflow:global_step/sec: 3.04019\n",
"2021-12-30 09:29:11,842 [INFO] tensorflow: global_step/sec: 3.04019\n",
"2021-12-30 09:29:13,738 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 34/120: loss: 0.00027 learning rate: 0.00100 Time taken: 0:00:31.021238 ETA: 0:44:27.826435\n",
"INFO:tensorflow:global_step/sec: 3.11219\n",
"2021-12-30 09:29:14,733 [INFO] tensorflow: global_step/sec: 3.11219\n",
"INFO:tensorflow:epoch = 34.072916666666664, learning_rate = 0.0009999999, loss = 0.0002140904, step = 3271 (5.538 sec)\n",
"2021-12-30 09:29:16,052 [INFO] tensorflow: epoch = 34.072916666666664, learning_rate = 0.0009999999, loss = 0.0002140904, step = 3271 (5.538 sec)\n",
"2021-12-30 09:29:16,982 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.690\n",
"INFO:tensorflow:global_step/sec: 3.09416\n",
"2021-12-30 09:29:17,642 [INFO] tensorflow: global_step/sec: 3.09416\n",
"INFO:tensorflow:global_step/sec: 3.09046\n",
"2021-12-30 09:29:20,554 [INFO] tensorflow: global_step/sec: 3.09046\n",
"INFO:tensorflow:epoch = 34.25, learning_rate = 0.0009999999, loss = 0.00023790609, step = 3288 (5.472 sec)\n",
"2021-12-30 09:29:21,523 [INFO] tensorflow: epoch = 34.25, learning_rate = 0.0009999999, loss = 0.00023790609, step = 3288 (5.472 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07999\n",
"2021-12-30 09:29:23,476 [INFO] tensorflow: global_step/sec: 3.07999\n",
"2021-12-30 09:29:25,120 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.577\n",
"INFO:tensorflow:global_step/sec: 3.04462\n",
"2021-12-30 09:29:26,432 [INFO] tensorflow: global_step/sec: 3.04462\n",
"INFO:tensorflow:epoch = 34.42708333333333, learning_rate = 0.0009999999, loss = 0.00026173465, step = 3305 (5.560 sec)\n",
"2021-12-30 09:29:27,084 [INFO] tensorflow: epoch = 34.42708333333333, learning_rate = 0.0009999999, loss = 0.00026173465, step = 3305 (5.560 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12675\n",
"2021-12-30 09:29:29,311 [INFO] tensorflow: global_step/sec: 3.12675\n",
"INFO:tensorflow:global_step/sec: 3.21623\n",
"2021-12-30 09:29:32,109 [INFO] tensorflow: global_step/sec: 3.21623\n",
"INFO:tensorflow:epoch = 34.604166666666664, learning_rate = 0.0009999999, loss = 0.00029122015, step = 3322 (5.356 sec)\n",
"2021-12-30 09:29:32,440 [INFO] tensorflow: epoch = 34.604166666666664, learning_rate = 0.0009999999, loss = 0.00029122015, step = 3322 (5.356 sec)\n",
"2021-12-30 09:29:33,089 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.100\n",
"INFO:tensorflow:global_step/sec: 3.11465\n",
"2021-12-30 09:29:34,999 [INFO] tensorflow: global_step/sec: 3.11465\n",
"INFO:tensorflow:epoch = 34.78125, learning_rate = 0.0009999999, loss = 0.00016799706, step = 3339 (5.478 sec)\n",
"2021-12-30 09:29:37,918 [INFO] tensorflow: epoch = 34.78125, learning_rate = 0.0009999999, loss = 0.00016799706, step = 3339 (5.478 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08212\n",
"2021-12-30 09:29:37,919 [INFO] tensorflow: global_step/sec: 3.08212\n",
"INFO:tensorflow:global_step/sec: 3.09319\n",
"2021-12-30 09:29:40,828 [INFO] tensorflow: global_step/sec: 3.09319\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"2021-12-30 09:29:41,144 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.828\n",
"INFO:tensorflow:epoch = 34.95833333333333, learning_rate = 0.0009999999, loss = 0.0002521411, step = 3356 (5.474 sec)\n",
"2021-12-30 09:29:43,392 [INFO] tensorflow: epoch = 34.95833333333333, learning_rate = 0.0009999999, loss = 0.0002521411, step = 3356 (5.474 sec)\n",
"INFO:tensorflow:global_step/sec: 3.0981\n",
"2021-12-30 09:29:43,733 [INFO] tensorflow: global_step/sec: 3.0981\n",
"2021-12-30 09:29:44,751 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 35/120: loss: 0.00023 learning rate: 0.00100 Time taken: 0:00:31.013570 ETA: 0:43:56.153436\n",
"INFO:tensorflow:global_step/sec: 3.03334\n",
"2021-12-30 09:29:46,700 [INFO] tensorflow: global_step/sec: 3.03334\n",
"INFO:tensorflow:epoch = 35.135416666666664, learning_rate = 0.0009999999, loss = 0.00026018888, step = 3373 (5.542 sec)\n",
"2021-12-30 09:29:48,934 [INFO] tensorflow: epoch = 35.135416666666664, learning_rate = 0.0009999999, loss = 0.00026018888, step = 3373 (5.542 sec)\n",
"2021-12-30 09:29:49,266 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.627\n",
"INFO:tensorflow:global_step/sec: 3.11848\n",
"2021-12-30 09:29:49,586 [INFO] tensorflow: global_step/sec: 3.11848\n",
"INFO:tensorflow:global_step/sec: 3.13584\n",
"2021-12-30 09:29:52,456 [INFO] tensorflow: global_step/sec: 3.13584\n",
"INFO:tensorflow:epoch = 35.3125, learning_rate = 0.0009999999, loss = 0.00021748221, step = 3390 (5.453 sec)\n",
"2021-12-30 09:29:54,387 [INFO] tensorflow: epoch = 35.3125, learning_rate = 0.0009999999, loss = 0.00021748221, step = 3390 (5.453 sec)\n",
"INFO:tensorflow:global_step/sec: 3.13133\n",
"2021-12-30 09:29:55,331 [INFO] tensorflow: global_step/sec: 3.13133\n",
"2021-12-30 09:29:57,261 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.018\n",
"INFO:tensorflow:global_step/sec: 3.12454\n",
"2021-12-30 09:29:58,211 [INFO] tensorflow: global_step/sec: 3.12454\n",
"INFO:tensorflow:epoch = 35.48958333333333, learning_rate = 0.0009999999, loss = 0.0002684542, step = 3407 (5.475 sec)\n",
"2021-12-30 09:29:59,862 [INFO] tensorflow: epoch = 35.48958333333333, learning_rate = 0.0009999999, loss = 0.0002684542, step = 3407 (5.475 sec)\n",
"INFO:tensorflow:global_step/sec: 3.04294\n",
"2021-12-30 09:30:01,169 [INFO] tensorflow: global_step/sec: 3.04294\n",
"INFO:tensorflow:global_step/sec: 3.09457\n",
"2021-12-30 09:30:04,077 [INFO] tensorflow: global_step/sec: 3.09457\n",
"INFO:tensorflow:epoch = 35.666666666666664, learning_rate = 0.0009999999, loss = 0.00024432334, step = 3424 (5.521 sec)\n",
"2021-12-30 09:30:05,383 [INFO] tensorflow: epoch = 35.666666666666664, learning_rate = 0.0009999999, loss = 0.00024432334, step = 3424 (5.521 sec)\n",
"2021-12-30 09:30:05,383 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.624\n",
"INFO:tensorflow:global_step/sec: 3.05921\n",
"2021-12-30 09:30:07,019 [INFO] tensorflow: global_step/sec: 3.05921\n",
"INFO:tensorflow:global_step/sec: 3.06456\n",
"2021-12-30 09:30:09,956 [INFO] tensorflow: global_step/sec: 3.06456\n",
"INFO:tensorflow:epoch = 35.84375, learning_rate = 0.0009999999, loss = 0.00019687116, step = 3441 (5.541 sec)\n",
"2021-12-30 09:30:10,924 [INFO] tensorflow: epoch = 35.84375, learning_rate = 0.0009999999, loss = 0.00019687116, step = 3441 (5.541 sec)\n",
"INFO:tensorflow:global_step/sec: 3.0835\n",
"2021-12-30 09:30:12,875 [INFO] tensorflow: global_step/sec: 3.0835\n",
"2021-12-30 09:30:13,515 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.595\n",
"INFO:tensorflow:global_step/sec: 3.06821\n",
"2021-12-30 09:30:15,808 [INFO] tensorflow: global_step/sec: 3.06821\n",
"2021-12-30 09:30:15,809 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 36/120: loss: 0.00024 learning rate: 0.00100 Time taken: 0:00:31.045637 ETA: 0:43:27.833519\n",
"INFO:tensorflow:epoch = 36.02083333333333, learning_rate = 0.0009999999, loss = 0.0002941037, step = 3458 (5.525 sec)\n",
"2021-12-30 09:30:16,449 [INFO] tensorflow: epoch = 36.02083333333333, learning_rate = 0.0009999999, loss = 0.0002941037, step = 3458 (5.525 sec)\n",
"INFO:tensorflow:global_step/sec: 3.06058\n",
"2021-12-30 09:30:18,748 [INFO] tensorflow: global_step/sec: 3.06058\n",
"INFO:tensorflow:global_step/sec: 3.0766\n",
"2021-12-30 09:30:21,674 [INFO] tensorflow: global_step/sec: 3.0766\n",
"2021-12-30 09:30:21,675 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.511\n",
"INFO:tensorflow:epoch = 36.197916666666664, learning_rate = 0.0009999999, loss = 0.00021702291, step = 3475 (5.537 sec)\n",
"2021-12-30 09:30:21,986 [INFO] tensorflow: epoch = 36.197916666666664, learning_rate = 0.0009999999, loss = 0.00021702291, step = 3475 (5.537 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11112\n",
"2021-12-30 09:30:24,567 [INFO] tensorflow: global_step/sec: 3.11112\n",
"INFO:tensorflow:epoch = 36.375, learning_rate = 0.0009999999, loss = 0.00027641328, step = 3492 (5.471 sec)\n",
"2021-12-30 09:30:27,457 [INFO] tensorflow: epoch = 36.375, learning_rate = 0.0009999999, loss = 0.00027641328, step = 3492 (5.471 sec)\n",
"INFO:tensorflow:global_step/sec: 3.1132\n",
"2021-12-30 09:30:27,458 [INFO] tensorflow: global_step/sec: 3.1132\n",
"2021-12-30 09:30:29,709 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.894\n",
"INFO:tensorflow:global_step/sec: 3.11066\n",
"2021-12-30 09:30:30,351 [INFO] tensorflow: global_step/sec: 3.11066\n",
"INFO:tensorflow:epoch = 36.55208333333333, learning_rate = 0.0009999999, loss = 0.00023247948, step = 3509 (5.458 sec)\n",
"2021-12-30 09:30:32,915 [INFO] tensorflow: epoch = 36.55208333333333, learning_rate = 0.0009999999, loss = 0.00023247948, step = 3509 (5.458 sec)\n",
"INFO:tensorflow:global_step/sec: 3.1134\n",
"2021-12-30 09:30:33,242 [INFO] tensorflow: global_step/sec: 3.1134\n",
"INFO:tensorflow:global_step/sec: 3.10704\n",
"2021-12-30 09:30:36,138 [INFO] tensorflow: global_step/sec: 3.10704\n",
"2021-12-30 09:30:37,761 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.839\n",
"INFO:tensorflow:epoch = 36.729166666666664, learning_rate = 0.0009999999, loss = 0.00025281054, step = 3526 (5.492 sec)\n",
"2021-12-30 09:30:38,407 [INFO] tensorflow: epoch = 36.729166666666664, learning_rate = 0.0009999999, loss = 0.00025281054, step = 3526 (5.492 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08786\n",
"2021-12-30 09:30:39,053 [INFO] tensorflow: global_step/sec: 3.08786\n",
"INFO:tensorflow:global_step/sec: 3.0846\n",
"2021-12-30 09:30:41,971 [INFO] tensorflow: global_step/sec: 3.0846\n",
"INFO:tensorflow:epoch = 36.90625, learning_rate = 0.0009999999, loss = 0.0003157978, step = 3543 (5.498 sec)\n",
"2021-12-30 09:30:43,904 [INFO] tensorflow: epoch = 36.90625, learning_rate = 0.0009999999, loss = 0.0003157978, step = 3543 (5.498 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08303\n",
"2021-12-30 09:30:44,890 [INFO] tensorflow: global_step/sec: 3.08303\n",
"2021-12-30 09:30:45,876 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.645\n",
"2021-12-30 09:30:46,856 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 37/120: loss: 0.00025 learning rate: 0.00100 Time taken: 0:00:31.054946 ETA: 0:42:57.560533\n",
"INFO:tensorflow:global_step/sec: 3.07404\n",
"2021-12-30 09:30:47,818 [INFO] tensorflow: global_step/sec: 3.07404\n",
"INFO:tensorflow:epoch = 37.08333333333333, learning_rate = 0.0009999999, loss = 0.00031789622, step = 3560 (5.507 sec)\n",
"2021-12-30 09:30:49,411 [INFO] tensorflow: epoch = 37.08333333333333, learning_rate = 0.0009999999, loss = 0.00031789622, step = 3560 (5.507 sec)\n",
"INFO:tensorflow:global_step/sec: 3.06539\n",
"2021-12-30 09:30:50,753 [INFO] tensorflow: global_step/sec: 3.06539\n",
"INFO:tensorflow:global_step/sec: 3.02985\n",
"2021-12-30 09:30:53,724 [INFO] tensorflow: global_step/sec: 3.02985\n",
"2021-12-30 09:30:54,023 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.551\n",
"INFO:tensorflow:epoch = 37.260416666666664, learning_rate = 0.0009999999, loss = 0.00022417336, step = 3577 (5.571 sec)\n",
"2021-12-30 09:30:54,982 [INFO] tensorflow: epoch = 37.260416666666664, learning_rate = 0.0009999999, loss = 0.00022417336, step = 3577 (5.571 sec)\n",
"INFO:tensorflow:global_step/sec: 3.1302\n",
"2021-12-30 09:30:56,599 [INFO] tensorflow: global_step/sec: 3.1302\n",
"INFO:tensorflow:global_step/sec: 3.1315\n",
"2021-12-30 09:30:59,473 [INFO] tensorflow: global_step/sec: 3.1315\n",
"INFO:tensorflow:epoch = 37.4375, learning_rate = 0.0009999999, loss = 0.00019636795, step = 3594 (5.440 sec)\n",
"2021-12-30 09:31:00,422 [INFO] tensorflow: epoch = 37.4375, learning_rate = 0.0009999999, loss = 0.00019636795, step = 3594 (5.440 sec)\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"2021-12-30 09:31:02,048 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.924\n",
"INFO:tensorflow:global_step/sec: 3.12133\n",
"2021-12-30 09:31:02,357 [INFO] tensorflow: global_step/sec: 3.12133\n",
"INFO:tensorflow:global_step/sec: 3.1237\n",
"2021-12-30 09:31:05,238 [INFO] tensorflow: global_step/sec: 3.1237\n",
"INFO:tensorflow:epoch = 37.61458333333333, learning_rate = 0.0009999999, loss = 0.0002096818, step = 3611 (5.468 sec)\n",
"2021-12-30 09:31:05,890 [INFO] tensorflow: epoch = 37.61458333333333, learning_rate = 0.0009999999, loss = 0.0002096818, step = 3611 (5.468 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07914\n",
"2021-12-30 09:31:08,161 [INFO] tensorflow: global_step/sec: 3.07914\n",
"2021-12-30 09:31:10,077 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.911\n",
"INFO:tensorflow:global_step/sec: 3.13718\n",
"2021-12-30 09:31:11,030 [INFO] tensorflow: global_step/sec: 3.13718\n",
"INFO:tensorflow:epoch = 37.791666666666664, learning_rate = 0.0009999999, loss = 0.00017775313, step = 3628 (5.454 sec)\n",
"2021-12-30 09:31:11,344 [INFO] tensorflow: epoch = 37.791666666666664, learning_rate = 0.0009999999, loss = 0.00017775313, step = 3628 (5.454 sec)\n",
"INFO:tensorflow:global_step/sec: 3.13234\n",
"2021-12-30 09:31:13,903 [INFO] tensorflow: global_step/sec: 3.13234\n",
"INFO:tensorflow:epoch = 37.96875, learning_rate = 0.0009999999, loss = 0.0001806503, step = 3645 (5.467 sec)\n",
"2021-12-30 09:31:16,811 [INFO] tensorflow: epoch = 37.96875, learning_rate = 0.0009999999, loss = 0.0001806503, step = 3645 (5.467 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09365\n",
"2021-12-30 09:31:16,812 [INFO] tensorflow: global_step/sec: 3.09365\n",
"2021-12-30 09:31:17,764 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 38/120: loss: 0.00019 learning rate: 0.00100 Time taken: 0:00:30.925752 ETA: 0:42:15.911638\n",
"2021-12-30 09:31:18,080 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.990\n",
"INFO:tensorflow:global_step/sec: 3.12373\n",
"2021-12-30 09:31:19,693 [INFO] tensorflow: global_step/sec: 3.12373\n",
"INFO:tensorflow:epoch = 38.14583333333333, learning_rate = 0.0009999999, loss = 0.0001628915, step = 3662 (5.441 sec)\n",
"2021-12-30 09:31:22,252 [INFO] tensorflow: epoch = 38.14583333333333, learning_rate = 0.0009999999, loss = 0.0001628915, step = 3662 (5.441 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12351\n",
"2021-12-30 09:31:22,574 [INFO] tensorflow: global_step/sec: 3.12351\n",
"INFO:tensorflow:global_step/sec: 3.05528\n",
"2021-12-30 09:31:25,520 [INFO] tensorflow: global_step/sec: 3.05528\n",
"2021-12-30 09:31:26,162 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.748\n",
"INFO:tensorflow:epoch = 38.322916666666664, learning_rate = 0.0009999999, loss = 0.00029461697, step = 3679 (5.544 sec)\n",
"2021-12-30 09:31:27,796 [INFO] tensorflow: epoch = 38.322916666666664, learning_rate = 0.0009999999, loss = 0.00029461697, step = 3679 (5.544 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08139\n",
"2021-12-30 09:31:28,441 [INFO] tensorflow: global_step/sec: 3.08139\n",
"INFO:tensorflow:global_step/sec: 3.0846\n",
"2021-12-30 09:31:31,359 [INFO] tensorflow: global_step/sec: 3.0846\n",
"INFO:tensorflow:epoch = 38.5, learning_rate = 0.0009999999, loss = 0.00031407957, step = 3696 (5.480 sec)\n",
"2021-12-30 09:31:33,276 [INFO] tensorflow: epoch = 38.5, learning_rate = 0.0009999999, loss = 0.00031407957, step = 3696 (5.480 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09192\n",
"2021-12-30 09:31:34,269 [INFO] tensorflow: global_step/sec: 3.09192\n",
"2021-12-30 09:31:34,270 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.667\n",
"INFO:tensorflow:global_step/sec: 3.06766\n",
"2021-12-30 09:31:37,203 [INFO] tensorflow: global_step/sec: 3.06766\n",
"INFO:tensorflow:epoch = 38.67708333333333, learning_rate = 0.0009999999, loss = 0.00020728525, step = 3713 (5.544 sec)\n",
"2021-12-30 09:31:38,820 [INFO] tensorflow: epoch = 38.67708333333333, learning_rate = 0.0009999999, loss = 0.00020728525, step = 3713 (5.544 sec)\n",
"INFO:tensorflow:global_step/sec: 3.0998\n",
"2021-12-30 09:31:40,107 [INFO] tensorflow: global_step/sec: 3.0998\n",
"2021-12-30 09:31:42,403 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.592\n",
"INFO:tensorflow:global_step/sec: 3.04694\n",
"2021-12-30 09:31:43,060 [INFO] tensorflow: global_step/sec: 3.04694\n",
"INFO:tensorflow:epoch = 38.854166666666664, learning_rate = 0.0009999999, loss = 0.00021449376, step = 3730 (5.560 sec)\n",
"2021-12-30 09:31:44,379 [INFO] tensorflow: epoch = 38.854166666666664, learning_rate = 0.0009999999, loss = 0.00021449376, step = 3730 (5.560 sec)\n",
"INFO:tensorflow:global_step/sec: 3.04486\n",
"2021-12-30 09:31:46,016 [INFO] tensorflow: global_step/sec: 3.04486\n",
"INFO:tensorflow:global_step/sec: 3.12569\n",
"2021-12-30 09:31:48,896 [INFO] tensorflow: global_step/sec: 3.12569\n",
"2021-12-30 09:31:48,897 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 39/120: loss: 0.00021 learning rate: 0.00100 Time taken: 0:00:31.115512 ETA: 0:42:00.356483\n",
"INFO:tensorflow:epoch = 39.03125, learning_rate = 0.0009999999, loss = 0.00022982704, step = 3747 (5.503 sec)\n",
"2021-12-30 09:31:49,882 [INFO] tensorflow: epoch = 39.03125, learning_rate = 0.0009999999, loss = 0.00022982704, step = 3747 (5.503 sec)\n",
"2021-12-30 09:31:50,545 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.566\n",
"INFO:tensorflow:global_step/sec: 3.08071\n",
"2021-12-30 09:31:51,817 [INFO] tensorflow: global_step/sec: 3.08071\n",
"INFO:tensorflow:global_step/sec: 3.11882\n",
"2021-12-30 09:31:54,703 [INFO] tensorflow: global_step/sec: 3.11882\n",
"INFO:tensorflow:epoch = 39.20833333333333, learning_rate = 0.0009999999, loss = 0.0002540529, step = 3764 (5.435 sec)\n",
"2021-12-30 09:31:55,317 [INFO] tensorflow: epoch = 39.20833333333333, learning_rate = 0.0009999999, loss = 0.0002540529, step = 3764 (5.435 sec)\n",
"INFO:tensorflow:global_step/sec: 3.1427\n",
"2021-12-30 09:31:57,567 [INFO] tensorflow: global_step/sec: 3.1427\n",
"2021-12-30 09:31:58,551 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.980\n",
"INFO:tensorflow:global_step/sec: 3.06591\n",
"2021-12-30 09:32:00,502 [INFO] tensorflow: global_step/sec: 3.06591\n",
"INFO:tensorflow:epoch = 39.385416666666664, learning_rate = 0.0009999999, loss = 0.00027751614, step = 3781 (5.503 sec)\n",
"2021-12-30 09:32:00,820 [INFO] tensorflow: epoch = 39.385416666666664, learning_rate = 0.0009999999, loss = 0.00027751614, step = 3781 (5.503 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12615\n",
"2021-12-30 09:32:03,381 [INFO] tensorflow: global_step/sec: 3.12615\n",
"INFO:tensorflow:epoch = 39.5625, learning_rate = 0.0009999999, loss = 0.00034317805, step = 3798 (5.506 sec)\n",
"2021-12-30 09:32:06,326 [INFO] tensorflow: epoch = 39.5625, learning_rate = 0.0009999999, loss = 0.00034317805, step = 3798 (5.506 sec)\n",
"INFO:tensorflow:global_step/sec: 3.05494\n",
"2021-12-30 09:32:06,327 [INFO] tensorflow: global_step/sec: 3.05494\n",
"2021-12-30 09:32:06,652 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.691\n",
"INFO:tensorflow:global_step/sec: 3.08944\n",
"2021-12-30 09:32:09,240 [INFO] tensorflow: global_step/sec: 3.08944\n",
"INFO:tensorflow:epoch = 39.73958333333333, learning_rate = 0.0009999999, loss = 0.00022842809, step = 3815 (5.507 sec)\n",
"2021-12-30 09:32:11,833 [INFO] tensorflow: epoch = 39.73958333333333, learning_rate = 0.0009999999, loss = 0.00022842809, step = 3815 (5.507 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10252\n",
"2021-12-30 09:32:12,141 [INFO] tensorflow: global_step/sec: 3.10252\n",
"2021-12-30 09:32:14,823 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.475\n",
"INFO:tensorflow:global_step/sec: 2.99444\n",
"2021-12-30 09:32:15,147 [INFO] tensorflow: global_step/sec: 2.99444\n",
"INFO:tensorflow:epoch = 39.916666666666664, learning_rate = 0.0009999999, loss = 0.0001889813, step = 3832 (5.567 sec)\n",
"2021-12-30 09:32:17,399 [INFO] tensorflow: epoch = 39.916666666666664, learning_rate = 0.0009999999, loss = 0.0001889813, step = 3832 (5.567 sec)\n",
"INFO:tensorflow:global_step/sec: 3.14545\n",
"2021-12-30 09:32:18,008 [INFO] tensorflow: global_step/sec: 3.14545\n",
"INFO:tensorflow:Saving checkpoints for step-3840.\n",
"2021-12-30 09:32:19,619 [INFO] tensorflow: Saving checkpoints for step-3840.\n",
"2021-12-30 09:32:23,157 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 23, 0.00s/step\n",
"2021-12-30 09:32:31,103 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 23, 0.79s/step\n",
"2021-12-30 09:32:39,427 [INFO] iva.detectnet_v2.evaluation.evaluation: step 20 / 23, 0.83s/step\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Matching predictions to ground truth, class 1/1.: 100%|█| 10649/10649 [00:00<00:00, 15841.72it/s]\n",
"Epoch 40/120\n",
"=========================\n",
"\n",
"Validation cost: 0.000258\n",
"Mean average_precision (in %): 50.1098\n",
"\n",
"class name average precision (in %)\n",
"------------ --------------------------\n",
"car 50.1098\n",
"\n",
"Median Inference Time: 0.017892\n",
"INFO:tensorflow:epoch = 40.0, learning_rate = 0.0009999999, loss = 0.00022762202, step = 3840 (25.653 sec)\n",
"2021-12-30 09:32:43,053 [INFO] tensorflow: epoch = 40.0, learning_rate = 0.0009999999, loss = 0.00022762202, step = 3840 (25.653 sec)\n",
"2021-12-30 09:32:43,053 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 40/120: loss: 0.00023 learning rate: 0.00100 Time taken: 0:00:54.146785 ETA: 1:12:11.742764\n",
"INFO:tensorflow:global_step/sec: 0.346186\n",
"2021-12-30 09:32:44,005 [INFO] tensorflow: global_step/sec: 0.346186\n",
"2021-12-30 09:32:45,915 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 6.433\n",
"INFO:tensorflow:global_step/sec: 3.12245\n",
"2021-12-30 09:32:46,888 [INFO] tensorflow: global_step/sec: 3.12245\n",
"INFO:tensorflow:epoch = 40.17708333333333, learning_rate = 0.0009999999, loss = 0.00022758974, step = 3857 (5.478 sec)\n",
"2021-12-30 09:32:48,531 [INFO] tensorflow: epoch = 40.17708333333333, learning_rate = 0.0009999999, loss = 0.00022758974, step = 3857 (5.478 sec)\n",
"INFO:tensorflow:global_step/sec: 3.0667\n",
"2021-12-30 09:32:49,823 [INFO] tensorflow: global_step/sec: 3.0667\n",
"INFO:tensorflow:global_step/sec: 3.08351\n",
"2021-12-30 09:32:52,741 [INFO] tensorflow: global_step/sec: 3.08351\n",
"INFO:tensorflow:epoch = 40.354166666666664, learning_rate = 0.0009999999, loss = 0.00027319402, step = 3874 (5.510 sec)\n",
"2021-12-30 09:32:54,040 [INFO] tensorflow: epoch = 40.354166666666664, learning_rate = 0.0009999999, loss = 0.00027319402, step = 3874 (5.510 sec)\n",
"2021-12-30 09:32:54,041 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.613\n",
"INFO:tensorflow:global_step/sec: 3.08309\n",
"2021-12-30 09:32:55,660 [INFO] tensorflow: global_step/sec: 3.08309\n",
"INFO:tensorflow:global_step/sec: 3.06476\n",
"2021-12-30 09:32:58,597 [INFO] tensorflow: global_step/sec: 3.06476\n",
"INFO:tensorflow:epoch = 40.53125, learning_rate = 0.0009999999, loss = 0.00026961544, step = 3891 (5.516 sec)\n",
"2021-12-30 09:32:59,557 [INFO] tensorflow: epoch = 40.53125, learning_rate = 0.0009999999, loss = 0.00026961544, step = 3891 (5.516 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10643\n",
"2021-12-30 09:33:01,494 [INFO] tensorflow: global_step/sec: 3.10643\n",
"2021-12-30 09:33:02,151 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.660\n",
"INFO:tensorflow:global_step/sec: 3.10475\n",
"2021-12-30 09:33:04,393 [INFO] tensorflow: global_step/sec: 3.10475\n",
"INFO:tensorflow:epoch = 40.70833333333333, learning_rate = 0.0009999999, loss = 0.00021671545, step = 3908 (5.471 sec)\n",
"2021-12-30 09:33:05,027 [INFO] tensorflow: epoch = 40.70833333333333, learning_rate = 0.0009999999, loss = 0.00021671545, step = 3908 (5.471 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08022\n",
"2021-12-30 09:33:07,315 [INFO] tensorflow: global_step/sec: 3.08022\n",
"INFO:tensorflow:global_step/sec: 3.06868\n",
"2021-12-30 09:33:10,248 [INFO] tensorflow: global_step/sec: 3.06868\n",
"2021-12-30 09:33:10,248 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.701\n",
"INFO:tensorflow:epoch = 40.885416666666664, learning_rate = 0.0009999999, loss = 0.00031232202, step = 3925 (5.539 sec)\n",
"2021-12-30 09:33:10,567 [INFO] tensorflow: epoch = 40.885416666666664, learning_rate = 0.0009999999, loss = 0.00031232202, step = 3925 (5.539 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12651\n",
"2021-12-30 09:33:13,126 [INFO] tensorflow: global_step/sec: 3.12651\n",
"2021-12-30 09:33:14,104 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 41/120: loss: 0.00019 learning rate: 0.00100 Time taken: 0:00:31.047232 ETA: 0:40:52.731302\n",
"INFO:tensorflow:epoch = 41.0625, learning_rate = 0.0009999999, loss = 0.00021240197, step = 3942 (5.504 sec)\n",
"2021-12-30 09:33:16,070 [INFO] tensorflow: epoch = 41.0625, learning_rate = 0.0009999999, loss = 0.00021240197, step = 3942 (5.504 sec)\n",
"INFO:tensorflow:global_step/sec: 3.0562\n",
"2021-12-30 09:33:16,071 [INFO] tensorflow: global_step/sec: 3.0562\n",
"2021-12-30 09:33:18,292 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.864\n",
"INFO:tensorflow:global_step/sec: 3.13621\n",
"2021-12-30 09:33:18,941 [INFO] tensorflow: global_step/sec: 3.13621\n",
"INFO:tensorflow:epoch = 41.23958333333333, learning_rate = 0.0009999999, loss = 0.0002668258, step = 3959 (5.425 sec)\n",
"2021-12-30 09:33:21,496 [INFO] tensorflow: epoch = 41.23958333333333, learning_rate = 0.0009999999, loss = 0.0002668258, step = 3959 (5.425 sec)\n",
"INFO:tensorflow:global_step/sec: 3.15175\n",
"2021-12-30 09:33:21,796 [INFO] tensorflow: global_step/sec: 3.15175\n",
"INFO:tensorflow:global_step/sec: 3.0786\n",
"2021-12-30 09:33:24,720 [INFO] tensorflow: global_step/sec: 3.0786\n",
"2021-12-30 09:33:26,353 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.812\n",
"INFO:tensorflow:epoch = 41.416666666666664, learning_rate = 0.0009999999, loss = 0.0002823872, step = 3976 (5.503 sec)\n",
"2021-12-30 09:33:26,999 [INFO] tensorflow: epoch = 41.416666666666664, learning_rate = 0.0009999999, loss = 0.0002823872, step = 3976 (5.503 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07451\n",
"2021-12-30 09:33:27,647 [INFO] tensorflow: global_step/sec: 3.07451\n",
"INFO:tensorflow:global_step/sec: 3.07738\n",
"2021-12-30 09:33:30,572 [INFO] tensorflow: global_step/sec: 3.07738\n",
"INFO:tensorflow:epoch = 41.59375, learning_rate = 0.0009999999, loss = 0.0003191193, step = 3993 (5.525 sec)\n",
"2021-12-30 09:33:32,524 [INFO] tensorflow: epoch = 41.59375, learning_rate = 0.0009999999, loss = 0.0003191193, step = 3993 (5.525 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08616\n",
"2021-12-30 09:33:33,488 [INFO] tensorflow: global_step/sec: 3.08616\n",
"2021-12-30 09:33:34,446 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.715\n",
"INFO:tensorflow:global_step/sec: 3.11804\n",
"2021-12-30 09:33:36,374 [INFO] tensorflow: global_step/sec: 3.11804\n",
"INFO:tensorflow:epoch = 41.77083333333333, learning_rate = 0.0009999999, loss = 0.00023407236, step = 4010 (5.468 sec)\n",
"2021-12-30 09:33:37,992 [INFO] tensorflow: epoch = 41.77083333333333, learning_rate = 0.0009999999, loss = 0.00023407236, step = 4010 (5.468 sec)\n",
"INFO:tensorflow:global_step/sec: 3.0607\n",
"2021-12-30 09:33:39,315 [INFO] tensorflow: global_step/sec: 3.0607\n",
"INFO:tensorflow:global_step/sec: 3.00647\n",
"2021-12-30 09:33:42,308 [INFO] tensorflow: global_step/sec: 3.00647\n",
"2021-12-30 09:33:42,631 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.435\n",
"INFO:tensorflow:epoch = 41.947916666666664, learning_rate = 0.0009999999, loss = 0.0002463283, step = 4027 (5.576 sec)\n",
"2021-12-30 09:33:43,568 [INFO] tensorflow: epoch = 41.947916666666664, learning_rate = 0.0009999999, loss = 0.0002463283, step = 4027 (5.576 sec)\n",
"INFO:tensorflow:global_step/sec: 3.13005\n",
"2021-12-30 09:33:45,184 [INFO] tensorflow: global_step/sec: 3.13005\n",
"2021-12-30 09:33:45,185 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 42/120: loss: 0.00025 learning rate: 0.00100 Time taken: 0:00:31.103383 ETA: 0:40:26.063898\n",
"INFO:tensorflow:global_step/sec: 3.09367\n",
"2021-12-30 09:33:48,093 [INFO] tensorflow: global_step/sec: 3.09367\n",
"INFO:tensorflow:epoch = 42.125, learning_rate = 0.0009999999, loss = 0.00019307976, step = 4044 (5.485 sec)\n",
"2021-12-30 09:33:49,053 [INFO] tensorflow: epoch = 42.125, learning_rate = 0.0009999999, loss = 0.00019307976, step = 4044 (5.485 sec)\n",
"2021-12-30 09:33:50,658 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.916\n",
"INFO:tensorflow:global_step/sec: 3.11057\n",
"2021-12-30 09:33:50,986 [INFO] tensorflow: global_step/sec: 3.11057\n",
"INFO:tensorflow:global_step/sec: 3.03284\n",
"2021-12-30 09:33:53,954 [INFO] tensorflow: global_step/sec: 3.03284\n",
"INFO:tensorflow:epoch = 42.30208333333333, learning_rate = 0.0009999999, loss = 0.00027837724, step = 4061 (5.525 sec)\n",
"2021-12-30 09:33:54,577 [INFO] tensorflow: epoch = 42.30208333333333, learning_rate = 0.0009999999, loss = 0.00027837724, step = 4061 (5.525 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11016\n",
"2021-12-30 09:33:56,848 [INFO] tensorflow: global_step/sec: 3.11016\n",
"2021-12-30 09:33:58,794 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.585\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO:tensorflow:global_step/sec: 3.0846\n",
"2021-12-30 09:33:59,765 [INFO] tensorflow: global_step/sec: 3.0846\n",
"INFO:tensorflow:epoch = 42.479166666666664, learning_rate = 0.0009999999, loss = 0.00016405058, step = 4078 (5.512 sec)\n",
"2021-12-30 09:34:00,089 [INFO] tensorflow: epoch = 42.479166666666664, learning_rate = 0.0009999999, loss = 0.00016405058, step = 4078 (5.512 sec)\n",
"INFO:tensorflow:global_step/sec: 3.0847\n",
"2021-12-30 09:34:02,683 [INFO] tensorflow: global_step/sec: 3.0847\n",
"INFO:tensorflow:epoch = 42.65625, learning_rate = 0.0009999999, loss = 0.00025825217, step = 4095 (5.505 sec)\n",
"2021-12-30 09:34:05,594 [INFO] tensorflow: epoch = 42.65625, learning_rate = 0.0009999999, loss = 0.00025825217, step = 4095 (5.505 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09061\n",
"2021-12-30 09:34:05,595 [INFO] tensorflow: global_step/sec: 3.09061\n",
"2021-12-30 09:34:06,899 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.676\n",
"INFO:tensorflow:global_step/sec: 3.05925\n",
"2021-12-30 09:34:08,537 [INFO] tensorflow: global_step/sec: 3.05925\n",
"INFO:tensorflow:epoch = 42.83333333333333, learning_rate = 0.0009999999, loss = 0.00023222785, step = 4112 (5.556 sec)\n",
"2021-12-30 09:34:11,150 [INFO] tensorflow: epoch = 42.83333333333333, learning_rate = 0.0009999999, loss = 0.00023222785, step = 4112 (5.556 sec)\n",
"INFO:tensorflow:global_step/sec: 3.06294\n",
"2021-12-30 09:34:11,475 [INFO] tensorflow: global_step/sec: 3.06294\n",
"INFO:tensorflow:global_step/sec: 3.15645\n",
"2021-12-30 09:34:14,327 [INFO] tensorflow: global_step/sec: 3.15645\n",
"2021-12-30 09:34:14,984 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.738\n",
"2021-12-30 09:34:16,332 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 43/120: loss: 0.00021 learning rate: 0.00100 Time taken: 0:00:31.130644 ETA: 0:39:57.059558\n",
"INFO:tensorflow:epoch = 43.010416666666664, learning_rate = 0.0009999999, loss = 0.00024233264, step = 4129 (5.525 sec)\n",
"2021-12-30 09:34:16,675 [INFO] tensorflow: epoch = 43.010416666666664, learning_rate = 0.0009999999, loss = 0.00024233264, step = 4129 (5.525 sec)\n",
"INFO:tensorflow:global_step/sec: 3.00609\n",
"2021-12-30 09:34:17,320 [INFO] tensorflow: global_step/sec: 3.00609\n",
"INFO:tensorflow:global_step/sec: 3.1138\n",
"2021-12-30 09:34:20,211 [INFO] tensorflow: global_step/sec: 3.1138\n",
"INFO:tensorflow:epoch = 43.1875, learning_rate = 0.0009999999, loss = 0.00029839188, step = 4146 (5.473 sec)\n",
"2021-12-30 09:34:22,148 [INFO] tensorflow: epoch = 43.1875, learning_rate = 0.0009999999, loss = 0.00029839188, step = 4146 (5.473 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10571\n",
"2021-12-30 09:34:23,109 [INFO] tensorflow: global_step/sec: 3.10571\n",
"2021-12-30 09:34:23,109 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.617\n",
"INFO:tensorflow:global_step/sec: 3.075\n",
"2021-12-30 09:34:26,036 [INFO] tensorflow: global_step/sec: 3.075\n",
"INFO:tensorflow:epoch = 43.36458333333333, learning_rate = 0.0009999999, loss = 0.00020983108, step = 4163 (5.521 sec)\n",
"2021-12-30 09:34:27,668 [INFO] tensorflow: epoch = 43.36458333333333, learning_rate = 0.0009999999, loss = 0.00020983108, step = 4163 (5.521 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08024\n",
"2021-12-30 09:34:28,957 [INFO] tensorflow: global_step/sec: 3.08024\n",
"2021-12-30 09:34:31,256 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.550\n",
"INFO:tensorflow:global_step/sec: 3.06713\n",
"2021-12-30 09:34:31,892 [INFO] tensorflow: global_step/sec: 3.06713\n",
"INFO:tensorflow:epoch = 43.541666666666664, learning_rate = 0.0009999999, loss = 0.00017259856, step = 4180 (5.523 sec)\n",
"2021-12-30 09:34:33,192 [INFO] tensorflow: epoch = 43.541666666666664, learning_rate = 0.0009999999, loss = 0.00017259856, step = 4180 (5.523 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09078\n",
"2021-12-30 09:34:34,804 [INFO] tensorflow: global_step/sec: 3.09078\n",
"INFO:tensorflow:global_step/sec: 3.11462\n",
"2021-12-30 09:34:37,693 [INFO] tensorflow: global_step/sec: 3.11462\n",
"INFO:tensorflow:epoch = 43.71875, learning_rate = 0.0009999999, loss = 0.00020179967, step = 4197 (5.478 sec)\n",
"2021-12-30 09:34:38,670 [INFO] tensorflow: epoch = 43.71875, learning_rate = 0.0009999999, loss = 0.00020179967, step = 4197 (5.478 sec)\n",
"2021-12-30 09:34:39,311 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.831\n",
"INFO:tensorflow:global_step/sec: 3.09142\n",
"2021-12-30 09:34:40,605 [INFO] tensorflow: global_step/sec: 3.09142\n",
"INFO:tensorflow:global_step/sec: 3.08388\n",
"2021-12-30 09:34:43,523 [INFO] tensorflow: global_step/sec: 3.08388\n",
"INFO:tensorflow:epoch = 43.89583333333333, learning_rate = 0.0009999999, loss = 0.00027891385, step = 4214 (5.492 sec)\n",
"2021-12-30 09:34:44,161 [INFO] tensorflow: epoch = 43.89583333333333, learning_rate = 0.0009999999, loss = 0.00027891385, step = 4214 (5.492 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08028\n",
"2021-12-30 09:34:46,445 [INFO] tensorflow: global_step/sec: 3.08028\n",
"2021-12-30 09:34:47,408 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 44/120: loss: 0.00024 learning rate: 0.00100 Time taken: 0:00:31.076196 ETA: 0:39:21.790893\n",
"2021-12-30 09:34:47,408 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.699\n",
"INFO:tensorflow:global_step/sec: 3.11363\n",
"2021-12-30 09:34:49,335 [INFO] tensorflow: global_step/sec: 3.11363\n",
"INFO:tensorflow:epoch = 44.072916666666664, learning_rate = 0.0009999999, loss = 0.0002582636, step = 4231 (5.507 sec)\n",
"2021-12-30 09:34:49,668 [INFO] tensorflow: epoch = 44.072916666666664, learning_rate = 0.0009999999, loss = 0.0002582636, step = 4231 (5.507 sec)\n",
"INFO:tensorflow:global_step/sec: 3.14104\n",
"2021-12-30 09:34:52,201 [INFO] tensorflow: global_step/sec: 3.14104\n",
"INFO:tensorflow:epoch = 44.25, learning_rate = 0.0009999999, loss = 0.00021697811, step = 4248 (5.435 sec)\n",
"2021-12-30 09:34:55,103 [INFO] tensorflow: epoch = 44.25, learning_rate = 0.0009999999, loss = 0.00021697811, step = 4248 (5.435 sec)\n",
"INFO:tensorflow:global_step/sec: 3.0994\n",
"2021-12-30 09:34:55,104 [INFO] tensorflow: global_step/sec: 3.0994\n",
"2021-12-30 09:34:55,435 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.919\n",
"INFO:tensorflow:global_step/sec: 3.09839\n",
"2021-12-30 09:34:58,009 [INFO] tensorflow: global_step/sec: 3.09839\n",
"INFO:tensorflow:epoch = 44.42708333333333, learning_rate = 0.0009999999, loss = 0.0002084161, step = 4265 (5.549 sec)\n",
"2021-12-30 09:35:00,653 [INFO] tensorflow: epoch = 44.42708333333333, learning_rate = 0.0009999999, loss = 0.0002084161, step = 4265 (5.549 sec)\n",
"INFO:tensorflow:global_step/sec: 3.01259\n",
"2021-12-30 09:35:00,997 [INFO] tensorflow: global_step/sec: 3.01259\n",
"2021-12-30 09:35:03,568 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.593\n",
"INFO:tensorflow:global_step/sec: 3.10662\n",
"2021-12-30 09:35:03,894 [INFO] tensorflow: global_step/sec: 3.10662\n",
"INFO:tensorflow:epoch = 44.604166666666664, learning_rate = 0.0009999999, loss = 0.00018577234, step = 4282 (5.528 sec)\n",
"2021-12-30 09:35:06,181 [INFO] tensorflow: epoch = 44.604166666666664, learning_rate = 0.0009999999, loss = 0.00018577234, step = 4282 (5.528 sec)\n",
"INFO:tensorflow:global_step/sec: 3.04703\n",
"2021-12-30 09:35:06,847 [INFO] tensorflow: global_step/sec: 3.04703\n",
"INFO:tensorflow:global_step/sec: 3.14965\n",
"2021-12-30 09:35:09,705 [INFO] tensorflow: global_step/sec: 3.14965\n",
"INFO:tensorflow:epoch = 44.78125, learning_rate = 0.0009999999, loss = 0.00018539667, step = 4299 (5.474 sec)\n",
"2021-12-30 09:35:11,655 [INFO] tensorflow: epoch = 44.78125, learning_rate = 0.0009999999, loss = 0.00018539667, step = 4299 (5.474 sec)\n",
"2021-12-30 09:35:11,655 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.729\n",
"INFO:tensorflow:global_step/sec: 3.07535\n",
"2021-12-30 09:35:12,631 [INFO] tensorflow: global_step/sec: 3.07535\n",
"INFO:tensorflow:global_step/sec: 3.09272\n",
"2021-12-30 09:35:15,541 [INFO] tensorflow: global_step/sec: 3.09272\n",
"INFO:tensorflow:epoch = 44.95833333333333, learning_rate = 0.0009999999, loss = 0.0002397768, step = 4316 (5.497 sec)\n",
"2021-12-30 09:35:17,153 [INFO] tensorflow: epoch = 44.95833333333333, learning_rate = 0.0009999999, loss = 0.0002397768, step = 4316 (5.497 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12005\n",
"2021-12-30 09:35:18,426 [INFO] tensorflow: global_step/sec: 3.12005\n",
"2021-12-30 09:35:18,427 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 45/120: loss: 0.00017 learning rate: 0.00100 Time taken: 0:00:31.016372 ETA: 0:38:46.227880\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"2021-12-30 09:35:19,729 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.773\n",
"INFO:tensorflow:global_step/sec: 3.06986\n",
"2021-12-30 09:35:21,358 [INFO] tensorflow: global_step/sec: 3.06986\n",
"INFO:tensorflow:epoch = 45.135416666666664, learning_rate = 0.0009999999, loss = 0.00019266657, step = 4333 (5.557 sec)\n",
"2021-12-30 09:35:22,710 [INFO] tensorflow: epoch = 45.135416666666664, learning_rate = 0.0009999999, loss = 0.00019266657, step = 4333 (5.557 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07105\n",
"2021-12-30 09:35:24,288 [INFO] tensorflow: global_step/sec: 3.07105\n",
"INFO:tensorflow:global_step/sec: 3.09493\n",
"2021-12-30 09:35:27,196 [INFO] tensorflow: global_step/sec: 3.09493\n",
"2021-12-30 09:35:27,826 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.700\n",
"INFO:tensorflow:epoch = 45.3125, learning_rate = 0.0009999999, loss = 0.00021266742, step = 4350 (5.449 sec)\n",
"2021-12-30 09:35:28,159 [INFO] tensorflow: epoch = 45.3125, learning_rate = 0.0009999999, loss = 0.00021266742, step = 4350 (5.449 sec)\n",
"INFO:tensorflow:global_step/sec: 3.06643\n",
"2021-12-30 09:35:30,131 [INFO] tensorflow: global_step/sec: 3.06643\n",
"INFO:tensorflow:global_step/sec: 3.07681\n",
"2021-12-30 09:35:33,056 [INFO] tensorflow: global_step/sec: 3.07681\n",
"INFO:tensorflow:epoch = 45.48958333333333, learning_rate = 0.0009999999, loss = 0.00019743232, step = 4367 (5.515 sec)\n",
"2021-12-30 09:35:33,674 [INFO] tensorflow: epoch = 45.48958333333333, learning_rate = 0.0009999999, loss = 0.00019743232, step = 4367 (5.515 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09\n",
"2021-12-30 09:35:35,969 [INFO] tensorflow: global_step/sec: 3.09\n",
"2021-12-30 09:35:35,970 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.561\n",
"INFO:tensorflow:global_step/sec: 3.11135\n",
"2021-12-30 09:35:38,861 [INFO] tensorflow: global_step/sec: 3.11135\n",
"INFO:tensorflow:epoch = 45.666666666666664, learning_rate = 0.0009999999, loss = 0.00024843818, step = 4384 (5.510 sec)\n",
"2021-12-30 09:35:39,185 [INFO] tensorflow: epoch = 45.666666666666664, learning_rate = 0.0009999999, loss = 0.00024843818, step = 4384 (5.510 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10697\n",
"2021-12-30 09:35:41,758 [INFO] tensorflow: global_step/sec: 3.10697\n",
"2021-12-30 09:35:44,016 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.857\n",
"INFO:tensorflow:epoch = 45.84375, learning_rate = 0.0009999999, loss = 0.00022437832, step = 4401 (5.484 sec)\n",
"2021-12-30 09:35:44,669 [INFO] tensorflow: epoch = 45.84375, learning_rate = 0.0009999999, loss = 0.00022437832, step = 4401 (5.484 sec)\n",
"INFO:tensorflow:global_step/sec: 3.0914\n",
"2021-12-30 09:35:44,670 [INFO] tensorflow: global_step/sec: 3.0914\n",
"INFO:tensorflow:global_step/sec: 3.04514\n",
"2021-12-30 09:35:47,625 [INFO] tensorflow: global_step/sec: 3.04514\n",
"2021-12-30 09:35:49,536 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 46/120: loss: 0.00025 learning rate: 0.00100 Time taken: 0:00:31.112762 ETA: 0:38:22.344421\n",
"INFO:tensorflow:epoch = 46.02083333333333, learning_rate = 0.0009999999, loss = 0.00022524448, step = 4418 (5.512 sec)\n",
"2021-12-30 09:35:50,181 [INFO] tensorflow: epoch = 46.02083333333333, learning_rate = 0.0009999999, loss = 0.00022524448, step = 4418 (5.512 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10525\n",
"2021-12-30 09:35:50,523 [INFO] tensorflow: global_step/sec: 3.10525\n",
"2021-12-30 09:35:52,153 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.579\n",
"INFO:tensorflow:global_step/sec: 3.05919\n",
"2021-12-30 09:35:53,465 [INFO] tensorflow: global_step/sec: 3.05919\n",
"INFO:tensorflow:epoch = 46.197916666666664, learning_rate = 0.0009999999, loss = 0.0002854688, step = 4435 (5.552 sec)\n",
"2021-12-30 09:35:55,732 [INFO] tensorflow: epoch = 46.197916666666664, learning_rate = 0.0009999999, loss = 0.0002854688, step = 4435 (5.552 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07784\n",
"2021-12-30 09:35:56,389 [INFO] tensorflow: global_step/sec: 3.07784\n",
"INFO:tensorflow:global_step/sec: 3.1042\n",
"2021-12-30 09:35:59,289 [INFO] tensorflow: global_step/sec: 3.1042\n",
"2021-12-30 09:36:00,241 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.729\n",
"INFO:tensorflow:epoch = 46.375, learning_rate = 0.0009999999, loss = 0.0002334767, step = 4452 (5.477 sec)\n",
"2021-12-30 09:36:01,209 [INFO] tensorflow: epoch = 46.375, learning_rate = 0.0009999999, loss = 0.0002334767, step = 4452 (5.477 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12374\n",
"2021-12-30 09:36:02,170 [INFO] tensorflow: global_step/sec: 3.12374\n",
"INFO:tensorflow:global_step/sec: 3.05046\n",
"2021-12-30 09:36:05,120 [INFO] tensorflow: global_step/sec: 3.05046\n",
"INFO:tensorflow:epoch = 46.55208333333333, learning_rate = 0.0009999999, loss = 0.00018884442, step = 4469 (5.532 sec)\n",
"2021-12-30 09:36:06,741 [INFO] tensorflow: epoch = 46.55208333333333, learning_rate = 0.0009999999, loss = 0.00018884442, step = 4469 (5.532 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09742\n",
"2021-12-30 09:36:08,026 [INFO] tensorflow: global_step/sec: 3.09742\n",
"2021-12-30 09:36:08,354 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.654\n",
"INFO:tensorflow:global_step/sec: 3.07132\n",
"2021-12-30 09:36:10,956 [INFO] tensorflow: global_step/sec: 3.07132\n",
"INFO:tensorflow:epoch = 46.729166666666664, learning_rate = 0.0009999999, loss = 0.0002715608, step = 4486 (5.530 sec)\n",
"2021-12-30 09:36:12,272 [INFO] tensorflow: epoch = 46.729166666666664, learning_rate = 0.0009999999, loss = 0.0002715608, step = 4486 (5.530 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07803\n",
"2021-12-30 09:36:13,880 [INFO] tensorflow: global_step/sec: 3.07803\n",
"2021-12-30 09:36:16,465 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.658\n",
"INFO:tensorflow:global_step/sec: 3.09958\n",
"2021-12-30 09:36:16,784 [INFO] tensorflow: global_step/sec: 3.09958\n",
"INFO:tensorflow:epoch = 46.90625, learning_rate = 0.0009999999, loss = 0.0002451991, step = 4503 (5.495 sec)\n",
"2021-12-30 09:36:17,767 [INFO] tensorflow: epoch = 46.90625, learning_rate = 0.0009999999, loss = 0.0002451991, step = 4503 (5.495 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09263\n",
"2021-12-30 09:36:19,694 [INFO] tensorflow: global_step/sec: 3.09263\n",
"2021-12-30 09:36:20,666 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 47/120: loss: 0.00023 learning rate: 0.00100 Time taken: 0:00:31.140258 ETA: 0:37:53.238805\n",
"INFO:tensorflow:global_step/sec: 3.11352\n",
"2021-12-30 09:36:22,585 [INFO] tensorflow: global_step/sec: 3.11352\n",
"INFO:tensorflow:epoch = 47.08333333333333, learning_rate = 0.0009999999, loss = 0.00019389059, step = 4520 (5.440 sec)\n",
"2021-12-30 09:36:23,206 [INFO] tensorflow: epoch = 47.08333333333333, learning_rate = 0.0009999999, loss = 0.00019389059, step = 4520 (5.440 sec)\n",
"2021-12-30 09:36:24,492 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.917\n",
"INFO:tensorflow:global_step/sec: 3.10155\n",
"2021-12-30 09:36:25,486 [INFO] tensorflow: global_step/sec: 3.10155\n",
"INFO:tensorflow:global_step/sec: 3.08127\n",
"2021-12-30 09:36:28,407 [INFO] tensorflow: global_step/sec: 3.08127\n",
"INFO:tensorflow:epoch = 47.260416666666664, learning_rate = 0.0009999999, loss = 0.00022550585, step = 4537 (5.528 sec)\n",
"2021-12-30 09:36:28,734 [INFO] tensorflow: epoch = 47.260416666666664, learning_rate = 0.0009999999, loss = 0.00022550585, step = 4537 (5.528 sec)\n",
"INFO:tensorflow:global_step/sec: 3.02137\n",
"2021-12-30 09:36:31,386 [INFO] tensorflow: global_step/sec: 3.02137\n",
"2021-12-30 09:36:32,720 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.307\n",
"INFO:tensorflow:epoch = 47.4375, learning_rate = 0.0009999999, loss = 0.0001965183, step = 4554 (5.620 sec)\n",
"2021-12-30 09:36:34,354 [INFO] tensorflow: epoch = 47.4375, learning_rate = 0.0009999999, loss = 0.0001965183, step = 4554 (5.620 sec)\n",
"INFO:tensorflow:global_step/sec: 3.03154\n",
"2021-12-30 09:36:34,355 [INFO] tensorflow: global_step/sec: 3.03154\n",
"INFO:tensorflow:global_step/sec: 3.0407\n",
"2021-12-30 09:36:37,315 [INFO] tensorflow: global_step/sec: 3.0407\n",
"INFO:tensorflow:epoch = 47.61458333333333, learning_rate = 0.0009999999, loss = 0.00023891173, step = 4571 (5.525 sec)\n",
"2021-12-30 09:36:39,879 [INFO] tensorflow: epoch = 47.61458333333333, learning_rate = 0.0009999999, loss = 0.00023891173, step = 4571 (5.525 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11692\n",
"2021-12-30 09:36:40,202 [INFO] tensorflow: global_step/sec: 3.11692\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"2021-12-30 09:36:40,849 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.604\n",
"INFO:tensorflow:global_step/sec: 3.03792\n",
"2021-12-30 09:36:43,165 [INFO] tensorflow: global_step/sec: 3.03792\n",
"INFO:tensorflow:epoch = 47.791666666666664, learning_rate = 0.0009999999, loss = 0.00025669375, step = 4588 (5.536 sec)\n",
"2021-12-30 09:36:45,415 [INFO] tensorflow: epoch = 47.791666666666664, learning_rate = 0.0009999999, loss = 0.00025669375, step = 4588 (5.536 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08203\n",
"2021-12-30 09:36:46,085 [INFO] tensorflow: global_step/sec: 3.08203\n",
"INFO:tensorflow:global_step/sec: 3.11085\n",
"2021-12-30 09:36:48,978 [INFO] tensorflow: global_step/sec: 3.11085\n",
"2021-12-30 09:36:48,979 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.601\n",
"INFO:tensorflow:epoch = 47.96875, learning_rate = 0.0009999999, loss = 0.00027893117, step = 4605 (5.507 sec)\n",
"2021-12-30 09:36:50,922 [INFO] tensorflow: epoch = 47.96875, learning_rate = 0.0009999999, loss = 0.00027893117, step = 4605 (5.507 sec)\n",
"INFO:tensorflow:global_step/sec: 3.06944\n",
"2021-12-30 09:36:51,910 [INFO] tensorflow: global_step/sec: 3.06944\n",
"2021-12-30 09:36:51,911 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 48/120: loss: 0.00021 learning rate: 0.00100 Time taken: 0:00:31.233389 ETA: 0:37:28.804001\n",
"INFO:tensorflow:global_step/sec: 3.11502\n",
"2021-12-30 09:36:54,799 [INFO] tensorflow: global_step/sec: 3.11502\n",
"INFO:tensorflow:epoch = 48.14583333333333, learning_rate = 0.0009999999, loss = 0.00021013248, step = 4622 (5.497 sec)\n",
"2021-12-30 09:36:56,419 [INFO] tensorflow: epoch = 48.14583333333333, learning_rate = 0.0009999999, loss = 0.00021013248, step = 4622 (5.497 sec)\n",
"2021-12-30 09:36:57,045 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.796\n",
"INFO:tensorflow:global_step/sec: 3.10234\n",
"2021-12-30 09:36:57,700 [INFO] tensorflow: global_step/sec: 3.10234\n",
"INFO:tensorflow:global_step/sec: 3.12208\n",
"2021-12-30 09:37:00,583 [INFO] tensorflow: global_step/sec: 3.12208\n",
"INFO:tensorflow:epoch = 48.322916666666664, learning_rate = 0.0009999999, loss = 0.0002358364, step = 4639 (5.489 sec)\n",
"2021-12-30 09:37:01,907 [INFO] tensorflow: epoch = 48.322916666666664, learning_rate = 0.0009999999, loss = 0.0002358364, step = 4639 (5.489 sec)\n",
"INFO:tensorflow:global_step/sec: 3.05178\n",
"2021-12-30 09:37:03,532 [INFO] tensorflow: global_step/sec: 3.05178\n",
"2021-12-30 09:37:05,173 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.605\n",
"INFO:tensorflow:global_step/sec: 3.07685\n",
"2021-12-30 09:37:06,457 [INFO] tensorflow: global_step/sec: 3.07685\n",
"INFO:tensorflow:epoch = 48.5, learning_rate = 0.0009999999, loss = 0.00020498177, step = 4656 (5.533 sec)\n",
"2021-12-30 09:37:07,440 [INFO] tensorflow: epoch = 48.5, learning_rate = 0.0009999999, loss = 0.00020498177, step = 4656 (5.533 sec)\n",
"INFO:tensorflow:global_step/sec: 3.06654\n",
"2021-12-30 09:37:09,392 [INFO] tensorflow: global_step/sec: 3.06654\n",
"INFO:tensorflow:global_step/sec: 3.09582\n",
"2021-12-30 09:37:12,299 [INFO] tensorflow: global_step/sec: 3.09582\n",
"INFO:tensorflow:epoch = 48.67708333333333, learning_rate = 0.0009999999, loss = 0.0002189246, step = 4673 (5.521 sec)\n",
"2021-12-30 09:37:12,961 [INFO] tensorflow: epoch = 48.67708333333333, learning_rate = 0.0009999999, loss = 0.0002189246, step = 4673 (5.521 sec)\n",
"2021-12-30 09:37:13,283 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.664\n",
"INFO:tensorflow:global_step/sec: 3.07522\n",
"2021-12-30 09:37:15,226 [INFO] tensorflow: global_step/sec: 3.07522\n",
"INFO:tensorflow:global_step/sec: 3.13125\n",
"2021-12-30 09:37:18,100 [INFO] tensorflow: global_step/sec: 3.13125\n",
"INFO:tensorflow:epoch = 48.854166666666664, learning_rate = 0.0009999999, loss = 0.00025429582, step = 4690 (5.458 sec)\n",
"2021-12-30 09:37:18,420 [INFO] tensorflow: epoch = 48.854166666666664, learning_rate = 0.0009999999, loss = 0.00025429582, step = 4690 (5.458 sec)\n",
"INFO:tensorflow:global_step/sec: 3.15708\n",
"2021-12-30 09:37:20,951 [INFO] tensorflow: global_step/sec: 3.15708\n",
"2021-12-30 09:37:21,273 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.029\n",
"2021-12-30 09:37:22,918 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 49/120: loss: 0.00021 learning rate: 0.00100 Time taken: 0:00:30.999862 ETA: 0:36:40.990182\n",
"INFO:tensorflow:epoch = 49.03125, learning_rate = 0.0009999999, loss = 0.00029787954, step = 4707 (5.477 sec)\n",
"2021-12-30 09:37:23,896 [INFO] tensorflow: epoch = 49.03125, learning_rate = 0.0009999999, loss = 0.00029787954, step = 4707 (5.477 sec)\n",
"INFO:tensorflow:global_step/sec: 3.05481\n",
"2021-12-30 09:37:23,897 [INFO] tensorflow: global_step/sec: 3.05481\n",
"INFO:tensorflow:global_step/sec: 3.06844\n",
"2021-12-30 09:37:26,830 [INFO] tensorflow: global_step/sec: 3.06844\n",
"INFO:tensorflow:epoch = 49.20833333333333, learning_rate = 0.0009999999, loss = 0.00021951024, step = 4724 (5.481 sec)\n",
"2021-12-30 09:37:29,377 [INFO] tensorflow: epoch = 49.20833333333333, learning_rate = 0.0009999999, loss = 0.00021951024, step = 4724 (5.481 sec)\n",
"2021-12-30 09:37:29,377 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.682\n",
"INFO:tensorflow:global_step/sec: 3.1422\n",
"2021-12-30 09:37:29,694 [INFO] tensorflow: global_step/sec: 3.1422\n",
"INFO:tensorflow:global_step/sec: 3.10066\n",
"2021-12-30 09:37:32,597 [INFO] tensorflow: global_step/sec: 3.10066\n",
"INFO:tensorflow:epoch = 49.385416666666664, learning_rate = 0.0009999999, loss = 0.0003749259, step = 4741 (5.466 sec)\n",
"2021-12-30 09:37:34,843 [INFO] tensorflow: epoch = 49.385416666666664, learning_rate = 0.0009999999, loss = 0.0003749259, step = 4741 (5.466 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12325\n",
"2021-12-30 09:37:35,479 [INFO] tensorflow: global_step/sec: 3.12325\n",
"2021-12-30 09:37:37,469 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.716\n",
"INFO:tensorflow:global_step/sec: 3.05463\n",
"2021-12-30 09:37:38,425 [INFO] tensorflow: global_step/sec: 3.05463\n",
"INFO:tensorflow:epoch = 49.5625, learning_rate = 0.0009999999, loss = 0.00029053763, step = 4758 (5.528 sec)\n",
"2021-12-30 09:37:40,370 [INFO] tensorflow: epoch = 49.5625, learning_rate = 0.0009999999, loss = 0.00029053763, step = 4758 (5.528 sec)\n",
"INFO:tensorflow:global_step/sec: 3.05454\n",
"2021-12-30 09:37:41,371 [INFO] tensorflow: global_step/sec: 3.05454\n",
"INFO:tensorflow:global_step/sec: 3.08815\n",
"2021-12-30 09:37:44,286 [INFO] tensorflow: global_step/sec: 3.08815\n",
"2021-12-30 09:37:45,576 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.671\n",
"INFO:tensorflow:epoch = 49.73958333333333, learning_rate = 0.0009999999, loss = 0.00018478357, step = 4775 (5.546 sec)\n",
"2021-12-30 09:37:45,917 [INFO] tensorflow: epoch = 49.73958333333333, learning_rate = 0.0009999999, loss = 0.00018478357, step = 4775 (5.546 sec)\n",
"INFO:tensorflow:global_step/sec: 3.06746\n",
"2021-12-30 09:37:47,220 [INFO] tensorflow: global_step/sec: 3.06746\n",
"INFO:tensorflow:global_step/sec: 3.1369\n",
"2021-12-30 09:37:50,089 [INFO] tensorflow: global_step/sec: 3.1369\n",
"INFO:tensorflow:epoch = 49.916666666666664, learning_rate = 0.0009999999, loss = 0.00021013418, step = 4792 (5.449 sec)\n",
"2021-12-30 09:37:51,365 [INFO] tensorflow: epoch = 49.916666666666664, learning_rate = 0.0009999999, loss = 0.00021013418, step = 4792 (5.449 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09854\n",
"2021-12-30 09:37:52,993 [INFO] tensorflow: global_step/sec: 3.09854\n",
"INFO:tensorflow:Saving checkpoints for step-4800.\n",
"2021-12-30 09:37:53,659 [INFO] tensorflow: Saving checkpoints for step-4800.\n",
"WARNING:tensorflow:Ignoring: /tmp/tmpxkzzn70p; No such file or directory\n",
"2021-12-30 09:37:53,806 [WARNING] tensorflow: Ignoring: /tmp/tmpxkzzn70p; No such file or directory\n",
"2021-12-30 09:37:57,228 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 23, 0.00s/step\n",
"2021-12-30 09:38:00,927 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 23, 0.37s/step\n",
"2021-12-30 09:38:04,475 [INFO] iva.detectnet_v2.evaluation.evaluation: step 20 / 23, 0.35s/step\n",
"Matching predictions to ground truth, class 1/1.: 100%|█| 3298/3298 [00:00<00:00, 16035.61it/s]\n",
"Epoch 50/120\n",
"=========================\n",
"\n",
"Validation cost: 0.000226\n",
"Mean average_precision (in %): 72.3448\n",
"\n",
"class name average precision (in %)\n",
"------------ --------------------------\n",
"car 72.3448\n",
"\n",
"Median Inference Time: 0.015533\n",
"2021-12-30 09:38:05,886 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 9.847\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO:tensorflow:epoch = 50.0, learning_rate = 0.0009999999, loss = 0.00016254658, step = 4800 (14.833 sec)\n",
"2021-12-30 09:38:06,198 [INFO] tensorflow: epoch = 50.0, learning_rate = 0.0009999999, loss = 0.00016254658, step = 4800 (14.833 sec)\n",
"2021-12-30 09:38:06,199 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 50/120: loss: 0.00016 learning rate: 0.00100 Time taken: 0:00:43.293750 ETA: 0:50:30.562470\n",
"INFO:tensorflow:global_step/sec: 0.594587\n",
"2021-12-30 09:38:08,130 [INFO] tensorflow: global_step/sec: 0.594587\n",
"INFO:tensorflow:global_step/sec: 3.13967\n",
"2021-12-30 09:38:10,996 [INFO] tensorflow: global_step/sec: 3.13967\n",
"INFO:tensorflow:epoch = 50.17708333333333, learning_rate = 0.0009999999, loss = 0.00022868972, step = 4817 (5.439 sec)\n",
"2021-12-30 09:38:11,637 [INFO] tensorflow: epoch = 50.17708333333333, learning_rate = 0.0009999999, loss = 0.00022868972, step = 4817 (5.439 sec)\n",
"INFO:tensorflow:global_step/sec: 3.05274\n",
"2021-12-30 09:38:13,945 [INFO] tensorflow: global_step/sec: 3.05274\n",
"2021-12-30 09:38:13,945 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.819\n",
"INFO:tensorflow:global_step/sec: 3.04161\n",
"2021-12-30 09:38:16,904 [INFO] tensorflow: global_step/sec: 3.04161\n",
"INFO:tensorflow:epoch = 50.354166666666664, learning_rate = 0.0009999999, loss = 0.00018608244, step = 4834 (5.586 sec)\n",
"2021-12-30 09:38:17,223 [INFO] tensorflow: epoch = 50.354166666666664, learning_rate = 0.0009999999, loss = 0.00018608244, step = 4834 (5.586 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08126\n",
"2021-12-30 09:38:19,824 [INFO] tensorflow: global_step/sec: 3.08126\n",
"2021-12-30 09:38:22,082 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.580\n",
"INFO:tensorflow:epoch = 50.53125, learning_rate = 0.0009999999, loss = 0.00019877488, step = 4851 (5.509 sec)\n",
"2021-12-30 09:38:22,732 [INFO] tensorflow: epoch = 50.53125, learning_rate = 0.0009999999, loss = 0.00019877488, step = 4851 (5.509 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09475\n",
"2021-12-30 09:38:22,733 [INFO] tensorflow: global_step/sec: 3.09475\n",
"INFO:tensorflow:global_step/sec: 3.08396\n",
"2021-12-30 09:38:25,651 [INFO] tensorflow: global_step/sec: 3.08396\n",
"INFO:tensorflow:epoch = 50.70833333333333, learning_rate = 0.0009999999, loss = 0.0002610277, step = 4868 (5.502 sec)\n",
"2021-12-30 09:38:28,234 [INFO] tensorflow: epoch = 50.70833333333333, learning_rate = 0.0009999999, loss = 0.0002610277, step = 4868 (5.502 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10042\n",
"2021-12-30 09:38:28,554 [INFO] tensorflow: global_step/sec: 3.10042\n",
"2021-12-30 09:38:30,166 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.742\n",
"INFO:tensorflow:global_step/sec: 3.10118\n",
"2021-12-30 09:38:31,456 [INFO] tensorflow: global_step/sec: 3.10118\n",
"INFO:tensorflow:epoch = 50.885416666666664, learning_rate = 0.0009999999, loss = 0.0002188585, step = 4885 (5.467 sec)\n",
"2021-12-30 09:38:33,701 [INFO] tensorflow: epoch = 50.885416666666664, learning_rate = 0.0009999999, loss = 0.0002188585, step = 4885 (5.467 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09896\n",
"2021-12-30 09:38:34,360 [INFO] tensorflow: global_step/sec: 3.09896\n",
"INFO:tensorflow:global_step/sec: 3.05286\n",
"2021-12-30 09:38:37,308 [INFO] tensorflow: global_step/sec: 3.05286\n",
"2021-12-30 09:38:37,309 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 51/120: loss: 0.00020 learning rate: 0.00100 Time taken: 0:00:31.094906 ETA: 0:35:45.548537\n",
"2021-12-30 09:38:38,287 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.629\n",
"INFO:tensorflow:epoch = 51.0625, learning_rate = 0.0009999999, loss = 0.00026515053, step = 4902 (5.550 sec)\n",
"2021-12-30 09:38:39,251 [INFO] tensorflow: epoch = 51.0625, learning_rate = 0.0009999999, loss = 0.00026515053, step = 4902 (5.550 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10889\n",
"2021-12-30 09:38:40,203 [INFO] tensorflow: global_step/sec: 3.10889\n",
"INFO:tensorflow:global_step/sec: 3.14598\n",
"2021-12-30 09:38:43,064 [INFO] tensorflow: global_step/sec: 3.14598\n",
"INFO:tensorflow:epoch = 51.23958333333333, learning_rate = 0.0009999999, loss = 0.00021856508, step = 4919 (5.435 sec)\n",
"2021-12-30 09:38:44,687 [INFO] tensorflow: epoch = 51.23958333333333, learning_rate = 0.0009999999, loss = 0.00021856508, step = 4919 (5.435 sec)\n",
"INFO:tensorflow:global_step/sec: 3.05331\n",
"2021-12-30 09:38:46,011 [INFO] tensorflow: global_step/sec: 3.05331\n",
"2021-12-30 09:38:46,310 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.929\n",
"INFO:tensorflow:global_step/sec: 3.14648\n",
"2021-12-30 09:38:48,872 [INFO] tensorflow: global_step/sec: 3.14648\n",
"INFO:tensorflow:epoch = 51.416666666666664, learning_rate = 0.0009999999, loss = 0.00014773724, step = 4936 (5.483 sec)\n",
"2021-12-30 09:38:50,169 [INFO] tensorflow: epoch = 51.416666666666664, learning_rate = 0.0009999999, loss = 0.00014773724, step = 4936 (5.483 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11436\n",
"2021-12-30 09:38:51,762 [INFO] tensorflow: global_step/sec: 3.11436\n",
"2021-12-30 09:38:54,357 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.853\n",
"INFO:tensorflow:global_step/sec: 3.08563\n",
"2021-12-30 09:38:54,678 [INFO] tensorflow: global_step/sec: 3.08563\n",
"INFO:tensorflow:epoch = 51.59375, learning_rate = 0.0009999999, loss = 0.00020669196, step = 4953 (5.494 sec)\n",
"2021-12-30 09:38:55,664 [INFO] tensorflow: epoch = 51.59375, learning_rate = 0.0009999999, loss = 0.00020669196, step = 4953 (5.494 sec)\n",
"INFO:tensorflow:global_step/sec: 3.04281\n",
"2021-12-30 09:38:57,636 [INFO] tensorflow: global_step/sec: 3.04281\n",
"INFO:tensorflow:global_step/sec: 3.03219\n",
"2021-12-30 09:39:00,604 [INFO] tensorflow: global_step/sec: 3.03219\n",
"INFO:tensorflow:epoch = 51.77083333333333, learning_rate = 0.0009999999, loss = 0.00016580292, step = 4970 (5.572 sec)\n",
"2021-12-30 09:39:01,236 [INFO] tensorflow: epoch = 51.77083333333333, learning_rate = 0.0009999999, loss = 0.00016580292, step = 4970 (5.572 sec)\n",
"2021-12-30 09:39:02,515 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.517\n",
"INFO:tensorflow:global_step/sec: 3.10637\n",
"2021-12-30 09:39:03,502 [INFO] tensorflow: global_step/sec: 3.10637\n",
"INFO:tensorflow:global_step/sec: 3.09103\n",
"2021-12-30 09:39:06,413 [INFO] tensorflow: global_step/sec: 3.09103\n",
"INFO:tensorflow:epoch = 51.947916666666664, learning_rate = 0.0009999999, loss = 0.00018627124, step = 4987 (5.490 sec)\n",
"2021-12-30 09:39:06,725 [INFO] tensorflow: epoch = 51.947916666666664, learning_rate = 0.0009999999, loss = 0.00018627124, step = 4987 (5.490 sec)\n",
"2021-12-30 09:39:08,352 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 52/120: loss: 0.00020 learning rate: 0.00100 Time taken: 0:00:31.036115 ETA: 0:35:10.455848\n",
"INFO:tensorflow:global_step/sec: 3.07355\n",
"2021-12-30 09:39:09,341 [INFO] tensorflow: global_step/sec: 3.07355\n",
"2021-12-30 09:39:10,641 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.616\n",
"INFO:tensorflow:epoch = 52.125, learning_rate = 0.0009999999, loss = 0.00027402295, step = 5004 (5.524 sec)\n",
"2021-12-30 09:39:12,250 [INFO] tensorflow: epoch = 52.125, learning_rate = 0.0009999999, loss = 0.00027402295, step = 5004 (5.524 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09398\n",
"2021-12-30 09:39:12,250 [INFO] tensorflow: global_step/sec: 3.09398\n",
"INFO:tensorflow:global_step/sec: 3.05465\n",
"2021-12-30 09:39:15,197 [INFO] tensorflow: global_step/sec: 3.05465\n",
"INFO:tensorflow:epoch = 52.30208333333333, learning_rate = 0.0009999999, loss = 0.00021165653, step = 5021 (5.535 sec)\n",
"2021-12-30 09:39:17,785 [INFO] tensorflow: epoch = 52.30208333333333, learning_rate = 0.0009999999, loss = 0.00021165653, step = 5021 (5.535 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11672\n",
"2021-12-30 09:39:18,084 [INFO] tensorflow: global_step/sec: 3.11672\n",
"2021-12-30 09:39:18,726 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.737\n",
"INFO:tensorflow:global_step/sec: 3.0224\n",
"2021-12-30 09:39:21,062 [INFO] tensorflow: global_step/sec: 3.0224\n",
"INFO:tensorflow:epoch = 52.479166666666664, learning_rate = 0.0009999999, loss = 0.00017279363, step = 5038 (5.535 sec)\n",
"2021-12-30 09:39:23,320 [INFO] tensorflow: epoch = 52.479166666666664, learning_rate = 0.0009999999, loss = 0.00017279363, step = 5038 (5.535 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10751\n",
"2021-12-30 09:39:23,958 [INFO] tensorflow: global_step/sec: 3.10751\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO:tensorflow:global_step/sec: 3.0492\n",
"2021-12-30 09:39:26,910 [INFO] tensorflow: global_step/sec: 3.0492\n",
"2021-12-30 09:39:26,911 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.436\n",
"INFO:tensorflow:epoch = 52.65625, learning_rate = 0.0009999999, loss = 0.00024269258, step = 5055 (5.584 sec)\n",
"2021-12-30 09:39:28,903 [INFO] tensorflow: epoch = 52.65625, learning_rate = 0.0009999999, loss = 0.00024269258, step = 5055 (5.584 sec)\n",
"INFO:tensorflow:global_step/sec: 3.03063\n",
"2021-12-30 09:39:29,880 [INFO] tensorflow: global_step/sec: 3.03063\n",
"INFO:tensorflow:global_step/sec: 3.14398\n",
"2021-12-30 09:39:32,742 [INFO] tensorflow: global_step/sec: 3.14398\n",
"INFO:tensorflow:epoch = 52.83333333333333, learning_rate = 0.0009999999, loss = 0.00019571744, step = 5072 (5.479 sec)\n",
"2021-12-30 09:39:34,382 [INFO] tensorflow: epoch = 52.83333333333333, learning_rate = 0.0009999999, loss = 0.00019571744, step = 5072 (5.479 sec)\n",
"2021-12-30 09:39:35,028 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.638\n",
"INFO:tensorflow:global_step/sec: 3.06264\n",
"2021-12-30 09:39:35,681 [INFO] tensorflow: global_step/sec: 3.06264\n",
"INFO:tensorflow:global_step/sec: 3.10718\n",
"2021-12-30 09:39:38,577 [INFO] tensorflow: global_step/sec: 3.10718\n",
"2021-12-30 09:39:39,564 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 53/120: loss: 0.00030 learning rate: 0.00100 Time taken: 0:00:31.227401 ETA: 0:34:52.235900\n",
"INFO:tensorflow:epoch = 53.010416666666664, learning_rate = 0.0009999999, loss = 0.00026285162, step = 5089 (5.510 sec)\n",
"2021-12-30 09:39:39,892 [INFO] tensorflow: epoch = 53.010416666666664, learning_rate = 0.0009999999, loss = 0.00026285162, step = 5089 (5.510 sec)\n",
"INFO:tensorflow:global_step/sec: 3.05242\n",
"2021-12-30 09:39:41,526 [INFO] tensorflow: global_step/sec: 3.05242\n",
"2021-12-30 09:39:43,135 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.671\n",
"INFO:tensorflow:global_step/sec: 3.12139\n",
"2021-12-30 09:39:44,409 [INFO] tensorflow: global_step/sec: 3.12139\n",
"INFO:tensorflow:epoch = 53.1875, learning_rate = 0.0009999999, loss = 0.00027590495, step = 5106 (5.474 sec)\n",
"2021-12-30 09:39:45,366 [INFO] tensorflow: epoch = 53.1875, learning_rate = 0.0009999999, loss = 0.00027590495, step = 5106 (5.474 sec)\n",
"INFO:tensorflow:global_step/sec: 3.0596\n",
"2021-12-30 09:39:47,351 [INFO] tensorflow: global_step/sec: 3.0596\n",
"INFO:tensorflow:global_step/sec: 3.10939\n",
"2021-12-30 09:39:50,245 [INFO] tensorflow: global_step/sec: 3.10939\n",
"INFO:tensorflow:epoch = 53.36458333333333, learning_rate = 0.0009999999, loss = 0.0002070835, step = 5123 (5.535 sec)\n",
"2021-12-30 09:39:50,901 [INFO] tensorflow: epoch = 53.36458333333333, learning_rate = 0.0009999999, loss = 0.0002070835, step = 5123 (5.535 sec)\n",
"2021-12-30 09:39:51,222 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.731\n",
"INFO:tensorflow:global_step/sec: 3.06996\n",
"2021-12-30 09:39:53,177 [INFO] tensorflow: global_step/sec: 3.06996\n",
"INFO:tensorflow:global_step/sec: 3.0532\n",
"2021-12-30 09:39:56,125 [INFO] tensorflow: global_step/sec: 3.0532\n",
"INFO:tensorflow:epoch = 53.541666666666664, learning_rate = 0.0009999999, loss = 0.00018008717, step = 5140 (5.549 sec)\n",
"2021-12-30 09:39:56,449 [INFO] tensorflow: epoch = 53.541666666666664, learning_rate = 0.0009999999, loss = 0.00018008717, step = 5140 (5.549 sec)\n",
"INFO:tensorflow:global_step/sec: 3.05003\n",
"2021-12-30 09:39:59,075 [INFO] tensorflow: global_step/sec: 3.05003\n",
"2021-12-30 09:39:59,403 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.449\n",
"INFO:tensorflow:epoch = 53.71875, learning_rate = 0.0009999999, loss = 0.00020198725, step = 5157 (5.506 sec)\n",
"2021-12-30 09:40:01,956 [INFO] tensorflow: epoch = 53.71875, learning_rate = 0.0009999999, loss = 0.00020198725, step = 5157 (5.506 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12383\n",
"2021-12-30 09:40:01,956 [INFO] tensorflow: global_step/sec: 3.12383\n",
"INFO:tensorflow:global_step/sec: 3.04385\n",
"2021-12-30 09:40:04,913 [INFO] tensorflow: global_step/sec: 3.04385\n",
"INFO:tensorflow:epoch = 53.89583333333333, learning_rate = 0.0009999999, loss = 0.00017273336, step = 5174 (5.538 sec)\n",
"2021-12-30 09:40:07,493 [INFO] tensorflow: epoch = 53.89583333333333, learning_rate = 0.0009999999, loss = 0.00017273336, step = 5174 (5.538 sec)\n",
"2021-12-30 09:40:07,494 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.721\n",
"INFO:tensorflow:global_step/sec: 3.09219\n",
"2021-12-30 09:40:07,824 [INFO] tensorflow: global_step/sec: 3.09219\n",
"INFO:tensorflow:global_step/sec: 3.06818\n",
"2021-12-30 09:40:10,757 [INFO] tensorflow: global_step/sec: 3.06818\n",
"2021-12-30 09:40:10,758 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 54/120: loss: 0.00025 learning rate: 0.00100 Time taken: 0:00:31.170252 ETA: 0:34:17.236622\n",
"INFO:tensorflow:epoch = 54.072916666666664, learning_rate = 0.0009999999, loss = 0.0002749422, step = 5191 (5.495 sec)\n",
"2021-12-30 09:40:12,989 [INFO] tensorflow: epoch = 54.072916666666664, learning_rate = 0.0009999999, loss = 0.0002749422, step = 5191 (5.495 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12404\n",
"2021-12-30 09:40:13,638 [INFO] tensorflow: global_step/sec: 3.12404\n",
"2021-12-30 09:40:15,569 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.766\n",
"INFO:tensorflow:global_step/sec: 3.08639\n",
"2021-12-30 09:40:16,554 [INFO] tensorflow: global_step/sec: 3.08639\n",
"INFO:tensorflow:epoch = 54.25, learning_rate = 0.0009999999, loss = 0.00018593861, step = 5208 (5.547 sec)\n",
"2021-12-30 09:40:18,536 [INFO] tensorflow: epoch = 54.25, learning_rate = 0.0009999999, loss = 0.00018593861, step = 5208 (5.547 sec)\n",
"INFO:tensorflow:global_step/sec: 3.04986\n",
"2021-12-30 09:40:19,505 [INFO] tensorflow: global_step/sec: 3.04986\n",
"INFO:tensorflow:global_step/sec: 3.10254\n",
"2021-12-30 09:40:22,406 [INFO] tensorflow: global_step/sec: 3.10254\n",
"2021-12-30 09:40:23,709 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.574\n",
"INFO:tensorflow:epoch = 54.42708333333333, learning_rate = 0.0009999999, loss = 0.00020612436, step = 5225 (5.514 sec)\n",
"2021-12-30 09:40:24,050 [INFO] tensorflow: epoch = 54.42708333333333, learning_rate = 0.0009999999, loss = 0.00020612436, step = 5225 (5.514 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07694\n",
"2021-12-30 09:40:25,331 [INFO] tensorflow: global_step/sec: 3.07694\n",
"INFO:tensorflow:global_step/sec: 3.03009\n",
"2021-12-30 09:40:28,301 [INFO] tensorflow: global_step/sec: 3.03009\n",
"INFO:tensorflow:epoch = 54.604166666666664, learning_rate = 0.0009999999, loss = 0.00022038813, step = 5242 (5.534 sec)\n",
"2021-12-30 09:40:29,584 [INFO] tensorflow: epoch = 54.604166666666664, learning_rate = 0.0009999999, loss = 0.00022038813, step = 5242 (5.534 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07441\n",
"2021-12-30 09:40:31,228 [INFO] tensorflow: global_step/sec: 3.07441\n",
"2021-12-30 09:40:31,865 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.520\n",
"INFO:tensorflow:global_step/sec: 3.11543\n",
"2021-12-30 09:40:34,117 [INFO] tensorflow: global_step/sec: 3.11543\n",
"INFO:tensorflow:epoch = 54.78125, learning_rate = 0.0009999999, loss = 0.00020172665, step = 5259 (5.510 sec)\n",
"2021-12-30 09:40:35,094 [INFO] tensorflow: epoch = 54.78125, learning_rate = 0.0009999999, loss = 0.00020172665, step = 5259 (5.510 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07853\n",
"2021-12-30 09:40:37,041 [INFO] tensorflow: global_step/sec: 3.07853\n",
"INFO:tensorflow:global_step/sec: 3.12245\n",
"2021-12-30 09:40:39,923 [INFO] tensorflow: global_step/sec: 3.12245\n",
"2021-12-30 09:40:39,924 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.819\n",
"INFO:tensorflow:epoch = 54.95833333333333, learning_rate = 0.0009999999, loss = 0.00020330295, step = 5276 (5.494 sec)\n",
"2021-12-30 09:40:40,587 [INFO] tensorflow: epoch = 54.95833333333333, learning_rate = 0.0009999999, loss = 0.00020330295, step = 5276 (5.494 sec)\n",
"2021-12-30 09:40:41,903 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 55/120: loss: 0.00019 learning rate: 0.00100 Time taken: 0:00:31.171603 ETA: 0:33:46.154224\n",
"INFO:tensorflow:global_step/sec: 3.05998\n",
"2021-12-30 09:40:42,864 [INFO] tensorflow: global_step/sec: 3.05998\n",
"INFO:tensorflow:global_step/sec: 3.10301\n",
"2021-12-30 09:40:45,765 [INFO] tensorflow: global_step/sec: 3.10301\n",
"INFO:tensorflow:epoch = 55.135416666666664, learning_rate = 0.0009999999, loss = 0.00024076148, step = 5293 (5.503 sec)\n",
"2021-12-30 09:40:46,090 [INFO] tensorflow: epoch = 55.135416666666664, learning_rate = 0.0009999999, loss = 0.00024076148, step = 5293 (5.503 sec)\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"2021-12-30 09:40:47,999 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.767\n",
"INFO:tensorflow:global_step/sec: 3.11852\n",
"2021-12-30 09:40:48,651 [INFO] tensorflow: global_step/sec: 3.11852\n",
"INFO:tensorflow:epoch = 55.3125, learning_rate = 0.0009999999, loss = 0.00021304096, step = 5310 (5.491 sec)\n",
"2021-12-30 09:40:51,581 [INFO] tensorflow: epoch = 55.3125, learning_rate = 0.0009999999, loss = 0.00021304096, step = 5310 (5.491 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07037\n",
"2021-12-30 09:40:51,582 [INFO] tensorflow: global_step/sec: 3.07037\n",
"INFO:tensorflow:global_step/sec: 3.09238\n",
"2021-12-30 09:40:54,492 [INFO] tensorflow: global_step/sec: 3.09238\n",
"2021-12-30 09:40:56,120 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.628\n",
"INFO:tensorflow:epoch = 55.48958333333333, learning_rate = 0.0009999999, loss = 0.00033611344, step = 5327 (5.504 sec)\n",
"2021-12-30 09:40:57,085 [INFO] tensorflow: epoch = 55.48958333333333, learning_rate = 0.0009999999, loss = 0.00033611344, step = 5327 (5.504 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08598\n",
"2021-12-30 09:40:57,409 [INFO] tensorflow: global_step/sec: 3.08598\n",
"INFO:tensorflow:global_step/sec: 3.13447\n",
"2021-12-30 09:41:00,280 [INFO] tensorflow: global_step/sec: 3.13447\n",
"INFO:tensorflow:epoch = 55.666666666666664, learning_rate = 0.0009999999, loss = 0.00023455582, step = 5344 (5.496 sec)\n",
"2021-12-30 09:41:02,581 [INFO] tensorflow: epoch = 55.666666666666664, learning_rate = 0.0009999999, loss = 0.00023455582, step = 5344 (5.496 sec)\n",
"INFO:tensorflow:global_step/sec: 3.05033\n",
"2021-12-30 09:41:03,231 [INFO] tensorflow: global_step/sec: 3.05033\n",
"2021-12-30 09:41:04,195 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.771\n",
"INFO:tensorflow:global_step/sec: 3.1271\n",
"2021-12-30 09:41:06,109 [INFO] tensorflow: global_step/sec: 3.1271\n",
"INFO:tensorflow:epoch = 55.84375, learning_rate = 0.0009999999, loss = 0.00022475774, step = 5361 (5.473 sec)\n",
"2021-12-30 09:41:08,054 [INFO] tensorflow: epoch = 55.84375, learning_rate = 0.0009999999, loss = 0.00022475774, step = 5361 (5.473 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08135\n",
"2021-12-30 09:41:09,029 [INFO] tensorflow: global_step/sec: 3.08135\n",
"INFO:tensorflow:global_step/sec: 3.03573\n",
"2021-12-30 09:41:11,994 [INFO] tensorflow: global_step/sec: 3.03573\n",
"2021-12-30 09:41:12,311 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.641\n",
"2021-12-30 09:41:12,960 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 56/120: loss: 0.00025 learning rate: 0.00100 Time taken: 0:00:31.046090 ETA: 0:33:06.949783\n",
"INFO:tensorflow:epoch = 56.02083333333333, learning_rate = 0.0009999999, loss = 0.00024123656, step = 5378 (5.539 sec)\n",
"2021-12-30 09:41:13,593 [INFO] tensorflow: epoch = 56.02083333333333, learning_rate = 0.0009999999, loss = 0.00024123656, step = 5378 (5.539 sec)\n",
"INFO:tensorflow:global_step/sec: 3.13846\n",
"2021-12-30 09:41:14,862 [INFO] tensorflow: global_step/sec: 3.13846\n",
"INFO:tensorflow:global_step/sec: 3.03648\n",
"2021-12-30 09:41:17,826 [INFO] tensorflow: global_step/sec: 3.03648\n",
"INFO:tensorflow:epoch = 56.197916666666664, learning_rate = 0.0009999999, loss = 0.00018562937, step = 5395 (5.517 sec)\n",
"2021-12-30 09:41:19,110 [INFO] tensorflow: epoch = 56.197916666666664, learning_rate = 0.0009999999, loss = 0.00018562937, step = 5395 (5.517 sec)\n",
"2021-12-30 09:41:20,421 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.664\n",
"INFO:tensorflow:global_step/sec: 3.07717\n",
"2021-12-30 09:41:20,750 [INFO] tensorflow: global_step/sec: 3.07717\n",
"INFO:tensorflow:global_step/sec: 3.03718\n",
"2021-12-30 09:41:23,714 [INFO] tensorflow: global_step/sec: 3.03718\n",
"INFO:tensorflow:epoch = 56.375, learning_rate = 0.0009999999, loss = 0.00032677903, step = 5412 (5.589 sec)\n",
"2021-12-30 09:41:24,699 [INFO] tensorflow: epoch = 56.375, learning_rate = 0.0009999999, loss = 0.00032677903, step = 5412 (5.589 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08887\n",
"2021-12-30 09:41:26,627 [INFO] tensorflow: global_step/sec: 3.08887\n",
"2021-12-30 09:41:28,565 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.558\n",
"INFO:tensorflow:global_step/sec: 3.09912\n",
"2021-12-30 09:41:29,531 [INFO] tensorflow: global_step/sec: 3.09912\n",
"INFO:tensorflow:epoch = 56.55208333333333, learning_rate = 0.0009999999, loss = 0.00022386354, step = 5429 (5.478 sec)\n",
"2021-12-30 09:41:30,177 [INFO] tensorflow: epoch = 56.55208333333333, learning_rate = 0.0009999999, loss = 0.00022386354, step = 5429 (5.478 sec)\n",
"INFO:tensorflow:global_step/sec: 3.04416\n",
"2021-12-30 09:41:32,488 [INFO] tensorflow: global_step/sec: 3.04416\n",
"INFO:tensorflow:global_step/sec: 3.06861\n",
"2021-12-30 09:41:35,421 [INFO] tensorflow: global_step/sec: 3.06861\n",
"INFO:tensorflow:epoch = 56.729166666666664, learning_rate = 0.0009999999, loss = 0.00026953706, step = 5446 (5.578 sec)\n",
"2021-12-30 09:41:35,755 [INFO] tensorflow: epoch = 56.729166666666664, learning_rate = 0.0009999999, loss = 0.00026953706, step = 5446 (5.578 sec)\n",
"2021-12-30 09:41:36,709 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.559\n",
"INFO:tensorflow:global_step/sec: 3.13602\n",
"2021-12-30 09:41:38,291 [INFO] tensorflow: global_step/sec: 3.13602\n",
"INFO:tensorflow:epoch = 56.90625, learning_rate = 0.0009999999, loss = 0.00024962303, step = 5463 (5.449 sec)\n",
"2021-12-30 09:41:41,204 [INFO] tensorflow: epoch = 56.90625, learning_rate = 0.0009999999, loss = 0.00024962303, step = 5463 (5.449 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08824\n",
"2021-12-30 09:41:41,205 [INFO] tensorflow: global_step/sec: 3.08824\n",
"INFO:tensorflow:global_step/sec: 3.06114\n",
"2021-12-30 09:41:44,145 [INFO] tensorflow: global_step/sec: 3.06114\n",
"2021-12-30 09:41:44,146 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 57/120: loss: 0.00018 learning rate: 0.00100 Time taken: 0:00:31.193090 ETA: 0:32:45.164698\n",
"2021-12-30 09:41:44,793 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.740\n",
"INFO:tensorflow:epoch = 57.08333333333333, learning_rate = 0.0009999999, loss = 0.00020600311, step = 5480 (5.605 sec)\n",
"2021-12-30 09:41:46,809 [INFO] tensorflow: epoch = 57.08333333333333, learning_rate = 0.0009999999, loss = 0.00020600311, step = 5480 (5.605 sec)\n",
"INFO:tensorflow:global_step/sec: 2.99281\n",
"2021-12-30 09:41:47,152 [INFO] tensorflow: global_step/sec: 2.99281\n",
"INFO:tensorflow:global_step/sec: 3.10721\n",
"2021-12-30 09:41:50,049 [INFO] tensorflow: global_step/sec: 3.10721\n",
"INFO:tensorflow:epoch = 57.260416666666664, learning_rate = 0.0009999999, loss = 0.00021522479, step = 5497 (5.475 sec)\n",
"2021-12-30 09:41:52,285 [INFO] tensorflow: epoch = 57.260416666666664, learning_rate = 0.0009999999, loss = 0.00021522479, step = 5497 (5.475 sec)\n",
"INFO:tensorflow:global_step/sec: 3.13611\n",
"2021-12-30 09:41:52,919 [INFO] tensorflow: global_step/sec: 3.13611\n",
"2021-12-30 09:41:52,919 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.614\n",
"INFO:tensorflow:global_step/sec: 3.12973\n",
"2021-12-30 09:41:55,794 [INFO] tensorflow: global_step/sec: 3.12973\n",
"INFO:tensorflow:epoch = 57.4375, learning_rate = 0.0009999999, loss = 0.00021561954, step = 5514 (5.452 sec)\n",
"2021-12-30 09:41:57,736 [INFO] tensorflow: epoch = 57.4375, learning_rate = 0.0009999999, loss = 0.00021561954, step = 5514 (5.452 sec)\n",
"INFO:tensorflow:global_step/sec: 3.05965\n",
"2021-12-30 09:41:58,736 [INFO] tensorflow: global_step/sec: 3.05965\n",
"2021-12-30 09:42:00,989 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.785\n",
"INFO:tensorflow:global_step/sec: 3.09095\n",
"2021-12-30 09:42:01,647 [INFO] tensorflow: global_step/sec: 3.09095\n",
"INFO:tensorflow:epoch = 57.61458333333333, learning_rate = 0.0009999999, loss = 0.00020949655, step = 5531 (5.557 sec)\n",
"2021-12-30 09:42:03,294 [INFO] tensorflow: epoch = 57.61458333333333, learning_rate = 0.0009999999, loss = 0.00020949655, step = 5531 (5.557 sec)\n",
"INFO:tensorflow:global_step/sec: 3.02151\n",
"2021-12-30 09:42:04,626 [INFO] tensorflow: global_step/sec: 3.02151\n",
"INFO:tensorflow:global_step/sec: 3.05955\n",
"2021-12-30 09:42:07,568 [INFO] tensorflow: global_step/sec: 3.05955\n",
"INFO:tensorflow:epoch = 57.791666666666664, learning_rate = 0.0009999999, loss = 0.00023869322, step = 5548 (5.603 sec)\n",
"2021-12-30 09:42:08,896 [INFO] tensorflow: epoch = 57.791666666666664, learning_rate = 0.0009999999, loss = 0.00023869322, step = 5548 (5.603 sec)\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"2021-12-30 09:42:09,211 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.324\n",
"INFO:tensorflow:global_step/sec: 3.04716\n",
"2021-12-30 09:42:10,521 [INFO] tensorflow: global_step/sec: 3.04716\n",
"INFO:tensorflow:global_step/sec: 3.09935\n",
"2021-12-30 09:42:13,425 [INFO] tensorflow: global_step/sec: 3.09935\n",
"INFO:tensorflow:epoch = 57.96875, learning_rate = 0.0009999999, loss = 0.00015433287, step = 5565 (5.501 sec)\n",
"2021-12-30 09:42:14,397 [INFO] tensorflow: epoch = 57.96875, learning_rate = 0.0009999999, loss = 0.00015433287, step = 5565 (5.501 sec)\n",
"2021-12-30 09:42:15,373 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 58/120: loss: 0.00025 learning rate: 0.00100 Time taken: 0:00:31.219094 ETA: 0:32:15.583801\n",
"INFO:tensorflow:global_step/sec: 3.08656\n",
"2021-12-30 09:42:16,341 [INFO] tensorflow: global_step/sec: 3.08656\n",
"2021-12-30 09:42:17,313 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.687\n",
"INFO:tensorflow:global_step/sec: 3.0774\n",
"2021-12-30 09:42:19,266 [INFO] tensorflow: global_step/sec: 3.0774\n",
"INFO:tensorflow:epoch = 58.14583333333333, learning_rate = 0.0009999999, loss = 0.000267832, step = 5582 (5.531 sec)\n",
"2021-12-30 09:42:19,928 [INFO] tensorflow: epoch = 58.14583333333333, learning_rate = 0.0009999999, loss = 0.000267832, step = 5582 (5.531 sec)\n",
"INFO:tensorflow:global_step/sec: 3.06825\n",
"2021-12-30 09:42:22,199 [INFO] tensorflow: global_step/sec: 3.06825\n",
"INFO:tensorflow:global_step/sec: 3.095\n",
"2021-12-30 09:42:25,107 [INFO] tensorflow: global_step/sec: 3.095\n",
"INFO:tensorflow:epoch = 58.322916666666664, learning_rate = 0.0009999999, loss = 0.00025748694, step = 5599 (5.492 sec)\n",
"2021-12-30 09:42:25,420 [INFO] tensorflow: epoch = 58.322916666666664, learning_rate = 0.0009999999, loss = 0.00025748694, step = 5599 (5.492 sec)\n",
"2021-12-30 09:42:25,420 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.669\n",
"INFO:tensorflow:global_step/sec: 3.08096\n",
"2021-12-30 09:42:28,028 [INFO] tensorflow: global_step/sec: 3.08096\n",
"INFO:tensorflow:epoch = 58.5, learning_rate = 0.0009999999, loss = 0.00020094954, step = 5616 (5.567 sec)\n",
"2021-12-30 09:42:30,987 [INFO] tensorflow: epoch = 58.5, learning_rate = 0.0009999999, loss = 0.00020094954, step = 5616 (5.567 sec)\n",
"INFO:tensorflow:global_step/sec: 3.04017\n",
"2021-12-30 09:42:30,988 [INFO] tensorflow: global_step/sec: 3.04017\n",
"2021-12-30 09:42:33,558 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.578\n",
"INFO:tensorflow:global_step/sec: 3.11686\n",
"2021-12-30 09:42:33,876 [INFO] tensorflow: global_step/sec: 3.11686\n",
"INFO:tensorflow:epoch = 58.67708333333333, learning_rate = 0.0009999999, loss = 0.00020002204, step = 5633 (5.508 sec)\n",
"2021-12-30 09:42:36,495 [INFO] tensorflow: epoch = 58.67708333333333, learning_rate = 0.0009999999, loss = 0.00020002204, step = 5633 (5.508 sec)\n",
"INFO:tensorflow:global_step/sec: 3.0612\n",
"2021-12-30 09:42:36,816 [INFO] tensorflow: global_step/sec: 3.0612\n",
"INFO:tensorflow:global_step/sec: 3.12213\n",
"2021-12-30 09:42:39,698 [INFO] tensorflow: global_step/sec: 3.12213\n",
"2021-12-30 09:42:41,629 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.781\n",
"INFO:tensorflow:epoch = 58.854166666666664, learning_rate = 0.0009999999, loss = 0.00022007289, step = 5650 (5.470 sec)\n",
"2021-12-30 09:42:41,966 [INFO] tensorflow: epoch = 58.854166666666664, learning_rate = 0.0009999999, loss = 0.00022007289, step = 5650 (5.470 sec)\n",
"INFO:tensorflow:global_step/sec: 3.05158\n",
"2021-12-30 09:42:42,648 [INFO] tensorflow: global_step/sec: 3.05158\n",
"INFO:tensorflow:global_step/sec: 3.12176\n",
"2021-12-30 09:42:45,531 [INFO] tensorflow: global_step/sec: 3.12176\n",
"2021-12-30 09:42:46,503 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 59/120: loss: 0.00025 learning rate: 0.00100 Time taken: 0:00:31.132528 ETA: 0:31:39.084212\n",
"INFO:tensorflow:epoch = 59.03125, learning_rate = 0.0009999999, loss = 0.00023058207, step = 5667 (5.533 sec)\n",
"2021-12-30 09:42:47,499 [INFO] tensorflow: epoch = 59.03125, learning_rate = 0.0009999999, loss = 0.00023058207, step = 5667 (5.533 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07428\n",
"2021-12-30 09:42:48,458 [INFO] tensorflow: global_step/sec: 3.07428\n",
"2021-12-30 09:42:49,762 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.593\n",
"INFO:tensorflow:global_step/sec: 3.08505\n",
"2021-12-30 09:42:51,376 [INFO] tensorflow: global_step/sec: 3.08505\n",
"INFO:tensorflow:epoch = 59.20833333333333, learning_rate = 0.0009999999, loss = 0.00017407708, step = 5684 (5.529 sec)\n",
"2021-12-30 09:42:53,028 [INFO] tensorflow: epoch = 59.20833333333333, learning_rate = 0.0009999999, loss = 0.00017407708, step = 5684 (5.529 sec)\n",
"INFO:tensorflow:global_step/sec: 3.05371\n",
"2021-12-30 09:42:54,323 [INFO] tensorflow: global_step/sec: 3.05371\n",
"INFO:tensorflow:global_step/sec: 3.08671\n",
"2021-12-30 09:42:57,238 [INFO] tensorflow: global_step/sec: 3.08671\n",
"2021-12-30 09:42:57,889 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.608\n",
"INFO:tensorflow:epoch = 59.385416666666664, learning_rate = 0.0009999999, loss = 0.00024867832, step = 5701 (5.516 sec)\n",
"2021-12-30 09:42:58,544 [INFO] tensorflow: epoch = 59.385416666666664, learning_rate = 0.0009999999, loss = 0.00024867832, step = 5701 (5.516 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07527\n",
"2021-12-30 09:43:00,165 [INFO] tensorflow: global_step/sec: 3.07527\n",
"INFO:tensorflow:global_step/sec: 3.05361\n",
"2021-12-30 09:43:03,112 [INFO] tensorflow: global_step/sec: 3.05361\n",
"INFO:tensorflow:epoch = 59.5625, learning_rate = 0.0009999999, loss = 0.000206337, step = 5718 (5.491 sec)\n",
"2021-12-30 09:43:04,035 [INFO] tensorflow: epoch = 59.5625, learning_rate = 0.0009999999, loss = 0.000206337, step = 5718 (5.491 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12023\n",
"2021-12-30 09:43:05,997 [INFO] tensorflow: global_step/sec: 3.12023\n",
"2021-12-30 09:43:05,998 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.667\n",
"INFO:tensorflow:global_step/sec: 3.11078\n",
"2021-12-30 09:43:08,890 [INFO] tensorflow: global_step/sec: 3.11078\n",
"INFO:tensorflow:epoch = 59.73958333333333, learning_rate = 0.0009999999, loss = 0.0003394548, step = 5735 (5.506 sec)\n",
"2021-12-30 09:43:09,541 [INFO] tensorflow: epoch = 59.73958333333333, learning_rate = 0.0009999999, loss = 0.0003394548, step = 5735 (5.506 sec)\n",
"INFO:tensorflow:global_step/sec: 3.06207\n",
"2021-12-30 09:43:11,829 [INFO] tensorflow: global_step/sec: 3.06207\n",
"2021-12-30 09:43:14,121 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.620\n",
"INFO:tensorflow:global_step/sec: 3.05766\n",
"2021-12-30 09:43:14,773 [INFO] tensorflow: global_step/sec: 3.05766\n",
"INFO:tensorflow:epoch = 59.916666666666664, learning_rate = 0.0009999999, loss = 0.0001945727, step = 5752 (5.566 sec)\n",
"2021-12-30 09:43:15,107 [INFO] tensorflow: epoch = 59.916666666666664, learning_rate = 0.0009999999, loss = 0.0001945727, step = 5752 (5.566 sec)\n",
"INFO:tensorflow:Saving checkpoints for step-5760.\n",
"2021-12-30 09:43:17,361 [INFO] tensorflow: Saving checkpoints for step-5760.\n",
"WARNING:tensorflow:Ignoring: /tmp/tmph4_19kaj; No such file or directory\n",
"2021-12-30 09:43:17,506 [WARNING] tensorflow: Ignoring: /tmp/tmph4_19kaj; No such file or directory\n",
"2021-12-30 09:43:20,993 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 23, 0.00s/step\n",
"2021-12-30 09:43:22,868 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 23, 0.19s/step\n",
"2021-12-30 09:43:24,649 [INFO] iva.detectnet_v2.evaluation.evaluation: step 20 / 23, 0.18s/step\n",
"Matching predictions to ground truth, class 1/1.: 100%|█| 1783/1783 [00:00<00:00, 15925.24it/s]\n",
"Epoch 60/120\n",
"=========================\n",
"\n",
"Validation cost: 0.000252\n",
"Mean average_precision (in %): 80.6683\n",
"\n",
"class name average precision (in %)\n",
"------------ --------------------------\n",
"car 80.6683\n",
"\n",
"Median Inference Time: 0.015601\n",
"INFO:tensorflow:epoch = 60.0, learning_rate = 0.0009999999, loss = 0.0002163398, step = 5760 (10.522 sec)\n",
"2021-12-30 09:43:25,629 [INFO] tensorflow: epoch = 60.0, learning_rate = 0.0009999999, loss = 0.0002163398, step = 5760 (10.522 sec)\n",
"INFO:tensorflow:global_step/sec: 0.828954\n",
"2021-12-30 09:43:25,630 [INFO] tensorflow: global_step/sec: 0.828954\n",
"2021-12-30 09:43:25,630 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 60/120: loss: 0.00022 learning rate: 0.00100 Time taken: 0:00:39.123619 ETA: 0:39:07.417145\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO:tensorflow:global_step/sec: 3.0484\n",
"2021-12-30 09:43:28,582 [INFO] tensorflow: global_step/sec: 3.0484\n",
"2021-12-30 09:43:30,209 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 12.432\n",
"INFO:tensorflow:epoch = 60.17708333333333, learning_rate = 0.0009999999, loss = 0.0002450766, step = 5777 (5.549 sec)\n",
"2021-12-30 09:43:31,178 [INFO] tensorflow: epoch = 60.17708333333333, learning_rate = 0.0009999999, loss = 0.0002450766, step = 5777 (5.549 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08029\n",
"2021-12-30 09:43:31,504 [INFO] tensorflow: global_step/sec: 3.08029\n",
"INFO:tensorflow:global_step/sec: 3.05788\n",
"2021-12-30 09:43:34,447 [INFO] tensorflow: global_step/sec: 3.05788\n",
"INFO:tensorflow:epoch = 60.354166666666664, learning_rate = 0.0009999999, loss = 0.00025938704, step = 5794 (5.524 sec)\n",
"2021-12-30 09:43:36,702 [INFO] tensorflow: epoch = 60.354166666666664, learning_rate = 0.0009999999, loss = 0.00025938704, step = 5794 (5.524 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10798\n",
"2021-12-30 09:43:37,343 [INFO] tensorflow: global_step/sec: 3.10798\n",
"2021-12-30 09:43:38,293 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.743\n",
"INFO:tensorflow:global_step/sec: 3.10814\n",
"2021-12-30 09:43:40,238 [INFO] tensorflow: global_step/sec: 3.10814\n",
"INFO:tensorflow:epoch = 60.53125, learning_rate = 0.0009999999, loss = 0.0002496457, step = 5811 (5.470 sec)\n",
"2021-12-30 09:43:42,172 [INFO] tensorflow: epoch = 60.53125, learning_rate = 0.0009999999, loss = 0.0002496457, step = 5811 (5.470 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09315\n",
"2021-12-30 09:43:43,148 [INFO] tensorflow: global_step/sec: 3.09315\n",
"INFO:tensorflow:global_step/sec: 3.09696\n",
"2021-12-30 09:43:46,054 [INFO] tensorflow: global_step/sec: 3.09696\n",
"2021-12-30 09:43:46,377 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.740\n",
"INFO:tensorflow:epoch = 60.70833333333333, learning_rate = 0.0009999999, loss = 0.0002186865, step = 5828 (5.485 sec)\n",
"2021-12-30 09:43:47,657 [INFO] tensorflow: epoch = 60.70833333333333, learning_rate = 0.0009999999, loss = 0.0002186865, step = 5828 (5.485 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10687\n",
"2021-12-30 09:43:48,951 [INFO] tensorflow: global_step/sec: 3.10687\n",
"INFO:tensorflow:global_step/sec: 3.0416\n",
"2021-12-30 09:43:51,910 [INFO] tensorflow: global_step/sec: 3.0416\n",
"INFO:tensorflow:epoch = 60.885416666666664, learning_rate = 0.0009999999, loss = 0.00025904234, step = 5845 (5.559 sec)\n",
"2021-12-30 09:43:53,216 [INFO] tensorflow: epoch = 60.885416666666664, learning_rate = 0.0009999999, loss = 0.00025904234, step = 5845 (5.559 sec)\n",
"2021-12-30 09:43:54,477 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.693\n",
"INFO:tensorflow:global_step/sec: 3.12341\n",
"2021-12-30 09:43:54,791 [INFO] tensorflow: global_step/sec: 3.12341\n",
"2021-12-30 09:43:56,746 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 61/120: loss: 0.00022 learning rate: 0.00100 Time taken: 0:00:31.117974 ETA: 0:30:35.960440\n",
"INFO:tensorflow:global_step/sec: 3.05632\n",
"2021-12-30 09:43:57,736 [INFO] tensorflow: global_step/sec: 3.05632\n",
"INFO:tensorflow:epoch = 61.0625, learning_rate = 0.0009999999, loss = 0.00023552464, step = 5862 (5.488 sec)\n",
"2021-12-30 09:43:58,704 [INFO] tensorflow: epoch = 61.0625, learning_rate = 0.0009999999, loss = 0.00023552464, step = 5862 (5.488 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11759\n",
"2021-12-30 09:44:00,623 [INFO] tensorflow: global_step/sec: 3.11759\n",
"2021-12-30 09:44:02,583 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.674\n",
"INFO:tensorflow:global_step/sec: 3.08434\n",
"2021-12-30 09:44:03,541 [INFO] tensorflow: global_step/sec: 3.08434\n",
"INFO:tensorflow:epoch = 61.23958333333333, learning_rate = 0.0009999999, loss = 0.00019884622, step = 5879 (5.496 sec)\n",
"2021-12-30 09:44:04,200 [INFO] tensorflow: epoch = 61.23958333333333, learning_rate = 0.0009999999, loss = 0.00019884622, step = 5879 (5.496 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08019\n",
"2021-12-30 09:44:06,463 [INFO] tensorflow: global_step/sec: 3.08019\n",
"INFO:tensorflow:global_step/sec: 3.11448\n",
"2021-12-30 09:44:09,353 [INFO] tensorflow: global_step/sec: 3.11448\n",
"INFO:tensorflow:epoch = 61.416666666666664, learning_rate = 0.0009999999, loss = 0.00026859136, step = 5896 (5.478 sec)\n",
"2021-12-30 09:44:09,679 [INFO] tensorflow: epoch = 61.416666666666664, learning_rate = 0.0009999999, loss = 0.00026859136, step = 5896 (5.478 sec)\n",
"2021-12-30 09:44:10,666 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.743\n",
"INFO:tensorflow:global_step/sec: 3.07978\n",
"2021-12-30 09:44:12,275 [INFO] tensorflow: global_step/sec: 3.07978\n",
"INFO:tensorflow:epoch = 61.59375, learning_rate = 0.0009999999, loss = 0.00022661529, step = 5913 (5.529 sec)\n",
"2021-12-30 09:44:15,208 [INFO] tensorflow: epoch = 61.59375, learning_rate = 0.0009999999, loss = 0.00022661529, step = 5913 (5.529 sec)\n",
"INFO:tensorflow:global_step/sec: 3.06779\n",
"2021-12-30 09:44:15,208 [INFO] tensorflow: global_step/sec: 3.06779\n",
"INFO:tensorflow:global_step/sec: 3.08861\n",
"2021-12-30 09:44:18,122 [INFO] tensorflow: global_step/sec: 3.08861\n",
"2021-12-30 09:44:18,740 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.771\n",
"INFO:tensorflow:epoch = 61.77083333333333, learning_rate = 0.0009999999, loss = 0.00022272152, step = 5930 (5.483 sec)\n",
"2021-12-30 09:44:20,690 [INFO] tensorflow: epoch = 61.77083333333333, learning_rate = 0.0009999999, loss = 0.00022272152, step = 5930 (5.483 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11047\n",
"2021-12-30 09:44:21,016 [INFO] tensorflow: global_step/sec: 3.11047\n",
"INFO:tensorflow:global_step/sec: 3.03614\n",
"2021-12-30 09:44:23,980 [INFO] tensorflow: global_step/sec: 3.03614\n",
"INFO:tensorflow:epoch = 61.947916666666664, learning_rate = 0.0009999999, loss = 0.0002286541, step = 5947 (5.520 sec)\n",
"2021-12-30 09:44:26,211 [INFO] tensorflow: epoch = 61.947916666666664, learning_rate = 0.0009999999, loss = 0.0002286541, step = 5947 (5.520 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08569\n",
"2021-12-30 09:44:26,897 [INFO] tensorflow: global_step/sec: 3.08569\n",
"2021-12-30 09:44:26,898 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.519\n",
"2021-12-30 09:44:27,886 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 62/120: loss: 0.00019 learning rate: 0.00100 Time taken: 0:00:31.129719 ETA: 0:30:05.523717\n",
"INFO:tensorflow:global_step/sec: 3.08829\n",
"2021-12-30 09:44:29,811 [INFO] tensorflow: global_step/sec: 3.08829\n",
"INFO:tensorflow:epoch = 62.125, learning_rate = 0.0009999999, loss = 0.00022139173, step = 5964 (5.582 sec)\n",
"2021-12-30 09:44:31,793 [INFO] tensorflow: epoch = 62.125, learning_rate = 0.0009999999, loss = 0.00022139173, step = 5964 (5.582 sec)\n",
"INFO:tensorflow:global_step/sec: 3.05961\n",
"2021-12-30 09:44:32,753 [INFO] tensorflow: global_step/sec: 3.05961\n",
"2021-12-30 09:44:35,007 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.664\n",
"INFO:tensorflow:global_step/sec: 3.10656\n",
"2021-12-30 09:44:35,650 [INFO] tensorflow: global_step/sec: 3.10656\n",
"INFO:tensorflow:epoch = 62.30208333333333, learning_rate = 0.0009999999, loss = 0.0002004339, step = 5981 (5.451 sec)\n",
"2021-12-30 09:44:37,244 [INFO] tensorflow: epoch = 62.30208333333333, learning_rate = 0.0009999999, loss = 0.0002004339, step = 5981 (5.451 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10223\n",
"2021-12-30 09:44:38,551 [INFO] tensorflow: global_step/sec: 3.10223\n",
"INFO:tensorflow:global_step/sec: 3.124\n",
"2021-12-30 09:44:41,432 [INFO] tensorflow: global_step/sec: 3.124\n",
"INFO:tensorflow:epoch = 62.479166666666664, learning_rate = 0.0009999999, loss = 0.00016891159, step = 5998 (5.517 sec)\n",
"2021-12-30 09:44:42,761 [INFO] tensorflow: epoch = 62.479166666666664, learning_rate = 0.0009999999, loss = 0.00016891159, step = 5998 (5.517 sec)\n",
"2021-12-30 09:44:43,080 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.775\n",
"INFO:tensorflow:global_step/sec: 3.08122\n",
"2021-12-30 09:44:44,353 [INFO] tensorflow: global_step/sec: 3.08122\n",
"INFO:tensorflow:global_step/sec: 3.0368\n",
"2021-12-30 09:44:47,316 [INFO] tensorflow: global_step/sec: 3.0368\n",
"INFO:tensorflow:epoch = 62.65625, learning_rate = 0.0009999999, loss = 0.00019995496, step = 6015 (5.540 sec)\n",
"2021-12-30 09:44:48,301 [INFO] tensorflow: epoch = 62.65625, learning_rate = 0.0009999999, loss = 0.00019995496, step = 6015 (5.540 sec)\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO:tensorflow:global_step/sec: 3.07032\n",
"2021-12-30 09:44:50,248 [INFO] tensorflow: global_step/sec: 3.07032\n",
"2021-12-30 09:44:51,222 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.562\n",
"INFO:tensorflow:global_step/sec: 3.08049\n",
"2021-12-30 09:44:53,169 [INFO] tensorflow: global_step/sec: 3.08049\n",
"INFO:tensorflow:epoch = 62.83333333333333, learning_rate = 0.0009999999, loss = 0.00018319156, step = 6032 (5.513 sec)\n",
"2021-12-30 09:44:53,814 [INFO] tensorflow: epoch = 62.83333333333333, learning_rate = 0.0009999999, loss = 0.00018319156, step = 6032 (5.513 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08536\n",
"2021-12-30 09:44:56,086 [INFO] tensorflow: global_step/sec: 3.08536\n",
"INFO:tensorflow:global_step/sec: 3.05624\n",
"2021-12-30 09:44:59,031 [INFO] tensorflow: global_step/sec: 3.05624\n",
"2021-12-30 09:44:59,032 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 63/120: loss: 0.00019 learning rate: 0.00100 Time taken: 0:00:31.151218 ETA: 0:29:35.619422\n",
"INFO:tensorflow:epoch = 63.010416666666664, learning_rate = 0.0009999999, loss = 0.00023036171, step = 6049 (5.559 sec)\n",
"2021-12-30 09:44:59,373 [INFO] tensorflow: epoch = 63.010416666666664, learning_rate = 0.0009999999, loss = 0.00023036171, step = 6049 (5.559 sec)\n",
"2021-12-30 09:44:59,373 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.538\n",
"INFO:tensorflow:global_step/sec: 3.12069\n",
"2021-12-30 09:45:01,915 [INFO] tensorflow: global_step/sec: 3.12069\n",
"INFO:tensorflow:epoch = 63.1875, learning_rate = 0.0009999999, loss = 0.00023338053, step = 6066 (5.462 sec)\n",
"2021-12-30 09:45:04,835 [INFO] tensorflow: epoch = 63.1875, learning_rate = 0.0009999999, loss = 0.00023338053, step = 6066 (5.462 sec)\n",
"INFO:tensorflow:global_step/sec: 3.0819\n",
"2021-12-30 09:45:04,835 [INFO] tensorflow: global_step/sec: 3.0819\n",
"2021-12-30 09:45:07,425 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.841\n",
"INFO:tensorflow:global_step/sec: 3.08029\n",
"2021-12-30 09:45:07,757 [INFO] tensorflow: global_step/sec: 3.08029\n",
"INFO:tensorflow:epoch = 63.36458333333333, learning_rate = 0.0009999999, loss = 0.00021511305, step = 6083 (5.525 sec)\n",
"2021-12-30 09:45:10,360 [INFO] tensorflow: epoch = 63.36458333333333, learning_rate = 0.0009999999, loss = 0.00021511305, step = 6083 (5.525 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08698\n",
"2021-12-30 09:45:10,673 [INFO] tensorflow: global_step/sec: 3.08698\n",
"INFO:tensorflow:global_step/sec: 3.06608\n",
"2021-12-30 09:45:13,608 [INFO] tensorflow: global_step/sec: 3.06608\n",
"2021-12-30 09:45:15,491 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.793\n",
"INFO:tensorflow:epoch = 63.541666666666664, learning_rate = 0.0009999999, loss = 0.0002792588, step = 6100 (5.449 sec)\n",
"2021-12-30 09:45:15,809 [INFO] tensorflow: epoch = 63.541666666666664, learning_rate = 0.0009999999, loss = 0.0002792588, step = 6100 (5.449 sec)\n",
"INFO:tensorflow:global_step/sec: 3.16557\n",
"2021-12-30 09:45:16,451 [INFO] tensorflow: global_step/sec: 3.16557\n",
"INFO:tensorflow:global_step/sec: 3.01774\n",
"2021-12-30 09:45:19,433 [INFO] tensorflow: global_step/sec: 3.01774\n",
"INFO:tensorflow:epoch = 63.71875, learning_rate = 0.0009999999, loss = 0.00021212832, step = 6117 (5.576 sec)\n",
"2021-12-30 09:45:21,384 [INFO] tensorflow: epoch = 63.71875, learning_rate = 0.0009999999, loss = 0.00021212832, step = 6117 (5.576 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08547\n",
"2021-12-30 09:45:22,350 [INFO] tensorflow: global_step/sec: 3.08547\n",
"2021-12-30 09:45:23,624 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.592\n",
"INFO:tensorflow:global_step/sec: 3.11549\n",
"2021-12-30 09:45:25,239 [INFO] tensorflow: global_step/sec: 3.11549\n",
"INFO:tensorflow:epoch = 63.89583333333333, learning_rate = 0.0009999999, loss = 0.00026226038, step = 6134 (5.448 sec)\n",
"2021-12-30 09:45:26,833 [INFO] tensorflow: epoch = 63.89583333333333, learning_rate = 0.0009999999, loss = 0.00026226038, step = 6134 (5.448 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09664\n",
"2021-12-30 09:45:28,145 [INFO] tensorflow: global_step/sec: 3.09664\n",
"2021-12-30 09:45:30,078 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 64/120: loss: 0.00029 learning rate: 0.00100 Time taken: 0:00:31.042695 ETA: 0:28:58.390923\n",
"INFO:tensorflow:global_step/sec: 3.0799\n",
"2021-12-30 09:45:31,068 [INFO] tensorflow: global_step/sec: 3.0799\n",
"2021-12-30 09:45:31,714 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.723\n",
"INFO:tensorflow:epoch = 64.07291666666666, learning_rate = 0.0009999999, loss = 0.0002012591, step = 6151 (5.560 sec)\n",
"2021-12-30 09:45:32,393 [INFO] tensorflow: epoch = 64.07291666666666, learning_rate = 0.0009999999, loss = 0.0002012591, step = 6151 (5.560 sec)\n",
"INFO:tensorflow:global_step/sec: 3.06041\n",
"2021-12-30 09:45:34,008 [INFO] tensorflow: global_step/sec: 3.06041\n",
"INFO:tensorflow:global_step/sec: 3.07711\n",
"2021-12-30 09:45:36,933 [INFO] tensorflow: global_step/sec: 3.07711\n",
"INFO:tensorflow:epoch = 64.25, learning_rate = 0.0009999999, loss = 0.00018843211, step = 6168 (5.515 sec)\n",
"2021-12-30 09:45:37,908 [INFO] tensorflow: epoch = 64.25, learning_rate = 0.0009999999, loss = 0.00018843211, step = 6168 (5.515 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07953\n",
"2021-12-30 09:45:39,856 [INFO] tensorflow: global_step/sec: 3.07953\n",
"2021-12-30 09:45:39,857 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.564\n",
"INFO:tensorflow:global_step/sec: 3.09885\n",
"2021-12-30 09:45:42,760 [INFO] tensorflow: global_step/sec: 3.09885\n",
"INFO:tensorflow:epoch = 64.42708333333333, learning_rate = 0.0009999999, loss = 0.0003230688, step = 6185 (5.500 sec)\n",
"2021-12-30 09:45:43,409 [INFO] tensorflow: epoch = 64.42708333333333, learning_rate = 0.0009999999, loss = 0.0003230688, step = 6185 (5.500 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07676\n",
"2021-12-30 09:45:45,685 [INFO] tensorflow: global_step/sec: 3.07676\n",
"2021-12-30 09:45:47,964 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.668\n",
"INFO:tensorflow:global_step/sec: 3.0777\n",
"2021-12-30 09:45:48,609 [INFO] tensorflow: global_step/sec: 3.0777\n",
"INFO:tensorflow:epoch = 64.60416666666666, learning_rate = 0.0009999999, loss = 0.00023527002, step = 6202 (5.523 sec)\n",
"2021-12-30 09:45:48,932 [INFO] tensorflow: epoch = 64.60416666666666, learning_rate = 0.0009999999, loss = 0.00023527002, step = 6202 (5.523 sec)\n",
"INFO:tensorflow:global_step/sec: 3.04818\n",
"2021-12-30 09:45:51,562 [INFO] tensorflow: global_step/sec: 3.04818\n",
"INFO:tensorflow:epoch = 64.78125, learning_rate = 0.0009999999, loss = 0.00017779661, step = 6219 (5.526 sec)\n",
"2021-12-30 09:45:54,458 [INFO] tensorflow: epoch = 64.78125, learning_rate = 0.0009999999, loss = 0.00017779661, step = 6219 (5.526 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10707\n",
"2021-12-30 09:45:54,459 [INFO] tensorflow: global_step/sec: 3.10707\n",
"2021-12-30 09:45:56,090 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.615\n",
"INFO:tensorflow:global_step/sec: 3.04484\n",
"2021-12-30 09:45:57,414 [INFO] tensorflow: global_step/sec: 3.04484\n",
"INFO:tensorflow:epoch = 64.95833333333333, learning_rate = 0.0009999999, loss = 0.00028504012, step = 6236 (5.558 sec)\n",
"2021-12-30 09:46:00,016 [INFO] tensorflow: epoch = 64.95833333333333, learning_rate = 0.0009999999, loss = 0.00028504012, step = 6236 (5.558 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07233\n",
"2021-12-30 09:46:00,344 [INFO] tensorflow: global_step/sec: 3.07233\n",
"2021-12-30 09:46:01,329 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 65/120: loss: 0.00024 learning rate: 0.00100 Time taken: 0:00:31.247153 ETA: 0:28:38.593417\n",
"INFO:tensorflow:global_step/sec: 3.1059\n",
"2021-12-30 09:46:03,242 [INFO] tensorflow: global_step/sec: 3.1059\n",
"2021-12-30 09:46:04,213 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.621\n",
"INFO:tensorflow:epoch = 65.13541666666666, learning_rate = 0.0009999999, loss = 0.00033119504, step = 6253 (5.519 sec)\n",
"2021-12-30 09:46:05,535 [INFO] tensorflow: epoch = 65.13541666666666, learning_rate = 0.0009999999, loss = 0.00033119504, step = 6253 (5.519 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09183\n",
"2021-12-30 09:46:06,152 [INFO] tensorflow: global_step/sec: 3.09183\n",
"INFO:tensorflow:global_step/sec: 3.11589\n",
"2021-12-30 09:46:09,041 [INFO] tensorflow: global_step/sec: 3.11589\n",
"INFO:tensorflow:epoch = 65.3125, learning_rate = 0.0009999999, loss = 0.0002511522, step = 6270 (5.480 sec)\n",
"2021-12-30 09:46:11,015 [INFO] tensorflow: epoch = 65.3125, learning_rate = 0.0009999999, loss = 0.0002511522, step = 6270 (5.480 sec)\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO:tensorflow:global_step/sec: 3.10108\n",
"2021-12-30 09:46:11,943 [INFO] tensorflow: global_step/sec: 3.10108\n",
"2021-12-30 09:46:12,260 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.857\n",
"INFO:tensorflow:global_step/sec: 3.06072\n",
"2021-12-30 09:46:14,884 [INFO] tensorflow: global_step/sec: 3.06072\n",
"INFO:tensorflow:epoch = 65.48958333333333, learning_rate = 0.0009999999, loss = 0.00023309077, step = 6287 (5.433 sec)\n",
"2021-12-30 09:46:16,447 [INFO] tensorflow: epoch = 65.48958333333333, learning_rate = 0.0009999999, loss = 0.00023309077, step = 6287 (5.433 sec)\n",
"INFO:tensorflow:global_step/sec: 3.22219\n",
"2021-12-30 09:46:17,677 [INFO] tensorflow: global_step/sec: 3.22219\n",
"2021-12-30 09:46:20,266 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.982\n",
"INFO:tensorflow:global_step/sec: 3.08317\n",
"2021-12-30 09:46:20,596 [INFO] tensorflow: global_step/sec: 3.08317\n",
"INFO:tensorflow:epoch = 65.66666666666666, learning_rate = 0.0009999999, loss = 0.00021256598, step = 6304 (5.443 sec)\n",
"2021-12-30 09:46:21,890 [INFO] tensorflow: epoch = 65.66666666666666, learning_rate = 0.0009999999, loss = 0.00021256598, step = 6304 (5.443 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10562\n",
"2021-12-30 09:46:23,494 [INFO] tensorflow: global_step/sec: 3.10562\n",
"INFO:tensorflow:global_step/sec: 3.04484\n",
"2021-12-30 09:46:26,450 [INFO] tensorflow: global_step/sec: 3.04484\n",
"INFO:tensorflow:epoch = 65.84375, learning_rate = 0.0009999999, loss = 0.00019074464, step = 6321 (5.523 sec)\n",
"2021-12-30 09:46:27,413 [INFO] tensorflow: epoch = 65.84375, learning_rate = 0.0009999999, loss = 0.00019074464, step = 6321 (5.523 sec)\n",
"2021-12-30 09:46:28,393 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.608\n",
"INFO:tensorflow:global_step/sec: 3.07266\n",
"2021-12-30 09:46:29,379 [INFO] tensorflow: global_step/sec: 3.07266\n",
"INFO:tensorflow:global_step/sec: 3.11768\n",
"2021-12-30 09:46:32,265 [INFO] tensorflow: global_step/sec: 3.11768\n",
"2021-12-30 09:46:32,266 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 66/120: loss: 0.00026 learning rate: 0.00100 Time taken: 0:00:30.944899 ETA: 0:27:51.024563\n",
"INFO:tensorflow:epoch = 66.02083333333333, learning_rate = 0.0009999999, loss = 0.00031863144, step = 6338 (5.511 sec)\n",
"2021-12-30 09:46:32,924 [INFO] tensorflow: epoch = 66.02083333333333, learning_rate = 0.0009999999, loss = 0.00031863144, step = 6338 (5.511 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07623\n",
"2021-12-30 09:46:35,191 [INFO] tensorflow: global_step/sec: 3.07623\n",
"2021-12-30 09:46:36,505 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.656\n",
"INFO:tensorflow:global_step/sec: 3.07476\n",
"2021-12-30 09:46:38,118 [INFO] tensorflow: global_step/sec: 3.07476\n",
"INFO:tensorflow:epoch = 66.19791666666666, learning_rate = 0.0009999999, loss = 0.00024005194, step = 6355 (5.541 sec)\n",
"2021-12-30 09:46:38,465 [INFO] tensorflow: epoch = 66.19791666666666, learning_rate = 0.0009999999, loss = 0.00024005194, step = 6355 (5.541 sec)\n",
"INFO:tensorflow:global_step/sec: 3.04878\n",
"2021-12-30 09:46:41,070 [INFO] tensorflow: global_step/sec: 3.04878\n",
"INFO:tensorflow:epoch = 66.375, learning_rate = 0.0009999999, loss = 0.00028774136, step = 6372 (5.520 sec)\n",
"2021-12-30 09:46:43,985 [INFO] tensorflow: epoch = 66.375, learning_rate = 0.0009999999, loss = 0.00028774136, step = 6372 (5.520 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08673\n",
"2021-12-30 09:46:43,986 [INFO] tensorflow: global_step/sec: 3.08673\n",
"2021-12-30 09:46:44,671 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.493\n",
"INFO:tensorflow:global_step/sec: 3.01064\n",
"2021-12-30 09:46:46,975 [INFO] tensorflow: global_step/sec: 3.01064\n",
"INFO:tensorflow:epoch = 66.55208333333333, learning_rate = 0.0009999999, loss = 0.00031464238, step = 6389 (5.578 sec)\n",
"2021-12-30 09:46:49,563 [INFO] tensorflow: epoch = 66.55208333333333, learning_rate = 0.0009999999, loss = 0.00031464238, step = 6389 (5.578 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10844\n",
"2021-12-30 09:46:49,871 [INFO] tensorflow: global_step/sec: 3.10844\n",
"INFO:tensorflow:global_step/sec: 3.09948\n",
"2021-12-30 09:46:52,774 [INFO] tensorflow: global_step/sec: 3.09948\n",
"2021-12-30 09:46:52,775 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.679\n",
"INFO:tensorflow:epoch = 66.72916666666666, learning_rate = 0.0009999999, loss = 0.00024163951, step = 6406 (5.433 sec)\n",
"2021-12-30 09:46:54,996 [INFO] tensorflow: epoch = 66.72916666666666, learning_rate = 0.0009999999, loss = 0.00024163951, step = 6406 (5.433 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12214\n",
"2021-12-30 09:46:55,657 [INFO] tensorflow: global_step/sec: 3.12214\n",
"INFO:tensorflow:global_step/sec: 3.09059\n",
"2021-12-30 09:46:58,569 [INFO] tensorflow: global_step/sec: 3.09059\n",
"INFO:tensorflow:epoch = 66.90625, learning_rate = 0.0009999999, loss = 0.00020179964, step = 6423 (5.508 sec)\n",
"2021-12-30 09:47:00,504 [INFO] tensorflow: epoch = 66.90625, learning_rate = 0.0009999999, loss = 0.00020179964, step = 6423 (5.508 sec)\n",
"2021-12-30 09:47:00,835 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.816\n",
"INFO:tensorflow:global_step/sec: 3.10097\n",
"2021-12-30 09:47:01,471 [INFO] tensorflow: global_step/sec: 3.10097\n",
"2021-12-30 09:47:03,396 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 67/120: loss: 0.00025 learning rate: 0.00100 Time taken: 0:00:31.127548 ETA: 0:27:29.760043\n",
"INFO:tensorflow:global_step/sec: 3.1134\n",
"2021-12-30 09:47:04,362 [INFO] tensorflow: global_step/sec: 3.1134\n",
"INFO:tensorflow:epoch = 67.08333333333333, learning_rate = 0.0009999999, loss = 0.00023302586, step = 6440 (5.421 sec)\n",
"2021-12-30 09:47:05,925 [INFO] tensorflow: epoch = 67.08333333333333, learning_rate = 0.0009999999, loss = 0.00023302586, step = 6440 (5.421 sec)\n",
"INFO:tensorflow:global_step/sec: 3.13345\n",
"2021-12-30 09:47:07,234 [INFO] tensorflow: global_step/sec: 3.13345\n",
"2021-12-30 09:47:08,854 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.940\n",
"INFO:tensorflow:global_step/sec: 3.09503\n",
"2021-12-30 09:47:10,142 [INFO] tensorflow: global_step/sec: 3.09503\n",
"INFO:tensorflow:epoch = 67.26041666666666, learning_rate = 0.0009999999, loss = 0.00017587777, step = 6457 (5.512 sec)\n",
"2021-12-30 09:47:11,437 [INFO] tensorflow: epoch = 67.26041666666666, learning_rate = 0.0009999999, loss = 0.00017587777, step = 6457 (5.512 sec)\n",
"INFO:tensorflow:global_step/sec: 3.0397\n",
"2021-12-30 09:47:13,103 [INFO] tensorflow: global_step/sec: 3.0397\n",
"INFO:tensorflow:global_step/sec: 3.10249\n",
"2021-12-30 09:47:16,004 [INFO] tensorflow: global_step/sec: 3.10249\n",
"INFO:tensorflow:epoch = 67.4375, learning_rate = 0.0009999999, loss = 0.00018511378, step = 6474 (5.505 sec)\n",
"2021-12-30 09:47:16,942 [INFO] tensorflow: epoch = 67.4375, learning_rate = 0.0009999999, loss = 0.00018511378, step = 6474 (5.505 sec)\n",
"2021-12-30 09:47:16,942 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.728\n",
"INFO:tensorflow:global_step/sec: 3.16671\n",
"2021-12-30 09:47:18,846 [INFO] tensorflow: global_step/sec: 3.16671\n",
"INFO:tensorflow:global_step/sec: 3.04197\n",
"2021-12-30 09:47:21,805 [INFO] tensorflow: global_step/sec: 3.04197\n",
"INFO:tensorflow:epoch = 67.61458333333333, learning_rate = 0.0009999999, loss = 0.0002064151, step = 6491 (5.531 sec)\n",
"2021-12-30 09:47:22,474 [INFO] tensorflow: epoch = 67.61458333333333, learning_rate = 0.0009999999, loss = 0.0002064151, step = 6491 (5.531 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07528\n",
"2021-12-30 09:47:24,731 [INFO] tensorflow: global_step/sec: 3.07528\n",
"2021-12-30 09:47:25,049 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.672\n",
"INFO:tensorflow:global_step/sec: 3.06873\n",
"2021-12-30 09:47:27,664 [INFO] tensorflow: global_step/sec: 3.06873\n",
"INFO:tensorflow:epoch = 67.79166666666666, learning_rate = 0.0009999999, loss = 0.00019480378, step = 6508 (5.508 sec)\n",
"2021-12-30 09:47:27,981 [INFO] tensorflow: epoch = 67.79166666666666, learning_rate = 0.0009999999, loss = 0.00019480378, step = 6508 (5.508 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08178\n",
"2021-12-30 09:47:30,584 [INFO] tensorflow: global_step/sec: 3.08178\n",
"2021-12-30 09:47:33,155 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.674\n",
"INFO:tensorflow:epoch = 67.96875, learning_rate = 0.0009999999, loss = 0.00020694101, step = 6525 (5.494 sec)\n",
"2021-12-30 09:47:33,475 [INFO] tensorflow: epoch = 67.96875, learning_rate = 0.0009999999, loss = 0.00020694101, step = 6525 (5.494 sec)\n",
"INFO:tensorflow:global_step/sec: 3.1123\n",
"2021-12-30 09:47:33,476 [INFO] tensorflow: global_step/sec: 3.1123\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"2021-12-30 09:47:34,412 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 68/120: loss: 0.00024 learning rate: 0.00100 Time taken: 0:00:31.017985 ETA: 0:26:52.935201\n",
"INFO:tensorflow:global_step/sec: 3.15991\n",
"2021-12-30 09:47:36,324 [INFO] tensorflow: global_step/sec: 3.15991\n",
"INFO:tensorflow:epoch = 68.14583333333333, learning_rate = 0.0009999999, loss = 0.0002937363, step = 6542 (5.427 sec)\n",
"2021-12-30 09:47:38,902 [INFO] tensorflow: epoch = 68.14583333333333, learning_rate = 0.0009999999, loss = 0.0002937363, step = 6542 (5.427 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10262\n",
"2021-12-30 09:47:39,225 [INFO] tensorflow: global_step/sec: 3.10262\n",
"2021-12-30 09:47:41,155 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.999\n",
"INFO:tensorflow:global_step/sec: 3.11416\n",
"2021-12-30 09:47:42,115 [INFO] tensorflow: global_step/sec: 3.11416\n",
"INFO:tensorflow:epoch = 68.32291666666666, learning_rate = 0.0009999999, loss = 0.00029822148, step = 6559 (5.468 sec)\n",
"2021-12-30 09:47:44,370 [INFO] tensorflow: epoch = 68.32291666666666, learning_rate = 0.0009999999, loss = 0.00029822148, step = 6559 (5.468 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08391\n",
"2021-12-30 09:47:45,033 [INFO] tensorflow: global_step/sec: 3.08391\n",
"INFO:tensorflow:global_step/sec: 3.03952\n",
"2021-12-30 09:47:47,994 [INFO] tensorflow: global_step/sec: 3.03952\n",
"2021-12-30 09:47:49,307 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.537\n",
"INFO:tensorflow:epoch = 68.5, learning_rate = 0.0009999999, loss = 0.000298041, step = 6576 (5.574 sec)\n",
"2021-12-30 09:47:49,944 [INFO] tensorflow: epoch = 68.5, learning_rate = 0.0009999999, loss = 0.000298041, step = 6576 (5.574 sec)\n",
"INFO:tensorflow:global_step/sec: 3.06371\n",
"2021-12-30 09:47:50,932 [INFO] tensorflow: global_step/sec: 3.06371\n",
"INFO:tensorflow:global_step/sec: 3.10353\n",
"2021-12-30 09:47:53,832 [INFO] tensorflow: global_step/sec: 3.10353\n",
"INFO:tensorflow:epoch = 68.67708333333333, learning_rate = 0.0009999999, loss = 0.0003346668, step = 6593 (5.512 sec)\n",
"2021-12-30 09:47:55,456 [INFO] tensorflow: epoch = 68.67708333333333, learning_rate = 0.0009999999, loss = 0.0003346668, step = 6593 (5.512 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08347\n",
"2021-12-30 09:47:56,751 [INFO] tensorflow: global_step/sec: 3.08347\n",
"2021-12-30 09:47:57,389 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.748\n",
"INFO:tensorflow:global_step/sec: 3.11039\n",
"2021-12-30 09:47:59,644 [INFO] tensorflow: global_step/sec: 3.11039\n",
"INFO:tensorflow:epoch = 68.85416666666666, learning_rate = 0.0009999999, loss = 0.00025518393, step = 6610 (5.477 sec)\n",
"2021-12-30 09:48:00,934 [INFO] tensorflow: epoch = 68.85416666666666, learning_rate = 0.0009999999, loss = 0.00025518393, step = 6610 (5.477 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12629\n",
"2021-12-30 09:48:02,523 [INFO] tensorflow: global_step/sec: 3.12629\n",
"INFO:tensorflow:global_step/sec: 3.17817\n",
"2021-12-30 09:48:05,355 [INFO] tensorflow: global_step/sec: 3.17817\n",
"2021-12-30 09:48:05,356 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 69/120: loss: 0.00032 learning rate: 0.00100 Time taken: 0:00:30.940322 ETA: 0:26:17.956418\n",
"2021-12-30 09:48:05,356 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.104\n",
"INFO:tensorflow:epoch = 69.03125, learning_rate = 0.0009999999, loss = 0.0002165939, step = 6627 (5.390 sec)\n",
"2021-12-30 09:48:06,324 [INFO] tensorflow: epoch = 69.03125, learning_rate = 0.0009999999, loss = 0.0002165939, step = 6627 (5.390 sec)\n",
"INFO:tensorflow:global_step/sec: 3.0884\n",
"2021-12-30 09:48:08,269 [INFO] tensorflow: global_step/sec: 3.0884\n",
"INFO:tensorflow:global_step/sec: 3.10878\n",
"2021-12-30 09:48:11,164 [INFO] tensorflow: global_step/sec: 3.10878\n",
"INFO:tensorflow:epoch = 69.20833333333333, learning_rate = 0.0009999999, loss = 0.00034628063, step = 6644 (5.476 sec)\n",
"2021-12-30 09:48:11,800 [INFO] tensorflow: epoch = 69.20833333333333, learning_rate = 0.0009999999, loss = 0.00034628063, step = 6644 (5.476 sec)\n",
"2021-12-30 09:48:13,431 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.769\n",
"INFO:tensorflow:global_step/sec: 3.09109\n",
"2021-12-30 09:48:14,076 [INFO] tensorflow: global_step/sec: 3.09109\n",
"INFO:tensorflow:global_step/sec: 3.05143\n",
"2021-12-30 09:48:17,025 [INFO] tensorflow: global_step/sec: 3.05143\n",
"INFO:tensorflow:epoch = 69.38541666666666, learning_rate = 0.0009999999, loss = 0.00024141677, step = 6661 (5.543 sec)\n",
"2021-12-30 09:48:17,343 [INFO] tensorflow: epoch = 69.38541666666666, learning_rate = 0.0009999999, loss = 0.00024141677, step = 6661 (5.543 sec)\n",
"INFO:tensorflow:global_step/sec: 3.0492\n",
"2021-12-30 09:48:19,977 [INFO] tensorflow: global_step/sec: 3.0492\n",
"2021-12-30 09:48:21,584 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.529\n",
"INFO:tensorflow:epoch = 69.5625, learning_rate = 0.0009999999, loss = 0.00030143856, step = 6678 (5.529 sec)\n",
"2021-12-30 09:48:22,872 [INFO] tensorflow: epoch = 69.5625, learning_rate = 0.0009999999, loss = 0.00030143856, step = 6678 (5.529 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10752\n",
"2021-12-30 09:48:22,873 [INFO] tensorflow: global_step/sec: 3.10752\n",
"INFO:tensorflow:global_step/sec: 3.14972\n",
"2021-12-30 09:48:25,730 [INFO] tensorflow: global_step/sec: 3.14972\n",
"INFO:tensorflow:epoch = 69.73958333333333, learning_rate = 0.0009999999, loss = 0.0002339421, step = 6695 (5.455 sec)\n",
"2021-12-30 09:48:28,327 [INFO] tensorflow: epoch = 69.73958333333333, learning_rate = 0.0009999999, loss = 0.0002339421, step = 6695 (5.455 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08291\n",
"2021-12-30 09:48:28,650 [INFO] tensorflow: global_step/sec: 3.08291\n",
"2021-12-30 09:48:29,608 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.926\n",
"INFO:tensorflow:global_step/sec: 3.12592\n",
"2021-12-30 09:48:31,529 [INFO] tensorflow: global_step/sec: 3.12592\n",
"INFO:tensorflow:epoch = 69.91666666666666, learning_rate = 0.0009999999, loss = 0.0002789432, step = 6712 (5.462 sec)\n",
"2021-12-30 09:48:33,790 [INFO] tensorflow: epoch = 69.91666666666666, learning_rate = 0.0009999999, loss = 0.0002789432, step = 6712 (5.462 sec)\n",
"INFO:tensorflow:global_step/sec: 3.0752\n",
"2021-12-30 09:48:34,455 [INFO] tensorflow: global_step/sec: 3.0752\n",
"INFO:tensorflow:Saving checkpoints for step-6720.\n",
"2021-12-30 09:48:36,053 [INFO] tensorflow: Saving checkpoints for step-6720.\n",
"WARNING:tensorflow:Ignoring: /tmp/tmparh8i37x; No such file or directory\n",
"2021-12-30 09:48:36,219 [WARNING] tensorflow: Ignoring: /tmp/tmparh8i37x; No such file or directory\n",
"2021-12-30 09:48:39,951 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 23, 0.00s/step\n",
"2021-12-30 09:48:41,820 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 23, 0.19s/step\n",
"2021-12-30 09:48:43,625 [INFO] iva.detectnet_v2.evaluation.evaluation: step 20 / 23, 0.18s/step\n",
"Matching predictions to ground truth, class 1/1.: 100%|█| 1839/1839 [00:00<00:00, 14396.84it/s]\n",
"Epoch 70/120\n",
"=========================\n",
"\n",
"Validation cost: 0.000227\n",
"Mean average_precision (in %): 87.7602\n",
"\n",
"class name average precision (in %)\n",
"------------ --------------------------\n",
"car 87.7602\n",
"\n",
"Median Inference Time: 0.018694\n",
"INFO:tensorflow:epoch = 70.0, learning_rate = 0.0009999999, loss = 0.00031668352, step = 6720 (10.839 sec)\n",
"2021-12-30 09:48:44,628 [INFO] tensorflow: epoch = 70.0, learning_rate = 0.0009999999, loss = 0.00031668352, step = 6720 (10.839 sec)\n",
"2021-12-30 09:48:44,628 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 70/120: loss: 0.00032 learning rate: 0.00100 Time taken: 0:00:39.283742 ETA: 0:32:44.187086\n",
"INFO:tensorflow:global_step/sec: 0.809668\n",
"2021-12-30 09:48:45,571 [INFO] tensorflow: global_step/sec: 0.809668\n",
"2021-12-30 09:48:45,874 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 12.296\n",
"INFO:tensorflow:global_step/sec: 3.08526\n",
"2021-12-30 09:48:48,488 [INFO] tensorflow: global_step/sec: 3.08526\n",
"INFO:tensorflow:epoch = 70.17708333333333, learning_rate = 0.0009999999, loss = 0.0003560237, step = 6737 (5.476 sec)\n",
"2021-12-30 09:48:50,104 [INFO] tensorflow: epoch = 70.17708333333333, learning_rate = 0.0009999999, loss = 0.0003560237, step = 6737 (5.476 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11671\n",
"2021-12-30 09:48:51,376 [INFO] tensorflow: global_step/sec: 3.11671\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"2021-12-30 09:48:53,964 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.724\n",
"INFO:tensorflow:global_step/sec: 3.07393\n",
"2021-12-30 09:48:54,304 [INFO] tensorflow: global_step/sec: 3.07393\n",
"INFO:tensorflow:epoch = 70.35416666666666, learning_rate = 0.0009999999, loss = 0.00020334452, step = 6754 (5.508 sec)\n",
"2021-12-30 09:48:55,613 [INFO] tensorflow: epoch = 70.35416666666666, learning_rate = 0.0009999999, loss = 0.00020334452, step = 6754 (5.508 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07583\n",
"2021-12-30 09:48:57,230 [INFO] tensorflow: global_step/sec: 3.07583\n",
"INFO:tensorflow:global_step/sec: 3.0666\n",
"2021-12-30 09:49:00,165 [INFO] tensorflow: global_step/sec: 3.0666\n",
"INFO:tensorflow:epoch = 70.53125, learning_rate = 0.0009999999, loss = 0.00025820165, step = 6771 (5.521 sec)\n",
"2021-12-30 09:49:01,133 [INFO] tensorflow: epoch = 70.53125, learning_rate = 0.0009999999, loss = 0.00025820165, step = 6771 (5.521 sec)\n",
"2021-12-30 09:49:02,086 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.625\n",
"INFO:tensorflow:global_step/sec: 3.12974\n",
"2021-12-30 09:49:03,040 [INFO] tensorflow: global_step/sec: 3.12974\n",
"INFO:tensorflow:global_step/sec: 3.04731\n",
"2021-12-30 09:49:05,994 [INFO] tensorflow: global_step/sec: 3.04731\n",
"INFO:tensorflow:epoch = 70.70833333333333, learning_rate = 0.0009999999, loss = 0.0002431914, step = 6788 (5.528 sec)\n",
"2021-12-30 09:49:06,661 [INFO] tensorflow: epoch = 70.70833333333333, learning_rate = 0.0009999999, loss = 0.0002431914, step = 6788 (5.528 sec)\n",
"INFO:tensorflow:global_step/sec: 3.06543\n",
"2021-12-30 09:49:08,930 [INFO] tensorflow: global_step/sec: 3.06543\n",
"2021-12-30 09:49:10,208 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.623\n",
"INFO:tensorflow:global_step/sec: 3.14125\n",
"2021-12-30 09:49:11,795 [INFO] tensorflow: global_step/sec: 3.14125\n",
"INFO:tensorflow:epoch = 70.88541666666666, learning_rate = 0.0009999999, loss = 0.00021419705, step = 6805 (5.454 sec)\n",
"2021-12-30 09:49:12,115 [INFO] tensorflow: epoch = 70.88541666666666, learning_rate = 0.0009999999, loss = 0.00021419705, step = 6805 (5.454 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10065\n",
"2021-12-30 09:49:14,697 [INFO] tensorflow: global_step/sec: 3.10065\n",
"2021-12-30 09:49:15,634 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 71/120: loss: 0.00029 learning rate: 0.00100 Time taken: 0:00:31.020456 ETA: 0:25:20.002336\n",
"INFO:tensorflow:epoch = 71.0625, learning_rate = 0.0009999999, loss = 0.00025662867, step = 6822 (5.428 sec)\n",
"2021-12-30 09:49:17,543 [INFO] tensorflow: epoch = 71.0625, learning_rate = 0.0009999999, loss = 0.00025662867, step = 6822 (5.428 sec)\n",
"INFO:tensorflow:global_step/sec: 3.16145\n",
"2021-12-30 09:49:17,544 [INFO] tensorflow: global_step/sec: 3.16145\n",
"2021-12-30 09:49:18,193 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.048\n",
"INFO:tensorflow:global_step/sec: 3.09626\n",
"2021-12-30 09:49:20,451 [INFO] tensorflow: global_step/sec: 3.09626\n",
"INFO:tensorflow:epoch = 71.23958333333333, learning_rate = 0.0009999999, loss = 0.000254815, step = 6839 (5.525 sec)\n",
"2021-12-30 09:49:23,068 [INFO] tensorflow: epoch = 71.23958333333333, learning_rate = 0.0009999999, loss = 0.000254815, step = 6839 (5.525 sec)\n",
"INFO:tensorflow:global_step/sec: 3.04802\n",
"2021-12-30 09:49:23,404 [INFO] tensorflow: global_step/sec: 3.04802\n",
"INFO:tensorflow:global_step/sec: 3.03701\n",
"2021-12-30 09:49:26,367 [INFO] tensorflow: global_step/sec: 3.03701\n",
"2021-12-30 09:49:26,368 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.467\n",
"INFO:tensorflow:epoch = 71.41666666666666, learning_rate = 0.0009999999, loss = 0.00020239854, step = 6856 (5.544 sec)\n",
"2021-12-30 09:49:28,612 [INFO] tensorflow: epoch = 71.41666666666666, learning_rate = 0.0009999999, loss = 0.00020239854, step = 6856 (5.544 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10495\n",
"2021-12-30 09:49:29,266 [INFO] tensorflow: global_step/sec: 3.10495\n",
"INFO:tensorflow:global_step/sec: 3.13607\n",
"2021-12-30 09:49:32,135 [INFO] tensorflow: global_step/sec: 3.13607\n",
"INFO:tensorflow:epoch = 71.59375, learning_rate = 0.0009999999, loss = 0.00025023575, step = 6873 (5.577 sec)\n",
"2021-12-30 09:49:34,189 [INFO] tensorflow: epoch = 71.59375, learning_rate = 0.0009999999, loss = 0.00025023575, step = 6873 (5.577 sec)\n",
"2021-12-30 09:49:34,507 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.574\n",
"INFO:tensorflow:global_step/sec: 2.96546\n",
"2021-12-30 09:49:35,170 [INFO] tensorflow: global_step/sec: 2.96546\n",
"INFO:tensorflow:global_step/sec: 3.06831\n",
"2021-12-30 09:49:38,104 [INFO] tensorflow: global_step/sec: 3.06831\n",
"INFO:tensorflow:epoch = 71.77083333333333, learning_rate = 0.0009999999, loss = 0.00021084612, step = 6890 (5.495 sec)\n",
"2021-12-30 09:49:39,684 [INFO] tensorflow: epoch = 71.77083333333333, learning_rate = 0.0009999999, loss = 0.00021084612, step = 6890 (5.495 sec)\n",
"INFO:tensorflow:global_step/sec: 3.16618\n",
"2021-12-30 09:49:40,946 [INFO] tensorflow: global_step/sec: 3.16618\n",
"2021-12-30 09:49:42,593 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.734\n",
"INFO:tensorflow:global_step/sec: 3.04849\n",
"2021-12-30 09:49:43,898 [INFO] tensorflow: global_step/sec: 3.04849\n",
"INFO:tensorflow:epoch = 71.94791666666666, learning_rate = 0.0009999999, loss = 0.00021439599, step = 6907 (5.514 sec)\n",
"2021-12-30 09:49:45,198 [INFO] tensorflow: epoch = 71.94791666666666, learning_rate = 0.0009999999, loss = 0.00021439599, step = 6907 (5.514 sec)\n",
"INFO:tensorflow:global_step/sec: 3.04033\n",
"2021-12-30 09:49:46,859 [INFO] tensorflow: global_step/sec: 3.04033\n",
"2021-12-30 09:49:46,859 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 72/120: loss: 0.00021 learning rate: 0.00100 Time taken: 0:00:31.181016 ETA: 0:24:56.688755\n",
"INFO:tensorflow:global_step/sec: 3.1099\n",
"2021-12-30 09:49:49,753 [INFO] tensorflow: global_step/sec: 3.1099\n",
"INFO:tensorflow:epoch = 72.125, learning_rate = 0.0009999999, loss = 0.00020212546, step = 6924 (5.538 sec)\n",
"2021-12-30 09:49:50,736 [INFO] tensorflow: epoch = 72.125, learning_rate = 0.0009999999, loss = 0.00020212546, step = 6924 (5.538 sec)\n",
"2021-12-30 09:49:50,736 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.561\n",
"INFO:tensorflow:global_step/sec: 3.1083\n",
"2021-12-30 09:49:52,648 [INFO] tensorflow: global_step/sec: 3.1083\n",
"INFO:tensorflow:global_step/sec: 3.05004\n",
"2021-12-30 09:49:55,599 [INFO] tensorflow: global_step/sec: 3.05004\n",
"INFO:tensorflow:epoch = 72.30208333333333, learning_rate = 0.0009999999, loss = 0.00020488664, step = 6941 (5.508 sec)\n",
"2021-12-30 09:49:56,244 [INFO] tensorflow: epoch = 72.30208333333333, learning_rate = 0.0009999999, loss = 0.00020488664, step = 6941 (5.508 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12124\n",
"2021-12-30 09:49:58,482 [INFO] tensorflow: global_step/sec: 3.12124\n",
"2021-12-30 09:49:58,806 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.783\n",
"INFO:tensorflow:global_step/sec: 3.08924\n",
"2021-12-30 09:50:01,396 [INFO] tensorflow: global_step/sec: 3.08924\n",
"INFO:tensorflow:epoch = 72.47916666666666, learning_rate = 0.0009999999, loss = 0.00027907075, step = 6958 (5.477 sec)\n",
"2021-12-30 09:50:01,721 [INFO] tensorflow: epoch = 72.47916666666666, learning_rate = 0.0009999999, loss = 0.00027907075, step = 6958 (5.477 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10873\n",
"2021-12-30 09:50:04,291 [INFO] tensorflow: global_step/sec: 3.10873\n",
"2021-12-30 09:50:06,871 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.800\n",
"INFO:tensorflow:epoch = 72.65625, learning_rate = 0.0009999999, loss = 0.00019706525, step = 6975 (5.477 sec)\n",
"2021-12-30 09:50:07,198 [INFO] tensorflow: epoch = 72.65625, learning_rate = 0.0009999999, loss = 0.00019706525, step = 6975 (5.477 sec)\n",
"INFO:tensorflow:global_step/sec: 3.095\n",
"2021-12-30 09:50:07,199 [INFO] tensorflow: global_step/sec: 3.095\n",
"INFO:tensorflow:global_step/sec: 3.01554\n",
"2021-12-30 09:50:10,183 [INFO] tensorflow: global_step/sec: 3.01554\n",
"INFO:tensorflow:epoch = 72.83333333333333, learning_rate = 0.0009999999, loss = 0.00021091955, step = 6992 (5.573 sec)\n",
"2021-12-30 09:50:12,771 [INFO] tensorflow: epoch = 72.83333333333333, learning_rate = 0.0009999999, loss = 0.00021091955, step = 6992 (5.573 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09617\n",
"2021-12-30 09:50:13,090 [INFO] tensorflow: global_step/sec: 3.09617\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"2021-12-30 09:50:15,043 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.473\n",
"INFO:tensorflow:global_step/sec: 3.07268\n",
"2021-12-30 09:50:16,019 [INFO] tensorflow: global_step/sec: 3.07268\n",
"2021-12-30 09:50:17,952 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 73/120: loss: 0.00018 learning rate: 0.00100 Time taken: 0:00:31.116398 ETA: 0:24:22.470711\n",
"INFO:tensorflow:epoch = 73.01041666666666, learning_rate = 0.0009999999, loss = 0.00023321885, step = 7009 (5.506 sec)\n",
"2021-12-30 09:50:18,277 [INFO] tensorflow: epoch = 73.01041666666666, learning_rate = 0.0009999999, loss = 0.00023321885, step = 7009 (5.506 sec)\n",
"INFO:tensorflow:global_step/sec: 3.1127\n",
"2021-12-30 09:50:18,910 [INFO] tensorflow: global_step/sec: 3.1127\n",
"INFO:tensorflow:global_step/sec: 3.03598\n",
"2021-12-30 09:50:21,875 [INFO] tensorflow: global_step/sec: 3.03598\n",
"2021-12-30 09:50:23,149 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.675\n",
"INFO:tensorflow:epoch = 73.1875, learning_rate = 0.0009999999, loss = 0.00023979762, step = 7026 (5.519 sec)\n",
"2021-12-30 09:50:23,796 [INFO] tensorflow: epoch = 73.1875, learning_rate = 0.0009999999, loss = 0.00023979762, step = 7026 (5.519 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11083\n",
"2021-12-30 09:50:24,768 [INFO] tensorflow: global_step/sec: 3.11083\n",
"INFO:tensorflow:global_step/sec: 3.10956\n",
"2021-12-30 09:50:27,662 [INFO] tensorflow: global_step/sec: 3.10956\n",
"INFO:tensorflow:epoch = 73.36458333333333, learning_rate = 0.0009999999, loss = 0.0003123232, step = 7043 (5.468 sec)\n",
"2021-12-30 09:50:29,264 [INFO] tensorflow: epoch = 73.36458333333333, learning_rate = 0.0009999999, loss = 0.0003123232, step = 7043 (5.468 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08836\n",
"2021-12-30 09:50:30,576 [INFO] tensorflow: global_step/sec: 3.08836\n",
"2021-12-30 09:50:31,230 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.748\n",
"INFO:tensorflow:global_step/sec: 3.078\n",
"2021-12-30 09:50:33,501 [INFO] tensorflow: global_step/sec: 3.078\n",
"INFO:tensorflow:epoch = 73.54166666666666, learning_rate = 0.0009999999, loss = 0.00018041329, step = 7060 (5.556 sec)\n",
"2021-12-30 09:50:34,820 [INFO] tensorflow: epoch = 73.54166666666666, learning_rate = 0.0009999999, loss = 0.00018041329, step = 7060 (5.556 sec)\n",
"INFO:tensorflow:global_step/sec: 3.0068\n",
"2021-12-30 09:50:36,494 [INFO] tensorflow: global_step/sec: 3.0068\n",
"INFO:tensorflow:global_step/sec: 3.06171\n",
"2021-12-30 09:50:39,433 [INFO] tensorflow: global_step/sec: 3.06171\n",
"2021-12-30 09:50:39,434 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.380\n",
"INFO:tensorflow:epoch = 73.71875, learning_rate = 0.0009999999, loss = 0.00020628101, step = 7077 (5.604 sec)\n",
"2021-12-30 09:50:40,424 [INFO] tensorflow: epoch = 73.71875, learning_rate = 0.0009999999, loss = 0.00020628101, step = 7077 (5.604 sec)\n",
"INFO:tensorflow:global_step/sec: 2.98995\n",
"2021-12-30 09:50:42,443 [INFO] tensorflow: global_step/sec: 2.98995\n",
"INFO:tensorflow:global_step/sec: 3.13566\n",
"2021-12-30 09:50:45,313 [INFO] tensorflow: global_step/sec: 3.13566\n",
"INFO:tensorflow:epoch = 73.89583333333333, learning_rate = 0.0009999999, loss = 0.00023889255, step = 7094 (5.547 sec)\n",
"2021-12-30 09:50:45,971 [INFO] tensorflow: epoch = 73.89583333333333, learning_rate = 0.0009999999, loss = 0.00023889255, step = 7094 (5.547 sec)\n",
"2021-12-30 09:50:47,599 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.495\n",
"INFO:tensorflow:global_step/sec: 3.05162\n",
"2021-12-30 09:50:48,263 [INFO] tensorflow: global_step/sec: 3.05162\n",
"2021-12-30 09:50:49,253 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 74/120: loss: 0.00016 learning rate: 0.00100 Time taken: 0:00:31.285368 ETA: 0:23:59.126937\n",
"INFO:tensorflow:global_step/sec: 3.04155\n",
"2021-12-30 09:50:51,222 [INFO] tensorflow: global_step/sec: 3.04155\n",
"INFO:tensorflow:epoch = 74.07291666666666, learning_rate = 0.0009999999, loss = 0.00018499364, step = 7111 (5.571 sec)\n",
"2021-12-30 09:50:51,542 [INFO] tensorflow: epoch = 74.07291666666666, learning_rate = 0.0009999999, loss = 0.00018499364, step = 7111 (5.571 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08462\n",
"2021-12-30 09:50:54,139 [INFO] tensorflow: global_step/sec: 3.08462\n",
"2021-12-30 09:50:55,756 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.520\n",
"INFO:tensorflow:epoch = 74.25, learning_rate = 0.0009999999, loss = 0.00031472062, step = 7128 (5.515 sec)\n",
"2021-12-30 09:50:57,058 [INFO] tensorflow: epoch = 74.25, learning_rate = 0.0009999999, loss = 0.00031472062, step = 7128 (5.515 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08344\n",
"2021-12-30 09:50:57,058 [INFO] tensorflow: global_step/sec: 3.08344\n",
"INFO:tensorflow:global_step/sec: 3.07837\n",
"2021-12-30 09:50:59,982 [INFO] tensorflow: global_step/sec: 3.07837\n",
"INFO:tensorflow:epoch = 74.42708333333333, learning_rate = 0.0009999999, loss = 0.0002435432, step = 7145 (5.472 sec)\n",
"2021-12-30 09:51:02,529 [INFO] tensorflow: epoch = 74.42708333333333, learning_rate = 0.0009999999, loss = 0.0002435432, step = 7145 (5.472 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12549\n",
"2021-12-30 09:51:02,861 [INFO] tensorflow: global_step/sec: 3.12549\n",
"2021-12-30 09:51:03,864 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.669\n",
"INFO:tensorflow:global_step/sec: 3.05058\n",
"2021-12-30 09:51:05,812 [INFO] tensorflow: global_step/sec: 3.05058\n",
"INFO:tensorflow:epoch = 74.60416666666666, learning_rate = 0.0009999999, loss = 0.00018251059, step = 7162 (5.559 sec)\n",
"2021-12-30 09:51:08,089 [INFO] tensorflow: epoch = 74.60416666666666, learning_rate = 0.0009999999, loss = 0.00018251059, step = 7162 (5.559 sec)\n",
"INFO:tensorflow:global_step/sec: 3.0752\n",
"2021-12-30 09:51:08,738 [INFO] tensorflow: global_step/sec: 3.0752\n",
"INFO:tensorflow:global_step/sec: 3.0821\n",
"2021-12-30 09:51:11,658 [INFO] tensorflow: global_step/sec: 3.0821\n",
"2021-12-30 09:51:11,980 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.643\n",
"INFO:tensorflow:epoch = 74.78125, learning_rate = 0.0009999999, loss = 0.0002601573, step = 7179 (5.489 sec)\n",
"2021-12-30 09:51:13,577 [INFO] tensorflow: epoch = 74.78125, learning_rate = 0.0009999999, loss = 0.0002601573, step = 7179 (5.489 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08564\n",
"2021-12-30 09:51:14,575 [INFO] tensorflow: global_step/sec: 3.08564\n",
"INFO:tensorflow:global_step/sec: 3.02962\n",
"2021-12-30 09:51:17,546 [INFO] tensorflow: global_step/sec: 3.02962\n",
"INFO:tensorflow:epoch = 74.95833333333333, learning_rate = 0.0009999999, loss = 0.0002157559, step = 7196 (5.574 sec)\n",
"2021-12-30 09:51:19,152 [INFO] tensorflow: epoch = 74.95833333333333, learning_rate = 0.0009999999, loss = 0.0002157559, step = 7196 (5.574 sec)\n",
"2021-12-30 09:51:20,114 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.590\n",
"INFO:tensorflow:global_step/sec: 3.10508\n",
"2021-12-30 09:51:20,444 [INFO] tensorflow: global_step/sec: 3.10508\n",
"2021-12-30 09:51:20,445 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 75/120: loss: 0.00023 learning rate: 0.00100 Time taken: 0:00:31.193042 ETA: 0:23:23.686870\n",
"INFO:tensorflow:global_step/sec: 3.12189\n",
"2021-12-30 09:51:23,327 [INFO] tensorflow: global_step/sec: 3.12189\n",
"INFO:tensorflow:epoch = 75.13541666666666, learning_rate = 0.0009999999, loss = 0.00021274602, step = 7213 (5.457 sec)\n",
"2021-12-30 09:51:24,608 [INFO] tensorflow: epoch = 75.13541666666666, learning_rate = 0.0009999999, loss = 0.00021274602, step = 7213 (5.457 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11246\n",
"2021-12-30 09:51:26,219 [INFO] tensorflow: global_step/sec: 3.11246\n",
"2021-12-30 09:51:28,151 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.885\n",
"INFO:tensorflow:global_step/sec: 3.09002\n",
"2021-12-30 09:51:29,131 [INFO] tensorflow: global_step/sec: 3.09002\n",
"INFO:tensorflow:epoch = 75.3125, learning_rate = 0.0009999999, loss = 0.00020353991, step = 7230 (5.512 sec)\n",
"2021-12-30 09:51:30,120 [INFO] tensorflow: epoch = 75.3125, learning_rate = 0.0009999999, loss = 0.00020353991, step = 7230 (5.512 sec)\n",
"INFO:tensorflow:global_step/sec: 3.06634\n",
"2021-12-30 09:51:32,066 [INFO] tensorflow: global_step/sec: 3.06634\n",
"INFO:tensorflow:global_step/sec: 3.06211\n",
"2021-12-30 09:51:35,006 [INFO] tensorflow: global_step/sec: 3.06211\n",
"INFO:tensorflow:epoch = 75.48958333333333, learning_rate = 0.0009999999, loss = 0.00018530976, step = 7247 (5.538 sec)\n",
"2021-12-30 09:51:35,658 [INFO] tensorflow: epoch = 75.48958333333333, learning_rate = 0.0009999999, loss = 0.00018530976, step = 7247 (5.538 sec)\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"2021-12-30 09:51:36,349 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.400\n",
"INFO:tensorflow:global_step/sec: 3.02033\n",
"2021-12-30 09:51:37,985 [INFO] tensorflow: global_step/sec: 3.02033\n",
"INFO:tensorflow:global_step/sec: 3.1175\n",
"2021-12-30 09:51:40,872 [INFO] tensorflow: global_step/sec: 3.1175\n",
"INFO:tensorflow:epoch = 75.66666666666666, learning_rate = 0.0009999999, loss = 0.00023226612, step = 7264 (5.548 sec)\n",
"2021-12-30 09:51:41,206 [INFO] tensorflow: epoch = 75.66666666666666, learning_rate = 0.0009999999, loss = 0.00023226612, step = 7264 (5.548 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07759\n",
"2021-12-30 09:51:43,797 [INFO] tensorflow: global_step/sec: 3.07759\n",
"2021-12-30 09:51:44,441 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.716\n",
"INFO:tensorflow:epoch = 75.84375, learning_rate = 0.0009999999, loss = 0.00020319423, step = 7281 (5.482 sec)\n",
"2021-12-30 09:51:46,688 [INFO] tensorflow: epoch = 75.84375, learning_rate = 0.0009999999, loss = 0.00020319423, step = 7281 (5.482 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11144\n",
"2021-12-30 09:51:46,689 [INFO] tensorflow: global_step/sec: 3.11144\n",
"INFO:tensorflow:global_step/sec: 3.073\n",
"2021-12-30 09:51:49,618 [INFO] tensorflow: global_step/sec: 3.073\n",
"2021-12-30 09:51:51,608 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 76/120: loss: 0.00019 learning rate: 0.00100 Time taken: 0:00:31.177002 ETA: 0:22:51.788075\n",
"INFO:tensorflow:epoch = 76.02083333333333, learning_rate = 0.0009999999, loss = 0.00022078335, step = 7298 (5.596 sec)\n",
"2021-12-30 09:51:52,285 [INFO] tensorflow: epoch = 76.02083333333333, learning_rate = 0.0009999999, loss = 0.00022078335, step = 7298 (5.596 sec)\n",
"INFO:tensorflow:global_step/sec: 3.00615\n",
"2021-12-30 09:51:52,612 [INFO] tensorflow: global_step/sec: 3.00615\n",
"2021-12-30 09:51:52,613 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.476\n",
"INFO:tensorflow:global_step/sec: 3.12464\n",
"2021-12-30 09:51:55,492 [INFO] tensorflow: global_step/sec: 3.12464\n",
"INFO:tensorflow:epoch = 76.19791666666666, learning_rate = 0.0009999999, loss = 0.00027532165, step = 7315 (5.494 sec)\n",
"2021-12-30 09:51:57,779 [INFO] tensorflow: epoch = 76.19791666666666, learning_rate = 0.0009999999, loss = 0.00027532165, step = 7315 (5.494 sec)\n",
"INFO:tensorflow:global_step/sec: 3.05343\n",
"2021-12-30 09:51:58,440 [INFO] tensorflow: global_step/sec: 3.05343\n",
"2021-12-30 09:52:00,741 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.607\n",
"INFO:tensorflow:global_step/sec: 3.05457\n",
"2021-12-30 09:52:01,386 [INFO] tensorflow: global_step/sec: 3.05457\n",
"INFO:tensorflow:epoch = 76.375, learning_rate = 0.0009999999, loss = 0.00020701857, step = 7332 (5.560 sec)\n",
"2021-12-30 09:52:03,338 [INFO] tensorflow: epoch = 76.375, learning_rate = 0.0009999999, loss = 0.00020701857, step = 7332 (5.560 sec)\n",
"INFO:tensorflow:global_step/sec: 3.06474\n",
"2021-12-30 09:52:04,323 [INFO] tensorflow: global_step/sec: 3.06474\n",
"INFO:tensorflow:global_step/sec: 3.08848\n",
"2021-12-30 09:52:07,237 [INFO] tensorflow: global_step/sec: 3.08848\n",
"INFO:tensorflow:epoch = 76.55208333333333, learning_rate = 0.0009999999, loss = 0.0001945587, step = 7349 (5.516 sec)\n",
"2021-12-30 09:52:08,855 [INFO] tensorflow: epoch = 76.55208333333333, learning_rate = 0.0009999999, loss = 0.0001945587, step = 7349 (5.516 sec)\n",
"2021-12-30 09:52:08,855 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.648\n",
"INFO:tensorflow:global_step/sec: 3.10721\n",
"2021-12-30 09:52:10,133 [INFO] tensorflow: global_step/sec: 3.10721\n",
"INFO:tensorflow:global_step/sec: 3.06824\n",
"2021-12-30 09:52:13,067 [INFO] tensorflow: global_step/sec: 3.06824\n",
"INFO:tensorflow:epoch = 76.72916666666666, learning_rate = 0.0009999999, loss = 0.0002061867, step = 7366 (5.503 sec)\n",
"2021-12-30 09:52:14,357 [INFO] tensorflow: epoch = 76.72916666666666, learning_rate = 0.0009999999, loss = 0.0002061867, step = 7366 (5.503 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10851\n",
"2021-12-30 09:52:15,962 [INFO] tensorflow: global_step/sec: 3.10851\n",
"2021-12-30 09:52:16,928 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.776\n",
"INFO:tensorflow:global_step/sec: 3.06225\n",
"2021-12-30 09:52:18,901 [INFO] tensorflow: global_step/sec: 3.06225\n",
"INFO:tensorflow:epoch = 76.90625, learning_rate = 0.0009999999, loss = 0.00017444692, step = 7383 (5.516 sec)\n",
"2021-12-30 09:52:19,873 [INFO] tensorflow: epoch = 76.90625, learning_rate = 0.0009999999, loss = 0.00017444692, step = 7383 (5.516 sec)\n",
"INFO:tensorflow:global_step/sec: 3.05317\n",
"2021-12-30 09:52:21,849 [INFO] tensorflow: global_step/sec: 3.05317\n",
"2021-12-30 09:52:22,813 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 77/120: loss: 0.00022 learning rate: 0.00100 Time taken: 0:00:31.199568 ETA: 0:22:21.581415\n",
"INFO:tensorflow:global_step/sec: 2.99872\n",
"2021-12-30 09:52:24,850 [INFO] tensorflow: global_step/sec: 2.99872\n",
"2021-12-30 09:52:25,178 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.241\n",
"INFO:tensorflow:epoch = 77.08333333333333, learning_rate = 0.0009999999, loss = 0.0001916799, step = 7400 (5.625 sec)\n",
"2021-12-30 09:52:25,499 [INFO] tensorflow: epoch = 77.08333333333333, learning_rate = 0.0009999999, loss = 0.0001916799, step = 7400 (5.625 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07133\n",
"2021-12-30 09:52:27,780 [INFO] tensorflow: global_step/sec: 3.07133\n",
"INFO:tensorflow:global_step/sec: 3.08694\n",
"2021-12-30 09:52:30,696 [INFO] tensorflow: global_step/sec: 3.08694\n",
"INFO:tensorflow:epoch = 77.26041666666666, learning_rate = 0.0009999999, loss = 0.00019160559, step = 7417 (5.537 sec)\n",
"2021-12-30 09:52:31,035 [INFO] tensorflow: epoch = 77.26041666666666, learning_rate = 0.0009999999, loss = 0.00019160559, step = 7417 (5.537 sec)\n",
"2021-12-30 09:52:33,331 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.534\n",
"INFO:tensorflow:global_step/sec: 3.04503\n",
"2021-12-30 09:52:33,651 [INFO] tensorflow: global_step/sec: 3.04503\n",
"INFO:tensorflow:epoch = 77.4375, learning_rate = 0.0009999999, loss = 0.00014427581, step = 7434 (5.552 sec)\n",
"2021-12-30 09:52:36,588 [INFO] tensorflow: epoch = 77.4375, learning_rate = 0.0009999999, loss = 0.00014427581, step = 7434 (5.552 sec)\n",
"INFO:tensorflow:global_step/sec: 3.06409\n",
"2021-12-30 09:52:36,589 [INFO] tensorflow: global_step/sec: 3.06409\n",
"INFO:tensorflow:global_step/sec: 3.06053\n",
"2021-12-30 09:52:39,529 [INFO] tensorflow: global_step/sec: 3.06053\n",
"2021-12-30 09:52:41,496 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.494\n",
"INFO:tensorflow:epoch = 77.61458333333333, learning_rate = 0.0009999999, loss = 0.00024838658, step = 7451 (5.553 sec)\n",
"2021-12-30 09:52:42,141 [INFO] tensorflow: epoch = 77.61458333333333, learning_rate = 0.0009999999, loss = 0.00024838658, step = 7451 (5.553 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07\n",
"2021-12-30 09:52:42,461 [INFO] tensorflow: global_step/sec: 3.07\n",
"INFO:tensorflow:global_step/sec: 3.06498\n",
"2021-12-30 09:52:45,397 [INFO] tensorflow: global_step/sec: 3.06498\n",
"INFO:tensorflow:epoch = 77.79166666666666, learning_rate = 0.0009999999, loss = 0.00023166003, step = 7468 (5.495 sec)\n",
"2021-12-30 09:52:47,636 [INFO] tensorflow: epoch = 77.79166666666666, learning_rate = 0.0009999999, loss = 0.00023166003, step = 7468 (5.495 sec)\n",
"INFO:tensorflow:global_step/sec: 3.14114\n",
"2021-12-30 09:52:48,262 [INFO] tensorflow: global_step/sec: 3.14114\n",
"2021-12-30 09:52:49,555 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.820\n",
"INFO:tensorflow:global_step/sec: 3.0799\n",
"2021-12-30 09:52:51,185 [INFO] tensorflow: global_step/sec: 3.0799\n",
"INFO:tensorflow:epoch = 77.96875, learning_rate = 0.0009999999, loss = 0.0002486659, step = 7485 (5.493 sec)\n",
"2021-12-30 09:52:53,129 [INFO] tensorflow: epoch = 77.96875, learning_rate = 0.0009999999, loss = 0.0002486659, step = 7485 (5.493 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10283\n",
"2021-12-30 09:52:54,085 [INFO] tensorflow: global_step/sec: 3.10283\n",
"2021-12-30 09:52:54,086 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 78/120: loss: 0.00021 learning rate: 0.00100 Time taken: 0:00:31.294577 ETA: 0:21:54.372239\n",
"INFO:tensorflow:global_step/sec: 3.13265\n",
"2021-12-30 09:52:56,958 [INFO] tensorflow: global_step/sec: 3.13265\n",
"2021-12-30 09:52:57,599 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.862\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO:tensorflow:epoch = 78.14583333333333, learning_rate = 0.0009999999, loss = 0.00023187764, step = 7502 (5.534 sec)\n",
"2021-12-30 09:52:58,664 [INFO] tensorflow: epoch = 78.14583333333333, learning_rate = 0.0009999999, loss = 0.00023187764, step = 7502 (5.534 sec)\n",
"INFO:tensorflow:global_step/sec: 2.98027\n",
"2021-12-30 09:52:59,978 [INFO] tensorflow: global_step/sec: 2.98027\n",
"INFO:tensorflow:global_step/sec: 3.06928\n",
"2021-12-30 09:53:02,910 [INFO] tensorflow: global_step/sec: 3.06928\n",
"INFO:tensorflow:epoch = 78.32291666666666, learning_rate = 0.0009999999, loss = 0.00016897186, step = 7519 (5.515 sec)\n",
"2021-12-30 09:53:04,179 [INFO] tensorflow: epoch = 78.32291666666666, learning_rate = 0.0009999999, loss = 0.00016897186, step = 7519 (5.515 sec)\n",
"INFO:tensorflow:global_step/sec: 3.14681\n",
"2021-12-30 09:53:05,770 [INFO] tensorflow: global_step/sec: 3.14681\n",
"2021-12-30 09:53:05,771 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.476\n",
"INFO:tensorflow:global_step/sec: 3.09638\n",
"2021-12-30 09:53:08,677 [INFO] tensorflow: global_step/sec: 3.09638\n",
"INFO:tensorflow:epoch = 78.5, learning_rate = 0.0009999999, loss = 0.00015003339, step = 7536 (5.472 sec)\n",
"2021-12-30 09:53:09,650 [INFO] tensorflow: epoch = 78.5, learning_rate = 0.0009999999, loss = 0.00015003339, step = 7536 (5.472 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10959\n",
"2021-12-30 09:53:11,571 [INFO] tensorflow: global_step/sec: 3.10959\n",
"2021-12-30 09:53:13,805 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.895\n",
"INFO:tensorflow:global_step/sec: 3.14589\n",
"2021-12-30 09:53:14,432 [INFO] tensorflow: global_step/sec: 3.14589\n",
"INFO:tensorflow:epoch = 78.67708333333333, learning_rate = 0.0009999999, loss = 0.00021886238, step = 7553 (5.426 sec)\n",
"2021-12-30 09:53:15,076 [INFO] tensorflow: epoch = 78.67708333333333, learning_rate = 0.0009999999, loss = 0.00021886238, step = 7553 (5.426 sec)\n",
"INFO:tensorflow:global_step/sec: 3.0862\n",
"2021-12-30 09:53:17,348 [INFO] tensorflow: global_step/sec: 3.0862\n",
"INFO:tensorflow:global_step/sec: 3.01861\n",
"2021-12-30 09:53:20,330 [INFO] tensorflow: global_step/sec: 3.01861\n",
"INFO:tensorflow:epoch = 78.85416666666666, learning_rate = 0.0009999999, loss = 0.0002045885, step = 7570 (5.572 sec)\n",
"2021-12-30 09:53:20,649 [INFO] tensorflow: epoch = 78.85416666666666, learning_rate = 0.0009999999, loss = 0.0002045885, step = 7570 (5.572 sec)\n",
"2021-12-30 09:53:21,914 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.666\n",
"INFO:tensorflow:global_step/sec: 3.13486\n",
"2021-12-30 09:53:23,201 [INFO] tensorflow: global_step/sec: 3.13486\n",
"2021-12-30 09:53:25,130 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 79/120: loss: 0.00019 learning rate: 0.00100 Time taken: 0:00:31.024963 ETA: 0:21:12.023479\n",
"INFO:tensorflow:epoch = 79.03125, learning_rate = 0.0009999999, loss = 0.00027470017, step = 7587 (5.463 sec)\n",
"2021-12-30 09:53:26,111 [INFO] tensorflow: epoch = 79.03125, learning_rate = 0.0009999999, loss = 0.00027470017, step = 7587 (5.463 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09138\n",
"2021-12-30 09:53:26,112 [INFO] tensorflow: global_step/sec: 3.09138\n",
"INFO:tensorflow:global_step/sec: 3.08675\n",
"2021-12-30 09:53:29,028 [INFO] tensorflow: global_step/sec: 3.08675\n",
"2021-12-30 09:53:29,996 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.747\n",
"INFO:tensorflow:epoch = 79.20833333333333, learning_rate = 0.0009999999, loss = 0.00023961505, step = 7604 (5.474 sec)\n",
"2021-12-30 09:53:31,586 [INFO] tensorflow: epoch = 79.20833333333333, learning_rate = 0.0009999999, loss = 0.00023961505, step = 7604 (5.474 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09512\n",
"2021-12-30 09:53:31,936 [INFO] tensorflow: global_step/sec: 3.09512\n",
"INFO:tensorflow:global_step/sec: 3.0154\n",
"2021-12-30 09:53:34,920 [INFO] tensorflow: global_step/sec: 3.0154\n",
"INFO:tensorflow:epoch = 79.38541666666666, learning_rate = 0.0009999999, loss = 0.00016730161, step = 7621 (5.596 sec)\n",
"2021-12-30 09:53:37,181 [INFO] tensorflow: epoch = 79.38541666666666, learning_rate = 0.0009999999, loss = 0.00016730161, step = 7621 (5.596 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10668\n",
"2021-12-30 09:53:37,817 [INFO] tensorflow: global_step/sec: 3.10668\n",
"2021-12-30 09:53:38,154 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.517\n",
"INFO:tensorflow:global_step/sec: 3.0697\n",
"2021-12-30 09:53:40,749 [INFO] tensorflow: global_step/sec: 3.0697\n",
"INFO:tensorflow:epoch = 79.5625, learning_rate = 0.0009999999, loss = 0.00023215264, step = 7638 (5.526 sec)\n",
"2021-12-30 09:53:42,708 [INFO] tensorflow: epoch = 79.5625, learning_rate = 0.0009999999, loss = 0.00023215264, step = 7638 (5.526 sec)\n",
"INFO:tensorflow:global_step/sec: 3.0756\n",
"2021-12-30 09:53:43,675 [INFO] tensorflow: global_step/sec: 3.0756\n",
"2021-12-30 09:53:46,268 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.648\n",
"INFO:tensorflow:global_step/sec: 3.09525\n",
"2021-12-30 09:53:46,583 [INFO] tensorflow: global_step/sec: 3.09525\n",
"INFO:tensorflow:epoch = 79.73958333333333, learning_rate = 0.0009999999, loss = 0.0002811979, step = 7655 (5.490 sec)\n",
"2021-12-30 09:53:48,197 [INFO] tensorflow: epoch = 79.73958333333333, learning_rate = 0.0009999999, loss = 0.0002811979, step = 7655 (5.490 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07194\n",
"2021-12-30 09:53:49,513 [INFO] tensorflow: global_step/sec: 3.07194\n",
"INFO:tensorflow:global_step/sec: 2.98078\n",
"2021-12-30 09:53:52,532 [INFO] tensorflow: global_step/sec: 2.98078\n",
"INFO:tensorflow:epoch = 79.91666666666666, learning_rate = 0.0009999999, loss = 0.00021133633, step = 7672 (5.606 sec)\n",
"2021-12-30 09:53:53,803 [INFO] tensorflow: epoch = 79.91666666666666, learning_rate = 0.0009999999, loss = 0.00021133633, step = 7672 (5.606 sec)\n",
"2021-12-30 09:53:54,453 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.436\n",
"INFO:tensorflow:global_step/sec: 3.14814\n",
"2021-12-30 09:53:55,391 [INFO] tensorflow: global_step/sec: 3.14814\n",
"INFO:tensorflow:Saving checkpoints for step-7680.\n",
"2021-12-30 09:53:56,032 [INFO] tensorflow: Saving checkpoints for step-7680.\n",
"WARNING:tensorflow:Ignoring: /tmp/tmpqbmshc4a; No such file or directory\n",
"2021-12-30 09:53:56,213 [WARNING] tensorflow: Ignoring: /tmp/tmpqbmshc4a; No such file or directory\n",
"2021-12-30 09:53:59,652 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 23, 0.00s/step\n",
"2021-12-30 09:54:01,293 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 23, 0.16s/step\n",
"2021-12-30 09:54:02,896 [INFO] iva.detectnet_v2.evaluation.evaluation: step 20 / 23, 0.16s/step\n",
"Matching predictions to ground truth, class 1/1.: 100%|█| 1374/1374 [00:00<00:00, 15996.53it/s]\n",
"Epoch 80/120\n",
"=========================\n",
"\n",
"Validation cost: 0.000223\n",
"Mean average_precision (in %): 82.0705\n",
"\n",
"class name average precision (in %)\n",
"------------ --------------------------\n",
"car 82.0705\n",
"\n",
"Median Inference Time: 0.015363\n",
"INFO:tensorflow:epoch = 80.0, learning_rate = 0.0009999999, loss = 0.00022284444, step = 7680 (9.981 sec)\n",
"2021-12-30 09:54:03,784 [INFO] tensorflow: epoch = 80.0, learning_rate = 0.0009999999, loss = 0.00022284444, step = 7680 (9.981 sec)\n",
"2021-12-30 09:54:03,784 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 80/120: loss: 0.00022 learning rate: 0.00100 Time taken: 0:00:38.657854 ETA: 0:25:46.314173\n",
"INFO:tensorflow:global_step/sec: 0.869295\n",
"2021-12-30 09:54:05,744 [INFO] tensorflow: global_step/sec: 0.869295\n",
"INFO:tensorflow:global_step/sec: 3.06145\n",
"2021-12-30 09:54:08,684 [INFO] tensorflow: global_step/sec: 3.06145\n",
"INFO:tensorflow:epoch = 80.17708333333333, learning_rate = 0.0009999999, loss = 0.00020570552, step = 7697 (5.540 sec)\n",
"2021-12-30 09:54:09,324 [INFO] tensorflow: epoch = 80.17708333333333, learning_rate = 0.0009999999, loss = 0.00020570552, step = 7697 (5.540 sec)\n",
"2021-12-30 09:54:09,987 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 12.875\n",
"INFO:tensorflow:global_step/sec: 3.0547\n",
"2021-12-30 09:54:11,630 [INFO] tensorflow: global_step/sec: 3.0547\n",
"INFO:tensorflow:global_step/sec: 3.0813\n",
"2021-12-30 09:54:14,551 [INFO] tensorflow: global_step/sec: 3.0813\n",
"INFO:tensorflow:epoch = 80.35416666666666, learning_rate = 0.0009999999, loss = 0.0002736857, step = 7714 (5.565 sec)\n",
"2021-12-30 09:54:14,889 [INFO] tensorflow: epoch = 80.35416666666666, learning_rate = 0.0009999999, loss = 0.0002736857, step = 7714 (5.565 sec)\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO:tensorflow:global_step/sec: 3.06789\n",
"2021-12-30 09:54:17,485 [INFO] tensorflow: global_step/sec: 3.06789\n",
"2021-12-30 09:54:18,117 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.602\n",
"INFO:tensorflow:epoch = 80.53125, learning_rate = 0.0009999999, loss = 0.00026451604, step = 7731 (5.505 sec)\n",
"2021-12-30 09:54:20,394 [INFO] tensorflow: epoch = 80.53125, learning_rate = 0.0009999999, loss = 0.00026451604, step = 7731 (5.505 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09269\n",
"2021-12-30 09:54:20,395 [INFO] tensorflow: global_step/sec: 3.09269\n",
"INFO:tensorflow:global_step/sec: 3.0809\n",
"2021-12-30 09:54:23,316 [INFO] tensorflow: global_step/sec: 3.0809\n",
"INFO:tensorflow:epoch = 80.70833333333333, learning_rate = 0.0009999999, loss = 0.00019114255, step = 7748 (5.551 sec)\n",
"2021-12-30 09:54:25,945 [INFO] tensorflow: epoch = 80.70833333333333, learning_rate = 0.0009999999, loss = 0.00019114255, step = 7748 (5.551 sec)\n",
"INFO:tensorflow:global_step/sec: 3.03202\n",
"2021-12-30 09:54:26,284 [INFO] tensorflow: global_step/sec: 3.03202\n",
"2021-12-30 09:54:26,285 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.487\n",
"INFO:tensorflow:global_step/sec: 3.06307\n",
"2021-12-30 09:54:29,223 [INFO] tensorflow: global_step/sec: 3.06307\n",
"INFO:tensorflow:epoch = 80.88541666666666, learning_rate = 0.0009999999, loss = 0.00021982522, step = 7765 (5.526 sec)\n",
"2021-12-30 09:54:31,471 [INFO] tensorflow: epoch = 80.88541666666666, learning_rate = 0.0009999999, loss = 0.00021982522, step = 7765 (5.526 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12207\n",
"2021-12-30 09:54:32,105 [INFO] tensorflow: global_step/sec: 3.12207\n",
"2021-12-30 09:54:34,385 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.693\n",
"INFO:tensorflow:global_step/sec: 3.04821\n",
"2021-12-30 09:54:35,058 [INFO] tensorflow: global_step/sec: 3.04821\n",
"2021-12-30 09:54:35,059 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 81/120: loss: 0.00020 learning rate: 0.00100 Time taken: 0:00:31.260716 ETA: 0:20:19.167913\n",
"INFO:tensorflow:epoch = 81.0625, learning_rate = 0.0009999999, loss = 0.00020432618, step = 7782 (5.513 sec)\n",
"2021-12-30 09:54:36,984 [INFO] tensorflow: epoch = 81.0625, learning_rate = 0.0009999999, loss = 0.00020432618, step = 7782 (5.513 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09445\n",
"2021-12-30 09:54:37,966 [INFO] tensorflow: global_step/sec: 3.09445\n",
"INFO:tensorflow:global_step/sec: 3.00147\n",
"2021-12-30 09:54:40,965 [INFO] tensorflow: global_step/sec: 3.00147\n",
"INFO:tensorflow:epoch = 81.23958333333333, learning_rate = 0.0009999999, loss = 0.00023844454, step = 7799 (5.592 sec)\n",
"2021-12-30 09:54:42,576 [INFO] tensorflow: epoch = 81.23958333333333, learning_rate = 0.0009999999, loss = 0.00023844454, step = 7799 (5.592 sec)\n",
"2021-12-30 09:54:42,576 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.418\n",
"INFO:tensorflow:global_step/sec: 3.08368\n",
"2021-12-30 09:54:43,883 [INFO] tensorflow: global_step/sec: 3.08368\n",
"INFO:tensorflow:global_step/sec: 3.0571\n",
"2021-12-30 09:54:46,827 [INFO] tensorflow: global_step/sec: 3.0571\n",
"INFO:tensorflow:epoch = 81.41666666666666, learning_rate = 0.0009999999, loss = 0.00019761972, step = 7816 (5.563 sec)\n",
"2021-12-30 09:54:48,138 [INFO] tensorflow: epoch = 81.41666666666666, learning_rate = 0.0009999999, loss = 0.00019761972, step = 7816 (5.563 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07136\n",
"2021-12-30 09:54:49,758 [INFO] tensorflow: global_step/sec: 3.07136\n",
"2021-12-30 09:54:50,739 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.500\n",
"INFO:tensorflow:global_step/sec: 3.10696\n",
"2021-12-30 09:54:52,654 [INFO] tensorflow: global_step/sec: 3.10696\n",
"INFO:tensorflow:epoch = 81.59375, learning_rate = 0.0009999999, loss = 0.0002200071, step = 7833 (5.500 sec)\n",
"2021-12-30 09:54:53,638 [INFO] tensorflow: epoch = 81.59375, learning_rate = 0.0009999999, loss = 0.0002200071, step = 7833 (5.500 sec)\n",
"INFO:tensorflow:global_step/sec: 3.04281\n",
"2021-12-30 09:54:55,612 [INFO] tensorflow: global_step/sec: 3.04281\n",
"INFO:tensorflow:global_step/sec: 3.1268\n",
"2021-12-30 09:54:58,491 [INFO] tensorflow: global_step/sec: 3.1268\n",
"2021-12-30 09:54:58,816 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.764\n",
"INFO:tensorflow:epoch = 81.77083333333333, learning_rate = 0.0009999999, loss = 0.00022027601, step = 7850 (5.495 sec)\n",
"2021-12-30 09:54:59,132 [INFO] tensorflow: epoch = 81.77083333333333, learning_rate = 0.0009999999, loss = 0.00022027601, step = 7850 (5.495 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09578\n",
"2021-12-30 09:55:01,398 [INFO] tensorflow: global_step/sec: 3.09578\n",
"INFO:tensorflow:global_step/sec: 3.06405\n",
"2021-12-30 09:55:04,335 [INFO] tensorflow: global_step/sec: 3.06405\n",
"INFO:tensorflow:epoch = 81.94791666666666, learning_rate = 0.0009999999, loss = 0.00025875698, step = 7867 (5.507 sec)\n",
"2021-12-30 09:55:04,640 [INFO] tensorflow: epoch = 81.94791666666666, learning_rate = 0.0009999999, loss = 0.00025875698, step = 7867 (5.507 sec)\n",
"2021-12-30 09:55:06,281 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 82/120: loss: 0.00024 learning rate: 0.00100 Time taken: 0:00:31.245263 ETA: 0:19:47.320007\n",
"2021-12-30 09:55:06,934 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.638\n",
"INFO:tensorflow:global_step/sec: 3.08427\n",
"2021-12-30 09:55:07,253 [INFO] tensorflow: global_step/sec: 3.08427\n",
"INFO:tensorflow:epoch = 82.125, learning_rate = 0.0009999999, loss = 0.00019990659, step = 7884 (5.502 sec)\n",
"2021-12-30 09:55:10,142 [INFO] tensorflow: epoch = 82.125, learning_rate = 0.0009999999, loss = 0.00019990659, step = 7884 (5.502 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11426\n",
"2021-12-30 09:55:10,143 [INFO] tensorflow: global_step/sec: 3.11426\n",
"INFO:tensorflow:global_step/sec: 3.0543\n",
"2021-12-30 09:55:13,090 [INFO] tensorflow: global_step/sec: 3.0543\n",
"2021-12-30 09:55:15,123 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.424\n",
"INFO:tensorflow:epoch = 82.30208333333333, learning_rate = 0.0009999999, loss = 0.00025245533, step = 7901 (5.618 sec)\n",
"2021-12-30 09:55:15,760 [INFO] tensorflow: epoch = 82.30208333333333, learning_rate = 0.0009999999, loss = 0.00025245533, step = 7901 (5.618 sec)\n",
"INFO:tensorflow:global_step/sec: 3.02731\n",
"2021-12-30 09:55:16,063 [INFO] tensorflow: global_step/sec: 3.02731\n",
"INFO:tensorflow:global_step/sec: 3.13552\n",
"2021-12-30 09:55:18,933 [INFO] tensorflow: global_step/sec: 3.13552\n",
"INFO:tensorflow:epoch = 82.47916666666666, learning_rate = 0.0009999999, loss = 0.0002477299, step = 7918 (5.436 sec)\n",
"2021-12-30 09:55:21,196 [INFO] tensorflow: epoch = 82.47916666666666, learning_rate = 0.0009999999, loss = 0.0002477299, step = 7918 (5.436 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10155\n",
"2021-12-30 09:55:21,835 [INFO] tensorflow: global_step/sec: 3.10155\n",
"2021-12-30 09:55:23,115 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.027\n",
"INFO:tensorflow:global_step/sec: 3.08968\n",
"2021-12-30 09:55:24,748 [INFO] tensorflow: global_step/sec: 3.08968\n",
"INFO:tensorflow:epoch = 82.65625, learning_rate = 0.0009999999, loss = 0.00021667572, step = 7935 (5.527 sec)\n",
"2021-12-30 09:55:26,723 [INFO] tensorflow: epoch = 82.65625, learning_rate = 0.0009999999, loss = 0.00021667572, step = 7935 (5.527 sec)\n",
"INFO:tensorflow:global_step/sec: 3.06188\n",
"2021-12-30 09:55:27,687 [INFO] tensorflow: global_step/sec: 3.06188\n",
"INFO:tensorflow:global_step/sec: 3.06471\n",
"2021-12-30 09:55:30,624 [INFO] tensorflow: global_step/sec: 3.06471\n",
"2021-12-30 09:55:31,274 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.514\n",
"INFO:tensorflow:epoch = 82.83333333333333, learning_rate = 0.0009999999, loss = 0.0002218005, step = 7952 (5.493 sec)\n",
"2021-12-30 09:55:32,216 [INFO] tensorflow: epoch = 82.83333333333333, learning_rate = 0.0009999999, loss = 0.0002218005, step = 7952 (5.493 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10719\n",
"2021-12-30 09:55:33,520 [INFO] tensorflow: global_step/sec: 3.10719\n",
"INFO:tensorflow:global_step/sec: 3.13314\n",
"2021-12-30 09:55:36,393 [INFO] tensorflow: global_step/sec: 3.13314\n",
"2021-12-30 09:55:37,373 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 83/120: loss: 0.00021 learning rate: 0.00100 Time taken: 0:00:31.075872 ETA: 0:19:09.807253\n",
"INFO:tensorflow:epoch = 83.01041666666666, learning_rate = 0.0009999999, loss = 0.00025370484, step = 7969 (5.496 sec)\n",
"2021-12-30 09:55:37,713 [INFO] tensorflow: epoch = 83.01041666666666, learning_rate = 0.0009999999, loss = 0.00025370484, step = 7969 (5.496 sec)\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO:tensorflow:global_step/sec: 3.08252\n",
"2021-12-30 09:55:39,312 [INFO] tensorflow: global_step/sec: 3.08252\n",
"2021-12-30 09:55:39,313 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.878\n",
"INFO:tensorflow:global_step/sec: 3.09959\n",
"2021-12-30 09:55:42,216 [INFO] tensorflow: global_step/sec: 3.09959\n",
"INFO:tensorflow:epoch = 83.1875, learning_rate = 0.0009999999, loss = 0.0002347774, step = 7986 (5.461 sec)\n",
"2021-12-30 09:55:43,174 [INFO] tensorflow: epoch = 83.1875, learning_rate = 0.0009999999, loss = 0.0002347774, step = 7986 (5.461 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08418\n",
"2021-12-30 09:55:45,134 [INFO] tensorflow: global_step/sec: 3.08418\n",
"2021-12-30 09:55:47,359 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.857\n",
"INFO:tensorflow:global_step/sec: 3.13967\n",
"2021-12-30 09:55:48,001 [INFO] tensorflow: global_step/sec: 3.13967\n",
"INFO:tensorflow:epoch = 83.36458333333333, learning_rate = 0.0009999999, loss = 0.00018955772, step = 8003 (5.522 sec)\n",
"2021-12-30 09:55:48,696 [INFO] tensorflow: epoch = 83.36458333333333, learning_rate = 0.0009999999, loss = 0.00018955772, step = 8003 (5.522 sec)\n",
"INFO:tensorflow:global_step/sec: 3.01395\n",
"2021-12-30 09:55:50,987 [INFO] tensorflow: global_step/sec: 3.01395\n",
"INFO:tensorflow:global_step/sec: 3.10784\n",
"2021-12-30 09:55:53,883 [INFO] tensorflow: global_step/sec: 3.10784\n",
"INFO:tensorflow:epoch = 83.54166666666666, learning_rate = 0.0009999999, loss = 0.00024020017, step = 8020 (5.520 sec)\n",
"2021-12-30 09:55:54,216 [INFO] tensorflow: epoch = 83.54166666666666, learning_rate = 0.0009999999, loss = 0.00024020017, step = 8020 (5.520 sec)\n",
"2021-12-30 09:55:55,519 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.512\n",
"INFO:tensorflow:global_step/sec: 3.0615\n",
"2021-12-30 09:55:56,822 [INFO] tensorflow: global_step/sec: 3.0615\n",
"INFO:tensorflow:epoch = 83.71875, learning_rate = 0.0009999999, loss = 0.0002193673, step = 8037 (5.553 sec)\n",
"2021-12-30 09:55:59,769 [INFO] tensorflow: epoch = 83.71875, learning_rate = 0.0009999999, loss = 0.0002193673, step = 8037 (5.553 sec)\n",
"INFO:tensorflow:global_step/sec: 3.05388\n",
"2021-12-30 09:55:59,769 [INFO] tensorflow: global_step/sec: 3.05388\n",
"INFO:tensorflow:global_step/sec: 3.07619\n",
"2021-12-30 09:56:02,695 [INFO] tensorflow: global_step/sec: 3.07619\n",
"2021-12-30 09:56:03,637 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.639\n",
"INFO:tensorflow:epoch = 83.89583333333333, learning_rate = 0.0009999999, loss = 0.00022271015, step = 8054 (5.439 sec)\n",
"2021-12-30 09:56:05,208 [INFO] tensorflow: epoch = 83.89583333333333, learning_rate = 0.0009999999, loss = 0.00022271015, step = 8054 (5.439 sec)\n",
"INFO:tensorflow:global_step/sec: 3.1624\n",
"2021-12-30 09:56:05,541 [INFO] tensorflow: global_step/sec: 3.1624\n",
"INFO:tensorflow:global_step/sec: 3.07165\n",
"2021-12-30 09:56:08,471 [INFO] tensorflow: global_step/sec: 3.07165\n",
"2021-12-30 09:56:08,472 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 84/120: loss: 0.00025 learning rate: 0.00100 Time taken: 0:00:31.094926 ETA: 0:18:39.417349\n",
"INFO:tensorflow:epoch = 84.07291666666666, learning_rate = 0.0009893251, loss = 0.00024231353, step = 8071 (5.501 sec)\n",
"2021-12-30 09:56:10,708 [INFO] tensorflow: epoch = 84.07291666666666, learning_rate = 0.0009893251, loss = 0.00024231353, step = 8071 (5.501 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11242\n",
"2021-12-30 09:56:11,363 [INFO] tensorflow: global_step/sec: 3.11242\n",
"2021-12-30 09:56:11,681 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.863\n",
"INFO:tensorflow:global_step/sec: 3.1212\n",
"2021-12-30 09:56:14,246 [INFO] tensorflow: global_step/sec: 3.1212\n",
"INFO:tensorflow:epoch = 84.25, learning_rate = 0.0009638744, loss = 0.00020416637, step = 8088 (5.495 sec)\n",
"2021-12-30 09:56:16,203 [INFO] tensorflow: epoch = 84.25, learning_rate = 0.0009638744, loss = 0.00020416637, step = 8088 (5.495 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08805\n",
"2021-12-30 09:56:17,161 [INFO] tensorflow: global_step/sec: 3.08805\n",
"2021-12-30 09:56:19,784 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.682\n",
"INFO:tensorflow:global_step/sec: 3.05788\n",
"2021-12-30 09:56:20,104 [INFO] tensorflow: global_step/sec: 3.05788\n",
"INFO:tensorflow:epoch = 84.42708333333333, learning_rate = 0.0009390784, loss = 0.00018419477, step = 8105 (5.510 sec)\n",
"2021-12-30 09:56:21,714 [INFO] tensorflow: epoch = 84.42708333333333, learning_rate = 0.0009390784, loss = 0.00018419477, step = 8105 (5.510 sec)\n",
"INFO:tensorflow:global_step/sec: 3.01209\n",
"2021-12-30 09:56:23,092 [INFO] tensorflow: global_step/sec: 3.01209\n",
"INFO:tensorflow:global_step/sec: 3.102\n",
"2021-12-30 09:56:25,993 [INFO] tensorflow: global_step/sec: 3.102\n",
"INFO:tensorflow:epoch = 84.60416666666666, learning_rate = 0.0009149199, loss = 0.00022928727, step = 8122 (5.559 sec)\n",
"2021-12-30 09:56:27,273 [INFO] tensorflow: epoch = 84.60416666666666, learning_rate = 0.0009149199, loss = 0.00022928727, step = 8122 (5.559 sec)\n",
"2021-12-30 09:56:27,915 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.600\n",
"INFO:tensorflow:global_step/sec: 3.10507\n",
"2021-12-30 09:56:28,892 [INFO] tensorflow: global_step/sec: 3.10507\n",
"INFO:tensorflow:global_step/sec: 3.03052\n",
"2021-12-30 09:56:31,862 [INFO] tensorflow: global_step/sec: 3.03052\n",
"INFO:tensorflow:epoch = 84.78125, learning_rate = 0.00089138286, loss = 0.00016203627, step = 8139 (5.567 sec)\n",
"2021-12-30 09:56:32,840 [INFO] tensorflow: epoch = 84.78125, learning_rate = 0.00089138286, loss = 0.00016203627, step = 8139 (5.567 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07238\n",
"2021-12-30 09:56:34,791 [INFO] tensorflow: global_step/sec: 3.07238\n",
"2021-12-30 09:56:36,065 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.541\n",
"INFO:tensorflow:global_step/sec: 3.14417\n",
"2021-12-30 09:56:37,653 [INFO] tensorflow: global_step/sec: 3.14417\n",
"INFO:tensorflow:epoch = 84.95833333333333, learning_rate = 0.0008684518, loss = 0.00022639414, step = 8156 (5.458 sec)\n",
"2021-12-30 09:56:38,298 [INFO] tensorflow: epoch = 84.95833333333333, learning_rate = 0.0008684518, loss = 0.00022639414, step = 8156 (5.458 sec)\n",
"2021-12-30 09:56:39,609 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 85/120: loss: 0.00019 learning rate: 0.00086 Time taken: 0:00:31.118930 ETA: 0:18:09.162554\n",
"INFO:tensorflow:global_step/sec: 3.06558\n",
"2021-12-30 09:56:40,589 [INFO] tensorflow: global_step/sec: 3.06558\n",
"INFO:tensorflow:global_step/sec: 3.10562\n",
"2021-12-30 09:56:43,487 [INFO] tensorflow: global_step/sec: 3.10562\n",
"INFO:tensorflow:epoch = 85.13541666666666, learning_rate = 0.0008461101, loss = 0.00014286752, step = 8173 (5.497 sec)\n",
"2021-12-30 09:56:43,796 [INFO] tensorflow: epoch = 85.13541666666666, learning_rate = 0.0008461101, loss = 0.00014286752, step = 8173 (5.497 sec)\n",
"2021-12-30 09:56:44,113 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.850\n",
"INFO:tensorflow:global_step/sec: 3.07804\n",
"2021-12-30 09:56:46,411 [INFO] tensorflow: global_step/sec: 3.07804\n",
"INFO:tensorflow:epoch = 85.3125, learning_rate = 0.0008243433, loss = 0.00020531267, step = 8190 (5.463 sec)\n",
"2021-12-30 09:56:49,259 [INFO] tensorflow: epoch = 85.3125, learning_rate = 0.0008243433, loss = 0.00020531267, step = 8190 (5.463 sec)\n",
"INFO:tensorflow:global_step/sec: 3.15936\n",
"2021-12-30 09:56:49,260 [INFO] tensorflow: global_step/sec: 3.15936\n",
"INFO:tensorflow:global_step/sec: 3.07175\n",
"2021-12-30 09:56:52,190 [INFO] tensorflow: global_step/sec: 3.07175\n",
"2021-12-30 09:56:52,190 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.762\n",
"INFO:tensorflow:epoch = 85.48958333333333, learning_rate = 0.0008031368, loss = 0.00020574764, step = 8207 (5.515 sec)\n",
"2021-12-30 09:56:54,774 [INFO] tensorflow: epoch = 85.48958333333333, learning_rate = 0.0008031368, loss = 0.00020574764, step = 8207 (5.515 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11027\n",
"2021-12-30 09:56:55,083 [INFO] tensorflow: global_step/sec: 3.11027\n",
"INFO:tensorflow:global_step/sec: 3.02851\n",
"2021-12-30 09:56:58,055 [INFO] tensorflow: global_step/sec: 3.02851\n",
"INFO:tensorflow:epoch = 85.66666666666666, learning_rate = 0.00078247546, loss = 0.00017599738, step = 8224 (5.592 sec)\n",
"2021-12-30 09:57:00,366 [INFO] tensorflow: epoch = 85.66666666666666, learning_rate = 0.00078247546, loss = 0.00017599738, step = 8224 (5.592 sec)\n",
"2021-12-30 09:57:00,367 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.462\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO:tensorflow:global_step/sec: 3.0412\n",
"2021-12-30 09:57:01,014 [INFO] tensorflow: global_step/sec: 3.0412\n",
"INFO:tensorflow:global_step/sec: 3.02663\n",
"2021-12-30 09:57:03,988 [INFO] tensorflow: global_step/sec: 3.02663\n",
"INFO:tensorflow:epoch = 85.84375, learning_rate = 0.000762346, loss = 0.00021473697, step = 8241 (5.564 sec)\n",
"2021-12-30 09:57:05,931 [INFO] tensorflow: epoch = 85.84375, learning_rate = 0.000762346, loss = 0.00021473697, step = 8241 (5.564 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10655\n",
"2021-12-30 09:57:06,885 [INFO] tensorflow: global_step/sec: 3.10655\n",
"2021-12-30 09:57:08,526 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.512\n",
"INFO:tensorflow:global_step/sec: 3.03784\n",
"2021-12-30 09:57:09,848 [INFO] tensorflow: global_step/sec: 3.03784\n",
"2021-12-30 09:57:10,849 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 86/120: loss: 0.00021 learning rate: 0.00075 Time taken: 0:00:31.251153 ETA: 0:17:42.539194\n",
"INFO:tensorflow:epoch = 86.02083333333333, learning_rate = 0.00074273406, loss = 0.00020405948, step = 8258 (5.579 sec)\n",
"2021-12-30 09:57:11,510 [INFO] tensorflow: epoch = 86.02083333333333, learning_rate = 0.00074273406, loss = 0.00020405948, step = 8258 (5.579 sec)\n",
"INFO:tensorflow:global_step/sec: 3.05303\n",
"2021-12-30 09:57:12,796 [INFO] tensorflow: global_step/sec: 3.05303\n",
"INFO:tensorflow:global_step/sec: 3.08517\n",
"2021-12-30 09:57:15,713 [INFO] tensorflow: global_step/sec: 3.08517\n",
"2021-12-30 09:57:16,670 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.559\n",
"INFO:tensorflow:epoch = 86.19791666666666, learning_rate = 0.0007236263, loss = 0.00021415716, step = 8275 (5.496 sec)\n",
"2021-12-30 09:57:17,006 [INFO] tensorflow: epoch = 86.19791666666666, learning_rate = 0.0007236263, loss = 0.00021415716, step = 8275 (5.496 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10599\n",
"2021-12-30 09:57:18,610 [INFO] tensorflow: global_step/sec: 3.10599\n",
"INFO:tensorflow:global_step/sec: 3.07699\n",
"2021-12-30 09:57:21,535 [INFO] tensorflow: global_step/sec: 3.07699\n",
"INFO:tensorflow:epoch = 86.375, learning_rate = 0.00070501043, loss = 0.00020478794, step = 8292 (5.498 sec)\n",
"2021-12-30 09:57:22,504 [INFO] tensorflow: epoch = 86.375, learning_rate = 0.00070501043, loss = 0.00020478794, step = 8292 (5.498 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09406\n",
"2021-12-30 09:57:24,444 [INFO] tensorflow: global_step/sec: 3.09406\n",
"2021-12-30 09:57:24,771 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.691\n",
"INFO:tensorflow:global_step/sec: 3.1076\n",
"2021-12-30 09:57:27,340 [INFO] tensorflow: global_step/sec: 3.1076\n",
"INFO:tensorflow:epoch = 86.55208333333333, learning_rate = 0.00068687345, loss = 0.00017284075, step = 8309 (5.493 sec)\n",
"2021-12-30 09:57:27,997 [INFO] tensorflow: epoch = 86.55208333333333, learning_rate = 0.00068687345, loss = 0.00017284075, step = 8309 (5.493 sec)\n",
"INFO:tensorflow:global_step/sec: 3.02145\n",
"2021-12-30 09:57:30,319 [INFO] tensorflow: global_step/sec: 3.02145\n",
"2021-12-30 09:57:32,937 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.491\n",
"INFO:tensorflow:global_step/sec: 3.06323\n",
"2021-12-30 09:57:33,257 [INFO] tensorflow: global_step/sec: 3.06323\n",
"INFO:tensorflow:epoch = 86.72916666666666, learning_rate = 0.0006692034, loss = 0.00017947772, step = 8326 (5.587 sec)\n",
"2021-12-30 09:57:33,584 [INFO] tensorflow: epoch = 86.72916666666666, learning_rate = 0.0006692034, loss = 0.00017947772, step = 8326 (5.587 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10015\n",
"2021-12-30 09:57:36,160 [INFO] tensorflow: global_step/sec: 3.10015\n",
"INFO:tensorflow:epoch = 86.90625, learning_rate = 0.0006519876, loss = 0.0002049467, step = 8343 (5.494 sec)\n",
"2021-12-30 09:57:39,078 [INFO] tensorflow: epoch = 86.90625, learning_rate = 0.0006519876, loss = 0.0002049467, step = 8343 (5.494 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08377\n",
"2021-12-30 09:57:39,079 [INFO] tensorflow: global_step/sec: 3.08377\n",
"2021-12-30 09:57:41,002 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.801\n",
"INFO:tensorflow:global_step/sec: 3.11377\n",
"2021-12-30 09:57:41,969 [INFO] tensorflow: global_step/sec: 3.11377\n",
"2021-12-30 09:57:41,970 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 87/120: loss: 0.00018 learning rate: 0.00064 Time taken: 0:00:31.127839 ETA: 0:17:07.218698\n",
"INFO:tensorflow:epoch = 87.08333333333333, learning_rate = 0.0006352147, loss = 0.00017522699, step = 8360 (5.477 sec)\n",
"2021-12-30 09:57:44,555 [INFO] tensorflow: epoch = 87.08333333333333, learning_rate = 0.0006352147, loss = 0.00017522699, step = 8360 (5.477 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12471\n",
"2021-12-30 09:57:44,849 [INFO] tensorflow: global_step/sec: 3.12471\n",
"INFO:tensorflow:global_step/sec: 3.13159\n",
"2021-12-30 09:57:47,723 [INFO] tensorflow: global_step/sec: 3.13159\n",
"2021-12-30 09:57:49,083 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.750\n",
"INFO:tensorflow:epoch = 87.26041666666666, learning_rate = 0.00061887363, loss = 0.000161887, step = 8377 (5.520 sec)\n",
"2021-12-30 09:57:50,075 [INFO] tensorflow: epoch = 87.26041666666666, learning_rate = 0.00061887363, loss = 0.000161887, step = 8377 (5.520 sec)\n",
"INFO:tensorflow:global_step/sec: 3.0009\n",
"2021-12-30 09:57:50,722 [INFO] tensorflow: global_step/sec: 3.0009\n",
"INFO:tensorflow:global_step/sec: 3.10186\n",
"2021-12-30 09:57:53,624 [INFO] tensorflow: global_step/sec: 3.10186\n",
"INFO:tensorflow:epoch = 87.4375, learning_rate = 0.0006029529, loss = 0.00020686141, step = 8394 (5.492 sec)\n",
"2021-12-30 09:57:55,567 [INFO] tensorflow: epoch = 87.4375, learning_rate = 0.0006029529, loss = 0.00020686141, step = 8394 (5.492 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10595\n",
"2021-12-30 09:57:56,522 [INFO] tensorflow: global_step/sec: 3.10595\n",
"2021-12-30 09:57:57,165 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.747\n",
"INFO:tensorflow:global_step/sec: 3.10903\n",
"2021-12-30 09:57:59,416 [INFO] tensorflow: global_step/sec: 3.10903\n",
"INFO:tensorflow:epoch = 87.61458333333333, learning_rate = 0.00058744143, loss = 0.0002645551, step = 8411 (5.473 sec)\n",
"2021-12-30 09:58:01,040 [INFO] tensorflow: epoch = 87.61458333333333, learning_rate = 0.00058744143, loss = 0.0002645551, step = 8411 (5.473 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11818\n",
"2021-12-30 09:58:02,303 [INFO] tensorflow: global_step/sec: 3.11818\n",
"INFO:tensorflow:global_step/sec: 3.02424\n",
"2021-12-30 09:58:05,279 [INFO] tensorflow: global_step/sec: 3.02424\n",
"2021-12-30 09:58:05,279 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.649\n",
"INFO:tensorflow:epoch = 87.79166666666666, learning_rate = 0.0005723293, loss = 0.00021970678, step = 8428 (5.521 sec)\n",
"2021-12-30 09:58:06,562 [INFO] tensorflow: epoch = 87.79166666666666, learning_rate = 0.0005723293, loss = 0.00021970678, step = 8428 (5.521 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12533\n",
"2021-12-30 09:58:08,158 [INFO] tensorflow: global_step/sec: 3.12533\n",
"INFO:tensorflow:global_step/sec: 3.08173\n",
"2021-12-30 09:58:11,079 [INFO] tensorflow: global_step/sec: 3.08173\n",
"INFO:tensorflow:epoch = 87.96875, learning_rate = 0.0005576057, loss = 0.00021028738, step = 8445 (5.480 sec)\n",
"2021-12-30 09:58:12,042 [INFO] tensorflow: epoch = 87.96875, learning_rate = 0.0005576057, loss = 0.00021028738, step = 8445 (5.480 sec)\n",
"2021-12-30 09:58:13,031 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 88/120: loss: 0.00017 learning rate: 0.00056 Time taken: 0:00:31.067105 ETA: 0:16:34.147346\n",
"2021-12-30 09:58:13,355 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.768\n",
"INFO:tensorflow:global_step/sec: 3.06674\n",
"2021-12-30 09:58:14,013 [INFO] tensorflow: global_step/sec: 3.06674\n",
"INFO:tensorflow:global_step/sec: 3.09511\n",
"2021-12-30 09:58:16,921 [INFO] tensorflow: global_step/sec: 3.09511\n",
"INFO:tensorflow:epoch = 88.14583333333333, learning_rate = 0.0005432609, loss = 0.00017595368, step = 8462 (5.515 sec)\n",
"2021-12-30 09:58:17,557 [INFO] tensorflow: epoch = 88.14583333333333, learning_rate = 0.0005432609, loss = 0.00017595368, step = 8462 (5.515 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08698\n",
"2021-12-30 09:58:19,837 [INFO] tensorflow: global_step/sec: 3.08698\n",
"2021-12-30 09:58:21,453 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.697\n",
"INFO:tensorflow:global_step/sec: 3.09331\n",
"2021-12-30 09:58:22,746 [INFO] tensorflow: global_step/sec: 3.09331\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO:tensorflow:epoch = 88.32291666666666, learning_rate = 0.0005292853, loss = 0.00020249978, step = 8479 (5.511 sec)\n",
"2021-12-30 09:58:23,068 [INFO] tensorflow: epoch = 88.32291666666666, learning_rate = 0.0005292853, loss = 0.00020249978, step = 8479 (5.511 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10127\n",
"2021-12-30 09:58:25,648 [INFO] tensorflow: global_step/sec: 3.10127\n",
"INFO:tensorflow:epoch = 88.5, learning_rate = 0.000515669, loss = 0.00019063393, step = 8496 (5.498 sec)\n",
"2021-12-30 09:58:28,566 [INFO] tensorflow: epoch = 88.5, learning_rate = 0.000515669, loss = 0.00019063393, step = 8496 (5.498 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08334\n",
"2021-12-30 09:58:28,567 [INFO] tensorflow: global_step/sec: 3.08334\n",
"2021-12-30 09:58:29,550 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.701\n",
"INFO:tensorflow:global_step/sec: 3.06651\n",
"2021-12-30 09:58:31,502 [INFO] tensorflow: global_step/sec: 3.06651\n",
"INFO:tensorflow:epoch = 88.67708333333333, learning_rate = 0.000502403, loss = 0.00017210525, step = 8513 (5.547 sec)\n",
"2021-12-30 09:58:34,113 [INFO] tensorflow: epoch = 88.67708333333333, learning_rate = 0.000502403, loss = 0.00017210525, step = 8513 (5.547 sec)\n",
"INFO:tensorflow:global_step/sec: 3.062\n",
"2021-12-30 09:58:34,441 [INFO] tensorflow: global_step/sec: 3.062\n",
"INFO:tensorflow:global_step/sec: 3.06999\n",
"2021-12-30 09:58:37,373 [INFO] tensorflow: global_step/sec: 3.06999\n",
"2021-12-30 09:58:37,694 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.558\n",
"INFO:tensorflow:epoch = 88.85416666666666, learning_rate = 0.0004894785, loss = 0.00017172426, step = 8530 (5.559 sec)\n",
"2021-12-30 09:58:39,673 [INFO] tensorflow: epoch = 88.85416666666666, learning_rate = 0.0004894785, loss = 0.00017172426, step = 8530 (5.559 sec)\n",
"INFO:tensorflow:global_step/sec: 3.074\n",
"2021-12-30 09:58:40,301 [INFO] tensorflow: global_step/sec: 3.074\n",
"INFO:tensorflow:global_step/sec: 3.10304\n",
"2021-12-30 09:58:43,201 [INFO] tensorflow: global_step/sec: 3.10304\n",
"2021-12-30 09:58:44,157 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 89/120: loss: 0.00026 learning rate: 0.00048 Time taken: 0:00:31.126715 ETA: 0:16:04.928171\n",
"INFO:tensorflow:epoch = 89.03125, learning_rate = 0.00047688655, loss = 0.00016481557, step = 8547 (5.502 sec)\n",
"2021-12-30 09:58:45,174 [INFO] tensorflow: epoch = 89.03125, learning_rate = 0.00047688655, loss = 0.00016481557, step = 8547 (5.502 sec)\n",
"2021-12-30 09:58:45,819 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.617\n",
"INFO:tensorflow:global_step/sec: 3.05472\n",
"2021-12-30 09:58:46,147 [INFO] tensorflow: global_step/sec: 3.05472\n",
"INFO:tensorflow:global_step/sec: 3.11523\n",
"2021-12-30 09:58:49,036 [INFO] tensorflow: global_step/sec: 3.11523\n",
"INFO:tensorflow:epoch = 89.20833333333333, learning_rate = 0.00046461824, loss = 0.00015700606, step = 8564 (5.446 sec)\n",
"2021-12-30 09:58:50,620 [INFO] tensorflow: epoch = 89.20833333333333, learning_rate = 0.00046461824, loss = 0.00015700606, step = 8564 (5.446 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11909\n",
"2021-12-30 09:58:51,922 [INFO] tensorflow: global_step/sec: 3.11909\n",
"2021-12-30 09:58:53,876 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.823\n",
"INFO:tensorflow:global_step/sec: 3.06524\n",
"2021-12-30 09:58:54,858 [INFO] tensorflow: global_step/sec: 3.06524\n",
"INFO:tensorflow:epoch = 89.38541666666666, learning_rate = 0.0004526658, loss = 0.00016166079, step = 8581 (5.541 sec)\n",
"2021-12-30 09:58:56,161 [INFO] tensorflow: epoch = 89.38541666666666, learning_rate = 0.0004526658, loss = 0.00016166079, step = 8581 (5.541 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08298\n",
"2021-12-30 09:58:57,777 [INFO] tensorflow: global_step/sec: 3.08298\n",
"INFO:tensorflow:global_step/sec: 3.1245\n",
"2021-12-30 09:59:00,658 [INFO] tensorflow: global_step/sec: 3.1245\n",
"INFO:tensorflow:epoch = 89.5625, learning_rate = 0.00044102062, loss = 0.00014053691, step = 8598 (5.464 sec)\n",
"2021-12-30 09:59:01,625 [INFO] tensorflow: epoch = 89.5625, learning_rate = 0.00044102062, loss = 0.00014053691, step = 8598 (5.464 sec)\n",
"2021-12-30 09:59:01,948 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.778\n",
"INFO:tensorflow:global_step/sec: 3.07793\n",
"2021-12-30 09:59:03,582 [INFO] tensorflow: global_step/sec: 3.07793\n",
"INFO:tensorflow:global_step/sec: 3.06986\n",
"2021-12-30 09:59:06,513 [INFO] tensorflow: global_step/sec: 3.06986\n",
"INFO:tensorflow:epoch = 89.73958333333333, learning_rate = 0.00042967498, loss = 0.00023653859, step = 8615 (5.536 sec)\n",
"2021-12-30 09:59:07,161 [INFO] tensorflow: epoch = 89.73958333333333, learning_rate = 0.00042967498, loss = 0.00023653859, step = 8615 (5.536 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11519\n",
"2021-12-30 09:59:09,402 [INFO] tensorflow: global_step/sec: 3.11519\n",
"2021-12-30 09:59:10,049 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.690\n",
"INFO:tensorflow:global_step/sec: 3.07292\n",
"2021-12-30 09:59:12,331 [INFO] tensorflow: global_step/sec: 3.07292\n",
"INFO:tensorflow:epoch = 89.91666666666666, learning_rate = 0.00041862146, loss = 0.00016365327, step = 8632 (5.506 sec)\n",
"2021-12-30 09:59:12,667 [INFO] tensorflow: epoch = 89.91666666666666, learning_rate = 0.00041862146, loss = 0.00016365327, step = 8632 (5.506 sec)\n",
"INFO:tensorflow:Saving checkpoints for step-8640.\n",
"2021-12-30 09:59:14,991 [INFO] tensorflow: Saving checkpoints for step-8640.\n",
"WARNING:tensorflow:Ignoring: /tmp/tmpwhalusm6; No such file or directory\n",
"2021-12-30 09:59:15,137 [WARNING] tensorflow: Ignoring: /tmp/tmpwhalusm6; No such file or directory\n",
"2021-12-30 09:59:18,473 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 23, 0.00s/step\n",
"2021-12-30 09:59:20,006 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 23, 0.15s/step\n",
"2021-12-30 09:59:21,798 [INFO] iva.detectnet_v2.evaluation.evaluation: step 20 / 23, 0.18s/step\n",
"Matching predictions to ground truth, class 1/1.: 100%|█| 998/998 [00:00<00:00, 15024.88it/s]\n",
"Epoch 90/120\n",
"=========================\n",
"\n",
"Validation cost: 0.000173\n",
"Mean average_precision (in %): 92.4343\n",
"\n",
"class name average precision (in %)\n",
"------------ --------------------------\n",
"car 92.4343\n",
"\n",
"Median Inference Time: 0.020538\n",
"INFO:tensorflow:epoch = 90.0, learning_rate = 0.00041351854, loss = 0.00017148374, step = 8640 (10.068 sec)\n",
"2021-12-30 09:59:22,736 [INFO] tensorflow: epoch = 90.0, learning_rate = 0.00041351854, loss = 0.00017148374, step = 8640 (10.068 sec)\n",
"INFO:tensorflow:global_step/sec: 0.864948\n",
"2021-12-30 09:59:22,737 [INFO] tensorflow: global_step/sec: 0.864948\n",
"2021-12-30 09:59:22,737 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 90/120: loss: 0.00017 learning rate: 0.00041 Time taken: 0:00:38.569443 ETA: 0:19:17.083275\n",
"INFO:tensorflow:global_step/sec: 3.07766\n",
"2021-12-30 09:59:25,661 [INFO] tensorflow: global_step/sec: 3.07766\n",
"2021-12-30 09:59:25,662 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 12.810\n",
"INFO:tensorflow:epoch = 90.17708333333333, learning_rate = 0.00040288043, loss = 0.00021027446, step = 8657 (5.418 sec)\n",
"2021-12-30 09:59:28,154 [INFO] tensorflow: epoch = 90.17708333333333, learning_rate = 0.00040288043, loss = 0.00021027446, step = 8657 (5.418 sec)\n",
"INFO:tensorflow:global_step/sec: 3.19409\n",
"2021-12-30 09:59:28,479 [INFO] tensorflow: global_step/sec: 3.19409\n",
"INFO:tensorflow:global_step/sec: 3.05377\n",
"2021-12-30 09:59:31,426 [INFO] tensorflow: global_step/sec: 3.05377\n",
"INFO:tensorflow:epoch = 90.35416666666666, learning_rate = 0.00039251623, loss = 0.00016386653, step = 8674 (5.546 sec)\n",
"2021-12-30 09:59:33,700 [INFO] tensorflow: epoch = 90.35416666666666, learning_rate = 0.00039251623, loss = 0.00016386653, step = 8674 (5.546 sec)\n",
"2021-12-30 09:59:33,700 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.881\n",
"INFO:tensorflow:global_step/sec: 3.07548\n",
"2021-12-30 09:59:34,352 [INFO] tensorflow: global_step/sec: 3.07548\n",
"INFO:tensorflow:global_step/sec: 3.10247\n",
"2021-12-30 09:59:37,253 [INFO] tensorflow: global_step/sec: 3.10247\n",
"INFO:tensorflow:epoch = 90.53125, learning_rate = 0.0003824184, loss = 0.00016908924, step = 8691 (5.476 sec)\n",
"2021-12-30 09:59:39,176 [INFO] tensorflow: epoch = 90.53125, learning_rate = 0.0003824184, loss = 0.00016908924, step = 8691 (5.476 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07296\n",
"2021-12-30 09:59:40,182 [INFO] tensorflow: global_step/sec: 3.07296\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"2021-12-30 09:59:41,798 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.700\n",
"INFO:tensorflow:global_step/sec: 3.11008\n",
"2021-12-30 09:59:43,076 [INFO] tensorflow: global_step/sec: 3.11008\n",
"INFO:tensorflow:epoch = 90.70833333333333, learning_rate = 0.00037258043, loss = 0.00018843211, step = 8708 (5.503 sec)\n",
"2021-12-30 09:59:44,679 [INFO] tensorflow: epoch = 90.70833333333333, learning_rate = 0.00037258043, loss = 0.00018843211, step = 8708 (5.503 sec)\n",
"INFO:tensorflow:global_step/sec: 3.16497\n",
"2021-12-30 09:59:45,919 [INFO] tensorflow: global_step/sec: 3.16497\n",
"INFO:tensorflow:global_step/sec: 3.06177\n",
"2021-12-30 09:59:48,859 [INFO] tensorflow: global_step/sec: 3.06177\n",
"2021-12-30 09:59:49,825 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.917\n",
"INFO:tensorflow:epoch = 90.88541666666666, learning_rate = 0.0003629951, loss = 0.00016345167, step = 8725 (5.473 sec)\n",
"2021-12-30 09:59:50,152 [INFO] tensorflow: epoch = 90.88541666666666, learning_rate = 0.0003629951, loss = 0.00016345167, step = 8725 (5.473 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09455\n",
"2021-12-30 09:59:51,767 [INFO] tensorflow: global_step/sec: 3.09455\n",
"2021-12-30 09:59:53,710 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 91/120: loss: 0.00015 learning rate: 0.00036 Time taken: 0:00:30.969270 ETA: 0:14:58.108830\n",
"INFO:tensorflow:global_step/sec: 3.12219\n",
"2021-12-30 09:59:54,650 [INFO] tensorflow: global_step/sec: 3.12219\n",
"INFO:tensorflow:epoch = 91.0625, learning_rate = 0.00035365697, loss = 0.0001612968, step = 8742 (5.503 sec)\n",
"2021-12-30 09:59:55,655 [INFO] tensorflow: epoch = 91.0625, learning_rate = 0.00035365697, loss = 0.0001612968, step = 8742 (5.503 sec)\n",
"INFO:tensorflow:global_step/sec: 3.04543\n",
"2021-12-30 09:59:57,605 [INFO] tensorflow: global_step/sec: 3.04543\n",
"2021-12-30 09:59:57,936 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.658\n",
"INFO:tensorflow:global_step/sec: 3.06737\n",
"2021-12-30 10:00:00,539 [INFO] tensorflow: global_step/sec: 3.06737\n",
"INFO:tensorflow:epoch = 91.23958333333333, learning_rate = 0.00034455885, loss = 0.00018286589, step = 8759 (5.542 sec)\n",
"2021-12-30 10:00:01,197 [INFO] tensorflow: epoch = 91.23958333333333, learning_rate = 0.00034455885, loss = 0.00018286589, step = 8759 (5.542 sec)\n",
"INFO:tensorflow:global_step/sec: 3.0616\n",
"2021-12-30 10:00:03,479 [INFO] tensorflow: global_step/sec: 3.0616\n",
"2021-12-30 10:00:06,081 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.554\n",
"INFO:tensorflow:global_step/sec: 3.07881\n",
"2021-12-30 10:00:06,402 [INFO] tensorflow: global_step/sec: 3.07881\n",
"INFO:tensorflow:epoch = 91.41666666666666, learning_rate = 0.000335695, loss = 0.00016428047, step = 8776 (5.518 sec)\n",
"2021-12-30 10:00:06,715 [INFO] tensorflow: epoch = 91.41666666666666, learning_rate = 0.000335695, loss = 0.00016428047, step = 8776 (5.518 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11504\n",
"2021-12-30 10:00:09,291 [INFO] tensorflow: global_step/sec: 3.11504\n",
"INFO:tensorflow:epoch = 91.59375, learning_rate = 0.0003270591, loss = 0.0001745156, step = 8793 (5.480 sec)\n",
"2021-12-30 10:00:12,194 [INFO] tensorflow: epoch = 91.59375, learning_rate = 0.0003270591, loss = 0.0001745156, step = 8793 (5.480 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09926\n",
"2021-12-30 10:00:12,195 [INFO] tensorflow: global_step/sec: 3.09926\n",
"2021-12-30 10:00:14,140 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.819\n",
"INFO:tensorflow:global_step/sec: 3.05318\n",
"2021-12-30 10:00:15,143 [INFO] tensorflow: global_step/sec: 3.05318\n",
"INFO:tensorflow:epoch = 91.77083333333333, learning_rate = 0.0003186454, loss = 0.00017714626, step = 8810 (5.491 sec)\n",
"2021-12-30 10:00:17,685 [INFO] tensorflow: epoch = 91.77083333333333, learning_rate = 0.0003186454, loss = 0.00017714626, step = 8810 (5.491 sec)\n",
"INFO:tensorflow:global_step/sec: 3.16782\n",
"2021-12-30 10:00:17,984 [INFO] tensorflow: global_step/sec: 3.16782\n",
"INFO:tensorflow:global_step/sec: 3.05189\n",
"2021-12-30 10:00:20,933 [INFO] tensorflow: global_step/sec: 3.05189\n",
"2021-12-30 10:00:22,237 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.702\n",
"INFO:tensorflow:epoch = 91.94791666666666, learning_rate = 0.00031044785, loss = 0.0001472796, step = 8827 (5.539 sec)\n",
"2021-12-30 10:00:23,224 [INFO] tensorflow: epoch = 91.94791666666666, learning_rate = 0.00031044785, loss = 0.0001472796, step = 8827 (5.539 sec)\n",
"INFO:tensorflow:global_step/sec: 3.04821\n",
"2021-12-30 10:00:23,885 [INFO] tensorflow: global_step/sec: 3.04821\n",
"2021-12-30 10:00:24,875 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 92/120: loss: 0.00017 learning rate: 0.00031 Time taken: 0:00:31.170770 ETA: 0:14:32.781571\n",
"INFO:tensorflow:global_step/sec: 3.10103\n",
"2021-12-30 10:00:26,788 [INFO] tensorflow: global_step/sec: 3.10103\n",
"INFO:tensorflow:epoch = 92.125, learning_rate = 0.0003024615, loss = 0.00014310695, step = 8844 (5.494 sec)\n",
"2021-12-30 10:00:28,717 [INFO] tensorflow: epoch = 92.125, learning_rate = 0.0003024615, loss = 0.00014310695, step = 8844 (5.494 sec)\n",
"INFO:tensorflow:global_step/sec: 3.06748\n",
"2021-12-30 10:00:29,722 [INFO] tensorflow: global_step/sec: 3.06748\n",
"2021-12-30 10:00:30,370 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.591\n",
"INFO:tensorflow:global_step/sec: 3.09613\n",
"2021-12-30 10:00:32,628 [INFO] tensorflow: global_step/sec: 3.09613\n",
"INFO:tensorflow:epoch = 92.30208333333333, learning_rate = 0.00029468027, loss = 0.00016960516, step = 8861 (5.527 sec)\n",
"2021-12-30 10:00:34,244 [INFO] tensorflow: epoch = 92.30208333333333, learning_rate = 0.00029468027, loss = 0.00016960516, step = 8861 (5.527 sec)\n",
"INFO:tensorflow:global_step/sec: 3.14237\n",
"2021-12-30 10:00:35,493 [INFO] tensorflow: global_step/sec: 3.14237\n",
"INFO:tensorflow:global_step/sec: 3.0725\n",
"2021-12-30 10:00:38,422 [INFO] tensorflow: global_step/sec: 3.0725\n",
"2021-12-30 10:00:38,422 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.839\n",
"INFO:tensorflow:epoch = 92.47916666666666, learning_rate = 0.00028709954, loss = 0.00017806343, step = 8878 (5.510 sec)\n",
"2021-12-30 10:00:39,754 [INFO] tensorflow: epoch = 92.47916666666666, learning_rate = 0.00028709954, loss = 0.00017806343, step = 8878 (5.510 sec)\n",
"INFO:tensorflow:global_step/sec: 3.0701\n",
"2021-12-30 10:00:41,353 [INFO] tensorflow: global_step/sec: 3.0701\n",
"INFO:tensorflow:global_step/sec: 3.11982\n",
"2021-12-30 10:00:44,238 [INFO] tensorflow: global_step/sec: 3.11982\n",
"INFO:tensorflow:epoch = 92.65625, learning_rate = 0.0002797138, loss = 0.00012543958, step = 8895 (5.456 sec)\n",
"2021-12-30 10:00:45,210 [INFO] tensorflow: epoch = 92.65625, learning_rate = 0.0002797138, loss = 0.00012543958, step = 8895 (5.456 sec)\n",
"2021-12-30 10:00:46,494 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.777\n",
"INFO:tensorflow:global_step/sec: 3.11519\n",
"2021-12-30 10:00:47,127 [INFO] tensorflow: global_step/sec: 3.11519\n",
"INFO:tensorflow:global_step/sec: 3.07222\n",
"2021-12-30 10:00:50,057 [INFO] tensorflow: global_step/sec: 3.07222\n",
"INFO:tensorflow:epoch = 92.83333333333333, learning_rate = 0.00027251805, loss = 0.00014818642, step = 8912 (5.510 sec)\n",
"2021-12-30 10:00:50,720 [INFO] tensorflow: epoch = 92.83333333333333, learning_rate = 0.00027251805, loss = 0.00014818642, step = 8912 (5.510 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07353\n",
"2021-12-30 10:00:52,985 [INFO] tensorflow: global_step/sec: 3.07353\n",
"2021-12-30 10:00:54,666 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.475\n",
"INFO:tensorflow:global_step/sec: 3.01734\n",
"2021-12-30 10:00:55,968 [INFO] tensorflow: global_step/sec: 3.01734\n",
"2021-12-30 10:00:55,968 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 93/120: loss: 0.00018 learning rate: 0.00027 Time taken: 0:00:31.096027 ETA: 0:13:59.592720\n",
"INFO:tensorflow:epoch = 93.01041666666666, learning_rate = 0.0002655072, loss = 0.00017762964, step = 8929 (5.566 sec)\n",
"2021-12-30 10:00:56,285 [INFO] tensorflow: epoch = 93.01041666666666, learning_rate = 0.0002655072, loss = 0.00017762964, step = 8929 (5.566 sec)\n",
"INFO:tensorflow:global_step/sec: 3.04953\n",
"2021-12-30 10:00:58,919 [INFO] tensorflow: global_step/sec: 3.04953\n",
"INFO:tensorflow:epoch = 93.1875, learning_rate = 0.00025867694, loss = 0.00020362425, step = 8946 (5.550 sec)\n",
"2021-12-30 10:01:01,835 [INFO] tensorflow: epoch = 93.1875, learning_rate = 0.00025867694, loss = 0.00020362425, step = 8946 (5.550 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08495\n",
"2021-12-30 10:01:01,836 [INFO] tensorflow: global_step/sec: 3.08495\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"2021-12-30 10:01:02,830 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.500\n",
"INFO:tensorflow:global_step/sec: 3.12081\n",
"2021-12-30 10:01:04,720 [INFO] tensorflow: global_step/sec: 3.12081\n",
"INFO:tensorflow:epoch = 93.36458333333333, learning_rate = 0.00025202238, loss = 0.00012836553, step = 8963 (5.496 sec)\n",
"2021-12-30 10:01:07,331 [INFO] tensorflow: epoch = 93.36458333333333, learning_rate = 0.00025202238, loss = 0.00012836553, step = 8963 (5.496 sec)\n",
"INFO:tensorflow:global_step/sec: 3.0597\n",
"2021-12-30 10:01:07,662 [INFO] tensorflow: global_step/sec: 3.0597\n",
"INFO:tensorflow:global_step/sec: 3.04417\n",
"2021-12-30 10:01:10,618 [INFO] tensorflow: global_step/sec: 3.04417\n",
"2021-12-30 10:01:10,940 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.662\n",
"INFO:tensorflow:epoch = 93.54166666666666, learning_rate = 0.0002455388, loss = 0.00014901487, step = 8980 (5.526 sec)\n",
"2021-12-30 10:01:12,857 [INFO] tensorflow: epoch = 93.54166666666666, learning_rate = 0.0002455388, loss = 0.00014901487, step = 8980 (5.526 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12251\n",
"2021-12-30 10:01:13,500 [INFO] tensorflow: global_step/sec: 3.12251\n",
"INFO:tensorflow:global_step/sec: 3.06339\n",
"2021-12-30 10:01:16,438 [INFO] tensorflow: global_step/sec: 3.06339\n",
"INFO:tensorflow:epoch = 93.71875, learning_rate = 0.00023922222, loss = 0.0001793719, step = 8997 (5.498 sec)\n",
"2021-12-30 10:01:18,356 [INFO] tensorflow: epoch = 93.71875, learning_rate = 0.00023922222, loss = 0.0001793719, step = 8997 (5.498 sec)\n",
"2021-12-30 10:01:19,002 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.807\n",
"INFO:tensorflow:global_step/sec: 3.12144\n",
"2021-12-30 10:01:19,322 [INFO] tensorflow: global_step/sec: 3.12144\n",
"INFO:tensorflow:global_step/sec: 3.11514\n",
"2021-12-30 10:01:22,211 [INFO] tensorflow: global_step/sec: 3.11514\n",
"INFO:tensorflow:epoch = 93.89583333333333, learning_rate = 0.00023306816, loss = 0.0002147496, step = 9014 (5.459 sec)\n",
"2021-12-30 10:01:23,815 [INFO] tensorflow: epoch = 93.89583333333333, learning_rate = 0.00023306816, loss = 0.0002147496, step = 9014 (5.459 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12378\n",
"2021-12-30 10:01:25,092 [INFO] tensorflow: global_step/sec: 3.12378\n",
"2021-12-30 10:01:27,034 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 94/120: loss: 0.00018 learning rate: 0.00023 Time taken: 0:00:31.066554 ETA: 0:13:27.730412\n",
"2021-12-30 10:01:27,034 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.902\n",
"INFO:tensorflow:global_step/sec: 3.05158\n",
"2021-12-30 10:01:28,041 [INFO] tensorflow: global_step/sec: 3.05158\n",
"INFO:tensorflow:epoch = 94.07291666666666, learning_rate = 0.0002270724, loss = 0.00014904395, step = 9031 (5.545 sec)\n",
"2021-12-30 10:01:29,360 [INFO] tensorflow: epoch = 94.07291666666666, learning_rate = 0.0002270724, loss = 0.00014904395, step = 9031 (5.545 sec)\n",
"INFO:tensorflow:global_step/sec: 3.0738\n",
"2021-12-30 10:01:30,969 [INFO] tensorflow: global_step/sec: 3.0738\n",
"INFO:tensorflow:global_step/sec: 3.07147\n",
"2021-12-30 10:01:33,899 [INFO] tensorflow: global_step/sec: 3.07147\n",
"INFO:tensorflow:epoch = 94.25, learning_rate = 0.00022123089, loss = 0.00014195462, step = 9048 (5.496 sec)\n",
"2021-12-30 10:01:34,856 [INFO] tensorflow: epoch = 94.25, learning_rate = 0.00022123089, loss = 0.00014195462, step = 9048 (5.496 sec)\n",
"2021-12-30 10:01:35,183 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.542\n",
"INFO:tensorflow:global_step/sec: 3.06575\n",
"2021-12-30 10:01:36,835 [INFO] tensorflow: global_step/sec: 3.06575\n",
"INFO:tensorflow:global_step/sec: 3.08282\n",
"2021-12-30 10:01:39,754 [INFO] tensorflow: global_step/sec: 3.08282\n",
"INFO:tensorflow:epoch = 94.42708333333333, learning_rate = 0.00021553945, loss = 0.00014953686, step = 9065 (5.558 sec)\n",
"2021-12-30 10:01:40,414 [INFO] tensorflow: epoch = 94.42708333333333, learning_rate = 0.00021553945, loss = 0.00014953686, step = 9065 (5.558 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07653\n",
"2021-12-30 10:01:42,680 [INFO] tensorflow: global_step/sec: 3.07653\n",
"2021-12-30 10:01:43,313 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.601\n",
"INFO:tensorflow:global_step/sec: 3.13028\n",
"2021-12-30 10:01:45,555 [INFO] tensorflow: global_step/sec: 3.13028\n",
"INFO:tensorflow:epoch = 94.60416666666666, learning_rate = 0.00020999463, loss = 0.0001833837, step = 9082 (5.457 sec)\n",
"2021-12-30 10:01:45,872 [INFO] tensorflow: epoch = 94.60416666666666, learning_rate = 0.00020999463, loss = 0.0001833837, step = 9082 (5.457 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08695\n",
"2021-12-30 10:01:48,470 [INFO] tensorflow: global_step/sec: 3.08695\n",
"INFO:tensorflow:epoch = 94.78125, learning_rate = 0.00020459226, loss = 0.00016718102, step = 9099 (5.521 sec)\n",
"2021-12-30 10:01:51,393 [INFO] tensorflow: epoch = 94.78125, learning_rate = 0.00020459226, loss = 0.00016718102, step = 9099 (5.521 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07906\n",
"2021-12-30 10:01:51,393 [INFO] tensorflow: global_step/sec: 3.07906\n",
"2021-12-30 10:01:51,394 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.751\n",
"INFO:tensorflow:global_step/sec: 3.11057\n",
"2021-12-30 10:01:54,287 [INFO] tensorflow: global_step/sec: 3.11057\n",
"INFO:tensorflow:epoch = 94.95833333333333, learning_rate = 0.00019932886, loss = 0.00017374974, step = 9116 (5.470 sec)\n",
"2021-12-30 10:01:56,863 [INFO] tensorflow: epoch = 94.95833333333333, learning_rate = 0.00019932886, loss = 0.00017374974, step = 9116 (5.470 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12149\n",
"2021-12-30 10:01:57,170 [INFO] tensorflow: global_step/sec: 3.12149\n",
"2021-12-30 10:01:58,129 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 95/120: loss: 0.00015 learning rate: 0.00020 Time taken: 0:00:31.100018 ETA: 0:12:57.500445\n",
"2021-12-30 10:01:59,423 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.909\n",
"INFO:tensorflow:global_step/sec: 3.08388\n",
"2021-12-30 10:02:00,088 [INFO] tensorflow: global_step/sec: 3.08388\n",
"INFO:tensorflow:epoch = 95.13541666666666, learning_rate = 0.00019420106, loss = 0.00016268846, step = 9133 (5.550 sec)\n",
"2021-12-30 10:02:02,413 [INFO] tensorflow: epoch = 95.13541666666666, learning_rate = 0.00019420106, loss = 0.00016268846, step = 9133 (5.550 sec)\n",
"INFO:tensorflow:global_step/sec: 3.053\n",
"2021-12-30 10:02:03,036 [INFO] tensorflow: global_step/sec: 3.053\n",
"INFO:tensorflow:global_step/sec: 3.09501\n",
"2021-12-30 10:02:05,944 [INFO] tensorflow: global_step/sec: 3.09501\n",
"2021-12-30 10:02:07,591 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.488\n",
"INFO:tensorflow:epoch = 95.3125, learning_rate = 0.00018920499, loss = 0.0001733328, step = 9150 (5.480 sec)\n",
"2021-12-30 10:02:07,893 [INFO] tensorflow: epoch = 95.3125, learning_rate = 0.00018920499, loss = 0.0001733328, step = 9150 (5.480 sec)\n",
"INFO:tensorflow:global_step/sec: 3.06853\n",
"2021-12-30 10:02:08,877 [INFO] tensorflow: global_step/sec: 3.06853\n",
"INFO:tensorflow:global_step/sec: 3.0848\n",
"2021-12-30 10:02:11,795 [INFO] tensorflow: global_step/sec: 3.0848\n",
"INFO:tensorflow:epoch = 95.48958333333333, learning_rate = 0.00018433764, loss = 0.00015449207, step = 9167 (5.508 sec)\n",
"2021-12-30 10:02:13,401 [INFO] tensorflow: epoch = 95.48958333333333, learning_rate = 0.00018433764, loss = 0.00015449207, step = 9167 (5.508 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08944\n",
"2021-12-30 10:02:14,708 [INFO] tensorflow: global_step/sec: 3.08944\n",
"2021-12-30 10:02:15,677 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.735\n",
"INFO:tensorflow:global_step/sec: 3.1101\n",
"2021-12-30 10:02:17,602 [INFO] tensorflow: global_step/sec: 3.1101\n",
"INFO:tensorflow:epoch = 95.66666666666666, learning_rate = 0.00017959549, loss = 0.00014047424, step = 9184 (5.512 sec)\n",
"2021-12-30 10:02:18,913 [INFO] tensorflow: epoch = 95.66666666666666, learning_rate = 0.00017959549, loss = 0.00014047424, step = 9184 (5.512 sec)\n",
"INFO:tensorflow:global_step/sec: 3.06676\n",
"2021-12-30 10:02:20,536 [INFO] tensorflow: global_step/sec: 3.06676\n",
"INFO:tensorflow:global_step/sec: 3.1099\n",
"2021-12-30 10:02:23,430 [INFO] tensorflow: global_step/sec: 3.1099\n",
"2021-12-30 10:02:23,755 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.758\n",
"INFO:tensorflow:epoch = 95.84375, learning_rate = 0.00017497533, loss = 0.00015625467, step = 9201 (5.486 sec)\n",
"2021-12-30 10:02:24,400 [INFO] tensorflow: epoch = 95.84375, learning_rate = 0.00017497533, loss = 0.00015625467, step = 9201 (5.486 sec)\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO:tensorflow:global_step/sec: 3.05883\n",
"2021-12-30 10:02:26,373 [INFO] tensorflow: global_step/sec: 3.05883\n",
"INFO:tensorflow:global_step/sec: 3.0781\n",
"2021-12-30 10:02:29,296 [INFO] tensorflow: global_step/sec: 3.0781\n",
"2021-12-30 10:02:29,297 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 96/120: loss: 0.00017 learning rate: 0.00017 Time taken: 0:00:31.151004 ETA: 0:12:27.624092\n",
"INFO:tensorflow:epoch = 96.02083333333333, learning_rate = 0.00017047403, loss = 0.00014952442, step = 9218 (5.539 sec)\n",
"2021-12-30 10:02:29,939 [INFO] tensorflow: epoch = 96.02083333333333, learning_rate = 0.00017047403, loss = 0.00014952442, step = 9218 (5.539 sec)\n",
"2021-12-30 10:02:31,855 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.694\n",
"INFO:tensorflow:global_step/sec: 3.14689\n",
"2021-12-30 10:02:32,156 [INFO] tensorflow: global_step/sec: 3.14689\n"
]
}
],
"source": [
"!tao detectnet_v2 train -e $SPECS_DIR/detectnet_v2_train_resnet18_kitti.txt \\\n",
" -r $USER_EXPERIMENT_DIR/experiment_dir_unpruned \\\n",
" -k tlt_encode \\\n",
" -n resnet18_detector \\\n",
" --gpus $NUM_GPUS"
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {},
"outputs": [
{
"ename": "SyntaxError",
"evalue": "invalid syntax (1907354398.py, line 1)",
"output_type": "error",
"traceback": [
"\u001b[0;36m File \u001b[0;32m\"/tmp/ipykernel_922/1907354398.py\"\u001b[0;36m, line \u001b[0;32m1\u001b[0m\n\u001b[0;31m local = $LOCAL_EXPERIMENT_DIR\u001b[0m\n\u001b[0m ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m invalid syntax\n"
]
}
],
"source": [
"local = $LOCAL_EXPERIMENT_DIR\n",
"local"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Model for each epoch:\n",
"---------------------\n",
"total 45M\r\n",
"-rw-r--r-- 1 guest guest 45M Dec 30 18:15 resnet18_detector.tlt\r\n"
]
}
],
"source": [
"print('Model for each epoch:')\n",
"print('---------------------')\n",
"!ls -lh $LOCAL_EXPERIMENT_DIR/experiment_dir_unpruned/weights"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 5. Evaluate the trained model "
]
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"2022-01-06 16:11:01,505 [INFO] root: Registry: ['nvcr.io']\n",
"Matplotlib created a temporary config/cache directory at /tmp/matplotlib-lv0msbns because the default path (/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.\n",
"Using TensorFlow backend.\n",
"WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.\n",
"Using TensorFlow backend.\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:43: The name tf.train.SessionRunHook is deprecated. Please use tf.estimator.SessionRunHook instead.\n",
"\n",
"2022-01-06 08:11:07,243 [INFO] iva.detectnet_v2.spec_handler.spec_loader: Merging specification from /workspace/tao-experiments/specs/detectnet_v2_train_resnet18_kitti.txt\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:153: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.\n",
"\n",
"2022-01-06 08:11:07,247 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:153: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.\n",
"\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.\n",
"\n",
"2022-01-06 08:11:07,601 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.\n",
"\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.\n",
"\n",
"2022-01-06 08:11:07,615 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.\n",
"\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1834: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead.\n",
"\n",
"2022-01-06 08:11:07,633 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1834: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead.\n",
"\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.\n",
"\n",
"2022-01-06 08:11:08,574 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.\n",
"\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:181: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.\n",
"\n",
"2022-01-06 08:11:08,574 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:181: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.\n",
"\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:186: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.\n",
"\n",
"2022-01-06 08:11:08,574 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:186: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.\n",
"\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:190: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.\n",
"\n",
"2022-01-06 08:11:08,897 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:190: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.\n",
"\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead.\n",
"\n",
"2022-01-06 08:11:08,897 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead.\n",
"\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead.\n",
"\n",
"2022-01-06 08:11:09,116 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead.\n",
"\n",
"/usr/local/lib/python3.6/dist-packages/keras/engine/saving.py:292: UserWarning: No training configuration found in save file: the model was *not* compiled. Compile it manually.\n",
" warnings.warn('No training configuration found in save file: '\n",
"2022-01-06 08:11:09,388 [INFO] iva.detectnet_v2.objectives.bbox_objective: Default L1 loss function will be used.\n",
"2022-01-06 08:11:09,491 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Serial augmentation enabled = False\n",
"2022-01-06 08:11:09,491 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Pseudo sharding enabled = False\n",
"2022-01-06 08:11:09,491 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Max Image Dimensions (all sources): (0, 0)\n",
"2022-01-06 08:11:09,491 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: number of cpus: 16, io threads: 32, compute threads: 16, buffered batches: 4\n",
"2022-01-06 08:11:09,492 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: total dataset size 190, number of sources: 1, batch size per gpu: 8, steps: 24\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.\n",
"\n",
"2022-01-06 08:11:09,519 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.\n",
"\n",
"WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code\n",
"2022-01-06 08:11:09,559 [WARNING] tensorflow: Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code\n",
"2022-01-06 08:11:09,574 [INFO] iva.detectnet_v2.dataloader.default_dataloader: Bounding box coordinates were detected in the input specification! Bboxes will be automatically converted to polygon coordinates.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"2022-01-06 08:11:09,779 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: shuffle: False - shard 0 of 1\n",
"2022-01-06 08:11:09,784 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: sampling 1 datasets with weights:\n",
"2022-01-06 08:11:09,784 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: source: 0 weight: 1.000000\n",
"WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code\n",
"2022-01-06 08:11:09,795 [WARNING] tensorflow: Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/core/build_wheel.runfiles/ai_infra/moduluspy/modulus/blocks/data_loaders/multi_source_loader/types/images2d_reference.py:427: The name tf.image.resize_images is deprecated. Please use tf.image.resize instead.\n",
"\n",
"2022-01-06 08:11:09,815 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/core/build_wheel.runfiles/ai_infra/moduluspy/modulus/blocks/data_loaders/multi_source_loader/types/images2d_reference.py:427: The name tf.image.resize_images is deprecated. Please use tf.image.resize instead.\n",
"\n",
"2022-01-06 08:11:10,006 [INFO] iva.detectnet_v2.evaluation.build_evaluator: Found 190 samples in validation set\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:107: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.\n",
"\n",
"2022-01-06 08:11:10,006 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:107: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:110: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.\n",
"\n",
"2022-01-06 08:11:10,007 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:110: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:113: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead.\n",
"\n",
"2022-01-06 08:11:10,008 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:113: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/rasterizers/bbox_rasterizer.py:347: The name tf.bincount is deprecated. Please use tf.math.bincount instead.\n",
"\n",
"2022-01-06 08:11:10,108 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/rasterizers/bbox_rasterizer.py:347: The name tf.bincount is deprecated. Please use tf.math.bincount instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_functions.py:17: The name tf.log is deprecated. Please use tf.math.log instead.\n",
"\n",
"2022-01-06 08:11:10,504 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_functions.py:17: The name tf.log is deprecated. Please use tf.math.log instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:235: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead.\n",
"\n",
"2022-01-06 08:11:10,512 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:235: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead.\n",
"\n",
"__________________________________________________________________________________________________\n",
"Layer (type) Output Shape Param # Connected to \n",
"==================================================================================================\n",
"input_1 (InputLayer) (None, 3, 544, 960) 0 \n",
"__________________________________________________________________________________________________\n",
"input_1_qdq (QDQ) (None, 3, 544, 960) 1 input_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"conv1 (QuantizedConv2D) (None, 64, 272, 480) 9472 input_1_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"bn_conv1 (BatchNormalization) (None, 64, 272, 480) 256 conv1[0][0] \n",
"__________________________________________________________________________________________________\n",
"activation_1 (ReLU) (None, 64, 272, 480) 0 bn_conv1[0][0] \n",
"__________________________________________________________________________________________________\n",
"activation_1_qdq (QDQ) (None, 64, 272, 480) 1 activation_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1a_conv_1 (QuantizedConv2 (None, 64, 136, 240) 36928 activation_1_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1a_bn_1 (BatchNormalizati (None, 64, 136, 240) 256 block_1a_conv_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1a_relu_1 (ReLU) (None, 64, 136, 240) 0 block_1a_bn_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1a_relu_1_qdq (QDQ) (None, 64, 136, 240) 1 block_1a_relu_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1a_conv_2 (QuantizedConv2 (None, 64, 136, 240) 36928 block_1a_relu_1_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1a_conv_shortcut (Quantiz (None, 64, 136, 240) 4160 activation_1_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1a_bn_2 (BatchNormalizati (None, 64, 136, 240) 256 block_1a_conv_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1a_bn_shortcut (BatchNorm (None, 64, 136, 240) 256 block_1a_conv_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1a_bn_2_qdq (QDQ) (None, 64, 136, 240) 1 block_1a_bn_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1a_bn_shortcut_qdq (QDQ) (None, 64, 136, 240) 1 block_1a_bn_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_1 (Add) (None, 64, 136, 240) 0 block_1a_bn_2_qdq[0][0] \n",
" block_1a_bn_shortcut_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_1_qdq (QDQ) (None, 64, 136, 240) 1 add_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1a_relu (ReLU) (None, 64, 136, 240) 0 add_1_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1a_relu_qdq (QDQ) (None, 64, 136, 240) 1 block_1a_relu[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1b_conv_1 (QuantizedConv2 (None, 64, 136, 240) 36928 block_1a_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1b_bn_1 (BatchNormalizati (None, 64, 136, 240) 256 block_1b_conv_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1b_relu_1 (ReLU) (None, 64, 136, 240) 0 block_1b_bn_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1b_relu_1_qdq (QDQ) (None, 64, 136, 240) 1 block_1b_relu_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1b_conv_2 (QuantizedConv2 (None, 64, 136, 240) 36928 block_1b_relu_1_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1b_conv_shortcut (Quantiz (None, 64, 136, 240) 4160 block_1a_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1b_bn_2 (BatchNormalizati (None, 64, 136, 240) 256 block_1b_conv_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1b_bn_shortcut (BatchNorm (None, 64, 136, 240) 256 block_1b_conv_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1b_bn_2_qdq (QDQ) (None, 64, 136, 240) 1 block_1b_bn_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1b_bn_shortcut_qdq (QDQ) (None, 64, 136, 240) 1 block_1b_bn_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_2 (Add) (None, 64, 136, 240) 0 block_1b_bn_2_qdq[0][0] \n",
" block_1b_bn_shortcut_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_2_qdq (QDQ) (None, 64, 136, 240) 1 add_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1b_relu (ReLU) (None, 64, 136, 240) 0 add_2_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1b_relu_qdq (QDQ) (None, 64, 136, 240) 1 block_1b_relu[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2a_conv_1 (QuantizedConv2 (None, 128, 68, 120) 73856 block_1b_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2a_bn_1 (BatchNormalizati (None, 128, 68, 120) 512 block_2a_conv_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2a_relu_1 (ReLU) (None, 128, 68, 120) 0 block_2a_bn_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2a_relu_1_qdq (QDQ) (None, 128, 68, 120) 1 block_2a_relu_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2a_conv_2 (QuantizedConv2 (None, 128, 68, 120) 147584 block_2a_relu_1_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2a_conv_shortcut (Quantiz (None, 128, 68, 120) 8320 block_1b_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2a_bn_2 (BatchNormalizati (None, 128, 68, 120) 512 block_2a_conv_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2a_bn_shortcut (BatchNorm (None, 128, 68, 120) 512 block_2a_conv_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2a_bn_2_qdq (QDQ) (None, 128, 68, 120) 1 block_2a_bn_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2a_bn_shortcut_qdq (QDQ) (None, 128, 68, 120) 1 block_2a_bn_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_3 (Add) (None, 128, 68, 120) 0 block_2a_bn_2_qdq[0][0] \n",
" block_2a_bn_shortcut_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_3_qdq (QDQ) (None, 128, 68, 120) 1 add_3[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2a_relu (ReLU) (None, 128, 68, 120) 0 add_3_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2a_relu_qdq (QDQ) (None, 128, 68, 120) 1 block_2a_relu[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2b_conv_1 (QuantizedConv2 (None, 128, 68, 120) 147584 block_2a_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2b_bn_1 (BatchNormalizati (None, 128, 68, 120) 512 block_2b_conv_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2b_relu_1 (ReLU) (None, 128, 68, 120) 0 block_2b_bn_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2b_relu_1_qdq (QDQ) (None, 128, 68, 120) 1 block_2b_relu_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2b_conv_2 (QuantizedConv2 (None, 128, 68, 120) 147584 block_2b_relu_1_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2b_conv_shortcut (Quantiz (None, 128, 68, 120) 16512 block_2a_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2b_bn_2 (BatchNormalizati (None, 128, 68, 120) 512 block_2b_conv_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2b_bn_shortcut (BatchNorm (None, 128, 68, 120) 512 block_2b_conv_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2b_bn_2_qdq (QDQ) (None, 128, 68, 120) 1 block_2b_bn_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2b_bn_shortcut_qdq (QDQ) (None, 128, 68, 120) 1 block_2b_bn_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_4 (Add) (None, 128, 68, 120) 0 block_2b_bn_2_qdq[0][0] \n",
" block_2b_bn_shortcut_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_4_qdq (QDQ) (None, 128, 68, 120) 1 add_4[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2b_relu (ReLU) (None, 128, 68, 120) 0 add_4_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2b_relu_qdq (QDQ) (None, 128, 68, 120) 1 block_2b_relu[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3a_conv_1 (QuantizedConv2 (None, 256, 34, 60) 295168 block_2b_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3a_bn_1 (BatchNormalizati (None, 256, 34, 60) 1024 block_3a_conv_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3a_relu_1 (ReLU) (None, 256, 34, 60) 0 block_3a_bn_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3a_relu_1_qdq (QDQ) (None, 256, 34, 60) 1 block_3a_relu_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3a_conv_2 (QuantizedConv2 (None, 256, 34, 60) 590080 block_3a_relu_1_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3a_conv_shortcut (Quantiz (None, 256, 34, 60) 33024 block_2b_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3a_bn_2 (BatchNormalizati (None, 256, 34, 60) 1024 block_3a_conv_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3a_bn_shortcut (BatchNorm (None, 256, 34, 60) 1024 block_3a_conv_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3a_bn_2_qdq (QDQ) (None, 256, 34, 60) 1 block_3a_bn_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3a_bn_shortcut_qdq (QDQ) (None, 256, 34, 60) 1 block_3a_bn_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_5 (Add) (None, 256, 34, 60) 0 block_3a_bn_2_qdq[0][0] \n",
" block_3a_bn_shortcut_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_5_qdq (QDQ) (None, 256, 34, 60) 1 add_5[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3a_relu (ReLU) (None, 256, 34, 60) 0 add_5_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3a_relu_qdq (QDQ) (None, 256, 34, 60) 1 block_3a_relu[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3b_conv_1 (QuantizedConv2 (None, 256, 34, 60) 590080 block_3a_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3b_bn_1 (BatchNormalizati (None, 256, 34, 60) 1024 block_3b_conv_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3b_relu_1 (ReLU) (None, 256, 34, 60) 0 block_3b_bn_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3b_relu_1_qdq (QDQ) (None, 256, 34, 60) 1 block_3b_relu_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3b_conv_2 (QuantizedConv2 (None, 256, 34, 60) 590080 block_3b_relu_1_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3b_conv_shortcut (Quantiz (None, 256, 34, 60) 65792 block_3a_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3b_bn_2 (BatchNormalizati (None, 256, 34, 60) 1024 block_3b_conv_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3b_bn_shortcut (BatchNorm (None, 256, 34, 60) 1024 block_3b_conv_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3b_bn_2_qdq (QDQ) (None, 256, 34, 60) 1 block_3b_bn_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3b_bn_shortcut_qdq (QDQ) (None, 256, 34, 60) 1 block_3b_bn_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_6 (Add) (None, 256, 34, 60) 0 block_3b_bn_2_qdq[0][0] \n",
" block_3b_bn_shortcut_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_6_qdq (QDQ) (None, 256, 34, 60) 1 add_6[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3b_relu (ReLU) (None, 256, 34, 60) 0 add_6_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3b_relu_qdq (QDQ) (None, 256, 34, 60) 1 block_3b_relu[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4a_conv_1 (QuantizedConv2 (None, 512, 34, 60) 1180160 block_3b_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4a_bn_1 (BatchNormalizati (None, 512, 34, 60) 2048 block_4a_conv_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4a_relu_1 (ReLU) (None, 512, 34, 60) 0 block_4a_bn_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4a_relu_1_qdq (QDQ) (None, 512, 34, 60) 1 block_4a_relu_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4a_conv_2 (QuantizedConv2 (None, 512, 34, 60) 2359808 block_4a_relu_1_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4a_conv_shortcut (Quantiz (None, 512, 34, 60) 131584 block_3b_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4a_bn_2 (BatchNormalizati (None, 512, 34, 60) 2048 block_4a_conv_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4a_bn_shortcut (BatchNorm (None, 512, 34, 60) 2048 block_4a_conv_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4a_bn_2_qdq (QDQ) (None, 512, 34, 60) 1 block_4a_bn_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4a_bn_shortcut_qdq (QDQ) (None, 512, 34, 60) 1 block_4a_bn_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_7 (Add) (None, 512, 34, 60) 0 block_4a_bn_2_qdq[0][0] \n",
" block_4a_bn_shortcut_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_7_qdq (QDQ) (None, 512, 34, 60) 1 add_7[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4a_relu (ReLU) (None, 512, 34, 60) 0 add_7_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4a_relu_qdq (QDQ) (None, 512, 34, 60) 1 block_4a_relu[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4b_conv_1 (QuantizedConv2 (None, 512, 34, 60) 2359808 block_4a_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4b_bn_1 (BatchNormalizati (None, 512, 34, 60) 2048 block_4b_conv_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4b_relu_1 (ReLU) (None, 512, 34, 60) 0 block_4b_bn_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4b_relu_1_qdq (QDQ) (None, 512, 34, 60) 1 block_4b_relu_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4b_conv_2 (QuantizedConv2 (None, 512, 34, 60) 2359808 block_4b_relu_1_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4b_conv_shortcut (Quantiz (None, 512, 34, 60) 262656 block_4a_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4b_bn_2 (BatchNormalizati (None, 512, 34, 60) 2048 block_4b_conv_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4b_bn_shortcut (BatchNorm (None, 512, 34, 60) 2048 block_4b_conv_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4b_bn_2_qdq (QDQ) (None, 512, 34, 60) 1 block_4b_bn_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4b_bn_shortcut_qdq (QDQ) (None, 512, 34, 60) 1 block_4b_bn_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_8 (Add) (None, 512, 34, 60) 0 block_4b_bn_2_qdq[0][0] \n",
" block_4b_bn_shortcut_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_8_qdq (QDQ) (None, 512, 34, 60) 1 add_8[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4b_relu (ReLU) (None, 512, 34, 60) 0 add_8_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4b_relu_qdq (QDQ) (None, 512, 34, 60) 1 block_4b_relu[0][0] \n",
"__________________________________________________________________________________________________\n",
"output_bbox (Conv2D) (None, 4, 34, 60) 2052 block_4b_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"output_cov (Conv2D) (None, 1, 34, 60) 513 block_4b_relu_qdq[0][0] \n",
"==================================================================================================\n",
"Total params: 11,550,895\n",
"Trainable params: 11,539,205\n",
"Non-trainable params: 11,690\n",
"__________________________________________________________________________________________________\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/utilities.py:139: The name tf.train.Scaffold is deprecated. Please use tf.compat.v1.train.Scaffold instead.\n",
"\n",
"2022-01-06 08:11:10,523 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/utilities.py:139: The name tf.train.Scaffold is deprecated. Please use tf.compat.v1.train.Scaffold instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:14: The name tf.local_variables_initializer is deprecated. Please use tf.compat.v1.local_variables_initializer instead.\n",
"\n",
"2022-01-06 08:11:10,523 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:14: The name tf.local_variables_initializer is deprecated. Please use tf.compat.v1.local_variables_initializer instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:15: The name tf.tables_initializer is deprecated. Please use tf.compat.v1.tables_initializer instead.\n",
"\n",
"2022-01-06 08:11:10,524 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:15: The name tf.tables_initializer is deprecated. Please use tf.compat.v1.tables_initializer instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:16: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead.\n",
"\n",
"2022-01-06 08:11:10,524 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:16: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/utilities.py:140: The name tf.train.SingularMonitoredSession is deprecated. Please use tf.compat.v1.train.SingularMonitoredSession instead.\n",
"\n",
"2022-01-06 08:11:10,525 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/utilities.py:140: The name tf.train.SingularMonitoredSession is deprecated. Please use tf.compat.v1.train.SingularMonitoredSession instead.\n",
"\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO:tensorflow:Graph was finalized.\n",
"2022-01-06 08:11:10,959 [INFO] tensorflow: Graph was finalized.\n",
"INFO:tensorflow:Running local_init_op.\n",
"2022-01-06 08:11:11,583 [INFO] tensorflow: Running local_init_op.\n",
"INFO:tensorflow:Done running local_init_op.\n",
"2022-01-06 08:11:11,821 [INFO] tensorflow: Done running local_init_op.\n",
"2022-01-06 08:11:12,410 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 24, 0.00s/step\n",
"2022-01-06 08:11:18,287 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 24, 0.59s/step\n",
"2022-01-06 08:11:19,846 [INFO] iva.detectnet_v2.evaluation.evaluation: step 20 / 24, 0.16s/step\n",
"Matching predictions to ground truth, class 1/1.: 100%|█| 990/990 [00:00<00:00, 15370.58it/s]\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:95: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead.\n",
"\n",
"2022-01-06 08:11:20,526 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:95: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead.\n",
"\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:98: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead.\n",
"\n",
"2022-01-06 08:11:20,526 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:98: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead.\n",
"\n",
"\n",
"Validation cost: 0.001124\n",
"Mean average_precision (in %): 92.5777\n",
"\n",
"class name average precision (in %)\n",
"------------ --------------------------\n",
"car 92.5777\n",
"\n",
"Median Inference Time: 0.015083\n",
"2022-01-06 08:11:20,568 [INFO] __main__: Evaluation complete.\n",
"Time taken to run __main__:main: 0:00:13.326326.\n",
"2022-01-06 16:11:21,794 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.\n"
]
}
],
"source": [
"!tao detectnet_v2 evaluate -e $SPECS_DIR/detectnet_v2_train_resnet18_kitti.txt\\\n",
" -m $USER_EXPERIMENT_DIR/experiment_dir_unpruned/weights/resnet18_detector.tlt \\\n",
" -k tlt_encode"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 6. Prune the trained model \n",
"* Specify pre-trained model\n",
"* Equalization criterion (`Applicable for resnets and mobilenets`)\n",
"* Threshold for pruning.\n",
"* A key to save and load the model\n",
"* Output directory to store the model\n",
"\n",
"*Usually, you just need to adjust `-pth` (threshold) for accuracy and model size trade off. Higher `pth` gives you smaller model (and thus higher inference speed) but worse accuracy. The threshold to use is dependent on the dataset. A pth value `5.2e-6` is just a start point. If the retrain accuracy is good, you can increase this value to get smaller models. Otherwise, lower this value to get better accuracy.*\n",
"\n",
"*For some internal studies, we have noticed that a pth value of 0.01 is a good starting point for detectnet_v2 models.*"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [],
"source": [
"# Create an output directory if it doesn't exist.\n",
"!mkdir -p $LOCAL_EXPERIMENT_DIR/experiment_dir_pruned"
]
},
{
"cell_type": "code",
"execution_count": 31,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"2022-01-06 16:11:38,259 [INFO] root: Registry: ['nvcr.io']\n",
"Matplotlib created a temporary config/cache directory at /tmp/matplotlib-gs4_jgeb because the default path (/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.\n",
"Using TensorFlow backend.\n",
"WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.\n",
"Using TensorFlow backend.\n",
"2022-01-06 08:11:44,217 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:44,251 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:44,281 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:44,337 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:44,339 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:44,342 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:44,344 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:44,374 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:44,430 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:44,432 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:44,435 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:44,438 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:44,467 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:44,524 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:44,526 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:44,528 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:44,531 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:44,561 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:44,617 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:44,619 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:44,622 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:44,624 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:44,654 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:44,710 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:44,712 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:44,715 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:44,717 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:44,747 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:44,803 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:44,805 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:44,808 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:44,811 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:44,841 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:44,896 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:44,899 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:44,901 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:44,904 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:44,933 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:44,989 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:44,991 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:44,994 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:44,997 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:45,993 [INFO] modulus.pruning.pruning: Exploring graph for retainable indices\n",
"2022-01-06 08:11:46,570 [INFO] modulus.pruning.pruning: Pruning model and appending pruned nodes to new graph\n",
"2022-01-06 08:11:46,572 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:47,029 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:47,893 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:47,895 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:48,466 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:48,469 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:48,471 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:49,457 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:49,459 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:50,035 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:50,038 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:50,041 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:51,123 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:51,125 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"2022-01-06 08:11:51,759 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:51,762 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:51,764 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:52,958 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:52,961 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:53,652 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:53,654 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:53,657 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:54,924 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:54,927 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:55,662 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:55,665 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:55,668 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:57,041 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:57,044 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:57,837 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:57,840 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:57,843 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:59,329 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:11:59,332 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:12:00,196 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:12:00,198 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:12:00,201 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:12:01,836 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:12:01,839 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:12:02,765 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:12:02,768 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:12:02,771 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-06 08:12:03,667 [INFO] iva.common.magnet_prune: Pruning ratio (pruned model / original model): 0.11881252491690038\n",
"2022-01-06 16:12:06,552 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.\n"
]
}
],
"source": [
"!tao detectnet_v2 prune \\\n",
" -m $USER_EXPERIMENT_DIR/experiment_dir_unpruned/weights/resnet18_detector.tlt \\\n",
" -o $USER_EXPERIMENT_DIR/experiment_dir_pruned/resnet18_nopool_bn_detectnet_v2_pruned.tlt \\\n",
" -eq union \\\n",
" -pth 0.05 \\\n",
" -k $KEY"
]
},
{
"cell_type": "code",
"execution_count": 32,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"total 11560\r\n",
"-rw-r--r-- 1 guest guest 5987320 Dec 31 09:27 resnet18_nopool_bn_detectnet_v2_pruned_qat.tlt\r\n",
"-rw-r--r-- 1 guest guest 5847776 Jan 6 16:12 resnet18_nopool_bn_detectnet_v2_pruned.tlt\r\n"
]
}
],
"source": [
"!ls -rlt $LOCAL_EXPERIMENT_DIR/experiment_dir_pruned/"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 7. Retrain the pruned model \n",
"* Model needs to be re-trained to bring back accuracy after pruning\n",
"* Specify re-training specification with pretrained weights as pruned model.\n",
"\n",
"*Note: For retraining, please set the `load_graph` option to `true` in the model_config to load the pruned model graph. Also, if after retraining, the model shows some decrease in mAP, it could be that the originally trained model was pruned a little too much. Please try reducing the pruning threshold (thereby reducing the pruning ratio) and use the new model to retrain.*\n",
"\n",
"*Note: DetectNet_v2 now supports Quantization Aware Training, to help with optmizing the model. By default, the training in the cell below doesn't run the model with QAT enabled. For information on training a model with QAT, please refer to the cells under [section 11](#head-11)*"
]
},
{
"cell_type": "code",
"execution_count": 37,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"random_seed: 42\r\n",
"dataset_config {\r\n",
" data_sources {\r\n",
" tfrecords_path: \"/workspace/tao-experiments/car_data/tfrecords/kitti_trainval/*\"\r\n",
" image_directory_path: \"/workspace/tao-experiments/car_data/training/\"\r\n",
" }\r\n",
" image_extension: \"png\"\r\n",
" target_class_mapping{\r\n",
" key:\"car\"\r\n",
" value:\"car\"\r\n",
" }\r\n",
" validation_fold: 0\r\n",
"}\r\n",
"augmentation_config {\r\n",
" preprocessing {\r\n",
" output_image_width: 960\r\n",
" output_image_height: 544\r\n",
" min_bbox_width: 1.0\r\n",
" min_bbox_height: 1.0\r\n",
" output_image_channel: 3\r\n",
" enable_auto_resize: true\r\n",
" }\r\n",
" spatial_augmentation {\r\n",
" hflip_probability: 0.5\r\n",
" vflip_probability: 0.0\r\n",
" zoom_min: 1.0\r\n",
" zoom_max: 1.0\r\n",
" translate_max_x: 8.0\r\n",
" translate_max_y: 8.0\r\n",
" }\r\n",
" color_augmentation {\r\n",
" hue_rotation_max: 25.0\r\n",
" saturation_shift_max: 0.20000000298\r\n",
" contrast_scale_max: 0.10000000149\r\n",
" contrast_center: 0.5\r\n",
" }\r\n",
"}\r\n",
"\r\n",
"postprocessing_config {\r\n",
" target_class_config {\r\n",
" key: \"car\"\r\n",
" value {\r\n",
" clustering_config {\r\n",
" clustering_algorithm: DBSCAN\r\n",
" coverage_threshold: 0.005\r\n",
" dbscan_eps: 0.15\r\n",
" dbscan_min_samples: 0.05\r\n",
" minimum_bounding_box_height: 4\r\n",
" dbscan_confidence_threshold: 0.9\r\n",
" }\r\n",
" }\r\n",
" }\r\n",
"}\r\n",
"model_config {\r\n",
" pretrained_model_file: \"/workspace/tao-experiments/detectnet_v2_car/pretrained_trafficcamnet/resnet18_trafficcamnet.tlt\"\r\n",
" num_layers: 18\r\n",
" use_batch_norm: true\r\n",
" objective_set {\r\n",
" bbox {\r\n",
" scale: 35.0\r\n",
" offset: 0.5\r\n",
" }\r\n",
" cov {\r\n",
" }\r\n",
" }\r\n",
" training_precision {\r\n",
" backend_floatx: FLOAT32\r\n",
" }\r\n",
" arch: \"resnet\"\r\n",
" all_projections: true\r\n",
"}\r\n",
"evaluation_config {\r\n",
" validation_period_during_training: 10\r\n",
" first_validation_epoch: 20\r\n",
" minimum_detection_ground_truth_overlap {\r\n",
" key: \"car\"\r\n",
" value: 0.5\r\n",
" }\r\n",
" evaluation_box_config {\r\n",
" key: \"car\"\r\n",
" value {\r\n",
" minimum_height: 20\r\n",
" maximum_height: 9999\r\n",
" minimum_width: 10\r\n",
" maximum_width: 9999\r\n",
" }\r\n",
" }\r\n",
" average_precision_mode: INTEGRATE\r\n",
"}\r\n",
"\r\n",
"cost_function_config {\r\n",
" target_classes {\r\n",
" name: \"car\"\r\n",
" class_weight: 1.0\r\n",
" coverage_foreground_weight: 0.05\r\n",
" objectives {\r\n",
" name: \"cov\"\r\n",
" initial_weight: 1.0\r\n",
" weight_target: 1.0\r\n",
" }\r\n",
" objectives {\r\n",
" name: \"bbox\"\r\n",
" initial_weight: 10.0\r\n",
" weight_target: 10.0\r\n",
" }\r\n",
" }\r\n",
" enable_autoweighting: true\r\n",
" max_objective_weight: 0.999899983406\r\n",
" min_objective_weight: 9.99999974738e-05\r\n",
"}\r\n",
"training_config {\r\n",
" batch_size_per_gpu: 8\r\n",
" num_epochs:120\r\n",
" enable_qat:true\r\n",
" learning_rate {\r\n",
" soft_start_annealing_schedule {\r\n",
" min_learning_rate: 5e-06\r\n",
" max_learning_rate: 1e-03\r\n",
" soft_start: 0.10000000149\r\n",
" annealing: 0.699999988079\r\n",
" }\r\n",
" }\r\n",
" regularizer {\r\n",
" type: L1\r\n",
" weight: 3.00000002618e-09\r\n",
" }\r\n",
" optimizer {\r\n",
" adam {\r\n",
" epsilon: 9.99999993923e-09\r\n",
" beta1: 0.899999976158\r\n",
" beta2: 0.999000012875\r\n",
" }\r\n",
" }\r\n",
" cost_scaling {\r\n",
" enabled: False\r\n",
" initial_exponent: 20.0\r\n",
" increment: 0.005\r\n",
" decrement: 1.0\r\n",
" }\r\n",
" checkpoint_interval: 10\r\n",
"}\r\n",
"bbox_rasterizer_config {\r\n",
" target_class_config {\r\n",
" key: \"car\"\r\n",
" value: {\r\n",
" cov_center_x: 0.5\r\n",
" cov_center_y: 0.5\r\n",
" cov_radius_x: 0.4\r\n",
" cov_radius_y: 0.4\r\n",
" bbox_min_radius: 1.0\r\n",
" }\r\n",
" }\r\n",
" deadzone_radius: 0.4\r\n",
"}\r\n",
"\r\n"
]
}
],
"source": [
"# Printing the retrain experiment file. \n",
"# Note: We have updated the experiment file to include the \n",
"# newly pruned model as a pretrained weights and, the\n",
"# load_graph option is set to true \n",
"!cat $LOCAL_SPECS_DIR/detectnet_v2_retrain_resnet18_kitti.txt"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"2021-12-31 09:41:01,171 [INFO] root: Registry: ['nvcr.io']\n",
"Matplotlib created a temporary config/cache directory at /tmp/matplotlib-4fv0ff_k because the default path (/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.\n",
"Using TensorFlow backend.\n",
"WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.\n",
"Using TensorFlow backend.\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:43: The name tf.train.SessionRunHook is deprecated. Please use tf.estimator.SessionRunHook instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/checkpoint_saver_hook.py:25: The name tf.train.CheckpointSaverHook is deprecated. Please use tf.estimator.CheckpointSaverHook instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/scripts/train.py:68: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/scripts/train.py:68: The name tf.logging.INFO is deprecated. Please use tf.compat.v1.logging.INFO instead.\n",
"\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod/tensorflow/__init__.py:117: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.\n",
"\n",
"2021-12-31 01:41:06,769 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/horovod/tensorflow/__init__.py:117: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.\n",
"\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod/tensorflow/__init__.py:143: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.\n",
"\n",
"2021-12-31 01:41:06,770 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/horovod/tensorflow/__init__.py:143: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.\n",
"\n",
"2021-12-31 01:41:07,169 [INFO] __main__: Loading experiment spec at /workspace/tao-experiments/detectnet_v2_car/specs/detectnet_v2_retrain_resnet18_kitti_car.txt.\n",
"2021-12-31 01:41:07,171 [INFO] iva.detectnet_v2.spec_handler.spec_loader: Merging specification from /workspace/tao-experiments/detectnet_v2_car/specs/detectnet_v2_retrain_resnet18_kitti_car.txt\n",
"2021-12-31 01:41:07,285 [INFO] __main__: Cannot iterate over exactly 761 samples with a batch size of 8; each epoch will therefore take one extra step.\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:107: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.\n",
"\n",
"2021-12-31 01:41:07,287 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:107: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:110: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.\n",
"\n",
"2021-12-31 01:41:07,287 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:110: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:113: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead.\n",
"\n",
"2021-12-31 01:41:07,289 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:113: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead.\n",
"\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.\n",
"\n",
"2021-12-31 01:41:07,308 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.\n",
"\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.\n",
"\n",
"2021-12-31 01:41:07,309 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.\n",
"\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1834: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead.\n",
"\n",
"2021-12-31 01:41:07,326 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1834: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead.\n",
"\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.\n",
"\n",
"2021-12-31 01:41:08,851 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.\n",
"\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead.\n",
"\n",
"2021-12-31 01:41:08,852 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead.\n",
"\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead.\n",
"\n",
"2021-12-31 01:41:09,102 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead.\n",
"\n",
"/usr/local/lib/python3.6/dist-packages/keras/engine/saving.py:292: UserWarning: No training configuration found in save file: the model was *not* compiled. Compile it manually.\n",
" warnings.warn('No training configuration found in save file: '\n",
"2021-12-31 01:41:15,046 [INFO] iva.detectnet_v2.objectives.bbox_objective: Default L1 loss function will be used.\n",
"2021-12-31 01:41:15,069 [INFO] iva.detectnet_v2.model.detectnet_model: Converting the keras model to quantize keras model.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"__________________________________________________________________________________________________\r\n",
"Layer (type) Output Shape Param # Connected to \r\n",
"==================================================================================================\r\n",
"input_1 (InputLayer) (None, 3, 544, 960) 0 \r\n",
"__________________________________________________________________________________________________\r\n",
"input_1_qdq (QDQ) (None, 3, 544, 960) 1 input_1[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"conv1 (QuantizedConv2D) (None, 64, 272, 480) 9472 input_1_qdq[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"bn_conv1 (BatchNormalization) (None, 64, 272, 480) 256 conv1[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"activation_1 (ReLU) (None, 64, 272, 480) 0 bn_conv1[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"activation_1_qdq (QDQ) (None, 64, 272, 480) 1 activation_1[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_1a_conv_1 (QuantizedConv2 (None, 64, 136, 240) 36928 activation_1_qdq[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_1a_bn_1 (BatchNormalizati (None, 64, 136, 240) 256 block_1a_conv_1[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_1a_relu_1 (ReLU) (None, 64, 136, 240) 0 block_1a_bn_1[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_1a_relu_1_qdq (QDQ) (None, 64, 136, 240) 1 block_1a_relu_1[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_1a_conv_2 (QuantizedConv2 (None, 64, 136, 240) 36928 block_1a_relu_1_qdq[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_1a_conv_shortcut (Quantiz (None, 64, 136, 240) 4160 activation_1_qdq[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_1a_bn_2 (BatchNormalizati (None, 64, 136, 240) 256 block_1a_conv_2[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_1a_bn_shortcut (BatchNorm (None, 64, 136, 240) 256 block_1a_conv_shortcut[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_1a_bn_2_qdq (QDQ) (None, 64, 136, 240) 1 block_1a_bn_2[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_1a_bn_shortcut_qdq (QDQ) (None, 64, 136, 240) 1 block_1a_bn_shortcut[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"add_1 (Add) (None, 64, 136, 240) 0 block_1a_bn_2_qdq[0][0] \r\n",
" block_1a_bn_shortcut_qdq[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"add_1_qdq (QDQ) (None, 64, 136, 240) 1 add_1[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_1a_relu (ReLU) (None, 64, 136, 240) 0 add_1_qdq[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_1a_relu_qdq (QDQ) (None, 64, 136, 240) 1 block_1a_relu[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_1b_conv_1 (QuantizedConv2 (None, 64, 136, 240) 36928 block_1a_relu_qdq[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_1b_bn_1 (BatchNormalizati (None, 64, 136, 240) 256 block_1b_conv_1[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_1b_relu_1 (ReLU) (None, 64, 136, 240) 0 block_1b_bn_1[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_1b_relu_1_qdq (QDQ) (None, 64, 136, 240) 1 block_1b_relu_1[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_1b_conv_2 (QuantizedConv2 (None, 64, 136, 240) 36928 block_1b_relu_1_qdq[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_1b_conv_shortcut (Quantiz (None, 64, 136, 240) 4160 block_1a_relu_qdq[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_1b_bn_2 (BatchNormalizati (None, 64, 136, 240) 256 block_1b_conv_2[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_1b_bn_shortcut (BatchNorm (None, 64, 136, 240) 256 block_1b_conv_shortcut[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_1b_bn_2_qdq (QDQ) (None, 64, 136, 240) 1 block_1b_bn_2[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_1b_bn_shortcut_qdq (QDQ) (None, 64, 136, 240) 1 block_1b_bn_shortcut[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"add_2 (Add) (None, 64, 136, 240) 0 block_1b_bn_2_qdq[0][0] \r\n",
" block_1b_bn_shortcut_qdq[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"add_2_qdq (QDQ) (None, 64, 136, 240) 1 add_2[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_1b_relu (ReLU) (None, 64, 136, 240) 0 add_2_qdq[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_1b_relu_qdq (QDQ) (None, 64, 136, 240) 1 block_1b_relu[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_2a_conv_1 (QuantizedConv2 (None, 128, 68, 120) 73856 block_1b_relu_qdq[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_2a_bn_1 (BatchNormalizati (None, 128, 68, 120) 512 block_2a_conv_1[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_2a_relu_1 (ReLU) (None, 128, 68, 120) 0 block_2a_bn_1[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_2a_relu_1_qdq (QDQ) (None, 128, 68, 120) 1 block_2a_relu_1[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_2a_conv_2 (QuantizedConv2 (None, 128, 68, 120) 147584 block_2a_relu_1_qdq[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_2a_conv_shortcut (Quantiz (None, 128, 68, 120) 8320 block_1b_relu_qdq[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_2a_bn_2 (BatchNormalizati (None, 128, 68, 120) 512 block_2a_conv_2[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_2a_bn_shortcut (BatchNorm (None, 128, 68, 120) 512 block_2a_conv_shortcut[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_2a_bn_2_qdq (QDQ) (None, 128, 68, 120) 1 block_2a_bn_2[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_2a_bn_shortcut_qdq (QDQ) (None, 128, 68, 120) 1 block_2a_bn_shortcut[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"add_3 (Add) (None, 128, 68, 120) 0 block_2a_bn_2_qdq[0][0] \r\n",
" block_2a_bn_shortcut_qdq[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"add_3_qdq (QDQ) (None, 128, 68, 120) 1 add_3[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_2a_relu (ReLU) (None, 128, 68, 120) 0 add_3_qdq[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_2a_relu_qdq (QDQ) (None, 128, 68, 120) 1 block_2a_relu[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_2b_conv_1 (QuantizedConv2 (None, 128, 68, 120) 147584 block_2a_relu_qdq[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_2b_bn_1 (BatchNormalizati (None, 128, 68, 120) 512 block_2b_conv_1[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_2b_relu_1 (ReLU) (None, 128, 68, 120) 0 block_2b_bn_1[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_2b_relu_1_qdq (QDQ) (None, 128, 68, 120) 1 block_2b_relu_1[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_2b_conv_2 (QuantizedConv2 (None, 128, 68, 120) 147584 block_2b_relu_1_qdq[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_2b_conv_shortcut (Quantiz (None, 128, 68, 120) 16512 block_2a_relu_qdq[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_2b_bn_2 (BatchNormalizati (None, 128, 68, 120) 512 block_2b_conv_2[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_2b_bn_shortcut (BatchNorm (None, 128, 68, 120) 512 block_2b_conv_shortcut[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_2b_bn_2_qdq (QDQ) (None, 128, 68, 120) 1 block_2b_bn_2[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_2b_bn_shortcut_qdq (QDQ) (None, 128, 68, 120) 1 block_2b_bn_shortcut[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"add_4 (Add) (None, 128, 68, 120) 0 block_2b_bn_2_qdq[0][0] \r\n",
" block_2b_bn_shortcut_qdq[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"add_4_qdq (QDQ) (None, 128, 68, 120) 1 add_4[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_2b_relu (ReLU) (None, 128, 68, 120) 0 add_4_qdq[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_2b_relu_qdq (QDQ) (None, 128, 68, 120) 1 block_2b_relu[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_3a_conv_1 (QuantizedConv2 (None, 256, 34, 60) 295168 block_2b_relu_qdq[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_3a_bn_1 (BatchNormalizati (None, 256, 34, 60) 1024 block_3a_conv_1[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_3a_relu_1 (ReLU) (None, 256, 34, 60) 0 block_3a_bn_1[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_3a_relu_1_qdq (QDQ) (None, 256, 34, 60) 1 block_3a_relu_1[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_3a_conv_2 (QuantizedConv2 (None, 256, 34, 60) 590080 block_3a_relu_1_qdq[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_3a_conv_shortcut (Quantiz (None, 256, 34, 60) 33024 block_2b_relu_qdq[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_3a_bn_2 (BatchNormalizati (None, 256, 34, 60) 1024 block_3a_conv_2[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_3a_bn_shortcut (BatchNorm (None, 256, 34, 60) 1024 block_3a_conv_shortcut[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_3a_bn_2_qdq (QDQ) (None, 256, 34, 60) 1 block_3a_bn_2[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_3a_bn_shortcut_qdq (QDQ) (None, 256, 34, 60) 1 block_3a_bn_shortcut[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"add_5 (Add) (None, 256, 34, 60) 0 block_3a_bn_2_qdq[0][0] \r\n",
" block_3a_bn_shortcut_qdq[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"add_5_qdq (QDQ) (None, 256, 34, 60) 1 add_5[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_3a_relu (ReLU) (None, 256, 34, 60) 0 add_5_qdq[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_3a_relu_qdq (QDQ) (None, 256, 34, 60) 1 block_3a_relu[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_3b_conv_1 (QuantizedConv2 (None, 256, 34, 60) 590080 block_3a_relu_qdq[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_3b_bn_1 (BatchNormalizati (None, 256, 34, 60) 1024 block_3b_conv_1[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_3b_relu_1 (ReLU) (None, 256, 34, 60) 0 block_3b_bn_1[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_3b_relu_1_qdq (QDQ) (None, 256, 34, 60) 1 block_3b_relu_1[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_3b_conv_2 (QuantizedConv2 (None, 256, 34, 60) 590080 block_3b_relu_1_qdq[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_3b_conv_shortcut (Quantiz (None, 256, 34, 60) 65792 block_3a_relu_qdq[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_3b_bn_2 (BatchNormalizati (None, 256, 34, 60) 1024 block_3b_conv_2[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_3b_bn_shortcut (BatchNorm (None, 256, 34, 60) 1024 block_3b_conv_shortcut[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_3b_bn_2_qdq (QDQ) (None, 256, 34, 60) 1 block_3b_bn_2[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_3b_bn_shortcut_qdq (QDQ) (None, 256, 34, 60) 1 block_3b_bn_shortcut[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"add_6 (Add) (None, 256, 34, 60) 0 block_3b_bn_2_qdq[0][0] \r\n",
" block_3b_bn_shortcut_qdq[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"add_6_qdq (QDQ) (None, 256, 34, 60) 1 add_6[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_3b_relu (ReLU) (None, 256, 34, 60) 0 add_6_qdq[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_3b_relu_qdq (QDQ) (None, 256, 34, 60) 1 block_3b_relu[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_4a_conv_1 (QuantizedConv2 (None, 512, 34, 60) 1180160 block_3b_relu_qdq[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_4a_bn_1 (BatchNormalizati (None, 512, 34, 60) 2048 block_4a_conv_1[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_4a_relu_1 (ReLU) (None, 512, 34, 60) 0 block_4a_bn_1[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_4a_relu_1_qdq (QDQ) (None, 512, 34, 60) 1 block_4a_relu_1[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_4a_conv_2 (QuantizedConv2 (None, 512, 34, 60) 2359808 block_4a_relu_1_qdq[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_4a_conv_shortcut (Quantiz (None, 512, 34, 60) 131584 block_3b_relu_qdq[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_4a_bn_2 (BatchNormalizati (None, 512, 34, 60) 2048 block_4a_conv_2[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_4a_bn_shortcut (BatchNorm (None, 512, 34, 60) 2048 block_4a_conv_shortcut[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_4a_bn_2_qdq (QDQ) (None, 512, 34, 60) 1 block_4a_bn_2[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_4a_bn_shortcut_qdq (QDQ) (None, 512, 34, 60) 1 block_4a_bn_shortcut[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"add_7 (Add) (None, 512, 34, 60) 0 block_4a_bn_2_qdq[0][0] \r\n",
" block_4a_bn_shortcut_qdq[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"add_7_qdq (QDQ) (None, 512, 34, 60) 1 add_7[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_4a_relu (ReLU) (None, 512, 34, 60) 0 add_7_qdq[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_4a_relu_qdq (QDQ) (None, 512, 34, 60) 1 block_4a_relu[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_4b_conv_1 (QuantizedConv2 (None, 512, 34, 60) 2359808 block_4a_relu_qdq[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_4b_bn_1 (BatchNormalizati (None, 512, 34, 60) 2048 block_4b_conv_1[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_4b_relu_1 (ReLU) (None, 512, 34, 60) 0 block_4b_bn_1[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_4b_relu_1_qdq (QDQ) (None, 512, 34, 60) 1 block_4b_relu_1[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_4b_conv_2 (QuantizedConv2 (None, 512, 34, 60) 2359808 block_4b_relu_1_qdq[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_4b_conv_shortcut (Quantiz (None, 512, 34, 60) 262656 block_4a_relu_qdq[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_4b_bn_2 (BatchNormalizati (None, 512, 34, 60) 2048 block_4b_conv_2[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_4b_bn_shortcut (BatchNorm (None, 512, 34, 60) 2048 block_4b_conv_shortcut[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_4b_bn_2_qdq (QDQ) (None, 512, 34, 60) 1 block_4b_bn_2[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_4b_bn_shortcut_qdq (QDQ) (None, 512, 34, 60) 1 block_4b_bn_shortcut[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"add_8 (Add) (None, 512, 34, 60) 0 block_4b_bn_2_qdq[0][0] \r\n",
" block_4b_bn_shortcut_qdq[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"add_8_qdq (QDQ) (None, 512, 34, 60) 1 add_8[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_4b_relu (ReLU) (None, 512, 34, 60) 0 add_8_qdq[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"block_4b_relu_qdq (QDQ) (None, 512, 34, 60) 1 block_4b_relu[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"output_bbox (Conv2D) (None, 4, 34, 60) 2052 block_4b_relu_qdq[0][0] \r\n",
"__________________________________________________________________________________________________\r\n",
"output_cov (Conv2D) (None, 1, 34, 60) 513 block_4b_relu_qdq[0][0] \r\n",
"==================================================================================================\r\n",
"Total params: 11,550,895\r\n",
"Trainable params: 11,539,205\r\n",
"Non-trainable params: 11,690\r\n",
"__________________________________________________________________________________________________\r\n",
"2021-12-31 01:41:38,637 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Serial augmentation enabled = False\r\n",
"2021-12-31 01:41:38,637 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Pseudo sharding enabled = False\r\n",
"2021-12-31 01:41:38,637 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Max Image Dimensions (all sources): (0, 0)\r\n",
"2021-12-31 01:41:38,637 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: number of cpus: 16, io threads: 32, compute threads: 16, buffered batches: 4\r\n",
"2021-12-31 01:41:38,637 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: total dataset size 761, number of sources: 1, batch size per gpu: 8, steps: 96\r\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.\n",
"\n",
"2021-12-31 01:41:38,668 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.\n",
"\n",
"WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code\n",
"2021-12-31 01:41:38,709 [WARNING] tensorflow: Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code\n",
"2021-12-31 01:41:38,728 [INFO] iva.detectnet_v2.dataloader.default_dataloader: Bounding box coordinates were detected in the input specification! Bboxes will be automatically converted to polygon coordinates.\n",
"2021-12-31 01:41:38,938 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: shuffle: True - shard 0 of 1\n",
"2021-12-31 01:41:38,943 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: sampling 1 datasets with weights:\n",
"2021-12-31 01:41:38,943 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: source: 0 weight: 1.000000\n",
"WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code\n",
"2021-12-31 01:41:38,955 [WARNING] tensorflow: Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/core/build_wheel.runfiles/ai_infra/moduluspy/modulus/blocks/data_loaders/multi_source_loader/types/images2d_reference.py:427: The name tf.image.resize_images is deprecated. Please use tf.image.resize instead.\n",
"\n",
"2021-12-31 01:41:38,975 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/core/build_wheel.runfiles/ai_infra/moduluspy/modulus/blocks/data_loaders/multi_source_loader/types/images2d_reference.py:427: The name tf.image.resize_images is deprecated. Please use tf.image.resize instead.\n",
"\n",
"2021-12-31 01:41:39,262 [INFO] __main__: Found 761 samples in training set\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/rasterizers/bbox_rasterizer.py:347: The name tf.bincount is deprecated. Please use tf.math.bincount instead.\n",
"\n",
"2021-12-31 01:41:39,349 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/rasterizers/bbox_rasterizer.py:347: The name tf.bincount is deprecated. Please use tf.math.bincount instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/training_proto_utilities.py:89: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.\n",
"\n",
"2021-12-31 01:41:39,507 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/training_proto_utilities.py:89: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/training_proto_utilities.py:36: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead.\n",
"\n",
"2021-12-31 01:41:39,521 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/training_proto_utilities.py:36: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_functions.py:17: The name tf.log is deprecated. Please use tf.math.log instead.\n",
"\n",
"2021-12-31 01:41:40,193 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_functions.py:17: The name tf.log is deprecated. Please use tf.math.log instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:235: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead.\n",
"\n",
"2021-12-31 01:41:40,201 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:235: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/model/detectnet_model.py:587: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead.\n",
"\n",
"2021-12-31 01:41:40,204 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/model/detectnet_model.py:587: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead.\n",
"\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"2021-12-31 01:41:41,539 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Serial augmentation enabled = False\n",
"2021-12-31 01:41:41,539 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Pseudo sharding enabled = False\n",
"2021-12-31 01:41:41,539 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Max Image Dimensions (all sources): (0, 0)\n",
"2021-12-31 01:41:41,540 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: number of cpus: 16, io threads: 32, compute threads: 16, buffered batches: 4\n",
"2021-12-31 01:41:41,540 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: total dataset size 190, number of sources: 1, batch size per gpu: 8, steps: 24\n",
"WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code\n",
"2021-12-31 01:41:41,548 [WARNING] tensorflow: Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code\n",
"2021-12-31 01:41:41,565 [INFO] iva.detectnet_v2.dataloader.default_dataloader: Bounding box coordinates were detected in the input specification! Bboxes will be automatically converted to polygon coordinates.\n",
"2021-12-31 01:41:41,769 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: shuffle: False - shard 0 of 1\n",
"2021-12-31 01:41:41,774 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: sampling 1 datasets with weights:\n",
"2021-12-31 01:41:41,774 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: source: 0 weight: 1.000000\n",
"WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code\n",
"2021-12-31 01:41:41,786 [WARNING] tensorflow: Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code\n",
"2021-12-31 01:41:41,986 [INFO] __main__: Found 190 samples in validation set\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/validation_hook.py:40: The name tf.summary.FileWriterCache is deprecated. Please use tf.compat.v1.summary.FileWriterCache instead.\n",
"\n",
"2021-12-31 01:41:42,566 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/validation_hook.py:40: The name tf.summary.FileWriterCache is deprecated. Please use tf.compat.v1.summary.FileWriterCache instead.\n",
"\n",
"2021-12-31 01:41:43,864 [INFO] __main__: Checkpoint interval: 10\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/scripts/train.py:108: The name tf.train.Scaffold is deprecated. Please use tf.compat.v1.train.Scaffold instead.\n",
"\n",
"2021-12-31 01:41:43,865 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/scripts/train.py:108: The name tf.train.Scaffold is deprecated. Please use tf.compat.v1.train.Scaffold instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:14: The name tf.local_variables_initializer is deprecated. Please use tf.compat.v1.local_variables_initializer instead.\n",
"\n",
"2021-12-31 01:41:43,865 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:14: The name tf.local_variables_initializer is deprecated. Please use tf.compat.v1.local_variables_initializer instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:15: The name tf.tables_initializer is deprecated. Please use tf.compat.v1.tables_initializer instead.\n",
"\n",
"2021-12-31 01:41:43,865 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:15: The name tf.tables_initializer is deprecated. Please use tf.compat.v1.tables_initializer instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:16: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead.\n",
"\n",
"2021-12-31 01:41:43,866 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:16: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/utils.py:59: The name tf.train.LoggingTensorHook is deprecated. Please use tf.estimator.LoggingTensorHook instead.\n",
"\n",
"2021-12-31 01:41:43,868 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/utils.py:59: The name tf.train.LoggingTensorHook is deprecated. Please use tf.estimator.LoggingTensorHook instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/utils.py:60: The name tf.train.StopAtStepHook is deprecated. Please use tf.estimator.StopAtStepHook instead.\n",
"\n",
"2021-12-31 01:41:43,868 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/utils.py:60: The name tf.train.StopAtStepHook is deprecated. Please use tf.estimator.StopAtStepHook instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/utils.py:73: The name tf.train.StepCounterHook is deprecated. Please use tf.estimator.StepCounterHook instead.\n",
"\n",
"2021-12-31 01:41:43,868 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/utils.py:73: The name tf.train.StepCounterHook is deprecated. Please use tf.estimator.StepCounterHook instead.\n",
"\n",
"INFO:tensorflow:Create CheckpointSaverHook.\n",
"2021-12-31 01:41:43,868 [INFO] tensorflow: Create CheckpointSaverHook.\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/utils.py:99: The name tf.train.SummarySaverHook is deprecated. Please use tf.estimator.SummarySaverHook instead.\n",
"\n",
"2021-12-31 01:41:43,868 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/utils.py:99: The name tf.train.SummarySaverHook is deprecated. Please use tf.estimator.SummarySaverHook instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/utilities.py:140: The name tf.train.SingularMonitoredSession is deprecated. Please use tf.compat.v1.train.SingularMonitoredSession instead.\n",
"\n",
"2021-12-31 01:41:43,869 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/utilities.py:140: The name tf.train.SingularMonitoredSession is deprecated. Please use tf.compat.v1.train.SingularMonitoredSession instead.\n",
"\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO:tensorflow:Graph was finalized.\n",
"2021-12-31 01:41:45,092 [INFO] tensorflow: Graph was finalized.\n",
"INFO:tensorflow:Running local_init_op.\n",
"2021-12-31 01:41:46,900 [INFO] tensorflow: Running local_init_op.\n",
"INFO:tensorflow:Done running local_init_op.\n",
"2021-12-31 01:41:47,428 [INFO] tensorflow: Done running local_init_op.\n",
"INFO:tensorflow:Saving checkpoints for step-0.\n",
"2021-12-31 01:41:55,602 [INFO] tensorflow: Saving checkpoints for step-0.\n",
"INFO:tensorflow:epoch = 0.0, learning_rate = 4.9999994e-06, loss = 0.058969934, step = 0\n",
"2021-12-31 01:42:30,675 [INFO] tensorflow: epoch = 0.0, learning_rate = 4.9999994e-06, loss = 0.058969934, step = 0\n",
"2021-12-31 01:42:30,677 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 0/120: loss: 0.05897 learning rate: 0.00000 Time taken: 0:00:00 ETA: 0:00:00\n",
"2021-12-31 01:42:30,678 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 1.083\n",
"INFO:tensorflow:global_step/sec: 2.04701\n",
"2021-12-31 01:42:35,073 [INFO] tensorflow: global_step/sec: 2.04701\n",
"INFO:tensorflow:epoch = 0.125, learning_rate = 5.2837117e-06, loss = 0.057994246, step = 12 (5.373 sec)\n",
"2021-12-31 01:42:36,048 [INFO] tensorflow: epoch = 0.125, learning_rate = 5.2837117e-06, loss = 0.057994246, step = 12 (5.373 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12693\n",
"2021-12-31 01:42:37,951 [INFO] tensorflow: global_step/sec: 3.12693\n",
"2021-12-31 01:42:39,890 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 12.047\n",
"INFO:tensorflow:global_step/sec: 3.10708\n",
"2021-12-31 01:42:40,848 [INFO] tensorflow: global_step/sec: 3.10708\n",
"INFO:tensorflow:epoch = 0.3020833333333333, learning_rate = 5.7134084e-06, loss = 0.05700021, step = 29 (5.452 sec)\n",
"2021-12-31 01:42:41,501 [INFO] tensorflow: epoch = 0.3020833333333333, learning_rate = 5.7134084e-06, loss = 0.05700021, step = 29 (5.452 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11212\n",
"2021-12-31 01:42:43,740 [INFO] tensorflow: global_step/sec: 3.11212\n",
"INFO:tensorflow:global_step/sec: 3.09211\n",
"2021-12-31 01:42:46,650 [INFO] tensorflow: global_step/sec: 3.09211\n",
"INFO:tensorflow:epoch = 0.47916666666666663, learning_rate = 6.1780506e-06, loss = 0.05604728, step = 46 (5.467 sec)\n",
"2021-12-31 01:42:46,968 [INFO] tensorflow: epoch = 0.47916666666666663, learning_rate = 6.1780506e-06, loss = 0.05604728, step = 46 (5.467 sec)\n",
"2021-12-31 01:42:47,913 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.929\n",
"INFO:tensorflow:global_step/sec: 3.12661\n",
"2021-12-31 01:42:49,529 [INFO] tensorflow: global_step/sec: 3.12661\n",
"INFO:tensorflow:epoch = 0.65625, learning_rate = 6.680479e-06, loss = 0.05479856, step = 63 (5.442 sec)\n",
"2021-12-31 01:42:52,410 [INFO] tensorflow: epoch = 0.65625, learning_rate = 6.680479e-06, loss = 0.05479856, step = 63 (5.442 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12288\n",
"2021-12-31 01:42:52,411 [INFO] tensorflow: global_step/sec: 3.12288\n",
"INFO:tensorflow:global_step/sec: 3.11473\n",
"2021-12-31 01:42:55,300 [INFO] tensorflow: global_step/sec: 3.11473\n",
"2021-12-31 01:42:55,946 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.899\n",
"INFO:tensorflow:epoch = 0.8333333333333333, learning_rate = 7.223768e-06, loss = 0.053019833, step = 80 (5.511 sec)\n",
"2021-12-31 01:42:57,921 [INFO] tensorflow: epoch = 0.8333333333333333, learning_rate = 7.223768e-06, loss = 0.053019833, step = 80 (5.511 sec)\n",
"INFO:tensorflow:global_step/sec: 3.05717\n",
"2021-12-31 01:42:58,244 [INFO] tensorflow: global_step/sec: 3.05717\n",
"INFO:tensorflow:global_step/sec: 3.15089\n",
"2021-12-31 01:43:01,101 [INFO] tensorflow: global_step/sec: 3.15089\n",
"f46173cbc852:60:96 [0] NCCL INFO Bootstrap : Using [0]lo:127.0.0.1<0> [1]eth0:172.17.0.31<0>\n",
"f46173cbc852:60:96 [0] NCCL INFO NET/Plugin : Plugin load returned 0 : libnccl-net.so: cannot open shared object file: No such file or directory.\n",
"f46173cbc852:60:96 [0] NCCL INFO NET/IB : No device found.\n",
"f46173cbc852:60:96 [0] NCCL INFO NET/Socket : Using [0]lo:127.0.0.1<0> [1]eth0:172.17.0.31<0>\n",
"f46173cbc852:60:96 [0] NCCL INFO Using network Socket\n",
"NCCL version 2.7.8+cuda11.1\n",
"f46173cbc852:60:96 [0] NCCL INFO Channel 00/32 : 0\n",
"f46173cbc852:60:96 [0] NCCL INFO Channel 01/32 : 0\n",
"f46173cbc852:60:96 [0] NCCL INFO Channel 02/32 : 0\n",
"f46173cbc852:60:96 [0] NCCL INFO Channel 03/32 : 0\n",
"f46173cbc852:60:96 [0] NCCL INFO Channel 04/32 : 0\n",
"f46173cbc852:60:96 [0] NCCL INFO Channel 05/32 : 0\n",
"f46173cbc852:60:96 [0] NCCL INFO Channel 06/32 : 0\n",
"f46173cbc852:60:96 [0] NCCL INFO Channel 07/32 : 0\n",
"f46173cbc852:60:96 [0] NCCL INFO Channel 08/32 : 0\n",
"f46173cbc852:60:96 [0] NCCL INFO Channel 09/32 : 0\n",
"f46173cbc852:60:96 [0] NCCL INFO Channel 10/32 : 0\n",
"f46173cbc852:60:96 [0] NCCL INFO Channel 11/32 : 0\n",
"f46173cbc852:60:96 [0] NCCL INFO Channel 12/32 : 0\n",
"f46173cbc852:60:96 [0] NCCL INFO Channel 13/32 : 0\n",
"f46173cbc852:60:96 [0] NCCL INFO Channel 14/32 : 0\n",
"f46173cbc852:60:96 [0] NCCL INFO Channel 15/32 : 0\n",
"f46173cbc852:60:96 [0] NCCL INFO Channel 16/32 : 0\n",
"f46173cbc852:60:96 [0] NCCL INFO Channel 17/32 : 0\n",
"f46173cbc852:60:96 [0] NCCL INFO Channel 18/32 : 0\n",
"f46173cbc852:60:96 [0] NCCL INFO Channel 19/32 : 0\n",
"f46173cbc852:60:96 [0] NCCL INFO Channel 20/32 : 0\n",
"f46173cbc852:60:96 [0] NCCL INFO Channel 21/32 : 0\n",
"f46173cbc852:60:96 [0] NCCL INFO Channel 22/32 : 0\n",
"f46173cbc852:60:96 [0] NCCL INFO Channel 23/32 : 0\n",
"f46173cbc852:60:96 [0] NCCL INFO Channel 24/32 : 0\n",
"f46173cbc852:60:96 [0] NCCL INFO Channel 25/32 : 0\n",
"f46173cbc852:60:96 [0] NCCL INFO Channel 26/32 : 0\n",
"f46173cbc852:60:96 [0] NCCL INFO Channel 27/32 : 0\n",
"f46173cbc852:60:96 [0] NCCL INFO Channel 28/32 : 0\n",
"f46173cbc852:60:96 [0] NCCL INFO Channel 29/32 : 0\n",
"f46173cbc852:60:96 [0] NCCL INFO Channel 30/32 : 0\n",
"f46173cbc852:60:96 [0] NCCL INFO Channel 31/32 : 0\n",
"f46173cbc852:60:96 [0] NCCL INFO Trees [0] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [1] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [2] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [3] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [4] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [5] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [6] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [7] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [8] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [9] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [10] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [11] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [12] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [13] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [14] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [15] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [16] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [17] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [18] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [19] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [20] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [21] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [22] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [23] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [24] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [25] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [26] -1/-1/-1->0->-1|-1->0->-1/-\n",
"f46173cbc852:60:96 [0] NCCL INFO 32 coll channels, 32 p2p channels, 32 p2p channels per peer\n",
"f46173cbc852:60:96 [0] NCCL INFO comm 0x7f00e8327070 rank 0 nranks 1 cudaDev 0 busId 1000 - Init COMPLETE\n",
"INFO:tensorflow:epoch = 1.0, learning_rate = 7.775394e-06, loss = 0.0023211944, step = 96 (5.433 sec)\n",
"2021-12-31 01:43:03,354 [INFO] tensorflow: epoch = 1.0, learning_rate = 7.775394e-06, loss = 0.0023211944, step = 96 (5.433 sec)\n",
"2021-12-31 01:43:03,355 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 1/120: loss: 0.00232 learning rate: 0.00001 Time taken: 0:00:39.755590 ETA: 1:18:50.915234\n",
"INFO:tensorflow:global_step/sec: 2.8074\n",
"2021-12-31 01:43:04,307 [INFO] tensorflow: global_step/sec: 2.8074\n",
"2021-12-31 01:43:04,307 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 23.921\n",
"INFO:tensorflow:global_step/sec: 3.11072\n",
"2021-12-31 01:43:07,200 [INFO] tensorflow: global_step/sec: 3.11072\n",
"INFO:tensorflow:epoch = 1.1770833333333333, learning_rate = 8.407727e-06, loss = 0.0027935752, step = 113 (5.489 sec)\n",
"2021-12-31 01:43:08,843 [INFO] tensorflow: epoch = 1.1770833333333333, learning_rate = 8.407727e-06, loss = 0.0027935752, step = 113 (5.489 sec)\n",
"INFO:tensorflow:global_step/sec: 3.03797\n",
"2021-12-31 01:43:10,162 [INFO] tensorflow: global_step/sec: 3.03797\n",
"2021-12-31 01:43:12,418 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.659\n",
"INFO:tensorflow:global_step/sec: 3.14592\n",
"2021-12-31 01:43:13,023 [INFO] tensorflow: global_step/sec: 3.14592\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO:tensorflow:epoch = 1.3541666666666665, learning_rate = 9.091485e-06, loss = 0.0023931498, step = 130 (5.468 sec)\n",
"2021-12-31 01:43:14,311 [INFO] tensorflow: epoch = 1.3541666666666665, learning_rate = 9.091485e-06, loss = 0.0023931498, step = 130 (5.468 sec)\n",
"INFO:tensorflow:global_step/sec: 3.06561\n",
"2021-12-31 01:43:15,959 [INFO] tensorflow: global_step/sec: 3.06561\n",
"INFO:tensorflow:global_step/sec: 3.13365\n",
"2021-12-31 01:43:18,831 [INFO] tensorflow: global_step/sec: 3.13365\n",
"INFO:tensorflow:epoch = 1.53125, learning_rate = 9.830848e-06, loss = 0.0024762782, step = 147 (5.477 sec)\n",
"2021-12-31 01:43:19,788 [INFO] tensorflow: epoch = 1.53125, learning_rate = 9.830848e-06, loss = 0.0024762782, step = 147 (5.477 sec)\n",
"2021-12-31 01:43:20,429 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.968\n",
"INFO:tensorflow:global_step/sec: 3.09446\n",
"2021-12-31 01:43:21,739 [INFO] tensorflow: global_step/sec: 3.09446\n",
"INFO:tensorflow:global_step/sec: 3.20456\n",
"2021-12-31 01:43:24,548 [INFO] tensorflow: global_step/sec: 3.20456\n",
"INFO:tensorflow:epoch = 1.7083333333333333, learning_rate = 1.0630339e-05, loss = 0.0025045061, step = 164 (5.396 sec)\n",
"2021-12-31 01:43:25,183 [INFO] tensorflow: epoch = 1.7083333333333333, learning_rate = 1.0630339e-05, loss = 0.0025045061, step = 164 (5.396 sec)\n",
"INFO:tensorflow:global_step/sec: 3.1596\n",
"2021-12-31 01:43:27,396 [INFO] tensorflow: global_step/sec: 3.1596\n",
"2021-12-31 01:43:28,335 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.297\n",
"INFO:tensorflow:global_step/sec: 3.11028\n",
"2021-12-31 01:43:30,290 [INFO] tensorflow: global_step/sec: 3.11028\n",
"INFO:tensorflow:epoch = 1.8854166666666665, learning_rate = 1.149485e-05, loss = 0.0019736807, step = 181 (5.425 sec)\n",
"2021-12-31 01:43:30,609 [INFO] tensorflow: epoch = 1.8854166666666665, learning_rate = 1.149485e-05, loss = 0.0019736807, step = 181 (5.425 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12254\n",
"2021-12-31 01:43:33,172 [INFO] tensorflow: global_step/sec: 3.12254\n",
"2021-12-31 01:43:34,175 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 2/120: loss: 0.00264 learning rate: 0.00001 Time taken: 0:00:30.786063 ETA: 1:00:32.755485\n",
"INFO:tensorflow:epoch = 2.0625, learning_rate = 1.2429667e-05, loss = 0.0022930084, step = 198 (5.454 sec)\n",
"2021-12-31 01:43:36,063 [INFO] tensorflow: epoch = 2.0625, learning_rate = 1.2429667e-05, loss = 0.0022930084, step = 198 (5.454 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11234\n",
"2021-12-31 01:43:36,064 [INFO] tensorflow: global_step/sec: 3.11234\n",
"2021-12-31 01:43:36,377 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.871\n",
"INFO:tensorflow:global_step/sec: 3.13573\n",
"2021-12-31 01:43:38,934 [INFO] tensorflow: global_step/sec: 3.13573\n",
"INFO:tensorflow:epoch = 2.239583333333333, learning_rate = 1.3440507e-05, loss = 0.0025554649, step = 215 (5.456 sec)\n",
"2021-12-31 01:43:41,519 [INFO] tensorflow: epoch = 2.239583333333333, learning_rate = 1.3440507e-05, loss = 0.0025554649, step = 215 (5.456 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10444\n",
"2021-12-31 01:43:41,833 [INFO] tensorflow: global_step/sec: 3.10444\n",
"2021-12-31 01:43:44,372 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.014\n",
"INFO:tensorflow:global_step/sec: 3.15191\n",
"2021-12-31 01:43:44,689 [INFO] tensorflow: global_step/sec: 3.15191\n",
"INFO:tensorflow:epoch = 2.4166666666666665, learning_rate = 1.4533554e-05, loss = 0.0028007568, step = 232 (5.358 sec)\n",
"2021-12-31 01:43:46,878 [INFO] tensorflow: epoch = 2.4166666666666665, learning_rate = 1.4533554e-05, loss = 0.0028007568, step = 232 (5.358 sec)\n",
"INFO:tensorflow:global_step/sec: 3.1895\n",
"2021-12-31 01:43:47,510 [INFO] tensorflow: global_step/sec: 3.1895\n",
"INFO:tensorflow:global_step/sec: 3.11823\n",
"2021-12-31 01:43:50,397 [INFO] tensorflow: global_step/sec: 3.11823\n",
"INFO:tensorflow:epoch = 2.59375, learning_rate = 1.5715494e-05, loss = 0.0023034266, step = 249 (5.438 sec)\n",
"2021-12-31 01:43:52,316 [INFO] tensorflow: epoch = 2.59375, learning_rate = 1.5715494e-05, loss = 0.0023034266, step = 249 (5.438 sec)\n",
"2021-12-31 01:43:52,316 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.178\n",
"INFO:tensorflow:global_step/sec: 3.12242\n",
"2021-12-31 01:43:53,279 [INFO] tensorflow: global_step/sec: 3.12242\n",
"INFO:tensorflow:global_step/sec: 3.0854\n",
"2021-12-31 01:43:56,196 [INFO] tensorflow: global_step/sec: 3.0854\n",
"INFO:tensorflow:epoch = 2.770833333333333, learning_rate = 1.6993554e-05, loss = 0.002050074, step = 266 (5.512 sec)\n",
"2021-12-31 01:43:57,828 [INFO] tensorflow: epoch = 2.770833333333333, learning_rate = 1.6993554e-05, loss = 0.002050074, step = 266 (5.512 sec)\n",
"INFO:tensorflow:global_step/sec: 3.06765\n",
"2021-12-31 01:43:59,130 [INFO] tensorflow: global_step/sec: 3.06765\n",
"2021-12-31 01:44:00,410 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.710\n",
"INFO:tensorflow:global_step/sec: 3.13753\n",
"2021-12-31 01:44:01,998 [INFO] tensorflow: global_step/sec: 3.13753\n",
"INFO:tensorflow:epoch = 2.9479166666666665, learning_rate = 1.8375551e-05, loss = 0.0018788145, step = 283 (5.458 sec)\n",
"2021-12-31 01:44:03,286 [INFO] tensorflow: epoch = 2.9479166666666665, learning_rate = 1.8375551e-05, loss = 0.0018788145, step = 283 (5.458 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09284\n",
"2021-12-31 01:44:04,908 [INFO] tensorflow: global_step/sec: 3.09284\n",
"2021-12-31 01:44:04,909 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 3/120: loss: 0.00228 learning rate: 0.00002 Time taken: 0:00:30.754512 ETA: 0:59:58.277884\n",
"INFO:tensorflow:global_step/sec: 3.13281\n",
"2021-12-31 01:44:07,781 [INFO] tensorflow: global_step/sec: 3.13281\n",
"2021-12-31 01:44:08,440 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.909\n",
"INFO:tensorflow:epoch = 3.125, learning_rate = 1.986994e-05, loss = 0.0017210282, step = 300 (5.494 sec)\n",
"2021-12-31 01:44:08,780 [INFO] tensorflow: epoch = 3.125, learning_rate = 1.986994e-05, loss = 0.0017210282, step = 300 (5.494 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09333\n",
"2021-12-31 01:44:10,691 [INFO] tensorflow: global_step/sec: 3.09333\n",
"INFO:tensorflow:global_step/sec: 3.04538\n",
"2021-12-31 01:44:13,646 [INFO] tensorflow: global_step/sec: 3.04538\n",
"INFO:tensorflow:epoch = 3.302083333333333, learning_rate = 2.1485861e-05, loss = 0.0017661453, step = 317 (5.485 sec)\n",
"2021-12-31 01:44:14,265 [INFO] tensorflow: epoch = 3.302083333333333, learning_rate = 2.1485861e-05, loss = 0.0017661453, step = 317 (5.485 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08164\n",
"2021-12-31 01:44:16,566 [INFO] tensorflow: global_step/sec: 3.08164\n",
"2021-12-31 01:44:16,567 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.609\n",
"INFO:tensorflow:global_step/sec: 3.16571\n",
"2021-12-31 01:44:19,409 [INFO] tensorflow: global_step/sec: 3.16571\n",
"INFO:tensorflow:epoch = 3.4791666666666665, learning_rate = 2.3233195e-05, loss = 0.0014621357, step = 334 (5.462 sec)\n",
"2021-12-31 01:44:19,727 [INFO] tensorflow: epoch = 3.4791666666666665, learning_rate = 2.3233195e-05, loss = 0.0014621357, step = 334 (5.462 sec)\n",
"INFO:tensorflow:global_step/sec: 3.0805\n",
"2021-12-31 01:44:22,331 [INFO] tensorflow: global_step/sec: 3.0805\n",
"2021-12-31 01:44:24,548 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.061\n",
"INFO:tensorflow:epoch = 3.65625, learning_rate = 2.512263e-05, loss = 0.001313116, step = 351 (5.444 sec)\n",
"2021-12-31 01:44:25,171 [INFO] tensorflow: epoch = 3.65625, learning_rate = 2.512263e-05, loss = 0.001313116, step = 351 (5.444 sec)\n",
"INFO:tensorflow:global_step/sec: 3.16802\n",
"2021-12-31 01:44:25,172 [INFO] tensorflow: global_step/sec: 3.16802\n",
"INFO:tensorflow:global_step/sec: 3.14677\n",
"2021-12-31 01:44:28,032 [INFO] tensorflow: global_step/sec: 3.14677\n",
"INFO:tensorflow:epoch = 3.833333333333333, learning_rate = 2.7165725e-05, loss = 0.001522504, step = 368 (5.440 sec)\n",
"2021-12-31 01:44:30,612 [INFO] tensorflow: epoch = 3.833333333333333, learning_rate = 2.7165725e-05, loss = 0.001522504, step = 368 (5.440 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09871\n",
"2021-12-31 01:44:30,936 [INFO] tensorflow: global_step/sec: 3.09871\n",
"2021-12-31 01:44:32,485 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.197\n",
"INFO:tensorflow:global_step/sec: 3.17968\n",
"2021-12-31 01:44:33,767 [INFO] tensorflow: global_step/sec: 3.17968\n",
"2021-12-31 01:44:35,700 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 4/120: loss: 0.00146 learning rate: 0.00003 Time taken: 0:00:30.792034 ETA: 0:59:31.875934\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO:tensorflow:epoch = 4.010416666666666, learning_rate = 2.937497e-05, loss = 0.0016864456, step = 385 (5.407 sec)\n",
"2021-12-31 01:44:36,019 [INFO] tensorflow: epoch = 4.010416666666666, learning_rate = 2.937497e-05, loss = 0.0016864456, step = 385 (5.407 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08037\n",
"2021-12-31 01:44:36,689 [INFO] tensorflow: global_step/sec: 3.08037\n",
"INFO:tensorflow:global_step/sec: 3.13591\n",
"2021-12-31 01:44:39,559 [INFO] tensorflow: global_step/sec: 3.13591\n",
"2021-12-31 01:44:40,522 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.886\n",
"INFO:tensorflow:epoch = 4.1875, learning_rate = 3.1763888e-05, loss = 0.000793205, step = 402 (5.454 sec)\n",
"2021-12-31 01:44:41,473 [INFO] tensorflow: epoch = 4.1875, learning_rate = 3.1763888e-05, loss = 0.000793205, step = 402 (5.454 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10295\n",
"2021-12-31 01:44:42,459 [INFO] tensorflow: global_step/sec: 3.10295\n",
"INFO:tensorflow:global_step/sec: 3.14244\n",
"2021-12-31 01:44:45,323 [INFO] tensorflow: global_step/sec: 3.14244\n",
"INFO:tensorflow:epoch = 4.364583333333333, learning_rate = 3.434708e-05, loss = 0.0017773899, step = 419 (5.481 sec)\n",
"2021-12-31 01:44:46,954 [INFO] tensorflow: epoch = 4.364583333333333, learning_rate = 3.434708e-05, loss = 0.0017773899, step = 419 (5.481 sec)\n",
"INFO:tensorflow:global_step/sec: 3.1007\n",
"2021-12-31 01:44:48,226 [INFO] tensorflow: global_step/sec: 3.1007\n",
"2021-12-31 01:44:48,534 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.964\n",
"INFO:tensorflow:global_step/sec: 3.09173\n",
"2021-12-31 01:44:51,137 [INFO] tensorflow: global_step/sec: 3.09173\n",
"INFO:tensorflow:epoch = 4.541666666666666, learning_rate = 3.7140348e-05, loss = 0.0009884587, step = 436 (5.456 sec)\n",
"2021-12-31 01:44:52,410 [INFO] tensorflow: epoch = 4.541666666666666, learning_rate = 3.7140348e-05, loss = 0.0009884587, step = 436 (5.456 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08905\n",
"2021-12-31 01:44:54,050 [INFO] tensorflow: global_step/sec: 3.08905\n",
"2021-12-31 01:44:56,636 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.687\n",
"INFO:tensorflow:global_step/sec: 3.12395\n",
"2021-12-31 01:44:56,931 [INFO] tensorflow: global_step/sec: 3.12395\n",
"INFO:tensorflow:epoch = 4.71875, learning_rate = 4.016078e-05, loss = 0.0011869274, step = 453 (5.499 sec)\n",
"2021-12-31 01:44:57,909 [INFO] tensorflow: epoch = 4.71875, learning_rate = 4.016078e-05, loss = 0.0011869274, step = 453 (5.499 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08327\n",
"2021-12-31 01:44:59,850 [INFO] tensorflow: global_step/sec: 3.08327\n",
"INFO:tensorflow:global_step/sec: 3.04109\n",
"2021-12-31 01:45:02,809 [INFO] tensorflow: global_step/sec: 3.04109\n",
"INFO:tensorflow:epoch = 4.895833333333333, learning_rate = 4.342685e-05, loss = 0.00089002214, step = 470 (5.557 sec)\n",
"2021-12-31 01:45:03,466 [INFO] tensorflow: epoch = 4.895833333333333, learning_rate = 4.342685e-05, loss = 0.00089002214, step = 470 (5.557 sec)\n",
"2021-12-31 01:45:04,761 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.616\n",
"INFO:tensorflow:global_step/sec: 3.10878\n",
"2021-12-31 01:45:05,704 [INFO] tensorflow: global_step/sec: 3.10878\n",
"2021-12-31 01:45:06,664 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 5/120: loss: 0.00068 learning rate: 0.00005 Time taken: 0:00:30.977990 ETA: 0:59:22.468895\n",
"INFO:tensorflow:global_step/sec: 3.08728\n",
"2021-12-31 01:45:08,620 [INFO] tensorflow: global_step/sec: 3.08728\n",
"INFO:tensorflow:epoch = 5.072916666666666, learning_rate = 4.6958532e-05, loss = 0.0012303976, step = 487 (5.473 sec)\n",
"2021-12-31 01:45:08,939 [INFO] tensorflow: epoch = 5.072916666666666, learning_rate = 4.6958532e-05, loss = 0.0012303976, step = 487 (5.473 sec)\n",
"INFO:tensorflow:global_step/sec: 3.14504\n",
"2021-12-31 01:45:11,481 [INFO] tensorflow: global_step/sec: 3.14504\n",
"2021-12-31 01:45:12,767 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.982\n",
"INFO:tensorflow:epoch = 5.25, learning_rate = 5.0777428e-05, loss = 0.00076111016, step = 504 (5.464 sec)\n",
"2021-12-31 01:45:14,404 [INFO] tensorflow: epoch = 5.25, learning_rate = 5.0777428e-05, loss = 0.00076111016, step = 504 (5.464 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07884\n",
"2021-12-31 01:45:14,404 [INFO] tensorflow: global_step/sec: 3.07884\n",
"INFO:tensorflow:global_step/sec: 3.09702\n",
"2021-12-31 01:45:17,311 [INFO] tensorflow: global_step/sec: 3.09702\n",
"INFO:tensorflow:epoch = 5.427083333333333, learning_rate = 5.4906894e-05, loss = 0.0009956401, step = 521 (5.475 sec)\n",
"2021-12-31 01:45:19,879 [INFO] tensorflow: epoch = 5.427083333333333, learning_rate = 5.4906894e-05, loss = 0.0009956401, step = 521 (5.475 sec)\n",
"INFO:tensorflow:global_step/sec: 3.117\n",
"2021-12-31 01:45:20,198 [INFO] tensorflow: global_step/sec: 3.117\n",
"2021-12-31 01:45:20,838 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.781\n",
"INFO:tensorflow:global_step/sec: 3.06975\n",
"2021-12-31 01:45:23,130 [INFO] tensorflow: global_step/sec: 3.06975\n",
"INFO:tensorflow:epoch = 5.604166666666666, learning_rate = 5.937219e-05, loss = 0.00063140277, step = 538 (5.473 sec)\n",
"2021-12-31 01:45:25,352 [INFO] tensorflow: epoch = 5.604166666666666, learning_rate = 5.937219e-05, loss = 0.00063140277, step = 538 (5.473 sec)\n",
"INFO:tensorflow:global_step/sec: 3.14108\n",
"2021-12-31 01:45:25,995 [INFO] tensorflow: global_step/sec: 3.14108\n",
"INFO:tensorflow:global_step/sec: 3.11869\n",
"2021-12-31 01:45:28,881 [INFO] tensorflow: global_step/sec: 3.11869\n",
"2021-12-31 01:45:28,882 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.865\n",
"INFO:tensorflow:epoch = 5.78125, learning_rate = 6.420062e-05, loss = 0.0007482808, step = 555 (5.445 sec)\n",
"2021-12-31 01:45:30,797 [INFO] tensorflow: epoch = 5.78125, learning_rate = 6.420062e-05, loss = 0.0007482808, step = 555 (5.445 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12629\n",
"2021-12-31 01:45:31,760 [INFO] tensorflow: global_step/sec: 3.12629\n",
"INFO:tensorflow:global_step/sec: 3.05212\n",
"2021-12-31 01:45:34,708 [INFO] tensorflow: global_step/sec: 3.05212\n",
"INFO:tensorflow:epoch = 5.958333333333333, learning_rate = 6.942172e-05, loss = 0.0006537111, step = 572 (5.513 sec)\n",
"2021-12-31 01:45:36,311 [INFO] tensorflow: epoch = 5.958333333333333, learning_rate = 6.942172e-05, loss = 0.0006537111, step = 572 (5.513 sec)\n",
"2021-12-31 01:45:36,954 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.777\n",
"INFO:tensorflow:global_step/sec: 3.13925\n",
"2021-12-31 01:45:37,575 [INFO] tensorflow: global_step/sec: 3.13925\n",
"2021-12-31 01:45:37,576 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 6/120: loss: 0.00068 learning rate: 0.00007 Time taken: 0:00:30.897338 ETA: 0:58:42.296495\n",
"INFO:tensorflow:global_step/sec: 3.0942\n",
"2021-12-31 01:45:40,484 [INFO] tensorflow: global_step/sec: 3.0942\n",
"INFO:tensorflow:epoch = 6.135416666666666, learning_rate = 7.506744e-05, loss = 0.00062102475, step = 589 (5.491 sec)\n",
"2021-12-31 01:45:41,802 [INFO] tensorflow: epoch = 6.135416666666666, learning_rate = 7.506744e-05, loss = 0.00062102475, step = 589 (5.491 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09134\n",
"2021-12-31 01:45:43,395 [INFO] tensorflow: global_step/sec: 3.09134\n",
"2021-12-31 01:45:45,003 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.849\n",
"INFO:tensorflow:global_step/sec: 3.12655\n",
"2021-12-31 01:45:46,274 [INFO] tensorflow: global_step/sec: 3.12655\n",
"INFO:tensorflow:epoch = 6.3125, learning_rate = 8.1172286e-05, loss = 0.0006389144, step = 606 (5.454 sec)\n",
"2021-12-31 01:45:47,256 [INFO] tensorflow: epoch = 6.3125, learning_rate = 8.1172286e-05, loss = 0.0006389144, step = 606 (5.454 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09139\n",
"2021-12-31 01:45:49,185 [INFO] tensorflow: global_step/sec: 3.09139\n",
"INFO:tensorflow:global_step/sec: 3.07791\n",
"2021-12-31 01:45:52,109 [INFO] tensorflow: global_step/sec: 3.07791\n",
"INFO:tensorflow:epoch = 6.489583333333333, learning_rate = 8.777361e-05, loss = 0.00036057908, step = 623 (5.495 sec)\n",
"2021-12-31 01:45:52,751 [INFO] tensorflow: epoch = 6.489583333333333, learning_rate = 8.777361e-05, loss = 0.00036057908, step = 623 (5.495 sec)\n",
"2021-12-31 01:45:53,074 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.780\n",
"INFO:tensorflow:global_step/sec: 3.0752\n",
"2021-12-31 01:45:55,036 [INFO] tensorflow: global_step/sec: 3.0752\n",
"INFO:tensorflow:global_step/sec: 3.15799\n",
"2021-12-31 01:45:57,886 [INFO] tensorflow: global_step/sec: 3.15799\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO:tensorflow:epoch = 6.666666666666666, learning_rate = 9.491178e-05, loss = 0.00081462215, step = 640 (5.447 sec)\n",
"2021-12-31 01:45:58,198 [INFO] tensorflow: epoch = 6.666666666666666, learning_rate = 9.491178e-05, loss = 0.00081462215, step = 640 (5.447 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12179\n",
"2021-12-31 01:46:00,769 [INFO] tensorflow: global_step/sec: 3.12179\n",
"2021-12-31 01:46:01,081 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.980\n",
"INFO:tensorflow:epoch = 6.84375, learning_rate = 0.00010263046, loss = 0.0003845542, step = 657 (5.450 sec)\n",
"2021-12-31 01:46:03,649 [INFO] tensorflow: epoch = 6.84375, learning_rate = 0.00010263046, loss = 0.0003845542, step = 657 (5.450 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12453\n",
"2021-12-31 01:46:03,649 [INFO] tensorflow: global_step/sec: 3.12453\n",
"INFO:tensorflow:global_step/sec: 3.12105\n",
"2021-12-31 01:46:06,533 [INFO] tensorflow: global_step/sec: 3.12105\n",
"2021-12-31 01:46:08,511 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 7/120: loss: 0.00055 learning rate: 0.00011 Time taken: 0:00:30.923176 ETA: 0:58:14.318840\n",
"INFO:tensorflow:epoch = 7.020833333333333, learning_rate = 0.00011097687, loss = 0.0004453143, step = 674 (5.492 sec)\n",
"2021-12-31 01:46:09,140 [INFO] tensorflow: epoch = 7.020833333333333, learning_rate = 0.00011097687, loss = 0.0004453143, step = 674 (5.492 sec)\n",
"2021-12-31 01:46:09,141 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.815\n",
"INFO:tensorflow:global_step/sec: 3.0709\n",
"2021-12-31 01:46:09,464 [INFO] tensorflow: global_step/sec: 3.0709\n",
"INFO:tensorflow:global_step/sec: 3.11622\n",
"2021-12-31 01:46:12,352 [INFO] tensorflow: global_step/sec: 3.11622\n",
"INFO:tensorflow:epoch = 7.197916666666666, learning_rate = 0.00012000204, loss = 0.00033354832, step = 691 (5.420 sec)\n",
"2021-12-31 01:46:14,560 [INFO] tensorflow: epoch = 7.197916666666666, learning_rate = 0.00012000204, loss = 0.00033354832, step = 691 (5.420 sec)\n",
"INFO:tensorflow:global_step/sec: 3.15665\n",
"2021-12-31 01:46:15,203 [INFO] tensorflow: global_step/sec: 3.15665\n",
"2021-12-31 01:46:17,111 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.093\n",
"INFO:tensorflow:global_step/sec: 3.14934\n",
"2021-12-31 01:46:18,061 [INFO] tensorflow: global_step/sec: 3.14934\n",
"INFO:tensorflow:epoch = 7.375, learning_rate = 0.0001297612, loss = 0.0004976982, step = 708 (5.439 sec)\n",
"2021-12-31 01:46:19,999 [INFO] tensorflow: epoch = 7.375, learning_rate = 0.0001297612, loss = 0.0004976982, step = 708 (5.439 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10536\n",
"2021-12-31 01:46:20,959 [INFO] tensorflow: global_step/sec: 3.10536\n",
"INFO:tensorflow:global_step/sec: 3.03658\n",
"2021-12-31 01:46:23,923 [INFO] tensorflow: global_step/sec: 3.03658\n",
"2021-12-31 01:46:25,226 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.646\n",
"INFO:tensorflow:epoch = 7.552083333333333, learning_rate = 0.000140314, loss = 0.0003321673, step = 725 (5.541 sec)\n",
"2021-12-31 01:46:25,540 [INFO] tensorflow: epoch = 7.552083333333333, learning_rate = 0.000140314, loss = 0.0003321673, step = 725 (5.541 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07437\n",
"2021-12-31 01:46:26,850 [INFO] tensorflow: global_step/sec: 3.07437\n",
"INFO:tensorflow:global_step/sec: 3.13185\n",
"2021-12-31 01:46:29,724 [INFO] tensorflow: global_step/sec: 3.13185\n",
"INFO:tensorflow:epoch = 7.729166666666666, learning_rate = 0.000151725, loss = 0.00041486855, step = 742 (5.513 sec)\n",
"2021-12-31 01:46:31,053 [INFO] tensorflow: epoch = 7.729166666666666, learning_rate = 0.000151725, loss = 0.00041486855, step = 742 (5.513 sec)\n",
"INFO:tensorflow:global_step/sec: 3.06736\n",
"2021-12-31 01:46:32,658 [INFO] tensorflow: global_step/sec: 3.06736\n",
"2021-12-31 01:46:33,301 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.769\n",
"INFO:tensorflow:global_step/sec: 3.13831\n",
"2021-12-31 01:46:35,526 [INFO] tensorflow: global_step/sec: 3.13831\n",
"INFO:tensorflow:epoch = 7.90625, learning_rate = 0.00016406403, loss = 0.0004383511, step = 759 (5.441 sec)\n",
"2021-12-31 01:46:36,494 [INFO] tensorflow: epoch = 7.90625, learning_rate = 0.00016406403, loss = 0.0004383511, step = 759 (5.441 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10883\n",
"2021-12-31 01:46:38,421 [INFO] tensorflow: global_step/sec: 3.10883\n",
"2021-12-31 01:46:39,427 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 8/120: loss: 0.00044 learning rate: 0.00017 Time taken: 0:00:30.900189 ETA: 0:57:40.821186\n",
"INFO:tensorflow:global_step/sec: 3.10754\n",
"2021-12-31 01:46:41,317 [INFO] tensorflow: global_step/sec: 3.10754\n",
"2021-12-31 01:46:41,318 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.949\n",
"INFO:tensorflow:epoch = 8.083333333333332, learning_rate = 0.0001774065, loss = 0.00051641563, step = 776 (5.455 sec)\n",
"2021-12-31 01:46:41,949 [INFO] tensorflow: epoch = 8.083333333333332, learning_rate = 0.0001774065, loss = 0.00051641563, step = 776 (5.455 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10833\n",
"2021-12-31 01:46:44,212 [INFO] tensorflow: global_step/sec: 3.10833\n",
"INFO:tensorflow:global_step/sec: 3.16535\n",
"2021-12-31 01:46:47,056 [INFO] tensorflow: global_step/sec: 3.16535\n",
"INFO:tensorflow:epoch = 8.260416666666666, learning_rate = 0.00019183406, loss = 0.00039191145, step = 793 (5.446 sec)\n",
"2021-12-31 01:46:47,395 [INFO] tensorflow: epoch = 8.260416666666666, learning_rate = 0.00019183406, loss = 0.00039191145, step = 793 (5.446 sec)\n",
"2021-12-31 01:46:49,356 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.880\n",
"INFO:tensorflow:global_step/sec: 3.04078\n",
"2021-12-31 01:46:50,015 [INFO] tensorflow: global_step/sec: 3.04078\n",
"INFO:tensorflow:epoch = 8.4375, learning_rate = 0.00020743473, loss = 0.0002986429, step = 810 (5.461 sec)\n",
"2021-12-31 01:46:52,856 [INFO] tensorflow: epoch = 8.4375, learning_rate = 0.00020743473, loss = 0.0002986429, step = 810 (5.461 sec)\n",
"INFO:tensorflow:global_step/sec: 3.16716\n",
"2021-12-31 01:46:52,857 [INFO] tensorflow: global_step/sec: 3.16716\n",
"INFO:tensorflow:global_step/sec: 3.08786\n",
"2021-12-31 01:46:55,772 [INFO] tensorflow: global_step/sec: 3.08786\n",
"2021-12-31 01:46:57,336 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.066\n",
"INFO:tensorflow:epoch = 8.614583333333332, learning_rate = 0.00022430453, loss = 0.0003126427, step = 827 (5.455 sec)\n",
"2021-12-31 01:46:58,311 [INFO] tensorflow: epoch = 8.614583333333332, learning_rate = 0.00022430453, loss = 0.0003126427, step = 827 (5.455 sec)\n",
"INFO:tensorflow:global_step/sec: 3.14698\n",
"2021-12-31 01:46:58,632 [INFO] tensorflow: global_step/sec: 3.14698\n",
"INFO:tensorflow:global_step/sec: 3.16867\n",
"2021-12-31 01:47:01,472 [INFO] tensorflow: global_step/sec: 3.16867\n",
"INFO:tensorflow:epoch = 8.791666666666666, learning_rate = 0.00024254584, loss = 0.0004375312, step = 844 (5.429 sec)\n",
"2021-12-31 01:47:03,740 [INFO] tensorflow: epoch = 8.791666666666666, learning_rate = 0.00024254584, loss = 0.0004375312, step = 844 (5.429 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08217\n",
"2021-12-31 01:47:04,392 [INFO] tensorflow: global_step/sec: 3.08217\n",
"2021-12-31 01:47:05,377 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.873\n",
"INFO:tensorflow:global_step/sec: 3.07055\n",
"2021-12-31 01:47:07,323 [INFO] tensorflow: global_step/sec: 3.07055\n",
"INFO:tensorflow:epoch = 8.96875, learning_rate = 0.00026227083, loss = 0.0003553498, step = 861 (5.489 sec)\n",
"2021-12-31 01:47:09,229 [INFO] tensorflow: epoch = 8.96875, learning_rate = 0.00026227083, loss = 0.0003553498, step = 861 (5.489 sec)\n",
"INFO:tensorflow:global_step/sec: 3.13923\n",
"2021-12-31 01:47:10,190 [INFO] tensorflow: global_step/sec: 3.13923\n",
"2021-12-31 01:47:10,191 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 9/120: loss: 0.00042 learning rate: 0.00027 Time taken: 0:00:30.783256 ETA: 0:56:56.941396\n",
"INFO:tensorflow:global_step/sec: 3.08024\n",
"2021-12-31 01:47:13,112 [INFO] tensorflow: global_step/sec: 3.08024\n",
"2021-12-31 01:47:13,434 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.824\n",
"INFO:tensorflow:epoch = 9.145833333333332, learning_rate = 0.00028359998, loss = 0.00037444063, step = 878 (5.478 sec)\n",
"2021-12-31 01:47:14,708 [INFO] tensorflow: epoch = 9.145833333333332, learning_rate = 0.00028359998, loss = 0.00037444063, step = 878 (5.478 sec)\n",
"INFO:tensorflow:global_step/sec: 3.1373\n",
"2021-12-31 01:47:15,980 [INFO] tensorflow: global_step/sec: 3.1373\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO:tensorflow:global_step/sec: 3.02985\n",
"2021-12-31 01:47:18,951 [INFO] tensorflow: global_step/sec: 3.02985\n",
"INFO:tensorflow:epoch = 9.322916666666666, learning_rate = 0.000306664, loss = 0.00035389428, step = 895 (5.524 sec)\n",
"2021-12-31 01:47:20,231 [INFO] tensorflow: epoch = 9.322916666666666, learning_rate = 0.000306664, loss = 0.00035389428, step = 895 (5.524 sec)\n",
"2021-12-31 01:47:21,532 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.698\n",
"INFO:tensorflow:global_step/sec: 3.09731\n",
"2021-12-31 01:47:21,857 [INFO] tensorflow: global_step/sec: 3.09731\n",
"INFO:tensorflow:global_step/sec: 3.13015\n",
"2021-12-31 01:47:24,732 [INFO] tensorflow: global_step/sec: 3.13015\n",
"INFO:tensorflow:epoch = 9.5, learning_rate = 0.00033160305, loss = 0.0005043094, step = 912 (5.456 sec)\n",
"2021-12-31 01:47:25,687 [INFO] tensorflow: epoch = 9.5, learning_rate = 0.00033160305, loss = 0.0005043094, step = 912 (5.456 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07979\n",
"2021-12-31 01:47:27,654 [INFO] tensorflow: global_step/sec: 3.07979\n",
"2021-12-31 01:47:29,623 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.719\n",
"INFO:tensorflow:global_step/sec: 3.06135\n",
"2021-12-31 01:47:30,594 [INFO] tensorflow: global_step/sec: 3.06135\n",
"INFO:tensorflow:epoch = 9.677083333333332, learning_rate = 0.00035857083, loss = 0.00032001833, step = 929 (5.573 sec)\n",
"2021-12-31 01:47:31,260 [INFO] tensorflow: epoch = 9.677083333333332, learning_rate = 0.00035857083, loss = 0.00032001833, step = 929 (5.573 sec)\n",
"INFO:tensorflow:global_step/sec: 3.16318\n",
"2021-12-31 01:47:33,439 [INFO] tensorflow: global_step/sec: 3.16318\n",
"INFO:tensorflow:global_step/sec: 3.04747\n",
"2021-12-31 01:47:36,393 [INFO] tensorflow: global_step/sec: 3.04747\n",
"INFO:tensorflow:epoch = 9.854166666666666, learning_rate = 0.0003877315, loss = 0.0004651723, step = 946 (5.467 sec)\n",
"2021-12-31 01:47:36,727 [INFO] tensorflow: epoch = 9.854166666666666, learning_rate = 0.0003877315, loss = 0.0004651723, step = 946 (5.467 sec)\n",
"2021-12-31 01:47:37,688 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.798\n",
"INFO:tensorflow:global_step/sec: 3.09933\n",
"2021-12-31 01:47:39,296 [INFO] tensorflow: global_step/sec: 3.09933\n",
"INFO:tensorflow:Saving checkpoints for step-960.\n",
"2021-12-31 01:47:40,910 [INFO] tensorflow: Saving checkpoints for step-960.\n",
"INFO:tensorflow:epoch = 10.0, learning_rate = 0.00041351854, loss = 0.0002994545, step = 960 (8.117 sec)\n",
"2021-12-31 01:47:44,844 [INFO] tensorflow: epoch = 10.0, learning_rate = 0.00041351854, loss = 0.0002994545, step = 960 (8.117 sec)\n",
"2021-12-31 01:47:44,844 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 10/120: loss: 0.00030 learning rate: 0.00041 Time taken: 0:00:34.627409 ETA: 1:03:29.014988\n",
"INFO:tensorflow:global_step/sec: 1.37855\n",
"2021-12-31 01:47:45,825 [INFO] tensorflow: global_step/sec: 1.37855\n",
"INFO:tensorflow:global_step/sec: 3.1084\n",
"2021-12-31 01:47:48,720 [INFO] tensorflow: global_step/sec: 3.1084\n",
"2021-12-31 01:47:49,371 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 17.119\n",
"INFO:tensorflow:epoch = 10.177083333333332, learning_rate = 0.00044714788, loss = 0.00043771122, step = 977 (5.488 sec)\n",
"2021-12-31 01:47:50,332 [INFO] tensorflow: epoch = 10.177083333333332, learning_rate = 0.00044714788, loss = 0.00043771122, step = 977 (5.488 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10385\n",
"2021-12-31 01:47:51,620 [INFO] tensorflow: global_step/sec: 3.10385\n",
"INFO:tensorflow:global_step/sec: 3.044\n",
"2021-12-31 01:47:54,577 [INFO] tensorflow: global_step/sec: 3.044\n",
"INFO:tensorflow:epoch = 10.354166666666666, learning_rate = 0.0004835121, loss = 0.00039788074, step = 994 (5.522 sec)\n",
"2021-12-31 01:47:55,855 [INFO] tensorflow: epoch = 10.354166666666666, learning_rate = 0.0004835121, loss = 0.00039788074, step = 994 (5.522 sec)\n",
"INFO:tensorflow:global_step/sec: 3.14587\n",
"2021-12-31 01:47:57,438 [INFO] tensorflow: global_step/sec: 3.14587\n",
"2021-12-31 01:47:57,438 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.793\n",
"INFO:tensorflow:global_step/sec: 3.08208\n",
"2021-12-31 01:48:00,358 [INFO] tensorflow: global_step/sec: 3.08208\n",
"INFO:tensorflow:epoch = 10.53125, learning_rate = 0.0005228336, loss = 0.00035509525, step = 1011 (5.481 sec)\n",
"2021-12-31 01:48:01,335 [INFO] tensorflow: epoch = 10.53125, learning_rate = 0.0005228336, loss = 0.00035509525, step = 1011 (5.481 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09282\n",
"2021-12-31 01:48:03,268 [INFO] tensorflow: global_step/sec: 3.09282\n",
"2021-12-31 01:48:05,548 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.664\n",
"INFO:tensorflow:global_step/sec: 3.07094\n",
"2021-12-31 01:48:06,198 [INFO] tensorflow: global_step/sec: 3.07094\n",
"INFO:tensorflow:epoch = 10.708333333333332, learning_rate = 0.000565353, loss = 0.00048184424, step = 1028 (5.457 sec)\n",
"2021-12-31 01:48:06,792 [INFO] tensorflow: epoch = 10.708333333333332, learning_rate = 0.000565353, loss = 0.00048184424, step = 1028 (5.457 sec)\n",
"INFO:tensorflow:global_step/sec: 3.18638\n",
"2021-12-31 01:48:09,023 [INFO] tensorflow: global_step/sec: 3.18638\n",
"INFO:tensorflow:global_step/sec: 3.12475\n",
"2021-12-31 01:48:11,903 [INFO] tensorflow: global_step/sec: 3.12475\n",
"INFO:tensorflow:epoch = 10.885416666666666, learning_rate = 0.00061133027, loss = 0.00041989962, step = 1045 (5.431 sec)\n",
"2021-12-31 01:48:12,223 [INFO] tensorflow: epoch = 10.885416666666666, learning_rate = 0.00061133027, loss = 0.00041989962, step = 1045 (5.431 sec)\n",
"2021-12-31 01:48:13,514 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.106\n",
"INFO:tensorflow:global_step/sec: 3.09789\n",
"2021-12-31 01:48:14,808 [INFO] tensorflow: global_step/sec: 3.09789\n",
"2021-12-31 01:48:15,769 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 11/120: loss: 0.00038 learning rate: 0.00064 Time taken: 0:00:30.974803 ETA: 0:56:16.253576\n",
"INFO:tensorflow:epoch = 11.0625, learning_rate = 0.0006610466, loss = 0.0004710684, step = 1062 (5.468 sec)\n",
"2021-12-31 01:48:17,691 [INFO] tensorflow: epoch = 11.0625, learning_rate = 0.0006610466, loss = 0.0004710684, step = 1062 (5.468 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12134\n",
"2021-12-31 01:48:17,692 [INFO] tensorflow: global_step/sec: 3.12134\n",
"INFO:tensorflow:global_step/sec: 3.11292\n",
"2021-12-31 01:48:20,583 [INFO] tensorflow: global_step/sec: 3.11292\n",
"2021-12-31 01:48:21,524 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.971\n",
"INFO:tensorflow:epoch = 11.239583333333332, learning_rate = 0.00071480573, loss = 0.00032609742, step = 1079 (5.457 sec)\n",
"2021-12-31 01:48:23,148 [INFO] tensorflow: epoch = 11.239583333333332, learning_rate = 0.00071480573, loss = 0.00032609742, step = 1079 (5.457 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11577\n",
"2021-12-31 01:48:23,471 [INFO] tensorflow: global_step/sec: 3.11577\n",
"INFO:tensorflow:global_step/sec: 3.18103\n",
"2021-12-31 01:48:26,301 [INFO] tensorflow: global_step/sec: 3.18103\n",
"INFO:tensorflow:epoch = 11.416666666666666, learning_rate = 0.0007729376, loss = 0.00039248687, step = 1096 (5.428 sec)\n",
"2021-12-31 01:48:28,576 [INFO] tensorflow: epoch = 11.416666666666666, learning_rate = 0.0007729376, loss = 0.00039248687, step = 1096 (5.428 sec)\n",
"INFO:tensorflow:global_step/sec: 3.05807\n",
"2021-12-31 01:48:29,244 [INFO] tensorflow: global_step/sec: 3.05807\n",
"2021-12-31 01:48:29,571 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.853\n",
"INFO:tensorflow:global_step/sec: 3.12808\n",
"2021-12-31 01:48:32,121 [INFO] tensorflow: global_step/sec: 3.12808\n",
"INFO:tensorflow:epoch = 11.59375, learning_rate = 0.00083579664, loss = 0.00029612635, step = 1113 (5.467 sec)\n",
"2021-12-31 01:48:34,043 [INFO] tensorflow: epoch = 11.59375, learning_rate = 0.00083579664, loss = 0.00029612635, step = 1113 (5.467 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11886\n",
"2021-12-31 01:48:35,007 [INFO] tensorflow: global_step/sec: 3.11886\n",
"2021-12-31 01:48:37,566 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.017\n",
"INFO:tensorflow:global_step/sec: 3.13563\n",
"2021-12-31 01:48:37,877 [INFO] tensorflow: global_step/sec: 3.13563\n",
"INFO:tensorflow:epoch = 11.770833333333332, learning_rate = 0.0009037676, loss = 0.00054383284, step = 1130 (5.429 sec)\n",
"2021-12-31 01:48:39,472 [INFO] tensorflow: epoch = 11.770833333333332, learning_rate = 0.0009037676, loss = 0.00054383284, step = 1130 (5.429 sec)\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO:tensorflow:global_step/sec: 3.13869\n",
"2021-12-31 01:48:40,744 [INFO] tensorflow: global_step/sec: 3.13869\n",
"INFO:tensorflow:global_step/sec: 3.03852\n",
"2021-12-31 01:48:43,706 [INFO] tensorflow: global_step/sec: 3.03852\n",
"INFO:tensorflow:epoch = 11.947916666666666, learning_rate = 0.0009772663, loss = 0.00030768485, step = 1147 (5.487 sec)\n",
"2021-12-31 01:48:44,959 [INFO] tensorflow: epoch = 11.947916666666666, learning_rate = 0.0009772663, loss = 0.00030768485, step = 1147 (5.487 sec)\n",
"2021-12-31 01:48:45,612 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.859\n",
"INFO:tensorflow:global_step/sec: 3.158\n",
"2021-12-31 01:48:46,556 [INFO] tensorflow: global_step/sec: 3.158\n",
"2021-12-31 01:48:46,557 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 12/120: loss: 0.00033 learning rate: 0.00100 Time taken: 0:00:30.774258 ETA: 0:55:23.619853\n",
"INFO:tensorflow:global_step/sec: 3.07127\n",
"2021-12-31 01:48:49,486 [INFO] tensorflow: global_step/sec: 3.07127\n",
"INFO:tensorflow:epoch = 12.125, learning_rate = 0.0009999999, loss = 0.00037400075, step = 1164 (5.552 sec)\n",
"2021-12-31 01:48:50,511 [INFO] tensorflow: epoch = 12.125, learning_rate = 0.0009999999, loss = 0.00037400075, step = 1164 (5.552 sec)\n",
"INFO:tensorflow:global_step/sec: 3.02999\n",
"2021-12-31 01:48:52,457 [INFO] tensorflow: global_step/sec: 3.02999\n",
"2021-12-31 01:48:53,756 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.557\n",
"INFO:tensorflow:global_step/sec: 3.10294\n",
"2021-12-31 01:48:55,357 [INFO] tensorflow: global_step/sec: 3.10294\n",
"INFO:tensorflow:epoch = 12.302083333333332, learning_rate = 0.0009999999, loss = 0.0002667844, step = 1181 (5.481 sec)\n",
"2021-12-31 01:48:55,992 [INFO] tensorflow: epoch = 12.302083333333332, learning_rate = 0.0009999999, loss = 0.0002667844, step = 1181 (5.481 sec)\n",
"INFO:tensorflow:global_step/sec: 3.04157\n",
"2021-12-31 01:48:58,316 [INFO] tensorflow: global_step/sec: 3.04157\n",
"INFO:tensorflow:global_step/sec: 3.13484\n",
"2021-12-31 01:49:01,187 [INFO] tensorflow: global_step/sec: 3.13484\n",
"INFO:tensorflow:epoch = 12.479166666666666, learning_rate = 0.0009999999, loss = 0.0003693278, step = 1198 (5.513 sec)\n",
"2021-12-31 01:49:01,505 [INFO] tensorflow: epoch = 12.479166666666666, learning_rate = 0.0009999999, loss = 0.0003693278, step = 1198 (5.513 sec)\n",
"2021-12-31 01:49:01,829 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.774\n",
"INFO:tensorflow:global_step/sec: 3.10975\n",
"2021-12-31 01:49:04,081 [INFO] tensorflow: global_step/sec: 3.10975\n",
"INFO:tensorflow:epoch = 12.65625, learning_rate = 0.0009999999, loss = 0.00029870466, step = 1215 (5.492 sec)\n",
"2021-12-31 01:49:06,998 [INFO] tensorflow: epoch = 12.65625, learning_rate = 0.0009999999, loss = 0.00029870466, step = 1215 (5.492 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08549\n",
"2021-12-31 01:49:06,998 [INFO] tensorflow: global_step/sec: 3.08549\n",
"INFO:tensorflow:global_step/sec: 3.15505\n",
"2021-12-31 01:49:09,851 [INFO] tensorflow: global_step/sec: 3.15505\n",
"2021-12-31 01:49:09,852 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.931\n",
"INFO:tensorflow:epoch = 12.833333333333332, learning_rate = 0.0009999999, loss = 0.0005125304, step = 1232 (5.381 sec)\n",
"2021-12-31 01:49:12,378 [INFO] tensorflow: epoch = 12.833333333333332, learning_rate = 0.0009999999, loss = 0.0005125304, step = 1232 (5.381 sec)\n",
"INFO:tensorflow:global_step/sec: 3.14805\n",
"2021-12-31 01:49:12,710 [INFO] tensorflow: global_step/sec: 3.14805\n",
"INFO:tensorflow:global_step/sec: 3.11234\n",
"2021-12-31 01:49:15,601 [INFO] tensorflow: global_step/sec: 3.11234\n",
"2021-12-31 01:49:17,536 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 13/120: loss: 0.00036 learning rate: 0.00100 Time taken: 0:00:30.976914 ETA: 0:55:14.529816\n",
"INFO:tensorflow:epoch = 13.010416666666666, learning_rate = 0.0009999999, loss = 0.00031723065, step = 1249 (5.491 sec)\n",
"2021-12-31 01:49:17,870 [INFO] tensorflow: epoch = 13.010416666666666, learning_rate = 0.0009999999, loss = 0.00031723065, step = 1249 (5.491 sec)\n",
"2021-12-31 01:49:17,870 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.942\n",
"INFO:tensorflow:global_step/sec: 3.11464\n",
"2021-12-31 01:49:18,491 [INFO] tensorflow: global_step/sec: 3.11464\n",
"INFO:tensorflow:global_step/sec: 3.07585\n",
"2021-12-31 01:49:21,417 [INFO] tensorflow: global_step/sec: 3.07585\n",
"INFO:tensorflow:epoch = 13.1875, learning_rate = 0.0009999999, loss = 0.0004643197, step = 1266 (5.486 sec)\n",
"2021-12-31 01:49:23,356 [INFO] tensorflow: epoch = 13.1875, learning_rate = 0.0009999999, loss = 0.0004643197, step = 1266 (5.486 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10442\n",
"2021-12-31 01:49:24,316 [INFO] tensorflow: global_step/sec: 3.10442\n",
"2021-12-31 01:49:25,902 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.901\n",
"INFO:tensorflow:global_step/sec: 3.10526\n",
"2021-12-31 01:49:27,214 [INFO] tensorflow: global_step/sec: 3.10526\n",
"INFO:tensorflow:epoch = 13.364583333333332, learning_rate = 0.0009999999, loss = 0.00024274483, step = 1283 (5.468 sec)\n",
"2021-12-31 01:49:28,824 [INFO] tensorflow: epoch = 13.364583333333332, learning_rate = 0.0009999999, loss = 0.00024274483, step = 1283 (5.468 sec)\n",
"INFO:tensorflow:global_step/sec: 3.13045\n",
"2021-12-31 01:49:30,089 [INFO] tensorflow: global_step/sec: 3.13045\n",
"INFO:tensorflow:global_step/sec: 3.10621\n",
"2021-12-31 01:49:32,987 [INFO] tensorflow: global_step/sec: 3.10621\n",
"2021-12-31 01:49:33,979 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.763\n",
"INFO:tensorflow:epoch = 13.541666666666666, learning_rate = 0.0009999999, loss = 0.00024446723, step = 1300 (5.457 sec)\n",
"2021-12-31 01:49:34,281 [INFO] tensorflow: epoch = 13.541666666666666, learning_rate = 0.0009999999, loss = 0.00024446723, step = 1300 (5.457 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11184\n",
"2021-12-31 01:49:35,879 [INFO] tensorflow: global_step/sec: 3.11184\n",
"INFO:tensorflow:global_step/sec: 3.07696\n",
"2021-12-31 01:49:38,804 [INFO] tensorflow: global_step/sec: 3.07696\n",
"INFO:tensorflow:epoch = 13.71875, learning_rate = 0.0009999999, loss = 0.00025024646, step = 1317 (5.500 sec)\n",
"2021-12-31 01:49:39,781 [INFO] tensorflow: epoch = 13.71875, learning_rate = 0.0009999999, loss = 0.00025024646, step = 1317 (5.500 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10634\n",
"2021-12-31 01:49:41,701 [INFO] tensorflow: global_step/sec: 3.10634\n",
"2021-12-31 01:49:42,029 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.846\n",
"INFO:tensorflow:global_step/sec: 3.11175\n",
"2021-12-31 01:49:44,594 [INFO] tensorflow: global_step/sec: 3.11175\n",
"INFO:tensorflow:epoch = 13.895833333333332, learning_rate = 0.0009999999, loss = 0.0002970668, step = 1334 (5.466 sec)\n",
"2021-12-31 01:49:45,247 [INFO] tensorflow: epoch = 13.895833333333332, learning_rate = 0.0009999999, loss = 0.0002970668, step = 1334 (5.466 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09122\n",
"2021-12-31 01:49:47,505 [INFO] tensorflow: global_step/sec: 3.09122\n",
"2021-12-31 01:49:48,499 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 14/120: loss: 0.00027 learning rate: 0.00100 Time taken: 0:00:30.961372 ETA: 0:54:41.905396\n",
"2021-12-31 01:49:50,130 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.688\n",
"INFO:tensorflow:global_step/sec: 3.04031\n",
"2021-12-31 01:49:50,465 [INFO] tensorflow: global_step/sec: 3.04031\n",
"INFO:tensorflow:epoch = 14.072916666666666, learning_rate = 0.0009999999, loss = 0.0002648344, step = 1351 (5.553 sec)\n",
"2021-12-31 01:49:50,801 [INFO] tensorflow: epoch = 14.072916666666666, learning_rate = 0.0009999999, loss = 0.0002648344, step = 1351 (5.553 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11386\n",
"2021-12-31 01:49:53,356 [INFO] tensorflow: global_step/sec: 3.11386\n",
"INFO:tensorflow:epoch = 14.25, learning_rate = 0.0009999999, loss = 0.0004238726, step = 1368 (5.469 sec)\n",
"2021-12-31 01:49:56,269 [INFO] tensorflow: epoch = 14.25, learning_rate = 0.0009999999, loss = 0.0004238726, step = 1368 (5.469 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08772\n",
"2021-12-31 01:49:56,270 [INFO] tensorflow: global_step/sec: 3.08772\n",
"2021-12-31 01:49:58,253 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.622\n",
"INFO:tensorflow:global_step/sec: 3.04037\n",
"2021-12-31 01:49:59,230 [INFO] tensorflow: global_step/sec: 3.04037\n",
"INFO:tensorflow:epoch = 14.427083333333332, learning_rate = 0.0009999999, loss = 0.00028553046, step = 1385 (5.510 sec)\n",
"2021-12-31 01:50:01,779 [INFO] tensorflow: epoch = 14.427083333333332, learning_rate = 0.0009999999, loss = 0.00028553046, step = 1385 (5.510 sec)\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO:tensorflow:global_step/sec: 3.1256\n",
"2021-12-31 01:50:02,110 [INFO] tensorflow: global_step/sec: 3.1256\n",
"INFO:tensorflow:global_step/sec: 3.21525\n",
"2021-12-31 01:50:04,909 [INFO] tensorflow: global_step/sec: 3.21525\n",
"2021-12-31 01:50:06,227 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.084\n",
"INFO:tensorflow:epoch = 14.604166666666666, learning_rate = 0.0009999999, loss = 0.00023231363, step = 1402 (5.408 sec)\n",
"2021-12-31 01:50:07,187 [INFO] tensorflow: epoch = 14.604166666666666, learning_rate = 0.0009999999, loss = 0.00023231363, step = 1402 (5.408 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09202\n",
"2021-12-31 01:50:07,820 [INFO] tensorflow: global_step/sec: 3.09202\n",
"INFO:tensorflow:global_step/sec: 3.12481\n",
"2021-12-31 01:50:10,700 [INFO] tensorflow: global_step/sec: 3.12481\n",
"INFO:tensorflow:epoch = 14.78125, learning_rate = 0.0009999999, loss = 0.000219284, step = 1419 (5.426 sec)\n",
"2021-12-31 01:50:12,613 [INFO] tensorflow: epoch = 14.78125, learning_rate = 0.0009999999, loss = 0.000219284, step = 1419 (5.426 sec)\n",
"INFO:tensorflow:global_step/sec: 3.13569\n",
"2021-12-31 01:50:13,570 [INFO] tensorflow: global_step/sec: 3.13569\n",
"2021-12-31 01:50:14,226 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.002\n",
"INFO:tensorflow:global_step/sec: 3.12047\n",
"2021-12-31 01:50:16,454 [INFO] tensorflow: global_step/sec: 3.12047\n",
"INFO:tensorflow:epoch = 14.958333333333332, learning_rate = 0.0009999999, loss = 0.00030893757, step = 1436 (5.452 sec)\n",
"2021-12-31 01:50:18,064 [INFO] tensorflow: epoch = 14.958333333333332, learning_rate = 0.0009999999, loss = 0.00030893757, step = 1436 (5.452 sec)\n",
"INFO:tensorflow:global_step/sec: 3.06218\n",
"2021-12-31 01:50:19,393 [INFO] tensorflow: global_step/sec: 3.06218\n",
"2021-12-31 01:50:19,394 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 15/120: loss: 0.00026 learning rate: 0.00100 Time taken: 0:00:30.874594 ETA: 0:54:01.832417\n",
"INFO:tensorflow:global_step/sec: 3.12212\n",
"2021-12-31 01:50:22,276 [INFO] tensorflow: global_step/sec: 3.12212\n",
"2021-12-31 01:50:22,277 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.844\n",
"INFO:tensorflow:epoch = 15.135416666666666, learning_rate = 0.0009999999, loss = 0.00026962464, step = 1453 (5.493 sec)\n",
"2021-12-31 01:50:23,557 [INFO] tensorflow: epoch = 15.135416666666666, learning_rate = 0.0009999999, loss = 0.00026962464, step = 1453 (5.493 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08638\n",
"2021-12-31 01:50:25,192 [INFO] tensorflow: global_step/sec: 3.08638\n",
"INFO:tensorflow:global_step/sec: 3.13022\n",
"2021-12-31 01:50:28,067 [INFO] tensorflow: global_step/sec: 3.13022\n",
"INFO:tensorflow:epoch = 15.3125, learning_rate = 0.0009999999, loss = 0.0002460035, step = 1470 (5.496 sec)\n",
"2021-12-31 01:50:29,053 [INFO] tensorflow: epoch = 15.3125, learning_rate = 0.0009999999, loss = 0.0002460035, step = 1470 (5.496 sec)\n",
"2021-12-31 01:50:30,341 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.803\n",
"INFO:tensorflow:global_step/sec: 3.09717\n",
"2021-12-31 01:50:30,973 [INFO] tensorflow: global_step/sec: 3.09717\n",
"INFO:tensorflow:global_step/sec: 3.0683\n",
"2021-12-31 01:50:33,906 [INFO] tensorflow: global_step/sec: 3.0683\n",
"INFO:tensorflow:epoch = 15.489583333333332, learning_rate = 0.0009999999, loss = 0.00025472968, step = 1487 (5.506 sec)\n",
"2021-12-31 01:50:34,558 [INFO] tensorflow: epoch = 15.489583333333332, learning_rate = 0.0009999999, loss = 0.00025472968, step = 1487 (5.506 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09346\n",
"2021-12-31 01:50:36,816 [INFO] tensorflow: global_step/sec: 3.09346\n",
"2021-12-31 01:50:38,418 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.763\n",
"INFO:tensorflow:global_step/sec: 3.11677\n",
"2021-12-31 01:50:39,703 [INFO] tensorflow: global_step/sec: 3.11677\n",
"INFO:tensorflow:epoch = 15.666666666666666, learning_rate = 0.0009999999, loss = 0.00024667347, step = 1504 (5.453 sec)\n",
"2021-12-31 01:50:40,011 [INFO] tensorflow: epoch = 15.666666666666666, learning_rate = 0.0009999999, loss = 0.00024667347, step = 1504 (5.453 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08411\n",
"2021-12-31 01:50:42,622 [INFO] tensorflow: global_step/sec: 3.08411\n",
"INFO:tensorflow:epoch = 15.84375, learning_rate = 0.0009999999, loss = 0.00032511502, step = 1521 (5.512 sec)\n",
"2021-12-31 01:50:45,523 [INFO] tensorflow: epoch = 15.84375, learning_rate = 0.0009999999, loss = 0.00032511502, step = 1521 (5.512 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10111\n",
"2021-12-31 01:50:45,524 [INFO] tensorflow: global_step/sec: 3.10111\n",
"2021-12-31 01:50:46,494 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.764\n",
"INFO:tensorflow:global_step/sec: 3.16293\n",
"2021-12-31 01:50:48,369 [INFO] tensorflow: global_step/sec: 3.16293\n",
"2021-12-31 01:50:50,299 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 16/120: loss: 0.00029 learning rate: 0.00100 Time taken: 0:00:30.916914 ETA: 0:53:35.359005\n",
"INFO:tensorflow:epoch = 16.020833333333332, learning_rate = 0.0009999999, loss = 0.00043479528, step = 1538 (5.407 sec)\n",
"2021-12-31 01:50:50,930 [INFO] tensorflow: epoch = 16.020833333333332, learning_rate = 0.0009999999, loss = 0.00043479528, step = 1538 (5.407 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11755\n",
"2021-12-31 01:50:51,256 [INFO] tensorflow: global_step/sec: 3.11755\n",
"INFO:tensorflow:global_step/sec: 3.13131\n",
"2021-12-31 01:50:54,130 [INFO] tensorflow: global_step/sec: 3.13131\n",
"2021-12-31 01:50:54,447 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.150\n",
"INFO:tensorflow:epoch = 16.197916666666664, learning_rate = 0.0009999999, loss = 0.00020423223, step = 1555 (5.414 sec)\n",
"2021-12-31 01:50:56,344 [INFO] tensorflow: epoch = 16.197916666666664, learning_rate = 0.0009999999, loss = 0.00020423223, step = 1555 (5.414 sec)\n",
"INFO:tensorflow:global_step/sec: 3.13861\n",
"2021-12-31 01:50:56,998 [INFO] tensorflow: global_step/sec: 3.13861\n",
"INFO:tensorflow:global_step/sec: 3.09344\n",
"2021-12-31 01:50:59,907 [INFO] tensorflow: global_step/sec: 3.09344\n",
"INFO:tensorflow:epoch = 16.375, learning_rate = 0.0009999999, loss = 0.00031184594, step = 1572 (5.516 sec)\n",
"2021-12-31 01:51:01,860 [INFO] tensorflow: epoch = 16.375, learning_rate = 0.0009999999, loss = 0.00031184594, step = 1572 (5.516 sec)\n",
"2021-12-31 01:51:02,508 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.810\n",
"INFO:tensorflow:global_step/sec: 3.07539\n",
"2021-12-31 01:51:02,834 [INFO] tensorflow: global_step/sec: 3.07539\n",
"INFO:tensorflow:global_step/sec: 3.15229\n",
"2021-12-31 01:51:05,689 [INFO] tensorflow: global_step/sec: 3.15229\n",
"INFO:tensorflow:epoch = 16.552083333333332, learning_rate = 0.0009999999, loss = 0.000413206, step = 1589 (5.407 sec)\n",
"2021-12-31 01:51:07,267 [INFO] tensorflow: epoch = 16.552083333333332, learning_rate = 0.0009999999, loss = 0.000413206, step = 1589 (5.407 sec)\n",
"INFO:tensorflow:global_step/sec: 3.15672\n",
"2021-12-31 01:51:08,540 [INFO] tensorflow: global_step/sec: 3.15672\n",
"2021-12-31 01:51:10,478 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.095\n",
"INFO:tensorflow:global_step/sec: 3.08711\n",
"2021-12-31 01:51:11,455 [INFO] tensorflow: global_step/sec: 3.08711\n",
"INFO:tensorflow:epoch = 16.729166666666664, learning_rate = 0.0009999999, loss = 0.00024962123, step = 1606 (5.492 sec)\n",
"2021-12-31 01:51:12,759 [INFO] tensorflow: epoch = 16.729166666666664, learning_rate = 0.0009999999, loss = 0.00024962123, step = 1606 (5.492 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09728\n",
"2021-12-31 01:51:14,361 [INFO] tensorflow: global_step/sec: 3.09728\n",
"INFO:tensorflow:global_step/sec: 3.1456\n",
"2021-12-31 01:51:17,222 [INFO] tensorflow: global_step/sec: 3.1456\n",
"INFO:tensorflow:epoch = 16.90625, learning_rate = 0.0009999999, loss = 0.0002297865, step = 1623 (5.434 sec)\n",
"2021-12-31 01:51:18,193 [INFO] tensorflow: epoch = 16.90625, learning_rate = 0.0009999999, loss = 0.0002297865, step = 1623 (5.434 sec)\n",
"2021-12-31 01:51:18,519 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.874\n",
"INFO:tensorflow:global_step/sec: 3.08134\n",
"2021-12-31 01:51:20,143 [INFO] tensorflow: global_step/sec: 3.08134\n",
"2021-12-31 01:51:21,093 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 17/120: loss: 0.00023 learning rate: 0.00100 Time taken: 0:00:30.786687 ETA: 0:52:51.028775\n",
"INFO:tensorflow:global_step/sec: 3.10101\n",
"2021-12-31 01:51:23,045 [INFO] tensorflow: global_step/sec: 3.10101\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO:tensorflow:epoch = 17.083333333333332, learning_rate = 0.0009999999, loss = 0.00022645936, step = 1640 (5.473 sec)\n",
"2021-12-31 01:51:23,666 [INFO] tensorflow: epoch = 17.083333333333332, learning_rate = 0.0009999999, loss = 0.00022645936, step = 1640 (5.473 sec)\n",
"INFO:tensorflow:global_step/sec: 3.18882\n",
"2021-12-31 01:51:25,867 [INFO] tensorflow: global_step/sec: 3.18882\n",
"2021-12-31 01:51:26,536 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.947\n",
"INFO:tensorflow:global_step/sec: 3.09031\n",
"2021-12-31 01:51:28,780 [INFO] tensorflow: global_step/sec: 3.09031\n",
"INFO:tensorflow:epoch = 17.260416666666664, learning_rate = 0.0009999999, loss = 0.00026644807, step = 1657 (5.438 sec)\n",
"2021-12-31 01:51:29,105 [INFO] tensorflow: epoch = 17.260416666666664, learning_rate = 0.0009999999, loss = 0.00026644807, step = 1657 (5.438 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07043\n",
"2021-12-31 01:51:31,711 [INFO] tensorflow: global_step/sec: 3.07043\n",
"INFO:tensorflow:epoch = 17.4375, learning_rate = 0.0009999999, loss = 0.000265828, step = 1674 (5.495 sec)\n",
"2021-12-31 01:51:34,599 [INFO] tensorflow: epoch = 17.4375, learning_rate = 0.0009999999, loss = 0.000265828, step = 1674 (5.495 sec)\n",
"INFO:tensorflow:global_step/sec: 3.115\n",
"2021-12-31 01:51:34,600 [INFO] tensorflow: global_step/sec: 3.115\n",
"2021-12-31 01:51:34,601 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.800\n",
"INFO:tensorflow:global_step/sec: 3.0859\n",
"2021-12-31 01:51:37,517 [INFO] tensorflow: global_step/sec: 3.0859\n",
"INFO:tensorflow:epoch = 17.614583333333332, learning_rate = 0.0009999999, loss = 0.00023460743, step = 1691 (5.488 sec)\n",
"2021-12-31 01:51:40,088 [INFO] tensorflow: epoch = 17.614583333333332, learning_rate = 0.0009999999, loss = 0.00023460743, step = 1691 (5.488 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11026\n",
"2021-12-31 01:51:40,410 [INFO] tensorflow: global_step/sec: 3.11026\n",
"2021-12-31 01:51:42,657 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.828\n",
"INFO:tensorflow:global_step/sec: 3.09534\n",
"2021-12-31 01:51:43,318 [INFO] tensorflow: global_step/sec: 3.09534\n",
"INFO:tensorflow:epoch = 17.791666666666664, learning_rate = 0.0009999999, loss = 0.00041015184, step = 1708 (5.538 sec)\n",
"2021-12-31 01:51:45,626 [INFO] tensorflow: epoch = 17.791666666666664, learning_rate = 0.0009999999, loss = 0.00041015184, step = 1708 (5.538 sec)\n",
"INFO:tensorflow:global_step/sec: 3.01936\n",
"2021-12-31 01:51:46,299 [INFO] tensorflow: global_step/sec: 3.01936\n",
"INFO:tensorflow:global_step/sec: 3.19109\n",
"2021-12-31 01:51:49,119 [INFO] tensorflow: global_step/sec: 3.19109\n",
"2021-12-31 01:51:50,728 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.781\n",
"INFO:tensorflow:epoch = 17.96875, learning_rate = 0.0009999999, loss = 0.00027786058, step = 1725 (5.418 sec)\n",
"2021-12-31 01:51:51,044 [INFO] tensorflow: epoch = 17.96875, learning_rate = 0.0009999999, loss = 0.00027786058, step = 1725 (5.418 sec)\n",
"INFO:tensorflow:global_step/sec: 3.14686\n",
"2021-12-31 01:51:51,979 [INFO] tensorflow: global_step/sec: 3.14686\n",
"2021-12-31 01:51:51,980 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 18/120: loss: 0.00025 learning rate: 0.00100 Time taken: 0:00:30.921691 ETA: 0:52:34.012452\n",
"INFO:tensorflow:global_step/sec: 3.06899\n",
"2021-12-31 01:51:54,912 [INFO] tensorflow: global_step/sec: 3.06899\n",
"INFO:tensorflow:epoch = 18.145833333333332, learning_rate = 0.0009999999, loss = 0.00034152667, step = 1742 (5.486 sec)\n",
"2021-12-31 01:51:56,529 [INFO] tensorflow: epoch = 18.145833333333332, learning_rate = 0.0009999999, loss = 0.00034152667, step = 1742 (5.486 sec)\n",
"INFO:tensorflow:global_step/sec: 3.06798\n",
"2021-12-31 01:51:57,845 [INFO] tensorflow: global_step/sec: 3.06798\n",
"2021-12-31 01:51:58,821 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.712\n",
"INFO:tensorflow:global_step/sec: 3.0707\n",
"2021-12-31 01:52:00,776 [INFO] tensorflow: global_step/sec: 3.0707\n",
"INFO:tensorflow:epoch = 18.322916666666664, learning_rate = 0.0009999999, loss = 0.00018357951, step = 1759 (5.542 sec)\n",
"2021-12-31 01:52:02,071 [INFO] tensorflow: epoch = 18.322916666666664, learning_rate = 0.0009999999, loss = 0.00018357951, step = 1759 (5.542 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10488\n",
"2021-12-31 01:52:03,675 [INFO] tensorflow: global_step/sec: 3.10488\n",
"INFO:tensorflow:global_step/sec: 3.15481\n",
"2021-12-31 01:52:06,528 [INFO] tensorflow: global_step/sec: 3.15481\n",
"2021-12-31 01:52:06,844 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.931\n",
"INFO:tensorflow:epoch = 18.5, learning_rate = 0.0009999999, loss = 0.0002744282, step = 1776 (5.434 sec)\n",
"2021-12-31 01:52:07,506 [INFO] tensorflow: epoch = 18.5, learning_rate = 0.0009999999, loss = 0.0002744282, step = 1776 (5.434 sec)\n",
"INFO:tensorflow:global_step/sec: 3.04815\n",
"2021-12-31 01:52:09,480 [INFO] tensorflow: global_step/sec: 3.04815\n",
"INFO:tensorflow:global_step/sec: 3.13544\n",
"2021-12-31 01:52:12,351 [INFO] tensorflow: global_step/sec: 3.13544\n",
"INFO:tensorflow:epoch = 18.677083333333332, learning_rate = 0.0009999999, loss = 0.0002699753, step = 1793 (5.483 sec)\n",
"2021-12-31 01:52:12,988 [INFO] tensorflow: epoch = 18.677083333333332, learning_rate = 0.0009999999, loss = 0.0002699753, step = 1793 (5.483 sec)\n",
"2021-12-31 01:52:14,937 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.712\n",
"INFO:tensorflow:global_step/sec: 3.0946\n",
"2021-12-31 01:52:15,259 [INFO] tensorflow: global_step/sec: 3.0946\n",
"INFO:tensorflow:global_step/sec: 3.08707\n",
"2021-12-31 01:52:18,174 [INFO] tensorflow: global_step/sec: 3.08707\n",
"INFO:tensorflow:epoch = 18.854166666666664, learning_rate = 0.0009999999, loss = 0.00025841006, step = 1810 (5.518 sec)\n",
"2021-12-31 01:52:18,506 [INFO] tensorflow: epoch = 18.854166666666664, learning_rate = 0.0009999999, loss = 0.00025841006, step = 1810 (5.518 sec)\n",
"INFO:tensorflow:global_step/sec: 3.15319\n",
"2021-12-31 01:52:21,029 [INFO] tensorflow: global_step/sec: 3.15319\n",
"2021-12-31 01:52:23,027 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 19/120: loss: 0.00026 learning rate: 0.00100 Time taken: 0:00:31.010104 ETA: 0:52:12.020474\n",
"2021-12-31 01:52:23,027 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.723\n",
"INFO:tensorflow:epoch = 19.03125, learning_rate = 0.0009999999, loss = 0.00022332484, step = 1827 (5.475 sec)\n",
"2021-12-31 01:52:23,981 [INFO] tensorflow: epoch = 19.03125, learning_rate = 0.0009999999, loss = 0.00022332484, step = 1827 (5.475 sec)\n",
"INFO:tensorflow:global_step/sec: 3.04748\n",
"2021-12-31 01:52:23,982 [INFO] tensorflow: global_step/sec: 3.04748\n",
"INFO:tensorflow:global_step/sec: 3.13235\n",
"2021-12-31 01:52:26,855 [INFO] tensorflow: global_step/sec: 3.13235\n",
"INFO:tensorflow:epoch = 19.208333333333332, learning_rate = 0.0009999999, loss = 0.00029236928, step = 1844 (5.358 sec)\n",
"2021-12-31 01:52:29,339 [INFO] tensorflow: epoch = 19.208333333333332, learning_rate = 0.0009999999, loss = 0.00029236928, step = 1844 (5.358 sec)\n",
"INFO:tensorflow:global_step/sec: 3.1927\n",
"2021-12-31 01:52:29,674 [INFO] tensorflow: global_step/sec: 3.1927\n",
"2021-12-31 01:52:30,923 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.329\n",
"INFO:tensorflow:global_step/sec: 3.14742\n",
"2021-12-31 01:52:32,533 [INFO] tensorflow: global_step/sec: 3.14742\n",
"INFO:tensorflow:epoch = 19.385416666666664, learning_rate = 0.0009999999, loss = 0.00037437247, step = 1861 (5.457 sec)\n",
"2021-12-31 01:52:34,796 [INFO] tensorflow: epoch = 19.385416666666664, learning_rate = 0.0009999999, loss = 0.00037437247, step = 1861 (5.457 sec)\n",
"INFO:tensorflow:global_step/sec: 3.06585\n",
"2021-12-31 01:52:35,469 [INFO] tensorflow: global_step/sec: 3.06585\n",
"INFO:tensorflow:global_step/sec: 3.13453\n",
"2021-12-31 01:52:38,340 [INFO] tensorflow: global_step/sec: 3.13453\n",
"2021-12-31 01:52:39,000 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.765\n",
"INFO:tensorflow:epoch = 19.5625, learning_rate = 0.0009999999, loss = 0.0003113672, step = 1878 (5.484 sec)\n",
"2021-12-31 01:52:40,281 [INFO] tensorflow: epoch = 19.5625, learning_rate = 0.0009999999, loss = 0.0003113672, step = 1878 (5.484 sec)\n",
"INFO:tensorflow:global_step/sec: 3.1091\n",
"2021-12-31 01:52:41,235 [INFO] tensorflow: global_step/sec: 3.1091\n",
"INFO:tensorflow:global_step/sec: 3.05292\n",
"2021-12-31 01:52:44,183 [INFO] tensorflow: global_step/sec: 3.05292\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO:tensorflow:epoch = 19.739583333333332, learning_rate = 0.0009999999, loss = 0.0002804278, step = 1895 (5.488 sec)\n",
"2021-12-31 01:52:45,769 [INFO] tensorflow: epoch = 19.739583333333332, learning_rate = 0.0009999999, loss = 0.0002804278, step = 1895 (5.488 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11145\n",
"2021-12-31 01:52:47,075 [INFO] tensorflow: global_step/sec: 3.11145\n",
"2021-12-31 01:52:47,076 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.764\n",
"INFO:tensorflow:global_step/sec: 3.16168\n",
"2021-12-31 01:52:49,922 [INFO] tensorflow: global_step/sec: 3.16168\n",
"INFO:tensorflow:epoch = 19.916666666666664, learning_rate = 0.0009999999, loss = 0.00027419688, step = 1912 (5.406 sec)\n",
"2021-12-31 01:52:51,175 [INFO] tensorflow: epoch = 19.916666666666664, learning_rate = 0.0009999999, loss = 0.00027419688, step = 1912 (5.406 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09613\n",
"2021-12-31 01:52:52,829 [INFO] tensorflow: global_step/sec: 3.09613\n",
"INFO:tensorflow:Saving checkpoints for step-1920.\n",
"2021-12-31 01:52:53,476 [INFO] tensorflow: Saving checkpoints for step-1920.\n",
"2021-12-31 01:52:57,047 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 23, 0.00s/step\n",
"2021-12-31 01:53:16,998 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 23, 2.00s/step\n",
"2021-12-31 01:53:35,926 [INFO] iva.detectnet_v2.evaluation.evaluation: step 20 / 23, 1.89s/step\n",
"Matching predictions to ground truth, class 1/1.: 100%|█| 227561/227561 [00:14<00:00, 15790.19it/s]\n",
"Epoch 20/120\n",
"=========================\n",
"\n",
"Validation cost: 0.000240\n",
"Mean average_precision (in %): 52.8653\n",
"\n",
"class name average precision (in %)\n",
"------------ --------------------------\n",
"car 52.8653\n",
"\n",
"Median Inference Time: 0.020122\n",
"INFO:tensorflow:epoch = 20.0, learning_rate = 0.0009999999, loss = 0.00030630492, step = 1920 (69.203 sec)\n",
"2021-12-31 01:54:00,378 [INFO] tensorflow: epoch = 20.0, learning_rate = 0.0009999999, loss = 0.00030630492, step = 1920 (69.203 sec)\n",
"2021-12-31 01:54:00,378 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 20/120: loss: 0.00031 learning rate: 0.00100 Time taken: 0:01:37.300722 ETA: 2:42:10.072236\n",
"2021-12-31 01:54:01,701 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 2.680\n",
"INFO:tensorflow:global_step/sec: 0.129474\n",
"2021-12-31 01:54:02,341 [INFO] tensorflow: global_step/sec: 0.129474\n",
"INFO:tensorflow:global_step/sec: 3.15364\n",
"2021-12-31 01:54:05,195 [INFO] tensorflow: global_step/sec: 3.15364\n",
"INFO:tensorflow:epoch = 20.177083333333332, learning_rate = 0.0009999999, loss = 0.00016714807, step = 1937 (5.459 sec)\n",
"2021-12-31 01:54:05,837 [INFO] tensorflow: epoch = 20.177083333333332, learning_rate = 0.0009999999, loss = 0.00016714807, step = 1937 (5.459 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11672\n",
"2021-12-31 01:54:08,082 [INFO] tensorflow: global_step/sec: 3.11672\n",
"2021-12-31 01:54:09,652 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.153\n",
"INFO:tensorflow:global_step/sec: 3.1613\n",
"2021-12-31 01:54:10,929 [INFO] tensorflow: global_step/sec: 3.1613\n",
"INFO:tensorflow:epoch = 20.354166666666664, learning_rate = 0.0009999999, loss = 0.0003008155, step = 1954 (5.420 sec)\n",
"2021-12-31 01:54:11,257 [INFO] tensorflow: epoch = 20.354166666666664, learning_rate = 0.0009999999, loss = 0.0003008155, step = 1954 (5.420 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12401\n",
"2021-12-31 01:54:13,810 [INFO] tensorflow: global_step/sec: 3.12401\n",
"INFO:tensorflow:epoch = 20.53125, learning_rate = 0.0009999999, loss = 0.00014596831, step = 1971 (5.472 sec)\n",
"2021-12-31 01:54:16,729 [INFO] tensorflow: epoch = 20.53125, learning_rate = 0.0009999999, loss = 0.00014596831, step = 1971 (5.472 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08312\n",
"2021-12-31 01:54:16,729 [INFO] tensorflow: global_step/sec: 3.08312\n",
"2021-12-31 01:54:17,704 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.840\n",
"INFO:tensorflow:global_step/sec: 3.1444\n",
"2021-12-31 01:54:19,592 [INFO] tensorflow: global_step/sec: 3.1444\n",
"INFO:tensorflow:epoch = 20.708333333333332, learning_rate = 0.0009999999, loss = 0.00019463911, step = 1988 (5.369 sec)\n",
"2021-12-31 01:54:22,098 [INFO] tensorflow: epoch = 20.708333333333332, learning_rate = 0.0009999999, loss = 0.00019463911, step = 1988 (5.369 sec)\n",
"INFO:tensorflow:global_step/sec: 3.18721\n",
"2021-12-31 01:54:22,415 [INFO] tensorflow: global_step/sec: 3.18721\n",
"INFO:tensorflow:global_step/sec: 3.07475\n",
"2021-12-31 01:54:25,343 [INFO] tensorflow: global_step/sec: 3.07475\n",
"2021-12-31 01:54:25,666 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.123\n",
"INFO:tensorflow:epoch = 20.885416666666664, learning_rate = 0.0009999999, loss = 0.00018789242, step = 2005 (5.501 sec)\n",
"2021-12-31 01:54:27,599 [INFO] tensorflow: epoch = 20.885416666666664, learning_rate = 0.0009999999, loss = 0.00018789242, step = 2005 (5.501 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12016\n",
"2021-12-31 01:54:28,227 [INFO] tensorflow: global_step/sec: 3.12016\n",
"INFO:tensorflow:global_step/sec: 3.13638\n",
"2021-12-31 01:54:31,097 [INFO] tensorflow: global_step/sec: 3.13638\n",
"2021-12-31 01:54:31,097 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 21/120: loss: 0.00021 learning rate: 0.00100 Time taken: 0:00:30.720605 ETA: 0:50:41.339932\n",
"INFO:tensorflow:epoch = 21.0625, learning_rate = 0.0009999999, loss = 0.00027513152, step = 2022 (5.427 sec)\n",
"2021-12-31 01:54:33,026 [INFO] tensorflow: epoch = 21.0625, learning_rate = 0.0009999999, loss = 0.00027513152, step = 2022 (5.427 sec)\n",
"2021-12-31 01:54:33,664 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.007\n",
"INFO:tensorflow:global_step/sec: 3.12048\n",
"2021-12-31 01:54:33,981 [INFO] tensorflow: global_step/sec: 3.12048\n",
"INFO:tensorflow:global_step/sec: 3.10843\n",
"2021-12-31 01:54:36,876 [INFO] tensorflow: global_step/sec: 3.10843\n",
"INFO:tensorflow:epoch = 21.239583333333332, learning_rate = 0.0009999999, loss = 0.00021614096, step = 2039 (5.464 sec)\n",
"2021-12-31 01:54:38,490 [INFO] tensorflow: epoch = 21.239583333333332, learning_rate = 0.0009999999, loss = 0.00021614096, step = 2039 (5.464 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11943\n",
"2021-12-31 01:54:39,761 [INFO] tensorflow: global_step/sec: 3.11943\n",
"2021-12-31 01:54:41,699 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.891\n",
"INFO:tensorflow:global_step/sec: 3.12557\n",
"2021-12-31 01:54:42,641 [INFO] tensorflow: global_step/sec: 3.12557\n",
"INFO:tensorflow:epoch = 21.416666666666664, learning_rate = 0.0009999999, loss = 0.00021882175, step = 2056 (5.421 sec)\n",
"2021-12-31 01:54:43,911 [INFO] tensorflow: epoch = 21.416666666666664, learning_rate = 0.0009999999, loss = 0.00021882175, step = 2056 (5.421 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10228\n",
"2021-12-31 01:54:45,542 [INFO] tensorflow: global_step/sec: 3.10228\n",
"INFO:tensorflow:global_step/sec: 3.12388\n",
"2021-12-31 01:54:48,423 [INFO] tensorflow: global_step/sec: 3.12388\n",
"INFO:tensorflow:epoch = 21.59375, learning_rate = 0.0009999999, loss = 0.00021170318, step = 2073 (5.488 sec)\n",
"2021-12-31 01:54:49,399 [INFO] tensorflow: epoch = 21.59375, learning_rate = 0.0009999999, loss = 0.00021170318, step = 2073 (5.488 sec)\n",
"2021-12-31 01:54:49,715 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.952\n",
"INFO:tensorflow:global_step/sec: 3.1125\n",
"2021-12-31 01:54:51,314 [INFO] tensorflow: global_step/sec: 3.1125\n",
"INFO:tensorflow:global_step/sec: 3.09682\n",
"2021-12-31 01:54:54,221 [INFO] tensorflow: global_step/sec: 3.09682\n",
"INFO:tensorflow:epoch = 21.770833333333332, learning_rate = 0.0009999999, loss = 0.00020895376, step = 2090 (5.469 sec)\n",
"2021-12-31 01:54:54,869 [INFO] tensorflow: epoch = 21.770833333333332, learning_rate = 0.0009999999, loss = 0.00020895376, step = 2090 (5.469 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08303\n",
"2021-12-31 01:54:57,140 [INFO] tensorflow: global_step/sec: 3.08303\n",
"2021-12-31 01:54:57,784 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.786\n",
"INFO:tensorflow:global_step/sec: 3.14396\n",
"2021-12-31 01:55:00,002 [INFO] tensorflow: global_step/sec: 3.14396\n",
"INFO:tensorflow:epoch = 21.947916666666664, learning_rate = 0.0009999999, loss = 0.00028925453, step = 2107 (5.444 sec)\n",
"2021-12-31 01:55:00,313 [INFO] tensorflow: epoch = 21.947916666666664, learning_rate = 0.0009999999, loss = 0.00028925453, step = 2107 (5.444 sec)\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"2021-12-31 01:55:01,924 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 22/120: loss: 0.00028 learning rate: 0.00100 Time taken: 0:00:30.849705 ETA: 0:50:23.271111\n",
"INFO:tensorflow:global_step/sec: 3.12439\n",
"2021-12-31 01:55:02,883 [INFO] tensorflow: global_step/sec: 3.12439\n",
"INFO:tensorflow:epoch = 22.125, learning_rate = 0.0009999999, loss = 0.0002457933, step = 2124 (5.397 sec)\n",
"2021-12-31 01:55:05,710 [INFO] tensorflow: epoch = 22.125, learning_rate = 0.0009999999, loss = 0.0002457933, step = 2124 (5.397 sec)\n",
"INFO:tensorflow:global_step/sec: 3.18275\n",
"2021-12-31 01:55:05,711 [INFO] tensorflow: global_step/sec: 3.18275\n",
"2021-12-31 01:55:05,711 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.229\n",
"INFO:tensorflow:global_step/sec: 3.10927\n",
"2021-12-31 01:55:08,605 [INFO] tensorflow: global_step/sec: 3.10927\n",
"INFO:tensorflow:epoch = 22.302083333333332, learning_rate = 0.0009999999, loss = 0.00023270797, step = 2141 (5.448 sec)\n",
"2021-12-31 01:55:11,157 [INFO] tensorflow: epoch = 22.302083333333332, learning_rate = 0.0009999999, loss = 0.00023270797, step = 2141 (5.448 sec)\n",
"INFO:tensorflow:global_step/sec: 3.13587\n",
"2021-12-31 01:55:11,475 [INFO] tensorflow: global_step/sec: 3.13587\n",
"2021-12-31 01:55:13,687 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.077\n",
"INFO:tensorflow:global_step/sec: 3.14804\n",
"2021-12-31 01:55:14,334 [INFO] tensorflow: global_step/sec: 3.14804\n",
"INFO:tensorflow:epoch = 22.479166666666664, learning_rate = 0.0009999999, loss = 0.0002169427, step = 2158 (5.448 sec)\n",
"2021-12-31 01:55:16,605 [INFO] tensorflow: epoch = 22.479166666666664, learning_rate = 0.0009999999, loss = 0.0002169427, step = 2158 (5.448 sec)\n",
"INFO:tensorflow:global_step/sec: 3.05397\n",
"2021-12-31 01:55:17,281 [INFO] tensorflow: global_step/sec: 3.05397\n",
"INFO:tensorflow:global_step/sec: 3.12936\n",
"2021-12-31 01:55:20,157 [INFO] tensorflow: global_step/sec: 3.12936\n",
"2021-12-31 01:55:21,754 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.793\n",
"INFO:tensorflow:epoch = 22.65625, learning_rate = 0.0009999999, loss = 0.00023978732, step = 2175 (5.468 sec)\n",
"2021-12-31 01:55:22,073 [INFO] tensorflow: epoch = 22.65625, learning_rate = 0.0009999999, loss = 0.00023978732, step = 2175 (5.468 sec)\n",
"INFO:tensorflow:global_step/sec: 3.13597\n",
"2021-12-31 01:55:23,027 [INFO] tensorflow: global_step/sec: 3.13597\n",
"INFO:tensorflow:global_step/sec: 3.13656\n",
"2021-12-31 01:55:25,897 [INFO] tensorflow: global_step/sec: 3.13656\n",
"INFO:tensorflow:epoch = 22.833333333333332, learning_rate = 0.0009999999, loss = 0.0001788536, step = 2192 (5.436 sec)\n",
"2021-12-31 01:55:27,509 [INFO] tensorflow: epoch = 22.833333333333332, learning_rate = 0.0009999999, loss = 0.0001788536, step = 2192 (5.436 sec)\n",
"INFO:tensorflow:global_step/sec: 3.0995\n",
"2021-12-31 01:55:28,800 [INFO] tensorflow: global_step/sec: 3.0995\n",
"2021-12-31 01:55:29,772 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.945\n",
"INFO:tensorflow:global_step/sec: 3.09257\n",
"2021-12-31 01:55:31,710 [INFO] tensorflow: global_step/sec: 3.09257\n",
"2021-12-31 01:55:32,645 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 23/120: loss: 0.00024 learning rate: 0.00100 Time taken: 0:00:30.740704 ETA: 0:49:41.848294\n",
"INFO:tensorflow:epoch = 23.010416666666664, learning_rate = 0.0009999999, loss = 0.00030232806, step = 2209 (5.459 sec)\n",
"2021-12-31 01:55:32,968 [INFO] tensorflow: epoch = 23.010416666666664, learning_rate = 0.0009999999, loss = 0.00030232806, step = 2209 (5.459 sec)\n",
"INFO:tensorflow:global_step/sec: 3.15826\n",
"2021-12-31 01:55:34,560 [INFO] tensorflow: global_step/sec: 3.15826\n",
"INFO:tensorflow:global_step/sec: 3.13683\n",
"2021-12-31 01:55:37,429 [INFO] tensorflow: global_step/sec: 3.13683\n",
"2021-12-31 01:55:37,771 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.003\n",
"INFO:tensorflow:epoch = 23.1875, learning_rate = 0.0009999999, loss = 0.0002608334, step = 2226 (5.453 sec)\n",
"2021-12-31 01:55:38,421 [INFO] tensorflow: epoch = 23.1875, learning_rate = 0.0009999999, loss = 0.0002608334, step = 2226 (5.453 sec)\n",
"INFO:tensorflow:global_step/sec: 3.03185\n",
"2021-12-31 01:55:40,398 [INFO] tensorflow: global_step/sec: 3.03185\n",
"INFO:tensorflow:global_step/sec: 3.07307\n",
"2021-12-31 01:55:43,326 [INFO] tensorflow: global_step/sec: 3.07307\n",
"INFO:tensorflow:epoch = 23.364583333333332, learning_rate = 0.0009999999, loss = 0.00030110948, step = 2243 (5.545 sec)\n",
"2021-12-31 01:55:43,966 [INFO] tensorflow: epoch = 23.364583333333332, learning_rate = 0.0009999999, loss = 0.00030110948, step = 2243 (5.545 sec)\n",
"2021-12-31 01:55:45,868 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.702\n",
"INFO:tensorflow:global_step/sec: 3.14163\n",
"2021-12-31 01:55:46,191 [INFO] tensorflow: global_step/sec: 3.14163\n",
"INFO:tensorflow:global_step/sec: 3.09284\n",
"2021-12-31 01:55:49,101 [INFO] tensorflow: global_step/sec: 3.09284\n",
"INFO:tensorflow:epoch = 23.541666666666664, learning_rate = 0.0009999999, loss = 0.00037726236, step = 2260 (5.470 sec)\n",
"2021-12-31 01:55:49,436 [INFO] tensorflow: epoch = 23.541666666666664, learning_rate = 0.0009999999, loss = 0.00037726236, step = 2260 (5.470 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10583\n",
"2021-12-31 01:55:51,999 [INFO] tensorflow: global_step/sec: 3.10583\n",
"2021-12-31 01:55:53,904 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.887\n",
"INFO:tensorflow:epoch = 23.71875, learning_rate = 0.0009999999, loss = 0.00027072575, step = 2277 (5.439 sec)\n",
"2021-12-31 01:55:54,875 [INFO] tensorflow: epoch = 23.71875, learning_rate = 0.0009999999, loss = 0.00027072575, step = 2277 (5.439 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12848\n",
"2021-12-31 01:55:54,876 [INFO] tensorflow: global_step/sec: 3.12848\n",
"INFO:tensorflow:global_step/sec: 3.1805\n",
"2021-12-31 01:55:57,705 [INFO] tensorflow: global_step/sec: 3.1805\n",
"INFO:tensorflow:epoch = 23.895833333333332, learning_rate = 0.0009999999, loss = 0.00017067703, step = 2294 (5.439 sec)\n",
"2021-12-31 01:56:00,314 [INFO] tensorflow: epoch = 23.895833333333332, learning_rate = 0.0009999999, loss = 0.00017067703, step = 2294 (5.439 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07718\n",
"2021-12-31 01:56:00,630 [INFO] tensorflow: global_step/sec: 3.07718\n",
"2021-12-31 01:56:01,965 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.813\n",
"INFO:tensorflow:global_step/sec: 3.05888\n",
"2021-12-31 01:56:03,572 [INFO] tensorflow: global_step/sec: 3.05888\n",
"2021-12-31 01:56:03,573 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 24/120: loss: 0.00019 learning rate: 0.00100 Time taken: 0:00:30.913691 ETA: 0:49:27.714340\n",
"INFO:tensorflow:epoch = 24.072916666666664, learning_rate = 0.0009999999, loss = 0.00019253616, step = 2311 (5.463 sec)\n",
"2021-12-31 01:56:05,777 [INFO] tensorflow: epoch = 24.072916666666664, learning_rate = 0.0009999999, loss = 0.00019253616, step = 2311 (5.463 sec)\n",
"INFO:tensorflow:global_step/sec: 3.13239\n",
"2021-12-31 01:56:06,446 [INFO] tensorflow: global_step/sec: 3.13239\n",
"INFO:tensorflow:global_step/sec: 3.08992\n",
"2021-12-31 01:56:09,358 [INFO] tensorflow: global_step/sec: 3.08992\n",
"2021-12-31 01:56:09,988 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.928\n",
"INFO:tensorflow:epoch = 24.25, learning_rate = 0.0009999999, loss = 0.00020398729, step = 2328 (5.546 sec)\n",
"2021-12-31 01:56:11,323 [INFO] tensorflow: epoch = 24.25, learning_rate = 0.0009999999, loss = 0.00020398729, step = 2328 (5.546 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07499\n",
"2021-12-31 01:56:12,285 [INFO] tensorflow: global_step/sec: 3.07499\n",
"INFO:tensorflow:global_step/sec: 3.11249\n",
"2021-12-31 01:56:15,177 [INFO] tensorflow: global_step/sec: 3.11249\n",
"INFO:tensorflow:epoch = 24.427083333333332, learning_rate = 0.0009999999, loss = 0.00024929407, step = 2345 (5.475 sec)\n",
"2021-12-31 01:56:16,799 [INFO] tensorflow: epoch = 24.427083333333332, learning_rate = 0.0009999999, loss = 0.00024929407, step = 2345 (5.475 sec)\n",
"INFO:tensorflow:global_step/sec: 3.06932\n",
"2021-12-31 01:56:18,109 [INFO] tensorflow: global_step/sec: 3.06932\n",
"2021-12-31 01:56:18,110 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.628\n",
"INFO:tensorflow:global_step/sec: 3.14342\n",
"2021-12-31 01:56:20,972 [INFO] tensorflow: global_step/sec: 3.14342\n",
"INFO:tensorflow:epoch = 24.604166666666664, learning_rate = 0.0009999999, loss = 0.00026869454, step = 2362 (5.451 sec)\n",
"2021-12-31 01:56:22,250 [INFO] tensorflow: epoch = 24.604166666666664, learning_rate = 0.0009999999, loss = 0.00026869454, step = 2362 (5.451 sec)\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO:tensorflow:global_step/sec: 3.10089\n",
"2021-12-31 01:56:23,874 [INFO] tensorflow: global_step/sec: 3.10089\n",
"2021-12-31 01:56:26,101 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.029\n",
"INFO:tensorflow:global_step/sec: 3.14177\n",
"2021-12-31 01:56:26,739 [INFO] tensorflow: global_step/sec: 3.14177\n",
"INFO:tensorflow:epoch = 24.78125, learning_rate = 0.0009999999, loss = 0.0001761469, step = 2379 (5.448 sec)\n",
"2021-12-31 01:56:27,698 [INFO] tensorflow: epoch = 24.78125, learning_rate = 0.0009999999, loss = 0.0001761469, step = 2379 (5.448 sec)\n",
"INFO:tensorflow:global_step/sec: 3.13058\n",
"2021-12-31 01:56:29,614 [INFO] tensorflow: global_step/sec: 3.13058\n",
"INFO:tensorflow:global_step/sec: 3.15482\n",
"2021-12-31 01:56:32,467 [INFO] tensorflow: global_step/sec: 3.15482\n",
"INFO:tensorflow:epoch = 24.958333333333332, learning_rate = 0.0009999999, loss = 0.00020437592, step = 2396 (5.394 sec)\n",
"2021-12-31 01:56:33,092 [INFO] tensorflow: epoch = 24.958333333333332, learning_rate = 0.0009999999, loss = 0.00020437592, step = 2396 (5.394 sec)\n",
"2021-12-31 01:56:34,020 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.254\n",
"2021-12-31 01:56:34,334 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 25/120: loss: 0.00022 learning rate: 0.00100 Time taken: 0:00:30.767394 ETA: 0:48:42.902436\n",
"INFO:tensorflow:global_step/sec: 3.15502\n",
"2021-12-31 01:56:35,319 [INFO] tensorflow: global_step/sec: 3.15502\n",
"INFO:tensorflow:global_step/sec: 3.11035\n",
"2021-12-31 01:56:38,213 [INFO] tensorflow: global_step/sec: 3.11035\n",
"INFO:tensorflow:epoch = 25.135416666666664, learning_rate = 0.0009999999, loss = 0.00022569609, step = 2413 (5.457 sec)\n",
"2021-12-31 01:56:38,549 [INFO] tensorflow: epoch = 25.135416666666664, learning_rate = 0.0009999999, loss = 0.00022569609, step = 2413 (5.457 sec)\n",
"INFO:tensorflow:global_step/sec: 3.13575\n",
"2021-12-31 01:56:41,083 [INFO] tensorflow: global_step/sec: 3.13575\n",
"2021-12-31 01:56:42,066 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.860\n",
"INFO:tensorflow:epoch = 25.3125, learning_rate = 0.0009999999, loss = 0.00022829151, step = 2430 (5.445 sec)\n",
"2021-12-31 01:56:43,994 [INFO] tensorflow: epoch = 25.3125, learning_rate = 0.0009999999, loss = 0.00022829151, step = 2430 (5.445 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09082\n",
"2021-12-31 01:56:43,995 [INFO] tensorflow: global_step/sec: 3.09082\n",
"INFO:tensorflow:global_step/sec: 3.08289\n",
"2021-12-31 01:56:46,914 [INFO] tensorflow: global_step/sec: 3.08289\n",
"INFO:tensorflow:epoch = 25.489583333333332, learning_rate = 0.0009999999, loss = 0.000245134, step = 2447 (5.496 sec)\n",
"2021-12-31 01:56:49,490 [INFO] tensorflow: epoch = 25.489583333333332, learning_rate = 0.0009999999, loss = 0.000245134, step = 2447 (5.496 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08314\n",
"2021-12-31 01:56:49,833 [INFO] tensorflow: global_step/sec: 3.08314\n",
"2021-12-31 01:56:50,156 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.723\n",
"INFO:tensorflow:global_step/sec: 3.18518\n",
"2021-12-31 01:56:52,659 [INFO] tensorflow: global_step/sec: 3.18518\n",
"INFO:tensorflow:epoch = 25.666666666666664, learning_rate = 0.0009999999, loss = 0.00030708834, step = 2464 (5.383 sec)\n",
"2021-12-31 01:56:54,873 [INFO] tensorflow: epoch = 25.666666666666664, learning_rate = 0.0009999999, loss = 0.00030708834, step = 2464 (5.383 sec)\n",
"INFO:tensorflow:global_step/sec: 3.15777\n",
"2021-12-31 01:56:55,509 [INFO] tensorflow: global_step/sec: 3.15777\n",
"2021-12-31 01:56:58,102 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.170\n",
"INFO:tensorflow:global_step/sec: 3.07733\n",
"2021-12-31 01:56:58,434 [INFO] tensorflow: global_step/sec: 3.07733\n",
"INFO:tensorflow:epoch = 25.84375, learning_rate = 0.0009999999, loss = 0.0002579501, step = 2481 (5.523 sec)\n",
"2021-12-31 01:57:00,396 [INFO] tensorflow: epoch = 25.84375, learning_rate = 0.0009999999, loss = 0.0002579501, step = 2481 (5.523 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07905\n",
"2021-12-31 01:57:01,357 [INFO] tensorflow: global_step/sec: 3.07905\n",
"INFO:tensorflow:global_step/sec: 3.10447\n",
"2021-12-31 01:57:04,256 [INFO] tensorflow: global_step/sec: 3.10447\n",
"2021-12-31 01:57:05,173 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 26/120: loss: 0.00029 learning rate: 0.00100 Time taken: 0:00:30.854927 ETA: 0:48:20.363122\n",
"INFO:tensorflow:epoch = 26.020833333333332, learning_rate = 0.0009999999, loss = 0.00022291686, step = 2498 (5.419 sec)\n",
"2021-12-31 01:57:05,815 [INFO] tensorflow: epoch = 26.020833333333332, learning_rate = 0.0009999999, loss = 0.00022291686, step = 2498 (5.419 sec)\n",
"2021-12-31 01:57:06,144 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.871\n",
"INFO:tensorflow:global_step/sec: 3.13286\n",
"2021-12-31 01:57:07,128 [INFO] tensorflow: global_step/sec: 3.13286\n",
"INFO:tensorflow:global_step/sec: 3.18198\n",
"2021-12-31 01:57:09,957 [INFO] tensorflow: global_step/sec: 3.18198\n",
"INFO:tensorflow:epoch = 26.197916666666664, learning_rate = 0.0009999999, loss = 0.00018194792, step = 2515 (5.424 sec)\n",
"2021-12-31 01:57:11,239 [INFO] tensorflow: epoch = 26.197916666666664, learning_rate = 0.0009999999, loss = 0.00018194792, step = 2515 (5.424 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09885\n",
"2021-12-31 01:57:12,861 [INFO] tensorflow: global_step/sec: 3.09885\n",
"2021-12-31 01:57:14,131 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.043\n",
"INFO:tensorflow:global_step/sec: 3.16075\n",
"2021-12-31 01:57:15,709 [INFO] tensorflow: global_step/sec: 3.16075\n",
"INFO:tensorflow:epoch = 26.375, learning_rate = 0.0009999999, loss = 0.00028747928, step = 2532 (5.425 sec)\n",
"2021-12-31 01:57:16,664 [INFO] tensorflow: epoch = 26.375, learning_rate = 0.0009999999, loss = 0.00028747928, step = 2532 (5.425 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10847\n",
"2021-12-31 01:57:18,604 [INFO] tensorflow: global_step/sec: 3.10847\n",
"INFO:tensorflow:global_step/sec: 3.10539\n",
"2021-12-31 01:57:21,502 [INFO] tensorflow: global_step/sec: 3.10539\n",
"INFO:tensorflow:epoch = 26.552083333333332, learning_rate = 0.0009999999, loss = 0.00024165685, step = 2549 (5.446 sec)\n",
"2021-12-31 01:57:22,109 [INFO] tensorflow: epoch = 26.552083333333332, learning_rate = 0.0009999999, loss = 0.00024165685, step = 2549 (5.446 sec)\n",
"2021-12-31 01:57:22,110 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.068\n",
"INFO:tensorflow:global_step/sec: 3.13624\n",
"2021-12-31 01:57:24,372 [INFO] tensorflow: global_step/sec: 3.13624\n",
"INFO:tensorflow:global_step/sec: 3.12163\n",
"2021-12-31 01:57:27,255 [INFO] tensorflow: global_step/sec: 3.12163\n",
"INFO:tensorflow:epoch = 26.729166666666664, learning_rate = 0.0009999999, loss = 0.00022220582, step = 2566 (5.489 sec)\n",
"2021-12-31 01:57:27,598 [INFO] tensorflow: epoch = 26.729166666666664, learning_rate = 0.0009999999, loss = 0.00022220582, step = 2566 (5.489 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07179\n",
"2021-12-31 01:57:30,185 [INFO] tensorflow: global_step/sec: 3.07179\n",
"2021-12-31 01:57:30,185 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.766\n",
"INFO:tensorflow:epoch = 26.90625, learning_rate = 0.0009999999, loss = 0.00021997553, step = 2583 (5.523 sec)\n",
"2021-12-31 01:57:33,121 [INFO] tensorflow: epoch = 26.90625, learning_rate = 0.0009999999, loss = 0.00021997553, step = 2583 (5.523 sec)\n",
"INFO:tensorflow:global_step/sec: 3.06438\n",
"2021-12-31 01:57:33,122 [INFO] tensorflow: global_step/sec: 3.06438\n",
"INFO:tensorflow:global_step/sec: 3.1242\n",
"2021-12-31 01:57:36,003 [INFO] tensorflow: global_step/sec: 3.1242\n",
"2021-12-31 01:57:36,003 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 27/120: loss: 0.00025 learning rate: 0.00100 Time taken: 0:00:30.809506 ETA: 0:47:45.284052\n",
"2021-12-31 01:57:38,235 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.847\n",
"INFO:tensorflow:epoch = 27.083333333333332, learning_rate = 0.0009999999, loss = 0.0001916479, step = 2600 (5.414 sec)\n",
"2021-12-31 01:57:38,535 [INFO] tensorflow: epoch = 27.083333333333332, learning_rate = 0.0009999999, loss = 0.0001916479, step = 2600 (5.414 sec)\n",
"INFO:tensorflow:global_step/sec: 3.15161\n",
"2021-12-31 01:57:38,858 [INFO] tensorflow: global_step/sec: 3.15161\n",
"INFO:tensorflow:global_step/sec: 3.10056\n",
"2021-12-31 01:57:41,761 [INFO] tensorflow: global_step/sec: 3.10056\n",
"INFO:tensorflow:epoch = 27.260416666666664, learning_rate = 0.0009999999, loss = 0.00020450028, step = 2617 (5.479 sec)\n",
"2021-12-31 01:57:44,014 [INFO] tensorflow: epoch = 27.260416666666664, learning_rate = 0.0009999999, loss = 0.00020450028, step = 2617 (5.479 sec)\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO:tensorflow:global_step/sec: 3.10632\n",
"2021-12-31 01:57:44,658 [INFO] tensorflow: global_step/sec: 3.10632\n",
"2021-12-31 01:57:46,214 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.067\n",
"INFO:tensorflow:global_step/sec: 3.14501\n",
"2021-12-31 01:57:47,520 [INFO] tensorflow: global_step/sec: 3.14501\n",
"INFO:tensorflow:epoch = 27.4375, learning_rate = 0.0009999999, loss = 0.00019256785, step = 2634 (5.427 sec)\n",
"2021-12-31 01:57:49,441 [INFO] tensorflow: epoch = 27.4375, learning_rate = 0.0009999999, loss = 0.00019256785, step = 2634 (5.427 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12231\n",
"2021-12-31 01:57:50,402 [INFO] tensorflow: global_step/sec: 3.12231\n",
"INFO:tensorflow:global_step/sec: 3.07739\n",
"2021-12-31 01:57:53,327 [INFO] tensorflow: global_step/sec: 3.07739\n",
"2021-12-31 01:57:54,289 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.768\n",
"INFO:tensorflow:epoch = 27.614583333333332, learning_rate = 0.0009999999, loss = 0.00020603626, step = 2651 (5.490 sec)\n",
"2021-12-31 01:57:54,931 [INFO] tensorflow: epoch = 27.614583333333332, learning_rate = 0.0009999999, loss = 0.00020603626, step = 2651 (5.490 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09426\n",
"2021-12-31 01:57:56,236 [INFO] tensorflow: global_step/sec: 3.09426\n",
"INFO:tensorflow:global_step/sec: 3.11004\n",
"2021-12-31 01:57:59,129 [INFO] tensorflow: global_step/sec: 3.11004\n",
"INFO:tensorflow:epoch = 27.791666666666664, learning_rate = 0.0009999999, loss = 0.0002125539, step = 2668 (5.424 sec)\n",
"2021-12-31 01:58:00,355 [INFO] tensorflow: epoch = 27.791666666666664, learning_rate = 0.0009999999, loss = 0.0002125539, step = 2668 (5.424 sec)\n",
"INFO:tensorflow:global_step/sec: 3.1556\n",
"2021-12-31 01:58:01,981 [INFO] tensorflow: global_step/sec: 3.1556\n",
"2021-12-31 01:58:02,313 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.925\n",
"INFO:tensorflow:global_step/sec: 3.15011\n",
"2021-12-31 01:58:04,839 [INFO] tensorflow: global_step/sec: 3.15011\n",
"INFO:tensorflow:epoch = 27.96875, learning_rate = 0.0009999999, loss = 0.00022524351, step = 2685 (5.469 sec)\n",
"2021-12-31 01:58:05,824 [INFO] tensorflow: epoch = 27.96875, learning_rate = 0.0009999999, loss = 0.00022524351, step = 2685 (5.469 sec)\n",
"2021-12-31 01:58:06,843 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 28/120: loss: 0.00022 learning rate: 0.00100 Time taken: 0:00:30.810175 ETA: 0:47:14.536117\n",
"INFO:tensorflow:global_step/sec: 3.02586\n",
"2021-12-31 01:58:07,813 [INFO] tensorflow: global_step/sec: 3.02586\n",
"2021-12-31 01:58:10,375 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.808\n",
"INFO:tensorflow:global_step/sec: 3.13164\n",
"2021-12-31 01:58:10,687 [INFO] tensorflow: global_step/sec: 3.13164\n",
"INFO:tensorflow:epoch = 28.145833333333332, learning_rate = 0.0009999999, loss = 0.00026755212, step = 2702 (5.507 sec)\n",
"2021-12-31 01:58:11,331 [INFO] tensorflow: epoch = 28.145833333333332, learning_rate = 0.0009999999, loss = 0.00026755212, step = 2702 (5.507 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09365\n",
"2021-12-31 01:58:13,596 [INFO] tensorflow: global_step/sec: 3.09365\n",
"INFO:tensorflow:global_step/sec: 3.12288\n",
"2021-12-31 01:58:16,478 [INFO] tensorflow: global_step/sec: 3.12288\n",
"INFO:tensorflow:epoch = 28.322916666666664, learning_rate = 0.0009999999, loss = 0.00023565102, step = 2719 (5.455 sec)\n",
"2021-12-31 01:58:16,786 [INFO] tensorflow: epoch = 28.322916666666664, learning_rate = 0.0009999999, loss = 0.00023565102, step = 2719 (5.455 sec)\n",
"2021-12-31 01:58:18,407 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.901\n",
"INFO:tensorflow:global_step/sec: 3.15836\n",
"2021-12-31 01:58:19,327 [INFO] tensorflow: global_step/sec: 3.15836\n",
"INFO:tensorflow:epoch = 28.5, learning_rate = 0.0009999999, loss = 0.00028118538, step = 2736 (5.370 sec)\n",
"2021-12-31 01:58:22,156 [INFO] tensorflow: epoch = 28.5, learning_rate = 0.0009999999, loss = 0.00028118538, step = 2736 (5.370 sec)\n",
"INFO:tensorflow:global_step/sec: 3.18097\n",
"2021-12-31 01:58:22,157 [INFO] tensorflow: global_step/sec: 3.18097\n",
"INFO:tensorflow:global_step/sec: 3.12828\n",
"2021-12-31 01:58:25,034 [INFO] tensorflow: global_step/sec: 3.12828\n",
"2021-12-31 01:58:26,318 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.281\n",
"INFO:tensorflow:epoch = 28.677083333333332, learning_rate = 0.0009999999, loss = 0.00023948813, step = 2753 (5.467 sec)\n",
"2021-12-31 01:58:27,623 [INFO] tensorflow: epoch = 28.677083333333332, learning_rate = 0.0009999999, loss = 0.00023948813, step = 2753 (5.467 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08942\n",
"2021-12-31 01:58:27,947 [INFO] tensorflow: global_step/sec: 3.08942\n",
"INFO:tensorflow:global_step/sec: 3.17395\n",
"2021-12-31 01:58:30,783 [INFO] tensorflow: global_step/sec: 3.17395\n",
"INFO:tensorflow:epoch = 28.854166666666664, learning_rate = 0.0009999999, loss = 0.00019599068, step = 2770 (5.375 sec)\n",
"2021-12-31 01:58:32,998 [INFO] tensorflow: epoch = 28.854166666666664, learning_rate = 0.0009999999, loss = 0.00019599068, step = 2770 (5.375 sec)\n",
"INFO:tensorflow:global_step/sec: 3.15879\n",
"2021-12-31 01:58:33,632 [INFO] tensorflow: global_step/sec: 3.15879\n",
"2021-12-31 01:58:34,286 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.102\n",
"INFO:tensorflow:global_step/sec: 3.1229\n",
"2021-12-31 01:58:36,514 [INFO] tensorflow: global_step/sec: 3.1229\n",
"2021-12-31 01:58:37,504 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 29/120: loss: 0.00019 learning rate: 0.00100 Time taken: 0:00:30.682151 ETA: 0:46:32.075770\n",
"INFO:tensorflow:epoch = 29.03125, learning_rate = 0.0009999999, loss = 0.00024140562, step = 2787 (5.464 sec)\n",
"2021-12-31 01:58:38,463 [INFO] tensorflow: epoch = 29.03125, learning_rate = 0.0009999999, loss = 0.00024140562, step = 2787 (5.464 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09659\n",
"2021-12-31 01:58:39,420 [INFO] tensorflow: global_step/sec: 3.09659\n",
"INFO:tensorflow:global_step/sec: 3.16497\n",
"2021-12-31 01:58:42,264 [INFO] tensorflow: global_step/sec: 3.16497\n",
"2021-12-31 01:58:42,265 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.068\n",
"INFO:tensorflow:epoch = 29.208333333333332, learning_rate = 0.0009999999, loss = 0.00020415211, step = 2804 (5.454 sec)\n",
"2021-12-31 01:58:43,917 [INFO] tensorflow: epoch = 29.208333333333332, learning_rate = 0.0009999999, loss = 0.00020415211, step = 2804 (5.454 sec)\n",
"INFO:tensorflow:global_step/sec: 3.06582\n",
"2021-12-31 01:58:45,199 [INFO] tensorflow: global_step/sec: 3.06582\n",
"INFO:tensorflow:global_step/sec: 3.1103\n",
"2021-12-31 01:58:48,093 [INFO] tensorflow: global_step/sec: 3.1103\n",
"INFO:tensorflow:epoch = 29.385416666666664, learning_rate = 0.0009999999, loss = 0.0002615858, step = 2821 (5.445 sec)\n",
"2021-12-31 01:58:49,362 [INFO] tensorflow: epoch = 29.385416666666664, learning_rate = 0.0009999999, loss = 0.0002615858, step = 2821 (5.445 sec)\n",
"2021-12-31 01:58:50,335 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.783\n",
"INFO:tensorflow:global_step/sec: 3.13487\n",
"2021-12-31 01:58:50,964 [INFO] tensorflow: global_step/sec: 3.13487\n",
"INFO:tensorflow:global_step/sec: 3.11121\n",
"2021-12-31 01:58:53,857 [INFO] tensorflow: global_step/sec: 3.11121\n",
"INFO:tensorflow:epoch = 29.5625, learning_rate = 0.0009999999, loss = 0.0002892708, step = 2838 (5.458 sec)\n",
"2021-12-31 01:58:54,819 [INFO] tensorflow: epoch = 29.5625, learning_rate = 0.0009999999, loss = 0.0002892708, step = 2838 (5.458 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09213\n",
"2021-12-31 01:58:56,767 [INFO] tensorflow: global_step/sec: 3.09213\n",
"2021-12-31 01:58:58,377 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.869\n",
"INFO:tensorflow:global_step/sec: 3.09605\n",
"2021-12-31 01:58:59,674 [INFO] tensorflow: global_step/sec: 3.09605\n",
"INFO:tensorflow:epoch = 29.739583333333332, learning_rate = 0.0009999999, loss = 0.00020180142, step = 2855 (5.459 sec)\n",
"2021-12-31 01:59:00,278 [INFO] tensorflow: epoch = 29.739583333333332, learning_rate = 0.0009999999, loss = 0.00020180142, step = 2855 (5.459 sec)\n",
"INFO:tensorflow:global_step/sec: 3.16731\n",
"2021-12-31 01:59:02,516 [INFO] tensorflow: global_step/sec: 3.16731\n",
"INFO:tensorflow:global_step/sec: 3.19697\n",
"2021-12-31 01:59:05,331 [INFO] tensorflow: global_step/sec: 3.19697\n",
"INFO:tensorflow:epoch = 29.916666666666664, learning_rate = 0.0009999999, loss = 0.00022788, step = 2872 (5.373 sec)\n",
"2021-12-31 01:59:05,651 [INFO] tensorflow: epoch = 29.916666666666664, learning_rate = 0.0009999999, loss = 0.00022788, step = 2872 (5.373 sec)\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"2021-12-31 01:59:06,301 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.242\n",
"INFO:tensorflow:Saving checkpoints for step-2880.\n",
"2021-12-31 01:59:07,896 [INFO] tensorflow: Saving checkpoints for step-2880.\n",
"2021-12-31 01:59:11,613 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 23, 0.00s/step\n",
"2021-12-31 01:59:29,611 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 23, 1.80s/step\n",
"2021-12-31 01:59:47,705 [INFO] iva.detectnet_v2.evaluation.evaluation: step 20 / 23, 1.81s/step\n",
"Matching predictions to ground truth, class 1/1.: 100%|█| 227520/227520 [00:15<00:00, 15150.37it/s]\n",
"Epoch 30/120\n",
"=========================\n",
"\n",
"Validation cost: 0.000272\n",
"Mean average_precision (in %): 36.6376\n",
"\n",
"class name average precision (in %)\n",
"------------ --------------------------\n",
"car 36.6376\n",
"\n",
"Median Inference Time: 0.018155\n",
"INFO:tensorflow:epoch = 30.0, learning_rate = 0.0009999999, loss = 0.00014572589, step = 2880 (67.029 sec)\n",
"2021-12-31 02:00:12,680 [INFO] tensorflow: epoch = 30.0, learning_rate = 0.0009999999, loss = 0.00014572589, step = 2880 (67.029 sec)\n",
"INFO:tensorflow:global_step/sec: 0.13363\n",
"2021-12-31 02:00:12,681 [INFO] tensorflow: global_step/sec: 0.13363\n",
"2021-12-31 02:00:12,682 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 30/120: loss: 0.00015 learning rate: 0.00100 Time taken: 0:01:35.106025 ETA: 2:22:39.542291\n",
"INFO:tensorflow:global_step/sec: 3.08966\n",
"2021-12-31 02:00:15,594 [INFO] tensorflow: global_step/sec: 3.08966\n",
"INFO:tensorflow:epoch = 30.177083333333332, learning_rate = 0.0009999999, loss = 0.00023693091, step = 2897 (5.480 sec)\n",
"2021-12-31 02:00:18,160 [INFO] tensorflow: epoch = 30.177083333333332, learning_rate = 0.0009999999, loss = 0.00023693091, step = 2897 (5.480 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12434\n",
"2021-12-31 02:00:18,475 [INFO] tensorflow: global_step/sec: 3.12434\n",
"2021-12-31 02:00:18,811 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 2.758\n",
"INFO:tensorflow:global_step/sec: 3.11877\n",
"2021-12-31 02:00:21,360 [INFO] tensorflow: global_step/sec: 3.11877\n",
"INFO:tensorflow:epoch = 30.354166666666664, learning_rate = 0.0009999999, loss = 0.00019017012, step = 2914 (5.508 sec)\n",
"2021-12-31 02:00:23,668 [INFO] tensorflow: epoch = 30.354166666666664, learning_rate = 0.0009999999, loss = 0.00019017012, step = 2914 (5.508 sec)\n",
"INFO:tensorflow:global_step/sec: 3.05469\n",
"2021-12-31 02:00:24,307 [INFO] tensorflow: global_step/sec: 3.05469\n",
"2021-12-31 02:00:26,860 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.851\n",
"INFO:tensorflow:global_step/sec: 3.13497\n",
"2021-12-31 02:00:27,177 [INFO] tensorflow: global_step/sec: 3.13497\n",
"INFO:tensorflow:epoch = 30.53125, learning_rate = 0.0009999999, loss = 0.00023556247, step = 2931 (5.443 sec)\n",
"2021-12-31 02:00:29,112 [INFO] tensorflow: epoch = 30.53125, learning_rate = 0.0009999999, loss = 0.00023556247, step = 2931 (5.443 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10196\n",
"2021-12-31 02:00:30,079 [INFO] tensorflow: global_step/sec: 3.10196\n",
"INFO:tensorflow:global_step/sec: 3.13182\n",
"2021-12-31 02:00:32,953 [INFO] tensorflow: global_step/sec: 3.13182\n",
"INFO:tensorflow:epoch = 30.708333333333332, learning_rate = 0.0009999999, loss = 0.00022839375, step = 2948 (5.471 sec)\n",
"2021-12-31 02:00:34,582 [INFO] tensorflow: epoch = 30.708333333333332, learning_rate = 0.0009999999, loss = 0.00022839375, step = 2948 (5.471 sec)\n",
"2021-12-31 02:00:34,898 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.882\n",
"INFO:tensorflow:global_step/sec: 3.10244\n",
"2021-12-31 02:00:35,854 [INFO] tensorflow: global_step/sec: 3.10244\n",
"INFO:tensorflow:global_step/sec: 3.11726\n",
"2021-12-31 02:00:38,741 [INFO] tensorflow: global_step/sec: 3.11726\n",
"INFO:tensorflow:epoch = 30.885416666666664, learning_rate = 0.0009999999, loss = 0.00016064616, step = 2965 (5.402 sec)\n",
"2021-12-31 02:00:39,984 [INFO] tensorflow: epoch = 30.885416666666664, learning_rate = 0.0009999999, loss = 0.00016064616, step = 2965 (5.402 sec)\n",
"INFO:tensorflow:global_step/sec: 3.18815\n",
"2021-12-31 02:00:41,564 [INFO] tensorflow: global_step/sec: 3.18815\n",
"2021-12-31 02:00:42,835 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.199\n",
"2021-12-31 02:00:43,493 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 31/120: loss: 0.00023 learning rate: 0.00100 Time taken: 0:00:30.842202 ETA: 0:45:44.956016\n",
"INFO:tensorflow:global_step/sec: 3.08479\n",
"2021-12-31 02:00:44,481 [INFO] tensorflow: global_step/sec: 3.08479\n",
"INFO:tensorflow:epoch = 31.0625, learning_rate = 0.0009999999, loss = 0.00025702466, step = 2982 (5.490 sec)\n",
"2021-12-31 02:00:45,474 [INFO] tensorflow: epoch = 31.0625, learning_rate = 0.0009999999, loss = 0.00025702466, step = 2982 (5.490 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07423\n",
"2021-12-31 02:00:47,409 [INFO] tensorflow: global_step/sec: 3.07423\n",
"INFO:tensorflow:global_step/sec: 3.145\n",
"2021-12-31 02:00:50,270 [INFO] tensorflow: global_step/sec: 3.145\n",
"INFO:tensorflow:epoch = 31.239583333333332, learning_rate = 0.0009999999, loss = 0.0002174182, step = 2999 (5.446 sec)\n",
"2021-12-31 02:00:50,920 [INFO] tensorflow: epoch = 31.239583333333332, learning_rate = 0.0009999999, loss = 0.0002174182, step = 2999 (5.446 sec)\n",
"2021-12-31 02:00:50,920 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.737\n",
"INFO:tensorflow:global_step/sec: 3.10315\n",
"2021-12-31 02:00:53,171 [INFO] tensorflow: global_step/sec: 3.10315\n",
"INFO:tensorflow:global_step/sec: 3.13706\n",
"2021-12-31 02:00:56,040 [INFO] tensorflow: global_step/sec: 3.13706\n",
"INFO:tensorflow:epoch = 31.416666666666664, learning_rate = 0.0009999999, loss = 0.00020670176, step = 3016 (5.442 sec)\n",
"2021-12-31 02:00:56,362 [INFO] tensorflow: epoch = 31.416666666666664, learning_rate = 0.0009999999, loss = 0.00020670176, step = 3016 (5.442 sec)\n",
"INFO:tensorflow:global_step/sec: 3.1144\n",
"2021-12-31 02:00:58,929 [INFO] tensorflow: global_step/sec: 3.1144\n",
"2021-12-31 02:00:58,930 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.970\n",
"INFO:tensorflow:epoch = 31.59375, learning_rate = 0.0009999999, loss = 0.00024619547, step = 3033 (5.442 sec)\n",
"2021-12-31 02:01:01,804 [INFO] tensorflow: epoch = 31.59375, learning_rate = 0.0009999999, loss = 0.00024619547, step = 3033 (5.442 sec)\n",
"INFO:tensorflow:global_step/sec: 3.13007\n",
"2021-12-31 02:01:01,805 [INFO] tensorflow: global_step/sec: 3.13007\n",
"INFO:tensorflow:global_step/sec: 3.1033\n",
"2021-12-31 02:01:04,705 [INFO] tensorflow: global_step/sec: 3.1033\n",
"2021-12-31 02:01:06,921 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.029\n",
"INFO:tensorflow:epoch = 31.770833333333332, learning_rate = 0.0009999999, loss = 0.00022062179, step = 3050 (5.438 sec)\n",
"2021-12-31 02:01:07,242 [INFO] tensorflow: epoch = 31.770833333333332, learning_rate = 0.0009999999, loss = 0.00022062179, step = 3050 (5.438 sec)\n",
"INFO:tensorflow:global_step/sec: 3.14991\n",
"2021-12-31 02:01:07,562 [INFO] tensorflow: global_step/sec: 3.14991\n",
"INFO:tensorflow:global_step/sec: 3.15098\n",
"2021-12-31 02:01:10,418 [INFO] tensorflow: global_step/sec: 3.15098\n",
"INFO:tensorflow:epoch = 31.947916666666664, learning_rate = 0.0009999999, loss = 0.0002557233, step = 3067 (5.429 sec)\n",
"2021-12-31 02:01:12,671 [INFO] tensorflow: epoch = 31.947916666666664, learning_rate = 0.0009999999, loss = 0.0002557233, step = 3067 (5.429 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09341\n",
"2021-12-31 02:01:13,328 [INFO] tensorflow: global_step/sec: 3.09341\n",
"2021-12-31 02:01:14,306 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 32/120: loss: 0.00021 learning rate: 0.00100 Time taken: 0:00:30.812075 ETA: 0:45:11.462612\n",
"2021-12-31 02:01:14,944 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.930\n",
"INFO:tensorflow:global_step/sec: 3.12876\n",
"2021-12-31 02:01:16,204 [INFO] tensorflow: global_step/sec: 3.12876\n",
"INFO:tensorflow:epoch = 32.125, learning_rate = 0.0009999999, loss = 0.00019032136, step = 3084 (5.496 sec)\n",
"2021-12-31 02:01:18,167 [INFO] tensorflow: epoch = 32.125, learning_rate = 0.0009999999, loss = 0.00019032136, step = 3084 (5.496 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12611\n",
"2021-12-31 02:01:19,083 [INFO] tensorflow: global_step/sec: 3.12611\n",
"INFO:tensorflow:global_step/sec: 3.10198\n",
"2021-12-31 02:01:21,985 [INFO] tensorflow: global_step/sec: 3.10198\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"2021-12-31 02:01:22,954 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.969\n",
"INFO:tensorflow:epoch = 32.30208333333333, learning_rate = 0.0009999999, loss = 0.00028179493, step = 3101 (5.441 sec)\n",
"2021-12-31 02:01:23,608 [INFO] tensorflow: epoch = 32.30208333333333, learning_rate = 0.0009999999, loss = 0.00028179493, step = 3101 (5.441 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07548\n",
"2021-12-31 02:01:24,911 [INFO] tensorflow: global_step/sec: 3.07548\n",
"INFO:tensorflow:global_step/sec: 3.13044\n",
"2021-12-31 02:01:27,786 [INFO] tensorflow: global_step/sec: 3.13044\n",
"INFO:tensorflow:epoch = 32.479166666666664, learning_rate = 0.0009999999, loss = 0.0002054078, step = 3118 (5.415 sec)\n",
"2021-12-31 02:01:29,023 [INFO] tensorflow: epoch = 32.479166666666664, learning_rate = 0.0009999999, loss = 0.0002054078, step = 3118 (5.415 sec)\n",
"INFO:tensorflow:global_step/sec: 3.18964\n",
"2021-12-31 02:01:30,608 [INFO] tensorflow: global_step/sec: 3.18964\n",
"2021-12-31 02:01:30,933 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.067\n",
"INFO:tensorflow:global_step/sec: 3.09494\n",
"2021-12-31 02:01:33,516 [INFO] tensorflow: global_step/sec: 3.09494\n",
"INFO:tensorflow:epoch = 32.65625, learning_rate = 0.0009999999, loss = 0.00023697459, step = 3135 (5.458 sec)\n",
"2021-12-31 02:01:34,481 [INFO] tensorflow: epoch = 32.65625, learning_rate = 0.0009999999, loss = 0.00023697459, step = 3135 (5.458 sec)\n",
"INFO:tensorflow:global_step/sec: 3.0875\n",
"2021-12-31 02:01:36,431 [INFO] tensorflow: global_step/sec: 3.0875\n",
"2021-12-31 02:01:39,026 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.712\n",
"INFO:tensorflow:global_step/sec: 3.07301\n",
"2021-12-31 02:01:39,359 [INFO] tensorflow: global_step/sec: 3.07301\n",
"INFO:tensorflow:epoch = 32.83333333333333, learning_rate = 0.0009999999, loss = 0.00020485572, step = 3152 (5.483 sec)\n",
"2021-12-31 02:01:39,964 [INFO] tensorflow: epoch = 32.83333333333333, learning_rate = 0.0009999999, loss = 0.00020485572, step = 3152 (5.483 sec)\n",
"INFO:tensorflow:global_step/sec: 3.13543\n",
"2021-12-31 02:01:42,230 [INFO] tensorflow: global_step/sec: 3.13543\n",
"INFO:tensorflow:global_step/sec: 3.12073\n",
"2021-12-31 02:01:45,114 [INFO] tensorflow: global_step/sec: 3.12073\n",
"2021-12-31 02:01:45,115 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 33/120: loss: 0.00024 learning rate: 0.00100 Time taken: 0:00:30.821438 ETA: 0:44:41.465133\n",
"INFO:tensorflow:epoch = 33.010416666666664, learning_rate = 0.0009999999, loss = 0.0002340886, step = 3169 (5.458 sec)\n",
"2021-12-31 02:01:45,422 [INFO] tensorflow: epoch = 33.010416666666664, learning_rate = 0.0009999999, loss = 0.0002340886, step = 3169 (5.458 sec)\n",
"2021-12-31 02:01:47,017 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.028\n",
"INFO:tensorflow:global_step/sec: 3.12017\n",
"2021-12-31 02:01:47,998 [INFO] tensorflow: global_step/sec: 3.12017\n",
"INFO:tensorflow:epoch = 33.1875, learning_rate = 0.0009999999, loss = 0.00038417822, step = 3186 (5.479 sec)\n",
"2021-12-31 02:01:50,901 [INFO] tensorflow: epoch = 33.1875, learning_rate = 0.0009999999, loss = 0.00038417822, step = 3186 (5.479 sec)\n",
"INFO:tensorflow:global_step/sec: 3.1\n",
"2021-12-31 02:01:50,901 [INFO] tensorflow: global_step/sec: 3.1\n",
"INFO:tensorflow:global_step/sec: 3.13812\n",
"2021-12-31 02:01:53,769 [INFO] tensorflow: global_step/sec: 3.13812\n",
"2021-12-31 02:01:55,050 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.900\n",
"INFO:tensorflow:epoch = 33.36458333333333, learning_rate = 0.0009999999, loss = 0.00022170536, step = 3203 (5.455 sec)\n",
"2021-12-31 02:01:56,355 [INFO] tensorflow: epoch = 33.36458333333333, learning_rate = 0.0009999999, loss = 0.00022170536, step = 3203 (5.455 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08546\n",
"2021-12-31 02:01:56,686 [INFO] tensorflow: global_step/sec: 3.08546\n",
"INFO:tensorflow:global_step/sec: 3.09395\n",
"2021-12-31 02:01:59,595 [INFO] tensorflow: global_step/sec: 3.09395\n",
"INFO:tensorflow:epoch = 33.541666666666664, learning_rate = 0.0009999999, loss = 0.00022379155, step = 3220 (5.493 sec)\n",
"2021-12-31 02:02:01,848 [INFO] tensorflow: epoch = 33.541666666666664, learning_rate = 0.0009999999, loss = 0.00022379155, step = 3220 (5.493 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12877\n",
"2021-12-31 02:02:02,472 [INFO] tensorflow: global_step/sec: 3.12877\n",
"2021-12-31 02:02:03,115 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.797\n",
"INFO:tensorflow:global_step/sec: 3.13066\n",
"2021-12-31 02:02:05,347 [INFO] tensorflow: global_step/sec: 3.13066\n",
"INFO:tensorflow:epoch = 33.71875, learning_rate = 0.0009999999, loss = 0.00024066477, step = 3237 (5.431 sec)\n",
"2021-12-31 02:02:07,279 [INFO] tensorflow: epoch = 33.71875, learning_rate = 0.0009999999, loss = 0.00024066477, step = 3237 (5.431 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07301\n",
"2021-12-31 02:02:08,275 [INFO] tensorflow: global_step/sec: 3.07301\n",
"INFO:tensorflow:global_step/sec: 3.09768\n",
"2021-12-31 02:02:11,181 [INFO] tensorflow: global_step/sec: 3.09768\n",
"2021-12-31 02:02:11,181 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.797\n",
"INFO:tensorflow:epoch = 33.89583333333333, learning_rate = 0.0009999999, loss = 0.00027595268, step = 3254 (5.466 sec)\n",
"2021-12-31 02:02:12,746 [INFO] tensorflow: epoch = 33.89583333333333, learning_rate = 0.0009999999, loss = 0.00027595268, step = 3254 (5.466 sec)\n",
"INFO:tensorflow:global_step/sec: 3.15961\n",
"2021-12-31 02:02:14,029 [INFO] tensorflow: global_step/sec: 3.15961\n",
"2021-12-31 02:02:15,949 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 34/120: loss: 0.00027 learning rate: 0.00100 Time taken: 0:00:30.820820 ETA: 0:44:10.590508\n",
"INFO:tensorflow:global_step/sec: 3.0989\n",
"2021-12-31 02:02:16,933 [INFO] tensorflow: global_step/sec: 3.0989\n",
"INFO:tensorflow:epoch = 34.072916666666664, learning_rate = 0.0009999999, loss = 0.00022298818, step = 3271 (5.470 sec)\n",
"2021-12-31 02:02:18,216 [INFO] tensorflow: epoch = 34.072916666666664, learning_rate = 0.0009999999, loss = 0.00022298818, step = 3271 (5.470 sec)\n",
"2021-12-31 02:02:19,169 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.040\n",
"INFO:tensorflow:global_step/sec: 3.12776\n",
"2021-12-31 02:02:19,811 [INFO] tensorflow: global_step/sec: 3.12776\n",
"INFO:tensorflow:global_step/sec: 3.2123\n",
"2021-12-31 02:02:22,613 [INFO] tensorflow: global_step/sec: 3.2123\n",
"INFO:tensorflow:epoch = 34.25, learning_rate = 0.0009999999, loss = 0.00022330222, step = 3288 (5.389 sec)\n",
"2021-12-31 02:02:23,605 [INFO] tensorflow: epoch = 34.25, learning_rate = 0.0009999999, loss = 0.00022330222, step = 3288 (5.389 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09327\n",
"2021-12-31 02:02:25,522 [INFO] tensorflow: global_step/sec: 3.09327\n",
"2021-12-31 02:02:27,159 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.030\n",
"INFO:tensorflow:global_step/sec: 3.11981\n",
"2021-12-31 02:02:28,407 [INFO] tensorflow: global_step/sec: 3.11981\n",
"INFO:tensorflow:epoch = 34.42708333333333, learning_rate = 0.0009999999, loss = 0.00027549744, step = 3305 (5.452 sec)\n",
"2021-12-31 02:02:29,057 [INFO] tensorflow: epoch = 34.42708333333333, learning_rate = 0.0009999999, loss = 0.00027549744, step = 3305 (5.452 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10408\n",
"2021-12-31 02:02:31,306 [INFO] tensorflow: global_step/sec: 3.10408\n",
"INFO:tensorflow:global_step/sec: 3.15005\n",
"2021-12-31 02:02:34,163 [INFO] tensorflow: global_step/sec: 3.15005\n",
"INFO:tensorflow:epoch = 34.604166666666664, learning_rate = 0.0009999999, loss = 0.00033780435, step = 3322 (5.447 sec)\n",
"2021-12-31 02:02:34,504 [INFO] tensorflow: epoch = 34.604166666666664, learning_rate = 0.0009999999, loss = 0.00033780435, step = 3322 (5.447 sec)\n",
"2021-12-31 02:02:35,144 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.051\n",
"INFO:tensorflow:global_step/sec: 3.13573\n",
"2021-12-31 02:02:37,033 [INFO] tensorflow: global_step/sec: 3.13573\n",
"INFO:tensorflow:epoch = 34.78125, learning_rate = 0.0009999999, loss = 0.00017757488, step = 3339 (5.442 sec)\n",
"2021-12-31 02:02:39,947 [INFO] tensorflow: epoch = 34.78125, learning_rate = 0.0009999999, loss = 0.00017757488, step = 3339 (5.442 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08866\n",
"2021-12-31 02:02:39,947 [INFO] tensorflow: global_step/sec: 3.08866\n",
"INFO:tensorflow:global_step/sec: 3.1252\n",
"2021-12-31 02:02:42,827 [INFO] tensorflow: global_step/sec: 3.1252\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"2021-12-31 02:02:43,149 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.983\n",
"INFO:tensorflow:epoch = 34.95833333333333, learning_rate = 0.0009999999, loss = 0.00028036637, step = 3356 (5.430 sec)\n",
"2021-12-31 02:02:45,376 [INFO] tensorflow: epoch = 34.95833333333333, learning_rate = 0.0009999999, loss = 0.00028036637, step = 3356 (5.430 sec)\n",
"INFO:tensorflow:global_step/sec: 3.13987\n",
"2021-12-31 02:02:45,694 [INFO] tensorflow: global_step/sec: 3.13987\n",
"2021-12-31 02:02:46,671 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 35/120: loss: 0.00023 learning rate: 0.00100 Time taken: 0:00:30.711346 ETA: 0:43:30.464382\n",
"INFO:tensorflow:global_step/sec: 3.10028\n",
"2021-12-31 02:02:48,597 [INFO] tensorflow: global_step/sec: 3.10028\n",
"INFO:tensorflow:epoch = 35.135416666666664, learning_rate = 0.0009999999, loss = 0.00024327867, step = 3373 (5.474 sec)\n",
"2021-12-31 02:02:50,851 [INFO] tensorflow: epoch = 35.135416666666664, learning_rate = 0.0009999999, loss = 0.00024327867, step = 3373 (5.474 sec)\n",
"2021-12-31 02:02:51,168 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.941\n",
"INFO:tensorflow:global_step/sec: 3.10518\n",
"2021-12-31 02:02:51,495 [INFO] tensorflow: global_step/sec: 3.10518\n",
"INFO:tensorflow:global_step/sec: 3.14777\n",
"2021-12-31 02:02:54,354 [INFO] tensorflow: global_step/sec: 3.14777\n",
"INFO:tensorflow:epoch = 35.3125, learning_rate = 0.0009999999, loss = 0.00022454353, step = 3390 (5.404 sec)\n",
"2021-12-31 02:02:56,255 [INFO] tensorflow: epoch = 35.3125, learning_rate = 0.0009999999, loss = 0.00022454353, step = 3390 (5.404 sec)\n",
"INFO:tensorflow:global_step/sec: 3.14609\n",
"2021-12-31 02:02:57,215 [INFO] tensorflow: global_step/sec: 3.14609\n",
"2021-12-31 02:02:59,134 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.109\n",
"INFO:tensorflow:global_step/sec: 3.12625\n",
"2021-12-31 02:03:00,094 [INFO] tensorflow: global_step/sec: 3.12625\n",
"INFO:tensorflow:epoch = 35.48958333333333, learning_rate = 0.0009999999, loss = 0.00027559517, step = 3407 (5.455 sec)\n",
"2021-12-31 02:03:01,711 [INFO] tensorflow: epoch = 35.48958333333333, learning_rate = 0.0009999999, loss = 0.00027559517, step = 3407 (5.455 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09801\n",
"2021-12-31 02:03:02,999 [INFO] tensorflow: global_step/sec: 3.09801\n",
"INFO:tensorflow:global_step/sec: 3.0745\n",
"2021-12-31 02:03:05,926 [INFO] tensorflow: global_step/sec: 3.0745\n",
"INFO:tensorflow:epoch = 35.666666666666664, learning_rate = 0.0009999999, loss = 0.00025373622, step = 3424 (5.489 sec)\n",
"2021-12-31 02:03:07,200 [INFO] tensorflow: epoch = 35.666666666666664, learning_rate = 0.0009999999, loss = 0.00025373622, step = 3424 (5.489 sec)\n",
"2021-12-31 02:03:07,200 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.795\n",
"INFO:tensorflow:global_step/sec: 3.12181\n",
"2021-12-31 02:03:08,809 [INFO] tensorflow: global_step/sec: 3.12181\n",
"INFO:tensorflow:global_step/sec: 3.10236\n",
"2021-12-31 02:03:11,710 [INFO] tensorflow: global_step/sec: 3.10236\n",
"INFO:tensorflow:epoch = 35.84375, learning_rate = 0.0009999999, loss = 0.00019812775, step = 3441 (5.492 sec)\n",
"2021-12-31 02:03:12,692 [INFO] tensorflow: epoch = 35.84375, learning_rate = 0.0009999999, loss = 0.00019812775, step = 3441 (5.492 sec)\n",
"INFO:tensorflow:global_step/sec: 3.05757\n",
"2021-12-31 02:03:14,653 [INFO] tensorflow: global_step/sec: 3.05757\n",
"2021-12-31 02:03:15,257 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.826\n",
"INFO:tensorflow:global_step/sec: 3.17889\n",
"2021-12-31 02:03:17,485 [INFO] tensorflow: global_step/sec: 3.17889\n",
"2021-12-31 02:03:17,486 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 36/120: loss: 0.00026 learning rate: 0.00100 Time taken: 0:00:30.816260 ETA: 0:43:08.565868\n",
"INFO:tensorflow:epoch = 36.02083333333333, learning_rate = 0.0009999999, loss = 0.00030115846, step = 3458 (5.418 sec)\n",
"2021-12-31 02:03:18,110 [INFO] tensorflow: epoch = 36.02083333333333, learning_rate = 0.0009999999, loss = 0.00030115846, step = 3458 (5.418 sec)\n",
"INFO:tensorflow:global_step/sec: 3.17189\n",
"2021-12-31 02:03:20,322 [INFO] tensorflow: global_step/sec: 3.17189\n",
"INFO:tensorflow:global_step/sec: 3.06893\n",
"2021-12-31 02:03:23,255 [INFO] tensorflow: global_step/sec: 3.06893\n",
"2021-12-31 02:03:23,255 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.004\n",
"INFO:tensorflow:epoch = 36.197916666666664, learning_rate = 0.0009999999, loss = 0.00021981688, step = 3475 (5.464 sec)\n",
"2021-12-31 02:03:23,574 [INFO] tensorflow: epoch = 36.197916666666664, learning_rate = 0.0009999999, loss = 0.00021981688, step = 3475 (5.464 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09389\n",
"2021-12-31 02:03:26,164 [INFO] tensorflow: global_step/sec: 3.09389\n",
"INFO:tensorflow:epoch = 36.375, learning_rate = 0.0009999999, loss = 0.0002635182, step = 3492 (5.514 sec)\n",
"2021-12-31 02:03:29,088 [INFO] tensorflow: epoch = 36.375, learning_rate = 0.0009999999, loss = 0.0002635182, step = 3492 (5.514 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07691\n",
"2021-12-31 02:03:29,089 [INFO] tensorflow: global_step/sec: 3.07691\n",
"2021-12-31 02:03:31,342 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.733\n",
"INFO:tensorflow:global_step/sec: 3.10372\n",
"2021-12-31 02:03:31,988 [INFO] tensorflow: global_step/sec: 3.10372\n",
"INFO:tensorflow:epoch = 36.55208333333333, learning_rate = 0.0009999999, loss = 0.00023157206, step = 3509 (5.435 sec)\n",
"2021-12-31 02:03:34,523 [INFO] tensorflow: epoch = 36.55208333333333, learning_rate = 0.0009999999, loss = 0.00023157206, step = 3509 (5.435 sec)\n",
"INFO:tensorflow:global_step/sec: 3.14823\n",
"2021-12-31 02:03:34,847 [INFO] tensorflow: global_step/sec: 3.14823\n",
"INFO:tensorflow:global_step/sec: 3.11476\n",
"2021-12-31 02:03:37,737 [INFO] tensorflow: global_step/sec: 3.11476\n",
"2021-12-31 02:03:39,333 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.029\n",
"INFO:tensorflow:epoch = 36.729166666666664, learning_rate = 0.0009999999, loss = 0.0002554315, step = 3526 (5.445 sec)\n",
"2021-12-31 02:03:39,968 [INFO] tensorflow: epoch = 36.729166666666664, learning_rate = 0.0009999999, loss = 0.0002554315, step = 3526 (5.445 sec)\n",
"INFO:tensorflow:global_step/sec: 3.1284\n",
"2021-12-31 02:03:40,614 [INFO] tensorflow: global_step/sec: 3.1284\n",
"INFO:tensorflow:global_step/sec: 3.12903\n",
"2021-12-31 02:03:43,490 [INFO] tensorflow: global_step/sec: 3.12903\n",
"INFO:tensorflow:epoch = 36.90625, learning_rate = 0.0009999999, loss = 0.00032151234, step = 3543 (5.463 sec)\n",
"2021-12-31 02:03:45,431 [INFO] tensorflow: epoch = 36.90625, learning_rate = 0.0009999999, loss = 0.00032151234, step = 3543 (5.463 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11319\n",
"2021-12-31 02:03:46,381 [INFO] tensorflow: global_step/sec: 3.11319\n",
"2021-12-31 02:03:47,350 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.947\n",
"2021-12-31 02:03:48,326 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 37/120: loss: 0.00025 learning rate: 0.00100 Time taken: 0:00:30.857602 ETA: 0:42:41.180976\n",
"INFO:tensorflow:global_step/sec: 3.07505\n",
"2021-12-31 02:03:49,308 [INFO] tensorflow: global_step/sec: 3.07505\n",
"INFO:tensorflow:epoch = 37.08333333333333, learning_rate = 0.0009999999, loss = 0.00031327084, step = 3560 (5.489 sec)\n",
"2021-12-31 02:03:50,920 [INFO] tensorflow: epoch = 37.08333333333333, learning_rate = 0.0009999999, loss = 0.00031327084, step = 3560 (5.489 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10528\n",
"2021-12-31 02:03:52,206 [INFO] tensorflow: global_step/sec: 3.10528\n",
"INFO:tensorflow:global_step/sec: 3.14709\n",
"2021-12-31 02:03:55,066 [INFO] tensorflow: global_step/sec: 3.14709\n",
"2021-12-31 02:03:55,387 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.886\n",
"INFO:tensorflow:epoch = 37.260416666666664, learning_rate = 0.0009999999, loss = 0.00023214112, step = 3577 (5.433 sec)\n",
"2021-12-31 02:03:56,354 [INFO] tensorflow: epoch = 37.260416666666664, learning_rate = 0.0009999999, loss = 0.00023214112, step = 3577 (5.433 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09991\n",
"2021-12-31 02:03:57,969 [INFO] tensorflow: global_step/sec: 3.09991\n",
"INFO:tensorflow:global_step/sec: 3.09839\n",
"2021-12-31 02:04:00,874 [INFO] tensorflow: global_step/sec: 3.09839\n",
"INFO:tensorflow:epoch = 37.4375, learning_rate = 0.0009999999, loss = 0.00018953544, step = 3594 (5.489 sec)\n",
"2021-12-31 02:04:01,842 [INFO] tensorflow: epoch = 37.4375, learning_rate = 0.0009999999, loss = 0.00018953544, step = 3594 (5.489 sec)\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"2021-12-31 02:04:03,448 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.813\n",
"INFO:tensorflow:global_step/sec: 3.09458\n",
"2021-12-31 02:04:03,782 [INFO] tensorflow: global_step/sec: 3.09458\n",
"INFO:tensorflow:global_step/sec: 3.13484\n",
"2021-12-31 02:04:06,653 [INFO] tensorflow: global_step/sec: 3.13484\n",
"INFO:tensorflow:epoch = 37.61458333333333, learning_rate = 0.0009999999, loss = 0.0002177629, step = 3611 (5.471 sec)\n",
"2021-12-31 02:04:07,313 [INFO] tensorflow: epoch = 37.61458333333333, learning_rate = 0.0009999999, loss = 0.0002177629, step = 3611 (5.471 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07231\n",
"2021-12-31 02:04:09,582 [INFO] tensorflow: global_step/sec: 3.07231\n",
"2021-12-31 02:04:11,552 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.679\n",
"INFO:tensorflow:global_step/sec: 3.06304\n",
"2021-12-31 02:04:12,521 [INFO] tensorflow: global_step/sec: 3.06304\n",
"INFO:tensorflow:epoch = 37.791666666666664, learning_rate = 0.0009999999, loss = 0.00018512359, step = 3628 (5.524 sec)\n",
"2021-12-31 02:04:12,837 [INFO] tensorflow: epoch = 37.791666666666664, learning_rate = 0.0009999999, loss = 0.00018512359, step = 3628 (5.524 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11124\n",
"2021-12-31 02:04:15,413 [INFO] tensorflow: global_step/sec: 3.11124\n",
"INFO:tensorflow:epoch = 37.96875, learning_rate = 0.0009999999, loss = 0.00018366278, step = 3645 (5.422 sec)\n",
"2021-12-31 02:04:18,259 [INFO] tensorflow: epoch = 37.96875, learning_rate = 0.0009999999, loss = 0.00018366278, step = 3645 (5.422 sec)\n",
"INFO:tensorflow:global_step/sec: 3.16152\n",
"2021-12-31 02:04:18,260 [INFO] tensorflow: global_step/sec: 3.16152\n",
"2021-12-31 02:04:19,216 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 38/120: loss: 0.00019 learning rate: 0.00100 Time taken: 0:00:30.887269 ETA: 0:42:12.756099\n",
"2021-12-31 02:04:19,512 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.126\n",
"INFO:tensorflow:global_step/sec: 3.17987\n",
"2021-12-31 02:04:21,090 [INFO] tensorflow: global_step/sec: 3.17987\n",
"INFO:tensorflow:epoch = 38.14583333333333, learning_rate = 0.0009999999, loss = 0.00018901107, step = 3662 (5.368 sec)\n",
"2021-12-31 02:04:23,628 [INFO] tensorflow: epoch = 38.14583333333333, learning_rate = 0.0009999999, loss = 0.00018901107, step = 3662 (5.368 sec)\n",
"INFO:tensorflow:global_step/sec: 3.15224\n",
"2021-12-31 02:04:23,945 [INFO] tensorflow: global_step/sec: 3.15224\n",
"INFO:tensorflow:global_step/sec: 3.1469\n",
"2021-12-31 02:04:26,805 [INFO] tensorflow: global_step/sec: 3.1469\n",
"2021-12-31 02:04:27,430 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.261\n",
"INFO:tensorflow:epoch = 38.322916666666664, learning_rate = 0.0009999999, loss = 0.00027874907, step = 3679 (5.439 sec)\n",
"2021-12-31 02:04:29,066 [INFO] tensorflow: epoch = 38.322916666666664, learning_rate = 0.0009999999, loss = 0.00027874907, step = 3679 (5.439 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09966\n",
"2021-12-31 02:04:29,709 [INFO] tensorflow: global_step/sec: 3.09966\n",
"INFO:tensorflow:global_step/sec: 3.13976\n",
"2021-12-31 02:04:32,575 [INFO] tensorflow: global_step/sec: 3.13976\n",
"INFO:tensorflow:epoch = 38.5, learning_rate = 0.0009999999, loss = 0.00032934928, step = 3696 (5.413 sec)\n",
"2021-12-31 02:04:34,479 [INFO] tensorflow: epoch = 38.5, learning_rate = 0.0009999999, loss = 0.00032934928, step = 3696 (5.413 sec)\n",
"INFO:tensorflow:global_step/sec: 3.15821\n",
"2021-12-31 02:04:35,425 [INFO] tensorflow: global_step/sec: 3.15821\n",
"2021-12-31 02:04:35,426 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.014\n",
"INFO:tensorflow:global_step/sec: 3.13531\n",
"2021-12-31 02:04:38,296 [INFO] tensorflow: global_step/sec: 3.13531\n",
"INFO:tensorflow:epoch = 38.67708333333333, learning_rate = 0.0009999999, loss = 0.00023085275, step = 3713 (5.401 sec)\n",
"2021-12-31 02:04:39,880 [INFO] tensorflow: epoch = 38.67708333333333, learning_rate = 0.0009999999, loss = 0.00023085275, step = 3713 (5.401 sec)\n",
"INFO:tensorflow:global_step/sec: 3.19594\n",
"2021-12-31 02:04:41,112 [INFO] tensorflow: global_step/sec: 3.19594\n",
"2021-12-31 02:04:43,375 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.162\n",
"INFO:tensorflow:global_step/sec: 3.09873\n",
"2021-12-31 02:04:44,016 [INFO] tensorflow: global_step/sec: 3.09873\n",
"INFO:tensorflow:epoch = 38.854166666666664, learning_rate = 0.0009999999, loss = 0.00021847687, step = 3730 (5.449 sec)\n",
"2021-12-31 02:04:45,329 [INFO] tensorflow: epoch = 38.854166666666664, learning_rate = 0.0009999999, loss = 0.00021847687, step = 3730 (5.449 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08643\n",
"2021-12-31 02:04:46,932 [INFO] tensorflow: global_step/sec: 3.08643\n",
"INFO:tensorflow:global_step/sec: 3.07102\n",
"2021-12-31 02:04:49,863 [INFO] tensorflow: global_step/sec: 3.07102\n",
"2021-12-31 02:04:49,864 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 39/120: loss: 0.00021 learning rate: 0.00100 Time taken: 0:00:30.638531 ETA: 0:41:21.721009\n",
"INFO:tensorflow:epoch = 39.03125, learning_rate = 0.0009999999, loss = 0.00023957586, step = 3747 (5.487 sec)\n",
"2021-12-31 02:04:50,816 [INFO] tensorflow: epoch = 39.03125, learning_rate = 0.0009999999, loss = 0.00023957586, step = 3747 (5.487 sec)\n",
"2021-12-31 02:04:51,468 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.711\n",
"INFO:tensorflow:global_step/sec: 3.1471\n",
"2021-12-31 02:04:52,723 [INFO] tensorflow: global_step/sec: 3.1471\n",
"INFO:tensorflow:global_step/sec: 3.10427\n",
"2021-12-31 02:04:55,622 [INFO] tensorflow: global_step/sec: 3.10427\n",
"INFO:tensorflow:epoch = 39.20833333333333, learning_rate = 0.0009999999, loss = 0.00030542281, step = 3764 (5.449 sec)\n",
"2021-12-31 02:04:56,265 [INFO] tensorflow: epoch = 39.20833333333333, learning_rate = 0.0009999999, loss = 0.00030542281, step = 3764 (5.449 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09555\n",
"2021-12-31 02:04:58,529 [INFO] tensorflow: global_step/sec: 3.09555\n",
"2021-12-31 02:04:59,461 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.024\n",
"INFO:tensorflow:global_step/sec: 3.16843\n",
"2021-12-31 02:05:01,370 [INFO] tensorflow: global_step/sec: 3.16843\n",
"INFO:tensorflow:epoch = 39.385416666666664, learning_rate = 0.0009999999, loss = 0.00026118432, step = 3781 (5.432 sec)\n",
"2021-12-31 02:05:01,697 [INFO] tensorflow: epoch = 39.385416666666664, learning_rate = 0.0009999999, loss = 0.00026118432, step = 3781 (5.432 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08668\n",
"2021-12-31 02:05:04,285 [INFO] tensorflow: global_step/sec: 3.08668\n",
"INFO:tensorflow:epoch = 39.5625, learning_rate = 0.0009999999, loss = 0.0003627761, step = 3798 (5.479 sec)\n",
"2021-12-31 02:05:07,176 [INFO] tensorflow: epoch = 39.5625, learning_rate = 0.0009999999, loss = 0.0003627761, step = 3798 (5.479 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11245\n",
"2021-12-31 02:05:07,177 [INFO] tensorflow: global_step/sec: 3.11245\n",
"2021-12-31 02:05:07,480 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.940\n",
"INFO:tensorflow:global_step/sec: 3.08999\n",
"2021-12-31 02:05:10,090 [INFO] tensorflow: global_step/sec: 3.08999\n",
"INFO:tensorflow:epoch = 39.73958333333333, learning_rate = 0.0009999999, loss = 0.0002601355, step = 3815 (5.450 sec)\n",
"2021-12-31 02:05:12,626 [INFO] tensorflow: epoch = 39.73958333333333, learning_rate = 0.0009999999, loss = 0.0002601355, step = 3815 (5.450 sec)\n",
"INFO:tensorflow:global_step/sec: 3.15523\n",
"2021-12-31 02:05:12,942 [INFO] tensorflow: global_step/sec: 3.15523\n",
"2021-12-31 02:05:15,503 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.930\n",
"INFO:tensorflow:global_step/sec: 3.10266\n",
"2021-12-31 02:05:15,843 [INFO] tensorflow: global_step/sec: 3.10266\n",
"INFO:tensorflow:epoch = 39.916666666666664, learning_rate = 0.0009999999, loss = 0.00019769114, step = 3832 (5.489 sec)\n",
"2021-12-31 02:05:18,115 [INFO] tensorflow: epoch = 39.916666666666664, learning_rate = 0.0009999999, loss = 0.00019769114, step = 3832 (5.489 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09172\n",
"2021-12-31 02:05:18,754 [INFO] tensorflow: global_step/sec: 3.09172\n",
"INFO:tensorflow:Saving checkpoints for step-3840.\n",
"2021-12-31 02:05:20,364 [INFO] tensorflow: Saving checkpoints for step-3840.\n",
"2021-12-31 02:05:24,053 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 23, 0.00s/step\n",
"2021-12-31 02:05:35,357 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 23, 1.13s/step\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"2021-12-31 02:05:46,437 [INFO] iva.detectnet_v2.evaluation.evaluation: step 20 / 23, 1.11s/step\n",
"Matching predictions to ground truth, class 1/1.: 100%|█| 24981/24981 [00:01<00:00, 15220.36it/s]\n",
"Epoch 40/120\n",
"=========================\n",
"\n",
"Validation cost: 0.000263\n",
"Mean average_precision (in %): 46.6108\n",
"\n",
"class name average precision (in %)\n",
"------------ --------------------------\n",
"car 46.6108\n",
"\n",
"Median Inference Time: 0.016943\n",
"INFO:tensorflow:epoch = 40.0, learning_rate = 0.0009999999, loss = 0.00024275055, step = 3840 (34.050 sec)\n",
"2021-12-31 02:05:52,164 [INFO] tensorflow: epoch = 40.0, learning_rate = 0.0009999999, loss = 0.00024275055, step = 3840 (34.050 sec)\n",
"2021-12-31 02:05:52,165 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 40/120: loss: 0.00024 learning rate: 0.00100 Time taken: 0:01:02.300122 ETA: 1:23:04.009781\n",
"INFO:tensorflow:global_step/sec: 0.261839\n",
"2021-12-31 02:05:53,126 [INFO] tensorflow: global_step/sec: 0.261839\n",
"2021-12-31 02:05:55,045 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 5.058\n",
"INFO:tensorflow:global_step/sec: 3.114\n",
"2021-12-31 02:05:56,016 [INFO] tensorflow: global_step/sec: 3.114\n",
"INFO:tensorflow:epoch = 40.17708333333333, learning_rate = 0.0009999999, loss = 0.00022768865, step = 3857 (5.465 sec)\n",
"2021-12-31 02:05:57,629 [INFO] tensorflow: epoch = 40.17708333333333, learning_rate = 0.0009999999, loss = 0.00022768865, step = 3857 (5.465 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12509\n",
"2021-12-31 02:05:58,896 [INFO] tensorflow: global_step/sec: 3.12509\n",
"INFO:tensorflow:global_step/sec: 3.14741\n",
"2021-12-31 02:06:01,756 [INFO] tensorflow: global_step/sec: 3.14741\n",
"INFO:tensorflow:epoch = 40.354166666666664, learning_rate = 0.0009999999, loss = 0.00025592765, step = 3874 (5.429 sec)\n",
"2021-12-31 02:06:03,058 [INFO] tensorflow: epoch = 40.354166666666664, learning_rate = 0.0009999999, loss = 0.00025592765, step = 3874 (5.429 sec)\n",
"2021-12-31 02:06:03,058 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.961\n",
"INFO:tensorflow:global_step/sec: 3.06277\n",
"2021-12-31 02:06:04,694 [INFO] tensorflow: global_step/sec: 3.06277\n",
"INFO:tensorflow:global_step/sec: 3.15204\n",
"2021-12-31 02:06:07,550 [INFO] tensorflow: global_step/sec: 3.15204\n",
"INFO:tensorflow:epoch = 40.53125, learning_rate = 0.0009999999, loss = 0.00026824954, step = 3891 (5.427 sec)\n",
"2021-12-31 02:06:08,484 [INFO] tensorflow: epoch = 40.53125, learning_rate = 0.0009999999, loss = 0.00026824954, step = 3891 (5.427 sec)\n",
"INFO:tensorflow:global_step/sec: 3.13477\n",
"2021-12-31 02:06:10,421 [INFO] tensorflow: global_step/sec: 3.13477\n",
"2021-12-31 02:06:11,077 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.943\n",
"INFO:tensorflow:global_step/sec: 3.13312\n",
"2021-12-31 02:06:13,293 [INFO] tensorflow: global_step/sec: 3.13312\n",
"INFO:tensorflow:epoch = 40.70833333333333, learning_rate = 0.0009999999, loss = 0.00023226727, step = 3908 (5.455 sec)\n",
"2021-12-31 02:06:13,939 [INFO] tensorflow: epoch = 40.70833333333333, learning_rate = 0.0009999999, loss = 0.00023226727, step = 3908 (5.455 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12905\n",
"2021-12-31 02:06:16,169 [INFO] tensorflow: global_step/sec: 3.12905\n",
"INFO:tensorflow:global_step/sec: 3.09483\n",
"2021-12-31 02:06:19,077 [INFO] tensorflow: global_step/sec: 3.09483\n",
"2021-12-31 02:06:19,078 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.997\n",
"INFO:tensorflow:epoch = 40.885416666666664, learning_rate = 0.0009999999, loss = 0.00028854975, step = 3925 (5.464 sec)\n",
"2021-12-31 02:06:19,403 [INFO] tensorflow: epoch = 40.885416666666664, learning_rate = 0.0009999999, loss = 0.00028854975, step = 3925 (5.464 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11312\n",
"2021-12-31 02:06:21,968 [INFO] tensorflow: global_step/sec: 3.11312\n",
"2021-12-31 02:06:22,902 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 41/120: loss: 0.00018 learning rate: 0.00100 Time taken: 0:00:30.738402 ETA: 0:40:28.333730\n",
"INFO:tensorflow:epoch = 41.0625, learning_rate = 0.0009999999, loss = 0.00021445828, step = 3942 (5.402 sec)\n",
"2021-12-31 02:06:24,805 [INFO] tensorflow: epoch = 41.0625, learning_rate = 0.0009999999, loss = 0.00021445828, step = 3942 (5.402 sec)\n",
"INFO:tensorflow:global_step/sec: 3.17182\n",
"2021-12-31 02:06:24,806 [INFO] tensorflow: global_step/sec: 3.17182\n",
"2021-12-31 02:06:27,040 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.119\n",
"INFO:tensorflow:global_step/sec: 3.13047\n",
"2021-12-31 02:06:27,681 [INFO] tensorflow: global_step/sec: 3.13047\n",
"INFO:tensorflow:epoch = 41.23958333333333, learning_rate = 0.0009999999, loss = 0.00029084214, step = 3959 (5.414 sec)\n",
"2021-12-31 02:06:30,219 [INFO] tensorflow: epoch = 41.23958333333333, learning_rate = 0.0009999999, loss = 0.00029084214, step = 3959 (5.414 sec)\n",
"INFO:tensorflow:global_step/sec: 3.14579\n",
"2021-12-31 02:06:30,542 [INFO] tensorflow: global_step/sec: 3.14579\n",
"INFO:tensorflow:global_step/sec: 3.13174\n",
"2021-12-31 02:06:33,416 [INFO] tensorflow: global_step/sec: 3.13174\n",
"2021-12-31 02:06:35,030 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.031\n",
"INFO:tensorflow:epoch = 41.416666666666664, learning_rate = 0.0009999999, loss = 0.00027365508, step = 3976 (5.449 sec)\n",
"2021-12-31 02:06:35,668 [INFO] tensorflow: epoch = 41.416666666666664, learning_rate = 0.0009999999, loss = 0.00027365508, step = 3976 (5.449 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08764\n",
"2021-12-31 02:06:36,331 [INFO] tensorflow: global_step/sec: 3.08764\n",
"INFO:tensorflow:global_step/sec: 3.18244\n",
"2021-12-31 02:06:39,159 [INFO] tensorflow: global_step/sec: 3.18244\n",
"INFO:tensorflow:epoch = 41.59375, learning_rate = 0.0009999999, loss = 0.00027586543, step = 3993 (5.429 sec)\n",
"2021-12-31 02:06:41,097 [INFO] tensorflow: epoch = 41.59375, learning_rate = 0.0009999999, loss = 0.00027586543, step = 3993 (5.429 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08536\n",
"2021-12-31 02:06:42,076 [INFO] tensorflow: global_step/sec: 3.08536\n",
"2021-12-31 02:06:43,036 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.983\n",
"INFO:tensorflow:global_step/sec: 3.0905\n",
"2021-12-31 02:06:44,988 [INFO] tensorflow: global_step/sec: 3.0905\n",
"INFO:tensorflow:epoch = 41.77083333333333, learning_rate = 0.0009999999, loss = 0.00020894797, step = 4010 (5.459 sec)\n",
"2021-12-31 02:06:46,556 [INFO] tensorflow: epoch = 41.77083333333333, learning_rate = 0.0009999999, loss = 0.00020894797, step = 4010 (5.459 sec)\n",
"INFO:tensorflow:global_step/sec: 3.21022\n",
"2021-12-31 02:06:47,791 [INFO] tensorflow: global_step/sec: 3.21022\n",
"INFO:tensorflow:global_step/sec: 3.06404\n",
"2021-12-31 02:06:50,729 [INFO] tensorflow: global_step/sec: 3.06404\n",
"2021-12-31 02:06:51,026 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.032\n",
"INFO:tensorflow:epoch = 41.947916666666664, learning_rate = 0.0009999999, loss = 0.0002663147, step = 4027 (5.425 sec)\n",
"2021-12-31 02:06:51,981 [INFO] tensorflow: epoch = 41.947916666666664, learning_rate = 0.0009999999, loss = 0.0002663147, step = 4027 (5.425 sec)\n",
"INFO:tensorflow:global_step/sec: 3.19239\n",
"2021-12-31 02:06:53,548 [INFO] tensorflow: global_step/sec: 3.19239\n",
"2021-12-31 02:06:53,549 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 42/120: loss: 0.00024 learning rate: 0.00100 Time taken: 0:00:30.646526 ETA: 0:39:50.429017\n",
"INFO:tensorflow:global_step/sec: 3.14376\n",
"2021-12-31 02:06:56,411 [INFO] tensorflow: global_step/sec: 3.14376\n",
"INFO:tensorflow:epoch = 42.125, learning_rate = 0.0009999999, loss = 0.00019230747, step = 4044 (5.414 sec)\n",
"2021-12-31 02:06:57,395 [INFO] tensorflow: epoch = 42.125, learning_rate = 0.0009999999, loss = 0.00019230747, step = 4044 (5.414 sec)\n",
"2021-12-31 02:06:59,007 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.062\n",
"INFO:tensorflow:global_step/sec: 3.08972\n",
"2021-12-31 02:06:59,323 [INFO] tensorflow: global_step/sec: 3.08972\n",
"INFO:tensorflow:global_step/sec: 3.08679\n",
"2021-12-31 02:07:02,239 [INFO] tensorflow: global_step/sec: 3.08679\n",
"INFO:tensorflow:epoch = 42.30208333333333, learning_rate = 0.0009999999, loss = 0.00026337092, step = 4061 (5.512 sec)\n",
"2021-12-31 02:07:02,907 [INFO] tensorflow: epoch = 42.30208333333333, learning_rate = 0.0009999999, loss = 0.00026337092, step = 4061 (5.512 sec)\n",
"INFO:tensorflow:global_step/sec: 3.06253\n",
"2021-12-31 02:07:05,178 [INFO] tensorflow: global_step/sec: 3.06253\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"2021-12-31 02:07:07,091 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.740\n",
"INFO:tensorflow:global_step/sec: 3.1694\n",
"2021-12-31 02:07:08,017 [INFO] tensorflow: global_step/sec: 3.1694\n",
"INFO:tensorflow:epoch = 42.479166666666664, learning_rate = 0.0009999999, loss = 0.00018202995, step = 4078 (5.425 sec)\n",
"2021-12-31 02:07:08,332 [INFO] tensorflow: epoch = 42.479166666666664, learning_rate = 0.0009999999, loss = 0.00018202995, step = 4078 (5.425 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10991\n",
"2021-12-31 02:07:10,911 [INFO] tensorflow: global_step/sec: 3.10991\n",
"INFO:tensorflow:epoch = 42.65625, learning_rate = 0.0009999999, loss = 0.00026308582, step = 4095 (5.526 sec)\n",
"2021-12-31 02:07:13,857 [INFO] tensorflow: epoch = 42.65625, learning_rate = 0.0009999999, loss = 0.00026308582, step = 4095 (5.526 sec)\n",
"INFO:tensorflow:global_step/sec: 3.05421\n",
"2021-12-31 02:07:13,858 [INFO] tensorflow: global_step/sec: 3.05421\n",
"2021-12-31 02:07:15,144 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.835\n",
"INFO:tensorflow:global_step/sec: 3.10968\n",
"2021-12-31 02:07:16,752 [INFO] tensorflow: global_step/sec: 3.10968\n",
"INFO:tensorflow:epoch = 42.83333333333333, learning_rate = 0.0009999999, loss = 0.00023425464, step = 4112 (5.467 sec)\n",
"2021-12-31 02:07:19,324 [INFO] tensorflow: epoch = 42.83333333333333, learning_rate = 0.0009999999, loss = 0.00023425464, step = 4112 (5.467 sec)\n",
"INFO:tensorflow:global_step/sec: 3.13169\n",
"2021-12-31 02:07:19,626 [INFO] tensorflow: global_step/sec: 3.13169\n",
"INFO:tensorflow:global_step/sec: 3.13808\n",
"2021-12-31 02:07:22,494 [INFO] tensorflow: global_step/sec: 3.13808\n",
"2021-12-31 02:07:23,111 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.104\n",
"2021-12-31 02:07:24,408 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 43/120: loss: 0.00022 learning rate: 0.00100 Time taken: 0:00:30.866551 ETA: 0:39:36.724403\n",
"INFO:tensorflow:epoch = 43.010416666666664, learning_rate = 0.0009999999, loss = 0.0002529886, step = 4129 (5.390 sec)\n",
"2021-12-31 02:07:24,715 [INFO] tensorflow: epoch = 43.010416666666664, learning_rate = 0.0009999999, loss = 0.0002529886, step = 4129 (5.390 sec)\n",
"INFO:tensorflow:global_step/sec: 3.15617\n",
"2021-12-31 02:07:25,346 [INFO] tensorflow: global_step/sec: 3.15617\n",
"INFO:tensorflow:global_step/sec: 3.17068\n",
"2021-12-31 02:07:28,184 [INFO] tensorflow: global_step/sec: 3.17068\n",
"INFO:tensorflow:epoch = 43.1875, learning_rate = 0.0009999999, loss = 0.0003409887, step = 4146 (5.413 sec)\n",
"2021-12-31 02:07:30,128 [INFO] tensorflow: epoch = 43.1875, learning_rate = 0.0009999999, loss = 0.0003409887, step = 4146 (5.413 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11762\n",
"2021-12-31 02:07:31,071 [INFO] tensorflow: global_step/sec: 3.11762\n",
"2021-12-31 02:07:31,072 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.125\n",
"INFO:tensorflow:global_step/sec: 3.13564\n",
"2021-12-31 02:07:33,941 [INFO] tensorflow: global_step/sec: 3.13564\n",
"INFO:tensorflow:epoch = 43.36458333333333, learning_rate = 0.0009999999, loss = 0.0002070026, step = 4163 (5.409 sec)\n",
"2021-12-31 02:07:35,537 [INFO] tensorflow: epoch = 43.36458333333333, learning_rate = 0.0009999999, loss = 0.0002070026, step = 4163 (5.409 sec)\n",
"INFO:tensorflow:global_step/sec: 3.0934\n",
"2021-12-31 02:07:36,851 [INFO] tensorflow: global_step/sec: 3.0934\n",
"2021-12-31 02:07:39,096 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.926\n",
"INFO:tensorflow:global_step/sec: 3.14991\n",
"2021-12-31 02:07:39,708 [INFO] tensorflow: global_step/sec: 3.14991\n",
"INFO:tensorflow:epoch = 43.541666666666664, learning_rate = 0.0009999999, loss = 0.000173652, step = 4180 (5.491 sec)\n",
"2021-12-31 02:07:41,028 [INFO] tensorflow: epoch = 43.541666666666664, learning_rate = 0.0009999999, loss = 0.000173652, step = 4180 (5.491 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07855\n",
"2021-12-31 02:07:42,631 [INFO] tensorflow: global_step/sec: 3.07855\n",
"INFO:tensorflow:global_step/sec: 3.10734\n",
"2021-12-31 02:07:45,528 [INFO] tensorflow: global_step/sec: 3.10734\n",
"INFO:tensorflow:epoch = 43.71875, learning_rate = 0.0009999999, loss = 0.00020460928, step = 4197 (5.425 sec)\n",
"2021-12-31 02:07:46,452 [INFO] tensorflow: epoch = 43.71875, learning_rate = 0.0009999999, loss = 0.00020460928, step = 4197 (5.425 sec)\n",
"2021-12-31 02:07:47,107 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.966\n",
"INFO:tensorflow:global_step/sec: 3.1378\n",
"2021-12-31 02:07:48,396 [INFO] tensorflow: global_step/sec: 3.1378\n",
"INFO:tensorflow:global_step/sec: 3.16415\n",
"2021-12-31 02:07:51,240 [INFO] tensorflow: global_step/sec: 3.16415\n",
"INFO:tensorflow:epoch = 43.89583333333333, learning_rate = 0.0009999999, loss = 0.00031253023, step = 4214 (5.404 sec)\n",
"2021-12-31 02:07:51,857 [INFO] tensorflow: epoch = 43.89583333333333, learning_rate = 0.0009999999, loss = 0.00031253023, step = 4214 (5.404 sec)\n",
"INFO:tensorflow:global_step/sec: 3.17255\n",
"2021-12-31 02:07:54,077 [INFO] tensorflow: global_step/sec: 3.17255\n",
"2021-12-31 02:07:55,033 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 44/120: loss: 0.00024 learning rate: 0.00100 Time taken: 0:00:30.644190 ETA: 0:38:48.958446\n",
"2021-12-31 02:07:55,033 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.234\n",
"INFO:tensorflow:global_step/sec: 3.13868\n",
"2021-12-31 02:07:56,945 [INFO] tensorflow: global_step/sec: 3.13868\n",
"INFO:tensorflow:epoch = 44.072916666666664, learning_rate = 0.0009999999, loss = 0.00028797207, step = 4231 (5.410 sec)\n",
"2021-12-31 02:07:57,266 [INFO] tensorflow: epoch = 44.072916666666664, learning_rate = 0.0009999999, loss = 0.00028797207, step = 4231 (5.410 sec)\n",
"INFO:tensorflow:global_step/sec: 3.13325\n",
"2021-12-31 02:07:59,817 [INFO] tensorflow: global_step/sec: 3.13325\n",
"INFO:tensorflow:epoch = 44.25, learning_rate = 0.0009999999, loss = 0.00022034133, step = 4248 (5.456 sec)\n",
"2021-12-31 02:08:02,723 [INFO] tensorflow: epoch = 44.25, learning_rate = 0.0009999999, loss = 0.00022034133, step = 4248 (5.456 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09679\n",
"2021-12-31 02:08:02,723 [INFO] tensorflow: global_step/sec: 3.09679\n",
"2021-12-31 02:08:03,040 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.979\n",
"INFO:tensorflow:global_step/sec: 3.1366\n",
"2021-12-31 02:08:05,593 [INFO] tensorflow: global_step/sec: 3.1366\n",
"INFO:tensorflow:epoch = 44.42708333333333, learning_rate = 0.0009999999, loss = 0.00021509474, step = 4265 (5.452 sec)\n",
"2021-12-31 02:08:08,175 [INFO] tensorflow: epoch = 44.42708333333333, learning_rate = 0.0009999999, loss = 0.00021509474, step = 4265 (5.452 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10429\n",
"2021-12-31 02:08:08,492 [INFO] tensorflow: global_step/sec: 3.10429\n",
"2021-12-31 02:08:11,038 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.007\n",
"INFO:tensorflow:global_step/sec: 3.147\n",
"2021-12-31 02:08:11,352 [INFO] tensorflow: global_step/sec: 3.147\n",
"INFO:tensorflow:epoch = 44.604166666666664, learning_rate = 0.0009999999, loss = 0.00019166914, step = 4282 (5.398 sec)\n",
"2021-12-31 02:08:13,572 [INFO] tensorflow: epoch = 44.604166666666664, learning_rate = 0.0009999999, loss = 0.00019166914, step = 4282 (5.398 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12261\n",
"2021-12-31 02:08:14,234 [INFO] tensorflow: global_step/sec: 3.12261\n",
"INFO:tensorflow:global_step/sec: 3.11574\n",
"2021-12-31 02:08:17,123 [INFO] tensorflow: global_step/sec: 3.11574\n",
"INFO:tensorflow:epoch = 44.78125, learning_rate = 0.0009999999, loss = 0.00019389593, step = 4299 (5.470 sec)\n",
"2021-12-31 02:08:19,043 [INFO] tensorflow: epoch = 44.78125, learning_rate = 0.0009999999, loss = 0.00019389593, step = 4299 (5.470 sec)\n",
"2021-12-31 02:08:19,043 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.984\n",
"INFO:tensorflow:global_step/sec: 3.10676\n",
"2021-12-31 02:08:20,019 [INFO] tensorflow: global_step/sec: 3.10676\n",
"INFO:tensorflow:global_step/sec: 3.11913\n",
"2021-12-31 02:08:22,905 [INFO] tensorflow: global_step/sec: 3.11913\n",
"INFO:tensorflow:epoch = 44.95833333333333, learning_rate = 0.0009999999, loss = 0.00024158414, step = 4316 (5.464 sec)\n",
"2021-12-31 02:08:24,507 [INFO] tensorflow: epoch = 44.95833333333333, learning_rate = 0.0009999999, loss = 0.00024158414, step = 4316 (5.464 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11237\n",
"2021-12-31 02:08:25,797 [INFO] tensorflow: global_step/sec: 3.11237\n",
"2021-12-31 02:08:25,797 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 45/120: loss: 0.00017 learning rate: 0.00100 Time taken: 0:00:30.741740 ETA: 0:38:25.630517\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"2021-12-31 02:08:27,080 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.885\n",
"INFO:tensorflow:global_step/sec: 3.0988\n",
"2021-12-31 02:08:28,701 [INFO] tensorflow: global_step/sec: 3.0988\n",
"INFO:tensorflow:epoch = 45.135416666666664, learning_rate = 0.0009999999, loss = 0.00021529435, step = 4333 (5.434 sec)\n",
"2021-12-31 02:08:29,941 [INFO] tensorflow: epoch = 45.135416666666664, learning_rate = 0.0009999999, loss = 0.00021529435, step = 4333 (5.434 sec)\n",
"INFO:tensorflow:global_step/sec: 3.17841\n",
"2021-12-31 02:08:31,533 [INFO] tensorflow: global_step/sec: 3.17841\n",
"INFO:tensorflow:global_step/sec: 3.14056\n",
"2021-12-31 02:08:34,398 [INFO] tensorflow: global_step/sec: 3.14056\n",
"2021-12-31 02:08:35,040 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.128\n",
"INFO:tensorflow:epoch = 45.3125, learning_rate = 0.0009999999, loss = 0.00022164083, step = 4350 (5.429 sec)\n",
"2021-12-31 02:08:35,370 [INFO] tensorflow: epoch = 45.3125, learning_rate = 0.0009999999, loss = 0.00022164083, step = 4350 (5.429 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08454\n",
"2021-12-31 02:08:37,316 [INFO] tensorflow: global_step/sec: 3.08454\n",
"INFO:tensorflow:global_step/sec: 3.13178\n",
"2021-12-31 02:08:40,190 [INFO] tensorflow: global_step/sec: 3.13178\n",
"INFO:tensorflow:epoch = 45.48958333333333, learning_rate = 0.0009999999, loss = 0.00019201306, step = 4367 (5.440 sec)\n",
"2021-12-31 02:08:40,809 [INFO] tensorflow: epoch = 45.48958333333333, learning_rate = 0.0009999999, loss = 0.00019201306, step = 4367 (5.440 sec)\n",
"INFO:tensorflow:global_step/sec: 3.14966\n",
"2021-12-31 02:08:43,047 [INFO] tensorflow: global_step/sec: 3.14966\n",
"2021-12-31 02:08:43,048 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.975\n",
"INFO:tensorflow:global_step/sec: 3.13618\n",
"2021-12-31 02:08:45,917 [INFO] tensorflow: global_step/sec: 3.13618\n",
"INFO:tensorflow:epoch = 45.666666666666664, learning_rate = 0.0009999999, loss = 0.00023129774, step = 4384 (5.412 sec)\n",
"2021-12-31 02:08:46,222 [INFO] tensorflow: epoch = 45.666666666666664, learning_rate = 0.0009999999, loss = 0.00023129774, step = 4384 (5.412 sec)\n",
"INFO:tensorflow:global_step/sec: 3.15537\n",
"2021-12-31 02:08:48,769 [INFO] tensorflow: global_step/sec: 3.15537\n",
"2021-12-31 02:08:51,022 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.080\n",
"INFO:tensorflow:epoch = 45.84375, learning_rate = 0.0009999999, loss = 0.00024337496, step = 4401 (5.447 sec)\n",
"2021-12-31 02:08:51,669 [INFO] tensorflow: epoch = 45.84375, learning_rate = 0.0009999999, loss = 0.00024337496, step = 4401 (5.447 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10292\n",
"2021-12-31 02:08:51,670 [INFO] tensorflow: global_step/sec: 3.10292\n",
"INFO:tensorflow:global_step/sec: 3.09262\n",
"2021-12-31 02:08:54,580 [INFO] tensorflow: global_step/sec: 3.09262\n",
"2021-12-31 02:08:56,505 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 46/120: loss: 0.00025 learning rate: 0.00100 Time taken: 0:00:30.713675 ETA: 0:37:52.811934\n",
"INFO:tensorflow:epoch = 46.02083333333333, learning_rate = 0.0009999999, loss = 0.00021427221, step = 4418 (5.478 sec)\n",
"2021-12-31 02:08:57,147 [INFO] tensorflow: epoch = 46.02083333333333, learning_rate = 0.0009999999, loss = 0.00021427221, step = 4418 (5.478 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10226\n",
"2021-12-31 02:08:57,481 [INFO] tensorflow: global_step/sec: 3.10226\n",
"2021-12-31 02:08:59,039 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.950\n",
"INFO:tensorflow:global_step/sec: 3.16259\n",
"2021-12-31 02:09:00,327 [INFO] tensorflow: global_step/sec: 3.16259\n",
"INFO:tensorflow:epoch = 46.197916666666664, learning_rate = 0.0009999999, loss = 0.00027523213, step = 4435 (5.376 sec)\n",
"2021-12-31 02:09:02,523 [INFO] tensorflow: epoch = 46.197916666666664, learning_rate = 0.0009999999, loss = 0.00027523213, step = 4435 (5.376 sec)\n",
"INFO:tensorflow:global_step/sec: 3.18993\n",
"2021-12-31 02:09:03,148 [INFO] tensorflow: global_step/sec: 3.18993\n",
"INFO:tensorflow:global_step/sec: 3.15175\n",
"2021-12-31 02:09:06,004 [INFO] tensorflow: global_step/sec: 3.15175\n",
"2021-12-31 02:09:06,959 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.252\n",
"INFO:tensorflow:epoch = 46.375, learning_rate = 0.0009999999, loss = 0.00021857015, step = 4452 (5.385 sec)\n",
"2021-12-31 02:09:07,908 [INFO] tensorflow: epoch = 46.375, learning_rate = 0.0009999999, loss = 0.00021857015, step = 4452 (5.385 sec)\n",
"INFO:tensorflow:global_step/sec: 3.16066\n",
"2021-12-31 02:09:08,851 [INFO] tensorflow: global_step/sec: 3.16066\n",
"INFO:tensorflow:global_step/sec: 3.13928\n",
"2021-12-31 02:09:11,718 [INFO] tensorflow: global_step/sec: 3.13928\n",
"INFO:tensorflow:epoch = 46.55208333333333, learning_rate = 0.0009999999, loss = 0.00018717645, step = 4469 (5.423 sec)\n",
"2021-12-31 02:09:13,331 [INFO] tensorflow: epoch = 46.55208333333333, learning_rate = 0.0009999999, loss = 0.00018717645, step = 4469 (5.423 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08887\n",
"2021-12-31 02:09:14,632 [INFO] tensorflow: global_step/sec: 3.08887\n",
"2021-12-31 02:09:14,948 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.036\n",
"INFO:tensorflow:global_step/sec: 3.11683\n",
"2021-12-31 02:09:17,519 [INFO] tensorflow: global_step/sec: 3.11683\n",
"INFO:tensorflow:epoch = 46.729166666666664, learning_rate = 0.0009999999, loss = 0.0002726344, step = 4486 (5.483 sec)\n",
"2021-12-31 02:09:18,814 [INFO] tensorflow: epoch = 46.729166666666664, learning_rate = 0.0009999999, loss = 0.0002726344, step = 4486 (5.483 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12777\n",
"2021-12-31 02:09:20,397 [INFO] tensorflow: global_step/sec: 3.12777\n",
"2021-12-31 02:09:22,931 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.054\n",
"INFO:tensorflow:global_step/sec: 3.14611\n",
"2021-12-31 02:09:23,257 [INFO] tensorflow: global_step/sec: 3.14611\n",
"INFO:tensorflow:epoch = 46.90625, learning_rate = 0.0009999999, loss = 0.00026218314, step = 4503 (5.421 sec)\n",
"2021-12-31 02:09:24,235 [INFO] tensorflow: epoch = 46.90625, learning_rate = 0.0009999999, loss = 0.00026218314, step = 4503 (5.421 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09623\n",
"2021-12-31 02:09:26,164 [INFO] tensorflow: global_step/sec: 3.09623\n",
"2021-12-31 02:09:27,124 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 47/120: loss: 0.00025 learning rate: 0.00100 Time taken: 0:00:30.598079 ETA: 0:37:13.659765\n",
"INFO:tensorflow:global_step/sec: 3.13636\n",
"2021-12-31 02:09:29,034 [INFO] tensorflow: global_step/sec: 3.13636\n",
"INFO:tensorflow:epoch = 47.08333333333333, learning_rate = 0.0009999999, loss = 0.00019468815, step = 4520 (5.433 sec)\n",
"2021-12-31 02:09:29,668 [INFO] tensorflow: epoch = 47.08333333333333, learning_rate = 0.0009999999, loss = 0.00019468815, step = 4520 (5.433 sec)\n",
"2021-12-31 02:09:30,952 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.934\n",
"INFO:tensorflow:global_step/sec: 3.11541\n",
"2021-12-31 02:09:31,923 [INFO] tensorflow: global_step/sec: 3.11541\n",
"INFO:tensorflow:global_step/sec: 3.08439\n",
"2021-12-31 02:09:34,841 [INFO] tensorflow: global_step/sec: 3.08439\n",
"INFO:tensorflow:epoch = 47.260416666666664, learning_rate = 0.0009999999, loss = 0.0002455582, step = 4537 (5.482 sec)\n",
"2021-12-31 02:09:35,150 [INFO] tensorflow: epoch = 47.260416666666664, learning_rate = 0.0009999999, loss = 0.0002455582, step = 4537 (5.482 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10217\n",
"2021-12-31 02:09:37,742 [INFO] tensorflow: global_step/sec: 3.10217\n",
"2021-12-31 02:09:39,037 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.739\n",
"INFO:tensorflow:epoch = 47.4375, learning_rate = 0.0009999999, loss = 0.00021689772, step = 4554 (5.486 sec)\n",
"2021-12-31 02:09:40,637 [INFO] tensorflow: epoch = 47.4375, learning_rate = 0.0009999999, loss = 0.00021689772, step = 4554 (5.486 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10819\n",
"2021-12-31 02:09:40,637 [INFO] tensorflow: global_step/sec: 3.10819\n",
"INFO:tensorflow:global_step/sec: 3.11466\n",
"2021-12-31 02:09:43,527 [INFO] tensorflow: global_step/sec: 3.11466\n",
"INFO:tensorflow:epoch = 47.61458333333333, learning_rate = 0.0009999999, loss = 0.00023821487, step = 4571 (5.492 sec)\n",
"2021-12-31 02:09:46,128 [INFO] tensorflow: epoch = 47.61458333333333, learning_rate = 0.0009999999, loss = 0.00023821487, step = 4571 (5.492 sec)\n",
"INFO:tensorflow:global_step/sec: 3.06935\n",
"2021-12-31 02:09:46,459 [INFO] tensorflow: global_step/sec: 3.06935\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"2021-12-31 02:09:47,121 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.742\n",
"INFO:tensorflow:global_step/sec: 3.0619\n",
"2021-12-31 02:09:49,398 [INFO] tensorflow: global_step/sec: 3.0619\n",
"INFO:tensorflow:epoch = 47.791666666666664, learning_rate = 0.0009999999, loss = 0.00028851727, step = 4588 (5.546 sec)\n",
"2021-12-31 02:09:51,674 [INFO] tensorflow: epoch = 47.791666666666664, learning_rate = 0.0009999999, loss = 0.00028851727, step = 4588 (5.546 sec)\n",
"INFO:tensorflow:global_step/sec: 3.0477\n",
"2021-12-31 02:09:52,352 [INFO] tensorflow: global_step/sec: 3.0477\n",
"INFO:tensorflow:global_step/sec: 3.11529\n",
"2021-12-31 02:09:55,241 [INFO] tensorflow: global_step/sec: 3.11529\n",
"2021-12-31 02:09:55,241 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.630\n",
"INFO:tensorflow:epoch = 47.96875, learning_rate = 0.0009999999, loss = 0.00027920722, step = 4605 (5.463 sec)\n",
"2021-12-31 02:09:57,137 [INFO] tensorflow: epoch = 47.96875, learning_rate = 0.0009999999, loss = 0.00027920722, step = 4605 (5.463 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12103\n",
"2021-12-31 02:09:58,124 [INFO] tensorflow: global_step/sec: 3.12103\n",
"2021-12-31 02:09:58,125 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 48/120: loss: 0.00023 learning rate: 0.00100 Time taken: 0:00:31.017924 ETA: 0:37:13.290499\n",
"INFO:tensorflow:global_step/sec: 3.10961\n",
"2021-12-31 02:10:01,018 [INFO] tensorflow: global_step/sec: 3.10961\n",
"INFO:tensorflow:epoch = 48.14583333333333, learning_rate = 0.0009999999, loss = 0.00020665534, step = 4622 (5.515 sec)\n",
"2021-12-31 02:10:02,652 [INFO] tensorflow: epoch = 48.14583333333333, learning_rate = 0.0009999999, loss = 0.00020665534, step = 4622 (5.515 sec)\n",
"2021-12-31 02:10:03,305 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.803\n",
"INFO:tensorflow:global_step/sec: 3.07957\n",
"2021-12-31 02:10:03,941 [INFO] tensorflow: global_step/sec: 3.07957\n",
"INFO:tensorflow:global_step/sec: 3.12005\n",
"2021-12-31 02:10:06,825 [INFO] tensorflow: global_step/sec: 3.12005\n",
"INFO:tensorflow:epoch = 48.322916666666664, learning_rate = 0.0009999999, loss = 0.00025195337, step = 4639 (5.470 sec)\n",
"2021-12-31 02:10:08,122 [INFO] tensorflow: epoch = 48.322916666666664, learning_rate = 0.0009999999, loss = 0.00025195337, step = 4639 (5.470 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08596\n",
"2021-12-31 02:10:09,742 [INFO] tensorflow: global_step/sec: 3.08596\n",
"2021-12-31 02:10:11,412 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.671\n",
"INFO:tensorflow:global_step/sec: 3.06963\n",
"2021-12-31 02:10:12,674 [INFO] tensorflow: global_step/sec: 3.06963\n",
"INFO:tensorflow:epoch = 48.5, learning_rate = 0.0009999999, loss = 0.00020538043, step = 4656 (5.527 sec)\n",
"2021-12-31 02:10:13,649 [INFO] tensorflow: epoch = 48.5, learning_rate = 0.0009999999, loss = 0.00020538043, step = 4656 (5.527 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09938\n",
"2021-12-31 02:10:15,578 [INFO] tensorflow: global_step/sec: 3.09938\n",
"INFO:tensorflow:global_step/sec: 3.08916\n",
"2021-12-31 02:10:18,491 [INFO] tensorflow: global_step/sec: 3.08916\n",
"INFO:tensorflow:epoch = 48.67708333333333, learning_rate = 0.0009999999, loss = 0.00020415505, step = 4673 (5.484 sec)\n",
"2021-12-31 02:10:19,133 [INFO] tensorflow: epoch = 48.67708333333333, learning_rate = 0.0009999999, loss = 0.00020415505, step = 4673 (5.484 sec)\n",
"2021-12-31 02:10:19,469 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.825\n",
"INFO:tensorflow:global_step/sec: 2.99769\n",
"2021-12-31 02:10:21,493 [INFO] tensorflow: global_step/sec: 2.99769\n",
"INFO:tensorflow:global_step/sec: 3.11618\n",
"2021-12-31 02:10:24,382 [INFO] tensorflow: global_step/sec: 3.11618\n",
"INFO:tensorflow:epoch = 48.854166666666664, learning_rate = 0.0009999999, loss = 0.0002558358, step = 4690 (5.566 sec)\n",
"2021-12-31 02:10:24,699 [INFO] tensorflow: epoch = 48.854166666666664, learning_rate = 0.0009999999, loss = 0.0002558358, step = 4690 (5.566 sec)\n",
"INFO:tensorflow:global_step/sec: 3.17413\n",
"2021-12-31 02:10:27,217 [INFO] tensorflow: global_step/sec: 3.17413\n",
"2021-12-31 02:10:27,552 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.743\n",
"2021-12-31 02:10:29,107 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 49/120: loss: 0.00021 learning rate: 0.00100 Time taken: 0:00:30.988764 ETA: 0:36:40.202230\n",
"INFO:tensorflow:epoch = 49.03125, learning_rate = 0.0009999999, loss = 0.0002811077, step = 4707 (5.369 sec)\n",
"2021-12-31 02:10:30,068 [INFO] tensorflow: epoch = 49.03125, learning_rate = 0.0009999999, loss = 0.0002811077, step = 4707 (5.369 sec)\n",
"INFO:tensorflow:global_step/sec: 3.15586\n",
"2021-12-31 02:10:30,069 [INFO] tensorflow: global_step/sec: 3.15586\n",
"INFO:tensorflow:global_step/sec: 3.08105\n",
"2021-12-31 02:10:32,990 [INFO] tensorflow: global_step/sec: 3.08105\n",
"INFO:tensorflow:epoch = 49.20833333333333, learning_rate = 0.0009999999, loss = 0.00021814539, step = 4724 (5.445 sec)\n",
"2021-12-31 02:10:35,513 [INFO] tensorflow: epoch = 49.20833333333333, learning_rate = 0.0009999999, loss = 0.00021814539, step = 4724 (5.445 sec)\n",
"2021-12-31 02:10:35,513 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.121\n",
"INFO:tensorflow:global_step/sec: 3.16115\n",
"2021-12-31 02:10:35,837 [INFO] tensorflow: global_step/sec: 3.16115\n",
"INFO:tensorflow:global_step/sec: 3.11059\n",
"2021-12-31 02:10:38,730 [INFO] tensorflow: global_step/sec: 3.11059\n",
"INFO:tensorflow:epoch = 49.385416666666664, learning_rate = 0.0009999999, loss = 0.0003454871, step = 4741 (5.472 sec)\n",
"2021-12-31 02:10:40,986 [INFO] tensorflow: epoch = 49.385416666666664, learning_rate = 0.0009999999, loss = 0.0003454871, step = 4741 (5.472 sec)\n",
"INFO:tensorflow:global_step/sec: 3.13963\n",
"2021-12-31 02:10:41,597 [INFO] tensorflow: global_step/sec: 3.13963\n",
"2021-12-31 02:10:43,536 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.930\n",
"INFO:tensorflow:global_step/sec: 3.11205\n",
"2021-12-31 02:10:44,489 [INFO] tensorflow: global_step/sec: 3.11205\n",
"INFO:tensorflow:epoch = 49.5625, learning_rate = 0.0009999999, loss = 0.00030548908, step = 4758 (5.441 sec)\n",
"2021-12-31 02:10:46,427 [INFO] tensorflow: epoch = 49.5625, learning_rate = 0.0009999999, loss = 0.00030548908, step = 4758 (5.441 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07504\n",
"2021-12-31 02:10:47,416 [INFO] tensorflow: global_step/sec: 3.07504\n",
"INFO:tensorflow:global_step/sec: 3.09781\n",
"2021-12-31 02:10:50,321 [INFO] tensorflow: global_step/sec: 3.09781\n",
"2021-12-31 02:10:51,605 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.786\n",
"INFO:tensorflow:epoch = 49.73958333333333, learning_rate = 0.0009999999, loss = 0.00019513202, step = 4775 (5.500 sec)\n",
"2021-12-31 02:10:51,927 [INFO] tensorflow: epoch = 49.73958333333333, learning_rate = 0.0009999999, loss = 0.00019513202, step = 4775 (5.500 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12051\n",
"2021-12-31 02:10:53,205 [INFO] tensorflow: global_step/sec: 3.12051\n",
"INFO:tensorflow:global_step/sec: 3.10188\n",
"2021-12-31 02:10:56,107 [INFO] tensorflow: global_step/sec: 3.10188\n",
"INFO:tensorflow:epoch = 49.916666666666664, learning_rate = 0.0009999999, loss = 0.00019703612, step = 4792 (5.446 sec)\n",
"2021-12-31 02:10:57,373 [INFO] tensorflow: epoch = 49.916666666666664, learning_rate = 0.0009999999, loss = 0.00019703612, step = 4792 (5.446 sec)\n",
"INFO:tensorflow:global_step/sec: 3.18392\n",
"2021-12-31 02:10:58,933 [INFO] tensorflow: global_step/sec: 3.18392\n",
"INFO:tensorflow:Saving checkpoints for step-4800.\n",
"2021-12-31 02:10:59,584 [INFO] tensorflow: Saving checkpoints for step-4800.\n",
"WARNING:tensorflow:Ignoring: /tmp/tmpu5j349v_; No such file or directory\n",
"2021-12-31 02:10:59,741 [WARNING] tensorflow: Ignoring: /tmp/tmpu5j349v_; No such file or directory\n",
"2021-12-31 02:11:03,284 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 23, 0.00s/step\n",
"2021-12-31 02:11:06,421 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 23, 0.31s/step\n",
"2021-12-31 02:11:09,639 [INFO] iva.detectnet_v2.evaluation.evaluation: step 20 / 23, 0.32s/step\n",
"Matching predictions to ground truth, class 1/1.: 100%|█| 3106/3106 [00:00<00:00, 15091.74it/s]\n",
"Epoch 50/120\n",
"=========================\n",
"\n",
"Validation cost: 0.000282\n",
"Mean average_precision (in %): 75.0445\n",
"\n",
"class name average precision (in %)\n",
"------------ --------------------------\n",
"car 75.0445\n",
"\n",
"Median Inference Time: 0.016083\n",
"2021-12-31 02:11:10,884 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 10.374\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO:tensorflow:epoch = 50.0, learning_rate = 0.0009999999, loss = 0.0001822824, step = 4800 (13.840 sec)\n",
"2021-12-31 02:11:11,213 [INFO] tensorflow: epoch = 50.0, learning_rate = 0.0009999999, loss = 0.0001822824, step = 4800 (13.840 sec)\n",
"2021-12-31 02:11:11,213 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 50/120: loss: 0.00018 learning rate: 0.00100 Time taken: 0:00:42.084785 ETA: 0:49:05.934932\n",
"INFO:tensorflow:global_step/sec: 0.632538\n",
"2021-12-31 02:11:13,162 [INFO] tensorflow: global_step/sec: 0.632538\n",
"INFO:tensorflow:global_step/sec: 3.12271\n",
"2021-12-31 02:11:16,044 [INFO] tensorflow: global_step/sec: 3.12271\n",
"INFO:tensorflow:epoch = 50.17708333333333, learning_rate = 0.0009999999, loss = 0.00025421774, step = 4817 (5.495 sec)\n",
"2021-12-31 02:11:16,708 [INFO] tensorflow: epoch = 50.17708333333333, learning_rate = 0.0009999999, loss = 0.00025421774, step = 4817 (5.495 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09806\n",
"2021-12-31 02:11:18,949 [INFO] tensorflow: global_step/sec: 3.09806\n",
"2021-12-31 02:11:18,949 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.800\n",
"INFO:tensorflow:global_step/sec: 3.16085\n",
"2021-12-31 02:11:21,796 [INFO] tensorflow: global_step/sec: 3.16085\n",
"INFO:tensorflow:epoch = 50.354166666666664, learning_rate = 0.0009999999, loss = 0.00026267546, step = 4834 (5.416 sec)\n",
"2021-12-31 02:11:22,124 [INFO] tensorflow: epoch = 50.354166666666664, learning_rate = 0.0009999999, loss = 0.00026267546, step = 4834 (5.416 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11504\n",
"2021-12-31 02:11:24,685 [INFO] tensorflow: global_step/sec: 3.11504\n",
"2021-12-31 02:11:26,966 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.949\n",
"INFO:tensorflow:epoch = 50.53125, learning_rate = 0.0009999999, loss = 0.00019830832, step = 4851 (5.496 sec)\n",
"2021-12-31 02:11:27,620 [INFO] tensorflow: epoch = 50.53125, learning_rate = 0.0009999999, loss = 0.00019830832, step = 4851 (5.496 sec)\n",
"INFO:tensorflow:global_step/sec: 3.06601\n",
"2021-12-31 02:11:27,621 [INFO] tensorflow: global_step/sec: 3.06601\n",
"INFO:tensorflow:global_step/sec: 3.17295\n",
"2021-12-31 02:11:30,457 [INFO] tensorflow: global_step/sec: 3.17295\n",
"INFO:tensorflow:epoch = 50.70833333333333, learning_rate = 0.0009999999, loss = 0.00031112588, step = 4868 (5.380 sec)\n",
"2021-12-31 02:11:33,000 [INFO] tensorflow: epoch = 50.70833333333333, learning_rate = 0.0009999999, loss = 0.00031112588, step = 4868 (5.380 sec)\n",
"INFO:tensorflow:global_step/sec: 3.14619\n",
"2021-12-31 02:11:33,318 [INFO] tensorflow: global_step/sec: 3.14619\n",
"2021-12-31 02:11:34,913 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.169\n",
"INFO:tensorflow:global_step/sec: 3.11736\n",
"2021-12-31 02:11:36,205 [INFO] tensorflow: global_step/sec: 3.11736\n",
"INFO:tensorflow:epoch = 50.885416666666664, learning_rate = 0.0009999999, loss = 0.00022085302, step = 4885 (5.482 sec)\n",
"2021-12-31 02:11:38,482 [INFO] tensorflow: epoch = 50.885416666666664, learning_rate = 0.0009999999, loss = 0.00022085302, step = 4885 (5.482 sec)\n",
"INFO:tensorflow:global_step/sec: 3.06744\n",
"2021-12-31 02:11:39,139 [INFO] tensorflow: global_step/sec: 3.06744\n",
"INFO:tensorflow:global_step/sec: 3.14456\n",
"2021-12-31 02:11:42,001 [INFO] tensorflow: global_step/sec: 3.14456\n",
"2021-12-31 02:11:42,002 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 51/120: loss: 0.00022 learning rate: 0.00100 Time taken: 0:00:30.829551 ETA: 0:35:27.239001\n",
"2021-12-31 02:11:42,966 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.835\n",
"INFO:tensorflow:epoch = 51.0625, learning_rate = 0.0009999999, loss = 0.00028381054, step = 4902 (5.462 sec)\n",
"2021-12-31 02:11:43,944 [INFO] tensorflow: epoch = 51.0625, learning_rate = 0.0009999999, loss = 0.00028381054, step = 4902 (5.462 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11839\n",
"2021-12-31 02:11:44,887 [INFO] tensorflow: global_step/sec: 3.11839\n",
"INFO:tensorflow:global_step/sec: 3.15296\n",
"2021-12-31 02:11:47,742 [INFO] tensorflow: global_step/sec: 3.15296\n",
"INFO:tensorflow:epoch = 51.23958333333333, learning_rate = 0.0009999999, loss = 0.00023730812, step = 4919 (5.395 sec)\n",
"2021-12-31 02:11:49,339 [INFO] tensorflow: epoch = 51.23958333333333, learning_rate = 0.0009999999, loss = 0.00023730812, step = 4919 (5.395 sec)\n",
"INFO:tensorflow:global_step/sec: 3.14008\n",
"2021-12-31 02:11:50,608 [INFO] tensorflow: global_step/sec: 3.14008\n",
"2021-12-31 02:11:50,927 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.123\n",
"INFO:tensorflow:global_step/sec: 3.14635\n",
"2021-12-31 02:11:53,468 [INFO] tensorflow: global_step/sec: 3.14635\n",
"INFO:tensorflow:epoch = 51.416666666666664, learning_rate = 0.0009999999, loss = 0.00014424529, step = 4936 (5.417 sec)\n",
"2021-12-31 02:11:54,756 [INFO] tensorflow: epoch = 51.416666666666664, learning_rate = 0.0009999999, loss = 0.00014424529, step = 4936 (5.417 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12961\n",
"2021-12-31 02:11:56,344 [INFO] tensorflow: global_step/sec: 3.12961\n",
"2021-12-31 02:11:58,905 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.071\n",
"INFO:tensorflow:global_step/sec: 3.13876\n",
"2021-12-31 02:11:59,211 [INFO] tensorflow: global_step/sec: 3.13876\n",
"INFO:tensorflow:epoch = 51.59375, learning_rate = 0.0009999999, loss = 0.0001981686, step = 4953 (5.417 sec)\n",
"2021-12-31 02:12:00,173 [INFO] tensorflow: epoch = 51.59375, learning_rate = 0.0009999999, loss = 0.0001981686, step = 4953 (5.417 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12204\n",
"2021-12-31 02:12:02,094 [INFO] tensorflow: global_step/sec: 3.12204\n",
"INFO:tensorflow:global_step/sec: 3.17097\n",
"2021-12-31 02:12:04,932 [INFO] tensorflow: global_step/sec: 3.17097\n",
"INFO:tensorflow:epoch = 51.77083333333333, learning_rate = 0.0009999999, loss = 0.00017278935, step = 4970 (5.415 sec)\n",
"2021-12-31 02:12:05,589 [INFO] tensorflow: epoch = 51.77083333333333, learning_rate = 0.0009999999, loss = 0.00017278935, step = 4970 (5.415 sec)\n",
"2021-12-31 02:12:06,884 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.065\n",
"INFO:tensorflow:global_step/sec: 3.09018\n",
"2021-12-31 02:12:07,845 [INFO] tensorflow: global_step/sec: 3.09018\n",
"INFO:tensorflow:global_step/sec: 3.17004\n",
"2021-12-31 02:12:10,684 [INFO] tensorflow: global_step/sec: 3.17004\n",
"INFO:tensorflow:epoch = 51.947916666666664, learning_rate = 0.0009999999, loss = 0.00019911549, step = 4987 (5.415 sec)\n",
"2021-12-31 02:12:11,004 [INFO] tensorflow: epoch = 51.947916666666664, learning_rate = 0.0009999999, loss = 0.00019911549, step = 4987 (5.415 sec)\n",
"2021-12-31 02:12:12,573 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 52/120: loss: 0.00021 learning rate: 0.00100 Time taken: 0:00:30.574318 ETA: 0:34:39.053603\n",
"INFO:tensorflow:global_step/sec: 3.14817\n",
"2021-12-31 02:12:13,543 [INFO] tensorflow: global_step/sec: 3.14817\n",
"2021-12-31 02:12:14,822 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.195\n",
"INFO:tensorflow:epoch = 52.125, learning_rate = 0.0009999999, loss = 0.00027346818, step = 5004 (5.426 sec)\n",
"2021-12-31 02:12:16,430 [INFO] tensorflow: epoch = 52.125, learning_rate = 0.0009999999, loss = 0.00027346818, step = 5004 (5.426 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11625\n",
"2021-12-31 02:12:16,431 [INFO] tensorflow: global_step/sec: 3.11625\n",
"INFO:tensorflow:global_step/sec: 3.13833\n",
"2021-12-31 02:12:19,299 [INFO] tensorflow: global_step/sec: 3.13833\n",
"INFO:tensorflow:epoch = 52.30208333333333, learning_rate = 0.0009999999, loss = 0.0002116377, step = 5021 (5.427 sec)\n",
"2021-12-31 02:12:21,857 [INFO] tensorflow: epoch = 52.30208333333333, learning_rate = 0.0009999999, loss = 0.0002116377, step = 5021 (5.427 sec)\n",
"INFO:tensorflow:global_step/sec: 3.16129\n",
"2021-12-31 02:12:22,145 [INFO] tensorflow: global_step/sec: 3.16129\n",
"2021-12-31 02:12:22,811 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.038\n",
"INFO:tensorflow:global_step/sec: 3.08766\n",
"2021-12-31 02:12:25,060 [INFO] tensorflow: global_step/sec: 3.08766\n",
"INFO:tensorflow:epoch = 52.479166666666664, learning_rate = 0.0009999999, loss = 0.00017864873, step = 5038 (5.474 sec)\n",
"2021-12-31 02:12:27,331 [INFO] tensorflow: epoch = 52.479166666666664, learning_rate = 0.0009999999, loss = 0.00017864873, step = 5038 (5.474 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10536\n",
"2021-12-31 02:12:27,958 [INFO] tensorflow: global_step/sec: 3.10536\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO:tensorflow:global_step/sec: 3.09067\n",
"2021-12-31 02:12:30,870 [INFO] tensorflow: global_step/sec: 3.09067\n",
"2021-12-31 02:12:30,871 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.813\n",
"INFO:tensorflow:epoch = 52.65625, learning_rate = 0.0009999999, loss = 0.0002503669, step = 5055 (5.505 sec)\n",
"2021-12-31 02:12:32,835 [INFO] tensorflow: epoch = 52.65625, learning_rate = 0.0009999999, loss = 0.0002503669, step = 5055 (5.505 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07037\n",
"2021-12-31 02:12:33,802 [INFO] tensorflow: global_step/sec: 3.07037\n",
"INFO:tensorflow:global_step/sec: 3.09705\n",
"2021-12-31 02:12:36,708 [INFO] tensorflow: global_step/sec: 3.09705\n",
"INFO:tensorflow:epoch = 52.83333333333333, learning_rate = 0.0009999999, loss = 0.00019338886, step = 5072 (5.497 sec)\n",
"2021-12-31 02:12:38,332 [INFO] tensorflow: epoch = 52.83333333333333, learning_rate = 0.0009999999, loss = 0.00019338886, step = 5072 (5.497 sec)\n",
"2021-12-31 02:12:38,932 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.813\n",
"INFO:tensorflow:global_step/sec: 3.15586\n",
"2021-12-31 02:12:39,560 [INFO] tensorflow: global_step/sec: 3.15586\n",
"INFO:tensorflow:global_step/sec: 3.1298\n",
"2021-12-31 02:12:42,435 [INFO] tensorflow: global_step/sec: 3.1298\n",
"2021-12-31 02:12:43,432 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 53/120: loss: 0.00029 learning rate: 0.00100 Time taken: 0:00:30.809845 ETA: 0:34:24.259645\n",
"INFO:tensorflow:epoch = 53.010416666666664, learning_rate = 0.0009999999, loss = 0.00026804704, step = 5089 (5.416 sec)\n",
"2021-12-31 02:12:43,748 [INFO] tensorflow: epoch = 53.010416666666664, learning_rate = 0.0009999999, loss = 0.00026804704, step = 5089 (5.416 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11938\n",
"2021-12-31 02:12:45,320 [INFO] tensorflow: global_step/sec: 3.11938\n",
"2021-12-31 02:12:46,958 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.919\n",
"INFO:tensorflow:global_step/sec: 3.09962\n",
"2021-12-31 02:12:48,224 [INFO] tensorflow: global_step/sec: 3.09962\n",
"INFO:tensorflow:epoch = 53.1875, learning_rate = 0.0009999999, loss = 0.0002400311, step = 5106 (5.425 sec)\n",
"2021-12-31 02:12:49,173 [INFO] tensorflow: epoch = 53.1875, learning_rate = 0.0009999999, loss = 0.0002400311, step = 5106 (5.425 sec)\n",
"INFO:tensorflow:global_step/sec: 3.15605\n",
"2021-12-31 02:12:51,076 [INFO] tensorflow: global_step/sec: 3.15605\n",
"INFO:tensorflow:global_step/sec: 3.10906\n",
"2021-12-31 02:12:53,970 [INFO] tensorflow: global_step/sec: 3.10906\n",
"INFO:tensorflow:epoch = 53.36458333333333, learning_rate = 0.0009999999, loss = 0.00023222376, step = 5123 (5.446 sec)\n",
"2021-12-31 02:12:54,619 [INFO] tensorflow: epoch = 53.36458333333333, learning_rate = 0.0009999999, loss = 0.00023222376, step = 5123 (5.446 sec)\n",
"2021-12-31 02:12:54,937 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.066\n",
"INFO:tensorflow:global_step/sec: 3.15133\n",
"2021-12-31 02:12:56,826 [INFO] tensorflow: global_step/sec: 3.15133\n",
"INFO:tensorflow:global_step/sec: 3.06932\n",
"2021-12-31 02:12:59,759 [INFO] tensorflow: global_step/sec: 3.06932\n",
"INFO:tensorflow:epoch = 53.541666666666664, learning_rate = 0.0009999999, loss = 0.00018954088, step = 5140 (5.469 sec)\n",
"2021-12-31 02:13:00,088 [INFO] tensorflow: epoch = 53.541666666666664, learning_rate = 0.0009999999, loss = 0.00018954088, step = 5140 (5.469 sec)\n",
"INFO:tensorflow:global_step/sec: 3.13145\n",
"2021-12-31 02:13:02,633 [INFO] tensorflow: global_step/sec: 3.13145\n",
"2021-12-31 02:13:02,964 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.919\n",
"INFO:tensorflow:epoch = 53.71875, learning_rate = 0.0009999999, loss = 0.00023142432, step = 5157 (5.351 sec)\n",
"2021-12-31 02:13:05,439 [INFO] tensorflow: epoch = 53.71875, learning_rate = 0.0009999999, loss = 0.00023142432, step = 5157 (5.351 sec)\n",
"INFO:tensorflow:global_step/sec: 3.20568\n",
"2021-12-31 02:13:05,440 [INFO] tensorflow: global_step/sec: 3.20568\n",
"INFO:tensorflow:global_step/sec: 3.09566\n",
"2021-12-31 02:13:08,347 [INFO] tensorflow: global_step/sec: 3.09566\n",
"INFO:tensorflow:epoch = 53.89583333333333, learning_rate = 0.0009999999, loss = 0.0001852567, step = 5174 (5.490 sec)\n",
"2021-12-31 02:13:10,929 [INFO] tensorflow: epoch = 53.89583333333333, learning_rate = 0.0009999999, loss = 0.0001852567, step = 5174 (5.490 sec)\n",
"2021-12-31 02:13:10,930 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.107\n",
"INFO:tensorflow:global_step/sec: 3.09326\n",
"2021-12-31 02:13:11,257 [INFO] tensorflow: global_step/sec: 3.09326\n",
"INFO:tensorflow:global_step/sec: 3.11349\n",
"2021-12-31 02:13:14,148 [INFO] tensorflow: global_step/sec: 3.11349\n",
"2021-12-31 02:13:14,148 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 54/120: loss: 0.00027 learning rate: 0.00100 Time taken: 0:00:30.734261 ETA: 0:33:48.461197\n",
"INFO:tensorflow:epoch = 54.072916666666664, learning_rate = 0.0009999999, loss = 0.00028139967, step = 5191 (5.460 sec)\n",
"2021-12-31 02:13:16,389 [INFO] tensorflow: epoch = 54.072916666666664, learning_rate = 0.0009999999, loss = 0.00028139967, step = 5191 (5.460 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12035\n",
"2021-12-31 02:13:17,032 [INFO] tensorflow: global_step/sec: 3.12035\n",
"2021-12-31 02:13:18,952 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.930\n",
"INFO:tensorflow:global_step/sec: 3.1369\n",
"2021-12-31 02:13:19,901 [INFO] tensorflow: global_step/sec: 3.1369\n",
"INFO:tensorflow:epoch = 54.25, learning_rate = 0.0009999999, loss = 0.0001926194, step = 5208 (5.414 sec)\n",
"2021-12-31 02:13:21,803 [INFO] tensorflow: epoch = 54.25, learning_rate = 0.0009999999, loss = 0.0001926194, step = 5208 (5.414 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12143\n",
"2021-12-31 02:13:22,784 [INFO] tensorflow: global_step/sec: 3.12143\n",
"INFO:tensorflow:global_step/sec: 3.12243\n",
"2021-12-31 02:13:25,667 [INFO] tensorflow: global_step/sec: 3.12243\n",
"2021-12-31 02:13:26,959 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.978\n",
"INFO:tensorflow:epoch = 54.42708333333333, learning_rate = 0.0009999999, loss = 0.00020052175, step = 5225 (5.472 sec)\n",
"2021-12-31 02:13:27,275 [INFO] tensorflow: epoch = 54.42708333333333, learning_rate = 0.0009999999, loss = 0.00020052175, step = 5225 (5.472 sec)\n",
"INFO:tensorflow:global_step/sec: 3.13882\n",
"2021-12-31 02:13:28,534 [INFO] tensorflow: global_step/sec: 3.13882\n",
"INFO:tensorflow:global_step/sec: 3.14909\n",
"2021-12-31 02:13:31,392 [INFO] tensorflow: global_step/sec: 3.14909\n",
"INFO:tensorflow:epoch = 54.604166666666664, learning_rate = 0.0009999999, loss = 0.00022143548, step = 5242 (5.418 sec)\n",
"2021-12-31 02:13:32,693 [INFO] tensorflow: epoch = 54.604166666666664, learning_rate = 0.0009999999, loss = 0.00022143548, step = 5242 (5.418 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11719\n",
"2021-12-31 02:13:34,279 [INFO] tensorflow: global_step/sec: 3.11719\n",
"2021-12-31 02:13:34,894 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.206\n",
"INFO:tensorflow:global_step/sec: 3.13181\n",
"2021-12-31 02:13:37,153 [INFO] tensorflow: global_step/sec: 3.13181\n",
"INFO:tensorflow:epoch = 54.78125, learning_rate = 0.0009999999, loss = 0.00020002536, step = 5259 (5.467 sec)\n",
"2021-12-31 02:13:38,160 [INFO] tensorflow: epoch = 54.78125, learning_rate = 0.0009999999, loss = 0.00020002536, step = 5259 (5.467 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07676\n",
"2021-12-31 02:13:40,078 [INFO] tensorflow: global_step/sec: 3.07676\n",
"INFO:tensorflow:global_step/sec: 3.11962\n",
"2021-12-31 02:13:42,963 [INFO] tensorflow: global_step/sec: 3.11962\n",
"2021-12-31 02:13:42,964 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.785\n",
"INFO:tensorflow:epoch = 54.95833333333333, learning_rate = 0.0009999999, loss = 0.00018826623, step = 5276 (5.447 sec)\n",
"2021-12-31 02:13:43,607 [INFO] tensorflow: epoch = 54.95833333333333, learning_rate = 0.0009999999, loss = 0.00018826623, step = 5276 (5.447 sec)\n",
"2021-12-31 02:13:44,929 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 55/120: loss: 0.00019 learning rate: 0.00100 Time taken: 0:00:30.752230 ETA: 0:33:18.894961\n",
"INFO:tensorflow:global_step/sec: 3.0925\n",
"2021-12-31 02:13:45,873 [INFO] tensorflow: global_step/sec: 3.0925\n",
"INFO:tensorflow:global_step/sec: 3.10218\n",
"2021-12-31 02:13:48,774 [INFO] tensorflow: global_step/sec: 3.10218\n",
"INFO:tensorflow:epoch = 55.135416666666664, learning_rate = 0.0009999999, loss = 0.0002561566, step = 5293 (5.489 sec)\n",
"2021-12-31 02:13:49,096 [INFO] tensorflow: epoch = 55.135416666666664, learning_rate = 0.0009999999, loss = 0.0002561566, step = 5293 (5.489 sec)\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"2021-12-31 02:13:51,003 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.880\n",
"INFO:tensorflow:global_step/sec: 3.13382\n",
"2021-12-31 02:13:51,646 [INFO] tensorflow: global_step/sec: 3.13382\n",
"INFO:tensorflow:epoch = 55.3125, learning_rate = 0.0009999999, loss = 0.00022903821, step = 5310 (5.405 sec)\n",
"2021-12-31 02:13:54,501 [INFO] tensorflow: epoch = 55.3125, learning_rate = 0.0009999999, loss = 0.00022903821, step = 5310 (5.405 sec)\n",
"INFO:tensorflow:global_step/sec: 3.15213\n",
"2021-12-31 02:13:54,502 [INFO] tensorflow: global_step/sec: 3.15213\n",
"INFO:tensorflow:global_step/sec: 3.12664\n",
"2021-12-31 02:13:57,380 [INFO] tensorflow: global_step/sec: 3.12664\n",
"2021-12-31 02:13:58,915 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.279\n",
"INFO:tensorflow:epoch = 55.48958333333333, learning_rate = 0.0009999999, loss = 0.0003117303, step = 5327 (5.370 sec)\n",
"2021-12-31 02:13:59,871 [INFO] tensorflow: epoch = 55.48958333333333, learning_rate = 0.0009999999, loss = 0.0003117303, step = 5327 (5.370 sec)\n",
"INFO:tensorflow:global_step/sec: 3.20505\n",
"2021-12-31 02:14:00,188 [INFO] tensorflow: global_step/sec: 3.20505\n",
"INFO:tensorflow:global_step/sec: 3.11667\n",
"2021-12-31 02:14:03,076 [INFO] tensorflow: global_step/sec: 3.11667\n",
"INFO:tensorflow:epoch = 55.666666666666664, learning_rate = 0.0009999999, loss = 0.00021246205, step = 5344 (5.502 sec)\n",
"2021-12-31 02:14:05,373 [INFO] tensorflow: epoch = 55.666666666666664, learning_rate = 0.0009999999, loss = 0.00021246205, step = 5344 (5.502 sec)\n",
"INFO:tensorflow:global_step/sec: 3.06871\n",
"2021-12-31 02:14:06,009 [INFO] tensorflow: global_step/sec: 3.06871\n",
"2021-12-31 02:14:06,964 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.847\n",
"INFO:tensorflow:global_step/sec: 3.15768\n",
"2021-12-31 02:14:08,859 [INFO] tensorflow: global_step/sec: 3.15768\n",
"INFO:tensorflow:epoch = 55.84375, learning_rate = 0.0009999999, loss = 0.00019778498, step = 5361 (5.416 sec)\n",
"2021-12-31 02:14:10,789 [INFO] tensorflow: epoch = 55.84375, learning_rate = 0.0009999999, loss = 0.00019778498, step = 5361 (5.416 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11438\n",
"2021-12-31 02:14:11,749 [INFO] tensorflow: global_step/sec: 3.11438\n",
"INFO:tensorflow:global_step/sec: 3.08607\n",
"2021-12-31 02:14:14,665 [INFO] tensorflow: global_step/sec: 3.08607\n",
"2021-12-31 02:14:14,985 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.935\n",
"2021-12-31 02:14:15,643 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 56/120: loss: 0.00023 learning rate: 0.00100 Time taken: 0:00:30.737879 ETA: 0:32:47.224243\n",
"INFO:tensorflow:epoch = 56.02083333333333, learning_rate = 0.0009999999, loss = 0.00023168212, step = 5378 (5.473 sec)\n",
"2021-12-31 02:14:16,261 [INFO] tensorflow: epoch = 56.02083333333333, learning_rate = 0.0009999999, loss = 0.00023168212, step = 5378 (5.473 sec)\n",
"INFO:tensorflow:global_step/sec: 3.13528\n",
"2021-12-31 02:14:17,536 [INFO] tensorflow: global_step/sec: 3.13528\n",
"INFO:tensorflow:global_step/sec: 3.0842\n",
"2021-12-31 02:14:20,454 [INFO] tensorflow: global_step/sec: 3.0842\n",
"INFO:tensorflow:epoch = 56.197916666666664, learning_rate = 0.0009999999, loss = 0.00018288865, step = 5395 (5.518 sec)\n",
"2021-12-31 02:14:21,779 [INFO] tensorflow: epoch = 56.197916666666664, learning_rate = 0.0009999999, loss = 0.00018288865, step = 5395 (5.518 sec)\n",
"2021-12-31 02:14:23,071 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.736\n",
"INFO:tensorflow:global_step/sec: 3.07625\n",
"2021-12-31 02:14:23,379 [INFO] tensorflow: global_step/sec: 3.07625\n",
"INFO:tensorflow:global_step/sec: 3.10173\n",
"2021-12-31 02:14:26,281 [INFO] tensorflow: global_step/sec: 3.10173\n",
"INFO:tensorflow:epoch = 56.375, learning_rate = 0.0009999999, loss = 0.00029991806, step = 5412 (5.460 sec)\n",
"2021-12-31 02:14:27,240 [INFO] tensorflow: epoch = 56.375, learning_rate = 0.0009999999, loss = 0.00029991806, step = 5412 (5.460 sec)\n",
"INFO:tensorflow:global_step/sec: 3.0822\n",
"2021-12-31 02:14:29,201 [INFO] tensorflow: global_step/sec: 3.0822\n",
"2021-12-31 02:14:31,136 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.800\n",
"INFO:tensorflow:global_step/sec: 3.12126\n",
"2021-12-31 02:14:32,084 [INFO] tensorflow: global_step/sec: 3.12126\n",
"INFO:tensorflow:epoch = 56.55208333333333, learning_rate = 0.0009999999, loss = 0.00020537036, step = 5429 (5.495 sec)\n",
"2021-12-31 02:14:32,734 [INFO] tensorflow: epoch = 56.55208333333333, learning_rate = 0.0009999999, loss = 0.00020537036, step = 5429 (5.495 sec)\n",
"INFO:tensorflow:global_step/sec: 3.02086\n",
"2021-12-31 02:14:35,064 [INFO] tensorflow: global_step/sec: 3.02086\n",
"INFO:tensorflow:global_step/sec: 3.02967\n",
"2021-12-31 02:14:38,034 [INFO] tensorflow: global_step/sec: 3.02967\n",
"INFO:tensorflow:epoch = 56.729166666666664, learning_rate = 0.0009999999, loss = 0.00025250454, step = 5446 (5.591 sec)\n",
"2021-12-31 02:14:38,325 [INFO] tensorflow: epoch = 56.729166666666664, learning_rate = 0.0009999999, loss = 0.00025250454, step = 5446 (5.591 sec)\n",
"2021-12-31 02:14:39,270 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.586\n",
"INFO:tensorflow:global_step/sec: 3.17206\n",
"2021-12-31 02:14:40,872 [INFO] tensorflow: global_step/sec: 3.17206\n",
"INFO:tensorflow:epoch = 56.90625, learning_rate = 0.0009999999, loss = 0.00024039985, step = 5463 (5.423 sec)\n",
"2021-12-31 02:14:43,748 [INFO] tensorflow: epoch = 56.90625, learning_rate = 0.0009999999, loss = 0.00024039985, step = 5463 (5.423 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12783\n",
"2021-12-31 02:14:43,749 [INFO] tensorflow: global_step/sec: 3.12783\n",
"INFO:tensorflow:global_step/sec: 3.09849\n",
"2021-12-31 02:14:46,654 [INFO] tensorflow: global_step/sec: 3.09849\n",
"2021-12-31 02:14:46,654 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 57/120: loss: 0.00019 learning rate: 0.00100 Time taken: 0:00:30.988193 ETA: 0:32:32.256131\n",
"2021-12-31 02:14:47,294 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.927\n",
"INFO:tensorflow:epoch = 57.08333333333333, learning_rate = 0.0009999999, loss = 0.00019845899, step = 5480 (5.482 sec)\n",
"2021-12-31 02:14:49,230 [INFO] tensorflow: epoch = 57.08333333333333, learning_rate = 0.0009999999, loss = 0.00019845899, step = 5480 (5.482 sec)\n",
"INFO:tensorflow:global_step/sec: 3.1251\n",
"2021-12-31 02:14:49,534 [INFO] tensorflow: global_step/sec: 3.1251\n",
"INFO:tensorflow:global_step/sec: 3.10843\n",
"2021-12-31 02:14:52,429 [INFO] tensorflow: global_step/sec: 3.10843\n",
"INFO:tensorflow:epoch = 57.260416666666664, learning_rate = 0.0009999999, loss = 0.00021955262, step = 5497 (5.501 sec)\n",
"2021-12-31 02:14:54,731 [INFO] tensorflow: epoch = 57.260416666666664, learning_rate = 0.0009999999, loss = 0.00021955262, step = 5497 (5.501 sec)\n",
"INFO:tensorflow:global_step/sec: 3.0578\n",
"2021-12-31 02:14:55,372 [INFO] tensorflow: global_step/sec: 3.0578\n",
"2021-12-31 02:14:55,373 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.757\n",
"INFO:tensorflow:global_step/sec: 3.11423\n",
"2021-12-31 02:14:58,262 [INFO] tensorflow: global_step/sec: 3.11423\n",
"INFO:tensorflow:epoch = 57.4375, learning_rate = 0.0009999999, loss = 0.00021705787, step = 5514 (5.475 sec)\n",
"2021-12-31 02:15:00,206 [INFO] tensorflow: epoch = 57.4375, learning_rate = 0.0009999999, loss = 0.00021705787, step = 5514 (5.475 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10544\n",
"2021-12-31 02:15:01,160 [INFO] tensorflow: global_step/sec: 3.10544\n",
"2021-12-31 02:15:03,400 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.917\n",
"INFO:tensorflow:global_step/sec: 3.13314\n",
"2021-12-31 02:15:04,033 [INFO] tensorflow: global_step/sec: 3.13314\n",
"INFO:tensorflow:epoch = 57.61458333333333, learning_rate = 0.0009999999, loss = 0.00021951186, step = 5531 (5.422 sec)\n",
"2021-12-31 02:15:05,628 [INFO] tensorflow: epoch = 57.61458333333333, learning_rate = 0.0009999999, loss = 0.00021951186, step = 5531 (5.422 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09716\n",
"2021-12-31 02:15:06,939 [INFO] tensorflow: global_step/sec: 3.09716\n",
"INFO:tensorflow:global_step/sec: 3.11089\n",
"2021-12-31 02:15:09,832 [INFO] tensorflow: global_step/sec: 3.11089\n",
"INFO:tensorflow:epoch = 57.791666666666664, learning_rate = 0.0009999999, loss = 0.00025929208, step = 5548 (5.480 sec)\n",
"2021-12-31 02:15:11,108 [INFO] tensorflow: epoch = 57.791666666666664, learning_rate = 0.0009999999, loss = 0.00025929208, step = 5548 (5.480 sec)\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"2021-12-31 02:15:11,431 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.902\n",
"INFO:tensorflow:global_step/sec: 3.10838\n",
"2021-12-31 02:15:12,727 [INFO] tensorflow: global_step/sec: 3.10838\n",
"INFO:tensorflow:global_step/sec: 3.09767\n",
"2021-12-31 02:15:15,633 [INFO] tensorflow: global_step/sec: 3.09767\n",
"INFO:tensorflow:epoch = 57.96875, learning_rate = 0.0009999999, loss = 0.00016327405, step = 5565 (5.467 sec)\n",
"2021-12-31 02:15:16,576 [INFO] tensorflow: epoch = 57.96875, learning_rate = 0.0009999999, loss = 0.00016327405, step = 5565 (5.467 sec)\n",
"2021-12-31 02:15:17,548 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 58/120: loss: 0.00028 learning rate: 0.00100 Time taken: 0:00:30.917719 ETA: 0:31:56.898571\n",
"INFO:tensorflow:global_step/sec: 3.15148\n",
"2021-12-31 02:15:18,488 [INFO] tensorflow: global_step/sec: 3.15148\n",
"2021-12-31 02:15:19,449 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.944\n",
"INFO:tensorflow:global_step/sec: 3.10904\n",
"2021-12-31 02:15:21,383 [INFO] tensorflow: global_step/sec: 3.10904\n",
"INFO:tensorflow:epoch = 58.14583333333333, learning_rate = 0.0009999999, loss = 0.00024927495, step = 5582 (5.455 sec)\n",
"2021-12-31 02:15:22,030 [INFO] tensorflow: epoch = 58.14583333333333, learning_rate = 0.0009999999, loss = 0.00024927495, step = 5582 (5.455 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12048\n",
"2021-12-31 02:15:24,267 [INFO] tensorflow: global_step/sec: 3.12048\n",
"INFO:tensorflow:global_step/sec: 3.05252\n",
"2021-12-31 02:15:27,216 [INFO] tensorflow: global_step/sec: 3.05252\n",
"INFO:tensorflow:epoch = 58.322916666666664, learning_rate = 0.0009999999, loss = 0.00029022855, step = 5599 (5.506 sec)\n",
"2021-12-31 02:15:27,536 [INFO] tensorflow: epoch = 58.322916666666664, learning_rate = 0.0009999999, loss = 0.00029022855, step = 5599 (5.506 sec)\n",
"2021-12-31 02:15:27,536 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.733\n",
"INFO:tensorflow:global_step/sec: 3.09497\n",
"2021-12-31 02:15:30,124 [INFO] tensorflow: global_step/sec: 3.09497\n",
"INFO:tensorflow:epoch = 58.5, learning_rate = 0.0009999999, loss = 0.00019651293, step = 5616 (5.471 sec)\n",
"2021-12-31 02:15:33,007 [INFO] tensorflow: epoch = 58.5, learning_rate = 0.0009999999, loss = 0.00019651293, step = 5616 (5.471 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12019\n",
"2021-12-31 02:15:33,008 [INFO] tensorflow: global_step/sec: 3.12019\n",
"2021-12-31 02:15:35,541 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.986\n",
"INFO:tensorflow:global_step/sec: 3.14971\n",
"2021-12-31 02:15:35,865 [INFO] tensorflow: global_step/sec: 3.14971\n",
"INFO:tensorflow:epoch = 58.67708333333333, learning_rate = 0.0009999999, loss = 0.00018958133, step = 5633 (5.443 sec)\n",
"2021-12-31 02:15:38,450 [INFO] tensorflow: epoch = 58.67708333333333, learning_rate = 0.0009999999, loss = 0.00018958133, step = 5633 (5.443 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08862\n",
"2021-12-31 02:15:38,779 [INFO] tensorflow: global_step/sec: 3.08862\n",
"INFO:tensorflow:global_step/sec: 3.15838\n",
"2021-12-31 02:15:41,629 [INFO] tensorflow: global_step/sec: 3.15838\n",
"2021-12-31 02:15:43,546 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.984\n",
"INFO:tensorflow:epoch = 58.854166666666664, learning_rate = 0.0009999999, loss = 0.00022863016, step = 5650 (5.418 sec)\n",
"2021-12-31 02:15:43,869 [INFO] tensorflow: epoch = 58.854166666666664, learning_rate = 0.0009999999, loss = 0.00022863016, step = 5650 (5.418 sec)\n",
"INFO:tensorflow:global_step/sec: 3.13126\n",
"2021-12-31 02:15:44,503 [INFO] tensorflow: global_step/sec: 3.13126\n",
"INFO:tensorflow:global_step/sec: 3.12265\n",
"2021-12-31 02:15:47,385 [INFO] tensorflow: global_step/sec: 3.12265\n",
"2021-12-31 02:15:48,346 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 59/120: loss: 0.00024 learning rate: 0.00100 Time taken: 0:00:30.782352 ETA: 0:31:17.723499\n",
"INFO:tensorflow:epoch = 59.03125, learning_rate = 0.0009999999, loss = 0.00020756725, step = 5667 (5.442 sec)\n",
"2021-12-31 02:15:49,310 [INFO] tensorflow: epoch = 59.03125, learning_rate = 0.0009999999, loss = 0.00020756725, step = 5667 (5.442 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09062\n",
"2021-12-31 02:15:50,297 [INFO] tensorflow: global_step/sec: 3.09062\n",
"2021-12-31 02:15:51,593 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.855\n",
"INFO:tensorflow:global_step/sec: 3.15237\n",
"2021-12-31 02:15:53,152 [INFO] tensorflow: global_step/sec: 3.15237\n",
"INFO:tensorflow:epoch = 59.20833333333333, learning_rate = 0.0009999999, loss = 0.000201433, step = 5684 (5.461 sec)\n",
"2021-12-31 02:15:54,771 [INFO] tensorflow: epoch = 59.20833333333333, learning_rate = 0.0009999999, loss = 0.000201433, step = 5684 (5.461 sec)\n",
"INFO:tensorflow:global_step/sec: 3.101\n",
"2021-12-31 02:15:56,055 [INFO] tensorflow: global_step/sec: 3.101\n",
"INFO:tensorflow:global_step/sec: 3.11417\n",
"2021-12-31 02:15:58,945 [INFO] tensorflow: global_step/sec: 3.11417\n",
"2021-12-31 02:15:59,633 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.876\n",
"INFO:tensorflow:epoch = 59.385416666666664, learning_rate = 0.0009999999, loss = 0.00023779727, step = 5701 (5.514 sec)\n",
"2021-12-31 02:16:00,286 [INFO] tensorflow: epoch = 59.385416666666664, learning_rate = 0.0009999999, loss = 0.00023779727, step = 5701 (5.514 sec)\n",
"INFO:tensorflow:global_step/sec: 3.04218\n",
"2021-12-31 02:16:01,903 [INFO] tensorflow: global_step/sec: 3.04218\n",
"INFO:tensorflow:global_step/sec: 3.11701\n",
"2021-12-31 02:16:04,790 [INFO] tensorflow: global_step/sec: 3.11701\n",
"INFO:tensorflow:epoch = 59.5625, learning_rate = 0.0009999999, loss = 0.00018094908, step = 5718 (5.418 sec)\n",
"2021-12-31 02:16:05,704 [INFO] tensorflow: epoch = 59.5625, learning_rate = 0.0009999999, loss = 0.00018094908, step = 5718 (5.418 sec)\n",
"INFO:tensorflow:global_step/sec: 3.16801\n",
"2021-12-31 02:16:07,631 [INFO] tensorflow: global_step/sec: 3.16801\n",
"2021-12-31 02:16:07,632 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.005\n",
"INFO:tensorflow:global_step/sec: 3.08125\n",
"2021-12-31 02:16:10,552 [INFO] tensorflow: global_step/sec: 3.08125\n",
"INFO:tensorflow:epoch = 59.73958333333333, learning_rate = 0.0009999999, loss = 0.0002995996, step = 5735 (5.492 sec)\n",
"2021-12-31 02:16:11,196 [INFO] tensorflow: epoch = 59.73958333333333, learning_rate = 0.0009999999, loss = 0.0002995996, step = 5735 (5.492 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07841\n",
"2021-12-31 02:16:13,476 [INFO] tensorflow: global_step/sec: 3.07841\n",
"2021-12-31 02:16:15,699 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.794\n",
"INFO:tensorflow:global_step/sec: 3.17524\n",
"2021-12-31 02:16:16,310 [INFO] tensorflow: global_step/sec: 3.17524\n",
"INFO:tensorflow:epoch = 59.916666666666664, learning_rate = 0.0009999999, loss = 0.00020735203, step = 5752 (5.435 sec)\n",
"2021-12-31 02:16:16,630 [INFO] tensorflow: epoch = 59.916666666666664, learning_rate = 0.0009999999, loss = 0.00020735203, step = 5752 (5.435 sec)\n",
"INFO:tensorflow:Saving checkpoints for step-5760.\n",
"2021-12-31 02:16:18,855 [INFO] tensorflow: Saving checkpoints for step-5760.\n",
"WARNING:tensorflow:Ignoring: /tmp/tmpgz2g0m6p; No such file or directory\n",
"2021-12-31 02:16:19,008 [WARNING] tensorflow: Ignoring: /tmp/tmpgz2g0m6p; No such file or directory\n",
"2021-12-31 02:16:22,548 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 23, 0.00s/step\n",
"2021-12-31 02:16:24,151 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 23, 0.16s/step\n",
"2021-12-31 02:16:25,700 [INFO] iva.detectnet_v2.evaluation.evaluation: step 20 / 23, 0.15s/step\n",
"Matching predictions to ground truth, class 1/1.: 100%|█| 299/299 [00:00<00:00, 14993.27it/s]\n",
"Epoch 60/120\n",
"=========================\n",
"\n",
"Validation cost: 0.001168\n",
"Mean average_precision (in %): 7.4982\n",
"\n",
"class name average precision (in %)\n",
"------------ --------------------------\n",
"car 7.49823\n",
"\n",
"Median Inference Time: 0.016016\n",
"INFO:tensorflow:epoch = 60.0, learning_rate = 0.0009999999, loss = 0.00025592744, step = 5760 (9.908 sec)\n",
"2021-12-31 02:16:26,538 [INFO] tensorflow: epoch = 60.0, learning_rate = 0.0009999999, loss = 0.00025592744, step = 5760 (9.908 sec)\n",
"INFO:tensorflow:global_step/sec: 0.879867\n",
"2021-12-31 02:16:26,539 [INFO] tensorflow: global_step/sec: 0.879867\n",
"2021-12-31 02:16:26,540 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 60/120: loss: 0.00026 learning rate: 0.00100 Time taken: 0:00:38.197542 ETA: 0:38:11.852517\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO:tensorflow:global_step/sec: 3.08222\n",
"2021-12-31 02:16:29,459 [INFO] tensorflow: global_step/sec: 3.08222\n",
"2021-12-31 02:16:31,053 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 13.026\n",
"INFO:tensorflow:epoch = 60.17708333333333, learning_rate = 0.0009999999, loss = 0.0002908432, step = 5777 (5.486 sec)\n",
"2021-12-31 02:16:32,025 [INFO] tensorflow: epoch = 60.17708333333333, learning_rate = 0.0009999999, loss = 0.0002908432, step = 5777 (5.486 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12999\n",
"2021-12-31 02:16:32,334 [INFO] tensorflow: global_step/sec: 3.12999\n",
"INFO:tensorflow:global_step/sec: 3.11828\n",
"2021-12-31 02:16:35,221 [INFO] tensorflow: global_step/sec: 3.11828\n",
"INFO:tensorflow:epoch = 60.354166666666664, learning_rate = 0.0009999999, loss = 0.0003057906, step = 5794 (5.361 sec)\n",
"2021-12-31 02:16:37,385 [INFO] tensorflow: epoch = 60.354166666666664, learning_rate = 0.0009999999, loss = 0.0003057906, step = 5794 (5.361 sec)\n",
"INFO:tensorflow:global_step/sec: 3.19024\n",
"2021-12-31 02:16:38,042 [INFO] tensorflow: global_step/sec: 3.19024\n",
"2021-12-31 02:16:39,026 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.085\n",
"INFO:tensorflow:global_step/sec: 3.08839\n",
"2021-12-31 02:16:40,956 [INFO] tensorflow: global_step/sec: 3.08839\n",
"INFO:tensorflow:epoch = 60.53125, learning_rate = 0.0009999999, loss = 0.00028665666, step = 5811 (5.504 sec)\n",
"2021-12-31 02:16:42,889 [INFO] tensorflow: epoch = 60.53125, learning_rate = 0.0009999999, loss = 0.00028665666, step = 5811 (5.504 sec)\n",
"INFO:tensorflow:global_step/sec: 3.1116\n",
"2021-12-31 02:16:43,848 [INFO] tensorflow: global_step/sec: 3.1116\n",
"INFO:tensorflow:global_step/sec: 3.11652\n",
"2021-12-31 02:16:46,736 [INFO] tensorflow: global_step/sec: 3.11652\n",
"2021-12-31 02:16:47,068 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.870\n",
"INFO:tensorflow:epoch = 60.70833333333333, learning_rate = 0.0009999999, loss = 0.00026772884, step = 5828 (5.463 sec)\n",
"2021-12-31 02:16:48,352 [INFO] tensorflow: epoch = 60.70833333333333, learning_rate = 0.0009999999, loss = 0.00026772884, step = 5828 (5.463 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12365\n",
"2021-12-31 02:16:49,617 [INFO] tensorflow: global_step/sec: 3.12365\n",
"INFO:tensorflow:global_step/sec: 3.22418\n",
"2021-12-31 02:16:52,409 [INFO] tensorflow: global_step/sec: 3.22418\n",
"INFO:tensorflow:epoch = 60.885416666666664, learning_rate = 0.0009999999, loss = 0.0003143124, step = 5845 (5.365 sec)\n",
"2021-12-31 02:16:53,716 [INFO] tensorflow: epoch = 60.885416666666664, learning_rate = 0.0009999999, loss = 0.0003143124, step = 5845 (5.365 sec)\n",
"2021-12-31 02:16:54,959 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.347\n",
"INFO:tensorflow:global_step/sec: 3.13406\n",
"2021-12-31 02:16:55,281 [INFO] tensorflow: global_step/sec: 3.13406\n",
"2021-12-31 02:16:57,210 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 61/120: loss: 0.00023 learning rate: 0.00100 Time taken: 0:00:30.671475 ETA: 0:30:09.617035\n",
"INFO:tensorflow:global_step/sec: 3.1438\n",
"2021-12-31 02:16:58,143 [INFO] tensorflow: global_step/sec: 3.1438\n",
"INFO:tensorflow:epoch = 61.0625, learning_rate = 0.0009999999, loss = 0.00030622148, step = 5862 (5.408 sec)\n",
"2021-12-31 02:16:59,124 [INFO] tensorflow: epoch = 61.0625, learning_rate = 0.0009999999, loss = 0.00030622148, step = 5862 (5.408 sec)\n",
"INFO:tensorflow:global_step/sec: 3.13396\n",
"2021-12-31 02:17:01,015 [INFO] tensorflow: global_step/sec: 3.13396\n",
"2021-12-31 02:17:02,963 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.989\n",
"INFO:tensorflow:global_step/sec: 3.10236\n",
"2021-12-31 02:17:03,916 [INFO] tensorflow: global_step/sec: 3.10236\n",
"INFO:tensorflow:epoch = 61.23958333333333, learning_rate = 0.0009999999, loss = 0.0002262259, step = 5879 (5.461 sec)\n",
"2021-12-31 02:17:04,585 [INFO] tensorflow: epoch = 61.23958333333333, learning_rate = 0.0009999999, loss = 0.0002262259, step = 5879 (5.461 sec)\n",
"INFO:tensorflow:global_step/sec: 3.06381\n",
"2021-12-31 02:17:06,854 [INFO] tensorflow: global_step/sec: 3.06381\n",
"INFO:tensorflow:global_step/sec: 3.14414\n",
"2021-12-31 02:17:09,716 [INFO] tensorflow: global_step/sec: 3.14414\n",
"INFO:tensorflow:epoch = 61.416666666666664, learning_rate = 0.0009999999, loss = 0.00029041187, step = 5896 (5.450 sec)\n",
"2021-12-31 02:17:10,035 [INFO] tensorflow: epoch = 61.416666666666664, learning_rate = 0.0009999999, loss = 0.00029041187, step = 5896 (5.450 sec)\n",
"2021-12-31 02:17:11,008 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.860\n",
"INFO:tensorflow:global_step/sec: 3.06187\n",
"2021-12-31 02:17:12,655 [INFO] tensorflow: global_step/sec: 3.06187\n",
"INFO:tensorflow:epoch = 61.59375, learning_rate = 0.0009999999, loss = 0.0002458653, step = 5913 (5.554 sec)\n",
"2021-12-31 02:17:15,589 [INFO] tensorflow: epoch = 61.59375, learning_rate = 0.0009999999, loss = 0.0002458653, step = 5913 (5.554 sec)\n",
"INFO:tensorflow:global_step/sec: 3.06744\n",
"2021-12-31 02:17:15,589 [INFO] tensorflow: global_step/sec: 3.06744\n",
"INFO:tensorflow:global_step/sec: 3.12374\n",
"2021-12-31 02:17:18,471 [INFO] tensorflow: global_step/sec: 3.12374\n",
"2021-12-31 02:17:19,101 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.713\n",
"INFO:tensorflow:epoch = 61.77083333333333, learning_rate = 0.0009999999, loss = 0.00022315713, step = 5930 (5.422 sec)\n",
"2021-12-31 02:17:21,011 [INFO] tensorflow: epoch = 61.77083333333333, learning_rate = 0.0009999999, loss = 0.00022315713, step = 5930 (5.422 sec)\n",
"INFO:tensorflow:global_step/sec: 3.13857\n",
"2021-12-31 02:17:21,338 [INFO] tensorflow: global_step/sec: 3.13857\n",
"INFO:tensorflow:global_step/sec: 3.14929\n",
"2021-12-31 02:17:24,196 [INFO] tensorflow: global_step/sec: 3.14929\n",
"INFO:tensorflow:epoch = 61.947916666666664, learning_rate = 0.0009999999, loss = 0.00022551135, step = 5947 (5.423 sec)\n",
"2021-12-31 02:17:26,434 [INFO] tensorflow: epoch = 61.947916666666664, learning_rate = 0.0009999999, loss = 0.00022551135, step = 5947 (5.423 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11901\n",
"2021-12-31 02:17:27,082 [INFO] tensorflow: global_step/sec: 3.11901\n",
"2021-12-31 02:17:27,082 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.060\n",
"2021-12-31 02:17:28,022 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 62/120: loss: 0.00021 learning rate: 0.00100 Time taken: 0:00:30.826897 ETA: 0:29:47.960048\n",
"INFO:tensorflow:global_step/sec: 3.12596\n",
"2021-12-31 02:17:29,961 [INFO] tensorflow: global_step/sec: 3.12596\n",
"INFO:tensorflow:epoch = 62.125, learning_rate = 0.0009999999, loss = 0.00024144864, step = 5964 (5.423 sec)\n",
"2021-12-31 02:17:31,857 [INFO] tensorflow: epoch = 62.125, learning_rate = 0.0009999999, loss = 0.00024144864, step = 5964 (5.423 sec)\n",
"INFO:tensorflow:global_step/sec: 3.14223\n",
"2021-12-31 02:17:32,825 [INFO] tensorflow: global_step/sec: 3.14223\n",
"2021-12-31 02:17:35,072 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.034\n",
"INFO:tensorflow:global_step/sec: 3.14178\n",
"2021-12-31 02:17:35,689 [INFO] tensorflow: global_step/sec: 3.14178\n",
"INFO:tensorflow:epoch = 62.30208333333333, learning_rate = 0.0009999999, loss = 0.0002066326, step = 5981 (5.399 sec)\n",
"2021-12-31 02:17:37,256 [INFO] tensorflow: epoch = 62.30208333333333, learning_rate = 0.0009999999, loss = 0.0002066326, step = 5981 (5.399 sec)\n",
"INFO:tensorflow:global_step/sec: 3.17649\n",
"2021-12-31 02:17:38,523 [INFO] tensorflow: global_step/sec: 3.17649\n",
"INFO:tensorflow:global_step/sec: 3.11674\n",
"2021-12-31 02:17:41,410 [INFO] tensorflow: global_step/sec: 3.11674\n",
"INFO:tensorflow:epoch = 62.479166666666664, learning_rate = 0.0009999999, loss = 0.00018762528, step = 5998 (5.440 sec)\n",
"2021-12-31 02:17:42,696 [INFO] tensorflow: epoch = 62.479166666666664, learning_rate = 0.0009999999, loss = 0.00018762528, step = 5998 (5.440 sec)\n",
"2021-12-31 02:17:43,014 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.182\n",
"INFO:tensorflow:global_step/sec: 3.13006\n",
"2021-12-31 02:17:44,286 [INFO] tensorflow: global_step/sec: 3.13006\n",
"INFO:tensorflow:global_step/sec: 3.16368\n",
"2021-12-31 02:17:47,131 [INFO] tensorflow: global_step/sec: 3.16368\n",
"INFO:tensorflow:epoch = 62.65625, learning_rate = 0.0009999999, loss = 0.00021068203, step = 6015 (5.413 sec)\n",
"2021-12-31 02:17:48,109 [INFO] tensorflow: epoch = 62.65625, learning_rate = 0.0009999999, loss = 0.00021068203, step = 6015 (5.413 sec)\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO:tensorflow:global_step/sec: 3.08351\n",
"2021-12-31 02:17:50,049 [INFO] tensorflow: global_step/sec: 3.08351\n",
"2021-12-31 02:17:51,016 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.995\n",
"INFO:tensorflow:global_step/sec: 3.12408\n",
"2021-12-31 02:17:52,930 [INFO] tensorflow: global_step/sec: 3.12408\n",
"INFO:tensorflow:epoch = 62.83333333333333, learning_rate = 0.0009999999, loss = 0.0001879006, step = 6032 (5.466 sec)\n",
"2021-12-31 02:17:53,575 [INFO] tensorflow: epoch = 62.83333333333333, learning_rate = 0.0009999999, loss = 0.0001879006, step = 6032 (5.466 sec)\n",
"INFO:tensorflow:global_step/sec: 3.16032\n",
"2021-12-31 02:17:55,778 [INFO] tensorflow: global_step/sec: 3.16032\n",
"INFO:tensorflow:global_step/sec: 3.12203\n",
"2021-12-31 02:17:58,661 [INFO] tensorflow: global_step/sec: 3.12203\n",
"2021-12-31 02:17:58,662 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 63/120: loss: 0.00019 learning rate: 0.00100 Time taken: 0:00:30.617789 ETA: 0:29:05.213948\n",
"INFO:tensorflow:epoch = 63.010416666666664, learning_rate = 0.0009999999, loss = 0.00027046746, step = 6049 (5.414 sec)\n",
"2021-12-31 02:17:58,990 [INFO] tensorflow: epoch = 63.010416666666664, learning_rate = 0.0009999999, loss = 0.00027046746, step = 6049 (5.414 sec)\n",
"2021-12-31 02:17:58,990 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.082\n",
"INFO:tensorflow:global_step/sec: 3.13138\n",
"2021-12-31 02:18:01,535 [INFO] tensorflow: global_step/sec: 3.13138\n",
"INFO:tensorflow:epoch = 63.1875, learning_rate = 0.0009999999, loss = 0.00025624284, step = 6066 (5.447 sec)\n",
"2021-12-31 02:18:04,437 [INFO] tensorflow: epoch = 63.1875, learning_rate = 0.0009999999, loss = 0.00025624284, step = 6066 (5.447 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10071\n",
"2021-12-31 02:18:04,437 [INFO] tensorflow: global_step/sec: 3.10071\n",
"2021-12-31 02:18:07,004 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.955\n",
"INFO:tensorflow:global_step/sec: 3.13355\n",
"2021-12-31 02:18:07,310 [INFO] tensorflow: global_step/sec: 3.13355\n",
"INFO:tensorflow:epoch = 63.36458333333333, learning_rate = 0.0009999999, loss = 0.00020785103, step = 6083 (5.456 sec)\n",
"2021-12-31 02:18:09,893 [INFO] tensorflow: epoch = 63.36458333333333, learning_rate = 0.0009999999, loss = 0.00020785103, step = 6083 (5.456 sec)\n",
"INFO:tensorflow:global_step/sec: 3.13682\n",
"2021-12-31 02:18:10,179 [INFO] tensorflow: global_step/sec: 3.13682\n",
"INFO:tensorflow:global_step/sec: 3.07099\n",
"2021-12-31 02:18:13,109 [INFO] tensorflow: global_step/sec: 3.07099\n",
"2021-12-31 02:18:14,998 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.021\n",
"INFO:tensorflow:epoch = 63.541666666666664, learning_rate = 0.0009999999, loss = 0.00027497724, step = 6100 (5.430 sec)\n",
"2021-12-31 02:18:15,322 [INFO] tensorflow: epoch = 63.541666666666664, learning_rate = 0.0009999999, loss = 0.00027497724, step = 6100 (5.430 sec)\n",
"INFO:tensorflow:global_step/sec: 3.13171\n",
"2021-12-31 02:18:15,983 [INFO] tensorflow: global_step/sec: 3.13171\n",
"INFO:tensorflow:global_step/sec: 3.11952\n",
"2021-12-31 02:18:18,868 [INFO] tensorflow: global_step/sec: 3.11952\n",
"INFO:tensorflow:epoch = 63.71875, learning_rate = 0.0009999999, loss = 0.00020174569, step = 6117 (5.472 sec)\n",
"2021-12-31 02:18:20,794 [INFO] tensorflow: epoch = 63.71875, learning_rate = 0.0009999999, loss = 0.00020174569, step = 6117 (5.472 sec)\n",
"INFO:tensorflow:global_step/sec: 3.1272\n",
"2021-12-31 02:18:21,746 [INFO] tensorflow: global_step/sec: 3.1272\n",
"2021-12-31 02:18:23,042 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.865\n",
"INFO:tensorflow:global_step/sec: 3.15143\n",
"2021-12-31 02:18:24,602 [INFO] tensorflow: global_step/sec: 3.15143\n",
"INFO:tensorflow:epoch = 63.89583333333333, learning_rate = 0.0009999999, loss = 0.0002714278, step = 6134 (5.413 sec)\n",
"2021-12-31 02:18:26,207 [INFO] tensorflow: epoch = 63.89583333333333, learning_rate = 0.0009999999, loss = 0.0002714278, step = 6134 (5.413 sec)\n",
"INFO:tensorflow:global_step/sec: 3.13317\n",
"2021-12-31 02:18:27,475 [INFO] tensorflow: global_step/sec: 3.13317\n",
"2021-12-31 02:18:29,451 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 64/120: loss: 0.00032 learning rate: 0.00100 Time taken: 0:00:30.778063 ETA: 0:28:43.571531\n",
"INFO:tensorflow:global_step/sec: 3.05168\n",
"2021-12-31 02:18:30,424 [INFO] tensorflow: global_step/sec: 3.05168\n",
"2021-12-31 02:18:31,071 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.911\n",
"INFO:tensorflow:epoch = 64.07291666666666, learning_rate = 0.0009999999, loss = 0.00021607369, step = 6151 (5.495 sec)\n",
"2021-12-31 02:18:31,702 [INFO] tensorflow: epoch = 64.07291666666666, learning_rate = 0.0009999999, loss = 0.00021607369, step = 6151 (5.495 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10386\n",
"2021-12-31 02:18:33,323 [INFO] tensorflow: global_step/sec: 3.10386\n",
"INFO:tensorflow:global_step/sec: 3.14869\n",
"2021-12-31 02:18:36,182 [INFO] tensorflow: global_step/sec: 3.14869\n",
"INFO:tensorflow:epoch = 64.25, learning_rate = 0.0009999999, loss = 0.00019266618, step = 6168 (5.454 sec)\n",
"2021-12-31 02:18:37,156 [INFO] tensorflow: epoch = 64.25, learning_rate = 0.0009999999, loss = 0.00019266618, step = 6168 (5.454 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09392\n",
"2021-12-31 02:18:39,091 [INFO] tensorflow: global_step/sec: 3.09392\n",
"2021-12-31 02:18:39,091 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.936\n",
"INFO:tensorflow:global_step/sec: 3.1748\n",
"2021-12-31 02:18:41,925 [INFO] tensorflow: global_step/sec: 3.1748\n",
"INFO:tensorflow:epoch = 64.42708333333333, learning_rate = 0.0009999999, loss = 0.00032748937, step = 6185 (5.424 sec)\n",
"2021-12-31 02:18:42,580 [INFO] tensorflow: epoch = 64.42708333333333, learning_rate = 0.0009999999, loss = 0.00032748937, step = 6185 (5.424 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10463\n",
"2021-12-31 02:18:44,824 [INFO] tensorflow: global_step/sec: 3.10463\n",
"2021-12-31 02:18:47,031 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.191\n",
"INFO:tensorflow:global_step/sec: 3.14035\n",
"2021-12-31 02:18:47,690 [INFO] tensorflow: global_step/sec: 3.14035\n",
"INFO:tensorflow:epoch = 64.60416666666666, learning_rate = 0.0009999999, loss = 0.00023559244, step = 6202 (5.416 sec)\n",
"2021-12-31 02:18:47,996 [INFO] tensorflow: epoch = 64.60416666666666, learning_rate = 0.0009999999, loss = 0.00023559244, step = 6202 (5.416 sec)\n",
"INFO:tensorflow:global_step/sec: 3.181\n",
"2021-12-31 02:18:50,520 [INFO] tensorflow: global_step/sec: 3.181\n",
"INFO:tensorflow:epoch = 64.78125, learning_rate = 0.0009999999, loss = 0.00018213262, step = 6219 (5.400 sec)\n",
"2021-12-31 02:18:53,396 [INFO] tensorflow: epoch = 64.78125, learning_rate = 0.0009999999, loss = 0.00018213262, step = 6219 (5.400 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12814\n",
"2021-12-31 02:18:53,397 [INFO] tensorflow: global_step/sec: 3.12814\n",
"2021-12-31 02:18:54,998 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.103\n",
"INFO:tensorflow:global_step/sec: 3.14067\n",
"2021-12-31 02:18:56,262 [INFO] tensorflow: global_step/sec: 3.14067\n",
"INFO:tensorflow:epoch = 64.95833333333333, learning_rate = 0.0009999999, loss = 0.00022505099, step = 6236 (5.436 sec)\n",
"2021-12-31 02:18:58,832 [INFO] tensorflow: epoch = 64.95833333333333, learning_rate = 0.0009999999, loss = 0.00022505099, step = 6236 (5.436 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10194\n",
"2021-12-31 02:18:59,164 [INFO] tensorflow: global_step/sec: 3.10194\n",
"2021-12-31 02:19:00,143 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 65/120: loss: 0.00020 learning rate: 0.00100 Time taken: 0:00:30.706986 ETA: 0:28:08.884214\n",
"INFO:tensorflow:global_step/sec: 3.15015\n",
"2021-12-31 02:19:02,021 [INFO] tensorflow: global_step/sec: 3.15015\n",
"2021-12-31 02:19:02,987 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.035\n",
"INFO:tensorflow:epoch = 65.13541666666666, learning_rate = 0.0009999999, loss = 0.00025406596, step = 6253 (5.455 sec)\n",
"2021-12-31 02:19:04,287 [INFO] tensorflow: epoch = 65.13541666666666, learning_rate = 0.0009999999, loss = 0.00025406596, step = 6253 (5.455 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08098\n",
"2021-12-31 02:19:04,942 [INFO] tensorflow: global_step/sec: 3.08098\n",
"INFO:tensorflow:global_step/sec: 3.13791\n",
"2021-12-31 02:19:07,810 [INFO] tensorflow: global_step/sec: 3.13791\n",
"INFO:tensorflow:epoch = 65.3125, learning_rate = 0.0009999999, loss = 0.00020526795, step = 6270 (5.461 sec)\n",
"2021-12-31 02:19:09,748 [INFO] tensorflow: epoch = 65.3125, learning_rate = 0.0009999999, loss = 0.00020526795, step = 6270 (5.461 sec)\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO:tensorflow:global_step/sec: 3.10414\n",
"2021-12-31 02:19:10,709 [INFO] tensorflow: global_step/sec: 3.10414\n",
"2021-12-31 02:19:11,031 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.865\n",
"INFO:tensorflow:global_step/sec: 3.10837\n",
"2021-12-31 02:19:13,605 [INFO] tensorflow: global_step/sec: 3.10837\n",
"INFO:tensorflow:epoch = 65.48958333333333, learning_rate = 0.0009999999, loss = 0.00018664336, step = 6287 (5.440 sec)\n",
"2021-12-31 02:19:15,188 [INFO] tensorflow: epoch = 65.48958333333333, learning_rate = 0.0009999999, loss = 0.00018664336, step = 6287 (5.440 sec)\n",
"INFO:tensorflow:global_step/sec: 3.15104\n",
"2021-12-31 02:19:16,461 [INFO] tensorflow: global_step/sec: 3.15104\n",
"2021-12-31 02:19:19,047 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.950\n",
"INFO:tensorflow:global_step/sec: 3.09198\n",
"2021-12-31 02:19:19,372 [INFO] tensorflow: global_step/sec: 3.09198\n",
"INFO:tensorflow:epoch = 65.66666666666666, learning_rate = 0.0009999999, loss = 0.00019480637, step = 6304 (5.448 sec)\n",
"2021-12-31 02:19:20,637 [INFO] tensorflow: epoch = 65.66666666666666, learning_rate = 0.0009999999, loss = 0.00019480637, step = 6304 (5.448 sec)\n",
"INFO:tensorflow:global_step/sec: 3.17656\n",
"2021-12-31 02:19:22,205 [INFO] tensorflow: global_step/sec: 3.17656\n",
"INFO:tensorflow:global_step/sec: 3.04145\n",
"2021-12-31 02:19:25,164 [INFO] tensorflow: global_step/sec: 3.04145\n",
"INFO:tensorflow:epoch = 65.84375, learning_rate = 0.0009999999, loss = 0.00016475077, step = 6321 (5.492 sec)\n",
"2021-12-31 02:19:26,128 [INFO] tensorflow: epoch = 65.84375, learning_rate = 0.0009999999, loss = 0.00016475077, step = 6321 (5.492 sec)\n",
"2021-12-31 02:19:27,090 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.868\n",
"INFO:tensorflow:global_step/sec: 3.12008\n",
"2021-12-31 02:19:28,049 [INFO] tensorflow: global_step/sec: 3.12008\n",
"INFO:tensorflow:global_step/sec: 3.18646\n",
"2021-12-31 02:19:30,873 [INFO] tensorflow: global_step/sec: 3.18646\n",
"2021-12-31 02:19:30,874 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 66/120: loss: 0.00021 learning rate: 0.00100 Time taken: 0:00:30.725753 ETA: 0:27:39.190640\n",
"INFO:tensorflow:epoch = 66.02083333333333, learning_rate = 0.0009999999, loss = 0.00026676062, step = 6338 (5.396 sec)\n",
"2021-12-31 02:19:31,525 [INFO] tensorflow: epoch = 66.02083333333333, learning_rate = 0.0009999999, loss = 0.00026676062, step = 6338 (5.396 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11022\n",
"2021-12-31 02:19:33,767 [INFO] tensorflow: global_step/sec: 3.11022\n",
"2021-12-31 02:19:35,063 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.086\n",
"INFO:tensorflow:global_step/sec: 3.04163\n",
"2021-12-31 02:19:36,726 [INFO] tensorflow: global_step/sec: 3.04163\n",
"INFO:tensorflow:epoch = 66.19791666666666, learning_rate = 0.0009999999, loss = 0.00021088615, step = 6355 (5.516 sec)\n",
"2021-12-31 02:19:37,041 [INFO] tensorflow: epoch = 66.19791666666666, learning_rate = 0.0009999999, loss = 0.00021088615, step = 6355 (5.516 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11943\n",
"2021-12-31 02:19:39,611 [INFO] tensorflow: global_step/sec: 3.11943\n",
"INFO:tensorflow:epoch = 66.375, learning_rate = 0.0009999999, loss = 0.0002372649, step = 6372 (5.447 sec)\n",
"2021-12-31 02:19:42,488 [INFO] tensorflow: epoch = 66.375, learning_rate = 0.0009999999, loss = 0.0002372649, step = 6372 (5.447 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12706\n",
"2021-12-31 02:19:42,489 [INFO] tensorflow: global_step/sec: 3.12706\n",
"2021-12-31 02:19:43,160 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.701\n",
"INFO:tensorflow:global_step/sec: 3.13157\n",
"2021-12-31 02:19:45,363 [INFO] tensorflow: global_step/sec: 3.13157\n",
"INFO:tensorflow:epoch = 66.55208333333333, learning_rate = 0.0009999999, loss = 0.00025395816, step = 6389 (5.424 sec)\n",
"2021-12-31 02:19:47,912 [INFO] tensorflow: epoch = 66.55208333333333, learning_rate = 0.0009999999, loss = 0.00025395816, step = 6389 (5.424 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12961\n",
"2021-12-31 02:19:48,239 [INFO] tensorflow: global_step/sec: 3.12961\n",
"INFO:tensorflow:global_step/sec: 3.15582\n",
"2021-12-31 02:19:51,091 [INFO] tensorflow: global_step/sec: 3.15582\n",
"2021-12-31 02:19:51,091 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.217\n",
"INFO:tensorflow:epoch = 66.72916666666666, learning_rate = 0.0009999999, loss = 0.00020779412, step = 6406 (5.443 sec)\n",
"2021-12-31 02:19:53,355 [INFO] tensorflow: epoch = 66.72916666666666, learning_rate = 0.0009999999, loss = 0.00020779412, step = 6406 (5.443 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08012\n",
"2021-12-31 02:19:54,012 [INFO] tensorflow: global_step/sec: 3.08012\n",
"INFO:tensorflow:global_step/sec: 3.09009\n",
"2021-12-31 02:19:56,925 [INFO] tensorflow: global_step/sec: 3.09009\n",
"INFO:tensorflow:epoch = 66.90625, learning_rate = 0.0009999999, loss = 0.00019410613, step = 6423 (5.533 sec)\n",
"2021-12-31 02:19:58,888 [INFO] tensorflow: epoch = 66.90625, learning_rate = 0.0009999999, loss = 0.00019410613, step = 6423 (5.533 sec)\n",
"2021-12-31 02:19:59,204 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.652\n",
"INFO:tensorflow:global_step/sec: 3.11294\n",
"2021-12-31 02:19:59,816 [INFO] tensorflow: global_step/sec: 3.11294\n",
"2021-12-31 02:20:01,725 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 67/120: loss: 0.00025 learning rate: 0.00100 Time taken: 0:00:30.867944 ETA: 0:27:16.001019\n",
"INFO:tensorflow:global_step/sec: 3.17059\n",
"2021-12-31 02:20:02,655 [INFO] tensorflow: global_step/sec: 3.17059\n",
"INFO:tensorflow:epoch = 67.08333333333333, learning_rate = 0.0009999999, loss = 0.00022489921, step = 6440 (5.373 sec)\n",
"2021-12-31 02:20:04,261 [INFO] tensorflow: epoch = 67.08333333333333, learning_rate = 0.0009999999, loss = 0.00022489921, step = 6440 (5.373 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12155\n",
"2021-12-31 02:20:05,538 [INFO] tensorflow: global_step/sec: 3.12155\n",
"2021-12-31 02:20:07,154 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.159\n",
"INFO:tensorflow:global_step/sec: 3.11576\n",
"2021-12-31 02:20:08,427 [INFO] tensorflow: global_step/sec: 3.11576\n",
"INFO:tensorflow:epoch = 67.26041666666666, learning_rate = 0.0009999999, loss = 0.00015930535, step = 6457 (5.461 sec)\n",
"2021-12-31 02:20:09,722 [INFO] tensorflow: epoch = 67.26041666666666, learning_rate = 0.0009999999, loss = 0.00015930535, step = 6457 (5.461 sec)\n",
"INFO:tensorflow:global_step/sec: 3.13507\n",
"2021-12-31 02:20:11,297 [INFO] tensorflow: global_step/sec: 3.13507\n",
"INFO:tensorflow:global_step/sec: 3.11164\n",
"2021-12-31 02:20:14,190 [INFO] tensorflow: global_step/sec: 3.11164\n",
"INFO:tensorflow:epoch = 67.4375, learning_rate = 0.0009999999, loss = 0.00016469126, step = 6474 (5.408 sec)\n",
"2021-12-31 02:20:15,129 [INFO] tensorflow: epoch = 67.4375, learning_rate = 0.0009999999, loss = 0.00016469126, step = 6474 (5.408 sec)\n",
"2021-12-31 02:20:15,130 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.078\n",
"INFO:tensorflow:global_step/sec: 3.13606\n",
"2021-12-31 02:20:17,059 [INFO] tensorflow: global_step/sec: 3.13606\n",
"INFO:tensorflow:global_step/sec: 3.09061\n",
"2021-12-31 02:20:19,972 [INFO] tensorflow: global_step/sec: 3.09061\n",
"INFO:tensorflow:epoch = 67.61458333333333, learning_rate = 0.0009999999, loss = 0.00018229302, step = 6491 (5.488 sec)\n",
"2021-12-31 02:20:20,617 [INFO] tensorflow: epoch = 67.61458333333333, learning_rate = 0.0009999999, loss = 0.00018229302, step = 6491 (5.488 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09548\n",
"2021-12-31 02:20:22,879 [INFO] tensorflow: global_step/sec: 3.09548\n",
"2021-12-31 02:20:23,211 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.748\n",
"INFO:tensorflow:global_step/sec: 3.09344\n",
"2021-12-31 02:20:25,788 [INFO] tensorflow: global_step/sec: 3.09344\n",
"INFO:tensorflow:epoch = 67.79166666666666, learning_rate = 0.0009999999, loss = 0.0001834989, step = 6508 (5.503 sec)\n",
"2021-12-31 02:20:26,120 [INFO] tensorflow: epoch = 67.79166666666666, learning_rate = 0.0009999999, loss = 0.0001834989, step = 6508 (5.503 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12968\n",
"2021-12-31 02:20:28,664 [INFO] tensorflow: global_step/sec: 3.12968\n",
"2021-12-31 02:20:31,251 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.877\n",
"INFO:tensorflow:epoch = 67.96875, learning_rate = 0.0009999999, loss = 0.00019346291, step = 6525 (5.464 sec)\n",
"2021-12-31 02:20:31,584 [INFO] tensorflow: epoch = 67.96875, learning_rate = 0.0009999999, loss = 0.00019346291, step = 6525 (5.464 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08128\n",
"2021-12-31 02:20:31,585 [INFO] tensorflow: global_step/sec: 3.08128\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"2021-12-31 02:20:32,572 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 68/120: loss: 0.00024 learning rate: 0.00100 Time taken: 0:00:30.833629 ETA: 0:26:43.348702\n",
"INFO:tensorflow:global_step/sec: 3.07581\n",
"2021-12-31 02:20:34,511 [INFO] tensorflow: global_step/sec: 3.07581\n",
"INFO:tensorflow:epoch = 68.14583333333333, learning_rate = 0.0009999999, loss = 0.00027407007, step = 6542 (5.450 sec)\n",
"2021-12-31 02:20:37,034 [INFO] tensorflow: epoch = 68.14583333333333, learning_rate = 0.0009999999, loss = 0.00027407007, step = 6542 (5.450 sec)\n",
"INFO:tensorflow:global_step/sec: 3.15916\n",
"2021-12-31 02:20:37,360 [INFO] tensorflow: global_step/sec: 3.15916\n",
"2021-12-31 02:20:39,304 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.837\n",
"INFO:tensorflow:global_step/sec: 3.10113\n",
"2021-12-31 02:20:40,262 [INFO] tensorflow: global_step/sec: 3.10113\n",
"INFO:tensorflow:epoch = 68.32291666666666, learning_rate = 0.0009999999, loss = 0.00020419603, step = 6559 (5.468 sec)\n",
"2021-12-31 02:20:42,502 [INFO] tensorflow: epoch = 68.32291666666666, learning_rate = 0.0009999999, loss = 0.00020419603, step = 6559 (5.468 sec)\n",
"INFO:tensorflow:global_step/sec: 3.13325\n",
"2021-12-31 02:20:43,134 [INFO] tensorflow: global_step/sec: 3.13325\n",
"INFO:tensorflow:global_step/sec: 3.12317\n",
"2021-12-31 02:20:46,016 [INFO] tensorflow: global_step/sec: 3.12317\n",
"2021-12-31 02:20:47,315 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.965\n",
"INFO:tensorflow:epoch = 68.5, learning_rate = 0.0009999999, loss = 0.00021399799, step = 6576 (5.439 sec)\n",
"2021-12-31 02:20:47,940 [INFO] tensorflow: epoch = 68.5, learning_rate = 0.0009999999, loss = 0.00021399799, step = 6576 (5.439 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08644\n",
"2021-12-31 02:20:48,932 [INFO] tensorflow: global_step/sec: 3.08644\n",
"INFO:tensorflow:global_step/sec: 3.10259\n",
"2021-12-31 02:20:51,833 [INFO] tensorflow: global_step/sec: 3.10259\n",
"INFO:tensorflow:epoch = 68.67708333333333, learning_rate = 0.0009999999, loss = 0.00025514496, step = 6593 (5.508 sec)\n",
"2021-12-31 02:20:53,449 [INFO] tensorflow: epoch = 68.67708333333333, learning_rate = 0.0009999999, loss = 0.00025514496, step = 6593 (5.508 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11142\n",
"2021-12-31 02:20:54,725 [INFO] tensorflow: global_step/sec: 3.11142\n",
"2021-12-31 02:20:55,373 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.819\n",
"INFO:tensorflow:global_step/sec: 3.06431\n",
"2021-12-31 02:20:57,663 [INFO] tensorflow: global_step/sec: 3.06431\n",
"INFO:tensorflow:epoch = 68.85416666666666, learning_rate = 0.0009999999, loss = 0.00022886533, step = 6610 (5.461 sec)\n",
"2021-12-31 02:20:58,910 [INFO] tensorflow: epoch = 68.85416666666666, learning_rate = 0.0009999999, loss = 0.00022886533, step = 6610 (5.461 sec)\n",
"INFO:tensorflow:global_step/sec: 3.17883\n",
"2021-12-31 02:21:00,494 [INFO] tensorflow: global_step/sec: 3.17883\n",
"INFO:tensorflow:global_step/sec: 3.16598\n",
"2021-12-31 02:21:03,336 [INFO] tensorflow: global_step/sec: 3.16598\n",
"2021-12-31 02:21:03,337 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 69/120: loss: 0.00021 learning rate: 0.00100 Time taken: 0:00:30.779441 ETA: 0:26:09.751497\n",
"2021-12-31 02:21:03,337 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.114\n",
"INFO:tensorflow:epoch = 69.03125, learning_rate = 0.0009999999, loss = 0.00017899284, step = 6627 (5.389 sec)\n",
"2021-12-31 02:21:04,298 [INFO] tensorflow: epoch = 69.03125, learning_rate = 0.0009999999, loss = 0.00017899284, step = 6627 (5.389 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11787\n",
"2021-12-31 02:21:06,223 [INFO] tensorflow: global_step/sec: 3.11787\n",
"INFO:tensorflow:global_step/sec: 3.13432\n",
"2021-12-31 02:21:09,094 [INFO] tensorflow: global_step/sec: 3.13432\n",
"INFO:tensorflow:epoch = 69.20833333333333, learning_rate = 0.0009999999, loss = 0.00022112772, step = 6644 (5.439 sec)\n",
"2021-12-31 02:21:09,738 [INFO] tensorflow: epoch = 69.20833333333333, learning_rate = 0.0009999999, loss = 0.00022112772, step = 6644 (5.439 sec)\n",
"2021-12-31 02:21:11,343 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.982\n",
"INFO:tensorflow:global_step/sec: 3.09762\n",
"2021-12-31 02:21:12,000 [INFO] tensorflow: global_step/sec: 3.09762\n",
"INFO:tensorflow:global_step/sec: 3.11987\n",
"2021-12-31 02:21:14,885 [INFO] tensorflow: global_step/sec: 3.11987\n",
"INFO:tensorflow:epoch = 69.38541666666666, learning_rate = 0.0009999999, loss = 0.0002007435, step = 6661 (5.463 sec)\n",
"2021-12-31 02:21:15,201 [INFO] tensorflow: epoch = 69.38541666666666, learning_rate = 0.0009999999, loss = 0.0002007435, step = 6661 (5.463 sec)\n",
"INFO:tensorflow:global_step/sec: 3.14711\n",
"2021-12-31 02:21:17,744 [INFO] tensorflow: global_step/sec: 3.14711\n",
"2021-12-31 02:21:19,337 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.020\n",
"INFO:tensorflow:epoch = 69.5625, learning_rate = 0.0009999999, loss = 0.00028268684, step = 6678 (5.422 sec)\n",
"2021-12-31 02:21:20,623 [INFO] tensorflow: epoch = 69.5625, learning_rate = 0.0009999999, loss = 0.00028268684, step = 6678 (5.422 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12537\n",
"2021-12-31 02:21:20,624 [INFO] tensorflow: global_step/sec: 3.12537\n",
"INFO:tensorflow:global_step/sec: 3.1465\n",
"2021-12-31 02:21:23,484 [INFO] tensorflow: global_step/sec: 3.1465\n",
"INFO:tensorflow:epoch = 69.73958333333333, learning_rate = 0.0009999999, loss = 0.00018598557, step = 6695 (5.439 sec)\n",
"2021-12-31 02:21:26,063 [INFO] tensorflow: epoch = 69.73958333333333, learning_rate = 0.0009999999, loss = 0.00018598557, step = 6695 (5.439 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10946\n",
"2021-12-31 02:21:26,379 [INFO] tensorflow: global_step/sec: 3.10946\n",
"2021-12-31 02:21:27,344 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.981\n",
"INFO:tensorflow:global_step/sec: 3.04757\n",
"2021-12-31 02:21:29,332 [INFO] tensorflow: global_step/sec: 3.04757\n",
"INFO:tensorflow:epoch = 69.91666666666666, learning_rate = 0.0009999999, loss = 0.00021765297, step = 6712 (5.509 sec)\n",
"2021-12-31 02:21:31,572 [INFO] tensorflow: epoch = 69.91666666666666, learning_rate = 0.0009999999, loss = 0.00021765297, step = 6712 (5.509 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12688\n",
"2021-12-31 02:21:32,210 [INFO] tensorflow: global_step/sec: 3.12688\n",
"INFO:tensorflow:Saving checkpoints for step-6720.\n",
"2021-12-31 02:21:33,794 [INFO] tensorflow: Saving checkpoints for step-6720.\n",
"WARNING:tensorflow:Ignoring: /tmp/tmpy5il9il2; No such file or directory\n",
"2021-12-31 02:21:33,945 [WARNING] tensorflow: Ignoring: /tmp/tmpy5il9il2; No such file or directory\n",
"2021-12-31 02:21:37,471 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 23, 0.00s/step\n",
"2021-12-31 02:21:39,290 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 23, 0.18s/step\n",
"2021-12-31 02:21:41,136 [INFO] iva.detectnet_v2.evaluation.evaluation: step 20 / 23, 0.18s/step\n",
"Matching predictions to ground truth, class 1/1.: 100%|█| 1921/1921 [00:00<00:00, 15298.64it/s]\n",
"Epoch 70/120\n",
"=========================\n",
"\n",
"Validation cost: 0.000228\n",
"Mean average_precision (in %): 70.7406\n",
"\n",
"class name average precision (in %)\n",
"------------ --------------------------\n",
"car 70.7406\n",
"\n",
"Median Inference Time: 0.015626\n",
"INFO:tensorflow:epoch = 70.0, learning_rate = 0.0009999999, loss = 0.0002533421, step = 6720 (10.581 sec)\n",
"2021-12-31 02:21:42,152 [INFO] tensorflow: epoch = 70.0, learning_rate = 0.0009999999, loss = 0.0002533421, step = 6720 (10.581 sec)\n",
"2021-12-31 02:21:42,153 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 70/120: loss: 0.00025 learning rate: 0.00100 Time taken: 0:00:38.793950 ETA: 0:32:19.697516\n",
"INFO:tensorflow:global_step/sec: 0.825737\n",
"2021-12-31 02:21:43,110 [INFO] tensorflow: global_step/sec: 0.825737\n",
"2021-12-31 02:21:43,427 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 12.435\n",
"INFO:tensorflow:global_step/sec: 3.11672\n",
"2021-12-31 02:21:45,997 [INFO] tensorflow: global_step/sec: 3.11672\n",
"INFO:tensorflow:epoch = 70.17708333333333, learning_rate = 0.0009999999, loss = 0.00026446863, step = 6737 (5.478 sec)\n",
"2021-12-31 02:21:47,630 [INFO] tensorflow: epoch = 70.17708333333333, learning_rate = 0.0009999999, loss = 0.00026446863, step = 6737 (5.478 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07963\n",
"2021-12-31 02:21:48,920 [INFO] tensorflow: global_step/sec: 3.07963\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"2021-12-31 02:21:51,473 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.859\n",
"INFO:tensorflow:global_step/sec: 3.11863\n",
"2021-12-31 02:21:51,806 [INFO] tensorflow: global_step/sec: 3.11863\n",
"INFO:tensorflow:epoch = 70.35416666666666, learning_rate = 0.0009999999, loss = 0.0001972542, step = 6754 (5.458 sec)\n",
"2021-12-31 02:21:53,088 [INFO] tensorflow: epoch = 70.35416666666666, learning_rate = 0.0009999999, loss = 0.0001972542, step = 6754 (5.458 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11907\n",
"2021-12-31 02:21:54,691 [INFO] tensorflow: global_step/sec: 3.11907\n",
"INFO:tensorflow:global_step/sec: 3.09339\n",
"2021-12-31 02:21:57,600 [INFO] tensorflow: global_step/sec: 3.09339\n",
"INFO:tensorflow:epoch = 70.53125, learning_rate = 0.0009999999, loss = 0.00021731402, step = 6771 (5.464 sec)\n",
"2021-12-31 02:21:58,551 [INFO] tensorflow: epoch = 70.53125, learning_rate = 0.0009999999, loss = 0.00021731402, step = 6771 (5.464 sec)\n",
"2021-12-31 02:21:59,525 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.837\n",
"INFO:tensorflow:global_step/sec: 3.12121\n",
"2021-12-31 02:22:00,484 [INFO] tensorflow: global_step/sec: 3.12121\n",
"INFO:tensorflow:global_step/sec: 3.13995\n",
"2021-12-31 02:22:03,350 [INFO] tensorflow: global_step/sec: 3.13995\n",
"INFO:tensorflow:epoch = 70.70833333333333, learning_rate = 0.0009999999, loss = 0.00025501798, step = 6788 (5.425 sec)\n",
"2021-12-31 02:22:03,977 [INFO] tensorflow: epoch = 70.70833333333333, learning_rate = 0.0009999999, loss = 0.00025501798, step = 6788 (5.425 sec)\n",
"INFO:tensorflow:global_step/sec: 3.15858\n",
"2021-12-31 02:22:06,200 [INFO] tensorflow: global_step/sec: 3.15858\n",
"2021-12-31 02:22:07,476 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.156\n",
"INFO:tensorflow:global_step/sec: 3.13873\n",
"2021-12-31 02:22:09,067 [INFO] tensorflow: global_step/sec: 3.13873\n",
"INFO:tensorflow:epoch = 70.88541666666666, learning_rate = 0.0009999999, loss = 0.00019845944, step = 6805 (5.415 sec)\n",
"2021-12-31 02:22:09,392 [INFO] tensorflow: epoch = 70.88541666666666, learning_rate = 0.0009999999, loss = 0.00019845944, step = 6805 (5.415 sec)\n",
"INFO:tensorflow:global_step/sec: 3.05977\n",
"2021-12-31 02:22:12,008 [INFO] tensorflow: global_step/sec: 3.05977\n",
"2021-12-31 02:22:12,983 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 71/120: loss: 0.00026 learning rate: 0.00100 Time taken: 0:00:30.831082 ETA: 0:25:10.723023\n",
"INFO:tensorflow:epoch = 71.0625, learning_rate = 0.0009999999, loss = 0.00024402466, step = 6822 (5.523 sec)\n",
"2021-12-31 02:22:14,914 [INFO] tensorflow: epoch = 71.0625, learning_rate = 0.0009999999, loss = 0.00024402466, step = 6822 (5.523 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09646\n",
"2021-12-31 02:22:14,915 [INFO] tensorflow: global_step/sec: 3.09646\n",
"2021-12-31 02:22:15,568 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.718\n",
"INFO:tensorflow:global_step/sec: 3.08621\n",
"2021-12-31 02:22:17,831 [INFO] tensorflow: global_step/sec: 3.08621\n",
"INFO:tensorflow:epoch = 71.23958333333333, learning_rate = 0.0009999999, loss = 0.0002740305, step = 6839 (5.492 sec)\n",
"2021-12-31 02:22:20,406 [INFO] tensorflow: epoch = 71.23958333333333, learning_rate = 0.0009999999, loss = 0.0002740305, step = 6839 (5.492 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11157\n",
"2021-12-31 02:22:20,724 [INFO] tensorflow: global_step/sec: 3.11157\n",
"INFO:tensorflow:global_step/sec: 3.1289\n",
"2021-12-31 02:22:23,600 [INFO] tensorflow: global_step/sec: 3.1289\n",
"2021-12-31 02:22:23,601 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.898\n",
"INFO:tensorflow:epoch = 71.41666666666666, learning_rate = 0.0009999999, loss = 0.00020245947, step = 6856 (5.459 sec)\n",
"2021-12-31 02:22:25,865 [INFO] tensorflow: epoch = 71.41666666666666, learning_rate = 0.0009999999, loss = 0.00020245947, step = 6856 (5.459 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10895\n",
"2021-12-31 02:22:26,495 [INFO] tensorflow: global_step/sec: 3.10895\n",
"INFO:tensorflow:global_step/sec: 3.15753\n",
"2021-12-31 02:22:29,345 [INFO] tensorflow: global_step/sec: 3.15753\n",
"INFO:tensorflow:epoch = 71.59375, learning_rate = 0.0009999999, loss = 0.00026298052, step = 6873 (5.385 sec)\n",
"2021-12-31 02:22:31,250 [INFO] tensorflow: epoch = 71.59375, learning_rate = 0.0009999999, loss = 0.00026298052, step = 6873 (5.385 sec)\n",
"2021-12-31 02:22:31,575 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.080\n",
"INFO:tensorflow:global_step/sec: 3.13893\n",
"2021-12-31 02:22:32,212 [INFO] tensorflow: global_step/sec: 3.13893\n",
"INFO:tensorflow:global_step/sec: 3.09143\n",
"2021-12-31 02:22:35,124 [INFO] tensorflow: global_step/sec: 3.09143\n",
"INFO:tensorflow:epoch = 71.77083333333333, learning_rate = 0.0009999999, loss = 0.00022285213, step = 6890 (5.497 sec)\n",
"2021-12-31 02:22:36,747 [INFO] tensorflow: epoch = 71.77083333333333, learning_rate = 0.0009999999, loss = 0.00022285213, step = 6890 (5.497 sec)\n",
"INFO:tensorflow:global_step/sec: 3.05994\n",
"2021-12-31 02:22:38,065 [INFO] tensorflow: global_step/sec: 3.05994\n",
"2021-12-31 02:22:39,700 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.618\n",
"INFO:tensorflow:global_step/sec: 3.07952\n",
"2021-12-31 02:22:40,988 [INFO] tensorflow: global_step/sec: 3.07952\n",
"INFO:tensorflow:epoch = 71.94791666666666, learning_rate = 0.0009999999, loss = 0.00020742175, step = 6907 (5.538 sec)\n",
"2021-12-31 02:22:42,285 [INFO] tensorflow: epoch = 71.94791666666666, learning_rate = 0.0009999999, loss = 0.00020742175, step = 6907 (5.538 sec)\n",
"INFO:tensorflow:global_step/sec: 3.1134\n",
"2021-12-31 02:22:43,878 [INFO] tensorflow: global_step/sec: 3.1134\n",
"2021-12-31 02:22:43,879 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 72/120: loss: 0.00023 learning rate: 0.00100 Time taken: 0:00:30.909193 ETA: 0:24:43.641266\n",
"INFO:tensorflow:global_step/sec: 3.11684\n",
"2021-12-31 02:22:46,766 [INFO] tensorflow: global_step/sec: 3.11684\n",
"INFO:tensorflow:epoch = 72.125, learning_rate = 0.0009999999, loss = 0.00021150528, step = 6924 (5.432 sec)\n",
"2021-12-31 02:22:47,717 [INFO] tensorflow: epoch = 72.125, learning_rate = 0.0009999999, loss = 0.00021150528, step = 6924 (5.432 sec)\n",
"2021-12-31 02:22:47,717 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.945\n",
"INFO:tensorflow:global_step/sec: 3.11654\n",
"2021-12-31 02:22:49,654 [INFO] tensorflow: global_step/sec: 3.11654\n",
"INFO:tensorflow:global_step/sec: 3.11434\n",
"2021-12-31 02:22:52,543 [INFO] tensorflow: global_step/sec: 3.11434\n",
"INFO:tensorflow:epoch = 72.30208333333333, learning_rate = 0.0009999999, loss = 0.00019201403, step = 6941 (5.448 sec)\n",
"2021-12-31 02:22:53,165 [INFO] tensorflow: epoch = 72.30208333333333, learning_rate = 0.0009999999, loss = 0.00019201403, step = 6941 (5.448 sec)\n",
"INFO:tensorflow:global_step/sec: 3.13196\n",
"2021-12-31 02:22:55,417 [INFO] tensorflow: global_step/sec: 3.13196\n",
"2021-12-31 02:22:55,738 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.938\n",
"INFO:tensorflow:global_step/sec: 3.0638\n",
"2021-12-31 02:22:58,355 [INFO] tensorflow: global_step/sec: 3.0638\n",
"INFO:tensorflow:epoch = 72.47916666666666, learning_rate = 0.0009999999, loss = 0.00027688072, step = 6958 (5.509 sec)\n",
"2021-12-31 02:22:58,675 [INFO] tensorflow: epoch = 72.47916666666666, learning_rate = 0.0009999999, loss = 0.00027688072, step = 6958 (5.509 sec)\n",
"INFO:tensorflow:global_step/sec: 3.04409\n",
"2021-12-31 02:23:01,311 [INFO] tensorflow: global_step/sec: 3.04409\n",
"2021-12-31 02:23:03,915 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.458\n",
"INFO:tensorflow:epoch = 72.65625, learning_rate = 0.0009999999, loss = 0.00019999848, step = 6975 (5.553 sec)\n",
"2021-12-31 02:23:04,228 [INFO] tensorflow: epoch = 72.65625, learning_rate = 0.0009999999, loss = 0.00019999848, step = 6975 (5.553 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08501\n",
"2021-12-31 02:23:04,228 [INFO] tensorflow: global_step/sec: 3.08501\n",
"INFO:tensorflow:global_step/sec: 3.08047\n",
"2021-12-31 02:23:07,150 [INFO] tensorflow: global_step/sec: 3.08047\n",
"INFO:tensorflow:epoch = 72.83333333333333, learning_rate = 0.0009999999, loss = 0.00021374633, step = 6992 (5.490 sec)\n",
"2021-12-31 02:23:09,718 [INFO] tensorflow: epoch = 72.83333333333333, learning_rate = 0.0009999999, loss = 0.00021374633, step = 6992 (5.490 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10401\n",
"2021-12-31 02:23:10,050 [INFO] tensorflow: global_step/sec: 3.10401\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"2021-12-31 02:23:11,967 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.840\n",
"INFO:tensorflow:global_step/sec: 3.10821\n",
"2021-12-31 02:23:12,945 [INFO] tensorflow: global_step/sec: 3.10821\n",
"2021-12-31 02:23:14,888 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 73/120: loss: 0.00019 learning rate: 0.00100 Time taken: 0:00:31.003034 ETA: 0:24:17.142592\n",
"INFO:tensorflow:epoch = 73.01041666666666, learning_rate = 0.0009999999, loss = 0.00026072646, step = 7009 (5.510 sec)\n",
"2021-12-31 02:23:15,227 [INFO] tensorflow: epoch = 73.01041666666666, learning_rate = 0.0009999999, loss = 0.00026072646, step = 7009 (5.510 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11509\n",
"2021-12-31 02:23:15,834 [INFO] tensorflow: global_step/sec: 3.11509\n",
"INFO:tensorflow:global_step/sec: 3.11245\n",
"2021-12-31 02:23:18,726 [INFO] tensorflow: global_step/sec: 3.11245\n",
"2021-12-31 02:23:20,023 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.828\n",
"INFO:tensorflow:epoch = 73.1875, learning_rate = 0.0009999999, loss = 0.00025162377, step = 7026 (5.443 sec)\n",
"2021-12-31 02:23:20,671 [INFO] tensorflow: epoch = 73.1875, learning_rate = 0.0009999999, loss = 0.00025162377, step = 7026 (5.443 sec)\n",
"INFO:tensorflow:global_step/sec: 3.14483\n",
"2021-12-31 02:23:21,588 [INFO] tensorflow: global_step/sec: 3.14483\n",
"INFO:tensorflow:global_step/sec: 3.10645\n",
"2021-12-31 02:23:24,485 [INFO] tensorflow: global_step/sec: 3.10645\n",
"INFO:tensorflow:epoch = 73.36458333333333, learning_rate = 0.0009999999, loss = 0.00030881102, step = 7043 (5.420 sec)\n",
"2021-12-31 02:23:26,090 [INFO] tensorflow: epoch = 73.36458333333333, learning_rate = 0.0009999999, loss = 0.00030881102, step = 7043 (5.420 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08788\n",
"2021-12-31 02:23:27,400 [INFO] tensorflow: global_step/sec: 3.08788\n",
"2021-12-31 02:23:28,053 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.906\n",
"INFO:tensorflow:global_step/sec: 3.04795\n",
"2021-12-31 02:23:30,352 [INFO] tensorflow: global_step/sec: 3.04795\n",
"INFO:tensorflow:epoch = 73.54166666666666, learning_rate = 0.0009999999, loss = 0.00018104416, step = 7060 (5.562 sec)\n",
"2021-12-31 02:23:31,653 [INFO] tensorflow: epoch = 73.54166666666666, learning_rate = 0.0009999999, loss = 0.00018104416, step = 7060 (5.562 sec)\n",
"INFO:tensorflow:global_step/sec: 3.02083\n",
"2021-12-31 02:23:33,332 [INFO] tensorflow: global_step/sec: 3.02083\n",
"INFO:tensorflow:global_step/sec: 3.14696\n",
"2021-12-31 02:23:36,192 [INFO] tensorflow: global_step/sec: 3.14696\n",
"2021-12-31 02:23:36,192 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.575\n",
"INFO:tensorflow:epoch = 73.71875, learning_rate = 0.0009999999, loss = 0.00022318763, step = 7077 (5.470 sec)\n",
"2021-12-31 02:23:37,123 [INFO] tensorflow: epoch = 73.71875, learning_rate = 0.0009999999, loss = 0.00022318763, step = 7077 (5.470 sec)\n",
"INFO:tensorflow:global_step/sec: 3.1766\n",
"2021-12-31 02:23:39,025 [INFO] tensorflow: global_step/sec: 3.1766\n",
"INFO:tensorflow:global_step/sec: 3.08531\n",
"2021-12-31 02:23:41,942 [INFO] tensorflow: global_step/sec: 3.08531\n",
"INFO:tensorflow:epoch = 73.89583333333333, learning_rate = 0.0009999999, loss = 0.00023189891, step = 7094 (5.475 sec)\n",
"2021-12-31 02:23:42,598 [INFO] tensorflow: epoch = 73.89583333333333, learning_rate = 0.0009999999, loss = 0.00023189891, step = 7094 (5.475 sec)\n",
"2021-12-31 02:23:44,213 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.936\n",
"INFO:tensorflow:global_step/sec: 3.08362\n",
"2021-12-31 02:23:44,860 [INFO] tensorflow: global_step/sec: 3.08362\n",
"2021-12-31 02:23:45,819 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 74/120: loss: 0.00014 learning rate: 0.00100 Time taken: 0:00:30.923106 ETA: 0:23:42.462874\n",
"INFO:tensorflow:global_step/sec: 3.0925\n",
"2021-12-31 02:23:47,771 [INFO] tensorflow: global_step/sec: 3.0925\n",
"INFO:tensorflow:epoch = 74.07291666666666, learning_rate = 0.0009999999, loss = 0.00020662685, step = 7111 (5.483 sec)\n",
"2021-12-31 02:23:48,081 [INFO] tensorflow: epoch = 74.07291666666666, learning_rate = 0.0009999999, loss = 0.00020662685, step = 7111 (5.483 sec)\n",
"INFO:tensorflow:global_step/sec: 3.14879\n",
"2021-12-31 02:23:50,629 [INFO] tensorflow: global_step/sec: 3.14879\n",
"2021-12-31 02:23:52,222 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.971\n",
"INFO:tensorflow:epoch = 74.25, learning_rate = 0.0009999999, loss = 0.00031725038, step = 7128 (5.463 sec)\n",
"2021-12-31 02:23:53,544 [INFO] tensorflow: epoch = 74.25, learning_rate = 0.0009999999, loss = 0.00031725038, step = 7128 (5.463 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08646\n",
"2021-12-31 02:23:53,545 [INFO] tensorflow: global_step/sec: 3.08646\n",
"INFO:tensorflow:global_step/sec: 3.1072\n",
"2021-12-31 02:23:56,441 [INFO] tensorflow: global_step/sec: 3.1072\n",
"INFO:tensorflow:epoch = 74.42708333333333, learning_rate = 0.0009999999, loss = 0.00023927617, step = 7145 (5.447 sec)\n",
"2021-12-31 02:23:58,991 [INFO] tensorflow: epoch = 74.42708333333333, learning_rate = 0.0009999999, loss = 0.00023927617, step = 7145 (5.447 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12716\n",
"2021-12-31 02:23:59,319 [INFO] tensorflow: global_step/sec: 3.12716\n",
"2021-12-31 02:24:00,269 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.857\n",
"INFO:tensorflow:global_step/sec: 3.12529\n",
"2021-12-31 02:24:02,199 [INFO] tensorflow: global_step/sec: 3.12529\n",
"INFO:tensorflow:epoch = 74.60416666666666, learning_rate = 0.0009999999, loss = 0.00018308447, step = 7162 (5.458 sec)\n",
"2021-12-31 02:24:04,449 [INFO] tensorflow: epoch = 74.60416666666666, learning_rate = 0.0009999999, loss = 0.00018308447, step = 7162 (5.458 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10644\n",
"2021-12-31 02:24:05,096 [INFO] tensorflow: global_step/sec: 3.10644\n",
"INFO:tensorflow:global_step/sec: 3.08842\n",
"2021-12-31 02:24:08,011 [INFO] tensorflow: global_step/sec: 3.08842\n",
"2021-12-31 02:24:08,331 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.805\n",
"INFO:tensorflow:epoch = 74.78125, learning_rate = 0.0009999999, loss = 0.00025437886, step = 7179 (5.471 sec)\n",
"2021-12-31 02:24:09,920 [INFO] tensorflow: epoch = 74.78125, learning_rate = 0.0009999999, loss = 0.00025437886, step = 7179 (5.471 sec)\n",
"INFO:tensorflow:global_step/sec: 3.13779\n",
"2021-12-31 02:24:10,879 [INFO] tensorflow: global_step/sec: 3.13779\n",
"INFO:tensorflow:global_step/sec: 3.09219\n",
"2021-12-31 02:24:13,789 [INFO] tensorflow: global_step/sec: 3.09219\n",
"INFO:tensorflow:epoch = 74.95833333333333, learning_rate = 0.0009999999, loss = 0.00022354284, step = 7196 (5.433 sec)\n",
"2021-12-31 02:24:15,353 [INFO] tensorflow: epoch = 74.95833333333333, learning_rate = 0.0009999999, loss = 0.00022354284, step = 7196 (5.433 sec)\n",
"2021-12-31 02:24:16,324 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.024\n",
"INFO:tensorflow:global_step/sec: 3.15685\n",
"2021-12-31 02:24:16,640 [INFO] tensorflow: global_step/sec: 3.15685\n",
"2021-12-31 02:24:16,641 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 75/120: loss: 0.00022 learning rate: 0.00100 Time taken: 0:00:30.833430 ETA: 0:23:07.504342\n",
"INFO:tensorflow:global_step/sec: 3.15708\n",
"2021-12-31 02:24:19,491 [INFO] tensorflow: global_step/sec: 3.15708\n",
"INFO:tensorflow:epoch = 75.13541666666666, learning_rate = 0.0009999999, loss = 0.00021582936, step = 7213 (5.446 sec)\n",
"2021-12-31 02:24:20,799 [INFO] tensorflow: epoch = 75.13541666666666, learning_rate = 0.0009999999, loss = 0.00021582936, step = 7213 (5.446 sec)\n",
"INFO:tensorflow:global_step/sec: 3.13105\n",
"2021-12-31 02:24:22,365 [INFO] tensorflow: global_step/sec: 3.13105\n",
"2021-12-31 02:24:24,273 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.162\n",
"INFO:tensorflow:global_step/sec: 3.12321\n",
"2021-12-31 02:24:25,247 [INFO] tensorflow: global_step/sec: 3.12321\n",
"INFO:tensorflow:epoch = 75.3125, learning_rate = 0.0009999999, loss = 0.00021144823, step = 7230 (5.407 sec)\n",
"2021-12-31 02:24:26,206 [INFO] tensorflow: epoch = 75.3125, learning_rate = 0.0009999999, loss = 0.00021144823, step = 7230 (5.407 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09894\n",
"2021-12-31 02:24:28,151 [INFO] tensorflow: global_step/sec: 3.09894\n",
"INFO:tensorflow:global_step/sec: 3.14173\n",
"2021-12-31 02:24:31,016 [INFO] tensorflow: global_step/sec: 3.14173\n",
"INFO:tensorflow:epoch = 75.48958333333333, learning_rate = 0.0009999999, loss = 0.0001849018, step = 7247 (5.454 sec)\n",
"2021-12-31 02:24:31,661 [INFO] tensorflow: epoch = 75.48958333333333, learning_rate = 0.0009999999, loss = 0.0001849018, step = 7247 (5.454 sec)\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"2021-12-31 02:24:32,300 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.917\n",
"INFO:tensorflow:global_step/sec: 3.1235\n",
"2021-12-31 02:24:33,897 [INFO] tensorflow: global_step/sec: 3.1235\n",
"INFO:tensorflow:global_step/sec: 3.09443\n",
"2021-12-31 02:24:36,806 [INFO] tensorflow: global_step/sec: 3.09443\n",
"INFO:tensorflow:epoch = 75.66666666666666, learning_rate = 0.0009999999, loss = 0.00021688137, step = 7264 (5.465 sec)\n",
"2021-12-31 02:24:37,125 [INFO] tensorflow: epoch = 75.66666666666666, learning_rate = 0.0009999999, loss = 0.00021688137, step = 7264 (5.465 sec)\n",
"INFO:tensorflow:global_step/sec: 3.1361\n",
"2021-12-31 02:24:39,676 [INFO] tensorflow: global_step/sec: 3.1361\n",
"2021-12-31 02:24:40,322 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.931\n",
"INFO:tensorflow:epoch = 75.84375, learning_rate = 0.0009999999, loss = 0.000215515, step = 7281 (5.408 sec)\n",
"2021-12-31 02:24:42,533 [INFO] tensorflow: epoch = 75.84375, learning_rate = 0.0009999999, loss = 0.000215515, step = 7281 (5.408 sec)\n",
"INFO:tensorflow:global_step/sec: 3.14871\n",
"2021-12-31 02:24:42,534 [INFO] tensorflow: global_step/sec: 3.14871\n",
"INFO:tensorflow:global_step/sec: 3.10189\n",
"2021-12-31 02:24:45,435 [INFO] tensorflow: global_step/sec: 3.10189\n",
"2021-12-31 02:24:47,428 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 76/120: loss: 0.00019 learning rate: 0.00100 Time taken: 0:00:30.774911 ETA: 0:22:34.096102\n",
"INFO:tensorflow:epoch = 76.02083333333333, learning_rate = 0.0009999999, loss = 0.00022375566, step = 7298 (5.532 sec)\n",
"2021-12-31 02:24:48,065 [INFO] tensorflow: epoch = 76.02083333333333, learning_rate = 0.0009999999, loss = 0.00022375566, step = 7298 (5.532 sec)\n",
"INFO:tensorflow:global_step/sec: 3.0508\n",
"2021-12-31 02:24:48,385 [INFO] tensorflow: global_step/sec: 3.0508\n",
"2021-12-31 02:24:48,386 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.803\n",
"INFO:tensorflow:global_step/sec: 3.13527\n",
"2021-12-31 02:24:51,256 [INFO] tensorflow: global_step/sec: 3.13527\n",
"INFO:tensorflow:epoch = 76.19791666666666, learning_rate = 0.0009999999, loss = 0.00027969066, step = 7315 (5.455 sec)\n",
"2021-12-31 02:24:53,520 [INFO] tensorflow: epoch = 76.19791666666666, learning_rate = 0.0009999999, loss = 0.00027969066, step = 7315 (5.455 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07234\n",
"2021-12-31 02:24:54,185 [INFO] tensorflow: global_step/sec: 3.07234\n",
"2021-12-31 02:24:56,428 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.872\n",
"INFO:tensorflow:global_step/sec: 3.13032\n",
"2021-12-31 02:24:57,060 [INFO] tensorflow: global_step/sec: 3.13032\n",
"INFO:tensorflow:epoch = 76.375, learning_rate = 0.0009999999, loss = 0.00020730004, step = 7332 (5.532 sec)\n",
"2021-12-31 02:24:59,052 [INFO] tensorflow: epoch = 76.375, learning_rate = 0.0009999999, loss = 0.00020730004, step = 7332 (5.532 sec)\n",
"INFO:tensorflow:global_step/sec: 3.04014\n",
"2021-12-31 02:25:00,021 [INFO] tensorflow: global_step/sec: 3.04014\n",
"INFO:tensorflow:global_step/sec: 3.06704\n",
"2021-12-31 02:25:02,955 [INFO] tensorflow: global_step/sec: 3.06704\n",
"INFO:tensorflow:epoch = 76.55208333333333, learning_rate = 0.0009999999, loss = 0.0002058598, step = 7349 (5.507 sec)\n",
"2021-12-31 02:25:04,559 [INFO] tensorflow: epoch = 76.55208333333333, learning_rate = 0.0009999999, loss = 0.0002058598, step = 7349 (5.507 sec)\n",
"2021-12-31 02:25:04,559 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.595\n",
"INFO:tensorflow:global_step/sec: 3.08539\n",
"2021-12-31 02:25:05,872 [INFO] tensorflow: global_step/sec: 3.08539\n",
"INFO:tensorflow:global_step/sec: 3.14172\n",
"2021-12-31 02:25:08,737 [INFO] tensorflow: global_step/sec: 3.14172\n",
"INFO:tensorflow:epoch = 76.72916666666666, learning_rate = 0.0009999999, loss = 0.00019932848, step = 7366 (5.499 sec)\n",
"2021-12-31 02:25:10,059 [INFO] tensorflow: epoch = 76.72916666666666, learning_rate = 0.0009999999, loss = 0.00019932848, step = 7366 (5.499 sec)\n",
"INFO:tensorflow:global_step/sec: 3.05893\n",
"2021-12-31 02:25:11,679 [INFO] tensorflow: global_step/sec: 3.05893\n",
"2021-12-31 02:25:12,633 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.774\n",
"INFO:tensorflow:global_step/sec: 3.10332\n",
"2021-12-31 02:25:14,579 [INFO] tensorflow: global_step/sec: 3.10332\n",
"INFO:tensorflow:epoch = 76.90625, learning_rate = 0.0009999999, loss = 0.00017971646, step = 7383 (5.490 sec)\n",
"2021-12-31 02:25:15,548 [INFO] tensorflow: epoch = 76.90625, learning_rate = 0.0009999999, loss = 0.00017971646, step = 7383 (5.490 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07069\n",
"2021-12-31 02:25:17,510 [INFO] tensorflow: global_step/sec: 3.07069\n",
"2021-12-31 02:25:18,448 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 77/120: loss: 0.00021 learning rate: 0.00100 Time taken: 0:00:31.029880 ETA: 0:22:14.284822\n",
"INFO:tensorflow:global_step/sec: 3.14903\n",
"2021-12-31 02:25:20,368 [INFO] tensorflow: global_step/sec: 3.14903\n",
"2021-12-31 02:25:20,700 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.792\n",
"INFO:tensorflow:epoch = 77.08333333333333, learning_rate = 0.0009999999, loss = 0.00019587143, step = 7400 (5.479 sec)\n",
"2021-12-31 02:25:21,027 [INFO] tensorflow: epoch = 77.08333333333333, learning_rate = 0.0009999999, loss = 0.00019587143, step = 7400 (5.479 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10895\n",
"2021-12-31 02:25:23,263 [INFO] tensorflow: global_step/sec: 3.10895\n",
"INFO:tensorflow:global_step/sec: 3.0908\n",
"2021-12-31 02:25:26,175 [INFO] tensorflow: global_step/sec: 3.0908\n",
"INFO:tensorflow:epoch = 77.26041666666666, learning_rate = 0.0009999999, loss = 0.00018863237, step = 7417 (5.494 sec)\n",
"2021-12-31 02:25:26,521 [INFO] tensorflow: epoch = 77.26041666666666, learning_rate = 0.0009999999, loss = 0.00018863237, step = 7417 (5.494 sec)\n",
"2021-12-31 02:25:28,789 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.725\n",
"INFO:tensorflow:global_step/sec: 3.05403\n",
"2021-12-31 02:25:29,122 [INFO] tensorflow: global_step/sec: 3.05403\n",
"INFO:tensorflow:epoch = 77.4375, learning_rate = 0.0009999999, loss = 0.00013236656, step = 7434 (5.529 sec)\n",
"2021-12-31 02:25:32,050 [INFO] tensorflow: epoch = 77.4375, learning_rate = 0.0009999999, loss = 0.00013236656, step = 7434 (5.529 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07269\n",
"2021-12-31 02:25:32,051 [INFO] tensorflow: global_step/sec: 3.07269\n",
"INFO:tensorflow:global_step/sec: 3.09571\n",
"2021-12-31 02:25:34,958 [INFO] tensorflow: global_step/sec: 3.09571\n",
"2021-12-31 02:25:36,852 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.804\n",
"INFO:tensorflow:epoch = 77.61458333333333, learning_rate = 0.0009999999, loss = 0.00024133905, step = 7451 (5.451 sec)\n",
"2021-12-31 02:25:37,501 [INFO] tensorflow: epoch = 77.61458333333333, learning_rate = 0.0009999999, loss = 0.00024133905, step = 7451 (5.451 sec)\n",
"INFO:tensorflow:global_step/sec: 3.13397\n",
"2021-12-31 02:25:37,830 [INFO] tensorflow: global_step/sec: 3.13397\n",
"INFO:tensorflow:global_step/sec: 3.15026\n",
"2021-12-31 02:25:40,687 [INFO] tensorflow: global_step/sec: 3.15026\n",
"INFO:tensorflow:epoch = 77.79166666666666, learning_rate = 0.0009999999, loss = 0.00023588668, step = 7468 (5.423 sec)\n",
"2021-12-31 02:25:42,923 [INFO] tensorflow: epoch = 77.79166666666666, learning_rate = 0.0009999999, loss = 0.00023588668, step = 7468 (5.423 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12899\n",
"2021-12-31 02:25:43,563 [INFO] tensorflow: global_step/sec: 3.12899\n",
"2021-12-31 02:25:44,852 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.003\n",
"INFO:tensorflow:global_step/sec: 3.12752\n",
"2021-12-31 02:25:46,441 [INFO] tensorflow: global_step/sec: 3.12752\n",
"INFO:tensorflow:epoch = 77.96875, learning_rate = 0.0009999999, loss = 0.00026528764, step = 7485 (5.492 sec)\n",
"2021-12-31 02:25:48,415 [INFO] tensorflow: epoch = 77.96875, learning_rate = 0.0009999999, loss = 0.00026528764, step = 7485 (5.492 sec)\n",
"INFO:tensorflow:global_step/sec: 3.13482\n",
"2021-12-31 02:25:49,312 [INFO] tensorflow: global_step/sec: 3.13482\n",
"2021-12-31 02:25:49,313 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 78/120: loss: 0.00024 learning rate: 0.00100 Time taken: 0:00:30.881901 ETA: 0:21:37.039823\n",
"INFO:tensorflow:global_step/sec: 3.09709\n",
"2021-12-31 02:25:52,218 [INFO] tensorflow: global_step/sec: 3.09709\n",
"2021-12-31 02:25:52,852 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.999\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO:tensorflow:epoch = 78.14583333333333, learning_rate = 0.0009999999, loss = 0.00024218127, step = 7502 (5.435 sec)\n",
"2021-12-31 02:25:53,850 [INFO] tensorflow: epoch = 78.14583333333333, learning_rate = 0.0009999999, loss = 0.00024218127, step = 7502 (5.435 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09891\n",
"2021-12-31 02:25:55,122 [INFO] tensorflow: global_step/sec: 3.09891\n",
"INFO:tensorflow:global_step/sec: 3.09088\n",
"2021-12-31 02:25:58,034 [INFO] tensorflow: global_step/sec: 3.09088\n",
"INFO:tensorflow:epoch = 78.32291666666666, learning_rate = 0.0009999999, loss = 0.00016379161, step = 7519 (5.444 sec)\n",
"2021-12-31 02:25:59,295 [INFO] tensorflow: epoch = 78.32291666666666, learning_rate = 0.0009999999, loss = 0.00016379161, step = 7519 (5.444 sec)\n",
"INFO:tensorflow:global_step/sec: 3.15911\n",
"2021-12-31 02:26:00,883 [INFO] tensorflow: global_step/sec: 3.15911\n",
"2021-12-31 02:26:00,883 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.904\n",
"INFO:tensorflow:global_step/sec: 3.1568\n",
"2021-12-31 02:26:03,734 [INFO] tensorflow: global_step/sec: 3.1568\n",
"INFO:tensorflow:epoch = 78.5, learning_rate = 0.0009999999, loss = 0.00015875562, step = 7536 (5.401 sec)\n",
"2021-12-31 02:26:04,696 [INFO] tensorflow: epoch = 78.5, learning_rate = 0.0009999999, loss = 0.00015875562, step = 7536 (5.401 sec)\n",
"INFO:tensorflow:global_step/sec: 3.14879\n",
"2021-12-31 02:26:06,592 [INFO] tensorflow: global_step/sec: 3.14879\n",
"2021-12-31 02:26:08,808 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.239\n",
"INFO:tensorflow:global_step/sec: 3.15303\n",
"2021-12-31 02:26:09,446 [INFO] tensorflow: global_step/sec: 3.15303\n",
"INFO:tensorflow:epoch = 78.67708333333333, learning_rate = 0.0009999999, loss = 0.00021253974, step = 7553 (5.407 sec)\n",
"2021-12-31 02:26:10,103 [INFO] tensorflow: epoch = 78.67708333333333, learning_rate = 0.0009999999, loss = 0.00021253974, step = 7553 (5.407 sec)\n",
"INFO:tensorflow:global_step/sec: 3.14596\n",
"2021-12-31 02:26:12,307 [INFO] tensorflow: global_step/sec: 3.14596\n",
"INFO:tensorflow:global_step/sec: 3.14081\n",
"2021-12-31 02:26:15,173 [INFO] tensorflow: global_step/sec: 3.14081\n",
"INFO:tensorflow:epoch = 78.85416666666666, learning_rate = 0.0009999999, loss = 0.00021158463, step = 7570 (5.361 sec)\n",
"2021-12-31 02:26:15,464 [INFO] tensorflow: epoch = 78.85416666666666, learning_rate = 0.0009999999, loss = 0.00021158463, step = 7570 (5.361 sec)\n",
"2021-12-31 02:26:16,705 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.327\n",
"INFO:tensorflow:global_step/sec: 3.18357\n",
"2021-12-31 02:26:18,000 [INFO] tensorflow: global_step/sec: 3.18357\n",
"2021-12-31 02:26:19,931 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 79/120: loss: 0.00020 learning rate: 0.00100 Time taken: 0:00:30.598691 ETA: 0:20:54.546340\n",
"INFO:tensorflow:epoch = 79.03125, learning_rate = 0.0009999999, loss = 0.0003108891, step = 7587 (5.427 sec)\n",
"2021-12-31 02:26:20,891 [INFO] tensorflow: epoch = 79.03125, learning_rate = 0.0009999999, loss = 0.0003108891, step = 7587 (5.427 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11185\n",
"2021-12-31 02:26:20,892 [INFO] tensorflow: global_step/sec: 3.11185\n",
"INFO:tensorflow:global_step/sec: 3.1108\n",
"2021-12-31 02:26:23,785 [INFO] tensorflow: global_step/sec: 3.1108\n",
"2021-12-31 02:26:24,719 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.957\n",
"INFO:tensorflow:epoch = 79.20833333333333, learning_rate = 0.0009999999, loss = 0.0002492683, step = 7604 (5.403 sec)\n",
"2021-12-31 02:26:26,294 [INFO] tensorflow: epoch = 79.20833333333333, learning_rate = 0.0009999999, loss = 0.0002492683, step = 7604 (5.403 sec)\n",
"INFO:tensorflow:global_step/sec: 3.17975\n",
"2021-12-31 02:26:26,615 [INFO] tensorflow: global_step/sec: 3.17975\n",
"INFO:tensorflow:global_step/sec: 3.17175\n",
"2021-12-31 02:26:29,453 [INFO] tensorflow: global_step/sec: 3.17175\n",
"INFO:tensorflow:epoch = 79.38541666666666, learning_rate = 0.0009999999, loss = 0.00017719714, step = 7621 (5.383 sec)\n",
"2021-12-31 02:26:31,677 [INFO] tensorflow: epoch = 79.38541666666666, learning_rate = 0.0009999999, loss = 0.00017719714, step = 7621 (5.383 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12532\n",
"2021-12-31 02:26:32,333 [INFO] tensorflow: global_step/sec: 3.12532\n",
"2021-12-31 02:26:32,646 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.229\n",
"INFO:tensorflow:global_step/sec: 3.13095\n",
"2021-12-31 02:26:35,207 [INFO] tensorflow: global_step/sec: 3.13095\n",
"INFO:tensorflow:epoch = 79.5625, learning_rate = 0.0009999999, loss = 0.0002237866, step = 7638 (5.494 sec)\n",
"2021-12-31 02:26:37,171 [INFO] tensorflow: epoch = 79.5625, learning_rate = 0.0009999999, loss = 0.0002237866, step = 7638 (5.494 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08493\n",
"2021-12-31 02:26:38,125 [INFO] tensorflow: global_step/sec: 3.08493\n",
"2021-12-31 02:26:40,731 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.740\n",
"INFO:tensorflow:global_step/sec: 3.07993\n",
"2021-12-31 02:26:41,047 [INFO] tensorflow: global_step/sec: 3.07993\n",
"INFO:tensorflow:epoch = 79.73958333333333, learning_rate = 0.0009999999, loss = 0.00025305708, step = 7655 (5.488 sec)\n",
"2021-12-31 02:26:42,659 [INFO] tensorflow: epoch = 79.73958333333333, learning_rate = 0.0009999999, loss = 0.00025305708, step = 7655 (5.488 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12893\n",
"2021-12-31 02:26:43,923 [INFO] tensorflow: global_step/sec: 3.12893\n",
"INFO:tensorflow:global_step/sec: 3.10116\n",
"2021-12-31 02:26:46,825 [INFO] tensorflow: global_step/sec: 3.10116\n",
"INFO:tensorflow:epoch = 79.91666666666666, learning_rate = 0.0009999999, loss = 0.0002140813, step = 7672 (5.460 sec)\n",
"2021-12-31 02:26:48,119 [INFO] tensorflow: epoch = 79.91666666666666, learning_rate = 0.0009999999, loss = 0.0002140813, step = 7672 (5.460 sec)\n",
"2021-12-31 02:26:48,764 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.896\n",
"INFO:tensorflow:global_step/sec: 3.12381\n",
"2021-12-31 02:26:49,706 [INFO] tensorflow: global_step/sec: 3.12381\n",
"INFO:tensorflow:Saving checkpoints for step-7680.\n",
"2021-12-31 02:26:50,333 [INFO] tensorflow: Saving checkpoints for step-7680.\n",
"WARNING:tensorflow:Ignoring: /tmp/tmp54z1zg9o; No such file or directory\n",
"2021-12-31 02:26:50,482 [WARNING] tensorflow: Ignoring: /tmp/tmp54z1zg9o; No such file or directory\n",
"2021-12-31 02:26:53,954 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 23, 0.00s/step\n",
"2021-12-31 02:26:55,701 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 23, 0.17s/step\n",
"2021-12-31 02:26:57,540 [INFO] iva.detectnet_v2.evaluation.evaluation: step 20 / 23, 0.18s/step\n",
"Matching predictions to ground truth, class 1/1.: 100%|█| 1593/1593 [00:00<00:00, 14886.01it/s]\n",
"Epoch 80/120\n",
"=========================\n",
"\n",
"Validation cost: 0.000226\n",
"Mean average_precision (in %): 81.9021\n",
"\n",
"class name average precision (in %)\n",
"------------ --------------------------\n",
"car 81.9021\n",
"\n",
"Median Inference Time: 0.016351\n",
"INFO:tensorflow:epoch = 80.0, learning_rate = 0.0009999999, loss = 0.00023144718, step = 7680 (10.385 sec)\n",
"2021-12-31 02:26:58,505 [INFO] tensorflow: epoch = 80.0, learning_rate = 0.0009999999, loss = 0.00023144718, step = 7680 (10.385 sec)\n",
"2021-12-31 02:26:58,505 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 80/120: loss: 0.00023 learning rate: 0.00100 Time taken: 0:00:38.568806 ETA: 0:25:42.752247\n",
"INFO:tensorflow:global_step/sec: 0.839065\n",
"2021-12-31 02:27:00,433 [INFO] tensorflow: global_step/sec: 0.839065\n",
"INFO:tensorflow:global_step/sec: 3.13072\n",
"2021-12-31 02:27:03,307 [INFO] tensorflow: global_step/sec: 3.13072\n",
"INFO:tensorflow:epoch = 80.17708333333333, learning_rate = 0.0009999999, loss = 0.00021025767, step = 7697 (5.452 sec)\n",
"2021-12-31 02:27:03,957 [INFO] tensorflow: epoch = 80.17708333333333, learning_rate = 0.0009999999, loss = 0.00021025767, step = 7697 (5.452 sec)\n",
"2021-12-31 02:27:04,592 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 12.636\n",
"INFO:tensorflow:global_step/sec: 3.09354\n",
"2021-12-31 02:27:06,217 [INFO] tensorflow: global_step/sec: 3.09354\n",
"INFO:tensorflow:global_step/sec: 3.10361\n",
"2021-12-31 02:27:09,116 [INFO] tensorflow: global_step/sec: 3.10361\n",
"INFO:tensorflow:epoch = 80.35416666666666, learning_rate = 0.0009999999, loss = 0.00023289569, step = 7714 (5.480 sec)\n",
"2021-12-31 02:27:09,438 [INFO] tensorflow: epoch = 80.35416666666666, learning_rate = 0.0009999999, loss = 0.00023289569, step = 7714 (5.480 sec)\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO:tensorflow:global_step/sec: 3.17417\n",
"2021-12-31 02:27:11,952 [INFO] tensorflow: global_step/sec: 3.17417\n",
"2021-12-31 02:27:12,572 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.066\n",
"INFO:tensorflow:epoch = 80.53125, learning_rate = 0.0009999999, loss = 0.00025123367, step = 7731 (5.353 sec)\n",
"2021-12-31 02:27:14,790 [INFO] tensorflow: epoch = 80.53125, learning_rate = 0.0009999999, loss = 0.00025123367, step = 7731 (5.353 sec)\n",
"INFO:tensorflow:global_step/sec: 3.16977\n",
"2021-12-31 02:27:14,791 [INFO] tensorflow: global_step/sec: 3.16977\n",
"INFO:tensorflow:global_step/sec: 3.15422\n",
"2021-12-31 02:27:17,644 [INFO] tensorflow: global_step/sec: 3.15422\n",
"INFO:tensorflow:epoch = 80.70833333333333, learning_rate = 0.0009999999, loss = 0.00019312736, step = 7748 (5.394 sec)\n",
"2021-12-31 02:27:20,184 [INFO] tensorflow: epoch = 80.70833333333333, learning_rate = 0.0009999999, loss = 0.00019312736, step = 7748 (5.394 sec)\n",
"INFO:tensorflow:global_step/sec: 3.14689\n",
"2021-12-31 02:27:20,504 [INFO] tensorflow: global_step/sec: 3.14689\n",
"2021-12-31 02:27:20,505 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.210\n",
"INFO:tensorflow:global_step/sec: 3.151\n",
"2021-12-31 02:27:23,361 [INFO] tensorflow: global_step/sec: 3.151\n",
"INFO:tensorflow:epoch = 80.88541666666666, learning_rate = 0.0009999999, loss = 0.00022258326, step = 7765 (5.359 sec)\n",
"2021-12-31 02:27:25,543 [INFO] tensorflow: epoch = 80.88541666666666, learning_rate = 0.0009999999, loss = 0.00022258326, step = 7765 (5.359 sec)\n",
"INFO:tensorflow:global_step/sec: 3.18746\n",
"2021-12-31 02:27:26,184 [INFO] tensorflow: global_step/sec: 3.18746\n",
"2021-12-31 02:27:28,384 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.384\n",
"INFO:tensorflow:global_step/sec: 3.16679\n",
"2021-12-31 02:27:29,026 [INFO] tensorflow: global_step/sec: 3.16679\n",
"2021-12-31 02:27:29,027 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 81/120: loss: 0.00021 learning rate: 0.00100 Time taken: 0:00:30.531827 ETA: 0:19:50.741271\n",
"INFO:tensorflow:epoch = 81.0625, learning_rate = 0.0009999999, loss = 0.00020768006, step = 7782 (5.409 sec)\n",
"2021-12-31 02:27:30,953 [INFO] tensorflow: epoch = 81.0625, learning_rate = 0.0009999999, loss = 0.00020768006, step = 7782 (5.409 sec)\n",
"INFO:tensorflow:global_step/sec: 3.1046\n",
"2021-12-31 02:27:31,925 [INFO] tensorflow: global_step/sec: 3.1046\n",
"INFO:tensorflow:global_step/sec: 3.12071\n",
"2021-12-31 02:27:34,809 [INFO] tensorflow: global_step/sec: 3.12071\n",
"INFO:tensorflow:epoch = 81.23958333333333, learning_rate = 0.0009999999, loss = 0.00023315694, step = 7799 (5.464 sec)\n",
"2021-12-31 02:27:36,416 [INFO] tensorflow: epoch = 81.23958333333333, learning_rate = 0.0009999999, loss = 0.00023315694, step = 7799 (5.464 sec)\n",
"2021-12-31 02:27:36,417 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.900\n",
"INFO:tensorflow:global_step/sec: 3.10464\n",
"2021-12-31 02:27:37,708 [INFO] tensorflow: global_step/sec: 3.10464\n",
"INFO:tensorflow:global_step/sec: 3.07604\n",
"2021-12-31 02:27:40,634 [INFO] tensorflow: global_step/sec: 3.07604\n",
"INFO:tensorflow:epoch = 81.41666666666666, learning_rate = 0.0009999999, loss = 0.00019775116, step = 7816 (5.506 sec)\n",
"2021-12-31 02:27:41,922 [INFO] tensorflow: epoch = 81.41666666666666, learning_rate = 0.0009999999, loss = 0.00019775116, step = 7816 (5.506 sec)\n",
"INFO:tensorflow:global_step/sec: 3.0827\n",
"2021-12-31 02:27:43,553 [INFO] tensorflow: global_step/sec: 3.0827\n",
"2021-12-31 02:27:44,563 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.552\n",
"INFO:tensorflow:global_step/sec: 3.09638\n",
"2021-12-31 02:27:46,460 [INFO] tensorflow: global_step/sec: 3.09638\n",
"INFO:tensorflow:epoch = 81.59375, learning_rate = 0.0009999999, loss = 0.00021089564, step = 7833 (5.490 sec)\n",
"2021-12-31 02:27:47,413 [INFO] tensorflow: epoch = 81.59375, learning_rate = 0.0009999999, loss = 0.00021089564, step = 7833 (5.490 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11075\n",
"2021-12-31 02:27:49,353 [INFO] tensorflow: global_step/sec: 3.11075\n",
"INFO:tensorflow:global_step/sec: 3.15657\n",
"2021-12-31 02:27:52,204 [INFO] tensorflow: global_step/sec: 3.15657\n",
"2021-12-31 02:27:52,518 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.140\n",
"INFO:tensorflow:epoch = 81.77083333333333, learning_rate = 0.0009999999, loss = 0.00022037266, step = 7850 (5.423 sec)\n",
"2021-12-31 02:27:52,835 [INFO] tensorflow: epoch = 81.77083333333333, learning_rate = 0.0009999999, loss = 0.00022037266, step = 7850 (5.423 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10221\n",
"2021-12-31 02:27:55,106 [INFO] tensorflow: global_step/sec: 3.10221\n",
"INFO:tensorflow:global_step/sec: 3.09659\n",
"2021-12-31 02:27:58,012 [INFO] tensorflow: global_step/sec: 3.09659\n",
"INFO:tensorflow:epoch = 81.94791666666666, learning_rate = 0.0009999999, loss = 0.0002764474, step = 7867 (5.488 sec)\n",
"2021-12-31 02:27:58,323 [INFO] tensorflow: epoch = 81.94791666666666, learning_rate = 0.0009999999, loss = 0.0002764474, step = 7867 (5.488 sec)\n",
"2021-12-31 02:27:59,985 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 82/120: loss: 0.00023 learning rate: 0.00100 Time taken: 0:00:30.941897 ETA: 0:19:35.792092\n",
"2021-12-31 02:28:00,631 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.654\n",
"INFO:tensorflow:global_step/sec: 3.05742\n",
"2021-12-31 02:28:00,956 [INFO] tensorflow: global_step/sec: 3.05742\n",
"INFO:tensorflow:epoch = 82.125, learning_rate = 0.0009999999, loss = 0.00019668054, step = 7884 (5.530 sec)\n",
"2021-12-31 02:28:03,854 [INFO] tensorflow: epoch = 82.125, learning_rate = 0.0009999999, loss = 0.00019668054, step = 7884 (5.530 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10456\n",
"2021-12-31 02:28:03,855 [INFO] tensorflow: global_step/sec: 3.10456\n",
"INFO:tensorflow:global_step/sec: 3.11412\n",
"2021-12-31 02:28:06,745 [INFO] tensorflow: global_step/sec: 3.11412\n",
"2021-12-31 02:28:08,676 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.861\n",
"INFO:tensorflow:epoch = 82.30208333333333, learning_rate = 0.0009999999, loss = 0.00028109158, step = 7901 (5.453 sec)\n",
"2021-12-31 02:28:09,307 [INFO] tensorflow: epoch = 82.30208333333333, learning_rate = 0.0009999999, loss = 0.00028109158, step = 7901 (5.453 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12152\n",
"2021-12-31 02:28:09,628 [INFO] tensorflow: global_step/sec: 3.12152\n",
"INFO:tensorflow:global_step/sec: 3.07932\n",
"2021-12-31 02:28:12,551 [INFO] tensorflow: global_step/sec: 3.07932\n",
"INFO:tensorflow:epoch = 82.47916666666666, learning_rate = 0.0009999999, loss = 0.000252699, step = 7918 (5.523 sec)\n",
"2021-12-31 02:28:14,830 [INFO] tensorflow: epoch = 82.47916666666666, learning_rate = 0.0009999999, loss = 0.000252699, step = 7918 (5.523 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08645\n",
"2021-12-31 02:28:15,467 [INFO] tensorflow: global_step/sec: 3.08645\n",
"2021-12-31 02:28:16,769 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.715\n",
"INFO:tensorflow:global_step/sec: 3.09079\n",
"2021-12-31 02:28:18,378 [INFO] tensorflow: global_step/sec: 3.09079\n",
"INFO:tensorflow:epoch = 82.65625, learning_rate = 0.0009999999, loss = 0.00020622474, step = 7935 (5.469 sec)\n",
"2021-12-31 02:28:20,299 [INFO] tensorflow: epoch = 82.65625, learning_rate = 0.0009999999, loss = 0.00020622474, step = 7935 (5.469 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09082\n",
"2021-12-31 02:28:21,290 [INFO] tensorflow: global_step/sec: 3.09082\n",
"INFO:tensorflow:global_step/sec: 3.11934\n",
"2021-12-31 02:28:24,175 [INFO] tensorflow: global_step/sec: 3.11934\n",
"2021-12-31 02:28:24,797 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.913\n",
"INFO:tensorflow:epoch = 82.83333333333333, learning_rate = 0.0009999999, loss = 0.00022422097, step = 7952 (5.461 sec)\n",
"2021-12-31 02:28:25,759 [INFO] tensorflow: epoch = 82.83333333333333, learning_rate = 0.0009999999, loss = 0.00022422097, step = 7952 (5.461 sec)\n",
"INFO:tensorflow:global_step/sec: 3.13581\n",
"2021-12-31 02:28:27,046 [INFO] tensorflow: global_step/sec: 3.13581\n",
"INFO:tensorflow:global_step/sec: 3.13157\n",
"2021-12-31 02:28:29,919 [INFO] tensorflow: global_step/sec: 3.13157\n",
"2021-12-31 02:28:30,917 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 83/120: loss: 0.00020 learning rate: 0.00100 Time taken: 0:00:30.923335 ETA: 0:19:04.163407\n",
"INFO:tensorflow:epoch = 83.01041666666666, learning_rate = 0.0009999999, loss = 0.00023292775, step = 7969 (5.477 sec)\n",
"2021-12-31 02:28:31,236 [INFO] tensorflow: epoch = 83.01041666666666, learning_rate = 0.0009999999, loss = 0.00023292775, step = 7969 (5.477 sec)\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO:tensorflow:global_step/sec: 3.045\n",
"2021-12-31 02:28:32,875 [INFO] tensorflow: global_step/sec: 3.045\n",
"2021-12-31 02:28:32,876 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.756\n",
"INFO:tensorflow:global_step/sec: 3.10973\n",
"2021-12-31 02:28:35,769 [INFO] tensorflow: global_step/sec: 3.10973\n",
"INFO:tensorflow:epoch = 83.1875, learning_rate = 0.0009999999, loss = 0.00021577808, step = 7986 (5.518 sec)\n",
"2021-12-31 02:28:36,754 [INFO] tensorflow: epoch = 83.1875, learning_rate = 0.0009999999, loss = 0.00021577808, step = 7986 (5.518 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08976\n",
"2021-12-31 02:28:38,682 [INFO] tensorflow: global_step/sec: 3.08976\n",
"2021-12-31 02:28:40,872 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.012\n",
"INFO:tensorflow:global_step/sec: 3.16801\n",
"2021-12-31 02:28:41,523 [INFO] tensorflow: global_step/sec: 3.16801\n",
"INFO:tensorflow:epoch = 83.36458333333333, learning_rate = 0.0009999999, loss = 0.00020284054, step = 8003 (5.398 sec)\n",
"2021-12-31 02:28:42,152 [INFO] tensorflow: epoch = 83.36458333333333, learning_rate = 0.0009999999, loss = 0.00020284054, step = 8003 (5.398 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12973\n",
"2021-12-31 02:28:44,399 [INFO] tensorflow: global_step/sec: 3.12973\n",
"INFO:tensorflow:global_step/sec: 3.11132\n",
"2021-12-31 02:28:47,291 [INFO] tensorflow: global_step/sec: 3.11132\n",
"INFO:tensorflow:epoch = 83.54166666666666, learning_rate = 0.0009999999, loss = 0.0002347276, step = 8020 (5.456 sec)\n",
"2021-12-31 02:28:47,608 [INFO] tensorflow: epoch = 83.54166666666666, learning_rate = 0.0009999999, loss = 0.0002347276, step = 8020 (5.456 sec)\n",
"2021-12-31 02:28:48,873 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.999\n",
"INFO:tensorflow:global_step/sec: 3.13563\n",
"2021-12-31 02:28:50,162 [INFO] tensorflow: global_step/sec: 3.13563\n",
"INFO:tensorflow:epoch = 83.71875, learning_rate = 0.0009999999, loss = 0.00020960139, step = 8037 (5.480 sec)\n",
"2021-12-31 02:28:53,087 [INFO] tensorflow: epoch = 83.71875, learning_rate = 0.0009999999, loss = 0.00020960139, step = 8037 (5.480 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07533\n",
"2021-12-31 02:28:53,088 [INFO] tensorflow: global_step/sec: 3.07533\n",
"INFO:tensorflow:global_step/sec: 3.11581\n",
"2021-12-31 02:28:55,977 [INFO] tensorflow: global_step/sec: 3.11581\n",
"2021-12-31 02:28:56,939 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.797\n",
"INFO:tensorflow:epoch = 83.89583333333333, learning_rate = 0.0009999999, loss = 0.00021027922, step = 8054 (5.444 sec)\n",
"2021-12-31 02:28:58,532 [INFO] tensorflow: epoch = 83.89583333333333, learning_rate = 0.0009999999, loss = 0.00021027922, step = 8054 (5.444 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12736\n",
"2021-12-31 02:28:58,854 [INFO] tensorflow: global_step/sec: 3.12736\n",
"INFO:tensorflow:global_step/sec: 3.11235\n",
"2021-12-31 02:29:01,746 [INFO] tensorflow: global_step/sec: 3.11235\n",
"2021-12-31 02:29:01,747 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 84/120: loss: 0.00025 learning rate: 0.00100 Time taken: 0:00:30.848573 ETA: 0:18:30.548644\n",
"INFO:tensorflow:epoch = 84.07291666666666, learning_rate = 0.0009893251, loss = 0.00022173265, step = 8071 (5.404 sec)\n",
"2021-12-31 02:29:03,936 [INFO] tensorflow: epoch = 84.07291666666666, learning_rate = 0.0009893251, loss = 0.00022173265, step = 8071 (5.404 sec)\n",
"INFO:tensorflow:global_step/sec: 3.18212\n",
"2021-12-31 02:29:04,574 [INFO] tensorflow: global_step/sec: 3.18212\n",
"2021-12-31 02:29:04,910 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.090\n",
"INFO:tensorflow:global_step/sec: 3.11056\n",
"2021-12-31 02:29:07,468 [INFO] tensorflow: global_step/sec: 3.11056\n",
"INFO:tensorflow:epoch = 84.25, learning_rate = 0.0009638744, loss = 0.00019869773, step = 8088 (5.433 sec)\n",
"2021-12-31 02:29:09,369 [INFO] tensorflow: epoch = 84.25, learning_rate = 0.0009638744, loss = 0.00019869773, step = 8088 (5.433 sec)\n",
"INFO:tensorflow:global_step/sec: 3.14521\n",
"2021-12-31 02:29:10,329 [INFO] tensorflow: global_step/sec: 3.14521\n",
"2021-12-31 02:29:12,900 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.031\n",
"INFO:tensorflow:global_step/sec: 3.11292\n",
"2021-12-31 02:29:13,220 [INFO] tensorflow: global_step/sec: 3.11292\n",
"INFO:tensorflow:epoch = 84.42708333333333, learning_rate = 0.0009390784, loss = 0.00017989332, step = 8105 (5.470 sec)\n",
"2021-12-31 02:29:14,839 [INFO] tensorflow: epoch = 84.42708333333333, learning_rate = 0.0009390784, loss = 0.00017989332, step = 8105 (5.470 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08311\n",
"2021-12-31 02:29:16,140 [INFO] tensorflow: global_step/sec: 3.08311\n",
"INFO:tensorflow:global_step/sec: 3.16076\n",
"2021-12-31 02:29:18,987 [INFO] tensorflow: global_step/sec: 3.16076\n",
"INFO:tensorflow:epoch = 84.60416666666666, learning_rate = 0.0009149199, loss = 0.00022611399, step = 8122 (5.428 sec)\n",
"2021-12-31 02:29:20,267 [INFO] tensorflow: epoch = 84.60416666666666, learning_rate = 0.0009149199, loss = 0.00022611399, step = 8122 (5.428 sec)\n",
"2021-12-31 02:29:20,905 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.988\n",
"INFO:tensorflow:global_step/sec: 3.09012\n",
"2021-12-31 02:29:21,900 [INFO] tensorflow: global_step/sec: 3.09012\n",
"INFO:tensorflow:global_step/sec: 3.13001\n",
"2021-12-31 02:29:24,775 [INFO] tensorflow: global_step/sec: 3.13001\n",
"INFO:tensorflow:epoch = 84.78125, learning_rate = 0.00089138286, loss = 0.00016238404, step = 8139 (5.481 sec)\n",
"2021-12-31 02:29:25,748 [INFO] tensorflow: epoch = 84.78125, learning_rate = 0.00089138286, loss = 0.00016238404, step = 8139 (5.481 sec)\n",
"INFO:tensorflow:global_step/sec: 3.1486\n",
"2021-12-31 02:29:27,633 [INFO] tensorflow: global_step/sec: 3.1486\n",
"2021-12-31 02:29:28,883 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.069\n",
"INFO:tensorflow:global_step/sec: 3.15592\n",
"2021-12-31 02:29:30,485 [INFO] tensorflow: global_step/sec: 3.15592\n",
"INFO:tensorflow:epoch = 84.95833333333333, learning_rate = 0.0008684518, loss = 0.00021510765, step = 8156 (5.396 sec)\n",
"2021-12-31 02:29:31,144 [INFO] tensorflow: epoch = 84.95833333333333, learning_rate = 0.0008684518, loss = 0.00021510765, step = 8156 (5.396 sec)\n",
"2021-12-31 02:29:32,473 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 85/120: loss: 0.00019 learning rate: 0.00086 Time taken: 0:00:30.711821 ETA: 0:17:54.913738\n",
"INFO:tensorflow:global_step/sec: 3.05572\n",
"2021-12-31 02:29:33,430 [INFO] tensorflow: global_step/sec: 3.05572\n",
"INFO:tensorflow:global_step/sec: 3.10748\n",
"2021-12-31 02:29:36,327 [INFO] tensorflow: global_step/sec: 3.10748\n",
"INFO:tensorflow:epoch = 85.13541666666666, learning_rate = 0.0008461101, loss = 0.00014195184, step = 8173 (5.504 sec)\n",
"2021-12-31 02:29:36,648 [INFO] tensorflow: epoch = 85.13541666666666, learning_rate = 0.0008461101, loss = 0.00014195184, step = 8173 (5.504 sec)\n",
"2021-12-31 02:29:36,976 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.714\n",
"INFO:tensorflow:global_step/sec: 3.08827\n",
"2021-12-31 02:29:39,241 [INFO] tensorflow: global_step/sec: 3.08827\n",
"INFO:tensorflow:epoch = 85.3125, learning_rate = 0.0008243433, loss = 0.0002028236, step = 8190 (5.438 sec)\n",
"2021-12-31 02:29:42,087 [INFO] tensorflow: epoch = 85.3125, learning_rate = 0.0008243433, loss = 0.0002028236, step = 8190 (5.438 sec)\n",
"INFO:tensorflow:global_step/sec: 3.16191\n",
"2021-12-31 02:29:42,087 [INFO] tensorflow: global_step/sec: 3.16191\n",
"INFO:tensorflow:global_step/sec: 3.09926\n",
"2021-12-31 02:29:44,991 [INFO] tensorflow: global_step/sec: 3.09926\n",
"2021-12-31 02:29:44,992 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.950\n",
"INFO:tensorflow:epoch = 85.48958333333333, learning_rate = 0.0008031368, loss = 0.00021021895, step = 8207 (5.456 sec)\n",
"2021-12-31 02:29:47,542 [INFO] tensorflow: epoch = 85.48958333333333, learning_rate = 0.0008031368, loss = 0.00021021895, step = 8207 (5.456 sec)\n",
"INFO:tensorflow:global_step/sec: 3.14168\n",
"2021-12-31 02:29:47,856 [INFO] tensorflow: global_step/sec: 3.14168\n",
"INFO:tensorflow:global_step/sec: 3.1824\n",
"2021-12-31 02:29:50,684 [INFO] tensorflow: global_step/sec: 3.1824\n",
"INFO:tensorflow:epoch = 85.66666666666666, learning_rate = 0.00078247546, loss = 0.0001817484, step = 8224 (5.368 sec)\n",
"2021-12-31 02:29:52,910 [INFO] tensorflow: epoch = 85.66666666666666, learning_rate = 0.00078247546, loss = 0.0001817484, step = 8224 (5.368 sec)\n",
"2021-12-31 02:29:52,911 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.258\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO:tensorflow:global_step/sec: 3.12469\n",
"2021-12-31 02:29:53,564 [INFO] tensorflow: global_step/sec: 3.12469\n",
"INFO:tensorflow:global_step/sec: 3.1005\n",
"2021-12-31 02:29:56,467 [INFO] tensorflow: global_step/sec: 3.1005\n",
"INFO:tensorflow:epoch = 85.84375, learning_rate = 0.000762346, loss = 0.0002100628, step = 8241 (5.461 sec)\n",
"2021-12-31 02:29:58,372 [INFO] tensorflow: epoch = 85.84375, learning_rate = 0.000762346, loss = 0.0002100628, step = 8241 (5.461 sec)\n",
"INFO:tensorflow:global_step/sec: 3.14253\n",
"2021-12-31 02:29:59,331 [INFO] tensorflow: global_step/sec: 3.14253\n",
"2021-12-31 02:30:00,907 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.013\n",
"INFO:tensorflow:global_step/sec: 3.13629\n",
"2021-12-31 02:30:02,201 [INFO] tensorflow: global_step/sec: 3.13629\n",
"2021-12-31 02:30:03,150 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 86/120: loss: 0.00021 learning rate: 0.00075 Time taken: 0:00:30.691654 ETA: 0:17:23.516227\n",
"INFO:tensorflow:epoch = 86.02083333333333, learning_rate = 0.00074273406, loss = 0.00019987373, step = 8258 (5.445 sec)\n",
"2021-12-31 02:30:03,817 [INFO] tensorflow: epoch = 86.02083333333333, learning_rate = 0.00074273406, loss = 0.00019987373, step = 8258 (5.445 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09925\n",
"2021-12-31 02:30:05,105 [INFO] tensorflow: global_step/sec: 3.09925\n",
"INFO:tensorflow:global_step/sec: 3.11828\n",
"2021-12-31 02:30:07,991 [INFO] tensorflow: global_step/sec: 3.11828\n",
"2021-12-31 02:30:08,936 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.910\n",
"INFO:tensorflow:epoch = 86.19791666666666, learning_rate = 0.0007236263, loss = 0.00020625738, step = 8275 (5.445 sec)\n",
"2021-12-31 02:30:09,262 [INFO] tensorflow: epoch = 86.19791666666666, learning_rate = 0.0007236263, loss = 0.00020625738, step = 8275 (5.445 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11626\n",
"2021-12-31 02:30:10,879 [INFO] tensorflow: global_step/sec: 3.11626\n",
"INFO:tensorflow:global_step/sec: 3.06292\n",
"2021-12-31 02:30:13,817 [INFO] tensorflow: global_step/sec: 3.06292\n",
"INFO:tensorflow:epoch = 86.375, learning_rate = 0.00070501043, loss = 0.00020658226, step = 8292 (5.522 sec)\n",
"2021-12-31 02:30:14,784 [INFO] tensorflow: epoch = 86.375, learning_rate = 0.00070501043, loss = 0.00020658226, step = 8292 (5.522 sec)\n",
"INFO:tensorflow:global_step/sec: 3.1247\n",
"2021-12-31 02:30:16,697 [INFO] tensorflow: global_step/sec: 3.1247\n",
"2021-12-31 02:30:17,008 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.776\n",
"INFO:tensorflow:global_step/sec: 3.11603\n",
"2021-12-31 02:30:19,586 [INFO] tensorflow: global_step/sec: 3.11603\n",
"INFO:tensorflow:epoch = 86.55208333333333, learning_rate = 0.00068687345, loss = 0.00018021333, step = 8309 (5.454 sec)\n",
"2021-12-31 02:30:20,238 [INFO] tensorflow: epoch = 86.55208333333333, learning_rate = 0.00068687345, loss = 0.00018021333, step = 8309 (5.454 sec)\n",
"INFO:tensorflow:global_step/sec: 3.15908\n",
"2021-12-31 02:30:22,435 [INFO] tensorflow: global_step/sec: 3.15908\n",
"2021-12-31 02:30:25,023 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.956\n",
"INFO:tensorflow:global_step/sec: 3.10135\n",
"2021-12-31 02:30:25,337 [INFO] tensorflow: global_step/sec: 3.10135\n",
"INFO:tensorflow:epoch = 86.72916666666666, learning_rate = 0.0006692034, loss = 0.00018173922, step = 8326 (5.438 sec)\n",
"2021-12-31 02:30:25,676 [INFO] tensorflow: epoch = 86.72916666666666, learning_rate = 0.0006692034, loss = 0.00018173922, step = 8326 (5.438 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10876\n",
"2021-12-31 02:30:28,232 [INFO] tensorflow: global_step/sec: 3.10876\n",
"INFO:tensorflow:epoch = 86.90625, learning_rate = 0.0006519876, loss = 0.00019990555, step = 8343 (5.434 sec)\n",
"2021-12-31 02:30:31,110 [INFO] tensorflow: epoch = 86.90625, learning_rate = 0.0006519876, loss = 0.00019990555, step = 8343 (5.434 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12621\n",
"2021-12-31 02:30:31,111 [INFO] tensorflow: global_step/sec: 3.12621\n",
"2021-12-31 02:30:33,071 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.851\n",
"INFO:tensorflow:global_step/sec: 3.08676\n",
"2021-12-31 02:30:34,026 [INFO] tensorflow: global_step/sec: 3.08676\n",
"2021-12-31 02:30:34,027 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 87/120: loss: 0.00019 learning rate: 0.00064 Time taken: 0:00:30.865652 ETA: 0:16:58.566519\n",
"INFO:tensorflow:epoch = 87.08333333333333, learning_rate = 0.0006352147, loss = 0.00018074345, step = 8360 (5.441 sec)\n",
"2021-12-31 02:30:36,551 [INFO] tensorflow: epoch = 87.08333333333333, learning_rate = 0.0006352147, loss = 0.00018074345, step = 8360 (5.441 sec)\n",
"INFO:tensorflow:global_step/sec: 3.14789\n",
"2021-12-31 02:30:36,885 [INFO] tensorflow: global_step/sec: 3.14789\n",
"INFO:tensorflow:global_step/sec: 3.13146\n",
"2021-12-31 02:30:39,759 [INFO] tensorflow: global_step/sec: 3.13146\n",
"2021-12-31 02:30:41,023 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.150\n",
"INFO:tensorflow:epoch = 87.26041666666666, learning_rate = 0.00061887363, loss = 0.00015664165, step = 8377 (5.420 sec)\n",
"2021-12-31 02:30:41,971 [INFO] tensorflow: epoch = 87.26041666666666, learning_rate = 0.00061887363, loss = 0.00015664165, step = 8377 (5.420 sec)\n",
"INFO:tensorflow:global_step/sec: 3.14259\n",
"2021-12-31 02:30:42,623 [INFO] tensorflow: global_step/sec: 3.14259\n",
"INFO:tensorflow:global_step/sec: 3.11579\n",
"2021-12-31 02:30:45,512 [INFO] tensorflow: global_step/sec: 3.11579\n",
"INFO:tensorflow:epoch = 87.4375, learning_rate = 0.0006029529, loss = 0.00019338421, step = 8394 (5.466 sec)\n",
"2021-12-31 02:30:47,437 [INFO] tensorflow: epoch = 87.4375, learning_rate = 0.0006029529, loss = 0.00019338421, step = 8394 (5.466 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10349\n",
"2021-12-31 02:30:48,412 [INFO] tensorflow: global_step/sec: 3.10349\n",
"2021-12-31 02:30:49,060 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.886\n",
"INFO:tensorflow:global_step/sec: 3.1108\n",
"2021-12-31 02:30:51,305 [INFO] tensorflow: global_step/sec: 3.1108\n",
"INFO:tensorflow:epoch = 87.61458333333333, learning_rate = 0.00058744143, loss = 0.00026332773, step = 8411 (5.446 sec)\n",
"2021-12-31 02:30:52,883 [INFO] tensorflow: epoch = 87.61458333333333, learning_rate = 0.00058744143, loss = 0.00026332773, step = 8411 (5.446 sec)\n",
"INFO:tensorflow:global_step/sec: 3.13275\n",
"2021-12-31 02:30:54,178 [INFO] tensorflow: global_step/sec: 3.13275\n",
"INFO:tensorflow:global_step/sec: 3.15201\n",
"2021-12-31 02:30:57,033 [INFO] tensorflow: global_step/sec: 3.15201\n",
"2021-12-31 02:30:57,034 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.083\n",
"INFO:tensorflow:epoch = 87.79166666666666, learning_rate = 0.0005723293, loss = 0.00022226713, step = 8428 (5.421 sec)\n",
"2021-12-31 02:30:58,304 [INFO] tensorflow: epoch = 87.79166666666666, learning_rate = 0.0005723293, loss = 0.00022226713, step = 8428 (5.421 sec)\n",
"INFO:tensorflow:global_step/sec: 3.17039\n",
"2021-12-31 02:30:59,872 [INFO] tensorflow: global_step/sec: 3.17039\n",
"INFO:tensorflow:global_step/sec: 3.12855\n",
"2021-12-31 02:31:02,749 [INFO] tensorflow: global_step/sec: 3.12855\n",
"INFO:tensorflow:epoch = 87.96875, learning_rate = 0.0005576057, loss = 0.0002040519, step = 8445 (5.419 sec)\n",
"2021-12-31 02:31:03,723 [INFO] tensorflow: epoch = 87.96875, learning_rate = 0.0005576057, loss = 0.0002040519, step = 8445 (5.419 sec)\n",
"2021-12-31 02:31:04,706 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 88/120: loss: 0.00017 learning rate: 0.00056 Time taken: 0:00:30.681135 ETA: 0:16:21.796326\n",
"2021-12-31 02:31:05,015 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.060\n",
"INFO:tensorflow:global_step/sec: 3.07984\n",
"2021-12-31 02:31:05,671 [INFO] tensorflow: global_step/sec: 3.07984\n",
"INFO:tensorflow:global_step/sec: 3.17267\n",
"2021-12-31 02:31:08,508 [INFO] tensorflow: global_step/sec: 3.17267\n",
"INFO:tensorflow:epoch = 88.14583333333333, learning_rate = 0.0005432609, loss = 0.00016582463, step = 8462 (5.430 sec)\n",
"2021-12-31 02:31:09,153 [INFO] tensorflow: epoch = 88.14583333333333, learning_rate = 0.0005432609, loss = 0.00016582463, step = 8462 (5.430 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08937\n",
"2021-12-31 02:31:11,421 [INFO] tensorflow: global_step/sec: 3.08937\n",
"2021-12-31 02:31:13,027 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.961\n",
"INFO:tensorflow:global_step/sec: 3.11409\n",
"2021-12-31 02:31:14,311 [INFO] tensorflow: global_step/sec: 3.11409\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO:tensorflow:epoch = 88.32291666666666, learning_rate = 0.0005292853, loss = 0.00020119053, step = 8479 (5.475 sec)\n",
"2021-12-31 02:31:14,628 [INFO] tensorflow: epoch = 88.32291666666666, learning_rate = 0.0005292853, loss = 0.00020119053, step = 8479 (5.475 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10034\n",
"2021-12-31 02:31:17,214 [INFO] tensorflow: global_step/sec: 3.10034\n",
"INFO:tensorflow:epoch = 88.5, learning_rate = 0.000515669, loss = 0.000196107, step = 8496 (5.481 sec)\n",
"2021-12-31 02:31:20,109 [INFO] tensorflow: epoch = 88.5, learning_rate = 0.000515669, loss = 0.000196107, step = 8496 (5.481 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10811\n",
"2021-12-31 02:31:20,109 [INFO] tensorflow: global_step/sec: 3.10811\n",
"2021-12-31 02:31:21,048 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.936\n",
"INFO:tensorflow:global_step/sec: 3.12467\n",
"2021-12-31 02:31:22,990 [INFO] tensorflow: global_step/sec: 3.12467\n",
"INFO:tensorflow:epoch = 88.67708333333333, learning_rate = 0.000502403, loss = 0.00017113365, step = 8513 (5.437 sec)\n",
"2021-12-31 02:31:25,545 [INFO] tensorflow: epoch = 88.67708333333333, learning_rate = 0.000502403, loss = 0.00017113365, step = 8513 (5.437 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10807\n",
"2021-12-31 02:31:25,885 [INFO] tensorflow: global_step/sec: 3.10807\n",
"INFO:tensorflow:global_step/sec: 3.10203\n",
"2021-12-31 02:31:28,787 [INFO] tensorflow: global_step/sec: 3.10203\n",
"2021-12-31 02:31:29,085 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.886\n",
"INFO:tensorflow:epoch = 88.85416666666666, learning_rate = 0.0004894785, loss = 0.00017804992, step = 8530 (5.476 sec)\n",
"2021-12-31 02:31:31,021 [INFO] tensorflow: epoch = 88.85416666666666, learning_rate = 0.0004894785, loss = 0.00017804992, step = 8530 (5.476 sec)\n",
"INFO:tensorflow:global_step/sec: 3.13307\n",
"2021-12-31 02:31:31,659 [INFO] tensorflow: global_step/sec: 3.13307\n",
"INFO:tensorflow:global_step/sec: 3.13211\n",
"2021-12-31 02:31:34,533 [INFO] tensorflow: global_step/sec: 3.13211\n",
"2021-12-31 02:31:35,522 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 89/120: loss: 0.00027 learning rate: 0.00048 Time taken: 0:00:30.799034 ETA: 0:15:54.770058\n",
"INFO:tensorflow:epoch = 89.03125, learning_rate = 0.00047688655, loss = 0.00016086872, step = 8547 (5.463 sec)\n",
"2021-12-31 02:31:36,484 [INFO] tensorflow: epoch = 89.03125, learning_rate = 0.00047688655, loss = 0.00016086872, step = 8547 (5.463 sec)\n",
"2021-12-31 02:31:37,118 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.899\n",
"INFO:tensorflow:global_step/sec: 3.09118\n",
"2021-12-31 02:31:37,444 [INFO] tensorflow: global_step/sec: 3.09118\n",
"INFO:tensorflow:global_step/sec: 3.15301\n",
"2021-12-31 02:31:40,299 [INFO] tensorflow: global_step/sec: 3.15301\n",
"INFO:tensorflow:epoch = 89.20833333333333, learning_rate = 0.00046461824, loss = 0.00015683114, step = 8564 (5.427 sec)\n",
"2021-12-31 02:31:41,912 [INFO] tensorflow: epoch = 89.20833333333333, learning_rate = 0.00046461824, loss = 0.00015683114, step = 8564 (5.427 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08829\n",
"2021-12-31 02:31:43,213 [INFO] tensorflow: global_step/sec: 3.08829\n",
"2021-12-31 02:31:45,138 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.937\n",
"INFO:tensorflow:global_step/sec: 3.12706\n",
"2021-12-31 02:31:46,091 [INFO] tensorflow: global_step/sec: 3.12706\n",
"INFO:tensorflow:epoch = 89.38541666666666, learning_rate = 0.0004526658, loss = 0.00016312698, step = 8581 (5.468 sec)\n",
"2021-12-31 02:31:47,379 [INFO] tensorflow: epoch = 89.38541666666666, learning_rate = 0.0004526658, loss = 0.00016312698, step = 8581 (5.468 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11153\n",
"2021-12-31 02:31:48,983 [INFO] tensorflow: global_step/sec: 3.11153\n",
"INFO:tensorflow:global_step/sec: 3.11486\n",
"2021-12-31 02:31:51,873 [INFO] tensorflow: global_step/sec: 3.11486\n",
"INFO:tensorflow:epoch = 89.5625, learning_rate = 0.00044102062, loss = 0.00013407067, step = 8598 (5.445 sec)\n",
"2021-12-31 02:31:52,824 [INFO] tensorflow: epoch = 89.5625, learning_rate = 0.00044102062, loss = 0.00013407067, step = 8598 (5.445 sec)\n",
"2021-12-31 02:31:53,150 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.963\n",
"INFO:tensorflow:global_step/sec: 3.11921\n",
"2021-12-31 02:31:54,758 [INFO] tensorflow: global_step/sec: 3.11921\n",
"INFO:tensorflow:global_step/sec: 3.15481\n",
"2021-12-31 02:31:57,611 [INFO] tensorflow: global_step/sec: 3.15481\n",
"INFO:tensorflow:epoch = 89.73958333333333, learning_rate = 0.00042967498, loss = 0.00023600094, step = 8615 (5.432 sec)\n",
"2021-12-31 02:31:58,257 [INFO] tensorflow: epoch = 89.73958333333333, learning_rate = 0.00042967498, loss = 0.00023600094, step = 8615 (5.432 sec)\n",
"INFO:tensorflow:global_step/sec: 3.1474\n",
"2021-12-31 02:32:00,470 [INFO] tensorflow: global_step/sec: 3.1474\n",
"2021-12-31 02:32:01,111 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.123\n",
"INFO:tensorflow:global_step/sec: 3.07346\n",
"2021-12-31 02:32:03,399 [INFO] tensorflow: global_step/sec: 3.07346\n",
"INFO:tensorflow:epoch = 89.91666666666666, learning_rate = 0.00041862146, loss = 0.0001629942, step = 8632 (5.453 sec)\n",
"2021-12-31 02:32:03,710 [INFO] tensorflow: epoch = 89.91666666666666, learning_rate = 0.00041862146, loss = 0.0001629942, step = 8632 (5.453 sec)\n",
"INFO:tensorflow:Saving checkpoints for step-8640.\n",
"2021-12-31 02:32:05,932 [INFO] tensorflow: Saving checkpoints for step-8640.\n",
"WARNING:tensorflow:Ignoring: /tmp/tmpcn0k4qpk; No such file or directory\n",
"2021-12-31 02:32:06,079 [WARNING] tensorflow: Ignoring: /tmp/tmpcn0k4qpk; No such file or directory\n",
"2021-12-31 02:32:09,573 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 23, 0.00s/step\n",
"2021-12-31 02:32:11,204 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 23, 0.16s/step\n",
"2021-12-31 02:32:12,803 [INFO] iva.detectnet_v2.evaluation.evaluation: step 20 / 23, 0.16s/step\n",
"Matching predictions to ground truth, class 1/1.: 100%|█| 1108/1108 [00:00<00:00, 14167.79it/s]\n",
"Epoch 90/120\n",
"=========================\n",
"\n",
"Validation cost: 0.000176\n",
"Mean average_precision (in %): 92.5234\n",
"\n",
"class name average precision (in %)\n",
"------------ --------------------------\n",
"car 92.5234\n",
"\n",
"Median Inference Time: 0.016275\n",
"INFO:tensorflow:epoch = 90.0, learning_rate = 0.00041351854, loss = 0.00016852404, step = 8640 (10.057 sec)\n",
"2021-12-31 02:32:13,767 [INFO] tensorflow: epoch = 90.0, learning_rate = 0.00041351854, loss = 0.00016852404, step = 8640 (10.057 sec)\n",
"INFO:tensorflow:global_step/sec: 0.868007\n",
"2021-12-31 02:32:13,767 [INFO] tensorflow: global_step/sec: 0.868007\n",
"2021-12-31 02:32:13,768 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 90/120: loss: 0.00017 learning rate: 0.00041 Time taken: 0:00:38.261456 ETA: 0:19:07.843673\n",
"INFO:tensorflow:global_step/sec: 3.09789\n",
"2021-12-31 02:32:16,673 [INFO] tensorflow: global_step/sec: 3.09789\n",
"2021-12-31 02:32:16,673 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 12.852\n",
"INFO:tensorflow:epoch = 90.17708333333333, learning_rate = 0.00040288043, loss = 0.00020004329, step = 8657 (5.370 sec)\n",
"2021-12-31 02:32:19,137 [INFO] tensorflow: epoch = 90.17708333333333, learning_rate = 0.00040288043, loss = 0.00020004329, step = 8657 (5.370 sec)\n",
"INFO:tensorflow:global_step/sec: 3.22797\n",
"2021-12-31 02:32:19,461 [INFO] tensorflow: global_step/sec: 3.22797\n",
"INFO:tensorflow:global_step/sec: 3.06729\n",
"2021-12-31 02:32:22,395 [INFO] tensorflow: global_step/sec: 3.06729\n",
"INFO:tensorflow:epoch = 90.35416666666666, learning_rate = 0.00039251623, loss = 0.0001566716, step = 8674 (5.522 sec)\n",
"2021-12-31 02:32:24,659 [INFO] tensorflow: epoch = 90.35416666666666, learning_rate = 0.00039251623, loss = 0.0001566716, step = 8674 (5.522 sec)\n",
"2021-12-31 02:32:24,659 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.044\n",
"INFO:tensorflow:global_step/sec: 3.11189\n",
"2021-12-31 02:32:25,287 [INFO] tensorflow: global_step/sec: 3.11189\n",
"INFO:tensorflow:global_step/sec: 3.13277\n",
"2021-12-31 02:32:28,160 [INFO] tensorflow: global_step/sec: 3.13277\n",
"INFO:tensorflow:epoch = 90.53125, learning_rate = 0.0003824184, loss = 0.00016862851, step = 8691 (5.384 sec)\n",
"2021-12-31 02:32:30,043 [INFO] tensorflow: epoch = 90.53125, learning_rate = 0.0003824184, loss = 0.00016862851, step = 8691 (5.384 sec)\n",
"INFO:tensorflow:global_step/sec: 3.14954\n",
"2021-12-31 02:32:31,017 [INFO] tensorflow: global_step/sec: 3.14954\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"2021-12-31 02:32:32,624 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.112\n",
"INFO:tensorflow:global_step/sec: 3.104\n",
"2021-12-31 02:32:33,917 [INFO] tensorflow: global_step/sec: 3.104\n",
"INFO:tensorflow:epoch = 90.70833333333333, learning_rate = 0.00037258043, loss = 0.00018155035, step = 8708 (5.485 sec)\n",
"2021-12-31 02:32:35,528 [INFO] tensorflow: epoch = 90.70833333333333, learning_rate = 0.00037258043, loss = 0.00018155035, step = 8708 (5.485 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09252\n",
"2021-12-31 02:32:36,827 [INFO] tensorflow: global_step/sec: 3.09252\n",
"INFO:tensorflow:global_step/sec: 3.1574\n",
"2021-12-31 02:32:39,678 [INFO] tensorflow: global_step/sec: 3.1574\n",
"2021-12-31 02:32:40,656 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.901\n",
"INFO:tensorflow:epoch = 90.88541666666666, learning_rate = 0.0003629951, loss = 0.00016156193, step = 8725 (5.425 sec)\n",
"2021-12-31 02:32:40,953 [INFO] tensorflow: epoch = 90.88541666666666, learning_rate = 0.0003629951, loss = 0.00016156193, step = 8725 (5.425 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09559\n",
"2021-12-31 02:32:42,585 [INFO] tensorflow: global_step/sec: 3.09559\n",
"2021-12-31 02:32:44,480 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 91/120: loss: 0.00015 learning rate: 0.00036 Time taken: 0:00:30.722107 ETA: 0:14:50.941115\n",
"INFO:tensorflow:global_step/sec: 3.14077\n",
"2021-12-31 02:32:45,451 [INFO] tensorflow: global_step/sec: 3.14077\n",
"INFO:tensorflow:epoch = 91.0625, learning_rate = 0.00035365697, loss = 0.00015743442, step = 8742 (5.451 sec)\n",
"2021-12-31 02:32:46,403 [INFO] tensorflow: epoch = 91.0625, learning_rate = 0.00035365697, loss = 0.00015743442, step = 8742 (5.451 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09423\n",
"2021-12-31 02:32:48,359 [INFO] tensorflow: global_step/sec: 3.09423\n",
"2021-12-31 02:32:48,663 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.980\n",
"INFO:tensorflow:global_step/sec: 3.11903\n",
"2021-12-31 02:32:51,245 [INFO] tensorflow: global_step/sec: 3.11903\n",
"INFO:tensorflow:epoch = 91.23958333333333, learning_rate = 0.00034455885, loss = 0.00018039507, step = 8759 (5.482 sec)\n",
"2021-12-31 02:32:51,885 [INFO] tensorflow: epoch = 91.23958333333333, learning_rate = 0.00034455885, loss = 0.00018039507, step = 8759 (5.482 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11282\n",
"2021-12-31 02:32:54,136 [INFO] tensorflow: global_step/sec: 3.11282\n",
"2021-12-31 02:32:56,734 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.781\n",
"INFO:tensorflow:global_step/sec: 3.09839\n",
"2021-12-31 02:32:57,041 [INFO] tensorflow: global_step/sec: 3.09839\n",
"INFO:tensorflow:epoch = 91.41666666666666, learning_rate = 0.000335695, loss = 0.00016672495, step = 8776 (5.473 sec)\n",
"2021-12-31 02:32:57,359 [INFO] tensorflow: epoch = 91.41666666666666, learning_rate = 0.000335695, loss = 0.00016672495, step = 8776 (5.473 sec)\n",
"INFO:tensorflow:global_step/sec: 3.14327\n",
"2021-12-31 02:32:59,904 [INFO] tensorflow: global_step/sec: 3.14327\n",
"INFO:tensorflow:epoch = 91.59375, learning_rate = 0.0003270591, loss = 0.00016697173, step = 8793 (5.423 sec)\n",
"2021-12-31 02:33:02,782 [INFO] tensorflow: epoch = 91.59375, learning_rate = 0.0003270591, loss = 0.00016697173, step = 8793 (5.423 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12635\n",
"2021-12-31 02:33:02,783 [INFO] tensorflow: global_step/sec: 3.12635\n",
"2021-12-31 02:33:04,692 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.132\n",
"INFO:tensorflow:global_step/sec: 3.1457\n",
"2021-12-31 02:33:05,644 [INFO] tensorflow: global_step/sec: 3.1457\n",
"INFO:tensorflow:epoch = 91.77083333333333, learning_rate = 0.0003186454, loss = 0.0001667777, step = 8810 (5.331 sec)\n",
"2021-12-31 02:33:08,113 [INFO] tensorflow: epoch = 91.77083333333333, learning_rate = 0.0003186454, loss = 0.0001667777, step = 8810 (5.331 sec)\n",
"INFO:tensorflow:global_step/sec: 3.21787\n",
"2021-12-31 02:33:08,441 [INFO] tensorflow: global_step/sec: 3.21787\n",
"INFO:tensorflow:global_step/sec: 3.04035\n",
"2021-12-31 02:33:11,401 [INFO] tensorflow: global_step/sec: 3.04035\n",
"2021-12-31 02:33:12,702 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.969\n",
"INFO:tensorflow:epoch = 91.94791666666666, learning_rate = 0.00031044785, loss = 0.00013874209, step = 8827 (5.573 sec)\n",
"2021-12-31 02:33:13,686 [INFO] tensorflow: epoch = 91.94791666666666, learning_rate = 0.00031044785, loss = 0.00013874209, step = 8827 (5.573 sec)\n",
"INFO:tensorflow:global_step/sec: 3.05037\n",
"2021-12-31 02:33:14,351 [INFO] tensorflow: global_step/sec: 3.05037\n",
"2021-12-31 02:33:15,321 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 92/120: loss: 0.00017 learning rate: 0.00031 Time taken: 0:00:30.829624 ETA: 0:14:23.229484\n",
"INFO:tensorflow:global_step/sec: 3.08042\n",
"2021-12-31 02:33:17,273 [INFO] tensorflow: global_step/sec: 3.08042\n",
"INFO:tensorflow:epoch = 92.125, learning_rate = 0.0003024615, loss = 0.00014028144, step = 8844 (5.533 sec)\n",
"2021-12-31 02:33:19,219 [INFO] tensorflow: epoch = 92.125, learning_rate = 0.0003024615, loss = 0.00014028144, step = 8844 (5.533 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07978\n",
"2021-12-31 02:33:20,195 [INFO] tensorflow: global_step/sec: 3.07978\n",
"2021-12-31 02:33:20,849 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.552\n",
"INFO:tensorflow:global_step/sec: 3.10133\n",
"2021-12-31 02:33:23,097 [INFO] tensorflow: global_step/sec: 3.10133\n",
"INFO:tensorflow:epoch = 92.30208333333333, learning_rate = 0.00029468027, loss = 0.00017352683, step = 8861 (5.435 sec)\n",
"2021-12-31 02:33:24,654 [INFO] tensorflow: epoch = 92.30208333333333, learning_rate = 0.00029468027, loss = 0.00017352683, step = 8861 (5.435 sec)\n",
"INFO:tensorflow:global_step/sec: 3.22998\n",
"2021-12-31 02:33:25,884 [INFO] tensorflow: global_step/sec: 3.22998\n",
"INFO:tensorflow:global_step/sec: 3.17979\n",
"2021-12-31 02:33:28,714 [INFO] tensorflow: global_step/sec: 3.17979\n",
"2021-12-31 02:33:28,715 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.426\n",
"INFO:tensorflow:epoch = 92.47916666666666, learning_rate = 0.00028709954, loss = 0.00018398758, step = 8878 (5.329 sec)\n",
"2021-12-31 02:33:29,983 [INFO] tensorflow: epoch = 92.47916666666666, learning_rate = 0.00028709954, loss = 0.00018398758, step = 8878 (5.329 sec)\n",
"INFO:tensorflow:global_step/sec: 3.18795\n",
"2021-12-31 02:33:31,537 [INFO] tensorflow: global_step/sec: 3.18795\n",
"INFO:tensorflow:global_step/sec: 3.19036\n",
"2021-12-31 02:33:34,358 [INFO] tensorflow: global_step/sec: 3.19036\n",
"INFO:tensorflow:epoch = 92.65625, learning_rate = 0.0002797138, loss = 0.0001245538, step = 8895 (5.331 sec)\n",
"2021-12-31 02:33:35,313 [INFO] tensorflow: epoch = 92.65625, learning_rate = 0.0002797138, loss = 0.0001245538, step = 8895 (5.331 sec)\n",
"2021-12-31 02:33:36,611 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.328\n",
"INFO:tensorflow:global_step/sec: 3.10887\n",
"2021-12-31 02:33:37,253 [INFO] tensorflow: global_step/sec: 3.10887\n",
"INFO:tensorflow:global_step/sec: 3.15651\n",
"2021-12-31 02:33:40,104 [INFO] tensorflow: global_step/sec: 3.15651\n",
"INFO:tensorflow:epoch = 92.83333333333333, learning_rate = 0.00027251805, loss = 0.00015153317, step = 8912 (5.451 sec)\n",
"2021-12-31 02:33:40,765 [INFO] tensorflow: epoch = 92.83333333333333, learning_rate = 0.00027251805, loss = 0.00015153317, step = 8912 (5.451 sec)\n",
"INFO:tensorflow:global_step/sec: 3.18736\n",
"2021-12-31 02:33:42,928 [INFO] tensorflow: global_step/sec: 3.18736\n",
"2021-12-31 02:33:44,499 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.357\n",
"INFO:tensorflow:global_step/sec: 3.1784\n",
"2021-12-31 02:33:45,760 [INFO] tensorflow: global_step/sec: 3.1784\n",
"2021-12-31 02:33:45,760 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 93/120: loss: 0.00017 learning rate: 0.00027 Time taken: 0:00:30.448468 ETA: 0:13:42.108642\n",
"INFO:tensorflow:epoch = 93.02083333333333, learning_rate = 0.00026510062, loss = 0.00015004519, step = 8930 (5.634 sec)\n",
"2021-12-31 02:33:46,398 [INFO] tensorflow: epoch = 93.02083333333333, learning_rate = 0.00026510062, loss = 0.00015004519, step = 8930 (5.634 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07088\n",
"2021-12-31 02:33:48,690 [INFO] tensorflow: global_step/sec: 3.07088\n",
"INFO:tensorflow:global_step/sec: 3.11426\n",
"2021-12-31 02:33:51,580 [INFO] tensorflow: global_step/sec: 3.11426\n",
"INFO:tensorflow:epoch = 93.19791666666666, learning_rate = 0.0002582808, loss = 0.0001753635, step = 8947 (5.510 sec)\n",
"2021-12-31 02:33:51,908 [INFO] tensorflow: epoch = 93.19791666666666, learning_rate = 0.0002582808, loss = 0.0001753635, step = 8947 (5.510 sec)\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"2021-12-31 02:33:52,560 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.811\n",
"INFO:tensorflow:global_step/sec: 3.13823\n",
"2021-12-31 02:33:54,448 [INFO] tensorflow: global_step/sec: 3.13823\n",
"INFO:tensorflow:epoch = 93.375, learning_rate = 0.00025163597, loss = 0.00015438857, step = 8964 (5.406 sec)\n",
"2021-12-31 02:33:57,315 [INFO] tensorflow: epoch = 93.375, learning_rate = 0.00025163597, loss = 0.00015438857, step = 8964 (5.406 sec)\n",
"INFO:tensorflow:global_step/sec: 3.13883\n",
"2021-12-31 02:33:57,315 [INFO] tensorflow: global_step/sec: 3.13883\n",
"INFO:tensorflow:global_step/sec: 3.14008\n",
"2021-12-31 02:34:00,182 [INFO] tensorflow: global_step/sec: 3.14008\n",
"2021-12-31 02:34:00,499 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.191\n",
"INFO:tensorflow:epoch = 93.55208333333333, learning_rate = 0.00024516255, loss = 0.00014850884, step = 8981 (5.435 sec)\n",
"2021-12-31 02:34:02,750 [INFO] tensorflow: epoch = 93.55208333333333, learning_rate = 0.00024516255, loss = 0.00014850884, step = 8981 (5.435 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11037\n",
"2021-12-31 02:34:03,075 [INFO] tensorflow: global_step/sec: 3.11037\n",
"INFO:tensorflow:global_step/sec: 3.17316\n",
"2021-12-31 02:34:05,911 [INFO] tensorflow: global_step/sec: 3.17316\n",
"INFO:tensorflow:epoch = 93.72916666666666, learning_rate = 0.00023885566, loss = 0.00019068367, step = 8998 (5.400 sec)\n",
"2021-12-31 02:34:08,149 [INFO] tensorflow: epoch = 93.72916666666666, learning_rate = 0.00023885566, loss = 0.00019068367, step = 8998 (5.400 sec)\n",
"2021-12-31 02:34:08,463 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.116\n",
"INFO:tensorflow:global_step/sec: 3.12464\n",
"2021-12-31 02:34:08,792 [INFO] tensorflow: global_step/sec: 3.12464\n",
"INFO:tensorflow:global_step/sec: 3.14064\n",
"2021-12-31 02:34:11,657 [INFO] tensorflow: global_step/sec: 3.14064\n",
"INFO:tensorflow:epoch = 93.90625, learning_rate = 0.00023271103, loss = 0.00016463723, step = 9015 (5.443 sec)\n",
"2021-12-31 02:34:13,592 [INFO] tensorflow: epoch = 93.90625, learning_rate = 0.00023271103, loss = 0.00016463723, step = 9015 (5.443 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11487\n",
"2021-12-31 02:34:14,547 [INFO] tensorflow: global_step/sec: 3.11487\n",
"2021-12-31 02:34:16,469 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 94/120: loss: 0.00017 learning rate: 0.00023 Time taken: 0:00:30.703146 ETA: 0:13:18.281789\n",
"2021-12-31 02:34:16,469 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.981\n",
"INFO:tensorflow:global_step/sec: 3.09466\n",
"2021-12-31 02:34:17,455 [INFO] tensorflow: global_step/sec: 3.09466\n",
"INFO:tensorflow:epoch = 94.08333333333333, learning_rate = 0.00022672446, loss = 0.00016287391, step = 9032 (5.423 sec)\n",
"2021-12-31 02:34:19,015 [INFO] tensorflow: epoch = 94.08333333333333, learning_rate = 0.00022672446, loss = 0.00016287391, step = 9032 (5.423 sec)\n",
"INFO:tensorflow:global_step/sec: 3.17407\n",
"2021-12-31 02:34:20,290 [INFO] tensorflow: global_step/sec: 3.17407\n",
"INFO:tensorflow:global_step/sec: 3.08702\n",
"2021-12-31 02:34:23,206 [INFO] tensorflow: global_step/sec: 3.08702\n",
"INFO:tensorflow:epoch = 94.26041666666666, learning_rate = 0.00022089168, loss = 0.00020044163, step = 9049 (5.478 sec)\n",
"2021-12-31 02:34:24,493 [INFO] tensorflow: epoch = 94.26041666666666, learning_rate = 0.00022089168, loss = 0.00020044163, step = 9049 (5.478 sec)\n",
"2021-12-31 02:34:24,494 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.924\n",
"INFO:tensorflow:global_step/sec: 3.14958\n",
"2021-12-31 02:34:26,063 [INFO] tensorflow: global_step/sec: 3.14958\n",
"INFO:tensorflow:global_step/sec: 3.13116\n",
"2021-12-31 02:34:28,938 [INFO] tensorflow: global_step/sec: 3.13116\n",
"INFO:tensorflow:epoch = 94.4375, learning_rate = 0.00021520916, loss = 0.000186937, step = 9066 (5.409 sec)\n",
"2021-12-31 02:34:29,902 [INFO] tensorflow: epoch = 94.4375, learning_rate = 0.00021520916, loss = 0.000186937, step = 9066 (5.409 sec)\n",
"INFO:tensorflow:global_step/sec: 3.13413\n",
"2021-12-31 02:34:31,809 [INFO] tensorflow: global_step/sec: 3.13413\n",
"2021-12-31 02:34:32,448 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.144\n",
"INFO:tensorflow:global_step/sec: 3.11\n",
"2021-12-31 02:34:34,703 [INFO] tensorflow: global_step/sec: 3.11\n",
"INFO:tensorflow:epoch = 94.61458333333333, learning_rate = 0.00020967284, loss = 0.00020168215, step = 9083 (5.437 sec)\n",
"2021-12-31 02:34:35,340 [INFO] tensorflow: epoch = 94.61458333333333, learning_rate = 0.00020967284, loss = 0.00020168215, step = 9083 (5.437 sec)\n",
"INFO:tensorflow:global_step/sec: 3.17093\n",
"2021-12-31 02:34:37,542 [INFO] tensorflow: global_step/sec: 3.17093\n",
"INFO:tensorflow:global_step/sec: 3.10238\n",
"2021-12-31 02:34:40,443 [INFO] tensorflow: global_step/sec: 3.10238\n",
"2021-12-31 02:34:40,443 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.016\n",
"INFO:tensorflow:epoch = 94.79166666666666, learning_rate = 0.00020427875, loss = 0.00018184115, step = 9100 (5.421 sec)\n",
"2021-12-31 02:34:40,761 [INFO] tensorflow: epoch = 94.79166666666666, learning_rate = 0.00020427875, loss = 0.00018184115, step = 9100 (5.421 sec)\n",
"INFO:tensorflow:global_step/sec: 3.13135\n",
"2021-12-31 02:34:43,317 [INFO] tensorflow: global_step/sec: 3.13135\n",
"INFO:tensorflow:epoch = 94.96875, learning_rate = 0.0001990236, loss = 0.00015686762, step = 9117 (5.429 sec)\n",
"2021-12-31 02:34:46,189 [INFO] tensorflow: epoch = 94.96875, learning_rate = 0.0001990236, loss = 0.00015686762, step = 9117 (5.429 sec)\n",
"INFO:tensorflow:global_step/sec: 3.13205\n",
"2021-12-31 02:34:46,190 [INFO] tensorflow: global_step/sec: 3.13205\n",
"2021-12-31 02:34:47,172 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 95/120: loss: 0.00016 learning rate: 0.00020 Time taken: 0:00:30.693372 ETA: 0:12:47.334294\n",
"2021-12-31 02:34:48,480 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.886\n",
"INFO:tensorflow:global_step/sec: 3.06851\n",
"2021-12-31 02:34:49,123 [INFO] tensorflow: global_step/sec: 3.06851\n",
"INFO:tensorflow:epoch = 95.14583333333333, learning_rate = 0.00019390366, loss = 0.0001407643, step = 9134 (5.503 sec)\n",
"2021-12-31 02:34:51,692 [INFO] tensorflow: epoch = 95.14583333333333, learning_rate = 0.00019390366, loss = 0.0001407643, step = 9134 (5.503 sec)\n",
"INFO:tensorflow:global_step/sec: 3.115\n",
"2021-12-31 02:34:52,012 [INFO] tensorflow: global_step/sec: 3.115\n",
"INFO:tensorflow:global_step/sec: 3.0898\n",
"2021-12-31 02:34:54,925 [INFO] tensorflow: global_step/sec: 3.0898\n",
"2021-12-31 02:34:56,546 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.796\n",
"INFO:tensorflow:epoch = 95.32291666666666, learning_rate = 0.00018891542, loss = 0.00015348643, step = 9151 (5.500 sec)\n",
"2021-12-31 02:34:57,193 [INFO] tensorflow: epoch = 95.32291666666666, learning_rate = 0.00018891542, loss = 0.00015348643, step = 9151 (5.500 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09928\n",
"2021-12-31 02:34:57,829 [INFO] tensorflow: global_step/sec: 3.09928\n",
"INFO:tensorflow:global_step/sec: 3.12769\n",
"2021-12-31 02:35:00,707 [INFO] tensorflow: global_step/sec: 3.12769\n",
"INFO:tensorflow:epoch = 95.5, learning_rate = 0.00018405533, loss = 0.00014253243, step = 9168 (5.489 sec)\n",
"2021-12-31 02:35:02,682 [INFO] tensorflow: epoch = 95.5, learning_rate = 0.00018405533, loss = 0.00014253243, step = 9168 (5.489 sec)\n",
"INFO:tensorflow:global_step/sec: 3.04581\n",
"2021-12-31 02:35:03,662 [INFO] tensorflow: global_step/sec: 3.04581\n",
"2021-12-31 02:35:04,622 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.766\n",
"INFO:tensorflow:global_step/sec: 3.13235\n",
"2021-12-31 02:35:06,535 [INFO] tensorflow: global_step/sec: 3.13235\n",
"INFO:tensorflow:epoch = 95.67708333333333, learning_rate = 0.00017932047, loss = 0.000165601, step = 9185 (5.435 sec)\n",
"2021-12-31 02:35:08,117 [INFO] tensorflow: epoch = 95.67708333333333, learning_rate = 0.00017932047, loss = 0.000165601, step = 9185 (5.435 sec)\n",
"INFO:tensorflow:global_step/sec: 3.1228\n",
"2021-12-31 02:35:09,417 [INFO] tensorflow: global_step/sec: 3.1228\n",
"INFO:tensorflow:global_step/sec: 3.10545\n",
"2021-12-31 02:35:12,315 [INFO] tensorflow: global_step/sec: 3.10545\n",
"2021-12-31 02:35:12,637 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.954\n",
"INFO:tensorflow:epoch = 95.85416666666666, learning_rate = 0.00017470738, loss = 0.00015945808, step = 9202 (5.455 sec)\n",
"2021-12-31 02:35:13,572 [INFO] tensorflow: epoch = 95.85416666666666, learning_rate = 0.00017470738, loss = 0.00015945808, step = 9202 (5.455 sec)\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO:tensorflow:global_step/sec: 3.13144\n",
"2021-12-31 02:35:15,189 [INFO] tensorflow: global_step/sec: 3.13144\n",
"INFO:tensorflow:global_step/sec: 3.16175\n",
"2021-12-31 02:35:18,036 [INFO] tensorflow: global_step/sec: 3.16175\n",
"2021-12-31 02:35:18,036 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 96/120: loss: 0.00017 learning rate: 0.00017 Time taken: 0:00:30.862694 ETA: 0:12:20.704662\n",
"INFO:tensorflow:epoch = 96.03125, learning_rate = 0.00017021279, loss = 0.00017275798, step = 9219 (5.375 sec)\n",
"2021-12-31 02:35:18,946 [INFO] tensorflow: epoch = 96.03125, learning_rate = 0.00017021279, loss = 0.00017275798, step = 9219 (5.375 sec)\n",
"2021-12-31 02:35:20,533 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.328\n",
"INFO:tensorflow:global_step/sec: 3.1987\n",
"2021-12-31 02:35:20,849 [INFO] tensorflow: global_step/sec: 3.1987\n",
"INFO:tensorflow:global_step/sec: 3.11133\n",
"2021-12-31 02:35:23,742 [INFO] tensorflow: global_step/sec: 3.11133\n",
"INFO:tensorflow:epoch = 96.20833333333333, learning_rate = 0.00016583403, loss = 0.00014478061, step = 9236 (5.396 sec)\n",
"2021-12-31 02:35:24,343 [INFO] tensorflow: epoch = 96.20833333333333, learning_rate = 0.00016583403, loss = 0.00014478061, step = 9236 (5.396 sec)\n",
"INFO:tensorflow:global_step/sec: 3.16547\n",
"2021-12-31 02:35:26,585 [INFO] tensorflow: global_step/sec: 3.16547\n",
"2021-12-31 02:35:28,562 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.913\n",
"INFO:tensorflow:global_step/sec: 3.0662\n",
"2021-12-31 02:35:29,520 [INFO] tensorflow: global_step/sec: 3.0662\n",
"INFO:tensorflow:epoch = 96.38541666666666, learning_rate = 0.00016156789, loss = 0.00015059316, step = 9253 (5.504 sec)\n",
"2021-12-31 02:35:29,847 [INFO] tensorflow: epoch = 96.38541666666666, learning_rate = 0.00016156789, loss = 0.00015059316, step = 9253 (5.504 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12489\n",
"2021-12-31 02:35:32,400 [INFO] tensorflow: global_step/sec: 3.12489\n",
"INFO:tensorflow:epoch = 96.5625, learning_rate = 0.0001574115, loss = 0.00014957045, step = 9270 (5.394 sec)\n",
"2021-12-31 02:35:35,241 [INFO] tensorflow: epoch = 96.5625, learning_rate = 0.0001574115, loss = 0.00014957045, step = 9270 (5.394 sec)\n",
"INFO:tensorflow:global_step/sec: 3.16776\n",
"2021-12-31 02:35:35,242 [INFO] tensorflow: global_step/sec: 3.16776\n",
"2021-12-31 02:35:36,579 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.947\n",
"INFO:tensorflow:global_step/sec: 3.03891\n",
"2021-12-31 02:35:38,203 [INFO] tensorflow: global_step/sec: 3.03891\n",
"INFO:tensorflow:epoch = 96.73958333333333, learning_rate = 0.00015336204, loss = 0.00014723095, step = 9287 (5.582 sec)\n",
"2021-12-31 02:35:40,823 [INFO] tensorflow: epoch = 96.73958333333333, learning_rate = 0.00015336204, loss = 0.00014723095, step = 9287 (5.582 sec)\n",
"INFO:tensorflow:global_step/sec: 3.06497\n",
"2021-12-31 02:35:41,140 [INFO] tensorflow: global_step/sec: 3.06497\n",
"INFO:tensorflow:global_step/sec: 3.10209\n",
"2021-12-31 02:35:44,041 [INFO] tensorflow: global_step/sec: 3.10209\n",
"2021-12-31 02:35:44,683 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.678\n",
"INFO:tensorflow:epoch = 96.91666666666666, learning_rate = 0.00014941662, loss = 0.00012564863, step = 9304 (5.448 sec)\n",
"2021-12-31 02:35:46,271 [INFO] tensorflow: epoch = 96.91666666666666, learning_rate = 0.00014941662, loss = 0.00012564863, step = 9304 (5.448 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12428\n",
"2021-12-31 02:35:46,921 [INFO] tensorflow: global_step/sec: 3.12428\n",
"2021-12-31 02:35:48,839 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 97/120: loss: 0.00015 learning rate: 0.00015 Time taken: 0:00:30.811469 ETA: 0:11:48.663789\n",
"INFO:tensorflow:global_step/sec: 3.12198\n",
"2021-12-31 02:35:49,804 [INFO] tensorflow: global_step/sec: 3.12198\n",
"INFO:tensorflow:epoch = 97.09375, learning_rate = 0.00014557282, loss = 0.00018413864, step = 9321 (5.419 sec)\n",
"2021-12-31 02:35:51,690 [INFO] tensorflow: epoch = 97.09375, learning_rate = 0.00014557282, loss = 0.00018413864, step = 9321 (5.419 sec)\n",
"INFO:tensorflow:global_step/sec: 3.16923\n",
"2021-12-31 02:35:52,644 [INFO] tensorflow: global_step/sec: 3.16923\n",
"2021-12-31 02:35:52,645 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.121\n",
"INFO:tensorflow:global_step/sec: 3.09744\n",
"2021-12-31 02:35:55,550 [INFO] tensorflow: global_step/sec: 3.09744\n",
"INFO:tensorflow:epoch = 97.27083333333333, learning_rate = 0.00014182765, loss = 0.0001475176, step = 9338 (5.483 sec)\n",
"2021-12-31 02:35:57,173 [INFO] tensorflow: epoch = 97.27083333333333, learning_rate = 0.00014182765, loss = 0.0001475176, step = 9338 (5.483 sec)\n",
"INFO:tensorflow:global_step/sec: 3.1089\n",
"2021-12-31 02:35:58,445 [INFO] tensorflow: global_step/sec: 3.1089\n",
"2021-12-31 02:36:00,727 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.747\n",
"INFO:tensorflow:global_step/sec: 3.09663\n",
"2021-12-31 02:36:01,351 [INFO] tensorflow: global_step/sec: 3.09663\n",
"INFO:tensorflow:epoch = 97.44791666666666, learning_rate = 0.00013817908, loss = 0.00013596992, step = 9355 (5.455 sec)\n",
"2021-12-31 02:36:02,628 [INFO] tensorflow: epoch = 97.44791666666666, learning_rate = 0.00013817908, loss = 0.00013596992, step = 9355 (5.455 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12268\n",
"2021-12-31 02:36:04,233 [INFO] tensorflow: global_step/sec: 3.12268\n",
"INFO:tensorflow:global_step/sec: 3.08511\n",
"2021-12-31 02:36:07,150 [INFO] tensorflow: global_step/sec: 3.08511\n",
"INFO:tensorflow:epoch = 97.625, learning_rate = 0.00013462438, loss = 0.00016642586, step = 9372 (5.479 sec)\n",
"2021-12-31 02:36:08,107 [INFO] tensorflow: epoch = 97.625, learning_rate = 0.00013462438, loss = 0.00016642586, step = 9372 (5.479 sec)\n",
"2021-12-31 02:36:08,765 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.881\n",
"INFO:tensorflow:global_step/sec: 3.1168\n",
"2021-12-31 02:36:10,038 [INFO] tensorflow: global_step/sec: 3.1168\n",
"INFO:tensorflow:global_step/sec: 3.13546\n",
"2021-12-31 02:36:12,908 [INFO] tensorflow: global_step/sec: 3.13546\n",
"INFO:tensorflow:epoch = 97.80208333333333, learning_rate = 0.000131161, loss = 0.00013217791, step = 9389 (5.432 sec)\n",
"2021-12-31 02:36:13,540 [INFO] tensorflow: epoch = 97.80208333333333, learning_rate = 0.000131161, loss = 0.00013217791, step = 9389 (5.432 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08843\n",
"2021-12-31 02:36:15,822 [INFO] tensorflow: global_step/sec: 3.08843\n",
"2021-12-31 02:36:16,776 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.968\n",
"INFO:tensorflow:global_step/sec: 3.10312\n",
"2021-12-31 02:36:18,723 [INFO] tensorflow: global_step/sec: 3.10312\n",
"INFO:tensorflow:epoch = 97.97916666666666, learning_rate = 0.00012778684, loss = 0.00013551794, step = 9406 (5.518 sec)\n",
"2021-12-31 02:36:19,058 [INFO] tensorflow: epoch = 97.97916666666666, learning_rate = 0.00012778684, loss = 0.00013551794, step = 9406 (5.518 sec)\n",
"2021-12-31 02:36:19,659 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 98/120: loss: 0.00014 learning rate: 0.00013 Time taken: 0:00:30.852059 ETA: 0:11:18.745290\n",
"INFO:tensorflow:global_step/sec: 3.1723\n",
"2021-12-31 02:36:21,560 [INFO] tensorflow: global_step/sec: 3.1723\n",
"INFO:tensorflow:epoch = 98.15625, learning_rate = 0.00012449948, loss = 0.00018377614, step = 9423 (5.333 sec)\n",
"2021-12-31 02:36:24,391 [INFO] tensorflow: epoch = 98.15625, learning_rate = 0.00012449948, loss = 0.00018377614, step = 9423 (5.333 sec)\n",
"INFO:tensorflow:global_step/sec: 3.17762\n",
"2021-12-31 02:36:24,392 [INFO] tensorflow: global_step/sec: 3.17762\n",
"2021-12-31 02:36:24,724 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.164\n",
"INFO:tensorflow:global_step/sec: 3.10318\n",
"2021-12-31 02:36:27,292 [INFO] tensorflow: global_step/sec: 3.10318\n",
"INFO:tensorflow:epoch = 98.33333333333333, learning_rate = 0.00012129669, loss = 0.0001495188, step = 9440 (5.443 sec)\n",
"2021-12-31 02:36:29,834 [INFO] tensorflow: epoch = 98.33333333333333, learning_rate = 0.00012129669, loss = 0.0001495188, step = 9440 (5.443 sec)\n",
"INFO:tensorflow:global_step/sec: 3.14526\n",
"2021-12-31 02:36:30,154 [INFO] tensorflow: global_step/sec: 3.14526\n",
"2021-12-31 02:36:32,718 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.020\n",
"INFO:tensorflow:global_step/sec: 3.11109\n",
"2021-12-31 02:36:33,047 [INFO] tensorflow: global_step/sec: 3.11109\n",
"INFO:tensorflow:epoch = 98.51041666666666, learning_rate = 0.00011817629, loss = 0.00014950712, step = 9457 (5.501 sec)\n",
"2021-12-31 02:36:35,336 [INFO] tensorflow: epoch = 98.51041666666666, learning_rate = 0.00011817629, loss = 0.00014950712, step = 9457 (5.501 sec)\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO:tensorflow:global_step/sec: 3.0515\n",
"2021-12-31 02:36:35,996 [INFO] tensorflow: global_step/sec: 3.0515\n",
"INFO:tensorflow:global_step/sec: 3.12682\n",
"2021-12-31 02:36:38,874 [INFO] tensorflow: global_step/sec: 3.12682\n",
"INFO:tensorflow:epoch = 98.6875, learning_rate = 0.00011513606, loss = 0.00016195548, step = 9474 (5.484 sec)\n",
"2021-12-31 02:36:40,820 [INFO] tensorflow: epoch = 98.6875, learning_rate = 0.00011513606, loss = 0.00016195548, step = 9474 (5.484 sec)\n",
"2021-12-31 02:36:40,820 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.686\n",
"INFO:tensorflow:global_step/sec: 3.09286\n",
"2021-12-31 02:36:41,784 [INFO] tensorflow: global_step/sec: 3.09286\n",
"INFO:tensorflow:global_step/sec: 3.17436\n",
"2021-12-31 02:36:44,620 [INFO] tensorflow: global_step/sec: 3.17436\n",
"INFO:tensorflow:epoch = 98.86458333333333, learning_rate = 0.000112174144, loss = 0.00015241868, step = 9491 (5.377 sec)\n",
"2021-12-31 02:36:46,197 [INFO] tensorflow: epoch = 98.86458333333333, learning_rate = 0.000112174144, loss = 0.00015241868, step = 9491 (5.377 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11268\n",
"2021-12-31 02:36:47,511 [INFO] tensorflow: global_step/sec: 3.11268\n",
"2021-12-31 02:36:48,769 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.163\n",
"INFO:tensorflow:global_step/sec: 3.18026\n",
"2021-12-31 02:36:50,341 [INFO] tensorflow: global_step/sec: 3.18026\n",
"2021-12-31 02:36:50,342 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 99/120: loss: 0.00016 learning rate: 0.00011 Time taken: 0:00:30.682968 ETA: 0:10:44.342336\n",
"INFO:tensorflow:epoch = 99.04166666666666, learning_rate = 0.00010928843, loss = 0.00013098572, step = 9508 (5.431 sec)\n",
"2021-12-31 02:36:51,627 [INFO] tensorflow: epoch = 99.04166666666666, learning_rate = 0.00010928843, loss = 0.00013098572, step = 9508 (5.431 sec)\n",
"INFO:tensorflow:global_step/sec: 3.18128\n",
"2021-12-31 02:36:53,170 [INFO] tensorflow: global_step/sec: 3.18128\n",
"INFO:tensorflow:global_step/sec: 3.08884\n",
"2021-12-31 02:36:56,084 [INFO] tensorflow: global_step/sec: 3.08884\n",
"2021-12-31 02:36:56,728 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.128\n",
"INFO:tensorflow:epoch = 99.21875, learning_rate = 0.00010647685, loss = 0.00013371903, step = 9525 (5.419 sec)\n",
"2021-12-31 02:36:57,046 [INFO] tensorflow: epoch = 99.21875, learning_rate = 0.00010647685, loss = 0.00013371903, step = 9525 (5.419 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09493\n",
"2021-12-31 02:36:58,992 [INFO] tensorflow: global_step/sec: 3.09493\n",
"INFO:tensorflow:global_step/sec: 3.16388\n",
"2021-12-31 02:37:01,836 [INFO] tensorflow: global_step/sec: 3.16388\n",
"INFO:tensorflow:epoch = 99.39583333333333, learning_rate = 0.00010373769, loss = 0.00014477258, step = 9542 (5.420 sec)\n",
"2021-12-31 02:37:02,467 [INFO] tensorflow: epoch = 99.39583333333333, learning_rate = 0.00010373769, loss = 0.00014477258, step = 9542 (5.420 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07402\n",
"2021-12-31 02:37:04,764 [INFO] tensorflow: global_step/sec: 3.07402\n",
"2021-12-31 02:37:04,765 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.887\n",
"INFO:tensorflow:global_step/sec: 3.05468\n",
"2021-12-31 02:37:07,710 [INFO] tensorflow: global_step/sec: 3.05468\n",
"INFO:tensorflow:epoch = 99.57291666666666, learning_rate = 0.00010106901, loss = 0.00015239236, step = 9559 (5.565 sec)\n",
"2021-12-31 02:37:08,032 [INFO] tensorflow: epoch = 99.57291666666666, learning_rate = 0.00010106901, loss = 0.00015239236, step = 9559 (5.565 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09532\n",
"2021-12-31 02:37:10,618 [INFO] tensorflow: global_step/sec: 3.09532\n",
"2021-12-31 02:37:12,852 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.729\n",
"INFO:tensorflow:epoch = 99.75, learning_rate = 9.846897e-05, loss = 0.0001369857, step = 9576 (5.476 sec)\n",
"2021-12-31 02:37:13,508 [INFO] tensorflow: epoch = 99.75, learning_rate = 9.846897e-05, loss = 0.0001369857, step = 9576 (5.476 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11286\n",
"2021-12-31 02:37:13,509 [INFO] tensorflow: global_step/sec: 3.11286\n",
"INFO:tensorflow:global_step/sec: 3.15592\n",
"2021-12-31 02:37:16,361 [INFO] tensorflow: global_step/sec: 3.15592\n",
"INFO:tensorflow:epoch = 99.92708333333333, learning_rate = 9.5935735e-05, loss = 0.00015331563, step = 9593 (5.464 sec)\n",
"2021-12-31 02:37:18,972 [INFO] tensorflow: epoch = 99.92708333333333, learning_rate = 9.5935735e-05, loss = 0.00015331563, step = 9593 (5.464 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07119\n",
"2021-12-31 02:37:19,291 [INFO] tensorflow: global_step/sec: 3.07119\n",
"INFO:tensorflow:Saving checkpoints for step-9600.\n",
"2021-12-31 02:37:20,916 [INFO] tensorflow: Saving checkpoints for step-9600.\n",
"WARNING:tensorflow:Ignoring: /tmp/tmpl8vha0hd; No such file or directory\n",
"2021-12-31 02:37:21,068 [WARNING] tensorflow: Ignoring: /tmp/tmpl8vha0hd; No such file or directory\n",
"2021-12-31 02:37:24,553 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 23, 0.00s/step\n",
"2021-12-31 02:37:26,133 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 23, 0.16s/step\n",
"2021-12-31 02:37:27,822 [INFO] iva.detectnet_v2.evaluation.evaluation: step 20 / 23, 0.17s/step\n",
"Matching predictions to ground truth, class 1/1.: 100%|█| 940/940 [00:00<00:00, 14898.54it/s]\n",
"Epoch 100/120\n",
"=========================\n",
"\n",
"Validation cost: 0.000165\n",
"Mean average_precision (in %): 93.7993\n",
"\n",
"class name average precision (in %)\n",
"------------ --------------------------\n",
"car 93.7993\n",
"\n",
"Median Inference Time: 0.016067\n",
"2021-12-31 02:37:28,392 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 12.871\n",
"INFO:tensorflow:epoch = 100.0, learning_rate = 9.491169e-05, loss = 0.00015050537, step = 9600 (9.742 sec)\n",
"2021-12-31 02:37:28,715 [INFO] tensorflow: epoch = 100.0, learning_rate = 9.491169e-05, loss = 0.00015050537, step = 9600 (9.742 sec)\n",
"2021-12-31 02:37:28,715 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 100/120: loss: 0.00015 learning rate: 0.00009 Time taken: 0:00:38.338516 ETA: 0:12:46.770310\n",
"INFO:tensorflow:global_step/sec: 0.865726\n",
"2021-12-31 02:37:29,687 [INFO] tensorflow: global_step/sec: 0.865726\n",
"INFO:tensorflow:global_step/sec: 3.13124\n",
"2021-12-31 02:37:32,562 [INFO] tensorflow: global_step/sec: 3.13124\n",
"INFO:tensorflow:epoch = 100.17708333333333, learning_rate = 9.246996e-05, loss = 0.00014004583, step = 9617 (5.456 sec)\n",
"2021-12-31 02:37:34,171 [INFO] tensorflow: epoch = 100.17708333333333, learning_rate = 9.246996e-05, loss = 0.00014004583, step = 9617 (5.456 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09152\n",
"2021-12-31 02:37:35,473 [INFO] tensorflow: global_step/sec: 3.09152\n",
"2021-12-31 02:37:36,468 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.767\n",
"INFO:tensorflow:global_step/sec: 3.07844\n",
"2021-12-31 02:37:38,396 [INFO] tensorflow: global_step/sec: 3.07844\n",
"INFO:tensorflow:epoch = 100.35416666666666, learning_rate = 9.009115e-05, loss = 0.00013848266, step = 9634 (5.535 sec)\n",
"2021-12-31 02:37:39,706 [INFO] tensorflow: epoch = 100.35416666666666, learning_rate = 9.009115e-05, loss = 0.00013848266, step = 9634 (5.535 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07221\n",
"2021-12-31 02:37:41,326 [INFO] tensorflow: global_step/sec: 3.07221\n",
"INFO:tensorflow:global_step/sec: 3.12414\n",
"2021-12-31 02:37:44,207 [INFO] tensorflow: global_step/sec: 3.12414\n",
"2021-12-31 02:37:44,531 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.804\n",
"INFO:tensorflow:epoch = 100.53125, learning_rate = 8.777352e-05, loss = 0.00014451219, step = 9651 (5.467 sec)\n",
"2021-12-31 02:37:45,173 [INFO] tensorflow: epoch = 100.53125, learning_rate = 8.777352e-05, loss = 0.00014451219, step = 9651 (5.467 sec)\n",
"INFO:tensorflow:global_step/sec: 3.0903\n",
"2021-12-31 02:37:47,119 [INFO] tensorflow: global_step/sec: 3.0903\n",
"INFO:tensorflow:global_step/sec: 3.11946\n",
"2021-12-31 02:37:50,004 [INFO] tensorflow: global_step/sec: 3.11946\n",
"INFO:tensorflow:epoch = 100.70833333333333, learning_rate = 8.5515436e-05, loss = 0.00013340452, step = 9668 (5.492 sec)\n",
"2021-12-31 02:37:50,666 [INFO] tensorflow: epoch = 100.70833333333333, learning_rate = 8.5515436e-05, loss = 0.00013340452, step = 9668 (5.492 sec)\n",
"2021-12-31 02:37:52,638 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.673\n",
"INFO:tensorflow:global_step/sec: 3.03125\n",
"2021-12-31 02:37:52,973 [INFO] tensorflow: global_step/sec: 3.03125\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO:tensorflow:global_step/sec: 3.09833\n",
"2021-12-31 02:37:55,878 [INFO] tensorflow: global_step/sec: 3.09833\n",
"INFO:tensorflow:epoch = 100.88541666666666, learning_rate = 8.331552e-05, loss = 0.0001583534, step = 9685 (5.544 sec)\n",
"2021-12-31 02:37:56,210 [INFO] tensorflow: epoch = 100.88541666666666, learning_rate = 8.331552e-05, loss = 0.0001583534, step = 9685 (5.544 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07079\n",
"2021-12-31 02:37:58,809 [INFO] tensorflow: global_step/sec: 3.07079\n",
"2021-12-31 02:37:59,780 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 101/120: loss: 0.00018 learning rate: 0.00008 Time taken: 0:00:31.065064 ETA: 0:09:50.236215\n",
"2021-12-31 02:38:00,763 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.616\n",
"INFO:tensorflow:epoch = 101.0625, learning_rate = 8.11722e-05, loss = 0.0001411916, step = 9702 (5.538 sec)\n",
"2021-12-31 02:38:01,748 [INFO] tensorflow: epoch = 101.0625, learning_rate = 8.11722e-05, loss = 0.0001411916, step = 9702 (5.538 sec)\n",
"INFO:tensorflow:global_step/sec: 3.06091\n",
"2021-12-31 02:38:01,749 [INFO] tensorflow: global_step/sec: 3.06091\n",
"INFO:tensorflow:global_step/sec: 3.12605\n",
"2021-12-31 02:38:04,628 [INFO] tensorflow: global_step/sec: 3.12605\n",
"INFO:tensorflow:epoch = 101.23958333333333, learning_rate = 7.908402e-05, loss = 0.00013547433, step = 9719 (5.440 sec)\n",
"2021-12-31 02:38:07,188 [INFO] tensorflow: epoch = 101.23958333333333, learning_rate = 7.908402e-05, loss = 0.00013547433, step = 9719 (5.440 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12639\n",
"2021-12-31 02:38:07,507 [INFO] tensorflow: global_step/sec: 3.12639\n",
"2021-12-31 02:38:08,719 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.137\n",
"INFO:tensorflow:global_step/sec: 3.15298\n",
"2021-12-31 02:38:10,361 [INFO] tensorflow: global_step/sec: 3.15298\n",
"INFO:tensorflow:epoch = 101.41666666666666, learning_rate = 7.704956e-05, loss = 0.00012285012, step = 9736 (5.398 sec)\n",
"2021-12-31 02:38:12,586 [INFO] tensorflow: epoch = 101.41666666666666, learning_rate = 7.704956e-05, loss = 0.00012285012, step = 9736 (5.398 sec)\n",
"INFO:tensorflow:global_step/sec: 3.13047\n",
"2021-12-31 02:38:13,236 [INFO] tensorflow: global_step/sec: 3.13047\n",
"INFO:tensorflow:global_step/sec: 3.11049\n",
"2021-12-31 02:38:16,130 [INFO] tensorflow: global_step/sec: 3.11049\n",
"2021-12-31 02:38:16,767 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.853\n",
"INFO:tensorflow:epoch = 101.59375, learning_rate = 7.506736e-05, loss = 0.00011939064, step = 9753 (5.435 sec)\n",
"2021-12-31 02:38:18,021 [INFO] tensorflow: epoch = 101.59375, learning_rate = 7.506736e-05, loss = 0.00011939064, step = 9753 (5.435 sec)\n",
"INFO:tensorflow:global_step/sec: 3.13732\n",
"2021-12-31 02:38:18,998 [INFO] tensorflow: global_step/sec: 3.13732\n",
"INFO:tensorflow:global_step/sec: 3.17726\n",
"2021-12-31 02:38:21,831 [INFO] tensorflow: global_step/sec: 3.17726\n",
"INFO:tensorflow:epoch = 101.77083333333333, learning_rate = 7.313623e-05, loss = 0.00018236988, step = 9770 (5.450 sec)\n",
"2021-12-31 02:38:23,472 [INFO] tensorflow: epoch = 101.77083333333333, learning_rate = 7.313623e-05, loss = 0.00018236988, step = 9770 (5.450 sec)\n",
"INFO:tensorflow:global_step/sec: 3.04948\n",
"2021-12-31 02:38:24,782 [INFO] tensorflow: global_step/sec: 3.04948\n",
"2021-12-31 02:38:24,783 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.951\n",
"INFO:tensorflow:global_step/sec: 3.11949\n",
"2021-12-31 02:38:27,667 [INFO] tensorflow: global_step/sec: 3.11949\n",
"INFO:tensorflow:epoch = 101.94791666666666, learning_rate = 7.125478e-05, loss = 0.00013593728, step = 9787 (5.489 sec)\n",
"2021-12-31 02:38:28,960 [INFO] tensorflow: epoch = 101.94791666666666, learning_rate = 7.125478e-05, loss = 0.00013593728, step = 9787 (5.489 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09232\n",
"2021-12-31 02:38:30,578 [INFO] tensorflow: global_step/sec: 3.09232\n",
"2021-12-31 02:38:30,579 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 102/120: loss: 0.00017 learning rate: 0.00007 Time taken: 0:00:30.773663 ETA: 0:09:13.925935\n",
"2021-12-31 02:38:32,821 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.883\n",
"INFO:tensorflow:global_step/sec: 3.08824\n",
"2021-12-31 02:38:33,492 [INFO] tensorflow: global_step/sec: 3.08824\n",
"INFO:tensorflow:epoch = 102.125, learning_rate = 6.942166e-05, loss = 0.00012506872, step = 9804 (5.487 sec)\n",
"2021-12-31 02:38:34,447 [INFO] tensorflow: epoch = 102.125, learning_rate = 6.942166e-05, loss = 0.00012506872, step = 9804 (5.487 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08679\n",
"2021-12-31 02:38:36,408 [INFO] tensorflow: global_step/sec: 3.08679\n",
"INFO:tensorflow:global_step/sec: 3.17773\n",
"2021-12-31 02:38:39,240 [INFO] tensorflow: global_step/sec: 3.17773\n",
"INFO:tensorflow:epoch = 102.30208333333333, learning_rate = 6.763577e-05, loss = 0.00015376492, step = 9821 (5.433 sec)\n",
"2021-12-31 02:38:39,880 [INFO] tensorflow: epoch = 102.30208333333333, learning_rate = 6.763577e-05, loss = 0.00015376492, step = 9821 (5.433 sec)\n",
"2021-12-31 02:38:40,851 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.907\n",
"INFO:tensorflow:global_step/sec: 3.12095\n",
"2021-12-31 02:38:42,124 [INFO] tensorflow: global_step/sec: 3.12095\n",
"INFO:tensorflow:global_step/sec: 3.07998\n",
"2021-12-31 02:38:45,046 [INFO] tensorflow: global_step/sec: 3.07998\n",
"INFO:tensorflow:epoch = 102.47916666666666, learning_rate = 6.589581e-05, loss = 0.000109862965, step = 9838 (5.483 sec)\n",
"2021-12-31 02:38:45,363 [INFO] tensorflow: epoch = 102.47916666666666, learning_rate = 6.589581e-05, loss = 0.000109862965, step = 9838 (5.483 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09057\n",
"2021-12-31 02:38:47,958 [INFO] tensorflow: global_step/sec: 3.09057\n",
"2021-12-31 02:38:48,916 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.798\n",
"INFO:tensorflow:epoch = 102.65625, learning_rate = 6.420062e-05, loss = 0.000120733814, step = 9855 (5.483 sec)\n",
"2021-12-31 02:38:50,846 [INFO] tensorflow: epoch = 102.65625, learning_rate = 6.420062e-05, loss = 0.000120733814, step = 9855 (5.483 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11487\n",
"2021-12-31 02:38:50,847 [INFO] tensorflow: global_step/sec: 3.11487\n",
"INFO:tensorflow:global_step/sec: 3.0553\n",
"2021-12-31 02:38:53,793 [INFO] tensorflow: global_step/sec: 3.0553\n",
"INFO:tensorflow:epoch = 102.83333333333333, learning_rate = 6.254898e-05, loss = 0.00011214214, step = 9872 (5.509 sec)\n",
"2021-12-31 02:38:56,355 [INFO] tensorflow: epoch = 102.83333333333333, learning_rate = 6.254898e-05, loss = 0.00011214214, step = 9872 (5.509 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11538\n",
"2021-12-31 02:38:56,682 [INFO] tensorflow: global_step/sec: 3.11538\n",
"2021-12-31 02:38:57,000 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.742\n",
"INFO:tensorflow:global_step/sec: 3.1363\n",
"2021-12-31 02:38:59,551 [INFO] tensorflow: global_step/sec: 3.1363\n",
"2021-12-31 02:39:01,492 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 103/120: loss: 0.00014 learning rate: 0.00006 Time taken: 0:00:30.940511 ETA: 0:08:45.988683\n",
"INFO:tensorflow:epoch = 103.01041666666666, learning_rate = 6.0939885e-05, loss = 0.0001462622, step = 9889 (5.462 sec)\n",
"2021-12-31 02:39:01,817 [INFO] tensorflow: epoch = 103.01041666666666, learning_rate = 6.0939885e-05, loss = 0.0001462622, step = 9889 (5.462 sec)\n",
"INFO:tensorflow:global_step/sec: 3.1054\n",
"2021-12-31 02:39:02,450 [INFO] tensorflow: global_step/sec: 3.1054\n",
"2021-12-31 02:39:04,952 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.151\n",
"INFO:tensorflow:global_step/sec: 3.19359\n",
"2021-12-31 02:39:05,268 [INFO] tensorflow: global_step/sec: 3.19359\n",
"INFO:tensorflow:epoch = 103.1875, learning_rate = 5.937219e-05, loss = 0.00013999503, step = 9906 (5.365 sec)\n",
"2021-12-31 02:39:07,183 [INFO] tensorflow: epoch = 103.1875, learning_rate = 5.937219e-05, loss = 0.00013999503, step = 9906 (5.365 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12605\n",
"2021-12-31 02:39:08,147 [INFO] tensorflow: global_step/sec: 3.12605\n",
"INFO:tensorflow:global_step/sec: 3.14067\n",
"2021-12-31 02:39:11,012 [INFO] tensorflow: global_step/sec: 3.14067\n",
"INFO:tensorflow:epoch = 103.36458333333333, learning_rate = 5.7844707e-05, loss = 0.00011897144, step = 9923 (5.450 sec)\n",
"2021-12-31 02:39:12,632 [INFO] tensorflow: epoch = 103.36458333333333, learning_rate = 5.7844707e-05, loss = 0.00011897144, step = 9923 (5.450 sec)\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"2021-12-31 02:39:12,960 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.976\n",
"INFO:tensorflow:global_step/sec: 3.08258\n",
"2021-12-31 02:39:13,932 [INFO] tensorflow: global_step/sec: 3.08258\n",
"INFO:tensorflow:global_step/sec: 3.1031\n",
"2021-12-31 02:39:16,832 [INFO] tensorflow: global_step/sec: 3.1031\n",
"INFO:tensorflow:epoch = 103.54166666666666, learning_rate = 5.6356632e-05, loss = 0.00013988047, step = 9940 (5.492 sec)\n",
"2021-12-31 02:39:18,124 [INFO] tensorflow: epoch = 103.54166666666666, learning_rate = 5.6356632e-05, loss = 0.00013988047, step = 9940 (5.492 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11806\n",
"2021-12-31 02:39:19,719 [INFO] tensorflow: global_step/sec: 3.11806\n",
"2021-12-31 02:39:20,985 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.922\n",
"INFO:tensorflow:global_step/sec: 3.1404\n",
"2021-12-31 02:39:22,585 [INFO] tensorflow: global_step/sec: 3.1404\n",
"INFO:tensorflow:epoch = 103.71875, learning_rate = 5.490684e-05, loss = 0.00015159059, step = 9957 (5.406 sec)\n",
"2021-12-31 02:39:23,530 [INFO] tensorflow: epoch = 103.71875, learning_rate = 5.490684e-05, loss = 0.00015159059, step = 9957 (5.406 sec)\n",
"INFO:tensorflow:global_step/sec: 3.13295\n",
"2021-12-31 02:39:25,457 [INFO] tensorflow: global_step/sec: 3.13295\n",
"INFO:tensorflow:global_step/sec: 3.10029\n",
"2021-12-31 02:39:28,360 [INFO] tensorflow: global_step/sec: 3.10029\n",
"INFO:tensorflow:epoch = 103.89583333333333, learning_rate = 5.3494296e-05, loss = 0.00014631462, step = 9974 (5.445 sec)\n",
"2021-12-31 02:39:28,975 [INFO] tensorflow: epoch = 103.89583333333333, learning_rate = 5.3494296e-05, loss = 0.00014631462, step = 9974 (5.445 sec)\n",
"2021-12-31 02:39:28,975 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.032\n",
"INFO:tensorflow:global_step/sec: 3.16198\n",
"2021-12-31 02:39:31,207 [INFO] tensorflow: global_step/sec: 3.16198\n",
"2021-12-31 02:39:32,183 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 104/120: loss: 0.00014 learning rate: 0.00005 Time taken: 0:00:30.674472 ETA: 0:08:10.791553\n",
"INFO:tensorflow:global_step/sec: 3.17347\n",
"2021-12-31 02:39:34,043 [INFO] tensorflow: global_step/sec: 3.17347\n",
"INFO:tensorflow:epoch = 104.07291666666666, learning_rate = 5.2118132e-05, loss = 0.00016223939, step = 9991 (5.353 sec)\n",
"2021-12-31 02:39:34,328 [INFO] tensorflow: epoch = 104.07291666666666, learning_rate = 5.2118132e-05, loss = 0.00016223939, step = 9991 (5.353 sec)\n",
"INFO:tensorflow:global_step/sec: 3.14243\n",
"2021-12-31 02:39:36,907 [INFO] tensorflow: global_step/sec: 3.14243\n",
"2021-12-31 02:39:36,907 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.215\n",
"INFO:tensorflow:epoch = 104.25, learning_rate = 5.0777377e-05, loss = 0.00013725745, step = 10008 (5.452 sec)\n",
"2021-12-31 02:39:39,780 [INFO] tensorflow: epoch = 104.25, learning_rate = 5.0777377e-05, loss = 0.00013725745, step = 10008 (5.452 sec)\n",
"INFO:tensorflow:global_step/sec: 3.13107\n",
"2021-12-31 02:39:39,781 [INFO] tensorflow: global_step/sec: 3.13107\n",
"INFO:tensorflow:global_step/sec: 3.16282\n",
"2021-12-31 02:39:42,627 [INFO] tensorflow: global_step/sec: 3.16282\n",
"2021-12-31 02:39:44,884 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.073\n",
"INFO:tensorflow:epoch = 104.42708333333333, learning_rate = 4.9471117e-05, loss = 0.00012815802, step = 10025 (5.428 sec)\n",
"2021-12-31 02:39:45,208 [INFO] tensorflow: epoch = 104.42708333333333, learning_rate = 4.9471117e-05, loss = 0.00012815802, step = 10025 (5.428 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09883\n",
"2021-12-31 02:39:45,531 [INFO] tensorflow: global_step/sec: 3.09883\n",
"INFO:tensorflow:global_step/sec: 3.12152\n",
"2021-12-31 02:39:48,414 [INFO] tensorflow: global_step/sec: 3.12152\n",
"INFO:tensorflow:epoch = 104.60416666666666, learning_rate = 4.8198453e-05, loss = 0.00012748632, step = 10042 (5.365 sec)\n",
"2021-12-31 02:39:50,573 [INFO] tensorflow: epoch = 104.60416666666666, learning_rate = 4.8198453e-05, loss = 0.00012748632, step = 10042 (5.365 sec)\n",
"INFO:tensorflow:global_step/sec: 3.21041\n",
"2021-12-31 02:39:51,218 [INFO] tensorflow: global_step/sec: 3.21041\n",
"2021-12-31 02:39:52,844 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.128\n",
"INFO:tensorflow:global_step/sec: 3.10249\n",
"2021-12-31 02:39:54,119 [INFO] tensorflow: global_step/sec: 3.10249\n",
"INFO:tensorflow:epoch = 104.78125, learning_rate = 4.695849e-05, loss = 0.00012360918, step = 10059 (5.474 sec)\n",
"2021-12-31 02:39:56,048 [INFO] tensorflow: epoch = 104.78125, learning_rate = 4.695849e-05, loss = 0.00012360918, step = 10059 (5.474 sec)\n",
"INFO:tensorflow:global_step/sec: 3.1278\n",
"2021-12-31 02:39:56,996 [INFO] tensorflow: global_step/sec: 3.1278\n",
"INFO:tensorflow:global_step/sec: 3.07065\n",
"2021-12-31 02:39:59,927 [INFO] tensorflow: global_step/sec: 3.07065\n",
"2021-12-31 02:40:00,892 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.852\n",
"INFO:tensorflow:epoch = 104.95833333333333, learning_rate = 4.5750465e-05, loss = 0.0001547207, step = 10076 (5.484 sec)\n",
"2021-12-31 02:40:01,532 [INFO] tensorflow: epoch = 104.95833333333333, learning_rate = 4.5750465e-05, loss = 0.0001547207, step = 10076 (5.484 sec)\n",
"INFO:tensorflow:global_step/sec: 3.1213\n",
"2021-12-31 02:40:02,810 [INFO] tensorflow: global_step/sec: 3.1213\n",
"2021-12-31 02:40:02,811 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 105/120: loss: 0.00017 learning rate: 0.00005 Time taken: 0:00:30.637483 ETA: 0:07:39.562240\n",
"INFO:tensorflow:global_step/sec: 3.14487\n",
"2021-12-31 02:40:05,672 [INFO] tensorflow: global_step/sec: 3.14487\n",
"INFO:tensorflow:epoch = 105.13541666666666, learning_rate = 4.4573477e-05, loss = 0.00016455869, step = 10093 (5.439 sec)\n",
"2021-12-31 02:40:06,971 [INFO] tensorflow: epoch = 105.13541666666666, learning_rate = 4.4573477e-05, loss = 0.00016455869, step = 10093 (5.439 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07484\n",
"2021-12-31 02:40:08,599 [INFO] tensorflow: global_step/sec: 3.07484\n",
"2021-12-31 02:40:08,912 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.938\n",
"INFO:tensorflow:global_step/sec: 3.11763\n",
"2021-12-31 02:40:11,486 [INFO] tensorflow: global_step/sec: 3.11763\n",
"INFO:tensorflow:epoch = 105.3125, learning_rate = 4.342681e-05, loss = 0.00015449108, step = 10110 (5.469 sec)\n",
"2021-12-31 02:40:12,439 [INFO] tensorflow: epoch = 105.3125, learning_rate = 4.342681e-05, loss = 0.00015449108, step = 10110 (5.469 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12221\n",
"2021-12-31 02:40:14,368 [INFO] tensorflow: global_step/sec: 3.12221\n",
"2021-12-31 02:40:16,894 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.059\n",
"INFO:tensorflow:global_step/sec: 3.16376\n",
"2021-12-31 02:40:17,213 [INFO] tensorflow: global_step/sec: 3.16376\n",
"INFO:tensorflow:epoch = 105.48958333333333, learning_rate = 4.230964e-05, loss = 0.00013281552, step = 10127 (5.417 sec)\n",
"2021-12-31 02:40:17,857 [INFO] tensorflow: epoch = 105.48958333333333, learning_rate = 4.230964e-05, loss = 0.00013281552, step = 10127 (5.417 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12792\n",
"2021-12-31 02:40:20,090 [INFO] tensorflow: global_step/sec: 3.12792\n",
"INFO:tensorflow:global_step/sec: 3.13507\n",
"2021-12-31 02:40:22,961 [INFO] tensorflow: global_step/sec: 3.13507\n",
"INFO:tensorflow:epoch = 105.66666666666666, learning_rate = 4.122121e-05, loss = 0.0001839644, step = 10144 (5.439 sec)\n",
"2021-12-31 02:40:23,296 [INFO] tensorflow: epoch = 105.66666666666666, learning_rate = 4.122121e-05, loss = 0.0001839644, step = 10144 (5.439 sec)\n",
"2021-12-31 02:40:24,870 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.076\n",
"INFO:tensorflow:global_step/sec: 3.14295\n",
"2021-12-31 02:40:25,825 [INFO] tensorflow: global_step/sec: 3.14295\n",
"INFO:tensorflow:epoch = 105.84375, learning_rate = 4.016078e-05, loss = 0.0001331382, step = 10161 (5.399 sec)\n",
"2021-12-31 02:40:28,695 [INFO] tensorflow: epoch = 105.84375, learning_rate = 4.016078e-05, loss = 0.0001331382, step = 10161 (5.399 sec)\n",
"INFO:tensorflow:global_step/sec: 3.13447\n",
"2021-12-31 02:40:28,696 [INFO] tensorflow: global_step/sec: 3.13447\n",
"INFO:tensorflow:global_step/sec: 3.11979\n",
"2021-12-31 02:40:31,581 [INFO] tensorflow: global_step/sec: 3.11979\n",
"2021-12-31 02:40:32,869 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.004\n",
"2021-12-31 02:40:33,520 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 106/120: loss: 0.00013 learning rate: 0.00004 Time taken: 0:00:30.709569 ETA: 0:07:09.933959\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO:tensorflow:epoch = 106.02083333333333, learning_rate = 3.9127593e-05, loss = 0.00016553479, step = 10178 (5.472 sec)\n",
"2021-12-31 02:40:34,167 [INFO] tensorflow: epoch = 106.02083333333333, learning_rate = 3.9127593e-05, loss = 0.00016553479, step = 10178 (5.472 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08729\n",
"2021-12-31 02:40:34,496 [INFO] tensorflow: global_step/sec: 3.08729\n",
"INFO:tensorflow:global_step/sec: 3.1182\n",
"2021-12-31 02:40:37,382 [INFO] tensorflow: global_step/sec: 3.1182\n",
"INFO:tensorflow:epoch = 106.19791666666666, learning_rate = 3.8121023e-05, loss = 0.00013836674, step = 10195 (5.445 sec)\n",
"2021-12-31 02:40:39,613 [INFO] tensorflow: epoch = 106.19791666666666, learning_rate = 3.8121023e-05, loss = 0.00013836674, step = 10195 (5.445 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12123\n",
"2021-12-31 02:40:40,266 [INFO] tensorflow: global_step/sec: 3.12123\n",
"2021-12-31 02:40:40,916 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.853\n",
"INFO:tensorflow:global_step/sec: 3.10674\n",
"2021-12-31 02:40:43,163 [INFO] tensorflow: global_step/sec: 3.10674\n",
"INFO:tensorflow:epoch = 106.375, learning_rate = 3.7140348e-05, loss = 0.0001403981, step = 10212 (5.442 sec)\n",
"2021-12-31 02:40:45,055 [INFO] tensorflow: epoch = 106.375, learning_rate = 3.7140348e-05, loss = 0.0001403981, step = 10212 (5.442 sec)\n",
"INFO:tensorflow:global_step/sec: 3.14745\n",
"2021-12-31 02:40:46,022 [INFO] tensorflow: global_step/sec: 3.14745\n",
"INFO:tensorflow:global_step/sec: 3.1443\n",
"2021-12-31 02:40:48,885 [INFO] tensorflow: global_step/sec: 3.1443\n",
"2021-12-31 02:40:48,885 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.097\n",
"INFO:tensorflow:epoch = 106.55208333333333, learning_rate = 3.6184865e-05, loss = 0.00012188773, step = 10229 (5.390 sec)\n",
"2021-12-31 02:40:50,445 [INFO] tensorflow: epoch = 106.55208333333333, learning_rate = 3.6184865e-05, loss = 0.00012188773, step = 10229 (5.390 sec)\n",
"INFO:tensorflow:global_step/sec: 3.22948\n",
"2021-12-31 02:40:51,671 [INFO] tensorflow: global_step/sec: 3.22948\n",
"INFO:tensorflow:global_step/sec: 3.12832\n",
"2021-12-31 02:40:54,548 [INFO] tensorflow: global_step/sec: 3.12832\n",
"INFO:tensorflow:epoch = 106.72916666666666, learning_rate = 3.5253997e-05, loss = 0.0001206238, step = 10246 (5.393 sec)\n",
"2021-12-31 02:40:55,837 [INFO] tensorflow: epoch = 106.72916666666666, learning_rate = 3.5253997e-05, loss = 0.0001206238, step = 10246 (5.393 sec)\n",
"2021-12-31 02:40:56,807 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.247\n",
"INFO:tensorflow:global_step/sec: 3.09806\n",
"2021-12-31 02:40:57,453 [INFO] tensorflow: global_step/sec: 3.09806\n",
"INFO:tensorflow:global_step/sec: 3.16457\n",
"2021-12-31 02:41:00,297 [INFO] tensorflow: global_step/sec: 3.16457\n",
"INFO:tensorflow:epoch = 106.90625, learning_rate = 3.434708e-05, loss = 0.00011907742, step = 10263 (5.434 sec)\n",
"2021-12-31 02:41:01,271 [INFO] tensorflow: epoch = 106.90625, learning_rate = 3.434708e-05, loss = 0.00011907742, step = 10263 (5.434 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09498\n",
"2021-12-31 02:41:03,205 [INFO] tensorflow: global_step/sec: 3.09498\n",
"2021-12-31 02:41:04,173 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 107/120: loss: 0.00013 learning rate: 0.00003 Time taken: 0:00:30.659608 ETA: 0:06:38.574906\n",
"2021-12-31 02:41:04,818 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.967\n",
"INFO:tensorflow:global_step/sec: 3.10806\n",
"2021-12-31 02:41:06,101 [INFO] tensorflow: global_step/sec: 3.10806\n",
"INFO:tensorflow:epoch = 107.08333333333333, learning_rate = 3.346349e-05, loss = 0.00013540336, step = 10280 (5.465 sec)\n",
"2021-12-31 02:41:06,736 [INFO] tensorflow: epoch = 107.08333333333333, learning_rate = 3.346349e-05, loss = 0.00013540336, step = 10280 (5.465 sec)\n",
"INFO:tensorflow:global_step/sec: 3.13969\n",
"2021-12-31 02:41:08,968 [INFO] tensorflow: global_step/sec: 3.13969\n",
"INFO:tensorflow:global_step/sec: 3.0524\n",
"2021-12-31 02:41:11,916 [INFO] tensorflow: global_step/sec: 3.0524\n",
"INFO:tensorflow:epoch = 107.26041666666666, learning_rate = 3.2602566e-05, loss = 0.00012322151, step = 10297 (5.492 sec)\n",
"2021-12-31 02:41:12,228 [INFO] tensorflow: epoch = 107.26041666666666, learning_rate = 3.2602566e-05, loss = 0.00012322151, step = 10297 (5.492 sec)\n",
"2021-12-31 02:41:12,886 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.790\n",
"INFO:tensorflow:global_step/sec: 3.09782\n",
"2021-12-31 02:41:14,821 [INFO] tensorflow: global_step/sec: 3.09782\n",
"INFO:tensorflow:epoch = 107.4375, learning_rate = 3.1763855e-05, loss = 0.000126844, step = 10314 (5.481 sec)\n",
"2021-12-31 02:41:17,710 [INFO] tensorflow: epoch = 107.4375, learning_rate = 3.1763855e-05, loss = 0.000126844, step = 10314 (5.481 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11487\n",
"2021-12-31 02:41:17,711 [INFO] tensorflow: global_step/sec: 3.11487\n",
"INFO:tensorflow:global_step/sec: 3.12406\n",
"2021-12-31 02:41:20,591 [INFO] tensorflow: global_step/sec: 3.12406\n",
"2021-12-31 02:41:20,922 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.889\n",
"INFO:tensorflow:epoch = 107.61458333333333, learning_rate = 3.0946718e-05, loss = 0.000155205, step = 10331 (5.428 sec)\n",
"2021-12-31 02:41:23,138 [INFO] tensorflow: epoch = 107.61458333333333, learning_rate = 3.0946718e-05, loss = 0.000155205, step = 10331 (5.428 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12549\n",
"2021-12-31 02:41:23,471 [INFO] tensorflow: global_step/sec: 3.12549\n",
"INFO:tensorflow:global_step/sec: 3.09736\n",
"2021-12-31 02:41:26,377 [INFO] tensorflow: global_step/sec: 3.09736\n",
"INFO:tensorflow:epoch = 107.79166666666666, learning_rate = 3.0150577e-05, loss = 0.00012097623, step = 10348 (5.516 sec)\n",
"2021-12-31 02:41:28,654 [INFO] tensorflow: epoch = 107.79166666666666, learning_rate = 3.0150577e-05, loss = 0.00012097623, step = 10348 (5.516 sec)\n",
"2021-12-31 02:41:28,978 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.827\n",
"INFO:tensorflow:global_step/sec: 3.08324\n",
"2021-12-31 02:41:29,296 [INFO] tensorflow: global_step/sec: 3.08324\n",
"INFO:tensorflow:global_step/sec: 3.1281\n",
"2021-12-31 02:41:32,173 [INFO] tensorflow: global_step/sec: 3.1281\n",
"INFO:tensorflow:epoch = 107.96875, learning_rate = 2.9374944e-05, loss = 0.00012659443, step = 10365 (5.427 sec)\n",
"2021-12-31 02:41:34,081 [INFO] tensorflow: epoch = 107.96875, learning_rate = 2.9374944e-05, loss = 0.00012659443, step = 10365 (5.427 sec)\n",
"INFO:tensorflow:global_step/sec: 3.15999\n",
"2021-12-31 02:41:35,021 [INFO] tensorflow: global_step/sec: 3.15999\n",
"2021-12-31 02:41:35,022 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 108/120: loss: 0.00015 learning rate: 0.00003 Time taken: 0:00:30.846182 ETA: 0:06:10.154188\n",
"2021-12-31 02:41:36,953 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.079\n",
"INFO:tensorflow:global_step/sec: 3.11094\n",
"2021-12-31 02:41:37,914 [INFO] tensorflow: global_step/sec: 3.11094\n",
"INFO:tensorflow:epoch = 108.14583333333333, learning_rate = 2.8619263e-05, loss = 0.00016044779, step = 10382 (5.514 sec)\n",
"2021-12-31 02:41:39,595 [INFO] tensorflow: epoch = 108.14583333333333, learning_rate = 2.8619263e-05, loss = 0.00016044779, step = 10382 (5.514 sec)\n",
"INFO:tensorflow:global_step/sec: 3.04494\n",
"2021-12-31 02:41:40,870 [INFO] tensorflow: global_step/sec: 3.04494\n",
"INFO:tensorflow:global_step/sec: 3.13631\n",
"2021-12-31 02:41:43,739 [INFO] tensorflow: global_step/sec: 3.13631\n",
"INFO:tensorflow:epoch = 108.32291666666666, learning_rate = 2.7882998e-05, loss = 0.00015443587, step = 10399 (5.430 sec)\n",
"2021-12-31 02:41:45,025 [INFO] tensorflow: epoch = 108.32291666666666, learning_rate = 2.7882998e-05, loss = 0.00015443587, step = 10399 (5.430 sec)\n",
"2021-12-31 02:41:45,025 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.779\n",
"INFO:tensorflow:global_step/sec: 3.11743\n",
"2021-12-31 02:41:46,626 [INFO] tensorflow: global_step/sec: 3.11743\n",
"INFO:tensorflow:global_step/sec: 3.12798\n",
"2021-12-31 02:41:49,504 [INFO] tensorflow: global_step/sec: 3.12798\n",
"INFO:tensorflow:epoch = 108.5, learning_rate = 2.7165697e-05, loss = 0.00012829533, step = 10416 (5.431 sec)\n",
"2021-12-31 02:41:50,455 [INFO] tensorflow: epoch = 108.5, learning_rate = 2.7165697e-05, loss = 0.00012829533, step = 10416 (5.431 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10803\n",
"2021-12-31 02:41:52,399 [INFO] tensorflow: global_step/sec: 3.10803\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"2021-12-31 02:41:53,041 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.951\n",
"INFO:tensorflow:global_step/sec: 3.11986\n",
"2021-12-31 02:41:55,284 [INFO] tensorflow: global_step/sec: 3.11986\n",
"INFO:tensorflow:epoch = 108.67708333333333, learning_rate = 2.646685e-05, loss = 0.00013514244, step = 10433 (5.449 sec)\n",
"2021-12-31 02:41:55,904 [INFO] tensorflow: epoch = 108.67708333333333, learning_rate = 2.646685e-05, loss = 0.00013514244, step = 10433 (5.449 sec)\n",
"INFO:tensorflow:global_step/sec: 3.14825\n",
"2021-12-31 02:41:58,143 [INFO] tensorflow: global_step/sec: 3.14825\n",
"INFO:tensorflow:global_step/sec: 3.13563\n",
"2021-12-31 02:42:01,013 [INFO] tensorflow: global_step/sec: 3.13563\n",
"2021-12-31 02:42:01,014 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.086\n",
"INFO:tensorflow:epoch = 108.85416666666666, learning_rate = 2.5785983e-05, loss = 0.00013020061, step = 10450 (5.438 sec)\n",
"2021-12-31 02:42:01,342 [INFO] tensorflow: epoch = 108.85416666666666, learning_rate = 2.5785983e-05, loss = 0.00013020061, step = 10450 (5.438 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07539\n",
"2021-12-31 02:42:03,940 [INFO] tensorflow: global_step/sec: 3.07539\n",
"2021-12-31 02:42:05,890 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 109/120: loss: 0.00015 learning rate: 0.00003 Time taken: 0:00:30.862455 ETA: 0:05:39.487004\n",
"INFO:tensorflow:epoch = 109.03125, learning_rate = 2.5122605e-05, loss = 0.00012909074, step = 10467 (5.516 sec)\n",
"2021-12-31 02:42:06,858 [INFO] tensorflow: epoch = 109.03125, learning_rate = 2.5122605e-05, loss = 0.00012909074, step = 10467 (5.516 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08307\n",
"2021-12-31 02:42:06,859 [INFO] tensorflow: global_step/sec: 3.08307\n",
"2021-12-31 02:42:09,089 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.769\n",
"INFO:tensorflow:global_step/sec: 3.12695\n",
"2021-12-31 02:42:09,737 [INFO] tensorflow: global_step/sec: 3.12695\n",
"INFO:tensorflow:epoch = 109.20833333333333, learning_rate = 2.4476318e-05, loss = 0.00013230942, step = 10484 (5.438 sec)\n",
"2021-12-31 02:42:12,296 [INFO] tensorflow: epoch = 109.20833333333333, learning_rate = 2.4476318e-05, loss = 0.00013230942, step = 10484 (5.438 sec)\n",
"INFO:tensorflow:global_step/sec: 3.13281\n",
"2021-12-31 02:42:12,610 [INFO] tensorflow: global_step/sec: 3.13281\n",
"INFO:tensorflow:global_step/sec: 3.10519\n",
"2021-12-31 02:42:15,508 [INFO] tensorflow: global_step/sec: 3.10519\n",
"2021-12-31 02:42:17,084 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.016\n",
"INFO:tensorflow:epoch = 109.38541666666666, learning_rate = 2.3846656e-05, loss = 0.000113790724, step = 10501 (5.434 sec)\n",
"2021-12-31 02:42:17,730 [INFO] tensorflow: epoch = 109.38541666666666, learning_rate = 2.3846656e-05, loss = 0.000113790724, step = 10501 (5.434 sec)\n",
"INFO:tensorflow:global_step/sec: 3.13156\n",
"2021-12-31 02:42:18,382 [INFO] tensorflow: global_step/sec: 3.13156\n",
"INFO:tensorflow:global_step/sec: 3.12688\n",
"2021-12-31 02:42:21,260 [INFO] tensorflow: global_step/sec: 3.12688\n",
"INFO:tensorflow:epoch = 109.5625, learning_rate = 2.3233171e-05, loss = 0.0001361549, step = 10518 (5.483 sec)\n",
"2021-12-31 02:42:23,213 [INFO] tensorflow: epoch = 109.5625, learning_rate = 2.3233171e-05, loss = 0.0001361549, step = 10518 (5.483 sec)\n",
"INFO:tensorflow:global_step/sec: 3.06829\n",
"2021-12-31 02:42:24,194 [INFO] tensorflow: global_step/sec: 3.06829\n",
"2021-12-31 02:42:25,179 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.708\n",
"INFO:tensorflow:global_step/sec: 3.12125\n",
"2021-12-31 02:42:27,077 [INFO] tensorflow: global_step/sec: 3.12125\n",
"INFO:tensorflow:epoch = 109.73958333333333, learning_rate = 2.2635491e-05, loss = 0.00017456386, step = 10535 (5.467 sec)\n",
"2021-12-31 02:42:28,680 [INFO] tensorflow: epoch = 109.73958333333333, learning_rate = 2.2635491e-05, loss = 0.00017456386, step = 10535 (5.467 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11145\n",
"2021-12-31 02:42:29,970 [INFO] tensorflow: global_step/sec: 3.11145\n",
"INFO:tensorflow:global_step/sec: 3.16919\n",
"2021-12-31 02:42:32,809 [INFO] tensorflow: global_step/sec: 3.16919\n",
"2021-12-31 02:42:33,133 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.145\n",
"INFO:tensorflow:epoch = 109.91666666666666, learning_rate = 2.2053186e-05, loss = 0.000117931166, step = 10552 (5.433 sec)\n",
"2021-12-31 02:42:34,112 [INFO] tensorflow: epoch = 109.91666666666666, learning_rate = 2.2053186e-05, loss = 0.000117931166, step = 10552 (5.433 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10318\n",
"2021-12-31 02:42:35,710 [INFO] tensorflow: global_step/sec: 3.10318\n",
"INFO:tensorflow:Saving checkpoints for step-10560.\n",
"2021-12-31 02:42:36,345 [INFO] tensorflow: Saving checkpoints for step-10560.\n",
"WARNING:tensorflow:Ignoring: /tmp/tmp3csn9e7j; No such file or directory\n",
"2021-12-31 02:42:36,499 [WARNING] tensorflow: Ignoring: /tmp/tmp3csn9e7j; No such file or directory\n",
"2021-12-31 02:42:39,998 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 23, 0.00s/step\n",
"2021-12-31 02:42:41,544 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 23, 0.15s/step\n",
"2021-12-31 02:42:42,991 [INFO] iva.detectnet_v2.evaluation.evaluation: step 20 / 23, 0.14s/step\n",
"Matching predictions to ground truth, class 1/1.: 100%|█| 939/939 [00:00<00:00, 15218.89it/s]\n",
"Epoch 110/120\n",
"=========================\n",
"\n",
"Validation cost: 0.000164\n",
"Mean average_precision (in %): 92.9904\n",
"\n",
"class name average precision (in %)\n",
"------------ --------------------------\n",
"car 92.9904\n",
"\n",
"Median Inference Time: 0.015517\n",
"INFO:tensorflow:epoch = 110.0, learning_rate = 2.1784352e-05, loss = 0.00014489322, step = 10560 (9.761 sec)\n",
"2021-12-31 02:42:43,873 [INFO] tensorflow: epoch = 110.0, learning_rate = 2.1784352e-05, loss = 0.00014489322, step = 10560 (9.761 sec)\n",
"2021-12-31 02:42:43,873 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 110/120: loss: 0.00014 learning rate: 0.00002 Time taken: 0:00:37.973679 ETA: 0:06:19.736793\n",
"INFO:tensorflow:global_step/sec: 0.8902\n",
"2021-12-31 02:42:45,820 [INFO] tensorflow: global_step/sec: 0.8902\n",
"2021-12-31 02:42:48,437 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 13.069\n",
"INFO:tensorflow:global_step/sec: 3.05759\n",
"2021-12-31 02:42:48,763 [INFO] tensorflow: global_step/sec: 3.05759\n",
"INFO:tensorflow:epoch = 110.17708333333333, learning_rate = 2.1223941e-05, loss = 0.00012704372, step = 10577 (5.527 sec)\n",
"2021-12-31 02:42:49,399 [INFO] tensorflow: epoch = 110.17708333333333, learning_rate = 2.1223941e-05, loss = 0.00012704372, step = 10577 (5.527 sec)\n",
"INFO:tensorflow:global_step/sec: 3.13433\n",
"2021-12-31 02:42:51,635 [INFO] tensorflow: global_step/sec: 3.13433\n",
"INFO:tensorflow:global_step/sec: 3.07144\n",
"2021-12-31 02:42:54,565 [INFO] tensorflow: global_step/sec: 3.07144\n",
"INFO:tensorflow:epoch = 110.35416666666666, learning_rate = 2.0677948e-05, loss = 0.00016754193, step = 10594 (5.478 sec)\n",
"2021-12-31 02:42:54,877 [INFO] tensorflow: epoch = 110.35416666666666, learning_rate = 2.0677948e-05, loss = 0.00016754193, step = 10594 (5.478 sec)\n",
"2021-12-31 02:42:56,485 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.850\n",
"INFO:tensorflow:global_step/sec: 3.16399\n",
"2021-12-31 02:42:57,409 [INFO] tensorflow: global_step/sec: 3.16399\n",
"INFO:tensorflow:epoch = 110.53125, learning_rate = 2.0145982e-05, loss = 0.00017092431, step = 10611 (5.436 sec)\n",
"2021-12-31 02:43:00,312 [INFO] tensorflow: epoch = 110.53125, learning_rate = 2.0145982e-05, loss = 0.00017092431, step = 10611 (5.436 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09949\n",
"2021-12-31 02:43:00,313 [INFO] tensorflow: global_step/sec: 3.09949\n",
"INFO:tensorflow:global_step/sec: 3.03918\n",
"2021-12-31 02:43:03,274 [INFO] tensorflow: global_step/sec: 3.03918\n",
"2021-12-31 02:43:04,528 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.866\n",
"INFO:tensorflow:epoch = 110.70833333333333, learning_rate = 1.962772e-05, loss = 0.0001525943, step = 10628 (5.495 sec)\n",
"2021-12-31 02:43:05,807 [INFO] tensorflow: epoch = 110.70833333333333, learning_rate = 1.962772e-05, loss = 0.0001525943, step = 10628 (5.495 sec)\n",
"INFO:tensorflow:global_step/sec: 3.14576\n",
"2021-12-31 02:43:06,135 [INFO] tensorflow: global_step/sec: 3.14576\n",
"INFO:tensorflow:global_step/sec: 3.11573\n",
"2021-12-31 02:43:09,024 [INFO] tensorflow: global_step/sec: 3.11573\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO:tensorflow:epoch = 110.88541666666666, learning_rate = 1.912279e-05, loss = 0.00013265897, step = 10645 (5.434 sec)\n",
"2021-12-31 02:43:11,241 [INFO] tensorflow: epoch = 110.88541666666666, learning_rate = 1.912279e-05, loss = 0.00013265897, step = 10645 (5.434 sec)\n",
"INFO:tensorflow:global_step/sec: 3.15653\n",
"2021-12-31 02:43:11,875 [INFO] tensorflow: global_step/sec: 3.15653\n",
"2021-12-31 02:43:12,500 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.088\n",
"INFO:tensorflow:global_step/sec: 3.08985\n",
"2021-12-31 02:43:14,788 [INFO] tensorflow: global_step/sec: 3.08985\n",
"2021-12-31 02:43:14,789 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 111/120: loss: 0.00016 learning rate: 0.00002 Time taken: 0:00:30.926534 ETA: 0:04:38.338803\n",
"INFO:tensorflow:epoch = 111.0625, learning_rate = 1.863085e-05, loss = 0.00014225669, step = 10662 (5.450 sec)\n",
"2021-12-31 02:43:16,691 [INFO] tensorflow: epoch = 111.0625, learning_rate = 1.863085e-05, loss = 0.00014225669, step = 10662 (5.450 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12967\n",
"2021-12-31 02:43:17,664 [INFO] tensorflow: global_step/sec: 3.12967\n",
"INFO:tensorflow:global_step/sec: 3.13493\n",
"2021-12-31 02:43:20,535 [INFO] tensorflow: global_step/sec: 3.13493\n",
"2021-12-31 02:43:20,535 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.893\n",
"INFO:tensorflow:epoch = 111.23958333333333, learning_rate = 1.8151566e-05, loss = 0.00011466992, step = 10679 (5.421 sec)\n",
"2021-12-31 02:43:22,112 [INFO] tensorflow: epoch = 111.23958333333333, learning_rate = 1.8151566e-05, loss = 0.00011466992, step = 10679 (5.421 sec)\n",
"INFO:tensorflow:global_step/sec: 3.13755\n",
"2021-12-31 02:43:23,403 [INFO] tensorflow: global_step/sec: 3.13755\n",
"INFO:tensorflow:global_step/sec: 3.13025\n",
"2021-12-31 02:43:26,278 [INFO] tensorflow: global_step/sec: 3.13025\n",
"INFO:tensorflow:epoch = 111.41666666666666, learning_rate = 1.7684593e-05, loss = 0.0001130897, step = 10696 (5.453 sec)\n",
"2021-12-31 02:43:27,565 [INFO] tensorflow: epoch = 111.41666666666666, learning_rate = 1.7684593e-05, loss = 0.0001130897, step = 10696 (5.453 sec)\n",
"2021-12-31 02:43:28,530 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.016\n",
"INFO:tensorflow:global_step/sec: 3.112\n",
"2021-12-31 02:43:29,170 [INFO] tensorflow: global_step/sec: 3.112\n",
"INFO:tensorflow:global_step/sec: 3.10371\n",
"2021-12-31 02:43:32,070 [INFO] tensorflow: global_step/sec: 3.10371\n",
"INFO:tensorflow:epoch = 111.59375, learning_rate = 1.7229651e-05, loss = 0.00014945658, step = 10713 (5.494 sec)\n",
"2021-12-31 02:43:33,059 [INFO] tensorflow: epoch = 111.59375, learning_rate = 1.7229651e-05, loss = 0.00014945658, step = 10713 (5.494 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10865\n",
"2021-12-31 02:43:34,965 [INFO] tensorflow: global_step/sec: 3.10865\n",
"2021-12-31 02:43:36,595 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.802\n",
"INFO:tensorflow:global_step/sec: 3.1058\n",
"2021-12-31 02:43:37,863 [INFO] tensorflow: global_step/sec: 3.1058\n",
"INFO:tensorflow:epoch = 111.77083333333333, learning_rate = 1.6786413e-05, loss = 0.0001740176, step = 10730 (5.463 sec)\n",
"2021-12-31 02:43:38,522 [INFO] tensorflow: epoch = 111.77083333333333, learning_rate = 1.6786413e-05, loss = 0.0001740176, step = 10730 (5.463 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12532\n",
"2021-12-31 02:43:40,743 [INFO] tensorflow: global_step/sec: 3.12532\n",
"INFO:tensorflow:global_step/sec: 3.15953\n",
"2021-12-31 02:43:43,591 [INFO] tensorflow: global_step/sec: 3.15953\n",
"INFO:tensorflow:epoch = 111.94791666666666, learning_rate = 1.6354546e-05, loss = 0.00012229677, step = 10747 (5.371 sec)\n",
"2021-12-31 02:43:43,893 [INFO] tensorflow: epoch = 111.94791666666666, learning_rate = 1.6354546e-05, loss = 0.00012229677, step = 10747 (5.371 sec)\n",
"2021-12-31 02:43:44,528 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.211\n",
"2021-12-31 02:43:45,471 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 112/120: loss: 0.00015 learning rate: 0.00002 Time taken: 0:00:30.700016 ETA: 0:04:05.600126\n",
"INFO:tensorflow:global_step/sec: 3.19094\n",
"2021-12-31 02:43:46,412 [INFO] tensorflow: global_step/sec: 3.19094\n",
"INFO:tensorflow:epoch = 112.125, learning_rate = 1.593382e-05, loss = 0.00012531076, step = 10764 (5.403 sec)\n",
"2021-12-31 02:43:49,295 [INFO] tensorflow: epoch = 112.125, learning_rate = 1.593382e-05, loss = 0.00012531076, step = 10764 (5.403 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12002\n",
"2021-12-31 02:43:49,296 [INFO] tensorflow: global_step/sec: 3.12002\n",
"INFO:tensorflow:global_step/sec: 3.16912\n",
"2021-12-31 02:43:52,136 [INFO] tensorflow: global_step/sec: 3.16912\n",
"2021-12-31 02:43:52,454 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.233\n",
"INFO:tensorflow:epoch = 112.30208333333333, learning_rate = 1.5523916e-05, loss = 0.00013217182, step = 10781 (5.418 sec)\n",
"2021-12-31 02:43:54,714 [INFO] tensorflow: epoch = 112.30208333333333, learning_rate = 1.5523916e-05, loss = 0.00013217182, step = 10781 (5.418 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09852\n",
"2021-12-31 02:43:55,041 [INFO] tensorflow: global_step/sec: 3.09852\n",
"INFO:tensorflow:global_step/sec: 3.12208\n",
"2021-12-31 02:43:57,923 [INFO] tensorflow: global_step/sec: 3.12208\n",
"INFO:tensorflow:epoch = 112.47916666666666, learning_rate = 1.5124544e-05, loss = 0.00011502146, step = 10798 (5.422 sec)\n",
"2021-12-31 02:44:00,136 [INFO] tensorflow: epoch = 112.47916666666666, learning_rate = 1.5124544e-05, loss = 0.00011502146, step = 10798 (5.422 sec)\n",
"2021-12-31 02:44:00,454 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.001\n",
"INFO:tensorflow:global_step/sec: 3.14032\n",
"2021-12-31 02:44:00,789 [INFO] tensorflow: global_step/sec: 3.14032\n",
"INFO:tensorflow:global_step/sec: 3.08947\n",
"2021-12-31 02:44:03,702 [INFO] tensorflow: global_step/sec: 3.08947\n",
"INFO:tensorflow:epoch = 112.65625, learning_rate = 1.473546e-05, loss = 0.00015259066, step = 10815 (5.513 sec)\n",
"2021-12-31 02:44:05,649 [INFO] tensorflow: epoch = 112.65625, learning_rate = 1.473546e-05, loss = 0.00015259066, step = 10815 (5.513 sec)\n",
"INFO:tensorflow:global_step/sec: 3.04217\n",
"2021-12-31 02:44:06,661 [INFO] tensorflow: global_step/sec: 3.04217\n",
"2021-12-31 02:44:08,583 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.605\n",
"INFO:tensorflow:global_step/sec: 3.13872\n",
"2021-12-31 02:44:09,528 [INFO] tensorflow: global_step/sec: 3.13872\n",
"INFO:tensorflow:epoch = 112.83333333333333, learning_rate = 1.4356387e-05, loss = 0.00012910762, step = 10832 (5.474 sec)\n",
"2021-12-31 02:44:11,123 [INFO] tensorflow: epoch = 112.83333333333333, learning_rate = 1.4356387e-05, loss = 0.00012910762, step = 10832 (5.474 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11992\n",
"2021-12-31 02:44:12,413 [INFO] tensorflow: global_step/sec: 3.11992\n",
"INFO:tensorflow:global_step/sec: 3.07851\n",
"2021-12-31 02:44:15,337 [INFO] tensorflow: global_step/sec: 3.07851\n",
"2021-12-31 02:44:16,316 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 113/120: loss: 0.00016 learning rate: 0.00001 Time taken: 0:00:30.819370 ETA: 0:03:35.735589\n",
"INFO:tensorflow:epoch = 113.01041666666666, learning_rate = 1.39870635e-05, loss = 0.00013738149, step = 10849 (5.516 sec)\n",
"2021-12-31 02:44:16,638 [INFO] tensorflow: epoch = 113.01041666666666, learning_rate = 1.39870635e-05, loss = 0.00013738149, step = 10849 (5.516 sec)\n",
"2021-12-31 02:44:16,639 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.827\n",
"INFO:tensorflow:global_step/sec: 3.07559\n",
"2021-12-31 02:44:18,263 [INFO] tensorflow: global_step/sec: 3.07559\n",
"INFO:tensorflow:global_step/sec: 3.09438\n",
"2021-12-31 02:44:21,171 [INFO] tensorflow: global_step/sec: 3.09438\n",
"INFO:tensorflow:epoch = 113.1875, learning_rate = 1.3627229e-05, loss = 0.00014177755, step = 10866 (5.515 sec)\n",
"2021-12-31 02:44:22,154 [INFO] tensorflow: epoch = 113.1875, learning_rate = 1.3627229e-05, loss = 0.00014177755, step = 10866 (5.515 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09556\n",
"2021-12-31 02:44:24,079 [INFO] tensorflow: global_step/sec: 3.09556\n",
"2021-12-31 02:44:24,712 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.774\n",
"INFO:tensorflow:global_step/sec: 3.09097\n",
"2021-12-31 02:44:26,990 [INFO] tensorflow: global_step/sec: 3.09097\n",
"INFO:tensorflow:epoch = 113.36458333333333, learning_rate = 1.3276664e-05, loss = 0.00014899964, step = 10883 (5.485 sec)\n",
"2021-12-31 02:44:27,639 [INFO] tensorflow: epoch = 113.36458333333333, learning_rate = 1.3276664e-05, loss = 0.00014899964, step = 10883 (5.485 sec)\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO:tensorflow:global_step/sec: 3.10294\n",
"2021-12-31 02:44:29,891 [INFO] tensorflow: global_step/sec: 3.10294\n",
"INFO:tensorflow:global_step/sec: 3.06893\n",
"2021-12-31 02:44:32,823 [INFO] tensorflow: global_step/sec: 3.06893\n",
"2021-12-31 02:44:32,824 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.655\n",
"INFO:tensorflow:epoch = 113.54166666666666, learning_rate = 1.2935117e-05, loss = 0.00013780282, step = 10900 (5.493 sec)\n",
"2021-12-31 02:44:33,132 [INFO] tensorflow: epoch = 113.54166666666666, learning_rate = 1.2935117e-05, loss = 0.00013780282, step = 10900 (5.493 sec)\n",
"INFO:tensorflow:global_step/sec: 3.19901\n",
"2021-12-31 02:44:35,637 [INFO] tensorflow: global_step/sec: 3.19901\n",
"INFO:tensorflow:epoch = 113.71875, learning_rate = 1.2602345e-05, loss = 0.0001215701, step = 10917 (5.429 sec)\n",
"2021-12-31 02:44:38,560 [INFO] tensorflow: epoch = 113.71875, learning_rate = 1.2602345e-05, loss = 0.0001215701, step = 10917 (5.429 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07774\n",
"2021-12-31 02:44:38,561 [INFO] tensorflow: global_step/sec: 3.07774\n",
"2021-12-31 02:44:40,785 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.124\n",
"INFO:tensorflow:global_step/sec: 3.14853\n",
"2021-12-31 02:44:41,420 [INFO] tensorflow: global_step/sec: 3.14853\n",
"INFO:tensorflow:epoch = 113.89583333333333, learning_rate = 1.2278146e-05, loss = 0.0001235164, step = 10934 (5.443 sec)\n",
"2021-12-31 02:44:44,003 [INFO] tensorflow: epoch = 113.89583333333333, learning_rate = 1.2278146e-05, loss = 0.0001235164, step = 10934 (5.443 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10904\n",
"2021-12-31 02:44:44,314 [INFO] tensorflow: global_step/sec: 3.10904\n",
"INFO:tensorflow:global_step/sec: 3.16273\n",
"2021-12-31 02:44:47,160 [INFO] tensorflow: global_step/sec: 3.16273\n",
"2021-12-31 02:44:47,161 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 114/120: loss: 0.00012 learning rate: 0.00001 Time taken: 0:00:30.868716 ETA: 0:03:05.212297\n",
"2021-12-31 02:44:48,780 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.016\n",
"INFO:tensorflow:epoch = 114.07291666666666, learning_rate = 1.19622855e-05, loss = 0.00017079899, step = 10951 (5.427 sec)\n",
"2021-12-31 02:44:49,431 [INFO] tensorflow: epoch = 114.07291666666666, learning_rate = 1.19622855e-05, loss = 0.00017079899, step = 10951 (5.427 sec)\n",
"INFO:tensorflow:global_step/sec: 3.08014\n",
"2021-12-31 02:44:50,082 [INFO] tensorflow: global_step/sec: 3.08014\n",
"INFO:tensorflow:global_step/sec: 3.12409\n",
"2021-12-31 02:44:52,963 [INFO] tensorflow: global_step/sec: 3.12409\n",
"INFO:tensorflow:epoch = 114.25, learning_rate = 1.1654552e-05, loss = 0.00017020533, step = 10968 (5.418 sec)\n",
"2021-12-31 02:44:54,849 [INFO] tensorflow: epoch = 114.25, learning_rate = 1.1654552e-05, loss = 0.00017020533, step = 10968 (5.418 sec)\n",
"INFO:tensorflow:global_step/sec: 3.13838\n",
"2021-12-31 02:44:55,830 [INFO] tensorflow: global_step/sec: 3.13838\n",
"2021-12-31 02:44:56,806 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.919\n",
"INFO:tensorflow:global_step/sec: 3.13382\n",
"2021-12-31 02:44:58,702 [INFO] tensorflow: global_step/sec: 3.13382\n",
"INFO:tensorflow:epoch = 114.42708333333333, learning_rate = 1.1354724e-05, loss = 0.00013432735, step = 10985 (5.450 sec)\n",
"2021-12-31 02:45:00,299 [INFO] tensorflow: epoch = 114.42708333333333, learning_rate = 1.1354724e-05, loss = 0.00013432735, step = 10985 (5.450 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11347\n",
"2021-12-31 02:45:01,593 [INFO] tensorflow: global_step/sec: 3.11347\n",
"INFO:tensorflow:global_step/sec: 3.11882\n",
"2021-12-31 02:45:04,479 [INFO] tensorflow: global_step/sec: 3.11882\n",
"2021-12-31 02:45:04,776 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.096\n",
"INFO:tensorflow:epoch = 114.60416666666666, learning_rate = 1.106262e-05, loss = 0.00014453108, step = 11002 (5.437 sec)\n",
"2021-12-31 02:45:05,736 [INFO] tensorflow: epoch = 114.60416666666666, learning_rate = 1.106262e-05, loss = 0.00014453108, step = 11002 (5.437 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11992\n",
"2021-12-31 02:45:07,363 [INFO] tensorflow: global_step/sec: 3.11992\n",
"INFO:tensorflow:global_step/sec: 3.12093\n",
"2021-12-31 02:45:10,247 [INFO] tensorflow: global_step/sec: 3.12093\n",
"INFO:tensorflow:epoch = 114.78125, learning_rate = 1.07780315e-05, loss = 0.00012335094, step = 11019 (5.470 sec)\n",
"2021-12-31 02:45:11,206 [INFO] tensorflow: epoch = 114.78125, learning_rate = 1.07780315e-05, loss = 0.00012335094, step = 11019 (5.470 sec)\n",
"2021-12-31 02:45:12,803 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.917\n",
"INFO:tensorflow:global_step/sec: 3.13043\n",
"2021-12-31 02:45:13,122 [INFO] tensorflow: global_step/sec: 3.13043\n",
"INFO:tensorflow:global_step/sec: 3.10208\n",
"2021-12-31 02:45:16,023 [INFO] tensorflow: global_step/sec: 3.10208\n",
"INFO:tensorflow:epoch = 114.95833333333333, learning_rate = 1.0500752e-05, loss = 0.00014424657, step = 11036 (5.464 sec)\n",
"2021-12-31 02:45:16,670 [INFO] tensorflow: epoch = 114.95833333333333, learning_rate = 1.0500752e-05, loss = 0.00014424657, step = 11036 (5.464 sec)\n",
"2021-12-31 02:45:17,939 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 115/120: loss: 0.00015 learning rate: 0.00001 Time taken: 0:00:30.788598 ETA: 0:02:33.942991\n",
"INFO:tensorflow:global_step/sec: 3.12618\n",
"2021-12-31 02:45:18,902 [INFO] tensorflow: global_step/sec: 3.12618\n",
"2021-12-31 02:45:20,844 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.872\n",
"INFO:tensorflow:global_step/sec: 3.09159\n",
"2021-12-31 02:45:21,814 [INFO] tensorflow: global_step/sec: 3.09159\n",
"INFO:tensorflow:epoch = 115.13541666666666, learning_rate = 1.0230618e-05, loss = 0.00014637099, step = 11053 (5.461 sec)\n",
"2021-12-31 02:45:22,130 [INFO] tensorflow: epoch = 115.13541666666666, learning_rate = 1.0230618e-05, loss = 0.00014637099, step = 11053 (5.461 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09214\n",
"2021-12-31 02:45:24,724 [INFO] tensorflow: global_step/sec: 3.09214\n",
"INFO:tensorflow:epoch = 115.3125, learning_rate = 9.967432e-06, loss = 0.00012523418, step = 11070 (5.459 sec)\n",
"2021-12-31 02:45:27,589 [INFO] tensorflow: epoch = 115.3125, learning_rate = 9.967432e-06, loss = 0.00012523418, step = 11070 (5.459 sec)\n",
"INFO:tensorflow:global_step/sec: 3.1405\n",
"2021-12-31 02:45:27,590 [INFO] tensorflow: global_step/sec: 3.1405\n",
"2021-12-31 02:45:28,902 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.821\n",
"INFO:tensorflow:global_step/sec: 3.11704\n",
"2021-12-31 02:45:30,477 [INFO] tensorflow: global_step/sec: 3.11704\n",
"INFO:tensorflow:epoch = 115.48958333333333, learning_rate = 9.711016e-06, loss = 0.0001291036, step = 11087 (5.405 sec)\n",
"2021-12-31 02:45:32,994 [INFO] tensorflow: epoch = 115.48958333333333, learning_rate = 9.711016e-06, loss = 0.0001291036, step = 11087 (5.405 sec)\n",
"INFO:tensorflow:global_step/sec: 3.17917\n",
"2021-12-31 02:45:33,308 [INFO] tensorflow: global_step/sec: 3.17917\n",
"INFO:tensorflow:global_step/sec: 3.11551\n",
"2021-12-31 02:45:36,197 [INFO] tensorflow: global_step/sec: 3.11551\n",
"2021-12-31 02:45:36,845 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.180\n",
"INFO:tensorflow:epoch = 115.66666666666666, learning_rate = 9.461188e-06, loss = 0.00011749414, step = 11104 (5.421 sec)\n",
"2021-12-31 02:45:38,415 [INFO] tensorflow: epoch = 115.66666666666666, learning_rate = 9.461188e-06, loss = 0.00011749414, step = 11104 (5.421 sec)\n",
"INFO:tensorflow:global_step/sec: 3.14649\n",
"2021-12-31 02:45:39,057 [INFO] tensorflow: global_step/sec: 3.14649\n",
"INFO:tensorflow:global_step/sec: 3.10549\n",
"2021-12-31 02:45:41,955 [INFO] tensorflow: global_step/sec: 3.10549\n",
"INFO:tensorflow:epoch = 115.84375, learning_rate = 9.217787e-06, loss = 0.00013511433, step = 11121 (5.454 sec)\n",
"2021-12-31 02:45:43,869 [INFO] tensorflow: epoch = 115.84375, learning_rate = 9.217787e-06, loss = 0.00013511433, step = 11121 (5.454 sec)\n",
"INFO:tensorflow:global_step/sec: 3.15179\n",
"2021-12-31 02:45:44,811 [INFO] tensorflow: global_step/sec: 3.15179\n",
"2021-12-31 02:45:44,812 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.107\n",
"INFO:tensorflow:global_step/sec: 3.19733\n",
"2021-12-31 02:45:47,626 [INFO] tensorflow: global_step/sec: 3.19733\n",
"2021-12-31 02:45:48,604 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 116/120: loss: 0.00018 learning rate: 0.00001 Time taken: 0:00:30.640092 ETA: 0:02:02.560369\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO:tensorflow:epoch = 116.02083333333333, learning_rate = 8.980656e-06, loss = 0.00011833891, step = 11138 (5.373 sec)\n",
"2021-12-31 02:45:49,241 [INFO] tensorflow: epoch = 116.02083333333333, learning_rate = 8.980656e-06, loss = 0.00011833891, step = 11138 (5.373 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11869\n",
"2021-12-31 02:45:50,512 [INFO] tensorflow: global_step/sec: 3.11869\n",
"2021-12-31 02:45:52,728 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.266\n",
"INFO:tensorflow:global_step/sec: 3.16797\n",
"2021-12-31 02:45:53,353 [INFO] tensorflow: global_step/sec: 3.16797\n",
"INFO:tensorflow:epoch = 116.19791666666666, learning_rate = 8.749617e-06, loss = 0.00015627059, step = 11155 (5.384 sec)\n",
"2021-12-31 02:45:54,625 [INFO] tensorflow: epoch = 116.19791666666666, learning_rate = 8.749617e-06, loss = 0.00015627059, step = 11155 (5.384 sec)\n",
"INFO:tensorflow:global_step/sec: 3.14015\n",
"2021-12-31 02:45:56,219 [INFO] tensorflow: global_step/sec: 3.14015\n",
"INFO:tensorflow:global_step/sec: 3.13151\n",
"2021-12-31 02:45:59,093 [INFO] tensorflow: global_step/sec: 3.13151\n",
"INFO:tensorflow:epoch = 116.375, learning_rate = 8.524531e-06, loss = 0.00014855113, step = 11172 (5.440 sec)\n",
"2021-12-31 02:46:00,065 [INFO] tensorflow: epoch = 116.375, learning_rate = 8.524531e-06, loss = 0.00014855113, step = 11172 (5.440 sec)\n",
"2021-12-31 02:46:00,705 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.073\n",
"INFO:tensorflow:global_step/sec: 3.11942\n",
"2021-12-31 02:46:01,978 [INFO] tensorflow: global_step/sec: 3.11942\n",
"INFO:tensorflow:global_step/sec: 3.10139\n",
"2021-12-31 02:46:04,880 [INFO] tensorflow: global_step/sec: 3.10139\n",
"INFO:tensorflow:epoch = 116.55208333333333, learning_rate = 8.305235e-06, loss = 0.00013425575, step = 11189 (5.472 sec)\n",
"2021-12-31 02:46:05,538 [INFO] tensorflow: epoch = 116.55208333333333, learning_rate = 8.305235e-06, loss = 0.00013425575, step = 11189 (5.472 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09432\n",
"2021-12-31 02:46:07,788 [INFO] tensorflow: global_step/sec: 3.09432\n",
"2021-12-31 02:46:08,754 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.846\n",
"INFO:tensorflow:global_step/sec: 3.0867\n",
"2021-12-31 02:46:10,704 [INFO] tensorflow: global_step/sec: 3.0867\n",
"INFO:tensorflow:epoch = 116.72916666666666, learning_rate = 8.09158e-06, loss = 0.00014706275, step = 11206 (5.483 sec)\n",
"2021-12-31 02:46:11,021 [INFO] tensorflow: epoch = 116.72916666666666, learning_rate = 8.09158e-06, loss = 0.00014706275, step = 11206 (5.483 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10271\n",
"2021-12-31 02:46:13,605 [INFO] tensorflow: global_step/sec: 3.10271\n",
"INFO:tensorflow:epoch = 116.90625, learning_rate = 7.883414e-06, loss = 0.00012494378, step = 11223 (5.482 sec)\n",
"2021-12-31 02:46:16,503 [INFO] tensorflow: epoch = 116.90625, learning_rate = 7.883414e-06, loss = 0.00012494378, step = 11223 (5.482 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10413\n",
"2021-12-31 02:46:16,504 [INFO] tensorflow: global_step/sec: 3.10413\n",
"2021-12-31 02:46:16,811 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.824\n",
"INFO:tensorflow:global_step/sec: 3.1071\n",
"2021-12-31 02:46:19,401 [INFO] tensorflow: global_step/sec: 3.1071\n",
"2021-12-31 02:46:19,402 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 117/120: loss: 0.00014 learning rate: 0.00001 Time taken: 0:00:30.789562 ETA: 0:01:32.368687\n",
"INFO:tensorflow:epoch = 117.08333333333333, learning_rate = 7.68061e-06, loss = 0.00011243073, step = 11240 (5.478 sec)\n",
"2021-12-31 02:46:21,982 [INFO] tensorflow: epoch = 117.08333333333333, learning_rate = 7.68061e-06, loss = 0.00011243073, step = 11240 (5.478 sec)\n",
"INFO:tensorflow:global_step/sec: 3.10682\n",
"2021-12-31 02:46:22,297 [INFO] tensorflow: global_step/sec: 3.10682\n",
"2021-12-31 02:46:24,866 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.829\n",
"INFO:tensorflow:global_step/sec: 3.1111\n",
"2021-12-31 02:46:25,190 [INFO] tensorflow: global_step/sec: 3.1111\n",
"INFO:tensorflow:epoch = 117.26041666666666, learning_rate = 7.483023e-06, loss = 0.00013696708, step = 11257 (5.413 sec)\n",
"2021-12-31 02:46:27,395 [INFO] tensorflow: epoch = 117.26041666666666, learning_rate = 7.483023e-06, loss = 0.00013696708, step = 11257 (5.413 sec)\n",
"INFO:tensorflow:global_step/sec: 3.15515\n",
"2021-12-31 02:46:28,043 [INFO] tensorflow: global_step/sec: 3.15515\n",
"INFO:tensorflow:global_step/sec: 3.16877\n",
"2021-12-31 02:46:30,883 [INFO] tensorflow: global_step/sec: 3.16877\n",
"INFO:tensorflow:epoch = 117.4375, learning_rate = 7.2905136e-06, loss = 0.00012579787, step = 11274 (5.395 sec)\n",
"2021-12-31 02:46:32,790 [INFO] tensorflow: epoch = 117.4375, learning_rate = 7.2905136e-06, loss = 0.00012579787, step = 11274 (5.395 sec)\n",
"2021-12-31 02:46:32,790 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 25.240\n",
"INFO:tensorflow:global_step/sec: 3.13713\n",
"2021-12-31 02:46:33,752 [INFO] tensorflow: global_step/sec: 3.13713\n",
"INFO:tensorflow:global_step/sec: 3.09187\n",
"2021-12-31 02:46:36,663 [INFO] tensorflow: global_step/sec: 3.09187\n",
"INFO:tensorflow:epoch = 117.61458333333333, learning_rate = 7.1029626e-06, loss = 0.00015214004, step = 11291 (5.485 sec)\n",
"2021-12-31 02:46:38,276 [INFO] tensorflow: epoch = 117.61458333333333, learning_rate = 7.1029626e-06, loss = 0.00015214004, step = 11291 (5.485 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11217\n",
"2021-12-31 02:46:39,555 [INFO] tensorflow: global_step/sec: 3.11217\n",
"2021-12-31 02:46:40,822 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.904\n",
"INFO:tensorflow:global_step/sec: 3.10832\n",
"2021-12-31 02:46:42,450 [INFO] tensorflow: global_step/sec: 3.10832\n",
"INFO:tensorflow:epoch = 117.79166666666666, learning_rate = 6.9202365e-06, loss = 0.00012910167, step = 11308 (5.464 sec)\n",
"2021-12-31 02:46:43,740 [INFO] tensorflow: epoch = 117.79166666666666, learning_rate = 6.9202365e-06, loss = 0.00012910167, step = 11308 (5.464 sec)\n",
"INFO:tensorflow:global_step/sec: 3.12206\n",
"2021-12-31 02:46:45,333 [INFO] tensorflow: global_step/sec: 3.12206\n",
"INFO:tensorflow:global_step/sec: 3.10022\n",
"2021-12-31 02:46:48,236 [INFO] tensorflow: global_step/sec: 3.10022\n",
"2021-12-31 02:46:48,877 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.829\n",
"INFO:tensorflow:epoch = 117.96875, learning_rate = 6.742211e-06, loss = 0.00015216904, step = 11325 (5.446 sec)\n",
"2021-12-31 02:46:49,185 [INFO] tensorflow: epoch = 117.96875, learning_rate = 6.742211e-06, loss = 0.00015216904, step = 11325 (5.446 sec)\n",
"2021-12-31 02:46:50,154 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 118/120: loss: 0.00016 learning rate: 0.00001 Time taken: 0:00:30.756523 ETA: 0:01:01.513046\n",
"INFO:tensorflow:global_step/sec: 3.16159\n",
"2021-12-31 02:46:51,082 [INFO] tensorflow: global_step/sec: 3.16159\n",
"INFO:tensorflow:global_step/sec: 3.10425\n",
"2021-12-31 02:46:53,982 [INFO] tensorflow: global_step/sec: 3.10425\n",
"INFO:tensorflow:epoch = 118.14583333333333, learning_rate = 6.568759e-06, loss = 0.00013407406, step = 11342 (5.426 sec)\n",
"2021-12-31 02:46:54,611 [INFO] tensorflow: epoch = 118.14583333333333, learning_rate = 6.568759e-06, loss = 0.00013407406, step = 11342 (5.426 sec)\n",
"INFO:tensorflow:global_step/sec: 3.0901\n",
"2021-12-31 02:46:56,894 [INFO] tensorflow: global_step/sec: 3.0901\n",
"2021-12-31 02:46:56,895 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.945\n",
"INFO:tensorflow:global_step/sec: 3.18174\n",
"2021-12-31 02:46:59,723 [INFO] tensorflow: global_step/sec: 3.18174\n",
"INFO:tensorflow:epoch = 118.32291666666666, learning_rate = 6.399776e-06, loss = 0.00012868403, step = 11359 (5.437 sec)\n",
"2021-12-31 02:47:00,049 [INFO] tensorflow: epoch = 118.32291666666666, learning_rate = 6.399776e-06, loss = 0.00012868403, step = 11359 (5.437 sec)\n",
"INFO:tensorflow:global_step/sec: 3.07008\n",
"2021-12-31 02:47:02,654 [INFO] tensorflow: global_step/sec: 3.07008\n",
"2021-12-31 02:47:04,898 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.991\n",
"INFO:tensorflow:epoch = 118.5, learning_rate = 6.2351396e-06, loss = 0.0001434735, step = 11376 (5.468 sec)\n",
"2021-12-31 02:47:05,517 [INFO] tensorflow: epoch = 118.5, learning_rate = 6.2351396e-06, loss = 0.0001434735, step = 11376 (5.468 sec)\n",
"INFO:tensorflow:global_step/sec: 3.14385\n",
"2021-12-31 02:47:05,517 [INFO] tensorflow: global_step/sec: 3.14385\n",
"INFO:tensorflow:global_step/sec: 3.07402\n",
"2021-12-31 02:47:08,445 [INFO] tensorflow: global_step/sec: 3.07402\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO:tensorflow:epoch = 118.67708333333333, learning_rate = 6.0747384e-06, loss = 0.00011625141, step = 11393 (5.486 sec)\n",
"2021-12-31 02:47:11,002 [INFO] tensorflow: epoch = 118.67708333333333, learning_rate = 6.0747384e-06, loss = 0.00011625141, step = 11393 (5.486 sec)\n",
"INFO:tensorflow:global_step/sec: 3.14265\n",
"2021-12-31 02:47:11,309 [INFO] tensorflow: global_step/sec: 3.14265\n",
"2021-12-31 02:47:12,900 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.995\n",
"INFO:tensorflow:global_step/sec: 3.0798\n",
"2021-12-31 02:47:14,231 [INFO] tensorflow: global_step/sec: 3.0798\n",
"INFO:tensorflow:epoch = 118.85416666666666, learning_rate = 5.9184586e-06, loss = 0.00016136347, step = 11410 (5.511 sec)\n",
"2021-12-31 02:47:16,514 [INFO] tensorflow: epoch = 118.85416666666666, learning_rate = 5.9184586e-06, loss = 0.00016136347, step = 11410 (5.511 sec)\n",
"INFO:tensorflow:global_step/sec: 3.09302\n",
"2021-12-31 02:47:17,141 [INFO] tensorflow: global_step/sec: 3.09302\n",
"INFO:tensorflow:global_step/sec: 3.10249\n",
"2021-12-31 02:47:20,042 [INFO] tensorflow: global_step/sec: 3.10249\n",
"2021-12-31 02:47:21,007 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 119/120: loss: 0.00013 learning rate: 0.00001 Time taken: 0:00:30.843189 ETA: 0:00:30.843189\n",
"2021-12-31 02:47:21,008 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.669\n",
"INFO:tensorflow:epoch = 119.03125, learning_rate = 5.7662037e-06, loss = 0.00010324842, step = 11427 (5.470 sec)\n",
"2021-12-31 02:47:21,984 [INFO] tensorflow: epoch = 119.03125, learning_rate = 5.7662037e-06, loss = 0.00010324842, step = 11427 (5.470 sec)\n",
"INFO:tensorflow:global_step/sec: 3.1137\n",
"2021-12-31 02:47:22,932 [INFO] tensorflow: global_step/sec: 3.1137\n",
"INFO:tensorflow:global_step/sec: 3.1008\n",
"2021-12-31 02:47:25,835 [INFO] tensorflow: global_step/sec: 3.1008\n",
"INFO:tensorflow:epoch = 119.20833333333333, learning_rate = 5.6178665e-06, loss = 0.0001413004, step = 11444 (5.451 sec)\n",
"2021-12-31 02:47:27,434 [INFO] tensorflow: epoch = 119.20833333333333, learning_rate = 5.6178665e-06, loss = 0.0001413004, step = 11444 (5.451 sec)\n",
"INFO:tensorflow:global_step/sec: 3.11028\n",
"2021-12-31 02:47:28,728 [INFO] tensorflow: global_step/sec: 3.11028\n",
"2021-12-31 02:47:29,045 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.885\n",
"INFO:tensorflow:global_step/sec: 3.13596\n",
"2021-12-31 02:47:31,598 [INFO] tensorflow: global_step/sec: 3.13596\n",
"INFO:tensorflow:epoch = 119.38541666666666, learning_rate = 5.47334e-06, loss = 0.00013335329, step = 11461 (5.422 sec)\n",
"2021-12-31 02:47:32,857 [INFO] tensorflow: epoch = 119.38541666666666, learning_rate = 5.47334e-06, loss = 0.00013335329, step = 11461 (5.422 sec)\n",
"INFO:tensorflow:global_step/sec: 3.187\n",
"2021-12-31 02:47:34,422 [INFO] tensorflow: global_step/sec: 3.187\n",
"2021-12-31 02:47:37,046 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.999\n",
"INFO:tensorflow:global_step/sec: 3.04762\n",
"2021-12-31 02:47:37,375 [INFO] tensorflow: global_step/sec: 3.04762\n",
"INFO:tensorflow:epoch = 119.5625, learning_rate = 5.332536e-06, loss = 0.00016154782, step = 11478 (5.492 sec)\n",
"2021-12-31 02:47:38,349 [INFO] tensorflow: epoch = 119.5625, learning_rate = 5.332536e-06, loss = 0.00016154782, step = 11478 (5.492 sec)\n",
"INFO:tensorflow:global_step/sec: 3.05874\n",
"2021-12-31 02:47:40,318 [INFO] tensorflow: global_step/sec: 3.05874\n",
"INFO:tensorflow:global_step/sec: 3.0813\n",
"2021-12-31 02:47:43,239 [INFO] tensorflow: global_step/sec: 3.0813\n",
"INFO:tensorflow:epoch = 119.73958333333333, learning_rate = 5.1953502e-06, loss = 0.00015278415, step = 11495 (5.507 sec)\n",
"2021-12-31 02:47:43,856 [INFO] tensorflow: epoch = 119.73958333333333, learning_rate = 5.1953502e-06, loss = 0.00015278415, step = 11495 (5.507 sec)\n",
"2021-12-31 02:47:45,135 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.725\n",
"INFO:tensorflow:global_step/sec: 3.14885\n",
"2021-12-31 02:47:46,097 [INFO] tensorflow: global_step/sec: 3.14885\n",
"INFO:tensorflow:global_step/sec: 3.08887\n",
"2021-12-31 02:47:49,010 [INFO] tensorflow: global_step/sec: 3.08887\n",
"INFO:tensorflow:epoch = 119.91666666666666, learning_rate = 5.061693e-06, loss = 0.00012841362, step = 11512 (5.489 sec)\n",
"2021-12-31 02:47:49,344 [INFO] tensorflow: epoch = 119.91666666666666, learning_rate = 5.061693e-06, loss = 0.00012841362, step = 11512 (5.489 sec)\n",
"INFO:tensorflow:Saving checkpoints for step-11520.\n",
"2021-12-31 02:47:51,604 [INFO] tensorflow: Saving checkpoints for step-11520.\n",
"WARNING:tensorflow:Ignoring: /tmp/tmp_8jkohhq; No such file or directory\n",
"2021-12-31 02:47:51,749 [WARNING] tensorflow: Ignoring: /tmp/tmp_8jkohhq; No such file or directory\n",
"2021-12-31 02:47:55,225 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 23, 0.00s/step\n",
"2021-12-31 02:47:56,929 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 23, 0.17s/step\n",
"2021-12-31 02:47:58,468 [INFO] iva.detectnet_v2.evaluation.evaluation: step 20 / 23, 0.15s/step\n",
"Matching predictions to ground truth, class 1/1.: 100%|█| 942/942 [00:00<00:00, 15479.20it/s]\n",
"Epoch 120/120\n",
"=========================\n",
"\n",
"Validation cost: 0.000163\n",
"Mean average_precision (in %): 92.9279\n",
"\n",
"class name average precision (in %)\n",
"------------ --------------------------\n",
"car 92.9279\n",
"\n",
"Median Inference Time: 0.016380\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:95: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead.\n",
"\n",
"2021-12-31 02:47:59,527 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:95: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead.\n",
"\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:98: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead.\n",
"\n",
"2021-12-31 02:47:59,528 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:98: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead.\n",
"\n",
"2021-12-31 02:47:59,529 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 24.725\n",
"Time taken to run __main__:main: 1:06:52.943099.\n",
"2021-12-31 10:48:01,293 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.\n"
]
}
],
"source": [
"# Retraining using the pruned model as pretrained weights \n",
"!tao detectnet_v2 train -e $SPECS_DIR/detectnet_v2_retrain_resnet18_kitti.txt \\\n",
" -r $USER_EXPERIMENT_DIR/experiment_dir_retrain \\\n",
" -k $KEY \\\n",
" -n resnet18_detector_pruned \\\n",
" --gpus $NUM_GPUS"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"total 45472\r\n",
"-rw-r--r-- 1 guest guest 46562000 Dec 31 10:47 resnet18_detector_pruned.tlt\r\n"
]
}
],
"source": [
"# Listing the newly retrained model.\n",
"!ls -rlt $LOCAL_EXPERIMENT_DIR/experiment_dir_retrain/weights"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 8. Evaluate the retrained model "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This section evaluates the pruned and retrained model, using the `evaluate` command."
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"2022-01-20 10:24:05,449 [INFO] root: Registry: ['nvcr.io']\n",
"Matplotlib created a temporary config/cache directory at /tmp/matplotlib-9f7b8ln9 because the default path (/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.\n",
"Using TensorFlow backend.\n",
"WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.\n",
"Using TensorFlow backend.\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:43: The name tf.train.SessionRunHook is deprecated. Please use tf.estimator.SessionRunHook instead.\n",
"\n",
"2022-01-20 02:24:11,405 [INFO] iva.detectnet_v2.spec_handler.spec_loader: Merging specification from /workspace/tao-experiments/specs/detectnet_v2_retrain_resnet18_kitti.txt\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:153: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.\n",
"\n",
"2022-01-20 02:24:11,409 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:153: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.\n",
"\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.\n",
"\n",
"2022-01-20 02:24:11,768 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.\n",
"\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.\n",
"\n",
"2022-01-20 02:24:11,784 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.\n",
"\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1834: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead.\n",
"\n",
"2022-01-20 02:24:11,801 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1834: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead.\n",
"\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.\n",
"\n",
"2022-01-20 02:24:12,743 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.\n",
"\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:181: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.\n",
"\n",
"2022-01-20 02:24:12,744 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:181: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.\n",
"\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:186: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.\n",
"\n",
"2022-01-20 02:24:12,744 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:186: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.\n",
"\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:190: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.\n",
"\n",
"2022-01-20 02:24:13,150 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:190: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.\n",
"\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead.\n",
"\n",
"2022-01-20 02:24:13,150 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead.\n",
"\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead.\n",
"\n",
"2022-01-20 02:24:13,370 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead.\n",
"\n",
"/usr/local/lib/python3.6/dist-packages/keras/engine/saving.py:292: UserWarning: No training configuration found in save file: the model was *not* compiled. Compile it manually.\n",
" warnings.warn('No training configuration found in save file: '\n",
"2022-01-20 02:24:13,642 [INFO] iva.detectnet_v2.objectives.bbox_objective: Default L1 loss function will be used.\n",
"2022-01-20 02:24:13,752 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Serial augmentation enabled = False\n",
"2022-01-20 02:24:13,752 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Pseudo sharding enabled = False\n",
"2022-01-20 02:24:13,753 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Max Image Dimensions (all sources): (0, 0)\n",
"2022-01-20 02:24:13,753 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: number of cpus: 16, io threads: 32, compute threads: 16, buffered batches: 4\n",
"2022-01-20 02:24:13,753 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: total dataset size 190, number of sources: 1, batch size per gpu: 8, steps: 24\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.\n",
"\n",
"2022-01-20 02:24:13,780 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.\n",
"\n",
"WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code\n",
"2022-01-20 02:24:13,820 [WARNING] tensorflow: Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code\n",
"2022-01-20 02:24:13,835 [INFO] iva.detectnet_v2.dataloader.default_dataloader: Bounding box coordinates were detected in the input specification! Bboxes will be automatically converted to polygon coordinates.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"2022-01-20 02:24:14,040 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: shuffle: False - shard 0 of 1\n",
"2022-01-20 02:24:14,045 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: sampling 1 datasets with weights:\n",
"2022-01-20 02:24:14,046 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: source: 0 weight: 1.000000\n",
"WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code\n",
"2022-01-20 02:24:14,057 [WARNING] tensorflow: Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/core/build_wheel.runfiles/ai_infra/moduluspy/modulus/blocks/data_loaders/multi_source_loader/types/images2d_reference.py:427: The name tf.image.resize_images is deprecated. Please use tf.image.resize instead.\n",
"\n",
"2022-01-20 02:24:14,077 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/core/build_wheel.runfiles/ai_infra/moduluspy/modulus/blocks/data_loaders/multi_source_loader/types/images2d_reference.py:427: The name tf.image.resize_images is deprecated. Please use tf.image.resize instead.\n",
"\n",
"2022-01-20 02:24:14,263 [INFO] iva.detectnet_v2.evaluation.build_evaluator: Found 190 samples in validation set\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:107: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.\n",
"\n",
"2022-01-20 02:24:14,263 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:107: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:110: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.\n",
"\n",
"2022-01-20 02:24:14,264 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:110: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:113: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead.\n",
"\n",
"2022-01-20 02:24:14,265 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:113: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/rasterizers/bbox_rasterizer.py:347: The name tf.bincount is deprecated. Please use tf.math.bincount instead.\n",
"\n",
"2022-01-20 02:24:14,364 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/rasterizers/bbox_rasterizer.py:347: The name tf.bincount is deprecated. Please use tf.math.bincount instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_functions.py:17: The name tf.log is deprecated. Please use tf.math.log instead.\n",
"\n",
"2022-01-20 02:24:14,759 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_functions.py:17: The name tf.log is deprecated. Please use tf.math.log instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:235: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead.\n",
"\n",
"2022-01-20 02:24:14,767 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:235: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead.\n",
"\n",
"__________________________________________________________________________________________________\n",
"Layer (type) Output Shape Param # Connected to \n",
"==================================================================================================\n",
"input_1 (InputLayer) (None, 3, 544, 960) 0 \n",
"__________________________________________________________________________________________________\n",
"input_1_qdq (QDQ) (None, 3, 544, 960) 1 input_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"conv1 (QuantizedConv2D) (None, 64, 272, 480) 9472 input_1_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"bn_conv1 (BatchNormalization) (None, 64, 272, 480) 256 conv1[0][0] \n",
"__________________________________________________________________________________________________\n",
"activation_1 (ReLU) (None, 64, 272, 480) 0 bn_conv1[0][0] \n",
"__________________________________________________________________________________________________\n",
"activation_1_qdq (QDQ) (None, 64, 272, 480) 1 activation_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1a_conv_1 (QuantizedConv2 (None, 64, 136, 240) 36928 activation_1_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1a_bn_1 (BatchNormalizati (None, 64, 136, 240) 256 block_1a_conv_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1a_relu_1 (ReLU) (None, 64, 136, 240) 0 block_1a_bn_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1a_relu_1_qdq (QDQ) (None, 64, 136, 240) 1 block_1a_relu_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1a_conv_2 (QuantizedConv2 (None, 64, 136, 240) 36928 block_1a_relu_1_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1a_conv_shortcut (Quantiz (None, 64, 136, 240) 4160 activation_1_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1a_bn_2 (BatchNormalizati (None, 64, 136, 240) 256 block_1a_conv_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1a_bn_shortcut (BatchNorm (None, 64, 136, 240) 256 block_1a_conv_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1a_bn_2_qdq (QDQ) (None, 64, 136, 240) 1 block_1a_bn_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1a_bn_shortcut_qdq (QDQ) (None, 64, 136, 240) 1 block_1a_bn_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_1 (Add) (None, 64, 136, 240) 0 block_1a_bn_2_qdq[0][0] \n",
" block_1a_bn_shortcut_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_1_qdq (QDQ) (None, 64, 136, 240) 1 add_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1a_relu (ReLU) (None, 64, 136, 240) 0 add_1_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1a_relu_qdq (QDQ) (None, 64, 136, 240) 1 block_1a_relu[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1b_conv_1 (QuantizedConv2 (None, 64, 136, 240) 36928 block_1a_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1b_bn_1 (BatchNormalizati (None, 64, 136, 240) 256 block_1b_conv_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1b_relu_1 (ReLU) (None, 64, 136, 240) 0 block_1b_bn_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1b_relu_1_qdq (QDQ) (None, 64, 136, 240) 1 block_1b_relu_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1b_conv_2 (QuantizedConv2 (None, 64, 136, 240) 36928 block_1b_relu_1_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1b_conv_shortcut (Quantiz (None, 64, 136, 240) 4160 block_1a_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1b_bn_2 (BatchNormalizati (None, 64, 136, 240) 256 block_1b_conv_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1b_bn_shortcut (BatchNorm (None, 64, 136, 240) 256 block_1b_conv_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1b_bn_2_qdq (QDQ) (None, 64, 136, 240) 1 block_1b_bn_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1b_bn_shortcut_qdq (QDQ) (None, 64, 136, 240) 1 block_1b_bn_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_2 (Add) (None, 64, 136, 240) 0 block_1b_bn_2_qdq[0][0] \n",
" block_1b_bn_shortcut_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_2_qdq (QDQ) (None, 64, 136, 240) 1 add_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1b_relu (ReLU) (None, 64, 136, 240) 0 add_2_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1b_relu_qdq (QDQ) (None, 64, 136, 240) 1 block_1b_relu[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2a_conv_1 (QuantizedConv2 (None, 128, 68, 120) 73856 block_1b_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2a_bn_1 (BatchNormalizati (None, 128, 68, 120) 512 block_2a_conv_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2a_relu_1 (ReLU) (None, 128, 68, 120) 0 block_2a_bn_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2a_relu_1_qdq (QDQ) (None, 128, 68, 120) 1 block_2a_relu_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2a_conv_2 (QuantizedConv2 (None, 128, 68, 120) 147584 block_2a_relu_1_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2a_conv_shortcut (Quantiz (None, 128, 68, 120) 8320 block_1b_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2a_bn_2 (BatchNormalizati (None, 128, 68, 120) 512 block_2a_conv_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2a_bn_shortcut (BatchNorm (None, 128, 68, 120) 512 block_2a_conv_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2a_bn_2_qdq (QDQ) (None, 128, 68, 120) 1 block_2a_bn_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2a_bn_shortcut_qdq (QDQ) (None, 128, 68, 120) 1 block_2a_bn_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_3 (Add) (None, 128, 68, 120) 0 block_2a_bn_2_qdq[0][0] \n",
" block_2a_bn_shortcut_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_3_qdq (QDQ) (None, 128, 68, 120) 1 add_3[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2a_relu (ReLU) (None, 128, 68, 120) 0 add_3_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2a_relu_qdq (QDQ) (None, 128, 68, 120) 1 block_2a_relu[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2b_conv_1 (QuantizedConv2 (None, 128, 68, 120) 147584 block_2a_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2b_bn_1 (BatchNormalizati (None, 128, 68, 120) 512 block_2b_conv_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2b_relu_1 (ReLU) (None, 128, 68, 120) 0 block_2b_bn_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2b_relu_1_qdq (QDQ) (None, 128, 68, 120) 1 block_2b_relu_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2b_conv_2 (QuantizedConv2 (None, 128, 68, 120) 147584 block_2b_relu_1_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2b_conv_shortcut (Quantiz (None, 128, 68, 120) 16512 block_2a_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2b_bn_2 (BatchNormalizati (None, 128, 68, 120) 512 block_2b_conv_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2b_bn_shortcut (BatchNorm (None, 128, 68, 120) 512 block_2b_conv_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2b_bn_2_qdq (QDQ) (None, 128, 68, 120) 1 block_2b_bn_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2b_bn_shortcut_qdq (QDQ) (None, 128, 68, 120) 1 block_2b_bn_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_4 (Add) (None, 128, 68, 120) 0 block_2b_bn_2_qdq[0][0] \n",
" block_2b_bn_shortcut_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_4_qdq (QDQ) (None, 128, 68, 120) 1 add_4[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2b_relu (ReLU) (None, 128, 68, 120) 0 add_4_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2b_relu_qdq (QDQ) (None, 128, 68, 120) 1 block_2b_relu[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3a_conv_1 (QuantizedConv2 (None, 256, 34, 60) 295168 block_2b_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3a_bn_1 (BatchNormalizati (None, 256, 34, 60) 1024 block_3a_conv_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3a_relu_1 (ReLU) (None, 256, 34, 60) 0 block_3a_bn_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3a_relu_1_qdq (QDQ) (None, 256, 34, 60) 1 block_3a_relu_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3a_conv_2 (QuantizedConv2 (None, 256, 34, 60) 590080 block_3a_relu_1_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3a_conv_shortcut (Quantiz (None, 256, 34, 60) 33024 block_2b_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3a_bn_2 (BatchNormalizati (None, 256, 34, 60) 1024 block_3a_conv_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3a_bn_shortcut (BatchNorm (None, 256, 34, 60) 1024 block_3a_conv_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3a_bn_2_qdq (QDQ) (None, 256, 34, 60) 1 block_3a_bn_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3a_bn_shortcut_qdq (QDQ) (None, 256, 34, 60) 1 block_3a_bn_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_5 (Add) (None, 256, 34, 60) 0 block_3a_bn_2_qdq[0][0] \n",
" block_3a_bn_shortcut_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_5_qdq (QDQ) (None, 256, 34, 60) 1 add_5[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3a_relu (ReLU) (None, 256, 34, 60) 0 add_5_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3a_relu_qdq (QDQ) (None, 256, 34, 60) 1 block_3a_relu[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3b_conv_1 (QuantizedConv2 (None, 256, 34, 60) 590080 block_3a_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3b_bn_1 (BatchNormalizati (None, 256, 34, 60) 1024 block_3b_conv_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3b_relu_1 (ReLU) (None, 256, 34, 60) 0 block_3b_bn_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3b_relu_1_qdq (QDQ) (None, 256, 34, 60) 1 block_3b_relu_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3b_conv_2 (QuantizedConv2 (None, 256, 34, 60) 590080 block_3b_relu_1_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3b_conv_shortcut (Quantiz (None, 256, 34, 60) 65792 block_3a_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3b_bn_2 (BatchNormalizati (None, 256, 34, 60) 1024 block_3b_conv_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3b_bn_shortcut (BatchNorm (None, 256, 34, 60) 1024 block_3b_conv_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3b_bn_2_qdq (QDQ) (None, 256, 34, 60) 1 block_3b_bn_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3b_bn_shortcut_qdq (QDQ) (None, 256, 34, 60) 1 block_3b_bn_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_6 (Add) (None, 256, 34, 60) 0 block_3b_bn_2_qdq[0][0] \n",
" block_3b_bn_shortcut_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_6_qdq (QDQ) (None, 256, 34, 60) 1 add_6[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3b_relu (ReLU) (None, 256, 34, 60) 0 add_6_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3b_relu_qdq (QDQ) (None, 256, 34, 60) 1 block_3b_relu[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4a_conv_1 (QuantizedConv2 (None, 512, 34, 60) 1180160 block_3b_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4a_bn_1 (BatchNormalizati (None, 512, 34, 60) 2048 block_4a_conv_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4a_relu_1 (ReLU) (None, 512, 34, 60) 0 block_4a_bn_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4a_relu_1_qdq (QDQ) (None, 512, 34, 60) 1 block_4a_relu_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4a_conv_2 (QuantizedConv2 (None, 512, 34, 60) 2359808 block_4a_relu_1_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4a_conv_shortcut (Quantiz (None, 512, 34, 60) 131584 block_3b_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4a_bn_2 (BatchNormalizati (None, 512, 34, 60) 2048 block_4a_conv_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4a_bn_shortcut (BatchNorm (None, 512, 34, 60) 2048 block_4a_conv_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4a_bn_2_qdq (QDQ) (None, 512, 34, 60) 1 block_4a_bn_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4a_bn_shortcut_qdq (QDQ) (None, 512, 34, 60) 1 block_4a_bn_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_7 (Add) (None, 512, 34, 60) 0 block_4a_bn_2_qdq[0][0] \n",
" block_4a_bn_shortcut_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_7_qdq (QDQ) (None, 512, 34, 60) 1 add_7[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4a_relu (ReLU) (None, 512, 34, 60) 0 add_7_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4a_relu_qdq (QDQ) (None, 512, 34, 60) 1 block_4a_relu[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4b_conv_1 (QuantizedConv2 (None, 512, 34, 60) 2359808 block_4a_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4b_bn_1 (BatchNormalizati (None, 512, 34, 60) 2048 block_4b_conv_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4b_relu_1 (ReLU) (None, 512, 34, 60) 0 block_4b_bn_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4b_relu_1_qdq (QDQ) (None, 512, 34, 60) 1 block_4b_relu_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4b_conv_2 (QuantizedConv2 (None, 512, 34, 60) 2359808 block_4b_relu_1_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4b_conv_shortcut (Quantiz (None, 512, 34, 60) 262656 block_4a_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4b_bn_2 (BatchNormalizati (None, 512, 34, 60) 2048 block_4b_conv_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4b_bn_shortcut (BatchNorm (None, 512, 34, 60) 2048 block_4b_conv_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4b_bn_2_qdq (QDQ) (None, 512, 34, 60) 1 block_4b_bn_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4b_bn_shortcut_qdq (QDQ) (None, 512, 34, 60) 1 block_4b_bn_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_8 (Add) (None, 512, 34, 60) 0 block_4b_bn_2_qdq[0][0] \n",
" block_4b_bn_shortcut_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_8_qdq (QDQ) (None, 512, 34, 60) 1 add_8[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4b_relu (ReLU) (None, 512, 34, 60) 0 add_8_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4b_relu_qdq (QDQ) (None, 512, 34, 60) 1 block_4b_relu[0][0] \n",
"__________________________________________________________________________________________________\n",
"output_bbox (Conv2D) (None, 4, 34, 60) 2052 block_4b_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"output_cov (Conv2D) (None, 1, 34, 60) 513 block_4b_relu_qdq[0][0] \n",
"==================================================================================================\n",
"Total params: 11,550,895\n",
"Trainable params: 11,539,205\n",
"Non-trainable params: 11,690\n",
"__________________________________________________________________________________________________\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/utilities.py:139: The name tf.train.Scaffold is deprecated. Please use tf.compat.v1.train.Scaffold instead.\n",
"\n",
"2022-01-20 02:24:14,778 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/utilities.py:139: The name tf.train.Scaffold is deprecated. Please use tf.compat.v1.train.Scaffold instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:14: The name tf.local_variables_initializer is deprecated. Please use tf.compat.v1.local_variables_initializer instead.\n",
"\n",
"2022-01-20 02:24:14,778 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:14: The name tf.local_variables_initializer is deprecated. Please use tf.compat.v1.local_variables_initializer instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:15: The name tf.tables_initializer is deprecated. Please use tf.compat.v1.tables_initializer instead.\n",
"\n",
"2022-01-20 02:24:14,778 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:15: The name tf.tables_initializer is deprecated. Please use tf.compat.v1.tables_initializer instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:16: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead.\n",
"\n",
"2022-01-20 02:24:14,779 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:16: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/utilities.py:140: The name tf.train.SingularMonitoredSession is deprecated. Please use tf.compat.v1.train.SingularMonitoredSession instead.\n",
"\n",
"2022-01-20 02:24:14,779 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/utilities.py:140: The name tf.train.SingularMonitoredSession is deprecated. Please use tf.compat.v1.train.SingularMonitoredSession instead.\n",
"\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO:tensorflow:Graph was finalized.\n",
"2022-01-20 02:24:15,215 [INFO] tensorflow: Graph was finalized.\n",
"INFO:tensorflow:Running local_init_op.\n",
"2022-01-20 02:24:15,846 [INFO] tensorflow: Running local_init_op.\n",
"INFO:tensorflow:Done running local_init_op.\n",
"2022-01-20 02:24:16,087 [INFO] tensorflow: Done running local_init_op.\n",
"2022-01-20 02:24:16,673 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 24, 0.00s/step\n",
"2022-01-20 02:24:22,679 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 24, 0.60s/step\n",
"2022-01-20 02:24:24,087 [INFO] iva.detectnet_v2.evaluation.evaluation: step 20 / 24, 0.14s/step\n",
"Matching predictions to ground truth, class 1/1.: 100%|█| 990/990 [00:00<00:00, 15854.94it/s]\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:95: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead.\n",
"\n",
"2022-01-20 02:24:24,832 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:95: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead.\n",
"\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:98: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead.\n",
"\n",
"2022-01-20 02:24:24,832 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:98: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead.\n",
"\n",
"\n",
"Validation cost: 0.001136\n",
"Mean average_precision (in %): 93.0563\n",
"\n",
"class name average precision (in %)\n",
"------------ --------------------------\n",
"car 93.0563\n",
"\n",
"Median Inference Time: 0.014622\n",
"2022-01-20 02:24:24,875 [INFO] __main__: Evaluation complete.\n",
"Time taken to run __main__:main: 0:00:13.471794.\n",
"2022-01-20 10:24:26,182 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.\n"
]
}
],
"source": [
"!tao detectnet_v2 evaluate -e $SPECS_DIR/detectnet_v2_retrain_resnet18_kitti.txt \\\n",
" -m $USER_EXPERIMENT_DIR/experiment_dir_retrain/weights/resnet18_detector_pruned.tlt \\\n",
" -k $KEY"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 9. Visualize inferences \n",
"In this section, we run the `inference` tool to generate inferences on the trained models. To render bboxes from more classes, please edit the spec file `detectnet_v2_inference_kitti_tlt.txt` to include all the classes you would like to visualize and edit the rest of the file accordingly."
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"2022-01-21 18:04:52,062 [INFO] root: Registry: ['nvcr.io']\n",
"Matplotlib created a temporary config/cache directory at /tmp/matplotlib-o5bd143n because the default path (/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.\n",
"Using TensorFlow backend.\n",
"Using TensorFlow backend.\n",
"WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.\n",
"2022-01-21 10:04:57,722 [INFO] iva.detectnet_v2.spec_handler.spec_loader: Merging specification from /workspace/tao-experiments/specs/detectnet_v2_inference_kitti_tlt.txt\n",
"2022-01-21 10:04:57,723 [INFO] __main__: Overlain images will be saved in the output path.\n",
"2022-01-21 10:04:57,723 [INFO] iva.detectnet_v2.inferencer.build_inferencer: Constructing inferencer\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/inferencer/tlt_inferencer.py:84: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.\n",
"\n",
"2022-01-21 10:04:57,723 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/inferencer/tlt_inferencer.py:84: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/inferencer/tlt_inferencer.py:87: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.\n",
"\n",
"2022-01-21 10:04:57,723 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/inferencer/tlt_inferencer.py:87: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.\n",
"\n",
"2022-01-21 10:04:58,004 [INFO] iva.detectnet_v2.inferencer.tlt_inferencer: Loading model from /workspace/tao-experiments/experiment/experiment_dir_retrain/weights/resnet18_detector_pruned.tlt:\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.\n",
"\n",
"2022-01-21 10:04:58,347 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.\n",
"\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:131: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.\n",
"\n",
"2022-01-21 10:04:58,360 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:131: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.\n",
"\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:133: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead.\n",
"\n",
"2022-01-21 10:04:58,360 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:133: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead.\n",
"\n",
"2022-01-21 10:04:58,361 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.\n",
"\n",
"2022-01-21 10:04:58,362 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.\n",
"\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1834: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead.\n",
"\n",
"2022-01-21 10:04:58,379 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1834: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead.\n",
"\n",
"2022-01-21 10:04:58,395 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:04:58,424 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:04:58,479 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:04:58,481 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:04:58,483 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:04:58,486 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:04:58,515 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:04:58,570 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:04:58,572 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:04:58,575 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:04:58,577 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:04:58,607 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:04:58,661 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:04:58,664 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:04:58,666 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:04:58,669 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:04:58,698 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:04:58,753 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:04:58,755 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:04:58,758 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:04:58,760 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:04:58,789 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:04:58,845 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:04:58,847 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:04:58,849 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:04:58,852 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:04:58,881 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"2022-01-21 10:04:58,937 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:04:58,939 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:04:58,942 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:04:58,945 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:04:58,975 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:04:59,030 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:04:59,032 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:04:59,035 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:04:59,037 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:04:59,067 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:04:59,122 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:04:59,124 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:04:59,126 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:04:59,129 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.\n",
"\n",
"2022-01-21 10:04:59,409 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.\n",
"\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:190: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.\n",
"\n",
"2022-01-21 10:04:59,409 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:190: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.\n",
"\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead.\n",
"\n",
"2022-01-21 10:04:59,409 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead.\n",
"\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead.\n",
"\n",
"2022-01-21 10:04:59,573 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead.\n",
"\n",
"/usr/local/lib/python3.6/dist-packages/keras/engine/saving.py:292: UserWarning: No training configuration found in save file: the model was *not* compiled. Compile it manually.\n",
" warnings.warn('No training configuration found in save file: '\n",
"2022-01-21 10:04:59,851 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:04:59,866 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:04:59,881 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:04:59,908 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:04:59,909 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:04:59,911 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:04:59,912 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:04:59,927 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:04:59,954 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:04:59,955 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:04:59,956 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:04:59,957 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:04:59,971 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:04:59,998 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:04:59,999 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:05:00,000 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:05:00,001 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:05:00,016 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:05:00,043 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:05:00,044 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:05:00,045 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:05:00,047 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:05:00,061 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:05:00,088 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:05:00,089 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:05:00,090 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:05:00,091 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:05:00,106 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:05:00,133 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:05:00,134 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:05:00,135 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:05:00,137 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:05:00,151 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"2022-01-21 10:05:00,178 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:05:00,179 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:05:00,180 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:05:00,181 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:05:00,196 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:05:00,223 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:05:00,224 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:05:00,225 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"2022-01-21 10:05:00,226 [WARNING] modulus.models.templates.qdq_layer: QDQ: Keras learning_phase was not set. Assuming evaluation phase.\n",
"_________________________________________________________________\n",
"Layer (type) Output Shape Param # \n",
"=================================================================\n",
"input_1 (InputLayer) (None, 3, 544, 960) 0 \n",
"_________________________________________________________________\n",
"model_1 (Model) [(None, 1, 34, 60), (None 11550895 \n",
"=================================================================\n",
"Total params: 11,550,895\n",
"Trainable params: 11,539,205\n",
"Non-trainable params: 11,690\n",
"_________________________________________________________________\n",
"2022-01-21 10:05:00,234 [INFO] __main__: Initialized model\n",
"2022-01-21 10:05:00,235 [INFO] __main__: Commencing inference\n",
"100%|███████████████████████████████████████████| 11/11 [00:23<00:00, 2.13s/it]\n",
"2022-01-21 10:05:23,635 [INFO] __main__: Inference complete\n",
"2022-01-21 18:05:24,755 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.\n"
]
}
],
"source": [
"# Running inference for detection on n images\n",
"!tao detectnet_v2 inference -e $SPECS_DIR/detectnet_v2_inference_kitti_tlt.txt \\\n",
" -o $USER_EXPERIMENT_DIR/tlt_infer_testing \\\n",
" -i $DATA_DOWNLOAD_DIR/training/image_t1 \\\n",
" -k $KEY"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The `inference` tool produces two outputs. \n",
"1. Overlain images in `$USER_EXPERIMENT_DIR/tlt_infer_testing/images_annotated`\n",
"2. Frame by frame bbox labels in kitti format located in `$USER_EXPERIMENT_DIR/tlt_infer_testing/labels`\n",
"\n",
"*Note: To run inferences for a single image, simply replace the path to the -i flag in `inference` command with the path to the image.*"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Package Version\r\n",
"------------------- ---------\r\n",
"argon2-cffi 21.1.0\r\n",
"async-generator 1.10\r\n",
"attrs 21.2.0\r\n",
"backcall 0.2.0\r\n",
"bleach 4.1.0\r\n",
"cached-property 1.5.2\r\n",
"certifi 2020.6.20\r\n",
"cffi 1.15.0\r\n",
"chardet 3.0.4\r\n",
"cycler 0.11.0\r\n",
"Cython 0.29.24\r\n",
"decorator 5.1.0\r\n",
"defusedxml 0.7.1\r\n",
"docker 4.3.1\r\n",
"docker-pycreds 0.4.0\r\n",
"entrypoints 0.3\r\n",
"h5py 3.1.0\r\n",
"idna 2.10\r\n",
"importlib-metadata 4.8.2\r\n",
"ipykernel 5.5.6\r\n",
"ipython 7.16.1\r\n",
"ipython-genutils 0.2.0\r\n",
"ipywidgets 7.6.5\r\n",
"jedi 0.18.1\r\n",
"Jinja2 3.0.3\r\n",
"joblib 1.0.1\r\n",
"jsonschema 3.2.0\r\n",
"jupyter 1.0.0\r\n",
"jupyter-client 7.0.6\r\n",
"jupyter-console 6.4.0\r\n",
"jupyter-core 4.9.1\r\n",
"jupyterlab-pygments 0.1.2\r\n",
"jupyterlab-widgets 1.0.2\r\n",
"kiwisolver 1.3.1\r\n",
"MarkupSafe 2.0.1\r\n",
"matplotlib 3.3.3\r\n",
"mistune 0.8.4\r\n",
"nbclient 0.5.8\r\n",
"nbconvert 6.0.7\r\n",
"nbformat 5.1.3\r\n",
"nest-asyncio 1.5.1\r\n",
"notebook 6.4.6\r\n",
"numpy 1.17.0\r\n",
"nvidia-pyindex 1.0.9\r\n",
"nvidia-tao 0.1.19\r\n",
"opencv-python 3.4.0.12\r\n",
"packaging 21.3\r\n",
"pandocfilters 1.5.0\r\n",
"parso 0.8.2\r\n",
"pexpect 4.8.0\r\n",
"pickleshare 0.7.5\r\n",
"Pillow 8.1.0\r\n",
"pip 21.2.2\r\n",
"prometheus-client 0.12.0\r\n",
"prompt-toolkit 3.0.22\r\n",
"ptyprocess 0.7.0\r\n",
"pycocotools 2.0.2\r\n",
"pycparser 2.21\r\n",
"Pygments 2.10.0\r\n",
"pyparsing 3.0.6\r\n",
"pyrsistent 0.18.0\r\n",
"python-dateutil 2.8.2\r\n",
"pyzmq 22.3.0\r\n",
"qtconsole 5.2.0\r\n",
"QtPy 1.11.2\r\n",
"requests 2.24.0\r\n",
"scipy 1.5.4\r\n",
"Send2Trash 1.8.0\r\n",
"setuptools 58.0.4\r\n",
"six 1.15.0\r\n",
"tabulate 0.8.7\r\n",
"terminado 0.12.1\r\n",
"testpath 0.5.0\r\n",
"tornado 6.1\r\n",
"tqdm 4.62.3\r\n",
"traitlets 4.3.3\r\n",
"typing_extensions 4.0.0\r\n",
"urllib3 1.25.10\r\n",
"wcwidth 0.2.5\r\n",
"webencodings 0.5.1\r\n",
"websocket-client 0.57.0\r\n",
"wheel 0.37.0\r\n",
"widgetsnbextension 3.5.2\r\n",
"zipp 3.6.0\r\n"
]
}
],
"source": [
"pip3 list"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com\r\n",
"Requirement already satisfied: matplotlib==3.3.3 in /home/guest/miniconda3/envs/taotoolkit/lib/python3.6/site-packages (3.3.3)\r\n",
"Requirement already satisfied: python-dateutil>=2.1 in /home/guest/miniconda3/envs/taotoolkit/lib/python3.6/site-packages (from matplotlib==3.3.3) (2.8.2)\r\n",
"Requirement already satisfied: cycler>=0.10 in /home/guest/miniconda3/envs/taotoolkit/lib/python3.6/site-packages (from matplotlib==3.3.3) (0.11.0)\r\n",
"Requirement already satisfied: pillow>=6.2.0 in /home/guest/miniconda3/envs/taotoolkit/lib/python3.6/site-packages (from matplotlib==3.3.3) (8.1.0)\r\n",
"Requirement already satisfied: numpy>=1.15 in /home/guest/miniconda3/envs/taotoolkit/lib/python3.6/site-packages (from matplotlib==3.3.3) (1.17.0)\r\n",
"Requirement already satisfied: kiwisolver>=1.0.1 in /home/guest/miniconda3/envs/taotoolkit/lib/python3.6/site-packages (from matplotlib==3.3.3) (1.3.1)\r\n",
"Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.3 in /home/guest/miniconda3/envs/taotoolkit/lib/python3.6/site-packages (from matplotlib==3.3.3) (3.0.6)\r\n",
"Requirement already satisfied: six>=1.5 in /home/guest/miniconda3/envs/taotoolkit/lib/python3.6/site-packages (from python-dateutil>=2.1->matplotlib==3.3.3) (1.15.0)\r\n"
]
}
],
"source": [
"!pip install matplotlib==3.3.3"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {
"scrolled": true
},
"outputs": [
{
"ename": "ModuleNotFoundError",
"evalue": "No module named 'matplotlib'",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mModuleNotFoundError\u001b[0m Traceback (most recent call last)",
"\u001b[0;32m/tmp/ipykernel_1391743/2971697587.py\u001b[0m in \u001b[0;36m\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0;32mimport\u001b[0m \u001b[0mmatplotlib\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
"\u001b[0;31mModuleNotFoundError\u001b[0m: No module named 'matplotlib'"
]
}
],
"source": [
"import matplotlib"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [
{
"ename": "ModuleNotFoundError",
"evalue": "No module named 'matplotlib'",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mModuleNotFoundError\u001b[0m Traceback (most recent call last)",
"\u001b[0;32m/tmp/ipykernel_1391743/3960136081.py\u001b[0m in \u001b[0;36m\u001b[0;34m\u001b[0m\n\u001b[1;32m 2\u001b[0m \u001b[0;31m# !pip3 install matplotlib==3.3.3\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 3\u001b[0m \u001b[0;31m# %matplotlib inline\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 4\u001b[0;31m \u001b[0;32mimport\u001b[0m \u001b[0mmatplotlib\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpyplot\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mplt\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 5\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mos\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 6\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0mmath\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mceil\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;31mModuleNotFoundError\u001b[0m: No module named 'matplotlib'"
]
}
],
"source": [
"# Simple grid visualizer\n",
"# !pip3 install matplotlib==3.3.3\n",
"# %matplotlib inline\n",
"import matplotlib.pyplot as plt\n",
"import os\n",
"from math import ceil\n",
"valid_image_ext = ['.jpg', '.png', '.jpeg', '.ppm']\n",
"\n",
"def visualize_images(image_dir, num_cols=4, num_images=10):\n",
" output_path = os.path.join(os.environ['LOCAL_EXPERIMENT_DIR'], image_dir)\n",
" num_rows = int(ceil(float(num_images) / float(num_cols)))\n",
" f, axarr = plt.subplots(num_rows, num_cols, figsize=[80,30])\n",
" f.tight_layout()\n",
" a = [os.path.join(output_path, image) for image in os.listdir(output_path) \n",
" if os.path.splitext(image)[1].lower() in valid_image_ext]\n",
" for idx, img_path in enumerate(a[:num_images]):\n",
" col_id = idx % num_cols\n",
" row_id = idx // num_cols\n",
" img = plt.imread(img_path)\n",
" axarr[row_id, col_id].imshow(img) "
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com\r\n",
"Requirement already satisfied: matplotlib in /home/guest/miniconda3/envs/taotoolkit/lib/python3.6/site-packages (3.3.3)\r\n",
"Requirement already satisfied: python-dateutil>=2.1 in /home/guest/miniconda3/envs/taotoolkit/lib/python3.6/site-packages (from matplotlib) (2.8.2)\r\n",
"Requirement already satisfied: cycler>=0.10 in /home/guest/miniconda3/envs/taotoolkit/lib/python3.6/site-packages (from matplotlib) (0.11.0)\r\n",
"Requirement already satisfied: pillow>=6.2.0 in /home/guest/miniconda3/envs/taotoolkit/lib/python3.6/site-packages (from matplotlib) (8.1.0)\r\n",
"Requirement already satisfied: numpy>=1.15 in /home/guest/miniconda3/envs/taotoolkit/lib/python3.6/site-packages (from matplotlib) (1.17.0)\r\n",
"Requirement already satisfied: kiwisolver>=1.0.1 in /home/guest/miniconda3/envs/taotoolkit/lib/python3.6/site-packages (from matplotlib) (1.3.1)\r\n",
"Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.3 in /home/guest/miniconda3/envs/taotoolkit/lib/python3.6/site-packages (from matplotlib) (3.0.6)\r\n",
"Requirement already satisfied: six>=1.5 in /home/guest/miniconda3/envs/taotoolkit/lib/python3.6/site-packages (from python-dateutil>=2.1->matplotlib) (1.15.0)\r\n"
]
}
],
"source": [
"!pip install matplotlib\n"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"scrolled": true
},
"outputs": [
{
"ename": "NameError",
"evalue": "name 'visualize_images' is not defined",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mNameError\u001b[0m Traceback (most recent call last)",
"\u001b[0;32m/tmp/ipykernel_1391743/276316856.py\u001b[0m in \u001b[0;36m\u001b[0;34m\u001b[0m\n\u001b[1;32m 4\u001b[0m \u001b[0mIMAGES\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m12\u001b[0m \u001b[0;31m# number of images to visualize.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 5\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 6\u001b[0;31m \u001b[0mvisualize_images\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mOUTPUT_PATH\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnum_cols\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mCOLS\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnum_images\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mIMAGES\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
"\u001b[0;31mNameError\u001b[0m: name 'visualize_images' is not defined"
]
}
],
"source": [
"# Visualizing the first 12 images.\n",
"OUTPUT_PATH = 'tlt_infer_testing/images_annotated' # relative path from $USER_EXPERIMENT_DIR.\n",
"COLS = 4 # number of columns in the visualizer grid.\n",
"IMAGES = 12 # number of images to visualize.\n",
"\n",
"visualize_images(OUTPUT_PATH, num_cols=COLS, num_images=IMAGES)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 10. Model Export "
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"2021-12-24 10:26:16,544 [INFO] root: Registry: ['nvcr.io']\n",
"Matplotlib created a temporary config/cache directory at /tmp/matplotlib-9d8ski3x because the default path (/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.\n",
"Using TensorFlow backend.\n",
"Using TensorFlow backend.\n",
"WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.\n",
"2021-12-24 02:26:23,902 [INFO] iva.common.export.keras_exporter: Using input nodes: ['input_1']\n",
"2021-12-24 02:26:23,902 [INFO] iva.common.export.keras_exporter: Using output nodes: ['output_cov/Sigmoid', 'output_bbox/BiasAdd']\n",
"/usr/local/lib/python3.6/dist-packages/keras/engine/saving.py:292: UserWarning: No training configuration found in save file: the model was *not* compiled. Compile it manually.\n",
" warnings.warn('No training configuration found in save file: '\n",
"NOTE: UFF has been tested with TensorFlow 1.14.0.\n",
"WARNING: The version of TensorFlow installed on this system is not guaranteed to work with UFF.\n",
"DEBUG [/usr/local/lib/python3.6/dist-packages/uff/converters/tensorflow/converter.py:96] Marking ['output_cov/Sigmoid', 'output_bbox/BiasAdd'] as outputs\n",
"2021-12-24 10:26:53,646 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.\n"
]
}
],
"source": [
"!mkdir -p $LOCAL_EXPERIMENT_DIR/experiment_dir_final\n",
"# Removing a pre-existing copy of the etlt if there has been any.\n",
"import os\n",
"output_file=os.path.join(os.environ['LOCAL_EXPERIMENT_DIR'],\n",
" \"experiment_dir_final/resnet18_detector.etlt\")\n",
"if os.path.exists(output_file):\n",
" os.system(\"rm {}\".format(output_file))\n",
"!tao detectnet_v2 export \\\n",
" -m $USER_EXPERIMENT_DIR/experiment_dir_retrain/weights/resnet18_detector_pruned.tlt \\\n",
" -o $USER_EXPERIMENT_DIR/experiment_dir_final/resnet18_detector.etlt \\\n",
" -k $KEY"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Exported model:\n",
"------------\n",
"total 244M\r\n",
"-rw-r--r-- 1 guest guest 200M Dec 22 16:27 calibration.tensor\r\n",
"-rw-r--r-- 1 guest guest 45M Dec 24 10:26 resnet18_detector.etlt\r\n"
]
}
],
"source": [
"print('Exported model:')\n",
"print('------------')\n",
"!ls -lh $LOCAL_EXPERIMENT_DIR/experiment_dir_final"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### A. Int8 Optimization \n",
"DetectNet_v2 model supports int8 inference mode in TensorRT. \n",
"In order to use int8 mode, we must calibrate the model to run 8-bit inferences -\n",
"\n",
"* Generate calibration tensorfile from the training data using detectnet_v2 calibration_tensorfile\n",
"* Use tao export to generate int8 calibration table.\n",
"\n",
"*Note: For this example, we generate a calibration tensorfile containing 10 batches of training data.\n",
"Ideally, it is best to use atleast 10-20% of the training data to do so. The more data provided during calibration, the closer int8 inferences are to fp32 inferences.*\n",
"\n",
"*Note: If the model was trained with QAT nodes available, please refrain from using the post training int8 optimization as mentioned below. Please export the model in int8 mode (using the arg `--data_type int8`) with just the path to the calibration cache file (using the argument `--cal_cache_file`)*"
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"2021-12-24 10:30:21,031 [INFO] root: Registry: ['nvcr.io']\n",
"Matplotlib created a temporary config/cache directory at /tmp/matplotlib-ow0gmh5m because the default path (/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.\n",
"Using TensorFlow backend.\n",
"usage: detectnet_v2 calibration_tensorfile [-h]\n",
" [--num_processes NUM_PROCESSES]\n",
" [--gpus GPUS]\n",
" [--gpu_index GPU_INDEX [GPU_INDEX ...]]\n",
" [--use_amp] [--log_file LOG_FILE]\n",
" [-e EXPERIMENT_SPEC_FILE]\n",
" [-o OUTPUT_PATH] [-m MAX_BATCHES]\n",
" [-v] [--use_validation_set]\n",
" {calibration_tensorfile,dataset_convert,evaluate,export,inference,prune,train}\n",
" ...\n",
"\n",
"optional arguments:\n",
" -h, --help show this help message and exit\n",
" --num_processes NUM_PROCESSES, -np NUM_PROCESSES\n",
" The number of horovod child processes to be spawned.\n",
" Default is -1(equal to --gpus).\n",
" --gpus GPUS The number of GPUs to be used for the job.\n",
" --gpu_index GPU_INDEX [GPU_INDEX ...]\n",
" The indices of the GPU's to be used.\n",
" --use_amp Flag to enable Auto Mixed Precision.\n",
" --log_file LOG_FILE Path to the output log file.\n",
" -e EXPERIMENT_SPEC_FILE, --experiment_spec_file EXPERIMENT_SPEC_FILE\n",
" Absolute path to the experiment spec file.\n",
" -o OUTPUT_PATH, --output_path OUTPUT_PATH\n",
" Path to the TensorFile that will be created.\n",
" -m MAX_BATCHES, --max_batches MAX_BATCHES\n",
" Maximum number of minibatches to dump. The default is\n",
" to dump the whole dataset.\n",
" -v, --verbose Set verbosity level for the logger.\n",
" --use_validation_set If set, then validation images are dumped. Otherwise,\n",
" training images are dumped.\n",
"\n",
"tasks:\n",
" {calibration_tensorfile,dataset_convert,evaluate,export,inference,prune,train}\n",
"2021-12-24 10:30:24,631 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.\n"
]
}
],
"source": [
"# !tao detectnet_v2 calibration_tensorfile -e $SPECS_DIR/detectnet_v2_train_resnet18_kitti_car.txt \\\n",
"# -m 10 \\\n",
"# -o $USER_EXPERIMENT_DIR/experiment_dir_final/calibration.tensor\n",
"!tao detectnet_v2 calibration_tensorfile -h"
]
},
{
"cell_type": "code",
"execution_count": 37,
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"2021-12-24 12:07:24,957 [INFO] root: Registry: ['nvcr.io']\n",
"Matplotlib created a temporary config/cache directory at /tmp/matplotlib-sctq5k5h because the default path (/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.\n",
"Using TensorFlow backend.\n",
"Using TensorFlow backend.\n",
"WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.\n",
"2021-12-24 04:07:32,210 [INFO] iva.common.export.keras_exporter: Using input nodes: ['input_1']\n",
"2021-12-24 04:07:32,210 [INFO] iva.common.export.keras_exporter: Using output nodes: ['output_cov/Sigmoid', 'output_bbox/BiasAdd']\n",
"/usr/local/lib/python3.6/dist-packages/keras/engine/saving.py:292: UserWarning: No training configuration found in save file: the model was *not* compiled. Compile it manually.\n",
" warnings.warn('No training configuration found in save file: '\n",
"2021-12-24 04:07:33,684 [DEBUG] iva.common.export.keras_exporter: Saving etlt model file at: /workspace/tao-experiments/detectnet_v2_car/experiment_dir_final/resnet18_detector.etlt.\n",
"2021-12-24 04:07:34,644 [DEBUG] modulus.export._uff: Patching keras BatchNormalization...\n",
"2021-12-24 04:07:34,644 [DEBUG] modulus.export._uff: Patching keras Dropout...\n",
"2021-12-24 04:07:34,644 [DEBUG] modulus.export._uff: Patching UFF TensorFlow converter apply_fused_padding...\n",
"2021-12-24 04:07:35,655 [DEBUG] modulus.export._uff: Unpatching keras BatchNormalization layer...\n",
"2021-12-24 04:07:35,655 [DEBUG] modulus.export._uff: Unpatching keras Dropout layer...\n",
"NOTE: UFF has been tested with TensorFlow 1.14.0.\n",
"WARNING: The version of TensorFlow installed on this system is not guaranteed to work with UFF.\n",
"DEBUG [/usr/local/lib/python3.6/dist-packages/uff/converters/tensorflow/converter.py:96] Marking ['output_cov/Sigmoid', 'output_bbox/BiasAdd'] as outputs\n",
"2021-12-24 04:07:38,404 [DEBUG] iva.common.export.base_exporter: Reading input dims from tensorfile.\n",
"2021-12-24 04:07:38,405 [DEBUG] iva.common.export.tensorfile: Opening /workspace/tao-experiments/detectnet_v2_car/experiment_dir_final/calibration.tensor with mode=r\n",
"2021-12-24 04:07:38,611 [DEBUG] iva.common.export.keras_exporter: Input dims: (3, 544, 960)\n",
"2021-12-24 04:07:38,624 [DEBUG] iva.common.export.tensorfile: Opening /workspace/tao-experiments/detectnet_v2_car/experiment_dir_final/calibration.tensor with mode=r\n",
"2021-12-24 04:07:38,624 [INFO] iva.common.export.keras_exporter: Calibration takes time especially if number of batches is large.\n",
"2021-12-24 04:07:44,710 [DEBUG] iva.common.export.base_calibrator: read_calibration_cache - no-op\n",
"2021-12-24 04:07:56,571 [DEBUG] iva.common.export.base_calibrator: read_calibration_cache - no-op\n",
"2021-12-24 04:07:56,571 [INFO] iva.common.export.base_calibrator: Saving calibration cache (size 4864) to /workspace/tao-experiments/detectnet_v2_car/experiment_dir_final/calibration.bin\n",
"2021-12-24 12:08:23,119 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.\n"
]
}
],
"source": [
"# !rm -rf $LOCAL_EXPERIMENT_DIR/experiment_dir_final/resnet18_detector.etlt\n",
"# !rm -rf $LOCAL_EXPERIMENT_DIR/experiment_dir_final/calibration.bin\n",
"!tao detectnet_v2 export \\\n",
" -m $USER_EXPERIMENT_DIR/experiment_dir_retrain/weights/resnet18_detector_pruned.tlt \\\n",
" -o $USER_EXPERIMENT_DIR/experiment_dir_final/resnet18_detector.etlt \\\n",
" -k $KEY \\\n",
" --cal_data_file $USER_EXPERIMENT_DIR/experiment_dir_final/calibration.tensor \\\n",
" --data_type int8 \\\n",
" --batches 10 \\\n",
" --batch_size 8 \\\n",
" --max_batch_size 8 \\\n",
" --engine_file $USER_EXPERIMENT_DIR/experiment_dir_final/resnet18_detector.trt.int8 \\\n",
" --cal_cache_file $USER_EXPERIMENT_DIR/experiment_dir_final/calibration.bin \\\n",
" --verbose"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### B. Generate TensorRT engine \n",
"Verify engine generation using the `tao-converter` utility included with the docker.\n",
"\n",
"The `tao-converter` produces optimized tensorrt engines for the platform that it resides on. Therefore, to get maximum performance, please instantiate this docker and execute the `tao-converter` command, with the exported `.etlt` file and calibration cache (for int8 mode) on your target device. The tao-converter utility included in this docker only works for x86 devices, with discrete NVIDIA GPU's. \n",
"\n",
"For the jetson devices, please download the tao-converter for jetson from the dev zone link [here](https://developer.nvidia.com/tao-converter). \n",
"\n",
"If you choose to integrate your model into deepstream directly, you may do so by simply copying the exported `.etlt` file along with the calibration cache to the target device and updating the spec file that configures the `gst-nvinfer` element to point to this newly exported model. Usually this file is called `config_infer_primary.txt` for detection models and `config_infer_secondary_*.txt` for classification models."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!tao converter $USER_EXPERIMENT_DIR/experiment_dir_final/resnet18_detector.etlt \\\n",
" -k $KEY \\\n",
" -c $USER_EXPERIMENT_DIR/experiment_dir_final/calibration.bin \\\n",
" -o output_cov/Sigmoid,output_bbox/BiasAdd \\\n",
" -d 3,384,1248 \\\n",
" -i nchw \\\n",
" -m 64 \\\n",
" -t int8 \\\n",
" -e $USER_EXPERIMENT_DIR/experiment_dir_final/resnet18_detector.trt \\\n",
" -b 4"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 11. Verify Deployed Model \n",
"Verify the exported model by visualizing inferences on TensorRT.\n",
"In addition to running inference on a `.tlt` model in [step 9](#head-9), the `inference` tool is also capable of consuming the converted `TensorRT engine` from [step 10.B](#head-10-2).\n",
"\n",
"*If after int-8 calibration the accuracy of the int-8 inferences seem to degrade, it could be because the there wasn't enough data in the calibration tensorfile used to calibrate thee model or, the training data is not entirely representative of your test images, and the calibration maybe incorrect. Therefore, you may either regenerate the calibration tensorfile with more batches of the training data and recalibrate the model, or calibrate the model on a few images from the test set. This may be done using `--cal_image_dir` flag in the `export` tool. For more information, please follow the instructions in the USER GUIDE."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### A. Inference using TensorRT engine "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!tao detectnet_v2 inference -e $SPECS_DIR/detectnet_v2_inference_kitti_etlt.txt \\\n",
" -o $USER_EXPERIMENT_DIR/etlt_infer_testing \\\n",
" -i $DATA_DOWNLOAD_DIR/testing/image_2 \\\n",
" -k $KEY"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# visualize the first 12 inferenced images.\n",
"OUTPUT_PATH = 'etlt_infer_testing/images_annotated' # relative path from $USER_EXPERIMENT_DIR.\n",
"COLS = 4 # number of columns in the visualizer grid.\n",
"IMAGES = 12 # number of images to visualize.\n",
"\n",
"visualize_images(OUTPUT_PATH, num_cols=COLS, num_images=IMAGES)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 11. QAT workflow \n",
"This section delves into the newly enabled Quantization Aware Training feature with DetectNet_v2. The workflow defined below converts a pruned model from section [5](#head-5) to enable QAT and retrain this model to while accounting the noise introduced due to quantization in the forward pass. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### A. Convert pruned model to QAT and retrain \n",
"All detectnet models, unpruned and pruned models can be converted to QAT models by setting the `enable_qat` parameter in the `training_config` component of the spec file to `true`."
]
},
{
"cell_type": "code",
"execution_count": 35,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"random_seed: 42\r\n",
"dataset_config {\r\n",
" data_sources {\r\n",
" tfrecords_path: \"/workspace/tao-experiments/data/tfrecords/kitti_trainval/*\"\r\n",
" image_directory_path: \"/workspace/tao-experiments/data/training/\"\r\n",
" }\r\n",
" image_extension: \"png\"\r\n",
" target_class_mapping{\r\n",
" key:\"car\"\r\n",
" value:\"car\"\r\n",
" }\r\n",
" validation_fold: 0\r\n",
"}\r\n",
"augmentation_config {\r\n",
" preprocessing {\r\n",
" output_image_width: 960\r\n",
" output_image_height: 544\r\n",
" min_bbox_width: 1.0\r\n",
" min_bbox_height: 1.0\r\n",
" output_image_channel: 3\r\n",
" enable_auto_resize: true\r\n",
" }\r\n",
" spatial_augmentation {\r\n",
" hflip_probability: 0.5\r\n",
" vflip_probability: 0.0\r\n",
" zoom_min: 1.0\r\n",
" zoom_max: 1.0\r\n",
" translate_max_x: 8.0\r\n",
" translate_max_y: 8.0\r\n",
" }\r\n",
" color_augmentation {\r\n",
" hue_rotation_max: 25.0\r\n",
" saturation_shift_max: 0.20000000298\r\n",
" contrast_scale_max: 0.10000000149\r\n",
" contrast_center: 0.5\r\n",
" }\r\n",
"}\r\n",
"\r\n",
"postprocessing_config {\r\n",
" target_class_config {\r\n",
" key: \"car\"\r\n",
" value {\r\n",
" clustering_config {\r\n",
" clustering_algorithm: DBSCAN\r\n",
" coverage_threshold: 0.005\r\n",
" dbscan_eps: 0.15\r\n",
" dbscan_min_samples: 0.05\r\n",
" minimum_bounding_box_height: 4\r\n",
" dbscan_confidence_threshold: 0.9\r\n",
" }\r\n",
" }\r\n",
" }\r\n",
"}\r\n",
"model_config {\r\n",
" pretrained_model_file: \"/workspace/tao-experiments/experiment/pretrained_trafficcamnet/resnet18_trafficcamnet.tlt\"\r\n",
" num_layers: 18\r\n",
" use_batch_norm: true\r\n",
" load_graph:true\r\n",
" objective_set {\r\n",
" bbox {\r\n",
" scale: 35.0\r\n",
" offset: 0.5\r\n",
" }\r\n",
" cov {\r\n",
" }\r\n",
" }\r\n",
" training_precision {\r\n",
" backend_floatx: FLOAT32\r\n",
" }\r\n",
" arch: \"resnet\"\r\n",
" all_projections: true\r\n",
"}\r\n",
"evaluation_config {\r\n",
" validation_period_during_training: 10\r\n",
" first_validation_epoch: 20\r\n",
" minimum_detection_ground_truth_overlap {\r\n",
" key: \"car\"\r\n",
" value: 0.5\r\n",
" }\r\n",
" evaluation_box_config {\r\n",
" key: \"car\"\r\n",
" value {\r\n",
" minimum_height: 20\r\n",
" maximum_height: 9999\r\n",
" minimum_width: 10\r\n",
" maximum_width: 9999\r\n",
" }\r\n",
" }\r\n",
" average_precision_mode: INTEGRATE\r\n",
"}\r\n",
"\r\n",
"cost_function_config {\r\n",
" target_classes {\r\n",
" name: \"car\"\r\n",
" class_weight: 1.0\r\n",
" coverage_foreground_weight: 0.05\r\n",
" objectives {\r\n",
" name: \"cov\"\r\n",
" initial_weight: 1.0\r\n",
" weight_target: 1.0\r\n",
" }\r\n",
" objectives {\r\n",
" name: \"bbox\"\r\n",
" initial_weight: 10.0\r\n",
" weight_target: 10.0\r\n",
" }\r\n",
" }\r\n",
" enable_autoweighting: true\r\n",
" max_objective_weight: 0.999899983406\r\n",
" min_objective_weight: 9.99999974738e-05\r\n",
"}\r\n",
"training_config {\r\n",
" batch_size_per_gpu: 8\r\n",
" num_epochs:120\r\n",
" enable_qat:true\r\n",
" learning_rate {\r\n",
" soft_start_annealing_schedule {\r\n",
" min_learning_rate: 5e-06\r\n",
" max_learning_rate: 1e-03\r\n",
" soft_start: 0.10000000149\r\n",
" annealing: 0.699999988079\r\n",
" }\r\n",
" }\r\n",
" regularizer {\r\n",
" type: L1\r\n",
" weight: 3.00000002618e-09\r\n",
" }\r\n",
" optimizer {\r\n",
" adam {\r\n",
" epsilon: 9.99999993923e-09\r\n",
" beta1: 0.899999976158\r\n",
" beta2: 0.999000012875\r\n",
" }\r\n",
" }\r\n",
" cost_scaling {\r\n",
" enabled: False\r\n",
" initial_exponent: 20.0\r\n",
" increment: 0.005\r\n",
" decrement: 1.0\r\n",
" }\r\n",
" checkpoint_interval: 10\r\n",
"}\r\n",
"bbox_rasterizer_config {\r\n",
" target_class_config {\r\n",
" key: \"car\"\r\n",
" value: {\r\n",
" cov_center_x: 0.5\r\n",
" cov_center_y: 0.5\r\n",
" cov_radius_x: 0.4\r\n",
" cov_radius_y: 0.4\r\n",
" bbox_min_radius: 1.0\r\n",
" }\r\n",
" }\r\n",
" deadzone_radius: 0.4\r\n",
"}\r\n",
"\r\n"
]
}
],
"source": [
"# Printing the retrain experiment file. \n",
"# Note: We have updated the experiment file to convert the\n",
"# pretrained model to qat mode by setting the enable_qat\n",
"# parameter.\n",
"!cat $LOCAL_SPECS_DIR/detectnet_v2_retrain_resnet18_kitti_qat.txt"
]
},
{
"cell_type": "code",
"execution_count": 36,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"2022-01-06 16:16:17,824 [INFO] root: Registry: ['nvcr.io']\n",
"Matplotlib created a temporary config/cache directory at /tmp/matplotlib-c0erns2p because the default path (/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.\n",
"Using TensorFlow backend.\n",
"WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.\n",
"Using TensorFlow backend.\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:43: The name tf.train.SessionRunHook is deprecated. Please use tf.estimator.SessionRunHook instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/checkpoint_saver_hook.py:25: The name tf.train.CheckpointSaverHook is deprecated. Please use tf.estimator.CheckpointSaverHook instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/scripts/train.py:68: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/scripts/train.py:68: The name tf.logging.INFO is deprecated. Please use tf.compat.v1.logging.INFO instead.\n",
"\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod/tensorflow/__init__.py:117: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.\n",
"\n",
"2022-01-06 08:16:23,560 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/horovod/tensorflow/__init__.py:117: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.\n",
"\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod/tensorflow/__init__.py:143: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.\n",
"\n",
"2022-01-06 08:16:23,560 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/horovod/tensorflow/__init__.py:143: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.\n",
"\n",
"2022-01-06 08:16:23,959 [INFO] iva.common.logging.logging: Log file already exists at /workspace/tao-experiments/experiment/experiment_dir_retrain_qat/status.json\n",
"2022-01-06 08:16:23,959 [INFO] __main__: Loading experiment spec at /workspace/tao-experiments/specs/detectnet_v2_retrain_resnet18_kitti_qat.txt.\n",
"2022-01-06 08:16:23,961 [INFO] iva.detectnet_v2.spec_handler.spec_loader: Merging specification from /workspace/tao-experiments/specs/detectnet_v2_retrain_resnet18_kitti_qat.txt\n",
"2022-01-06 08:16:24,074 [INFO] __main__: Cannot iterate over exactly 761 samples with a batch size of 8; each epoch will therefore take one extra step.\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:107: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.\n",
"\n",
"2022-01-06 08:16:24,075 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:107: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:110: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.\n",
"\n",
"2022-01-06 08:16:24,076 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:110: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:113: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead.\n",
"\n",
"2022-01-06 08:16:24,077 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:113: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead.\n",
"\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.\n",
"\n",
"2022-01-06 08:16:24,476 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.\n",
"\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.\n",
"\n",
"2022-01-06 08:16:24,485 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.\n",
"\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1834: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead.\n",
"\n",
"2022-01-06 08:16:24,501 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1834: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead.\n",
"\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.\n",
"\n",
"2022-01-06 08:16:25,114 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.\n",
"\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead.\n",
"\n",
"2022-01-06 08:16:25,114 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead.\n",
"\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead.\n",
"\n",
"2022-01-06 08:16:25,302 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead.\n",
"\n",
"/usr/local/lib/python3.6/dist-packages/keras/engine/saving.py:292: UserWarning: No training configuration found in save file: the model was *not* compiled. Compile it manually.\n",
" warnings.warn('No training configuration found in save file: '\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"2022-01-06 08:16:42,071 [INFO] iva.detectnet_v2.objectives.bbox_objective: Default L1 loss function will be used.\n",
"__________________________________________________________________________________________________\n",
"Layer (type) Output Shape Param # Connected to \n",
"==================================================================================================\n",
"input_1 (InputLayer) (None, 3, 544, 960) 0 \n",
"__________________________________________________________________________________________________\n",
"input_1_qdq (QDQ) (None, 3, 544, 960) 1 input_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"conv1 (QuantizedConv2D) (None, 64, 272, 480) 9472 input_1_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"bn_conv1 (BatchNormalization) (None, 64, 272, 480) 256 conv1[0][0] \n",
"__________________________________________________________________________________________________\n",
"activation_1 (ReLU) (None, 64, 272, 480) 0 bn_conv1[0][0] \n",
"__________________________________________________________________________________________________\n",
"activation_1_qdq (QDQ) (None, 64, 272, 480) 1 activation_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1a_conv_1 (QuantizedConv2 (None, 64, 136, 240) 36928 activation_1_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1a_bn_1 (BatchNormalizati (None, 64, 136, 240) 256 block_1a_conv_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1a_relu_1 (ReLU) (None, 64, 136, 240) 0 block_1a_bn_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1a_relu_1_qdq (QDQ) (None, 64, 136, 240) 1 block_1a_relu_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1a_conv_2 (QuantizedConv2 (None, 64, 136, 240) 36928 block_1a_relu_1_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1a_conv_shortcut (Quantiz (None, 64, 136, 240) 4160 activation_1_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1a_bn_2 (BatchNormalizati (None, 64, 136, 240) 256 block_1a_conv_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1a_bn_shortcut (BatchNorm (None, 64, 136, 240) 256 block_1a_conv_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1a_bn_2_qdq (QDQ) (None, 64, 136, 240) 1 block_1a_bn_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1a_bn_shortcut_qdq (QDQ) (None, 64, 136, 240) 1 block_1a_bn_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_1 (Add) (None, 64, 136, 240) 0 block_1a_bn_2_qdq[0][0] \n",
" block_1a_bn_shortcut_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_1_qdq (QDQ) (None, 64, 136, 240) 1 add_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1a_relu (ReLU) (None, 64, 136, 240) 0 add_1_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1a_relu_qdq (QDQ) (None, 64, 136, 240) 1 block_1a_relu[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1b_conv_1 (QuantizedConv2 (None, 64, 136, 240) 36928 block_1a_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1b_bn_1 (BatchNormalizati (None, 64, 136, 240) 256 block_1b_conv_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1b_relu_1 (ReLU) (None, 64, 136, 240) 0 block_1b_bn_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1b_relu_1_qdq (QDQ) (None, 64, 136, 240) 1 block_1b_relu_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1b_conv_2 (QuantizedConv2 (None, 64, 136, 240) 36928 block_1b_relu_1_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1b_conv_shortcut (Quantiz (None, 64, 136, 240) 4160 block_1a_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1b_bn_2 (BatchNormalizati (None, 64, 136, 240) 256 block_1b_conv_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1b_bn_shortcut (BatchNorm (None, 64, 136, 240) 256 block_1b_conv_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1b_bn_2_qdq (QDQ) (None, 64, 136, 240) 1 block_1b_bn_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1b_bn_shortcut_qdq (QDQ) (None, 64, 136, 240) 1 block_1b_bn_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_2 (Add) (None, 64, 136, 240) 0 block_1b_bn_2_qdq[0][0] \n",
" block_1b_bn_shortcut_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_2_qdq (QDQ) (None, 64, 136, 240) 1 add_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1b_relu (ReLU) (None, 64, 136, 240) 0 add_2_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1b_relu_qdq (QDQ) (None, 64, 136, 240) 1 block_1b_relu[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2a_conv_1 (QuantizedConv2 (None, 128, 68, 120) 73856 block_1b_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2a_bn_1 (BatchNormalizati (None, 128, 68, 120) 512 block_2a_conv_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2a_relu_1 (ReLU) (None, 128, 68, 120) 0 block_2a_bn_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2a_relu_1_qdq (QDQ) (None, 128, 68, 120) 1 block_2a_relu_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2a_conv_2 (QuantizedConv2 (None, 128, 68, 120) 147584 block_2a_relu_1_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2a_conv_shortcut (Quantiz (None, 128, 68, 120) 8320 block_1b_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2a_bn_2 (BatchNormalizati (None, 128, 68, 120) 512 block_2a_conv_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2a_bn_shortcut (BatchNorm (None, 128, 68, 120) 512 block_2a_conv_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2a_bn_2_qdq (QDQ) (None, 128, 68, 120) 1 block_2a_bn_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2a_bn_shortcut_qdq (QDQ) (None, 128, 68, 120) 1 block_2a_bn_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_3 (Add) (None, 128, 68, 120) 0 block_2a_bn_2_qdq[0][0] \n",
" block_2a_bn_shortcut_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_3_qdq (QDQ) (None, 128, 68, 120) 1 add_3[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2a_relu (ReLU) (None, 128, 68, 120) 0 add_3_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2a_relu_qdq (QDQ) (None, 128, 68, 120) 1 block_2a_relu[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2b_conv_1 (QuantizedConv2 (None, 128, 68, 120) 147584 block_2a_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2b_bn_1 (BatchNormalizati (None, 128, 68, 120) 512 block_2b_conv_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2b_relu_1 (ReLU) (None, 128, 68, 120) 0 block_2b_bn_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2b_relu_1_qdq (QDQ) (None, 128, 68, 120) 1 block_2b_relu_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2b_conv_2 (QuantizedConv2 (None, 128, 68, 120) 147584 block_2b_relu_1_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2b_conv_shortcut (Quantiz (None, 128, 68, 120) 16512 block_2a_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2b_bn_2 (BatchNormalizati (None, 128, 68, 120) 512 block_2b_conv_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2b_bn_shortcut (BatchNorm (None, 128, 68, 120) 512 block_2b_conv_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2b_bn_2_qdq (QDQ) (None, 128, 68, 120) 1 block_2b_bn_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2b_bn_shortcut_qdq (QDQ) (None, 128, 68, 120) 1 block_2b_bn_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_4 (Add) (None, 128, 68, 120) 0 block_2b_bn_2_qdq[0][0] \n",
" block_2b_bn_shortcut_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_4_qdq (QDQ) (None, 128, 68, 120) 1 add_4[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2b_relu (ReLU) (None, 128, 68, 120) 0 add_4_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2b_relu_qdq (QDQ) (None, 128, 68, 120) 1 block_2b_relu[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3a_conv_1 (QuantizedConv2 (None, 256, 34, 60) 295168 block_2b_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3a_bn_1 (BatchNormalizati (None, 256, 34, 60) 1024 block_3a_conv_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3a_relu_1 (ReLU) (None, 256, 34, 60) 0 block_3a_bn_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3a_relu_1_qdq (QDQ) (None, 256, 34, 60) 1 block_3a_relu_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3a_conv_2 (QuantizedConv2 (None, 256, 34, 60) 590080 block_3a_relu_1_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3a_conv_shortcut (Quantiz (None, 256, 34, 60) 33024 block_2b_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3a_bn_2 (BatchNormalizati (None, 256, 34, 60) 1024 block_3a_conv_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3a_bn_shortcut (BatchNorm (None, 256, 34, 60) 1024 block_3a_conv_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3a_bn_2_qdq (QDQ) (None, 256, 34, 60) 1 block_3a_bn_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3a_bn_shortcut_qdq (QDQ) (None, 256, 34, 60) 1 block_3a_bn_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_5 (Add) (None, 256, 34, 60) 0 block_3a_bn_2_qdq[0][0] \n",
" block_3a_bn_shortcut_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_5_qdq (QDQ) (None, 256, 34, 60) 1 add_5[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3a_relu (ReLU) (None, 256, 34, 60) 0 add_5_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3a_relu_qdq (QDQ) (None, 256, 34, 60) 1 block_3a_relu[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3b_conv_1 (QuantizedConv2 (None, 256, 34, 60) 590080 block_3a_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3b_bn_1 (BatchNormalizati (None, 256, 34, 60) 1024 block_3b_conv_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3b_relu_1 (ReLU) (None, 256, 34, 60) 0 block_3b_bn_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3b_relu_1_qdq (QDQ) (None, 256, 34, 60) 1 block_3b_relu_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3b_conv_2 (QuantizedConv2 (None, 256, 34, 60) 590080 block_3b_relu_1_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3b_conv_shortcut (Quantiz (None, 256, 34, 60) 65792 block_3a_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3b_bn_2 (BatchNormalizati (None, 256, 34, 60) 1024 block_3b_conv_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3b_bn_shortcut (BatchNorm (None, 256, 34, 60) 1024 block_3b_conv_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3b_bn_2_qdq (QDQ) (None, 256, 34, 60) 1 block_3b_bn_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3b_bn_shortcut_qdq (QDQ) (None, 256, 34, 60) 1 block_3b_bn_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_6 (Add) (None, 256, 34, 60) 0 block_3b_bn_2_qdq[0][0] \n",
" block_3b_bn_shortcut_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_6_qdq (QDQ) (None, 256, 34, 60) 1 add_6[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3b_relu (ReLU) (None, 256, 34, 60) 0 add_6_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3b_relu_qdq (QDQ) (None, 256, 34, 60) 1 block_3b_relu[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4a_conv_1 (QuantizedConv2 (None, 512, 34, 60) 1180160 block_3b_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4a_bn_1 (BatchNormalizati (None, 512, 34, 60) 2048 block_4a_conv_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4a_relu_1 (ReLU) (None, 512, 34, 60) 0 block_4a_bn_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4a_relu_1_qdq (QDQ) (None, 512, 34, 60) 1 block_4a_relu_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4a_conv_2 (QuantizedConv2 (None, 512, 34, 60) 2359808 block_4a_relu_1_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4a_conv_shortcut (Quantiz (None, 512, 34, 60) 131584 block_3b_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4a_bn_2 (BatchNormalizati (None, 512, 34, 60) 2048 block_4a_conv_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4a_bn_shortcut (BatchNorm (None, 512, 34, 60) 2048 block_4a_conv_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4a_bn_2_qdq (QDQ) (None, 512, 34, 60) 1 block_4a_bn_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4a_bn_shortcut_qdq (QDQ) (None, 512, 34, 60) 1 block_4a_bn_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_7 (Add) (None, 512, 34, 60) 0 block_4a_bn_2_qdq[0][0] \n",
" block_4a_bn_shortcut_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_7_qdq (QDQ) (None, 512, 34, 60) 1 add_7[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4a_relu (ReLU) (None, 512, 34, 60) 0 add_7_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4a_relu_qdq (QDQ) (None, 512, 34, 60) 1 block_4a_relu[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4b_conv_1 (QuantizedConv2 (None, 512, 34, 60) 2359808 block_4a_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4b_bn_1 (BatchNormalizati (None, 512, 34, 60) 2048 block_4b_conv_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4b_relu_1 (ReLU) (None, 512, 34, 60) 0 block_4b_bn_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4b_relu_1_qdq (QDQ) (None, 512, 34, 60) 1 block_4b_relu_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4b_conv_2 (QuantizedConv2 (None, 512, 34, 60) 2359808 block_4b_relu_1_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4b_conv_shortcut (Quantiz (None, 512, 34, 60) 262656 block_4a_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4b_bn_2 (BatchNormalizati (None, 512, 34, 60) 2048 block_4b_conv_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4b_bn_shortcut (BatchNorm (None, 512, 34, 60) 2048 block_4b_conv_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4b_bn_2_qdq (QDQ) (None, 512, 34, 60) 1 block_4b_bn_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4b_bn_shortcut_qdq (QDQ) (None, 512, 34, 60) 1 block_4b_bn_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_8 (Add) (None, 512, 34, 60) 0 block_4b_bn_2_qdq[0][0] \n",
" block_4b_bn_shortcut_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_8_qdq (QDQ) (None, 512, 34, 60) 1 add_8[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4b_relu (ReLU) (None, 512, 34, 60) 0 add_8_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4b_relu_qdq (QDQ) (None, 512, 34, 60) 1 block_4b_relu[0][0] \n",
"__________________________________________________________________________________________________\n",
"output_bbox (Conv2D) (None, 16, 34, 60) 8208 block_4b_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"output_cov (Conv2D) (None, 4, 34, 60) 2052 block_4b_relu_qdq[0][0] \n",
"==================================================================================================\n",
"Total params: 11,558,590\n",
"Trainable params: 11,546,900\n",
"Non-trainable params: 11,690\n",
"__________________________________________________________________________________________________\n",
"2022-01-06 08:16:44,324 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Serial augmentation enabled = False\n",
"2022-01-06 08:16:44,324 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Pseudo sharding enabled = False\n",
"2022-01-06 08:16:44,324 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Max Image Dimensions (all sources): (0, 0)\n",
"2022-01-06 08:16:44,324 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: number of cpus: 16, io threads: 32, compute threads: 16, buffered batches: 4\n",
"2022-01-06 08:16:44,324 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: total dataset size 761, number of sources: 1, batch size per gpu: 8, steps: 96\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.\n",
"\n",
"2022-01-06 08:16:44,359 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.\n",
"\n",
"WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code\n",
"2022-01-06 08:16:44,401 [WARNING] tensorflow: Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code\n",
"2022-01-06 08:16:44,420 [INFO] iva.detectnet_v2.dataloader.default_dataloader: Bounding box coordinates were detected in the input specification! Bboxes will be automatically converted to polygon coordinates.\n",
"2022-01-06 08:16:44,626 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: shuffle: True - shard 0 of 1\n",
"2022-01-06 08:16:44,631 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: sampling 1 datasets with weights:\n",
"2022-01-06 08:16:44,631 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: source: 0 weight: 1.000000\n",
"WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code\n",
"2022-01-06 08:16:44,642 [WARNING] tensorflow: Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/core/build_wheel.runfiles/ai_infra/moduluspy/modulus/blocks/data_loaders/multi_source_loader/types/images2d_reference.py:427: The name tf.image.resize_images is deprecated. Please use tf.image.resize instead.\n",
"\n",
"2022-01-06 08:16:44,662 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/core/build_wheel.runfiles/ai_infra/moduluspy/modulus/blocks/data_loaders/multi_source_loader/types/images2d_reference.py:427: The name tf.image.resize_images is deprecated. Please use tf.image.resize instead.\n",
"\n",
"2022-01-06 08:16:44,943 [INFO] __main__: Found 761 samples in training set\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/rasterizers/bbox_rasterizer.py:347: The name tf.bincount is deprecated. Please use tf.math.bincount instead.\n",
"\n",
"2022-01-06 08:16:45,029 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/rasterizers/bbox_rasterizer.py:347: The name tf.bincount is deprecated. Please use tf.math.bincount instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/training_proto_utilities.py:89: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.\n",
"\n",
"2022-01-06 08:16:45,118 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/training_proto_utilities.py:89: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/training_proto_utilities.py:36: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead.\n",
"\n",
"2022-01-06 08:16:45,131 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/training_proto_utilities.py:36: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead.\n",
"\n",
"Traceback (most recent call last):\n",
" File \"/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py\", line 1607, in _create_c_op\n",
" c_op = c_api.TF_FinishOperation(op_desc)\n",
"tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot reshape a tensor with 261120 elements to shape [8,1,4,34,60] (65280 elements) for 'reshape_1_1/Reshape' (op: 'Reshape') with input shapes: [8,16,34,60], [5] and with input tensors computed as partial shapes: input[1] = [8,1,4,34,60].\n",
"\n",
"During handling of the above exception, another exception occurred:\n",
"\n",
"Traceback (most recent call last):\n",
" File \"/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/scripts/train.py\", line 843, in \n",
" File \"/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/scripts/train.py\", line 832, in \n",
" File \"\", line 2, in main\n",
" File \"/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/utilities/timer.py\", line 46, in wrapped_fn\n",
" File \"/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/scripts/train.py\", line 821, in main\n",
" File \"/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/scripts/train.py\", line 702, in run_experiment\n",
" File \"/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/scripts/train.py\", line 613, in train_gridbox\n",
" File \"/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/scripts/train.py\", line 468, in build_training_graph\n",
" File \"/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/model/detectnet_model.py\", line 582, in build_training_graph\n",
" File \"/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/model/detectnet_model.py\", line 302, in predictions_to_dict\n",
" File \"/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/objectives/base_objective.py\", line 98, in reshape_output\n",
" File \"/usr/local/lib/python3.6/dist-packages/keras/engine/base_layer.py\", line 457, in __call__\n",
" output = self.call(inputs, **kwargs)\n",
" File \"/usr/local/lib/python3.6/dist-packages/keras/layers/core.py\", line 401, in call\n",
" return K.reshape(inputs, (K.shape(inputs)[0],) + self.target_shape)\n",
" File \"/usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py\", line 1969, in reshape\n",
" return tf.reshape(x, shape)\n",
" File \"/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/array_ops.py\", line 131, in reshape\n",
" result = gen_array_ops.reshape(tensor, shape, name)\n",
" File \"/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gen_array_ops.py\", line 8115, in reshape\n",
" \"Reshape\", tensor=tensor, shape=shape, name=name)\n",
" File \"/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/op_def_library.py\", line 794, in _apply_op_helper\n",
" op_def=op_def)\n",
" File \"/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/util/deprecation.py\", line 513, in new_func\n",
" return func(*args, **kwargs)\n",
" File \"/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py\", line 3357, in create_op\n",
" attrs, op_def, compute_device)\n",
" File \"/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py\", line 3426, in _create_op_internal\n",
" op_def=op_def)\n",
" File \"/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py\", line 1770, in __init__\n",
" control_input_ops)\n",
" File \"/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py\", line 1610, in _create_c_op\n",
" raise ValueError(str(e))\n",
"ValueError: Cannot reshape a tensor with 261120 elements to shape [8,1,4,34,60] (65280 elements) for 'reshape_1_1/Reshape' (op: 'Reshape') with input shapes: [8,16,34,60], [5] and with input tensors computed as partial shapes: input[1] = [8,1,4,34,60].\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"2022-01-06 16:16:46,566 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.\r\n"
]
}
],
"source": [
"!tao detectnet_v2 train -e $SPECS_DIR/detectnet_v2_retrain_resnet18_kitti_qat.txt \\\n",
" -r $USER_EXPERIMENT_DIR/experiment_dir_retrain_qat \\\n",
" -k $KEY \\\n",
" -n resnet18_detector_pruned_qat \\\n",
" --gpus $NUM_GPUS"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"total 45472\r\n",
"-rw-r--r-- 1 guest guest 46562000 Dec 31 12:04 resnet18_detector_pruned_qat.tlt\r\n"
]
}
],
"source": [
"!ls -rlt $LOCAL_EXPERIMENT_DIR/experiment_dir_retrain_qat/weights"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### B. Evaluate QAT converted model \n",
"This section evaluates a QAT enabled pruned retrained model. The mAP of this model should be comparable to that of the pruned retrained model without QAT. However, due to quantization, it is possible sometimes to see a drop in the mAP value for certain datasets."
]
},
{
"cell_type": "code",
"execution_count": 38,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"2022-01-06 16:20:00,401 [INFO] root: Registry: ['nvcr.io']\n",
"Matplotlib created a temporary config/cache directory at /tmp/matplotlib-wx4hs8w2 because the default path (/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.\n",
"Using TensorFlow backend.\n",
"WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.\n",
"Using TensorFlow backend.\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:43: The name tf.train.SessionRunHook is deprecated. Please use tf.estimator.SessionRunHook instead.\n",
"\n",
"2022-01-06 08:20:06,043 [INFO] iva.detectnet_v2.spec_handler.spec_loader: Merging specification from /workspace/tao-experiments/specs/detectnet_v2_retrain_resnet18_kitti_qat.txt\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:153: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.\n",
"\n",
"2022-01-06 08:20:06,047 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:153: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.\n",
"\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.\n",
"\n",
"2022-01-06 08:20:06,399 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.\n",
"\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.\n",
"\n",
"2022-01-06 08:20:06,413 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.\n",
"\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1834: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead.\n",
"\n",
"2022-01-06 08:20:06,431 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1834: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead.\n",
"\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.\n",
"\n",
"2022-01-06 08:20:07,339 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.\n",
"\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:181: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.\n",
"\n",
"2022-01-06 08:20:07,340 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:181: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.\n",
"\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:186: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.\n",
"\n",
"2022-01-06 08:20:07,340 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:186: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.\n",
"\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:190: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.\n",
"\n",
"2022-01-06 08:20:07,660 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:190: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.\n",
"\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead.\n",
"\n",
"2022-01-06 08:20:07,660 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead.\n",
"\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead.\n",
"\n",
"2022-01-06 08:20:07,873 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead.\n",
"\n",
"/usr/local/lib/python3.6/dist-packages/keras/engine/saving.py:292: UserWarning: No training configuration found in save file: the model was *not* compiled. Compile it manually.\n",
" warnings.warn('No training configuration found in save file: '\n",
"2022-01-06 08:20:08,144 [INFO] iva.detectnet_v2.objectives.bbox_objective: Default L1 loss function will be used.\n",
"2022-01-06 08:20:08,243 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Serial augmentation enabled = False\n",
"2022-01-06 08:20:08,243 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Pseudo sharding enabled = False\n",
"2022-01-06 08:20:08,243 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Max Image Dimensions (all sources): (0, 0)\n",
"2022-01-06 08:20:08,243 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: number of cpus: 16, io threads: 32, compute threads: 16, buffered batches: 4\n",
"2022-01-06 08:20:08,244 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: total dataset size 190, number of sources: 1, batch size per gpu: 8, steps: 24\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.\n",
"\n",
"2022-01-06 08:20:08,271 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.\n",
"\n",
"WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code\n",
"2022-01-06 08:20:08,311 [WARNING] tensorflow: Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code\n",
"2022-01-06 08:20:08,325 [INFO] iva.detectnet_v2.dataloader.default_dataloader: Bounding box coordinates were detected in the input specification! Bboxes will be automatically converted to polygon coordinates.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"2022-01-06 08:20:08,525 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: shuffle: False - shard 0 of 1\n",
"2022-01-06 08:20:08,530 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: sampling 1 datasets with weights:\n",
"2022-01-06 08:20:08,530 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: source: 0 weight: 1.000000\n",
"WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code\n",
"2022-01-06 08:20:08,541 [WARNING] tensorflow: Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/core/build_wheel.runfiles/ai_infra/moduluspy/modulus/blocks/data_loaders/multi_source_loader/types/images2d_reference.py:427: The name tf.image.resize_images is deprecated. Please use tf.image.resize instead.\n",
"\n",
"2022-01-06 08:20:08,560 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/core/build_wheel.runfiles/ai_infra/moduluspy/modulus/blocks/data_loaders/multi_source_loader/types/images2d_reference.py:427: The name tf.image.resize_images is deprecated. Please use tf.image.resize instead.\n",
"\n",
"2022-01-06 08:20:08,742 [INFO] iva.detectnet_v2.evaluation.build_evaluator: Found 190 samples in validation set\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:107: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.\n",
"\n",
"2022-01-06 08:20:08,742 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:107: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:110: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.\n",
"\n",
"2022-01-06 08:20:08,743 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:110: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:113: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead.\n",
"\n",
"2022-01-06 08:20:08,744 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:113: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/rasterizers/bbox_rasterizer.py:347: The name tf.bincount is deprecated. Please use tf.math.bincount instead.\n",
"\n",
"2022-01-06 08:20:08,841 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/rasterizers/bbox_rasterizer.py:347: The name tf.bincount is deprecated. Please use tf.math.bincount instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_functions.py:17: The name tf.log is deprecated. Please use tf.math.log instead.\n",
"\n",
"2022-01-06 08:20:09,221 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_functions.py:17: The name tf.log is deprecated. Please use tf.math.log instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:235: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead.\n",
"\n",
"2022-01-06 08:20:09,229 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:235: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead.\n",
"\n",
"__________________________________________________________________________________________________\n",
"Layer (type) Output Shape Param # Connected to \n",
"==================================================================================================\n",
"input_1 (InputLayer) (None, 3, 544, 960) 0 \n",
"__________________________________________________________________________________________________\n",
"input_1_qdq (QDQ) (None, 3, 544, 960) 1 input_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"conv1 (QuantizedConv2D) (None, 64, 272, 480) 9472 input_1_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"bn_conv1 (BatchNormalization) (None, 64, 272, 480) 256 conv1[0][0] \n",
"__________________________________________________________________________________________________\n",
"activation_1 (ReLU) (None, 64, 272, 480) 0 bn_conv1[0][0] \n",
"__________________________________________________________________________________________________\n",
"activation_1_qdq (QDQ) (None, 64, 272, 480) 1 activation_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1a_conv_1 (QuantizedConv2 (None, 64, 136, 240) 36928 activation_1_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1a_bn_1 (BatchNormalizati (None, 64, 136, 240) 256 block_1a_conv_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1a_relu_1 (ReLU) (None, 64, 136, 240) 0 block_1a_bn_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1a_relu_1_qdq (QDQ) (None, 64, 136, 240) 1 block_1a_relu_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1a_conv_2 (QuantizedConv2 (None, 64, 136, 240) 36928 block_1a_relu_1_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1a_conv_shortcut (Quantiz (None, 64, 136, 240) 4160 activation_1_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1a_bn_2 (BatchNormalizati (None, 64, 136, 240) 256 block_1a_conv_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1a_bn_shortcut (BatchNorm (None, 64, 136, 240) 256 block_1a_conv_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1a_bn_2_qdq (QDQ) (None, 64, 136, 240) 1 block_1a_bn_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1a_bn_shortcut_qdq (QDQ) (None, 64, 136, 240) 1 block_1a_bn_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_1 (Add) (None, 64, 136, 240) 0 block_1a_bn_2_qdq[0][0] \n",
" block_1a_bn_shortcut_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_1_qdq (QDQ) (None, 64, 136, 240) 1 add_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1a_relu (ReLU) (None, 64, 136, 240) 0 add_1_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1a_relu_qdq (QDQ) (None, 64, 136, 240) 1 block_1a_relu[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1b_conv_1 (QuantizedConv2 (None, 64, 136, 240) 36928 block_1a_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1b_bn_1 (BatchNormalizati (None, 64, 136, 240) 256 block_1b_conv_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1b_relu_1 (ReLU) (None, 64, 136, 240) 0 block_1b_bn_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1b_relu_1_qdq (QDQ) (None, 64, 136, 240) 1 block_1b_relu_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1b_conv_2 (QuantizedConv2 (None, 64, 136, 240) 36928 block_1b_relu_1_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1b_conv_shortcut (Quantiz (None, 64, 136, 240) 4160 block_1a_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1b_bn_2 (BatchNormalizati (None, 64, 136, 240) 256 block_1b_conv_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1b_bn_shortcut (BatchNorm (None, 64, 136, 240) 256 block_1b_conv_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1b_bn_2_qdq (QDQ) (None, 64, 136, 240) 1 block_1b_bn_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1b_bn_shortcut_qdq (QDQ) (None, 64, 136, 240) 1 block_1b_bn_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_2 (Add) (None, 64, 136, 240) 0 block_1b_bn_2_qdq[0][0] \n",
" block_1b_bn_shortcut_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_2_qdq (QDQ) (None, 64, 136, 240) 1 add_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1b_relu (ReLU) (None, 64, 136, 240) 0 add_2_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_1b_relu_qdq (QDQ) (None, 64, 136, 240) 1 block_1b_relu[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2a_conv_1 (QuantizedConv2 (None, 128, 68, 120) 73856 block_1b_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2a_bn_1 (BatchNormalizati (None, 128, 68, 120) 512 block_2a_conv_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2a_relu_1 (ReLU) (None, 128, 68, 120) 0 block_2a_bn_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2a_relu_1_qdq (QDQ) (None, 128, 68, 120) 1 block_2a_relu_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2a_conv_2 (QuantizedConv2 (None, 128, 68, 120) 147584 block_2a_relu_1_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2a_conv_shortcut (Quantiz (None, 128, 68, 120) 8320 block_1b_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2a_bn_2 (BatchNormalizati (None, 128, 68, 120) 512 block_2a_conv_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2a_bn_shortcut (BatchNorm (None, 128, 68, 120) 512 block_2a_conv_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2a_bn_2_qdq (QDQ) (None, 128, 68, 120) 1 block_2a_bn_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2a_bn_shortcut_qdq (QDQ) (None, 128, 68, 120) 1 block_2a_bn_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_3 (Add) (None, 128, 68, 120) 0 block_2a_bn_2_qdq[0][0] \n",
" block_2a_bn_shortcut_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_3_qdq (QDQ) (None, 128, 68, 120) 1 add_3[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2a_relu (ReLU) (None, 128, 68, 120) 0 add_3_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2a_relu_qdq (QDQ) (None, 128, 68, 120) 1 block_2a_relu[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2b_conv_1 (QuantizedConv2 (None, 128, 68, 120) 147584 block_2a_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2b_bn_1 (BatchNormalizati (None, 128, 68, 120) 512 block_2b_conv_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2b_relu_1 (ReLU) (None, 128, 68, 120) 0 block_2b_bn_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2b_relu_1_qdq (QDQ) (None, 128, 68, 120) 1 block_2b_relu_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2b_conv_2 (QuantizedConv2 (None, 128, 68, 120) 147584 block_2b_relu_1_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2b_conv_shortcut (Quantiz (None, 128, 68, 120) 16512 block_2a_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2b_bn_2 (BatchNormalizati (None, 128, 68, 120) 512 block_2b_conv_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2b_bn_shortcut (BatchNorm (None, 128, 68, 120) 512 block_2b_conv_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2b_bn_2_qdq (QDQ) (None, 128, 68, 120) 1 block_2b_bn_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2b_bn_shortcut_qdq (QDQ) (None, 128, 68, 120) 1 block_2b_bn_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_4 (Add) (None, 128, 68, 120) 0 block_2b_bn_2_qdq[0][0] \n",
" block_2b_bn_shortcut_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_4_qdq (QDQ) (None, 128, 68, 120) 1 add_4[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2b_relu (ReLU) (None, 128, 68, 120) 0 add_4_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_2b_relu_qdq (QDQ) (None, 128, 68, 120) 1 block_2b_relu[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3a_conv_1 (QuantizedConv2 (None, 256, 34, 60) 295168 block_2b_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3a_bn_1 (BatchNormalizati (None, 256, 34, 60) 1024 block_3a_conv_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3a_relu_1 (ReLU) (None, 256, 34, 60) 0 block_3a_bn_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3a_relu_1_qdq (QDQ) (None, 256, 34, 60) 1 block_3a_relu_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3a_conv_2 (QuantizedConv2 (None, 256, 34, 60) 590080 block_3a_relu_1_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3a_conv_shortcut (Quantiz (None, 256, 34, 60) 33024 block_2b_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3a_bn_2 (BatchNormalizati (None, 256, 34, 60) 1024 block_3a_conv_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3a_bn_shortcut (BatchNorm (None, 256, 34, 60) 1024 block_3a_conv_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3a_bn_2_qdq (QDQ) (None, 256, 34, 60) 1 block_3a_bn_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3a_bn_shortcut_qdq (QDQ) (None, 256, 34, 60) 1 block_3a_bn_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_5 (Add) (None, 256, 34, 60) 0 block_3a_bn_2_qdq[0][0] \n",
" block_3a_bn_shortcut_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_5_qdq (QDQ) (None, 256, 34, 60) 1 add_5[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3a_relu (ReLU) (None, 256, 34, 60) 0 add_5_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3a_relu_qdq (QDQ) (None, 256, 34, 60) 1 block_3a_relu[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3b_conv_1 (QuantizedConv2 (None, 256, 34, 60) 590080 block_3a_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3b_bn_1 (BatchNormalizati (None, 256, 34, 60) 1024 block_3b_conv_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3b_relu_1 (ReLU) (None, 256, 34, 60) 0 block_3b_bn_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3b_relu_1_qdq (QDQ) (None, 256, 34, 60) 1 block_3b_relu_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3b_conv_2 (QuantizedConv2 (None, 256, 34, 60) 590080 block_3b_relu_1_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3b_conv_shortcut (Quantiz (None, 256, 34, 60) 65792 block_3a_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3b_bn_2 (BatchNormalizati (None, 256, 34, 60) 1024 block_3b_conv_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3b_bn_shortcut (BatchNorm (None, 256, 34, 60) 1024 block_3b_conv_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3b_bn_2_qdq (QDQ) (None, 256, 34, 60) 1 block_3b_bn_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3b_bn_shortcut_qdq (QDQ) (None, 256, 34, 60) 1 block_3b_bn_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_6 (Add) (None, 256, 34, 60) 0 block_3b_bn_2_qdq[0][0] \n",
" block_3b_bn_shortcut_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_6_qdq (QDQ) (None, 256, 34, 60) 1 add_6[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3b_relu (ReLU) (None, 256, 34, 60) 0 add_6_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_3b_relu_qdq (QDQ) (None, 256, 34, 60) 1 block_3b_relu[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4a_conv_1 (QuantizedConv2 (None, 512, 34, 60) 1180160 block_3b_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4a_bn_1 (BatchNormalizati (None, 512, 34, 60) 2048 block_4a_conv_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4a_relu_1 (ReLU) (None, 512, 34, 60) 0 block_4a_bn_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4a_relu_1_qdq (QDQ) (None, 512, 34, 60) 1 block_4a_relu_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4a_conv_2 (QuantizedConv2 (None, 512, 34, 60) 2359808 block_4a_relu_1_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4a_conv_shortcut (Quantiz (None, 512, 34, 60) 131584 block_3b_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4a_bn_2 (BatchNormalizati (None, 512, 34, 60) 2048 block_4a_conv_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4a_bn_shortcut (BatchNorm (None, 512, 34, 60) 2048 block_4a_conv_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4a_bn_2_qdq (QDQ) (None, 512, 34, 60) 1 block_4a_bn_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4a_bn_shortcut_qdq (QDQ) (None, 512, 34, 60) 1 block_4a_bn_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_7 (Add) (None, 512, 34, 60) 0 block_4a_bn_2_qdq[0][0] \n",
" block_4a_bn_shortcut_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_7_qdq (QDQ) (None, 512, 34, 60) 1 add_7[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4a_relu (ReLU) (None, 512, 34, 60) 0 add_7_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4a_relu_qdq (QDQ) (None, 512, 34, 60) 1 block_4a_relu[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4b_conv_1 (QuantizedConv2 (None, 512, 34, 60) 2359808 block_4a_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4b_bn_1 (BatchNormalizati (None, 512, 34, 60) 2048 block_4b_conv_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4b_relu_1 (ReLU) (None, 512, 34, 60) 0 block_4b_bn_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4b_relu_1_qdq (QDQ) (None, 512, 34, 60) 1 block_4b_relu_1[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4b_conv_2 (QuantizedConv2 (None, 512, 34, 60) 2359808 block_4b_relu_1_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4b_conv_shortcut (Quantiz (None, 512, 34, 60) 262656 block_4a_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4b_bn_2 (BatchNormalizati (None, 512, 34, 60) 2048 block_4b_conv_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4b_bn_shortcut (BatchNorm (None, 512, 34, 60) 2048 block_4b_conv_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4b_bn_2_qdq (QDQ) (None, 512, 34, 60) 1 block_4b_bn_2[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4b_bn_shortcut_qdq (QDQ) (None, 512, 34, 60) 1 block_4b_bn_shortcut[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_8 (Add) (None, 512, 34, 60) 0 block_4b_bn_2_qdq[0][0] \n",
" block_4b_bn_shortcut_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"add_8_qdq (QDQ) (None, 512, 34, 60) 1 add_8[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4b_relu (ReLU) (None, 512, 34, 60) 0 add_8_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"block_4b_relu_qdq (QDQ) (None, 512, 34, 60) 1 block_4b_relu[0][0] \n",
"__________________________________________________________________________________________________\n",
"output_bbox (Conv2D) (None, 4, 34, 60) 2052 block_4b_relu_qdq[0][0] \n",
"__________________________________________________________________________________________________\n",
"output_cov (Conv2D) (None, 1, 34, 60) 513 block_4b_relu_qdq[0][0] \n",
"==================================================================================================\n",
"Total params: 11,550,895\n",
"Trainable params: 11,539,205\n",
"Non-trainable params: 11,690\n",
"__________________________________________________________________________________________________\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/utilities.py:139: The name tf.train.Scaffold is deprecated. Please use tf.compat.v1.train.Scaffold instead.\n",
"\n",
"2022-01-06 08:20:09,239 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/utilities.py:139: The name tf.train.Scaffold is deprecated. Please use tf.compat.v1.train.Scaffold instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:14: The name tf.local_variables_initializer is deprecated. Please use tf.compat.v1.local_variables_initializer instead.\n",
"\n",
"2022-01-06 08:20:09,240 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:14: The name tf.local_variables_initializer is deprecated. Please use tf.compat.v1.local_variables_initializer instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:15: The name tf.tables_initializer is deprecated. Please use tf.compat.v1.tables_initializer instead.\n",
"\n",
"2022-01-06 08:20:09,240 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:15: The name tf.tables_initializer is deprecated. Please use tf.compat.v1.tables_initializer instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:16: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead.\n",
"\n",
"2022-01-06 08:20:09,240 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:16: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead.\n",
"\n",
"WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/utilities.py:140: The name tf.train.SingularMonitoredSession is deprecated. Please use tf.compat.v1.train.SingularMonitoredSession instead.\n",
"\n",
"2022-01-06 08:20:09,241 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/utilities.py:140: The name tf.train.SingularMonitoredSession is deprecated. Please use tf.compat.v1.train.SingularMonitoredSession instead.\n",
"\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO:tensorflow:Graph was finalized.\n",
"2022-01-06 08:20:09,661 [INFO] tensorflow: Graph was finalized.\n",
"INFO:tensorflow:Running local_init_op.\n",
"2022-01-06 08:20:10,289 [INFO] tensorflow: Running local_init_op.\n",
"INFO:tensorflow:Done running local_init_op.\n",
"2022-01-06 08:20:10,529 [INFO] tensorflow: Done running local_init_op.\n",
"2022-01-06 08:20:11,113 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 24, 0.00s/step\n",
"2022-01-06 08:20:17,140 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 24, 0.60s/step\n",
"2022-01-06 08:20:18,702 [INFO] iva.detectnet_v2.evaluation.evaluation: step 20 / 24, 0.16s/step\n",
"Matching predictions to ground truth, class 1/1.: 100%|█| 1015/1015 [00:00<00:00, 15749.92it/s]\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:95: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead.\n",
"\n",
"2022-01-06 08:20:19,413 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:95: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead.\n",
"\n",
"WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:98: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead.\n",
"\n",
"2022-01-06 08:20:19,414 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:98: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead.\n",
"\n",
"\n",
"Validation cost: 0.001126\n",
"Mean average_precision (in %): 92.8748\n",
"\n",
"class name average precision (in %)\n",
"------------ --------------------------\n",
"car 92.8748\n",
"\n",
"Median Inference Time: 0.016298\n",
"2022-01-06 08:20:19,454 [INFO] __main__: Evaluation complete.\n",
"Time taken to run __main__:main: 0:00:13.413126.\n",
"2022-01-06 16:20:20,676 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.\n"
]
}
],
"source": [
"!tao detectnet_v2 evaluate -e $SPECS_DIR/detectnet_v2_retrain_resnet18_kitti_qat.txt \\\n",
" -m $USER_EXPERIMENT_DIR/experiment_dir_retrain_qat/weights/resnet18_detector_pruned_qat.tlt \\\n",
" -k $KEY \\\n",
" -f tlt"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### C. Export QAT trained model to int8 \n",
"Export a QAT trained model to TensorRT parsable model. This command generates an .etlt file from the trained model and the serializes corresponding int8 scales as a TRT readable calibration cache file."
]
},
{
"cell_type": "code",
"execution_count": 39,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"2022-01-06 16:20:39,163 [INFO] root: Registry: ['nvcr.io']\n",
"Matplotlib created a temporary config/cache directory at /tmp/matplotlib-iliiqnbj because the default path (/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.\n",
"Using TensorFlow backend.\n",
"Using TensorFlow backend.\n",
"WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.\n",
"2022-01-06 08:20:46,924 [INFO] iva.common.export.keras_exporter: Using input nodes: ['input_1']\n",
"2022-01-06 08:20:46,924 [INFO] iva.common.export.keras_exporter: Using output nodes: ['output_cov/Sigmoid', 'output_bbox/BiasAdd']\n",
"/usr/local/lib/python3.6/dist-packages/keras/engine/saving.py:292: UserWarning: No training configuration found in save file: the model was *not* compiled. Compile it manually.\n",
" warnings.warn('No training configuration found in save file: '\n",
"2022-01-06 08:21:02,948 [DEBUG] iva.common.export.keras_exporter: Saving etlt model file at: /workspace/tao-experiments/experiment/experiment_dir_final/resnet18_detector_qat.etlt.\n",
"2022-01-06 08:21:03,134 [DEBUG] modulus.export._uff: Patching keras BatchNormalization...\n",
"2022-01-06 08:21:03,134 [DEBUG] modulus.export._uff: Patching keras Dropout...\n",
"2022-01-06 08:21:03,134 [DEBUG] modulus.export._uff: Patching UFF TensorFlow converter apply_fused_padding...\n",
"2022-01-06 08:21:04,207 [DEBUG] modulus.export._uff: Unpatching keras BatchNormalization layer...\n",
"2022-01-06 08:21:04,207 [DEBUG] modulus.export._uff: Unpatching keras Dropout layer...\n",
"NOTE: UFF has been tested with TensorFlow 1.14.0.\n",
"WARNING: The version of TensorFlow installed on this system is not guaranteed to work with UFF.\n",
"DEBUG [/usr/local/lib/python3.6/dist-packages/uff/converters/tensorflow/converter.py:96] Marking ['output_cov/Sigmoid', 'output_bbox/BiasAdd'] as outputs\n",
"2022-01-06 08:21:06,861 [DEBUG] iva.common.export.base_exporter: Data file doesn't exist. Pulling input dimensions from the network.\n",
"2022-01-06 08:21:06,861 [DEBUG] iva.common.export.keras_exporter: Input dims: (3, 544, 960)\n",
"2022-01-06 16:22:47,081 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.\n"
]
}
],
"source": [
"!rm -rf $LOCAL_EXPERIMENT_DIR/experiment_dir_final/resnet18_detector_qat.etlt\n",
"!rm -rf $LOCAL_EXPERIMENT_DIR/experiment_dir_final/calibration_qat.bin\n",
"!tao detectnet_v2 export \\\n",
" -m $USER_EXPERIMENT_DIR/experiment_dir_retrain_qat/weights/resnet18_detector_pruned_qat.tlt \\\n",
" -o $USER_EXPERIMENT_DIR/experiment_dir_final/resnet18_detector_qat.etlt \\\n",
" -k $KEY \\\n",
" --data_type int8 \\\n",
" --batch_size 8 \\\n",
" --max_batch_size 16 \\\n",
" --engine_file $USER_EXPERIMENT_DIR/experiment_dir_final/resnet18_detector_qat.trt.int8 \\\n",
" --cal_cache_file $USER_EXPERIMENT_DIR/experiment_dir_final/calibration_qat.bin \\\n",
" --verbose"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### D. Evaluate a QAT trained model using the exported TensorRT engine \n",
"This section evaluates a QAT enabled pruned retrained model using the TensorRT int8 engine that was exported in [Section C](#head-12-3). Please note that there maybe a slight difference (~0.1-0.5%) in the mAP from [Section B](#head-12-2), oweing to some differences in the implementation of quantization in TensorRT.\n",
"\n",
"*Note: The TensorRT evaluator might be slightly slower than the TAO evaluator here, because the evaluation dataloader is pinned to the CPU to avoid any clashes between TensorRT and TAO instances in the GPU. Please note that this tool was not intended and has not been developed for profiling the model. It is just a means to qualitatively analyse the model.*\n",
"\n",
"*Please use native TensorRT or DeepStream for the most optimized inferences.*"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!tao detectnet_v2 evaluate -e $SPECS_DIR/detectnet_v2_retrain_resnet18_kitti_qat.txt \\\n",
" -m $USER_EXPERIMENT_DIR/experiment_dir_final/resnet18_detector_qat.trt.int8 \\\n",
" -f tensorrt"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### E. Inference using QAT engine \n",
"Run inference and visualize detections on test images, using the exported TensorRT engine from [Section C](#head-12-3)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!tao detectnet_v2 inference -e $SPECS_DIR/detectnet_v2_inference_kitti_etlt_qat.txt \\\n",
" -o $USER_EXPERIMENT_DIR/tlt_infer_testing_qat \\\n",
" -i $DATA_DOWNLOAD_DIR/testing/image_2 \\\n",
" -k $KEY"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# visualize the first 12 inferenced images.\n",
"OUTPUT_PATH = 'tlt_infer_testing_qat/images_annotated' # relative path from $USER_EXPERIMENT_DIR.\n",
"COLS = 4 # number of columns in the visualizer grid.\n",
"IMAGES = 12 # number of images to visualize.\n",
"\n",
"visualize_images(OUTPUT_PATH, num_cols=COLS, num_images=IMAGES)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.10"
}
},
"nbformat": 4,
"nbformat_minor": 2
}