The Average Precision of the model is showing 0 from the begining. I have followed all the steps described in TLT-Detectnet_v2 example.
Here I am sharing the log
Using TensorFlow backend.
2020-11-09 15:24:32.241622: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-11-09 15:24:36.075907: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-11-09 15:24:36.102671: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-09 15:24:36.103073: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: GeForce GTX 1660 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.59
pciBusID: 0000:01:00.0
2020-11-09 15:24:36.103101: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-11-09 15:24:36.103158: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-11-09 15:24:36.104116: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-11-09 15:24:36.104391: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-11-09 15:24:36.105614: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-11-09 15:24:36.106537: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-11-09 15:24:36.106592: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-11-09 15:24:36.106689: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-09 15:24:36.107152: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-09 15:24:36.107525: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2020-11-09 15:24:36.107554: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-11-09 15:24:36.699424: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-11-09 15:24:36.699460: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0
2020-11-09 15:24:36.699467: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N
2020-11-09 15:24:36.699637: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-09 15:24:36.700048: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-09 15:24:36.700407: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-09 15:24:36.700728: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4485 MB memory) → physical GPU (device: 0, name: GeForce GTX 1660 Ti, pci bus id: 0000:01:00.0, compute capability: 7.5)
2020-11-09 15:24:36,701 [INFO] iva.detectnet_v2.scripts.train: Loading experiment spec at /workspace/examples/detectnet_v2/specs/detectnet_v2_train_resnet18_kitti.txt.
2020-11-09 15:24:36,702 [INFO] iva.detectnet_v2.spec_handler.spec_loader: Merging specification from /workspace/examples/detectnet_v2/specs/detectnet_v2_train_resnet18_kitti.txt
2020-11-09 15:24:45,003 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Serial augmentation enabled = False
2020-11-09 15:24:45,003 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Pseudo sharding enabled = False
2020-11-09 15:24:45,004 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Max Image Dimensions (all sources): (0, 0)
2020-11-09 15:24:45,004 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: number of cpus: 12, io threads: 24, compute threads: 12, buffered batches: 4
2020-11-09 15:24:45,004 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: total dataset size 460, number of sources: 1, batch size per gpu: 4, steps: 115
2020-11-09 15:24:45,102 [INFO] iva.detectnet_v2.dataloader.default_dataloader: Bounding box coordinates were detected in the input specification! Bboxes will be automatically converted to polygon coordinates.
2020-11-09 15:24:45.132292: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-09 15:24:45.132626: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: GeForce GTX 1660 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.59
pciBusID: 0000:01:00.0
2020-11-09 15:24:45.132651: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-11-09 15:24:45.132681: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-11-09 15:24:45.132705: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-11-09 15:24:45.132723: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-11-09 15:24:45.132742: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-11-09 15:24:45.132761: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-11-09 15:24:45.132777: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-11-09 15:24:45.132837: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-09 15:24:45.133171: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-09 15:24:45.133458: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2020-11-09 15:24:45,346 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: shuffle: True - shard 0 of 1
2020-11-09 15:24:45,352 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: sampling 1 datasets with weights:
2020-11-09 15:24:45,352 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: source: 0 weight: 1.000000
2020-11-09 15:24:45,834 [INFO] iva.detectnet_v2.scripts.train: Found 460 samples in training set
2020-11-09 15:24:47,942 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Serial augmentation enabled = False
2020-11-09 15:24:47,942 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Pseudo sharding enabled = False
2020-11-09 15:24:47,942 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Max Image Dimensions (all sources): (0, 0)
2020-11-09 15:24:47,942 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: number of cpus: 12, io threads: 24, compute threads: 12, buffered batches: 4
2020-11-09 15:24:47,943 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: total dataset size 74, number of sources: 1, batch size per gpu: 4, steps: 19
2020-11-09 15:24:47,971 [INFO] iva.detectnet_v2.dataloader.default_dataloader: Bounding box coordinates were detected in the input specification! Bboxes will be automatically converted to polygon coordinates.
2020-11-09 15:24:48,308 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: shuffle: False - shard 0 of 1
2020-11-09 15:24:48,313 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: sampling 1 datasets with weights:
2020-11-09 15:24:48,314 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: source: 0 weight: 1.000000
2020-11-09 15:24:48,647 [INFO] iva.detectnet_v2.scripts.train: Found 74 samples in validation set
2020-11-09 15:24:52.826196: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-09 15:24:52.826538: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: GeForce GTX 1660 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.59
pciBusID: 0000:01:00.0
2020-11-09 15:24:52.826570: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-11-09 15:24:52.826606: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-11-09 15:24:52.826632: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-11-09 15:24:52.826651: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-11-09 15:24:52.826671: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-11-09 15:24:52.826689: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-11-09 15:24:52.826706: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-11-09 15:24:52.826772: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-09 15:24:52.827105: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-09 15:24:52.827403: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2020-11-09 15:24:52.828443: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-11-09 15:24:52.828458: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0
2020-11-09 15:24:52.828465: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N
2020-11-09 15:24:52.828552: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-09 15:24:52.828890: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-11-09 15:24:52.829190: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4485 MB memory) → physical GPU (device: 0, name: GeForce GTX 1660 Ti, pci bus id: 0000:01:00.0, compute capability: 7.5)
2020-11-09 15:25:13.784563: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-11-09 15:25:14.000827: I tensorflow/core/kernels/cuda_solvers.cc:159] Creating CudaSolver handles for stream 0x6a4c3d0
2020-11-09 15:25:14.000931: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-11-09 15:25:14.175887: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-11-09 15:25:14.187955: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-11-09 15:25:18,066 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 0/120: loss: 0.08668 Time taken: 0:00:00 ETA: 0:00:00
2020-11-09 15:25:18,066 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 0.705
2020-11-09 15:25:27,167 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 6.768
2020-11-09 15:25:32,383 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.173
2020-11-09 15:25:37,631 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.056
2020-11-09 15:25:43,000 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.627
2020-11-09 15:25:46,575 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 1/120: loss: 0.27268 Time taken: 0:00:33.973992 ETA: 1:07:22.905033
2020-11-09 15:25:48,466 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.295
2020-11-09 15:25:53,725 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.015
2020-11-09 15:25:58,974 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.055
2020-11-09 15:26:04,230 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.023
2020-11-09 15:26:09,487 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.026
2020-11-09 15:26:10,762 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 2/120: loss: 0.01615 Time taken: 0:00:24.180101 ETA: 0:47:33.251880
2020-11-09 15:26:14,763 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.954
2020-11-09 15:26:20,023 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.010
2020-11-09 15:26:25,293 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.978
2020-11-09 15:26:30,564 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.971
2020-11-09 15:26:34,993 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 3/120: loss: 0.00834 Time taken: 0:00:24.234537 ETA: 0:47:15.440871
2020-11-09 15:26:35,842 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.947
2020-11-09 15:26:41,138 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.883
2020-11-09 15:26:46,414 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.955
2020-11-09 15:26:51,700 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.920
2020-11-09 15:26:56,987 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.914
2020-11-09 15:26:59,327 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 4/120: loss: 0.00326 Time taken: 0:00:24.328542 ETA: 0:47:02.110927
2020-11-09 15:27:02,296 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.836
2020-11-09 15:27:07,590 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.889
2020-11-09 15:27:12,892 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.862
2020-11-09 15:27:18,193 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.866
2020-11-09 15:27:23,505 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.825
2020-11-09 15:27:23,721 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 5/120: loss: 0.00186 Time taken: 0:00:24.393186 ETA: 0:46:45.216373
2020-11-09 15:27:28,815 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.834
2020-11-09 15:27:34,127 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.829
2020-11-09 15:27:39,437 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.831
2020-11-09 15:27:44,749 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.828
2020-11-09 15:27:48,151 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 6/120: loss: 0.00123 Time taken: 0:00:24.432409 ETA: 0:46:25.294604
2020-11-09 15:27:50,067 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.803
2020-11-09 15:27:55,378 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.829
2020-11-09 15:28:00,697 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.803
2020-11-09 15:28:06,011 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.819
2020-11-09 15:28:11,326 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.815
2020-11-09 15:28:12,619 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 7/120: loss: 0.00108 Time taken: 0:00:24.462360 ETA: 0:46:04.246642
2020-11-09 15:28:16,660 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.748
2020-11-09 15:28:21,983 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.786
2020-11-09 15:28:27,318 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.744
2020-11-09 15:28:32,643 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.780
2020-11-09 15:28:37,118 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 8/120: loss: 0.00081 Time taken: 0:00:24.501054 ETA: 0:45:44.118053
2020-11-09 15:28:37,981 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.737
2020-11-09 15:28:43,322 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.722
2020-11-09 15:28:48,651 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.767
2020-11-09 15:28:53,975 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.782
2020-11-09 15:28:59,309 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.749
2020-11-09 15:29:01,660 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 9/120: loss: 0.00073 Time taken: 0:00:24.543428 ETA: 0:45:24.320475
2020-11-09 15:29:04,642 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.751
2020-11-09 15:29:09,982 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.729
2020-11-09 15:29:15,320 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.732
2020-11-09 15:29:20,656 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.742
2020-11-09 15:29:28,927 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 12.091
2020-11-09 15:29:29,160 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 10/120: loss: 0.00069 Time taken: 0:00:27.479451 ETA: 0:50:22.739577
2020-11-09 15:29:34,272 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.712
2020-11-09 15:29:39,592 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.797
2020-11-09 15:29:44,923 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.760
2020-11-09 15:29:50,255 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.754
2020-11-09 15:29:53,685 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 11/120: loss: 0.00066 Time taken: 0:00:24.537799 ETA: 0:44:34.620052
2020-11-09 15:29:55,610 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.677
2020-11-09 15:30:00,945 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.743
2020-11-09 15:30:06,278 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.751
2020-11-09 15:30:11,612 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.747
2020-11-09 15:30:16,943 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.760
2020-11-09 15:30:18,230 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 12/120: loss: 0.00065 Time taken: 0:00:24.548827 ETA: 0:44:11.273283
2020-11-09 15:30:22,284 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.723
2020-11-09 15:30:27,621 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.739
2020-11-09 15:30:32,956 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.747
2020-11-09 15:30:38,302 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.705
2020-11-09 15:30:42,812 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 13/120: loss: 0.00063 Time taken: 0:00:24.583797 ETA: 0:43:50.466226
2020-11-09 15:30:43,669 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.632
2020-11-09 15:30:49,155 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.230
2020-11-09 15:30:54,556 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.516
2020-11-09 15:30:59,987 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.413
2020-11-09 15:31:05,327 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.727
2020-11-09 15:31:07,686 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 14/120: loss: 0.00061 Time taken: 0:00:24.865604 ETA: 0:43:55.753991
2020-11-09 15:31:10,674 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.701
2020-11-09 15:31:16,012 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.734
2020-11-09 15:31:21,346 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.751
2020-11-09 15:31:26,681 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.744
2020-11-09 15:31:32,020 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.731
2020-11-09 15:31:32,236 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 15/120: loss: 0.00061 Time taken: 0:00:24.552267 ETA: 0:42:57.988068
2020-11-09 15:31:37,360 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.727
2020-11-09 15:31:42,695 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.746
2020-11-09 15:31:48,032 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.737
2020-11-09 15:31:53,374 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.722
2020-11-09 15:31:56,905 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 16/120: loss: 0.00063 Time taken: 0:00:24.663935 ETA: 0:42:45.049284
2020-11-09 15:31:58,842 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.288
2020-11-09 15:32:04,457 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 17.810
2020-11-09 15:32:09,837 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.586
2020-11-09 15:32:15,180 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.720
2020-11-09 15:32:20,523 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.717
2020-11-09 15:32:21,819 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 17/120: loss: 0.00061 Time taken: 0:00:24.913709 ETA: 0:42:46.111995
2020-11-09 15:32:25,882 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.660
2020-11-09 15:32:31,226 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.713
2020-11-09 15:32:36,571 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.710
2020-11-09 15:32:41,935 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.643
2020-11-09 15:32:46,429 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 18/120: loss: 0.00060 Time taken: 0:00:24.608681 ETA: 0:41:50.085483
2020-11-09 15:32:47,287 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.686
2020-11-09 15:32:52,637 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.692
2020-11-09 15:32:57,984 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.703
2020-11-09 15:33:03,330 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.707
2020-11-09 15:33:08,679 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.695
2020-11-09 15:33:11,040 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 19/120: loss: 0.00060 Time taken: 0:00:24.615119 ETA: 0:41:26.126969
2020-11-09 15:33:14,035 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.670
2020-11-09 15:33:19,382 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.705
2020-11-09 15:33:24,730 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.700
2020-11-09 15:33:30,077 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.703
2020-11-09 15:33:38,240 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 12.249
2020-11-09 15:33:38,471 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 20/120: loss: 0.00060 Time taken: 0:00:27.414317 ETA: 0:45:41.431713
2020-11-09 15:33:43,602 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.653
2020-11-09 15:33:48,937 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.747
2020-11-09 15:33:54,280 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.714
2020-11-09 15:33:59,624 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.716
2020-11-09 15:34:03,053 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 21/120: loss: 0.00060 Time taken: 0:00:24.597913 ETA: 0:40:35.193342
2020-11-09 15:34:04,981 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.665
2020-11-09 15:34:10,327 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.709
2020-11-09 15:34:15,673 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.706
2020-11-09 15:34:21,025 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.685
2020-11-09 15:34:26,371 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.706
2020-11-09 15:34:27,660 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 22/120: loss: 0.00060 Time taken: 0:00:24.605171 ETA: 0:40:11.306778
2020-11-09 15:34:31,720 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.697
2020-11-09 15:34:37,065 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.709
2020-11-09 15:34:42,427 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.650
2020-11-09 15:34:47,774 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.703
2020-11-09 15:34:52,277 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 23/120: loss: 0.00060 Time taken: 0:00:24.610870 ETA: 0:39:47.254356
2020-11-09 15:34:53,137 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.647
2020-11-09 15:34:58,487 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.694
2020-11-09 15:35:03,837 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.693
2020-11-09 15:35:09,182 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.708
2020-11-09 15:35:14,536 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.680
2020-11-09 15:35:16,902 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 24/120: loss: 0.00060 Time taken: 0:00:24.625373 ETA: 0:39:24.035774
2020-11-09 15:35:19,906 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.623
2020-11-09 15:35:25,252 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.705
2020-11-09 15:35:30,598 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.708
2020-11-09 15:35:35,944 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.705
2020-11-09 15:35:41,293 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.695
2020-11-09 15:35:41,511 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 25/120: loss: 0.00059 Time taken: 0:00:24.609301 ETA: 0:38:57.883604
2020-11-09 15:35:46,647 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.680
2020-11-09 15:35:51,999 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.688
2020-11-09 15:35:57,350 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.689
2020-11-09 15:36:02,700 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.691
2020-11-09 15:36:06,136 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 26/120: loss: 0.00059 Time taken: 0:00:24.627062 ETA: 0:38:34.943858
2020-11-09 15:36:08,066 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.637
2020-11-09 15:36:13,419 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.681
2020-11-09 15:36:18,771 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.688
2020-11-09 15:36:24,120 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.694
2020-11-09 15:36:29,476 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.674
2020-11-09 15:36:30,766 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 27/120: loss: 0.00059 Time taken: 0:00:24.630316 ETA: 0:38:10.619390
2020-11-09 15:36:34,837 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.654
2020-11-09 15:36:40,201 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.644
2020-11-09 15:36:45,547 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.705
2020-11-09 15:36:50,898 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.688
2020-11-09 15:36:55,397 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 28/120: loss: 0.00059 Time taken: 0:00:24.630022 ETA: 0:37:45.962007
2020-11-09 15:36:56,257 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.662
2020-11-09 15:37:01,608 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.688
2020-11-09 15:37:07,021 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.476
2020-11-09 15:37:12,501 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.248
2020-11-09 15:37:17,960 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.319
2020-11-09 15:37:20,314 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 29/120: loss: 0.00059 Time taken: 0:00:24.917419 ETA: 0:37:47.485125
2020-11-09 15:37:23,312 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.686
2020-11-09 15:37:28,659 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.702
2020-11-09 15:37:34,007 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.700
2020-11-09 15:37:39,362 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.676
2020-11-09 15:37:47,526 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 18, 0.00s/step
2020-11-09 15:37:48,825 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 18, 0.13s/step
Epoch 30/120Validation cost: -0.000006
Mean average_precision (in %): 0.0000class name average precision (in %)
Barcode 0
Median Inference Time: 0.018677
2020-11-09 15:37:49,437 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 9.926
2020-11-09 15:37:49,655 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 30/120: loss: 0.00059 Time taken: 0:00:29.335681 ETA: 0:44:00.211287
2020-11-09 15:37:54,782 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.710
2020-11-09 15:38:00,128 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.705
2020-11-09 15:38:05,473 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.712
2020-11-09 15:38:10,815 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.720
2020-11-09 15:38:14,247 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 31/120: loss: 0.00059 Time taken: 0:00:24.591709 ETA: 0:36:28.662134
2020-11-09 15:38:16,173 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.665
2020-11-09 15:38:21,525 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.684
2020-11-09 15:38:26,988 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.306
2020-11-09 15:38:32,359 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.617
2020-11-09 15:38:37,877 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.126
2020-11-09 15:38:39,242 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 32/120: loss: 0.00059 Time taken: 0:00:24.994376 ETA: 0:36:39.505083
2020-11-09 15:38:43,571 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 17.564
2020-11-09 15:38:49,111 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.049
2020-11-09 15:38:54,797 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 17.589
2020-11-09 15:39:00,347 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.017
2020-11-09 15:39:04,859 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 33/120: loss: 0.00059 Time taken: 0:00:25.617651 ETA: 0:37:08.735615
2020-11-09 15:39:05,717 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.624
2020-11-09 15:39:11,075 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.666
2020-11-09 15:39:16,432 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.666
2020-11-09 15:39:21,787 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.674
2020-11-09 15:39:27,141 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.677
2020-11-09 15:39:29,507 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 34/120: loss: 0.00058 Time taken: 0:00:24.649906 ETA: 0:35:19.891889
2020-11-09 15:39:32,510 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.626
2020-11-09 15:39:37,873 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.649
2020-11-09 15:39:43,232 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.660
2020-11-09 15:39:48,589 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.670
2020-11-09 15:39:53,946 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.667
2020-11-09 15:39:54,167 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 35/120: loss: 0.00058 Time taken: 0:00:24.651760 ETA: 0:34:55.399588
2020-11-09 15:39:59,315 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.628
2020-11-09 15:40:04,673 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.664
2020-11-09 15:40:10,030 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.667
2020-11-09 15:40:15,387 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.668
2020-11-09 15:40:18,835 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 36/120: loss: 0.00058 Time taken: 0:00:24.666444 ETA: 0:34:31.981261
2020-11-09 15:40:20,763 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.602
2020-11-09 15:40:26,120 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.669
2020-11-09 15:40:31,477 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.666
2020-11-09 15:40:36,836 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.661
2020-11-09 15:40:42,211 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.607
2020-11-09 15:40:43,503 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 37/120: loss: 0.00058 Time taken: 0:00:24.674596 ETA: 0:34:07.991454
2020-11-09 15:40:47,573 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.649
2020-11-09 15:40:52,933 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.658
2020-11-09 15:40:58,293 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.659
2020-11-09 15:41:03,654 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.654
2020-11-09 15:41:08,160 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 38/120: loss: 0.00058 Time taken: 0:00:24.654824 ETA: 0:33:41.695589
2020-11-09 15:41:09,021 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.634
2020-11-09 15:41:14,378 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.667
2020-11-09 15:41:19,736 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.662
2020-11-09 15:41:25,102 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.637
2020-11-09 15:41:30,462 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.659
2020-11-09 15:41:32,834 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 39/120: loss: 0.00057 Time taken: 0:00:24.669656 ETA: 0:33:18.242158
2020-11-09 15:41:35,838 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.600
2020-11-09 15:41:41,199 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.653
2020-11-09 15:41:46,558 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.664
2020-11-09 15:41:51,914 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.672
2020-11-09 15:42:00,153 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 18, 0.00s/step
2020-11-09 15:42:00,924 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 18, 0.08s/step
Epoch 40/120Validation cost: -0.000008
Mean average_precision (in %): 0.0000class name average precision (in %)
Barcode 0
Median Inference Time: 0.018767
2020-11-09 15:42:01,538 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 10.391
2020-11-09 15:42:01,756 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 40/120: loss: 0.00057 Time taken: 0:00:28.921932 ETA: 0:38:33.754520
2020-11-09 15:42:06,909 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.620
2020-11-09 15:42:12,265 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.672
2020-11-09 15:42:17,620 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.675
2020-11-09 15:42:22,973 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.679
2020-11-09 15:42:26,403 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 41/120: loss: 0.00057 Time taken: 0:00:24.650249 ETA: 0:32:27.369653
2020-11-09 15:42:28,333 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.657
2020-11-09 15:42:33,687 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.680
2020-11-09 15:42:39,049 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.650
2020-11-09 15:42:44,410 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.654
2020-11-09 15:42:49,764 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.678
2020-11-09 15:42:51,055 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 42/120: loss: 0.00057 Time taken: 0:00:24.652049 ETA: 0:32:02.859808
2020-11-09 15:42:55,126 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.651
2020-11-09 15:43:00,483 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.666
2020-11-09 15:43:05,833 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.695
2020-11-09 15:43:11,190 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.669
2020-11-09 15:43:15,690 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 43/120: loss: 0.00057 Time taken: 0:00:24.634120 ETA: 0:31:36.827257
2020-11-09 15:43:16,550 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.655
2020-11-09 15:43:21,929 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.591
2020-11-09 15:43:27,624 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 17.562
2020-11-09 15:43:33,154 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.082
2020-11-09 15:43:38,583 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.420
2020-11-09 15:43:40,956 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 44/120: loss: 0.00056 Time taken: 0:00:25.262087 ETA: 0:31:59.918602
2020-11-09 15:43:43,964 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.585
2020-11-09 15:43:49,329 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.641
2020-11-09 15:43:54,679 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.693
2020-11-09 15:44:00,035 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.671
2020-11-09 15:44:05,391 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.669
2020-11-09 15:44:05,610 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 45/120: loss: 0.00056 Time taken: 0:00:24.652169 ETA: 0:30:48.912710
2020-11-09 15:44:10,748 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.672
2020-11-09 15:44:16,113 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.637
2020-11-09 15:44:21,472 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.661
2020-11-09 15:44:26,829 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.669
2020-11-09 15:44:30,260 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 46/120: loss: 0.00056 Time taken: 0:00:24.654373 ETA: 0:30:24.423615
2020-11-09 15:44:32,190 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.656
2020-11-09 15:44:37,548 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.665
2020-11-09 15:44:42,925 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.598
2020-11-09 15:44:48,291 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.635
2020-11-09 15:44:53,651 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.657
2020-11-09 15:44:54,952 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 47/120: loss: 0.00056 Time taken: 0:00:24.686080 ETA: 0:30:02.083838
2020-11-09 15:44:59,030 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.593
2020-11-09 15:45:04,394 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.643
2020-11-09 15:45:09,748 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.678
2020-11-09 15:45:15,108 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.657
2020-11-09 15:45:19,618 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 48/120: loss: 0.00055 Time taken: 0:00:24.669990 ETA: 0:29:36.239250
2020-11-09 15:45:20,479 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.620
2020-11-09 15:45:25,837 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.663
2020-11-09 15:45:31,193 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.673
2020-11-09 15:45:36,551 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.663
2020-11-09 15:45:41,911 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.659
2020-11-09 15:45:44,276 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 49/120: loss: 0.00055 Time taken: 0:00:24.657773 ETA: 0:29:10.701918
2020-11-09 15:45:47,281 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.622
2020-11-09 15:45:52,643 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.651
2020-11-09 15:45:58,001 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.665
2020-11-09 15:46:03,360 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.659
2020-11-09 15:46:11,539 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 18, 0.00s/step
2020-11-09 15:46:12,306 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 18, 0.08s/step
Epoch 50/120Validation cost: -0.000009
Mean average_precision (in %): 0.0000class name average precision (in %)
Barcode 0
Median Inference Time: 0.018706
2020-11-09 15:46:12,911 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 10.471
2020-11-09 15:46:13,130 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 50/120: loss: 0.00055 Time taken: 0:00:28.847298 ETA: 0:33:39.310837
2020-11-09 15:46:18,267 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.672
2020-11-09 15:46:23,619 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.686
2020-11-09 15:46:28,970 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.689
2020-11-09 15:46:34,323 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.680
2020-11-09 15:46:37,763 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 51/120: loss: 0.00055 Time taken: 0:00:24.633155 ETA: 0:28:19.687670
2020-11-09 15:46:39,703 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.588
2020-11-09 15:46:45,057 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.679
2020-11-09 15:46:50,406 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.696
2020-11-09 15:46:55,762 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.671
2020-11-09 15:47:01,117 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.675
2020-11-09 15:47:02,416 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 52/120: loss: 0.00054 Time taken: 0:00:24.649637 ETA: 0:27:56.175282
2020-11-09 15:47:06,481 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.643
2020-11-09 15:47:11,832 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.691
2020-11-09 15:47:17,186 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.678
2020-11-09 15:47:22,541 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.674
2020-11-09 15:47:27,046 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 53/120: loss: 0.00054 Time taken: 0:00:24.635251 ETA: 0:27:30.561820
2020-11-09 15:47:27,906 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.641
2020-11-09 15:47:33,260 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.677
2020-11-09 15:47:38,615 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.674
2020-11-09 15:47:43,972 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.670
2020-11-09 15:47:49,330 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.664
2020-11-09 15:47:51,699 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 54/120: loss: 0.00054 Time taken: 0:00:24.647254 ETA: 0:27:06.718732
2020-11-09 15:47:54,699 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.627
2020-11-09 15:48:00,050 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.686
2020-11-09 15:48:05,407 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.669
2020-11-09 15:48:10,761 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.678
2020-11-09 15:48:16,121 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.658
2020-11-09 15:48:16,354 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 55/120: loss: 0.00053 Time taken: 0:00:24.640178 ETA: 0:26:41.611568
2020-11-09 15:48:21,498 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.604
2020-11-09 15:48:26,847 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.693
2020-11-09 15:48:32,200 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.684
2020-11-09 15:48:37,553 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.682
2020-11-09 15:48:41,103 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 56/120: loss: 0.00053 Time taken: 0:00:24.759493 ETA: 0:26:24.607544
2020-11-09 15:48:43,088 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.067
2020-11-09 15:48:48,516 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.425
2020-11-09 15:48:53,952 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.396
2020-11-09 15:48:59,306 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.677
2020-11-09 15:49:04,655 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.698
2020-11-09 15:49:05,945 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 57/120: loss: 0.00052 Time taken: 0:00:24.849053 ETA: 0:26:05.490363
2020-11-09 15:49:10,011 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.669
2020-11-09 15:49:15,358 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.704
2020-11-09 15:49:20,710 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.686
2020-11-09 15:49:26,060 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.692
2020-11-09 15:49:30,560 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 58/120: loss: 0.00052 Time taken: 0:00:24.613310 ETA: 0:25:26.025241
2020-11-09 15:49:31,418 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.664
2020-11-09 15:49:36,774 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.671
2020-11-09 15:49:42,130 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.674
2020-11-09 15:49:47,486 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.671
2020-11-09 15:49:52,832 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.706
2020-11-09 15:49:55,193 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 59/120: loss: 0.00051 Time taken: 0:00:24.631893 ETA: 0:25:02.545454
2020-11-09 15:49:58,190 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.662
2020-11-09 15:50:03,538 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.701
2020-11-09 15:50:08,886 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.700
2020-11-09 15:50:14,230 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.711
2020-11-09 15:50:22,431 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 18, 0.00s/step
2020-11-09 15:50:23,192 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 18, 0.08s/step
Epoch 60/120Validation cost: -0.000009
Mean average_precision (in %): 0.0000class name average precision (in %)
Barcode 0
Median Inference Time: 0.018649
2020-11-09 15:50:23,797 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 10.453
2020-11-09 15:50:24,016 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 60/120: loss: 0.00051 Time taken: 0:00:28.818508 ETA: 0:28:49.110460
2020-11-09 15:50:29,158 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.657
2020-11-09 15:50:34,505 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.702
2020-11-09 15:50:39,856 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.689
2020-11-09 15:50:45,209 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.684
2020-11-09 15:50:48,637 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 61/120: loss: 0.00051 Time taken: 0:00:24.624450 ETA: 0:24:12.842562
2020-11-09 15:50:50,566 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.668
2020-11-09 15:50:55,915 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.694
2020-11-09 15:51:01,260 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.710
2020-11-09 15:51:06,608 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.700
2020-11-09 15:51:11,961 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.682
2020-11-09 15:51:13,257 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 62/120: loss: 0.00050 Time taken: 0:00:24.611703 ETA: 0:23:47.478783
2020-11-09 15:51:17,322 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.654
2020-11-09 15:51:22,669 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.703
2020-11-09 15:51:28,019 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.692
2020-11-09 15:51:33,375 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.670
2020-11-09 15:51:37,871 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 63/120: loss: 0.00049 Time taken: 0:00:24.621742 ETA: 0:23:23.439322
2020-11-09 15:51:38,731 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.673
2020-11-09 15:51:44,080 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.694
2020-11-09 15:51:49,431 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.689
2020-11-09 15:51:54,779 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.699
2020-11-09 15:52:00,128 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.696
2020-11-09 15:52:02,486 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 64/120: loss: 0.00049 Time taken: 0:00:24.615403 ETA: 0:22:58.462564
2020-11-09 15:52:05,485 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.671
2020-11-09 15:52:10,835 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.689
2020-11-09 15:52:16,184 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.696
2020-11-09 15:52:21,535 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.689
2020-11-09 15:52:26,895 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.658
2020-11-09 15:52:27,111 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 65/120: loss: 0.00048 Time taken: 0:00:24.621719 ETA: 0:22:34.194539
2020-11-09 15:52:32,245 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.696
2020-11-09 15:52:37,594 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.694
2020-11-09 15:52:42,951 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.669
2020-11-09 15:52:48,297 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.707
2020-11-09 15:52:51,726 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 66/120: loss: 0.00048 Time taken: 0:00:24.615527 ETA: 0:22:09.238453
2020-11-09 15:52:53,652 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.674
2020-11-09 15:52:59,001 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.696
2020-11-09 15:53:04,352 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.689
2020-11-09 15:53:09,707 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.673
2020-11-09 15:53:15,054 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.702
2020-11-09 15:53:16,345 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 67/120: loss: 0.00047 Time taken: 0:00:24.618950 ETA: 0:21:44.804369
2020-11-09 15:53:20,410 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.673
2020-11-09 15:53:25,758 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.698
2020-11-09 15:53:31,105 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.703
2020-11-09 15:53:36,453 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.702
2020-11-09 15:53:40,955 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 68/120: loss: 0.00046 Time taken: 0:00:24.604668 ETA: 0:21:19.442731
2020-11-09 15:53:41,814 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.652
2020-11-09 15:53:47,162 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.699
2020-11-09 15:53:52,511 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.697
2020-11-09 15:53:57,865 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.680
2020-11-09 15:54:03,211 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.707
2020-11-09 15:54:05,575 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 69/120: loss: 0.00046 Time taken: 0:00:24.623955 ETA: 0:20:55.821706
2020-11-09 15:54:08,574 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.647
2020-11-09 15:54:13,923 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.695
2020-11-09 15:54:19,279 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.672
2020-11-09 15:54:24,631 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.683
2020-11-09 15:54:32,830 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 18, 0.00s/step
2020-11-09 15:54:33,596 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 18, 0.08s/step
Epoch 70/120Validation cost: -0.000009
Mean average_precision (in %): 0.0000class name average precision (in %)
Barcode 0
Median Inference Time: 0.018721
2020-11-09 15:54:34,201 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 10.450
2020-11-09 15:54:34,420 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 70/120: loss: 0.00045 Time taken: 0:00:28.839478 ETA: 0:24:01.973913
2020-11-09 15:54:39,559 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.666
2020-11-09 15:54:44,905 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.706
2020-11-09 15:54:50,260 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.676
2020-11-09 15:54:55,612 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.686
2020-11-09 15:54:59,048 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 71/120: loss: 0.00044 Time taken: 0:00:24.632549 ETA: 0:20:06.994915
2020-11-09 15:55:00,972 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.655
2020-11-09 15:55:06,318 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.708
2020-11-09 15:55:11,664 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.705
2020-11-09 15:55:17,010 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.709
2020-11-09 15:55:22,358 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.698
2020-11-09 15:55:23,657 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 72/120: loss: 0.00044 Time taken: 0:00:24.604086 ETA: 0:19:40.996147
2020-11-09 15:55:27,723 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.640
2020-11-09 15:55:33,070 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.705
2020-11-09 15:55:38,525 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.332
2020-11-09 15:55:44,074 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.022
2020-11-09 15:55:48,867 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 73/120: loss: 0.00043 Time taken: 0:00:25.198520 ETA: 0:19:44.330426
2020-11-09 15:55:49,731 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 17.676
2020-11-09 15:55:55,301 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 17.956
2020-11-09 15:56:00,976 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 17.621
2020-11-09 15:56:06,619 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 17.722
2020-11-09 15:56:11,979 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.658
2020-11-09 15:56:14,367 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 74/120: loss: 0.00042 Time taken: 0:00:25.493479 ETA: 0:19:32.700035
2020-11-09 15:56:17,412 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.406
2020-11-09 15:56:22,898 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.228
2020-11-09 15:56:28,412 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.138
2020-11-09 15:56:33,991 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 17.925
2020-11-09 15:56:39,583 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 17.884
2020-11-09 15:56:39,804 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 75/120: loss: 0.00041 Time taken: 0:00:25.448140 ETA: 0:19:05.166296
2020-11-09 15:56:45,093 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.149
2020-11-09 15:56:50,569 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.261
2020-11-09 15:56:56,019 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.351
2020-11-09 15:57:01,483 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.302
2020-11-09 15:57:04,966 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 76/120: loss: 0.00040 Time taken: 0:00:25.164729 ETA: 0:18:27.248060
2020-11-09 15:57:06,946 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.304
2020-11-09 15:57:12,315 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.627
2020-11-09 15:57:17,790 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.266
2020-11-09 15:57:23,171 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.585
2020-11-09 15:57:28,531 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.657
2020-11-09 15:57:29,835 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 77/120: loss: 0.00039 Time taken: 0:00:24.862974 ETA: 0:17:49.107899
2020-11-09 15:57:33,906 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.606
2020-11-09 15:57:39,266 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.657
2020-11-09 15:57:44,626 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.657
2020-11-09 15:57:49,992 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.639
2020-11-09 15:57:54,497 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 78/120: loss: 0.00038 Time taken: 0:00:24.670586 ETA: 0:17:16.164597
2020-11-09 15:57:55,361 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.625
2020-11-09 15:58:00,727 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.638
2020-11-09 15:58:06,089 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.651
2020-11-09 15:58:11,445 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.668
2020-11-09 15:58:16,808 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.649
2020-11-09 15:58:19,176 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 79/120: loss: 0.00038 Time taken: 0:00:24.677868 ETA: 0:16:51.792574
2020-11-09 15:58:22,183 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.605
2020-11-09 15:58:27,548 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.641
2020-11-09 15:58:32,909 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.652
2020-11-09 15:58:38,282 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.614
2020-11-09 15:58:46,512 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 18, 0.00s/step
2020-11-09 15:58:47,279 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 18, 0.08s/step
Epoch 80/120Validation cost: -0.000009
Mean average_precision (in %): 0.0000class name average precision (in %)
Barcode 0
Median Inference Time: 0.018731
2020-11-09 15:58:47,887 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 10.412
2020-11-09 15:58:48,108 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 80/120: loss: 0.00037 Time taken: 0:00:28.924486 ETA: 0:19:16.979446
2020-11-09 15:58:53,243 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.672
2020-11-09 15:58:58,605 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.649
2020-11-09 15:59:03,966 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.653
2020-11-09 15:59:09,329 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.647
2020-11-09 15:59:12,769 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 81/120: loss: 0.00036 Time taken: 0:00:24.662795 ETA: 0:16:01.848998
2020-11-09 15:59:14,701 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.617
2020-11-09 15:59:20,059 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.663
2020-11-09 15:59:25,420 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.657
2020-11-09 15:59:30,783 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.645
2020-11-09 15:59:36,142 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.660
2020-11-09 15:59:37,435 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 82/120: loss: 0.00035 Time taken: 0:00:24.668991 ETA: 0:15:37.421661
2020-11-09 15:59:41,510 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.632
2020-11-09 15:59:46,869 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.660
2020-11-09 15:59:52,232 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.647
2020-11-09 15:59:57,596 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.645
2020-11-09 16:00:02,112 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 83/120: loss: 0.00034 Time taken: 0:00:24.671679 ETA: 0:15:12.852141
2020-11-09 16:00:02,975 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.591
2020-11-09 16:00:08,334 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.660
2020-11-09 16:00:13,694 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.657
2020-11-09 16:00:19,054 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.657
2020-11-09 16:00:24,412 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.664
2020-11-09 16:00:26,775 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 84/120: loss: 0.00033 Time taken: 0:00:24.667840 ETA: 0:14:48.042249
2020-11-09 16:00:29,779 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.634
2020-11-09 16:00:35,136 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.667
2020-11-09 16:00:40,518 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.583
2020-11-09 16:00:45,879 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.652
2020-11-09 16:00:51,243 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.646
2020-11-09 16:00:51,466 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 85/120: loss: 0.00032 Time taken: 0:00:24.680628 ETA: 0:14:23.821974
2020-11-09 16:00:56,613 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.624
2020-11-09 16:01:01,972 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.659
2020-11-09 16:01:07,331 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.662
2020-11-09 16:01:12,691 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.659
2020-11-09 16:01:16,130 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 86/120: loss: 0.00031 Time taken: 0:00:24.672608 ETA: 0:13:58.868660
2020-11-09 16:01:18,062 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.619
2020-11-09 16:01:23,423 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.653
2020-11-09 16:01:28,781 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.663
2020-11-09 16:01:34,141 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.660
2020-11-09 16:01:39,504 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.647
2020-11-09 16:01:40,796 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 87/120: loss: 0.00030 Time taken: 0:00:24.665144 ETA: 0:13:33.949759
2020-11-09 16:01:44,872 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.630
2020-11-09 16:01:50,237 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.638
2020-11-09 16:01:55,600 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.648
2020-11-09 16:02:00,961 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.653
2020-11-09 16:02:05,467 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 88/120: loss: 0.00029 Time taken: 0:00:24.669179 ETA: 0:13:09.413719
2020-11-09 16:02:06,330 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.628
2020-11-09 16:02:11,693 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.645
2020-11-09 16:02:17,047 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.681
2020-11-09 16:02:22,444 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.530
2020-11-09 16:02:27,951 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.157
2020-11-09 16:02:30,347 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 89/120: loss: 0.00028 Time taken: 0:00:24.874244 ETA: 0:12:51.101571
2020-11-09 16:02:33,449 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.190
2020-11-09 16:02:38,808 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.659
2020-11-09 16:02:44,167 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.663
2020-11-09 16:02:49,527 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.657
2020-11-09 16:02:57,769 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 18, 0.00s/step
2020-11-09 16:02:58,534 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 18, 0.08s/step
Epoch 90/120Validation cost: -0.000009
Mean average_precision (in %): 0.0000class name average precision (in %)
Barcode 0
Median Inference Time: 0.018663
2020-11-09 16:02:59,137 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 10.405
2020-11-09 16:02:59,355 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 90/120: loss: 0.00028 Time taken: 0:00:29.010012 ETA: 0:14:30.300357
2020-11-09 16:03:04,489 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.687
2020-11-09 16:03:09,839 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.692
2020-11-09 16:03:15,191 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.687
2020-11-09 16:03:20,545 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.678
2020-11-09 16:03:23,977 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 91/120: loss: 0.00027 Time taken: 0:00:24.624875 ETA: 0:11:54.121384
2020-11-09 16:03:25,905 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.658
2020-11-09 16:03:31,260 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.674
2020-11-09 16:03:36,608 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.701
2020-11-09 16:03:41,960 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.683
2020-11-09 16:03:47,318 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.667
2020-11-09 16:03:48,614 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 92/120: loss: 0.00027 Time taken: 0:00:24.635386 ETA: 0:11:29.790808
2020-11-09 16:03:52,685 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.631
2020-11-09 16:03:58,039 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.680
2020-11-09 16:04:03,391 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.683
2020-11-09 16:04:08,742 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.691
2020-11-09 16:04:13,240 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 93/120: loss: 0.00026 Time taken: 0:00:24.626832 ETA: 0:11:04.924464
2020-11-09 16:04:14,099 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.666
2020-11-09 16:04:19,451 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.685
2020-11-09 16:04:24,800 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.696
2020-11-09 16:04:30,153 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.683
2020-11-09 16:04:35,511 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.665
2020-11-09 16:04:37,877 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 94/120: loss: 0.00026 Time taken: 0:00:24.634906 ETA: 0:10:40.507551
2020-11-09 16:04:40,879 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.627
2020-11-09 16:04:46,229 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.692
2020-11-09 16:04:51,579 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.692
2020-11-09 16:04:56,927 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.699
2020-11-09 16:05:02,280 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.684
2020-11-09 16:05:02,497 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 95/120: loss: 0.00026 Time taken: 0:00:24.617615 ETA: 0:10:15.440363
2020-11-09 16:05:07,636 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.674
2020-11-09 16:05:12,991 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.673
2020-11-09 16:05:18,344 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.683
2020-11-09 16:05:23,696 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.685
2020-11-09 16:05:27,134 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 96/120: loss: 0.00025 Time taken: 0:00:24.635570 ETA: 0:09:51.253675
2020-11-09 16:05:29,064 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.630
2020-11-09 16:05:34,413 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.695
2020-11-09 16:05:39,764 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.691
2020-11-09 16:05:45,110 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.703
2020-11-09 16:05:50,459 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.697
2020-11-09 16:05:51,751 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 97/120: loss: 0.00025 Time taken: 0:00:24.619668 ETA: 0:09:26.252375
2020-11-09 16:05:55,819 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.658
2020-11-09 16:06:01,180 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.654
2020-11-09 16:06:06,531 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.690
2020-11-09 16:06:11,881 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.690
2020-11-09 16:06:16,382 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 98/120: loss: 0.00025 Time taken: 0:00:24.630991 ETA: 0:09:01.881791
2020-11-09 16:06:17,242 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.654
2020-11-09 16:06:22,601 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.664
2020-11-09 16:06:27,951 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.691
2020-11-09 16:06:33,308 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.669
2020-11-09 16:06:38,660 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.683
2020-11-09 16:06:41,023 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 99/120: loss: 0.00024 Time taken: 0:00:24.638321 ETA: 0:08:37.404734
2020-11-09 16:06:44,020 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.657
2020-11-09 16:06:49,374 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.680
2020-11-09 16:06:54,724 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.693
2020-11-09 16:07:00,075 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.687
2020-11-09 16:07:08,261 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 18, 0.00s/step
2020-11-09 16:07:09,027 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 18, 0.08s/step
Epoch 100/120Validation cost: -0.000009
Mean average_precision (in %): 0.0000class name average precision (in %)
Barcode 0
Median Inference Time: 0.018703
2020-11-09 16:07:09,631 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 10.465
2020-11-09 16:07:09,849 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 100/120: loss: 0.00024 Time taken: 0:00:28.822534 ETA: 0:09:36.450672
2020-11-09 16:07:14,978 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.704
2020-11-09 16:07:20,324 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.705
2020-11-09 16:07:25,672 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.699
2020-11-09 16:07:31,020 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.699
2020-11-09 16:07:34,453 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 101/120: loss: 0.00024 Time taken: 0:00:24.608407 ETA: 0:07:47.559733
2020-11-09 16:07:36,381 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.655
2020-11-09 16:07:41,732 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.687
2020-11-09 16:07:47,088 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.674
2020-11-09 16:07:52,447 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.659
2020-11-09 16:07:57,796 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.696
2020-11-09 16:07:59,102 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 102/120: loss: 0.00024 Time taken: 0:00:24.634665 ETA: 0:07:23.423970
2020-11-09 16:08:03,168 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.617
2020-11-09 16:08:08,530 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.651
2020-11-09 16:08:13,875 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.707
2020-11-09 16:08:19,228 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.683
2020-11-09 16:08:23,728 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 103/120: loss: 0.00024 Time taken: 0:00:24.638016 ETA: 0:06:58.846280
2020-11-09 16:08:24,588 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.657
2020-11-09 16:08:29,936 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.700
2020-11-09 16:08:35,285 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.694
2020-11-09 16:08:40,645 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.658
2020-11-09 16:08:46,000 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.677
2020-11-09 16:08:48,371 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 104/120: loss: 0.00024 Time taken: 0:00:24.636763 ETA: 0:06:34.188206
2020-11-09 16:08:51,370 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.623
2020-11-09 16:08:56,723 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.681
2020-11-09 16:09:02,079 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.671
2020-11-09 16:09:07,431 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.687
2020-11-09 16:09:12,788 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.666
2020-11-09 16:09:13,005 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 105/120: loss: 0.00023 Time taken: 0:00:24.635604 ETA: 0:06:09.534062
2020-11-09 16:09:18,146 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.667
2020-11-09 16:09:23,500 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.678
2020-11-09 16:09:28,856 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.671
2020-11-09 16:09:34,211 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.677
2020-11-09 16:09:37,643 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 106/120: loss: 0.00023 Time taken: 0:00:24.639756 ETA: 0:05:44.956583
2020-11-09 16:09:39,582 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.616
2020-11-09 16:09:44,941 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.661
2020-11-09 16:09:50,297 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.673
2020-11-09 16:09:55,653 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.670
2020-11-09 16:10:01,010 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.670
2020-11-09 16:10:02,305 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 107/120: loss: 0.00023 Time taken: 0:00:24.661431 ETA: 0:05:20.598598
2020-11-09 16:10:06,384 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.606
2020-11-09 16:10:11,748 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.646
2020-11-09 16:10:17,104 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.672
2020-11-09 16:10:22,460 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.670
2020-11-09 16:10:26,960 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 108/120: loss: 0.00023 Time taken: 0:00:24.653965 ETA: 0:04:55.847574
2020-11-09 16:10:27,823 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.647
2020-11-09 16:10:33,174 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.687
2020-11-09 16:10:38,530 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.672
2020-11-09 16:10:43,902 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.616
2020-11-09 16:10:49,254 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.684
2020-11-09 16:10:51,614 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 109/120: loss: 0.00023 Time taken: 0:00:24.653509 ETA: 0:04:31.188601
2020-11-09 16:10:54,609 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.675
2020-11-09 16:10:59,961 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.687
2020-11-09 16:11:05,314 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.680
2020-11-09 16:11:10,668 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.679
2020-11-09 16:11:18,866 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 18, 0.00s/step
2020-11-09 16:11:19,636 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 18, 0.08s/step
Epoch 110/120Validation cost: -0.000009
Mean average_precision (in %): 0.0000class name average precision (in %)
Barcode 0
Median Inference Time: 0.018713
2020-11-09 16:11:20,241 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 10.447
2020-11-09 16:11:20,460 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 110/120: loss: 0.00023 Time taken: 0:00:28.840487 ETA: 0:04:48.404870
2020-11-09 16:11:25,596 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.673
2020-11-09 16:11:30,946 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.695
2020-11-09 16:11:36,295 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.696
2020-11-09 16:11:41,644 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.695
2020-11-09 16:11:45,075 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 111/120: loss: 0.00023 Time taken: 0:00:24.619414 ETA: 0:03:41.574725
2020-11-09 16:11:47,004 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.658
2020-11-09 16:11:52,362 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.665
2020-11-09 16:11:57,713 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.687
2020-11-09 16:12:03,062 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.696
2020-11-09 16:12:08,404 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.719
2020-11-09 16:12:09,698 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 112/120: loss: 0.00023 Time taken: 0:00:24.619006 ETA: 0:03:16.952049
2020-11-09 16:12:13,764 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.658
2020-11-09 16:12:19,111 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.703
2020-11-09 16:12:24,460 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.696
2020-11-09 16:12:29,808 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.700
2020-11-09 16:12:34,309 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 113/120: loss: 0.00023 Time taken: 0:00:24.606572 ETA: 0:02:52.246002
2020-11-09 16:12:35,172 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.645
2020-11-09 16:12:40,528 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.670
2020-11-09 16:12:45,877 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.694
2020-11-09 16:12:51,235 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.664
2020-11-09 16:12:56,587 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.686
2020-11-09 16:12:58,948 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 114/120: loss: 0.00022 Time taken: 0:00:24.645556 ETA: 0:02:27.873334
2020-11-09 16:13:01,950 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.648
2020-11-09 16:13:07,295 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.709
2020-11-09 16:13:12,647 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.687
2020-11-09 16:13:18,000 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.682
2020-11-09 16:13:23,351 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.686
2020-11-09 16:13:23,569 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 115/120: loss: 0.00022 Time taken: 0:00:24.616805 ETA: 0:02:03.084024
2020-11-09 16:13:28,713 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.654
2020-11-09 16:13:34,066 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.681
2020-11-09 16:13:39,423 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.668
2020-11-09 16:13:44,779 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.670
2020-11-09 16:13:48,211 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 116/120: loss: 0.00022 Time taken: 0:00:24.644392 ETA: 0:01:38.577568
2020-11-09 16:13:50,138 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.661
2020-11-09 16:13:55,496 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.667
2020-11-09 16:14:00,849 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.682
2020-11-09 16:14:06,201 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.683
2020-11-09 16:14:11,554 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.685
2020-11-09 16:14:12,844 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 117/120: loss: 0.00022 Time taken: 0:00:24.632753 ETA: 0:01:13.898258
2020-11-09 16:14:16,913 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.660
2020-11-09 16:14:22,271 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.665
2020-11-09 16:14:27,626 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.676
2020-11-09 16:14:32,978 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.684
2020-11-09 16:14:37,578 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 118/120: loss: 0.00022 Time taken: 0:00:24.733698 ETA: 0:00:49.467396
2020-11-09 16:14:38,443 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.298
2020-11-09 16:14:43,839 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.535
2020-11-09 16:14:49,326 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.225
2020-11-09 16:14:54,732 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.499
2020-11-09 16:15:00,089 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.667
2020-11-09 16:15:02,449 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 119/120: loss: 0.00022 Time taken: 0:00:24.871270 ETA: 0:00:24.871270
2020-11-09 16:15:05,452 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.648
2020-11-09 16:15:10,805 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.680
2020-11-09 16:15:16,168 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.648
2020-11-09 16:15:21,525 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 18.669
2020-11-09 16:15:29,724 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 18, 0.00s/step
2020-11-09 16:15:30,492 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 18, 0.08s/step
Epoch 120/120Validation cost: -0.000009
Mean average_precision (in %): 0.0000class name average precision (in %)
Barcode 0
Median Inference Time: 0.018723
2020-11-09 16:15:31,097 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 10.447
2020-11-09 16:15:31,564 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 10.447
Time taken to run iva.detectnet_v2.scripts.train:main: 0:50:57.088636.