2020-10-22 09:23:46,161 [INFO] iva.detectnet_v2.spec_handler.spec_loader: Merging specification from /workspace/data/specs/detectnet_v2_train_resnet18_kitti.txt 2020-10-22 09:23:48.302719: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2020-10-22 09:23:48.387336: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: name: TITAN RTX major: 7 minor: 5 memoryClockRate(GHz): 1.77 pciBusID: 0000:86:00.0 2020-10-22 09:23:48.387400: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2020-10-22 09:23:48.387476: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2020-10-22 09:23:48.389195: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2020-10-22 09:23:48.389655: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2020-10-22 09:23:48.392057: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2020-10-22 09:23:48.393845: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2020-10-22 09:23:48.393950: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2020-10-22 09:23:48.396702: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0 2020-10-22 09:23:48.396745: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2020-10-22 09:23:49.386309: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix: 2020-10-22 09:23:49.386370: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0 2020-10-22 09:23:49.386385: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N 2020-10-22 09:23:49.390740: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 22394 MB memory) -> physical GPU (device: 0, name: TITAN RTX, pci bus id: 0000:86:00.0, compute capability: 7.5) 2020-10-22 09:23:51,482 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Serial augmentation enabled = False 2020-10-22 09:23:51,482 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Pseudo sharding enabled = False 2020-10-22 09:23:51,483 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Max Image Dimensions (all sources): (0, 0) 2020-10-22 09:23:51,483 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: number of cpus: 48, io threads: 96, compute threads: 48, buffered batches: 4 2020-10-22 09:23:51,483 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: total dataset size 1117, number of sources: 1, batch size per gpu: 32, steps: 35 2020-10-22 09:23:51,629 [INFO] iva.detectnet_v2.dataloader.default_dataloader: Bounding box coordinates were detected in the input specification! Bboxes will be automatically converted to polygon coordinates. 2020-10-22 09:23:51.667316: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: name: TITAN RTX major: 7 minor: 5 memoryClockRate(GHz): 1.77 pciBusID: 0000:86:00.0 2020-10-22 09:23:51.667379: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2020-10-22 09:23:51.667421: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2020-10-22 09:23:51.667438: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2020-10-22 09:23:51.667454: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2020-10-22 09:23:51.667470: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2020-10-22 09:23:51.667486: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2020-10-22 09:23:51.667508: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2020-10-22 09:23:51.668843: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0 2020-10-22 09:23:52,111 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: shuffle: False - shard 0 of 1 2020-10-22 09:23:52,122 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: sampling 1 datasets with weights: 2020-10-22 09:23:52,122 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: source: 0 weight: 1.000000 2020-10-22 09:23:52,580 [INFO] iva.detectnet_v2.evaluation.build_evaluator: Found 1117 samples in validation set __________________________________________________________________________________________________ Layer (type) Output Shape Param # Connected to ================================================================================================== input_1 (InputLayer) (None, 3, 96, 224) 0 __________________________________________________________________________________________________ conv1 (Conv2D) (None, 64, 48, 112) 9472 input_1[0][0] __________________________________________________________________________________________________ bn_conv1 (BatchNormalization) (None, 64, 48, 112) 256 conv1[0][0] __________________________________________________________________________________________________ activation_1 (Activation) (None, 64, 48, 112) 0 bn_conv1[0][0] __________________________________________________________________________________________________ block_1a_conv_1 (Conv2D) (None, 64, 24, 56) 36928 activation_1[0][0] __________________________________________________________________________________________________ block_1a_bn_1 (BatchNormalizati (None, 64, 24, 56) 256 block_1a_conv_1[0][0] __________________________________________________________________________________________________ block_1a_relu_1 (Activation) (None, 64, 24, 56) 0 block_1a_bn_1[0][0] __________________________________________________________________________________________________ block_1a_conv_2 (Conv2D) (None, 64, 24, 56) 36928 block_1a_relu_1[0][0] __________________________________________________________________________________________________ block_1a_conv_shortcut (Conv2D) (None, 64, 24, 56) 4160 activation_1[0][0] __________________________________________________________________________________________________ block_1a_bn_2 (BatchNormalizati (None, 64, 24, 56) 256 block_1a_conv_2[0][0] __________________________________________________________________________________________________ block_1a_bn_shortcut (BatchNorm (None, 64, 24, 56) 256 block_1a_conv_shortcut[0][0] __________________________________________________________________________________________________ add_1 (Add) (None, 64, 24, 56) 0 block_1a_bn_2[0][0] block_1a_bn_shortcut[0][0] __________________________________________________________________________________________________ block_1a_relu (Activation) (None, 64, 24, 56) 0 add_1[0][0] __________________________________________________________________________________________________ block_1b_conv_1 (Conv2D) (None, 64, 24, 56) 36928 block_1a_relu[0][0] __________________________________________________________________________________________________ block_1b_bn_1 (BatchNormalizati (None, 64, 24, 56) 256 block_1b_conv_1[0][0] __________________________________________________________________________________________________ block_1b_relu_1 (Activation) (None, 64, 24, 56) 0 block_1b_bn_1[0][0] __________________________________________________________________________________________________ block_1b_conv_2 (Conv2D) (None, 64, 24, 56) 36928 block_1b_relu_1[0][0] __________________________________________________________________________________________________ block_1b_bn_2 (BatchNormalizati (None, 64, 24, 56) 256 block_1b_conv_2[0][0] __________________________________________________________________________________________________ add_2 (Add) (None, 64, 24, 56) 0 block_1b_bn_2[0][0] block_1a_relu[0][0] __________________________________________________________________________________________________ block_1b_relu (Activation) (None, 64, 24, 56) 0 add_2[0][0] __________________________________________________________________________________________________ block_2a_conv_1 (Conv2D) (None, 128, 12, 28) 73856 block_1b_relu[0][0] __________________________________________________________________________________________________ block_2a_bn_1 (BatchNormalizati (None, 128, 12, 28) 512 block_2a_conv_1[0][0] __________________________________________________________________________________________________ block_2a_relu_1 (Activation) (None, 128, 12, 28) 0 block_2a_bn_1[0][0] __________________________________________________________________________________________________ block_2a_conv_2 (Conv2D) (None, 128, 12, 28) 147584 block_2a_relu_1[0][0] __________________________________________________________________________________________________ block_2a_conv_shortcut (Conv2D) (None, 128, 12, 28) 8320 block_1b_relu[0][0] __________________________________________________________________________________________________ block_2a_bn_2 (BatchNormalizati (None, 128, 12, 28) 512 block_2a_conv_2[0][0] __________________________________________________________________________________________________ block_2a_bn_shortcut (BatchNorm (None, 128, 12, 28) 512 block_2a_conv_shortcut[0][0] __________________________________________________________________________________________________ add_3 (Add) (None, 128, 12, 28) 0 block_2a_bn_2[0][0] block_2a_bn_shortcut[0][0] __________________________________________________________________________________________________ block_2a_relu (Activation) (None, 128, 12, 28) 0 add_3[0][0] __________________________________________________________________________________________________ block_2b_conv_1 (Conv2D) (None, 128, 12, 28) 147584 block_2a_relu[0][0] __________________________________________________________________________________________________ block_2b_bn_1 (BatchNormalizati (None, 128, 12, 28) 512 block_2b_conv_1[0][0] __________________________________________________________________________________________________ block_2b_relu_1 (Activation) (None, 128, 12, 28) 0 block_2b_bn_1[0][0] __________________________________________________________________________________________________ block_2b_conv_2 (Conv2D) (None, 128, 12, 28) 147584 block_2b_relu_1[0][0] __________________________________________________________________________________________________ block_2b_bn_2 (BatchNormalizati (None, 128, 12, 28) 512 block_2b_conv_2[0][0] __________________________________________________________________________________________________ add_4 (Add) (None, 128, 12, 28) 0 block_2b_bn_2[0][0] block_2a_relu[0][0] __________________________________________________________________________________________________ block_2b_relu (Activation) (None, 128, 12, 28) 0 add_4[0][0] __________________________________________________________________________________________________ block_3a_conv_1 (Conv2D) (None, 256, 6, 14) 295168 block_2b_relu[0][0] __________________________________________________________________________________________________ block_3a_bn_1 (BatchNormalizati (None, 256, 6, 14) 1024 block_3a_conv_1[0][0] __________________________________________________________________________________________________ block_3a_relu_1 (Activation) (None, 256, 6, 14) 0 block_3a_bn_1[0][0] __________________________________________________________________________________________________ block_3a_conv_2 (Conv2D) (None, 256, 6, 14) 590080 block_3a_relu_1[0][0] __________________________________________________________________________________________________ block_3a_conv_shortcut (Conv2D) (None, 256, 6, 14) 33024 block_2b_relu[0][0] __________________________________________________________________________________________________ block_3a_bn_2 (BatchNormalizati (None, 256, 6, 14) 1024 block_3a_conv_2[0][0] __________________________________________________________________________________________________ block_3a_bn_shortcut (BatchNorm (None, 256, 6, 14) 1024 block_3a_conv_shortcut[0][0] __________________________________________________________________________________________________ add_5 (Add) (None, 256, 6, 14) 0 block_3a_bn_2[0][0] block_3a_bn_shortcut[0][0] __________________________________________________________________________________________________ block_3a_relu (Activation) (None, 256, 6, 14) 0 add_5[0][0] __________________________________________________________________________________________________ block_3b_conv_1 (Conv2D) (None, 256, 6, 14) 590080 block_3a_relu[0][0] __________________________________________________________________________________________________ block_3b_bn_1 (BatchNormalizati (None, 256, 6, 14) 1024 block_3b_conv_1[0][0] __________________________________________________________________________________________________ block_3b_relu_1 (Activation) (None, 256, 6, 14) 0 block_3b_bn_1[0][0] __________________________________________________________________________________________________ block_3b_conv_2 (Conv2D) (None, 256, 6, 14) 590080 block_3b_relu_1[0][0] __________________________________________________________________________________________________ block_3b_bn_2 (BatchNormalizati (None, 256, 6, 14) 1024 block_3b_conv_2[0][0] __________________________________________________________________________________________________ add_6 (Add) (None, 256, 6, 14) 0 block_3b_bn_2[0][0] block_3a_relu[0][0] __________________________________________________________________________________________________ block_3b_relu (Activation) (None, 256, 6, 14) 0 add_6[0][0] __________________________________________________________________________________________________ block_4a_conv_1 (Conv2D) (None, 512, 6, 14) 1180160 block_3b_relu[0][0] __________________________________________________________________________________________________ block_4a_bn_1 (BatchNormalizati (None, 512, 6, 14) 2048 block_4a_conv_1[0][0] __________________________________________________________________________________________________ block_4a_relu_1 (Activation) (None, 512, 6, 14) 0 block_4a_bn_1[0][0] __________________________________________________________________________________________________ block_4a_conv_2 (Conv2D) (None, 512, 6, 14) 2359808 block_4a_relu_1[0][0] __________________________________________________________________________________________________ block_4a_conv_shortcut (Conv2D) (None, 512, 6, 14) 131584 block_3b_relu[0][0] __________________________________________________________________________________________________ block_4a_bn_2 (BatchNormalizati (None, 512, 6, 14) 2048 block_4a_conv_2[0][0] __________________________________________________________________________________________________ block_4a_bn_shortcut (BatchNorm (None, 512, 6, 14) 2048 block_4a_conv_shortcut[0][0] __________________________________________________________________________________________________ add_7 (Add) (None, 512, 6, 14) 0 block_4a_bn_2[0][0] block_4a_bn_shortcut[0][0] __________________________________________________________________________________________________ block_4a_relu (Activation) (None, 512, 6, 14) 0 add_7[0][0] __________________________________________________________________________________________________ block_4b_conv_1 (Conv2D) (None, 512, 6, 14) 2359808 block_4a_relu[0][0] __________________________________________________________________________________________________ block_4b_bn_1 (BatchNormalizati (None, 512, 6, 14) 2048 block_4b_conv_1[0][0] __________________________________________________________________________________________________ block_4b_relu_1 (Activation) (None, 512, 6, 14) 0 block_4b_bn_1[0][0] __________________________________________________________________________________________________ block_4b_conv_2 (Conv2D) (None, 512, 6, 14) 2359808 block_4b_relu_1[0][0] __________________________________________________________________________________________________ block_4b_bn_2 (BatchNormalizati (None, 512, 6, 14) 2048 block_4b_conv_2[0][0] __________________________________________________________________________________________________ add_8 (Add) (None, 512, 6, 14) 0 block_4b_bn_2[0][0] block_4a_relu[0][0] __________________________________________________________________________________________________ block_4b_relu (Activation) (None, 512, 6, 14) 0 add_8[0][0] __________________________________________________________________________________________________ output_bbox (Conv2D) (None, 140, 6, 14) 71820 block_4b_relu[0][0] __________________________________________________________________________________________________ output_cov (Conv2D) (None, 35, 6, 14) 17955 block_4b_relu[0][0] ================================================================================================== Total params: 11,285,103 Trainable params: 11,275,375 Non-trainable params: 9,728 __________________________________________________________________________________________________ 2020-10-22 09:23:57.004555: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: name: TITAN RTX major: 7 minor: 5 memoryClockRate(GHz): 1.77 pciBusID: 0000:86:00.0 2020-10-22 09:23:57.004631: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2020-10-22 09:23:57.004688: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2020-10-22 09:23:57.004707: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2020-10-22 09:23:57.004724: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2020-10-22 09:23:57.004740: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2020-10-22 09:23:57.004757: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2020-10-22 09:23:57.004773: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2020-10-22 09:23:57.006113: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0 2020-10-22 09:23:57.008203: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix: 2020-10-22 09:23:57.008229: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0 2020-10-22 09:23:57.008238: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N 2020-10-22 09:23:57.009696: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 22394 MB memory) -> physical GPU (device: 0, name: TITAN RTX, pci bus id: 0000:86:00.0, compute capability: 7.5) 2020-10-22 09:23:59,790 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 35, 0.00s/step 2020-10-22 09:24:01.431842: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2020-10-22 09:24:01.433697: I tensorflow/core/kernels/cuda_solvers.cc:159] Creating CudaSolver handles for stream 0x27b71af0 2020-10-22 09:24:01.433942: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2020-10-22 09:24:02.199159: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2020-10-22 09:24:02.406079: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2020-10-22 09:24:09,084 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 35, 0.93s/step 2020-10-22 09:24:14,031 [INFO] iva.detectnet_v2.evaluation.evaluation: step 20 / 35, 0.49s/step 2020-10-22 09:24:18,642 [INFO] iva.detectnet_v2.evaluation.evaluation: step 30 / 35, 0.46s/step Matching predictions to ground truth, class 1/35.: 100%|█| 579/579 [00:00<00:00, 11425.40it/s] Matching predictions to ground truth, class 2/35.: 100%|█| 535/535 [00:00<00:00, 13824.17it/s] Matching predictions to ground truth, class 3/35.: 100%|█| 589/589 [00:00<00:00, 11695.36it/s] Matching predictions to ground truth, class 4/35.: 100%|█| 434/434 [00:00<00:00, 11431.06it/s] Matching predictions to ground truth, class 5/35.: 100%|█| 647/647 [00:00<00:00, 13616.09it/s] Matching predictions to ground truth, class 6/35.: 100%|█| 1/1 [00:00<00:00, 4025.24it/s] Matching predictions to ground truth, class 7/35.: 100%|█| 669/669 [00:00<00:00, 15006.09it/s] Matching predictions to ground truth, class 8/35.: 100%|█| 813/813 [00:00<00:00, 12378.33it/s] Matching predictions to ground truth, class 9/35.: 100%|█| 556/556 [00:00<00:00, 11744.90it/s] Matching predictions to ground truth, class 10/35.: 100%|█| 630/630 [00:00<00:00, 11795.11it/s] Matching predictions to ground truth, class 11/35.: 100%|█| 382/382 [00:00<00:00, 11649.40it/s] Matching predictions to ground truth, class 13/35.: 100%|█| 179/179 [00:00<00:00, 23517.01it/s] Matching predictions to ground truth, class 14/35.: 100%|█| 920/920 [00:00<00:00, 30139.50it/s] Matching predictions to ground truth, class 15/35.: 100%|█| 244/244 [00:00<00:00, 30778.33it/s] Matching predictions to ground truth, class 17/35.: 100%|█| 138/138 [00:00<00:00, 13114.92it/s] Matching predictions to ground truth, class 18/35.: 100%|█| 624/624 [00:00<00:00, 28883.46it/s] Matching predictions to ground truth, class 20/35.: 100%|█| 287/287 [00:00<00:00, 32403.71it/s] Matching predictions to ground truth, class 21/35.: 100%|█| 387/387 [00:00<00:00, 12749.35it/s] Matching predictions to ground truth, class 22/35.: 100%|█| 405/405 [00:00<00:00, 12154.18it/s] Matching predictions to ground truth, class 23/35.: 100%|█| 561/561 [00:00<00:00, 32147.95it/s] Matching predictions to ground truth, class 24/35.: 100%|█| 487/487 [00:00<00:00, 31769.59it/s] Matching predictions to ground truth, class 26/35.: 100%|█| 1/1 [00:00<00:00, 7989.15it/s] Matching predictions to ground truth, class 27/35.: 100%|█| 300/300 [00:00<00:00, 39725.06it/s] Matching predictions to ground truth, class 29/35.: 100%|█| 596/596 [00:00<00:00, 31864.54it/s] Matching predictions to ground truth, class 31/35.: 100%|█| 1/1 [00:00<00:00, 6797.90it/s] Matching predictions to ground truth, class 32/35.: 100%|█| 587/587 [00:00<00:00, 27148.95it/s] Matching predictions to ground truth, class 33/35.: 100%|█| 141/141 [00:00<00:00, 14990.29it/s] Matching predictions to ground truth, class 35/35.: 100%|█| 203/203 [00:00<00:00, 13914.30it/s] Validation cost: 0.000917 Mean average_precision (in %): 50.8764 class name average precision (in %) ------------ -------------------------- a 99.5633 a0 98.9199 a1 98.6738 a2 99.2469 a3 99.2547 a4 0 a5 98.5794 a6 99.1117 a7 99.2181 a8 98.2053 a9 99.1157 b 0 c 55.7202 d 28.8427 e 14.5717 f 0 g 98.7936 h 38.6502 i 0 j 7.34229 k 99.301 l 99.1454 m 14.0127 n 14.8827 p 0 q 0 r 2.86495 s 0 t 52.0325 u 0 v 0 w 69.0906 x 97.6779 y 0 z 97.8575 Median Inference Time: 0.001501 2020-10-22 09:24:22,586 [INFO] iva.detectnet_v2.scripts.evaluate: Evaluation complete. Time taken to run iva.detectnet_v2.scripts.evaluate:main: 0:00:36.427874.