Using TensorFlow backend. 2020-10-27 01:36:38.701121: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2020-10-27 01:36:42,533 [INFO] iva.detectnet_v2.spec_handler.spec_loader: Merging specification from /workspace/data/specs/detectnet_v2_train_resnet18_kitti.txt 2020-10-27 01:36:46.580639: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2020-10-27 01:36:46.682273: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: name: TITAN RTX major: 7 minor: 5 memoryClockRate(GHz): 1.77 pciBusID: 0000:86:00.0 2020-10-27 01:36:46.682334: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2020-10-27 01:36:46.682408: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2020-10-27 01:36:46.684221: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2020-10-27 01:36:46.684711: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2020-10-27 01:36:46.699944: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2020-10-27 01:36:46.701562: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2020-10-27 01:36:46.701649: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2020-10-27 01:36:46.703935: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0 2020-10-27 01:36:46.703979: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2020-10-27 01:36:47.720879: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix: 2020-10-27 01:36:47.720935: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0 2020-10-27 01:36:47.720961: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N 2020-10-27 01:36:47.724607: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 22394 MB memory) -> physical GPU (device: 0, name: TITAN RTX, pci bus id: 0000:86:00.0, compute capability: 7.5) 2020-10-27 01:36:50,651 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Serial augmentation enabled = False 2020-10-27 01:36:50,651 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Pseudo sharding enabled = False 2020-10-27 01:36:50,652 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Max Image Dimensions (all sources): (0, 0) 2020-10-27 01:36:50,652 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: number of cpus: 48, io threads: 96, compute threads: 48, buffered batches: 4 2020-10-27 01:36:50,652 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: total dataset size 1117, number of sources: 1, batch size per gpu: 32, steps: 35 2020-10-27 01:36:50,800 [INFO] iva.detectnet_v2.dataloader.default_dataloader: Bounding box coordinates were detected in the input specification! Bboxes will be automatically converted to polygon coordinates. 2020-10-27 01:36:50.839308: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: name: TITAN RTX major: 7 minor: 5 memoryClockRate(GHz): 1.77 pciBusID: 0000:86:00.0 2020-10-27 01:36:50.839375: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2020-10-27 01:36:50.839434: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2020-10-27 01:36:50.839456: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2020-10-27 01:36:50.839478: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2020-10-27 01:36:50.839499: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2020-10-27 01:36:50.839520: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2020-10-27 01:36:50.839540: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2020-10-27 01:36:50.840929: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0 2020-10-27 01:36:51,099 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: shuffle: False - shard 0 of 1 2020-10-27 01:36:51,107 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: sampling 1 datasets with weights: 2020-10-27 01:36:51,107 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: source: 0 weight: 1.000000 2020-10-27 01:36:51,532 [INFO] iva.detectnet_v2.evaluation.build_evaluator: Found 1117 samples in validation set __________________________________________________________________________________________________ Layer (type) Output Shape Param # Connected to ================================================================================================== input_1 (InputLayer) (None, 3, 288, 672) 0 __________________________________________________________________________________________________ conv1 (Conv2D) (None, 64, 144, 336) 9472 input_1[0][0] __________________________________________________________________________________________________ bn_conv1 (BatchNormalization) (None, 64, 144, 336) 256 conv1[0][0] __________________________________________________________________________________________________ activation_1 (Activation) (None, 64, 144, 336) 0 bn_conv1[0][0] __________________________________________________________________________________________________ block_1a_conv_1 (Conv2D) (None, 64, 72, 168) 4160 activation_1[0][0] __________________________________________________________________________________________________ block_1a_bn_1 (BatchNormalizati (None, 64, 72, 168) 256 block_1a_conv_1[0][0] __________________________________________________________________________________________________ block_1a_relu_1 (Activation) (None, 64, 72, 168) 0 block_1a_bn_1[0][0] __________________________________________________________________________________________________ block_1a_conv_2 (Conv2D) (None, 64, 72, 168) 36928 block_1a_relu_1[0][0] __________________________________________________________________________________________________ block_1a_bn_2 (BatchNormalizati (None, 64, 72, 168) 256 block_1a_conv_2[0][0] __________________________________________________________________________________________________ block_1a_relu_2 (Activation) (None, 64, 72, 168) 0 block_1a_bn_2[0][0] __________________________________________________________________________________________________ block_1a_conv_3 (Conv2D) (None, 256, 72, 168) 16640 block_1a_relu_2[0][0] __________________________________________________________________________________________________ block_1a_conv_shortcut (Conv2D) (None, 256, 72, 168) 16640 activation_1[0][0] __________________________________________________________________________________________________ block_1a_bn_3 (BatchNormalizati (None, 256, 72, 168) 1024 block_1a_conv_3[0][0] __________________________________________________________________________________________________ block_1a_bn_shortcut (BatchNorm (None, 256, 72, 168) 1024 block_1a_conv_shortcut[0][0] __________________________________________________________________________________________________ add_1 (Add) (None, 256, 72, 168) 0 block_1a_bn_3[0][0] block_1a_bn_shortcut[0][0] __________________________________________________________________________________________________ block_1a_relu (Activation) (None, 256, 72, 168) 0 add_1[0][0] __________________________________________________________________________________________________ block_1b_conv_1 (Conv2D) (None, 64, 72, 168) 16448 block_1a_relu[0][0] __________________________________________________________________________________________________ block_1b_bn_1 (BatchNormalizati (None, 64, 72, 168) 256 block_1b_conv_1[0][0] __________________________________________________________________________________________________ block_1b_relu_1 (Activation) (None, 64, 72, 168) 0 block_1b_bn_1[0][0] __________________________________________________________________________________________________ block_1b_conv_2 (Conv2D) (None, 64, 72, 168) 36928 block_1b_relu_1[0][0] __________________________________________________________________________________________________ block_1b_bn_2 (BatchNormalizati (None, 64, 72, 168) 256 block_1b_conv_2[0][0] __________________________________________________________________________________________________ block_1b_relu_2 (Activation) (None, 64, 72, 168) 0 block_1b_bn_2[0][0] __________________________________________________________________________________________________ block_1b_conv_3 (Conv2D) (None, 256, 72, 168) 16640 block_1b_relu_2[0][0] __________________________________________________________________________________________________ block_1b_bn_3 (BatchNormalizati (None, 256, 72, 168) 1024 block_1b_conv_3[0][0] __________________________________________________________________________________________________ add_2 (Add) (None, 256, 72, 168) 0 block_1b_bn_3[0][0] block_1a_relu[0][0] __________________________________________________________________________________________________ block_1b_relu (Activation) (None, 256, 72, 168) 0 add_2[0][0] __________________________________________________________________________________________________ block_1c_conv_1 (Conv2D) (None, 64, 72, 168) 16448 block_1b_relu[0][0] __________________________________________________________________________________________________ block_1c_bn_1 (BatchNormalizati (None, 64, 72, 168) 256 block_1c_conv_1[0][0] __________________________________________________________________________________________________ block_1c_relu_1 (Activation) (None, 64, 72, 168) 0 block_1c_bn_1[0][0] __________________________________________________________________________________________________ block_1c_conv_2 (Conv2D) (None, 64, 72, 168) 36928 block_1c_relu_1[0][0] __________________________________________________________________________________________________ block_1c_bn_2 (BatchNormalizati (None, 64, 72, 168) 256 block_1c_conv_2[0][0] __________________________________________________________________________________________________ block_1c_relu_2 (Activation) (None, 64, 72, 168) 0 block_1c_bn_2[0][0] __________________________________________________________________________________________________ block_1c_conv_3 (Conv2D) (None, 256, 72, 168) 16640 block_1c_relu_2[0][0] __________________________________________________________________________________________________ block_1c_bn_3 (BatchNormalizati (None, 256, 72, 168) 1024 block_1c_conv_3[0][0] __________________________________________________________________________________________________ add_3 (Add) (None, 256, 72, 168) 0 block_1c_bn_3[0][0] block_1b_relu[0][0] __________________________________________________________________________________________________ block_1c_relu (Activation) (None, 256, 72, 168) 0 add_3[0][0] __________________________________________________________________________________________________ block_2a_conv_1 (Conv2D) (None, 128, 36, 84) 32896 block_1c_relu[0][0] __________________________________________________________________________________________________ block_2a_bn_1 (BatchNormalizati (None, 128, 36, 84) 512 block_2a_conv_1[0][0] __________________________________________________________________________________________________ block_2a_relu_1 (Activation) (None, 128, 36, 84) 0 block_2a_bn_1[0][0] __________________________________________________________________________________________________ block_2a_conv_2 (Conv2D) (None, 128, 36, 84) 147584 block_2a_relu_1[0][0] __________________________________________________________________________________________________ block_2a_bn_2 (BatchNormalizati (None, 128, 36, 84) 512 block_2a_conv_2[0][0] __________________________________________________________________________________________________ block_2a_relu_2 (Activation) (None, 128, 36, 84) 0 block_2a_bn_2[0][0] __________________________________________________________________________________________________ block_2a_conv_3 (Conv2D) (None, 512, 36, 84) 66048 block_2a_relu_2[0][0] __________________________________________________________________________________________________ block_2a_conv_shortcut (Conv2D) (None, 512, 36, 84) 131584 block_1c_relu[0][0] __________________________________________________________________________________________________ block_2a_bn_3 (BatchNormalizati (None, 512, 36, 84) 2048 block_2a_conv_3[0][0] __________________________________________________________________________________________________ block_2a_bn_shortcut (BatchNorm (None, 512, 36, 84) 2048 block_2a_conv_shortcut[0][0] __________________________________________________________________________________________________ add_4 (Add) (None, 512, 36, 84) 0 block_2a_bn_3[0][0] block_2a_bn_shortcut[0][0] __________________________________________________________________________________________________ block_2a_relu (Activation) (None, 512, 36, 84) 0 add_4[0][0] __________________________________________________________________________________________________ block_2b_conv_1 (Conv2D) (None, 128, 36, 84) 65664 block_2a_relu[0][0] __________________________________________________________________________________________________ block_2b_bn_1 (BatchNormalizati (None, 128, 36, 84) 512 block_2b_conv_1[0][0] __________________________________________________________________________________________________ block_2b_relu_1 (Activation) (None, 128, 36, 84) 0 block_2b_bn_1[0][0] __________________________________________________________________________________________________ block_2b_conv_2 (Conv2D) (None, 128, 36, 84) 147584 block_2b_relu_1[0][0] __________________________________________________________________________________________________ block_2b_bn_2 (BatchNormalizati (None, 128, 36, 84) 512 block_2b_conv_2[0][0] __________________________________________________________________________________________________ block_2b_relu_2 (Activation) (None, 128, 36, 84) 0 block_2b_bn_2[0][0] __________________________________________________________________________________________________ block_2b_conv_3 (Conv2D) (None, 512, 36, 84) 66048 block_2b_relu_2[0][0] __________________________________________________________________________________________________ block_2b_bn_3 (BatchNormalizati (None, 512, 36, 84) 2048 block_2b_conv_3[0][0] __________________________________________________________________________________________________ add_5 (Add) (None, 512, 36, 84) 0 block_2b_bn_3[0][0] block_2a_relu[0][0] __________________________________________________________________________________________________ block_2b_relu (Activation) (None, 512, 36, 84) 0 add_5[0][0] __________________________________________________________________________________________________ block_2c_conv_1 (Conv2D) (None, 128, 36, 84) 65664 block_2b_relu[0][0] __________________________________________________________________________________________________ block_2c_bn_1 (BatchNormalizati (None, 128, 36, 84) 512 block_2c_conv_1[0][0] __________________________________________________________________________________________________ block_2c_relu_1 (Activation) (None, 128, 36, 84) 0 block_2c_bn_1[0][0] __________________________________________________________________________________________________ block_2c_conv_2 (Conv2D) (None, 128, 36, 84) 147584 block_2c_relu_1[0][0] __________________________________________________________________________________________________ block_2c_bn_2 (BatchNormalizati (None, 128, 36, 84) 512 block_2c_conv_2[0][0] __________________________________________________________________________________________________ block_2c_relu_2 (Activation) (None, 128, 36, 84) 0 block_2c_bn_2[0][0] __________________________________________________________________________________________________ block_2c_conv_3 (Conv2D) (None, 512, 36, 84) 66048 block_2c_relu_2[0][0] __________________________________________________________________________________________________ block_2c_bn_3 (BatchNormalizati (None, 512, 36, 84) 2048 block_2c_conv_3[0][0] __________________________________________________________________________________________________ add_6 (Add) (None, 512, 36, 84) 0 block_2c_bn_3[0][0] block_2b_relu[0][0] __________________________________________________________________________________________________ block_2c_relu (Activation) (None, 512, 36, 84) 0 add_6[0][0] __________________________________________________________________________________________________ block_2d_conv_1 (Conv2D) (None, 128, 36, 84) 65664 block_2c_relu[0][0] __________________________________________________________________________________________________ block_2d_bn_1 (BatchNormalizati (None, 128, 36, 84) 512 block_2d_conv_1[0][0] __________________________________________________________________________________________________ block_2d_relu_1 (Activation) (None, 128, 36, 84) 0 block_2d_bn_1[0][0] __________________________________________________________________________________________________ block_2d_conv_2 (Conv2D) (None, 128, 36, 84) 147584 block_2d_relu_1[0][0] __________________________________________________________________________________________________ block_2d_bn_2 (BatchNormalizati (None, 128, 36, 84) 512 block_2d_conv_2[0][0] __________________________________________________________________________________________________ block_2d_relu_2 (Activation) (None, 128, 36, 84) 0 block_2d_bn_2[0][0] __________________________________________________________________________________________________ block_2d_conv_3 (Conv2D) (None, 512, 36, 84) 66048 block_2d_relu_2[0][0] __________________________________________________________________________________________________ block_2d_bn_3 (BatchNormalizati (None, 512, 36, 84) 2048 block_2d_conv_3[0][0] __________________________________________________________________________________________________ add_7 (Add) (None, 512, 36, 84) 0 block_2d_bn_3[0][0] block_2c_relu[0][0] __________________________________________________________________________________________________ block_2d_relu (Activation) (None, 512, 36, 84) 0 add_7[0][0] __________________________________________________________________________________________________ block_3a_conv_1 (Conv2D) (None, 256, 18, 42) 131328 block_2d_relu[0][0] __________________________________________________________________________________________________ block_3a_bn_1 (BatchNormalizati (None, 256, 18, 42) 1024 block_3a_conv_1[0][0] __________________________________________________________________________________________________ block_3a_relu_1 (Activation) (None, 256, 18, 42) 0 block_3a_bn_1[0][0] __________________________________________________________________________________________________ block_3a_conv_2 (Conv2D) (None, 256, 18, 42) 590080 block_3a_relu_1[0][0] __________________________________________________________________________________________________ block_3a_bn_2 (BatchNormalizati (None, 256, 18, 42) 1024 block_3a_conv_2[0][0] __________________________________________________________________________________________________ block_3a_relu_2 (Activation) (None, 256, 18, 42) 0 block_3a_bn_2[0][0] __________________________________________________________________________________________________ block_3a_conv_3 (Conv2D) (None, 1024, 18, 42) 263168 block_3a_relu_2[0][0] __________________________________________________________________________________________________ block_3a_conv_shortcut (Conv2D) (None, 1024, 18, 42) 525312 block_2d_relu[0][0] __________________________________________________________________________________________________ block_3a_bn_3 (BatchNormalizati (None, 1024, 18, 42) 4096 block_3a_conv_3[0][0] __________________________________________________________________________________________________ block_3a_bn_shortcut (BatchNorm (None, 1024, 18, 42) 4096 block_3a_conv_shortcut[0][0] __________________________________________________________________________________________________ add_8 (Add) (None, 1024, 18, 42) 0 block_3a_bn_3[0][0] block_3a_bn_shortcut[0][0] __________________________________________________________________________________________________ block_3a_relu (Activation) (None, 1024, 18, 42) 0 add_8[0][0] __________________________________________________________________________________________________ block_3b_conv_1 (Conv2D) (None, 256, 18, 42) 262400 block_3a_relu[0][0] __________________________________________________________________________________________________ block_3b_bn_1 (BatchNormalizati (None, 256, 18, 42) 1024 block_3b_conv_1[0][0] __________________________________________________________________________________________________ block_3b_relu_1 (Activation) (None, 256, 18, 42) 0 block_3b_bn_1[0][0] __________________________________________________________________________________________________ block_3b_conv_2 (Conv2D) (None, 256, 18, 42) 590080 block_3b_relu_1[0][0] __________________________________________________________________________________________________ block_3b_bn_2 (BatchNormalizati (None, 256, 18, 42) 1024 block_3b_conv_2[0][0] __________________________________________________________________________________________________ block_3b_relu_2 (Activation) (None, 256, 18, 42) 0 block_3b_bn_2[0][0] __________________________________________________________________________________________________ block_3b_conv_3 (Conv2D) (None, 1024, 18, 42) 263168 block_3b_relu_2[0][0] __________________________________________________________________________________________________ block_3b_bn_3 (BatchNormalizati (None, 1024, 18, 42) 4096 block_3b_conv_3[0][0] __________________________________________________________________________________________________ add_9 (Add) (None, 1024, 18, 42) 0 block_3b_bn_3[0][0] block_3a_relu[0][0] __________________________________________________________________________________________________ block_3b_relu (Activation) (None, 1024, 18, 42) 0 add_9[0][0] __________________________________________________________________________________________________ block_3c_conv_1 (Conv2D) (None, 256, 18, 42) 262400 block_3b_relu[0][0] __________________________________________________________________________________________________ block_3c_bn_1 (BatchNormalizati (None, 256, 18, 42) 1024 block_3c_conv_1[0][0] __________________________________________________________________________________________________ block_3c_relu_1 (Activation) (None, 256, 18, 42) 0 block_3c_bn_1[0][0] __________________________________________________________________________________________________ block_3c_conv_2 (Conv2D) (None, 256, 18, 42) 590080 block_3c_relu_1[0][0] __________________________________________________________________________________________________ block_3c_bn_2 (BatchNormalizati (None, 256, 18, 42) 1024 block_3c_conv_2[0][0] __________________________________________________________________________________________________ block_3c_relu_2 (Activation) (None, 256, 18, 42) 0 block_3c_bn_2[0][0] __________________________________________________________________________________________________ block_3c_conv_3 (Conv2D) (None, 1024, 18, 42) 263168 block_3c_relu_2[0][0] __________________________________________________________________________________________________ block_3c_bn_3 (BatchNormalizati (None, 1024, 18, 42) 4096 block_3c_conv_3[0][0] __________________________________________________________________________________________________ add_10 (Add) (None, 1024, 18, 42) 0 block_3c_bn_3[0][0] block_3b_relu[0][0] __________________________________________________________________________________________________ block_3c_relu (Activation) (None, 1024, 18, 42) 0 add_10[0][0] __________________________________________________________________________________________________ block_3d_conv_1 (Conv2D) (None, 256, 18, 42) 262400 block_3c_relu[0][0] __________________________________________________________________________________________________ block_3d_bn_1 (BatchNormalizati (None, 256, 18, 42) 1024 block_3d_conv_1[0][0] __________________________________________________________________________________________________ block_3d_relu_1 (Activation) (None, 256, 18, 42) 0 block_3d_bn_1[0][0] __________________________________________________________________________________________________ block_3d_conv_2 (Conv2D) (None, 256, 18, 42) 590080 block_3d_relu_1[0][0] __________________________________________________________________________________________________ block_3d_bn_2 (BatchNormalizati (None, 256, 18, 42) 1024 block_3d_conv_2[0][0] __________________________________________________________________________________________________ block_3d_relu_2 (Activation) (None, 256, 18, 42) 0 block_3d_bn_2[0][0] __________________________________________________________________________________________________ block_3d_conv_3 (Conv2D) (None, 1024, 18, 42) 263168 block_3d_relu_2[0][0] __________________________________________________________________________________________________ block_3d_bn_3 (BatchNormalizati (None, 1024, 18, 42) 4096 block_3d_conv_3[0][0] __________________________________________________________________________________________________ add_11 (Add) (None, 1024, 18, 42) 0 block_3d_bn_3[0][0] block_3c_relu[0][0] __________________________________________________________________________________________________ block_3d_relu (Activation) (None, 1024, 18, 42) 0 add_11[0][0] __________________________________________________________________________________________________ block_3e_conv_1 (Conv2D) (None, 256, 18, 42) 262400 block_3d_relu[0][0] __________________________________________________________________________________________________ block_3e_bn_1 (BatchNormalizati (None, 256, 18, 42) 1024 block_3e_conv_1[0][0] __________________________________________________________________________________________________ block_3e_relu_1 (Activation) (None, 256, 18, 42) 0 block_3e_bn_1[0][0] __________________________________________________________________________________________________ block_3e_conv_2 (Conv2D) (None, 256, 18, 42) 590080 block_3e_relu_1[0][0] __________________________________________________________________________________________________ block_3e_bn_2 (BatchNormalizati (None, 256, 18, 42) 1024 block_3e_conv_2[0][0] __________________________________________________________________________________________________ block_3e_relu_2 (Activation) (None, 256, 18, 42) 0 block_3e_bn_2[0][0] __________________________________________________________________________________________________ block_3e_conv_3 (Conv2D) (None, 1024, 18, 42) 263168 block_3e_relu_2[0][0] __________________________________________________________________________________________________ block_3e_bn_3 (BatchNormalizati (None, 1024, 18, 42) 4096 block_3e_conv_3[0][0] __________________________________________________________________________________________________ add_12 (Add) (None, 1024, 18, 42) 0 block_3e_bn_3[0][0] block_3d_relu[0][0] __________________________________________________________________________________________________ block_3e_relu (Activation) (None, 1024, 18, 42) 0 add_12[0][0] __________________________________________________________________________________________________ block_3f_conv_1 (Conv2D) (None, 256, 18, 42) 262400 block_3e_relu[0][0] __________________________________________________________________________________________________ block_3f_bn_1 (BatchNormalizati (None, 256, 18, 42) 1024 block_3f_conv_1[0][0] __________________________________________________________________________________________________ block_3f_relu_1 (Activation) (None, 256, 18, 42) 0 block_3f_bn_1[0][0] __________________________________________________________________________________________________ block_3f_conv_2 (Conv2D) (None, 256, 18, 42) 590080 block_3f_relu_1[0][0] __________________________________________________________________________________________________ block_3f_bn_2 (BatchNormalizati (None, 256, 18, 42) 1024 block_3f_conv_2[0][0] __________________________________________________________________________________________________ block_3f_relu_2 (Activation) (None, 256, 18, 42) 0 block_3f_bn_2[0][0] __________________________________________________________________________________________________ block_3f_conv_3 (Conv2D) (None, 1024, 18, 42) 263168 block_3f_relu_2[0][0] __________________________________________________________________________________________________ block_3f_bn_3 (BatchNormalizati (None, 1024, 18, 42) 4096 block_3f_conv_3[0][0] __________________________________________________________________________________________________ add_13 (Add) (None, 1024, 18, 42) 0 block_3f_bn_3[0][0] block_3e_relu[0][0] __________________________________________________________________________________________________ block_3f_relu (Activation) (None, 1024, 18, 42) 0 add_13[0][0] __________________________________________________________________________________________________ block_4a_conv_1 (Conv2D) (None, 512, 18, 42) 524800 block_3f_relu[0][0] __________________________________________________________________________________________________ block_4a_bn_1 (BatchNormalizati (None, 512, 18, 42) 2048 block_4a_conv_1[0][0] __________________________________________________________________________________________________ block_4a_relu_1 (Activation) (None, 512, 18, 42) 0 block_4a_bn_1[0][0] __________________________________________________________________________________________________ block_4a_conv_2 (Conv2D) (None, 512, 18, 42) 2359808 block_4a_relu_1[0][0] __________________________________________________________________________________________________ block_4a_bn_2 (BatchNormalizati (None, 512, 18, 42) 2048 block_4a_conv_2[0][0] __________________________________________________________________________________________________ block_4a_relu_2 (Activation) (None, 512, 18, 42) 0 block_4a_bn_2[0][0] __________________________________________________________________________________________________ block_4a_conv_3 (Conv2D) (None, 2048, 18, 42) 1050624 block_4a_relu_2[0][0] __________________________________________________________________________________________________ block_4a_conv_shortcut (Conv2D) (None, 2048, 18, 42) 2099200 block_3f_relu[0][0] __________________________________________________________________________________________________ block_4a_bn_3 (BatchNormalizati (None, 2048, 18, 42) 8192 block_4a_conv_3[0][0] __________________________________________________________________________________________________ block_4a_bn_shortcut (BatchNorm (None, 2048, 18, 42) 8192 block_4a_conv_shortcut[0][0] __________________________________________________________________________________________________ add_14 (Add) (None, 2048, 18, 42) 0 block_4a_bn_3[0][0] block_4a_bn_shortcut[0][0] __________________________________________________________________________________________________ block_4a_relu (Activation) (None, 2048, 18, 42) 0 add_14[0][0] __________________________________________________________________________________________________ block_4b_conv_1 (Conv2D) (None, 512, 18, 42) 1049088 block_4a_relu[0][0] __________________________________________________________________________________________________ block_4b_bn_1 (BatchNormalizati (None, 512, 18, 42) 2048 block_4b_conv_1[0][0] __________________________________________________________________________________________________ block_4b_relu_1 (Activation) (None, 512, 18, 42) 0 block_4b_bn_1[0][0] __________________________________________________________________________________________________ block_4b_conv_2 (Conv2D) (None, 512, 18, 42) 2359808 block_4b_relu_1[0][0] __________________________________________________________________________________________________ block_4b_bn_2 (BatchNormalizati (None, 512, 18, 42) 2048 block_4b_conv_2[0][0] __________________________________________________________________________________________________ block_4b_relu_2 (Activation) (None, 512, 18, 42) 0 block_4b_bn_2[0][0] __________________________________________________________________________________________________ block_4b_conv_3 (Conv2D) (None, 2048, 18, 42) 1050624 block_4b_relu_2[0][0] __________________________________________________________________________________________________ block_4b_bn_3 (BatchNormalizati (None, 2048, 18, 42) 8192 block_4b_conv_3[0][0] __________________________________________________________________________________________________ add_15 (Add) (None, 2048, 18, 42) 0 block_4b_bn_3[0][0] block_4a_relu[0][0] __________________________________________________________________________________________________ block_4b_relu (Activation) (None, 2048, 18, 42) 0 add_15[0][0] __________________________________________________________________________________________________ block_4c_conv_1 (Conv2D) (None, 512, 18, 42) 1049088 block_4b_relu[0][0] __________________________________________________________________________________________________ block_4c_bn_1 (BatchNormalizati (None, 512, 18, 42) 2048 block_4c_conv_1[0][0] __________________________________________________________________________________________________ block_4c_relu_1 (Activation) (None, 512, 18, 42) 0 block_4c_bn_1[0][0] __________________________________________________________________________________________________ block_4c_conv_2 (Conv2D) (None, 512, 18, 42) 2359808 block_4c_relu_1[0][0] __________________________________________________________________________________________________ block_4c_bn_2 (BatchNormalizati (None, 512, 18, 42) 2048 block_4c_conv_2[0][0] __________________________________________________________________________________________________ block_4c_relu_2 (Activation) (None, 512, 18, 42) 0 block_4c_bn_2[0][0] __________________________________________________________________________________________________ block_4c_conv_3 (Conv2D) (None, 2048, 18, 42) 1050624 block_4c_relu_2[0][0] __________________________________________________________________________________________________ block_4c_bn_3 (BatchNormalizati (None, 2048, 18, 42) 8192 block_4c_conv_3[0][0] __________________________________________________________________________________________________ add_16 (Add) (None, 2048, 18, 42) 0 block_4c_bn_3[0][0] block_4b_relu[0][0] __________________________________________________________________________________________________ block_4c_relu (Activation) (None, 2048, 18, 42) 0 add_16[0][0] __________________________________________________________________________________________________ output_bbox (Conv2D) (None, 140, 18, 42) 286860 block_4c_relu[0][0] __________________________________________________________________________________________________ output_cov (Conv2D) (None, 35, 18, 42) 71715 block_4c_relu[0][0] ================================================================================================== Total params: 23,946,287 Trainable params: 23,893,167 Non-trainable params: 53,120 __________________________________________________________________________________________________ 2020-10-27 01:36:57.244194: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: name: TITAN RTX major: 7 minor: 5 memoryClockRate(GHz): 1.77 pciBusID: 0000:86:00.0 2020-10-27 01:36:57.244324: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2020-10-27 01:36:57.244435: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2020-10-27 01:36:57.244492: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2020-10-27 01:36:57.244546: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2020-10-27 01:36:57.244598: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2020-10-27 01:36:57.244651: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2020-10-27 01:36:57.244703: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2020-10-27 01:36:57.247803: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0 2020-10-27 01:36:57.252294: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix: 2020-10-27 01:36:57.252355: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0 2020-10-27 01:36:57.252393: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N 2020-10-27 01:36:57.255772: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 22394 MB memory) -> physical GPU (device: 0, name: TITAN RTX, pci bus id: 0000:86:00.0, compute capability: 7.5) 2020-10-27 01:37:01,760 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 35, 0.00s/step 2020-10-27 01:37:03.857009: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2020-10-27 01:37:03.858781: I tensorflow/core/kernels/cuda_solvers.cc:159] Creating CudaSolver handles for stream 0x28d81db0 2020-10-27 01:37:03.858952: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2020-10-27 01:37:05.388626: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2020-10-27 01:37:06.447184: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2020-10-27 01:37:19,311 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 35, 1.76s/step 2020-10-27 01:37:30,724 [INFO] iva.detectnet_v2.evaluation.evaluation: step 20 / 35, 1.14s/step 2020-10-27 01:37:41,982 [INFO] iva.detectnet_v2.evaluation.evaluation: step 30 / 35, 1.13s/step Matching predictions to ground truth, class 1/35.: 100%|█| 870/870 [00:00<00:00, 12677.38it/s] Matching predictions to ground truth, class 2/35.: 100%|█| 1011/1011 [00:00<00:00, 16557.56it/s] Matching predictions to ground truth, class 3/35.: 100%|█| 919/919 [00:00<00:00, 15081.23it/s] Matching predictions to ground truth, class 4/35.: 100%|█| 823/823 [00:00<00:00, 14169.13it/s] Matching predictions to ground truth, class 5/35.: 100%|█| 1209/1209 [00:00<00:00, 18445.45it/s] Matching predictions to ground truth, class 7/35.: 100%|█| 1206/1206 [00:00<00:00, 18780.74it/s] Matching predictions to ground truth, class 8/35.: 100%|█| 1112/1112 [00:00<00:00, 13668.99it/s] Matching predictions to ground truth, class 9/35.: 100%|█| 727/727 [00:00<00:00, 12960.84it/s] Matching predictions to ground truth, class 10/35.: 100%|█| 870/870 [00:00<00:00, 13836.55it/s] Matching predictions to ground truth, class 11/35.: 100%|█| 905/905 [00:00<00:00, 16231.69it/s] Matching predictions to ground truth, class 14/35.: 100%|█| 846/846 [00:00<00:00, 25922.73it/s] Matching predictions to ground truth, class 15/35.: 100%|█| 709/709 [00:00<00:00, 33199.31it/s] Matching predictions to ground truth, class 17/35.: 100%|█| 769/769 [00:00<00:00, 31319.62it/s] Matching predictions to ground truth, class 18/35.: 100%|█| 822/822 [00:00<00:00, 31516.52it/s] Matching predictions to ground truth, class 20/35.: 100%|█| 20/20 [00:00<00:00, 35499.82it/s] Matching predictions to ground truth, class 21/35.: 100%|█| 1297/1297 [00:00<00:00, 22115.40it/s] Matching predictions to ground truth, class 22/35.: 100%|█| 667/667 [00:00<00:00, 15162.33it/s] Matching predictions to ground truth, class 23/35.: 100%|█| 735/735 [00:00<00:00, 36072.33it/s] Matching predictions to ground truth, class 24/35.: 100%|█| 526/526 [00:00<00:00, 39268.87it/s] Matching predictions to ground truth, class 27/35.: 100%|█| 340/340 [00:00<00:00, 43583.84it/s] Matching predictions to ground truth, class 29/35.: 100%|█| 616/616 [00:00<00:00, 28991.15it/s] Matching predictions to ground truth, class 32/35.: 100%|█| 830/830 [00:00<00:00, 32609.62it/s] Matching predictions to ground truth, class 33/35.: 100%|█| 494/494 [00:00<00:00, 24484.33it/s] Matching predictions to ground truth, class 35/35.: 100%|█| 291/291 [00:00<00:00, 17834.28it/s] Validation cost: 0.000440 Mean average_precision (in %): 44.8783 class name average precision (in %) ------------ -------------------------- a 99.6147 a0 96.587 a1 98.3116 a2 98.8142 a3 97.6739 a4 0 a5 98.2325 a6 99.2009 a7 99.3841 a8 97.9869 a9 98.7267 b 0 c 0 d 27.5937 e 11.4607 f 0 g 13.6759 h 34.9136 i 0 j 0 k 87.8861 l 98.5676 m 12.8376 n 1.43191 p 0 q 0 r 0.016835 s 0 t 86.7987 u 0 v 0 w 17.7831 x 96.9121 y 0 z 96.3308 Median Inference Time: 0.007454 2020-10-27 01:37:49,635 [INFO] iva.detectnet_v2.scripts.evaluate: Evaluation complete. Time taken to run iva.detectnet_v2.scripts.evaluate:main: 0:01:07.104787.