Error on tlt-training detectnet_v2?

ai12 · June 15, 2020, 3:29am

Hello everyone.
I’m having a little trouble while running tlt-train for the detectnet_v2 net example.

Error happens when I run this cell

!tlt-train detectnet_v2 -e $SPECS_DIR/detectnet_v2_train_resnet18_kitti.txt \
                        -r $USER_EXPERIMENT_DIR/experiment_dir_unpruned \
                        -k $KEY \
                        -n resnet18_detector

what I’m getting is the following

Using TensorFlow backend.
2020-06-15 02:32:12,974 [INFO] iva.detectnet_v2.scripts.train: Loading experiment spec at /workspace/examples/detectnet_v2/specs/detectnet_v2_train_resnet18_kitti.txt.
2020-06-15 02:32:12,975 [INFO] iva.detectnet_v2.spec_handler.spec_loader: Merging specification from /workspace/examples/detectnet_v2/specs/detectnet_v2_train_resnet18_kitti.txt
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
input_1 (InputLayer)            (None, 3, 188, 621)  0                                            
__________________________________________________________________________________________________
conv1 (Conv2D)                  (None, 64, 94, 311)  9472        input_1[0][0]                    
__________________________________________________________________________________________________
bn_conv1 (BatchNormalization)   (None, 64, 94, 311)  256         conv1[0][0]                      
__________________________________________________________________________________________________
activation_1 (Activation)       (None, 64, 94, 311)  0           bn_conv1[0][0]                   
__________________________________________________________________________________________________
block_1a_conv_1 (Conv2D)        (None, 64, 47, 156)  36928       activation_1[0][0]               
__________________________________________________________________________________________________
block_1a_bn_1 (BatchNormalizati (None, 64, 47, 156)  256         block_1a_conv_1[0][0]            
__________________________________________________________________________________________________
block_1a_relu_1 (Activation)    (None, 64, 47, 156)  0           block_1a_bn_1[0][0]              
__________________________________________________________________________________________________
block_1a_conv_2 (Conv2D)        (None, 64, 47, 156)  36928       block_1a_relu_1[0][0]            
__________________________________________________________________________________________________
block_1a_conv_shortcut (Conv2D) (None, 64, 47, 156)  4160        activation_1[0][0]               
__________________________________________________________________________________________________
block_1a_bn_2 (BatchNormalizati (None, 64, 47, 156)  256         block_1a_conv_2[0][0]            
__________________________________________________________________________________________________
block_1a_bn_shortcut (BatchNorm (None, 64, 47, 156)  256         block_1a_conv_shortcut[0][0]     
__________________________________________________________________________________________________
add_1 (Add)                     (None, 64, 47, 156)  0           block_1a_bn_2[0][0]              
                                                                 block_1a_bn_shortcut[0][0]       
__________________________________________________________________________________________________
block_1a_relu (Activation)      (None, 64, 47, 156)  0           add_1[0][0]                      
__________________________________________________________________________________________________
block_1b_conv_1 (Conv2D)        (None, 64, 47, 156)  36928       block_1a_relu[0][0]              
__________________________________________________________________________________________________
block_1b_bn_1 (BatchNormalizati (None, 64, 47, 156)  256         block_1b_conv_1[0][0]            
__________________________________________________________________________________________________
block_1b_relu_1 (Activation)    (None, 64, 47, 156)  0           block_1b_bn_1[0][0]              
__________________________________________________________________________________________________
block_1b_conv_2 (Conv2D)        (None, 64, 47, 156)  36928       block_1b_relu_1[0][0]            
__________________________________________________________________________________________________
block_1b_bn_2 (BatchNormalizati (None, 64, 47, 156)  256         block_1b_conv_2[0][0]            
__________________________________________________________________________________________________
add_2 (Add)                     (None, 64, 47, 156)  0           block_1b_bn_2[0][0]              
                                                                 block_1a_relu[0][0]              
__________________________________________________________________________________________________
block_1b_relu (Activation)      (None, 64, 47, 156)  0           add_2[0][0]                      
__________________________________________________________________________________________________
block_2a_conv_1 (Conv2D)        (None, 128, 24, 78)  73856       block_1b_relu[0][0]              
__________________________________________________________________________________________________
block_2a_bn_1 (BatchNormalizati (None, 128, 24, 78)  512         block_2a_conv_1[0][0]            
__________________________________________________________________________________________________
block_2a_relu_1 (Activation)    (None, 128, 24, 78)  0           block_2a_bn_1[0][0]              
__________________________________________________________________________________________________
block_2a_conv_2 (Conv2D)        (None, 128, 24, 78)  147584      block_2a_relu_1[0][0]            
__________________________________________________________________________________________________
block_2a_conv_shortcut (Conv2D) (None, 128, 24, 78)  8320        block_1b_relu[0][0]              
__________________________________________________________________________________________________
block_2a_bn_2 (BatchNormalizati (None, 128, 24, 78)  512         block_2a_conv_2[0][0]            
__________________________________________________________________________________________________
block_2a_bn_shortcut (BatchNorm (None, 128, 24, 78)  512         block_2a_conv_shortcut[0][0]     
__________________________________________________________________________________________________
add_3 (Add)                     (None, 128, 24, 78)  0           block_2a_bn_2[0][0]              
                                                                 block_2a_bn_shortcut[0][0]       
__________________________________________________________________________________________________
block_2a_relu (Activation)      (None, 128, 24, 78)  0           add_3[0][0]                      
__________________________________________________________________________________________________
block_2b_conv_1 (Conv2D)        (None, 128, 24, 78)  147584      block_2a_relu[0][0]              
__________________________________________________________________________________________________
block_2b_bn_1 (BatchNormalizati (None, 128, 24, 78)  512         block_2b_conv_1[0][0]            
__________________________________________________________________________________________________
block_2b_relu_1 (Activation)    (None, 128, 24, 78)  0           block_2b_bn_1[0][0]              
__________________________________________________________________________________________________
block_2b_conv_2 (Conv2D)        (None, 128, 24, 78)  147584      block_2b_relu_1[0][0]            
__________________________________________________________________________________________________
block_2b_bn_2 (BatchNormalizati (None, 128, 24, 78)  512         block_2b_conv_2[0][0]            
__________________________________________________________________________________________________
add_4 (Add)                     (None, 128, 24, 78)  0           block_2b_bn_2[0][0]              
                                                                 block_2a_relu[0][0]              
__________________________________________________________________________________________________
block_2b_relu (Activation)      (None, 128, 24, 78)  0           add_4[0][0]                      
__________________________________________________________________________________________________
block_3a_conv_1 (Conv2D)        (None, 256, 12, 39)  295168      block_2b_relu[0][0]              
__________________________________________________________________________________________________
block_3a_bn_1 (BatchNormalizati (None, 256, 12, 39)  1024        block_3a_conv_1[0][0]            
__________________________________________________________________________________________________
block_3a_relu_1 (Activation)    (None, 256, 12, 39)  0           block_3a_bn_1[0][0]              
__________________________________________________________________________________________________
block_3a_conv_2 (Conv2D)        (None, 256, 12, 39)  590080      block_3a_relu_1[0][0]            
__________________________________________________________________________________________________
block_3a_conv_shortcut (Conv2D) (None, 256, 12, 39)  33024       block_2b_relu[0][0]              
__________________________________________________________________________________________________
block_3a_bn_2 (BatchNormalizati (None, 256, 12, 39)  1024        block_3a_conv_2[0][0]            
__________________________________________________________________________________________________
block_3a_bn_shortcut (BatchNorm (None, 256, 12, 39)  1024        block_3a_conv_shortcut[0][0]     
__________________________________________________________________________________________________
add_5 (Add)                     (None, 256, 12, 39)  0           block_3a_bn_2[0][0]              
                                                                 block_3a_bn_shortcut[0][0]       
__________________________________________________________________________________________________
block_3a_relu (Activation)      (None, 256, 12, 39)  0           add_5[0][0]                      
__________________________________________________________________________________________________
block_3b_conv_1 (Conv2D)        (None, 256, 12, 39)  590080      block_3a_relu[0][0]              
__________________________________________________________________________________________________
block_3b_bn_1 (BatchNormalizati (None, 256, 12, 39)  1024        block_3b_conv_1[0][0]            
__________________________________________________________________________________________________
block_3b_relu_1 (Activation)    (None, 256, 12, 39)  0           block_3b_bn_1[0][0]              
__________________________________________________________________________________________________
block_3b_conv_2 (Conv2D)        (None, 256, 12, 39)  590080      block_3b_relu_1[0][0]            
__________________________________________________________________________________________________
block_3b_bn_2 (BatchNormalizati (None, 256, 12, 39)  1024        block_3b_conv_2[0][0]            
__________________________________________________________________________________________________
add_6 (Add)                     (None, 256, 12, 39)  0           block_3b_bn_2[0][0]              
                                                                 block_3a_relu[0][0]              
__________________________________________________________________________________________________
block_3b_relu (Activation)      (None, 256, 12, 39)  0           add_6[0][0]                      
__________________________________________________________________________________________________
block_4a_conv_1 (Conv2D)        (None, 512, 12, 39)  1180160     block_3b_relu[0][0]              
__________________________________________________________________________________________________
block_4a_bn_1 (BatchNormalizati (None, 512, 12, 39)  2048        block_4a_conv_1[0][0]            
__________________________________________________________________________________________________
block_4a_relu_1 (Activation)    (None, 512, 12, 39)  0           block_4a_bn_1[0][0]              
__________________________________________________________________________________________________
block_4a_conv_2 (Conv2D)        (None, 512, 12, 39)  2359808     block_4a_relu_1[0][0]            
__________________________________________________________________________________________________
block_4a_conv_shortcut (Conv2D) (None, 512, 12, 39)  131584      block_3b_relu[0][0]              
__________________________________________________________________________________________________
block_4a_bn_2 (BatchNormalizati (None, 512, 12, 39)  2048        block_4a_conv_2[0][0]            
__________________________________________________________________________________________________
block_4a_bn_shortcut (BatchNorm (None, 512, 12, 39)  2048        block_4a_conv_shortcut[0][0]     
__________________________________________________________________________________________________
add_7 (Add)                     (None, 512, 12, 39)  0           block_4a_bn_2[0][0]              
                                                                 block_4a_bn_shortcut[0][0]       
__________________________________________________________________________________________________
block_4a_relu (Activation)      (None, 512, 12, 39)  0           add_7[0][0]                      
__________________________________________________________________________________________________
block_4b_conv_1 (Conv2D)        (None, 512, 12, 39)  2359808     block_4a_relu[0][0]              
__________________________________________________________________________________________________
block_4b_bn_1 (BatchNormalizati (None, 512, 12, 39)  2048        block_4b_conv_1[0][0]            
__________________________________________________________________________________________________
block_4b_relu_1 (Activation)    (None, 512, 12, 39)  0           block_4b_bn_1[0][0]              
__________________________________________________________________________________________________
block_4b_conv_2 (Conv2D)        (None, 512, 12, 39)  2359808     block_4b_relu_1[0][0]            
__________________________________________________________________________________________________
block_4b_bn_2 (BatchNormalizati (None, 512, 12, 39)  2048        block_4b_conv_2[0][0]            
__________________________________________________________________________________________________
add_8 (Add)                     (None, 512, 12, 39)  0           block_4b_bn_2[0][0]              
                                                                 block_4a_relu[0][0]              
__________________________________________________________________________________________________
block_4b_relu (Activation)      (None, 512, 12, 39)  0           add_8[0][0]                      
__________________________________________________________________________________________________
output_bbox (Conv2D)            (None, 12, 12, 39)   6156        block_4b_relu[0][0]              
__________________________________________________________________________________________________
output_cov (Conv2D)             (None, 3, 12, 39)    1539        block_4b_relu[0][0]              
==================================================================================================
Total params: 11,203,023
Trainable params: 11,193,295
Non-trainable params: 9,728
__________________________________________________________________________________________________
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
2020-06-15 02:32:26,924 [INFO] iva.detectnet_v2.scripts.train: Found 6434 samples in training set
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
2020-06-15 02:32:30,459 [INFO] iva.detectnet_v2.scripts.train: Found 1047 samples in validation set
2020-06-15 02:33:03,304 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 0/120: loss: 0.12597 Time taken: 0:00:00 ETA: 0:00:00
2020-06-15 02:33:11,378 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 1.794
2020-06-15 02:33:14,003 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.522
2020-06-15 02:33:16,619 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.560
2020-06-15 02:33:19,251 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.498
2020-06-15 02:33:21,871 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.546
2020-06-15 02:33:24,479 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.586
2020-06-15 02:33:27,088 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.583
2020-06-15 02:33:29,696 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.586
2020-06-15 02:33:32,302 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.596
2020-06-15 02:33:34,906 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.599
2020-06-15 02:33:37,520 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.565
2020-06-15 02:33:40,128 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.589
2020-06-15 02:33:42,732 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.599
2020-06-15 02:33:45,389 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.413
2020-06-15 02:33:48,075 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.307
2020-06-15 02:33:50,734 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.402

… and goes on like this undefinedly.
Is this normal? When I ran the Faster RCNN example, the output looked like a typical keras training, but here I’m not really sure of what is happening.
Might the error be that my card isn’t powerfull enought, even when I took some actions to lower the burden on my GPU?

Thank you in advance, any help is appreciated!

=========== SOME MORE INFO ===========
I’m using almost the same dataset provided as example, but I resized all images to half height and width, and then I did the math on the labels too.
I ran other examples before, so it is not related to not knowing how to configure the key or the docker image.
I changed every batch size to 1
I changed the config size data augmentation to match the new images size and the evaluation_bx_config too
I am running the tlt on the docker image provided.

=========== CONFIG FILE ===============

random_seed: 42
dataset_config {
  data_sources {
    tfrecords_path: "/workspace/tlt-experiments/data/tfrecords/kitti_trainval/*"
    image_directory_path: "/workspace/tlt-experiments/data/training"
  }
  image_extension: "png"
  target_class_mapping {
    key: "car"
    value: "car"
  }
  target_class_mapping {
    key: "cyclist"
    value: "cyclist"
  }
  target_class_mapping {
    key: "pedestrian"
    value: "pedestrian"
  }
  target_class_mapping {
    key: "person_sitting"
    value: "pedestrian"
  }
  target_class_mapping {
    key: "van"
    value: "car"
  }
  validation_fold: 0
}
augmentation_config {
  preprocessing {
    output_image_width: 621 #1248
    output_image_height: 188 #384
    min_bbox_width: 1.0
    min_bbox_height: 1.0
    output_image_channel: 3
  }
  spatial_augmentation {
    hflip_probability: 0.5
    zoom_min: 1.0
    zoom_max: 1.0
    translate_max_x: 8.0
    translate_max_y: 8.0
  }
  color_augmentation {
    hue_rotation_max: 25.0
    saturation_shift_max: 0.20000000298
    contrast_scale_max: 0.10000000149
    contrast_center: 0.5
  }
}
postprocessing_config {
  target_class_config {
    key: "car"
    value {
      clustering_config {
        coverage_threshold: 0.00499999988824
        dbscan_eps: 0.20000000298
        dbscan_min_samples: 0.0500000007451
        minimum_bounding_box_height: 20
      }
    }
  }
  target_class_config {
    key: "cyclist"
    value {
      clustering_config {
        coverage_threshold: 0.00499999988824
        dbscan_eps: 0.15000000596
        dbscan_min_samples: 0.0500000007451
        minimum_bounding_box_height: 20
      }
    }
  }
  target_class_config {
    key: "pedestrian"
    value {
      clustering_config {
        coverage_threshold: 0.00749999983236
        dbscan_eps: 0.230000004172
        dbscan_min_samples: 0.0500000007451
        minimum_bounding_box_height: 20
      }
    }
  }
}
model_config {
  pretrained_model_file: "/workspace/tlt-experiments/detectnet_v2/pretrained_resnet18/tlt_pretrained_detectnet_v2_vresnet18/resnet18.hdf5"
  num_layers: 18
  use_batch_norm: true
  objective_set {
    bbox {
      scale: 35.0
      offset: 0.5
    }
    cov {
    }
  }
  training_precision {
    backend_floatx: FLOAT32
  }
  arch: "resnet"
}
evaluation_config {
  validation_period_during_training: 10
  first_validation_epoch: 30
  minimum_detection_ground_truth_overlap {
    key: "car"
    value: 0.699999988079
  }
  minimum_detection_ground_truth_overlap {
    key: "cyclist"
    value: 0.5
  }
  minimum_detection_ground_truth_overlap {
    key: "pedestrian"
    value: 0.5
  }
  evaluation_box_config {
    key: "car"
    value {
      minimum_height: 9 #20
      maximum_height: 9999
      minimum_width: 4 #10
      maximum_width: 9999
    }
  }
  evaluation_box_config {
    key: "cyclist"
    value {
      minimum_height: 9 #20
      maximum_height: 9999
      minimum_width: 4 #10
      maximum_width: 9999
    }
  }
  evaluation_box_config {
    key: "pedestrian"
    value {
      minimum_height: 9 #20
      maximum_height: 9999
      minimum_width: 4 #10
      maximum_width: 9999
    }
  }
  average_precision_mode: INTEGRATE
}
cost_function_config {
  target_classes {
    name: "car"
    class_weight: 1.0
    coverage_foreground_weight: 0.0500000007451
    objectives {
      name: "cov"
      initial_weight: 1.0
      weight_target: 1.0
    }
    objectives {
      name: "bbox"
      initial_weight: 10.0
      weight_target: 10.0
    }
  }
  target_classes {
    name: "cyclist"
    class_weight: 8.0
    coverage_foreground_weight: 0.0500000007451
    objectives {
      name: "cov"
      initial_weight: 1.0
      weight_target: 1.0
    }
    objectives {
      name: "bbox"
      initial_weight: 10.0
      weight_target: 1.0
    }
  }
  target_classes {
    name: "pedestrian"
    class_weight: 4.0
    coverage_foreground_weight: 0.0500000007451
    objectives {
      name: "cov"
      initial_weight: 1.0
      weight_target: 1.0
    }
    objectives {
      name: "bbox"
      initial_weight: 10.0
      weight_target: 10.0
    }
  }
  enable_autoweighting: true
  max_objective_weight: 0.999899983406
  min_objective_weight: 9.99999974738e-05
}
training_config {
  batch_size_per_gpu: 1 #4
  num_epochs: 120
  learning_rate {
    soft_start_annealing_schedule {
      min_learning_rate: 5e-06
      max_learning_rate: 5e-04
      soft_start: 0.10000000149
      annealing: 0.699999988079
    }
  }
  regularizer {
    type: L1
    weight: 3.00000002618e-09
  }
  optimizer {
    adam {
      epsilon: 9.99999993923e-09
      beta1: 0.899999976158
      beta2: 0.999000012875
    }
  }
  cost_scaling {
    initial_exponent: 20.0
    increment: 0.005
    decrement: 1.0
  }
  checkpoint_interval: 10
}
bbox_rasterizer_config {
  target_class_config {
    key: "car"
    value {
      cov_center_x: 0.5
      cov_center_y: 0.5
      cov_radius_x: 0.40000000596
      cov_radius_y: 0.40000000596
      bbox_min_radius: 1.0
    }
  }
  target_class_config {
    key: "cyclist"
    value {
      cov_center_x: 0.5
      cov_center_y: 0.5
      cov_radius_x: 1.0
      cov_radius_y: 1.0
      bbox_min_radius: 1.0
    }
  }
  target_class_config {
    key: "pedestrian"
    value {
      cov_center_x: 0.5
      cov_center_y: 0.5
      cov_radius_x: 1.0
      cov_radius_y: 1.0
      bbox_min_radius: 1.0
    }
  }
  deadzone_radius: 0.400000154972
}

=========== HARDWARE DETAILS ===========

 Machine:   Device: laptop System: Dell product: Inspiron 7559 v: 1.2.9 serial: N/A
            UEFI: Dell v: 1.2.9 date: 09/03/2018
 CPU:       Quad core Intel Core i7-6700HQ (-MT-MCP-) cache: 6144 KB
            clock speeds: max: 3500 MHz 1: 1513 MHz 2: 2220 MHz 3: 1842 MHz
            4: 1430 MHz 5: 1860 MHz 6: 1892 MHz 7: 1971 MHz 8: 1995 MHz
 Graphics:  Card-1: Intel HD Graphics 530
            Card-2: NVIDIA GM107M [GeForce GTX 960M]
            Card-2 memory: 2 GB GDDR5 5GHz 128 bits
            Display Server: x11 (X.Org 1.20.5 )
            drivers: modesetting,nvidia (unloaded: fbdev,vesa,nouveau)
            Resolution: 1920x1080@60.02hz
            OpenGL: renderer: GeForce GTX 960M/PCIe/SSE2
            version: 4.6.0 NVIDIA 440.82
 Drives:    HDD Total Size: 750.2GB (14.9% used)
            ID-1: /dev/sda model: Samsung_SSD_860 size: 250.1GB
            ID-2: /dev/sdb model: CT500MX500SSD1 size: 500.1GB
 Info:      Processes: 365 Uptime: 3:00 Memory: 3920.5/7827.7MB
            Client: Shell (bash) inxi: 2.3.56

=========== SOFTWARE DETAILS ===========

  System:    Host: jpablo-Inspiron-7559 Kernel: 5.3.0-53-generic x86_64
             bits: 64
             Desktop: Gnome 3.28.4 Distro: Ubuntu 18.04.4 LTS
  Docker:    Server Version: 19.03.8
  	         Image repository: nvcr.io/nvidia/tlt-streamanalytics
  	         Image tag: v2.0_dp_py2
  Nvidia:    Driver Version: 440.82
  	         CUDA Version 10.2.89
  Machine:   Device: laptop System: Dell product: Inspiron 7559 v: 1.2.9 serial: N/A
             UEFI: Dell v: 1.2.9 date: 09/03/2018

Morganh · June 15, 2020, 6:59am

I cannot see the error in your log. Could you elaborate more?

More, for detectnet_v2, see https://docs.nvidia.com/metropolis/TLT/tlt-getting-started-guide/index.html#requirements, your setting does not match W > =480, H >=272 and W, H are multiples of 16

DetectNet_v2
Input size: C * W * H (where C = 1 or 3, W > =480, H >=272 and W, H are multiples of 16)
Image format: JPG, JPEG, PNG
Label format: KITTI detection
Note: The tlt-train tool does not support training on images of multiple resolutions, or resizing images during training. All of the images must be resized offline to the final training size and the corresponding bounding boxes must be scaled accordingly.

ai12 · June 15, 2020, 2:47pm

Thank you for your your help @Morganh, but the “problem” is still going on.

I was aware of that and checked that resizing the images to half their size would fit the restriction… then I just didn’t notice my new dimensions were not exactly half of the original height and width.

I resized the images properly now to 624 x 192 instead of 621 x 188.

I got a better output now, it seems like working, but it still looks strange.
After the net description and before of that it looks like a proper training, i get a lot of lines like the one below as output… Is that normal, can I just ignore it, or does that line means something odd is happening?

2020-06-15 14:27:31,851 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.077

Here below is the full output (I removed some redundant lines becouse of forum limits n coments, but you can get the idea).

Using TensorFlow backend.
2020-06-15 14:10:29,523 [INFO] iva.detectnet_v2.scripts.train: Loading experiment spec at /workspace/examples/detectnet_v2/specs/detectnet_v2_train_resnet18_kitti.txt.
2020-06-15 14:10:29,524 [INFO] iva.detectnet_v2.spec_handler.spec_loader: Merging specification from /workspace/examples/detectnet_v2/specs/detectnet_v2_train_resnet18_kitti.txt
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
input_1 (InputLayer)            (None, 3, 192, 624)  0                                            
__________________________________________________________________________________________________
conv1 (Conv2D)                  (None, 64, 96, 312)  9472        input_1[0][0]                    
__________________________________________________________________________________________________
bn_conv1 (BatchNormalization)   (None, 64, 96, 312)  256         conv1[0][0]                      
__________________________________________________________________________________________________
activation_1 (Activation)       (None, 64, 96, 312)  0           bn_conv1[0][0]                   
__________________________________________________________________________________________________
block_1a_conv_1 (Conv2D)        (None, 64, 48, 156)  36928       activation_1[0][0]               
__________________________________________________________________________________________________
block_1a_bn_1 (BatchNormalizati (None, 64, 48, 156)  256         block_1a_conv_1[0][0]            
__________________________________________________________________________________________________
block_1a_relu_1 (Activation)    (None, 64, 48, 156)  0           block_1a_bn_1[0][0]              
__________________________________________________________________________________________________
block_1a_conv_2 (Conv2D)        (None, 64, 48, 156)  36928       block_1a_relu_1[0][0]            
__________________________________________________________________________________________________
block_1a_conv_shortcut (Conv2D) (None, 64, 48, 156)  4160        activation_1[0][0]               
__________________________________________________________________________________________________
block_1a_bn_2 (BatchNormalizati (None, 64, 48, 156)  256         block_1a_conv_2[0][0]            
__________________________________________________________________________________________________
block_1a_bn_shortcut (BatchNorm (None, 64, 48, 156)  256         block_1a_conv_shortcut[0][0]     
__________________________________________________________________________________________________
add_1 (Add)                     (None, 64, 48, 156)  0           block_1a_bn_2[0][0]              
                                                                 block_1a_bn_shortcut[0][0]       
__________________________________________________________________________________________________
block_1a_relu (Activation)      (None, 64, 48, 156)  0           add_1[0][0]                      
__________________________________________________________________________________________________
block_1b_conv_1 (Conv2D)        (None, 64, 48, 156)  36928       block_1a_relu[0][0]              
__________________________________________________________________________________________________
block_1b_bn_1 (BatchNormalizati (None, 64, 48, 156)  256         block_1b_conv_1[0][0]            
__________________________________________________________________________________________________
block_1b_relu_1 (Activation)    (None, 64, 48, 156)  0           block_1b_bn_1[0][0]              
__________________________________________________________________________________________________
block_1b_conv_2 (Conv2D)        (None, 64, 48, 156)  36928       block_1b_relu_1[0][0]            
__________________________________________________________________________________________________
block_1b_bn_2 (BatchNormalizati (None, 64, 48, 156)  256         block_1b_conv_2[0][0]            
__________________________________________________________________________________________________
add_2 (Add)                     (None, 64, 48, 156)  0           block_1b_bn_2[0][0]              
                                                                 block_1a_relu[0][0]              
__________________________________________________________________________________________________
block_1b_relu (Activation)      (None, 64, 48, 156)  0           add_2[0][0]                      
__________________________________________________________________________________________________
block_2a_conv_1 (Conv2D)        (None, 128, 24, 78)  73856       block_1b_relu[0][0]              
__________________________________________________________________________________________________
block_2a_bn_1 (BatchNormalizati (None, 128, 24, 78)  512         block_2a_conv_1[0][0]            
__________________________________________________________________________________________________
block_2a_relu_1 (Activation)    (None, 128, 24, 78)  0           block_2a_bn_1[0][0]              
__________________________________________________________________________________________________
block_2a_conv_2 (Conv2D)        (None, 128, 24, 78)  147584      block_2a_relu_1[0][0]            
__________________________________________________________________________________________________
block_2a_conv_shortcut (Conv2D) (None, 128, 24, 78)  8320        block_1b_relu[0][0]              
__________________________________________________________________________________________________
block_2a_bn_2 (BatchNormalizati (None, 128, 24, 78)  512         block_2a_conv_2[0][0]            
__________________________________________________________________________________________________
block_2a_bn_shortcut (BatchNorm (None, 128, 24, 78)  512         block_2a_conv_shortcut[0][0]     
__________________________________________________________________________________________________
add_3 (Add)                     (None, 128, 24, 78)  0           block_2a_bn_2[0][0]              
                                                                 block_2a_bn_shortcut[0][0]       
__________________________________________________________________________________________________
block_2a_relu (Activation)      (None, 128, 24, 78)  0           add_3[0][0]                      
__________________________________________________________________________________________________
block_2b_conv_1 (Conv2D)        (None, 128, 24, 78)  147584      block_2a_relu[0][0]              
__________________________________________________________________________________________________
block_2b_bn_1 (BatchNormalizati (None, 128, 24, 78)  512         block_2b_conv_1[0][0]            
__________________________________________________________________________________________________
block_2b_relu_1 (Activation)    (None, 128, 24, 78)  0           block_2b_bn_1[0][0]              
__________________________________________________________________________________________________
block_2b_conv_2 (Conv2D)        (None, 128, 24, 78)  147584      block_2b_relu_1[0][0]            
__________________________________________________________________________________________________
block_2b_bn_2 (BatchNormalizati (None, 128, 24, 78)  512         block_2b_conv_2[0][0]            
__________________________________________________________________________________________________
add_4 (Add)                     (None, 128, 24, 78)  0           block_2b_bn_2[0][0]              
                                                                 block_2a_relu[0][0]              
__________________________________________________________________________________________________
block_2b_relu (Activation)      (None, 128, 24, 78)  0           add_4[0][0]                      
__________________________________________________________________________________________________
block_3a_conv_1 (Conv2D)        (None, 256, 12, 39)  295168      block_2b_relu[0][0]              
__________________________________________________________________________________________________
block_3a_bn_1 (BatchNormalizati (None, 256, 12, 39)  1024        block_3a_conv_1[0][0]            
__________________________________________________________________________________________________
block_3a_relu_1 (Activation)    (None, 256, 12, 39)  0           block_3a_bn_1[0][0]              
__________________________________________________________________________________________________
block_3a_conv_2 (Conv2D)        (None, 256, 12, 39)  590080      block_3a_relu_1[0][0]            
__________________________________________________________________________________________________
block_3a_conv_shortcut (Conv2D) (None, 256, 12, 39)  33024       block_2b_relu[0][0]              
__________________________________________________________________________________________________
block_3a_bn_2 (BatchNormalizati (None, 256, 12, 39)  1024        block_3a_conv_2[0][0]            
__________________________________________________________________________________________________
block_3a_bn_shortcut (BatchNorm (None, 256, 12, 39)  1024        block_3a_conv_shortcut[0][0]     
__________________________________________________________________________________________________
add_5 (Add)                     (None, 256, 12, 39)  0           block_3a_bn_2[0][0]              
                                                                 block_3a_bn_shortcut[0][0]       
__________________________________________________________________________________________________
block_3a_relu (Activation)      (None, 256, 12, 39)  0           add_5[0][0]                      
__________________________________________________________________________________________________
block_3b_conv_1 (Conv2D)        (None, 256, 12, 39)  590080      block_3a_relu[0][0]              
__________________________________________________________________________________________________
block_3b_bn_1 (BatchNormalizati (None, 256, 12, 39)  1024        block_3b_conv_1[0][0]            
__________________________________________________________________________________________________
block_3b_relu_1 (Activation)    (None, 256, 12, 39)  0           block_3b_bn_1[0][0]              
__________________________________________________________________________________________________
block_3b_conv_2 (Conv2D)        (None, 256, 12, 39)  590080      block_3b_relu_1[0][0]            
__________________________________________________________________________________________________
block_3b_bn_2 (BatchNormalizati (None, 256, 12, 39)  1024        block_3b_conv_2[0][0]            
__________________________________________________________________________________________________
add_6 (Add)                     (None, 256, 12, 39)  0           block_3b_bn_2[0][0]              
                                                                 block_3a_relu[0][0]              
__________________________________________________________________________________________________
block_3b_relu (Activation)      (None, 256, 12, 39)  0           add_6[0][0]                      
__________________________________________________________________________________________________
block_4a_conv_1 (Conv2D)        (None, 512, 12, 39)  1180160     block_3b_relu[0][0]              
__________________________________________________________________________________________________
block_4a_bn_1 (BatchNormalizati (None, 512, 12, 39)  2048        block_4a_conv_1[0][0]            
__________________________________________________________________________________________________
block_4a_relu_1 (Activation)    (None, 512, 12, 39)  0           block_4a_bn_1[0][0]              
__________________________________________________________________________________________________
block_4a_conv_2 (Conv2D)        (None, 512, 12, 39)  2359808     block_4a_relu_1[0][0]            
__________________________________________________________________________________________________
block_4a_conv_shortcut (Conv2D) (None, 512, 12, 39)  131584      block_3b_relu[0][0]              
__________________________________________________________________________________________________
block_4a_bn_2 (BatchNormalizati (None, 512, 12, 39)  2048        block_4a_conv_2[0][0]            
__________________________________________________________________________________________________
block_4a_bn_shortcut (BatchNorm (None, 512, 12, 39)  2048        block_4a_conv_shortcut[0][0]     
__________________________________________________________________________________________________
add_7 (Add)                     (None, 512, 12, 39)  0           block_4a_bn_2[0][0]              
                                                                 block_4a_bn_shortcut[0][0]       
__________________________________________________________________________________________________
block_4a_relu (Activation)      (None, 512, 12, 39)  0           add_7[0][0]                      
__________________________________________________________________________________________________
block_4b_conv_1 (Conv2D)        (None, 512, 12, 39)  2359808     block_4a_relu[0][0]              
__________________________________________________________________________________________________
block_4b_bn_1 (BatchNormalizati (None, 512, 12, 39)  2048        block_4b_conv_1[0][0]            
__________________________________________________________________________________________________
block_4b_relu_1 (Activation)    (None, 512, 12, 39)  0           block_4b_bn_1[0][0]              
__________________________________________________________________________________________________
block_4b_conv_2 (Conv2D)        (None, 512, 12, 39)  2359808     block_4b_relu_1[0][0]            
__________________________________________________________________________________________________
block_4b_bn_2 (BatchNormalizati (None, 512, 12, 39)  2048        block_4b_conv_2[0][0]            
__________________________________________________________________________________________________
add_8 (Add)                     (None, 512, 12, 39)  0           block_4b_bn_2[0][0]              
                                                                 block_4a_relu[0][0]              
__________________________________________________________________________________________________
block_4b_relu (Activation)      (None, 512, 12, 39)  0           add_8[0][0]                      
__________________________________________________________________________________________________
output_bbox (Conv2D)            (None, 12, 12, 39)   6156        block_4b_relu[0][0]              
__________________________________________________________________________________________________
output_cov (Conv2D)             (None, 3, 12, 39)    1539        block_4b_relu[0][0]              
==================================================================================================
Total params: 11,203,023
Trainable params: 11,193,295
Non-trainable params: 9,728
__________________________________________________________________________________________________
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
2020-06-15 14:15:35,877 [INFO] iva.detectnet_v2.scripts.train: Found 6434 samples in training set
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
2020-06-15 14:15:40,262 [INFO] iva.detectnet_v2.scripts.train: Found 1047 samples in validation set
2020-06-15 14:16:16,725 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 0/2: loss: 0.12803 Time taken: 0:00:00 ETA: 0:00:00
2020-06-15 14:16:25,767 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 1.461
2020-06-15 14:16:28,643 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.692
2020-06-15 14:16:31,533 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.653
2020-06-15 14:16:34,423 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.652
2020-06-15 14:16:37,295 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.706
2020-06-15 14:16:40,152 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.751
2020-06-15 14:16:43,025 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.702
2020-06-15 14:16:45,913 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.659
2020-06-15 14:16:48,808 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.637
2020-06-15 14:16:51,696 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.656
2020-06-15 14:16:54,573 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.692
2020-06-15 14:16:57,446 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.701
2020-06-15 14:17:00,329 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.671
2020-06-15 14:17:03,254 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.548
2020-06-15 14:17:06,153 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.626
2020-06-15 14:17:09,115 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.439
2020-06-15 14:17:12,142 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.260
2020-06-15 14:17:15,196 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.188
2020-06-15 14:17:18,308 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.034
2020-06-15 14:17:21,207 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.623
2020-06-15 14:17:24,002 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.948
2020-06-15 14:17:26,717 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.209
2020-06-15 14:17:29,434 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.203
2020-06-15 14:17:32,182 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.096
2020-06-15 14:17:34,928 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.108
2020-06-15 14:17:40,889 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 4.194
2020-06-15 14:17:43,635 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.107
2020-06-15 14:17:46,380 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.107
2020-06-15 14:17:49,122 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.120
2020-06-15 14:17:51,812 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.294
2020-06-15 14:17:54,659 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.783
2020-06-15 14:17:57,426 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.035
2020-06-15 14:18:00,168 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.119
2020-06-15 14:18:02,935 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.036
2020-06-15 14:18:05,721 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.973
2020-06-15 14:18:08,449 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.166
2020-06-15 14:18:11,242 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.951
2020-06-15 14:18:13,994 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.085
2020-06-15 14:18:16,668 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.350
2020-06-15 14:18:19,462 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.948
2020-06-15 14:18:22,169 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.239
2020-06-15 14:18:24,915 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.105
2020-06-15 14:18:27,639 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.178
2020-06-15 14:18:30,414 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.011
2020-06-15 14:18:33,139 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.176
2020-06-15 14:18:35,895 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.072
2020-06-15 14:18:38,669 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.012
2020-06-15 14:18:41,398 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.164
2020-06-15 14:18:44,197 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.932
2020-06-15 14:18:46,895 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.267
2020-06-15 14:18:49,620 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.174
2020-06-15 14:18:52,353 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.147
2020-06-15 14:18:55,108 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.078
2020-06-15 14:18:57,829 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.187
2020-06-15 14:19:00,550 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.192
2020-06-15 14:19:03,379 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.835
2020-06-15 14:19:06,088 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.230
2020-06-15 14:19:08,822 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.147
2020-06-15 14:19:11,539 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.201
2020-06-15 14:19:14,352 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.887
2020-06-15 14:19:17,067 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.211
2020-06-15 14:19:19,779 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.217
2020-06-15 14:19:22,578 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.933
2020-06-15 14:19:25,359 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.992
2020-06-15 14:19:28,147 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.966
2020-06-15 14:19:30,869 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.186
2020-06-15 14:19:33,583 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.213
2020-06-15 14:19:36,327 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.113
2020-06-15 14:19:39,095 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.031
2020-06-15 14:19:41,819 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.179
2020-06-15 14:19:44,531 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.218
2020-06-15 14:19:47,290 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.062
2020-06-15 14:19:50,116 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.850
2020-06-15 14:19:52,838 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.183
2020-06-15 14:19:55,639 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.926
2020-06-15 14:19:58,375 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.141
2020-06-15 14:20:01,105 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.156
2020-06-15 14:20:03,827 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.188
2020-06-15 14:20:06,582 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.075
2020-06-15 14:20:09,337 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.074
2020-06-15 14:20:12,145 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.904
2020-06-15 14:20:14,894 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.097
2020-06-15 14:20:17,703 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.899
2020-06-15 14:20:20,435 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.151
2020-06-15 14:20:23,130 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.277
2020-06-15 14:20:25,931 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.928
2020-06-15 14:20:28,675 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.112
2020-06-15 14:20:31,399 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.178
2020-06-15 14:20:34,305 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.604
2020-06-15 14:20:37,086 [INFO] /usr/local/lib/python2.7/dist-
2020-06-15 14:25:41,285 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.647
2020-06-15 14:25:44,005 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.195
2020-06-15 14:25:46,750 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.106
2020-06-15 14:25:49,535 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.980
2020-06-15 14:25:52,263 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.164
2020-06-15 14:25:54,987 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.177
2020-06-15 14:25:57,745 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.068
2020-06-15 14:26:00,514 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.027
2020-06-15 14:26:03,228 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.215
2020-06-15 14:26:06,048 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.863
2020-06-15 14:26:08,800 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.088
2020-06-15 14:26:11,531 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.154
2020-06-15 14:26:14,256 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.174
2020-06-15 14:26:17,004 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.100
2020-06-15 14:26:19,735 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.156
2020-06-15 14:26:22,604 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.715
2020-06-15 14:26:25,447 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.793
2020-06-15 14:26:28,263 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.879
2020-06-15 14:26:30,994 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.156
2020-06-15 14:26:33,776 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.987
2020-06-15 14:26:36,522 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.104
2020-06-15 14:26:39,200 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.335
2020-06-15 14:26:41,981 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.993
2020-06-15 14:26:44,709 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.164
2020-06-15 14:26:47,449 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.125
2020-06-15 14:26:50,209 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.059
2020-06-15 14:26:52,953 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.113
2020-06-15 14:26:55,720 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.036
2020-06-15 14:26:58,540 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.864
2020-06-15 14:27:01,279 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.128
2020-06-15 14:27:04,003 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.179
2020-06-15 14:27:06,719 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.207
2020-06-15 14:27:09,536 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.875
2020-06-15 14:27:12,370 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.822
2020-06-15 14:27:15,176 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.909
2020-06-15 14:27:17,934 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.066
2020-06-15 14:27:20,750 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.881
2020-06-15 14:27:23,561 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.892
2020-06-15 14:27:26,390 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.838
2020-06-15 14:27:29,097 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.238
2020-06-15 14:27:31,851 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.077
2020-06-15 14:27:34,570 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.195
2020-06-15 14:27:37,377 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.907
2020-06-15 14:27:40,295 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.569
2020-06-15 14:27:43,217 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.554
2020-06-15 14:27:46,081 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.731
2020-06-15 14:27:48,876 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.945
2020-06-15 14:27:51,547 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.362
2020-06-15 14:27:54,333 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.975
2020-06-15 14:27:57,033 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.258
2020-06-15 14:27:59,767 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.147
2020-06-15 14:28:02,536 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.029
2020-06-15 14:28:05,304 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.031
2020-06-15 14:28:08,638 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 7.499
2020-06-15 14:28:11,526 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.659
2020-06-15 14:28:14,400 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.698
2020-06-15 14:28:17,419 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.283
2020-06-15 14:28:20,474 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.183
2020-06-15 14:28:23,369 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.636
2020-06-15 14:28:26,413 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.214
2020-06-15 14:28:27,760 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 1047, 0.00s/step
2020-06-15 14:28:28,992 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 1047, 0.12s/step
2020-06-15 14:28:29,404 [INFO] iva.detectnet_v2.evaluation.evaluation: step 20 / 1047, 0.04s/step
2020-06-15 14:28:29,864 [INFO] iva.detectnet_v2.evaluation.evaluation: step 30 / 1047, 0.05s/step
2020-06-15 14:28:30,328 [INFO] iva.detectnet_v2.evaluation.evaluation: step 40 / 1047, 0.05s/step
2020-06-15 14:28:30,770 [INFO] iva.detectnet_v2.evaluation.evaluation: step 50 / 1047, 0.04s/step
2020-06-15 14:28:31,277 [INFO] iva.detectnet_v2.evaluation.evaluation: step 60 / 1047, 0.05s/step
2020-06-15 14:28:31,738 [INFO] iva.detectnet_v2.evaluation.evaluation: step 70 / 1047, 0.05s/step
2020-06-15 14:28:32,140 [INFO] iva.detectnet_v2.evaluation.evaluation: step 80 / 1047, 0.04s/step
2020-06-15 14:28:32,544 [INFO] iva.detectnet_v2.evaluation.evaluation: step 90 / 1047, 0.04s/step
2020-06-15 14:28:32,963 [INFO] iva.detectnet_v2.evaluation.evaluation: step 100 / 1047, 0.04s/step
2020-06-15 14:28:33,359 [INFO] iva.detectnet_v2.evaluation.evaluation: step 110 / 1047, 0.04s/step
2020-06-15 14:28:33,755 [INFO] iva.detectnet_v2.evaluation.evaluation: step 120 / 1047, 0.04s/step
2020-06-15 14:28:34,129 [INFO] iva.detectnet_v2.evaluation.evaluation: step 130 / 1047, 0.04s/step
2020-06-15 14:28:34,523 [INFO] iva.detectnet_v2.evaluation.evaluation: step 140 / 1047, 0.04s/step
2020-06-15 14:28:34,925 [INFO] iva.detectnet_v2.evaluation.evaluation: step 150 / 1047, 0.04s/step
2020-06-15 14:28:35,337 [INFO] iva.detectnet_v2.evaluation.evaluation: step 160 / 1047, 0.04s/step
2020-06-15 14:28:35,720 [INFO] iva.detectnet_v2.evaluation.evaluation: step 170 / 1047, 0.04s/step
2020-06-15 14:28:36,095 [INFO] iva.detectnet_v2.evaluation.evaluation: step 180 / 1047, 0.04s/step
2020-06-15 14:28:36,507 [INFO] iva.detectnet_v2.evaluation.evaluation: step 190 / 1047, 0.04s/step
2020-06-15 14:28:36,921 [INFO] iva.detectnet_v2.evaluation.evaluation: step 200 / 1047, 0.04s/step
2020-06-15 14:28:37,320 [INFO] iva.detectnet_v2.evaluation.evaluation: step 210 / 1047, 0.04s/step
2020-06-15 14:28:37,704 [INFO] iva.detectnet_v2.evaluation.evaluation: step 220 / 1047, 0.04s/step
2020-06-15 14:28:38,109 [INFO] iva.detectnet_v2.evaluation.evaluation: step 230 / 1047, 0.04s/step
2020-06-15 14:28:38,511 [INFO] iva.detectnet_v2.evaluation.evaluation: step 240 / 1047, 0.04s/step
2020-06-15 14:28:38,894 [INFO] iva.detectnet_v2.evaluation.evaluation: step 250 / 1047, 0.04s/step
2020-06-15 14:28:39,270 [INFO] iva.detectnet_v2.evaluation.evaluation: step 260 / 1047, 0.04s/step
2020-06-15 14:28:39,724 [INFO] iva.detectnet_v2.evaluation.evaluation: step 270 / 1047, 0.05s/step
2020-06-15 14:28:40,102 [INFO] iva.detectnet_v2.evaluation.evaluation: step 280 / 1047, 0.04s/step
2020-06-15 14:28:40,491 [INFO] iva.detectnet_v2.evaluation.evaluation: step 290 / 1047, 0.04s/step
2020-06-15 14:28:40,889 [INFO] iva.detectnet_v2.evaluation.evaluation: step 300 / 1047, 0.04s/step
2020-06-15 14:28:41,303 [INFO] iva.detectnet_v2.evaluation.evaluation: step 310 / 1047, 0.04s/step
2020-06-15 14:28:41,718 [INFO] iva.detectnet_v2.evaluation.evaluation: step 320 / 1047, 0.04s/step
2020-06-15 14:28:42,116 [INFO] iva.detectnet_v2.evaluation.evaluation: step 330 / 1047, 0.04s/step
2020-06-15 14:28:42,497 [INFO] iva.detectnet_v2.evaluation.evaluation: step 340 / 1047, 0.04s/step
2020-06-15 14:28:42,892 [INFO] iva.detectnet_v2.evaluation.evaluation: step 350 / 1047, 0.04s/step
2020-06-15 14:28:43,336 [INFO] iva.detectnet_v2.evaluation.evaluation: step 360 / 1047, 0.04s/step
2020-06-15 14:28:43,750 [INFO] iva.detectnet_v2.evaluation.evaluation: step 370 / 1047, 0.04s/step
2020-06-15 14:28:44,177 [INFO] iva.detectnet_v2.evaluation.evaluation: step 380 / 1047, 0.04s/step
2020-06-15 14:28:44,570 [INFO] iva.detectnet_v2.evaluation.evaluation: step 390 / 1047, 0.04s/step
2020-06-15 14:28:44,973 [INFO] iva.detectnet_v2.evaluation.evaluation: step 400 / 1047, 0.04s/step
2020-06-15 14:28:45,399 [INFO] iva.detectnet_v2.evaluation.evaluation: step 410 / 1047, 0.04s/step
2020-06-15 14:28:45,848 [INFO] iva.detectnet_v2.evaluation.evaluation: step 420 / 1047, 0.04s/step
2020-06-15 14:28:46,318 [INFO] iva.detectnet_v2.evaluation.evaluation: step 430 / 1047, 0.05s/step
2020-06-15 14:28:46,798 [INFO] iva.detectnet_v2.evaluation.evaluation: step 440 / 1047, 0.05s/step
2020-06-15 14:28:47,269 [INFO] iva.detectnet_v2.evaluation.evaluation: step 450 / 1047, 0.05s/step
2020-06-15 14:28:47,716 [INFO] iva.detectnet_v2.evaluation.evaluation: step 460 / 1047, 0.04s/step
2020-06-15 14:28:48,171 [INFO] iva.detectnet_v2.evaluation.evaluation: step 470 / 1047, 0.05s/step
2020-06-15 14:28:48,635 [INFO] iva.detectnet_v2.evaluation.evaluation: step 480 / 1047, 0.05s/step
2020-06-15 14:28:49,061 [INFO] iva.detectnet_v2.evaluation.evaluation: step 490 / 1047, 0.04s/step
2020-06-15 14:28:49,523 [INFO] iva.detectnet_v2.evaluation.evaluation: step 500 / 1047, 0.05s/step
2020-06-15 14:28:49,988 [INFO] iva.detectnet_v2.evaluation.evaluation: step 510 / 1047, 0.05s/step
2020-06-15 14:28:50,426 [INFO] iva.detectnet_v2.evaluation.evaluation: step 520 / 1047, 0.04s/step
2020-06-15 14:28:50,865 [INFO] iva.detectnet_v2.evaluation.evaluation: step 530 / 1047, 0.04s/step
2020-06-15 14:28:51,319 [INFO] iva.detectnet_v2.evaluation.evaluation: step 540 / 1047, 0.05s/step
2020-06-15 14:28:51,779 [INFO] iva.detectnet_v2.evaluation.evaluation: step 550 / 1047, 0.05s/step
2020-06-15 14:28:52,247 [INFO] iva.detectnet_v2.evaluation.evaluation: step 560 / 1047, 0.05s/step
2020-06-15 14:28:52,694 [INFO] iva.detectnet_v2.evaluation.evaluation: step 570 / 1047, 0.04s/step
2020-06-15 14:28:53,151 [INFO] iva.detectnet_v2.evaluation.evaluation: step 580 / 1047, 0.05s/step
2020-06-15 14:28:53,599 [INFO] iva.detectnet_v2.evaluation.evaluation: step 590 / 1047, 0.04s/step
2020-06-15 14:28:54,046 [INFO] iva.detectnet_v2.evaluation.evaluation: step 600 / 1047, 0.04s/step
2020-06-15 14:28:54,476 [INFO] iva.detectnet_v2.evaluation.evaluation: step 610 / 1047, 0.04s/step
2020-06-15 14:28:54,894 [INFO] iva.detectnet_v2.evaluation.evaluation: step 620 / 1047, 0.04s/step
2020-06-15 14:28:55,331 [INFO] iva.detectnet_v2.evaluation.evaluation: step 630 / 1047, 0.04s/step
2020-06-15 14:28:55,730 [INFO] iva.detectnet_v2.evaluation.evaluation: step 640 / 1047, 0.04s/step
2020-06-15 14:28:56,184 [INFO] iva.detectnet_v2.evaluation.evaluation: step 650 / 1047, 0.05s/step
2020-06-15 14:28:56,575 [INFO] iva.detectnet_v2.evaluation.evaluation: step 660 / 1047, 0.04s/step
2020-06-15 14:28:56,952 [INFO] iva.detectnet_v2.evaluation.evaluation: step 670 / 1047, 0.04s/step
2020-06-15 14:28:57,359 [INFO] iva.detectnet_v2.evaluation.evaluation: step 680 / 1047, 0.04s/step
2020-06-15 14:28:57,760 [INFO] iva.detectnet_v2.evaluation.evaluation: step 690 / 1047, 0.04s/step
2020-06-15 14:28:58,176 [INFO] iva.detectnet_v2.evaluation.evaluation: step 700 / 1047, 0.04s/step
2020-06-15 14:28:58,579 [INFO] iva.detectnet_v2.evaluation.evaluation: step 710 / 1047, 0.04s/step
2020-06-15 14:28:58,976 [INFO] iva.detectnet_v2.evaluation.evaluation: step 720 / 1047, 0.04s/step
2020-06-15 14:28:59,368 [INFO] iva.detectnet_v2.evaluation.evaluation: step 730 / 1047, 0.04s/step
2020-06-15 14:28:59,773 [INFO] iva.detectnet_v2.evaluation.evaluation: step 740 / 1047, 0.04s/step
2020-06-15 14:29:00,240 [INFO] iva.detectnet_v2.evaluation.evaluation: step 750 / 1047, 0.05s/step
2020-06-15 14:29:00,691 [INFO] iva.detectnet_v2.evaluation.evaluation: step 760 / 1047, 0.05s/step
2020-06-15 14:29:01,124 [INFO] iva.detectnet_v2.evaluation.evaluation: step 770 / 1047, 0.04s/step
2020-06-15 14:29:01,562 [INFO] iva.detectnet_v2.evaluation.evaluation: step 780 / 1047, 0.04s/step
2020-06-15 14:29:01,989 [INFO] iva.detectnet_v2.evaluation.evaluation: step 790 / 1047, 0.04s/step
2020-06-15 14:29:02,397 [INFO] iva.detectnet_v2.evaluation.evaluation: step 800 / 1047, 0.04s/step
2020-06-15 14:29:02,850 [INFO] iva.detectnet_v2.evaluation.evaluation: step 810 / 1047, 0.05s/step
2020-06-15 14:29:03,293 [INFO] iva.detectnet_v2.evaluation.evaluation: step 820 / 1047, 0.04s/step
2020-06-15 14:29:03,709 [INFO] iva.detectnet_v2.evaluation.evaluation: step 830 / 1047, 0.04s/step
2020-06-15 14:29:04,155 [INFO] iva.detectnet_v2.evaluation.evaluation: step 840 / 1047, 0.04s/step
2020-06-15 14:29:04,586 [INFO] iva.detectnet_v2.evaluation.evaluation: step 850 / 1047, 0.04s/step
2020-06-15 14:29:05,018 [INFO] iva.detectnet_v2.evaluation.evaluation: step 860 / 1047, 0.04s/step
2020-06-15 14:29:05,447 [INFO] iva.detectnet_v2.evaluation.evaluation: step 870 / 1047, 0.04s/step
2020-06-15 14:29:05,840 [INFO] iva.detectnet_v2.evaluation.evaluation: step 880 / 1047, 0.04s/step
2020-06-15 14:29:06,273 [INFO] iva.detectnet_v2.evaluation.evaluation: step 890 / 1047, 0.04s/step
2020-06-15 14:29:06,709 [INFO] iva.detectnet_v2.evaluation.evaluation: step 900 / 1047, 0.04s/step
2020-06-15 14:29:07,125 [INFO] iva.detectnet_v2.evaluation.evaluation: step 910 / 1047, 0.04s/step
2020-06-15 14:29:07,570 [INFO] iva.detectnet_v2.evaluation.evaluation: step 920 / 1047, 0.04s/step
2020-06-15 14:29:07,983 [INFO] iva.detectnet_v2.evaluation.evaluation: step 930 / 1047, 0.04s/step
2020-06-15 14:29:08,371 [INFO] iva.detectnet_v2.evaluation.evaluation: step 940 / 1047, 0.04s/step
2020-06-15 14:29:08,804 [INFO] iva.detectnet_v2.evaluation.evaluation: step 950 / 1047, 0.04s/step
2020-06-15 14:29:09,222 [INFO] iva.detectnet_v2.evaluation.evaluation: step 960 / 1047, 0.04s/step
2020-06-15 14:29:09,621 [INFO] iva.detectnet_v2.evaluation.evaluation: step 970 / 1047, 0.04s/step
2020-06-15 14:29:10,062 [INFO] iva.detectnet_v2.evaluation.evaluation: step 980 / 1047, 0.04s/step
2020-06-15 14:29:10,485 [INFO] iva.detectnet_v2.evaluation.evaluation: step 990 / 1047, 0.04s/step
2020-06-15 14:29:10,921 [INFO] iva.detectnet_v2.evaluation.evaluation: step 1000 / 1047, 0.04s/step
2020-06-15 14:29:11,333 [INFO] iva.detectnet_v2.evaluation.evaluation: step 1010 / 1047, 0.04s/step
2020-06-15 14:29:11,729 [INFO] iva.detectnet_v2.evaluation.evaluation: step 1020 / 1047, 0.04s/step
2020-06-15 14:29:12,158 [INFO] iva.detectnet_v2.evaluation.evaluation: step 1030 / 1047, 0.04s/step
2020-06-15 14:29:12,586 [INFO] iva.detectnet_v2.evaluation.evaluation: step 1040 / 1047, 0.04s/step
Matching predictions to ground truth, class 1/3.: 100%|█| 24106/24106 [00:01<00:
Matching predictions to ground truth, class 2/3.: 100%|█| 576/576 [00:00<00:00, 
Matching predictions to ground truth, class 3/3.: 100%|█| 9884/9884 [00:00<00:00
Epoch 1/2
=========================

Validation cost: 0.000294
Mean average_precision (in %): 8.7194

class name      average precision (in %)
------------  --------------------------
car                              9.18049
cyclist                          3.563
pedestrian                      13.4148

Median Inference Time: 0.030580
2020-06-15 14:29:15,760 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 1/2: loss: 0.00086 Time taken: 0:13:06.842792 ETA: 0:13:06.842792
2020-06-15 14:29:17,512 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 0.489
2020-06-15 14:29:20,239 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.169
2020-06-15 14:29:23,090 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.771
2020-06-15 14:29:25,947 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.752
2020-06-15 14:29:28,801 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.758
2020-06-15 14:29:31,517 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.208
2020-06-15 14:29:34,202 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.310
2020-06-15 14:29:37,007 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.914
2020-06-15 14:29:40,056 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.200
2020-06-15 14:29:42,965 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.596
2020-06-15 14:29:46,050 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.104
2020-06-15 14:29:48,768 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.201
2020-06-15 14:29:51,584 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.878
2020-06-15 14:29:54,528 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.494
2020-06-15 14:29:57,563 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.238
2020-06-15 14:30:00,506 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.494
2020-06-15 14:30:03,445 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.508
2020-06-15 14:30:06,372 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.543
2020-06-15 14:30:09,326 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.464
2020-06-15 14:30:12,198 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.706
2020-06-15 14:30:15,086 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.659
2020-06-15 14:30:17,909 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.855
2020-06-15 14:30:20,771 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.738
2020-06-15 14:30:23,491 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.191
2020-06-15 14:30:26,227 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.139
2020-06-15 14:30:29,017 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.960
2020-06-15 14:30:31,775 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.068
2020-06-15 14:30:34,463 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.301
2020-06-15 14:30:37,267 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.917
2020-06-15 14:30:39,999 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.150
2020-06-15 14:30:42,707 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.235
2020-06-15 14:30:45,420 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.216
2020-06-15 14:30:48,109 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.295
2020-06-15 14:30:50,861 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.086
2020-06-15 14:30:53,590 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.163
2020-06-15 14:30:56,383 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.951
2020-06-15 14:30:59,139 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.072
2020-06-15 14:31:01,887 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.096
2020-06-15 14:31:04,570 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.319
2020-06-15 14:31:07,378 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.904
2020-06-15 14:31:10,119 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.121
2020-06-15 14:31:12,871 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.087
2020-06-15 14:31:15,587 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.208
2020-06-15 14:31:18,393 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.908
2020-06-15 14:31:21,200 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.906
2020-06-15 14:31:23,926 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.174
2020-06-15 14:31:26,673 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.103
2020-06-15 14:31:29,559 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.662
2020-06-15 14:31:32,557 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.339
2020-06-15 14:31:35,495 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.510
2020-06-15 14:31:38,260 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.043
2020-06-15 14:31:41,155 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.638
2020-06-15 14:31:44,081 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.547
2020-06-15 14:31:46,991 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.590
2020-06-15 14:31:49,834 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.797
2020-06-15 14:31:52,832 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.338
2020-06-15 14:31:55,546 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.213
2020-06-15 14:31:58,420 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.699
2020-06-15 14:32:01,285 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.729
2020-06-15 14:32:04,433 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 7.943
2020-06-15 14:32:07,277 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.789
2020-06-15 14:32:10,272 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.349
2020-06-15 14:32:13,827 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 7.033
2020-06-15 14:32:16,992 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 7.903
2020-06-15 14:32:19,946 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.462
2020-06-15 14:32:22,876 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.534
2020-06-15 14:32:25,792 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.575
2020-06-15 14:32:28,930 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 7.967
2020-06-15 14:32:31,918 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.368
2020-06-15 14:32:34,688 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.028
2020-06-15 14:32:37,602 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.579
2020-06-15 14:32:40,515 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.584
2020-06-15 14:32:43,236 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.190
2020-06-15 14:32:46,080 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.791
2020-06-15 14:32:48,798 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.199
2020-06-15 14:32:51,689 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.649
2020-06-15 14:32:54,522 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.825
2020-06-15 14:32:57,249 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.168
2020-06-15 14:33:00,009 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.058
2020-06-15 14:33:02,784 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.011
2020-06-15 14:33:05,538 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.076
2020-06-15 14:33:08,403 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.728
2020-06-15 14:33:11,190 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.972
2020-06-15 14:33:14,506 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 7.539
2020-06-15 14:33:17,445 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.508
2020-06-15 14:33:20,252 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.908
2020-06-15 14:33:22,990 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.131
2020-06-15 14:33:25,720 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.159
2020-06-15 14:33:28,442 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.186
2020-06-15 14:33:31,333 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.649
2020-06-15 14:33:34,271 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.507
2020-06-15 14:33:37,021 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.095
2020-06-15 14:33:39,896 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.696
2020-06-15 14:33:42,685 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.965
2020-06-15 14:33:45,462 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.003
2020-06-15 14:33:48,235 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.018
2020-06-15 14:33:51,318 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.110
2020-06-15 14:33:54,231 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.583
2020-06-15 14:33:56,949 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.198
2020-06-15 14:33:59,745 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.942
2020-06-15 14:34:02,494 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.098
2020-06-15 14:34:05,247 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.081
2020-06-15 14:34:08,122 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.696
2020-06-15 14:34:11,031 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.596
2020-06-15 14:34:13,827 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.944
2020-06-15 14:34:16,655 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.839
2020-06-15 14:34:19,393 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.134
2020-06-15 14:34:22,227 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.823
2020-06-15 14:34:25,107 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.679
2020-06-15 14:34:27,952 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.788
2020-06-15 14:34:30,870 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.570
2020-06-15 14:34:33,616 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.106
2020-06-15 14:34:36,347 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.153
2020-06-15 14:34:39,094 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.101
2020-06-15 14:34:42,009 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.577
2020-06-15 14:34:44,764 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.076
2020-06-15 14:34:47,614 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.774
2020-06-15 14:34:50,358 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.112
2020-06-15 14:34:53,129 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.022
2020-06-15 14:34:55,862 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.147
2020-06-15 14:34:58,724 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.736
2020-06-15 14:35:01,411 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.306
2020-06-15 14:35:04,194 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.985
2020-06-15 14:35:06,871 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.339
2020-06-15 14:35:09,585 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.214
2020-06-15 14:35:12,298 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.215
2020-06-15 14:35:14,990 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.287
2020-06-15 14:35:17,675 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.314
2020-06-15 14:35:20,571 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.632
2020-06-15 14:35:23,413 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.797
2020-06-15 14:35:26,214 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.926
2020-06-15 14:35:28,945 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.155
2020-06-15 14:35:31,784 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.806
2020-06-15 14:35:34,492 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.235
2020-06-15 14:35:37,474 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.383
2020-06-15 14:35:40,195 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.189
2020-06-15 14:35:43,097 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.615
2020-06-15 14:35:45,857 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.061
2020-06-15 14:35:48,856 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.337
2020-06-15 14:35:51,841 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.376
2020-06-15 14:35:54,722 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.678
2020-06-15 14:35:57,766 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.214
2020-06-15 14:36:00,729 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.438
2020-06-15 14:36:03,635 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.603
2020-06-15 14:36:06,534 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.626
2020-06-15 14:36:09,432 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.627
2020-06-15 14:36:12,309 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.691
2020-06-15 14:36:15,171 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.734
2020-06-15 14:36:18,086 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.579
2020-06-15 14:36:20,718 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.497
2020-06-15 14:36:23,445 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.169
2020-06-15 14:36:26,384 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.509
2020-06-15 14:36:29,006 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.535
2020-06-15 14:36:31,810 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.918
2020-06-15 14:36:34,487 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.339
2020-06-15 14:36:37,096 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.581
2020-06-15 14:36:39,745 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.439
2020-06-15 14:36:42,714 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.424
2020-06-15 14:36:45,563 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.773
2020-06-15 14:36:48,338 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.011
2020-06-15 14:36:51,049 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.224
2020-06-15 14:36:53,844 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.946
2020-06-15 14:36:56,709 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.726
2020-06-15 14:36:59,325 [INFO] /usr/local/lib/python2.7/dist-
2020-06-15 14:39:13,794 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.587
2020-06-15 14:39:16,734 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.502
2020-06-15 14:39:19,702 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.424
2020-06-15 14:39:22,412 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.228
2020-06-15 14:39:25,083 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.359
2020-06-15 14:39:27,743 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.402
2020-06-15 14:39:30,389 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.447
2020-06-15 14:39:33,158 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.029
2020-06-15 14:39:35,825 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.377
2020-06-15 14:39:38,623 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.935
2020-06-15 14:39:41,422 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.933
2020-06-15 14:39:44,073 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.430
2020-06-15 14:39:47,037 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.437
2020-06-15 14:39:50,055 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.284
2020-06-15 14:39:52,956 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.617
2020-06-15 14:39:55,769 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.890
2020-06-15 14:39:58,620 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.769
2020-06-15 14:40:01,330 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.226
2020-06-15 14:40:04,272 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.498
2020-06-15 14:40:07,061 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.965
2020-06-15 14:40:09,947 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.665
2020-06-15 14:40:12,624 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.340
2020-06-15 14:40:15,352 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.164
2020-06-15 14:40:18,082 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.161
2020-06-15 14:40:21,059 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.399
2020-06-15 14:40:24,036 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.397
2020-06-15 14:40:26,734 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.268
2020-06-15 14:40:29,365 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.503
2020-06-15 14:40:32,046 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.326
2020-06-15 14:40:34,935 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.654
2020-06-15 14:40:37,826 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.649
2020-06-15 14:40:40,710 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.670
2020-06-15 14:40:43,480 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.026
2020-06-15 14:40:46,135 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.420
2020-06-15 14:40:48,850 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.206
2020-06-15 14:40:51,520 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.365
2020-06-15 14:40:54,182 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.395
2020-06-15 14:40:56,847 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.380
2020-06-15 14:40:59,504 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.413
2020-06-15 14:41:02,142 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.475
2020-06-15 14:41:04,802 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.402
2020-06-15 14:41:07,513 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.223
2020-06-15 14:41:10,450 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.515
2020-06-15 14:41:13,276 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.846
2020-06-15 14:41:16,141 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.727
2020-06-15 14:41:19,140 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.336
2020-06-15 14:41:22,097 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.457
2020-06-15 14:41:24,193 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 1047, 0.00s/step
2020-06-15 14:41:24,549 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 1047, 0.04s/step
2020-06-15 14:41:24,908 [INFO] iva.detectnet_v2.evaluation.evaluation: step 20 / 1047, 0.04s/step
2020-06-15 14:41:25,251 [INFO] iva.detectnet_v2.evaluation.evaluation: step 30 / 1047, 0.03s/step
2020-06-15 14:41:25,598 [INFO] iva.detectnet_v2.evaluation.evaluation: step 40 / 1047, 0.03s/step
2020-06-15 14:41:25,996 [INFO] iva.detectnet_v2.evaluation.evaluation: step 50 / 1047, 0.04s/step
2020-06-15 14:41:26,353 [INFO] iva.detectnet_v2.evaluation.evaluation: step 60 / 1047, 0.04s/step
2020-06-15 14:41:26,723 [INFO] iva.detectnet_v2.evaluation.evaluation: step 70 / 1047, 0.04s/step
2020-06-15 14:41:27,123 [INFO] iva.detectnet_v2.evaluation.evaluation: step 80 / 1047, 0.04s/step
2020-06-15 14:41:27,505 [INFO] iva.detectnet_v2.evaluation.evaluation: step 90 / 1047, 0.04s/step
2020-06-15 14:41:28,136 [INFO] iva.detectnet_v2.evaluation.evaluation: step 100 / 1047, 0.06s/step
2020-06-15 14:41:28,511 [INFO] iva.detectnet_v2.evaluation.evaluation: step 110 / 1047, 0.04s/step
2020-06-15 14:41:28,889 [INFO] iva.detectnet_v2.evaluation.evaluation: step 120 / 1047, 0.04s/step
2020-06-15 14:41:29,244 [INFO] iva.detectnet_v2.evaluation.evaluation: step 130 / 1047, 0.04s/step
2020-06-15 14:41:29,633 [INFO] iva.detectnet_v2.evaluation.evaluation: step 140 / 1047, 0.04s/step
2020-06-15 14:41:30,006 [INFO] iva.detectnet_v2.evaluation.evaluation: step 150 / 1047, 0.04s/step
2020-06-15 14:41:30,400 [INFO] iva.detectnet_v2.evaluation.evaluation: step 160 / 1047, 0.04s/step
2020-06-15 14:41:30,764 [INFO] iva.detectnet_v2.evaluation.evaluation: step 170 / 1047, 0.04s/step
2020-06-15 14:41:31,123 [INFO] iva.detectnet_v2.evaluation.evaluation: step 180 / 1047, 0.04s/step
2020-06-15 14:41:31,509 [INFO] iva.detectnet_v2.evaluation.evaluation: step 190 / 1047, 0.04s/step
2020-06-15 14:41:31,902 [INFO] iva.detectnet_v2.evaluation.evaluation: step 200 / 1047, 0.04s/step
2020-06-15 14:41:32,278 [INFO] iva.detectnet_v2.evaluation.evaluation: step 210 / 1047, 0.04s/step
2020-06-15 14:41:32,660 [INFO] iva.detectnet_v2.evaluation.evaluation: step 220 / 1047, 0.04s/step
2020-06-15 14:41:33,038 [INFO] iva.detectnet_v2.evaluation.evaluation: step 230 / 1047, 0.04s/step
2020-06-15 14:41:33,414 [INFO] iva.detectnet_v2.evaluation.evaluation: step 240 / 1047, 0.04s/step
2020-06-15 14:41:33,776 [INFO] iva.detectnet_v2.evaluation.evaluation: step 250 / 1047, 0.04s/step
2020-06-15 14:41:34,145 [INFO] iva.detectnet_v2.evaluation.evaluation: step 260 / 1047, 0.04s/step
2020-06-15 14:41:34,522 [INFO] iva.detectnet_v2.evaluation.evaluation: step 270 / 1047, 0.04s/step
2020-06-15 14:41:34,885 [INFO] iva.detectnet_v2.evaluation.evaluation: step 280 / 1047, 0.04s/step
2020-06-15 14:41:35,262 [INFO] iva.detectnet_v2.evaluation.evaluation: step 290 / 1047, 0.04s/step
2020-06-15 14:41:35,643 [INFO] iva.detectnet_v2.evaluation.evaluation: step 300 / 1047, 0.04s/step
2020-06-15 14:41:36,031 [INFO] iva.detectnet_v2.evaluation.evaluation: step 310 / 1047, 0.04s/step
2020-06-15 14:41:36,422 [INFO] iva.detectnet_v2.evaluation.evaluation: step 320 / 1047, 0.04s/step
2020-06-15 14:41:36,806 [INFO] iva.detectnet_v2.evaluation.evaluation: step 330 / 1047, 0.04s/step
2020-06-15 14:41:37,174 [INFO] iva.detectnet_v2.evaluation.evaluation: step 340 / 1047, 0.04s/step
2020-06-15 14:41:37,546 [INFO] iva.detectnet_v2.evaluation.evaluation: step 350 / 1047, 0.04s/step
2020-06-15 14:41:37,926 [INFO] iva.detectnet_v2.evaluation.evaluation: step 360 / 1047, 0.04s/step
2020-06-15 14:41:38,305 [INFO] iva.detectnet_v2.evaluation.evaluation: step 370 / 1047, 0.04s/step
2020-06-15 14:41:38,706 [INFO] iva.detectnet_v2.evaluation.evaluation: step 380 / 1047, 0.04s/step
2020-06-15 14:41:39,079 [INFO] iva.detectnet_v2.evaluation.evaluation: step 390 / 1047, 0.04s/step
2020-06-15 14:41:39,452 [INFO] iva.detectnet_v2.evaluation.evaluation: step 400 / 1047, 0.04s/step
2020-06-15 14:41:39,879 [INFO] iva.detectnet_v2.evaluation.evaluation: step 410 / 1047, 0.04s/step
2020-06-15 14:41:40,229 [INFO] iva.detectnet_v2.evaluation.evaluation: step 420 / 1047, 0.03s/step
2020-06-15 14:41:40,640 [INFO] iva.detectnet_v2.evaluation.evaluation: step 430 / 1047, 0.04s/step
2020-06-15 14:41:40,993 [INFO] iva.detectnet_v2.evaluation.evaluation: step 440 / 1047, 0.04s/step
2020-06-15 14:41:41,355 [INFO] iva.detectnet_v2.evaluation.evaluation: step 450 / 1047, 0.04s/step
2020-06-15 14:41:41,737 [INFO] iva.detectnet_v2.evaluation.evaluation: step 460 / 1047, 0.04s/step
2020-06-15 14:41:42,118 [INFO] iva.detectnet_v2.evaluation.evaluation: step 470 / 1047, 0.04s/step
2020-06-15 14:41:42,470 [INFO] iva.detectnet_v2.evaluation.evaluation: step 480 / 1047, 0.04s/step
2020-06-15 14:41:42,805 [INFO] iva.detectnet_v2.evaluation.evaluation: step 490 / 1047, 0.03s/step
2020-06-15 14:41:43,201 [INFO] iva.detectnet_v2.evaluation.evaluation: step 500 / 1047, 0.04s/step
2020-06-15 14:41:43,594 [INFO] iva.detectnet_v2.evaluation.evaluation: step 510 / 1047, 0.04s/step
2020-06-15 14:41:43,976 [INFO] iva.detectnet_v2.evaluation.evaluation: step 520 / 1047, 0.04s/step
2020-06-15 14:41:44,325 [INFO] iva.detectnet_v2.evaluation.evaluation: step 530 / 1047, 0.03s/step
2020-06-15 14:41:44,674 [INFO] iva.detectnet_v2.evaluation.evaluation: step 540 / 1047, 0.03s/step
2020-06-15 14:41:45,062 [INFO] iva.detectnet_v2.evaluation.evaluation: step 550 / 1047, 0.04s/step
2020-06-15 14:41:45,445 [INFO] iva.detectnet_v2.evaluation.evaluation: step 560 / 1047, 0.04s/step
2020-06-15 14:41:45,815 [INFO] iva.detectnet_v2.evaluation.evaluation: step 570 / 1047, 0.04s/step
2020-06-15 14:41:46,199 [INFO] iva.detectnet_v2.evaluation.evaluation: step 580 / 1047, 0.04s/step
2020-06-15 14:41:46,583 [INFO] iva.detectnet_v2.evaluation.evaluation: step 590 / 1047, 0.04s/step
2020-06-15 14:41:46,946 [INFO] iva.detectnet_v2.evaluation.evaluation: step 600 / 1047, 0.04s/step
2020-06-15 14:41:47,324 [INFO] iva.detectnet_v2.evaluation.evaluation: step 610 / 1047, 0.04s/step
2020-06-15 14:41:47,703 [INFO] iva.detectnet_v2.evaluation.evaluation: step 620 / 1047, 0.04s/step
2020-06-15 14:41:48,082 [INFO] iva.detectnet_v2.evaluation.evaluation: step 630 / 1047, 0.04s/step
2020-06-15 14:41:48,481 [INFO] iva.detectnet_v2.evaluation.evaluation: step 640 / 1047, 0.04s/step
2020-06-15 14:41:48,861 [INFO] iva.detectnet_v2.evaluation.evaluation: step 650 / 1047, 0.04s/step
2020-06-15 14:41:49,232 [INFO] iva.detectnet_v2.evaluation.evaluation: step 660 / 1047, 0.04s/step
2020-06-15 14:41:49,589 [INFO] iva.detectnet_v2.evaluation.evaluation: step 670 / 1047, 0.04s/step
2020-06-15 14:41:49,989 [INFO] iva.detectnet_v2.evaluation.evaluation: step 680 / 1047, 0.04s/step
2020-06-15 14:41:50,362 [INFO] iva.detectnet_v2.evaluation.evaluation: step 690 / 1047, 0.04s/step
2020-06-15 14:41:50,753 [INFO] iva.detectnet_v2.evaluation.evaluation: step 700 / 1047, 0.04s/step
2020-06-15 14:41:51,135 [INFO] iva.detectnet_v2.evaluation.evaluation: step 710 / 1047, 0.04s/step
2020-06-15 14:41:51,500 [INFO] iva.detectnet_v2.evaluation.evaluation: step 720 / 1047, 0.04s/step
2020-06-15 14:41:51,903 [INFO] iva.detectnet_v2.evaluation.evaluation: step 730 / 1047, 0.04s/step
2020-06-15 14:41:52,291 [INFO] iva.detectnet_v2.evaluation.evaluation: step 740 / 1047, 0.04s/step
2020-06-15 14:41:52,672 [INFO] iva.detectnet_v2.evaluation.evaluation: step 750 / 1047, 0.04s/step
2020-06-15 14:41:53,047 [INFO] iva.detectnet_v2.evaluation.evaluation: step 760 / 1047, 0.04s/step
2020-06-15 14:41:53,421 [INFO] iva.detectnet_v2.evaluation.evaluation: step 770 / 1047, 0.04s/step
2020-06-15 14:41:53,804 [INFO] iva.detectnet_v2.evaluation.evaluation: step 780 / 1047, 0.04s/step
2020-06-15 14:41:54,148 [INFO] iva.detectnet_v2.evaluation.evaluation: step 790 / 1047, 0.03s/step
2020-06-15 14:41:54,542 [INFO] iva.detectnet_v2.evaluation.evaluation: step 800 / 1047, 0.04s/step
2020-06-15 14:41:54,948 [INFO] iva.detectnet_v2.evaluation.evaluation: step 810 / 1047, 0.04s/step
2020-06-15 14:41:55,343 [INFO] iva.detectnet_v2.evaluation.evaluation: step 820 / 1047, 0.04s/step
2020-06-15 14:41:55,732 [INFO] iva.detectnet_v2.evaluation.evaluation: step 830 / 1047, 0.04s/step
2020-06-15 14:41:56,118 [INFO] iva.detectnet_v2.evaluation.evaluation: step 840 / 1047, 0.04s/step
2020-06-15 14:41:56,501 [INFO] iva.detectnet_v2.evaluation.evaluation: step 850 / 1047, 0.04s/step
2020-06-15 14:41:56,864 [INFO] iva.detectnet_v2.evaluation.evaluation: step 860 / 1047, 0.04s/step
2020-06-15 14:41:57,262 [INFO] iva.detectnet_v2.evaluation.evaluation: step 870 / 1047, 0.04s/step
2020-06-15 14:41:57,634 [INFO] iva.detectnet_v2.evaluation.evaluation: step 880 / 1047, 0.04s/step
2020-06-15 14:41:58,008 [INFO] iva.detectnet_v2.evaluation.evaluation: step 890 / 1047, 0.04s/step
2020-06-15 14:41:58,423 [INFO] iva.detectnet_v2.evaluation.evaluation: step 900 / 1047, 0.04s/step
2020-06-15 14:41:58,806 [INFO] iva.detectnet_v2.evaluation.evaluation: step 910 / 1047, 0.04s/step
2020-06-15 14:41:59,206 [INFO] iva.detectnet_v2.evaluation.evaluation: step 920 / 1047, 0.04s/step
2020-06-15 14:41:59,643 [INFO] iva.detectnet_v2.evaluation.evaluation: step 930 / 1047, 0.04s/step
2020-06-15 14:42:00,041 [INFO] iva.detectnet_v2.evaluation.evaluation: step 940 / 1047, 0.04s/step
2020-06-15 14:42:00,460 [INFO] iva.detectnet_v2.evaluation.evaluation: step 950 / 1047, 0.04s/step
2020-06-15 14:42:00,824 [INFO] iva.detectnet_v2.evaluation.evaluation: step 960 / 1047, 0.04s/step
2020-06-15 14:42:01,190 [INFO] iva.detectnet_v2.evaluation.evaluation: step 970 / 1047, 0.04s/step
2020-06-15 14:42:01,556 [INFO] iva.detectnet_v2.evaluation.evaluation: step 980 / 1047, 0.04s/step
2020-06-15 14:42:01,937 [INFO] iva.detectnet_v2.evaluation.evaluation: step 990 / 1047, 0.04s/step
2020-06-15 14:42:02,294 [INFO] iva.detectnet_v2.evaluation.evaluation: step 1000 / 1047, 0.04s/step
2020-06-15 14:42:02,633 [INFO] iva.detectnet_v2.evaluation.evaluation: step 1010 / 1047, 0.03s/step
2020-06-15 14:42:02,967 [INFO] iva.detectnet_v2.evaluation.evaluation: step 1020 / 1047, 0.03s/step
2020-06-15 14:42:03,329 [INFO] iva.detectnet_v2.evaluation.evaluation: step 1030 / 1047, 0.04s/step
2020-06-15 14:42:03,673 [INFO] iva.detectnet_v2.evaluation.evaluation: step 1040 / 1047, 0.03s/step
Matching predictions to ground truth, class 1/3.: 100%|█| 13648/13648 [00:00<00:
Matching predictions to ground truth, class 2/3.: 100%|█| 1092/1092 [00:00<00:00
Matching predictions to ground truth, class 3/3.: 100%|█| 2970/2970 [00:00<00:00
Epoch 2/2
=========================

Validation cost: 0.000187
Mean average_precision (in %): 28.7512

class name      average precision (in %)
------------  --------------------------
car                              38.0343
cyclist                          12.9367
pedestrian                       35.2825

Median Inference Time: 0.030535
Time taken to run iva.detectnet_v2.scripts.train:main: 0:31:40.542805.

I’m not sure if there is too much more info to add here, I’m not making any more changes than the absolutely necessary to run the example yet.

Again, thank you for your help, I appreciate it!

Morganh · June 15, 2020, 3:33pm

The log is normal.
But 624 x 192 seems to mismatch W > =480, H >=272.
Anyway, you can just let the training continue.

ai12 · June 15, 2020, 3:43pm

True, I was foccused on the multiple of 16 condition.
For the moment is doing well tho, I’ll comment my results when done, I already trained, pruned, and retrain. Now I’m running tlt-infer and getting ready for deploy.

ai12 · June 15, 2020, 4:48pm

Hello, I’m reporting as promised.
The model worked fine and I was able to get the .etlt file, run the deepstream-app on my Nano and got the .engine file and everything is running.
I can’t report on how using lower resolution images affected the end result since I just trained 2 epochs (my current goal was just to prove that it worked and now I’ll be doing some real transfer learning).
Thank you so much for your help @Morganh, I really appreciate your dedication.

Topic		Replies	Views
Core dump Illegal Instruction on detectnet_v2 example TAO Toolkit	17	1987	October 12, 2021
Training detectnet_v2 Issue TAO Toolkit	15	1845	October 12, 2021
Tlt-train loss is minimal but performances are bad TAO Toolkit	11	518	October 12, 2021
tlt-train error when deploy mobilenet_v2 by using DetectNet TAO Toolkit	28	2365	October 12, 2021
TLT training error : Key cost_sums/cyclist-bbox not found in checkpoint TAO Toolkit	6	1194	October 12, 2021
0 map over 120 epoch on detectnet v2 pre-trained model TAO Toolkit	5	761	October 12, 2021
Retraining with pretrained tlt models TAO Toolkit	33	2712	October 12, 2021
"tlt-train detectnet_v2" lead core dump TAO Toolkit	7	967	October 12, 2021
Run detectnet_v2.ipynb error with my own data TAO Toolkit tao	23	1397	March 4, 2022
Problem of tao detectnet_v2 evaluate 0% TAO Toolkit python	21	391	July 7, 2023

Error on tlt-training detectnet_v2?

Related topics