0 map over 120 epoch on detectnet v2 pre-trained model

Hi.
After couple of testing found out that evaluation of my training on detectnet still maintained 0 mAP.

Here’s the detail.

Device : Ubuntu 18.04 CUDA 11 GPU Geforce GTX 1650 4GB
Total training label : 558
Total training image : 558

Image size: 480*288

example label (kitti) :
car 0.0 0 0.0 268.4 391.1 580.6 572.9 0.0 0.0 0.0 0.0 0.0 0.0 0.0
person 0.0 0 0.0 281.1 300.2 315.9 379.9 0.0 0.0 0.0 0.0 0.0 0.0 0.0

Convert file:

kitti_config {
root_directory_path: “/workspace/dataset/training”
image_dir_name: “image_2”
label_dir_name: “label_2”
image_extension: “.jpg”
partition_mode: “random”
num_partitions:2
val_split: 20
num_shards: 10
}
image_directory_path: “/workspace/dataset/training”

Log Convert:

2020-09-23 02:19:53.770882: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
Using TensorFlow backend.
2020-09-23 02:19:55,506 - iva.detectnet_v2.dataio.build_converter - INFO - Instantiating a kitti converter
2020-09-23 02:19:55,507 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Creating output directory /workspace/tf_records
2020-09-23 02:19:55,508 - iva.detectnet_v2.dataio.kitti_converter_lib - INFO - Num images in
Train: 502 Val: 55
2020-09-23 02:19:55,508 - iva.detectnet_v2.dataio.kitti_converter_lib - INFO - Validation data in partition 0. Hence, while choosing the validationset during training choose validation_fold 0.
2020-09-23 02:19:55,509 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 0
WARNING:tensorflow:From /home/vpraveen/.cache/dazel/_dazel_vpraveen/715c8bafe7816f3bb6f309cd506049bb/execroot/ai_infra/bazel-out/k8-py3-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/dataio/dataset_converter_lib.py:142: The name tf.python_io.TFRecordWriter is deprecated. Please use tf.io.TFRecordWriter instead.

2020-09-23 02:19:55,509 - tensorflow - WARNING - From /home/vpraveen/.cache/dazel/_dazel_vpraveen/715c8bafe7816f3bb6f309cd506049bb/execroot/ai_infra/bazel-out/k8-py3-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/dataio/dataset_converter_lib.py:142: The name tf.python_io.TFRecordWriter is deprecated. Please use tf.io.TFRecordWriter instead.

/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: VisibleDeprecationWarning: Reading unicode strings without specifying the encoding argument is deprecated. Set the encoding, use None for the system default.
2020-09-23 02:19:55,517 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 1
2020-09-23 02:19:55,520 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 2
2020-09-23 02:19:55,524 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 3
2020-09-23 02:19:55,527 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 4
2020-09-23 02:19:55,531 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 5
2020-09-23 02:19:55,535 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 6
2020-09-23 02:19:55,538 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 7
2020-09-23 02:19:55,543 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 8
2020-09-23 02:19:55,546 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 9
**2020-09-23 02:19:55,554 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - **
Wrote the following numbers of objects:
b’person’: 54
b’car’: 48
b’bus’: 15
b’truck’: 2
b’motorcycle’: 1
b’bicycle’: 4

2020-09-23 02:19:55,554 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 0
2020-09-23 02:19:55,588 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 1
2020-09-23 02:19:55,622 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 2
2020-09-23 02:19:55,657 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 3
2020-09-23 02:19:55,692 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 4
2020-09-23 02:19:55,727 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 5
2020-09-23 02:19:55,761 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 6
2020-09-23 02:19:55,795 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 7
2020-09-23 02:19:55,829 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 8
2020-09-23 02:19:55,864 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 9
**2020-09-23 02:19:55,900 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - **
Wrote the following numbers of objects:
b’car’: 488
b’person’: 523
b’motorcycle’: 2
b’bus’: 90
b’truck’: 24
b’bicycle’: 13

2020-09-23 02:19:55,900 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Cumulative object statistics
**2020-09-23 02:19:55,900 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - **
Wrote the following numbers of objects:
b’person’: 577
b’car’: 536
b’bus’: 105
b’truck’: 26
b’motorcycle’: 3
b’bicycle’: 17

**2020-09-23 02:19:55,900 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Class map. **
**Label in GT: Label in tfrecords file **
b’person’: b’person’
b’car’: b’car’
b’bus’: b’bus’
b’truck’: b’truck’
b’motorcycle’: b’motorcycle’
b’bicycle’: b’bicycle’
For the dataset_config in the experiment_spec, please use labels in the tfrecords file, while writing the classmap.

2020-09-23 02:19:55,900 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Tfrecords generation complete.

Train File:
model_config {
arch: “resnet”
pretrained_model_file: “/workspace/pretrained_model/tlt_resnet18_detectnet_v2_v1/resnet18.hdf5”
freeze_blocks: 0
freeze_blocks: 1
all_projections: True
num_layers: 18
use_pooling: False
use_batch_norm: True
dropout_rate: 0.0
training_precision: {
backend_floatx: FLOAT32
}
objective_set: {
cov {}
bbox {
scale: 35.0
offset: 0.5
}
}
}

bbox_rasterizer_config {
target_class_config {
key: “car”
value: {
cov_center_x: 0.5
cov_center_y: 0.5
cov_radius_x: 0.4
cov_radius_y: 0.4
bbox_min_radius: 1.0
}
}
target_class_config {
key: “bus”
value: {
cov_center_x: 0.5
cov_center_y: 0.5
cov_radius_x: 0.4
cov_radius_y: 0.4
bbox_min_radius: 1.0
}
}
target_class_config {
key: “person”
value: {
cov_center_x: 0.5
cov_center_y: 0.5
cov_radius_x: 0.4
cov_radius_y: 0.4
bbox_min_radius: 1.0
}
}
target_class_config {
key: “motorcycle”
value: {
cov_center_x: 0.5
cov_center_y: 0.5
cov_radius_x: 0.4
cov_radius_y: 0.4
bbox_min_radius: 1.0
}
}
target_class_config {
key: “bicycle”
value: {
cov_center_x: 0.5
cov_center_y: 0.5
cov_radius_x: 0.4
cov_radius_y: 0.4
bbox_min_radius: 1.0
}
}
target_class_config {
key: “truck”
value: {
cov_center_x: 0.5
cov_center_y: 0.5
cov_radius_x: 0.4
cov_radius_y: 0.4
bbox_min_radius: 1.0
}
}
deadzone_radius: 0.67
}

postprocessing_config {
target_class_config {
key: “car”
value: {
clustering_config {
coverage_threshold: 0.005
dbscan_eps: 0.13
dbscan_min_samples: 0.05
minimum_bounding_box_height: 20
}
}
}
target_class_config {
key: “bus”
value: {
clustering_config {
coverage_threshold: 0.005
dbscan_eps: 0.15
dbscan_min_samples: 0.05
minimum_bounding_box_height: 20
}
}
}
target_class_config {
key: “person”
value: {
clustering_config {
coverage_threshold: 0.005
dbscan_eps: 0.15
dbscan_min_samples: 0.05
minimum_bounding_box_height: 20
}
}
}
target_class_config {
key: “motorcycle”
value: {
clustering_config {
coverage_threshold: 0.005
dbscan_eps: 0.15
dbscan_min_samples: 0.05
minimum_bounding_box_height: 20
}
}
}
target_class_config {
key: “bicycle”
value: {
clustering_config {
coverage_threshold: 0.005
dbscan_eps: 0.15
dbscan_min_samples: 0.05
minimum_bounding_box_height: 20
}
}
}
target_class_config {
key: “truck”
value: {
clustering_config {
coverage_threshold: 0.005
dbscan_eps: 0.15
dbscan_min_samples: 0.05
minimum_bounding_box_height: 20
}
}
}
}

cost_function_config {
target_classes {
name: “car”
class_weight: 1.0
coverage_foreground_weight: 0.05
objectives {
name: “cov”
initial_weight: 1.0
weight_target: 1.0
}
objectives {
name: “bbox”
initial_weight: 10.0
weight_target: 10.0
}
}
target_classes {
name: “bus”
class_weight: 1.0
coverage_foreground_weight: 0.05
objectives {
name: “cov”
initial_weight: 1.0
weight_target: 1.0
}
objectives {
name: “bbox”
initial_weight: 10.0
weight_target: 1.0
}
}
target_classes {
name: “person”
class_weight: 1.0
coverage_foreground_weight: 0.05
objectives {
name: “cov”
initial_weight: 1.0
weight_target: 1.0
}
objectives {
name: “bbox”
initial_weight: 10.0
weight_target: 10.0
}
}
target_classes {
name: “motorcycle”
class_weight: 1.0
coverage_foreground_weight: 0.05
objectives {
name: “cov”
initial_weight: 1.0
weight_target: 1.0
}
objectives {
name: “bbox”
initial_weight: 10.0
weight_target: 10.0
}
}
target_classes {
name: “bicycle”
class_weight: 1.0
coverage_foreground_weight: 0.05
objectives {
name: “cov”
initial_weight: 1.0
weight_target: 1.0
}
objectives {
name: “bbox”
initial_weight: 10.0
weight_target: 10.0
}
}
target_classes {
name: “truck”
class_weight: 1.0
coverage_foreground_weight: 0.05
objectives {
name: “cov”
initial_weight: 1.0
weight_target: 1.0
}
objectives {
name: “bbox”
initial_weight: 10.0
weight_target: 10.0
}
}
enable_autoweighting: True
max_objective_weight: 0.9999
min_objective_weight: 0.0001
}

training_config {
batch_size_per_gpu: 5
num_epochs: 120
learning_rate {
soft_start_annealing_schedule {
min_learning_rate: 5e-6
max_learning_rate: 5e-4
soft_start: 0.1
annealing: 0.7
}
}
regularizer {
type: L1
weight: 3e-9
}
optimizer {
adam {
epsilon: 1e-08
beta1: 0.9
beta2: 0.999
}
}
cost_scaling {
enabled: False
initial_exponent: 20.0
increment: 0.005
decrement: 1.0
}
checkpoint_interval: 10
}

augmentation_config {
preprocessing {
output_image_width: 480
output_image_height: 288
output_image_channel: 3
min_bbox_width: 1.0
min_bbox_height: 1.0
}
spatial_augmentation {
hflip_probability: 0.5
vflip_probability: 0.0
zoom_min: 1.0
zoom_max: 1.0
translate_max_x: 8.0
translate_max_y: 8.0
}
color_augmentation {
color_shift_stddev: 0.0
hue_rotation_max: 25.0
saturation_shift_max: 0.2
contrast_scale_max: 0.1
contrast_center: 0.5
}
}

dataset_config {
data_sources: {
tfrecords_path: “/workspace/tf_records/*”
image_directory_path: “/workspace/dataset/training”
}
image_extension: “jpg”
target_class_mapping {
key: “person”
value: “person”
}
target_class_mapping {
key: “car”
value: “car”
}
target_class_mapping {
key: “bus”
value: “bus”
}
target_class_mapping {
key: “truck”
value: “truck”
}

target_class_mapping {
  key: "motorcycle"
  value: "motorcycle"

}
target_class_mapping {
key: “bicycle”
value: “bicycle”
}

validation_fold: 0
}

evaluation_config {
average_precision_mode: INTEGRATE
validation_period_during_training: 120
minimum_detection_ground_truth_overlap {
key: “person”
value: 0.5
}
minimum_detection_ground_truth_overlap {
key: “car”
value: 0.5
}
minimum_detection_ground_truth_overlap {
key: “bus”
value: 0.5
}
minimum_detection_ground_truth_overlap {
key: “truck”
value: 0.5
}

minimum_detection_ground_truth_overlap {
key: “motorcycle”
value: 0.5
}
minimum_detection_ground_truth_overlap {
key: “bicycle”
value: 0.5
}
evaluation_box_config {
key: “person”
value {
minimum_height: 4
maximum_height: 9999
minimum_width: 4
maximum_width: 9999
}
}
evaluation_box_config {
key: “car”
value {
minimum_height: 4
maximum_height: 9999
minimum_width: 4
maximum_width: 9999
}
}
evaluation_box_config {
key: “bus”
value {
minimum_height: 4
maximum_height: 9999
minimum_width: 4
maximum_width: 9999
}
}
evaluation_box_config {
key: “truck”
value {
minimum_height: 4
maximum_height: 9999
minimum_width: 4
maximum_width: 9999
}
}
evaluation_box_config {
key: “motorcycle”
value {
minimum_height: 4
maximum_height: 9999
minimum_width: 4
maximum_width: 9999
}
}
evaluation_box_config {
key: “bicycle”
value {
minimum_height: 4
maximum_height: 9999
minimum_width: 4
maximum_width: 9999
}
}
}

Train Log:

Using TensorFlow backend.
2020-09-25 05:19:50.352578: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-09-25 05:19:52.362182: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-09-25 05:19:52.376912: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-25 05:19:52.377163: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: GeForce GTX 1650 major: 7 minor: 5 memoryClockRate(GHz): 1.665
pciBusID: 0000:01:00.0
2020-09-25 05:19:52.377199: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-09-25 05:19:52.377271: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-09-25 05:19:52.378514: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-09-25 05:19:52.378797: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-09-25 05:19:52.380222: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-09-25 05:19:52.381335: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-09-25 05:19:52.381411: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-09-25 05:19:52.381571: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-25 05:19:52.381891: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-25 05:19:52.382168: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2020-09-25 05:19:52.382193: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-09-25 05:19:52.840727: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-09-25 05:19:52.840797: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0
2020-09-25 05:19:52.840804: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N
2020-09-25 05:19:52.841034: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-25 05:19:52.841377: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-25 05:19:52.841608: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-25 05:19:52.841813: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2145 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1650, pci bus id: 0000:01:00.0, compute capability: 7.5)
2020-09-25 05:19:52,842 [INFO] iva.detectnet_v2.scripts.train: Loading experiment spec at spec_files/train.txt.
2020-09-25 05:19:52,843 [INFO] iva.detectnet_v2.spec_handler.spec_loader: Merging specification from spec_files/train.txt
2020-09-25 05:19:53,040 [INFO] iva.detectnet_v2.scripts.train: Cannot iterate over exactly 446 samples with a batch size of 5; each epoch will therefore take one extra step.


Layer (type) Output Shape Param # Connected to

input_1 (InputLayer) (None, 3, 288, 480) 0


conv1 (Conv2D) (None, 64, 144, 240) 9472 input_1[0][0]


bn_conv1 (BatchNormalization) (None, 64, 144, 240) 256 conv1[0][0]


activation_1 (Activation) (None, 64, 144, 240) 0 bn_conv1[0][0]


block_1a_conv_1 (Conv2D) (None, 64, 72, 120) 36928 activation_1[0][0]


block_1a_bn_1 (BatchNormalizati (None, 64, 72, 120) 256 block_1a_conv_1[0][0]


block_1a_relu_1 (Activation) (None, 64, 72, 120) 0 block_1a_bn_1[0][0]


block_1a_conv_2 (Conv2D) (None, 64, 72, 120) 36928 block_1a_relu_1[0][0]


block_1a_conv_shortcut (Conv2D) (None, 64, 72, 120) 4160 activation_1[0][0]


block_1a_bn_2 (BatchNormalizati (None, 64, 72, 120) 256 block_1a_conv_2[0][0]


block_1a_bn_shortcut (BatchNorm (None, 64, 72, 120) 256 block_1a_conv_shortcut[0][0]


add_1 (Add) (None, 64, 72, 120) 0 block_1a_bn_2[0][0]
block_1a_bn_shortcut[0][0]


block_1a_relu (Activation) (None, 64, 72, 120) 0 add_1[0][0]


block_1b_conv_1 (Conv2D) (None, 64, 72, 120) 36928 block_1a_relu[0][0]


block_1b_bn_1 (BatchNormalizati (None, 64, 72, 120) 256 block_1b_conv_1[0][0]


block_1b_relu_1 (Activation) (None, 64, 72, 120) 0 block_1b_bn_1[0][0]


block_1b_conv_2 (Conv2D) (None, 64, 72, 120) 36928 block_1b_relu_1[0][0]


block_1b_conv_shortcut (Conv2D) (None, 64, 72, 120) 4160 block_1a_relu[0][0]


block_1b_bn_2 (BatchNormalizati (None, 64, 72, 120) 256 block_1b_conv_2[0][0]


block_1b_bn_shortcut (BatchNorm (None, 64, 72, 120) 256 block_1b_conv_shortcut[0][0]


add_2 (Add) (None, 64, 72, 120) 0 block_1b_bn_2[0][0]
block_1b_bn_shortcut[0][0]


block_1b_relu (Activation) (None, 64, 72, 120) 0 add_2[0][0]


block_2a_conv_1 (Conv2D) (None, 128, 36, 60) 73856 block_1b_relu[0][0]


block_2a_bn_1 (BatchNormalizati (None, 128, 36, 60) 512 block_2a_conv_1[0][0]


block_2a_relu_1 (Activation) (None, 128, 36, 60) 0 block_2a_bn_1[0][0]


block_2a_conv_2 (Conv2D) (None, 128, 36, 60) 147584 block_2a_relu_1[0][0]


block_2a_conv_shortcut (Conv2D) (None, 128, 36, 60) 8320 block_1b_relu[0][0]


block_2a_bn_2 (BatchNormalizati (None, 128, 36, 60) 512 block_2a_conv_2[0][0]


block_2a_bn_shortcut (BatchNorm (None, 128, 36, 60) 512 block_2a_conv_shortcut[0][0]


add_3 (Add) (None, 128, 36, 60) 0 block_2a_bn_2[0][0]
block_2a_bn_shortcut[0][0]


block_2a_relu (Activation) (None, 128, 36, 60) 0 add_3[0][0]


block_2b_conv_1 (Conv2D) (None, 128, 36, 60) 147584 block_2a_relu[0][0]


block_2b_bn_1 (BatchNormalizati (None, 128, 36, 60) 512 block_2b_conv_1[0][0]


block_2b_relu_1 (Activation) (None, 128, 36, 60) 0 block_2b_bn_1[0][0]


block_2b_conv_2 (Conv2D) (None, 128, 36, 60) 147584 block_2b_relu_1[0][0]


block_2b_conv_shortcut (Conv2D) (None, 128, 36, 60) 16512 block_2a_relu[0][0]


block_2b_bn_2 (BatchNormalizati (None, 128, 36, 60) 512 block_2b_conv_2[0][0]


block_2b_bn_shortcut (BatchNorm (None, 128, 36, 60) 512 block_2b_conv_shortcut[0][0]


add_4 (Add) (None, 128, 36, 60) 0 block_2b_bn_2[0][0]
block_2b_bn_shortcut[0][0]


block_2b_relu (Activation) (None, 128, 36, 60) 0 add_4[0][0]


block_3a_conv_1 (Conv2D) (None, 256, 18, 30) 295168 block_2b_relu[0][0]


block_3a_bn_1 (BatchNormalizati (None, 256, 18, 30) 1024 block_3a_conv_1[0][0]


block_3a_relu_1 (Activation) (None, 256, 18, 30) 0 block_3a_bn_1[0][0]


block_3a_conv_2 (Conv2D) (None, 256, 18, 30) 590080 block_3a_relu_1[0][0]


block_3a_conv_shortcut (Conv2D) (None, 256, 18, 30) 33024 block_2b_relu[0][0]


block_3a_bn_2 (BatchNormalizati (None, 256, 18, 30) 1024 block_3a_conv_2[0][0]


block_3a_bn_shortcut (BatchNorm (None, 256, 18, 30) 1024 block_3a_conv_shortcut[0][0]


add_5 (Add) (None, 256, 18, 30) 0 block_3a_bn_2[0][0]
block_3a_bn_shortcut[0][0]


block_3a_relu (Activation) (None, 256, 18, 30) 0 add_5[0][0]


block_3b_conv_1 (Conv2D) (None, 256, 18, 30) 590080 block_3a_relu[0][0]


block_3b_bn_1 (BatchNormalizati (None, 256, 18, 30) 1024 block_3b_conv_1[0][0]


block_3b_relu_1 (Activation) (None, 256, 18, 30) 0 block_3b_bn_1[0][0]


block_3b_conv_2 (Conv2D) (None, 256, 18, 30) 590080 block_3b_relu_1[0][0]


block_3b_conv_shortcut (Conv2D) (None, 256, 18, 30) 65792 block_3a_relu[0][0]


block_3b_bn_2 (BatchNormalizati (None, 256, 18, 30) 1024 block_3b_conv_2[0][0]


block_3b_bn_shortcut (BatchNorm (None, 256, 18, 30) 1024 block_3b_conv_shortcut[0][0]


add_6 (Add) (None, 256, 18, 30) 0 block_3b_bn_2[0][0]
block_3b_bn_shortcut[0][0]


block_3b_relu (Activation) (None, 256, 18, 30) 0 add_6[0][0]


block_4a_conv_1 (Conv2D) (None, 512, 18, 30) 1180160 block_3b_relu[0][0]


block_4a_bn_1 (BatchNormalizati (None, 512, 18, 30) 2048 block_4a_conv_1[0][0]


block_4a_relu_1 (Activation) (None, 512, 18, 30) 0 block_4a_bn_1[0][0]


block_4a_conv_2 (Conv2D) (None, 512, 18, 30) 2359808 block_4a_relu_1[0][0]


block_4a_conv_shortcut (Conv2D) (None, 512, 18, 30) 131584 block_3b_relu[0][0]


block_4a_bn_2 (BatchNormalizati (None, 512, 18, 30) 2048 block_4a_conv_2[0][0]


block_4a_bn_shortcut (BatchNorm (None, 512, 18, 30) 2048 block_4a_conv_shortcut[0][0]


add_7 (Add) (None, 512, 18, 30) 0 block_4a_bn_2[0][0]
block_4a_bn_shortcut[0][0]


block_4a_relu (Activation) (None, 512, 18, 30) 0 add_7[0][0]


block_4b_conv_1 (Conv2D) (None, 512, 18, 30) 2359808 block_4a_relu[0][0]


block_4b_bn_1 (BatchNormalizati (None, 512, 18, 30) 2048 block_4b_conv_1[0][0]


block_4b_relu_1 (Activation) (None, 512, 18, 30) 0 block_4b_bn_1[0][0]


block_4b_conv_2 (Conv2D) (None, 512, 18, 30) 2359808 block_4b_relu_1[0][0]


block_4b_conv_shortcut (Conv2D) (None, 512, 18, 30) 262656 block_4a_relu[0][0]


block_4b_bn_2 (BatchNormalizati (None, 512, 18, 30) 2048 block_4b_conv_2[0][0]


block_4b_bn_shortcut (BatchNorm (None, 512, 18, 30) 2048 block_4b_conv_shortcut[0][0]


add_8 (Add) (None, 512, 18, 30) 0 block_4b_bn_2[0][0]
block_4b_bn_shortcut[0][0]


block_4b_relu (Activation) (None, 512, 18, 30) 0 add_8[0][0]


output_bbox (Conv2D) (None, 24, 18, 30) 12312 block_4b_relu[0][0]


output_cov (Conv2D) (None, 6, 18, 30) 3078 block_4b_relu[0][0]

Total params: 11,563,678
Trainable params: 11,386,526
Non-trainable params: 177,152


2020-09-25 05:20:01,985 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Serial augmentation enabled = False
2020-09-25 05:20:01,985 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Pseudo sharding enabled = False
2020-09-25 05:20:01,985 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Max Image Dimensions (all sources): (0, 0)
2020-09-25 05:20:01,985 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: number of cpus: 8, io threads: 16, compute threads: 8, buffered batches: 4
2020-09-25 05:20:01,985 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: total dataset size 446, number of sources: 1, batch size per gpu: 5, steps: 90
2020-09-25 05:20:02,076 [INFO] iva.detectnet_v2.dataloader.default_dataloader: Bounding box coordinates were detected in the input specification! Bboxes will be automatically converted to polygon coordinates.
2020-09-25 05:20:02.102279: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-25 05:20:02.102490: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: GeForce GTX 1650 major: 7 minor: 5 memoryClockRate(GHz): 1.665
pciBusID: 0000:01:00.0
2020-09-25 05:20:02.102513: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-09-25 05:20:02.102541: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-09-25 05:20:02.102564: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-09-25 05:20:02.102580: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-09-25 05:20:02.102596: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-09-25 05:20:02.102612: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-09-25 05:20:02.102625: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-09-25 05:20:02.102678: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-25 05:20:02.102879: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-25 05:20:02.103042: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2020-09-25 05:20:02,276 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: shuffle: True - shard 0 of 1
2020-09-25 05:20:02,281 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: sampling 1 datasets with weights:
2020-09-25 05:20:02,281 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: source: 0 weight: 1.000000
2020-09-25 05:20:02,674 [INFO] iva.detectnet_v2.scripts.train: Found 446 samples in training set
2020-09-25 05:20:04,943 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Serial augmentation enabled = False
2020-09-25 05:20:04,943 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Pseudo sharding enabled = False
2020-09-25 05:20:04,943 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Max Image Dimensions (all sources): (0, 0)
2020-09-25 05:20:04,943 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: number of cpus: 8, io threads: 16, compute threads: 8, buffered batches: 4
2020-09-25 05:20:04,944 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: total dataset size 111, number of sources: 1, batch size per gpu: 5, steps: 23
2020-09-25 05:20:04,966 [INFO] iva.detectnet_v2.dataloader.default_dataloader: Bounding box coordinates were detected in the input specification! Bboxes will be automatically converted to polygon coordinates.
2020-09-25 05:20:05,158 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: shuffle: False - shard 0 of 1
2020-09-25 05:20:05,163 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: sampling 1 datasets with weights:
2020-09-25 05:20:05,163 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: source: 0 weight: 1.000000
2020-09-25 05:20:05,429 [INFO] iva.detectnet_v2.scripts.train: Found 111 samples in validation set
2020-09-25 05:20:08.106165: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-25 05:20:08.106447: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: GeForce GTX 1650 major: 7 minor: 5 memoryClockRate(GHz): 1.665
pciBusID: 0000:01:00.0
2020-09-25 05:20:08.106526: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-09-25 05:20:08.106596: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-09-25 05:20:08.106642: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-09-25 05:20:08.106690: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-09-25 05:20:08.106720: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-09-25 05:20:08.106736: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-09-25 05:20:08.106783: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-09-25 05:20:08.106886: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-25 05:20:08.107153: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-25 05:20:08.107320: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2020-09-25 05:20:08.109603: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-09-25 05:20:08.109618: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0
2020-09-25 05:20:08.109624: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N
2020-09-25 05:20:08.109730: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-25 05:20:08.109946: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-25 05:20:08.110180: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2145 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1650, pci bus id: 0000:01:00.0, compute capability: 7.5)
2020-09-25 05:20:31.558489: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-09-25 05:20:31.704984: I tensorflow/core/kernels/cuda_solvers.cc:159] Creating CudaSolver handles for stream 0x24209590
2020-09-25 05:20:31.705107: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-09-25 05:20:31.839434: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-09-25 05:20:31.840041: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-09-25 05:20:33.250112: W tensorflow/core/common_runtime/bfc_allocator.cc:305] Garbage collection: deallocate free memory regions (i.e., allocations) so that we can re-allocate a larger region to avoid OOM due to memory fragmentation. If you see this message frequently, you are running near the threshold of the available device memory and re-allocation may incur great performance overhead. You may try smaller batch sizes to observe the performance impact. Set TF_ENABLE_GPU_GARBAGE_COLLECTION=false if you’d like to disable this feature.
2020-09-25 05:20:33.479382: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.17GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2020-09-25 05:20:33.479417: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.17GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2020-09-25 05:20:33.492237: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.10GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2020-09-25 05:20:33.492257: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.10GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2020-09-25 05:20:33.523150: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.16GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2020-09-25 05:20:33.523181: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.16GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2020-09-25 05:20:33.659521: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.12GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2020-09-25 05:20:33.659558: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.12GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2020-09-25 05:20:33.728502: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.15GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2020-09-25 05:20:33.728537: W tensorflow/core/common_runtime/bfc_allocator.cc:239] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.15GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2020-09-25 05:20:34,788 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 0/120: loss: 0.11059 Time taken: 0:00:00 ETA: 0:00:00
2020-09-25 05:20:34,788 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 1.045
2020-09-25 05:20:43,621 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 9.178
2020-09-25 05:20:47,437 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 32.760
2020-09-25 05:20:51,364 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 31.837
2020-09-25 05:20:53,965 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 1/120: loss: 0.38297 Time taken: 0:00:23.811672 ETA: 0:47:13.588965
2020-09-25 05:20:55,334 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 31.484
2020-09-25 05:20:59,104 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.160
2020-09-25 05:21:02,773 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 34.065
2020-09-25 05:21:06,440 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 34.096
2020-09-25 05:21:07,337 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 2/120: loss: 0.03791 Time taken: 0:00:13.370991 ETA: 0:26:17.776965
2020-09-25 05:21:10,126 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.910
2020-09-25 05:21:13,792 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 34.098
2020-09-25 05:21:17,471 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.976
2020-09-25 05:21:20,571 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 3/120: loss: 0.01536 Time taken: 0:00:13.235514 ETA: 0:25:48.555101
2020-09-25 05:21:21,159 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.899
2020-09-25 05:21:24,830 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 34.057
2020-09-25 05:21:28,520 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.876
2020-09-25 05:21:32,194 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 34.020
2020-09-25 05:21:33,819 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 4/120: loss: 0.00664 Time taken: 0:00:13.248857 ETA: 0:25:36.867470
2020-09-25 05:21:35,878 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.929
2020-09-25 05:21:39,556 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.992
2020-09-25 05:21:43,238 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.953
2020-09-25 05:21:46,922 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.933
2020-09-25 05:21:47,074 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 5/120: loss: 0.00508 Time taken: 0:00:13.249818 ETA: 0:25:23.729025
2020-09-25 05:21:50,614 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.860
2020-09-25 05:21:54,297 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.948
2020-09-25 05:21:57,980 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.939
2020-09-25 05:22:00,345 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 6/120: loss: 0.00330 Time taken: 0:00:13.274091 ETA: 0:25:13.246320
2020-09-25 05:22:01,671 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.866
2020-09-25 05:22:05,355 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.928
2020-09-25 05:22:09,046 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.872
2020-09-25 05:22:12,737 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.866
2020-09-25 05:22:13,632 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 7/120: loss: 0.00185 Time taken: 0:00:13.285375 ETA: 0:25:01.247388
2020-09-25 05:22:16,437 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.785
2020-09-25 05:22:20,123 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.918
2020-09-25 05:22:23,821 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.800
2020-09-25 05:22:26,935 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 8/120: loss: 0.00160 Time taken: 0:00:13.302382 ETA: 0:24:49.866810
2020-09-25 05:22:27,528 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.722
2020-09-25 05:22:31,220 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.864
2020-09-25 05:22:34,911 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.861
2020-09-25 05:22:38,599 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.903
2020-09-25 05:22:40,231 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 9/120: loss: 0.00127 Time taken: 0:00:13.296301 ETA: 0:24:35.889372
2020-09-25 05:22:42,297 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.798
2020-09-25 05:22:45,994 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.819
2020-09-25 05:22:49,692 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.797
2020-09-25 05:22:56,032 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.717
2020-09-25 05:22:56,209 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 10/120: loss: 0.00109 Time taken: 0:00:15.948317 ETA: 0:29:14.314902
2020-09-25 05:22:59,751 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.631
2020-09-25 05:23:03,463 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.679
2020-09-25 05:23:07,162 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.790
2020-09-25 05:23:09,535 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 11/120: loss: 0.00093 Time taken: 0:00:13.351778 ETA: 0:24:15.343779
2020-09-25 05:23:10,868 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.730
2020-09-25 05:23:14,571 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.761
2020-09-25 05:23:18,267 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.827
2020-09-25 05:23:21,969 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.760
2020-09-25 05:23:22,868 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 12/120: loss: 0.00086 Time taken: 0:00:13.330067 ETA: 0:23:59.647202
2020-09-25 05:23:25,687 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.626
2020-09-25 05:23:29,382 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.833
2020-09-25 05:23:33,086 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.748
2020-09-25 05:23:36,194 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 13/120: loss: 0.00082 Time taken: 0:00:13.326625 ETA: 0:23:45.948912
packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 17/120: loss: 0.00074 Time taken: 0:00:13.343099 ETA: 0:22:54.339160
2020-09-25 05:24:32,327 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.708
2020-09-25 05:24:36,027 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.783
2020-09-25 05:24:39,730 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.755
2020-09-25 05:24:42,851 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 18/120: loss: 0.00071 Time taken: 0:00:13.330578 ETA: 0:22:39.718989
2020-09-25 05:24:43,447 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.630
2020-09-25 05:24:47,141 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.842
2020-09-25 05:24:50,840 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.798
2020-09-25 05:24:54,559 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.614
2020-09-25 05:24:56,210 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 19/120: loss: 0.00070 Time taken: 0:00:13.344863 ETA: 0:22:27.831205
2020-09-25 05:24:58,280 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.594
2020-09-25 05:25:01,982 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.770
2020-09-25 05:25:05,686 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.747
2020-09-25 05:25:12,037 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.682
2020-09-25 05:25:12,202 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 20/120: loss: 0.00071 Time taken: 0:00:15.990352 ETA: 0:26:39.035168
2020-09-25 05:25:15,747 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.715
2020-09-25 05:25:19,444 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.809
2020-09-25 05:25:23,140 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.828
2020-09-25 05:25:25,514 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 21/120: loss: 0.00069 Time taken: 0:00:13.322730 ETA: 0:21:58.950229
2020-09-25 05:25:26,846 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.724
2020-09-25 05:25:30,542 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.823
2020-09-25 05:25:34,243 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.775
2020-09-25 05:25:37,942 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.799
2020-09-25 05:25:38,843 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 22/120: loss: 0.00069 Time taken: 0:00:13.331492 ETA: 0:21:46.486234
2020-09-25 05:25:41,653 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.688
2020-09-25 05:25:45,352 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.796
2020-09-25 05:25:49,047 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.827
2020-09-25 05:25:52,154 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 23/120: loss: 0.00068 Time taken: 0:00:13.308590 ETA: 0:21:30.933224
2020-09-25 05:25:52,748 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.778
2020-09-25 05:25:56,440 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.854
2020-09-25 05:26:00,399 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 31.576
2020-09-25 05:26:04,306 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 31.998
2020-09-25 05:26:05,988 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 24/120: loss: 0.00068 Time taken: 0:00:13.827487 ETA: 0:22:07.438774
2020-09-25 05:26:08,089 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.049
2020-09-25 05:26:11,770 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.955
2020-09-25 05:26:15,440 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 34.062
2020-09-25 05:26:19,122 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.955
2020-09-25 05:26:19,271 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 25/120: loss: 0.00067 Time taken: 0:00:13.288818 ETA: 0:21:02.437676
2020-09-25 05:26:22,800 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.988
2020-09-25 05:26:26,485 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.921
2020-09-25 05:26:30,172 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.911
2020-09-25 05:26:32,528 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 26/120: loss: 0.00067 Time taken: 0:00:13.257048 ETA: 0:20:46.162479
2020-09-25 05:26:33,854 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.946
2020-09-25 05:26:37,533 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.981
2020-09-25 05:26:41,208 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 34.018
2020-09-25 05:26:44,888 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.967
2020-09-25 05:26:45,776 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 27/120: loss: 0.00068 Time taken: 0:00:13.246758 ETA: 0:20:31.948448
2020-09-25 05:26:48,569 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.954
2020-09-25 05:26:52,243 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 34.031
2020-09-25 05:26:55,934 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.862
2020-09-25 05:26:59,036 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 28/120: loss: 0.00067 Time taken: 0:00:13.258445 ETA: 0:20:19.776964
2020-09-25 05:26:59,624 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.877
2020-09-25 05:27:03,305 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.964
2020-09-25 05:27:06,984 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.975
2020-09-25 05:27:10,664 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.975
2020-09-25 05:27:12,295 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 29/120: loss: 0.00067 Time taken: 0:00:13.260096 ETA: 0:20:06.668743
2020-09-25 05:27:14,362 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.800
2020-09-25 05:27:18,038 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 34.008
2020-09-25 05:27:21,716 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.987
2020-09-25 05:27:28,075 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.659
2020-09-25 05:27:28,241 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 30/120: loss: 0.00066 Time taken: 0:00:15.926454 ETA: 0:23:53.380866
2020-09-25 05:27:31,783 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.723
2020-09-25 05:27:35,460 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.997
2020-09-25 05:27:39,142 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.959
2020-09-25 05:27:41,504 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 31/120: loss: 0.00066 Time taken: 0:00:13.277555 ETA: 0:19:41.702394
2020-09-25 05:27:42,827 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.917
2020-09-25 05:27:46,506 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.981
2020-09-25 05:27:50,185 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.976
2020-09-25 05:27:53,865 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.970
2020-09-25 05:27:54,753 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 32/120: loss: 0.00065 Time taken: 0:00:13.248086 ETA: 0:19:25.831608
2020-09-25 05:27:57,542 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 34.001
2020-09-25 05:28:01,227 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.922
2020-09-25 05:28:04,913 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.915
2020-09-25 05:28:08,019 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 33/120: loss: 0.00065 Time taken: 0:00:13.265800 ETA: 0:19:14.124621
2020-09-25 05:28:08,610 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.808
2020-09-25 05:28:12,291 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.962
2020-09-25 05:28:15,968 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.996
2020-09-25 05:28:19,650 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.952
2020-09-25 05:28:21,273 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 34/120: loss: 0.00758 Time taken: 0:00:13.254507 ETA: 0:18:59.887608
2020-09-25 05:28:23,333 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.938
2020-09-25 05:28:27,011 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.989
2020-09-25 05:28:30,692 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.958
2020-09-25 05:28:34,378 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.919
2020-09-25 05:28:34,529 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 35/120: loss: 0.00078 Time taken: 0:00:13.251895 ETA: 0:18:46.411091
2020-09-25 05:28:38,064 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.921
2020-09-25 05:28:41,748 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.928
2020-09-25 05:28:45,428 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.968
2020-09-25 05:28:47,791 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 36/120: loss: 0.00074 Time taken: 0:00:13.262972 ETA: 0:18:34.089658
2020-09-25 05:28:49,118 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.882
2020-09-25 05:28:52,801 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.938
2020-09-25 05:28:56,487 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.912
2020-09-25 05:29:00,166 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.984
2020-09-25 05:29:01,067 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 37/120: loss: 0.00071 Time taken: 0:00:13.269735 ETA: 0:18:21.387993
2020-09-25 05:29:03,867 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.774
2020-09-25 05:29:07,545 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.988
2020-09-25 05:29:11,230 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.923
2020-09-25 05:29:14,331 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 38/120: loss: 0.00070 Time taken: 0:00:13.268082 ETA: 0:18:07.982716
2020-09-25 05:29:14,921 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.872
2020-09-25 05:29:18,607 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.908
2020-09-25 05:29:22,292 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.928
2020-09-25 05:29:25,972 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.970
2020-09-25 05:29:27,595 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 39/120: loss: 0.00070 Time taken: 0:00:13.263396 ETA: 0:17:54.335059
2020-09-25 05:29:29,650 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.984
2020-09-25 05:29:33,335 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.918
2020-09-25 05:29:37,021 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.915
2020-09-25 05:29:43,378 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.665
2020-09-25 05:29:43,540 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 40/120: loss: 0.00069 Time taken: 0:00:15.930472 ETA: 0:21:14.437752
2020-09-25 05:29:47,065 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.908
2020-09-25 05:29:50,742 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.999
2020-09-25 05:29:54,421 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.976
2020-09-25 05:29:56,789 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 41/120: loss: 0.00069 Time taken: 0:00:13.257629 ETA: 0:17:27.352722
2020-09-25 05:29:58,116 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.839
2020-09-25 05:30:01,794 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.980
2020-09-25 05:30:05,471 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.998
2020-09-25 05:30:09,152 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.964
2020-09-25 05:30:10,049 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 42/120: loss: 0.00068 Time taken: 0:00:13.259108 ETA: 0:17:14.210392
2020-09-25 05:30:12,843 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.866
2020-09-25 05:30:16,524 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.957
2020-09-25 05:30:20,207 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.947
2020-09-25 05:30:23,299 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 43/120: loss: 0.00068 Time taken: 0:00:13.252171 ETA: 0:17:00.417188
2020-09-25 05:30:23,888 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.957
2020-09-25 05:30:27,567 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.982
2020-09-25 05:30:31,240 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 34.027
2020-09-25 05:30:34,917 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.998
2020-09-25 05:30:36,543 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 44/120: loss: 0.00069 Time taken: 0:00:13.242719 ETA: 0:16:46.446639
2020-09-25 05:30:38,607 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.877
2020-09-25 05:30:42,289 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.955
2020-09-25 05:30:45,975 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.910
2020-09-25 05:30:49,662 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.904
2020-09-25 05:30:49,811 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 45/120: loss: 0.00067 Time taken: 0:00:13.267299 ETA: 0:16:35.047438
2020-09-25 05:30:53,346 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.938
2020-09-25 05:30:57,031 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.925
2020-09-25 05:31:00,713 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.954
2020-09-25 05:31:03,080 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 46/120: loss: 0.00067 Time taken: 0:00:13.268861 ETA: 0:16:21.895736
2020-09-25 05:31:04,408 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.829
2020-09-25 05:31:08,084 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 34.004
2020-09-25 05:31:11,759 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 34.019
2020-09-25 05:31:15,439 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.969
2020-09-25 05:31:16,328 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 47/120: loss: 0.00067 Time taken: 0:00:13.244247 ETA: 0:16:06.830028
2020-09-25 05:31:19,125 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.909
2020-09-25 05:31:22,808 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.945
2020-09-25 05:31:26,492 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.933
2020-09-25 05:31:29,599 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 48/120: loss: 0.00067 Time taken: 0:00:13.270696 ETA: 0:15:55.490124
2020-09-25 05:31:30,187 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.833
2020-09-25 05:31:33,869 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.948
2020-09-25 05:31:37,552 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.939
2020-09-25 05:31:41,233 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.964
2020-09-25 05:31:42,860 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 49/120: loss: 0.00067 Time taken: 0:00:13.259679 ETA: 0:15:41.437215
2020-09-25 05:31:44,922 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.888
2020-09-25 05:31:48,601 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.979
2020-09-25 05:31:52,279 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.984
2020-09-25 05:31:58,631 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.679
2020-09-25 05:31:58,795 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 50/120: loss: 0.00066 Time taken: 0:00:15.921100 ETA: 0:18:34.477026
2020-09-25 05:32:02,321 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.885
2020-09-25 05:32:05,996 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 34.018
2020-09-25 05:32:09,677 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.959
2020-09-25 05:32:12,041 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 51/120: loss: 0.00066 Time taken: 0:00:13.260289 ETA: 0:15:14.959938
2020-09-25 05:32:13,365 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.895
2020-09-25 05:32:17,035 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 34.057
2020-09-25 05:32:20,712 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 34.002
2020-09-25 05:32:24,382 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 34.055
2020-09-25 05:32:25,272 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 52/120: loss: 0.00066 Time taken: 0:00:13.229429 ETA: 0:14:59.601189
2020-09-25 05:32:28,064 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.954
2020-09-25 05:32:31,745 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.962
2020-09-25 05:32:35,432 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.905
2020-09-25 05:32:38,527 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 53/120: loss: 0.00066 Time taken: 0:00:13.255558 ETA: 0:14:48.122387
2020-09-25 05:32:39,116 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.926
2020-09-25 05:32:42,804 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.896
2020-09-25 05:32:46,490 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.919
2020-09-25 05:32:50,167 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.998
2020-09-25 05:32:51,796 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 54/120: loss: 0.00066 Time taken: 0:00:13.267325 ETA: 0:14:35.643429
2020-09-25 05:32:53,855 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.894
2020-09-25 05:32:57,544 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.884
2020-09-25 05:33:01,231 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.903
2020-09-25 05:33:04,921 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.881
2020-09-25 05:33:05,070 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 55/120: loss: 0.00067 Time taken: 0:00:13.272705 ETA: 0:14:22.725830
2020-09-25 05:33:08,604 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.945
2020-09-25 05:33:12,288 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.937
2020-09-25 05:33:15,976 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.896
2020-09-25 05:33:18,336 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 56/120: loss: 0.00065 Time taken: 0:00:13.262952 ETA: 0:14:08.828918
2020-09-25 05:33:19,659 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.939
2020-09-25 05:33:23,347 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.892
2020-09-25 05:33:27,034 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.911
2020-09-25 05:33:30,713 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.975
2020-09-25 05:33:31,608 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 57/120: loss: 0.00065 Time taken: 0:00:13.268925 ETA: 0:13:55.942287
2020-09-25 05:33:34,402 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.882
2020-09-25 05:33:38,091 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.888
2020-09-25 05:33:41,773 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.953
2020-09-25 05:33:44,874 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 58/120: loss: 0.00065 Time taken: 0:00:13.267169 ETA: 0:13:42.564448
2020-09-25 05:33:45,461 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.899
2020-09-25 05:33:49,140 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.975
2020-09-25 05:33:52,819 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.978
2020-09-25 05:33:56,503 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.933
2020-09-25 05:33:58,130 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 59/120: loss: 0.00064 Time taken: 0:00:13.258810 ETA: 0:13:28.787413
2020-09-25 05:34:00,196 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.853
2020-09-25 05:34:03,880 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.933
2020-09-25 05:34:07,558 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.980
2020-09-25 05:34:13,825 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.947
2020-09-25 05:34:13,988 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 60/120: loss: 0.00064 Time taken: 0:00:15.842089 ETA: 0:15:50.525322
2020-09-25 05:34:17,519 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.848
2020-09-25 05:34:21,188 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 34.072
2020-09-25 05:34:24,875 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.904
2020-09-25 05:34:27,244 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 61/120: loss: 0.00064 Time taken: 0:00:13.259550 ETA: 0:13:02.313442
2020-09-25 05:34:28,568 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.849
2020-09-25 05:34:32,247 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.977
2020-09-25 05:34:35,931 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.935
2020-09-25 05:34:39,607 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 34.010
2020-09-25 05:34:40,493 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 62/120: loss: 0.00064 Time taken: 0:00:13.256494 ETA: 0:12:48.876668
2020-09-25 05:34:43,290 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.941
2020-09-25 05:34:46,970 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.964
2020-09-25 05:34:50,649 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.982
2020-09-25 05:34:53,756 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 63/120: loss: 0.00064 Time taken: 0:00:13.259691 ETA: 0:12:35.802387
2020-09-25 05:40:45,564 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.924
2020-09-25 05:40:49,244 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.968
2020-09-25 05:40:52,927 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.943
2020-09-25 05:40:59,174 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 20.008
2020-09-25 05:40:59,337 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 90/120: loss: 0.00052 Time taken: 0:00:15.818256 ETA: 0:07:54.547691
2020-09-25 05:41:02,867 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.860
2020-09-25 05:41:06,545 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.984
2020-09-25 05:41:10,221 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 34.004
2020-09-25 05:41:12,582 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 91/120: loss: 0.00052 Time taken: 0:00:13.258513 ETA: 0:06:24.496890
2020-09-25 05:41:13,907 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.913
2020-09-25 05:41:17,581 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 34.030
2020-09-25 05:41:21,261 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.968
2020-09-25 05:41:24,936 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 34.013
2020-09-25 05:41:25,826 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 92/120: loss: 0.00052 Time taken: 0:00:13.242242 ETA: 0:06:10.782772
2020-09-25 05:41:28,622 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.911
2020-09-25 05:41:32,310 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.899
2020-09-25 05:41:35,993 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.942
2020-09-25 05:41:39,085 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 93/120: loss: 0.00052 Time taken: 0:00:13.259746 ETA: 0:05:58.013131
2020-09-25 05:41:39,673 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.972
2020-09-25 05:41:43,352 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.973
2020-09-25 05:41:47,029 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 34.001
2020-09-25 05:41:50,705 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 34.001
2020-09-25 05:41:52,331 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 94/120: loss: 0.00051 Time taken: 0:00:13.244038 ETA: 0:05:44.344997
2020-09-25 05:41:54,394 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.890
2020-09-25 05:41:58,076 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.948
2020-09-25 05:42:01,752 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 34.007
2020-09-25 05:42:05,445 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.852
2020-09-25 05:42:05,594 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 95/120: loss: 0.00051 Time taken: 0:00:13.262260 ETA: 0:05:31.556511
2020-09-25 05:42:09,128 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.946
2020-09-25 05:42:12,812 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.929
2020-09-25 05:42:16,492 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.972
2020-09-25 05:42:18,856 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 96/120: loss: 0.00051 Time taken: 0:00:13.259517 ETA: 0:05:18.228418
2020-09-25 05:42:20,181 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.884
2020-09-25 05:42:23,973 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 32.966
2020-09-25 05:42:27,709 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.460
2020-09-25 05:42:31,605 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 32.088
2020-09-25 05:42:32,562 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 97/120: loss: 0.00050 Time taken: 0:00:13.698739 ETA: 0:05:15.070993
2020-09-25 05:42:35,577 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 31.474
2020-09-25 05:42:39,364 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.009
2020-09-25 05:42:43,151 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.008
2020-09-25 05:42:46,256 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 98/120: loss: 0.00050 Time taken: 0:00:13.701441 ETA: 0:05:01.431708
2020-09-25 05:42:46,852 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.778
2020-09-25 05:42:50,555 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.757
2020-09-25 05:42:54,277 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.582
2020-09-25 05:42:57,991 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.661
2020-09-25 05:42:59,634 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 99/120: loss: 0.00050 Time taken: 0:00:13.377351 ETA: 0:04:40.924367
2020-09-25 05:43:01,708 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.629
2020-09-25 05:43:05,405 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.814
2020-09-25 05:43:09,110 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.738
2020-09-25 05:43:15,365 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.986
2020-09-25 05:43:15,536 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 100/120: loss: 0.00050 Time taken: 0:00:15.878635 ETA: 0:05:17.572703
2020-09-25 05:43:19,089 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.573
2020-09-25 05:43:22,825 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.455
2020-09-25 05:43:26,794 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 31.504
2020-09-25 05:43:29,188 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 101/120: loss: 0.00050 Time taken: 0:00:13.670250 ETA: 0:04:19.734758
2020-09-25 05:43:30,543 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.342
2020-09-25 05:43:34,438 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 32.095
2020-09-25 05:43:38,295 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 32.409
2020-09-25 05:43:42,407 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 30.401
2020-09-25 05:43:43,326 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 102/120: loss: 0.00050 Time taken: 0:00:14.132929 ETA: 0:04:14.392724
2020-09-25 05:43:46,425 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 31.112
2020-09-25 05:43:50,249 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 32.695
2020-09-25 05:43:53,943 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.838
2020-09-25 05:43:57,063 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 103/120: loss: 0.00050 Time taken: 0:00:13.739446 ETA: 0:03:53.570581
2020-09-25 05:43:57,659 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.635
2020-09-25 05:44:01,362 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.765
2020-09-25 05:44:05,057 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.830
2020-09-25 05:44:08,752 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.826
2020-09-25 05:44:10,387 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 104/120: loss: 0.00050 Time taken: 0:00:13.325872 ETA: 0:03:33.213959
2020-09-25 05:44:12,455 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.761
2020-09-25 05:44:16,157 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.765
2020-09-25 05:44:19,866 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.708
2020-09-25 05:44:23,572 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.728
2020-09-25 05:44:23,724 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 105/120: loss: 0.00050 Time taken: 0:00:13.333118 ETA: 0:03:19.996777
2020-09-25 05:44:27,281 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.713
2020-09-25 05:44:30,981 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.786
2020-09-25 05:44:34,683 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.762
2020-09-25 05:44:37,060 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 106/120: loss: 0.00049 Time taken: 0:00:13.336954 ETA: 0:03:06.717358
2020-09-25 05:44:38,395 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.678
2020-09-25 05:44:42,096 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.781
2020-09-25 05:44:45,793 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.810
2020-09-25 05:44:49,494 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.771
2020-09-25 05:44:50,394 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 107/120: loss: 0.00049 Time taken: 0:00:13.332589 ETA: 0:02:53.323653
2020-09-25 05:44:53,211 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.633
2020-09-25 05:44:56,951 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.424
2020-09-25 05:45:00,666 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.648
2020-09-25 05:45:04,075 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 108/120: loss: 0.00049 Time taken: 0:00:13.649482 ETA: 0:02:43.793787
2020-09-25 05:45:04,743 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 30.662
2020-09-25 05:45:08,524 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.066
2020-09-25 05:45:12,390 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 32.336
2020-09-25 05:45:16,084 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.834
2020-09-25 05:45:17,715 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 109/120: loss: 0.00049 Time taken: 0:00:13.671419 ETA: 0:02:30.385613
2020-09-25 05:45:19,785 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.784
2020-09-25 05:45:23,488 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.757
2020-09-25 05:45:27,192 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.746
2020-09-25 05:45:33,489 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 19.850
2020-09-25 05:45:33,656 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 110/120: loss: 0.00049 Time taken: 0:00:15.922817 ETA: 0:02:39.228168
2020-09-25 05:45:37,203 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.665
2020-09-25 05:45:40,897 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.840
2020-09-25 05:45:44,589 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.863
2020-09-25 05:45:46,965 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 111/120: loss: 0.00049 Time taken: 0:00:13.324021 ETA: 0:01:59.916186
2020-09-25 05:45:48,296 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.724
2020-09-25 05:45:51,996 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.779
2020-09-25 05:45:55,936 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 31.734
2020-09-25 05:45:59,707 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.148
2020-09-25 05:46:00,611 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 112/120: loss: 0.00049 Time taken: 0:00:13.645939 ETA: 0:01:49.167509
2020-09-25 05:46:03,433 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.551
2020-09-25 05:46:07,135 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.768
2020-09-25 05:46:10,835 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.784
2020-09-25 05:46:13,952 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 113/120: loss: 0.00049 Time taken: 0:00:13.340894 ETA: 0:01:33.386256
2020-09-25 05:46:14,545 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.695
2020-09-25 05:46:18,253 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.709
2020-09-25 05:46:22,008 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.288
2020-09-25 05:46:25,713 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.739
2020-09-25 05:46:27,345 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 114/120: loss: 0.00049 Time taken: 0:00:13.392524 ETA: 0:01:20.355143
2020-09-25 05:46:29,419 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.736
2020-09-25 05:46:33,114 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.828
2020-09-25 05:46:36,811 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.812
2020-09-25 05:46:40,519 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.711
2020-09-25 05:46:40,673 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 115/120: loss: 0.00049 Time taken: 0:00:13.322469 ETA: 0:01:06.612343
2020-09-25 05:46:44,219 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.795
2020-09-25 05:46:47,920 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.776
2020-09-25 05:46:51,620 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.791
2020-09-25 05:46:53,999 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 116/120: loss: 0.00049 Time taken: 0:00:13.328723 ETA: 0:00:53.314891
2020-09-25 05:46:55,360 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.423
2020-09-25 05:46:59,082 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.580
2020-09-25 05:47:02,778 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.829
2020-09-25 05:47:06,480 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.764
2020-09-25 05:47:07,377 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 117/120: loss: 0.00049 Time taken: 0:00:13.377530 ETA: 0:00:40.132590
2020-09-25 05:47:10,191 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.688
2020-09-25 05:47:13,911 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.600
2020-09-25 05:47:17,610 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.797
2020-09-25 05:47:20,753 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 118/120: loss: 0.00049 Time taken: 0:00:13.369365 ETA: 0:00:26.738730
2020-09-25 05:47:21,402 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 32.968
2020-09-25 05:47:25,388 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 31.358
2020-09-25 05:47:29,101 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.667
2020-09-25 05:47:32,800 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.792
2020-09-25 05:47:34,439 [INFO] /usr/local/lib/python3.6/dist-packages/modulus/hooks/task_progress_monitor_hook.pyc: Epoch 119/120: loss: 0.00049 Time taken: 0:00:13.687866 ETA: 0:00:13.687866
2020-09-25 05:47:36,514 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.666
2020-09-25 05:47:40,219 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.738
2020-09-25 05:47:43,918 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 33.797
2020-09-25 05:47:50,203 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 22, 0.00s/step
2020-09-25 05:47:51,509 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 22, 0.13s/step
2020-09-25 05:47:52,229 [INFO] iva.detectnet_v2.evaluation.evaluation: step 20 / 22, 0.07s/step
Epoch 120/120

Validation cost: -0.000004
Mean average_precision (in %): 0.0000

class name average precision (in %)


bicycle 0
bus 0
car 0
motorcycle 0
person 0
truck 0

Median Inference Time: 0.014106
2020-09-25 05:47:52,400 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 14.737
2020-09-25 05:47:52,848 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 14.737
Time taken to run iva.detectnet_v2.scripts.train:main: 0:28:00.696113.

Is there anything wrong with the configuration files?

There should be something wrong in your label.

Image size: 480*288

example label (kitti) :
car 0.0 0 0.0 268.4 391.1 580.6 572.9 0.0 0.0 0.0 0.0 0.0 0.0 0.0
person 0.0 0 0.0 281.1 300.2 315.9 379.9 0.0 0.0 0.0 0.0 0.0 0.0 0.0

Why one 480x288 image has a bbox (268.4 391.1 580.6 572.9) ?

More,

  1. suggest training only two or three classes. Car, person, bus.
    Because in your dataset, other classes’ training images are not enough.
  2. set lower minimum_bounding_box_height, for example, set to 4.

Hi MorganH, thanks for the reply.

I create the dataset using cvat annotation tool. and thats the result of creating bounding boxes.
Is TLT only accept integer value instead of float?

  1. I tried just train “car” class and still got same value (0 map after 120 epoch)
  2. I did this one too.

Ahh… I just realised I haven’t change the label coordinates to 480*288 resolution.
Let me try it again

Yes, please make sure your label is correct.
That’s why I asked “Why one 480x288 image has a bbox (268.4 391.1 580.6 572.9) ?”