Error while training on coco dataset

Accelerated Computing Intelligent Video Analytics TAO Toolkit
1

I’m getting the following error, while trying to train a retinanet resnet_18 model with tao toolkit, any idea what this error means or how I can resolve?

File “/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/retinanet/scripts/train.py”, line 390, in \n’
b’ File “/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/common/utils.py”, line 528, in return_func\n’
b’ File “/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/common/utils.py”, line 516, in return_func\n’
b’ File “/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/retinanet/scripts/train.py”, line 386, in main\n’
b’ File “/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/retinanet/scripts/train.py”, line 308, in run_experiment\n’
b’ File “/usr/local/lib/python3.6/dist-packages/keras/engine/training.py”, line 1039, in fit\n’
b’ validation_steps=validation_steps)\n’
b’ File “/usr/local/lib/python3.6/dist-packages/keras/engine/training_arrays.py”, line 154, in fit_loop\n’
b’ outs = f(ins)\n’
b’ File “/usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py”, line 2715, in call\n’
b’ return self._call(inputs)\n’
b’ File “/usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py”, line 2675, in _call\n’
b’ fetched = self._callable_fn(*array_vals)\n’
b’ File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py”, line 1472, in call\n’
b’ run_metadata_ptr)\n’
b’tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.\n’
b’ (0) Invalid argument: ConcatOp : Ranks of all input tensors should match: shape[0] = [1] vs. shape[1] = \n’
b’\t [[{{node concat}}]]\n’
b’\t [[cond_3/SliceReplace_2/range/_3453]]\n’
b’ (1) Invalid argument: ConcatOp : Ranks of all input tensors should match: shape[0] = [1] vs. shape[1] = \n’
b’\t [[{{node concat}}]]\n’
b’0 successful operations.\n’
b’0 derived errors ignored.\n’
b’libpng warning: iCCP: known incorrect sRGB profile\n’

2

Can you share your training spec file and training command?

3

random_seed: 42
retinanet_config {
aspect_ratios_global: “[1.0, 2.0, 0.5]”
scales: “[0.045, 0.09, 0.2, 0.4, 0.55, 0.7]”
two_boxes_for_ar1: false
clip_boxes: false
loss_loc_weight: 0.8
focal_loss_alpha: 0.25
focal_loss_gamma: 2.0
variances: “[0.1, 0.1, 0.2, 0.2]”
arch: “resnet”
nlayers: 18
n_kernels: 1
n_anchor_levels: 1
feature_size: 256
freeze_bn: False
freeze_blocks: 0
}
training_config {
enable_qat: False
resume_model_path: “/workspace/models/30954d01-3592-47c3-aeea-baa5faf464a6/unpruned_model/weights/retinanet_resnet18_epoch_040.tlt”
batch_size_per_gpu: 8
num_epochs: 40
n_workers: 2
checkpoint_interval: 10
learning_rate {
soft_start_annealing_schedule {
min_learning_rate: 4e-5
max_learning_rate: 1.5e-2
soft_start: 0.1
annealing: 0.3
}
}
regularizer {
type: L1
weight: 2e-5
}
optimizer {
sgd {
momentum: 0.9
nesterov: True
}
}
}
eval_config {
validation_period_during_training: 10
average_precision_mode: SAMPLE
batch_size: 8
matching_iou_threshold: 0.5
}
nms_config {
confidence_threshold: 0.01
clustering_iou_threshold: 0.6
top_k: 200
}
augmentation_config {
output_width: 640
output_height: 480
output_channel: 3
}
dataset_config {
data_sources: {
tfrecords_path: “/workspace/models/30954d01-3592-47c3-aeea-baa5faf464a6/data/train/tfrecord/train-*”
}
target_class_mapping {
key: “object”
value: “object”
}
validation_data_sources: {
image_directory_path: “/workspace/models/30954d01-3592-47c3-aeea-baa5faf464a6/data/val/image”
label_directory_path: “/workspace/models/30954d01-3592-47c3-aeea-baa5faf464a6/data/val/label”
}

Screen Shot 2022-05-24 at 8.55.12 AM
Screen Shot 2022-05-24 at 8.55.12 AM1898×646 159 KB

4

Could you please attach the full log?

5

Using TensorFlow backend.
Using TensorFlow backend.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/retinanet/scripts/train.py:61: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

2022-05-24 14:22:41,657 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/retinanet/scripts/train.py:61: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/retinanet/scripts/train.py:64: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

2022-05-24 14:22:41,658 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/retinanet/scripts/train.py:64: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

2022-05-24 14:22:42,102 [INFO] main: Loading experiment spec at /workspace/models/30954d01-3592-47c3-aeea-baa5faf464a6/specs/train-specs.txt.
2022-05-24 14:22:42,103 [INFO] iva.retinanet.utils.spec_loader: Merging specification from /workspace/models/30954d01-3592-47c3-aeea-baa5faf464a6/specs/train-specs.txt
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:153: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.

2022-05-24 14:22:42,104 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:153: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.

2022-05-24 14:22:42,105 [INFO] main: Using DALI dataloader…
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.

2022-05-24 14:22:42,192 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1834: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead.

2022-05-24 14:22:42,202 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1834: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4185: The name tf.truncated_normal is deprecated. Please use tf.random.truncated_normal instead.

2022-05-24 14:22:42,479 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4185: The name tf.truncated_normal is deprecated. Please use tf.random.truncated_normal instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:2018: The name tf.image.resize_nearest_neighbor is deprecated. Please use tf.compat.v1.image.resize_nearest_neighbor instead.

2022-05-24 14:22:42,493 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:2018: The name tf.image.resize_nearest_neighbor is deprecated. Please use tf.compat.v1.image.resize_nearest_neighbor instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4115: The name tf.random_normal is deprecated. Please use tf.random.normal instead.

2022-05-24 14:22:42,590 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4115: The name tf.random_normal is deprecated. Please use tf.random.normal instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

2022-05-24 14:22:42,696 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

WARNING:tensorflow:From /opt/nvidia/third_party/keras/tensorflow_backend.py:187: The name tf.nn.avg_pool is deprecated. Please use tf.nn.avg_pool2d instead.

2022-05-24 14:22:43,503 [WARNING] tensorflow: From /opt/nvidia/third_party/keras/tensorflow_backend.py:187: The name tf.nn.avg_pool is deprecated. Please use tf.nn.avg_pool2d instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.

2022-05-24 14:22:43,673 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:190: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.

2022-05-24 14:22:43,673 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:190: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead.

2022-05-24 14:22:43,673 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead.

2022-05-24 14:22:43,962 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/optimizers.py:790: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.

2022-05-24 14:22:44,324 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/optimizers.py:790: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:3295: The name tf.log is deprecated. Please use tf.math.log instead.

2022-05-24 14:22:44,326 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:3295: The name tf.log is deprecated. Please use tf.math.log instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:986: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead.

2022-05-24 14:22:44,701 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:986: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:973: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead.

2022-05-24 14:22:44,794 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:973: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead.

2022-05-24 14:22:45,634 [INFO] iva.retinanet.utils.model_io: Loading model weights…

Layer (type) Output Shape Param # Connected to

Input (InputLayer) (8, 3, 480, 640) 0

conv1 (Conv2D) (8, 64, 240, 320) 9408 Input[0][0]

bn_conv1 (BatchNormalization) (8, 64, 240, 320) 256 conv1[0][0]

activation_1 (Activation) (8, 64, 240, 320) 0 bn_conv1[0][0]

block_1a_conv_1 (Conv2D) (8, 64, 120, 160) 36864 activation_1[0][0]

block_1a_bn_1 (BatchNormalizati (8, 64, 120, 160) 256 block_1a_conv_1[0][0]

block_1a_relu_1 (Activation) (8, 64, 120, 160) 0 block_1a_bn_1[0][0]

block_1a_conv_2 (Conv2D) (8, 64, 120, 160) 36864 block_1a_relu_1[0][0]

block_1a_conv_shortcut (Conv2D) (8, 64, 120, 160) 4096 activation_1[0][0]

block_1a_bn_2 (BatchNormalizati (8, 64, 120, 160) 256 block_1a_conv_2[0][0]

block_1a_bn_shortcut (BatchNorm (8, 64, 120, 160) 256 block_1a_conv_shortcut[0][0]

add_1 (Add) (8, 64, 120, 160) 0 block_1a_bn_2[0][0]
block_1a_bn_shortcut[0][0]

block_1a_relu (Activation) (8, 64, 120, 160) 0 add_1[0][0]

block_1b_conv_1 (Conv2D) (8, 64, 120, 160) 36864 block_1a_relu[0][0]

block_1b_bn_1 (BatchNormalizati (8, 64, 120, 160) 256 block_1b_conv_1[0][0]

block_1b_relu_1 (Activation) (8, 64, 120, 160) 0 block_1b_bn_1[0][0]

block_1b_conv_2 (Conv2D) (8, 64, 120, 160) 36864 block_1b_relu_1[0][0]

block_1b_conv_shortcut (Conv2D) (8, 64, 120, 160) 4096 block_1a_relu[0][0]

block_1b_bn_2 (BatchNormalizati (8, 64, 120, 160) 256 block_1b_conv_2[0][0]

block_1b_bn_shortcut (BatchNorm (8, 64, 120, 160) 256 block_1b_conv_shortcut[0][0]

add_2 (Add) (8, 64, 120, 160) 0 block_1b_bn_2[0][0]
block_1b_bn_shortcut[0][0]

block_1b_relu (Activation) (8, 64, 120, 160) 0 add_2[0][0]

block_2a_conv_1 (Conv2D) (8, 128, 60, 80) 73728 block_1b_relu[0][0]

block_2a_bn_1 (BatchNormalizati (8, 128, 60, 80) 512 block_2a_conv_1[0][0]

block_2a_relu_1 (Activation) (8, 128, 60, 80) 0 block_2a_bn_1[0][0]

block_2a_conv_2 (Conv2D) (8, 128, 60, 80) 147456 block_2a_relu_1[0][0]

block_2a_conv_shortcut (Conv2D) (8, 128, 60, 80) 8192 block_1b_relu[0][0]

block_2a_bn_2 (BatchNormalizati (8, 128, 60, 80) 512 block_2a_conv_2[0][0]

block_2a_bn_shortcut (BatchNorm (8, 128, 60, 80) 512 block_2a_conv_shortcut[0][0]

add_3 (Add) (8, 128, 60, 80) 0 block_2a_bn_2[0][0]
block_2a_bn_shortcut[0][0]

block_2a_relu (Activation) (8, 128, 60, 80) 0 add_3[0][0]

block_2b_conv_1 (Conv2D) (8, 128, 60, 80) 147456 block_2a_relu[0][0]

block_2b_bn_1 (BatchNormalizati (8, 128, 60, 80) 512 block_2b_conv_1[0][0]

block_2b_relu_1 (Activation) (8, 128, 60, 80) 0 block_2b_bn_1[0][0]

block_2b_conv_2 (Conv2D) (8, 128, 60, 80) 147456 block_2b_relu_1[0][0]

block_2b_conv_shortcut (Conv2D) (8, 128, 60, 80) 16384 block_2a_relu[0][0]

block_2b_bn_2 (BatchNormalizati (8, 128, 60, 80) 512 block_2b_conv_2[0][0]

block_2b_bn_shortcut (BatchNorm (8, 128, 60, 80) 512 block_2b_conv_shortcut[0][0]

add_4 (Add) (8, 128, 60, 80) 0 block_2b_bn_2[0][0]
block_2b_bn_shortcut[0][0]

block_2b_relu (Activation) (8, 128, 60, 80) 0 add_4[0][0]

block_3a_conv_1 (Conv2D) (8, 256, 30, 40) 294912 block_2b_relu[0][0]

block_3a_bn_1 (BatchNormalizati (8, 256, 30, 40) 1024 block_3a_conv_1[0][0]

block_3a_relu_1 (Activation) (8, 256, 30, 40) 0 block_3a_bn_1[0][0]

block_3a_conv_2 (Conv2D) (8, 256, 30, 40) 589824 block_3a_relu_1[0][0]

block_3a_conv_shortcut (Conv2D) (8, 256, 30, 40) 32768 block_2b_relu[0][0]

block_3a_bn_2 (BatchNormalizati (8, 256, 30, 40) 1024 block_3a_conv_2[0][0]

block_3a_bn_shortcut (BatchNorm (8, 256, 30, 40) 1024 block_3a_conv_shortcut[0][0]

add_5 (Add) (8, 256, 30, 40) 0 block_3a_bn_2[0][0]
block_3a_bn_shortcut[0][0]

block_3a_relu (Activation) (8, 256, 30, 40) 0 add_5[0][0]

block_3b_conv_1 (Conv2D) (8, 256, 30, 40) 589824 block_3a_relu[0][0]

block_3b_bn_1 (BatchNormalizati (8, 256, 30, 40) 1024 block_3b_conv_1[0][0]

block_3b_relu_1 (Activation) (8, 256, 30, 40) 0 block_3b_bn_1[0][0]

block_3b_conv_2 (Conv2D) (8, 256, 30, 40) 589824 block_3b_relu_1[0][0]

block_3b_conv_shortcut (Conv2D) (8, 256, 30, 40) 65536 block_3a_relu[0][0]

block_3b_bn_2 (BatchNormalizati (8, 256, 30, 40) 1024 block_3b_conv_2[0][0]

block_3b_bn_shortcut (BatchNorm (8, 256, 30, 40) 1024 block_3b_conv_shortcut[0][0]

add_6 (Add) (8, 256, 30, 40) 0 block_3b_bn_2[0][0]
block_3b_bn_shortcut[0][0]

block_3b_relu (Activation) (8, 256, 30, 40) 0 add_6[0][0]

block_4a_conv_1 (Conv2D) (8, 512, 30, 40) 1179648 block_3b_relu[0][0]

block_4a_bn_1 (BatchNormalizati (8, 512, 30, 40) 2048 block_4a_conv_1[0][0]

block_4a_relu_1 (Activation) (8, 512, 30, 40) 0 block_4a_bn_1[0][0]

block_4a_conv_2 (Conv2D) (8, 512, 30, 40) 2359296 block_4a_relu_1[0][0]

block_4a_conv_shortcut (Conv2D) (8, 512, 30, 40) 131072 block_3b_relu[0][0]

block_4a_bn_2 (BatchNormalizati (8, 512, 30, 40) 2048 block_4a_conv_2[0][0]

block_4a_bn_shortcut (BatchNorm (8, 512, 30, 40) 2048 block_4a_conv_shortcut[0][0]

add_7 (Add) (8, 512, 30, 40) 0 block_4a_bn_2[0][0]
block_4a_bn_shortcut[0][0]

block_4a_relu (Activation) (8, 512, 30, 40) 0 add_7[0][0]

block_4b_conv_1 (Conv2D) (8, 512, 30, 40) 2359296 block_4a_relu[0][0]

block_4b_bn_1 (BatchNormalizati (8, 512, 30, 40) 2048 block_4b_conv_1[0][0]

block_4b_relu_1 (Activation) (8, 512, 30, 40) 0 block_4b_bn_1[0][0]

block_4b_conv_2 (Conv2D) (8, 512, 30, 40) 2359296 block_4b_relu_1[0][0]

block_4b_conv_shortcut (Conv2D) (8, 512, 30, 40) 262144 block_4a_relu[0][0]

block_4b_bn_2 (BatchNormalizati (8, 512, 30, 40) 2048 block_4b_conv_2[0][0]

block_4b_bn_shortcut (BatchNorm (8, 512, 30, 40) 2048 block_4b_conv_shortcut[0][0]

add_8 (Add) (8, 512, 30, 40) 0 block_4b_bn_2[0][0]
block_4b_bn_shortcut[0][0]

block_4b_relu (Activation) (8, 512, 30, 40) 0 add_8[0][0]

expand_conv1 (Conv2D) (8, 256, 15, 20) 1179904 block_4b_relu[0][0]

expand1_relu (ReLU) (8, 256, 15, 20) 0 expand_conv1[0][0]

C5_reduced (Conv2D) (8, 256, 15, 20) 65792 expand1_relu[0][0]

P5_upsampled (UpSampling2D) (8, 256, 30, 40) 0 C5_reduced[0][0]

C4_reduced (Conv2D) (8, 256, 30, 40) 131328 block_4b_relu[0][0]

P4_merged (Add) (8, 256, 30, 40) 0 P5_upsampled[0][0]
C4_reduced[0][0]

P4_upsampled (UpSampling2D) (8, 256, 60, 80) 0 P4_merged[0][0]

C3_reduced (Conv2D) (8, 256, 60, 80) 33024 block_2b_relu[0][0]

P6 (Conv2D) (8, 256, 8, 10) 590080 expand1_relu[0][0]

P3_merged (Add) (8, 256, 60, 80) 0 P4_upsampled[0][0]
C3_reduced[0][0]

P6_relu (ReLU) (8, 256, 8, 10) 0 P6[0][0]

P3 (Conv2D) (8, 256, 60, 80) 590080 P3_merged[0][0]

P4 (Conv2D) (8, 256, 30, 40) 590080 P4_merged[0][0]

P5 (Conv2D) (8, 256, 15, 20) 590080 C5_reduced[0][0]

P7 (Conv2D) (8, 256, 4, 5) 590080 P6_relu[0][0]

P3_relu (ReLU) (8, 256, 60, 80) 0 P3[0][0]

P4_relu (ReLU) (8, 256, 30, 40) 0 P4[0][0]

P5_relu (ReLU) (8, 256, 15, 20) 0 P5[0][0]

P7_relu (ReLU) (8, 256, 4, 5) 0 P7[0][0]

retinanet_class_subn_0 (Conv2D) multiple 590080 P3_relu[0][0]
P4_relu[0][0]
P5_relu[0][0]
P6_relu[0][0]
P7_relu[0][0]

retinanet_conf_regressor (Conv2 multiple 48405 retinanet_class_subn_0[0][0]
retinanet_class_subn_0[1][0]
retinanet_class_subn_0[2][0]
retinanet_class_subn_0[3][0]
retinanet_class_subn_0[4][0]

retinanet_loc_subn_0 (Conv2D) multiple 590080 P3_relu[0][0]
P4_relu[0][0]
P5_relu[0][0]
P6_relu[0][0]
P7_relu[0][0]

permute_1 (Permute) (8, 60, 80, 21) 0 retinanet_conf_regressor[0][0]

permute_3 (Permute) (8, 30, 40, 21) 0 retinanet_conf_regressor[1][0]

permute_5 (Permute) (8, 15, 20, 21) 0 retinanet_conf_regressor[2][0]

permute_7 (Permute) (8, 8, 10, 21) 0 retinanet_conf_regressor[3][0]

permute_9 (Permute) (8, 4, 5, 21) 0 retinanet_conf_regressor[4][0]

retinanet_loc_regressor (Conv2D multiple 27660 retinanet_loc_subn_0[0][0]
retinanet_loc_subn_0[1][0]
retinanet_loc_subn_0[2][0]
retinanet_loc_subn_0[3][0]
retinanet_loc_subn_0[4][0]

conf_reshape_0 (Reshape) (8, 14400, 1, 7) 0 permute_1[0][0]

conf_reshape_1 (Reshape) (8, 3600, 1, 7) 0 permute_3[0][0]

conf_reshape_2 (Reshape) (8, 900, 1, 7) 0 permute_5[0][0]

conf_reshape_3 (Reshape) (8, 240, 1, 7) 0 permute_7[0][0]

conf_reshape_4 (Reshape) (8, 60, 1, 7) 0 permute_9[0][0]

permute_2 (Permute) (8, 60, 80, 12) 0 retinanet_loc_regressor[0][0]

permute_4 (Permute) (8, 30, 40, 12) 0 retinanet_loc_regressor[1][0]

permute_6 (Permute) (8, 15, 20, 12) 0 retinanet_loc_regressor[2][0]

permute_8 (Permute) (8, 8, 10, 12) 0 retinanet_loc_regressor[3][0]

permute_10 (Permute) (8, 4, 5, 12) 0 retinanet_loc_regressor[4][0]

retinanet_anchor_0 (RetinaAncho (8, 4800, 3, 8) 0 retinanet_loc_regressor[0][0]

retinanet_anchor_1 (RetinaAncho (8, 1200, 3, 8) 0 retinanet_loc_regressor[1][0]

retinanet_anchor_2 (RetinaAncho (8, 300, 3, 8) 0 retinanet_loc_regressor[2][0]

retinanet_anchor_3 (RetinaAncho (8, 80, 3, 8) 0 retinanet_loc_regressor[3][0]

retinanet_anchor_4 (RetinaAncho (8, 20, 3, 8) 0 retinanet_loc_regressor[4][0]

mbox_conf (Concatenate) (8, 19200, 1, 7) 0 conf_reshape_0[0][0]
conf_reshape_1[0][0]
conf_reshape_2[0][0]
conf_reshape_3[0][0]
conf_reshape_4[0][0]

loc_reshape_0 (Reshape) (8, 14400, 1, 4) 0 permute_2[0][0]

loc_reshape_1 (Reshape) (8, 3600, 1, 4) 0 permute_4[0][0]

loc_reshape_2 (Reshape) (8, 900, 1, 4) 0 permute_6[0][0]

loc_reshape_3 (Reshape) (8, 240, 1, 4) 0 permute_8[0][0]

loc_reshape_4 (Reshape) (8, 60, 1, 4) 0 permute_10[0][0]

anchor_reshape_0 (Reshape) (8, 14400, 1, 8) 0 retinanet_anchor_0[0][0]

anchor_reshape_1 (Reshape) (8, 3600, 1, 8) 0 retinanet_anchor_1[0][0]

anchor_reshape_2 (Reshape) (8, 900, 1, 8) 0 retinanet_anchor_2[0][0]

anchor_reshape_3 (Reshape) (8, 240, 1, 8) 0 retinanet_anchor_3[0][0]

anchor_reshape_4 (Reshape) (8, 60, 1, 8) 0 retinanet_anchor_4[0][0]

mbox_conf_sigmoid (Activation) (8, 19200, 1, 7) 0 mbox_conf[0][0]

mbox_loc (Concatenate) (8, 19200, 1, 4) 0 loc_reshape_0[0][0]
loc_reshape_1[0][0]
loc_reshape_2[0][0]
loc_reshape_3[0][0]
loc_reshape_4[0][0]

mbox_priorbox (Concatenate) (8, 19200, 1, 8) 0 anchor_reshape_0[0][0]
anchor_reshape_1[0][0]
anchor_reshape_2[0][0]
anchor_reshape_3[0][0]
anchor_reshape_4[0][0]

concatenate_1 (Concatenate) (8, 19200, 1, 19) 0 mbox_conf_sigmoid[0][0]
mbox_loc[0][0]
mbox_priorbox[0][0]

retinanet_predictions (Reshape) (8, 19200, 19) 0 concatenate_1[0][0]

Total params: 17,159,137
Trainable params: 17,138,081
Non-trainable params: 21,056

2022-05-24 14:22:59,184 [INFO] main: Number of samples in the training dataset: 6461
2022-05-24 14:22:59,184 [INFO] main: Number of samples in the validation dataset: 1615
Epoch 1/40
libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: profile ‘ICC Profile’: 1000000h: invalid rendering intent
libpng warning: iCCP: known incorrect sRGB profile

1/808 […] - ETA: 2:32:15 - loss: 9.6297libpng warning: iCCP: known incorrect sRGB profile

2/808 […] - ETA: 1:24:14 - loss: 11.4549/usr/local/lib/python3.6/dist-packages/keras/callbacks.py:122: UserWarning: Method on_batch_end() is slow compared to the batch update (0.495214). Check your callbacks.
% delta_t_median)
libpng warning: iCCP: known incorrect sRGB profile

3/808 […] - ETA: 57:07 - loss: 11.1547
4/808 […] - ETA: 43:34 - loss: 11.0048libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile

5/808 […] - ETA: 35:26 - loss: 11.0107
6/808 […] - ETA: 30:01 - loss: 10.8235
7/808 […] - ETA: 26:08 - loss: 11.1047libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile

8/808 […] - ETA: 23:14 - loss: 11.0295
9/808 […] - ETA: 20:58 - loss: 11.0131
10/808 […] - ETA: 19:09 - loss: 10.9658
11/808 […] - ETA: 17:40 - loss: 10.8491libpng warning: iCCP: known incorrect sRGB profile

12/808 […] - ETA: 16:26 - loss: 10.6727libpng warning: iCCP: known incorrect sRGB profile

13/808 […] - ETA: 15:23 - loss: 10.6469
14/808 […] - ETA: 14:30 - loss: 10.6666
15/808 […] - ETA: 13:43 - loss: 10.5771libpng warning: iCCP: known incorrect sRGB profile

16/808 […] - ETA: 13:02 - loss: 10.5365
17/808 […] - ETA: 12:26 - loss: 10.5096
18/808 […] - ETA: 11:54 - loss: 10.4490
19/808 […] - ETA: 11:25 - loss: 10.3882
20/808 […] - ETA: 10:59 - loss: 10.3374libpng warning: iCCP: known incorrect sRGB profile

21/808 […] - ETA: 10:36 - loss: 10.3024libpng warning: iCCP: known incorrect sRGB profile

22/808 […] - ETA: 10:14 - loss: 10.2472
23/808 […] - ETA: 9:55 - loss: 10.2101
24/808 […] - ETA: 9:37 - loss: 10.2576
25/808 […] - ETA: 9:20 - loss: 10.2155
26/808 […] - ETA: 9:05 - loss: 10.1781libpng warning: iCCP: known incorrect sRGB profile

27/808 [>…] - ETA: 8:51 - loss: 10.1382libpng warning: iCCP: known incorrect sRGB profile

28/808 [>…] - ETA: 8:38 - loss: 10.1153
29/808 [>…] - ETA: 8:25 - loss: 10.0989
30/808 [>…] - ETA: 8:14 - loss: 10.0746
31/808 [>…] - ETA: 8:03 - loss: 10.0541
32/808 [>…] - ETA: 7:53 - loss: 10.0469libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile

33/808 [>…] - ETA: 7:44 - loss: 10.0248
34/808 [>…] - ETA: 7:35 - loss: 10.0004libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile

35/808 [>…] - ETA: 7:26 - loss: 9.9834 libpng warning: iCCP: known incorrect sRGB profile

36/808 [>…] - ETA: 7:18 - loss: 9.9538
37/808 [>…] - ETA: 7:11 - loss: 9.9493libpng warning: iCCP: profile ‘ICC Profile’: 1000000h: invalid rendering intent
libpng warning: iCCP: known incorrect sRGB profile

38/808 [>…] - ETA: 7:04 - loss: 9.9318libpng warning: iCCP: known incorrect sRGB profile

39/808 [>…] - ETA: 6:57 - loss: 9.9054libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile

40/808 [>…] - ETA: 6:50 - loss: 9.8955libpng warning: iCCP: known incorrect sRGB profile

41/808 [>…] - ETA: 6:44 - loss: 9.8984
42/808 [>…] - ETA: 6:38 - loss: 9.8741
43/808 [>…] - ETA: 6:33 - loss: 9.8552libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile

44/808 [>…] - ETA: 6:27 - loss: 9.8441libpng warning: iCCP: known incorrect sRGB profile

45/808 [>…] - ETA: 6:22 - loss: 9.8268
46/808 [>…] - ETA: 6:17 - loss: 9.8125libpng warning: iCCP: known incorrect sRGB profile

47/808 [>…] - ETA: 6:12 - loss: 9.7903
48/808 [>…] - ETA: 6:08 - loss: 9.7684
49/808 [>…] - ETA: 6:04 - loss: 9.7426libpng warning: iCCP: known incorrect sRGB profile

50/808 [>…] - ETA: 5:59 - loss: 9.7243
51/808 [>…] - ETA: 5:55 - loss: 9.7048libpng warning: iCCP: known incorrect sRGB profile

52/808 [>…] - ETA: 5:51 - loss: 9.6955libpng warning: iCCP: known incorrect sRGB profile

53/808 [>…] - ETA: 5:48 - loss: 9.6828
54/808 [=>…] - ETA: 5:44 - loss: 9.6666
55/808 [=>…] - ETA: 5:41 - loss: 9.6759libpng warning: iCCP: known incorrect sRGB profile

56/808 [=>…] - ETA: 5:37 - loss: 9.6661libpng warning: iCCP: known incorrect sRGB profile

57/808 [=>…] - ETA: 5:34 - loss: 9.6514
58/808 [=>…] - ETA: 5:31 - loss: 9.6367
59/808 [=>…] - ETA: 5:28 - loss: 9.6372libpng warning: iCCP: known incorrect sRGB profile

60/808 [=>…] - ETA: 5:25 - loss: 9.6263
61/808 [=>…] - ETA: 5:22 - loss: 9.6222
62/808 [=>…] - ETA: 5:19 - loss: 9.6166libpng warning: iCCP: known incorrect sRGB profile

63/808 [=>…] - ETA: 5:16 - loss: 9.6073
64/808 [=>…] - ETA: 5:14 - loss: 9.5978libpng warning: iCCP: known incorrect sRGB profile

65/808 [=>…] - ETA: 5:11 - loss: 9.5849
66/808 [=>…] - ETA: 5:09 - loss: 9.5781libpng warning: iCCP: known incorrect sRGB profile

67/808 [=>…] - ETA: 5:06 - loss: 9.5663
68/808 [=>…] - ETA: 5:04 - loss: 9.5569libpng warning: iCCP: known incorrect sRGB profile

69/808 [=>…] - ETA: 5:01 - loss: 9.5434libpng warning: iCCP: known incorrect sRGB profile

70/808 [=>…] - ETA: 4:59 - loss: 9.5305
71/808 [=>…] - ETA: 4:57 - loss: 9.5215
72/808 [=>…] - ETA: 4:55 - loss: 9.5134
73/808 [=>…] - ETA: 4:53 - loss: 9.5093
74/808 [=>…] - ETA: 4:51 - loss: 9.5009
75/808 [=>…] - ETA: 4:49 - loss: 9.4954libpng warning: iCCP: known incorrect sRGB profile

76/808 [=>…] - ETA: 4:47 - loss: 9.4861
77/808 [=>…] - ETA: 4:45 - loss: 9.4869
78/808 [=>…] - ETA: 4:43 - loss: 9.4780
79/808 [=>…] - ETA: 4:41 - loss: 9.4995
80/808 [=>…] - ETA: 4:39 - loss: 9.5042libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile

81/808 [==>…] - ETA: 4:38 - loss: 9.4995
82/808 [==>…] - ETA: 4:36 - loss: 9.4958
83/808 [==>…] - ETA: 4:34 - loss: 9.4945
84/808 [==>…] - ETA: 4:33 - loss: 9.4836
85/808 [==>…] - ETA: 4:31 - loss: 9.4811
86/808 [==>…] - ETA: 4:30 - loss: 9.4707
87/808 [==>…] - ETA: 4:28 - loss: 9.4661
88/808 [==>…] - ETA: 4:26 - loss: 9.4646libpng warning: iCCP: known incorrect sRGB profile

89/808 [==>…] - ETA: 4:25 - loss: 9.4606libpng warning: iCCP: known incorrect sRGB profile

90/808 [==>…] - ETA: 4:24 - loss: 9.4559libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile

91/808 [==>…] - ETA: 4:22 - loss: 9.4541
92/808 [==>…] - ETA: 4:21 - loss: 9.4496
93/808 [==>…] - ETA: 4:19 - loss: 9.4425
94/808 [==>…] - ETA: 4:18 - loss: 9.4428libpng warning: iCCP: known incorrect sRGB profile

95/808 [==>…] - ETA: 4:17 - loss: 9.4345libpng warning: iCCP: known incorrect sRGB profile

96/808 [==>…] - ETA: 4:15 - loss: 9.4309
97/808 [==>…] - ETA: 4:14 - loss: 9.4253libpng warning: iCCP: known incorrect sRGB profile

98/808 [==>…] - ETA: 4:13 - loss: 9.4248
99/808 [==>…] - ETA: 4:12 - loss: 9.4310
100/808 [==>…] - ETA: 4:10 - loss: 9.4275
101/808 [==>…] - ETA: 4:09 - loss: 9.4307
102/808 [==>…] - ETA: 4:08 - loss: 9.4241libpng warning: iCCP: known incorrect sRGB profile

103/808 [==>…] - ETA: 4:07 - loss: 9.4185libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile

104/808 [==>…] - ETA: 4:06 - loss: 9.4120libpng warning: iCCP: known incorrect sRGB profile

105/808 [==>…] - ETA: 4:05 - loss: 9.4061libpng warning: iCCP: known incorrect sRGB profile

106/808 [==>…] - ETA: 4:04 - loss: 9.3974
107/808 [==>…] - ETA: 4:02 - loss: 9.3925
108/808 [===>…] - ETA: 4:01 - loss: 9.3886libpng warning: iCCP: known incorrect sRGB profile

109/808 [===>…] - ETA: 4:00 - loss: 9.3836
110/808 [===>…] - ETA: 3:59 - loss: 9.3766
111/808 [===>…] - ETA: 3:58 - loss: 9.3764libpng warning: iCCP: known incorrect sRGB profile

112/808 [===>…] - ETA: 3:57 - loss: 9.3723
113/808 [===>…] - ETA: 3:56 - loss: 9.3750libpng warning: iCCP: known incorrect sRGB profile

114/808 [===>…] - ETA: 3:55 - loss: 9.3726
115/808 [===>…] - ETA: 3:54 - loss: 9.3662libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile

116/808 [===>…] - ETA: 3:53 - loss: 9.3600
117/808 [===>…] - ETA: 3:52 - loss: 9.3556
118/808 [===>…] - ETA: 3:51 - loss: 9.3521
119/808 [===>…] - ETA: 3:50 - loss: 9.3514
120/808 [===>…] - ETA: 3:50 - loss: 9.3456libpng warning: iCCP: known incorrect sRGB profile

121/808 [===>…] - ETA: 3:49 - loss: 9.3458
122/808 [===>…] - ETA: 3:48 - loss: 9.3411
123/808 [===>…] - ETA: 3:47 - loss: 9.3350
124/808 [===>…] - ETA: 3:46 - loss: 9.3356
125/808 [===>…] - ETA: 3:45 - loss: 9.3321libpng warning: iCCP: known incorrect sRGB profile

126/808 [===>…] - ETA: 3:44 - loss: 9.3253
127/808 [===>…] - ETA: 3:43 - loss: 9.3253libpng warning: iCCP: known incorrect sRGB profile

128/808 [===>…] - ETA: 3:43 - loss: 9.3193libpng warning: iCCP: known incorrect sRGB profile

129/808 [===>…] - ETA: 3:42 - loss: 9.3145
130/808 [===>…] - ETA: 3:41 - loss: 9.3178libpng warning: iCCP: profile ‘ICC Profile’: 0h: PCS illuminant is not D50

131/808 [===>…] - ETA: 3:40 - loss: 9.3172libpng warning: iCCP: known incorrect sRGB profile

132/808 [===>…] - ETA: 3:39 - loss: 9.3141libpng warning: iCCP: known incorrect sRGB profile

133/808 [===>…] - ETA: 3:39 - loss: 9.3110
134/808 [===>…] - ETA: 3:38 - loss: 9.3045libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile

135/808 [====>…] - ETA: 3:37 - loss: 9.3005libpng warning: iCCP: known incorrect sRGB profile

136/808 [====>…] - ETA: 3:36 - loss: 9.2973
137/808 [====>…] - ETA: 3:36 - loss: 9.2909libpng warning: iCCP: known incorrect sRGB profile

138/808 [====>…] - ETA: 3:35 - loss: 9.2867libpng warning: iCCP: known incorrect sRGB profile

139/808 [====>…] - ETA: 3:34 - loss: 9.2850
140/808 [====>…] - ETA: 3:33 - loss: 9.2833libpng warning: iCCP: known incorrect sRGB profile

141/808 [====>…] - ETA: 3:33 - loss: 9.2814libpng warning: iCCP: known incorrect sRGB profile

142/808 [====>…] - ETA: 3:32 - loss: 9.2762libpng warning: iCCP: known incorrect sRGB profile

143/808 [====>…] - ETA: 3:31 - loss: 9.2713
144/808 [====>…] - ETA: 3:31 - loss: 9.2671libpng warning: iCCP: profile ‘ICC Profile’: ‘desc’: ICC profile tag start not a multiple of 4
libpng warning: iCCP: profile ‘ICC Profile’: ‘wtpt’: ICC profile tag start not a multiple of 4
libpng warning: iCCP: profile ‘ICC Profile’: ‘bkpt’: ICC profile tag start not a multiple of 4
libpng warning: iCCP: profile ‘ICC Profile’: ‘rXYZ’: ICC profile tag start not a multiple of 4
libpng warning: iCCP: profile ‘ICC Profile’: ‘gXYZ’: ICC profile tag start not a multiple of 4
libpng warning: iCCP: profile ‘ICC Profile’: ‘bXYZ’: ICC profile tag start not a multiple of 4
libpng warning: iCCP: profile ‘ICC Profile’: ‘dmnd’: ICC profile tag start not a multiple of 4
libpng warning: iCCP: profile ‘ICC Profile’: ‘dmdd’: ICC profile tag start not a multiple of 4
libpng warning: iCCP: profile ‘ICC Profile’: ‘vued’: ICC profile tag start not a multiple of 4
libpng warning: iCCP: profile ‘ICC Profile’: ‘view’: ICC profile tag start not a multiple of 4
libpng warning: iCCP: profile ‘ICC Profile’: ‘lumi’: ICC profile tag start not a multiple of 4
libpng warning: iCCP: profile ‘ICC Profile’: ‘meas’: ICC profile tag start not a multiple of 4
libpng warning: iCCP: profile ‘ICC Profile’: ‘tech’: ICC profile tag start not a multiple of 4
libpng warning: iCCP: profile ‘ICC Profile’: ‘rTRC’: ICC profile tag start not a multiple of 4
libpng warning: iCCP: profile ‘ICC Profile’: ‘gTRC’: ICC profile tag start not a multiple of 4
libpng warning: iCCP: profile ‘ICC Profile’: ‘bTRC’: ICC profile tag start not a multiple of 4

145/808 [====>…] - ETA: 3:30 - loss: 9.2645
146/808 [====>…] - ETA: 3:29 - loss: 9.2626
147/808 [====>…] - ETA: 3:28 - loss: 9.2589
148/808 [====>…] - ETA: 3:28 - loss: 9.2580
149/808 [====>…] - ETA: 3:27 - loss: 9.2542
150/808 [====>…] - ETA: 3:26 - loss: 9.2533libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile

151/808 [====>…] - ETA: 3:26 - loss: 9.2519libpng warning: iCCP: known incorrect sRGB profile

152/808 [====>…] - ETA: 3:25 - loss: 9.2533
153/808 [====>…] - ETA: 3:24 - loss: 9.2528libpng warning: iCCP: known incorrect sRGB profile

154/808 [====>…] - ETA: 3:24 - loss: 9.2483
155/808 [====>…] - ETA: 3:23 - loss: 9.2428
156/808 [====>…] - ETA: 3:23 - loss: 9.2490
157/808 [====>…] - ETA: 3:22 - loss: 9.2455
158/808 [====>…] - ETA: 3:21 - loss: 9.2465libpng warning: iCCP: known incorrect sRGB profile

159/808 [====>…] - ETA: 3:21 - loss: 9.2421
160/808 [====>…] - ETA: 3:20 - loss: 9.2415
161/808 [====>…] - ETA: 3:19 - loss: 9.2383
162/808 [=====>…] - ETA: 3:19 - loss: 9.2362
163/808 [=====>…] - ETA: 3:18 - loss: 9.2349libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile

164/808 [=====>…] - ETA: 3:18 - loss: 9.2383libpng warning: iCCP: known incorrect sRGB profile

165/808 [=====>…] - ETA: 3:17 - loss: 9.2338libpng warning: iCCP: known incorrect sRGB profile

166/808 [=====>…] - ETA: 3:16 - loss: 9.2314
167/808 [=====>…] - ETA: 3:16 - loss: 9.2297
168/808 [=====>…] - ETA: 3:15 - loss: 9.2301
169/808 [=====>…] - ETA: 3:15 - loss: 9.2278libpng warning: iCCP: known incorrect sRGB profile

170/808 [=====>…] - ETA: 3:14 - loss: 9.2250libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile

171/808 [=====>…] - ETA: 3:14 - loss: 9.2223libpng warning: iCCP: known incorrect sRGB profile

172/808 [=====>…] - ETA: 3:13 - loss: 9.2205
173/808 [=====>…] - ETA: 3:12 - loss: 9.2197
174/808 [=====>…] - ETA: 3:12 - loss: 9.2195
175/808 [=====>…] - ETA: 3:11 - loss: 9.2164libpng warning: iCCP: known incorrect sRGB profile

176/808 [=====>…] - ETA: 3:11 - loss: 9.2144libpng warning: iCCP: known incorrect sRGB profile

177/808 [=====>…] - ETA: 3:10 - loss: 9.2112libpng warning: iCCP: known incorrect sRGB profile

178/808 [=====>…] - ETA: 3:10 - loss: 9.2095libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile

179/808 [=====>…] - ETA: 3:09 - loss: 9.2084
180/808 [=====>…] - ETA: 3:09 - loss: 9.2062
181/808 [=====>…] - ETA: 3:08 - loss: 9.2056libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile

182/808 [=====>…] - ETA: 3:08 - loss: 9.2096libpng warning: iCCP: known incorrect sRGB profile

183/808 [=====>…] - ETA: 3:07 - loss: 9.2081libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile

184/808 [=====>…] - ETA: 3:06 - loss: 9.2076libpng warning: iCCP: known incorrect sRGB profile

185/808 [=====>…] - ETA: 3:06 - loss: 9.2030
186/808 [=====>…] - ETA: 3:05 - loss: 9.1998
187/808 [=====>…] - ETA: 3:05 - loss: 9.1943libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile

188/808 [=====>…] - ETA: 3:04 - loss: 9.1925
189/808 [======>…] - ETA: 3:04 - loss: 9.1923
190/808 [======>…] - ETA: 3:03 - loss: 9.1892
191/808 [======>…] - ETA: 3:03 - loss: 9.1881
192/808 [======>…] - ETA: 3:02 - loss: 9.1889
193/808 [======>…] - ETA: 3:02 - loss: 9.1870
194/808 [======>…] - ETA: 3:01 - loss: 9.1884
195/808 [======>…] - ETA: 3:01 - loss: 9.1866
196/808 [======>…] - ETA: 3:01 - loss: 9.1827libpng warning: iCCP: known incorrect sRGB profile

197/808 [======>…] - ETA: 3:00 - loss: 9.1811libpng warning: iCCP: known incorrect sRGB profile

198/808 [======>…] - ETA: 3:00 - loss: 9.1825libpng warning: iCCP: known incorrect sRGB profile

199/808 [======>…] - ETA: 2:59 - loss: 9.1806
200/808 [======>…] - ETA: 2:59 - loss: 9.1778
201/808 [======>…] - ETA: 2:58 - loss: 9.1842
202/808 [======>…] - ETA: 2:58 - loss: 9.1810libpng warning: iCCP: known incorrect sRGB profile

203/808 [======>…] - ETA: 2:57 - loss: 9.1799libpng warning: iCCP: known incorrect sRGB profile

204/808 [======>…] - ETA: 2:57 - loss: 9.1777
205/808 [======>…] - ETA: 2:56 - loss: 9.1754
206/808 [======>…] - ETA: 2:56 - loss: 9.1750libpng warning: iCCP: known incorrect sRGB profile

207/808 [======>…] - ETA: 2:55 - loss: 9.1750libpng warning: iCCP: known incorrect sRGB profile

208/808 [======>…] - ETA: 2:55 - loss: 9.1743libpng warning: iCCP: known incorrect sRGB profile

209/808 [======>…] - ETA: 2:54 - loss: 9.1718
210/808 [======>…] - ETA: 2:54 - loss: 9.1752
211/808 [======>…] - ETA: 2:53 - loss: 9.1737
212/808 [======>…] - ETA: 2:53 - loss: 9.1715
213/808 [======>…] - ETA: 2:53 - loss: 9.1707libpng warning: iCCP: known incorrect sRGB profile

214/808 [======>…] - ETA: 2:52 - loss: 9.1712libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile

215/808 [======>…] - ETA: 2:52 - loss: 9.1711
216/808 [=======>…] - ETA: 2:51 - loss: 9.1670libpng warning: iCCP: known incorrect sRGB profile

217/808 [=======>…] - ETA: 2:51 - loss: 9.1665
218/808 [=======>…] - ETA: 2:50 - loss: 9.1660libpng warning: iCCP: known incorrect sRGB profile

219/808 [=======>…] - ETA: 2:50 - loss: 9.1625libpng warning: iCCP: profile ‘ICC Profile’: ‘desc’: ICC profile tag start not a multiple of 4
libpng warning: iCCP: profile ‘ICC Profile’: ‘wtpt’: ICC profile tag start not a multiple of 4
libpng warning: iCCP: profile ‘ICC Profile’: ‘bkpt’: ICC profile tag start not a multiple of 4
libpng warning: iCCP: profile ‘ICC Profile’: ‘rXYZ’: ICC profile tag start not a multiple of 4
libpng warning: iCCP: profile ‘ICC Profile’: ‘gXYZ’: ICC profile tag start not a multiple of 4
libpng warning: iCCP: profile ‘ICC Profile’: ‘bXYZ’: ICC profile tag start not a multiple of 4
libpng warning: iCCP: profile ‘ICC Profile’: ‘dmnd’: ICC profile tag start not a multiple of 4
libpng warning: iCCP: profile ‘ICC Profile’: ‘dmdd’: ICC profile tag start not a multiple of 4
libpng warning: iCCP: profile ‘ICC Profile’: ‘vued’: ICC profile tag start not a multiple of 4
libpng warning: iCCP: profile ‘ICC Profile’: ‘view’: ICC profile tag start not a multiple of 4
libpng warning: iCCP: profile ‘ICC Profile’: ‘lumi’: ICC profile tag start not a multiple of 4
libpng warning: iCCP: profile ‘ICC Profile’: ‘meas’: ICC profile tag start not a multiple of 4
libpng warning: iCCP: profile ‘ICC Profile’: ‘tech’: ICC profile tag start not a multiple of 4
libpng warning: iCCP: profile ‘ICC Profile’: ‘rTRC’: ICC profile tag start not a multiple of 4
libpng warning: iCCP: profile ‘ICC Profile’: ‘gTRC’: ICC profile tag start not a multiple of 4
libpng warning: iCCP: profile ‘ICC Profile’: ‘bTRC’: ICC profile tag start not a multiple of 4
libpng warning: iCCP: known incorrect sRGB profile

220/808 [=======>…] - ETA: 2:50 - loss: 9.1616
221/808 [=======>…] - ETA: 2:49 - loss: 9.1588libpng warning: iCCP: known incorrect sRGB profile

222/808 [=======>…] - ETA: 2:49 - loss: 9.1564
223/808 [=======>…] - ETA: 2:48 - loss: 9.1582
224/808 [=======>…] - ETA: 2:48 - loss: 9.1541
225/808 [=======>…] - ETA: 2:47 - loss: 9.1509
226/808 [=======>…] - ETA: 2:47 - loss: 9.1504
227/808 [=======>…] - ETA: 2:47 - loss: 9.1467libpng warning: iCCP: known incorrect sRGB profile

228/808 [=======>…] - ETA: 2:46 - loss: 9.1504
229/808 [=======>…] - ETA: 2:46 - loss: 9.1500libpng warning: iCCP: known incorrect sRGB profile

230/808 [=======>…] - ETA: 2:45 - loss: 9.1525libpng warning: iCCP: known incorrect sRGB profile

231/808 [=======>…] - ETA: 2:45 - loss: 9.1509
232/808 [=======>…] - ETA: 2:44 - loss: 9.1481
233/808 [=======>…] - ETA: 2:44 - loss: 9.1466
234/808 [=======>…] - ETA: 2:44 - loss: 9.1514
235/808 [=======>…] - ETA: 2:43 - loss: 9.1516
236/808 [=======>…] - ETA: 2:43 - loss: 9.1506
237/808 [=======>…] - ETA: 2:42 - loss: 9.1508
238/808 [=======>…] - ETA: 2:42 - loss: 9.1498
239/808 [=======>…] - ETA: 2:42 - loss: 9.1499libpng warning: iCCP: known incorrect sRGB profile

240/808 [=======>…] - ETA: 2:41 - loss: 9.1488
241/808 [=======>…] - ETA: 2:41 - loss: 9.1467libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile

242/808 [=======>…] - ETA: 2:40 - loss: 9.1459libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile

243/808 [========>…] - ETA: 2:40 - loss: 9.1435
244/808 [========>…] - ETA: 2:40 - loss: 9.1428
245/808 [========>…] - ETA: 2:39 - loss: 9.1442
246/808 [========>…] - ETA: 2:39 - loss: 9.1412libpng warning: iCCP: known incorrect sRGB profile

247/808 [========>…] - ETA: 2:38 - loss: 9.1413libpng warning: iCCP: known incorrect sRGB profile

248/808 [========>…] - ETA: 2:38 - loss: 9.1398
249/808 [========>…] - ETA: 2:38 - loss: 9.1459libpng warning: iCCP: known incorrect sRGB profile

250/808 [========>…] - ETA: 2:37 - loss: 9.1445libpng warning: iCCP: known incorrect sRGB profile

251/808 [========>…] - ETA: 2:37 - loss: 9.1452
252/808 [========>…] - ETA: 2:36 - loss: 9.1430libpng warning: iCCP: known incorrect sRGB profile

253/808 [========>…] - ETA: 2:36 - loss: 9.1411
254/808 [========>…] - ETA: 2:36 - loss: 9.1393libpng warning: iCCP: known incorrect sRGB profile

255/808 [========>…] - ETA: 2:35 - loss: 9.1356libpng warning: iCCP: known incorrect sRGB profile

256/808 [========>…] - ETA: 2:35 - loss: 9.1335libpng warning: iCCP: known incorrect sRGB profile

257/808 [========>…] - ETA: 2:35 - loss: 9.1323libpng warning: iCCP: known incorrect sRGB profile

258/808 [========>…] - ETA: 2:34 - loss: 9.1295
259/808 [========>…] - ETA: 2:34 - loss: 9.1323
260/808 [========>…] - ETA: 2:33 - loss: 9.1318libpng warning: iCCP: known incorrect sRGB profile

261/808 [========>…] - ETA: 2:33 - loss: 9.1305
262/808 [========>…] - ETA: 2:33 - loss: 9.1283
263/808 [========>…] - ETA: 2:32 - loss: 9.1279
264/808 [========>…] - ETA: 2:32 - loss: 9.1263
265/808 [========>…] - ETA: 2:32 - loss: 9.1259libpng warning: iCCP: known incorrect sRGB profile

266/808 [========>…] - ETA: 2:31 - loss: 9.1254libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile

267/808 [========>…] - ETA: 2:31 - loss: 9.1233
268/808 [========>…] - ETA: 2:30 - loss: 9.1206
269/808 [========>…] - ETA: 2:30 - loss: 9.1191libpng warning: iCCP: known incorrect sRGB profile

270/808 [=========>…] - ETA: 2:30 - loss: 9.1222
271/808 [=========>…] - ETA: 2:29 - loss: 9.1197
272/808 [=========>…] - ETA: 2:29 - loss: 9.1207
273/808 [=========>…] - ETA: 2:29 - loss: 9.1178libpng warning: iCCP: known incorrect sRGB profile

274/808 [=========>…] - ETA: 2:28 - loss: 9.1164
275/808 [=========>…] - ETA: 2:28 - loss: 9.1126
276/808 [=========>…] - ETA: 2:27 - loss: 9.1101libpng warning: iCCP: profile ‘ICC Profile’: ‘desc’: ICC profile tag start not a multiple of 4
libpng warning: iCCP: profile ‘ICC Profile’: ‘wtpt’: ICC profile tag start not a multiple of 4
libpng warning: iCCP: profile ‘ICC Profile’: ‘bkpt’: ICC profile tag start not a multiple of 4
libpng warning: iCCP: profile ‘ICC Profile’: ‘rXYZ’: ICC profile tag start not a multiple of 4
libpng warning: iCCP: profile ‘ICC Profile’: ‘gXYZ’: ICC profile tag start not a multiple of 4
libpng warning: iCCP: profile ‘ICC Profile’: ‘bXYZ’: ICC profile tag start not a multiple of 4
libpng warning: iCCP: profile ‘ICC Profile’: ‘dmnd’: ICC profile tag start not a multiple of 4
libpng warning: iCCP: profile ‘ICC Profile’: ‘dmdd’: ICC profile tag start not a multiple of 4
libpng warning: iCCP: profile ‘ICC Profile’: ‘vued’: ICC profile tag start not a multiple of 4
libpng warning: iCCP: profile ‘ICC Profile’: ‘view’: ICC profile tag start not a multiple of 4
libpng warning: iCCP: profile ‘ICC Profile’: ‘lumi’: ICC profile tag start not a multiple of 4
libpng warning: iCCP: profile ‘ICC Profile’: ‘meas’: ICC profile tag start not a multiple of 4
libpng warning: iCCP: profile ‘ICC Profile’: ‘tech’: ICC profile tag start not a multiple of 4
libpng warning: iCCP: profile ‘ICC Profile’: ‘rTRC’: ICC profile tag start not a multiple of 4
libpng warning: iCCP: profile ‘ICC Profile’: ‘gTRC’: ICC profile tag start not a multiple of 4
libpng warning: iCCP: profile ‘ICC Profile’: ‘bTRC’: ICC profile tag start not a multiple of 4

277/808 [=========>…] - ETA: 2:27 - loss: 9.1085
278/808 [=========>…] - ETA: 2:27 - loss: 9.1103
279/808 [=========>…] - ETA: 2:26 - loss: 9.1124libpng warning: iCCP: known incorrect sRGB profile

280/808 [=========>…] - ETA: 2:26 - loss: 9.1113libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile

281/808 [=========>…] - ETA: 2:26 - loss: 9.1096
282/808 [=========>…] - ETA: 2:25 - loss: 9.1093
283/808 [=========>…] - ETA: 2:25 - loss: 9.1071libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile

284/808 [=========>…] - ETA: 2:25 - loss: 9.1074
285/808 [=========>…] - ETA: 2:24 - loss: 9.1068
286/808 [=========>…] - ETA: 2:24 - loss: 9.1048
287/808 [=========>…] - ETA: 2:24 - loss: 9.1038
288/808 [=========>…] - ETA: 2:23 - loss: 9.1011
289/808 [=========>…] - ETA: 2:23 - loss: 9.1003
290/808 [=========>…] - ETA: 2:23 - loss: 9.0986
291/808 [=========>…] - ETA: 2:22 - loss: 9.0971
292/808 [=========>…] - ETA: 2:22 - loss: 9.0980libpng warning: iCCP: known incorrect sRGB profile

293/808 [=========>…] - ETA: 2:22 - loss: 9.0960
294/808 [=========>…] - ETA: 2:21 - loss: 9.0934libpng warning: iCCP: profile ‘ICC Profile’: 1000000h: invalid rendering intent

295/808 [=========>…] - ETA: 2:21 - loss: 9.0926libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile

296/808 [=========>…] - ETA: 2:20 - loss: 9.0916
297/808 [==========>…] - ETA: 2:20 - loss: 9.0930libpng warning: iCCP: known incorrect sRGB profile

298/808 [==========>…] - ETA: 2:20 - loss: 9.0924
299/808 [==========>…] - ETA: 2:19 - loss: 9.0901
300/808 [==========>…] - ETA: 2:19 - loss: 9.0904libpng warning: iCCP: known incorrect sRGB profile

301/808 [==========>…] - ETA: 2:19 - loss: 9.0900
302/808 [==========>…] - ETA: 2:18 - loss: 9.0877libpng warning: iCCP: known incorrect sRGB profile

303/808 [==========>…] - ETA: 2:18 - loss: 9.0855libpng warning: iCCP: known incorrect sRGB profile

304/808 [==========>…] - ETA: 2:18 - loss: 9.0836libpng warning: iCCP: known incorrect sRGB profile

305/808 [==========>…] - ETA: 2:17 - loss: 9.0824libpng warning: iCCP: known incorrect sRGB profile

306/808 [==========>…] - ETA: 2:17 - loss: 9.0803libpng warning: iCCP: known incorrect sRGB profile

307/808 [==========>…] - ETA: 2:17 - loss: 9.0790libpng warning: iCCP: known incorrect sRGB profile

308/808 [==========>…] - ETA: 2:16 - loss: 9.0774
309/808 [==========>…] - ETA: 2:16 - loss: 9.0766libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile

310/808 [==========>…] - ETA: 2:16 - loss: 9.0749
311/808 [==========>…] - ETA: 2:15 - loss: 9.0734libpng warning: iCCP: known incorrect sRGB profile

312/808 [==========>…] - ETA: 2:15 - loss: 9.0742libpng warning: iCCP: known incorrect sRGB profile

313/808 [==========>…] - ETA: 2:15 - loss: 9.0747libpng warning: iCCP: known incorrect sRGB profile

314/808 [==========>…] - ETA: 2:14 - loss: 9.0739
315/808 [==========>…] - ETA: 2:14 - loss: 9.0716
316/808 [==========>…] - ETA: 2:14 - loss: 9.0727
317/808 [==========>…] - ETA: 2:13 - loss: 9.0706libpng warning: iCCP: known incorrect sRGB profile

318/808 [==========>…] - ETA: 2:13 - loss: 9.0697libpng warning: iCCP: known incorrect sRGB profile

319/808 [==========>…] - ETA: 2:13 - loss: 9.0681libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile

320/808 [==========>…] - ETA: 2:12 - loss: 9.0665
321/808 [==========>…] - ETA: 2:12 - loss: 9.0651libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile

322/808 [==========>…] - ETA: 2:12 - loss: 9.0652libpng warning: iCCP: known incorrect sRGB profile

323/808 [==========>…] - ETA: 2:11 - loss: 9.0653libpng warning: iCCP: known incorrect sRGB profile

324/808 [===========>…] - ETA: 2:11 - loss: 9.0623
325/808 [===========>…] - ETA: 2:11 - loss: 9.0664libpng warning: iCCP: known incorrect sRGB profile

326/808 [===========>…] - ETA: 2:10 - loss: 9.0648
327/808 [===========>…] - ETA: 2:10 - loss: 9.0629
328/808 [===========>…] - ETA: 2:10 - loss: 9.0635
329/808 [===========>…] - ETA: 2:09 - loss: 9.0630
330/808 [===========>…] - ETA: 2:09 - loss: 9.0618libpng warning: iCCP: known incorrect sRGB profile

331/808 [===========>…] - ETA: 2:09 - loss: 9.0603libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile

332/808 [===========>…] - ETA: 2:08 - loss: 9.0587
333/808 [===========>…] - ETA: 2:08 - loss: 9.0597libpng warning: iCCP: known incorrect sRGB profile

334/808 [===========>…] - ETA: 2:08 - loss: 9.0579
335/808 [===========>…] - ETA: 2:08 - loss: 9.0570
336/808 [===========>…] - ETA: 2:07 - loss: 9.0567
337/808 [===========>…] - ETA: 2:07 - loss: 9.0553
338/808 [===========>…] - ETA: 2:07 - loss: 9.0537
339/808 [===========>…] - ETA: 2:06 - loss: 9.0536
340/808 [===========>…] - ETA: 2:06 - loss: 9.0531libpng warning: iCCP: known incorrect sRGB profile

341/808 [===========>…] - ETA: 2:06 - loss: 9.0529libpng warning: iCCP: known incorrect sRGB profile

342/808 [===========>…] - ETA: 2:05 - loss: 9.0518libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile

343/808 [===========>…] - ETA: 2:05 - loss: 9.0503
344/808 [===========>…] - ETA: 2:05 - loss: 9.0502
345/808 [===========>…] - ETA: 2:04 - loss: 9.0495libpng warning: iCCP: known incorrect sRGB profile

346/808 [===========>…] - ETA: 2:04 - loss: 9.0490
347/808 [===========>…] - ETA: 2:04 - loss: 9.0498
348/808 [===========>…] - ETA: 2:03 - loss: 9.0475libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile

349/808 [===========>…] - ETA: 2:03 - loss: 9.0532
350/808 [===========>…] - ETA: 2:03 - loss: 9.0523
351/808 [============>…] - ETA: 2:02 - loss: 9.0504libpng warning: iCCP: known incorrect sRGB profile

352/808 [============>…] - ETA: 2:02 - loss: 9.0493
353/808 [============>…] - ETA: 2:02 - loss: 9.0477
354/808 [============>…] - ETA: 2:02 - loss: 9.0469
355/808 [============>…] - ETA: 2:01 - loss: 9.0459
356/808 [============>…] - ETA: 2:01 - loss: 9.0450
357/808 [============>…] - ETA: 2:01 - loss: 9.0435libpng warning: iCCP: known incorrect sRGB profile

358/808 [============>…] - ETA: 2:00 - loss: 9.0419
359/808 [============>…] - ETA: 2:00 - loss: 9.0409
360/808 [============>…] - ETA: 2:00 - loss: 9.0395libpng warning: iCCP: known incorrect sRGB profile

361/808 [============>…] - ETA: 1:59 - loss: 9.0379
362/808 [============>…] - ETA: 1:59 - loss: 9.0369
363/808 [============>…] - ETA: 1:59 - loss: 9.0351libpng warning: iCCP: known incorrect sRGB profile

364/808 [============>…] - ETA: 1:58 - loss: 9.0344
365/808 [============>…] - ETA: 1:58 - loss: 9.0325
366/808 [============>…] - ETA: 1:58 - loss: 9.0303libpng warning: iCCP: known incorrect sRGB profile

367/808 [============>…] - ETA: 1:58 - loss: 9.0302
368/808 [============>…] - ETA: 1:57 - loss: 9.0300
369/808 [============>…] - ETA: 1:57 - loss: 9.0278
370/808 [============>…] - ETA: 1:57 - loss: 9.0269
371/808 [============>…] - ETA: 1:56 - loss: 9.0258libpng warning: iCCP: known incorrect sRGB profile

372/808 [============>…] - ETA: 1:56 - loss: 9.0250libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile

373/808 [============>…] - ETA: 1:56 - loss: 9.0244libpng warning: iCCP: known incorrect sRGB profile

374/808 [============>…] - ETA: 1:55 - loss: 9.0226
375/808 [============>…] - ETA: 1:55 - loss: 9.0211
376/808 [============>…] - ETA: 1:55 - loss: 9.0199
377/808 [============>…] - ETA: 1:55 - loss: 9.0196
378/808 [=============>…] - ETA: 1:54 - loss: 9.0183libpng warning: iCCP: known incorrect sRGB profile

379/808 [=============>…] - ETA: 1:54 - loss: 9.0169libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile

380/808 [=============>…] - ETA: 1:54 - loss: 9.0165
381/808 [=============>…] - ETA: 1:53 - loss: 9.0158libpng warning: iCCP: known incorrect sRGB profile

382/808 [=============>…] - ETA: 1:53 - loss: 9.0151
383/808 [=============>…] - ETA: 1:53 - loss: 9.0142libpng warning: iCCP: known incorrect sRGB profile

384/808 [=============>…] - ETA: 1:52 - loss: 9.0148libpng warning: iCCP: known incorrect sRGB profile

385/808 [=============>…] - ETA: 1:52 - loss: 9.0140libpng warning: iCCP: known incorrect sRGB profile

386/808 [=============>…] - ETA: 1:52 - loss: 9.0131
387/808 [=============>…] - ETA: 1:51 - loss: 9.0118libpng warning: iCCP: known incorrect sRGB profile

388/808 [=============>…] - ETA: 1:51 - loss: 9.0100libpng warning: iCCP: known incorrect sRGB profile

389/808 [=============>…] - ETA: 1:51 - loss: 9.0091libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile

390/808 [=============>…] - ETA: 1:51 - loss: 9.0087libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile

391/808 [=============>…] - ETA: 1:50 - loss: 9.0077
392/808 [=============>…] - ETA: 1:50 - loss: 9.0067libpng warning: iCCP: known incorrect sRGB profile

393/808 [=============>…] - ETA: 1:50 - loss: 9.0073libpng warning: iCCP: known incorrect sRGB profile

394/808 [=============>…] - ETA: 1:49 - loss: 9.0052libpng warning: iCCP: known incorrect sRGB profile

395/808 [=============>…] - ETA: 1:49 - loss: 9.0049
396/808 [=============>…] - ETA: 1:49 - loss: 9.0044
397/808 [=============>…] - ETA: 1:49 - loss: 9.0042
398/808 [=============>…] - ETA: 1:48 - loss: 9.0046libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile

399/808 [=============>…] - ETA: 1:48 - loss: 9.0029
400/808 [=============>…] - ETA: 1:48 - loss: 9.0039
401/808 [=============>…] - ETA: 1:47 - loss: 9.0021
402/808 [=============>…] - ETA: 1:47 - loss: 9.0046
403/808 [=============>…] - ETA: 1:47 - loss: 9.0029
404/808 [==============>…] - ETA: 1:46 - loss: 9.0031
405/808 [==============>…] - ETA: 1:46 - loss: 9.0025libpng warning: iCCP: known incorrect sRGB profile

406/808 [==============>…] - ETA: 1:46 - loss: 9.0026
407/808 [==============>…] - ETA: 1:46 - loss: 9.0023libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile

408/808 [==============>…] - ETA: 1:45 - loss: 9.0016
409/808 [==============>…] - ETA: 1:45 - loss: 9.0017
410/808 [==============>…] - ETA: 1:45 - loss: 8.9999
411/808 [==============>…] - ETA: 1:44 - loss: 8.9975
412/808 [==============>…] - ETA: 1:44 - loss: 8.9968
413/808 [==============>…] - ETA: 1:44 - loss: 8.9958
414/808 [==============>…] - ETA: 1:44 - loss: 8.9948
415/808 [==============>…] - ETA: 1:43 - loss: 8.9954
416/808 [==============>…] - ETA: 1:43 - loss: 8.9957
417/808 [==============>…] - ETA: 1:43 - loss: 8.9943
418/808 [==============>…] - ETA: 1:42 - loss: 8.9932libpng warning: iCCP: known incorrect sRGB profile

419/808 [==============>…] - ETA: 1:42 - loss: 8.9965
420/808 [==============>…] - ETA: 1:42 - loss: 8.9958
421/808 [==============>…] - ETA: 1:42 - loss: 8.9941
422/808 [==============>…] - ETA: 1:41 - loss: 8.9937libpng warning: iCCP: known incorrect sRGB profile

423/808 [==============>…] - ETA: 1:41 - loss: 8.9937
424/808 [==============>…] - ETA: 1:41 - loss: 8.9925
425/808 [==============>…] - ETA: 1:40 - loss: 8.9915
426/808 [==============>…] - ETA: 1:40 - loss: 8.9902libpng warning: iCCP: known incorrect sRGB profile

427/808 [==============>…] - ETA: 1:40 - loss: 8.9890libpng warning: iCCP: known incorrect sRGB profile

428/808 [==============>…] - ETA: 1:39 - loss: 8.9887libpng warning: iCCP: known incorrect sRGB profile

429/808 [==============>…] - ETA: 1:39 - loss: 8.9887libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile

430/808 [==============>…] - ETA: 1:39 - loss: 8.9876
431/808 [===============>…] - ETA: 1:39 - loss: 8.9864libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile

432/808 [===============>…] - ETA: 1:38 - loss: 8.9846
433/808 [===============>…] - ETA: 1:38 - loss: 8.9835
434/808 [===============>…] - ETA: 1:38 - loss: 8.9826
435/808 [===============>…] - ETA: 1:37 - loss: 8.9821
436/808 [===============>…] - ETA: 1:37 - loss: 8.9811libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile

437/808 [===============>…] - ETA: 1:37 - loss: 8.9818libpng warning: iCCP: profile ‘ICC Profile’: 1000000h: invalid rendering intent
libpng warning: iCCP: known incorrect sRGB profile

438/808 [===============>…] - ETA: 1:37 - loss: 8.9807
439/808 [===============>…] - ETA: 1:36 - loss: 8.9814libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile

440/808 [===============>…] - ETA: 1:36 - loss: 8.9808
441/808 [===============>…] - ETA: 1:36 - loss: 8.9808
442/808 [===============>…] - ETA: 1:35 - loss: 8.9818
443/808 [===============>…] - ETA: 1:35 - loss: 8.9820libpng warning: iCCP: known incorrect sRGB profile

444/808 [===============>…] - ETA: 1:35 - loss: 8.9811
445/808 [===============>…] - ETA: 1:35 - loss: 8.9798libpng warning: iCCP: known incorrect sRGB profile

446/808 [===============>…] - ETA: 1:34 - loss: 8.9795
447/808 [===============>…] - ETA: 1:34 - loss: 8.9797
448/808 [===============>…] - ETA: 1:34 - loss: 8.9782libpng warning: iCCP: known incorrect sRGB profile

449/808 [===============>…] - ETA: 1:34 - loss: 8.9771
450/808 [===============>…] - ETA: 1:33 - loss: 8.9773
451/808 [===============>…] - ETA: 1:33 - loss: 8.9775
452/808 [===============>…] - ETA: 1:33 - loss: 8.9773
453/808 [===============>…] - ETA: 1:32 - loss: 8.9764libpng warning: iCCP: known incorrect sRGB profile

454/808 [===============>…] - ETA: 1:32 - loss: 8.9757
455/808 [===============>…] - ETA: 1:32 - loss: 8.9746
456/808 [===============>…] - ETA: 1:32 - loss: 8.9734libpng warning: iCCP: known incorrect sRGB profile

457/808 [===============>…] - ETA: 1:31 - loss: 8.9724libpng warning: iCCP: known incorrect sRGB profile

458/808 [================>…] - ETA: 1:31 - loss: 8.9726
459/808 [================>…] - ETA: 1:31 - loss: 8.9727libpng warning: iCCP: known incorrect sRGB profile

460/808 [================>…] - ETA: 1:30 - loss: 8.9737libpng warning: iCCP: known incorrect sRGB profile

461/808 [================>…] - ETA: 1:30 - loss: 8.9728
462/808 [================>…] - ETA: 1:30 - loss: 8.9716libpng warning: iCCP: known incorrect sRGB profile

463/808 [================>…] - ETA: 1:30 - loss: 8.9723
464/808 [================>…] - ETA: 1:29 - loss: 8.9722
465/808 [================>…] - ETA: 1:29 - loss: 8.9717
466/808 [================>…] - ETA: 1:29 - loss: 8.9724libpng warning: iCCP: known incorrect sRGB profile

467/808 [================>…] - ETA: 1:28 - loss: 8.9716libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile

468/808 [================>…] - ETA: 1:28 - loss: 8.9702
469/808 [================>…] - ETA: 1:28 - loss: 8.9685
470/808 [================>…] - ETA: 1:28 - loss: 8.9689libpng warning: iCCP: known incorrect sRGB profile

471/808 [================>…] - ETA: 1:27 - loss: 8.9679
472/808 [================>…] - ETA: 1:27 - loss: 8.9670
473/808 [================>…] - ETA: 1:27 - loss: 8.9666
474/808 [================>…] - ETA: 1:27 - loss: 8.9657
475/808 [================>…] - ETA: 1:26 - loss: 8.9646
476/808 [================>…] - ETA: 1:26 - loss: 8.9638
477/808 [================>…] - ETA: 1:26 - loss: 8.9630
478/808 [================>…] - ETA: 1:25 - loss: 8.9617
479/808 [================>…] - ETA: 1:25 - loss: 8.9594libpng warning: iCCP: known incorrect sRGB profile

480/808 [================>…] - ETA: 1:25 - loss: 8.9588libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile

481/808 [================>…] - ETA: 1:25 - loss: 8.9591
482/808 [================>…] - ETA: 1:24 - loss: 8.9587
483/808 [================>…] - ETA: 1:24 - loss: 8.9594
484/808 [================>…] - ETA: 1:24 - loss: 8.9586
485/808 [=================>…] - ETA: 1:23 - loss: 8.9579
486/808 [=================>…] - ETA: 1:23 - loss: 8.9572libpng warning: iCCP: known incorrect sRGB profile

487/808 [=================>…] - ETA: 1:23 - loss: 8.9561
488/808 [=================>…] - ETA: 1:23 - loss: 8.9553
489/808 [=================>…] - ETA: 1:22 - loss: 8.9549libpng warning: iCCP: known incorrect sRGB profile

490/808 [=================>…] - ETA: 1:22 - loss: 8.9543libpng warning: iCCP: known incorrect sRGB profile

491/808 [=================>…] - ETA: 1:22 - loss: 8.9540
492/808 [=================>…] - ETA: 1:22 - loss: 8.9527
493/808 [=================>…] - ETA: 1:21 - loss: 8.9526
494/808 [=================>…] - ETA: 1:21 - loss: 8.9524
495/808 [=================>…] - ETA: 1:21 - loss: 8.9504libpng warning: iCCP: known incorrect sRGB profile

496/808 [=================>…] - ETA: 1:20 - loss: 8.9510libpng warning: iCCP: known incorrect sRGB profile

497/808 [=================>…] - ETA: 1:20 - loss: 8.9514
498/808 [=================>…] - ETA: 1:20 - loss: 8.9519
499/808 [=================>…] - ETA: 1:20 - loss: 8.9507libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile

500/808 [=================>…] - ETA: 1:19 - loss: 8.9500
501/808 [=================>…] - ETA: 1:19 - loss: 8.9496
502/808 [=================>…] - ETA: 1:19 - loss: 8.9479
503/808 [=================>…] - ETA: 1:19 - loss: 8.9469libpng warning: iCCP: known incorrect sRGB profile

504/808 [=================>…] - ETA: 1:18 - loss: 8.9462libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile

505/808 [=================>…] - ETA: 1:18 - loss: 8.9470libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile

506/808 [=================>…] - ETA: 1:18 - loss: 8.9460
507/808 [=================>…] - ETA: 1:17 - loss: 8.9458
508/808 [=================>…] - ETA: 1:17 - loss: 8.9460
509/808 [=================>…] - ETA: 1:17 - loss: 8.9447
510/808 [=================>…] - ETA: 1:17 - loss: 8.9455libpng warning: iCCP: known incorrect sRGB profile

511/808 [=================>…] - ETA: 1:16 - loss: 8.9457
512/808 [==================>…] - ETA: 1:16 - loss: 8.9451
513/808 [==================>…] - ETA: 1:16 - loss: 8.9458
514/808 [==================>…] - ETA: 1:16 - loss: 8.9446libpng warning: iCCP: known incorrect sRGB profile

515/808 [==================>…] - ETA: 1:15 - loss: 8.9426libpng warning: iCCP: known incorrect sRGB profile

516/808 [==================>…] - ETA: 1:15 - loss: 8.9422
517/808 [==================>…] - ETA: 1:15 - loss: 8.9417libpng warning: iCCP: known incorrect sRGB profile

518/808 [==================>…] - ETA: 1:14 - loss: 8.9405libpng warning: iCCP: known incorrect sRGB profile

519/808 [==================>…] - ETA: 1:14 - loss: 8.9397
520/808 [==================>…] - ETA: 1:14 - loss: 8.9396
521/808 [==================>…] - ETA: 1:14 - loss: 8.9376libpng warning: iCCP: known incorrect sRGB profile

522/808 [==================>…] - ETA: 1:13 - loss: 8.9375libpng warning: iCCP: known incorrect sRGB profile

523/808 [==================>…] - ETA: 1:13 - loss: 8.9376
524/808 [==================>…] - ETA: 1:13 - loss: 8.9373
525/808 [==================>…] - ETA: 1:13 - loss: 8.9365
526/808 [==================>…] - ETA: 1:12 - loss: 8.9356
527/808 [==================>…] - ETA: 1:12 - loss: 8.9337libpng warning: iCCP: profile ‘ICC Profile’: 1000000h: invalid rendering intent
libpng warning: iCCP: known incorrect sRGB profile

528/808 [==================>…] - ETA: 1:12 - loss: 8.9325
529/808 [==================>…] - ETA: 1:11 - loss: 8.9317libpng warning: iCCP: known incorrect sRGB profile

530/808 [==================>…] - ETA: 1:11 - loss: 8.9310
531/808 [==================>…] - ETA: 1:11 - loss: 8.9300
532/808 [==================>…] - ETA: 1:11 - loss: 8.9302
533/808 [==================>…] - ETA: 1:10 - loss: 8.9289libpng warning: iCCP: known incorrect sRGB profile

534/808 [==================>…] - ETA: 1:10 - loss: 8.9284libpng warning: iCCP: known incorrect sRGB profile

535/808 [==================>…] - ETA: 1:10 - loss: 8.9281libpng warning: iCCP: known incorrect sRGB profile

536/808 [==================>…] - ETA: 1:10 - loss: 8.9269libpng warning: iCCP: known incorrect sRGB profile

537/808 [==================>…] - ETA: 1:09 - loss: 8.9263
538/808 [==================>…] - ETA: 1:09 - loss: 8.9252libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile

539/808 [===================>…] - ETA: 1:09 - loss: 8.9247
540/808 [===================>…] - ETA: 1:09 - loss: 8.9225
541/808 [===================>…] - ETA: 1:08 - loss: 8.9216
542/808 [===================>…] - ETA: 1:08 - loss: 8.9209libpng warning: iCCP: known incorrect sRGB profile

543/808 [===================>…] - ETA: 1:08 - loss: 8.9213
544/808 [===================>…] - ETA: 1:07 - loss: 8.9206libpng warning: iCCP: known incorrect sRGB profile

545/808 [===================>…] - ETA: 1:07 - loss: 8.9199libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile

546/808 [===================>…] - ETA: 1:07 - loss: 8.9182libpng warning: iCCP: known incorrect sRGB profile

547/808 [===================>…] - ETA: 1:07 - loss: 8.9180libpng warning: iCCP: known incorrect sRGB profile

548/808 [===================>…] - ETA: 1:06 - loss: 8.9177
549/808 [===================>…] - ETA: 1:06 - loss: 8.9172
550/808 [===================>…] - ETA: 1:06 - loss: 8.9160libpng warning: iCCP: profile ‘ICC Profile’: 1000000h: invalid rendering intent

551/808 [===================>…] - ETA: 1:06 - loss: 8.9160libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile

552/808 [===================>…] - ETA: 1:05 - loss: 8.9161libpng warning: iCCP: known incorrect sRGB profile

553/808 [===================>…] - ETA: 1:05 - loss: 8.9146libpng warning: iCCP: known incorrect sRGB profile

554/808 [===================>…] - ETA: 1:05 - loss: 8.9136
555/808 [===================>…] - ETA: 1:05 - loss: 8.9130libpng warning: iCCP: known incorrect sRGB profile

556/808 [===================>…] - ETA: 1:04 - loss: 8.9124
557/808 [===================>…] - ETA: 1:04 - loss: 8.9117libpng warning: iCCP: known incorrect sRGB profile

558/808 [===================>…] - ETA: 1:04 - loss: 8.9106libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile

559/808 [===================>…] - ETA: 1:03 - loss: 8.9104
560/808 [===================>…] - ETA: 1:03 - loss: 8.9095
561/808 [===================>…] - ETA: 1:03 - loss: 8.9090libpng warning: iCCP: known incorrect sRGB profile

562/808 [===================>…] - ETA: 1:03 - loss: 8.9083libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile

563/808 [===================>…] - ETA: 1:02 - loss: 8.9078libpng warning: iCCP: known incorrect sRGB profile

564/808 [===================>…] - ETA: 1:02 - loss: 8.9063libpng warning: iCCP: known incorrect sRGB profile

565/808 [===================>…] - ETA: 1:02 - loss: 8.9063
566/808 [====================>…] - ETA: 1:02 - loss: 8.9053libpng warning: iCCP: known incorrect sRGB profile

567/808 [====================>…] - ETA: 1:01 - loss: 8.9048libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile

568/808 [====================>…] - ETA: 1:01 - loss: 8.9039
569/808 [====================>…] - ETA: 1:01 - loss: 8.9042libpng warning: iCCP: known incorrect sRGB profile

570/808 [====================>…] - ETA: 1:01 - loss: 8.9029
571/808 [====================>…] - ETA: 1:00 - loss: 8.9016libpng warning: iCCP: known incorrect sRGB profile

572/808 [====================>…] - ETA: 1:00 - loss: 8.9014libpng warning: iCCP: known incorrect sRGB profile

573/808 [====================>…] - ETA: 1:00 - loss: 8.9010libpng warning: iCCP: known incorrect sRGB profile

574/808 [====================>…] - ETA: 59s - loss: 8.9009 libpng warning: iCCP: known incorrect sRGB profile

575/808 [====================>…] - ETA: 59s - loss: 8.8994
576/808 [====================>…] - ETA: 59s - loss: 8.9001
577/808 [====================>…] - ETA: 59s - loss: 8.8988
578/808 [====================>…] - ETA: 58s - loss: 8.8982libpng warning: iCCP: known incorrect sRGB profile

579/808 [====================>…] - ETA: 58s - loss: 8.8966
580/808 [====================>…] - ETA: 58s - loss: 8.8948libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile

581/808 [====================>…] - ETA: 58s - loss: 8.8940
582/808 [====================>…] - ETA: 57s - loss: 8.8938libpng warning: iCCP: profile ‘ICC Profile’: 1000000h: invalid rendering intent

583/808 [====================>…] - ETA: 57s - loss: 8.8932libpng warning: iCCP: known incorrect sRGB profile

584/808 [====================>…] - ETA: 57s - loss: 8.8935libpng warning: iCCP: known incorrect sRGB profile

585/808 [====================>…] - ETA: 57s - loss: 8.8923libpng warning: iCCP: known incorrect sRGB profile

586/808 [====================>…] - ETA: 56s - loss: 8.8922libpng warning: iCCP: known incorrect sRGB profile

587/808 [====================>…] - ETA: 56s - loss: 8.8906
588/808 [====================>…] - ETA: 56s - loss: 8.8895
589/808 [====================>…] - ETA: 56s - loss: 8.8884libpng warning: iCCP: known incorrect sRGB profile

590/808 [====================>…] - ETA: 55s - loss: 8.8882
591/808 [====================>…] - ETA: 55s - loss: 8.8875libpng warning: iCCP: known incorrect sRGB profile

592/808 [====================>…] - ETA: 55s - loss: 8.8865
593/808 [=====================>…] - ETA: 54s - loss: 8.8851
594/808 [=====================>…] - ETA: 54s - loss: 8.8845libpng warning: iCCP: profile ‘ICC Profile’: 0h: PCS illuminant is not D50
libpng warning: iCCP: known incorrect sRGB profile

595/808 [=====================>…] - ETA: 54s - loss: 8.8847libpng warning: iCCP: known incorrect sRGB profile

596/808 [=====================>…] - ETA: 54s - loss: 8.8847
597/808 [=====================>…] - ETA: 53s - loss: 8.8845libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile

598/808 [=====================>…] - ETA: 53s - loss: 8.8842
599/808 [=====================>…] - ETA: 53s - loss: 8.8835
600/808 [=====================>…] - ETA: 53s - loss: 8.8826
601/808 [=====================>…] - ETA: 52s - loss: 8.8818
602/808 [=====================>…] - ETA: 52s - loss: 8.8809libpng warning: iCCP: known incorrect sRGB profile

603/808 [=====================>…] - ETA: 52s - loss: 8.8804libpng warning: iCCP: known incorrect sRGB profile

604/808 [=====================>…] - ETA: 52s - loss: 8.8801
605/808 [=====================>…] - ETA: 51s - loss: 8.8798libpng warning: iCCP: known incorrect sRGB profile

606/808 [=====================>…] - ETA: 51s - loss: 8.8786libpng warning: iCCP: known incorrect sRGB profile

607/808 [=====================>…] - ETA: 51s - loss: 8.8784libpng warning: iCCP: known incorrect sRGB profile

608/808 [=====================>…] - ETA: 51s - loss: 8.8773libpng warning: iCCP: known incorrect sRGB profile

609/808 [=====================>…] - ETA: 50s - loss: 8.8763libpng warning: iCCP: known incorrect sRGB profile

610/808 [=====================>…] - ETA: 50s - loss: 8.8756libpng warning: iCCP: known incorrect sRGB profile

611/808 [=====================>…] - ETA: 50s - loss: 8.8758
612/808 [=====================>…] - ETA: 50s - loss: 8.8751
613/808 [=====================>…] - ETA: 49s - loss: 8.8736
614/808 [=====================>…] - ETA: 49s - loss: 8.8742libpng warning: iCCP: known incorrect sRGB profile

615/808 [=====================>…] - ETA: 49s - loss: 8.8745
616/808 [=====================>…] - ETA: 48s - loss: 8.8736libpng warning: iCCP: known incorrect sRGB profile

617/808 [=====================>…] - ETA: 48s - loss: 8.8727libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile

618/808 [=====================>…] - ETA: 48s - loss: 8.8722
619/808 [=====================>…] - ETA: 48s - loss: 8.8715libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile

620/808 [======================>…] - ETA: 47s - loss: 8.8711
621/808 [======================>…] - ETA: 47s - loss: 8.8716libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile

622/808 [======================>…] - ETA: 47s - loss: 8.8707
623/808 [======================>…] - ETA: 47s - loss: 8.8694libpng warning: iCCP: known incorrect sRGB profile

624/808 [======================>…] - ETA: 46s - loss: 8.8684
625/808 [======================>…] - ETA: 46s - loss: 8.8679libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile

626/808 [======================>…] - ETA: 46s - loss: 8.8671
627/808 [======================>…] - ETA: 46s - loss: 8.8670libpng warning: iCCP: known incorrect sRGB profile

628/808 [======================>…] - ETA: 45s - loss: 8.8664
629/808 [======================>…] - ETA: 45s - loss: 8.8662
630/808 [======================>…] - ETA: 45s - loss: 8.8651libpng warning: iCCP: known incorrect sRGB profile

631/808 [======================>…] - ETA: 45s - loss: 8.8645
632/808 [======================>…] - ETA: 44s - loss: 8.8629
633/808 [======================>…] - ETA: 44s - loss: 8.8630libpng warning: iCCP: known incorrect sRGB profile

634/808 [======================>…] - ETA: 44s - loss: 8.8613
635/808 [======================>…] - ETA: 44s - loss: 8.8619libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile

636/808 [======================>…] - ETA: 43s - loss: 8.8612
637/808 [======================>…] - ETA: 43s - loss: 8.8604libpng warning: iCCP: known incorrect sRGB profile

638/808 [======================>…] - ETA: 43s - loss: 8.8604
639/808 [======================>…] - ETA: 42s - loss: 8.8596
640/808 [======================>…] - ETA: 42s - loss: 8.8590libpng warning: iCCP: known incorrect sRGB profile

641/808 [======================>…] - ETA: 42s - loss: 8.8584
642/808 [======================>…] - ETA: 42s - loss: 8.8580
643/808 [======================>…] - ETA: 41s - loss: 8.8570
644/808 [======================>…] - ETA: 41s - loss: 8.8577
645/808 [======================>…] - ETA: 41s - loss: 8.8574libpng warning: iCCP: known incorrect sRGB profile

646/808 [======================>…] - ETA: 41s - loss: 8.8567libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile

647/808 [=======================>…] - ETA: 40s - loss: 8.8564libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile

648/808 [=======================>…] - ETA: 40s - loss: 8.8558
649/808 [=======================>…] - ETA: 40s - loss: 8.8545
650/808 [=======================>…] - ETA: 40s - loss: 8.8558
651/808 [=======================>…] - ETA: 39s - loss: 8.8550libpng warning: iCCP: profile ‘ICC Profile’: ‘desc’: ICC profile tag start not a multiple of 4
libpng warning: iCCP: profile ‘ICC Profile’: ‘wtpt’: ICC profile tag start not a multiple of 4
libpng warning: iCCP: profile ‘ICC Profile’: ‘bkpt’: ICC profile tag start not a multiple of 4
libpng warning: iCCP: profile ‘ICC Profile’: ‘rXYZ’: ICC profile tag start not a multiple of 4
libpng warning: iCCP: profile ‘ICC Profile’: ‘gXYZ’: ICC profile tag start not a multiple of 4
libpng warning: iCCP: profile ‘ICC Profile’: ‘bXYZ’: ICC profile tag start not a multiple of 4
libpng warning: iCCP: profile ‘ICC Profile’: ‘dmnd’: ICC profile tag start not a multiple of 4
libpng warning: iCCP: profile ‘ICC Profile’: ‘dmdd’: ICC profile tag start not a multiple of 4
libpng warning: iCCP: profile ‘ICC Profile’: ‘vued’: ICC profile tag start not a multiple of 4
libpng warning: iCCP: profile ‘ICC Profile’: ‘view’: ICC profile tag start not a multiple of 4
libpng warning: iCCP: profile ‘ICC Profile’: ‘lumi’: ICC profile tag start not a multiple of 4
libpng warning: iCCP: profile ‘ICC Profile’: ‘meas’: ICC profile tag start not a multiple of 4
libpng warning: iCCP: profile ‘ICC Profile’: ‘tech’: ICC profile tag start not a multiple of 4
libpng warning: iCCP: profile ‘ICC Profile’: ‘rTRC’: ICC profile tag start not a multiple of 4
libpng warning: iCCP: profile ‘ICC Profile’: ‘gTRC’: ICC profile tag start not a multiple of 4
libpng warning: iCCP: profile ‘ICC Profile’: ‘bTRC’: ICC profile tag start not a multiple of 4
libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile

652/808 [=======================>…] - ETA: 39s - loss: 8.8546libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile

653/808 [=======================>…] - ETA: 39s - loss: 8.8534Traceback (most recent call last):
File “/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/retinanet/scripts/train.py”, line 390, in
File “/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/common/utils.py”, line 528, in return_func
File “/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/common/utils.py”, line 516, in return_func
File “/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/retinanet/scripts/train.py”, line 386, in main
File “/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/retinanet/scripts/train.py”, line 308, in run_experiment
File “/usr/local/lib/python3.6/dist-packages/keras/engine/training.py”, line 1039, in fit
validation_steps=validation_steps)
File “/usr/local/lib/python3.6/dist-packages/keras/engine/training_arrays.py”, line 154, in fit_loop
outs = f(ins)
File “/usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py”, line 2715, in call
return self._call(inputs)
File “/usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py”, line 2675, in _call
fetched = self._callable_fn(*array_vals)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py”, line 1472, in call
run_metadata_ptr)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
(0) Invalid argument: ConcatOp : Ranks of all input tensors should match: shape[0] = [1] vs. shape[1] =
[[{{node concat_4}}]]
[[cond/BipartiteMatch/Cast/_3417]]
(1) Invalid argument: ConcatOp : Ranks of all input tensors should match: shape[0] = [1] vs. shape[1] =
[[{{node concat_4}}]]
0 successful operations.
0 derived errors ignored.

6

I am afraid some dataset has the mismatching issue. To narrow down, please use part of training tfrecord files and re-run.

7

What do you mean mismatching issue? I tried by just using 2 classes from the dataset and still having the same issue.

8

There are some warning like “libpng warning: iCCP: known incorrect sRGB profile
libpng warning: iCCP: known incorrect sRGB profile” or “libpng warning: iCCP: profile ‘ICC Profile’: ‘bXYZ’: ICC profile tag start not a multiple of 4”

Please use one tfrecord file and retry. If it works, add one more tfrecord file and retry …

9

I added the tfrecord one by one, training run successfully on all except one specific record

train-fold-000-of-002-shard-00008-of-00010

Screen Shot 2022-05-24 at 3.42.56 PM
Screen Shot 2022-05-24 at 3.42.56 PM2000×126 78.6 KB

How can I further debug this file to see what’s causing the problem?

10

I think I may have solved the issue, the dataset_convert specs file was missing the class mapping of some of the classes in my dataset which caused the tfrecord files to be generated incorrectly.

kitti_config {
root_directory_path: “/workspace/tao-experiments/data/”
image_dir_name: “image”
label_dir_name: “label”
image_extension: “.png”
partition_mode: “random”
num_partitions: 2
val_split: 0
num_shards: 10
}
image_directory_path: “/workspace/tao-experiments/data/”
target_class_mapping {
key: “object”
value: “object”
}
target_class_mapping {
key: “remote”
value: “remote”
}
target_class_mapping {
key: “cell_phone”
value: “cell_phone”
}
target_class_mapping {
key: “toothbrush”
value: “toothbrush”
}