Fluctuating validation accuracy of YOLOv4-tiny in TAO 5.0.0

I prepared a simple dataset containing people smoking and the annotation data in KITTI format.
training: 762images
validation: 183 images

The annotation example is as followed:

smoking 0 0 0.0 158 74 1229 819 0 0 0 0 0 0 0 1.0, where

158 74 1229 819 represents x_min, y_min, x_max, y_max, respectively.
They form the bounding box shown in the image below.

The last value 1.0 represents score, but I think it’s of no use here.
However, according to this page

The shown KITTI format there doesn’t include score. I’m not sure if it matters as I was still able to run tao training using the annotated data I had prepared.

I ran kmeans on my own dataset and modified the anchor values with the results generated by kmeans.
Still, I also made other modifications in the config file below.
yolo_v4_tiny_train_kitti_seq_smoke_only_new.txt (2.1 KB)

I’m here to list all of them one by one:

big_anchor_shape and mid_anchor_shape: replaced the default values with the generated results from kmean

batch_size_per_gpu: Modified from 8 to 32

parameters in soft_start_cosine_annealing_schedule(or soft_start_annealing_schedule):
min_learning_rate: from 1e-7 to 1e-6
max_learning_rate: fromt 1e-4 to 1e-2(or 1e-3)
I also set annealing to 0.3 when I tried soft_start_annealing_schedule

output_width: from 1248 to 384

target_class_mapping: only smoking class

training and validation_data_sources

During training, the mAP weirdly fluctuated.

INFO: Starting Training Loop.
Epoch 1/100
25/25 [==============================] - 88s 4s/step - loss: 2673.3562
Producing predictions: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 25/25 [00:08<00:00, 2.78it/s]
Start to calculate AP for each class


smoking AP 0.07707
mAP 0.07707


Validation loss: 2572.7906013257575
INFO: Training loop in progress
Epoch 2/100
25/25 [==============================] - 71s 3s/step - loss: 4816.4711
Producing predictions: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 25/25 [00:06<00:00, 3.65it/s]
Start to calculate AP for each class


smoking AP 0.04018
mAP 0.04018


Validation loss: 2078.263588028725
INFO: Training loop in progress
Epoch 3/100
25/25 [==============================] - 54s 2s/step - loss: 3044.0435
Producing predictions: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 25/25 [00:06<00:00, 4.00it/s]
Start to calculate AP for each class


smoking AP 0.02312
mAP 0.02312


Validation loss: 1063.2112580886994
INFO: Training loop in progress
Epoch 4/100
25/25 [==============================] - 66s 3s/step - loss: 1638.7887
Producing predictions: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 25/25 [00:06<00:00, 3.65it/s]
Start to calculate AP for each class


smoking AP 0.00351
mAP 0.00351


Validation loss: 3531.0549760298295
INFO: Training loop in progress
Epoch 5/100
25/25 [==============================] - 70s 3s/step - loss: 866.2483
Producing predictions: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 25/25 [00:06<00:00, 3.88it/s]
Start to calculate AP for each class


smoking AP 0.0011
mAP 0.0011


Validation loss: 32408.78073705808
INFO: Training loop in progress
Epoch 6/100
25/25 [==============================] - 66s 3s/step - loss: 392.1233
Producing predictions: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 25/25 [00:06<00:00, 3.88it/s]
Start to calculate AP for each class


smoking AP 0.00901
mAP 0.00901


Validation loss: 167.35042810921718
INFO: Training loop in progress
Epoch 7/100
25/25 [==============================] - 66s 3s/step - loss: 205.9445
Producing predictions: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 25/25 [00:06<00:00, 3.93it/s]
Start to calculate AP for each class


smoking AP 0.01001
mAP 0.01001


Validation loss: 376.28182212752523
INFO: Training loop in progress
Epoch 8/100
25/25 [==============================] - 70s 3s/step - loss: 155.3828
Producing predictions: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 25/25 [00:06<00:00, 4.14it/s]
Start to calculate AP for each class


smoking AP 0.01251
mAP 0.01251


Validation loss: 327.0372024690262
INFO: Training loop in progress
Epoch 9/100
25/25 [==============================] - 55s 2s/step - loss: 125.0881
Producing predictions: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 25/25 [00:06<00:00, 3.65it/s]
Start to calculate AP for each class


smoking AP 0.11301
mAP 0.11301


Validation loss: 270.55066842743844
INFO: Training loop in progress
Epoch 10/100
25/25 [==============================] - 66s 3s/step - loss: 118.1201
Producing predictions: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 25/25 [00:06<00:00, 4.03it/s]
Start to calculate AP for each class


smoking AP 0.08624
mAP 0.08624


Validation loss: 217.81548039600102

Epoch 00010: saving model to /workspace/tao-experiments/yolo_v4_tiny/experiment_dir_unpruned_smoking/weights/yolov4_cspdarknet_tiny_epoch_010.hdf5
INFO: Training loop in progress
Epoch 11/100
25/25 [==============================] - 81s 3s/step - loss: 131.8296
Producing predictions: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 25/25 [00:06<00:00, 4.04it/s]
Start to calculate AP for each class


smoking AP 0.09134
mAP 0.09134


Validation loss: 135.1073814931542
INFO: Training loop in progress
Epoch 12/100
25/25 [==============================] - 49s 2s/step - loss: 123.6221
Producing predictions: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 25/25 [00:06<00:00, 4.00it/s]
Start to calculate AP for each class


smoking AP 0.2068
mAP 0.2068


Validation loss: 111.23007556645557
INFO: Training loop in progress
Epoch 13/100
25/25 [==============================] - 52s 2s/step - loss: 85.0279
Producing predictions: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 25/25 [00:05<00:00, 4.21it/s]
Start to calculate AP for each class


smoking AP 0.5603
mAP 0.5603


Validation loss: 104.5875682638149
INFO: Training loop in progress
Epoch 14/100
25/25 [==============================] - 81s 3s/step - loss: 87.0963
Producing predictions: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 25/25 [00:06<00:00, 3.80it/s]
Start to calculate AP for each class


smoking AP 0.24214
mAP 0.24214


Validation loss: 75.83081440010456
INFO: Training loop in progress
Epoch 15/100
25/25 [==============================] - 56s 2s/step - loss: 78.5005
Producing predictions: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 25/25 [00:06<00:00, 3.90it/s]
Start to calculate AP for each class


smoking AP 0.27755
mAP 0.27755


Validation loss: 71.13194953070746
INFO: Training loop in progress
Epoch 16/100
25/25 [==============================] - 88s 4s/step - loss: 80.3134
Producing predictions: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 25/25 [00:06<00:00, 3.64it/s]
Start to calculate AP for each class


smoking AP 0.35114
mAP 0.35114


Validation loss: 71.0700780695135
INFO: Training loop in progress
Epoch 17/100
25/25 [==============================] - 58s 2s/step - loss: 76.2839
Producing predictions: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 25/25 [00:07<00:00, 3.57it/s]
Start to calculate AP for each class


smoking AP 0.41444
mAP 0.41444


Validation loss: 60.80281228730173
INFO: Training loop in progress
Epoch 18/100
25/25 [==============================] - 83s 3s/step - loss: 72.6713
Producing predictions: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 25/25 [00:06<00:00, 3.87it/s]
Start to calculate AP for each class


smoking AP 0.06757
mAP 0.06757


Validation loss: 53.45103755141749
INFO: Training loop in progress
Epoch 19/100
25/25 [==============================] - 57s 2s/step - loss: 77.6783
Producing predictions: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 25/25 [00:06<00:00, 4.07it/s]
Start to calculate AP for each class


smoking AP 0.50432
mAP 0.50432


Validation loss: 51.597843979344226
INFO: Training loop in progress
Epoch 20/100
25/25 [==============================] - 62s 2s/step - loss: 69.0703
Producing predictions: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 25/25 [00:06<00:00, 3.90it/s]
Start to calculate AP for each class


smoking AP 0.30715
mAP 0.30715


Validation loss: 48.57911666715988

Epoch 00020: saving model to /workspace/tao-experiments/yolo_v4_tiny/experiment_dir_unpruned_smoking/weights/yolov4_cspdarknet_tiny_epoch_020.hdf5
INFO: Training loop in progress
Epoch 21/100
25/25 [==============================] - 51s 2s/step - loss: 74.3389
Producing predictions: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 25/25 [00:06<00:00, 3.98it/s]
Start to calculate AP for each class


smoking AP 0.24655
mAP 0.24655


Validation loss: 46.32186658454664
INFO: Training loop in progress
Epoch 22/100
25/25 [==============================] - 83s 3s/step - loss: 67.2906
Producing predictions: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 25/25 [00:06<00:00, 3.93it/s]
Start to calculate AP for each class


smoking AP 0.12553
mAP 0.12553


Validation loss: 46.86462209682272
INFO: Training loop in progress
Epoch 23/100
25/25 [==============================] - 62s 2s/step - loss: 68.4061
Producing predictions: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 25/25 [00:06<00:00, 3.90it/s]
Start to calculate AP for each class


smoking AP 0.36286
mAP 0.36286


Validation loss: 43.366982932042596
INFO: Training loop in progress
Epoch 24/100
25/25 [==============================] - 57s 2s/step - loss: 62.9102
Producing predictions: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 25/25 [00:06<00:00, 4.01it/s]
Start to calculate AP for each class


smoking AP 0.31774
mAP 0.31774


Validation loss: 42.14983980583422
INFO: Training loop in progress
Epoch 25/100
25/25 [==============================] - 85s 3s/step - loss: 70.7289
Producing predictions: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 25/25 [00:06<00:00, 3.57it/s]
Start to calculate AP for each class


smoking AP 0.34921
mAP 0.34921


Validation loss: 46.02775396481909
INFO: Training loop in progress
Epoch 26/100
25/25 [==============================] - 53s 2s/step - loss: 64.0946
Producing predictions: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 25/25 [00:06<00:00, 3.77it/s]
Start to calculate AP for each class


smoking AP 0.46719
mAP 0.46719


Validation loss: 40.331729079737805
INFO: Training loop in progress
Epoch 27/100
25/25 [==============================] - 61s 2s/step - loss: 58.9223
Producing predictions: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 25/25 [00:06<00:00, 3.73it/s]
Start to calculate AP for each class


smoking AP 0.57853
mAP 0.57853


Validation loss: 59.00261229698104
INFO: Training loop in progress
Epoch 28/100
25/25 [==============================] - 51s 2s/step - loss: 62.3263
Producing predictions: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 25/25 [00:05<00:00, 4.17it/s]
Start to calculate AP for each class


smoking AP 0.74894
mAP 0.74894


Validation loss: 35.31016413370768
INFO: Training loop in progress
Epoch 29/100
25/25 [==============================] - 62s 2s/step - loss: 57.3677
Producing predictions: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 25/25 [00:06<00:00, 3.93it/s]
Start to calculate AP for each class


smoking AP 0.69051
mAP 0.69051


Validation loss: 31.09010052921796
INFO: Training loop in progress
Epoch 30/100
25/25 [==============================] - 58s 2s/step - loss: 54.5162
Producing predictions: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 25/25 [00:06<00:00, 3.90it/s]
Start to calculate AP for each class


smoking AP 0.17569
mAP 0.17569


Validation loss: 37.61037811125168

Epoch 00030: saving model to /workspace/tao-experiments/yolo_v4_tiny/experiment_dir_unpruned_smoking/weights/yolov4_cspdarknet_tiny_epoch_030.hdf5
INFO: Training loop in progress
Epoch 31/100
25/25 [==============================] - 81s 3s/step - loss: 71.8543
Producing predictions: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 25/25 [00:06<00:00, 3.93it/s]
Start to calculate AP for each class


smoking AP 0.12494
mAP 0.12494


Validation loss: 35.03747435290404
INFO: Training loop in progress
Epoch 32/100
25/25 [==============================] - 57s 2s/step - loss: 60.4696
Producing predictions: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 25/25 [00:06<00:00, 3.87it/s]
Start to calculate AP for each class


smoking AP 0.54802
mAP 0.54802


Validation loss: 30.464541618270104
INFO: Training loop in progress
Epoch 33/100
25/25 [==============================] - 58s 2s/step - loss: 56.1650
Producing predictions: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 25/25 [00:06<00:00, 3.94it/s]
Start to calculate AP for each class


smoking AP 0.41138
mAP 0.41138


Validation loss: 32.67412964021317
INFO: Training loop in progress
Epoch 34/100
25/25 [==============================] - 51s 2s/step - loss: 62.4648
Producing predictions: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 25/25 [00:06<00:00, 3.83it/s]
Start to calculate AP for each class


smoking AP 0.23967
mAP 0.23967


Validation loss: 35.15605672200521
INFO: Training loop in progress
Epoch 35/100
25/25 [==============================] - 82s 3s/step - loss: 52.5718
Producing predictions: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 25/25 [00:07<00:00, 3.48it/s]
Start to calculate AP for each class


smoking AP 0.56368
mAP 0.56368


Validation loss: 31.7618452129942
INFO: Training loop in progress
Epoch 36/100
25/25 [==============================] - 54s 2s/step - loss: 48.1929
Producing predictions: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 25/25 [00:06<00:00, 3.82it/s]
Start to calculate AP for each class


smoking AP 0.55678
mAP 0.55678


Validation loss: 31.313318175498885
INFO: Training loop in progress
Epoch 37/100
25/25 [==============================] - 71s 3s/step - loss: 51.7320
Producing predictions: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 25/25 [00:06<00:00, 3.88it/s]
Start to calculate AP for each class


smoking AP 0.38675
mAP 0.38675


Validation loss: 34.50323201189138
INFO: Training loop in progress
Epoch 38/100
25/25 [==============================] - 54s 2s/step - loss: 55.7664
Producing predictions: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 25/25 [00:06<00:00, 3.72it/s]
Start to calculate AP for each class


smoking AP 0.35641
mAP 0.35641


Validation loss: 32.14118668527314
INFO: Training loop in progress
Epoch 39/100

Not sure what step I did incorrectly.

Though heavily fluctuating, the mAP literally kept increasing as the training went on.

Also, because I modified the output_width, I guess the pretrained model didn’t work and the network was trained from scratch instead of transfer learning, right?

Pretrained model should be working. Please set a lower training batch_size and retry.
For example, batch_size_per_gpu: 4

I could run training with the downloaded pretrained model used. It’s just that I wondered if it’s still of some use when I intend to train a model with different input size.

Do you mean that the phenomenon of the fluctuating accuracy may have something to do with the batch size?

Yes, you can use a lower batch-size.

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.