I prepared a simple dataset containing people smoking and the annotation data in KITTI format.
training: 762images
validation: 183 images
The annotation example is as followed:
smoking 0 0 0.0 158 74 1229 819 0 0 0 0 0 0 0 1.0, where
158 74 1229 819 represents x_min, y_min, x_max, y_max, respectively.
They form the bounding box shown in the image below.
The last value 1.0 represents score, but I think it’s of no use here.
However, according to this page
The shown KITTI format there doesn’t include score. I’m not sure if it matters as I was still able to run tao training using the annotated data I had prepared.
I ran kmeans on my own dataset and modified the anchor values with the results generated by kmeans.
Still, I also made other modifications in the config file below.
yolo_v4_tiny_train_kitti_seq_smoke_only_new.txt (2.1 KB)
I’m here to list all of them one by one:
big_anchor_shape and mid_anchor_shape: replaced the default values with the generated results from kmean
batch_size_per_gpu: Modified from 8 to 32
parameters in soft_start_cosine_annealing_schedule(or soft_start_annealing_schedule):
min_learning_rate: from 1e-7 to 1e-6
max_learning_rate: fromt 1e-4 to 1e-2(or 1e-3)
I also set annealing to 0.3 when I tried soft_start_annealing_schedule
output_width: from 1248 to 384
target_class_mapping: only smoking class
training and validation_data_sources
During training, the mAP weirdly fluctuated.
INFO: Starting Training Loop.
Epoch 1/100
25/25 [==============================] - 88s 4s/step - loss: 2673.3562
Producing predictions: 100%|████████████████████| 25/25 [00:08<00:00, 2.78it/s]
Start to calculate AP for each class
smoking AP 0.07707
mAP 0.07707
Validation loss: 2572.7906013257575
INFO: Training loop in progress
Epoch 2/100
25/25 [==============================] - 71s 3s/step - loss: 4816.4711
Producing predictions: 100%|████████████████████| 25/25 [00:06<00:00, 3.65it/s]
Start to calculate AP for each class
smoking AP 0.04018
mAP 0.04018
Validation loss: 2078.263588028725
INFO: Training loop in progress
Epoch 3/100
25/25 [==============================] - 54s 2s/step - loss: 3044.0435
Producing predictions: 100%|████████████████████| 25/25 [00:06<00:00, 4.00it/s]
Start to calculate AP for each class
smoking AP 0.02312
mAP 0.02312
Validation loss: 1063.2112580886994
INFO: Training loop in progress
Epoch 4/100
25/25 [==============================] - 66s 3s/step - loss: 1638.7887
Producing predictions: 100%|████████████████████| 25/25 [00:06<00:00, 3.65it/s]
Start to calculate AP for each class
smoking AP 0.00351
mAP 0.00351
Validation loss: 3531.0549760298295
INFO: Training loop in progress
Epoch 5/100
25/25 [==============================] - 70s 3s/step - loss: 866.2483
Producing predictions: 100%|████████████████████| 25/25 [00:06<00:00, 3.88it/s]
Start to calculate AP for each class
smoking AP 0.0011
mAP 0.0011
Validation loss: 32408.78073705808
INFO: Training loop in progress
Epoch 6/100
25/25 [==============================] - 66s 3s/step - loss: 392.1233
Producing predictions: 100%|████████████████████| 25/25 [00:06<00:00, 3.88it/s]
Start to calculate AP for each class
smoking AP 0.00901
mAP 0.00901
Validation loss: 167.35042810921718
INFO: Training loop in progress
Epoch 7/100
25/25 [==============================] - 66s 3s/step - loss: 205.9445
Producing predictions: 100%|████████████████████| 25/25 [00:06<00:00, 3.93it/s]
Start to calculate AP for each class
smoking AP 0.01001
mAP 0.01001
Validation loss: 376.28182212752523
INFO: Training loop in progress
Epoch 8/100
25/25 [==============================] - 70s 3s/step - loss: 155.3828
Producing predictions: 100%|████████████████████| 25/25 [00:06<00:00, 4.14it/s]
Start to calculate AP for each class
smoking AP 0.01251
mAP 0.01251
Validation loss: 327.0372024690262
INFO: Training loop in progress
Epoch 9/100
25/25 [==============================] - 55s 2s/step - loss: 125.0881
Producing predictions: 100%|████████████████████| 25/25 [00:06<00:00, 3.65it/s]
Start to calculate AP for each class
smoking AP 0.11301
mAP 0.11301
Validation loss: 270.55066842743844
INFO: Training loop in progress
Epoch 10/100
25/25 [==============================] - 66s 3s/step - loss: 118.1201
Producing predictions: 100%|████████████████████| 25/25 [00:06<00:00, 4.03it/s]
Start to calculate AP for each class
smoking AP 0.08624
mAP 0.08624
Validation loss: 217.81548039600102
Epoch 00010: saving model to /workspace/tao-experiments/yolo_v4_tiny/experiment_dir_unpruned_smoking/weights/yolov4_cspdarknet_tiny_epoch_010.hdf5
INFO: Training loop in progress
Epoch 11/100
25/25 [==============================] - 81s 3s/step - loss: 131.8296
Producing predictions: 100%|████████████████████| 25/25 [00:06<00:00, 4.04it/s]
Start to calculate AP for each class
smoking AP 0.09134
mAP 0.09134
Validation loss: 135.1073814931542
INFO: Training loop in progress
Epoch 12/100
25/25 [==============================] - 49s 2s/step - loss: 123.6221
Producing predictions: 100%|████████████████████| 25/25 [00:06<00:00, 4.00it/s]
Start to calculate AP for each class
smoking AP 0.2068
mAP 0.2068
Validation loss: 111.23007556645557
INFO: Training loop in progress
Epoch 13/100
25/25 [==============================] - 52s 2s/step - loss: 85.0279
Producing predictions: 100%|████████████████████| 25/25 [00:05<00:00, 4.21it/s]
Start to calculate AP for each class
smoking AP 0.5603
mAP 0.5603
Validation loss: 104.5875682638149
INFO: Training loop in progress
Epoch 14/100
25/25 [==============================] - 81s 3s/step - loss: 87.0963
Producing predictions: 100%|████████████████████| 25/25 [00:06<00:00, 3.80it/s]
Start to calculate AP for each class
smoking AP 0.24214
mAP 0.24214
Validation loss: 75.83081440010456
INFO: Training loop in progress
Epoch 15/100
25/25 [==============================] - 56s 2s/step - loss: 78.5005
Producing predictions: 100%|████████████████████| 25/25 [00:06<00:00, 3.90it/s]
Start to calculate AP for each class
smoking AP 0.27755
mAP 0.27755
Validation loss: 71.13194953070746
INFO: Training loop in progress
Epoch 16/100
25/25 [==============================] - 88s 4s/step - loss: 80.3134
Producing predictions: 100%|████████████████████| 25/25 [00:06<00:00, 3.64it/s]
Start to calculate AP for each class
smoking AP 0.35114
mAP 0.35114
Validation loss: 71.0700780695135
INFO: Training loop in progress
Epoch 17/100
25/25 [==============================] - 58s 2s/step - loss: 76.2839
Producing predictions: 100%|████████████████████| 25/25 [00:07<00:00, 3.57it/s]
Start to calculate AP for each class
smoking AP 0.41444
mAP 0.41444
Validation loss: 60.80281228730173
INFO: Training loop in progress
Epoch 18/100
25/25 [==============================] - 83s 3s/step - loss: 72.6713
Producing predictions: 100%|████████████████████| 25/25 [00:06<00:00, 3.87it/s]
Start to calculate AP for each class
smoking AP 0.06757
mAP 0.06757
Validation loss: 53.45103755141749
INFO: Training loop in progress
Epoch 19/100
25/25 [==============================] - 57s 2s/step - loss: 77.6783
Producing predictions: 100%|████████████████████| 25/25 [00:06<00:00, 4.07it/s]
Start to calculate AP for each class
smoking AP 0.50432
mAP 0.50432
Validation loss: 51.597843979344226
INFO: Training loop in progress
Epoch 20/100
25/25 [==============================] - 62s 2s/step - loss: 69.0703
Producing predictions: 100%|████████████████████| 25/25 [00:06<00:00, 3.90it/s]
Start to calculate AP for each class
smoking AP 0.30715
mAP 0.30715
Validation loss: 48.57911666715988
Epoch 00020: saving model to /workspace/tao-experiments/yolo_v4_tiny/experiment_dir_unpruned_smoking/weights/yolov4_cspdarknet_tiny_epoch_020.hdf5
INFO: Training loop in progress
Epoch 21/100
25/25 [==============================] - 51s 2s/step - loss: 74.3389
Producing predictions: 100%|████████████████████| 25/25 [00:06<00:00, 3.98it/s]
Start to calculate AP for each class
smoking AP 0.24655
mAP 0.24655
Validation loss: 46.32186658454664
INFO: Training loop in progress
Epoch 22/100
25/25 [==============================] - 83s 3s/step - loss: 67.2906
Producing predictions: 100%|████████████████████| 25/25 [00:06<00:00, 3.93it/s]
Start to calculate AP for each class
smoking AP 0.12553
mAP 0.12553
Validation loss: 46.86462209682272
INFO: Training loop in progress
Epoch 23/100
25/25 [==============================] - 62s 2s/step - loss: 68.4061
Producing predictions: 100%|████████████████████| 25/25 [00:06<00:00, 3.90it/s]
Start to calculate AP for each class
smoking AP 0.36286
mAP 0.36286
Validation loss: 43.366982932042596
INFO: Training loop in progress
Epoch 24/100
25/25 [==============================] - 57s 2s/step - loss: 62.9102
Producing predictions: 100%|████████████████████| 25/25 [00:06<00:00, 4.01it/s]
Start to calculate AP for each class
smoking AP 0.31774
mAP 0.31774
Validation loss: 42.14983980583422
INFO: Training loop in progress
Epoch 25/100
25/25 [==============================] - 85s 3s/step - loss: 70.7289
Producing predictions: 100%|████████████████████| 25/25 [00:06<00:00, 3.57it/s]
Start to calculate AP for each class
smoking AP 0.34921
mAP 0.34921
Validation loss: 46.02775396481909
INFO: Training loop in progress
Epoch 26/100
25/25 [==============================] - 53s 2s/step - loss: 64.0946
Producing predictions: 100%|████████████████████| 25/25 [00:06<00:00, 3.77it/s]
Start to calculate AP for each class
smoking AP 0.46719
mAP 0.46719
Validation loss: 40.331729079737805
INFO: Training loop in progress
Epoch 27/100
25/25 [==============================] - 61s 2s/step - loss: 58.9223
Producing predictions: 100%|████████████████████| 25/25 [00:06<00:00, 3.73it/s]
Start to calculate AP for each class
smoking AP 0.57853
mAP 0.57853
Validation loss: 59.00261229698104
INFO: Training loop in progress
Epoch 28/100
25/25 [==============================] - 51s 2s/step - loss: 62.3263
Producing predictions: 100%|████████████████████| 25/25 [00:05<00:00, 4.17it/s]
Start to calculate AP for each class
smoking AP 0.74894
mAP 0.74894
Validation loss: 35.31016413370768
INFO: Training loop in progress
Epoch 29/100
25/25 [==============================] - 62s 2s/step - loss: 57.3677
Producing predictions: 100%|████████████████████| 25/25 [00:06<00:00, 3.93it/s]
Start to calculate AP for each class
smoking AP 0.69051
mAP 0.69051
Validation loss: 31.09010052921796
INFO: Training loop in progress
Epoch 30/100
25/25 [==============================] - 58s 2s/step - loss: 54.5162
Producing predictions: 100%|████████████████████| 25/25 [00:06<00:00, 3.90it/s]
Start to calculate AP for each class
smoking AP 0.17569
mAP 0.17569
Validation loss: 37.61037811125168
Epoch 00030: saving model to /workspace/tao-experiments/yolo_v4_tiny/experiment_dir_unpruned_smoking/weights/yolov4_cspdarknet_tiny_epoch_030.hdf5
INFO: Training loop in progress
Epoch 31/100
25/25 [==============================] - 81s 3s/step - loss: 71.8543
Producing predictions: 100%|████████████████████| 25/25 [00:06<00:00, 3.93it/s]
Start to calculate AP for each class
smoking AP 0.12494
mAP 0.12494
Validation loss: 35.03747435290404
INFO: Training loop in progress
Epoch 32/100
25/25 [==============================] - 57s 2s/step - loss: 60.4696
Producing predictions: 100%|████████████████████| 25/25 [00:06<00:00, 3.87it/s]
Start to calculate AP for each class
smoking AP 0.54802
mAP 0.54802
Validation loss: 30.464541618270104
INFO: Training loop in progress
Epoch 33/100
25/25 [==============================] - 58s 2s/step - loss: 56.1650
Producing predictions: 100%|████████████████████| 25/25 [00:06<00:00, 3.94it/s]
Start to calculate AP for each class
smoking AP 0.41138
mAP 0.41138
Validation loss: 32.67412964021317
INFO: Training loop in progress
Epoch 34/100
25/25 [==============================] - 51s 2s/step - loss: 62.4648
Producing predictions: 100%|████████████████████| 25/25 [00:06<00:00, 3.83it/s]
Start to calculate AP for each class
smoking AP 0.23967
mAP 0.23967
Validation loss: 35.15605672200521
INFO: Training loop in progress
Epoch 35/100
25/25 [==============================] - 82s 3s/step - loss: 52.5718
Producing predictions: 100%|████████████████████| 25/25 [00:07<00:00, 3.48it/s]
Start to calculate AP for each class
smoking AP 0.56368
mAP 0.56368
Validation loss: 31.7618452129942
INFO: Training loop in progress
Epoch 36/100
25/25 [==============================] - 54s 2s/step - loss: 48.1929
Producing predictions: 100%|████████████████████| 25/25 [00:06<00:00, 3.82it/s]
Start to calculate AP for each class
smoking AP 0.55678
mAP 0.55678
Validation loss: 31.313318175498885
INFO: Training loop in progress
Epoch 37/100
25/25 [==============================] - 71s 3s/step - loss: 51.7320
Producing predictions: 100%|████████████████████| 25/25 [00:06<00:00, 3.88it/s]
Start to calculate AP for each class
smoking AP 0.38675
mAP 0.38675
Validation loss: 34.50323201189138
INFO: Training loop in progress
Epoch 38/100
25/25 [==============================] - 54s 2s/step - loss: 55.7664
Producing predictions: 100%|████████████████████| 25/25 [00:06<00:00, 3.72it/s]
Start to calculate AP for each class
smoking AP 0.35641
mAP 0.35641
Validation loss: 32.14118668527314
INFO: Training loop in progress
Epoch 39/100
Not sure what step I did incorrectly.
Though heavily fluctuating, the mAP literally kept increasing as the training went on.
Also, because I modified the output_width, I guess the pretrained model didn’t work and the network was trained from scratch instead of transfer learning, right?