I ran into another problem trying to train a new model with my own data. I’m trying to create a model to track a climbing helmet. I’ve successfully trained the model with about 1000 images which I extraced from a video file.
Now I wanted to improve the detection rate and created another set of images which show the helmet from all possible angles. When I try to train these files I get:
2021-04-12 20:45:14 - Epoch: 0, Validation Loss: nan, Validation Regression Loss nan, Validation Classification Loss: nan
To make sure my dataset is in the correct format, I created two simple test-folders. One with an image + annotations from the first training session which works and a second one from my new set which returns nan.
I’ve attached a zip-file with the datasets. ssd_training_test.zip (1.6 MB)
I’m running them with the following command line:
Test1 works fine:
python3 train_ssd.py --dataset-type=voc --data=data/test1 --model-dir=models/test1 --num-epochs=100 --num-workers=10 --batch-size=1
Test2 returns nan:
python3 train_ssd.py --dataset-type=voc --data=data/test2 --model-dir=models/test2 --num-epochs=100 --num-workers=10 --batch-size=1
I have no clue where my error is? Anybody has any ideas?