Cannot retrain SSD Mobilenet with Custom Dataset

jerryfan08 · May 26, 2023, 10:55pm

I am currently working on a project that uses object detection to sort rubbish into recyclable and non-recyclable. I have taken some images (around 200 I think) and I am trying to retrain SSD Mobilenet within the docker container. However, I keep getting an OSError and supposedly “trainval.txt” does not exist, but when i look in the local file manager, the file is present.

Here’s the log:
root@ubuntu:/jetson-inference/python/training/detection/ssd# python3 train_ssd.py --dataset-type=voc --data=data/RubbishData --model-dir=models/rubbish_sorter_models --batch-size=2 --workers=1 --epoch=1
2023-05-26 22:41:43 - Using CUDA…
2023-05-26 22:41:43 - Namespace(balance_data=False, base_net=None, base_net_lr=0.001, batch_size=2, checkpoint_folder=‘models/rubbish_sorter_models’, dataset_type=‘voc’, datasets=[‘data/RubbishData’], debug_steps=10, extra_layers_lr=None, freeze_base_net=False, freeze_net=False, gamma=0.1, log_level=‘info’, lr=0.01, mb2_width_mult=1.0, milestones=‘80,100’, momentum=0.9, net=‘mb1-ssd’, num_epochs=1, num_workers=1, pretrained_ssd=‘models/mobilenet-v1-ssd-mp-0_675.pth’, resolution=300, resume=None, scheduler=‘cosine’, t_max=100, use_cuda=True, validation_epochs=1, validation_mean_ap=False, weight_decay=0.0005)
2023-05-26 22:44:43 - model resolution 300x300
2023-05-26 22:44:43 - SSDSpec(feature_map_size=19, shrinkage=16, box_sizes=SSDBoxSizes(min=60, max=105), aspect_ratios=[2, 3])
2023-05-26 22:44:43 - SSDSpec(feature_map_size=10, shrinkage=32, box_sizes=SSDBoxSizes(min=105, max=150), aspect_ratios=[2, 3])
2023-05-26 22:44:43 - SSDSpec(feature_map_size=5, shrinkage=64, box_sizes=SSDBoxSizes(min=150, max=195), aspect_ratios=[2, 3])
2023-05-26 22:44:43 - SSDSpec(feature_map_size=3, shrinkage=100, box_sizes=SSDBoxSizes(min=195, max=240), aspect_ratios=[2, 3])
2023-05-26 22:44:43 - SSDSpec(feature_map_size=2, shrinkage=150, box_sizes=SSDBoxSizes(min=240, max=285), aspect_ratios=[2, 3])
2023-05-26 22:44:43 - SSDSpec(feature_map_size=1, shrinkage=300, box_sizes=SSDBoxSizes(min=285, max=330), aspect_ratios=[2, 3])
2023-05-26 22:44:45 - Prepare training datasets.
Traceback (most recent call last):
File “train_ssd.py”, line 260, in
target_transform=target_transform)
File “/jetson-inference/python/training/detection/ssd/vision/datasets/voc_dataset.py”, line 44, in init
raise IOError(f"missing ImageSet file {image_sets_file}")
OSError: missing ImageSet file data/RubbishData/ImageSets/Main/trainval.txt

I would really appreciate it if anyone could help me out here.

Also, I was wondering if it is possible to output extra text after the detection box text while running detectNet - e.g. (Plastic Bottle 89.96% - Recyclable). If that is not possible, is there an option to change the colours of the boxes, such that there may be something like a green box for recyclable and a red box for non recyclable? I hope that’s clear.

AastaLLL · May 29, 2023, 4:01am

Hi,

Since you train it within a container, have you mounted the data folder when launching the container?

To output the detection result, you can check the Python script below.
Just adding what you want by checking the detection variable.

github.com

dusty-nv/jetson-inference/blob/master/python/examples/detectnet.py#L77


      
          
          
if img is None: # timeout
              continue  
              
          # detect objects in the image (with overlay)
          detections = net.Detect(img, overlay=args.overlay)
          
          
# print the detections
          print("detected {:d} objects in image".format(len(detections)))
          
          
for detection in detections:
              print(detection)
          
          
# render the image
          output.Render(img)
          
          
# update the title bar
          output.SetStatus("{:s} | Network {:.0f} FPS".format(args.network, net.GetNetworkFPS()))
          
          
# print out performance info
          net.PrintProfilerTimes()

Thanks.

dusty_nv · May 30, 2023, 3:44pm

jerryfan08:

However, I keep getting an OSError and supposedly “trainval.txt” does not exist, but when i look in the local file manager, the file is present.

Here’s the log:

root@ubuntu:/jetson-inference/python/training/detection/ssd# python3 train_ssd.py --dataset-type=voc --data=data/RubbishData --model-dir=models/rubbish_sorter_models --batch-size=2 --workers=1 --epoch=1
...
File “/jetson-inference/python/training/detection/ssd/vision/datasets/voc_dataset.py”, line 44, in **init**
raise IOError(f"missing ImageSet file {image_sets_file}")
OSError: missing ImageSet file data/RubbishData/ImageSets/Main/trainval.txt

Hi @jerryfan08, the /jetson-inference/python/training/detection/ssd/data directory (and the ssd/models directory) should automatically be mounted into the container if you started it with docker/run.sh. Can you try running this from within the container to double-check that it’s mounted okay?

# from within the container
cd /jetson-inference/python/training/detection/ssd/
ls -ll data/RubbishData/ImageSets/Main

You should see the test.txt train.txt trainval.txt val.txt files listed there in the container, mounted from your host device.

If you are okay with it reading like Plastic Bottle Recyclable - 89.96%, it would be easiest just to edit the models/rubbish_sorter_models/labels.txt with the class name as you wish for it to appear. The copy of labels.txt stored under your model’s directory is used for inference, not training, so you can change it as you wish. Alternatively, you can modify the lower-level code that generates the text that gets rendered here:

https://github.com/dusty-nv/jetson-inference/blob/e6de5e403c2ccbea90b074968627a8562a23c253/c/detectNet.cpp#L1043

Yes, you can create a colors.txt for your model that has one RGB/RGBA color per line (corresponding to the class in labels.txt) and you can load it with the --colors command-line option to detectnet.py or with the optional colors argument to the detectNet initializer.

The format of colors.txt should be like this (typically the first line is the ignored BACKGROUND class)

0 0 0
0 255 0
255 0 0 127

(you can optionally specify the fourth alpha component to the color)

jerryfan08 · May 30, 2023, 3:59pm

Dear all, thanks for your suggestions. I had already solved the problem and it turned out that I had just mistakenly structured the Pascal VOC data structure wrong and forgot the Main folder.

I will try changing the labels and also having coloured boxes, thanks!!!