I’m trying to make my own custom object detection on Jetson AGX Orin using Nvidia ‘Hello AI world’ tutorial.
I would launch the model on live video stream by using an USB Camera, but before I’m testing the model on a video.
I started labelling just a hundred of frames (from 12th to 114th). However I looked it doesn’t detect 2 labels out of 11: they are NA_15 and LE_05. It’s very strange because I can see them labelled in annotation files.
To explain better how I’m working, the steps are as follow:
Label video source on CVAT and export data in PASCAL VOC type;
Hi @domenico.depascale, I would run train_ssd.py with the --validation-mean-ap flag, and it will print out the per-class accuracies (Mean Average Precision) during the validation step. That should let you know how well the model is trained on each of the classes. You may need to add more data to your dataset for the classes that are underperforming.
Dear @dusty_nv and @AastaLLL , like Dusty suggested I have tried to add more data to my dataset and enable the --validation-mean-ap flag (these are the results eval_results.7z (277.1 KB)). I labelled the full video source on CVAT (about 1950 frames, Annotations.7z (15.8 KB)), but similar error still appears. Now the label not detected is NA_16 and not NA_15. This is very strange beacause I labelled the same number of frame as for NA_15 as for NA_16. In add, this two are the labels which appears frequently during the video. In fact I tracked more than 1 thousand frame with this two labels.
More strange it’s this situation: before this I made the same work on the same video source, which has been cropped, and the all process works very properly!
Hi @domenico.depascale, those eval_results seem to be the raw detection bounding box outputs, so it’s hard for me to say if the model is balanced in terms of mean average precision across classes. At the end of each training epoch, it should have printed out a per-class summary when run with the --validation-mean-ap flag that will let you know the per-class accuracy over the test dataset.
Hi @domenico.depascale, it makes sense that NA_16 class isn’t being detected during inference, because the mAP of that class is zero (along with TC_25):
Hi @dusty_nv I understood that the zero value of NA_16 class mAP is the problem (like TC_25), but I don’t understand why. The dataset is the same used for the other classes. In particular the entire dataset cames from the same video source. So I can’t get a reason to this problem. I repeat that the same process works properly on the same video makes cropped.
Do you need more informations or other data? Hope you have an advice for this which can help me.
Thanks in advance.
PS: I attach the annotation files used for the training. How you can see the NA_16 annotations are present from frame 12 to 1936 one.
Is there anything printed out when you first start the training like ‘unknown class NA_16’ or ‘missing labels for NA_16’ which may indicate they are mislabeled? The fact that it’s literally 0.0000% accuracy would seem to point to some problem with the dataset linked to that class.
Is the content material of the NA_16 class inherently more challenging to detect in some way? (i.e. the NA_16 objects are less discernable/identifiable, they are odd shapes/aspect ratios, they have larger variations, ect)
Hi @dusty_nv there isn’t anything printed out which may indicate the label is unknown. There is just this warning message:
/usr/local/lib/python3.8/dist-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction=‘sum’ instead.
warnings.warn(warning.format(ret))
About the second question: The entire dataset cames from the same video source. During labelling phase on CVAT, the tool divided the video into 1952 frames. I used only these ones. In add, NA_16 class indicates an arrow up-down and NA_15 class indicates an arrow down-up: there are the same in dimension, shapes, aspect ratios, but just NA_16 is recognized.
I’m available if you need something else. If you want a direct contact we can arrange a meeting.
Hmm okay - I wonder if perhaps the NA_16 arrows and the NA_15 arrows look so similar that the object detector has trouble telling them apart. Sometimes when in-depth classification is used, it can be done in a two stage process where first detection is run, and then a secondary classifier is run on the detections (i.e. the detector finds all ‘arrows’, and then the classifier picks the type of ‘arrow’).
In the earlier training epochs, NA_16 is actually being detected with an mAP of up to 90%. But then it drops to 0, presumably because the model fits to the other arrow type instead. You could try running the training with the --balance-data flag to see if that helps. However, it may be that this level of discernibility between classes is beyond the capabilities of this SSD-Mobilenet network (which is optimized for speed and performance to be deployed on embedded platforms). I’m not an expert on training, so you may need to experiment further or look to other types of detection models out there.
You should just be able to run it with --balance-data flag, you don’t need to set it to --balance-data=true or anything (since the action is already store_true)
I’m not exactly sure, sorry. Another idea I had is to select your ssd-mobilenet model from an epoch which trained the classes evenly. You can look at the Mean Average Precision report for each epoch and find one that is most balanced for all your classes. Then select this model checkpoint to export to ONNX.
