Test: Warning -- some classes may be missing validation examples while training custom segmentation model

Hi @dusty_nv ,

We are training custom segmentation model (refering to this repo) with 3 classes i.e background, person, robot on custom dataset having 10k training images and masks and 2.5k validation images and masks.

On completing an epoch while validation during training we are getting this warning which shows the metric average row correct for robot class as nan and also the IoU is 0.0

Test:  Warning -- some classes may be missing validation examples
global correct: 98.1
average row correct: ['87.5', '98.7', 'nan']
IoU: ['70.6', '98.0', '0.0']
mean IoU: 56.2

This warning is present in utils.py

Can you please let us know what the warning indicates during training the custom segmentation model and why is this happening?

Does this affect the model accuracy during inference?

Hi @nishantshrivastav23, I believe what this means is that there aren’t instances of the 3rd class in your validation dataset. Can you confirm if that is the case?

Hi @dusty_nv,

We have instances of the third class i.e. robot in validation dataset still we are getting the mentioned warning.

Can you please confirm what is the root cause w.r.t to compute function in utils.py?

I’m not entirely sure, sorry about that - it’s been some years since I have needed to train segmentation model. The training code originally came from torchvision: https://github.com/pytorch/vision/tree/main/references/segmentation

I had added support for FCN-ResNet18 and made it ONNX exportable. Then Onixaz packaged it up into a self-contained repo. It appears there have been some updates to the original torchvision code in the meantime (including that error/exception you are getting has now been removed)

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.