Test: Warning -- some classes may be missing validation examples while training custom segmentation model

nishantshrivastav23 · February 22, 2022, 10:20am

We are training custom segmentation model (refering to this repo) with 3 classes i.e background, person, robot on custom dataset having 10k training images and masks and 2.5k validation images and masks.

On completing an epoch while validation during training we are getting this warning which shows the metric average row correct for robot class as nan and also the IoU is 0.0

Test:  Warning -- some classes may be missing validation examples
global correct: 98.1
average row correct: ['87.5', '98.7', 'nan']
IoU: ['70.6', '98.0', '0.0']
mean IoU: 56.2

This warning is present in utils.py

github.com

dusty-nv/pytorch-segmentation/blob/16882772bc767511d892d134918722011d1ea771/utils.py#L102

      
        
                self.mat.zero_()
            
            
def compute(self):
                h = self.mat.float()
              
                # not all object classes may have data from the val category
                h_sum1 = h.sum(1)
                h_sum1[h_sum1 == 0] = 1
            
            
    if not torch.equal(h.sum(1), h_sum1):
                   print('Test:  Warning -- some classes may be missing validation examples')
            
            
    acc_global = torch.diag(h).sum() / h.sum()
                acc = torch.diag(h) / h.sum(1)
                iu = torch.diag(h) / (h_sum1 + h.sum(0) - torch.diag(h))
                self.acc_global = acc_global.item() * 100
                self.mean_IoU = iu.mean().item() * 100
                return acc_global, acc, iu
            
            
def reduce_from_all_processes(self):
                if not torch.distributed.is_available():

Can you please let us know what the warning indicates during training the custom segmentation model and why is this happening?

Does this affect the model accuracy during inference?

dusty_nv · February 22, 2022, 6:25pm

Hi @nishantshrivastav23, I believe what this means is that there aren’t instances of the 3rd class in your validation dataset. Can you confirm if that is the case?

nishantshrivastav23 · February 23, 2022, 6:26am

Hi @dusty_nv,

We have instances of the third class i.e. robot in validation dataset still we are getting the mentioned warning.

Can you please confirm what is the root cause w.r.t to compute function in utils.py?

dusty_nv · February 23, 2022, 8:01pm

I’m not entirely sure, sorry about that - it’s been some years since I have needed to train segmentation model. The training code originally came from torchvision: https://github.com/pytorch/vision/tree/main/references/segmentation

I had added support for FCN-ResNet18 and made it ONNX exportable. Then Onixaz packaged it up into a self-contained repo. It appears there have been some updates to the original torchvision code in the meantime (including that error/exception you are getting has now been removed)

system · March 23, 2022, 5:10am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.