Please provide the following information when requesting support.
• Hardware - RTX 3090
• Network Type - Vanilla Unet Dynamic
• TAO Toolkit 4.0.0 on WSL2 - Ubuntu 20.04.6 LTS inside Windows 11
• Training spec file -
scratch_v1_1.txt (1.6 KB)
• I’m trying to train a binary segmentation model to identify scratches on metal sheets.
Sample Image -
I’m referring to Nvidia’s implementation of U-Net for Industrial Defect Detection on DAGM2007 dataset (which is weakly labeled so I have also labeled my data in similar elliptical manner).
My issue is that regardless of how many epochs I run, the final model gives a blank black output image & doesn’t segment anything, essentially classifying everything as background. The overlayed inferenced output looks like -
The evaluation results are -
Recall : 0.5 Precision: 0.9808197021484375 F1 score: 0.9903169895620691 Mean IOU: 0.49040985107421875
I have tried unet, vanilla_unet_dynamic, resnet, vgg. I have referred to other posts where Morgan had suggested to make sure all the images are png etc.
I’m not understanding what I’m doing wrong. I have previously successfully trained Unet Segmentation models on TAO. But many times when I add more data to those successfully trained models in a new training iteration, I come across this same issue where the final model stops segmenting anything and just generates a blank output.