TAO unet producing nan values

Running unet isbi example notebook with this data and spec produces this training output 2022 04 04.txt (76.1 KB) output in the training step, and in the evaluate step evaluate output 2022 04 04.txt (41.6 KB) this output.

And also tried following laurim’s advice here Training multi-class UNet does not converge - #31 by laurim? to set weight to 0. and removed the augmentation_config section to same results.

Also tried recreating the masks such that the background label is 2 and plant and leaf as 0 and 1 with same results.