Issue during Segmentation Training using custom network

Hi,

I am encountering this strange error where,
Upon training with Lr 1e-3, using Adam its giving below error and upon reducing the lr to 1e-4, training continues fine.
Just wanted to know the reason behind this strange behavior.

/opt/nvidia/medical/ai4med/utils/metric_accumulator.py:85: RuntimeWarning: Mean of empty slice.
/usr/local/lib/python3.6/dist-packages/numpy/core/_methods.py:161: RuntimeWarning: invalid value encountered in double_scalars
ret = ret.dtype.type(ret / rcount)

segResNetCPU.py (6.2 KB)