Issue during Segmentation Training using custom network


I am encountering this strange error where,
Upon training with Lr 1e-3, using Adam its giving below error and upon reducing the lr to 1e-4, training continues fine.
Just wanted to know the reason behind this strange behavior.

/opt/nvidia/medical/ai4med/utils/ RuntimeWarning: Mean of empty slice.
/usr/local/lib/python3.6/dist-packages/numpy/core/ RuntimeWarning: invalid value encountered in double_scalars
ret = ret.dtype.type(ret / rcount) (6.2 KB)