Input contains NaN or infinity

I use Clara SDK version3.0 to classify chest x-ray.
when I run train.sh, a program raise this exception.
“Exception: <class ‘ValueError’>: Input contains NaN, infinty or a value too large for dtype(‘float32’).”

Thank you for your interest in Clara Train. Would you please share some more details about the hardware your running this, some more details on the training scripts and perhaps the training log file?

Sure, I use g4dn.2xlarge instance (AWS), NVIDIA Tesla T4, 8vCPUs and 32GB RAM.
training_log.zip (3.1 KB)
And I don’t know why it can’t detect GPUs

Hi,
It seems there’s something with the node you’re using. Are you able to run nvidia-smi to make sure that the GPU is present and recognized. I believe we have to resolve this issue first as it is likely impacting the subsequent training.