To narrow down, please double check below.
- Do your training meet below requirement?
- Input size : C * W * H (where C = 3, W > =128, H >=128 and W, H are multiples of 32)
- Image format : JPG
- Label format : COCO detection
- Can you try to train with the public dataset mentioned in the jupyter notebook again?
- Try to reboot
- Try to train with a smaller network
- Try to train with smaller image_size
More reference for OOM issue:
Maskrcnn:
Other networks