While I train YOLO with my own dataset on cmd window,
I got cudaDeviceSynchronize() error on 5069th iteration,
Do you think it’s just an accidental error or are there any guide lines for below error?
Until 5068th iteration, the training seems under a good condition.
3.30GHz CPU i5 was mostly fully occupied.
GPU GTX 1080 was used in 0~89% range.
13GB RAM had enough space.
[ERROR message]
5069: 0.051683, 0.079799 avg loss, 0.001000 rate, 12.920081 seconds, 324416 images
Loaded: 9.079948 seconds
CUDA status = cudaDeviceSynchronize() Error: file: d:\darknet\src\cuda.c : cuda_get_device() : line: 23 : build time: Feb 22 2019 - 11:20:47
CUDA Error: unknown error