So i was training my first CNN with tensorflow but at the 7th epoch i get this error message that stated “GPU sync failed”. I was looking for an answer and i found that it may be because my GPU ran out of memory (i’ve got a RTX 2060). And it’s very possible because i’ve left my PC alone, training my CNN, and i found my brother was playing a game, so that may have caused the lack of memory.
Beyond all of that, i have a few questions that surged after this episode:
Did the error message appear because i ran out of memory or could it be caused for other reason?
If i get the “GPU Sync failed” error message, that means i’m using my GPU for the training process, right? because it was going too slow and i was thinking that it was running on the CPU.
If that error message does not mean i’m using my GPU, how can i check that?
What can i do to train my CNN faster?
That would be all, thanks!