Hi,
This could happen for a few reasons.
- As you mentioned, it may be a a memory issue, which you could try to verify by allocating less memory to the GPU and seeing if that error still occurs. You can do this in TF 2.0 like so (https://github.com/tensorflow/tensorflow/issues/25138#issuecomment-484428798):
import tensorflow as tf
tf.config.gpu.set_per_process_memory_fraction(0.75)
tf.config.gpu.set_per_process_memory_growth(True)
# your model creation, etc.
model = MyModel(...)
I see the code you’re running sets dynamic memory growth if you have > 1 GPU (https://github.com/zzh8829/yolov3-tf2/blob/master/train.py#L46-L47), but since you only have 1 GPU, then it is likely just trying to allocate all memory (>90%) at the start.
-
Some users seem to have experienced this on Windows when there were other TensorFlow or similar processes using the GPU simultaneously, either by you or by other users: https://stackoverflow.com/a/53707323/10993413
-
As always, make sure your PATH variables are correct. Sometimes if you tried multiple installations and didn’t clean things up properly, the PATHs may be finding the wrong version first, and using that, causing an issue. If you add new paths to the beginning of PATH, they should be found first: https://www.tensorflow.org/install/gpu#windows_setup
-
As mentioned on your Stack Overflow post by another user, you could try upgrading to a newer version of CUDNN, though I’m not sure this will help since your config is listed as supported on TF docs: 從原始碼開始建構 | TensorFlow. If this does solve it, it may have been PATH issue after all since you will likely update the PATHs after installing the newer version.