Ran out of memory on GPU

AI & Data Science Deep Learning (Training & Inference) Frameworks
#1 
2019-05-06 14:02:27.264629: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally
2019-05-06 14:02:30.554284: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.45GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-05-06 14:02:30.555624: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.02GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-05-06 14:02:30.884528: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.72GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-05-06 14:02:32.482456: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.72GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-05-06 14:02:32.482999: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.54GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-05-06 14:02:34.231668: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.32GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-05-06 14:02:35.367897: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.27GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-05-06 14:02:35.949636: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.09GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-05-06 14:05:05.327823: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.11GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-05-06 14:05:05.629307: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.08GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.

Does this mean that GPU is not being used? I am using a tensorflow:latest-gpu-py3 image

#2

The GPU is definitely being used. Such warnings are fairly normal as you push the batch size toward its upper limit. One place they can arise is in autotuning convolution algorithms. Some GPU conv algorithms require allocating temporary workspaces. If allocation of the workspace fails, the above warning is issued, but autotuning continues over GPU algos with smaller workspace requirements.

#3

InvalidArgumentError: Tensor input_image:0, specified in either feed_devices or fetch_devices was not found in the Graph

I’m getting this error and I can’t get it fixed. Could it be due to those warnings stated above?