CUB segmented reduce errortoo many resources requested for launch

Hey guys, I flashed my system new, loaded necessary dependency for object detection model. At first, tensorflow is working but its for cpu, gave the similiar error at below. I supposed that the error occurs because it didn’t use GPU. Then installed necessary packages to use gpu. Now, it seems that GPU does work but gives the same error. My weight back up matrix has 16 MB but when I doing testing, it gives memory alloc error.

I dont think the error is caused by my model. Its too small, and image size is small too.

Can anybody give any idea about what is wrong with my necessary packages or any clue that I can’t see.

NOT: I use tensorflow 1.7.0 now.

Batch images : Tensor(shape=( 480 , 320), dtype=float32)

2018-08-02 11:10:53: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:865] ARM64 does not support NUMA - returning NUMA node zero
2018-08-02 11:10:53: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1344] Found device 0 with properties:
name: NVIDIA Tegra X2 major: 6 minor: 2 memoryClockRate(GHz): 1.3005
pciBusID: 0000:00:00.0
totalMemory: 7.66GiB freeMemory: 6.05GiB
2018-08-02 11:10:53: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-08-02 11:10:55: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-08-02 11:10:55: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917] 0
2018-08-02 11:10:55: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0: N
2018-08-02 11:10:55: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 784 MB memory) → physical GPU (device: 0, name: NVIDIA Tegra X2, pci bus id: 0000:00:00.0, compute capability: 6.2)
Model x restored.
2018-08-02 11:11:10: W tensorflow/core/common_runtime/bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.05GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2018-08-02 11:11:10: W tensorflow/core/common_runtime/bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.05GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2018-08-02 11:11:10: W tensorflow/core/common_runtime/bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.04GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2018-08-02 11:11:10: W tensorflow/core/common_runtime/bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.05GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.

Traceback (most recent call last):

tensorflow.python.framework.errors_impl.InternalError: CUB segmented reduce errortoo many resources requested for launch
[[Node: ArgMax/_645 = _Recvclient_terminated=false, recv_device=“/job:localhost/replica:0/task:0/device:CPU:0”, send_device=“/job:localhost/replica:0/task:0/device:GPU:0”, send_device_incarnation=1, tensor_type=DT_INT64, _device=“/job:localhost/replica:0/task:0/device:CPU:0”]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
tensorflow.python.framework.errors_impl.InternalError: CUB segmented reduce errortoo many resources requested for launch
[[Node: ArgMax/_645 = _Recvclient_terminated=false, recv_device=“/job:localhost/replica:0/task:0/device:CPU:0”, send_device=“/job:localhost/replica:0/task:0/device:GPU:0”, send_device_incarnation=1, tensor_type=DT_INT64, _device=“/job:localhost/replica:0/task:0/device:CPU:0”]]

Hi,

This error is caused by unified physical memory.
TensorFlow by default allocates all the available memory which is not working in an integrated platform.

Please apply this configuration to avoid this error:

config = tf.ConfigProto()
config.gpu_options.allow_growth = True

session = tf.Session(config=config, ...)

Thanks.