tensorflow/stream_executor/cuda/cuda_driver.cc:965] failed to alloc 2304 bytes on host: CUDA_ERROR_UNKNOWN

Autonomous Machines Jetson & Embedded Systems Jetson TX2
1

ubuntu@tegra-ubuntu:~$ python3
Python 3.5.2 (default, Sep 14 2017, 22:51:06)
[GCC 5.4.0 20160609] on linux
Type “help”, “copyright”, “credits” or “license” for more information.

import tensorflow as tf
hello = tf.constant(‘Hello, TensorFlow!’)
sess = tf.Session()
2017-11-21 06:22:38.820962: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:857] ARM64 does not support NUMA - returning NUMA node zero
2017-11-21 06:22:38.821085: I tensorflow/core/common_runtime/gpu/gpu_device.cc:955] Found device 0 with properties:
name: NVIDIA Tegra X2
major: 6 minor: 2 memoryClockRate (GHz) 1.3005
pciBusID 0000:00:00.0
Total memory: 7.67GiB
Free memory: 5.34GiB
2017-11-21 06:22:38.821134: I tensorflow/core/common_runtime/gpu/gpu_device.cc:976] DMA: 0
2017-11-21 06:22:38.821159: I tensorflow/core/common_runtime/gpu/gpu_device.cc:986] 0: Y
2017-11-21 06:22:38.821195: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1045] Creating TensorFlow device (/gpu:0) → (device: 0, name: NVIDIA Tegra X2, pci bus id: 0000:00:00.0)
print(tf.Session())
2017-11-21 06:23:00.939320: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1045] Creating TensorFlow device (/gpu:0) → (device: 0, name: NVIDIA Tegra X2, pci bus id: 0000:00:00.0)
<tensorflow.python.client.session.Session object at 0x7f97f03ef0>
a=tf.constant(10)
b=tf.constant(32)
print(sess.run(a+b))

2018-09-19 15:00:55.674557: W ./tensorflow/core/common_runtime/gpu/pool_allocator.h:195] could not allocate pinned host memory of size: 2304
2018-09-19 15:00:55.674580: E tensorflow/stream_executor/cuda/cuda_driver.cc:965] failed to alloc 2304 bytes on host: CUDA_ERROR_UNKNOWN
2018-09-19 15:00:55.674602: W ./tensorflow/core/common_runtime/gpu/pool_allocator.h:195] could not allocate pinned host memory of size: 2304
2018-09-19 15:00:55.674625: E tensorflow/stream_executor/cuda/cuda_driver.cc:965] failed to alloc 2304 bytes on host: CUDA_ERROR_UNKNOWN
2018-09-19 15:00:55.674646: W ./tensorflow/core/common_runtime/gpu/pool_allocator.h:195] could not allocate pinned host memory of size: 2304
2018-09-19 15:00:55.674669: E tensorflow/stream_executor/cuda/cuda_driver.cc:965] failed to alloc 2304 bytes on host: CUDA_ERROR_UNKNOWN
2018-09-19 15:00:55.674691: W ./tensorflow/core/common_runtime/gpu/pool_allocator.h:195] could not allocate pinned host memory of size: 2304
2018-09-19 15:00:55.674714: E tensorflow/stream_executor/cuda/cuda_driver.cc:965] failed to alloc 2304 bytes on host: CUDA_ERROR_UNKNOWN
2018-09-19 15:00:55.674736: W ./tensorflow/core/common_runtime/gpu/pool_allocator.h:195] could not allocate pinned host memory of size: 2304
2018-09-19 15:00:55.674759: E tensorflow/stream_executor/cuda/cuda_driver.cc:965] failed to alloc 2304 bytes on host: CUDA_ERROR_UNKNOWN
2018-09-19 15:00:55.674781: W ./tensorflow/core/common_runtime/gpu/pool_allocator.h:195] could not allocate pinned host memory of size: 2304
2018-09-19 15:00:55.674803: E tensorflow/stream_executor/cuda/cuda_driver.cc:965] failed to alloc 2304 bytes on host: CUDA_ERROR_UNKNOWN
2018-09-19 15:00:55.674825: W ./tensorflow/core/common_runtime/gpu/pool_allocator.h:195] could not allocate pinned host memory of size: 2304
2018-09-19 15:00:55.674848: E tensorflow/stream_executor/cuda/cuda_driver.cc:965] failed to alloc 2304 bytes on host: CUDA_ERROR_UNKNOWN

2

Hi,

Please try this:

config = tf.ConfigProto()
config.gpu_options.allow_growth = True

session = tf.Session(config=config, ...)

Thanks.

3

Thanks.

4

Worked for me in Python 2.7.12 on the TX2 (JetPack 3.3/L4T 28.2.1) running Tensorflow 1.9.