CUDA stop working after rebooting TX2

Autonomous Machines Jetson & Embedded Systems Jetson TX2
1

Hello,

I have a problem with run my c++ tensorflow app in TX2 directly after reboot.

normally I build app by bazel build

bazel build -c opt --local_resources 3072,4.0,1.0 --verbose_failures --config=monolithic //tensorflow/app/...

after build, when i execute

bazel-bin/tensorflow/app/run

everything is work perfectly but, when I reboot TX2 and execute firstly bazel-bin I have spam of this errors.

May 11 12:29:05 tegra-ubuntu start.sh[800]: 2018-05-11 12:29:04.622723: E tensorflow/stream_executor/cuda/cuda_driver.cc:967] failed to alloc 2304 bytes on host: CUDA_ERROR_UNKNOWN
May 11 12:29:05 tegra-ubuntu start.sh[800]: 2018-05-11 12:29:04.622849: E tensorflow/stream_executor/cuda/cuda_driver.cc:967] failed to alloc 2304 bytes on host: CUDA_ERROR_UNKNOWN

I guess that is a problem in loaded packages. When bazel build is executing before bazel-run, loading packages is correct.
There is a way to only load missing packages without earlier building? I need to run app imidiatly after start but my building process is to long to building before run app :(

2

Hi,

This error occurs when TensorFlow try to allocate a huge amount of memory.

TensorFlow by default allocate all the available GPU memory.
On a fresh boot device, the available memory is very high (6.2 GB).
On iGPU environment, such a huge memory allocation will fail in general as host and GPU share the same memory.

Currently, a workaround is to restrict the amount of each memory allocation.
We apply these configure in the python interface:

config = tf.ConfigProto()
config.gpu_options.allow_growth = True
session = tf.Session(config=config, ...)

You can update your program with the corresponding configuration in C++ interface.
Thanks.