Odd behavior with Jetpack 3.2 and tensorflow

Robotics & Edge Computing Jetson & Embedded Systems Jetson TX2
1

hello, I tried to install tensorflow for 3.2, cuda 9.0 and cudnn 7.0.5 via the following site and can report that I was able to install after multiple attempts, running commands like ‘bazel clean’ and etc.

However, when running validation test for tensorflow, there was odd behavior. First time I had to kill the process as this repeated without end:

2017-12-29 09:45:39.801035: E tensorflow/stream_executor/cuda/cuda_driver.cc:967] failed to alloc 2304 bytes on host: CUDA_ERROR_UNKNOWN
2017-12-29 09:45:39.801065: W ./tensorflow/core/common_runtime/gpu/pool_allocator.h:195] could not allocate pinned host memory of size: 2304

After this, it sort of worked but with odd warnings and results:

~$ python3
Python 3.5.2 (default, Nov 23 2017, 16:37:01) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
>>> hello = tf.constant('hello, all')
>>> sess = tf.Session()
2017-12-29 10:21:23.017060: E tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:878] could not open file to read NUMA node: /sys/bus/pci/devices/0000:00:00.0/numa_node
Your kernel may have been built without NUMA support.
2017-12-29 10:21:23.017180: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 0 with properties: 
name: NVIDIA Tegra X2 major: 6 minor: 2 memoryClockRate(GHz): 1.3005
pciBusID: 0000:00:00.0
totalMemory: 7.66GiB freeMemory: 144.84MiB
2017-12-29 10:21:23.017230: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: NVIDIA Tegra X2, pci bus id: 0000:00:00.0, compute capability: 6.2)
2017-12-29 10:21:23.017267: I tensorflow/core/common_runtime/gpu/gpu_device.cc:732] Could not identify NUMA node of /job:localhost/replica:0/task:0/device:GPU:0, defaulting to 0.  Your kernel may not have been built with NUMA support.
>>> print(sess.run(hello))
b'hello, all'

This doesn’t make me feel very confident in the installation, so if anyone can make any sense of this and how to resolve so that tensorflow functions more normally, it would be appreciated.

Perhaps I should note that installed with bazel 5.4 python 3.5, which differ from what the site author installed with. Also, I’d rather not revert to cuda 8.0 or 3.1 for the other known successful installations.

2

P.S. I am also uncertain if cudnn 7.0.5 was located correctly by the install - where should this library normally be located on the 3.2 jetpack architecture?

3

Hi,

Check cuDNN installation via this command:

ll /usr/lib/aarch64-linux-gnu/libcudnn*

From your log, TensorFlow is launched successfully.
There is no NUMA support on the ARM platform and it’s a harmless warning.

This information also can be found in the installation guide you shared in #1:
Step 5: Install TensorFlow
3.Open the …

Thanks.

4

Okay, will ignore the warning. This is the result from the cuDNN intallation check

nvidia@tegra-ubuntu:~$ ll /usr/lib/aarch64-linux-gnu/libcudnn*
lrwxrwxrwx 1 root root        29 Dec 21 04:45 /usr/lib/aarch64-linux-gnu/libcudnn.so -> /etc/alternatives/libcudnn_so
lrwxrwxrwx 1 root root        17 Nov 15 09:19 /usr/lib/aarch64-linux-gnu/libcudnn.so.7 -> libcudnn.so.7.0.5
-rw-r--r-- 1 root root 246463352 Nov 15 09:19 /usr/lib/aarch64-linux-gnu/libcudnn.so.7.0.5
lrwxrwxrwx 1 root root        32 Dec 21 04:45 /usr/lib/aarch64-linux-gnu/libcudnn_static.a -> /etc/alternatives/libcudnn_stlib
-rw-r--r-- 1 root root 249280608 Nov 15 09:19 /usr/lib/aarch64-linux-gnu/libcudnn_static_v7.a