Tensorflow error in NVIDIA TX1

Autonomous Machines Jetson & Embedded Systems Jetson TX1
#1

HI,

Am working on NVIDIA TX1, i want to run my deep learning algorithm in my NVIDIA TX1,

i followed the below steps to installed the Tensorflow in NVIDIA TX1,

https://www.youtube.com/watch?v=5l5aeJ6BoVM

but when i tried to run my code am getting the following erros,

2017-10-13 16:22:42.879537: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:857] ARM64 does not support NUMA - returning NUMA node zero
2017-10-13 16:22:42.879688: I tensorflow/core/common_runtime/gpu/gpu_device.cc:955] Found device 0 with properties:
name: NVIDIA Tegra X1
major: 5 minor: 3 memoryClockRate (GHz) 0.9984
pciBusID 0000:00:00.0
Total memory: 3.89GiB
Free memory: 83.39MiB
2017-10-13 16:22:42.879741: I tensorflow/core/common_runtime/gpu/gpu_device.cc:976] DMA: 0
2017-10-13 16:22:42.879772: I tensorflow/core/common_runtime/gpu/gpu_device.cc:986] 0: Y
2017-10-13 16:22:42.879802: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1045] Creating TensorFlow device (/gpu:0) -> (device: 0, name: NVIDIA Tegra X1, pci bus id: 0000:00:00.0)
WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/python/util/tf_should_use.py:175: initialize_all_variables (from tensorflow.python.ops.variables) is deprecated and will be removed after 2017-03-02.
Instructions for updating:
Use tf.global_variables_initializer instead.

Please let me know the proper procedure to install the Tensorflow with GPU support…in NVIDIA TX1

Am new to this… please help me…

Regards,
Niranjan B

#2

Hi,

Could you try MNist sample to check if this issue also occurs?

Thanks.

#3

Hi Niranantany,

Have you tried MNist sample to check if this issue also occurs?
Any result can be shared?

Thanks

#4

I ran into similar issue here. I am using python 3.5 on Jetson TX1 (R28.1).

And I used this wheel file to install tensorflow directly: https://github.com/jetsonhacks/installTensorFlowJetsonTX/blob/master/TX1/tensorflow-1.3.0-cp35-cp35m-linux_aarch64.whl.

I am trying to run this MNIST script https://raw.githubusercontent.com/tensorflow/tensorflow/master/tensorflow/examples/tutorials/mnist/mnist_deep.py

Successfully downloaded train-images-idx3-ubyte.gz 9912422 bytes.
Extracting /tmp/tensorflow/mnist/input_data/train-images-idx3-ubyte.gz
Successfully downloaded train-labels-idx1-ubyte.gz 28881 bytes.
Extracting /tmp/tensorflow/mnist/input_data/train-labels-idx1-ubyte.gz
Successfully downloaded t10k-images-idx3-ubyte.gz 1648877 bytes.
Extracting /tmp/tensorflow/mnist/input_data/t10k-images-idx3-ubyte.gz
Successfully downloaded t10k-labels-idx1-ubyte.gz 4542 bytes.
Extracting /tmp/tensorflow/mnist/input_data/t10k-labels-idx1-ubyte.gz
Saving graph to: /tmp/tmpvzmvs0m5
2017-12-30 06:22:48.342137: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:857] ARM64 does not support NUMA - returning NUMA node zero
2017-12-30 06:22:48.342261: I tensorflow/core/common_runtime/gpu/gpu_device.cc:955] Found device 0 with properties: 
name: NVIDIA Tegra X1
major: 5 minor: 3 memoryClockRate (GHz) 0.9984
pciBusID 0000:00:00.0
Total memory: 3.89GiB
Free memory: 1.19GiB
2017-12-30 06:22:48.342308: I tensorflow/core/common_runtime/gpu/gpu_device.cc:976] DMA: 0 
2017-12-30 06:22:48.342338: I tensorflow/core/common_runtime/gpu/gpu_device.cc:986] 0:   Y 
2017-12-30 06:22:48.342369: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1045] Creating TensorFlow device (/gpu:0) -> (device: 0, name: NVIDIA Tegra X1, pci bus id: 0000:00:00.0)
step 0, training accuracy 0.06

I think we need to fix the issue corresponding to NUMA here:

2017-12-30 06:22:48.342137: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:857] ARM64 does not support NUMA - returning NUMA node zero
2017-12-30 06:22:48.342261: I tensorflow/core/common_runtime/gpu/gpu_device.cc:955] Found device 0 with properties: 
name: NVIDIA Tegra X1

Since in some successfully installed tensorflow, maybe in python 2.7, no NUMA errors exist.

Have you figured out how to fix this?

#5

Hi Jetboyyerong,
What’s the problem ?
Tensorflow says that it will not use numa specs, because this option isn t supported by our ARMv8.
Until we don t have this support on kernel, we will not be able to use it.

Your program works, finds the gpu, but sends you warnings… no problem.

#6

The problem is, the tensorflow obtained from jethacks is not running fast. And when running a test script I mentioned, the whole machine becomes very slow.

While I am working on TX1 and it only has 4GB memory. Do you think this is a memory issue?

#7

Not running fast ? How did you compare ?

What kind of job do you send to TF ?

Be sure to have 10go min swapdisk
Boost card if you can ( i don t remember if you have a *.sh boost)

And don t remember: it s a TX1! A powerfull low power board, but low ram (cpu/gpu)

#8

You are absolutely right, after rebuilding R28.1 kernel and create a 10GB swap file every works out fine. again. Thanks for your discussion!.