Tensorflow error in NVIDIA TX1

HI,

Am working on NVIDIA TX1, i want to run my deep learning algorithm in my NVIDIA TX1,

i followed the below steps to installed the Tensorflow in NVIDIA TX1,

https://www.youtube.com/watch?v=5l5aeJ6BoVM

but when i tried to run my code am getting the following erros,

2017-10-13 16:22:42.879537: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:857] ARM64 does not support NUMA - returning NUMA node zero
2017-10-13 16:22:42.879688: I tensorflow/core/common_runtime/gpu/gpu_device.cc:955] Found device 0 with properties:
name: NVIDIA Tegra X1
major: 5 minor: 3 memoryClockRate (GHz) 0.9984
pciBusID 0000:00:00.0
Total memory: 3.89GiB
Free memory: 83.39MiB
2017-10-13 16:22:42.879741: I tensorflow/core/common_runtime/gpu/gpu_device.cc:976] DMA: 0
2017-10-13 16:22:42.879772: I tensorflow/core/common_runtime/gpu/gpu_device.cc:986] 0: Y
2017-10-13 16:22:42.879802: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1045] Creating TensorFlow device (/gpu:0) -> (device: 0, name: NVIDIA Tegra X1, pci bus id: 0000:00:00.0)
WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/python/util/tf_should_use.py:175: initialize_all_variables (from tensorflow.python.ops.variables) is deprecated and will be removed after 2017-03-02.
Instructions for updating:
Use tf.global_variables_initializer instead.

Please let me know the proper procedure to install the Tensorflow with GPU support…in NVIDIA TX1

Am new to this… please help me…

Regards,
Niranjan B

Hi,

Could you try MNist sample to check if this issue also occurs?

Thanks.

Hi Niranantany,

Have you tried MNist sample to check if this issue also occurs?
Any result can be shared?

Thanks

I ran into similar issue here. I am using python 3.5 on Jetson TX1 (R28.1).

And I used this wheel file to install tensorflow directly: https://github.com/jetsonhacks/installTensorFlowJetsonTX/blob/master/TX1/tensorflow-1.3.0-cp35-cp35m-linux_aarch64.whl.

I am trying to run this MNIST script https://raw.githubusercontent.com/tensorflow/tensorflow/master/tensorflow/examples/tutorials/mnist/mnist_deep.py

Successfully downloaded train-images-idx3-ubyte.gz 9912422 bytes.
Extracting /tmp/tensorflow/mnist/input_data/train-images-idx3-ubyte.gz
Successfully downloaded train-labels-idx1-ubyte.gz 28881 bytes.
Extracting /tmp/tensorflow/mnist/input_data/train-labels-idx1-ubyte.gz
Successfully downloaded t10k-images-idx3-ubyte.gz 1648877 bytes.
Extracting /tmp/tensorflow/mnist/input_data/t10k-images-idx3-ubyte.gz
Successfully downloaded t10k-labels-idx1-ubyte.gz 4542 bytes.
Extracting /tmp/tensorflow/mnist/input_data/t10k-labels-idx1-ubyte.gz
Saving graph to: /tmp/tmpvzmvs0m5
2017-12-30 06:22:48.342137: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:857] ARM64 does not support NUMA - returning NUMA node zero
2017-12-30 06:22:48.342261: I tensorflow/core/common_runtime/gpu/gpu_device.cc:955] Found device 0 with properties: 
name: NVIDIA Tegra X1
major: 5 minor: 3 memoryClockRate (GHz) 0.9984
pciBusID 0000:00:00.0
Total memory: 3.89GiB
Free memory: 1.19GiB
2017-12-30 06:22:48.342308: I tensorflow/core/common_runtime/gpu/gpu_device.cc:976] DMA: 0 
2017-12-30 06:22:48.342338: I tensorflow/core/common_runtime/gpu/gpu_device.cc:986] 0:   Y 
2017-12-30 06:22:48.342369: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1045] Creating TensorFlow device (/gpu:0) -> (device: 0, name: NVIDIA Tegra X1, pci bus id: 0000:00:00.0)
step 0, training accuracy 0.06

I think we need to fix the issue corresponding to NUMA here:

2017-12-30 06:22:48.342137: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:857] ARM64 does not support NUMA - returning NUMA node zero
2017-12-30 06:22:48.342261: I tensorflow/core/common_runtime/gpu/gpu_device.cc:955] Found device 0 with properties: 
name: NVIDIA Tegra X1

Since in some successfully installed tensorflow, maybe in python 2.7, no NUMA errors exist.

Have you figured out how to fix this?

Hi Jetboyyerong,
What’s the problem ?
Tensorflow says that it will not use numa specs, because this option isn t supported by our ARMv8.
Until we don t have this support on kernel, we will not be able to use it.

Your program works, finds the gpu, but sends you warnings… no problem.

The problem is, the tensorflow obtained from jethacks is not running fast. And when running a test script I mentioned, the whole machine becomes very slow.

While I am working on TX1 and it only has 4GB memory. Do you think this is a memory issue?

Not running fast ? How did you compare ?

What kind of job do you send to TF ?

Be sure to have 10go min swapdisk
Boost card if you can ( i don t remember if you have a *.sh boost)

And don t remember: it s a TX1! A powerfull low power board, but low ram (cpu/gpu)

You are absolutely right, after rebuilding R28.1 kernel and create a 10GB swap file every works out fine. again. Thanks for your discussion!.