NUMA Error running Tensorflow on Jetson Tx2

Robotics & Edge Computing Jetson & Embedded Systems Jetson TX2
1

Hey Developers,

i just build tensorflow on my jetson tx2 and ran the “Hello Tensorflow” Test and got following output.
It says something about “NUMA” and “numa_nodes” which it searches somewhere in “bus/pci/devices”. How can i change this for the tegra to work?

>>> print tf.Session().run(hello)
2017-12-13 10:00:03.781946: E tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:878] could not open file to read NUMA node: /sys/bus/pci/devices/0000:00:00.0/numa_node
Your kernel may have been built without NUMA support.
2017-12-13 10:00:03.782058: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 0 with properties: 
name: NVIDIA Tegra X2 major: 6 minor: 2 memoryClockRate(GHz): 1.3005
pciBusID: 0000:00:00.0
totalMemory: 7.67GiB freeMemory: 157.59MiB
2017-12-13 10:00:03.782100: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: NVIDIA Tegra X2, pci bus id: 0000:00:00.0, compute capability: 6.2)
2017-12-13 10:00:03.782145: I tensorflow/core/common_runtime/gpu/gpu_device.cc:732] Could not identify NUMA node of /job:localhost/replica:0/task:0/device:GPU:0, defaulting to 0.  Your kernel may not have been built with NUMA support.
Hello,TensorFlow!
3

Don’t swear on the forum.

The NUMA information is a non-harmful warning.
Tensorflow can run correctly with the warning.

Please remember to remove the inappropriate word in comment #2.
Thanks.

3 Likes
4

after running
self.gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=gpu_memory_fraction)

the console print → does not support NUMA - returning NUMA node zero
e: NVIDIA Tegra X2 major: 6 minor: 2 memoryClockRate(GHz): 1.02
2019-09-30 19:29:57.556148: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-09-30 19:29:57.556624: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:972] ARM64 does not support NUMA - returning NUMA node zero
2019-09-30 19:29:57.556947: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:972] ARM64 does not support NUMA - returning NUMA node zero
2019-09-30 19:29:57.557038: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2019-09-30 19:29:57.557302: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2019-09-30 19:29:59.313177: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-30 19:29:59.313285: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187] 0
2019-09-30 19:29:59.313324: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0: N
2019-09-30 19:29:59.314187: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:972] ARM64 does not support NUMA - returning NUMA node zero
2019-09-30 19:29:59.314843: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:972] ARM64 does not support NUMA - returning NUMA node zero

so the ARM has no NUMA node just a warning, does my code run still pass and ok?

5

Hi,

YES. It should be a harmless warning.
Thanks.

1 Like
6

sorry for pumping this question again but is there anyway I can get rid/turn off that message in the log? Thank you, I’m using WSL2.

1 Like