Cannot import TF 2.6.0 correctly on Xavier NX

Robotics & Edge Computing Jetson & Embedded Systems Jetson Xavier NX
1

Hello,

I see an error during the import TensorFlow to my project. When I tried it I got:

**ubuntu@NVIDIA-01** : **~** $ python3
Python 3.6.9 (default, Jan 26 2021, 15:33:00)
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
>>> tf.random.normal((5,))
2021-10-04 12:56:47.553686: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1017] ARM64 does not support NUMA - returning NUMA node zero
2021-10-04 12:56:47.915038: W tensorflow/stream_executor/platform/default/dso_loader.cc:65] Could not load dynamic library 'libcutensor.so.1'; dlerror: libcutensor.so.1: cannot open shared object file: No such file or directory
2021-10-04 12:56:48.131866: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1835] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
<tf.Tensor: shape=(5,), dtype=float32, numpy=
array([-0.02318512, -0.41199052, -0.8519837 , -0.7407376 , 1.5291699 ],
dtype=float32)>
>>>

All I do is like in instructions: Installing TensorFlow for Jetson Platform :: NVIDIA Deep Learning Frameworks Documentation

Thanks.

1 Like
3

Hi,

It looks like a warning message rather than an error.
Do you meet any error or crash when using TensorFlow?

Thanks.

4

After about 1 day of continual calculations, I got NaN values with my project, but with TF 2.5.0 all was working well. Is really now using GPU during calculation, because of Skipping registering GPU devices?

I think that now all calculations are done on CPUā€¦

5

Hi,

Just want to confirm the working case (TF 2.5.0).
Is this also applied to XavierNX?

Thanks.

6

Hello @AastaLLL,

the same issue with TF 2.6.0 on Xavier AGX, looks like after installing the target with the SDK I had to follow the installation instructions in Installing TensorFlow For Jetson Platform :: NVIDIA Deep Learning Frameworks Documentation, with the same error/warning that it cannot find the libcutensor.so.1 library with the same instruction as @markub3327 stated.

Regards.

2 Likes
7

@AastaLLL All was working before with TF 2.5.0 on Xavier NX.

8

Hi, both

We can reproduce this error in our environment as well.
Our internal team is working on this.

Will share more information with you later.
Thanks.

3 Likes
10

@AastaLLL Thanks a lotā€¦

1 Like
11

Hi,
I would like to report a similar issue on Jetson Nano.
Iā€™ve installed tensorflow 2.6.0 in accordance with the instructions on Nvidia website. Tensorflow was installed properly as no errors or warnings were detected when loading the library.
However, when I run a simple script just to validate the operation I get the following message:

I2021-10-10 15:48:02.405837: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1017] ARM64 does not support NUMA - returning NUMA node zero
2021-10-10 15:48:02.414289: W tensorflow/stream_executor/platform/default/dso_loader.cc:65] Could not load dynamic library ā€˜libcutensor.so.1ā€™; dlerror: libcutensor.so.1: cannot open shared object file: No such file or directory
2021-10-10 15:48:02.423279: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1835] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at ä½æē”Ø pip å®‰č£… TensorFlow for how to download and setup the required libraries for your platform.
Skipping registering GPU devicesā€¦

Iā€™m affraid my Jetson is running on CPU and not on GPU.
Would be very thankfull to get some help/feedback on this problem.

Thanks!

1 Like
12

I can also confirm this happening on Jetson Xavier AGX
Similar environment.

1 Like
13

Just throwing in to say that Iā€™m having the same problem on my Jetson Nano

In case it matters, Iā€™m creating a virtualenv rather than using ā€œsudo pip installā€, but otherwise using the same instructions

python3 -m venv mpai_venv
source mpai_venv/bin/activate
python3 -m pip install cython wheel numpy pkgconfig
env H5PY_SETUP_REQUIRES=0 python3 -m pip install h5py scipy
python3 -m pip install --pre --extra-index-url https://developer.download.nvidia.com/compute/redist/jp/v46  tensorflow

Then:

(mpai_venv) chunky@ckjetson:/mnt/sup2rtam/rtam_openai$ python3
Python 3.6.9 (default, Jan 26 2021, 15:33:00)
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow
>>> import tensorflow as tf
>>> tf.test.is_gpu_available()
WARNING:tensorflow:From <stdin>:1: is_gpu_available (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.config.list_physical_devices('GPU')` instead.
2021-10-18 22:47:55.618920: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1017] ARM64 does not support NUMA - returning NUMA node zero
2021-10-18 22:47:55.981493: W tensorflow/stream_executor/platform/default/dso_loader.cc:65] Could not load dynamic library 'libcutensor.so.1'; dlerror: libcutensor.so.1: cannot open shared object file: No such file or directory
2021-10-18 22:47:56.199765: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1835] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
False
>>>

Cheers
Gary

14

Hi @AastaLLL

Facing exacly same issue at same situtation w/ @igal.kroyter . I didnt run tf yet. But in basic script calls such as list_physical_devices(ā€˜GPUā€™) or device_lib.list_local_devices(); getting warnings about 1- ARM64 not support NUMA and 2- No such file libcutensor.so at LD_LIBRARY_PATH at cuda-10.2/lib64

15

Hi

I can confirm the same issue here with fresh install of Jetpack 4.6. Then following the Tensorflow installation instructions from Nvidia website:

2021-10-22 22:14:43.509309: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1017] ARM64 does not support NUMA - returning NUMA node zero
2021-10-22 22:14:43.636463: W tensorflow/stream_executor/platform/default/dso_loader.cc:65] Could not load dynamic library ā€˜libcutensor.so.1ā€™; dlerror: libcutensor.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda-10.2/lib64:
2021-10-22 22:14:43.711188: W tensorflow/stream_executor/platform/default/dso_loader.cc:65] Could not load dynamic library ā€˜libcudnn.so.8ā€™; dlerror: libcudnn.so.8: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda-10.2/lib64:
2021-10-22 22:14:43.711373: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1835] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU.
Skipping registering GPU devicesā€¦
GPU is NOT AVAILABLE

16

I dug through the archives of libcutensor: https://developer.nvidia.com/cutensor/downloads

Unfortunately, all the aarch64 ones say they require CUDA 11, but CUDA 10.2 is all thatā€™s in the devkitā€¦ Iā€™m not sure how to proceed. For grins, I tried grabbing the latest and pointing to it using LD_LIBRARY_PATH, but then I just got an error on the next library it tried to load:

2021-10-24 20:01:28.568057: W tensorflow/stream_executor/platform/default/dso_loader.cc:65] Could not load dynamic library ā€˜libcutensor.so.1ā€™; dlerror: libcublasLt.so.11: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/chunky/Downloads/libcutensor-linux-sbsa-1.3.3.2-archive/lib/11:

Itā€™s now more than two weeks since @AastaLLL said you were able to reproduce this in the lab; do you have any updates on when a patch will be available?

Cheers
Gary

17

I have the same problem.
This is so annoying, after spending 2 weeks to get things done, I must tell the customer, cannot use Jetson, this is the LAST thing that need to be fixed. But I cannot do it here.
libcutensor.so.1 not being found, so GPU aceleration is ignored, and now Yolo runs on some demoā€™s 1 frame per 1,5 second. Also tensorflow 2.6 sucks in dnn.

I have TX1, Architecture 5.3

18

I was able to make tensorflow 2 work with the GPU by forcing it to version 2.5.0. Thatā€™s really not an ideal solution, but at least I have GPU support, I guess?

python3 -m pip install --pre --extra-index-url https://developer.download.nvidia.com/compute/redist/jp/v46 tensorflow==2.5.0

Iā€™ve also been incredibly frustrated by the thing where the SDK is still on Ubuntu 18.04, and leans so heavily on python3.6. I see the next version is expected in Q1 of 2022; @AastaLLL: does that mean we can expect it in two months, or would I be wiser to anticipate it six months from now?

Gary

19

Hi @chunkyks, we donā€™t currently have further details to share regarding the schedule for the JetPack 5.0 release other than is whatā€™s included on these pages linked to below, so please stay tuned for further updates.

20

I just re-read your roadmap; apparently the Nano [which I have] isnā€™t getting JetPack 5. I guess that moots the whole question [and takes the nano off the table for this project]

22

Hello, is there any update? I ran across the same issue. Thank you!

1 Like
23

Hi,

Please test our latest v2.6.0+nv21.11 release.
The GPU mode now can run correctly.

$ python3
>>> import tensorflow as tf
>>> tf.random.normal((5,))
2021-11-22 02:48:26.034594: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1017] ARM64 does not support NUMA - returning NUMA node zero
2021-11-22 02:48:26.177009: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1017] ARM64 does not support NUMA - returning NUMA node zero
2021-11-22 02:48:26.177398: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1017] ARM64 does not support NUMA - returning NUMA node zero
2021-11-22 02:48:26.181837: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1017] ARM64 does not support NUMA - returning NUMA node zero
2021-11-22 02:48:26.182441: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1017] ARM64 does not support NUMA - returning NUMA node zero
2021-11-22 02:48:26.182685: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1017] ARM64 does not support NUMA - returning NUMA node zero
2021-11-22 02:48:27.771993: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1017] ARM64 does not support NUMA - returning NUMA node zero
2021-11-22 02:48:27.772281: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1017] ARM64 does not support NUMA - returning NUMA node zero
2021-11-22 02:48:27.772447: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1017] ARM64 does not support NUMA - returning NUMA node zero
2021-11-22 02:48:27.772601: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1510] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 9999 MB memory:  -> device: 0, name: Xavier, pci bus id: 0000:00:00.0, compute capability: 7.2
<tf.Tensor: shape=(5,), dtype=float32, numpy=
array([ 0.3516134 ,  0.85141206,  0.32087272, -0.38743106,  0.3919686 ],
      dtype=float32)>

Thanks.

1 Like