Cannot import TF 2.6.0 correctly on Xavier NX

Hello,

I see an error during the import TensorFlow to my project. When I tried it I got:

**ubuntu@NVIDIA-01** : **~** $ python3
Python 3.6.9 (default, Jan 26 2021, 15:33:00)
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
>>> tf.random.normal((5,))
2021-10-04 12:56:47.553686: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1017] ARM64 does not support NUMA - returning NUMA node zero
2021-10-04 12:56:47.915038: W tensorflow/stream_executor/platform/default/dso_loader.cc:65] Could not load dynamic library 'libcutensor.so.1'; dlerror: libcutensor.so.1: cannot open shared object file: No such file or directory
2021-10-04 12:56:48.131866: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1835] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
<tf.Tensor: shape=(5,), dtype=float32, numpy=
array([-0.02318512, -0.41199052, -0.8519837 , -0.7407376 , 1.5291699 ],
dtype=float32)>
>>>

All I do is like in instructions: Installing TensorFlow For Jetson Platform :: NVIDIA Deep Learning Frameworks Documentation

Thanks.

1 Like

Hi,

It looks like a warning message rather than an error.
Do you meet any error or crash when using TensorFlow?

Thanks.

After about 1 day of continual calculations, I got NaN values with my project, but with TF 2.5.0 all was working well. Is really now using GPU during calculation, because of Skipping registering GPU devices?

I think that now all calculations are done on CPU…

Hi,

Just want to confirm the working case (TF 2.5.0).
Is this also applied to XavierNX?

Thanks.

Hello @AastaLLL,

the same issue with TF 2.6.0 on Xavier AGX, looks like after installing the target with the SDK I had to follow the installation instructions in Installing TensorFlow For Jetson Platform :: NVIDIA Deep Learning Frameworks Documentation, with the same error/warning that it cannot find the libcutensor.so.1 library with the same instruction as @markub3327 stated.

Regards.

2 Likes

@AastaLLL All was working before with TF 2.5.0 on Xavier NX.

Hi, both

We can reproduce this error in our environment as well.
Our internal team is working on this.

Will share more information with you later.
Thanks.

3 Likes

@AastaLLL Thanks a lot…

1 Like

Hi,
I would like to report a similar issue on Jetson Nano.
I’ve installed tensorflow 2.6.0 in accordance with the instructions on Nvidia website. Tensorflow was installed properly as no errors or warnings were detected when loading the library.
However, when I run a simple script just to validate the operation I get the following message:

I2021-10-10 15:48:02.405837: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1017] ARM64 does not support NUMA - returning NUMA node zero
2021-10-10 15:48:02.414289: W tensorflow/stream_executor/platform/default/dso_loader.cc:65] Could not load dynamic library ‘libcutensor.so.1’; dlerror: libcutensor.so.1: cannot open shared object file: No such file or directory
2021-10-10 15:48:02.423279: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1835] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at GPU support  |  TensorFlow for how to download and setup the required libraries for your platform.
Skipping registering GPU devices…

I’m affraid my Jetson is running on CPU and not on GPU.
Would be very thankfull to get some help/feedback on this problem.

Thanks!

1 Like

I can also confirm this happening on Jetson Xavier AGX
Similar environment.

1 Like

Just throwing in to say that I’m having the same problem on my Jetson Nano

In case it matters, I’m creating a virtualenv rather than using “sudo pip install”, but otherwise using the same instructions

python3 -m venv mpai_venv
source mpai_venv/bin/activate
python3 -m pip install cython wheel numpy pkgconfig
env H5PY_SETUP_REQUIRES=0 python3 -m pip install h5py scipy
python3 -m pip install --pre --extra-index-url https://developer.download.nvidia.com/compute/redist/jp/v46  tensorflow

Then:

(mpai_venv) chunky@ckjetson:/mnt/sup2rtam/rtam_openai$ python3
Python 3.6.9 (default, Jan 26 2021, 15:33:00)
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow
>>> import tensorflow as tf
>>> tf.test.is_gpu_available()
WARNING:tensorflow:From <stdin>:1: is_gpu_available (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.config.list_physical_devices('GPU')` instead.
2021-10-18 22:47:55.618920: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1017] ARM64 does not support NUMA - returning NUMA node zero
2021-10-18 22:47:55.981493: W tensorflow/stream_executor/platform/default/dso_loader.cc:65] Could not load dynamic library 'libcutensor.so.1'; dlerror: libcutensor.so.1: cannot open shared object file: No such file or directory
2021-10-18 22:47:56.199765: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1835] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
False
>>>

Cheers
Gary

Hi @AastaLLL

Facing exacly same issue at same situtation w/ @igal.kroyter . I didnt run tf yet. But in basic script calls such as list_physical_devices(‘GPU’) or device_lib.list_local_devices(); getting warnings about 1- ARM64 not support NUMA and 2- No such file libcutensor.so at LD_LIBRARY_PATH at cuda-10.2/lib64

Hi

I can confirm the same issue here with fresh install of Jetpack 4.6. Then following the Tensorflow installation instructions from Nvidia website:

2021-10-22 22:14:43.509309: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1017] ARM64 does not support NUMA - returning NUMA node zero
2021-10-22 22:14:43.636463: W tensorflow/stream_executor/platform/default/dso_loader.cc:65] Could not load dynamic library ‘libcutensor.so.1’; dlerror: libcutensor.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda-10.2/lib64:
2021-10-22 22:14:43.711188: W tensorflow/stream_executor/platform/default/dso_loader.cc:65] Could not load dynamic library ‘libcudnn.so.8’; dlerror: libcudnn.so.8: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda-10.2/lib64:
2021-10-22 22:14:43.711373: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1835] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU.
Skipping registering GPU devices…
GPU is NOT AVAILABLE

I dug through the archives of libcutensor: cuTENSOR Download Page | NVIDIA Developer

Unfortunately, all the aarch64 ones say they require CUDA 11, but CUDA 10.2 is all that’s in the devkit… I’m not sure how to proceed. For grins, I tried grabbing the latest and pointing to it using LD_LIBRARY_PATH, but then I just got an error on the next library it tried to load:

2021-10-24 20:01:28.568057: W tensorflow/stream_executor/platform/default/dso_loader.cc:65] Could not load dynamic library ‘libcutensor.so.1’; dlerror: libcublasLt.so.11: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/chunky/Downloads/libcutensor-linux-sbsa-1.3.3.2-archive/lib/11:

It’s now more than two weeks since @AastaLLL said you were able to reproduce this in the lab; do you have any updates on when a patch will be available?

Cheers
Gary