Tensorflow not able to detect GPU on NX

[gg22]

My env:

Jetson Xavier NX 8GB.
R34 (release), REVISION: 1.1, GCID: 30414990, BOARD: t186ref, EABI: aarch64

Running jetpack 5.0.

I followed the guide to install CUDA, and then installed tensorflow with:

sudo pip3 install --pre --extra-index-url https://developer.download.nvidia.com/compute/redist/jp/v50 tensorflow==1.15.5+nv22.05

But when I enter python and try out:

>>> import tensorflow as tf
>>> print("Num GPUs Available: ", len(tf.config.experimental.list_physical_devices('GPU'))) 

I get this output:

2022-06-01 17:04:18.242739: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2022-06-01 17:04:18.360009: E tensorflow/stream_executor/cuda/cuda_driver.cc:282] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
2022-06-01 17:04:18.360247: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (ubuntu): /proc/driver/nvidia/version does not exist
Num GPUs Available:  0

What might be the fix for this?
Thank you so much in advance!

[AastaLLL]

Hi,

We test TensorFlow 1.15.5+nv22.05 on a JetPack 5.0.1 DP environment and it works well.

$ python3
Python 3.8.10 (default, Mar 15 2022, 12:22:08)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
2022-06-02 03:37:00.419303: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
>>>

Not sure if this relates to CUDA 11.
Would you mind sharing your setting with us?

$ sudo apt show nvidia-jetpack

Thanks.

[gg22]

Hi! Thanks for the reply. I actually had the same output as you after importing tensorflow in python, and the errors only appeared when I tried to list out GPU devices.

My output for show nvidia-jetpack is:

Package: nvidia-jetpack
Version: 5.0.1-b118
Priority: standard
Section: metapackages
Maintainer: NVIDIA Corporation
Installed-Size: 199 kB
Depends: nvidia-cuda (= 5.0.1-b118), nvidia-opencv (= 5.0.1-b118), nvidia-cudnn8 (= 5.0.1-b118), nvidia-tensorrt (= 5.0.1-b118), nvidia-container (= 5.0.1-b118), nvidia-vpi (= 5.0.1-b118), nvidia-nsight-sys (= 5.0.1-b118), nvidia-l4t-jetson-multimedia-api (>> 34.1-0), nvidia-l4t-jetson-multimedia-api (<< 34.2-0)
Homepage: http://developer.nvidia.com/jetson
Download-Size: 29.4 kB
APT-Sources: https://repo.download.nvidia.com/jetson/t194 r34.1/main arm64 Packages
Description: NVIDIA Jetpack Meta Package

Hope this helps!

[AastaLLL]

Hi,

We test it on an Orin board and we can get the output correctly.
Based on your log, it seems that there is an issue when TensorFlow queries the driver information.

Could you help to check the following output as well?

$ cat /proc/driver/nvidia/version

We are going to double-check this on a XavierNX board and share more information with you.
Thanks.

[gg22]

Thank again for the reply, I think you might have identified part of the issue.

cat /proc/driver/nvidia/version outputs the following:

cat: /proc/driver/nvidia/version: No such file or directory

Would this be a driver corruption issue?

[AastaLLL]

Hi,

Would you mind reflashing the device with JetPack again?

We test the TensorFlow package on a XavierNX board and it can work correctly.
The /proc/driver/nvidia/version also shows normally in our environment.

Thanks.

[gg22]

Reflashing the Jetson + installing CUDA upon flashing with SDK manager fixed it for me, tensorflow is now working with the GPU. Thank you so much for your help!