JetPack 4.3 tensorflow can't find Cudart 10.0 shared library

Robotics & Edge Computing Jetson & Embedded Systems Jetson AGX Xavier
1

I did a new install with the latest April 22, 2020 JetPack 4.3 and installed tensorflow following the instructions here doing the tensorflow<2 install. When I import tensorflow in python3 it says that it can’t find libcudart.so.10.0 in the LD_LIBRARY_PATH and then python3 crashes. The 10.2 cuda libraries came preinstalled and I do not see the older version of cuda in the package listings for Ubuntu. What is the recommended fix for this?

3

Hi,

The new release in April 22, 2020 is JetPack 4.4DP.
Our prebuilt from JetPack4.4 will be available soon.

Thanks.

4

In addition to sdkmanager_1.1.0-6343, I’ve also tried installing using 0.9.12-4180 that fails with downloading OpenCV packages, and 1.0.1-5538 that fails with Visionworks packages. Is there a known “good” version right now that I can install and use with Tensorflow and TensorRT?

5

Hi,

Please use the latest sdkmanager (v1.1.0-6343) and select JetPack4.3 for the installation.
Here is the Tensorflow package for JetPack4.3:

$ sudo pip3 install --pre --extra-index-url https://developer.download.nvidia.com/compute/redist/jp/v43 tensorflow      #v2.1
$ sudo pip3 install --pre --extra-index-url https://developer.download.nvidia.com/compute/redist/jp/v43 ‘tensorflow<2’  #v1.15.0

Thanks.

6

I did that. I linked to those instructions in my original post. That install lead to the Cuda problem I had.

7

Hi,

Just want to clarify first.

Please noticed that the JetPack version released in April 22, 2020 is version 4.4.
But the recommended JetPack is version 4.3 which is released last year.

The default option in our sdkmanager is v4.4.
Please manually changed the choice back to v4.3.

Thanks.

8

The 4.3 jetpack install using the sdkmanager_1.1.0-6343, fails to download libvisionworks: ‘Version 1.6.0.500n for libvisionworks was not found’. The 4.4 jetpack install was successful, I was not able to install Tensorflow 1.15 without tensorflow crashing on startup each time with the problem described in the top post. Tensorflow v2.1 install was successful, but it can’t find the cuda libraries and prints out messages that nothing will be GPU accelerated. The v2.1 install does seem to be working though.

1 Like
9

I have the same problem here. I have jetpack 4.4 and installed tensorflow 2.1 via the link given in the original post. When importing tensorflow in the python3 shell it complains about not finding the cuda 10.0 libraries. Actually I fixed that (or at least stopped tensorflow throwing warnings) by linking the 10.2 libraries to the 10.0 files tensorflow is missing. Like so

sudo ln -s /usr/lib/aarch64-linux-gnu/libcudnn.so.8.0.0 /usr/lib/aarch64-linux-gnu/libcudnn.so.7
sudo ln -s /usr/lib/aarch64-linux-gnu/libcublas.so /usr/lib/aarch64-linux-gnu/libcublas.so.10.0
sudo ln -s /usr/local/cuda-10.2/targets/aarch64-linux/lib/libcusolver.so.10 /usr/local/cuda-10.2/targets/aarch64-linux/lib/libcusolver.so.10.0
sudo ln -s /usr/local/cuda-10.2/targets/aarch64-linux/lib/libcurand.so.10 /usr/local/cuda-10.2/targets/aarch64-linux/lib/libcurand.so.10.0
sudo ln -s /usr/local/cuda-10.2/targets/aarch64-linux/lib/libcufft.so.10 /usr/local/cuda-10.2/targets/aarch64-linux/lib/libcufft.so.10.0
sudo ln -s /usr/local/cuda-10.2/targets/aarch64-linux/lib/libcudart.so /usr/local/cuda-10.2/targets/aarch64-linux/lib/libcudart.so.10.0
sudo ln -s /usr/local/cuda-10.2/targets/aarch64-linux/lib/libcusparse.so.10 /usr/local/cuda-10.2/targets/aarch64-linux/lib/libcusparse.so.10.0
sudo ln -s /usr/lib/aarch64-linux-gnu/libnvinfer.so /usr/lib/aarch64-linux-gnu/libnvinfer.so.6
sudo ln -s /usr/lib/aarch64-linux-gnu/libnvinfer_plugin.so.7 /usr/lib/aarch64-linux-gnu/libnvinfer_plugin.so.6

and adding the library locations to the $LD_LIBRARY_PATH and $PATH

export PATH=/usr/local/cuda/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda/lib64
export PATH=/usr/local/cuda-10.2/bin/:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda-10.2/lib64:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib/aarch64-linux-gnu/

(this worked for me on other systems at least for 10.1 → 10.0). Now tensorflow seems to be able to load all the libraries but it still crashes after creating the cuda device:

...
2020-05-01 13:05:08.011310: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 282 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X1, pci bus id: 0000:00:00.0, compute capability: 5.3)

2020-05-01 13:05:08.042946: F ./tensorflow/core/kernels/random_op_gpu.h:232] Non-OK-status: GpuLaunchKernel(FillPhiloxRandomKernelLaunch<Distribution>, num_blocks, block_size, 0, d.stream(), gen, data, size, dist) status: Internal: invalid device function

I suspect it is due to some operators being not backwards compatable from cuda 10.2 to 10.0.

Long story short: I still would like to make a proper cuda 10.0 install on jetpack 4.4. Is there any way to do so?

10

Hi,

There are several TensorFlow package built for Jetson.
The one supports JetPack4.4 is version 2.1.0+nv20.4.

It should be installed with this commnad:

$ sudo pip3 install --pre --extra-index-url https://developer.download.nvidia.com/compute/redist/jp/v44 tensorflow

Please make sure you are using the correct version first.
Thanks.