cuDNN & TensorRT on SDK 6.0.6

servanti · August 28, 2023, 8:50pm

Please provide the following info (tick the boxes after creating this topic):
Software Version
DRIVE OS 6.0.6
DRIVE OS 6.0.5
DRIVE OS 6.0.4 (rev. 1)
DRIVE OS 6.0.4 SDK
other

Target Operating System
Linux
QNX
other

Hardware Platform
DRIVE AGX Orin Developer Kit (940-63710-0010-300)
DRIVE AGX Orin Developer Kit (940-63710-0010-200)
DRIVE AGX Orin Developer Kit (940-63710-0010-100)
DRIVE AGX Orin Developer Kit (940-63710-0010-D00)
DRIVE AGX Orin Developer Kit (940-63710-0010-C00)
DRIVE AGX Orin Developer Kit (not sure its number)
other

SDK Manager Version
1.9.3.10904
other

Host Machine Version
native Ubuntu Linux 20.04 Host installed with SDK Manager
native Ubuntu Linux 20.04 Host installed with DRIVE OS Docker Containers
native Ubuntu Linux 18.04 Host installed with DRIVE OS Docker Containers
other

Hello

After looking at this thread:

I do not see any of the cuDNN and TensorRT headers in /usr/include/aarch64-linux-gnu on the Target , the are installed on the host after I installed the SDK via sdkmanger but I don’t even see CUDA-X AI as a target component. is cuDNN and TensorRT installed by sdkmanger?

SivaRamaKrishnaNV · August 29, 2023, 2:25am

Dear @servanti,
Yes. Sdkmanager installs CUDA, cudnn, TRT on target.

servanti · August 29, 2023, 5:04pm

Dear SivaRamaKrishnaNV
So is it normal that CUDA-X AI is not listed as one of the target components in the sdkmanager? The libs (.so) are installed but no headers are installed on the target. I did a find and it did not find anything. Do we need to copy the headers from the host system? which is fine if we need to do that but, I never had to do that before and I tried to go through the release notes and documentation but did not see anything.

SivaRamaKrishnaNV · August 29, 2023, 5:36pm

Dear @servanti,
The TRT samples are not part of target file system. We recommend to cross compile the samples on host/docker and run on target. Please check Build and Run Sample Applications for DRIVE OS 6.x Linux | NVIDIA Docs

servanti · August 29, 2023, 6:07pm

Dear SivaRamaKrishnaNV
It is not just the samples, we build opencv, torch etc cuda , cudnn support since sometimes it is difficult to find pre built packages with the compatible versions of cuda. We do this on Jetson Orin all the time, I suppose I can just install those wheels and debs to avoid cross compiling all these different packages. So basically we can no longer compile any TRT applications directly on this target and it all has to be cross compiled on a host or a docker?

servanti · August 29, 2023, 10:18pm

Dear SivaRamaKrishnaNV

Will it be possible to install cudnn-local-repo-ubuntu2004-8.6.0.163_1.0-1_arm64.deb
On the Orin Drive to get full cuDNN development , but still I am not able to find a compatible TensorRT installer , I suppose I can use the debs from Jetpack where I install on Jetson Orin. Will that work?

SivaRamaKrishnaNV · August 30, 2023, 1:40am

Dear @servanti ,
No. You can copy required headers/samples from host.

servanti · September 5, 2023, 5:04pm

Dear SivaRamaKrishnaNV
Thank you for the reply, I managed to get Pytorch and OpenCV directly compiled on the target after copying the appropriate headers/libs from host.
Python and tensroRT was a different story. There was no tensorrt.so in the dist package folder of python3.8 and I can not copy the host lib as it is x86. Or at least I could not find the aarch64 lib, but we were lucky to have a Jetson Orin that has everything installed on the target via JetPack and I copied those libs and it worked.
At lease we can , in a rush, just build and run on the target now after some “set ups” post flashing

SivaRamaKrishnaNV · September 6, 2023, 12:59pm

Dear @servanti,
Thank you sharing your experience.
But you may encounter few issues if you use libs from Jetson release.
Could you share the steps you used to compile PyTorch/OpenCV on target to help others in community?

servanti · September 7, 2023, 6:45pm

Dear SivaRamaKrishnaNV

I could not find aarch64 tensorrt.so for python so I just figured I’ll try the one on my Jetson Orin. So far it works, if there is an “official” way to get that lib I’d be happy to try it,

Now as far as compiling cuDNN and TensorRT code on the target. These are the steps that I followed.

Tar all libcudnn* and libnvinfer* in target lib/ aarch64-linux-gnu and /etc/alternative in and all cudnn.* and nvinfer* include/ aarch64-linux-gnu on the host. These are the libs and headers. The idea is to untar these on the target in /usr/include/aarch64-linux-gnu and /usr/lib/aarch64-linux-gnu and /etc/alternatives. Do not copy or scp since tar preserves all the symbolic links.
Once this is done then the target has all the headers and libs for compiling cuDNN and TensorRT apps.

OpenCV on Target
Download openCV and openCV-Contrib source, I downloaded version 4.5.4 Release OpenCV 4.5.4 · opencv/opencv · GitHub
Install dependencies: sudo apt install -y cmake libavcodec-dev libavformat-dev libavutil-dev libeigen3-dev libglew-dev libgtk2.0-dev libgtk-3-dev libjpeg-dev libpng-dev libpostproc-dev libswscale-dev libtbb-dev libtiff5-dev libv4l-dev libxvidcore-dev libx264-dev qt5-default zlib1g-dev python-dev python-numpy python-py python-pytest python3-dev python3-py python3-pytest libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libopenblas-dev
Here is part of a shell script to compile with cuDNN and CUDA on Orin drive, you may need to change the install directory

OpenCV defines

OPENCV_VERSION=4.5.4
ARCH_BIN=8.7
CUDNN_VERSION=8.6
NUM_JOBS=$(nproc)
OPENCV_DOWNLOAD_DIR=$HOME/opencv
OPENCV_DIR=opencv-$OPENCV_VERSION
OPENCV_CONTRIB_DIR=opencv_contrib-$OPENCV_VERSION
OPENCV_BUILD_DIR=$OPENCV_DIR/build
OPENCV_GITHUB_REPO=OpenCV · GitHub
OPENCV_REPO=$OPENCV_GITHUB_REPO/opencv/archive/refs/tags/$OPENCV_VERSION.tar.gz
OPENCV_CONTRIB_REPO=$OPENCV_GITHUB_REPO/opencv_contrib/archive/refs/tags/$OPENCV_VERSION.tar.gz
OPENCV_INSTALL_LOCATION=/usr

cd $OPENCV_DOWNLOAD_DIR/$OPENCV_BUILD_DIR
cmake -D CMAKE_BUILD_TYPE=RELEASE
-D CMAKE_INSTALL_PREFIX=${OPENCV_INSTALL_LOCATION}
-D WITH_CUDA=ON
-D WITH_CUDNN=ON
-D CUDNN_VERSION=${CUDNN_VERSION}
-D CUDA_ARCH_BIN=${ARCH_BIN}
-D CUDA_ARCH_PTX=“”
-D ENABLE_FAST_MATH=ON
-D CUDA_FAST_MATH=ON
-D WITH_CUBLAS=ON
-D WITH_LIBV4L=ON
-D WITH_V4L=ON
-D WITH_GSTREAMER=ON
-D WITH_GSTREAMER_0_10=OFF
-D WITH_QT=OFF
-D WITH_GTK=ON
-D WITH_GTK_2_X=ON
-D WITH_OPENGL=ON
-D BUILD_opencv_python2=ON
-D BUILD_opencv_python3=ON
-D BUILD_TESTS=OFF
-D BUILD_PERF_TESTS=OFF
-D OPENCV_GENERATE_PKGCONFIG=YES
-D PYTHON_EXECUTABLE=$(which python3)
-D OPENCV_PYTHON3_INSTALL_PATH=$(python3 -c “from distutils.sysconfig import get_python_lib; print(get_python_lib())”)
-D OPENCV_EXTRA_MODULES_PATH=…/…/${OPENCV_CONTRIB_DIR}/modules
-D CPACK_PACKAGE_VERSION=${OPENCV_VERSION}
-D EXTRAMODULES_VCSVERSION=${OPENCV_VERSION}
-D OPENCV_VCSVERSION=${OPENCV_VERSION}
…/
make -j${NUM_JOBS}
make install

Test OpenCV in python:
python3 -c “import cv2; print(‘OpenCV version:’, str(cv2.version)); print(cv2.getBuildInformation())”

Pytorch:

Here is a script that builds the latest pytorch but it can be changed to download whatever version that you need.
Also , I think the cmake version “out of the box” on the target 3.16 is not sufficient for pytorch compile so I install cmake 3.27 from source:
Installing | CMake

The option TORCH_CXX_FLAGS=D_GLIBCXX_USE_CXX11_ABI=1 is something we need in regards to compile and link with ROS (Long story ) , you can omit that

git clone https://github.com/pytorch/pytorch.git && \
cd pytorch && \
pip3 install pyyaml==3.13 && \
pip3 install -r requirements.txt && \  
sudo apt update
sudo apt-get install apt-utils
sudo apt-get install python3-setuptools
USE_OPENCV=1 \
BUILD_TORCH=ON \
CMAKE_PREFIX_PATH="/usr/bin/" \
LD_LIBRARY_PATH=/usr/local/cuda-11.4/lib64:/usr/local/lib:$LD_LIBRARY_PATH \
CUDA_BIN_PATH=/usr/local/cuda-11.4/bin \
CUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda-11.4/ \
CUDNN_LIB_DIR=/usr/lib/aarch64-linux-gnu \
CUDNN_INCLUDE_DIR=/usr/include/aarch64-linux-gnu \
CUDNN_LIBRAY=/usr/lib/aarch64-linux-gnu/libcudnn.so \
CUDA_HOST_COMPILER=cc \
USE_CUDA=1 \
USE_CUDNN=1 \
USE_NNPACK=1 \
TORCH_CXX_FLAGS=D_GLIBCXX_USE_CXX11_ABI=1 \
CC=cc \
CXX=c++ \
TORCH_CUDA_ARCH_LIST="8.7" \
TORCH_NVCC_FLAGS="-Xfatbin -compress-all" \
python3 setup.py bdist_wheel

Install the Python wheel (includes LibTorch)

pip3 install dist/*.whl

Clean up resources

rm -fr pytorch

I also had to set up swap file on target