Jetson AGX Xavier Cuda Manual installation error with sample application

Robotics & Edge Computing Jetson & Embedded Systems Jetson AGX Xavier
1

We have installed cuda 10-2 packages manually by downloading through SDK manager to host PC, then copied the “cuda-repo-l4t-10-2-local-10.2.89_1.0-1_arm64.deb” to jetson agx which is flashed OS manually (not through SDK Manager) and installed it. While executing the sample application we are seeing the below error:

cudaGetDeviceProperties returned 35
→ CUDA driver version is insufficient for CUDA runtime version
CUDA error at bandwithTest.cu:242 code=35(cudaErrorInsufficientDriver) “cudaSetDevice(currentDevice)”

But when installed OS + Cuda through SDK manager it works fine with no errors. What is the package we are missing ???

2

Did you only install the “cuda-repo-l4t-10-2-local-10.2.89_1.0-1_arm64.deb”? Or did you install other packages?

With the repo installed you can “sudo apt update”, and then install actual CUDA. The repo just tells apt how to find the repository which has the CUDA content.

The following is from an NX, but the list should similar to what you see with command “dpkg -l | egrep -i 'cuda'”:
EDIT: This was from my PC, same idea, but will add the packages I see for NX below (I remote ssh to my NX, must’ve used the wrong terminal).

ii  cuda-command-line-tools-10-2                    10.2.89-1                                           amd64        CUDA command-line tools
ii  cuda-compiler-10-2                              10.2.89-1                                           amd64        CUDA compiler
ii  cuda-cross-aarch64-10-2                         10.2.89-1                                           all          CUDA 10.2 cross-platform meta-package
ii  cuda-cudart-10-2                                10.2.89-1                                           amd64        CUDA Runtime native Libraries
ii  cuda-cudart-cross-aarch64-10-2                  10.2.89-1                                           all          CUDA Runtime cross-aarch64 dev links, headers
ii  cuda-cudart-dev-10-2                            10.2.89-1                                           amd64        CUDA Runtime native dev links, headers
ii  cuda-cufft-10-2                                 10.2.89-1                                           amd64        CUFFT native runtime libraries
ii  cuda-cufft-cross-aarch64-10-2                   10.2.89-1                                           all          CUFFT cross-aarch64 dev links, headers
ii  cuda-cufft-dev-10-2                             10.2.89-1                                           amd64        CUFFT native dev links, headers
ii  cuda-cuobjdump-10-2                             10.2.89-1                                           amd64        CUDA cuobjdump
ii  cuda-cupti-10-2                                 10.2.89-1                                           amd64        CUDA profiling tools runtime libs.
ii  cuda-cupti-cross-aarch64-10-2                   10.2.89-1                                           all          CUDA cross-aarch64 profiling tools interface.
ii  cuda-cupti-dev-10-2                             10.2.89-1                                           amd64        CUDA profiling tools interface.
ii  cuda-curand-10-2                                10.2.89-1                                           amd64        CURAND native runtime libraries
ii  cuda-curand-cross-aarch64-10-2                  10.2.89-1                                           all          CURAND cross-aarch64 dev links, headers
ii  cuda-curand-dev-10-2                            10.2.89-1                                           amd64        CURAND native dev links, headers
ii  cuda-cusolver-10-2                              10.2.89-1                                           amd64        CUDA solver native runtime libraries
ii  cuda-cusolver-cross-aarch64-10-2                10.2.89-1                                           all          CUSOLVER cross-aarch64 dev links, headers
ii  cuda-cusolver-dev-10-2                          10.2.89-1                                           amd64        CUDA solver native dev links, headers
ii  cuda-cusparse-10-2                              10.2.89-1                                           amd64        CUSPARSE native runtime libraries
ii  cuda-cusparse-cross-aarch64-10-2                10.2.89-1                                           all          CUSPARSE cross-aarch64 dev links, headers
ii  cuda-cusparse-dev-10-2                          10.2.89-1                                           amd64        CUSPARSE native dev links, headers
ii  cuda-documentation-10-2                         10.2.89-1                                           amd64        CUDA documentation
ii  cuda-driver-cross-aarch64-10-2                  10.2.89-1                                           all          CUDA Driver cross-aarch64 dev links, headers
ii  cuda-driver-dev-10-2                            10.2.89-1                                           amd64        CUDA Driver native dev stub library
ii  cuda-gdb-10-2                                   10.2.89-1                                           amd64        CUDA-GDB
ii  cuda-libraries-10-2                             10.2.89-1                                           amd64        CUDA Libraries 10.2 meta-package
ii  cuda-libraries-dev-10-2                         10.2.89-1                                           amd64        CUDA Libraries 10.2 development meta-package
ii  cuda-license-10-2                               10.2.89-1                                           amd64        CUDA licenses
ii  cuda-memcheck-10-2                              10.2.89-1                                           amd64        CUDA-MEMCHECK
ii  cuda-misc-headers-10-2                          10.2.89-1                                           amd64        CUDA miscellaneous headers
ii  cuda-misc-headers-cross-aarch64-10-2            10.2.89-1                                           all          CUDA headers for cross-aarch64
ii  cuda-npp-10-2                                   10.2.89-1                                           amd64        NPP native runtime libraries
ii  cuda-npp-cross-aarch64-10-2                     10.2.89-1                                           all          NPP cross-aarch64 dev links, headers
ii  cuda-npp-dev-10-2                               10.2.89-1                                           amd64        NPP native dev links, headers
ii  cuda-nsight-10-2                                10.2.89-1                                           amd64        CUDA nsight
ii  cuda-nsight-compute-10-2                        10.2.89-1                                           amd64        NVIDIA Nsight Compute
ii  cuda-nsight-compute-addon-l4t-10-2              10.2.89-1                                           all          NVIDIA Nsight Compute Addon
ii  cuda-nsight-systems-10-2                        10.2.89-1                                           amd64        NVIDIA Nsight Systems
ii  cuda-nvcc-10-2                                  10.2.89-1                                           amd64        CUDA nvcc
ii  cuda-nvdisasm-10-2                              10.2.89-1                                           amd64        CUDA disassembler
ii  cuda-nvgraph-10-2                               10.2.89-1                                           amd64        NVGRAPH native runtime libraries
ii  cuda-nvgraph-cross-aarch64-10-2                 10.2.89-1                                           all          NVGRAPH cross-aarch64 dev links, headers
ii  cuda-nvgraph-dev-10-2                           10.2.89-1                                           amd64        NVGRAPH native dev links, headers
ii  cuda-nvjpeg-10-2                                10.2.89-1                                           amd64        NVJPEG native runtime libraries
ii  cuda-nvjpeg-dev-10-2                            10.2.89-1                                           amd64        NVJPEG native dev links, headers
ii  cuda-nvml-cross-aarch64-10-2                    10.2.89-1                                           all          NVML cross-aarch64 dev links, headers
ii  cuda-nvml-dev-10-2                              10.2.89-1                                           amd64        NVML native dev links, headers
ii  cuda-nvprof-10-2                                10.2.89-1                                           amd64        CUDA Profiler tools
ii  cuda-nvprune-10-2                               10.2.89-1                                           amd64        CUDA nvprune
ii  cuda-nvrtc-10-2                                 10.2.89-1                                           amd64        NVRTC native runtime libraries
ii  cuda-nvrtc-cross-aarch64-10-2                   10.2.89-1                                           all          NVRTC cross-aarch64 dev links, headers
ii  cuda-nvrtc-dev-10-2                             10.2.89-1                                           amd64        NVRTC native dev links, headers
ii  cuda-nvtx-10-2                                  10.2.89-1                                           amd64        NVIDIA Tools Extension
ii  cuda-nvvp-10-2                                  10.2.89-1                                           amd64        CUDA nvvp
ii  cuda-repo-cross-aarch64-10-2-local-10.2.89      1.0-1                                               all          cuda repository configuration files
ii  cuda-repo-ubuntu1804-10-2-local-10.2.89-440.40  1.0-1                                               amd64        cuda repository configuration files
ii  cuda-samples-10-2                               10.2.89-1                                           amd64        CUDA example applications
ii  cuda-sanitizer-api-10-2                         10.2.89-1                                           amd64        CUDA Sanitizer API
ii  cuda-toolkit-10-2                               10.2.89-1                                           amd64        CUDA Toolkit 10.2 meta-package
ii  cuda-tools-10-2                                 10.2.89-1                                           amd64        CUDA Tools meta-package
ii  cuda-visual-tools-10-2                          10.2.89-1                                           amd64        CUDA visual tools
ii  libcudart9.1:amd64                              9.1.85-3ubuntu1                                     amd64        NVIDIA CUDA Runtime Library
ii  libnvrtc9.1:amd64                               9.1.85-3ubuntu1                                     amd64        CUDA Runtime Compilation (NVIDIA NVRTC Library)
ii  nvidia-cuda-doc                                 9.1.85-3ubuntu1                                     all          NVIDIA CUDA and OpenCL documentation
ii  nvidia-cuda-gdb                                 9.1.85-3ubuntu1                                     amd64        NVIDIA CUDA Debugger (GDB)

ARM64 (there are other packages you can get, this is just what was on the NX at the moment):

ii  cuda-compiler-10-2                         10.2.89-1                                        arm64        CUDA compiler
ii  cuda-core-10-2                             10.2.89-1                                        arm64        CUDA core tools
ii  cuda-cuobjdump-10-2                        10.2.89-1                                        arm64        CUDA cuobjdump
ii  cuda-license-10-2                          10.2.89-1                                        arm64        CUDA licenses
ii  cuda-misc-headers-10-2                     10.2.89-1                                        arm64        CUDA miscellaneous headers
ii  cuda-nvcc-10-2                             10.2.89-1                                        arm64        CUDA nvcc
ii  cuda-nvprune-10-2                          10.2.89-1                                        arm64        CUDA nvprune
ii  nvidia-l4t-cuda                            32.4.3-20200625213407                            arm64        NVIDIA CUDA Package

Those were installed when telling JetPack/SDKM to install CUDA to the Jetson. If you manually installed, then you would also have to consider using apt-get to install those other packages as well. The advantage of JetPack/SDKM being that it knows about all of those packages and just installs them.

NOTE: If you have those, then your CUDA may just not know where to find the CUDA content. You’d need your “$PATH” environment variable to include the right “bin/” directory at “/usr/local/cuda/bin/”.