Cuda 13+ needed for TensorRT-Edge LLM on Drive AGX Thor?

Autonomous Vehicles DRIVE AGX Thor DRIVE AGX Thor General
1

DRIVE OS Version: 7.0.3

Issue Description: Hello, I can’t build the TensorRT-Edge LLM on my Drive AGX Thor Development Kit which has the officially available DriveOS 7.0.3 installed. This only seems to come with Cuda 12.8 Toolkit. When preparing build configuration on the Thor I get the following logs:

cmake .. -DCMAKE_BUILD_TYPE=Release -DTRT_PACKAGE_DIR=/usr -DCMAKE_TOOLCHAIN_FILE=cmake/aarch64_linux_toolchain.cmake -DEMBEDDED_TARGET=auto-thor
– Configurable variable CUDA_CTK_VERSION set to 13.2
– Using CUDA toolkit dir: /usr/local/cuda/targets/aarch64-linux
– Configurable variable CUDA_CTK_VERSION set to 13.2
– Using CUDA toolkit dir: /usr/local/cuda/targets/aarch64-linux
– The CXX compiler identification is GNU 13.2.0
– The CUDA compiler identification is NVIDIA 12.8.93
– Detecting CXX compiler ABI info
– Detecting CXX compiler ABI info - done
– Check for working CXX compiler: /usr/bin/aarch64-linux-gnu-g++ - skipped
– Detecting CXX compile features
– Detecting CXX compile features - done
– Configurable variable CUDA_CTK_VERSION set to 13.2
– Configurable variable CUDA_DIR set to /usr/local/cuda/targets/aarch64-linux
– NVTX profiling DISABLED (use -DENABLE_NVTX_PROFILING=ON to enable)
– FMHA Kernels: Excluding SM architectures: EXCLUDE_SM_80;EXCLUDE_SM_86;EXCLUDE_SM_87;EXCLUDE_SM_89;EXCLUDE_SM_100;EXCLUDE_SM_120;EXCLUDE_SM_121
– Configuring done (0.9s)
– Generating done (0.1s)

And when actually building it will fail with the following logs:

[ 92%] Building CUDA object cpp/CMakeFiles/edgellmCore.dir/kernels/int4GroupwiseGemmKernels/int4WoQGemvCuda.cu.o
nvcc fatal : Unsupported gpu architecture ‘compute_110’
make[2]: *** [cpp/CMakeFiles/edgellmCore.dir/build.make:5859: cpp/CMakeFiles/edgellmCore.dir/kernels/contextAttentionKernels/utilKernels.cu.o] Error 1
make[2]: *** Waiting for unfinished jobs…
[ 92%] Building CUDA object cpp/CMakeFiles/edgellmCore.dir/kernels/int4GroupwiseGemmKernels/int4WoqGemmCuda.cu.o
nvcc fatal : Unsupported gpu architecture ‘compute_110’
make[2]: *** [cpp/CMakeFiles/edgellmCore.dir/build.make:5874: cpp/CMakeFiles/edgellmCore.dir/kernels/embeddingKernels/embeddingKernels.cu.o] Error 1
nvcc fatal : Unsupported gpu architecture ‘compute_110’
make[2]: *** [cpp/CMakeFiles/edgellmCore.dir/build.make:5889: cpp/CMakeFiles/edgellmCore.dir/kernels/int4GroupwiseGemmKernels/int4WoQGemvCuda.cu.o] Error 1
nvcc fatal : Unsupported gpu architecture ‘compute_110’
make[2]: *** [cpp/CMakeFiles/edgellmCore.dir/build.make:5904: cpp/CMakeFiles/edgellmCore.dir/kernels/int4GroupwiseGemmKernels/int4WoqGemmCuda.cu.o] Error 1
[ 92%] Building CUDA object cpp/CMakeFiles/edgellmCore.dir/kernels/kvCacheUtilKernels/kvCacheUtilsKernels.cu.o
nvcc fatal : Unsupported gpu architecture ‘compute_110’
make[2]: *** [cpp/CMakeFiles/edgellmCore.dir/build.make:5919: cpp/CMakeFiles/edgellmCore.dir/kernels/kvCacheUtilKernels/kvCacheUtilsKernels.cu.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:199: cpp/CMakeFiles/edgellmKernels.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs…
make[1]: *** [CMakeFiles/Makefile2:225: cpp/CMakeFiles/edgellmCore.dir/all] Error 2
[ 93%] Linking CXX static library libedgellmTokenizer.a
[ 93%] Built target edgellmTokenizer
make: *** [Makefile:91: all] Error 2

How can I fix this issue and get the TensorRT-Edge LLM running? Thank you very much for your support.

4

Dear @adas.developer ,
Please use DRIVE OS LLM sdk(DriveOS LLM SDK: TensorRT’s Large Language Model Inference Framework for Auto Platforms — NVIDIA DriveOS 7.0.3 Linux SDK Developer Guide) on DRIVE OS 7.0.3 release.

5

Thanks fo your answer. I need support for Qwen3VL models, so I need to use TensorRT Edge-LLM. Can i upgrade my DriveOS or at least CUDA version? There was a version 7.2.4 mentioned being supported in the release notes.

6

DRIVE OS Edge LLM is renamed as TensorRT edge LLM in next DRIVE OS release. For latest DevZone release is DRIVE OS 7.0.3. for that, you need to use DRIVE OS Edge LLM.
CUDA can’t be upgraded alone and is tied to DRIVE OS.