AGX Orin Error: cudaErrorUnsupportedPtxVersion: the provided PTX was compiled with an unsupported toolchain

[tnwilly]

I am using AGX Orin and just update OS and all toolkit by SDK manager.
Then I upgrade CUDA toolkit to 12.5 according to the official instructions ([CUDA Toolkit 12.5 Downloads | NVIDIA Developer]).

I see this ERROR in runtime on the AGX Orin, and the test case works on my desktop GPU.

You can reproduce with the code: https://github.com/weimin023/testcuda.git

[AastaLLL]

Hi,

We are not able to reproduce this issue.

Test on AGX Orin with JetPack 6.0 GA with CUDA 12.5 upgrade.
And the test passed:

$ ./unittest
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from CUDAfunction
[ RUN      ] CUDAfunction.test_cuMath_vec
[       OK ] CUDAfunction.test_cuMath_vec (83 ms)
[----------] 1 test from CUDAfunction (83 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (83 ms total)
[  PASSED  ] 1 test.

Thanks.

[tnwilly]

May I know how did you upgrade the CUDA toolkit?

My steps:

1. Flash OS and install Jetpack 6.0 by SDK manager (Jetpack 6.0 REV.1)
2. Download and install CUDA toolkits 12.5 CUDA toolkits 12.5

Thanks!

[AastaLLL]

Hi,

Have you added this to the environment variable?

export PATH=/usr/local/cuda-12/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda-12.5/compat

Thanks.

[tnwilly]

Hi, this is what I use.
Any different?

export PATH=/usr/local/cuda-12.5/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda-12.5/lib64:$LD_LIBRARY_PATH

Even I use your env variable settings I still see the error: C++ exception with description “parallel_for failed: cudaErrorUnsupportedPtxVersion: the provided PTX was compiled with an unsupported toolchain.” thrown in the test body.

My steps for switching CUDA version from 12.2 to 12.5:

1. Switch CUDA toolkits version by this script: switch-cuda
2. Revise ~/.bashrc with:

export PATH=/usr/local/cuda-12.5/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda-12.5/lib64:$LD_LIBRARY_PATH

3. source ~/.bashrc and check CUDA version by nvcc --version

Is there any script or tool for safely switch CUDA version?

[tnwilly]

I got it.
After revise bashrc, reboot is necessary, can not just re-source it.

[AastaLLL]

Hi,

In our case, we don’t reboot it but just resource.
But the whole cmake file needs to be removed and re-created.

Based on the last comment, is everything working now?
Thanks.