nvprof: incompatible CUDA driver version on TX2

Autonomous Machines Jetson & Embedded Systems Jetson TX2
1

Hi,

I am trying to use nvprof to trace my CUDA application, however, when I use “nvprof ./vectorAdd”, there is an error message:
[Vector addition of 50000 elements]
==14380== NVPROF is profiling process 14380, command: ./vectorAdd
======== Error: incompatible CUDA driver version.

vectorAdd is a sample from NVIDIA_CUDA-8.0_Samples/0_Simple/vectorAdd. If I directly run vectorAdd, the output is:
[Vector addition of 50000 elements]
Copy input data from the host memory to the CUDA device
CUDA kernel launch with 196 blocks of 256 threads
Copy output data from the CUDA device to the host memory
Test PASSED
Done

The version of my CUDA is (nvcc --version):
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2016 NVIDIA Corporation
Built on Mon_Mar_20_17:07:33_CDT_2017
Cuda compilation tools, release 8.0, V8.0.72

The version of nvprof is (nvprof --version):
nvprof: NVIDIA (R) Cuda command line profiler
Copyright (c) 2012 - 2016 NVIDIA Corporation
Release version 8.0.84 (21)

It seems they are all 8.0. Could any one can help?

2

Hi,

Your CUDA toolkit and nvprof are in the different version. (8.0.72 v.s. 8.0.84)
Please install them from JetPack.

Thanks.

3

Thanks for your comments. I reinstall them from JetPack, but the version of CUDA and nvprof didn’t change. Nvprof still doesn’t work. Same error message. My JetPack is JetPack-L4T-3.1-linux-x64.

BTW, nvprof works well in my TK1 board. On this board, the CUDA version is 6.5.45, but the version of nvprof is 6.5.53(21).

4

Hi,

Thanks for your testing and update.

We are checking this issue internally.
Will update information with you later.

5

Hi,

nvprof can run successfully on JetPack3.2 DP.
You can use it as temporal alternative. (We are checking the status of JetPack3.1.)

Thanks.

6

Hi,

We can run nvprof successfully on JetPack 3.1.
Could you check it again?

nvidia@tegra-ubuntu:~/NVIDIA_CUDA-8.0_Samples/0_Simple/vectorAdd$ nvprof ./vectorAdd
[Vector addition of 50000 elements]
==21468== NVPROF is profiling process 21468, command: ./vectorAdd
==21468== Warning: Unified Memory Profiling is not supported on the underlying platform. System requirements for unified memory can be found at: http://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#um-requirements
Copy input data from the host memory to the CUDA device
CUDA kernel launch with 196 blocks of 256 threads
Copy output data from the CUDA device to the host memory
Test PASSED
Done
==21468== Profiling application: ./vectorAdd
==21468== Profiling result:
Time(%)      Time     Calls       Avg       Min       Max  Name
 50.58%  167.58us         2  83.790us  78.859us  88.721us  [CUDA memcpy HtoD]
 26.29%  87.120us         1  87.120us  87.120us  87.120us  vectorAdd(float const *, float const *, float*, int)
 23.13%  76.618us         1  76.618us  76.618us  76.618us  [CUDA memcpy DtoH]

==21468== API calls:
Time(%)      Time     Calls       Avg       Min       Max  Name
 99.42%  246.25ms         3  82.084ms  18.560us  246.21ms  cudaMalloc
  0.37%  928.53us         3  309.51us  227.97us  437.27us  cudaMemcpy
  0.12%  299.20us         3  99.732us  18.080us  231.55us  cudaFree
  0.04%  94.271us        91  1.0350us     608ns  21.279us  cuDeviceGetAttribute
  0.03%  74.719us         1  74.719us  74.719us  74.719us  cudaLaunch
  0.00%  10.688us         4  2.6720us     640ns  7.6160us  cudaSetupArgument
  0.00%  6.9430us         1  6.9430us  6.9430us  6.9430us  cuDeviceTotalMem
  0.00%  5.8240us         3  1.9410us     864ns  2.8160us  cuDeviceGetCount
  0.00%  3.4880us         1  3.4880us  3.4880us  3.4880us  cudaConfigureCall
  0.00%  3.1360us         3  1.0450us     800ns  1.2480us  cuDeviceGet
  0.00%  1.8240us         1  1.8240us  1.8240us  1.8240us  cuDeviceGetName
  0.00%  1.5360us         1  1.5360us  1.5360us  1.5360us  cudaGetLastError

Thanks.

7

Sorry it still doesn’t work. Could you please share your CUDA version and NVPROF version?

8

Hi,

Do you set up the environment completely from JetPack3.1?
It should be compatible if all the package comes from same JetPack version.

nvidia@tegra-ubuntu:~$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2016 NVIDIA Corporation
Built on Mon_Mar_20_17:07:33_CDT_2017
Cuda compilation tools, release 8.0, V8.0.72

nvidia@tegra-ubuntu:~$ nvprof --version
nvprof: NVIDIA (R) Cuda command line profiler
Copyright (c) 2012 - 2016 NVIDIA Corporation
Release version 8.0.84 (21)

To help us better know about your problem, please share the result of deviceQuery.

$ cp -r /usr/local/cuda-8.0/samples .
$ cd samples/1_Utilities/deviceQuery 
$ make 
$ ./deviceQuery

Thanks.

9

Thanks for your reply. The followings are the information of my board.

nvidia@tegra-ubuntu:~$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2016 NVIDIA Corporation
Built on Mon_Mar_20_17:07:33_CDT_2017
Cuda compilation tools, release 8.0, V8.0.72

nvidia@tegra-ubuntu:~$ nvprof --version
nvprof: NVIDIA (R) Cuda command line profiler
Copyright (c) 2012 - 2016 NVIDIA Corporation
Release version 8.0.84 (21)

nvidia@tegra-ubuntu:~/NVIDIA_CUDA-8.0_Samples/1_Utilities/deviceQuery$ ./deviceQuery
./deviceQuery Starting…

CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 1 CUDA Capable device(s)

Device 0: “GP10B”
CUDA Driver Version / Runtime Version 8.5 / 8.0
CUDA Capability Major/Minor version number: 6.2
Total amount of global memory: 7853 MBytes (8234323968 bytes)
( 2) Multiprocessors, (128) CUDA Cores/MP: 256 CUDA Cores
GPU Max Clock rate: 1301 MHz (1.30 GHz)
Memory Clock rate: 13 Mhz
Memory Bus Width: 64-bit
L2 Cache Size: 524288 bytes
Maximum Texture Dimension Size (x,y,z) 1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)
Maximum Layered 1D Texture Size, (num) layers 1D=(32768), 2048 layers
Maximum Layered 2D Texture Size, (num) layers 2D=(32768, 32768), 2048 layers
Total amount of constant memory: 65536 bytes
Total amount of shared memory per block: 49152 bytes
Total number of registers available per block: 32768
Warp size: 32
Maximum number of threads per multiprocessor: 2048
Maximum number of threads per block: 1024
Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
Max dimension size of a grid size (x,y,z): (2147483647, 65535, 65535)
Maximum memory pitch: 2147483647 bytes
Texture alignment: 512 bytes
Concurrent copy and kernel execution: Yes with 1 copy engine(s)
Run time limit on kernels: No
Integrated GPU sharing Host Memory: Yes
Support host page-locked memory mapping: Yes
Alignment requirement for Surfaces: Yes
Device has ECC support: Disabled
Device supports Unified Addressing (UVA): Yes
Device PCI Domain ID / Bus ID / location ID: 0 / 0 / 0
Compute Mode:
< Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 8.5, CUDA Runtime Version = 8.0, NumDevs = 1, Device0 = GP10B
Result = PASS

10

Hi,

It looks like there are something wrong in your environment.
Please re-flash your device and install all the packages from the same JetPack version.

Suppose you should get the same CUDA driver/runtime version like this:
Device 0: “NVIDIA Tegra X2”
CUDA Driver Version / Runtime Version 8.0 / 8.0
CUDA Capability Major/Minor version number: 6.2
…

Thanks.

11

Hi， I have met the same problem ! I set up the environment completely from JetPack3.1,but the result of deviceQuery just the same as yma1, what can I do next to solve this problem?

12

Hi,

Have you also re-flashed your device with JetPack3.1?
If yes, could you retry it or try JetPack3.2 DP?

Thanks.