Bug: nvprof driver version (Solved)

Autonomous Machines Jetson & Embedded Systems Jetson AGX Xavier
1

This example works fine for me:

This is using the built-in nvcc that JetPack installed.

nvidia@xavier:/tmp$ nvcc add.cu -o add_cuda --gpu-architecture compute_72
nvidia@xavier:/tmp$ ./add_cuda 
Max error: 0
nvidia@xavier:/tmp$

However, not everything works:

nvidia@xavier:/tmp$ nvprof ./add_cuda 
==9879== NVPROF is profiling process 9879, command: ./add_cuda
======== Error: incompatible CUDA driver version.

This means that the version of nvprof that’s shipped with JetPack isn’t compatible with the Xavier CUDA driver that ships with JetPack.

2

Hi,

This is a known issue.
Please run nvprof with root authority.

For example:

sudo /usr/local/cuda-10.0/bin/nvprof NVIDIA_CUDA-10.0_Samples/0_Simple/vectorAdd/vectorAdd

We will replace the error log with a hint for execution in the next release.

Thanks.

3

There’s two problems with that:

  1. nvprof is not in the default path for root, and thus I’d need to add the full path
  2. my program actually wants to run with the environment and user ID of a particular user

I can work around both of these in my code, but it would be better if you fixed it, instead of just documenting it.
Specifically, you would want to:

  1. set the suid bit on the nvprof executable, so it always runs as root even when called from a regular user
  2. make sure to call seteuid(getuid()) and setegid(getgid()) before starting the debugged program, so that the debugged program is run using regular user privileges
  3. you’ll need to set up whatever is needed for the debugger in the wrapping nvprof tool, so that it can run with regular privileges once the normal program starts
4

Hi,

The root cause is much more complicated.
There are some security issue of our CUDA toolkit and you will have to run nvprof as root in the near future.

Sorry that we cannot illustrate too much about the real cause.
But we are working on fixing it and will share with you once the root authority is no more required.

Thanks.

5

Is this security issue specific to Jetson/Xavier or does it also apply to other systems, e.g., V100s in a standard HPC system?

The motivating question is that if upgrading to CUDA 10.0 on an HPC system with K80s, V100, P100s, etc., would nvprof have to be run as root.

Please advise.

Thanks.

6

Hi,

This issue is specific to Xavier.
We check nvprof with desktop GPU, it can work correctly without root authority.

Thanks.