No kernel to profile when using nsight compute

Developer Tools Nsight Compute
1

Hi, I am using jetson orin nano kit and trying to use ncu to profile a hello world kernel, but got

ncu -o cudaHelloWorld -k regex:hello_world_cuda --set=full python3 ./hello_
world_cuda.py
==PROF== Connected to process 11093 (/usr/bin/python3.8)
==ERROR== Failed to prepare kernel for profiling

==ERROR== Unknown Error on device 0.
==ERROR== Failed to profile “hello_world_cuda()” in process 11093
==PROF== Trying to shutdown target application
==ERROR== The application returned an error code (9).
==ERROR== An error occurred while trying to profile.
==WARNING== No kernels were profiled.
==WARNING== Profiling kernels launched by child processes requires the --target-processes all option.

ncu version
$ ncu --version
NVIDIA (R) Nsight Compute Command Line Profiler
Copyright (c) 2018-2023 NVIDIA Corporation
Version 2023.2.0.0 (build 32895467) (public-release)

Any help would be greatly appreciated!

Best,

2

some more info, there are no nsight compute or nsight system come with the jetpack, I install nsight system and compute myself, the nsys works fine, but ncu is not…

3

Can you share the output of “python3 ./hello_world_cuda.py” without ncu and then try this command:

“ncu -o cudaHelloWorld --target-processes all python3 ./hello_world_cuda.py”

4

Thank you for the reply!
the output for this program is
$ python3 ./hello_world_cuda.py
Hello world from GPU.
Just say hello world from thread 1, and then sync all threads.

__global__ void hello_world_cuda() {
    if (blockIdx.x==0 && threadIdx.x==0) printf("Hello world from GPU.\n");
    return;
}

int hello_world_cuda_host() {
    hello_world_cuda<<<16, 32>>>();
    cudaDeviceSynchronize();
    return 0;
}

and when using the cmd you provide, I got the same error log…
ncu -o cudaHelloWorld --target-processes all python3 ./hello_world_cuda.py
==PROF== Connected to process 4812 (/usr/bin/python3.8)
==ERROR== Failed to prepare kernel for profiling

==ERROR== Unknown Error on device 0.
==ERROR== Failed to profile “hello_world_cuda()” in process 4812
==PROF== Trying to shutdown target application
==ERROR== The application returned an error code (9).
==ERROR== An error occurred while trying to profile.
==WARNING== No kernels were profiled.

5

Also, I could not open ncu-ui, it says:
$ ncu-ui
/usr/local/NVIDIA-Nsight-Compute/ncu-ui: 9: /usr/local/NVIDIA-Nsight-Compute-2023.2/host/linux-desktop-t210-a64/ncu-ui: not found

Is that indicating I had some issue installing it?

6

Hi Jmarusarz,

I heard that the nsys and ncu on jetson orin nano should come with the jetpack — do you know how I could access those two tools with jetpack? Thank you so much for your help!

Best

7

What version of jetpack do you have? It’s not clear that there is an installation issue. What is the output of “which ncu”? Whatever location the “ncu” binary is should may also have the ui. Can you do an “ls” of whatever directory “ncu” is located in?

8

Hi Jmarusarz,

Thank you for your help and sry for the late reply.

I think I am using jetpack 5.1.1 cause I downloaded the image for jetson orin nano from JetPack SDK | NVIDIA Developer and here are the outputs:

chengzhe-xu@chengzhexu-desktop:/usr/local/NVIDIA-Nsight-Compute$ which ncu
/usr/local/NVIDIA-Nsight-Compute/ncu
chengzhe-xu@chengzhexu-desktop:/usr/local/NVIDIA-Nsight-Compute$ ll
total 72
drwxr-xr-x  7 root root  4096 Jul 16 22:20 ./
drwxr-xr-x 12 root root  4096 Jul 16 22:20 ../
drwxrwxrwx 14 root root  4096 May 24 19:29 docs/
-rw-rw-rw-  1 root root 33024 May 24 19:29 EULA.txt
drwxrwxrwx  6 root root  4096 May 24 19:29 extras/
drwxrwxrwx  4 root root  4096 May 24 19:29 host/
-rwxrwxrwx  1 root root   410 May 24 19:29 ncu*
-rwxrwxrwx  1 root root   312 May 24 19:29 ncu-ui*
drwxrwxrwx  2 root root  4096 May 24 19:29 sections/
drwxrwxrwx  6 root root  4096 May 24 19:29 target/
chengzhe-xu@chengzhexu-desktop:/usr/local/NVIDIA-Nsight-Compute$ ./ncu-ui 
./ncu-ui: 9: /usr/local/NVIDIA-Nsight-Compute-2023.2/host/linux-desktop-t210-a64/ncu-ui: not found
chengzhe-xu@chengzhexu-desktop:/usr/local/NVIDIA-Nsight-Compute$

Best,
Chengzhe

How to install and use nsight compute with jetpack 5.0?
9

It looks like this is coming from the Desktop version of Nsight Compute 2023. I’m not sure if that would have the Jetson GUI. Did you use sdk manager to install Jetpack? There should be a developer tools section for “Target Components” that allows you to install an Nsight Compute version for the Nano. Did you see this, and if so, did you select it? If not, can you try installing that version? Do you have anything in your home directory/nvidia/nvidia_sdk?

image
image1413×911 237 KB