Xavier NX - Nsight Compute "Failing to prepare kernel for profiling"

Hi,

I have an Xavier NX with Jetpack 4.5 that I wish to profile CUDA kernels using Nsight Compute. I am connecting from my host computer and went through the process to enable root access over ssh already. When I connect I put the User Name as root and this is the output I get after I launch and attach to one of the CUDA sample applications:

==ERROR== An error occurred while trying to profile.
==WARNING== No kernels were profiled.
==PROF== Attempting to connect to ncu-ui at 10.0.0.72:50152…
==PROF== Connected to ncu-ui at 10.0.0.72:50152.
==WARNING== Note: Running with unmodified GPU clocks. If not controlled otherwise, profiling results may be inconsistent.
==PROF== Connected to process 11495 (/usr/local/cuda-10.2/samples/0_Simple/matrixMul/matrixMul)
==ERROR== Failed to prepare kernel for profiling
==ERROR== Failed to profile kernel “MatrixMulCUDA” in process 11495
==ERROR== UnknownError
==PROF== Disconnected from process 11495
==ERROR== An error occurred while trying to profile.
==WARNING== No kernels were profiled.

I’m also confused about Nsight Compute with Jetpack. I believe it’s part of the CUDA toolkit but I cannot find it anywhere locally on the Xavier? Or is it just a host program?

Any help would be greatly appreciated!

Hi,

Nsight compute doesn’t support the Jetson host.
You can find this information on the below page:
https://developer.nvidia.com/nsight-compute

Supported platforms

Host
  • Linux x86_64[1]
  • Windows x86_64[1]
  • MacOS[1]
Target
  • Linux x86_64[1]
  • Windows x86_64[1]
  • Linux PowerPC[1]
  • Linux aarch64 sbsa[1]
  • DRIVE OS QNX aarch64[2][3]
  • DRIVE OS Linux aarch64[2][3]

The “Failed to prepare …” error is caused by an authority issue.
Could you double confirm that if you ssh the device with root?

Thanks.

Hi!
I have the same problem and I ssh the device with root. what else can be the problem?

Hi,

We want to reproduce this issue internally.
Could you share which Nsight Compute version do you use?

If you are not using the latest one, could you update it and try it again?
https://developer.nvidia.com/nsight-compute

Thanks.

Hi,

I was already logging in as root. I performed the same setup as: No timeline for profiler Xavier: nvvp and nsight compute is not working in Jetson Xavier

Per the suggestion, I updated to Nsight Compute 2020.3.1 but get the same issue. I was previously using 2020.1.2. The targeted program will execute and produce the correct output but kernel profiling fails.

Just to be sure, I have my target platform set to Linux (aarch64 sbsa). Is this correct?

Thanks

Hi,

Thanks for your feedback.

We are checking this issue with our internal team.
Will share you more information later.

Hi,

Thanks for your patience.

After checking internally, please install the Nsight Compute via SDKmanager (within CUDA toolkit).
You can find it under the /opt/nvidia/nsight-compute/2019.5.0 folder.
The tool shared in our webpage is for other system, not for Jetson.

Thanks.

Hi,

Thank you for the response. Does the host or Jetson need this install? Also, is the /opt/nvidia/nsight-compute/2019.5.0 on the host or Jetson side? I see this path on the host but not the Jetson.

Thank you

Hi,

The tool need to be installed on the host side.
And connect to the Jeston via ssh.

Thanks.

Apologies I’m still having some issues. I suspect my install of Nsight-compute 2019.5.0 is incomplete.

When I go to /opt/nvidia/nsight-compute/2019.5.0 I only two folders - host and target. Comparing to another version I had installed this is only a subset.

I would like to reinstall 2019.5.0 but doing so via the SDKmanager seems to install a newer version. Is there a way to specify that I want to reinstall 2019.5.0 since you mentioned it needed to be installed via the SDKmanager?

Is there any update on this. I am also facing the same issue.

Hi,

Please reinstall it with SDK manager.
Nsight Compute is part of CUDA toolkit and the latest version for Jetson is still 2019.05.

You may need to uninstall CUDA first.
In STEP01 page, you can find “Repair/Uninstall” bottom in the lower left.

Thanks.

Hi,

I confirmed the solution. The issue was an incomplete install of Nsight compute 2019.5.0 from the SDK. After starting from scratch, installing from the SDK I can now profile kernels on the Xavier NX.

Thank you so much for the help!

Hi @AastaLLL, @TylerMG26,

There is no Linux (aarch64 sbsa) in the section target of Nsight Compute 2019.5 version, so how i can select the Jetson Xavier as the target ?

Thanks

Hi, chakibdace

Please find information in the below comment:

Thanks.