Nsight Compute getting confused between kernels?

[asterix_obelix]

Hi, I have been trying to profile a couple of kernels that appear in succession in a code I am working on -

  fKernelSpecBase_CUDA<<< >>>();
  device dynchronize call;
  fKernelSpecOpt_CUDA<<< >>>();

I am on an A100 GPU, using CUDA 12.2, and profiling using Version 2023.2.1.0. I launch ncu as follows -

srun -N1 -n1 -c1 --threads-per-core=1 ncu --target-processes all --kernel-name-base demangled --kernel-id ::regex:KernelSpec:600 --call-stack --import-source yes --replay-mode application -f -o test <executable>

This results in both the above kernels being profiled as expected -

However, if I add --set full to the profiling call, ncu appears to confuse the two kernels reporting identical numbers for some of the metrics -

It is almost as if the profiler is getting confused across the two kernels? Am I doing something wrong? Is there a more appropriate way to profile kernels with similar names?

[felix_dt]

That looks unexpected and potentially like a bug. There is nothing wrong about adding --set full to this command line. Could you try the latest ncu version 2024.2, and see if the issue still reproduces?

Do you have a reproducer code that you could share for debugging this on our end?

[asterix_obelix]

Hi, I just edited the original post. I was mistaken about the version I was using. I am on Version 2023.2.1. I am working on a shared cluster, and do not have admin rights to install the newer version. I tried copying over the linux-desktop-glibc_2_11_3-x64 folder from under the target directory in my MacOS install, but that didn’t work. Do you have any suggestions on how could I proceed without admin rights to install on a remote server? Is there a preinstalled version that I could copy over?

[felix_dt]

I think you should be able to install it in a user-local directory, it doesn’t need to be in /opt or /usr. Have you tried that?

[asterix_obelix]

Upon testing, I believe the issue is resolved in the latest version. It might have been resolved in a prior version too, but have not tested anything between 2023.2.1 and 2024.2.

[felix_dt]

Thanks for the confirmation.