Hello all,
I’m confused to see L1 cache section till 340 ms when compute is completed at 200ms. Am I missing something?
It would be helpful to know the ncu version, GPU and platform (Linux/Windows, WSL/vGPU/…) this was collected on and how what duration ncu reported for this the workload overall.
In general, it is possible that the timeline shows data before and after the profiled kernel launch, as data collection is not started/stopped precisely at kernel boundaries. For some metrics that are shared beyond the SM, it is also expected to see non-zero data there.
It is furthermore relevant whether or not the report was collected with context-switch trace. It allows the tool to filter all metrics in the timeline to only data relevant for the target CUDA context. Without it, data is unfiltered and samples from different passes are aligned with best effort only. The timeline indicates if you have this information e.g. in the right-click context menu, which would either say “Disable Context Switch Filter” or “Disable Trim Filter”.
Knowing which ncu version and GPU this was collected with would help to understand the tool support level and the expected number of passes needed to collect all metrics.
If this was collected with an ncu version older than 2024.2.1, it’s possible this is a bug fixed in that version.
Fixed an issue with PM sampling reporting incomplete data.
Hello,
I profiled by application on Quadro RTX 5000 on Ubuntu 20.04. The earlier screenshot was from an older version of ncu (2023)
I upgraded my OS to ubuntu 24.04, CUDA Toolkit to cuda 13 and ncu version to 2025.3.1. The latest profiler result is as below.
I’m attaching the complete ncu report as well. Thank you for your response.
Why do I have PM metrics till 190ms whereas the worklaod execution goes beyong 300ms? Also, it would be helpful to understand the limitations of current cuda code based on the profiler result.
It seems there are some issues with calculating the needed buffer size/sampling interval on this Turing chip for the long kernel. Compared to later architectures, the sampling interval is computed in cycles here, rather than in (nano)seconds, which is less precise. I would suggest to manually set the buffer size and/or the sampling interval, e.g. like this
ncu --pm-sampling-interval 400000 --pm-sampling-buffer-size 50000000 …
If this is the kernel you are primarily interested in for profiling, you can also add -k identifyNeighborsGrid to profile only this one.
The tool uses different metrics with different internal definitions for the L1 hit rate on PM sampling (single-pass collection) and regular profiling (multi-pass collection) on Turing. We are checking on this, but you can consider the multi-pass metric reported on the Details page to be more reliable and use its value. The issue should be specific to Turing, and doesn’t exist on newer architectures.