Hello, I’m working with Nsight Compute/Cuda-11.7 version to profile a Fortran application. The runtime of the job is 3 minutes. But with Nsight compute profiler it doesn’t finish even in 4 hours. So looking for options to reduce profiling time (1) profile kernels from a single process i.e. process…

The Nsight Systems (nsys) profiler output shows that following kernel consumes most of runtime: [7/9] Executing 'cuda_gpu_kern_sum' stats report Time (%) Total Time (ns) Instances Avg (ns) Med (ns) Min (ns) Max (ns) StdDev (ns) …

You have to specify the --kernel-name-base option to tell ncu which version of the kernel/function name it should consider for matching the filter. In your case, it would be --kernel-name-base demangled. Note that you can furthermore use regular expressions with the regex: prefix (as clarified in th…

Now with these NCU options ncu --target-processes all -o report.ncu.tpall.out.%h.%p -f --kernel-name regex:cutlass_80_tensorop --kernel-name-base demangled It fails with following error: ==ERROR== UnknownError ==ERROR== Failed to profile kernel "Kernel" in process 61987 ==PROF== Trying to shutd…

In general, we recommend profiling a single rank per node for MPI applications. Nsight Compute, by default, serializes kernels within a process, replays them, and needs a global lock on the profiling hardware. Because of this, multi-process applications with communication etc… like MPI applications,…

Ok Thanks. Without NCU profiler, the job’s run time is 3 minutes. But with NCU, and single kernel profiling [ncu --target-processes=all -o report.ncu.tpall.out.%h.%p -f --kernel-name regex:cutlass_80_tensorop --kernel-name-base demangled] it doesn’t complete in 4 hours also. Please suggest what…

It’s likely that you’re profiling many instances of the same kernel. One thing to try is profiling a subset using the --launch-count flag. You can optionally add the --launch-skip and --launch-skip-before-match flags. See here for more details on that Nsight Compute CLI :: Nsight Compute Documentati…

Thanks. Our objective of profiling this Fortran application is: to get general characteristics of code such as Time split between CPU & GPU. Summary of time spent in MPI, Data transfer between Host & Device and vice versa. % of each GPU Utilization. Memory Utilization etc To get these details, w…

Nsight Systems is going to give you more of this type of information than Nsight Compute.

The imported output from a Nsight Compute profiler report shows some of the values as “nan”: void cutlass::Kernel<cutlass_80_tensorop_d884gemm_64x32_16x4_tn_align1>(T1::Params), 2023-Sep-06 01:39:44, Context 1, Stream 7 Section: GPU Speed Of Light Throughput -------------------------------…

Option to profile only master process

Developer Tools Nsight Compute

felix_dt August 29, 2023, 6:09am 2

You can find further information on these topics here and here in the documentation.

Topic		Replies	Views
Nsight-Compute returns “No kernels were profiled” warning Nsight Compute	9	1800	July 27, 2023
Question about profiling nccl kernels with Nsight Compute Nsight Compute	22	5926	October 10, 2025
Can't Get NCU GUI To Import Properly Nsight Compute	8	1592	October 5, 2020
Nsight Compute not detecting kernel launch Nsight Compute profiling	13	3414	May 6, 2021
Kernel output all correct but got NAN when profiling with nsight-compute Nsight Compute cuda	4	1010	January 12, 2024
Cannot profile kernel from CUDA samples Nsight Compute	6	640	May 31, 2023
Nsight-compute print "the application returned an error code (249)" Nsight Compute	5	1626	February 13, 2023
==ERROR== Failed to profile "createVersionVisualization" in process 12840 Nsight Compute	20	313	September 1, 2025
Nsight compute option to profile only 1 process Nsight Compute cuda	1	426	August 28, 2023
Nsight compute can not profile when use openmpi and nvshmem in multi-gpus Nsight Compute cuda , kernel	2	1325	January 17, 2024

Option to profile only master process

Related topics