Hello. I want to ask if these two mechanisms are possible with NCU.
Profile for a given time (ex. 5 min) and then turn off the profiler while the profile target program keeps running.
: Similar to ‘duration’ option in dlprof & ‘launch-count’ + ‘kill no’ in NCU
Furthermore, profile for a given time, turn off the profiler, and later on continue profiling.
: Similar to sending SIGSTOP to only the profiler and the SIGCONT after few time has passed.
There is no option to stop profiling after a pre-defined amount of time. This would be tricky to use, as the overhead incurred by the tool can vary wildly, depending on the selected options and target application. Besides setting an upper limit on the number of collected results, as you’ve seen, you can instrument you application using the CUDA profiler start/stop API which is recognized by the tool. Nsight Compute has several options related to these APIs, too, in its command line guide.
Similarly, you can instrument you application or library with NVTX to specify the included or excluded ranges based on application logic, rather than on the number of CUDA kernel launches. Many DL and NVIDIA libraries already include support for NVTX.