Is there a way to control the number of runs the Visual Profiler requires to profile an application?
I’m trying to profile an application that takes a minute or two to run. The profiler runs it 12 times, so it takes nearly 1/2 hour to do one profiling run. Unfortunately, the results of a half-our run are usually not very informative - the profiler records 20,002 rows on all but a couple of runs (runs that fail or exceed the allotted runtime), and it drops 19,995 of those rows.
I’d like to be able to:
Profile single runs. Yes, the results are not as statistically significant, but they can still be very useful.
Instead of dropping the rows that are not there in every run, drop the runs that do not complete successfully.
Get useful profiling information on my code in some way. I’d be interested in alternate suggestions.
Use the profiler without the GUI.
If you set CUDA_PROFILE=1, the driver will generate a complete log file. You can select the counters you want to analyze.
Look at the manual/readme for more info.
OK - thanks. I’ll try that. Since the CLI documentation in the Users’ Guide is only 10 pages at the end, I thought that the preferred approach would be to use the GUI, and that I was missing something. I’d still love to be able to use the profiler through the GUI, with its automated analysis.
I’m now using the command-line interface. I keep getting messages like the following in my output:
NV_Warning: Counter <91>l1_global_load_hit<92> is not compatible with other selected counters and it cannot be profiled in this run.
Where can I find documentation on what counters are compatible with what other counters? Where can I find documentation on how many counters are available on a particular device?
Does the GUI run the programs multiple times because I have selected counters that cannot be collected at the same time?
So - clearly the GUI driver runs the application multiple times, with different settings each time. Is there a way to find out what parameters it is using on each run? Some of the profiling settings consistently cause my application to crash, and I’d like to avoid that.
Profile single runs. Yes, the results are not as statistically significant, but they can still be very useful.
You can disable all profiler counters (In the “Session settings dialog” under the “Profiler Counters” tab).
In this case the application will be run only one time & only basic timing information & other options for kernels and memory copies will be collected. Note that in this case most of the “automated analysis” features will not be available as they require profiler counters.
Instead of dropping the rows that are not there in every run, drop the runs that do not complete successfully.
Does your application use multiple streams? Visual Profiler version 4.0 has a bug due to which rows are dropped in this case.
Refer The Official NVIDIA Forums | NVIDIA
You should check why some runs are failing? In case the execution timeout is reached you can try increasing the timeout limit (the default is 30 seconds). Also note that the following counters can cause GPU kernels to run longer than the Timeout Detection and Recovery (TDR) limit.
“gld instructions 8/16/32/64/128 bit”
“gst instructions 8/16/32/64/128 bit”
In this case the application will terminate with an error and profiling data will not be available. Disabling the TDR timeout is recommended when these counters are selected. Or you can disable these counters.
Where can I find documentation on how many counters are available on a particular device?
You can refer the “profiler counters” section in the Compute Visual Profiler User Guide.
Does the GUI run the programs multiple times because I have selected counters that cannot be collected at the same time?
Yes Visual Profiler runs the program multiple times to collect data for all the selected counters.
To find which counters are collected in each run you can save the project and look at the “.cvp” project file.
Pasting information from the Visual Profiler 4.0 sample project “analysis.cvp”:
gld_inst_8bit,gld_inst_16bit,gld_inst_32bit,gld_inst_64bit,gld_inst_128bit
gst_inst_8bit,gst_inst_16bit,gst_inst_32bit,gst_inst_64bit,gst_inst_128bit
local_load,local_store,gld_request,gst_request
shared_load,shared_store,sm_cta_launched,l1_local_load_hit
branch,l1_local_load_miss,l1_local_store_hit
divergent_branch,l1_local_store_miss,l1_global_load_hit
inst_issued,l1_global_load_miss
inst_executed,uncached_global_load_transaction,global_store_transaction
warps_launched,threads_launched,l1_shared_bank_conflict
active_warps
active_cycles
Note that run number 1 is used to collect basic timing information and no counters are enabled in this run.
Thank you. That was quite helpful. While the Visual Profiler Users Guide does not seem to list the number of counters available, it does list the things that they can count. It seems that the global loads and stores, qualified by size, are not available with the compute architecture 2.0, but only with 2.1. That is apparently what was causing runs 2 and 3 of my program under the profiler to crash every time.
Thank you also for the pointer to the .cvp file. That’s very helpful for understanding just what the GUI is doing.
The “gld instructions 8/16/32/64/128 bit” and “gst instructions 8/16/32/64/128 bit” counters are available for GPU devices with compute capability 2.0 and 2.1. (Also Visual Profiler will not allow you to select counters which are not supported).
So for these counters you are either:
a) hitting the application timeout limit (you can try increasing the value in the session settings dialog) or
b) the kernel is hitting the TDR limit (you are try disabling TDR timeout).
You can find out based on the message in the Visual Profiler output window for application runs 2 and 3.
Thank you for your help on this. Please bear with me a bit longer - I’m still confused, and trying to work through this.
I didn’t think that the Visual Profiler would allow me to select counters which are not supported, either. However, the table on page 64 of “Compute Visual Profiler Users Guide”, DU-05162-001_v04, says that gld{32,64,128}b and gst{32,64,128}b are supported in 2.0 but not 2.1.
Ah - here’s why I’m confused about this. The gld and gst instruction counters are on page 67-8. The counters that are not supported in 2.0 are the transaction counters on page 64.
So, the global load/store instruction counters should work. However, when I enable them, my software exits with messages like the following, which do not look to me like timeouts. Are these really timeouts, or is something else likely to be going on here?
I’m running under Scientific Linux 6.1, 64-bit, with current updates, on a GeForce GTX 480. I think that the TDR limits are Windows-only, and the runs that die under the profiler do so quite quickly.
OK - thanks - will do. Just for reference, I’m running kernel 2.6.32-131.17.1.el6.x86_64, with driver nvidia-x11-drv-285.05.09-1.el6.elrepo.x86_64, and the CUDA 4.0 dev kit. Its runtime links are as follows:
I have been unable to isolate a reasonable test case to demonstrate this bug. I’m attaching the output of nvidia-bug-report.sh, in case that is useful even without a test case.