How to use a script to get sass information from a .ncu-rep file

[783161219]

Hi, I have a .ncu-rep file and I want to get the sass information from it to do some analysis. In the past I just open nsight compute, change to Source page and use Copy as Image to output .csv file of all sass in the kernel, and then do some analysis based on this csv file.

But this process is a bit complex since I have many .ncu-rep files. So is there any way to get stall information of each sass by a script. I find out that there is a python library called ncu_report, but when I use sass_by_pc it always return an empty string.

For example, I use
ncu_file = ncu_report.load_report(‘xxx.ncu-rep’)
ncu_range = ncu_file.range_by_idx(0)
kernel = ncu_range.action_by_idx(0)
sass = kernel.sass_by_pc(0xXXXXXXXX)
and there is no return whether I pass any pc in.

So is there any example to get sass information by ncu_report, or is there any way to generate .csv file when I run ncu. Thanks

[felix_dt]

The API you were trying is the right approach for fine-grained access to metric and SASS information. As explained in the documentation for sass_by_pc(), you need to pass absolute addresses to this function, as they are reported as correlation IDs for the SASS-based metrics. A working example code would look like below. You can replace inst_executed with sampling-based warp stall metrics, too.

a = r.action_by_idx(0)
m = a.metric_by_name("inst_executed")
pcs = m.correlation_ids()

for cid in range(0, pcs.num_instances()):
    pc = pcs.as_uint64(cid)
    sass = a.sass_by_pc(pc).strip()

Also since 2021.3, Nsight Compute CLI ncu supports printing the source and sass using the new option –print-source. This option can be combined with --csv.

[783161219]

Thank you for the reply

ncu_file = ncu_report.load_report('temp.ncu-rep')
ncu_range = ncu_file.range_by_idx(0)
kernel = ncu_range.action_by_idx(0)
m = kernel.metric_by_name("inst_executed")
pcs = m.correlation_ids()

for cid in range(0, pcs.num_instances()):
  pc = pcs.as_uint64(cid)
  sass = kernel.sass_by_pc(pc).strip()

My code is as above now, the pc is identical to the address shows in nsight compute, but the sass is still an empty string. Is this rely on something other than _ncu_report.so and ncu_report.py?

[felix_dt]

No, the ncu_report files you listed are sufficient. Please state which exact Nsight Compute version you are using (ncu --version) and for which GPU architecture your report is collected?

Also, check if the command-line csv output I mentioned is working for this report.

[783161219]

Oh, the .ncu-rep file I used is generated by nv-nsight-cu-cli. I use ncu to generate a new file and it works.

But it seems that we can only get the sass code by ncu_report, but can’t get stall status of each sass?

I will try to use command line to gen the csv file, thank you very much.

[felix_dt]

Stalls are sass-correlated metrics, just like inst_executed. You can get the individual stall reason values as metrics, too. See e.g. the SourceCounters.section file for their names. You would need to iterate all of them and add up the stall values per instruction offset.

m = a.metric_by_name("smsp__pcsamp_warps_issue_stalled_long_scoreboard")
pcs = m.correlation_ids()

for cid in range(0, pcs.num_instances()):
    pc = pcs.as_uint64(cid)
    sass = a.sass_by_pc(pc).strip()
    print(sass)
    print(m.as_uint64(cid))

The metric value for “group:smsp__pcsamp_warp_stall_reasons” also contains a string with a comma-separated list of the stall reason metric names, which you could parse in your script, too.

[783161219]

I see, thank you very much, this helps me a lot.