Hello!
On ga10x GPUs, I can include the following item in a --gpu-metrics-set file to obtain FMA pipeline throughput:
name: SM FMA Light Throughput
id: SM_C.TriageAC.smsp__inst_executed_pipe_fmalite
However, it seems that specifying a --gpu-metrics-set file is not supported on Orin, which is ga10b. Could you help me figure out the correct way to obtain FMA throughput on Orin?
Thank you so much!
You can make a copy of the reference ga10b config and edit that:
$ cp target-linux-tegra-armv8/GpuMetrics/ga10b.config target-linux-tegra-armv8/GpuMetrics/ga10b-new.config
$ vim target-linux-tegra-armv8/GpuMetrics/ga10b-new.config # replace alias: ga10b -> ga10b-new
$ vim target-linux-tegra-armv8/GpuMetrics/index.config # append ga10b-new
$ nsys profile --gpu-metrics-device=0 --gpu-metrics-set=ga10b-new ...
target-linux-tegra-armv8 is a directory where nsys executable resides.
Thank you for your reply! I actually used a relatively old version of nsys on Orin. After upgrading to 2024.2.2, I found the file ga10b-gfxt.config which has the FMA metrics.
Another question is that when including many GPU metrics and set --gpu-metrics-frequency to a relatively large value, e.g. 200000, a large portion of the nsys report displays “missing data”. Would it be possible to mitigate this?
Thank you so much!
That’s what you have to choose from. Having more metrics comes at a cost of spending more CPU time to collect each sample, so the more metrics you want, the less sustained sampling rate you can have.
I see. Thank you for the explanation!
Another question is that after switching from 2022.5 to 2024.2, the kernel information is lost from the report.
Before, I have CUDA HW(Orin) → [All. Streams] which shows all the kernels displayed in blue bars, which can be mapped to CUPTI_ACTIVITY_KIND_KERNEL.
After the upgrade, there is only Orin → [Memory usage, Static memory usage, etc]
I did notice an error message emerged after the upgrade:
Loaded CUPTI library: /opt/nvidia/nsight-systems/2024.2.2/target-linux-tegra-armv8/libcupti-tegra.so.11.4
EnableActivity(handle, kind) returned 14: CUPTI_ERROR_NOT_COMPATIBLE
Could this error be relevant to the loss of kernel information?
@mpopov is that the right CUPTI?
Yes, it appears to be correct.