Cannot get tensor core metrics with latest NSight system

Developer Tools Nsight Systems Profiling Linux Targets
,
1

I would like to use NSight system to profile a cudnn program test.cpp. However, I found that some GPU metrics were not showing up. I found a related topic in this forum (here). It was because the version is too old. I updated my target Nsight system to the latest version, but this problem still exists. (I used apt-get purge to uninstall the old version of the Nsight system which was installed with cuda, then I installed the new one.) I didn’t use the CLI on the target platform. I used nsight GUI on the host system to start profiling. Here is what I saw on the host Nsight system.

1
12560×1390 404 KB

If I run nsys profile --gpu-metrics-device all, I will get

GPU 0: General Metrics for NVIDIA GA100 (any frequency)
WARNING: duration = 0 for system-wide collection. You'll have to stop the collection with CTRL+C or execute `nsys stop --session=`.
/dvs/p4/build/sw/devtools/Agora/Rel/QuadD_Main/QuadD/Common/AgentAPI/Src/Controller.cpp(611): Throw in function Agent::Client::{anonymous}::ProfileController::ProfileController(const Agent::Client::ControllerParams&, const Ptr&)
Dynamic exception type: boost::wrapexcept<QuadDCommon::InvalidArgumentException>
std::exception::what: InvalidArgumentException

System details
Host Nsight version: 2023.1.1.127-32365746v0 Windows-x64.
Target platform is Microsoft Azure server.
Target Nsight version (nsys --version): 2023.1.1.127-32365746v0.
GPU: A100.
Driver version: 520.61.05.
Cuda version: 11.8.

nsys status -e:

Timestamp counter supported: Yes

CPU Profiling Environment Check
Root privilege: disabled
Linux Kernel Paranoid Level = 0
Linux Distribution = Ubuntu
Linux Kernel Version = 5.15.0-1033-azure: OK
Linux perf_event_open syscall available: OK
Sampling trigger event available: OK
Intel(c) Last Branch Record support: Not Available
CPU Profiling Environment (process-tree): OK
CPU Profiling Environment (system-wide): OK

apt list --installed | grep nsight

cuda-nsight-11-8/unknown,now 11.8.86-1 amd64 [installed,auto-removable]
nsight-systems-2023.1.1/now 2023.1.1.127-32365746v0 amd64 [installed,local]

How can I see the missing metrics?

2

@pkovalenko can you respond

3

Apologies for the late response. This is a known issue that should be resolved in our recent release. Please let me know if it works for you.

4

I seem to have a similar issue… Is this issue expected to be fixed in 2023.2.1?

❯ nsys --version
NVIDIA Nsight Systems version 2023.2.1.122-32598524v0
❯ nsys profile --gpu-metrics-device all
GPU 0: General Metrics for NVIDIA TU10x (any frequency)
WARNING: duration = 0 for system-wide collection. You'll have to stop the collection with CTRL+C or execute `nsys stop --session=`.
/dvs/p4/build/sw/devtools/Agora/Rel/QuadD_Main/QuadD/Common/AgentAPI/Src/Controller.cpp(622): Throw in function Agent::Client::{anonymous}::ProfileController::ProfileController(const Agent::Client::ControllerParams&, const Ptr&)
Dynamic exception type: boost::wrapexcept<QuadDCommon::InvalidArgumentException>
std::exception::what: InvalidArgumentException
5

No, it will be fixed in 2023.3.1, currently expected to go out the door later this summer.