Nothing shown in Cuda HW while SM Warp occupancy is more than 0

[ftatp5901]

Hi there!

Does anybody knows why the Cuda HW is empty even though there is some SM occupied, just like the picture? Is this possible? Why?

Thanks

[hwilper]

It’s a little hard to tell from a screenshot, but a couple of initial questions. Do you have more than one GPU on this system? Could the GPU be busy doing functions that we do not trace (see User Guide — nsight-systems 2024.7 documentation for the list of default traced functions)?

@pkovalenko do you have other suggestions?

[ftatp5901]

Thanks for your reply,
Yes my system do have 2 GPUs. But the second device is never used. When I check the nvidia-smi, the memory usage of the second device is always near to 0.

[Greg]

The most likely case is that another process is running on the GPU. The Performance Counters are collected at a device level, not a context level. The other context that is running is not being traced by NSYS. NSYS has a GPU Context Switch trace feature that can be used to identify what process is active on the GPU.

[ftatp5901]

Thank you @hwilper and @Greg for your help!

I’d found out the answer : the program was using CudaGraph, which is incompatible with nsys.

It is the first time for me to look at what CudaGraph is, leading me to confusion, so please forgive me if the question was silly lol

[hwilper]

CudaGraph isn’t incompatible with Nsys, you just need to use an option for it.

(that is assuming that you have a relatively recent version of Nsys)

[ftatp5901]

Wow, thanks! It’s showing!!!