I noticed significant overhead coming from libcuda.so after enabling kernel-level profiling for cuda graphs. Is it expected? If so, is there anything you guys could do to reduce the overhead?
I also wonder if cuptiActivityEnableHWTrace supports tracing individual kernels within a cuda graph? And could it reduce the libcuda.so overhead if this is the case? Just wanted to confirm before I try.
Hi Keren,
Yes, it is possible to get node-level tracing information for CUDA graphs using HES (Hardware Event System).
Our current software patching method for CUDA graph node-level tracing does introduce additional overhead. Using HES can significantly reduce this overhead. However, HES is only supported on specific platforms and GPU(Blackwell), as documented here:
https://docs.nvidia.com/cupti/main/main.html#hardware-event-system-hes
That section also points to a sample (activity_trace_async) you can use as a reference.
One detail not explicitly mentioned in the docs: to ensure HES works, you need to have the nvperf host library available along with the CUPTI library:
On Linux: libnvperf_host.so
On Windows: nvperf_host.dll
In the CUDA Toolkit, these nvperf libraries are shipped in the same directory as the CUPTI library, so normally no extra setup is required. If the nvperf library is missing or not found at runtime, calls to cuptiActivityEnableHWTrace may fail with CUPTI_ERROR_UNKNOWN or CUPTI_ERROR_NOT_INITIALIZED.
Thanks for the information. Do you mean libnvperf_host.so has to be loaded in the memory space as well using LD_LIBRARY_PATH or LD_PRELOAD?
The CUPTI library automatically loads the nvperf library, so explicit loading is unnecessary.
Awesome! I got it work now. Thanks very much!