I would like to retrieve the link between NVTX and CUPTI events from Nsight Systems sqlite database profiling output.
I have NVTX events in my TensorRT application. Each NVTX event is associated with kernel executions. I would like to get the name of the executed kernels for each NVTX events.
The tables in the sqlite database are the following :
There is a “shortname” integer field in CUPTI_ACTIVITY_KIND_KERNEL and I can get the kernel name from it in the StringIds table.
There is also a correlationId field but it doesn’t match the NVTX id field of the NvtxPushPopRanges table.
How to link “correlationId” from the CUPTI_ACTIVITY_KIND_KERNEL table with “id” from the NvtxPushPopRanges table.
There is no direct way to correlate the NVTX events with the CUDA kernel execution events. NVTX events are not associated with any GPU events. These are markers/ranges captured during the execution of application on the CPU.
The “correlationId” field in the CUPTI kernel record is used to correlate it with the corresponding CUDA API (cuLaunch* etc) event. On the other hand, the “id” field in the NvtxPushPopRanges is typically used to correlate the range push event with the range pop event.
One can build the correlation between the NVTX ranges and CUDA APIs based on the timestamps.
Hi @mjain,
Sorry for the delay but thanks a lot for your helping answer.
I built the correlation between NVTX ranges and CUDA kernels based on timestamps but not between NVTX ranges and CUDA kernel launchers.
Do you know in which of the CUPTI tables I can find the execution times of kernel launchers ?
Thanks in advance,
Kernel timestamps would be available in the table CUPTI_ACTIVITY_KIND_KERNEL. Check out fields ‘start’ and ‘end’ for start and end timestamps. And I guess by CUDA kernel launchers, you mean corresponding CUDA APIs. Those can be found in the table CUPTI_ACTIVITY_KIND_RUNTIME or CUPTI_ACTIVITY_KIND_DRIVER depending on which API is used to launch the kernel. Basis of correlation would be correlationId field value.