Cupti instrumentation overhead

Developer Tools Nsight Systems Profiling Linux Targets
1

Hi,
I’m new to using nsight systems

I want to profile the DLRM model inference using nsight systems on ubuntu.
But It seems that the overhead of the profiler is too high on the first iteration of inference because of the CUPTI instrumentation as below
(the unrolled_elementwise_kernel part is the first iteration.)

image
image1033×364 27.9 KB

My command was nsys profile -c cudaProfilerApi -t nvtx,cuda python dlrm_s_pytorch_inference.py <OTHER_ARGS>

Why is this happening?
Can I somehow filter out the first iteration with CUPTI instrumentation overhead?
Or can I reduce the overhead in some way?

Thanks in advance

2

I am running into the exact same issue with the PyTorch profiler which is based off of CUPTI. I am seeing 4 seconds of the CUpti_ActivityOverhead activities before start seeing my runtime and kernel activities.