I’m new to using nsight systems
I want to profile the DLRM model inference using nsight systems on ubuntu.
But It seems that the overhead of the profiler is too high on the first iteration of inference because of the CUPTI instrumentation as below
(the unrolled_elementwise_kernel part is the first iteration.)
My command was
nsys profile -c cudaProfilerApi -t nvtx,cuda python dlrm_s_pytorch_inference.py <OTHER_ARGS>
Why is this happening?
Can I somehow filter out the first iteration with CUPTI instrumentation overhead?
Or can I reduce the overhead in some way?
Thanks in advance