Excessive CUDA profiling data flush

Greetings, thanks for making nsys, it is an exceptionally useful tool!

I’m encountering excessive profiler overhead (“CUDA profiling data flush” in the screenshot):

See in the image the regularly occuring 100% CPU usage in “CUPTI worker thead” and NSys, that are blocking gpu utilization.

I believe this is caused because one of the libraries is repeatedly calling cudaProfilerStart/Stop:


Since I don’t have access to the source code for this library (Gst-nvtracker — DeepStream 6.2 Release documentation) I can’t remove those calls.

The cudaProfilerStart/Stop calls are already being ignored by nsys, I tried with:

nsys profile -t cuda,nvtx -c none app
nsys profile -t cuda,nvtx -c cudaProfilerApi --capture-range-end=none app

But the profiler still initiates the “CUDA profiling data flush” every time cudaProfilerStop is called. Is there a way to override this behavior?

Ubuntu 20.04
NVIDIA driver 525.85.12
CUDA 11.8
NSight Systems 2022.4.2, 2023.1.1 (same behavior)
CPU Ryzen 2600 + GPU RTX 2060

The behavior can be reproduced to a lesser extent by running this demo app: deepstream_python_apps/apps/deepstream-nvdsanalytics at master · NVIDIA-AI-IOT/deepstream_python_apps · GitHub
python deepstream_nvdsanalytics.py file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4

Unfortunately, that is the default behavior.

Fortunately, we are adding an option to change that, --flush-on-cudaprofilerstop=false

That change will ship in our next release, 2023.2, scheduled to go out in three weeks.

@ztasoulas , is there anything that can be done in the meantime?

Unfortunately, currently there is no work around for this overhead, unless CUDA is not traced at all.

As Holly mentioned, the next release will provide the --flush-on-cudaprofilerstop switch to control this behavior. The option will be available for profiling from the GUI as well.

Thanks for the quick response, that is good news. I’ll be looking forward to the new release!

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.