Error: Some events were lost. How do I fix this?

Developer Tools Nsight Systems Profiling Linux Targets
1

I’m profiling Python code which uses CUDA (e.g. CuPy) and always get an error about events being lost e.g.

Some events (328,938) were lost. Certain charts (including CPU utilization) on the timeline may display incorrect data. Try to decrease sampling rate and start a new profiling session.

Does anyone know if this error is something to worry about? Decreasing the sampling rate didn’t help.

I also get a couple of warnings, I don’t know if they’re related:

  • Installed CUDA driver version (11.2) is not supported by this build of Nsight Systems. CUDA trace will be collected using libraries for driver version 11.1
  • The operating system throttled the collection of sampling data 321 times.

Thanks for any help.

2

If the system is throttling, there is a good chance that you are going to not get all the sampling information that you expect. We don’t want you to think you are getting a clean report with all the data if you are not.

May I ask how long you are collecting for and how wide?

I’ll also ask my sampling expert to have a look at this.

3

Hi Jonathan,

What platform are you on (CPU arch, OS combination)? What nsys CLI command line did you use to collect the data? How long does your collection last?

4

Hello, it’s Intel(R) Xeon(R) CPU E5-2670 v3 CPU & Ubuntu 16.04.6

nsys profile -f true -t cublas,cudnn -o pt_cifar10 python cuda.py

Collection is typically 40-60s.

Thanks