Nsys Profiler- Wrong event order

Developer Tools Nsight Systems Profiling Linux Targets
1

I am trying to profile page faults for my cuda program using below command

nsys profile -t cuda --force-overwrite true --cuda-um-cpu-page-faults=true --cuda-um-gpu-page-faults=true -o trace_report ./cuda_program

It shows the below error

Items {
        Type: ErrorText
        Value: "Wrong event order has been detected when adding events to the collection:\nnew event ={ StartNs=190185147777 StopNs=190185371995 GlobalId=347136872939520 Event={ CudaUvmGpuPageFaultEvent=[{ Address=140386738634752 NumberOfPageFaults=3 FaultAccessType=1 DeviceId=0 },] } Type=120 }\nlast event ={ StartNs=190526069698 StopNs=190526324124 GlobalId=347136872939520 Event={ CudaUvmGpuPageFaultEvent=[{ Address=140389102125056 NumberOfPageFaults=3 FaultAccessType=1 DeviceId=0 },] } Type=120 }"
      }
2

@skottapalli can you take a look at this one?

3

@turbocandy - thank you for reporting the bug.

  1. Which version of nsys are you using?
  2. Which target platform and OS are you trying to profile?
  3. What is the output of nvidia-smi command on the target system?
  4. Could you share the .nsys-rep file?
  5. To allow me to debug, could you share the repro? Does it repro 100% of the time when profiling your cuda program?
4

The version of nsys is ‘NVIDIA Nsight Systems version 2021.5.2.53-28d0e6e’
OS version ‘Ubuntu 20.04.4’,
Nvidia GeForce 3090 driver 470.199.02,
CUDA version 11.4
nyc_0.nsys-rep (13.8 MB)

Yes the above issue arises every time profiling the CUDA program

5

Thank you for the info. The version is pretty old. We have fixed a few bugs in nsys and the underlying CUPTI w.r.t --cuda-um-cpu-page-faults feature. Please use the latest 2023.3 version from https://developer.download.nvidia.com/devtools/repos or Nsight Systems | NVIDIA Developer and let us know if you are still seeing the bug.