Nsys Profiler- Wrong event order

I am trying to profile page faults for my cuda program using below command

nsys profile -t cuda --force-overwrite true --cuda-um-cpu-page-faults=true --cuda-um-gpu-page-faults=true -o trace_report ./cuda_program

It shows the below error

Items {
        Type: ErrorText
        Value: "Wrong event order has been detected when adding events to the collection:\nnew event ={ StartNs=190185147777 StopNs=190185371995 GlobalId=347136872939520 Event={ CudaUvmGpuPageFaultEvent=[{ Address=140386738634752 NumberOfPageFaults=3 FaultAccessType=1 DeviceId=0 },] } Type=120 }\nlast event ={ StartNs=190526069698 StopNs=190526324124 GlobalId=347136872939520 Event={ CudaUvmGpuPageFaultEvent=[{ Address=140389102125056 NumberOfPageFaults=3 FaultAccessType=1 DeviceId=0 },] } Type=120 }"
      }

@skottapalli can you take a look at this one?

@turbocandy - thank you for reporting the bug.

  1. Which version of nsys are you using?
  2. Which target platform and OS are you trying to profile?
  3. What is the output of nvidia-smi command on the target system?
  4. Could you share the .nsys-rep file?
  5. To allow me to debug, could you share the repro? Does it repro 100% of the time when profiling your cuda program?

The version of nsys is ‘NVIDIA Nsight Systems version 2021.5.2.53-28d0e6e’
OS version ‘Ubuntu 20.04.4’,
Nvidia GeForce 3090 driver 470.199.02,
CUDA version 11.4
nyc_0.nsys-rep (13.8 MB)

Yes the above issue arises every time profiling the CUDA program

Thank you for the info. The version is pretty old. We have fixed a few bugs in nsys and the underlying CUPTI w.r.t --cuda-um-cpu-page-faults feature. Please use the latest 2023.3 version from https://developer.download.nvidia.com/devtools/repos or Nsight Systems | NVIDIA Developer and let us know if you are still seeing the bug.

Hi

I got the same issue here:

{
Type: ErrorText
Value: “Wrong event order has been detected when adding events to the collection:\nnew event ={ StartNs=90567415755 StopNs=90567420233 GlobalId=283116778389059 Event={ TraceProcessEvent=[{ Correlation=27094481 EventClass=0 TextId=1468 ReturnValue=0 },] } Type=48 }\nlast event ={ StartNs=90671527721 StopNs=90671545254 GlobalId=283116778389059 Event={ TraceProcessEvent=[{ Correlation=27124593 EventClass=0 TextId=1468 ReturnValue=0 },] } Type=48 }”
}

Here’s my command:

nsys profile python raven/main_nvsa_marg.py --mode test … (a bunch of run options)

Here’s my system info:

  • NVIDIA Nsight Systems version 2024.1.1.59-241133802077v0
  • Ubuntu 18.04.6 LTS
  • NVIDIA GeForce RTX 2070, CUDA Version: 12.2 (got by “nvidia-smi”)

Note that if I run “nvcc --version” it will show CUDA version as 9.1.85, which doesn’t match nvidia-smi. Not sure if that’s an issue.

Looking forward to your response.

Thanks!

My issue resolved by moving to newer version. You can change nvcc version by exporting the PATH variable , check first response here

Thanks for your response. Do you mean it’s resolved after moving to a newer version of Nsight Systems?

1 Like

Hello @hancheny11 yes, please move the latest version of Nsight Systems 2024.2. It has a fix for the out-of-order error that you are seeing.