Nsight Systems Issue: Unable to configure the collection of CPU IP samples

I get the following issue when trying to profile my program with Nsight Systems (which apparently eliminate Nsight Systems from collecting CUDA events for profiling):
“Source: Daemon
Description: Unable to configure the collection of CPU IP samples, backtraces, and/or scheduling data. Try the ‘nsys status --environment’ command to learn more.”

This is the output of command: “nsys status --environment”:
Timestamp counter supported: Yes
Sampling Environment Check
Linux Kernel Paranoid Level = 4: OK
Linux Distribution = Ubuntu
Linux Kernel Version = 5.11.0-38-generic: OK
Linux perf_event_open syscall available: Fail
Sampling trigger event available: Fail
Intel(c) Last Branch Record support: Not Available
Sampling Environment: Fail

GPU: RTX A6000
OS: Ubuntu 20.04
Driver version: 470.57.02
CUDA version: 11.4 (update 2)
Nsight Systems version: 2021.4.1

Any help is appreciated

In order to collect CPU samples, Nsight Systems uses the linux perf functionality under the covers. This functionality is not available when Paranoid is set >2.

(from the documentation)

Requirements for x86_64, Power, and ARM SBSA Targets on Linux

Use of Linux Perf : To collect thread scheduling data and IP (instruction pointer) samples, the Perf paranoid level on the target system must be 2 or less. Use the following command to check:

cat /proc/sys/kernel/perf_event_paranoid

If the output is >2, then do the following to temporarily adjust the paranoid level (note that this has to be done after each reboot):

sudo sh -c ‘echo 2 >/proc/sys/kernel/perf_event_paranoid’

To make the change permanent, use the following command:

sudo sh -c ‘echo kernel.perf_event_paranoid=2 > /etc/sysctl.d/local.conf’

1 Like

If you are running in a container see - Nsight Systems User Guide :: Nsight Systems Documentation

Thank you.
I had to decrease it to level 1 otherwise it won’t start.
Now I have the following issue:

Failed to start profiling. The target application <path to target application>

ApplicationExitedBeforeProfilingStarted (1104) {
    RuntimeError (120) {
        OriginalExceptionClass: N5boost16exception_detail10clone_implIN11QuadDCommon16RuntimeExceptionEEE
        OriginalFile: /build/agent/work/20a3cfcd1c25021d/QuadD/Host/Analysis/Clients/AnalysisHelper/AnalysisStatus.cpp
        OriginalLine: 79
        OriginalFunction: static QuadDAnalysis::AnalysisHelper::AnalysisStatus::StatusInfo QuadDAnalysis::AnalysisHelper::AnalysisStatus::MakeFromErrorString(QuadDAnalysis::AnalysisHelper::AnalysisStatus::StatusType, QuadDAnalysis::AnalysisHelper::AnalysisStatus::ErrorType, const string&, const DevicePtr&)
        ErrorText: The target application exited before profiling started.
    }
}
Full error information:
StopAnalysisError (4106) {
    RuntimeError (120) {
        OriginalExceptionClass: N5boost16exception_detail10clone_implIN11QuadDCommon16RuntimeExceptionEEE
        OriginalFile: /build/agent/work/20a3cfcd1c25021d/QuadD/Host/Analysis/Clients/AnalysisHelper/AnalysisStatus.cpp
        OriginalLine: 79
        OriginalFunction: static QuadDAnalysis::AnalysisHelper::AnalysisStatus::StatusInfo QuadDAnalysis::AnalysisHelper::AnalysisStatus::MakeFromErrorString(QuadDAnalysis::AnalysisHelper::AnalysisStatus::StatusType, QuadDAnalysis::AnalysisHelper::AnalysisStatus::ErrorType, const string&, const DevicePtr&)
        ErrorText: Analysis failed.
    }
}

Profiling options:
DeviceId: "Local"
EventTypes {
  Items: CpuCycles
  Items: OSRuntime
  Items: Cuda
}
HowToStart: Delayed
HowToStop: Delayed
DeviceType: Unix
DeviceDisplayName: "MyPC"
OSRuntimeOptions {
  DurationThresholdNs: 1000
}

Processes {
  HowToAttach: LaunchAnother
  Command: "/home/workspace/program/CMake/x64/Release/programExecute"
  WorkingDirectory: "/home/workspace/program/CMake/x64/Release/"
  UserName: "david"
  CollectCudaTrace: true
  EnvironmentVariables {
    Name: "DESKTOP_SESSION"
    Value: "ubuntu"
  }
  EnvironmentVariables {
    Name: "GDMSESSION"
    Value: "ubuntu"
  }
  EnvironmentVariables {
    Name: "GIO_LAUNCHED_DESKTOP_FILE"
    Value: "/usr/share/applications/nsys-ui-2021.4.1.desktop"
  }
  EnvironmentVariables {
    Name: "GIO_LAUNCHED_DESKTOP_FILE_PID"
    Value: "87580"
  }
  EnvironmentVariables {
    Name: "GJS_DEBUG_OUTPUT"
    Value: "stderr"
  }
  EnvironmentVariables {
    Name: "GJS_DEBUG_TOPICS"
    Value: "JS ERROR;JS LOG"
  }
  EnvironmentVariables {
    Name: "GNOME_DESKTOP_SESSION_ID"
    Value: "this-is-deprecated"
  }
  EnvironmentVariables {
    Name: "GNOME_SHELL_SESSION_MODE"
    Value: "ubuntu"
  }
  EnvironmentVariables {
    Name: "GPG_AGENT_INFO"
    Value: "/run/user/1000/gnupg/S.gpg-agent:0:1"
  }
  EnvironmentVariables {
    Name: "GTK_MODULES"
    Value: "gail:atk-bridge"
  }
  EnvironmentVariables {
    Name: "HOME"
    Value: "/home/david"
  }
  EnvironmentVariables {
    Name: "IM_CONFIG_PHASE"
    Value: "1"
  }
  EnvironmentVariables {
    Name: "INVOCATION_ID"
    Value: "d57d37a06db44f53ad952c48ea075210"
  }
  EnvironmentVariables {
    Name: "JOURNAL_STREAM"
    Value: "8:52902"
  }
  EnvironmentVariables {
    Name: "LANGUAGE"
    Value: "en_IL:en"
  }
  EnvironmentVariables {
    Name: "LD_LIBRARY_PATH"
    Value: "/usr/local/cuda/lib64:/opt/nvidia/nsight-systems/2021.4.1/target-linux-x64"
  }
  EnvironmentVariables {
    Name: "LOGNAME"
    Value: "david"
  }
  EnvironmentVariables {
    Name: "MANAGERPID"
    Value: "1881"
  }
  EnvironmentVariables {
    Name: "NV_AGORA_CRASH_FD"
    Value: "6"
  }
  EnvironmentVariables {
    Name: "NV_AGORA_PATH"
    Value: "/opt/nvidia/nsight-systems/2021.4.1/host-linux-x64"
  }
  EnvironmentVariables {
    Name: "NV_QUADD_PATH"
    Value: "/opt/nvidia/nsight-systems/2021.4.1/host-linux-x64/nsys-ui.bin"
  }
  EnvironmentVariables {
    Name: "PATH"
    Value: "/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin"
  }
  EnvironmentVariables {
    Name: "PWD"
    Value: "/home/david"
  }
  EnvironmentVariables {
    Name: "QT_ACCESSIBILITY"
    Value: "1"
  }
  EnvironmentVariables {
    Name: "QT_DPI_ADJUSTMENT_POLICY"
    Value: "AdjustDpi"
  }
  EnvironmentVariables {
    Name: "QT_IM_MODULE"
    Value: "ibus"
  }
  EnvironmentVariables {
    Name: "QT_PLUGIN_PATH"
    Value: "/opt/nvidia/nsight-systems/2021.4.1/host-linux-x64/Plugins:"
  }
  EnvironmentVariables {
    Name: "QUADD_INSTALL_DIR"
    Value: "/opt/nvidia/nsight-systems/2021.4.1/target-linux-x64"
  }
  EnvironmentVariables {
    Name: "SESSION_MANAGER"
    Value: "local/MyPC:@/tmp/.ICE-unix/2102,unix/MyPC:/tmp/.ICE-unix/2102"
  }
  EnvironmentVariables {
    Name: "SHELL"
    Value: "/bin/bash"
  }
  EnvironmentVariables {
    Name: "SHLVL"
    Value: "1"
  }
  EnvironmentVariables {
    Name: "SP_LD_LIBRARY_PATH"
    Value: "/usr/local/cuda/lib64"
  }
  EnvironmentVariables {
    Name: "SSH_AGENT_PID"
    Value: "2067"
  }
  EnvironmentVariables {
    Name: "SSH_AUTH_SOCK"
    Value: "/run/user/1000/keyring/ssh"
  }
  EnvironmentVariables {
    Name: "TSP_LD_LIBRARY_PATH"
    Value: "/usr/local/cuda/lib64"
  }
  EnvironmentVariables {
    Name: "USER"
    Value: "david"
  }
  EnvironmentVariables {
    Name: "USERNAME"
    Value: "david"
  }
  EnvironmentVariables {
    Name: "WINDOWPATH"
    Value: "2"
  }
  EnvironmentVariables {
    Name: "XDG_CONFIG_DIRS"
    Value: "/etc/xdg/xdg-ubuntu:/etc/xdg"
  }
  EnvironmentVariables {
    Name: "XDG_CURRENT_DESKTOP"
    Value: "ubuntu:GNOME"
  }
  EnvironmentVariables {
    Name: "XDG_DATA_DIRS"
    Value: "/usr/share/ubuntu:/usr/local/share/:/usr/share/:/var/lib/snapd/desktop"
  }
  EnvironmentVariables {
    Name: "XDG_MENU_PREFIX"
    Value: "gnome-"
  }
  EnvironmentVariables {
    Name: "XDG_RUNTIME_DIR"
    Value: "/run/user/1000"
  }
  EnvironmentVariables {
    Name: "XDG_SESSION_CLASS"
    Value: "user"
  }
  EnvironmentVariables {
    Name: "XDG_SESSION_DESKTOP"
    Value: "ubuntu"
  }
  EnvironmentVariables {
    Name: "XDG_SESSION_TYPE"
    Value: "x11"
  }
  EnvironmentVariables {
    Name: "_"
    Value: "/opt/nvidia/nsight-systems/2021.4.1/host-linux-x64/CrashReporter"
  }
  CudaFlushPeriodically: true
  CudaFlushPeriod: 10000000000
  CudaSkipSomeApiCalls: true
  CollectGPUMemoryUsage: false
}
ShowBacktrace: true
StartDelayMilliseconds: 10000
DurationMilliseconds: 30000
UseDWARF: true
IncludeChildren: true
UseLinuxPerf: true
LinuxPerfOptions {
  CollectCpuCtxswTrace: true
  SamplingPeriod: 1000000
  CollectIPBacktraceSamples: true
}

What is your command line? It looks like analysis is not triggering until the application has exited.

The command line is a full path to the target executable.
Our application doesn’t require any arguments.
I have orange exclamation marks but without any warning/error messages.


Hello davidLe,

Could you do the following to collect the log for further investigation?

  1. Save nvlog.config (491 Bytes) on the target side.
  2. Set the environment variable “NVLOG_CONFIG_FILE” to the path of “nvlog.config” in the GUI.
  3. Re-run the collection and share the file “/tmp/nsys-ui.log” that gets created.

Thanks!

Thanks for your help.
nsys-ui.log (83.5 KB)

The error messages show that CUPTI, the API we use to trace CUDA, could not be initialized because its version is not compatible with the driver version.

Could you please try to use Nsight Systems installed as part of the CUDA toolkit located in “/usr/local/cuda-[version]/nsight-systems-[version]”?

I don’t have the nsight-systems as a part of CUDA toolkit

Could you please try to install the latest driver and see if the problem persists?