Nsight Systems Issue: Unable to configure the collection of CPU IP samples

davidLe · November 2, 2021, 2:29pm

I get the following issue when trying to profile my program with Nsight Systems (which apparently eliminate Nsight Systems from collecting CUDA events for profiling):
“Source: Daemon
Description: Unable to configure the collection of CPU IP samples, backtraces, and/or scheduling data. Try the ‘nsys status --environment’ command to learn more.”

This is the output of command: “nsys status --environment”:
Timestamp counter supported: Yes
Sampling Environment Check
Linux Kernel Paranoid Level = 4: OK
Linux Distribution = Ubuntu
Linux Kernel Version = 5.11.0-38-generic: OK
Linux perf_event_open syscall available: Fail
Sampling trigger event available: Fail
Intel(c) Last Branch Record support: Not Available
Sampling Environment: Fail

GPU: RTX A6000
OS: Ubuntu 20.04
Driver version: 470.57.02
CUDA version: 11.4 (update 2)
Nsight Systems version: 2021.4.1

Any help is appreciated

hwilper · November 2, 2021, 2:47pm

In order to collect CPU samples, Nsight Systems uses the linux perf functionality under the covers. This functionality is not available when Paranoid is set >2.

(from the documentation)

Requirements for x86_64, Power, and ARM SBSA Targets on Linux

Use of Linux Perf : To collect thread scheduling data and IP (instruction pointer) samples, the Perf paranoid level on the target system must be 2 or less. Use the following command to check:

cat /proc/sys/kernel/perf_event_paranoid

If the output is >2, then do the following to temporarily adjust the paranoid level (note that this has to be done after each reboot):

sudo sh -c ‘echo 2 >/proc/sys/kernel/perf_event_paranoid’

To make the change permanent, use the following command:

sudo sh -c ‘echo kernel.perf_event_paranoid=2 > /etc/sysctl.d/local.conf’

hwilper · November 2, 2021, 2:48pm

If you are running in a container see - User Guide :: Nsight Systems Documentation

davidLe · November 3, 2021, 7:52am

Thank you.
I had to decrease it to level 1 otherwise it won’t start.
Now I have the following issue:

Failed to start profiling. The target application <path to target application>

ApplicationExitedBeforeProfilingStarted (1104) {
    RuntimeError (120) {
        OriginalExceptionClass: N5boost16exception_detail10clone_implIN11QuadDCommon16RuntimeExceptionEEE
        OriginalFile: /build/agent/work/20a3cfcd1c25021d/QuadD/Host/Analysis/Clients/AnalysisHelper/AnalysisStatus.cpp
        OriginalLine: 79
        OriginalFunction: static QuadDAnalysis::AnalysisHelper::AnalysisStatus::StatusInfo QuadDAnalysis::AnalysisHelper::AnalysisStatus::MakeFromErrorString(QuadDAnalysis::AnalysisHelper::AnalysisStatus::StatusType, QuadDAnalysis::AnalysisHelper::AnalysisStatus::ErrorType, const string&, const DevicePtr&)
        ErrorText: The target application exited before profiling started.
    }
}

Full error information:
StopAnalysisError (4106) {
    RuntimeError (120) {
        OriginalExceptionClass: N5boost16exception_detail10clone_implIN11QuadDCommon16RuntimeExceptionEEE
        OriginalFile: /build/agent/work/20a3cfcd1c25021d/QuadD/Host/Analysis/Clients/AnalysisHelper/AnalysisStatus.cpp
        OriginalLine: 79
        OriginalFunction: static QuadDAnalysis::AnalysisHelper::AnalysisStatus::StatusInfo QuadDAnalysis::AnalysisHelper::AnalysisStatus::MakeFromErrorString(QuadDAnalysis::AnalysisHelper::AnalysisStatus::StatusType, QuadDAnalysis::AnalysisHelper::AnalysisStatus::ErrorType, const string&, const DevicePtr&)
        ErrorText: Analysis failed.
    }
}

Profiling options:
DeviceId: "Local"
EventTypes {
  Items: CpuCycles
  Items: OSRuntime
  Items: Cuda
}
HowToStart: Delayed
HowToStop: Delayed
DeviceType: Unix
DeviceDisplayName: "MyPC"
OSRuntimeOptions {
  DurationThresholdNs: 1000
}

Processes {
  HowToAttach: LaunchAnother
  Command: "/home/workspace/program/CMake/x64/Release/programExecute"
  WorkingDirectory: "/home/workspace/program/CMake/x64/Release/"
  UserName: "david"
  CollectCudaTrace: true
  EnvironmentVariables {
    Name: "DESKTOP_SESSION"
    Value: "ubuntu"
  }
  EnvironmentVariables {
    Name: "GDMSESSION"
    Value: "ubuntu"
  }
  EnvironmentVariables {
    Name: "GIO_LAUNCHED_DESKTOP_FILE"
    Value: "/usr/share/applications/nsys-ui-2021.4.1.desktop"
  }
  EnvironmentVariables {
    Name: "GIO_LAUNCHED_DESKTOP_FILE_PID"
    Value: "87580"
  }
  EnvironmentVariables {
    Name: "GJS_DEBUG_OUTPUT"
    Value: "stderr"
  }
  EnvironmentVariables {
    Name: "GJS_DEBUG_TOPICS"
    Value: "JS ERROR;JS LOG"
  }
  EnvironmentVariables {
    Name: "GNOME_DESKTOP_SESSION_ID"
    Value: "this-is-deprecated"
  }
  EnvironmentVariables {
    Name: "GNOME_SHELL_SESSION_MODE"
    Value: "ubuntu"
  }
  EnvironmentVariables {
    Name: "GPG_AGENT_INFO"
    Value: "/run/user/1000/gnupg/S.gpg-agent:0:1"
  }
  EnvironmentVariables {
    Name: "GTK_MODULES"
    Value: "gail:atk-bridge"
  }
  EnvironmentVariables {
    Name: "HOME"
    Value: "/home/david"
  }
  EnvironmentVariables {
    Name: "IM_CONFIG_PHASE"
    Value: "1"
  }
  EnvironmentVariables {
    Name: "INVOCATION_ID"
    Value: "d57d37a06db44f53ad952c48ea075210"
  }
  EnvironmentVariables {
    Name: "JOURNAL_STREAM"
    Value: "8:52902"
  }
  EnvironmentVariables {
    Name: "LANGUAGE"
    Value: "en_IL:en"
  }
  EnvironmentVariables {
    Name: "LD_LIBRARY_PATH"
    Value: "/usr/local/cuda/lib64:/opt/nvidia/nsight-systems/2021.4.1/target-linux-x64"
  }
  EnvironmentVariables {
    Name: "LOGNAME"
    Value: "david"
  }
  EnvironmentVariables {
    Name: "MANAGERPID"
    Value: "1881"
  }
  EnvironmentVariables {
    Name: "NV_AGORA_CRASH_FD"
    Value: "6"
  }
  EnvironmentVariables {
    Name: "NV_AGORA_PATH"
    Value: "/opt/nvidia/nsight-systems/2021.4.1/host-linux-x64"
  }
  EnvironmentVariables {
    Name: "NV_QUADD_PATH"
    Value: "/opt/nvidia/nsight-systems/2021.4.1/host-linux-x64/nsys-ui.bin"
  }
  EnvironmentVariables {
    Name: "PATH"
    Value: "/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin"
  }
  EnvironmentVariables {
    Name: "PWD"
    Value: "/home/david"
  }
  EnvironmentVariables {
    Name: "QT_ACCESSIBILITY"
    Value: "1"
  }
  EnvironmentVariables {
    Name: "QT_DPI_ADJUSTMENT_POLICY"
    Value: "AdjustDpi"
  }
  EnvironmentVariables {
    Name: "QT_IM_MODULE"
    Value: "ibus"
  }
  EnvironmentVariables {
    Name: "QT_PLUGIN_PATH"
    Value: "/opt/nvidia/nsight-systems/2021.4.1/host-linux-x64/Plugins:"
  }
  EnvironmentVariables {
    Name: "QUADD_INSTALL_DIR"
    Value: "/opt/nvidia/nsight-systems/2021.4.1/target-linux-x64"
  }
  EnvironmentVariables {
    Name: "SESSION_MANAGER"
    Value: "local/MyPC:@/tmp/.ICE-unix/2102,unix/MyPC:/tmp/.ICE-unix/2102"
  }
  EnvironmentVariables {
    Name: "SHELL"
    Value: "/bin/bash"
  }
  EnvironmentVariables {
    Name: "SHLVL"
    Value: "1"
  }
  EnvironmentVariables {
    Name: "SP_LD_LIBRARY_PATH"
    Value: "/usr/local/cuda/lib64"
  }
  EnvironmentVariables {
    Name: "SSH_AGENT_PID"
    Value: "2067"
  }
  EnvironmentVariables {
    Name: "SSH_AUTH_SOCK"
    Value: "/run/user/1000/keyring/ssh"
  }
  EnvironmentVariables {
    Name: "TSP_LD_LIBRARY_PATH"
    Value: "/usr/local/cuda/lib64"
  }
  EnvironmentVariables {
    Name: "USER"
    Value: "david"
  }
  EnvironmentVariables {
    Name: "USERNAME"
    Value: "david"
  }
  EnvironmentVariables {
    Name: "WINDOWPATH"
    Value: "2"
  }
  EnvironmentVariables {
    Name: "XDG_CONFIG_DIRS"
    Value: "/etc/xdg/xdg-ubuntu:/etc/xdg"
  }
  EnvironmentVariables {
    Name: "XDG_CURRENT_DESKTOP"
    Value: "ubuntu:GNOME"
  }
  EnvironmentVariables {
    Name: "XDG_DATA_DIRS"
    Value: "/usr/share/ubuntu:/usr/local/share/:/usr/share/:/var/lib/snapd/desktop"
  }
  EnvironmentVariables {
    Name: "XDG_MENU_PREFIX"
    Value: "gnome-"
  }
  EnvironmentVariables {
    Name: "XDG_RUNTIME_DIR"
    Value: "/run/user/1000"
  }
  EnvironmentVariables {
    Name: "XDG_SESSION_CLASS"
    Value: "user"
  }
  EnvironmentVariables {
    Name: "XDG_SESSION_DESKTOP"
    Value: "ubuntu"
  }
  EnvironmentVariables {
    Name: "XDG_SESSION_TYPE"
    Value: "x11"
  }
  EnvironmentVariables {
    Name: "_"
    Value: "/opt/nvidia/nsight-systems/2021.4.1/host-linux-x64/CrashReporter"
  }
  CudaFlushPeriodically: true
  CudaFlushPeriod: 10000000000
  CudaSkipSomeApiCalls: true
  CollectGPUMemoryUsage: false
}
ShowBacktrace: true
StartDelayMilliseconds: 10000
DurationMilliseconds: 30000
UseDWARF: true
IncludeChildren: true
UseLinuxPerf: true
LinuxPerfOptions {
  CollectCpuCtxswTrace: true
  SamplingPeriod: 1000000
  CollectIPBacktraceSamples: true
}

hwilper · November 3, 2021, 7:24pm

What is your command line? It looks like analysis is not triggering until the application has exited.

davidLe · November 9, 2021, 3:08pm

The command line is a full path to the target executable.
Our application doesn’t require any arguments.
I have orange exclamation marks but without any warning/error messages.

jyi · November 10, 2021, 1:34am

Hello davidLe,

Could you do the following to collect the log for further investigation?

Save nvlog.config (491 Bytes) on the target side.
Set the environment variable “NVLOG_CONFIG_FILE” to the path of “nvlog.config” in the GUI.

image1858×722 116 KB
Re-run the collection and share the file “/tmp/nsys-ui.log” that gets created.

Thanks!

davidLe · November 10, 2021, 8:31am

Thanks for your help.
nsys-ui.log (83.5 KB)

jyi · November 11, 2021, 1:22am

The error messages show that CUPTI, the API we use to trace CUDA, could not be initialized because its version is not compatible with the driver version.

Could you please try to use Nsight Systems installed as part of the CUDA toolkit located in “/usr/local/cuda-[version]/nsight-systems-[version]”?

davidLe · November 17, 2021, 10:00am

I don’t have the nsight-systems as a part of CUDA toolkit

jyi · November 17, 2021, 7:48pm

Could you please try to install the latest driver and see if the problem persists?

davidLe · December 13, 2021, 1:56pm

Using the Nsight systems installed with the CUDA toolkit solved my issue.
Thanks!

system · December 27, 2021, 1:57pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.