Hello,
I am trying to profile my hybrid code parallelized with MPI and OpenACC, using Nsight Systems.
I followed the documentation and finally make the program running but couldn’t collect the OpenACC trace data so far. I also tried with the newer version (nsys 4.1) though. Your help will be really appreciated. Thank you!
Yongsuk
By 4.1 do you mean 2023.4.1? (Just checking)
How long was the application running for?
Can you send me the CLI command (or the GUI project settings you used).
OS and archetecture information might be useful as well.
Hi hwilper,
Yes, I’m using ver. 2023.4.1.
The application was running till the end, as I specified.
Here are the GUI project settings:
The OS and architecture info is here:
I tried different compiling frags (-g, -pg, and/or -C) but in vain. :(
Thank you for your help!
Yongsuk
@yongsukjo93 In the Nsight Systems GUI, you just need to specify the application and its parameters In the text field below Command line with arguments. Please do not prefix nsys profile here.
If you are profiling via the command line interface, you would need to use the option openacc with the -t|--trace switch. The full command could look as follows: nsys profile -t openacc,nvtx,mpi,cuda mpirun -np 1 ./SPH.
Hi @rdietrich,
Thank you so much! I tried it with the CLI and it worked!
However, I tried the GUI version without using , but it gave me another error message. I thought I set the path and placed the executable in the current directory. Could you help me about this issue?
Full error information:
RuntimeError (120) {
RuntimeError (120) {
OriginalExceptionClass: N5boost10wrapexceptIN11QuadDCommon16RuntimeExceptionEEE
OriginalFile: /dvs/p4/build/sw/devtools/Agora/Rel/CUDA12.3/QuadD/Host/Analysis/Clients/AnalysisHelper/AnalysisStatus.cpp
OriginalLine: 79
OriginalFunction: static QuadDAnalysis::AnalysisHelper::AnalysisStatus::StatusInfo QuadDAnalysis::AnalysisHelper::AnalysisStatus::MakeFromErrorString(QuadDAnalysis::AnalysisHelper::AnalysisStatus::StatusType, QuadDAnalysis::AnalysisHelper::AnalysisStatus::ErrorType, const string&, const DevicePtr&)
ErrorText: Executable not found in current directory or standard search paths
}
}
Profiling options:
DeviceId: “Local”
EventTypes {
Items: CpuCycles
Items: OSRuntime
Items: Cuda
Items: NvtxEvents
Items: CuDNN
Items: CuBLAS
Items: OpenACC
}
HowToStart: Manual
HowToStop: Manual
DeviceType: Unix
DeviceDisplayName: “XPS-8960”
OSRuntimeOptions {
DurationThresholdNs: 1000
}
MPITraceOptions {
collectMpiTrace: true
}
Processes {
HowToAttach: LaunchAnother
Command: “mpirun”
Arguments: “-np”
Arguments: “1”
Arguments: “./SPH”
WorkingDirectory: “/home/yongcho/MPI_OpenACC_SPH”
UserName: “yongcho”
CollectNvtxTrace: true
CollectCudaTrace: true
EnvironmentVariables {
Name: “DESKTOP_SESSION”
Value: “ubuntu”
}
EnvironmentVariables {
Name: “DISPLAY”
Value: “:1”
}
EnvironmentVariables {
Name: “GDMSESSION”
Value: “ubuntu”
}
EnvironmentVariables {
Name: “GIO_LAUNCHED_DESKTOP_FILE”
Value: “/usr/share/applications/nsys-ui-2023.3.3.desktop”
}
EnvironmentVariables {
Name: “GIO_LAUNCHED_DESKTOP_FILE_PID”
Value: “147012”
}
EnvironmentVariables {
Name: “GJS_DEBUG_OUTPUT”
Value: “stderr”
}
EnvironmentVariables {
Name: “GJS_DEBUG_TOPICS”
Value: “JS ERROR;JS LOG”
}
EnvironmentVariables {
Name: “GNOME_DESKTOP_SESSION_ID”
Value: “this-is-deprecated”
}
EnvironmentVariables {
Name: “GNOME_SHELL_SESSION_MODE”
Value: “ubuntu”
}
EnvironmentVariables {
Name: “GPG_AGENT_INFO”
Value: “/run/user/1000/gnupg/S.gpg-agent:0:1”
}
EnvironmentVariables {
Name: “GTK_IM_MODULE”
Value: “ibus”
}
EnvironmentVariables {
Name: “GTK_MODULES”
Value: “gail:atk-bridge”
}
EnvironmentVariables {
Name: “HOME”
Value: “/home/yongcho”
}
EnvironmentVariables {
Name: “INVOCATION_ID”
Value: “5cae3aa7adb54054a5cd389394008163”
}
EnvironmentVariables {
Name: “JOURNAL_STREAM”
Value: “8:61653”
}
EnvironmentVariables {
Name: “LD_LIBRARY_PATH”
Value: “”
}
EnvironmentVariables {
Name: “LD_PRELOAD”
Value: “:{}”
}
EnvironmentVariables {
Name: “LOGNAME”
Value: “yongcho”
}
EnvironmentVariables {
Name: “MANAGERPID”
Value: “1834”
}
EnvironmentVariables {
Name: “NSYS_LD_LIBRARY_PATH”
Value: “”
}
EnvironmentVariables {
Name: “NSYS_QT_PLUGIN_PATH”
Value: “”
}
EnvironmentVariables {
Name: “NV_AGORA_CRASH_FD”
Value: “7”
}
EnvironmentVariables {
Name: “NV_AGORA_PATH”
Value: “/opt/nvidia/nsight-systems/2023.3.3/host-linux-x64”
}
EnvironmentVariables {
Name: “NV_QUADD_PATH”
Value: “/opt/nvidia/nsight-systems/2023.3.3/host-linux-x64/nsys-ui.bin”
}
EnvironmentVariables {
Name: “PATH”
Value: “/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin:/opt/nvidia/hpc_sdk/Linux_x86_64/2023//compilers/bin/:/opt/nvidia/hpc_sdk/Linux_x86_64/2023//comm_libs/openmpi4/bin/:/opt/nvidia/nsight-systems/2023.3.3/target-linux-x64/:/opt/nvidia/nsight-systems/2023.3.3/host-linux-x64/:/opt/nvidia/nsight-systems/2023.3.3/bin/\r\n”
}
EnvironmentVariables {
Name: “PWD”
Value: “/home/yongcho”
}
EnvironmentVariables {
Name: “QT_ACCESSIBILITY”
Value: “1”
}
EnvironmentVariables {
Name: “QT_IM_MODULE”
Value: “ibus”
}
EnvironmentVariables {
Name: “QT_PLUGIN_PATH”
Value: “”
}
EnvironmentVariables {
Name: “QUADD_INSTALL_DIR”
Value: “/opt/nvidia/nsight-systems/2023.3.3/target-linux-x64”
}
EnvironmentVariables {
Name: “SESSION_MANAGER”
Value: “local/yongcho-XPS-8960:@/tmp/.ICE-unix/2159,unix/yongcho-XPS-8960:/tmp/.ICE-unix/2159”
}
EnvironmentVariables {
Name: “SHELL”
Value: “/bin/bash”
}
EnvironmentVariables {
Name: “SHLVL”
Value: “1”
}
EnvironmentVariables {
Name: “SSH_AGENT_LAUNCHER”
Value: “gnome-keyring”
}
EnvironmentVariables {
Name: “SSH_AUTH_SOCK”
Value: “/run/user/1000/keyring/ssh”
}
EnvironmentVariables {
Name: “SYSTEMD_EXEC_PID”
Value: “2178”
}
EnvironmentVariables {
Name: “USER”
Value: “yongcho”
}
EnvironmentVariables {
Name: “USERNAME”
Value: “yongcho”
}
EnvironmentVariables {
Name: “WINDOWPATH”
Value: “2”
}
EnvironmentVariables {
Name: “XAUTHORITY”
Value: “/run/user/1000/gdm/Xauthority”
}
EnvironmentVariables {
Name: “XDG_CONFIG_DIRS”
Value: “/etc/xdg/xdg-ubuntu:/etc/xdg”
}
EnvironmentVariables {
Name: “XDG_CURRENT_DESKTOP”
Value: “ubuntu:GNOME”
}
EnvironmentVariables {
Name: “XDG_DATA_DIRS”
Value: “/usr/share/ubuntu:/usr/share/gnome:/usr/local/share/:/usr/share/:/var/lib/snapd/desktop”
}
EnvironmentVariables {
Name: “XDG_MENU_PREFIX”
Value: “gnome-”
}
EnvironmentVariables {
Name: “XDG_RUNTIME_DIR”
Value: “/run/user/1000”
}
EnvironmentVariables {
Name: “XDG_SESSION_CLASS”
Value: “user”
}
EnvironmentVariables {
Name: “XDG_SESSION_DESKTOP”
Value: “ubuntu”
}
EnvironmentVariables {
Name: “XDG_SESSION_TYPE”
Value: “x11”
}
EnvironmentVariables {
Name: “_”
Value: “/opt/nvidia/nsight-systems/2023.3.3/host-linux-x64/CrashReporter”
}
CudaFlushPeriodically: true
CudaFlushPeriod: 7000000000
CudaSkipSomeApiCalls: false
CollectGPUMemoryUsage: true
CollectUvmCpuPageFaults: true
CUDABacktraceOptions {
CollectOtherBacktraces: true
OtherBacktracesThresholdNs: 80000
CollectKernelBacktraces: true
KernelBacktracesThresholdNs: 80000
CollectMemoryBacktraces: true
MemoryBacktracesThresholdNs: 80000
CollectSynchronizationBacktraces: true
SynchronizationBacktracesThresholdNs: 80000
CollectPythonBacktrace: false
}
CollectUvmGpuPageFaults: true
CudaGraphTraceOptions {
Mode: Graph
}
CudaFlushOnCudaProfilerStop: true
}
ShowBacktrace: true
UseDWARF: true
IncludeChildren: true
UseLinuxPerf: true
LinuxPerfOptions {
CollectCpuCtxswTrace: true
SamplingPeriod: 1000000
CollectIPBacktraceSamples: true
SamplesPerBacktrace: 4
Mode: ProcessTree
cpuIPSamplingTriggerEventIndex: 0
triggerType: Hardware
}
SymbolResolutionOptions {
ResolveSymbols: true
}
NetworkProfilingOptions {
ShouldCollectNicMetrics: false
}
Looks like mpirun cannot be found. Two options:
mpirun in the text field: /opt/nvidia/hpc_sdk/Linux_x86_64/2023/comm_libs/openmpi4/bin/mpirun -n 1 ./SPHPATH before starting the Nsight Systems GUI (export PATH=/opt/nvidia/hpc_sdk/Linux_x86_64/2023/comm_libs/openmpi4/bin:/$PATH)I’ll check why setting PATH via the Environment variables box in the GUI doesn’t work.
Hi rdietrich,
Perfect! Now, it works on the GUI as well. Problem resolved!
Yeah, but I thought setting the PATH via the Environment variables box in the GUI would work.
Thank you so much!
Yongsuk