We use nsys commands to profile our application(very large), our commands like:
nsys launch ./application
nsys start
nsys stop
When nsys start begin, the application subtask collapsed and got coredump.
The application run well alone. Anyone know why? and how to debug.
Can you tell me a little more about your system?
What is the OS version, what version of Nsight System?
What are the results if you call “nsys status -e”?
OS is ubuntu 20.04,
‘nsys status -e’ runs okay
I tried again. It seems a lot of data missing Like:
So now two phenomena appear accidentally:
First, my application has two processes. After nsys start starts running, one of the processes will crash, while the other process can be collected by nsys normally.
Second, both processes can collect data, but the collected data is abnormal and there are a large number of blank areas, as shown in the figure.
What version of Nsys, and can you show me the output of -e?
Can you try with
“nsys launch --trace=nvtx,cuda ./application”
That will run the same options as usual except without the OS runtime trace.
Nsys version:
NVIDIA Nsight Systems version 2023.4.1.97-234133557503v0
My commad is :
“nsys launch --trace=nvtx,cuda,cudnn,cuda ./application”
Without OS runtime trace, but it still failed
@liuyis can you take a look at this?
@herrywangyi163 Could you try the last version and see if there’s any change? It can be downloaded from Nsight Systems - Get Started | NVIDIA Developer. Currently the latest is 2024.2, but 2024.3 is expected to be out perry soon (probably within this week)
Regarding the “Missing Data” on the timeline - how long was the profiling session? And did you change the GPU metrics sampling frequency? It might be related to the amount of sampled data, if it exceeded the buffer size before Nsys has a chance to process, some will be lost.
Is it possible to share the report file and the full Nsys commands you used?
