I have been using Nsight Systems for profiling my application, but recently I’ve encountered a problem where I get frequent “Missing Data” segments in my profiling timeline, which didn’t occur before. I’ve attached a screenshot for reference:
Details:
GPU: NVIDIA RTX 6000
Nsight Systems Version: Version 2023.2.2.0
OS: Ubuntu 22.04.3 LTS
Application: Python-based workload
Steps Taken So Far:
Updated Drivers and Nsight Systems: Ensured both are the latest versions.
Checked System Resources: Verified that there is no significant CPU, memory, or disk I/O contention during profiling.
Despite these steps, the issue persists. The frequent “Missing Data” makes it challenging to analyze the performance of my application effectively.
what can be the solution or problem in my situation?
Any insights or suggestions would be greatly appreciated.
When was the last time you didn’t encounter this issue? What changed since then - Nsys, GPU driver, the workload? Can share a minimal workload that reproduces the issue?
2023.2.2.0 is not the latest Nsys version, but I believe using the latest one won’t be enough.
A week ago, I ran the summarization stage using Mamba’s inferenc), and it worked fine.
However, this week I tried running the generation stage and encountered an error.
Now, even when I run the summarization stage again, the problem persists.
It seems like there might be an issue with the sampling buffer or some leftover data somewhere. Is there a way to check for this?
