Software Version
Nsight Systems 2023.4 for Linux Target
Nsight GUI for macOS host
Device
V100 and A100 GPUs.
Currently, in the Stats System View, it is not possible to get the number of bytes sent or received. Notice the empty column Bytes in the pictures below.
V100 Report
A100 Report
However, when hovering over the timeline of ncclSend , shown below, the the bytes are visible.
V100 Report
A100 Report
How can I get this byte transmitted data as a generated stats report?
Currently the stats script does not pull this information.
@jkreibich & @rdietrich is this information available in the sqlite and how can the stats script be customized here.
The NCCL API parameter information is not available in the SQLite export and, thus, cannot be pulled out by the stats scripts. There are plans to add this information to the export formats, but there is no ETA.
The stats report you’re looking at (CUDA GPU Trace) provides a trace of all CUDA Kernels and CUDA memory operations. This particular report knows nothing about what the CUDA Kernels are trying to do, and in specific knows nothing about the NCCL library. The Bytes, Throughput, and memory-kind columns are specific to the CUDA memory operations (memset and memcpy), and will never show data for rows displaying CUDA Kernels, including NCCL or MPI kernels.
As @rdietrich mentioned, at this time we don’t have any NCCL specific stats reports, because the full data is unavailable to the exports used by the stats report system.
NVTX events are integrated into the NCCL library, however, so some of the timing information is available in the export, even if the more API specific data (such as data size, rank, etc.) is not. Unfortunatly, that’s not a big help if you specifically want the bytes information.
Thank you @rdietrich and @jkreibich
I will close this issue since you both have clarified why the data is not available.
@rdietrich I will be looking forward to the next release that has this feature.
For motivation, I wanted to point out that the stats reports are exportable and therefore allow external analysis. In my case, I use Nsight for research and the exported data from stats is what I use for my plots.
The NCCL API view, on the other hand, to the best of my knowledge is not exportable, so I can see the data in the timelines but cannot analyze externally.
As I’m sure you’re aware, nsys export can be used to generate an SQLite file from an .nsys-rep file. That exported SQLite file is what nsys stats uses as input. The various stats reports are snippets of Python code that issue SQL queries to the SQLite export and generate the reports.
The disadvantage of this process is that data needs to be exported in order to be accessible to the stats system and, as seen in this case, the exports sometimes lag behind other data handling features. We try to keep the exports as up to date as possible, but there are sometimes delays with the more complex data formats.
The advantage of this process, and something that might be particularly useful for your situation, is that you can write your own stats reports. If you look in the Nsight install directory, you should find a .../reports/ directory full of Python modules. Each of those represents a stats report. You are free to copy/modify/edit/create your own report files, so that you can generate your own stats reports. The nsys stats command should be able to find a custom stats report in the current directory, or there are options to setup a personal directory of reports and let nsys know where to find it (see nsys help stats for more info). We don’t have a lot of documentation on how to write your own reports, but hopefully if you’re familiar with Python, you should be able to figure a lot of it out from the existing reports. If you can write your queries into a report file, then nsys stats can take care of all the formatting and export features you might want to use.
If that’s not quite good enough, you can also access the raw data in the SQLite file. If you’re using an analytic environment like Pandas inside Jupyter, it is pretty easy to import the SQLite data and play with it however you want. You can also, of course, just run SQL queries against the SQLite file. Most BI tools, such as Tableau, also understand how to read SQLite files. The Nsight Systems website has pretty good docs on the format of the SQLite files and what all the tables mean. If you dump the schema, we’ve also tried to include comments in the SQL to explain the meaning of the tables and columns.
If you get really stuck on anything, feel free to post a question to the forum and tag me on it. Good luck!
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.
