I’m profiling a PyTorch model on a DGX Workstation that has four A100 GPUs using the Nvidia pytorch-22.03 container (Pytorch 1.12.0a0). nsys version 2021.5.2.52, CUDA 11.6 with driver version 510.47.03 with kernel driver version 470.103.01 come within the container.
On my MacOS host, I’m using the Nsight 2021.5.1 and also tried 2022.5.1. No luck. Can’t get any streams result. Here goes how I am executing nsys:
could you verify that the application creates streams, i.e. calls torch.cuda.Stream? With a sample pytorch app that creates streams, I can see that the information is presented on the GUI. Using the same docker image and nsys versions 2021.5.2 and 2023.1.1.
If the application does create streams, could you share the application or a part that reproduces the behavior?
Also, you could take advantage of the latest CLI version, 2023.1.1. That will provide bug fixes and more features.
Note, that version would help when used inside the docker, to collect the profile.
To view the profile you would need the same or a later version on your host.
Hello @ztasoulas
I’ve used an example that explicitly calls torch.cuda.Stream and I saw them in the nsight host. Please find attached the example and report.
Thanks for the help and heads up. streams.py (1.0 KB) streams_profile.nsys-rep (452.1 KB)
I’ve used an example that explicitly calls torch.cuda.Stream and I saw them in the nsight host. Please find attached the example and report.
Thanks for the help and heads up.
I’m glad this resolved the issue. The GUI will not mention anything about streams, if streams are not explicitly created in the source code.
Is there a way to install the latest CLI version 2023.1.1 on a Pytorch container (say 23.02 ) without being superuser ?
Sure, in this case you can download Nsight Systems 2023.1.1 (Linux Host .run Installer), then execute the script inside the container, you can use chmod +x NsightSystems-linux-public-2023.1.1.127-3236574.run to give executable permissions to the script. This does not require sudo priveleges to install, by default the tool will be installed at /home/<username>/nsight-systems-2023.1.1. You can cleanly uninstall it by deleting the aforementioned directory. The tool executable will be at <installation path>/nsight-systems-2023.1.1/bin/nsys.
If space is an issue, as the above option will install the GUI binaries as well, you can download Nsight Systems 2023.1.1 (Linux CLI only .deb Installer) and simply unpack it, dpkg -x NsightSystems-linux-cli-public-2023.1.1.127-3236574.deb <path to place the extracted artifacts>. The tool executable will be at <extraction path>/opt/nvidia/nsight-systems-cli/2023.1.1/bin/nsys.
The latter approach will install only the CLI tool.