Hi, I’m trying to get nsight working on manjaro linux. I’ve installed the cuda-tools package, ran ncu-ui, built my binary, and launched it through nsight, and although the application runs, it’s unable to attach. The progress log dialogue just gets stuck in this loop:
Preparing to launch...
Launched process: eng-gfx-consumer (pid: 206607)
Attempting to automatically connect...
Searching for attachable processes on localhost:49152-49215...
Searching for attachable processes on localhost:49152-49215...
Searching for attachable processes on localhost:49152-49215...
Searching for attachable processes on localhost:49152-49215...
Searching for attachable processes on localhost:49152-49215...
Searching for attachable processes on localhost:49152-49215...
Searching for attachable processes on localhost:49152-49215...
Can you check that one of these ports is accessible on your system for the current user? Alternatively, you can change the default port range in the UI’s options dialog (Tools > Options > Connection). During the first CUDA API call, the launched target process attempts to open a socket on one of these ports, and the host UI tries to find and connect to them.
It can also help to
double-check that your app is in fact running CUDA
try with a small CUDA sample application first, to debug if the issue is specific to your app, or general on your system
try to collect non-interactively through the Profile activity, or the ncu command line interface (and open the report written with the -o flag in the UI afterwards).
I managed to get it work, but only by launching the app in console (Administrator permissions) using ncu --mode launch app.exe and the Nsight Compute / Interactive Profile.
No way to launch it directly using Nsight Compute (localhost / Launch). Might be I do something wrong, but with Ubuntu my workflow works properly.
I’d prefer to launch and profile the app right away using Nsight Compute, i.s.o. starting an console first.
For my understanding … is it (theoretically) possible to launch and attach an app directly through Nsight Compute (2021.1.0.0) on Windows 10, or is it “normal” that I have to start the app first using the console ncu --mode launch command?
On Ubuntu, the launch and attach from Nsight Compute seems to work. On Windows 10 I didn’t manage.
It is certainly expected that you can launch and profile applications directly using both the ncu command line as well as the ncu-ui GUI with the Interactive Profile activity on Windows 10. This is the default option for the CLI flag --mode, named launch-and-attach.
In your first post, you mentioned this was on Manjaro Linux. Later you mentioned your Ubuntu workflow work properly, but your latest question is about a problem on Windows 10. Can you please clarify on which platform you are having the issue right now?
This is the line where it blocks (right at the begining of the code) HANDLE_ERROR( cudaMallocManaged((void**)&m_block , sizeof(uint64_t)) );
Whether I start using “Interactive Profile” or “Profile” … it doesn’t matter. It doesn’t go beyond the cudaMallocManaged() function.
When you say it blocks there, you mean it doesn’t even fully connect in interactive mode, right? I.e. it doesn’t show you the first CUDA API call in the API stream view tool window, like here ? Instead, it keeps showing the connection log that you mentioned in your first post?
I truly don’t see why it would behave like this, especially since you say that launching from the CLI and attaching from the UI works for you, which is essentially the same underneath. This seems to imply there aren’t any conceptual errors with how you application behaves that Nsight Compute can’t handle, and the relevant ports are accessible. Still, it would be good if you could try a small CUDA sample app as well, to see if that behaves better.
One thing that appears to be different is that you mentioned your ncu --mode launch command was run as administrator, but when trying to launch from the UI, the UI process wasn’t, correct? Can you check the following:
try profiling directly from the CLI in a non-admin console, i.e. ncu .
try profiling directly from the CLI in an admin console, i.e. ncu .
try launching interactively from the UI, when the UI itself is started as admin
I’m using UI Nsight Compute (2021.2.2.0) to connect from Windows 10 host to Linux CentOS 7 with using “Interactive Profiling” with user and/or admin permissions. The target program uses CUDA. I double-checked Windows firewall allows the port range connection as well as linux iptables is empty and allows anything. netcat server-client works well over 49152 TCP port.
Still, I get the persistent message:
Searching for attachable processes on :49152-49215…
Searching for attachable processes on :49152-49215…
After a few hours of research I found out what is the root cause. The environment Environment field should be set to “LD_PRELOAD=/tmp/var/target/linux-desktop-glibc_2_11_3-x64/libcuda-injection.so”
By default, when Environment not set, "bash -c" starts the Linux target with LD_PRELOAD="C:/Program Files/NVIDIA Corporation/Nsight Compute 2021.2.2/host/windows-desktop-win7-x64/~/tmp/var/target/linux-desktop-glibc_2_11_3-x64/libcuda-injection.so", which obviously doesn’t make sense on the Linux target.
Therefore remove interactive profiling from Windows host to Linux target doesn’t seem to work by default and requires fixing environment variable to a correct value.
We are not aware of such an issue. Interactive profiling from Windows 10 to CentOS7 should not require to set any special Environment values and we can’t reproduce this issue in local testing on such a setup. In general, setting LD_PRELOAD is not required for ncu at all, as this will be set internally.
Can you share your exact settings in the Interactive Profiling connection dialog?
The environment variables can be set only in the Remote Launch dialog. I use Launch.
Exact settings go as follows:
Application Executable: ./build/my_exe
Working directory: /home/akss/src/nvidia_project
Comman Line Arguments: -param1=foo -param2=bar
LD_PRELOAD=/home/UNIXHOME/akochepasov/tmp/var/target/linux-desktop-glibc_2_11_3-x64/libcuda-injection.so
Profiler settings attached
