Connection to Agent lost for latest nsys releases. Application never starts

Developer Tools Nsight Systems Profiling Linux Targets
1

I encounter an error when launching a profile session with “nsys profile” with the CLI:

Connection to Agent lost. This is most likely a bug. Internal reason: ‘End of file’.
Please refer to the troubleshooting section of the docs:
https://docs.nvidia.com/nsight-systems/UserGuide/index.html#troubleshooting

This is all the output that I get, nothing about the application itself. This happens with the most simplest profiling configuration, like:

nsys profile -t cuda --sample=none --cpuctxsw=none   ./mm

The thing is that this only happens for newer versions of Nsight Systems. With the same system and environment, running the program with Nsight Systems version 2025.1.1 works and outputs a good trace. Instead, versions 2025.6.1 and above produce the mentioned error. I don’t know where exactly the breaking point is between 2025.1 and 2025.6.

The only obvious thing I checked is the environment. I don’t know if there are specific versions of the SDK or CUDA that you need for each version of nsys, but as far as I know, nsys is not tied to any CUDA version. Still, the tests I have done were all under cuda 12.6 and cuda 12.8. The environment is pretty controlled on my machine (software installed via module files), so I don’t think there could be other libraries interfering. However, it is true that on the same system I can run any arbitrary late version of Nsight Systems when executing inside Singularity containers. One of the containers used, to give an example, is based on nvidia/cuda:12.6.3-cudnn-devel-ubuntu24.04, so nothing very complicated, and the Nsight Systems binary used was installed on the host and used inside the container via a binding of the filesystem.

My system is a Red Hat Enterprise Linux 9.2 on Intel x86 CPUs, and I have H100 GPUs.

I could not dig any deeper, as the error message is not very informative. Given this, I assume this is some kind of unexpected bug and probably a hand from somebody from NVIDIA or somebody that has encountered this before will ease a lot.

Thanks!

Marc

2

@sanjivanis can you look into this.

3

Hi @marc.clasca , could you try running
nsys profile -t cuda-sw --sample=none --cpuctxsw=none ./mm
instead? I suspect this might be an error related to hardware trace feature that was made default in Nsys in 2025.6. Also, please share if there are any errors or warnings in the Diagnostic report section in the Nsys GUI.

4

Hey @sanjivanis, thanks for your fast response!

Unfortunately, I tried this and got the exact same behavior. :(

Regarding the Diagnostic report from the GUI, maybe I don’t get what you mean. With my current setup, I launch the profile with the CLI so the GUI never intervenes, and I never get a report back when I get this error, so I cannot open anything.

Please tell me if you want me to:

  1. Provide the report for one of the profiling sessions with one of the nsys versions that works; or
  2. Try launching the profiling sessions with the GUI and tell you what’s there (I never did this).

Marc

5

Thanks, Marc. Since cuda-sw fails identically, this does not appear to be caused by hardware-based CUDA tracing. And you are correct, because no report is generated, there is no Diagnostics section to inspect. There is no need to try the GUI or provide a successful report right now.

Could you please provide:

nsys --version
readlink -f "$(command -v nsys)"
nvidia-smi
uname -a
ldd --version | head -1
getenforce

Please run these two tests with the failing Nsight Systems version:

nsys profile --trace=none --sample=none --cpuctxsw=none /bin/true
nsys profile --trace=none --sample=none --cpuctxsw=none ./mm

These will tell us whether the agent fails during basic startup/application launch, before CUDA tracing is involved.

Most importantly, could you collect the verbose agent log? In the failing installation’s target-linux-x64 directory:

  1. Copy nvlog.config.template to nvlog.config.

  2. Change:

    $ nsys-ui.log

    to:

    $ nsys-agent.log

  3. Reproduce the failure and attach the resulting nsys-agent.log.

Immediately after reproducing, please also run:

coredumpctl --since "10 minutes ago" list
journalctl -k --since "10 minutes ago" | grep -Ei 'nsys|segfault|avc|denied'

The fact that the same Nsight Systems installation works inside Singularity but fails directly on the RHEL host makes me think it is a host userspace, library-loading, or security-policy difference. The agent log should give us more info.