Nsys Failed to initialize client to server communication. Aborting

nsys profile python res50.py

Raise Error

Warning: LBR backtrace method is not supported on this platform. DWARF backtrace method will be used.
Failed to initialize client to server communication. Aborting.
/build/agent/work/20a3cfcd1c25021d/QuadD/Target/quadd_d/quadd_d/jni/CLI/CLIInteractiveExecute.cpp(2201): Throw in function void QuadDDaemon::CLIInteractiveImpl::InitServer(const CLICommandParserPtr&, const AgentControlConfigPtr&, const AnalysisStartOptionsPtr&)
Dynamic exception type: boost::wrapexcept<QuadDCommon::NotInitializedException>
std::exception::what: NotInitializedException
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Sun_Aug_15_21:14:11_PDT_2021
Cuda compilation tools, release 11.4, V11.4.120
Build cuda_11.4.r11.4/compiler.30300941_0
nsys status -e
Timestamp counter supported: Yes
Sampling Environment Check
Linux Kernel Paranoid Level = 2: OK
Linux Distribution = CentOS
Linux Kernel Version = 3.10.0-1160.42.2.el7.x86_64: OK
Linux perf_event_open syscall available: OK
Sampling trigger event available: OK
Intel(c) Last Branch Record support: Not Available
Sampling Environment: OK

Has anyone met this before?
I could run nsighy system 2 days ago, I didn’t install anything and don’t change my env variables.

Best
Max

Can you do a quick check and see if there are any other Nsys processes running on the system? Sometimes we see something like this when there are “zombie” processes.

There is no other running Nsys process.
I also tried ‘reboot’, it still didn’t work.

@skottapalli for further investigation.

@tianyu9748 - could you check if the directory /tmp/nvidia/nsight-systems is writable? Could you share the output of ls -l /tmp/nvidia/nsight-systems and ls -l /tmp/nvidia?

Could you try removing the /tmp/nvida directory and try profiling again?

ls -l /tmp/nvidia/nsight-systems the return is: ls: cannot access '/tmp/nvidia/nsight-systems': No such file or directory

ls -l /tmp/nvidia the return is: ls: cannot access '/tmp/nvidia': No such file or directory

I have changed TMPDIR to /home/tianyu/tmp, does this influence NCU?

Best
Tianyu

Which version of nsys are you using? Do you have permissions to read/write to the path $TMPDIR?

Nsys process become zombie and its not waiting for inference to be done .without using nsys my application do the inference but after enabling the nsys profile my application is not executing properly
And i could see nsys process are in defunc state

@rajesh.k1 can you give us a few more details on what you are doing?