"The timeout expired." with Nsight Systems 2023.2.1

Developer Tools Nsight Systems Profiling Linux Targets
1

I get the following error with 2023.2.1

├ nsys profile sleep 1
The timeout expired.

However 2023.1.1 works correctly

├ nsys profile sleep 1
Generating '/tmp/nsys-report-772f.qdstrm'
[1/1] [========================100%] report6.nsys-rep
Generated:
    /central/home/spjbyrne/report6.nsys-rep

Unfortunately I don’t have sudo access, so I can’t use nvidia-persistenced. If I try to run it I get

├ nvidia-persistenced
nvidia-persistenced failed to initialize. Check syslog for more details.
2

Would I be correct in assuming that “nsys profile sleep 1” is profiling a one second sleep?

My guess would be that there has been a performance change in the set up code that causing that to not be a sufficient time interval for sleep. Can you try with a longer sleep?

3

No, that doesn’t seem to help. In fact the time it takes seems to be independent of the length of the sleep

├ time nsys profile sleep 1
The timeout expired.

real	0m30.219s
user	0m0.096s
sys	0m0.122s

├ time nsys profile sleep 60
The timeout expired.

real	0m30.214s
user	0m0.093s
sys	0m0.106s

Somewhat strangely, we only see it on certain nodes. And on some nodes we actually have the opposite (it works with 2023.2.1, but times out with 2023.1.1).