Nsight Systems fails to start profiling on Win 10 x64 with runtime target side error

Hi,

I am having trouble getting the latest version of Nsight Systems (2023.1.1) to work on my Windows 10 x64 system. The error I am getting is as follows and appears to be related to permissions:

RuntimeError (120) {
    TargetSideError (1100) {
        ErrorText: C:\dvs\p4\build\sw\devtools\Agora\Rel\QuadD_Main\QuadD\Common\ProcessResumer\Windows\ProcessResumerImpl.cpp(45): Throw in function __cdecl QuadDCommon::ProcessResumer::WindowsProcessResumer::WindowsProcessResumer(class QuadDCommon::StrongType<class QuadDCommon::LimitedNumber<unsigned int,4294967295>,class QuadDCommon::ProcessIdTag>)
            Dynamic exception type: struct boost::wrapexcept<class QuadDCommon::SystemException>
            std::exception::what: SystemException
            [struct QuadDCommon::tag_errno_text * __ptr64] = Access is denied
            [struct QuadDCommon::tag_message * __ptr64] = Failed to open thread 19884 from process 10604
            [struct QuadDCommon::tag_winapi_error_code * __ptr64] = 5
            [struct boost::errinfo_api_function_ * __ptr64] = OpenThread
            
        ServiceName: AnalysisService
        MethodName: Init
        SystemError (124) {
            ErrorText: Failed to open thread 19884 from process 10604
            ApiFunction: OpenThread
            OriginalFile: C:\dvs\p4\build\sw\devtools\Agora\Rel\QuadD_Main\QuadD\Common\ProcessResumer\Windows\ProcessResumerImpl.cpp
            OriginalLine: 45
            OriginalFunction: __cdecl QuadDCommon::ProcessResumer::WindowsProcessResumer::WindowsProcessResumer(class QuadDCommon::StrongType<class QuadDCommon::LimitedNumber<unsigned int,4294967295>,class QuadDCommon::ProcessIdTag>)
            OriginalExceptionClass: struct boost::wrapexcept<class QuadDCommon::SystemException>
        }
    }
}

I updated to the latest driver and tried running as administrator but no difference. Interestingly, I can get the 2020.3.4 version to start profiling, but this now complains about a driver mismatch and fails to collect any CUDA events. I would appreciate any insight into what could be going on here.

@dofek can you respond to this?

I have the same error, did you manage to fix this?

hi @rinkertt

I’ll ping Doron on this again, but as he is in Israel, he will not be responding today. Meanwhile, can you give me some details on the OS version and Nsys version you are using?

Are you running from the CLI or from the GUI? If CLI, can you give me the exact command line you used?

@dofek

Thanks @hwilper. To answer your question @rinkertt, no the issue is still ongoing. It’s been a couple of months since I last tried to get this working, I can still use Nsight Compute so managing with that for now.

Additional details:

  • OS: Windows 10 Enterprise 22H2
  • Nsight Systems 2023.1.1 GUI
  • This is a company managed system, my regular user account does not have admin rights but I have a separate admin account that usually gives me privileges when required.

Thanks,

Will

This is indeed a permissions issue. nsys.exe expects to have permission to launch target processes and suspend/resume their threads. This is why it requires administrative privileges to run. Running from an admin account should provide such permissions. Is there anything special about the target process that nsys.exe is trying to launch, or its location?

Doron

Similar situation for me as @will_thompson2000

  • Windows 10 Enterprise LTSC 21H2
  • Nsight Systems 2023.2.1 and 2022.4.2
  • I use the GUI, when selecting the local PC as target, I get a UAC prompt, and proceed with a localadmin account.
  • GPU: RTX A4500

Thanks @dofek, some additional info on the target process:
I’m trying to profile a C++ program that I can from Matlab using MEX. So I start a Matlab session without GUI, and run the script that will call the MEX routine with GPU accelerated code. So the target application with arguments is

"C:\Program Files\MATLAB\R2021b\bin\win64\matlab.exe" -wait -nojvm -nosplash -r "test_gpu_bf_large_blocks_threaded"

and the working directory is the location of the script and mex file. This worked fine in the past on a different pc, and works for Nsight Compute. Nsight Systems cannot start Matlab. When I press start I immediately get:

RuntimeError (120) {
    TargetSideError (1100) {
        ErrorText: C:\dvs\p4\build\sw\devtools\Agora\Rel\QuadD_Main\QuadD\Common\ProcessResumer\Windows\ProcessResumerImpl.cpp(45): Throw in function __cdecl QuadDCommon::ProcessResumer::WindowsProcessResumer::WindowsProcessResumer(class QuadDCommon::StrongType<class QuadDCommon::LimitedNumber<unsigned int,4294967295>,class QuadDCommon::ProcessIdTag>)
            Dynamic exception type: struct boost::wrapexcept<class QuadDCommon::SystemException>
            std::exception::what: SystemException
            [struct QuadDCommon::tag_errno_text * __ptr64] = Access is denied
            [struct QuadDCommon::tag_message * __ptr64] = Failed to open thread 17616 from process 17612
            [struct QuadDCommon::tag_winapi_error_code * __ptr64] = 5
            [struct boost::errinfo_api_function_ * __ptr64] = OpenThread
            
        ServiceName: AnalysisService
        MethodName: Init
        SystemError (124) {
            ErrorText: Failed to open thread 17616 from process 17612
            ApiFunction: OpenThread
            OriginalFile: C:\dvs\p4\build\sw\devtools\Agora\Rel\QuadD_Main\QuadD\Common\ProcessResumer\Windows\ProcessResumerImpl.cpp
            OriginalLine: 45
            OriginalFunction: __cdecl QuadDCommon::ProcessResumer::WindowsProcessResumer::WindowsProcessResumer(class QuadDCommon::StrongType<class QuadDCommon::LimitedNumber<unsigned int,4294967295>,class QuadDCommon::ProcessIdTag>)
            OriginalExceptionClass: struct boost::wrapexcept<class QuadDCommon::SystemException>
        }
    }
}
Full error information:
StartAnalysisError (4105) {
    TargetSideError (1100) {
        ErrorText: C:\dvs\p4\build\sw\devtools\Agora\Rel\QuadD_Main\QuadD\Common\ProcessResumer\Windows\ProcessResumerImpl.cpp(45): Throw in function __cdecl QuadDCommon::ProcessResumer::WindowsProcessResumer::WindowsProcessResumer(class QuadDCommon::StrongType<class QuadDCommon::LimitedNumber<unsigned int,4294967295>,class QuadDCommon::ProcessIdTag>)
            Dynamic exception type: struct boost::wrapexcept<class QuadDCommon::SystemException>
            std::exception::what: SystemException
            [struct QuadDCommon::tag_errno_text * __ptr64] = Access is denied
            [struct QuadDCommon::tag_message * __ptr64] = Failed to open thread 17616 from process 17612
            [struct QuadDCommon::tag_winapi_error_code * __ptr64] = 5
            [struct boost::errinfo_api_function_ * __ptr64] = OpenThread
            
        ServiceName: AnalysisService
        MethodName: Init
        SystemError (124) {
            ErrorText: Failed to open thread 17616 from process 17612
            ApiFunction: OpenThread
            OriginalFile: C:\dvs\p4\build\sw\devtools\Agora\Rel\QuadD_Main\QuadD\Common\ProcessResumer\Windows\ProcessResumerImpl.cpp
            OriginalLine: 45
            OriginalFunction: __cdecl QuadDCommon::ProcessResumer::WindowsProcessResumer::WindowsProcessResumer(class QuadDCommon::StrongType<class QuadDCommon::LimitedNumber<unsigned int,4294967295>,class QuadDCommon::ProcessIdTag>)
            OriginalExceptionClass: struct boost::wrapexcept<class QuadDCommon::SystemException>
        }
    }
}

Profiling options:
DeviceId: "Local"
EventTypes {
  Items: CpuCycles
  Items: Cuda
  Items: NvtxEvents
  Items: Wddm
}
RateHz: 1000
HowToStart: Immediate
HowToStop: Manual
DeviceType: Windows
DeviceDisplayName: "TUD1005912"
WindowsPerfOptions {
  CollectThreadActivity: true
  CollectThreadBacktrace: true
  retainEtwFiles: true
  SymbolSearchVerboseLog: false
  AutomaticallyGenerateReportFileNames: false
}
Processes {
  HowToAttach: LaunchAnother
  Command: "C:\\Program Files\\MATLAB\\R2021b\\bin\\win64\\matlab.exe"
  Arguments: "-wait"
  Arguments: "-nojvm"
  Arguments: "-nosplash"
  Arguments: "-r"
  Arguments: "test_gpu_bf_large_blocks_threaded"
  WorkingDirectory: "C:\\Users\\verasonics\\Documents\\Rick\\REPOS\\OOPBF\\GPU_BF\\test"
  UserName: "verasonics"
  CollectNvtxTrace: true
  CollectCudaTrace: true
  CudaFlushPeriodically: true
  CudaFlushPeriod: 10000000000
  CudaSkipSomeApiCalls: true
  CollectGPUMemoryUsage: true
  CudaGraphTraceOptions {
    Mode: Graph
  }
  CudaFlushOnCudaProfilerStop: true
}
ShowBacktrace: true
IncludeChildren: true
CollectWddmEvents: true
WDDMOptions {
  CollectEtwDxgKrnlEvents: true
}
SymbolResolutionOptions {
  ResolveSymbols: false
}

Thanks rinkertt for the detailed report.

This certainly sheds more light on the scenario. I’d like to run a few tests and hope to have an update soon.

Doron

Some more info. I got 2022 working.

Going to the NVIDIA control panel and setting “allow acces to the GPU performance counters to all users”, and then starting Nsight systems 2022.4.2 as admin did the trick.
So using 2022 i can do my profiling.

For whatever reason I cannot start nsight systems 2023.2.1 as adminstrator from my normal user account nor the localadmin account. Nothing happens.

1 Like

Thanks for looking into this @dofek. I had actually all but given up on this but just tried again running as administrator. This was not working before but it will now start profiling. However, after the profiling run has finished it now reports an error saying it has failed to collect any CUDA events. The application ran as normal and produced the expected output.

Got exactly the same issue. Access to counters for all users is on and I run Systems as an admin.
Downloading 2022.5 didn’t help and I don’t have access to older versions unfortunately.
(Edit: Ok, joined Developer Program, thought it will take more than just one click. :) But it seems that installing 2022.4.1.21-0db2c85 did not help at all and the issue persists.)

Here is another thread with this problem - Unable to start profiling NSight Systems

Version information: NVIDIA Nsight Systems, 2023.2.1.122-32598524v0 Windows-x64

RuntimeError (120) {
    TargetSideError (1100) {
        ErrorText: C:\dvs\p4\build\sw\devtools\Agora\Rel\QuadD_Main\QuadD\Common\ProcessResumer\Windows\ProcessResumerImpl.cpp(45): Throw in function __cdecl QuadDCommon::ProcessResumer::WindowsProcessResumer::WindowsProcessResumer(class QuadDCommon::StrongType<class QuadDCommon::LimitedNumber<unsigned int,4294967295>,class QuadDCommon::ProcessIdTag>)
            Dynamic exception type: struct boost::wrapexcept<class QuadDCommon::SystemException>
            std::exception::what: SystemException
            [struct QuadDCommon::tag_errno_text * __ptr64] = Odmowa dost�pu
            [struct QuadDCommon::tag_message * __ptr64] = Failed to open thread 12616 from process 10800
            [struct QuadDCommon::tag_winapi_error_code * __ptr64] = 5
            [struct boost::errinfo_api_function_ * __ptr64] = OpenThread
            
        ServiceName: AnalysisService
        MethodName: Init
        SystemError (124) {
            ErrorText: Failed to open thread 12616 from process 10800
            ApiFunction: OpenThread
            OriginalFile: C:\dvs\p4\build\sw\devtools\Agora\Rel\QuadD_Main\QuadD\Common\ProcessResumer\Windows\ProcessResumerImpl.cpp
            OriginalLine: 45
            OriginalFunction: __cdecl QuadDCommon::ProcessResumer::WindowsProcessResumer::WindowsProcessResumer(class QuadDCommon::StrongType<class QuadDCommon::LimitedNumber<unsigned int,4294967295>,class QuadDCommon::ProcessIdTag>)
            OriginalExceptionClass: struct boost::wrapexcept<class QuadDCommon::SystemException>
        }
    }
}
1 Like