W17:54:09:950|quadd_common_core|10560|FileSystem.cpp:169[FindInstalledFile]: File 'nsys-config.ini' is not found: /opt/nvidia/nsight-systems/2023.1.1/host-linux-x64; /opt/nvidia/nsight-systems/2023.1.1/linux-desktop-glibc_2_11_3-x64; /opt/nvidia/nsight-systems/2023.1.1/target-linux-x64; /opt/nvidia/nsight-systems/2023.1.1/target-linux-x64/host-linux-x64; /opt/nvidia/nsight-systems/2023.1.1/target-linux-x64/linux-desktop-glibc_2_11_3-x64; /opt/nvidia/nsight-systems/2023.1.1/target-linux-x64/target-linux-x64; W17:54:09:950|quadd_common_core|10560|FileSystem.cpp:169[FindInstalledFile]: File 'config.ini' is not found: /opt/nvidia/nsight-systems/2023.1.1/host-linux-x64; /opt/nvidia/nsight-systems/2023.1.1/linux-desktop-glibc_2_11_3-x64; /opt/nvidia/nsight-systems/2023.1.1/target-linux-x64; /opt/nvidia/nsight-systems/2023.1.1/target-linux-x64/host-linux-x64; /opt/nvidia/nsight-systems/2023.1.1/target-linux-x64/linux-desktop-glibc_2_11_3-x64; /opt/nvidia/nsight-systems/2023.1.1/target-linux-x64/target-linux-x64; I17:54:09:950|quadd_daemon|10560|main.cpp:183[InitConfiguration]: NSys version[ @ ] built[] I17:54:09:950|quadd_daemon|10560|main.cpp:185[InitConfiguration]: Got config file path: I17:54:09:950|quadd_daemon|10560|main.cpp:126[InitPython]: Initing Embedded Python I17:54:09:950|quadd_common_core|10560|FileSystem.cpp:153[FindInstalledFile]: File 'python/lib' is found: /opt/nvidia/nsight-systems/2023.1.1/host-linux-x64/python/lib. I17:54:09:958|quadd_daemon|10560|main.cpp:133[InitPython]: Embedded Python Version: 3.8.3-final-0 I17:54:09:971|quadd_gpu_health|10560|GpuHealth.cpp:70[OpenAllDevicesImpl]: Opening device 0 (status=0) took 5335255 ns I17:54:09:976|quadd_gpu_health|10560|GpuHealth.cpp:70[OpenAllDevicesImpl]: Opening device 1 (status=0) took 5337808 ns I17:54:10:010|quadd_linux_perf|10560|event_fd.cpp:49[OpenEventFile]: perf_event_open(event CPU Clock (sw), attr.type=0x1, attr.config=0x0, tid 10560, cpu -1) returned fd 12 I17:54:10:010|quadd_linux_perf|10560|environment.cpp:250[CheckOSAndKernel]: Detected distribution: Ubuntu. Kernel minimal requirement: 4.3.0-0. Detected kernel: 5.15.0-67-generic I17:54:10:010|quadd_linux_perf|10560|event_fd.cpp:49[OpenEventFile]: perf_event_open(event Reference Cycles, attr.type=0x0, attr.config=0x9, tid 10560, cpu -1) returned fd 12 I17:54:10:010|quadd_linux_perf|10560|event_selection_set.cpp:595[GetDefaultTriggerEventIndex]: GetDefaultTriggerEventIndex(): Reference Cycles(0) event is default sampling trigger event I17:54:10:011|quadd_linux_perf|10560|event_fd.cpp:49[OpenEventFile]: perf_event_open(event Reference Cycles, attr.type=0x0, attr.config=0x9, tid 10560, cpu -1) returned fd 12 I17:54:10:011|quadd_linux_perf|10560|event_fd.cpp:49[OpenEventFile]: perf_event_open(event Reference Cycles, attr.type=0x0, attr.config=0x9, tid 10560, cpu -1) returned fd 12 I17:54:10:011|quadd_linux_perf|10560|event_fd.cpp:49[OpenEventFile]: perf_event_open(event CPU Clock (sw), attr.type=0x1, attr.config=0x0, tid 10560, cpu -1) returned fd 12 I17:54:10:011|quadd_linux_perf|10560|environment.cpp:250[CheckOSAndKernel]: Detected distribution: Ubuntu. Kernel minimal requirement: 4.3.0-0. Detected kernel: 5.15.0-67-generic I17:54:10:011|quadd_linux_perf|10560|event_fd.cpp:49[OpenEventFile]: perf_event_open(event Reference Cycles, attr.type=0x0, attr.config=0x9, tid 10560, cpu -1) returned fd 12 I17:54:10:011|quadd_linux_perf|10560|event_selection_set.cpp:595[GetDefaultTriggerEventIndex]: GetDefaultTriggerEventIndex(): Reference Cycles(0) event is default sampling trigger event I17:54:10:011|quadd_linux_perf|10560|event_fd.cpp:49[OpenEventFile]: perf_event_open(event Reference Cycles, attr.type=0x0, attr.config=0x9, tid 10560, cpu -1) returned fd 12 I17:54:10:011|quadd_linux_perf|10560|event_fd.cpp:49[OpenEventFile]: perf_event_open(event Reference Cycles, attr.type=0x0, attr.config=0x9, tid 10560, cpu -1) returned fd 12 I17:54:10:012|drvacc|10560|:64[]: Driver module override for Cuda I17:54:10:013|quadd_linux_perf|10560|environment.cpp:250[CheckOSAndKernel]: Detected distribution: Ubuntu. Kernel minimal requirement: 4.3.0-0. Detected kernel: 5.15.0-67-generic I17:54:10:013|quadd_gpuinfo_dta|10560|DevToolsApi.cpp:25[CreatePersistenceGuard]: EnableGpuPersistence = 1 I17:54:10:015|drvacc|10560|:145[]: Trying to load driver module: 'libcuda.so.1': 0x4c3e840 I17:54:18:045|drvacc|10560|:64[]: Driver module override for Cuda I17:54:18:045|sp_cli|10560|ParseCLICommandLine.cpp:546[ConfigureLinuxPerf]: Setting LP mode to processtree. I17:54:18:066|quadd_linux_perf|10560|event_fd.cpp:49[OpenEventFile]: perf_event_open(event Reference Cycles, attr.type=0x0, attr.config=0x9, tid 10560, cpu -1) returned fd 12 I17:54:18:066|quadd_linux_perf|10560|event_selection_set.cpp:595[GetDefaultTriggerEventIndex]: GetDefaultTriggerEventIndex(): Reference Cycles(0) event is default sampling trigger event I17:54:18:066|quadd_gpuinfo_dta|10560|DevToolsApi.cpp:25[CreatePersistenceGuard]: EnableGpuPersistence = 1 I17:54:26:089|drvacc|10560|:64[]: Driver module override for Cuda I17:54:26:089|sp_cli|10560|ParseCLICommandLine.cpp:1046[ConfigureCuda]: cudabacktrace collection disabled I17:54:26:098|sp_cli|10560|ParseCLICommandLine.cpp:546[ConfigureLinuxPerf]: Setting LP mode to processtree. I17:54:26:114|quadd_linux_perf|10560|event_fd.cpp:49[OpenEventFile]: perf_event_open(event Reference Cycles, attr.type=0x0, attr.config=0x9, tid 10560, cpu -1) returned fd 12 I17:54:26:114|quadd_linux_perf|10560|event_selection_set.cpp:595[GetDefaultTriggerEventIndex]: GetDefaultTriggerEventIndex(): Reference Cycles(0) event is default sampling trigger event I17:54:26:114|quadd_gpuinfo_dta|10560|DevToolsApi.cpp:25[CreatePersistenceGuard]: EnableGpuPersistence = 1 I17:54:34:132|drvacc|10560|:64[]: Driver module override for Cuda I17:54:34:133|sp_cli|10560|ParseCLICommandLine.cpp:1046[ConfigureCuda]: cudabacktrace collection disabled I17:54:34:140|sp_cli|10560|ParseCLICommandLine.cpp:546[ConfigureLinuxPerf]: Setting LP mode to processtree. I17:54:34:158|quadd_linux_perf|10560|event_fd.cpp:49[OpenEventFile]: perf_event_open(event Reference Cycles, attr.type=0x0, attr.config=0x9, tid 10560, cpu -1) returned fd 12 I17:54:34:158|quadd_linux_perf|10560|event_selection_set.cpp:595[GetDefaultTriggerEventIndex]: GetDefaultTriggerEventIndex(): Reference Cycles(0) event is default sampling trigger event I17:54:34:158|quadd_gpuinfo_dta|10560|DevToolsApi.cpp:25[CreatePersistenceGuard]: EnableGpuPersistence = 1 I17:54:42:180|drvacc|10560|:64[]: Driver module override for Cuda I17:54:42:181|sp_cli|10560|ParseCLICommandLine.cpp:1046[ConfigureCuda]: cudabacktrace collection disabled I17:54:42:190|quadd_common_core|10560|AsyncProcessorHolder.h:28[AsyncProcessorHolder]: AsyncProcessorHolder[0x7ffe508fe220]: 2 AsyncProcessors I17:54:42:190|quadd_common_core|10560|AsyncProcessor.cpp:39[AsyncProcessor]: 1Comm@AsyncProcessor[0x4bcafb0] is creating. I17:54:42:190|quadd_common_core|10560|AsyncProcessor.cpp:84[AsyncProcessor]: 1Comm@AsyncProcessor[0x4bcafb0] created. I17:54:42:190|quadd_common_core|10560|AsyncProcessor.cpp:39[AsyncProcessor]: 2Worker@AsyncProcessor[0x4c3c7c0] is creating. I17:54:42:190|quadd_common_core|10563|AsyncProcessor.cpp:55[operator()]: Thread[10563] started servicing 1Comm@AsyncProcessor. I17:54:42:190|quadd_common_core|10560|AsyncProcessor.cpp:84[AsyncProcessor]: 2Worker@AsyncProcessor[0x4c3c7c0] created. I17:54:42:190|quadd_common_core|10564|AsyncProcessor.cpp:55[operator()]: Thread[10564] started servicing 1Comm@AsyncProcessor. I17:54:42:191|quadd_common_core|10566|AsyncProcessor.cpp:55[operator()]: Thread[10566] started servicing 2Worker@AsyncProcessor. I17:54:42:191|quadd_agent_api|10560|AgentAPI.cpp:381[CreateAPI]: Create local AgentAPI. I17:54:42:190|quadd_common_core|10565|AsyncProcessor.cpp:55[operator()]: Thread[10565] started servicing 2Worker@AsyncProcessor. I17:54:42:191|quadd_common_core|10563|FileSystem.cpp:153[FindInstalledFile]: File 'nsys' is found: /opt/nvidia/nsight-systems/2023.1.1/target-linux-x64/nsys. I17:54:42:191|quadd_agent_api|10563|Agent.cpp:141[MakeCommandString]: Agent start command: "/opt/nvidia/nsight-systems/2023.1.1/target-linux-x64/nsys" --start-agent --session-name=profile-echo-10560 --shm-name=quadd-create-agent-86fee657-e46c-4a4a-aac6-27faec82f6c7 W17:54:42:198|quadd_common_core|10567|FileSystem.cpp:169[FindInstalledFile]: File 'nsys-config.ini' is not found: /opt/nvidia/nsight-systems/2023.1.1/host-linux-x64; /opt/nvidia/nsight-systems/2023.1.1/linux-desktop-glibc_2_11_3-x64; /opt/nvidia/nsight-systems/2023.1.1/target-linux-x64; /opt/nvidia/nsight-systems/2023.1.1/target-linux-x64/host-linux-x64; /opt/nvidia/nsight-systems/2023.1.1/target-linux-x64/linux-desktop-glibc_2_11_3-x64; /opt/nvidia/nsight-systems/2023.1.1/target-linux-x64/target-linux-x64; W17:54:42:198|quadd_common_core|10567|FileSystem.cpp:169[FindInstalledFile]: File 'config.ini' is not found: /opt/nvidia/nsight-systems/2023.1.1/host-linux-x64; /opt/nvidia/nsight-systems/2023.1.1/linux-desktop-glibc_2_11_3-x64; /opt/nvidia/nsight-systems/2023.1.1/target-linux-x64; /opt/nvidia/nsight-systems/2023.1.1/target-linux-x64/host-linux-x64; /opt/nvidia/nsight-systems/2023.1.1/target-linux-x64/linux-desktop-glibc_2_11_3-x64; /opt/nvidia/nsight-systems/2023.1.1/target-linux-x64/target-linux-x64; I17:54:42:198|quadd_daemon|10567|main.cpp:183[InitConfiguration]: NSys version[ @ ] built[] I17:54:42:198|quadd_daemon|10567|main.cpp:185[InitConfiguration]: Got config file path: I17:54:42:199|quadd_daemon|10569|RestartAsDaemon.cpp:131[RestartAsDaemon]: daemon is started I17:54:42:200|quadd_common_core|10569|AsyncProcessorHolder.h:28[AsyncProcessorHolder]: AsyncProcessorHolder[0x7ffe09b78ae0]: 3 AsyncProcessors I17:54:42:200|quadd_common_core|10569|AsyncProcessor.cpp:39[AsyncProcessor]: 1MainService@AsyncProcessor[0x3a91fb0] is creating. I17:54:42:200|quadd_common_core|10569|AsyncProcessor.cpp:84[AsyncProcessor]: 1MainService@AsyncProcessor[0x3a91fb0] created. I17:54:42:200|quadd_common_core|10569|AsyncProcessor.cpp:39[AsyncProcessor]: 2TaskRunner@AsyncProcessor[0x3a92230] is creating. I17:54:42:200|quadd_common_core|10570|AsyncProcessor.cpp:55[operator()]: Thread[10570] started servicing 1MainService@AsyncProcessor. I17:54:42:200|quadd_common_core|10571|AsyncProcessor.cpp:55[operator()]: Thread[10571] started servicing 1MainService@AsyncProcessor. I17:54:42:200|quadd_common_core|10573|AsyncProcessor.cpp:55[operator()]: Thread[10573] started servicing 2TaskRunner@AsyncProcessor. I17:54:42:200|quadd_common_core|10569|AsyncProcessor.cpp:84[AsyncProcessor]: 2TaskRunner@AsyncProcessor[0x3a92230] created. I17:54:42:200|quadd_common_core|10569|AsyncProcessor.cpp:39[AsyncProcessor]: 3AnalysisServiceImpl@AsyncProcessor[0x3a93c70] is creating. I17:54:42:200|quadd_common_core|10572|AsyncProcessor.cpp:55[operator()]: Thread[10572] started servicing 2TaskRunner@AsyncProcessor. I17:54:42:200|quadd_common_core|10569|AsyncProcessor.cpp:84[AsyncProcessor]: 3AnalysisServiceImpl@AsyncProcessor[0x3a93c70] created. I17:54:42:200|quadd_common_session_manager|10569|Helper.cpp:70[GetFileLock]: Getting session manager file lock: mode=0 path=/tmp/nvidia/nsight_systems/SessionManager-32365746v0.lock. I17:54:42:200|quadd_common_core|10574|AsyncProcessor.cpp:55[operator()]: Thread[10574] started servicing 3AnalysisServiceImpl@AsyncProcessor. I17:54:42:200|quadd_common_core|10575|AsyncProcessor.cpp:55[operator()]: Thread[10575] started servicing 3AnalysisServiceImpl@AsyncProcessor. I17:54:42:203|quadd_common_session_manager|10569|Helper.cpp:70[GetFileLock]: Getting session manager file lock: mode=1 path=/tmp/nvidia/nsight_systems/SessionManager-32365746v0.lock. I17:54:42:204|quadd_common_session_manager|10569|BaseSessionManager.cpp:447[CreateUser]: Creating user "sscott" I17:54:42:204|quadd_common_session_manager|10569|BaseSessionManager.cpp:267[CreateSession]: Creating session "profile-echo-10560" with ID 1010569 I17:54:42:204|quadd_agent|10569|SessionRegistry.cpp:40[SessionRegistry]: Session registered: id=1010569 name=profile-echo-10560. I17:54:42:204|quadd_daemon_timesync|10569|TimeSyncService.cpp:391[DoBegin]: TimeSyncServiceImpl[0x3a97040]: #DoBegin begin I17:54:42:204|quadd_daemon_timesync|10569|TimeSyncService.cpp:561[Reset]: TimeSyncServiceImpl[0x3a97040]: #Reset begin I17:54:42:204|quadd_daemon_timesync|10569|TimeSyncService.cpp:581[Reset]: TimeSyncServiceImpl[0x3a97040]: #Reset end I17:54:42:204|quadd_utc_timecor|10569|UtcTimeCorrelation.cpp:48[GetResult]: GetResult[(cpu=3035104862184, utc=1678298082204108921)] offset=-1678295047099246737 I17:54:42:204|quadd_local_timecor|10569|LocaltimeCorrelation.cpp:33[GetResult]: GetResult[(utc=1678298082, local=1678298082)] offset=0 I17:54:42:204|drvacc|10569|:64[]: Driver module override for Cuda E17:55:12:191|quadd_agent_api|10563|Agent.cpp:74[operator()]: Agent launcher exceeded its timeout. I17:55:12:191|quadd_common_core|10560|AsyncProcessorHolder.h:62[Terminate]: AsyncProcessorHolder[0x7ffe508fe220]: Stopping I17:55:12:191|quadd_common_core|10560|AsyncProcessor.cpp:110[Stop]: 1Comm@AsyncProcessor[0x4bcafb0] is stopping. I17:55:12:191|quadd_common_core|10560|AsyncProcessor.cpp:124[Stop]: 1Comm@AsyncProcessor[0x4bcafb0] is waiting for work threads finish. I17:55:12:191|quadd_common_core|10564|AsyncProcessor.cpp:78[operator()]: Thread[10564] stopped servicing 1Comm@AsyncProcessor. I17:55:12:192|quadd_agent_api|10563|Agent.cpp:61[operator()]: Agent launcher exited: exit code=9 E17:55:12:192|quadd_agent_api|10563|LocalAgentAPI.cpp:244[operator()]: Failed to create session: /dvs/p4/build/sw/devtools/Agora/Rel/QuadD_Main/QuadD/Common/AgentAPI/Src/Agent.cpp(105): Throw in function void {anonymous}::SpawnAgentProcess(Agent::CreateParams&&) Dynamic exception type: boost::wrapexcept std::exception::what: RuntimeException [QuadDCommon::tag_message*] = Agent launcher failed. [QuadDCommon::tag_exit_code*] = 9 . I17:55:12:192|quadd_common_core|10563|AsyncProcessor.cpp:78[operator()]: Thread[10563] stopped servicing 1Comm@AsyncProcessor. I17:55:12:192|quadd_common_core|10560|AsyncProcessor.cpp:132[Stop]: 1Comm@AsyncProcessor[0x4bcafb0] stopped. I17:55:12:192|quadd_common_core|10560|AsyncProcessor.cpp:110[Stop]: 2Worker@AsyncProcessor[0x4c3c7c0] is stopping. I17:55:12:192|quadd_common_core|10560|AsyncProcessor.cpp:124[Stop]: 2Worker@AsyncProcessor[0x4c3c7c0] is waiting for work threads finish. I17:55:12:192|quadd_common_core|10566|AsyncProcessor.cpp:78[operator()]: Thread[10566] stopped servicing 2Worker@AsyncProcessor. I17:55:12:192|quadd_common_core|10565|AsyncProcessor.cpp:78[operator()]: Thread[10565] stopped servicing 2Worker@AsyncProcessor. I17:55:12:192|quadd_common_core|10560|AsyncProcessor.cpp:132[Stop]: 2Worker@AsyncProcessor[0x4c3c7c0] stopped. I17:55:12:192|quadd_common_core|10560|AsyncProcessorHolder.h:78[Terminate]: AsyncProcessorHolder[0x7ffe508fe220]: Destroying F17:55:12:192|quadd_daemon|10560|main.cpp:642[main]: Fatal error: /dvs/p4/build/sw/devtools/Agora/Rel/QuadD_Main/QuadD/Common/AgentAPI/Src/Controller.cpp(825): Throw in function Agent::ISession::Ptr Agent::Client::{anonymous}::CreateSession(const Agent::Command&, const Ptr&) Dynamic exception type: boost::wrapexcept std::exception::what: RuntimeException [QuadDCommon::tag_message*] = The timeout expired. I17:55:14:283|quadd_common_core|10569|FileSystem.cpp:153[FindInstalledFile]: File 'CudaGpuInfoDumper' is found: /opt/nvidia/nsight-systems/2023.1.1/target-linux-x64/CudaGpuInfoDumper. I17:55:14:283|quadd_gpuinfo_dta|10569|DevToolsApi.cpp:25[CreatePersistenceGuard]: EnableGpuPersistence = 1 I17:55:22:321|drvacc|10569|:64[]: Driver module override for Cuda I17:55:30:354|quadd_gpuinfo_cta|10569|CudaToolsApi.cpp:441[InitializeGpuInfoListOutOfProcess]: Launching CudaGpuInfoDumper to collect gpu details for 1 processes I17:55:30:355|quadd_gputraits|10569|GpuTicksConverter.cpp:160[DiscreteGpuDevice]: GpuTimeInterpolateFromFirstToLast = 0 I17:55:30:356|quadd_gputraits|10569|GpuTicksConverter.cpp:160[DiscreteGpuDevice]: GpuTimeInterpolateFromFirstToLast = 0 I17:55:30:356|quadd_daemon|10569|TimeConverter.cpp:138[Refine]: rdtsc -> monotonic conversion factors: I17:55:30:356|quadd_daemon|10569|TimeConverter.cpp:139[Refine]: Start sync point = {3989854687285795, 3035104889198} I17:55:30:356|quadd_daemon|10569|TimeConverter.cpp:142[Refine]: End sync point = {3989994328531544, 3083257235539} I17:55:30:356|quadd_daemon|10569|TimeConverter.cpp:145[Refine]: Multiplier = 0.344828965702240 I17:55:30:356|quadd_daemon|10569|TimeConverter.cpp:218[Refine]: CLOCK_MONOTONIC_RAW -> CLOCK_MONOTONIC conversion factors: I17:55:30:356|quadd_daemon|10569|TimeConverter.cpp:219[Refine]: Start sync point = {3033787701471, 3035104889720} I17:55:30:356|quadd_daemon|10569|TimeConverter.cpp:222[Refine]: End sync point = {3081939894203, 3083257257582} I17:55:30:356|quadd_daemon|10569|TimeConverter.cpp:225[Refine]: Multiplier = 1.000003637009865 I17:55:30:356|quadd_gputraits|10569|GpuTicksConverter.cpp:185[Refine]: GPU 12544 time -> CPU time conversion factors: N sync point = {1678298130830870240, 3989994326499874} N+1 sync point = {1678298130831791328, 3989994329171077} Multiplier from N to N+1 = 2.900051895150083 Multiplier from first to N+1 = 2.900051895150083 I17:55:30:357|quadd_gputraits|10569|GpuTicksConverter.cpp:185[Refine]: GPU 19200 time -> CPU time conversion factors: N sync point = {1678298130837380640, 3989994328361349} N+1 sync point = {1678298130837977792, 3989994330093112} Multiplier from N to N+1 = 2.900037176464284 Multiplier from first to N+1 = 2.900037176464284 I17:55:30:357|quadd_tsc_timecor|10569|TscGlobalTimeCorrelation.cpp:38[GetResult]: GetResult[(cpu=3083257228253, tsc=3083257228252)] offset=1 I17:55:30:357|quadd_daemon_timesync|10569|TimeSyncService.cpp:509[DoEnd]: TimeSyncServiceImpl[0x3a97040]: #DoEnd begin I17:55:30:357|quadd_daemon_timesync|10569|TimeSyncService.cpp:561[Reset]: TimeSyncServiceImpl[0x3a97040]: #Reset begin I17:55:30:357|quadd_daemon_timesync|10569|TimeSyncService.cpp:581[Reset]: TimeSyncServiceImpl[0x3a97040]: #Reset end I17:55:30:357|quadd_daemon_timesync|10569|TimeSyncService.cpp:519[DoEnd]: TimeSyncServiceImpl[0x3a97040]: Sync adjustment 18446741038604717632 I17:55:30:367|quadd_daemon_timesync|10569|TimeSyncService.cpp:561[Reset]: TimeSyncServiceImpl[0x3a97040]: #Reset begin I17:55:30:367|quadd_daemon_timesync|10569|TimeSyncService.cpp:581[Reset]: TimeSyncServiceImpl[0x3a97040]: #Reset end I17:55:30:367|quadd_agent_session|10569|Session.cpp:365[Session]: Session created. I17:55:30:368|quadd_pbcomm_server|10569|Server.cpp:44[Server]: Server[0x3aaa4b0]: service[0] AgentService I17:55:30:368|quadd_pbcomm_server|10569|Server.cpp:48[Server]: Server[0x3aaa4b0] created. I17:55:30:368|quadd_pbcomm_server|10569|Server.cpp:77[Start]: Server[0x3aaa4b0] starts session acceptor. I17:55:30:368|quadd_pbcomm_tcp|10569|Communicator.cpp:195[Acceptor]: Acceptor[0x3a95000] created. I17:55:30:368|quadd_pbcomm_tcp|10569|Communicator.cpp:545[CommunicatorCreator]: CommunicatorCreator[0x3acf8d0] created: port=0 I17:55:30:368|quadd_pbcomm_server|10569|Server.cpp:83[Start]: Server[0x3aaa4b0] started. I17:55:30:368|quadd_pbcomm_tcp|10570|Communicator.cpp:208[Start]: Acceptor[0x3a95000]: starts accepting. I17:55:30:368|quadd_pbcomm_server|10569|Server.cpp:55[~Server]: Server[0x3aaa4b0] is closing communicators. I17:55:30:368|quadd_pbcomm_server|10569|Server.cpp:60[~Server]: Server[0x3aaa4b0] destroyed. I17:55:30:368|quadd_pbcomm_tcp|10569|Communicator.cpp:250[Terminate]: Acceptor[0x3a95000] terminating I17:55:30:368|quadd_pbcomm_tcp|10569|Communicator.cpp:554[~CommunicatorCreator]: CommunicatorCreator[0x3acf8d0] destroyed. I17:55:30:368|quadd_common_session_manager|10569|BaseSessionManager.cpp:355[RemoveUser]: Removing user "sscott" I17:55:30:368|quadd_common_session_manager|10569|BaseSessionManager.cpp:554[RemoveSession]: Removing session "profile-echo-10560" with ID 1010569 I17:55:30:368|quadd_agent|10569|SessionRegistry.cpp:52[~SessionRegistry]: Session unregistered: id=1010569. I17:55:30:368|quadd_common_session_manager|10569|Helper.cpp:70[GetFileLock]: Getting session manager file lock: mode=1 path=/tmp/nvidia/nsight_systems/SessionManager-32365746v0.lock. E17:55:30:368|quadd_pbcomm_tcp|10571|Communicator.cpp:219[Handle]: Acceptor[0x3a95000]: accept failed: Operation canceled I17:55:30:368|quadd_pbcomm_tcp|10571|Communicator.cpp:200[~Acceptor]: Acceptor[0x3a95000] destroyed. I17:55:30:368|quadd_common_session_manager|10569|Helper.cpp:70[GetFileLock]: Getting session manager file lock: mode=1 path=/tmp/nvidia/nsight_systems/SessionManager-32365746v0.lock. E17:55:30:368|quadd_common_session_manager|10569|BaseSessionManager.cpp:63[OpenSharedMemoryObjects]: Failed to open ManagedSharedMemory(NSys-32365746v0): Dynamic exception type: boost::interprocess::interprocess_exception std::exception::what: No such file or directory . I17:55:30:368|quadd_common_core|10569|AsyncProcessorHolder.h:62[Terminate]: AsyncProcessorHolder[0x7ffe09b78ae0]: Stopping I17:55:30:368|quadd_common_core|10569|AsyncProcessor.cpp:110[Stop]: 1MainService@AsyncProcessor[0x3a91fb0] is stopping. I17:55:30:368|quadd_common_core|10569|AsyncProcessor.cpp:124[Stop]: 1MainService@AsyncProcessor[0x3a91fb0] is waiting for work threads finish. I17:55:30:368|quadd_common_core|10570|AsyncProcessor.cpp:78[operator()]: Thread[10570] stopped servicing 1MainService@AsyncProcessor. I17:55:30:368|quadd_common_core|10571|AsyncProcessor.cpp:78[operator()]: Thread[10571] stopped servicing 1MainService@AsyncProcessor. I17:55:30:368|quadd_common_core|10569|AsyncProcessor.cpp:132[Stop]: 1MainService@AsyncProcessor[0x3a91fb0] stopped. I17:55:30:368|quadd_common_core|10569|AsyncProcessor.cpp:110[Stop]: 2TaskRunner@AsyncProcessor[0x3a92230] is stopping. I17:55:30:368|quadd_common_core|10569|AsyncProcessor.cpp:124[Stop]: 2TaskRunner@AsyncProcessor[0x3a92230] is waiting for work threads finish. I17:55:30:368|quadd_common_core|10572|AsyncProcessor.cpp:78[operator()]: Thread[10572] stopped servicing 2TaskRunner@AsyncProcessor. I17:55:30:368|quadd_common_core|10573|AsyncProcessor.cpp:78[operator()]: Thread[10573] stopped servicing 2TaskRunner@AsyncProcessor. I17:55:30:368|quadd_common_core|10569|AsyncProcessor.cpp:132[Stop]: 2TaskRunner@AsyncProcessor[0x3a92230] stopped. I17:55:30:368|quadd_common_core|10569|AsyncProcessor.cpp:110[Stop]: 3AnalysisServiceImpl@AsyncProcessor[0x3a93c70] is stopping. I17:55:30:368|quadd_common_core|10569|AsyncProcessor.cpp:124[Stop]: 3AnalysisServiceImpl@AsyncProcessor[0x3a93c70] is waiting for work threads finish. I17:55:30:369|quadd_common_core|10575|AsyncProcessor.cpp:78[operator()]: Thread[10575] stopped servicing 3AnalysisServiceImpl@AsyncProcessor. I17:55:30:369|quadd_common_core|10574|AsyncProcessor.cpp:78[operator()]: Thread[10574] stopped servicing 3AnalysisServiceImpl@AsyncProcessor. I17:55:30:369|quadd_common_core|10569|AsyncProcessor.cpp:132[Stop]: 3AnalysisServiceImpl@AsyncProcessor[0x3a93c70] stopped. I17:55:30:369|quadd_common_core|10569|AsyncProcessorHolder.h:78[Terminate]: AsyncProcessorHolder[0x7ffe09b78ae0]: Destroying W17:55:46:667|quadd_common_core|10588|FileSystem.cpp:169[FindInstalledFile]: File 'nsys-config.ini' is not found: /opt/nvidia/nsight-systems/2023.1.1/host-linux-x64; /opt/nvidia/nsight-systems/2023.1.1/linux-desktop-glibc_2_11_3-x64; /opt/nvidia/nsight-systems/2023.1.1/target-linux-x64; /opt/nvidia/nsight-systems/2023.1.1/target-linux-x64/host-linux-x64; /opt/nvidia/nsight-systems/2023.1.1/target-linux-x64/linux-desktop-glibc_2_11_3-x64; /opt/nvidia/nsight-systems/2023.1.1/target-linux-x64/target-linux-x64; W17:55:46:667|quadd_common_core|10588|FileSystem.cpp:169[FindInstalledFile]: File 'config.ini' is not found: /opt/nvidia/nsight-systems/2023.1.1/host-linux-x64; /opt/nvidia/nsight-systems/2023.1.1/linux-desktop-glibc_2_11_3-x64; /opt/nvidia/nsight-systems/2023.1.1/target-linux-x64; /opt/nvidia/nsight-systems/2023.1.1/target-linux-x64/host-linux-x64; /opt/nvidia/nsight-systems/2023.1.1/target-linux-x64/linux-desktop-glibc_2_11_3-x64; /opt/nvidia/nsight-systems/2023.1.1/target-linux-x64/target-linux-x64; I17:55:46:667|quadd_daemon|10588|main.cpp:183[InitConfiguration]: NSys version[ @ ] built[] I17:55:46:667|quadd_daemon|10588|main.cpp:185[InitConfiguration]: Got config file path: I17:55:46:667|quadd_daemon|10588|main.cpp:126[InitPython]: Initing Embedded Python I17:55:46:667|quadd_common_core|10588|FileSystem.cpp:153[FindInstalledFile]: File 'python/lib' is found: /opt/nvidia/nsight-systems/2023.1.1/host-linux-x64/python/lib. I17:55:46:675|quadd_daemon|10588|main.cpp:133[InitPython]: Embedded Python Version: 3.8.3-final-0 I17:55:46:688|quadd_gpu_health|10588|GpuHealth.cpp:70[OpenAllDevicesImpl]: Opening device 0 (status=0) took 5325685 ns I17:55:46:693|quadd_gpu_health|10588|GpuHealth.cpp:70[OpenAllDevicesImpl]: Opening device 1 (status=0) took 5327574 ns I17:55:46:714|quadd_linux_perf|10588|event_fd.cpp:49[OpenEventFile]: perf_event_open(event CPU Clock (sw), attr.type=0x1, attr.config=0x0, tid 10588, cpu -1) returned fd 12 I17:55:46:714|quadd_linux_perf|10588|environment.cpp:250[CheckOSAndKernel]: Detected distribution: Ubuntu. Kernel minimal requirement: 4.3.0-0. Detected kernel: 5.15.0-67-generic I17:55:46:714|quadd_linux_perf|10588|event_fd.cpp:49[OpenEventFile]: perf_event_open(event Reference Cycles, attr.type=0x0, attr.config=0x9, tid 10588, cpu -1) returned fd 12 I17:55:46:714|quadd_linux_perf|10588|event_selection_set.cpp:595[GetDefaultTriggerEventIndex]: GetDefaultTriggerEventIndex(): Reference Cycles(0) event is default sampling trigger event I17:55:46:714|quadd_linux_perf|10588|event_fd.cpp:49[OpenEventFile]: perf_event_open(event Reference Cycles, attr.type=0x0, attr.config=0x9, tid 10588, cpu -1) returned fd 12 I17:55:46:715|quadd_linux_perf|10588|event_fd.cpp:49[OpenEventFile]: perf_event_open(event Reference Cycles, attr.type=0x0, attr.config=0x9, tid 10588, cpu -1) returned fd 12 I17:55:46:715|quadd_linux_perf|10588|event_fd.cpp:49[OpenEventFile]: perf_event_open(event CPU Clock (sw), attr.type=0x1, attr.config=0x0, tid 10588, cpu -1) returned fd 12 I17:55:46:715|quadd_linux_perf|10588|environment.cpp:250[CheckOSAndKernel]: Detected distribution: Ubuntu. Kernel minimal requirement: 4.3.0-0. Detected kernel: 5.15.0-67-generic I17:55:46:715|quadd_linux_perf|10588|event_fd.cpp:49[OpenEventFile]: perf_event_open(event Reference Cycles, attr.type=0x0, attr.config=0x9, tid 10588, cpu -1) returned fd 12 I17:55:46:715|quadd_linux_perf|10588|event_selection_set.cpp:595[GetDefaultTriggerEventIndex]: GetDefaultTriggerEventIndex(): Reference Cycles(0) event is default sampling trigger event I17:55:46:715|quadd_linux_perf|10588|event_fd.cpp:49[OpenEventFile]: perf_event_open(event Reference Cycles, attr.type=0x0, attr.config=0x9, tid 10588, cpu -1) returned fd 12 I17:55:46:715|quadd_linux_perf|10588|event_fd.cpp:49[OpenEventFile]: perf_event_open(event Reference Cycles, attr.type=0x0, attr.config=0x9, tid 10588, cpu -1) returned fd 12 I17:55:46:716|drvacc|10588|:64[]: Driver module override for Cuda I17:55:46:716|quadd_linux_perf|10588|environment.cpp:250[CheckOSAndKernel]: Detected distribution: Ubuntu. Kernel minimal requirement: 4.3.0-0. Detected kernel: 5.15.0-67-generic I17:55:46:716|quadd_gpuinfo_dta|10588|DevToolsApi.cpp:25[CreatePersistenceGuard]: EnableGpuPersistence = 1 I17:55:46:718|drvacc|10588|:145[]: Trying to load driver module: 'libcuda.so.1': 0x30158e0 I17:55:54:749|drvacc|10588|:64[]: Driver module override for Cuda I17:55:54:750|sp_cli|10588|ParseCLICommandLine.cpp:546[ConfigureLinuxPerf]: Setting LP mode to processtree. I17:55:54:774|quadd_linux_perf|10588|event_fd.cpp:49[OpenEventFile]: perf_event_open(event Reference Cycles, attr.type=0x0, attr.config=0x9, tid 10588, cpu -1) returned fd 12 I17:55:54:774|quadd_linux_perf|10588|event_selection_set.cpp:595[GetDefaultTriggerEventIndex]: GetDefaultTriggerEventIndex(): Reference Cycles(0) event is default sampling trigger event I17:55:54:774|sp_cli|10588|ParseCLICommandLine.cpp:546[ConfigureLinuxPerf]: Setting LP mode to processtree. I17:55:54:774|quadd_linux_perf|10588|event_fd.cpp:49[OpenEventFile]: perf_event_open(event Reference Cycles, attr.type=0x0, attr.config=0x9, tid 10588, cpu -1) returned fd 12 I17:55:54:774|quadd_linux_perf|10588|event_selection_set.cpp:595[GetDefaultTriggerEventIndex]: GetDefaultTriggerEventIndex(): Reference Cycles(0) event is default sampling trigger event I17:55:54:774|sp_cli|10588|ParseCLICommandLine.cpp:546[ConfigureLinuxPerf]: Setting LP mode to processtree. I17:55:54:774|quadd_linux_perf|10588|event_fd.cpp:49[OpenEventFile]: perf_event_open(event Reference Cycles, attr.type=0x0, attr.config=0x9, tid 10588, cpu -1) returned fd 12 I17:55:54:774|quadd_linux_perf|10588|event_selection_set.cpp:595[GetDefaultTriggerEventIndex]: GetDefaultTriggerEventIndex(): Reference Cycles(0) event is default sampling trigger event I17:55:54:774|quadd_common_core|10588|AsyncProcessorHolder.h:28[AsyncProcessorHolder]: AsyncProcessorHolder[0x7ffddde6f770]: 2 AsyncProcessors I17:55:54:775|quadd_common_core|10588|AsyncProcessor.cpp:39[AsyncProcessor]: 1Comm@AsyncProcessor[0x2fa1fb0] is creating. I17:55:54:775|quadd_common_core|10588|AsyncProcessor.cpp:84[AsyncProcessor]: 1Comm@AsyncProcessor[0x2fa1fb0] created. I17:55:54:775|quadd_common_core|10588|AsyncProcessor.cpp:39[AsyncProcessor]: 2Worker@AsyncProcessor[0x3014ec0] is creating. I17:55:54:775|quadd_common_core|10588|AsyncProcessor.cpp:84[AsyncProcessor]: 2Worker@AsyncProcessor[0x3014ec0] created. I17:55:54:775|quadd_agent_api|10588|AgentAPI.cpp:381[CreateAPI]: Create local AgentAPI. I17:55:54:775|quadd_common_core|10589|AsyncProcessor.cpp:55[operator()]: Thread[10589] started servicing 1Comm@AsyncProcessor. I17:55:54:775|quadd_common_core|10592|AsyncProcessor.cpp:55[operator()]: Thread[10592] started servicing 2Worker@AsyncProcessor. I17:55:54:775|quadd_common_core|10590|AsyncProcessor.cpp:55[operator()]: Thread[10590] started servicing 1Comm@AsyncProcessor. I17:55:54:775|quadd_common_core|10591|AsyncProcessor.cpp:55[operator()]: Thread[10591] started servicing 2Worker@AsyncProcessor. I17:55:54:775|quadd_common_core|10589|FileSystem.cpp:153[FindInstalledFile]: File 'nsys' is found: /opt/nvidia/nsight-systems/2023.1.1/target-linux-x64/nsys. I17:55:54:775|quadd_agent_api|10589|Agent.cpp:141[MakeCommandString]: Agent start command: "/opt/nvidia/nsight-systems/2023.1.1/target-linux-x64/nsys" --start-agent --session-name=profile-esat-10588 --shm-name=quadd-create-agent-ed6a401d-2025-49ff-b8a0-8ca496a83009 W17:55:54:780|quadd_common_core|10593|FileSystem.cpp:169[FindInstalledFile]: File 'nsys-config.ini' is not found: /opt/nvidia/nsight-systems/2023.1.1/host-linux-x64; /opt/nvidia/nsight-systems/2023.1.1/linux-desktop-glibc_2_11_3-x64; /opt/nvidia/nsight-systems/2023.1.1/target-linux-x64; /opt/nvidia/nsight-systems/2023.1.1/target-linux-x64/host-linux-x64; /opt/nvidia/nsight-systems/2023.1.1/target-linux-x64/linux-desktop-glibc_2_11_3-x64; /opt/nvidia/nsight-systems/2023.1.1/target-linux-x64/target-linux-x64; W17:55:54:780|quadd_common_core|10593|FileSystem.cpp:169[FindInstalledFile]: File 'config.ini' is not found: /opt/nvidia/nsight-systems/2023.1.1/host-linux-x64; /opt/nvidia/nsight-systems/2023.1.1/linux-desktop-glibc_2_11_3-x64; /opt/nvidia/nsight-systems/2023.1.1/target-linux-x64; /opt/nvidia/nsight-systems/2023.1.1/target-linux-x64/host-linux-x64; /opt/nvidia/nsight-systems/2023.1.1/target-linux-x64/linux-desktop-glibc_2_11_3-x64; /opt/nvidia/nsight-systems/2023.1.1/target-linux-x64/target-linux-x64; I17:55:54:780|quadd_daemon|10593|main.cpp:183[InitConfiguration]: NSys version[ @ ] built[] I17:55:54:780|quadd_daemon|10593|main.cpp:185[InitConfiguration]: Got config file path: I17:55:54:781|quadd_daemon|10595|RestartAsDaemon.cpp:131[RestartAsDaemon]: daemon is started I17:55:54:781|quadd_common_core|10595|AsyncProcessorHolder.h:28[AsyncProcessorHolder]: AsyncProcessorHolder[0x7ffd2d6b2fe0]: 3 AsyncProcessors I17:55:54:781|quadd_common_core|10595|AsyncProcessor.cpp:39[AsyncProcessor]: 1MainService@AsyncProcessor[0x3870fb0] is creating. I17:55:54:781|quadd_common_core|10595|AsyncProcessor.cpp:84[AsyncProcessor]: 1MainService@AsyncProcessor[0x3870fb0] created. I17:55:54:781|quadd_common_core|10595|AsyncProcessor.cpp:39[AsyncProcessor]: 2TaskRunner@AsyncProcessor[0x3871230] is creating. I17:55:54:781|quadd_common_core|10595|AsyncProcessor.cpp:84[AsyncProcessor]: 2TaskRunner@AsyncProcessor[0x3871230] created. I17:55:54:781|quadd_common_core|10595|AsyncProcessor.cpp:39[AsyncProcessor]: 3AnalysisServiceImpl@AsyncProcessor[0x3872c70] is creating. I17:55:54:781|quadd_common_core|10596|AsyncProcessor.cpp:55[operator()]: Thread[10596] started servicing 1MainService@AsyncProcessor. I17:55:54:781|quadd_common_core|10597|AsyncProcessor.cpp:55[operator()]: Thread[10597] started servicing 1MainService@AsyncProcessor. I17:55:54:781|quadd_common_core|10598|AsyncProcessor.cpp:55[operator()]: Thread[10598] started servicing 2TaskRunner@AsyncProcessor. I17:55:54:781|quadd_common_core|10599|AsyncProcessor.cpp:55[operator()]: Thread[10599] started servicing 2TaskRunner@AsyncProcessor. I17:55:54:781|quadd_common_core|10595|AsyncProcessor.cpp:84[AsyncProcessor]: 3AnalysisServiceImpl@AsyncProcessor[0x3872c70] created. I17:55:54:782|quadd_common_session_manager|10595|Helper.cpp:70[GetFileLock]: Getting session manager file lock: mode=0 path=/tmp/nvidia/nsight_systems/SessionManager-32365746v0.lock. I17:55:54:781|quadd_common_core|10600|AsyncProcessor.cpp:55[operator()]: Thread[10600] started servicing 3AnalysisServiceImpl@AsyncProcessor. I17:55:54:782|quadd_common_core|10601|AsyncProcessor.cpp:55[operator()]: Thread[10601] started servicing 3AnalysisServiceImpl@AsyncProcessor. I17:55:54:785|quadd_common_session_manager|10595|Helper.cpp:70[GetFileLock]: Getting session manager file lock: mode=1 path=/tmp/nvidia/nsight_systems/SessionManager-32365746v0.lock. I17:55:54:786|quadd_common_session_manager|10595|BaseSessionManager.cpp:447[CreateUser]: Creating user "sscott" I17:55:54:786|quadd_common_session_manager|10595|BaseSessionManager.cpp:267[CreateSession]: Creating session "profile-esat-10588" with ID 1010595 I17:55:54:786|quadd_agent|10595|SessionRegistry.cpp:40[SessionRegistry]: Session registered: id=1010595 name=profile-esat-10588. I17:55:54:786|quadd_daemon_timesync|10595|TimeSyncService.cpp:391[DoBegin]: TimeSyncServiceImpl[0x3876040]: #DoBegin begin I17:55:54:786|quadd_daemon_timesync|10595|TimeSyncService.cpp:561[Reset]: TimeSyncServiceImpl[0x3876040]: #Reset begin I17:55:54:786|quadd_daemon_timesync|10595|TimeSyncService.cpp:581[Reset]: TimeSyncServiceImpl[0x3876040]: #Reset end I17:55:54:786|quadd_utc_timecor|10595|UtcTimeCorrelation.cpp:48[GetResult]: GetResult[(cpu=3107686884233, utc=1678298154786130970)] offset=-1678295047099246737 I17:55:54:786|quadd_local_timecor|10595|LocaltimeCorrelation.cpp:33[GetResult]: GetResult[(utc=1678298154, local=1678298154)] offset=0 I17:55:54:786|drvacc|10595|:64[]: Driver module override for Cuda I17:56:24:775|quadd_common_core|10588|AsyncProcessorHolder.h:62[Terminate]: AsyncProcessorHolder[0x7ffddde6f770]: Stopping E17:56:24:775|quadd_agent_api|10589|Agent.cpp:74[operator()]: Agent launcher exceeded its timeout. I17:56:24:775|quadd_common_core|10588|AsyncProcessor.cpp:110[Stop]: 1Comm@AsyncProcessor[0x2fa1fb0] is stopping. I17:56:24:775|quadd_common_core|10588|AsyncProcessor.cpp:124[Stop]: 1Comm@AsyncProcessor[0x2fa1fb0] is waiting for work threads finish. I17:56:24:775|quadd_common_core|10590|AsyncProcessor.cpp:78[operator()]: Thread[10590] stopped servicing 1Comm@AsyncProcessor. I17:56:24:776|quadd_agent_api|10589|Agent.cpp:61[operator()]: Agent launcher exited: exit code=9 E17:56:24:776|quadd_agent_api|10589|LocalAgentAPI.cpp:244[operator()]: Failed to create session: /dvs/p4/build/sw/devtools/Agora/Rel/QuadD_Main/QuadD/Common/AgentAPI/Src/Agent.cpp(105): Throw in function void {anonymous}::SpawnAgentProcess(Agent::CreateParams&&) Dynamic exception type: boost::wrapexcept std::exception::what: RuntimeException [QuadDCommon::tag_message*] = Agent launcher failed. [QuadDCommon::tag_exit_code*] = 9 . I17:56:24:776|quadd_common_core|10589|AsyncProcessor.cpp:78[operator()]: Thread[10589] stopped servicing 1Comm@AsyncProcessor. I17:56:24:776|quadd_common_core|10588|AsyncProcessor.cpp:132[Stop]: 1Comm@AsyncProcessor[0x2fa1fb0] stopped. I17:56:24:776|quadd_common_core|10588|AsyncProcessor.cpp:110[Stop]: 2Worker@AsyncProcessor[0x3014ec0] is stopping. I17:56:24:776|quadd_common_core|10588|AsyncProcessor.cpp:124[Stop]: 2Worker@AsyncProcessor[0x3014ec0] is waiting for work threads finish. I17:56:24:776|quadd_common_core|10591|AsyncProcessor.cpp:78[operator()]: Thread[10591] stopped servicing 2Worker@AsyncProcessor. I17:56:24:776|quadd_common_core|10592|AsyncProcessor.cpp:78[operator()]: Thread[10592] stopped servicing 2Worker@AsyncProcessor. I17:56:24:776|quadd_common_core|10588|AsyncProcessor.cpp:132[Stop]: 2Worker@AsyncProcessor[0x3014ec0] stopped. I17:56:24:776|quadd_common_core|10588|AsyncProcessorHolder.h:78[Terminate]: AsyncProcessorHolder[0x7ffddde6f770]: Destroying F17:56:24:776|quadd_daemon|10588|main.cpp:642[main]: Fatal error: /dvs/p4/build/sw/devtools/Agora/Rel/QuadD_Main/QuadD/Common/AgentAPI/Src/Controller.cpp(825): Throw in function Agent::ISession::Ptr Agent::Client::{anonymous}::CreateSession(const Agent::Command&, const Ptr&) Dynamic exception type: boost::wrapexcept std::exception::what: RuntimeException [QuadDCommon::tag_message*] = The timeout expired. I17:56:26:868|quadd_common_core|10595|FileSystem.cpp:153[FindInstalledFile]: File 'CudaGpuInfoDumper' is found: /opt/nvidia/nsight-systems/2023.1.1/target-linux-x64/CudaGpuInfoDumper. I17:56:26:868|quadd_gpuinfo_dta|10595|DevToolsApi.cpp:25[CreatePersistenceGuard]: EnableGpuPersistence = 1 I17:56:34:897|drvacc|10595|:64[]: Driver module override for Cuda I17:56:42:933|quadd_gpuinfo_cta|10595|CudaToolsApi.cpp:441[InitializeGpuInfoListOutOfProcess]: Launching CudaGpuInfoDumper to collect gpu details for 1 processes I17:56:42:934|quadd_gputraits|10595|GpuTicksConverter.cpp:160[DiscreteGpuDevice]: GpuTimeInterpolateFromFirstToLast = 0 I17:56:42:935|quadd_gputraits|10595|GpuTicksConverter.cpp:160[DiscreteGpuDevice]: GpuTimeInterpolateFromFirstToLast = 0 I17:56:42:935|quadd_daemon|10595|TimeConverter.cpp:138[Refine]: rdtsc -> monotonic conversion factors: I17:56:42:935|quadd_daemon|10595|TimeConverter.cpp:139[Refine]: Start sync point = {3990065174299080, 3107686910823} I17:56:42:935|quadd_daemon|10595|TimeConverter.cpp:142[Refine]: End sync point = {3990204805569175, 3155835826908} I17:56:42:935|quadd_daemon|10595|TimeConverter.cpp:145[Refine]: Multiplier = 0.344829034730123 I17:56:42:935|quadd_daemon|10595|TimeConverter.cpp:218[Refine]: CLOCK_MONOTONIC_RAW -> CLOCK_MONOTONIC conversion factors: I17:56:42:935|quadd_daemon|10595|TimeConverter.cpp:219[Refine]: Start sync point = {3106369456678, 3107686911446} I17:56:42:935|quadd_daemon|10595|TimeConverter.cpp:222[Refine]: End sync point = {3154518207532, 3155835847053} I17:56:42:935|quadd_daemon|10595|TimeConverter.cpp:225[Refine]: Multiplier = 1.000003837129660 I17:56:42:935|quadd_gputraits|10595|GpuTicksConverter.cpp:185[Refine]: GPU 12544 time -> CPU time conversion factors: N sync point = {1678298203408952768, 3990204802935924} N+1 sync point = {1678298203410112864, 3990204806300214} Multiplier from N to N+1 = 2.900009999172482 Multiplier from first to N+1 = 2.900009999172482 I17:56:42:935|quadd_gputraits|10595|GpuTicksConverter.cpp:185[Refine]: GPU 19200 time -> CPU time conversion factors: N sync point = {1678298203415768096, 3990204805344427} N+1 sync point = {1678298203416464768, 3990204807364745} Multiplier from N to N+1 = 2.899955789812136 Multiplier from first to N+1 = 2.899955789812136 I17:56:42:935|quadd_tsc_timecor|10595|TscGlobalTimeCorrelation.cpp:38[GetResult]: GetResult[(cpu=3155835808039, tsc=3155835808038)] offset=1 I17:56:42:935|quadd_daemon_timesync|10595|TimeSyncService.cpp:509[DoEnd]: TimeSyncServiceImpl[0x3876040]: #DoEnd begin I17:56:42:935|quadd_daemon_timesync|10595|TimeSyncService.cpp:561[Reset]: TimeSyncServiceImpl[0x3876040]: #Reset begin I17:56:42:935|quadd_daemon_timesync|10595|TimeSyncService.cpp:581[Reset]: TimeSyncServiceImpl[0x3876040]: #Reset end I17:56:42:935|quadd_daemon_timesync|10595|TimeSyncService.cpp:519[DoEnd]: TimeSyncServiceImpl[0x3876040]: Sync adjustment 18446740966022692892 I17:56:42:946|quadd_daemon_timesync|10595|TimeSyncService.cpp:561[Reset]: TimeSyncServiceImpl[0x3876040]: #Reset begin I17:56:42:946|quadd_daemon_timesync|10595|TimeSyncService.cpp:581[Reset]: TimeSyncServiceImpl[0x3876040]: #Reset end I17:56:42:946|quadd_agent_session|10595|Session.cpp:365[Session]: Session created. I17:56:42:947|quadd_pbcomm_server|10595|Server.cpp:44[Server]: Server[0x38894b0]: service[0] AgentService I17:56:42:947|quadd_pbcomm_server|10595|Server.cpp:48[Server]: Server[0x38894b0] created. I17:56:42:947|quadd_pbcomm_server|10595|Server.cpp:77[Start]: Server[0x38894b0] starts session acceptor. I17:56:42:947|quadd_pbcomm_tcp|10595|Communicator.cpp:195[Acceptor]: Acceptor[0x3874000] created. I17:56:42:947|quadd_pbcomm_tcp|10595|Communicator.cpp:545[CommunicatorCreator]: CommunicatorCreator[0x38ae8d0] created: port=0 I17:56:42:947|quadd_pbcomm_server|10595|Server.cpp:83[Start]: Server[0x38894b0] started. I17:56:42:947|quadd_pbcomm_tcp|10596|Communicator.cpp:208[Start]: Acceptor[0x3874000]: starts accepting. I17:56:42:947|quadd_pbcomm_server|10595|Server.cpp:55[~Server]: Server[0x38894b0] is closing communicators. I17:56:42:947|quadd_pbcomm_server|10595|Server.cpp:60[~Server]: Server[0x38894b0] destroyed. I17:56:42:947|quadd_pbcomm_tcp|10595|Communicator.cpp:250[Terminate]: Acceptor[0x3874000] terminating I17:56:42:947|quadd_pbcomm_tcp|10595|Communicator.cpp:554[~CommunicatorCreator]: CommunicatorCreator[0x38ae8d0] destroyed. I17:56:42:947|quadd_common_session_manager|10595|BaseSessionManager.cpp:355[RemoveUser]: Removing user "sscott" I17:56:42:947|quadd_common_session_manager|10595|BaseSessionManager.cpp:554[RemoveSession]: Removing session "profile-esat-10588" with ID 1010595 I17:56:42:947|quadd_agent|10595|SessionRegistry.cpp:52[~SessionRegistry]: Session unregistered: id=1010595. I17:56:42:947|quadd_common_session_manager|10595|Helper.cpp:70[GetFileLock]: Getting session manager file lock: mode=1 path=/tmp/nvidia/nsight_systems/SessionManager-32365746v0.lock. E17:56:42:947|quadd_pbcomm_tcp|10596|Communicator.cpp:219[Handle]: Acceptor[0x3874000]: accept failed: Operation canceled I17:56:42:947|quadd_pbcomm_tcp|10596|Communicator.cpp:200[~Acceptor]: Acceptor[0x3874000] destroyed. I17:56:42:947|quadd_common_session_manager|10595|Helper.cpp:70[GetFileLock]: Getting session manager file lock: mode=1 path=/tmp/nvidia/nsight_systems/SessionManager-32365746v0.lock. E17:56:42:947|quadd_common_session_manager|10595|BaseSessionManager.cpp:63[OpenSharedMemoryObjects]: Failed to open ManagedSharedMemory(NSys-32365746v0): Dynamic exception type: boost::interprocess::interprocess_exception std::exception::what: No such file or directory . I17:56:42:947|quadd_common_core|10595|AsyncProcessorHolder.h:62[Terminate]: AsyncProcessorHolder[0x7ffd2d6b2fe0]: Stopping I17:56:42:947|quadd_common_core|10595|AsyncProcessor.cpp:110[Stop]: 1MainService@AsyncProcessor[0x3870fb0] is stopping. I17:56:42:947|quadd_common_core|10595|AsyncProcessor.cpp:124[Stop]: 1MainService@AsyncProcessor[0x3870fb0] is waiting for work threads finish. I17:56:42:947|quadd_common_core|10596|AsyncProcessor.cpp:78[operator()]: Thread[10596] stopped servicing 1MainService@AsyncProcessor. I17:56:42:947|quadd_common_core|10597|AsyncProcessor.cpp:78[operator()]: Thread[10597] stopped servicing 1MainService@AsyncProcessor. I17:56:42:947|quadd_common_core|10595|AsyncProcessor.cpp:132[Stop]: 1MainService@AsyncProcessor[0x3870fb0] stopped. I17:56:42:947|quadd_common_core|10595|AsyncProcessor.cpp:110[Stop]: 2TaskRunner@AsyncProcessor[0x3871230] is stopping. I17:56:42:947|quadd_common_core|10595|AsyncProcessor.cpp:124[Stop]: 2TaskRunner@AsyncProcessor[0x3871230] is waiting for work threads finish. I17:56:42:947|quadd_common_core|10598|AsyncProcessor.cpp:78[operator()]: Thread[10598] stopped servicing 2TaskRunner@AsyncProcessor. I17:56:42:947|quadd_common_core|10599|AsyncProcessor.cpp:78[operator()]: Thread[10599] stopped servicing 2TaskRunner@AsyncProcessor. I17:56:42:948|quadd_common_core|10595|AsyncProcessor.cpp:132[Stop]: 2TaskRunner@AsyncProcessor[0x3871230] stopped. I17:56:42:948|quadd_common_core|10595|AsyncProcessor.cpp:110[Stop]: 3AnalysisServiceImpl@AsyncProcessor[0x3872c70] is stopping. I17:56:42:948|quadd_common_core|10595|AsyncProcessor.cpp:124[Stop]: 3AnalysisServiceImpl@AsyncProcessor[0x3872c70] is waiting for work threads finish. I17:56:42:948|quadd_common_core|10600|AsyncProcessor.cpp:78[operator()]: Thread[10600] stopped servicing 3AnalysisServiceImpl@AsyncProcessor. I17:56:42:948|quadd_common_core|10601|AsyncProcessor.cpp:78[operator()]: Thread[10601] stopped servicing 3AnalysisServiceImpl@AsyncProcessor. I17:56:42:948|quadd_common_core|10595|AsyncProcessor.cpp:132[Stop]: 3AnalysisServiceImpl@AsyncProcessor[0x3872c70] stopped. I17:56:42:948|quadd_common_core|10595|AsyncProcessorHolder.h:78[Terminate]: AsyncProcessorHolder[0x7ffd2d6b2fe0]: Destroying W17:57:39:971|quadd_common_core|10614|FileSystem.cpp:169[FindInstalledFile]: File 'nsys-config.ini' is not found: /opt/nvidia/nsight-systems/2023.1.1/host-linux-x64; /opt/nvidia/nsight-systems/2023.1.1/linux-desktop-glibc_2_11_3-x64; /opt/nvidia/nsight-systems/2023.1.1/target-linux-x64; /opt/nvidia/nsight-systems/2023.1.1/target-linux-x64/host-linux-x64; /opt/nvidia/nsight-systems/2023.1.1/target-linux-x64/linux-desktop-glibc_2_11_3-x64; /opt/nvidia/nsight-systems/2023.1.1/target-linux-x64/target-linux-x64; W17:57:39:972|quadd_common_core|10614|FileSystem.cpp:169[FindInstalledFile]: File 'config.ini' is not found: /opt/nvidia/nsight-systems/2023.1.1/host-linux-x64; /opt/nvidia/nsight-systems/2023.1.1/linux-desktop-glibc_2_11_3-x64; /opt/nvidia/nsight-systems/2023.1.1/target-linux-x64; /opt/nvidia/nsight-systems/2023.1.1/target-linux-x64/host-linux-x64; /opt/nvidia/nsight-systems/2023.1.1/target-linux-x64/linux-desktop-glibc_2_11_3-x64; /opt/nvidia/nsight-systems/2023.1.1/target-linux-x64/target-linux-x64; I17:57:39:972|quadd_daemon|10614|main.cpp:183[InitConfiguration]: NSys version[ @ ] built[] I17:57:39:972|quadd_daemon|10614|main.cpp:185[InitConfiguration]: Got config file path: I17:57:39:972|quadd_daemon|10614|main.cpp:126[InitPython]: Initing Embedded Python I17:57:39:972|quadd_common_core|10614|FileSystem.cpp:153[FindInstalledFile]: File 'python/lib' is found: /opt/nvidia/nsight-systems/2023.1.1/host-linux-x64/python/lib. I17:57:39:980|quadd_daemon|10614|main.cpp:133[InitPython]: Embedded Python Version: 3.8.3-final-0 I17:57:39:993|quadd_gpu_health|10614|GpuHealth.cpp:70[OpenAllDevicesImpl]: Opening device 0 (status=0) took 5323818 ns I17:57:39:998|quadd_gpu_health|10614|GpuHealth.cpp:70[OpenAllDevicesImpl]: Opening device 1 (status=0) took 5335372 ns I17:57:40:034|quadd_linux_perf|10614|event_fd.cpp:49[OpenEventFile]: perf_event_open(event CPU Clock (sw), attr.type=0x1, attr.config=0x0, tid 10614, cpu -1) returned fd 12 I17:57:40:034|quadd_linux_perf|10614|environment.cpp:250[CheckOSAndKernel]: Detected distribution: Ubuntu. Kernel minimal requirement: 4.3.0-0. Detected kernel: 5.15.0-67-generic I17:57:40:034|quadd_linux_perf|10614|event_fd.cpp:49[OpenEventFile]: perf_event_open(event Reference Cycles, attr.type=0x0, attr.config=0x9, tid 10614, cpu -1) returned fd 12 I17:57:40:034|quadd_linux_perf|10614|event_selection_set.cpp:595[GetDefaultTriggerEventIndex]: GetDefaultTriggerEventIndex(): Reference Cycles(0) event is default sampling trigger event I17:57:40:034|quadd_linux_perf|10614|event_fd.cpp:49[OpenEventFile]: perf_event_open(event Reference Cycles, attr.type=0x0, attr.config=0x9, tid 10614, cpu -1) returned fd 12 I17:57:40:035|quadd_linux_perf|10614|event_fd.cpp:49[OpenEventFile]: perf_event_open(event Reference Cycles, attr.type=0x0, attr.config=0x9, tid 10614, cpu -1) returned fd 12 I17:57:40:035|quadd_linux_perf|10614|event_fd.cpp:49[OpenEventFile]: perf_event_open(event CPU Clock (sw), attr.type=0x1, attr.config=0x0, tid 10614, cpu -1) returned fd 12 I17:57:40:035|quadd_linux_perf|10614|environment.cpp:250[CheckOSAndKernel]: Detected distribution: Ubuntu. Kernel minimal requirement: 4.3.0-0. Detected kernel: 5.15.0-67-generic I17:57:40:035|quadd_linux_perf|10614|event_fd.cpp:49[OpenEventFile]: perf_event_open(event Reference Cycles, attr.type=0x0, attr.config=0x9, tid 10614, cpu -1) returned fd 12 I17:57:40:035|quadd_linux_perf|10614|event_selection_set.cpp:595[GetDefaultTriggerEventIndex]: GetDefaultTriggerEventIndex(): Reference Cycles(0) event is default sampling trigger event I17:57:40:035|quadd_linux_perf|10614|event_fd.cpp:49[OpenEventFile]: perf_event_open(event Reference Cycles, attr.type=0x0, attr.config=0x9, tid 10614, cpu -1) returned fd 12 I17:57:40:035|quadd_linux_perf|10614|event_fd.cpp:49[OpenEventFile]: perf_event_open(event Reference Cycles, attr.type=0x0, attr.config=0x9, tid 10614, cpu -1) returned fd 12 I17:57:40:036|drvacc|10614|:64[]: Driver module override for Cuda I17:57:40:036|quadd_gpuinfo_dta|10614|DevToolsApi.cpp:25[CreatePersistenceGuard]: EnableGpuPersistence = 1 I17:57:40:038|drvacc|10614|:145[]: Trying to load driver module: 'libcuda.so.1': 0x4274dd0 I17:57:48:069|drvacc|10614|:64[]: Driver module override for Cuda I17:57:48:070|quadd_common_core|10614|AsyncProcessorHolder.h:28[AsyncProcessorHolder]: AsyncProcessorHolder[0x7ffebacbf7c0]: 2 AsyncProcessors I17:57:48:070|quadd_common_core|10614|AsyncProcessor.cpp:39[AsyncProcessor]: 1Comm@AsyncProcessor[0x4200fb0] is creating. I17:57:48:070|quadd_common_core|10614|AsyncProcessor.cpp:84[AsyncProcessor]: 1Comm@AsyncProcessor[0x4200fb0] created. I17:57:48:070|quadd_common_core|10614|AsyncProcessor.cpp:39[AsyncProcessor]: 2Worker@AsyncProcessor[0x42727c0] is creating. I17:57:48:070|quadd_common_core|10616|AsyncProcessor.cpp:55[operator()]: Thread[10616] started servicing 1Comm@AsyncProcessor. I17:57:48:070|quadd_common_core|10618|AsyncProcessor.cpp:55[operator()]: Thread[10618] started servicing 2Worker@AsyncProcessor. I17:57:48:070|quadd_common_core|10615|AsyncProcessor.cpp:55[operator()]: Thread[10615] started servicing 1Comm@AsyncProcessor. I17:57:48:070|quadd_common_core|10617|AsyncProcessor.cpp:55[operator()]: Thread[10617] started servicing 2Worker@AsyncProcessor. I17:57:48:070|quadd_common_core|10614|AsyncProcessor.cpp:84[AsyncProcessor]: 2Worker@AsyncProcessor[0x42727c0] created. I17:57:48:071|quadd_agent_api|10614|AgentAPI.cpp:381[CreateAPI]: Create local AgentAPI. I17:57:48:071|quadd_common_core|10616|FileSystem.cpp:153[FindInstalledFile]: File 'nsys' is found: /opt/nvidia/nsight-systems/2023.1.1/target-linux-x64/nsys. I17:57:48:071|quadd_agent_api|10616|Agent.cpp:141[MakeCommandString]: Agent start command: "/opt/nvidia/nsight-systems/2023.1.1/target-linux-x64/nsys" --start-agent --session-name=profile-esat-10614 --shm-name=quadd-create-agent-e70c7c02-926a-4670-9cf4-275d0788283d W17:57:48:078|quadd_common_core|10619|FileSystem.cpp:169[FindInstalledFile]: File 'nsys-config.ini' is not found: /opt/nvidia/nsight-systems/2023.1.1/host-linux-x64; /opt/nvidia/nsight-systems/2023.1.1/linux-desktop-glibc_2_11_3-x64; /opt/nvidia/nsight-systems/2023.1.1/target-linux-x64; /opt/nvidia/nsight-systems/2023.1.1/target-linux-x64/host-linux-x64; /opt/nvidia/nsight-systems/2023.1.1/target-linux-x64/linux-desktop-glibc_2_11_3-x64; /opt/nvidia/nsight-systems/2023.1.1/target-linux-x64/target-linux-x64; W17:57:48:078|quadd_common_core|10619|FileSystem.cpp:169[FindInstalledFile]: File 'config.ini' is not found: /opt/nvidia/nsight-systems/2023.1.1/host-linux-x64; /opt/nvidia/nsight-systems/2023.1.1/linux-desktop-glibc_2_11_3-x64; /opt/nvidia/nsight-systems/2023.1.1/target-linux-x64; /opt/nvidia/nsight-systems/2023.1.1/target-linux-x64/host-linux-x64; /opt/nvidia/nsight-systems/2023.1.1/target-linux-x64/linux-desktop-glibc_2_11_3-x64; /opt/nvidia/nsight-systems/2023.1.1/target-linux-x64/target-linux-x64; I17:57:48:078|quadd_daemon|10619|main.cpp:183[InitConfiguration]: NSys version[ @ ] built[] I17:57:48:078|quadd_daemon|10619|main.cpp:185[InitConfiguration]: Got config file path: I17:57:48:079|quadd_daemon|10621|RestartAsDaemon.cpp:131[RestartAsDaemon]: daemon is started I17:57:48:079|quadd_common_core|10621|AsyncProcessorHolder.h:28[AsyncProcessorHolder]: AsyncProcessorHolder[0x7ffc563f58e0]: 3 AsyncProcessors I17:57:48:079|quadd_common_core|10621|AsyncProcessor.cpp:39[AsyncProcessor]: 1MainService@AsyncProcessor[0x429dfb0] is creating. I17:57:48:079|quadd_common_core|10621|AsyncProcessor.cpp:84[AsyncProcessor]: 1MainService@AsyncProcessor[0x429dfb0] created. I17:57:48:079|quadd_common_core|10621|AsyncProcessor.cpp:39[AsyncProcessor]: 2TaskRunner@AsyncProcessor[0x429e230] is creating. I17:57:48:079|quadd_common_core|10621|AsyncProcessor.cpp:84[AsyncProcessor]: 2TaskRunner@AsyncProcessor[0x429e230] created. I17:57:48:079|quadd_common_core|10622|AsyncProcessor.cpp:55[operator()]: Thread[10622] started servicing 1MainService@AsyncProcessor. I17:57:48:079|quadd_common_core|10625|AsyncProcessor.cpp:55[operator()]: Thread[10625] started servicing 2TaskRunner@AsyncProcessor. I17:57:48:079|quadd_common_core|10621|AsyncProcessor.cpp:39[AsyncProcessor]: 3AnalysisServiceImpl@AsyncProcessor[0x429fc70] is creating. I17:57:48:079|quadd_common_core|10623|AsyncProcessor.cpp:55[operator()]: Thread[10623] started servicing 1MainService@AsyncProcessor. I17:57:48:080|quadd_common_core|10626|AsyncProcessor.cpp:55[operator()]: Thread[10626] started servicing 3AnalysisServiceImpl@AsyncProcessor. I17:57:48:079|quadd_common_core|10624|AsyncProcessor.cpp:55[operator()]: Thread[10624] started servicing 2TaskRunner@AsyncProcessor. I17:57:48:080|quadd_common_core|10621|AsyncProcessor.cpp:84[AsyncProcessor]: 3AnalysisServiceImpl@AsyncProcessor[0x429fc70] created. I17:57:48:080|quadd_common_core|10627|AsyncProcessor.cpp:55[operator()]: Thread[10627] started servicing 3AnalysisServiceImpl@AsyncProcessor. I17:57:48:080|quadd_common_session_manager|10621|Helper.cpp:70[GetFileLock]: Getting session manager file lock: mode=0 path=/tmp/nvidia/nsight_systems/SessionManager-32365746v0.lock. I17:57:48:083|quadd_common_session_manager|10621|Helper.cpp:70[GetFileLock]: Getting session manager file lock: mode=1 path=/tmp/nvidia/nsight_systems/SessionManager-32365746v0.lock. I17:57:48:083|quadd_common_session_manager|10621|BaseSessionManager.cpp:447[CreateUser]: Creating user "sscott" I17:57:48:083|quadd_common_session_manager|10621|BaseSessionManager.cpp:267[CreateSession]: Creating session "profile-esat-10614" with ID 1010621 I17:57:48:083|quadd_agent|10621|SessionRegistry.cpp:40[SessionRegistry]: Session registered: id=1010621 name=profile-esat-10614. I17:57:48:083|quadd_daemon_timesync|10621|TimeSyncService.cpp:391[DoBegin]: TimeSyncServiceImpl[0x42a3040]: #DoBegin begin I17:57:48:083|quadd_daemon_timesync|10621|TimeSyncService.cpp:561[Reset]: TimeSyncServiceImpl[0x42a3040]: #Reset begin I17:57:48:083|quadd_daemon_timesync|10621|TimeSyncService.cpp:581[Reset]: TimeSyncServiceImpl[0x42a3040]: #Reset end I17:57:48:083|quadd_utc_timecor|10621|UtcTimeCorrelation.cpp:48[GetResult]: GetResult[(cpu=3220984435910, utc=1678298268083682647)] offset=-1678295047099246737 I17:57:48:083|quadd_local_timecor|10621|LocaltimeCorrelation.cpp:33[GetResult]: GetResult[(utc=1678298268, local=1678298268)] offset=0 I17:57:48:084|drvacc|10621|:64[]: Driver module override for Cuda E17:58:18:071|quadd_agent_api|10616|Agent.cpp:74[operator()]: Agent launcher exceeded its timeout. I17:58:18:071|quadd_common_core|10614|AsyncProcessorHolder.h:62[Terminate]: AsyncProcessorHolder[0x7ffebacbf7c0]: Stopping I17:58:18:071|quadd_common_core|10614|AsyncProcessor.cpp:110[Stop]: 1Comm@AsyncProcessor[0x4200fb0] is stopping. I17:58:18:071|quadd_common_core|10614|AsyncProcessor.cpp:124[Stop]: 1Comm@AsyncProcessor[0x4200fb0] is waiting for work threads finish. I17:58:18:071|quadd_common_core|10615|AsyncProcessor.cpp:78[operator()]: Thread[10615] stopped servicing 1Comm@AsyncProcessor. I17:58:18:072|quadd_agent_api|10616|Agent.cpp:61[operator()]: Agent launcher exited: exit code=9 E17:58:18:072|quadd_agent_api|10616|LocalAgentAPI.cpp:244[operator()]: Failed to create session: /dvs/p4/build/sw/devtools/Agora/Rel/QuadD_Main/QuadD/Common/AgentAPI/Src/Agent.cpp(105): Throw in function void {anonymous}::SpawnAgentProcess(Agent::CreateParams&&) Dynamic exception type: boost::wrapexcept std::exception::what: RuntimeException [QuadDCommon::tag_message*] = Agent launcher failed. [QuadDCommon::tag_exit_code*] = 9 . I17:58:18:072|quadd_common_core|10616|AsyncProcessor.cpp:78[operator()]: Thread[10616] stopped servicing 1Comm@AsyncProcessor. I17:58:18:072|quadd_common_core|10614|AsyncProcessor.cpp:132[Stop]: 1Comm@AsyncProcessor[0x4200fb0] stopped. I17:58:18:072|quadd_common_core|10614|AsyncProcessor.cpp:110[Stop]: 2Worker@AsyncProcessor[0x42727c0] is stopping. I17:58:18:072|quadd_common_core|10614|AsyncProcessor.cpp:124[Stop]: 2Worker@AsyncProcessor[0x42727c0] is waiting for work threads finish. I17:58:18:072|quadd_common_core|10617|AsyncProcessor.cpp:78[operator()]: Thread[10617] stopped servicing 2Worker@AsyncProcessor. I17:58:18:072|quadd_common_core|10618|AsyncProcessor.cpp:78[operator()]: Thread[10618] stopped servicing 2Worker@AsyncProcessor. I17:58:18:072|quadd_common_core|10614|AsyncProcessor.cpp:132[Stop]: 2Worker@AsyncProcessor[0x42727c0] stopped. I17:58:18:072|quadd_common_core|10614|AsyncProcessorHolder.h:78[Terminate]: AsyncProcessorHolder[0x7ffebacbf7c0]: Destroying F17:58:18:072|quadd_daemon|10614|main.cpp:642[main]: Fatal error: /dvs/p4/build/sw/devtools/Agora/Rel/QuadD_Main/QuadD/Common/AgentAPI/Src/Controller.cpp(825): Throw in function Agent::ISession::Ptr Agent::Client::{anonymous}::CreateSession(const Agent::Command&, const Ptr&) Dynamic exception type: boost::wrapexcept std::exception::what: RuntimeException [QuadDCommon::tag_message*] = The timeout expired. I17:58:20:164|quadd_common_core|10621|FileSystem.cpp:153[FindInstalledFile]: File 'CudaGpuInfoDumper' is found: /opt/nvidia/nsight-systems/2023.1.1/target-linux-x64/CudaGpuInfoDumper. I17:58:20:164|quadd_gpuinfo_dta|10621|DevToolsApi.cpp:25[CreatePersistenceGuard]: EnableGpuPersistence = 1 I17:58:28:193|drvacc|10621|:64[]: Driver module override for Cuda I17:58:36:229|quadd_gpuinfo_cta|10621|CudaToolsApi.cpp:441[InitializeGpuInfoListOutOfProcess]: Launching CudaGpuInfoDumper to collect gpu details for 1 processes I17:58:36:229|quadd_gputraits|10621|GpuTicksConverter.cpp:160[DiscreteGpuDevice]: GpuTimeInterpolateFromFirstToLast = 0 I17:58:36:230|quadd_gputraits|10621|GpuTicksConverter.cpp:160[DiscreteGpuDevice]: GpuTimeInterpolateFromFirstToLast = 0 I17:58:36:230|quadd_daemon|10621|TimeConverter.cpp:138[Refine]: rdtsc -> monotonic conversion factors: I17:58:36:230|quadd_daemon|10621|TimeConverter.cpp:139[Refine]: Start sync point = {3990393735791784, 3220984462150} I17:58:36:230|quadd_daemon|10621|TimeConverter.cpp:142[Refine]: End sync point = {3990533361295451, 3269131402363} I17:58:36:230|quadd_daemon|10621|TimeConverter.cpp:145[Refine]: Multiplier = 0.344829124683612 I17:58:36:230|quadd_daemon|10621|TimeConverter.cpp:218[Refine]: CLOCK_MONOTONIC_RAW -> CLOCK_MONOTONIC conversion factors: I17:58:36:230|quadd_daemon|10621|TimeConverter.cpp:219[Refine]: Start sync point = {3219666564297, 3220984462748} I17:58:36:230|quadd_daemon|10621|TimeConverter.cpp:222[Refine]: End sync point = {3267813324131, 3269131419891} I17:58:36:230|quadd_daemon|10621|TimeConverter.cpp:225[Refine]: Multiplier = 1.000004098074319 I17:58:36:230|quadd_gputraits|10621|GpuTicksConverter.cpp:185[Refine]: GPU 12544 time -> CPU time conversion factors: N sync point = {1678298316704218624, 3990533359193786} N+1 sync point = {1678298316705255712, 3990533362201375} Multiplier from N to N+1 = 2.900032591255516 Multiplier from first to N+1 = 2.900032591255516 I17:58:36:231|quadd_gputraits|10621|GpuTicksConverter.cpp:185[Refine]: GPU 19200 time -> CPU time conversion factors: N sync point = {1678298316711056704, 3990533361120835} N+1 sync point = {1678298316711771232, 3990533363192972} Multiplier from N to N+1 = 2.900008117246630 Multiplier from first to N+1 = 2.900008117246630 I17:58:36:231|quadd_tsc_timecor|10621|TscGlobalTimeCorrelation.cpp:38[GetResult]: GetResult[(cpu=3269131388334, tsc=3269131388333)] offset=1 I17:58:36:231|quadd_daemon_timesync|10621|TimeSyncService.cpp:509[DoEnd]: TimeSyncServiceImpl[0x42a3040]: #DoEnd begin I17:58:36:231|quadd_daemon_timesync|10621|TimeSyncService.cpp:561[Reset]: TimeSyncServiceImpl[0x42a3040]: #Reset begin I17:58:36:231|quadd_daemon_timesync|10621|TimeSyncService.cpp:581[Reset]: TimeSyncServiceImpl[0x42a3040]: #Reset end I17:58:36:231|quadd_daemon_timesync|10621|TimeSyncService.cpp:519[DoEnd]: TimeSyncServiceImpl[0x42a3040]: Sync adjustment 18446740852725143630 I17:58:36:241|quadd_daemon_timesync|10621|TimeSyncService.cpp:561[Reset]: TimeSyncServiceImpl[0x42a3040]: #Reset begin I17:58:36:242|quadd_daemon_timesync|10621|TimeSyncService.cpp:581[Reset]: TimeSyncServiceImpl[0x42a3040]: #Reset end I17:58:36:242|quadd_agent_session|10621|Session.cpp:365[Session]: Session created. I17:58:36:242|quadd_pbcomm_server|10621|Server.cpp:44[Server]: Server[0x42b64b0]: service[0] AgentService I17:58:36:242|quadd_pbcomm_server|10621|Server.cpp:48[Server]: Server[0x42b64b0] created. I17:58:36:242|quadd_pbcomm_server|10621|Server.cpp:77[Start]: Server[0x42b64b0] starts session acceptor. I17:58:36:242|quadd_pbcomm_tcp|10621|Communicator.cpp:195[Acceptor]: Acceptor[0x42a1000] created. I17:58:36:242|quadd_pbcomm_tcp|10621|Communicator.cpp:545[CommunicatorCreator]: CommunicatorCreator[0x42db8d0] created: port=0 I17:58:36:242|quadd_pbcomm_server|10621|Server.cpp:83[Start]: Server[0x42b64b0] started. I17:58:36:242|quadd_pbcomm_tcp|10623|Communicator.cpp:208[Start]: Acceptor[0x42a1000]: starts accepting. I17:58:36:242|quadd_pbcomm_server|10621|Server.cpp:55[~Server]: Server[0x42b64b0] is closing communicators. I17:58:36:242|quadd_pbcomm_server|10621|Server.cpp:60[~Server]: Server[0x42b64b0] destroyed. I17:58:36:242|quadd_common_session_manager|10621|BaseSessionManager.cpp:355[RemoveUser]: Removing user "sscott" I17:58:36:242|quadd_common_session_manager|10621|BaseSessionManager.cpp:554[RemoveSession]: Removing session "profile-esat-10614" with ID 1010621 I17:58:36:242|quadd_agent|10621|SessionRegistry.cpp:52[~SessionRegistry]: Session unregistered: id=1010621. I17:58:36:242|quadd_common_session_manager|10621|Helper.cpp:70[GetFileLock]: Getting session manager file lock: mode=1 path=/tmp/nvidia/nsight_systems/SessionManager-32365746v0.lock. I17:58:36:242|quadd_pbcomm_tcp|10623|Communicator.cpp:250[Terminate]: Acceptor[0x42a1000] terminating I17:58:36:242|quadd_pbcomm_tcp|10623|Communicator.cpp:554[~CommunicatorCreator]: CommunicatorCreator[0x42db8d0] destroyed. E17:58:36:242|quadd_pbcomm_tcp|10623|Communicator.cpp:219[Handle]: Acceptor[0x42a1000]: accept failed: Operation canceled I17:58:36:242|quadd_pbcomm_tcp|10623|Communicator.cpp:200[~Acceptor]: Acceptor[0x42a1000] destroyed. I17:58:36:242|quadd_common_session_manager|10621|Helper.cpp:70[GetFileLock]: Getting session manager file lock: mode=1 path=/tmp/nvidia/nsight_systems/SessionManager-32365746v0.lock. E17:58:36:242|quadd_common_session_manager|10621|BaseSessionManager.cpp:63[OpenSharedMemoryObjects]: Failed to open ManagedSharedMemory(NSys-32365746v0): Dynamic exception type: boost::interprocess::interprocess_exception std::exception::what: No such file or directory . I17:58:36:242|quadd_common_core|10621|AsyncProcessorHolder.h:62[Terminate]: AsyncProcessorHolder[0x7ffc563f58e0]: Stopping I17:58:36:242|quadd_common_core|10621|AsyncProcessor.cpp:110[Stop]: 1MainService@AsyncProcessor[0x429dfb0] is stopping. I17:58:36:242|quadd_common_core|10621|AsyncProcessor.cpp:124[Stop]: 1MainService@AsyncProcessor[0x429dfb0] is waiting for work threads finish. I17:58:36:242|quadd_common_core|10623|AsyncProcessor.cpp:78[operator()]: Thread[10623] stopped servicing 1MainService@AsyncProcessor. I17:58:36:243|quadd_common_core|10622|AsyncProcessor.cpp:78[operator()]: Thread[10622] stopped servicing 1MainService@AsyncProcessor. I17:58:36:243|quadd_common_core|10621|AsyncProcessor.cpp:132[Stop]: 1MainService@AsyncProcessor[0x429dfb0] stopped. I17:58:36:243|quadd_common_core|10621|AsyncProcessor.cpp:110[Stop]: 2TaskRunner@AsyncProcessor[0x429e230] is stopping. I17:58:36:243|quadd_common_core|10621|AsyncProcessor.cpp:124[Stop]: 2TaskRunner@AsyncProcessor[0x429e230] is waiting for work threads finish. I17:58:36:243|quadd_common_core|10625|AsyncProcessor.cpp:78[operator()]: Thread[10625] stopped servicing 2TaskRunner@AsyncProcessor. I17:58:36:243|quadd_common_core|10624|AsyncProcessor.cpp:78[operator()]: Thread[10624] stopped servicing 2TaskRunner@AsyncProcessor. I17:58:36:243|quadd_common_core|10621|AsyncProcessor.cpp:132[Stop]: 2TaskRunner@AsyncProcessor[0x429e230] stopped. I17:58:36:243|quadd_common_core|10621|AsyncProcessor.cpp:110[Stop]: 3AnalysisServiceImpl@AsyncProcessor[0x429fc70] is stopping. I17:58:36:243|quadd_common_core|10621|AsyncProcessor.cpp:124[Stop]: 3AnalysisServiceImpl@AsyncProcessor[0x429fc70] is waiting for work threads finish. I17:58:36:243|quadd_common_core|10627|AsyncProcessor.cpp:78[operator()]: Thread[10627] stopped servicing 3AnalysisServiceImpl@AsyncProcessor. I17:58:36:243|quadd_common_core|10626|AsyncProcessor.cpp:78[operator()]: Thread[10626] stopped servicing 3AnalysisServiceImpl@AsyncProcessor. I17:58:36:243|quadd_common_core|10621|AsyncProcessor.cpp:132[Stop]: 3AnalysisServiceImpl@AsyncProcessor[0x429fc70] stopped. I17:58:36:243|quadd_common_core|10621|AsyncProcessorHolder.h:78[Terminate]: AsyncProcessorHolder[0x7ffc563f58e0]: Destroying