I14:38:08:277|Injection|13360|Init.cpp:1877[InitializeInjection]: Starting CUDA injection initialization
I14:38:08:280|quadd_common_core|13360|Config.cpp:87[InternalReset]: Loaded config file: /tmp/injection_config_0f42de8d
I14:38:08:280|Injection|13360|Init.cpp:1306[InitializeInjectionCommon]: UseAgentAPI = 1

I14:38:08:280|Injection|13360|Manager.cpp:190[Initialize]: Initialize::start.
I14:38:08:280|quadd_common_core|13360|AsyncProcessorHolder.h:28[AsyncProcessorHolder]: AsyncProcessorHolder[0x2155b0b3b20]: 2 AsyncProcessors
I14:38:08:280|quadd_common_core|13360|AsyncProcessor.cpp:39[AsyncProcessor]: 1TaskRunner@AsyncProcessor[0x2155b1a0100] is creating.
I14:38:08:281|quadd_common_core|13360|AsyncProcessor.cpp:84[AsyncProcessor]: 1TaskRunner@AsyncProcessor[0x2155b1a0100] created.
I14:38:08:281|quadd_common_core|13360|AsyncProcessor.cpp:39[AsyncProcessor]: 2CommsProcessor@AsyncProcessor[0x2155b1a0180] is creating.
I14:38:08:281|quadd_common_core|13369|AsyncProcessor.cpp:55[operator()]: Thread[13369] started servicing 1TaskRunner@AsyncProcessor.
I14:38:08:281|quadd_common_core|13360|AsyncProcessor.cpp:84[AsyncProcessor]: 2CommsProcessor@AsyncProcessor[0x2155b1a0180] created.
I14:38:08:281|quadd_common_core|13370|AsyncProcessor.cpp:55[operator()]: Thread[13370] started servicing 2CommsProcessor@AsyncProcessor.
I14:38:08:281|quadd_agent_api|13360|AgentAPI.cpp:381[CreateAPI]: Create local AgentAPI.
I14:38:08:283|quadd_pbcomm_tcp|13370|Communicator.cpp:536[CommunicatorCreator]: CommunicatorCreator[0x2155b840000] created.
I14:38:08:283|quadd_pbcomm_proxy|13370|ClientProxy.cpp:55[ClientProxy]: ClientProxy[0x2155b820280] created.
I14:38:08:283|quadd_pbcomm_proxy|13370|ClientProxy.cpp:138[HandleStart]: ClientProxy[0x2155b820280] is starting.
I14:38:08:283|quadd_pbcomm_tcp|13370|Communicator.cpp:296[Connector]: Connector[0x2155b8c0000] created.
I14:38:08:283|quadd_pbcomm_tcp|13370|Communicator.cpp:301[Start]: Connector[0x2155b8c0000] is connecting to 127.0.0.1:40455 .
I14:38:08:283|quadd_pbcomm_tcp|13370|Communicator.cpp:311[Start]: Connector[0x2155b8c0000] set timeout 60 seconds.
I14:38:08:283|quadd_pbcomm_tcp|13370|Communicator.cpp:358[HandleConnect]: Connector[0x2155b8c0000] connected.
I14:38:08:283|quadd_pbcomm_tcp|13370|Communicator.cpp:361[HandleConnect]: Connector[0x2155b8c0000]: reading BuildId.
I14:38:08:283|quadd_pbcomm_tcp|13370|Communicator.cpp:443[HandleRead]: Connector[0x2155b8c0000]: BuildId checked: df9881ff65bb8845f5d270f4cf683ed5cb39a957
I14:38:08:283|quadd_pbcomm_tcp|13370|Communicator.cpp:71[Communicator]: Communicator[0x2155b8b01d0] created.
I14:38:08:283|quadd_pbcomm_tcp|13370|Communicator.cpp:284[~Connector]: Connector[0x2155b8c0000] destroyed.
I14:38:08:283|quadd_pbcomm_proxy|13370|ClientProxy.cpp:156[HandleConnect]: ClientProxy[0x2155b820280] connected to the server.
I14:38:08:283|Injection|13360|Manager.cpp:241[Initialize]: Injection library attached to session successfully.
I14:38:08:283|Injection|13360|EventHandlerImpl.cpp:755[RegisterStartStopAnalysisHandler]: Registered NVTX start/stop handler (0x2155b0a1140)
I14:38:08:283|Injection|13360|AppTraceController.cpp:32[AppTraceController]: AppTraceController[0x2155b360010,13341]: 
I14:38:08:283|quadd_common_core|13360|Config.cpp:87[InternalReset]: Loaded config file: /tmp/injection_config_0f42de8d
I14:38:08:481|Injection|13370|ProfilerApiImpl.cpp:461[Notify]: Capture range trigger = 0, capture range end action = 2
I14:38:08:481|Injection|13370|ProfilerApiImpl.cpp:473[Notify]: Agent state = Collection
I14:38:08:482|Injection|13360|Init.cpp:1157[IsRecordingFromSessionState]: Session state = Collection
I14:38:08:482|Injection|13360|Init.cpp:1410[InitializeInjectionCommon]: Starting recording.
I14:38:08:482|Injection|13360|EventHandlerImpl.cpp:268[StartRecording]: Starting the recording
I14:38:08:482|Injection|13360|StorageWriterCreator.cpp:44[Create]: Injection exchange file: '/tmp/nvidia/nsight_systems/quadd_session_1013348/injection_files/injection_13360_0_storage.dat'
I14:38:08:482|Injection|13360|EventHandlerImpl.cpp:276[StartRecording]: Calling the start handlers
I14:38:08:482|Injection|13360|Init.cpp:434[RecordServiceTraceEventOfType]: Sending service event: NVTXStart.
I14:38:08:483|Injection|13360|EventHandlerImpl.cpp:284[StartRecording]: Recording started
I14:38:08:483|Injection|13360|Init.cpp:1466[InitializeInjectionCommon]: Injection Common Init OK
I14:38:08:483|Injection|13360|CudaInjectionInit.cpp:1984[RegisterEventHandlerCUDA]: RegisterEventHandlerCUDA() START.
I14:38:08:483|Injection|13370|Init.cpp:470[NameServiceThread]: Naming service thread 13370 as [NSys Comms].
W14:38:08:483|Injection|13360|CudaInjectionInit.cpp:1586[PrepareAgentConfiguration]: Failed to get Agent configuration
I14:38:08:483|quadd_common_core|13360|Config.cpp:87[InternalReset]: Loaded config file: /tmp/injection_config_d31b3df1
I14:38:08:483|Injection|13360|CudaInjectionInit.cpp:186[CheckFlushOnCudaProfilerStop]: Buffers holding CUDA trace data will be flushed on CudaProfilerStop() call.
I14:38:08:483|Injection|13360|common.cpp:72[CuptiUseRawGpuTimestamps]: CUPTI raw GPU timestamp mode: 1
I14:38:08:483|Injection|13369|Init.cpp:470[NameServiceThread]: Naming service thread 13369 as [NSys].
I14:38:08:485|quadd_cuda_tracing|13360|CuptiLibrary.cpp:89[LoadCuptiLibrary]: Loaded CUPTI library /opt/nvidia/nsight-systems/2022.4.2/target-linux-x64/libcupti.so.11.6
W14:38:08:487|quadd_cuda_tracing|13360|Configuration.cpp:72[SanitizeActivities]: DTCUPTI-2456: Disabling CUDA event activity
I14:38:08:489|nvetbls|13360|:186[]: Initialize etbl: 7

I14:38:08:490|quadd_cuda_tracing|13360|Configuration.cpp:126[SanitizeOptions]: Flush interval set to 0
I14:38:08:490|quadd_cuda_tracing|13360|InjectionInterface.cpp:476[ConfigureCupti]: CUPTI raw CPU timestamp mode: 0
I14:38:08:496|Injection|13360|CudaInjectionInit.cpp:286[AddBuffer]: New CUPTI buffer allocated
I14:38:08:502|Injection|13360|CudaInjectionInit.cpp:286[AddBuffer]: New CUPTI buffer allocated
I14:38:08:508|Injection|13360|CudaInjectionInit.cpp:286[AddBuffer]: New CUPTI buffer allocated
I14:38:08:513|Injection|13360|CudaInjectionInit.cpp:286[AddBuffer]: New CUPTI buffer allocated
I14:38:08:519|Injection|13360|CudaInjectionInit.cpp:286[AddBuffer]: New CUPTI buffer allocated
I14:38:08:525|Injection|13360|CudaInjectionInit.cpp:286[AddBuffer]: New CUPTI buffer allocated
I14:38:08:531|Injection|13360|CudaInjectionInit.cpp:286[AddBuffer]: New CUPTI buffer allocated
I14:38:08:537|Injection|13360|CudaInjectionInit.cpp:286[AddBuffer]: New CUPTI buffer allocated
I14:38:08:543|Injection|13360|CudaInjectionInit.cpp:286[AddBuffer]: New CUPTI buffer allocated
I14:38:08:548|Injection|13360|CudaInjectionInit.cpp:286[AddBuffer]: New CUPTI buffer allocated
I14:38:08:554|Injection|13360|CudaInjectionInit.cpp:286[AddBuffer]: New CUPTI buffer allocated
I14:38:08:560|Injection|13360|CudaInjectionInit.cpp:286[AddBuffer]: New CUPTI buffer allocated
I14:38:08:566|Injection|13360|CudaInjectionInit.cpp:286[AddBuffer]: New CUPTI buffer allocated
I14:38:08:572|Injection|13360|CudaInjectionInit.cpp:286[AddBuffer]: New CUPTI buffer allocated
I14:38:08:578|Injection|13360|CudaInjectionInit.cpp:286[AddBuffer]: New CUPTI buffer allocated
I14:38:08:584|Injection|13360|CudaInjectionInit.cpp:286[AddBuffer]: New CUPTI buffer allocated
I14:38:08:589|Injection|13360|CudaInjectionInit.cpp:286[AddBuffer]: New CUPTI buffer allocated
I14:38:08:595|Injection|13360|CudaInjectionInit.cpp:286[AddBuffer]: New CUPTI buffer allocated
I14:38:08:601|Injection|13360|CudaInjectionInit.cpp:286[AddBuffer]: New CUPTI buffer allocated
I14:38:08:607|Injection|13360|CudaInjectionInit.cpp:286[AddBuffer]: New CUPTI buffer allocated
I14:38:08:612|Injection|13360|CudaInjectionInit.cpp:286[AddBuffer]: New CUPTI buffer allocated
I14:38:08:618|Injection|13360|CudaInjectionInit.cpp:286[AddBuffer]: New CUPTI buffer allocated
I14:38:08:624|Injection|13360|CudaInjectionInit.cpp:286[AddBuffer]: New CUPTI buffer allocated
I14:38:08:630|Injection|13360|CudaInjectionInit.cpp:286[AddBuffer]: New CUPTI buffer allocated
I14:38:08:636|Injection|13360|CudaInjectionInit.cpp:286[AddBuffer]: New CUPTI buffer allocated
I14:38:08:642|Injection|13360|CudaInjectionInit.cpp:286[AddBuffer]: New CUPTI buffer allocated
I14:38:08:647|Injection|13360|CudaInjectionInit.cpp:286[AddBuffer]: New CUPTI buffer allocated
I14:38:08:653|Injection|13360|CudaInjectionInit.cpp:286[AddBuffer]: New CUPTI buffer allocated
I14:38:08:659|Injection|13360|CudaInjectionInit.cpp:286[AddBuffer]: New CUPTI buffer allocated
I14:38:08:665|Injection|13360|CudaInjectionInit.cpp:286[AddBuffer]: New CUPTI buffer allocated
I14:38:08:671|Injection|13360|CudaInjectionInit.cpp:286[AddBuffer]: New CUPTI buffer allocated
I14:38:08:676|Injection|13360|CudaInjectionInit.cpp:286[AddBuffer]: New CUPTI buffer allocated
I14:38:08:682|Injection|13360|CudaInjectionInit.cpp:286[AddBuffer]: New CUPTI buffer allocated
I14:38:08:688|Injection|13360|CudaInjectionInit.cpp:286[AddBuffer]: New CUPTI buffer allocated
I14:38:08:694|Injection|13360|CudaInjectionInit.cpp:286[AddBuffer]: New CUPTI buffer allocated
I14:38:08:700|Injection|13360|CudaInjectionInit.cpp:286[AddBuffer]: New CUPTI buffer allocated
I14:38:08:705|Injection|13360|CudaInjectionInit.cpp:286[AddBuffer]: New CUPTI buffer allocated
I14:38:08:711|Injection|13360|CudaInjectionInit.cpp:286[AddBuffer]: New CUPTI buffer allocated
I14:38:08:717|Injection|13360|CudaInjectionInit.cpp:286[AddBuffer]: New CUPTI buffer allocated
I14:38:08:723|Injection|13360|CudaInjectionInit.cpp:286[AddBuffer]: New CUPTI buffer allocated
I14:38:08:729|Injection|13360|CudaInjectionInit.cpp:286[AddBuffer]: New CUPTI buffer allocated
I14:38:08:734|Injection|13360|CudaInjectionInit.cpp:286[AddBuffer]: New CUPTI buffer allocated
I14:38:08:740|Injection|13360|CudaInjectionInit.cpp:286[AddBuffer]: New CUPTI buffer allocated
I14:38:08:746|Injection|13360|CudaInjectionInit.cpp:286[AddBuffer]: New CUPTI buffer allocated
I14:38:08:752|Injection|13360|CudaInjectionInit.cpp:286[AddBuffer]: New CUPTI buffer allocated
I14:38:08:758|Injection|13360|CudaInjectionInit.cpp:286[AddBuffer]: New CUPTI buffer allocated
I14:38:08:763|Injection|13360|CudaInjectionInit.cpp:286[AddBuffer]: New CUPTI buffer allocated
I14:38:08:769|Injection|13360|CudaInjectionInit.cpp:286[AddBuffer]: New CUPTI buffer allocated
I14:38:08:775|Injection|13360|CudaInjectionInit.cpp:286[AddBuffer]: New CUPTI buffer allocated
I14:38:08:781|Injection|13360|CudaInjectionInit.cpp:286[AddBuffer]: New CUPTI buffer allocated
I14:38:08:781|Injection|13360|CudaInjectionInit.cpp:250[CuptiExpandableBufferManager]: Initial no. of CUPTI buffers = 50 of size 10485760 bytes each. Expandable on demand to 71017949184 bytes total
I14:38:08:781|quadd_common_core|13360|Config.cpp:87[InternalReset]: Loaded config file: /tmp/injection_config_d31b3df1
I14:38:08:781|Injection|13360|EventHandlerImpl.cpp:755[RegisterStartStopAnalysisHandler]: Registered CUDA start/stop handler (0x2155b0c0740)
I14:38:08:781|Injection|13360|CudaInjectionInit.cpp:840[OnStartAnalysis]: CUDA: OnStartAnalysis: START.
I14:38:08:781|quadd_common_core|13360|Config.cpp:87[InternalReset]: Loaded config file: /tmp/injection_config_d31b3df1
I14:38:08:789|quadd_cuda_tracing|13360|InjectionInterface.cpp:723[StartCudaTracing]: CUDA tracing started
I14:38:08:789|Injection|13360|Init.cpp:434[RecordServiceTraceEventOfType]: Sending service event: CUDAStart.
I14:38:08:789|Injection|13360|CudaInjectionInit.cpp:880[OnStartAnalysis]: CUDA: OnStartAnalysis: DONE.
I14:38:08:789|Injection|13360|CudaInjectionInit.cpp:2014[RegisterEventHandlerCUDA]: RegisterEventHandlerCUDA() DONE.
I14:38:08:844|Injection|13360|CudaInjectionInit.cpp:981[bufferRequested]: Provided a new buffer for CUPTI.
I14:38:08:846|quadd_cuda_tracing|13360|CallbackHandler.cpp:411[EnableUvmActivity]: CUDA device 0: Unified Memory trace is supported.
I14:38:08:848|quadd_cuda_tracing|13360|Handle.cpp:184[EnableUnifiedMemoryActivity]: UVM enabled
I14:38:09:213|Injection|13360|CudaInjectionInit.cpp:1343[OnContextCreated]: Context with ID 1 created on device 0
I14:38:09:217|Injection|13360|Manager.cpp:299[Deinitialize]: Deinitialize::start.
I14:38:09:217|Injection|13360|CudaInjectionInit.cpp:886[OnApplicationExit]: CUDA: OnApplicationExit: START.
I14:38:09:217|Injection|13372|CudaInjectionInit.cpp:995[bufferCompleted]: CUPTI buffer completed, validSize=5640
I14:38:09:217|Injection|13372|Init.cpp:470[NameServiceThread]: Naming service thread 13372 as CUPTI worker thread.
I14:38:09:218|Injection|13372|CuptiToFlatDataConverter.cpp:970[process_CUPTI_ACTIVITY_KIND_DEVICE]: ordinal[0] name[Tesla V100-SXM2-32GB] uuid[daebe6e5-2de1-b558-7ec8-2749b04987b1]
I14:38:09:218|Injection|13372|CudaInjectionInit.cpp:1133[HandleBuffer]: Processed 30 CUPTI events.
I14:38:09:219|Injection|13360|CudaInjectionInit.cpp:719[ReportCUDADone]: CUDA: ReportCUDADone: START.
I14:38:09:219|Injection|13360|Init.cpp:434[RecordServiceTraceEventOfType]: Sending service event: CUDAFinish.
I14:38:09:219|Injection|13360|CudaInjectionInit.cpp:722[ReportCUDADone]: CUDA: ReportCUDADone: DONE.
I14:38:09:219|Injection|13360|CudaInjectionInit.cpp:1212[FinalDiagnosticHandler]: CUPTI: Number of CUPTI events produced: 30, CUPTI buffers in use: 0.
I14:38:09:219|Injection|13360|CudaInjectionInit.cpp:900[OnApplicationExit]: CUDA: OnApplicationExit: DONE.
I14:38:09:219|Injection|13360|EventHandlerImpl.cpp:311[StopRecording]: Calling analysis stop handlers
I14:38:09:219|Injection|13360|Init.cpp:434[RecordServiceTraceEventOfType]: Sending service event: NVTXFinish.
I14:38:09:219|Injection|13360|EventHandlerImpl.cpp:338[StopRecording]: Flushing the rest of events.
I14:38:09:220|Injection|13360|EventHandlerImpl.cpp:370[StopRecording]: The storage is about to be reset.
I14:38:09:220|Injection|13360|EventHandlerImpl.cpp:377[StopRecording]: The storage has been reset.
I14:38:09:220|Injection|13360|EventHandlerImpl.cpp:771[DeregisterStartStopAnalysisHandler]: Deregistered NVTX start/stop handler (0x2155b0a1140)
I14:38:09:220|Injection|13360|EventHandlerImpl.cpp:193[~EventHandlerImpl]: Event handler is being destructed 13360
I14:38:09:220|Injection|13360|EventHandlerImpl.cpp:53[~ThreadData]: ThreadData 0x2155b0a0b00 is destructed 13360
I14:38:09:220|Injection|13360|EventHandlerImpl.cpp:209[~EventHandlerImpl]: EventHandlerImpl: queues - 4 (0) stopped queues - 0
I14:38:09:220|quadd_pbcomm_proxy|13370|ClientProxy.cpp:70[HandleTerminate]: ClientProxy[0x2155b820280] is terminating.
I14:38:09:220|Injection|13360|EventHandlerImpl.cpp:209[~EventHandlerImpl]: EventHandlerImpl: queues - 4 (0) stopped queues - 0
I14:38:09:220|quadd_pbcomm_tcp|13370|Communicator.cpp:554[~CommunicatorCreator]: CommunicatorCreator[0x2155b840000] destroyed.
I14:38:09:220|quadd_common_core|13360|AsyncProcessorHolder.h:62[Terminate]: AsyncProcessorHolder[0x2155b0b3b20]: Stopping
I14:38:09:220|quadd_pbcomm_tcp|13370|Communicator.cpp:87[Terminate]: Communicator[0x2155b8b01d0]: Socket is shutting down.
I14:38:09:220|quadd_common_core|13360|AsyncProcessor.cpp:110[Stop]: 1TaskRunner@AsyncProcessor[0x2155b1a0100] is stopping.
I14:38:09:220|quadd_common_core|13360|AsyncProcessor.cpp:124[Stop]: 1TaskRunner@AsyncProcessor[0x2155b1a0100] is waiting for work threads finish.
I14:38:09:220|quadd_pbcomm_tcp|13370|Communicator.cpp:82[~Communicator]: Communicator[0x2155b8b01d0] destroyed.
I14:38:09:220|quadd_common_core|13369|AsyncProcessor.cpp:78[operator()]: Thread[13369] stopped servicing 1TaskRunner@AsyncProcessor.
W14:38:09:220|quadd_pbcomm_proxy|13370|ClientProxy.cpp:253[HandleReadMessage]: ClientProxy[0x2155b820280]: Read message failed: Operation canceled
I14:38:09:220|quadd_pbcomm_proxy|13370|ClientProxy.cpp:256[HandleReadMessage]: ClientProxy[0x2155b820280] is canceling all the outstanding requests.
I14:38:09:220|quadd_pbcomm_proxy|13370|ClientProxy.cpp:60[~ClientProxy]: ClientProxy[0x2155b820280] is destroying.
I14:38:09:220|Injection|13369|EventHandlerImpl.cpp:53[~ThreadData]: ThreadData 0x2155e8100c0 is destructed 13369
I14:38:09:220|quadd_common_core|13360|AsyncProcessor.cpp:132[Stop]: 1TaskRunner@AsyncProcessor[0x2155b1a0100] stopped.
I14:38:09:220|quadd_common_core|13360|AsyncProcessor.cpp:110[Stop]: 2CommsProcessor@AsyncProcessor[0x2155b1a0180] is stopping.
I14:38:09:220|quadd_common_core|13360|AsyncProcessor.cpp:124[Stop]: 2CommsProcessor@AsyncProcessor[0x2155b1a0180] is waiting for work threads finish.
I14:38:09:220|quadd_common_core|13370|AsyncProcessor.cpp:78[operator()]: Thread[13370] stopped servicing 2CommsProcessor@AsyncProcessor.
I14:38:09:220|Injection|13370|EventHandlerImpl.cpp:53[~ThreadData]: ThreadData 0x2155b8718c0 is destructed 13370
I14:38:09:220|quadd_common_core|13360|AsyncProcessor.cpp:132[Stop]: 2CommsProcessor@AsyncProcessor[0x2155b1a0180] stopped.
I14:38:09:220|quadd_common_core|13360|AsyncProcessorHolder.h:78[Terminate]: AsyncProcessorHolder[0x2155b0b3b20]: Destroying
I14:38:09:220|Injection|13360|Manager.cpp:346[Deinitialize]: Deinitialize::end.