Deepstream can't debug with cuda-gdb successfully

Hello there,

Recently, I try to work with deepstream-test1 and can debug like nvdsinfer_context_impl_capi.cpp file with normal gdb debugger.
But it can’t debug with cuda-gdb for deepstream_test1_app.c when step into gst_element_set_state (pipeline, GST_STATE_PLAYING);
Can you help me find why can’t debug deepstream app with cuda-gdb?
Thank you.

Following is partial cuda-gdb infomation :

NVIDIA (R) CUDA Debugger

11.4 release Portions Copyright (C) 2007-2021 NVIDIA Corporation

GNU gdb (GDB) 10.1

Copyright (C) 2020 Free Software Foundation, Inc.

License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details.

This GDB was configured as "aarch64-elf-linux-gnu".

Type "show configuration" for configuration details.

For bug reporting instructions, please see:

<https://www.gnu.org/software/gdb/bugs/>.

Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>.
Type "apropos word" to search for commands related to "word".

[Thread debugging using libthread_db enabled]

Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".

[Detaching after fork from child process 1697669]

[Detaching after fork from child process 1697676]

[New Thread 0xffffec0ed900 (LWP 1697697)]

[Detaching after vfork from child process 1697698]

[Detaching after vfork from child process 1697701]

Added elements to bin

Using file: /opt/nvidia/deepstream/deepstream-6.3/samples/streams/sample_720p.h264

[New Thread 0xffffe3c14900 (LWP 1697708)]
GDB unhandled notify: library-unloaded: {"id":"/usr/lib/aarch64-linux-gnu/libv4l/plugins/nv/libv4l2_nvargus.so","target-name":"/usr/lib/aarch64-linux-gnu/libv4l/plugins/nv/libv4l2_nvargus.so","host-name":"/usr/lib/aarch64-linux-gnu/libv4l/plugins/nv/libv4l2_nvargus.so","thread-group":"i1"}

GDB unhandled notify: library-unloaded: {"id":"/usr/lib/aarch64-linux-gnu/tegra/libnvargus_socketclient.so","target-name":"/usr/lib/aarch64-linux-gnu/tegra/libnvargus_socketclient.so","host-name":"/usr/lib/aarch64-linux-gnu/tegra/libnvargus_socketclient.so","thread-group":"i1"}

GDB unhandled notify: library-unloaded: {"id":"/lib/aarch64-linux-gnu/libgtk-3.so.0","target-name":"/lib/aarch64-linux-gnu/libgtk-3.so.0","host-name":"/lib/aarch64-linux-gnu/libgtk-3.so.0","thread-group":"i1"}

GDB unhandled notify: library-unloaded: {"id":"/lib/aarch64-linux-gnu/libgdk-3.so.0","target-name":"/lib/aarch64-linux-gnu/libgdk-3.so.0","host-name":"/lib/aarch64-linux-gnu/libgdk-3.so.0","thread-group":"i1"}

GDB unhandled notify: library-unloaded: {"id":"/lib/aarch64-linux-gnu/libXi.so.6","target-name":"/lib/aarch64-linux-gnu/libXi.so.6","host-name":"/lib/aarch64-linux-gnu/libXi.so.6","thread-group":"i1"}

GDB unhandled notify: library-unloaded: {"id":"/lib/aarch64-linux-gnu/libXfixes.so.3","target-name":"/lib/aarch64-linux-gnu/libXfixes.so.3","host-name":"/lib/aarch64-linux-gnu/libXfixes.so.3","thread-group":"i1"}
GDB unhandled notify: library-unloaded: {"id":"/lib/aarch64-linux-gnu/libcairo-gobject.so.2","target-name":"/lib/aarch64-linux-gnu/libcairo-gobject.so.2","host-name":"/lib/aarch64-linux-gnu/libcairo-gobject.so.2","thread-group":"i1"}

GDB unhandled notify: library-unloaded: {"id":"/lib/aarch64-linux-gnu/libgdk_pixbuf-2.0.so.0","target-name":"/lib/aarch64-linux-gnu/libgdk_pixbuf-2.0.so.0","host-name":"/lib/aarch64-linux-gnu/libgdk_pixbuf-2.0.so.0","thread-group":"i1"}

GDB unhandled notify: library-unloaded: {"id":"/lib/aarch64-linux-gnu/libatk-1.0.so.0","target-name":"/lib/aarch64-linux-gnu/libatk-1.0.so.0","host-name":"/lib/aarch64-linux-gnu/libatk-1.0.so.0","thread-group":"i1"}

GDB unhandled notify: library-unloaded: {"id":"/lib/aarch64-linux-gnu/libatk-bridge-2.0.so.0","target-name":"/lib/aarch64-linux-gnu/libatk-bridge-2.0.so.0","host-name":"/lib/aarch64-linux-gnu/libatk-bridge-2.0.so.0","thread-group":"i1"}

GDB unhandled notify: library-unloaded: {"id":"/lib/aarch64-linux-gnu/libepoxy.so.0","target-name":"/lib/aarch64-linux-gnu/libepoxy.so.0","host-name":"/lib/aarch64-linux-gnu/libepoxy.so.0","thread-group":"i1"}

GDB unhandled notify: library-unloaded: {"id":"/lib/aarch64-linux-gnu/libXinerama.so.1","target-name":"/lib/aarch64-linux-gnu/libXinerama.so.1","host-name":"/lib/aarch64-linux-gnu/libXinerama.so.1","thread-group":"i1"}

GDB unhandled notify: library-unloaded: {"id":"/lib/aarch64-linux-gnu/libXrandr.so.2","target-name":"/lib/aarch64-linux-gnu/libXrandr.so.2","host-name":"/lib/aarch64-linux-gnu/libXrandr.so.2","thread-group":"i1"}

Hi @JasonWu,
Based on the cuda-gdb output you are using the 11.4 version, which is quite old. Any chance you can try the same scenario with a newer CUDA version? CUDA Toolkit 12.4 Update 1 Downloads | NVIDIA Developer

Thank you for your reply.

I will try newer CUDA version and if any thing I found I will let you know as soon as possible.

By the way, above issue is developing on Jetson Orin NX and Deepstream6.3

Thanks.

Hi AKravets,

After use Deepstream 7.0 and GTX4090, I can stop breakpoint at the device code of nvdsinfer_conversion.cu.
But cuda-gdb take over 10 minutes to stop breakpoint at device code, and normal gdb can stop at host code of nvdsinfer_conversion.cu in couple of seconds.
Is it possible to fix this kind of issue?

Thank you.

Following message is shown by cuda-gdb :

NVIDIA (R) CUDA Debugger
CUDA Toolkit 12.2 release
Portions Copyright (C) 2007-2023 NVIDIA Corporation
GNU gdb (GDB) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word".
No source file named /opt/nvidia/deepstream/deepstream-7.0/sources/libs/nvdsinfer/nvdsinfer_context_impl.cpp.
2
No source file named /opt/nvidia/deepstream/deepstream-7.0/sources/libs/nvdsinfer/nvdsinfer_conversion.cu.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff477f000 (LWP 2744695)]
[New Thread 0x7ffff349c000 (LWP 2744696)]
[Detaching after fork from child process 2744697]
[New Thread 0x7fffea02c000 (LWP 2744715)]
Added elements to bin
Using file: /opt/nvidia/deepstream/deepstream-7.0/samples/streams/sample_720p.h264
[New Thread 0x7fffd19ff000 (LWP 2744719)]
[New Thread 0x7fff8abff000 (LWP 2744721)]
GDB unhandled notify: library-unloaded: {"id":"/lib/x86_64-linux-gnu/libnvinfer_builder_resource.so.8.6.1","target-name":"/lib/x86_64-linux-gnu/libnvinfer_builder_resource.so.8.6.1","host-name":"/lib/x86_64-linux-gnu/libnvinfer_builder_resource.so.8.6.1","thread-group":"i1"}
0:07:09.870489867 2744685 0x5555ba562e50 INFO                 nvinfer gstnvinfer.cpp:682:gst_nvinfer_logger:<primary-nvinference-engine> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:2095> [UID = 1]: deserialized trt engine from :/opt/nvidia/deepstream/deepstream-7.0/samples/models/Primary_Detector/resnet18_trafficcamnet.etlt_b1_gpu0_int8.engine
0:09:16.313350650 2744685 0x5555ba562e50 INFO                 nvinfer gstnvinfer.cpp:682:gst_nvinfer_logger:<primary-nvinference-engine> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2198> [UID = 1]: Use deserialized engine model: /opt/nvidia/deepstream/deepstream-7.0/samples/models/Primary_Detector/resnet18_trafficcamnet.etlt_b1_gpu0_int8.engine
[New Thread 0x7fff88fff000 (LWP 2751964)]
[New Thread 0x7fff69fff000 (LWP 2751965)]
[New Thread 0x7fff697fe000 (LWP 2751966)]
0:09:16.544174634 2744685 0x5555ba562e50 INFO                 nvinfer gstnvinfer_impl.cpp:343:notifyLoadModelStatus:<primary-nvinference-engine> [UID 1]: Load new model:/opt/nvidia/deepstream/deepstream-7.0/sources/apps/sample_apps/deepstream-test1/dstest1_pgie_config.txt sucessfully
[New Thread 0x7fff68ffd000 (LWP 2751967)]
[New Thread 0x7fff5bfff000 (LWP 2751968)]
INFO: [Implicit Engine Info]: layers num: 3
   INPUT  kFLOAT input_1         3x544x960       
   OUTPUT kFLOAT output_bbox/BiasAdd 16x34x60        
Running...
[New Thread 0x7fff5b7fe000 (LWP 2751969)]
[New Thread 0x7fff5affd000 (LWP 2751970)]
[New Thread 0x7fff5a7fc000 (LWP 2751987)]
[New Thread 0x7fff59ffb000 (LWP 2751988)]
[Switching to Thread 0x7fff69fff000 (LWP 2751965)]

Thread 8 "deepstream-test" hit Breakpoint 1, nvdsinfer::InferPreprocessor::transform (this=0x55555f95d530, batchInput=..., devBuf=0x7fff1d000000, mainStream=..., waitingEvent=0x555561605f20) at nvdsinfer_context_impl.cpp:499
499	        if (convertFcn) {

Thread 8 "deepstream-test" hit Breakpoint 2, NvDsInferConvert_C3ToP3Float (outBuffer=0x7fff1d000000, inBuffer=0x7fff1c200000 "", width=960, height=544, pitch=3072, scaleFactor=0.00392156886, meanDataBuffer=0x0, stream=0x55555b7a7e30) at /opt/nvidia/deepstream/deepstream-7.0/sources/libs/nvdsinfer/nvdsinfer_conversion.cu:302
302	    if (meanDataBuffer == NULL)
[New Thread 0x7fff597fa000 (LWP 2801654)]

Thread 8 "deepstream-test" hit Breakpoint 1, nvdsinfer::InferPreprocessor::transform (this=0x55555f95d530, batchInput=..., devBuf=0x7fff1d000000, mainStream=..., waitingEvent=0x555561605f20) at nvdsinfer_context_impl.cpp:499
499	        if (convertFcn) {

Thread 8 "deepstream-test" hit Breakpoint 2, NvDsInferConvert_C3ToP3Float (outBuffer=0x7fff1d000000, inBuffer=0x7fff1d600000 "", width=960, height=544, pitch=3072, scaleFactor=0.00392156886, meanDataBuffer=0x0 <NvDsInferConvert_FtFTensorKernel(float*, float*, unsigned int, unsigned int, unsigned int, float)>, stream=0x55555b7a7e30) at /opt/nvidia/deepstream/deepstream-7.0/sources/libs/nvdsinfer/nvdsinfer_conversion.cu:302
302	    if (meanDataBuffer == NULL)
Frame Number = 0 Number of objects = 15 Vehicle Count = 10 Person Count = 5

Thread 7 "deepstream-test" hit Breakpoint 4, NvDsInferConvert_CxToP3FloatKernel<<<(30,17,1),(32,32,1)>>> (outBuffer=0x7fff1d000000, inBuffer=0x7fff1d600000 "T\207\272T\207\272T\207\272T\207\272S\211\273S\211\273S\211\273S\211\273S\211\273T\212\274T\212\274T\212\274T\212\274T\212\274T\212\274T\212\274T\212\274T\212\274T\212\274T\212\274T\212\274T\212\274T\212\274T\212\274T\212\274V\213\275V\213\275V\213\275V\213\275V\213\275V\213\275V\213\275V\213\275V\213\275V\213\275V\213\275V\213\275W\214\276W\214\276W\214\276W\214\276W\214\276W\214\276W\214\276W\214\276W\214\276W\214\276W\214\276W\214\276W\214\276W\214\276W\214\276W\214\276W\214\276W\214\276W\214\276W\214\276W\214\276W\214\276W\214\276W\214\276X\215\300X\215\300X\215\300X\215\300X\215\300X\215", width=960, height=544, pitch=3072, inputPixelSize=3, scaleFactor=0.00392156886) at nvdsinfer_conversion.cu:35
35	            outBuffer[width * height * k + row * width + col] =
cuda block (0, 0, 0) thread (0, 0, 0)

To help us debug the problem, could you collect and share additional logs with us?

  • Add NVLOG_CONFIG_FILE variable pointing the nvlog.config file (attached). E.g.: NVLOG_CONFIG_FILE=${HOME}/nvlog.config
    nvlog.config (539 Bytes)
  • Run the debugging session.
  • You should see the /tmp/debugger.log file created - could you share it with us?

Hi AKravets,

I use cuda-gdb to set a breakpoint at /opt/nvidia/deepstream/deepstream-7.0/sources/libs/nvdsinfer/nvdsinfer_conversion.cu:501, and debugger can stop at line 501.
After that I quit the debugger, and upload debugger.log for you.
debugger.zip (10.0 MB)

Thank you for your support.

Hi @JasonWu
Thank you for the log! Based on these logs it looks like you might be hitting an issue, which was fixed after the 12.2 release (the one, you collected the logs with). Any chance you can try a more recent version (e.g. 12.4?)

Hi AKravets,

Here is the log with cuda-gdb 12.5.
debugger_cuda-gdb12.5.zip (3.6 MB)

Thank you

Hi @JasonWu
Thank you for the new log, we are investigating the issue. Could you also confirm that you still see the original problem (it takes ~ significant amount of time to hit the breakpoint in the device code)?

Hi AKravets,

Yes, it still takes very long time to stop at breakpoint comparing to normal gdb.

Thank you.

Hello,

Can you please show us the cuda-gdb commands used? I assume the -ex was executed here. Do you see any difference in execution speed when issuing these commands directly to the cuda-gdb prompt versus passing them in via -ex?

Can you also show us:

  1. The output from nvidia-smi
  2. Does execution speed get better when setting:
export CUDA_MODULE_LOADING=EAGER
  1. Can you set the following, continue to the breakpoint that takes a long time to reach, issue a ctrl-c and provide the output:
set debug cuda trace event

This will generate a lot of debug output messages.
4. With the same method from request 3, can you also show the output of:

maint print cuda_stats

Hi agontarek,

Here is the commands which I typed sequentially :

cuda-gdb ./deepstream-test1-app

b /opt/nvidia/deepstream/deepstream-7.0/sources/libs/nvdsinfer/nvdsinfer_context_impl.cpp:494

r /opt/nvidia/deepstream/deepstream-7.0/samples/streams/sample_720p.h264

Here is the result of nvidia-smi.

After export CUDA_MODULE_LOADING=EAGER has no effect on execution speed , as slow as before.

Following information comes from r /opt/nvidia/deepstream/deepstream-7.0/samples/streams/sample_720p.h264 for few seconds, then I hit ctrl-c and “set debug cuda trace event” .

Similar process for :

maint print cuda_stats

(cuda-gdb) set debug cuda trace event
(cuda-gdb) c
Continuing.
[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=55555f5f1e30
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.55555f5f1e30.o.eL2eWL 16872

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=555560e0df00
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.555560e0df00.o.zwaf2w 680

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=55555a809ec0
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.55555a809ec0.o.RyWgCz 54120

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=55555ae7a860
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.55555ae7a860.o.77IBbI 980072

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=555560d34cf0
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.555560d34cf0.o.WoGvrP 12392

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=55555cceff40
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.55555cceff40.o.uGcMtR 7464

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=555561db2de0
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.555561db2de0.o.UOl4Ba 1898464

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=555563e10fb0
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.555563e10fb0.o.AAx7pS 919400

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=55555c997b20
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.55555c997b20.o.YFqoTc 12136

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=55555cce3060
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.55555cce3060.o.WXRA6e 680

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=555560fe1dc0
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.555560fe1dc0.o.2ONg0Y 4264

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=555561ace5f0
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.555561ace5f0.o.2sVSXE 679144

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=55555c440a00
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.55555c440a00.o.RxJDaJ 5160

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=555561ce5150
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.555561ce5150.o.4IZSaN 680

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=555563e562e0
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.555563e562e0.o.T2F2PJ 1890464

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=555561d7be10
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.555561d7be10.o.YxUdGT 1218536

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=5555612891d0
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.5555612891d0.o.NgKGKe 677992

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=555559de9560
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.555559de9560.o.JZuTJN 680

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=5555595b1a20
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.5555595b1a20.o.QrkiFz 680

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=55555f5723a0
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.55555f5723a0.o.eN0zyN 907360

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=55555eee2720
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.55555eee2720.o.oLPxhc 680

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=55555b5e12d0
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.55555b5e12d0.o.Aftx75 680

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=5555665ae7d0
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.5555665ae7d0.o.wg7dS9 2728

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=5555614dd620
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.5555614dd620.o.kVIuxk 680

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=5555615a5860
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.5555615a5860.o.qlDMQw 680

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=5555669774c0
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.5555669774c0.o.1hFKTm 1899936

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=55556698b4f0
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.55556698b4f0.o.Oa3Ie2 23784

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=55555b5df870
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.55555b5df870.o.WCEfYj 680

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=55556720d0c0
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.55556720d0c0.o.uKIuOg 28264

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=55555b3ab1e0
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.55555b3ab1e0.o.cdK7cP 687392

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=55555af26420
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.55555af26420.o.9qcblu 82272

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=5555617dd130
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.5555617dd130.o.kmBG3d 680

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=55555a9013f0
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.55555a9013f0.o.BheRvG 680

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=5555618c0bb0
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.5555618c0bb0.o.0T1Q2g 8872

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=555561667e90
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.555561667e90.o.5BGKpt 2472

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=5555639e1200
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.5555639e1200.o.2Kv1tF 3744

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=5555618e2aa0
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.5555618e2aa0.o.bHjkvx 3678944

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=5555681df3d0
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.5555681df3d0.o.TpydUw 8811808

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=55555e922fa0
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.55555e922fa0.o.rDwhJn 5032

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=55555f85c9e0
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.55555f85c9e0.o.OXypq8 1048096

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=55555d7a6c50
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.55555d7a6c50.o.nljdwm 8224

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=555561662560
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.555561662560.o.rLqdAN 8552

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=55555dbe83b0
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.55555dbe83b0.o.6Otqum 52000

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=555558793610
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.555558793610.o.wID1xI 680

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=5555611512f0
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.5555611512f0.o.ur281D 13928

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=55555c566540
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.55555c566540.o.e2XXGe 680

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=555564e004f0
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.555564e004f0.o.630R1y 47720

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=55555dc02bf0
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.55555dc02bf0.o.hWTXLE 16872

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=5555609c5290
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.5555609c5290.o.BSZXaY 13032

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=55556fa750c0
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.55556fa750c0.o.YEOfIr 144680

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=55556fb18e90
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.55556fb18e90.o.V7QrN3 1938400

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=55556fa0ba60
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.55556fa0ba60.o.oejeGx 680

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=55555df21ca0
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.55555df21ca0.o.NYVgyP 680

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=55556fbe94a0
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.55556fbe94a0.o.PkEB5K 53993384

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=555572a647e0
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.555572a647e0.o.Ouxd4L 680

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=555572ab8020
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.555572ab8020.o.cdR4DA 680

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=555572ecb9f0
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.555572ecb9f0.o.akgdI2 41960

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=555573699f90
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.555573699f90.o.uo6ACW 35232

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=555571de7f10
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.555571de7f10.o.AMZaMc 680

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=5555724890a0
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.5555724890a0.o.xTOFap 306400

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=555571d9cb90
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.555571d9cb90.o.SI1oFN 41640

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=55556f7e8a20
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.55556f7e8a20.o.jiYYkh 680

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=55557391e8c0
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.55557391e8c0.o.AZGQAc 1893024

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=555570cf5630
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.555570cf5630.o.5uSWX2 7400

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=55556e6c1e70
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.55556e6c1e70.o.j2v2FZ 452256

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=555573601460
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.555573601460.o.LRppVP 225896

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=55557388c950
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.55557388c950.o.23lg87 680

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=55556080d200
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.55556080d200.o.1IrMiH 817568

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=55556f67e730
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.55556f67e730.o.u6w0OZ 680

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=555573dc2b90
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.555573dc2b90.o.iY2VRN 7210216

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=5555728f0d00
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.5555728f0d00.o.UUGJaQ 11560

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=5555716f3d50
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.5555716f3d50.o.rTBgvF 1900768

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=5555713a41d0
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.5555713a41d0.o.vhjm3o 680

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=55556f868d30
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.55556f868d30.o.dJc3Tt 20648

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=555578db7f50
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.555578db7f50.o.qirFPp 16872

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=555578db8160
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.555578db8160.o.dO2BII 680

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=55557602cdc0
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.55557602cdc0.o.BKAMgL 255720

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=555572c5e9f0
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.555572c5e9f0.o.Si5MMP 1345768

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=555579069560
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.555579069560.o.NtkwLj 22120

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=555572dfc170
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.555572dfc170.o.R53wmk 16872

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=5555778c3210
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.5555778c3210.o.wwZevx 680

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=555570e8f0f0
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.555570e8f0f0.o.xrzMPs 19944

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=5555790a4010
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.5555790a4010.o.aFAAdO 16872

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=555578fdbdb0
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.555578fdbdb0.o.kLcsIk 680

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=555578fb6420
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.555578fb6420.o.BJ7Ex9 16872

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=555578e6e760
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.555578e6e760.o.TzxUPu 1094120

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=5555736aad00
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.5555736aad00.o.1YQHtK 16872

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=555578f92e80
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.555578f92e80.o.mXlzTN 28136

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=555578e2e390
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.555578e2e390.o.Hya8Cc 475936

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=5555760e2c40
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.5555760e2c40.o.ajDbZd 1940960

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=55557a572b80
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.55557a572b80.o.7N20vU 680

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=55557a2a90a0
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.55557a2a90a0.o.hLfRyD 680

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=555578df8b70
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.555578df8b70.o.Eq60D4 680

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=55557778d480
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.55557778d480.o.1toB4b 680

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=555577abd9d0
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.555577abd9d0.o.NcWRTF 6944

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=555577c9b6a0
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.555577c9b6a0.o.Hwjoya 3744

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=555577e51160
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.555577e51160.o.Thd9qb 16872

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=55557a2a8660
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.55557a2a8660.o.dgZ2qH 1577832

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=55557bb27d20
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.55557bb27d20.o.26O6Ca 680

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=55557a317770
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.55557a317770.o.C6wQoZ 251624

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=55557a515560
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.55557a515560.o.bW7Zsn 6120

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=55557b2117f0
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.55557b2117f0.o.So80WO 680

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=55557a569510
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.55557a569510.o.i5MyhT 16872

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=55557b580740
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.55557b580740.o.B6s7Rz 3744

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=55557b3282c0
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.55557b3282c0.o.qeGpKf 680

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=55557b2c4cf0
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.55557b2c4cf0.o.Ed7bHT 680

[CUDAGDB-EVT] CUDBG_EVENT_ELF_IMAGE_LOADED dev_id=0 context=555555694730 module=55557b2c5020
[CUDAGDB-EVT]     -> ELF image loaded /tmp/cuda-dbg/4104350/session2/elf.555555694730.55557b2c5020.o.oZZfq2 680
(cuda-gdb) maint print cuda_stats                                                                                                                                                                        
API name                        Number of calls Average call time(usec) Min call time(usec) Max call time(usec) Total call time(usec) 
suspendDevice                              9280                    2870                  19               49848              26638063 
resumeDevice                               9278                    1659                  17               45200              15394316 
getDeviceType                                 2                      11                  10                  13                    23 
getSmType                                     2                      12                  11                  13                    24 
getNumDevices                                 2                      46                  14                  79                    93 
getNumSMs                                     2                      47                  13                  82                    95 
getNumWarps                                   2                      13                  13                  13                    26 
getNumLanes                                   2                      11                  10                  13                    23 
getNumRegisters                               2                      11                  10                  13                    23 
acknowledgeSyncEvents                      9251                     116                  21                 862               1080931 
setKernelLaunchNotificationMode               2                     105                  92                 119                   211 
getNextEvent                              27810                      64                   8                3015               1799478 
getElfImageByHandle                        5789                     196                  17               79864               1138142 
getNumPredicates                              2                      11                  10                  13                    23 
getDeviceName                                 2                      12                  11                  14                    25 
readDeviceExceptionState                   9258                     123                  41                 782               1139909 
getNumUniformRegisters                        2                      11                  10                  12                    22 
getNumUniformPredicates                       2                      10                   9                  12                    21 
getErrorStringEx                          18581                     113                  15                 564               2101085 
getDeviceInfoSizes                            2                      17                  10                  24                    34 
getDeviceInfo                              9280                     119                  15                 221               1107764 
Total time spend in CUDBG API is 50.400331 sec

Thank you.