program using CUPTI failed when running under cuda-gdb

Hello,

I’m having this weird issue: I have a program that uses CUPTI callbackAPI to monitor the kernels in the program. It runs well when it’s directly launched; but when I put it under cuda-gdb and run, it failed with the following error:

error: function cuptiSubscribe(&subscriber, (CUpti_CallbackFunc)getStackTraceCallback, NULL) failed with error CUPTI_ERROR_NOT_INITIALIZED

I’ve tried CUDA 8.0, 9.0, but failed with the same error. Any thoughts?

Thanks a lot!

Add:
I’ve tried to use an empty callback function and still got the same error, so it’s nothing to do with my callback function.

Update:
Here’s the version information:
cat /proc/driver/nvidia/version
NVRM version: NVIDIA UNIX x86_64 Kernel Module 384.81 Sat Sep 2 02:43:11 PDT 2017
GCC version: gcc version 4.8.5 20150623 (Red Hat 4.8.5-16) (GCC)

nvcc --version
nvcc: NVIDIA ® Cuda compiler driver
Copyright © 2005-2016 NVIDIA Corporation
Built on Tue_Jan_10_13:22:03_CST_2017
Cuda compilation tools, release 8.0, V8.0.61

Any ideas? I’m still stuck in this problem.

Update#3:
I’ve tried all examples in CUPTI/samples and concluded that programs that use callbackAPI and activityAPI will fail under cuda-gdb. (They are all well-behaved without cuda-gdb) But the fail reason differs:
If I have calls from activityAPI, then once run it under cuda-gdb, it’ll hang for a minute then exit with error:
The CUDA driver has hit an internal error.
Error code: 0x100ff00000001c
Further execution or debugging is unreliable.
Please ensure that your temporary directory is mounted with write and exec permissions.

If I have calls from callbackAPI, then it’ll fail out much sooner with error:
error: function cuptiSubscribe(&subscriber, (CUpti_CallbackFunc)getStackTraceCallback, NULL) failed with error CUPTI_ERROR_NOT_INITIALIZED

Hi,hzhang86

we don’t have support for running two cuda tools together. That is we can have only one of CUPTI or nvprof or cuda-gdb or cuda-memcheck applied to an application at one time.

Hi, Veraj

I appreciate the information.
Besides, I found there’s also a Debugger API but very limited documentation and no samples. Is that what you use to build cuda-gdb? Basically, you can’t have calls from CUPTI and Debugger API in the same program right?

Thanks

Yes. That’s right