CUPTI cleanup error

Developer Tools CUPTI – CUDA Profiler Tools Interface
1

Hey guys i was trying to profile my cuda application and use cupti as profiling tool

so basically im trying to make the cupti profiling application silent error without interrupting the main thread

so i created cleanup function that detach the cupti profiling if any small error happens in cupti profiling app

void cleanup(char *function){ 
 fprintf(stderr, “started the sync\n”); 
 cudaDeviceSynchronize(); fprintf(stderr, “completed the sync\n”);                           cuptiActivityFlushAll(CUPTI_ACTIVITY_FLAG_FLUSH_FORCED);

 fprintf(stderr, “completed the flushing\n”); 
 cuptiFinalize(); 
 if(arena){  
  arena_destroy(arena);  
  arena = NULL; 
 } 
 if(k_arena){ 
   arena_destroy(k_arena); 
   k_arena = NULL; 
 }
}

static void CUPTIAPI bufferCompleted(CUcontext context, uint32_t streamId, uint8_t *buffer, size_t size, size_t validSize){
        int ret;
        //struct timespec start, end;
 
        //int kernel_count = 0;
  
        int is_kernel_recorded = 0;
        CUpti_Activity *record = NULL;
        CUptiResult res = CUPTI_SUCCESS;
   
        //clock_gettime(CLOCK_MONOTONIC, &start);

        if(validSize > 0){
                while((res = cuptiActivityGetNextRecord(buffer,validSize,&record)) == CUPTI_SUCCESS){
                         if(record->kind == CUPTI_ACTIVITY_KIND_CONCURRENT_KERNEL){
                                        CUpti_ActivityKernel9 *kernel = (CUpti_ActivityKernel9 *)record;
                                        record_kernel_fn(&tasks,kernel);
                                        is_kernel_recorded = 1;
                                }
}
}

here the synchronization is happening correctly but flushing is not happening, its looking like main thread cuda process stuck at the time

Example usage of cleanup

        *buffer = (uint8_t *)malloc(BUF_REQ_SZE);

        //fprintf(stdout, "req address %p", (void *)buffer);
        if(!buffer){ 
                perror("malloc");
                cleanup("malloc");
        }     

int InitializeInjection(void){
        atexit(cleanup);                                                               CUPTI_CHECK(cuptiActivityRegisterCallbacks(bufferRequest,bufferCompleted));
        CUPTI_CHECK(cuptiActivityEnable(CUPTI_ACTIVITY_KIND_DEVICE));
        CUPTI_CHECK(cuptiActivityEnable(CUPTI_ACTIVITY_KIND_CONCURRENT_KERNEL));
        CUPTI_CHECK(cuptiActivityEnable(CUPTI_ACTIVITY_KIND_MEMORY2));
        return 1;
}

do you guys know why its not cleaning up but at_exit() it is completing i think

is there any thing i missed to include or is there any other approach to detach cupti from running process ?

here i tried without using cuptiFinalizer() instead used cuptiDisableActivity(), even though its not working
also tried removing cudaSynchronize(), not working

basically how to do on fly cupti detach without synchronizing or affecting the current running cuda process?

2

Thanks for sharing these details.

As a general rule, cuptiActivityFlushAll() should not be called from most CUPTI callbacks. If it must be invoked from a callback, it is only safe to do so from CUDA API trace callbacks, specifically those within the CUPTI_CB_DOMAIN_RUNTIME_API or CUPTI_CB_DOMAIN_DRIVER_API domains of CUpti_CallbackDomain. It must not be called from activity buffer requested/completed callbacks, nor from callbacks in any other domain of CUpti_CallbackDomain. Invoking cuptiActivityFlushAll() from these unsupported domains can result in a hang.

For your cleanup path, we recommend setting a flag within the buffer callback to record the need for cleanup, and then immediately returning. The actual flush and finalization should be deferred and executed later from a dedicated application/control thread, or at process exit.