setup1:
gpu: 1080 Ti
ubuntu 16.04
cuda: 10.0.130
driver: 430.50
cupti event collection Mode: CUPTI_EVENT_COLLECTION_MODE_KERNEL
I want to read GPU performance counters to get cache miss rate. Before I do anything, I tried to compile and run the sample programs under /usr/local/cuda-10.0/extras/CUPTI/sample/event_sampling folder, but the program always stops at “cuptiEventGroupEnable” function and gives “CUPTI_ERROR_INVALID_PARAMETER”.
Setup2:
gpu: DGX station V100
ubuntu 18
cuda: 10.0.130
driver: 418.67
cupti event collection Mode: CUPTI_EVENT_COLLECTION_MODE_CONTINOUS
I also tried the sample programs under /usr/local/cuda-10.0/extras/CUPTI/sample/event_sampling folder. Same error is reported.
I also tried some other examples, and all these examples stop at “cuptiEventGroupEnable”. I also printed out eventGroup parameter, and I find it is not NULL. Do I miss something?
Here is how I run the program on 1080Ti and the corresponding error message:
$./event_sampling 0 active_cycles
Usage: ./event_sampling [device_num] [event_name]
CUDA Device Number: 0
CUDA Device Name: GeForce GTX 1080 Ti
Creating sampling thread
eventGroup: 0x9c003140
event_sampling.cu:103:Error CUPTI_ERROR_INVALID_PARAMETER for CUPTI API function ‘cuptiEventGroupEnable’.
Segmentation fault (core dumped)
Here is how I run the program on DGX Station with V100 and the corresponding error message:
Usage: ./event_sampling [device_num] [event_name]
CUDA Device Number: 0
CUDA Device Name: Tesla V100-DGXS-32GB
Creating sampling thread
eventGroup: 0x7c0039e0
event_sampling.cu:102:Error CUPTI_ERROR_INVALID_PARAMETER for CUPTI API function ‘cuptiEventGroupEnable’.
This is part of the code:
void *
sampling_func(void *arg)
{
CUptiResult cuptiErr;
CUpti_EventGroup eventGroup;
CUpti_EventID eventId;
size_t bytesRead, valueSize;
uint32_t numInstances = 0, j = 0;
uint64_t *eventValues = NULL, eventVal = 0;
uint32_t profile_all = 1;
cuptiErr = cuptiSetEventCollectionMode(context,
CUPTI_EVENT_COLLECTION_MODE_KERNEL);
//CUPTI_EVENT_COLLECTION_MODE_CONTINUOUS);
CHECK_CUPTI_ERROR(cuptiErr, "cuptiSetEventCollectionMode");
cuptiErr = cuptiEventGroupCreate(context, &eventGroup, 0);
CHECK_CUPTI_ERROR(cuptiErr, "cuptiEventGroupCreate");
cuptiErr = cuptiEventGetIdFromName(device, eventName, &eventId);
CHECK_CUPTI_ERROR(cuptiErr, "cuptiEventGetIdFromName");
cuptiErr = cuptiEventGroupAddEvent(eventGroup, eventId);
CHECK_CUPTI_ERROR(cuptiErr, "cuptiEventGroupAddEvent");
cuptiErr = cuptiEventGroupSetAttribute(eventGroup,
CUPTI_EVENT_GROUP_ATTR_PROFILE_ALL_DOMAIN_INSTANCES,
sizeof(profile_all), &profile_all);
CHECK_CUPTI_ERROR(cuptiErr, "cuptiEventGroupSetAttribute");
cuptiErr = cuptiEventGroupEnable(eventGroup);
fprintf(stderr, "eventGroup: 0x%x\n",eventGroup);
CHECK_CUPTI_ERROR(cuptiErr, "cuptiEventGroupEnable");